IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks

1 INTRODUCTION . Proximal Policy Optimization ( Schulman et al. , 2017 ) is one of the most sample-efficient on-policy algorithms . However , it relies on a synchronous architecture for collecting experiences , which is closely tied to its trust region optimization objective . Other architectures such as IMPALA can achieve much higher throughputs due to the asynchronous collection of samples from workers . Yet , IMPALA suffers from reduced sample efficiency since it can not safely take multiple SGD steps per batch as PPO can . The new agent , Importance Weighted Asynchronous Architectures with Clipped Target Networks ( IMPACT ) , mitigates this inherent mismatch . Not only is the algorithm highly sample efficient , it can learn quickly , training 30 percent faster than IMPALA . At the same time , we propose a novel method to stabilize agents in distributed asynchronous setups and , through our ablation studies , show how the agent can learn in both a time and sample efficient manner . In our paper , we show that the algorithm IMPACT realizes greater gains by striking the balance between high sample throughput and sample efficiency . In our experiments , we demonstrate in the experiments that IMPACT exceeds state-of-the-art agents in training time ( with same hardware ) while maintaining similar sample efficiency with PPO ’ s . The contributions of this paper are as follows : 1 . We show that when collecting experiences asynchronously , introducing a target network allows for a stabilized surrogate objective and multiple SGD steps per batch ( Section 3.1 ) . 2 . We show that using a circular buffer for storing asynchronously collected experiences allows for smooth trade-off between real-time performance and sample efficiency ( Section 3.2 ) . 3 . We show that IMPACT , when evaluated using identical hardware and neural network models , improves both in real-time and timestep efficiency over both synchronous PPO and IMPALA ( Section 4 ) . 2 BACKGROUND . Reinforcement Learning assumes a Markov Decision Process ( MDP ) setup defined by the tuple ( S , A , p , γ , r ) where S and A represent the state and action space , γ ∈ [ 0 , 1 ] is the discount factor , and p : S ×A× S → R and R : S ×A→ R are the transition dynamics and reward function that models an environment . Let π ( at|st ) : S ×A→ [ 0 , 1 ] denote a stochastic policy mapping that returns an action distribution given state st ∈ S. Rolling out policy π ( at|st ) in the environment is equivalent to sampling a trajectory τ ∼ P ( τ ) , where τ : = ( s0 , a0 , .... , aT−1 , sT , aT ) . We can compactly define state and state-action marginals of the trajectory distribution pπ ( st ) and pπ ( st , at ) induced by the policy π ( at|st ) .The goal for reinforcement learning aims to maximize the following objective : J ( θ ) = E ( st , at ) ∼pπ [ ∑T t=0 γ tR ( st , at ) ] . When θ parameterizes π ( at|st ) , the policy is updated according to the Policy Gradient Theorem ( Sutton et al. , 2000 ) : ∇θJ ( θ ) = E ( st , at ) ∼pπ ( · ) [ ∇θ log πθ ( at|st ) Âπθ ( st , at ) ] , where Âπθ ( st , at ) is an estimator of the advantage function . The advantage estimator is usually defined as the 1-step TD error , Âπθ ( st , at ) = r ( st , at ) + γV̂ ( st+1 ) − V̂ ( st ) , where V̂ ( st ) is an estimation of the value function . Policy gradients , however , suffer from high variance and large update-step sizes , oftentimes leading to sudden drops in performance . 2.1 DISTRIBUTED PPO . Per iteration , Proximal Policy Optimization ( PPO ) optimizes policy πθ from target πθold via the following objective function L ( θ ) = Epπθold [ min ( rt ( θ ) Ât , clip ( rt ( θ ) , 1− , 1 + ) Ât ) ] , where rt ( θ ) = πθ ( at|st ) πθold ( at|st ) and is the clipping hyperparameter . In addition , many PPO implementations use GAE-λ as a low bias , low variance advantage estimator for Ât ( Schulman et al. , 2015b ) . PPO ’ s surrogate objective contains the importance sampling ratio rt ( θ ) , which can potentially explode if πθold is too far from πθ . ( Han & Sung , 2017 ) . PPO ’ s surrogate loss mitigates this with the clipping function , which ensures that the agent makes reasonable steps . Alternatively , PPO can also be seen as an adaptive trust region introduced in TRPO ( Schulman et al. , 2015a ) . In Figure 1a , distributed PPO agents implement a synchronous data-gathering scheme . Before data collection , workers are updated to πold and aggregate worker batches to training batch Dtrain . The learner performs many mini-batch gradient steps on Dtrain . Once the learner is done , learner weights are broadcast to all workers , who start sampling again . 2.2 IMPORTANCE WEIGHTED ACTOR-LEARNER ARCHITECTURES . In Figure 1b , IMPALA decouples acting and learning by having the learner threads send actions , observations , and values while the master thread computes and applies the gradients from a queue of learners experience ( Espeholt et al. , 2018 ) . This maximizes GPU utilization and allows for increased sample throughput , leading to high training speeds on easier environments such as Pong . As the number of learners grows , worker policies begin to diverge from the learner policy , resulting in stale policy gradients . To correct this , the IMPALA paper utilizes V-trace to correct the distributional shift : vst = Vφ ( st ) + t+n−1∑ i=t γi−t i−1∏ j=t cj ρi ( ri+1 + γVφ ( si+1 ) − Vφ ( si ) ) where , Vφ is the value network , πθ is the policy network of the master thread , µθ′ is the policy network of the learner thread , and cj = min ( c̄ , πθ ( aj |sj ) µθ′ ( aj |sj ) ) and ρi = min ( ρ̄ , πθ ( ai|si ) µθ′ ( ai|si ) ) are clipped IS ratios . Algorithm 1 IMPACT Input : Batch size M , number of workers W , circular buffer size N , replay coefficient K , target update frequency ttarget , weight broadcast frequency tfrequency , learning rates α and β 1 : Randomly initialize network weights ( θ , w ) 2 : Initialize target network ( θ′ , w′ ) ← ( θ , w ) 3 : Create W workers and duplicate ( θ , w ) to each worker 4 : Initialize circular buffer C ( N , K ) 5 : for t = 1 , .. , T do 6 : Obtain batch B of size M traversed k times from C ( N , K ) 7 : If k = 0 , evaluate B on target θ′ , append target output to B 8 : Compute policy and value network gradients ∇θJ ( θ ) = 1 M ∑ ( i , j ) ∈B ∇θπθ ( sj |aj ) max ( πtarget ( sj |aj ) , βπworkeri ( sj |aj ) ) ÂV -GAE − η∇θKL ( πtarget , πθ ) ∇wL ( w ) = 1 M ∑ j ( Vw ( sj ) − V̂V -GAE ( sj ) ) ∇wVw ( sj ) 9 : Update policy and value network weights θ ← θ + αt∇θJ ( θ ) , w ← w − βt∇wL ( w ) 10 : If k = K , discard batch B from C ( N , K ) 11 : If t ≡ 0 ( mod ttarget ) , update target network ( θ′ , w′ ) ← ( θ , w ) 12 : If t ≡ 0 ( mod tfrequency ) , broadcast weights to workers 13 : end for Worker-i Input : Worker sample batch size S 1 : repeat 2 : Bi = ∅ 3 : for t = 1 , ... , S do 4 : Store ( st , at , rt , st+1 ) ran by θi in batch Bi 5 : end for 6 : Send Bi to C ( N , K ) 7 : If broadcasted weights exist , set θi ← θ 8 : until learner finishes 3 IMPACT ALGORITHM . Like IMPALA , IMPACT separates sampling workers from learner workers . Algorithm 1 and Figure 1c describe the main training loop and architecture of IMPACT . In the beginning , each worker copies weights from the master network . Then , each worker uses their own policy to collect trajectories and sends the data ( st , at , rt ) to the circular buffer . Simultaneously , workers also asynchronously pull policy weights from the master learner . In the meantime , the target network occasionally syncs with the master learner every ttarget iterations . The master learner then repeatedly draws experience from the circular buffer . Each sample is weighted by the importance ratio of πθπworkeri as well as clipped with target network ratio πworkeriπtarget . The target network is used to provide a stable trust region ( Figure 2 ) , allowing multiple steps per batch ( i.e. , like PPO ) even in the asynchronous setting ( i.e. , with the IMPALA architecture ) . In the next section , we describe the design of this improved objective . 3.1 MAXIMAL TARGET-WORKER CLIPPING . PPO gathers experience from previous iteration ’ s policy πθold , and the current policy trains by importance sampling off-policy experience with respect to πθ . In the asynchronous setting , worker i ’ s policy , denoted as πworkeri , generates experience for the policy network πθ . The probability that batch B comes from worker i can be parameterized as a categorical distribution i ∼ D ( α1 , ... , αn ) . We include this by adding an extra expectation to the importance-sampled policy gradient objective ( IS-PG ) ( Jie & Abbeel , 2010 ) : JIS ( θ ) = Ei∼D ( α ) [ E ( st , at ) ∼πworkeri [ πθ πworkeri Ât ] ] . Since each worker contains a different policy , the agent introduces a target network for stability ( Figure 2 ) . Off-policy agents such as DDPG and DQN update target networks with a moving average . For IMPACT , we periodically update the target network with the master network . However , training with importance weighted ratio πθπtarget can lead to numerical instability , as shown in Figure 3 . To prevent this , we clip the importance sampling ratio from worker policy , πworkeri , to target policy , πtarget : JAIS ( θ ) = Ei∼D ( α ) [ E ( st , at ) ∼πworkeri [ min ( πworkeri πtarget , ρ ) πθ πworkeri Ât ] ] = Ei∼D ( α ) [ E ( st , at ) ∼πworkeri [ πθ max ( πtarget , βπworkeri ) Ât ] ] , where β = 1ρ . In the experiments , we set ρ as a hyperparameter with ρ ≥ 1 and β ≤ 1 . To see why clipping is necessary , when master network ’ s action distribution changes significantly over few training iterations , worker i ’ s policy , πworkeri , samples data outside that of target policy , πtarget , leading to large likelihood ratios , πworkeri πtarget . The clipping function min ( πworkeriπtarget , ρ ) pulls back πworkeri , R3 = πθmax ( πtarget , βπworkeri ) . In ( b ) , we show the target network update frequency is robust to a range of choices . We try target network update frequency ttarget equal to the multiple ( ranging from 1/16 and 16 ) of n = N ·K , the product of the size of circular buffer and the replay times for each batch in the buffer . large IS ratios to ρ . Figure 10 in Appendix E provides additional intuition behind the target clipping objective . We show that the target network clipping is a lower bound of the IS-PG objective . For ρ > 1 , the clipped target ratio is larger and serves to augment advantage estimator Ât . This incentivizes the agent toward good actions while avoiding bad actions . Thus , higher values of ρ encourages the agent to learn faster at the cost of instability . We use GAE-λ with V-trace ( Han & Sung , 2019 ) . The V-trace GAE-λ modifies the advantage function by adding clipped importance sampling terms to the summation of TD errors : ÂV -GAE = t+n−1∑ i=t ( λγ ) i−t i−1∏ j=t cj δiV , where ci = min ( c̄ , πtarget ( aj |sj ) πworkeri ( aj |sj ) ) ( we use the convention ∏t−1 j=t cj = 1 ) and δiV is the importance sampled 1-step TD error introduced in V-trace .

Summary: Reinforcement learning (RL) training speed is broadly evaluated on two dimensions:  sample efficiency (the number of environment interactions required) and wall-clock time.  Improved wall-clock training time has been achieved through distributed actors and learners, but often at the expense of sample efficiency.  IMPACT repurposes successful concepts from deep RL - the target network, importance sampling and a replay buffer to demonstrate improvements on both axes in on three continuous environments and three games from the Atari Learning Environment.