DNA: Proximal Policy Optimization with a Dual Network Architecture
Authors: Matthew Aitchison
Penny Sweetser
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Policy Gradient
Deep Reinforcement Learning
Noise Scale
Atari
Procgen
Mujoco

Abstract: This paper explores the problem of simultaneously learning a value function and policy in deep actor-critic reinforcement learning models. We find that the common practice of learning these functions jointly is sub-optimal due to an order-of-magnitude difference in noise levels between the two tasks. Instead, we show that learning these tasks independently, but with a constrained distillation phase, significantly improves performance. Furthermore, we find that policy gradient noise levels decrease when using a lower \textit{variance} return estimate. Whereas, value learning noise level decreases with a lower \textit{bias} estimate. Together these insights inform an extension to Proximal Policy Optimization we call \textit{Dual Network Architecture} (DNA), which significantly outperforms its predecessor. DNA also exceeds the performance of the popular Rainbow DQN algorithm on four of the five environments tested, even under more difficult stochastic control settings.

DNA: Proximal Policy Optimization with a Dual Network Architecture
Matthew Aitchison The Australian National University matthew. ai aitchison@anu. edu. au
Penny Sweetser The Australian National University
Abstract
This paper explores the problem of simultaneously learning a value function and policy in deep actor-critic reinforcement learning models. We find that the common practice of learning these functions jointly is sub-optimal due to an order-of- magnitude difference in noise levels between the two tasks. Instead, we show that learning these tasks independently. but with a constrained distillation phase, significantly improves performance. Furthermore, we find that policy gradient noise levels decrease when using a lower variance return estimate. Whereas, value learning noise level decreases with lower bias estimate. Together these insights inform an extension to Proximal Policy Optimization we call Dual Network Architecture (DNA), which significantly outperforms its predecessor. DNA also exceeds the performance of the popular Rainbow DQN algorithm on four of the five environments tested, even under more difficult stochastic control settings.
Introduction
Combining deep neural networks with reinforcement learning has produced impressive results challenging problems, such as playing Chess [32], Atari games [25], and robotics tasks [31]. However, results have bifurcated between two competing approaches: Q-learning-base learning an (action conditioned) value estimate and actor-critic policy gradient (AC-PG, or just PG) approaches that learn both a policy and value estimate. PG offers many theoretical advantages over Q-lcarning, such as natural support for continuous control problems and the ability to learn stochastic policies. Until recently, PG approaches [24, 31, 11], while strong on continuous control problems. have under- pedfor-e Q-learning approaches on vision-based tasks [25, 16, 4], restricting the use of PG this domain.
approaches that require
This paper aims to close the gap between PG and Q-learning methods by showing that the common practice of jointly learning value and policy with shared features negatively affects the algorithm's performance," 2 We also demonstrate an onder-of-magnitue difference in noise levels between these two tasks and argue that this makes these tasks poorly aligned.
In light of this result, we introduce a new algorithm, based on Proximal Policy Optimization (PPO) [31], called Dual Network Architecture (DNA). We test DNA empirically Learning Environment (ALE) [6], Atari-5 [2]. Our results show a strong increase in performance on this benchmark compared to both PPO and another dual network model Phasic Policy Gradient (PPG) [11]. Our model also outperforms Rainbow DQN [16] on four of the five games tested, even under more challenging environmental settings.
subset of the Arcade
For simplicity, we refer to actor-critic policy gradient as policy gradient, even though some early policy gradient approaches, such REINFORCE [36], do not learn value function.
"PP [31], A3C [24], and Muslie [15] all use joint network when training on dision-based discrete action environments, PPG [11] uses a dual network with an auxiliary task, which we discuss in more detail in Section 2
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We summarize our contributions. First, we provide empirical results showing an order-of maggitude difference in the noise scale between the policy gradient and the value loss gradient. Second, we give evidence for the benefit of using low bias value estimates for value learning, but low variance estimates for advantage estimates. Finally, we introduce and justify our dual network constrained distillation algorithm DNA with empirical results on ALE.
2 Preliminaries and Related Work
Proximal Policy Optimization Our algorithm builds the well-established Proximal Policy Optimization (PPO) algorithm [31]. PPO is a policy gradient, reinforcement learning algorithm, with many of the advantages of its predecessor Trust Region Policy Optimization [29], while being much simpler to implement. Various aspects of PPO have been studied in recent years, such as the impact of implementation choices [12, 19], the ability to generalize [11, 27], and performance under multi-agent settings [37]. There has also been a growing understanding that the value and policy learning tasks involved in actor-critic models like PPO have important asymmetries [27, I1]. Our work builds on this by investigating how differences in value and policy noise levels can be accommodated to improve PPO's performance on video-based discrete action tasks.
TD(A) Return Estimation Policy gradient algorithms often make use of value estimate baseline [24), as well as for estimating the value of truncated trajectories [9]. To better facilitate control over the noise levels for policy and value learning,
work makes of two different return
estimations, both using TD(A) [33]. Given value estimates V(.) from the value network we define the n-step value estimate for some state s, taken at time t, and their exponentially weighted sum as
k-
NSTEP(T.)(s)) = Σ
Y14 + 7 (((44))
TD TD(:A) := (1 A) k- NSTEE((7.)
(2)
For values of À close to 1, more weight is assigned to longer n-step return estimates, and less to shorter ones. There has been a long-standing belief that shorter n-step returns generate more biased estimates, whereas longer n-step estimates, due to summing over many stochastic rewards, have higher v For a more thorough discussion on this topic see [20].
Noise Scale. Noise scale is a measure of noise-to-signal in stochastic gradient descent (SGD). While SGD provides unbiased gradient estimates, these estimates are typically very noisy. Each sampled gradient estimate G can be thought of as J. A method for estimating the ratio of the magnitude of this implied noise vector to the magnitude of the true gradient was proposed by [23] who show that their efficient-to-ee co-lllllle simple noise scale is a good match for the noise scale. They also show that the noise scale provides useful information about the choice of mini-batch size use when estimating gradients.
of the true gradient G and some noise vector
Phasic Policy Gradient. Most similar to our work is Phasic Policy Gradient (PPG) [11]. Like DNA, PPG has three distinct phases during training and uses two independent networks. However, there are several important distinctions in our work. First, we forgo the large replay buffer, relying instead on learning entirely from recent experience. This sigmificantly reduces the memory requirements of our algorithm. Second, we make of distillation phase rather than auxiliary phase." Finally, our work reduces gradient noise by using two different return estimators calibrated for the characteristics of each task. We assess the impact of these differences the performance of the agent in Section 6.
'While there are cases (i.e. where rewards are negatively correlated) where this is not the case, empirical experiments confirm the intuition that longer n-step estimates generally have higher variance and lower bias. *PPG refer to their third phase as a distillation phase, however, because the update trains both the policy and value networks on value estimates generated from rollouts, it is better thought of as an auxiliary task. We discuss this in more detail in Section 4.3

---

3 The Noise Properties of Value Learning and Policy Gradient
Here we examine the noise properties of the policy gradient and value learning tasks and show that the policy gradient has much higher noise level than value learning. This implies different challenges in the optimization problem. Specifically, that policy should be learned with a much larger mini mini-batch size than value learning and that return estimates should be adapted appropriately.
3.1 Motivating Example
To motivate our investigation into noise levels, we consider the task of learning two independent functions Fi (a) := (5x) + N(0, of) and (2()) cos(52) N(0. oτ). where N(H: o²) is Gaussian noise with mean μ and standard deviation o, over the domain [-T, тT. We fix = 1, and vary σ on a log scale from 0.1 to 100. We trained two 3-layer multi-layer perceptions (MLPs) on this problem. The first was joint model, using first hidden layer of 1024 units and second hidden layer of 2048 units, followed by two output heads. Our second model was a dual network consisting of two independent MLPs with 1024 units each hidden layer. All models used ReLU activations in between linear layers, and error was measured as mean-squared-erss (MSE) between the predicted value and the noise free true value. The results levels, the tasks did not interfere, but as the noise of the first task increased, performance on the second task eventually degraded for the joint network, but not the dual network. While simple, this experiment demonstrates the impact one noisy task can have on another when learned jointly by a deep neural network.
presented in Figure 1. At low noise
0.200
0.5
Dual Lr
Lu7
Joint Ln
0.175
Joint
0.4
0.150
0.125
MSF
0.100
0.2
0.075
0.050
0.1
0.0
0.000
10
10a
101
10
10
10a
10a
Figure I: A depiction of the problem of destructive interference on a toy problem. LTi and LT2 refer to the meaa-squured-error on the first and second task respectively. Shading indicates standard error over 100 seeds.
3.2
Noise Scale in Reinforcement Learning
To assess the differences in noise levels in our experiments, we measure the gradient noise scale of the policy gradient and value loss gradient. We did this by calculating an estimate of the noise scale, developed by [23] called simple noise scale defined as
Beimple
G
where Σ is the gradient covariant matrix, and G is the true gradient. For convenience we also use the notation : / /Bample. which can interpreted as the ratio of the length of the implied noise vector to the length the true gradient vector. That is, kind of noise-to-signal ratio. As [23] have shown, low bias estimate Bsiple can be found efficiently by generating gradient estimates Bunn using small mini-batch of size Bsmall well the gradient G Buy using large mini-batch of size Boig. then calculating
'It is worth noting we did not find the same result when using shallow networks. That is, dual and joint models performed similarly when using a 2-layer MI.P. 'This number of hidden units chosen that both models had the same number of parameters.

---

Pnicy (Asauti)
Policy (MaPacman)
Policy (YarsRevenge)
Value (Asssut)
2.0
(Tanalleerrg))
1.5
1.0
0.5
0.0
25
50
75
100
125
150
175
200
Frame (H)
Figure 2: Noise scale for the two learning objectives. For clarity all values are further smoothed using an exponential rolling average (with non-smoothed values presented a lighter shade). Smoothing may introduce bias, as while | and S unbiased estimators, their ratio may not be.
1
19
(Brg G Beis
12 Bamall G GBannn
(4)
Boig
B
S = 1/ small
(IG Bunan
12 |GBugl 12),
(5)
1/ Baig
where [|[g| 2 and E[S] tr()).
To evaluate the noise scale of value learning and policy gradient we trained our dual network model
(described fully in section 4) on the three Atari games from the Atari-3 validation set [2]. Like [23]. we found it necessary to smooth out noise by maintaining exponential moving average of |G Bus |2. We used Bsmall = 32 and Boig = 16, 384 given in Appendix N.
our experiments. Pscudocode for this procedure is
The results are presented Figure 2. We found that policy noise (0xx) reduced by about half during training and that value noise (o,)) remained constant. Both noise levels varied very little between the three environments. A large variation was observed between the value loss and policy gradient, consistent between environments. The noise levels, measured at the end of training and averaged over all three environments, were found be ar 220.5 and av 17.5, representing 12.6 times difference.
4 Dual Network Architecture
Based on our experimental results in Section we propose an architecture which takes into account the large difference in noise levels between the two tasks. This architecture consists of three improvements to PPO. First, to reduce negative interference from the noisy policy gradient, policy and value should be learned by independent networks with different hyperparameters (e.g. training epochs and mini-batch size). Second, the variance/bias trade-off in the return estimations should be calibrated each task. Finally, constrained distillation phase should be applied to take advantage of any constructive interference between the two tasks.?
4.1 Independent Networks
Important to DNA is the of dual network architecture." Not only does this setup allow for learning of the value and policy without destructive interference, also enables specialized set of hyperparameters to be calibrated for the distinct tasks of policy and value learning. Specifically, the
'The source code used to generate the results in this paper is provided in the supplementary material. We also provide an implementation of our algorithm https:// //gi figt. com/co////a///o com/maite wetaaiicchiooooiee wttassstt///////D//// By dual network, we mean two independent networks. Not to be confused with dual-head network (which refer a joint network), with dualing networks [35].

---

((+)) V++(s)
(++(+)
T(())
V(s))
fx By
0
Figure 3: Architecture of DNA (left) compared to a single network setup (right).
calibration of the mini-bated size, which it has been suggested is best set roughly proportional to [23]. The policy network outputs a policy π and a value estimate Vr, whereas the value network only outputs value Vv, depicted in Figure 3.
4.2 Decoupled Return Estimation
To better facilitate control over the noise levels for policy and value learning, DNA makes use of two different return estimations, both using TD(A) [33] as follows
Vung (s:) TD():r)
(6)
Vadv (8t) := TD ))))
(7)
where Xy and 7 are hyperparameters controlling the variance / bias trade-off of each estimate. The Viny estimates are used as targets for training the value function, whereas Vidv are used for advantage estimates used for policy gradient estimates given by
Å = Vadv(Se) - (v(
(8)
This formulation of the advantages is equivalent to the general advantage estimation (GAE) [30], that is, for any À E [0..1], E [0..1] we have," 9
A AGAE(s.A) TD (st)
Because of this, it is common when using PPO generate value targets, Vung. for the value function by adding the network's 's value estimates to the (unnormalized) advantages, implicitly settings xy Ax 10 We hypothesize that due to bootstrapping, estimates used for Vang would benefit from being low bias, conversely, because estimates used for advantage estimation contribute to the noise of the (already very noisy) policy gradient, they would benefit from being low variance. That is, the bootstrapping process may cause symmetrically distributed error, but not biased error, to cancel out, This suggests setting AS- < \v, a thought which we test empirically in section 6.2.
4.3 Distillation
Because value learning a simpler task in terms of noise level, it makes sense try transfer some of the knowledge learned by the value network to the policy network. If the value network learns to identify important features in the environment, features also. To facilitate this, we employ constrained distillation update [17]. Distillation between two identical networks has the property that the student's target function (since an identical network generated it) is guaranteed to be in the model's solution space. Also, because the distillation targets are deterministic." the process is also very low noise, which we verify Appendix
would like the policy network identify those
A proof for this claim is provided in Appendix For example, see line 65 of the popular baselines implementation PPO. https://github. com/opnaai/c/ccc//mmmca/cc basel iiees/ne////e wttbb//a22h9e8b344eeeeeo httpo224466eela44383..o.. /tt.///f....//444499////aaaeeee.
-yy
"That is to say. Vang (st) are samples drawn from the random variable R(sx). where A is (noisy) return estimator, whereas V (st) a deterministic function that produces estimate of [[[)].))].
5

---

We contrast this with the auxiliary update of PPG which trains both the value function and the policy on auxiliary task. This auxiliary task equates to learning N Σi o Vingg where Viarg is value estimate for the policy from updates prior. For quickly changing policies, and large N, these updates may cause tension with the value estimates of the current policy. 12
We use (s)s) as the distillation targets, with the input states being taken from the current rollout. However, unlike policy and value training, distillation state inputs need not be on-policy. We explored other distillation targets, covered in Appendix J, but found (s) to be the best of those tried. Distillation is performed, like PPG's auxiliary task, using mean squared error, and under soft constraint on the policy network's 's policy, specifically
(0) = [V((((s)) (v()))) + + E, [L Tata (-(s))) x-((s)))]
(10)
where 3 is the policy constraint coefficient, and is copy of the policy before the distillation update. During updates gradients are only propagated through the policy network, and not the value network.
4.4 Training
Like Phasic Policy Gradient (PPG) [11], DNA splits training into three distinct phases (policy, value, and distil), but unlike PPG, rather than using large replay buffer, we perform all updates on-policy on the current batch rollout data. Each of the three phases optimizes single objective, for some number of epochs, using its own optimizer, with unique set of hyperparameters. The optimization objective for the policy phase of DNA is the clipped surrogate object from PPO [31] including the entropy bonus
CLIP E, (iii
e)).) + ceb S[T(s:)]
(1I)
=sa,ls))
where S the entropy in nats, Pt is the ratio saulas|s.) at time f, e is the clipping coefficient, and cab the entropy bonus coefficient, For the value phase, we we use the squared-error value loss,
IV == E. (s
(12)
summary, the DNA algorithm separates the tasks of policy learning and value learning into network to handle the high noise of policy learning and separate network to handle the lower noise task value learning. Knowledge from the value learning network is transferred to the policy network through separate constrained distillation phase, which allows for constructive interference between the learning tasks while minimizing the destructive. We formalize the algorithm as follows.
12 better way auxiliary tasks would to separate value network output head, dedicated to the auxiliary task. This way the agent could lear the value of the current policy, along with the moving average value. We believe, however, that the better solution to overfitting the value function to recent data is simply to train less. An idea which discuss further in Section 7.

---

Algorithm Proximal Policy Optimization with Dual Network Architecture
1: procedure PPO-DNA
2: Input N E z rollout horizon
3: Input A E Z number of agents
4: Input π the initial policy.
5: for i = 1 to......0 do
6:
for a== to A do
7:
Run policy in environment for N timesteps
Compute Vwg + TD(
8:
Compute Adv <- TD(::A.)
9:
10:
Compute t Vady - Vv
for j = 1 to Ex do
12:
Optimize CCLP wrt 0x
13:
for j = 1 to Ey do Optimize LVF wrt 0y
14:
15:
Told f
for j = to ED do Optimize D. wrt 0x
17:
18:
Output
5
Evaluation
To evaluate our algorithm's performance, used the Atari-5 benchmark [2]. Scores in Atari-5 are generated using a weighted geometric average over five specific games and produce results that correlate well with the median score if all 57-games had been evaluated. This allowed to perform multiple seeded runs and defined a clear training and test split between the games. In all cases, we fit hyperparameters to the 3-game validation set and only used the 5-game test set for final evaluations.
We opted for the more difficult stochastic ALE settings recommended as best practice by [22]. However, to better understand our results in the context of prior work, we also provide results under the simpler deterministic settings Appendix F, and additionally provide for reference a single evaluation the full 57-game set in Appendix M. Unless otherwise specified, agents were scored according to their average performance over the previously completed 100-episodes at the end of training.
coarse hyperparameter sweep found initial hyperparameters for our model on the Atari-3 validation set. Notably, we found the optimal mini-bated batch size for value and distillation to be the minimum tested (256), while the optimal mini-band batch size for policy was the largest tested (2048). For optimization, we used Adam [21], over the standard 200 million frames." Full hyperparameter details for our experiments are given in Appendix B.
In order to understand the impact of the hyperparameters introduced by our algorithm, namely Ex, Ev, ED and X λy, we performed the following experiments on the Atari-3 validation set. We started by setting Eg = Ev = ED = 2, then searched over Ev E [1, 2, 3, 4], and selected the best Ev. We then searched over E5 and E'D while keeping the other two parameters constant. We also performed similar experiment on the impact of xy and A by setting xv to 0.95 then sweeping across A [0.6, 0.8, 0.9, 0.95, 0.975). then setting xy to the best A and repeating. This process allowed us to verify if the best settings occur at xy Xx or at Xv * > During our À tuning, we also recorded noise levels for one of the seeds, the results which we discuss in Section 6.2.
We evaluated our algorithm DNA, against PPO, and PPG. All models used the 'NatureCNN'' encoder from [25]. However, because DNA and PPG use networks and thus twice the parameters, doubled the number of channels of the PPO encoder which we refer to as PPO (2x) and which is very similar the encoder used by [4]. All models have have approximately 3.5 million parameters in total. For fairness we repeated the same sweep on PPO for the number of training epochs (Erro) and
used for GAE (Arro). To verify that our settings did not inadvertently degrade the performance of
That is 50 million environment interactions.
7

---

3339
12
Figure 4: Training curves for DNA with various epochs. The DNA algorithm is most sensitive to the number of policy epochs, and is optimal our validation with E 2. Applying policy updates 3 or 4 times significantly reduces the agents performance.
245
Ap- C25
255
144
12
Figure 5: Performance of DNA across range of À values for return estimations. Left: The optimal settings for λy and Åę differed. Shading indicates standard eπrоr over 5 seeds. Mid/right: Noise levels from single seed, σx increased with X where σy decreased with λy. Shading indicates standard deviation over the three environments.
PPO we also tested against the settings used by [31] and refer this as PPO (original). These results are included in the ablation study in Appendix C.
6 Main Experimental Results
In this section we present the results of our main study comparing DNA to PPO and PPG on Atari-5. We also include supplementary experiments on MuloCo [34] and Procgen [10] in Appendix K, and L. respectively.
6.1 Impact of Training Epochs
We start by investigating the impact of the Ex. Ey, and Ep hyperparameters. We found that using less than three policy epochs for DNA dramatically increased the performance of the agent while also decreasing the computation required to train the agent (Figure 4). We also found that DNA was robust the choice of value and distillation epochs but that overtraining on value (Fy 4) marginally decreased performance. This was not the case for distillation updates. The distillation phase demonstrated some benefit indicated by the ED = 0 results under-perform the others. We selected Ex 2, Ev 1, Ep 2 the optimal settings and used these for the remainder of the experiments. These settings require total of four updates for the policy network, but only one update for the value network (we discuss this further in Section 7). The training curves also suggest DNA may benefit from training beyond the standard 200 million frames.
6.2 Return Estimation
We found that the choice of \y made large difference to the performance of our agent. As was hypothesised the optimal settings for > differed that of xv with X preferring lower variance, higher bias value of 0.8, while the value targets, Ay preferred a lower bias, higher variance setting of 0.95 (Figure 5). Notable that the non-homogeneous setting xy 0.95, X
outperformed both homogeneous choices X Av 0.8 and X Av 0.95. We found, expected, that policy gradient noise can be reduced by selecting low value for > However, against our expectations,
setting \y lower actually increased, not decreased the noise level of the value learning task. Once

---

DNA
250
PPG
PPO (2x)
Score (Atari-5)
200
Rainbow
150
100
50
50
75
100
125
150
175
200
Frame (M)
BattleZone
1e1
DoubleDunk
NagNammeeie
2
5.0
100
200
100 Qbert
200
100
200
1e5
Phoenix
1e4
0.5
2
0.0
100
200
100
200
Figure Top: Results on the Atari-5 benchmark. DNA outperforms PPO by wide margin, and even exceeds Rainbow DQN despite the more difficult settings used in our experiment. Bottom: Individual training curves for each of the five games.
discovered this we ran addition experiments for broader choice of λy, and include the results in Figure 5.
6.3
Comparison to PPO and PPG
We found DNA to outperform both PPO and PPG on the Atari-5 dataset by wide margin (Figure 6). We give Atari-5 scores, as well as training plots for each individual game. We also include references for Rainbow DQN, although we note that these scores were generated under the simpler deterministic settings.
7 Discussion
Reduced Training as an Alternative to Experience Replay We found that less training, both on policy and value updates, typically improves agent's performance. One explanation for this could be, as [11] have suggested, that in the case value learning, this could be due to overfitting to recent experience." In terms of policy updates, where the largest performance degradation was observed. this could be due to increased training epochs causing a large change the policy, which may suggest better constraint than PPO's 's clipping is required." That being said, limiting the training two epochs provides straightforward solution to the problem while having the additional benefit of decreasing the computational resources required train an agent.
"This plausible, especially with high variance returns.
It is worth noting that PPO does clip the entropy bonus, and therefore updates with non-zen entropy bonus could make arbitrarily large changes to the policy regardless of the choice of

---

Choice of X Our results give evidence for the benefit of selecting A < Av. As expected, picking lower values X reduces the policy gradient noise. However, counterintuitively, noise on value learning reduces with higher values of xv. That is, policy learning prefers low variance, but value learning prefers low bias. This result may explain why clipping, which introduces bias, performs poorly value learning but quite well on policy learning. 16 Our results also suggest that an even higher choice of λy than the 0.95 we used in our experiments may be appropriate.
Update-to-Data Ratio Our work also raises an interesting question about the update-to-data (UTD) ratio differences between value and policy learning. We found DNA to work best with only a single value network update but four policy network updates. This asymmetry could be due to the extremely high noise levels for policy updates. The single value update contrasts with other work where much higher value learning UTD ratios was found to be preferred [1. 8].
Limitations
Our paper's primary focus was improving results on discrete action vision-based problems, this is where PG has traditionally unperformed Deep Q-learning. However, do also provide an initial look into continuous control problems on the MuJoCo [34] dataset in Appendix K.
Broader Impact In our work, we have shown that deep actor-critic reinforcement learning models can effectively solve complex vision-based, discrete-action environments, even without the use of large replay buffers. Surprisingly this be achieved using less not more policy and value network updates. Because Q-learning approaches generally produce deterministic policies, algorithms based on them may 'lock in' to decision based on minimal differences in outcome. This is not always the case with stochastic policies, which are able to randomly split over actions of roughly equal quality, which may lead to fairer outcomes. We do not foresee any direct negative societal impacts from this research.
Future work Our algorithm is already highly competitive with Rainbow DQN, and could likely be extended with orthogonal improvements such as adding recurrence [14], distributional value learning [5], improved feature encoders [13], better exploration bonuses [7], and making use of a learned model [28].
8 Conclusion
In this paper, we have highlighted noise level key difference between policy and value learning. We have introduced an algorithm that accounts for this order-of-magnitude difference by limiting negative interference through the training of two independent networks for value and policy but retaining constructive interference through a constrained distillation process. Furthermore, we have shown that the variance/bias tradeoff differs for value learning and policy gradient and that return estimation should cater for this. Together these changes result in novel algorithm, DNA, that outperforms its predecessor PPO, and even surpasses the popular Q-learning approach Rainbow DQN, while under more challenging environmental settings.
Acknowledgments and Disclosure of Funding
This research was supported by an Australian Government Research Training Program (RTP) Schol- arship
BB value clipping, we mean the value trust-region suggested by [30], but which has been shown hurt performance by [3].
10

---

References
[1] M. Aitchison. Optimal use of experience in first person shooter environments. In 2019 IEEE Conference on Games (CoG), pages 1-4. IEEE, 2019.
[2] M. Aitchison, P. Sweetser, and M. Hutter. Atari-5: Distilling the arcade learning environment down five games. arXiv preprint arXiv:2210.02019, 2022.
[3] M. Andrychowicz, A. Raichuk, P. Stañczyk. M. Orsini, Girgin, Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, et al. What matters for on-policy deep actor-critic methods?
a large-scale study. In International conference on learning representations, 2020.
[4] A. P. Badia, B. Piot, S. Kapturowski. P. Sprechmann. A. Vitvitskyi. Z. D. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine
Learning, pages 507-517. PMLR, 2020.
[5] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[6] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[7] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12894, 2018.
[8] X. Chen, C. Wang. Z. Zhou, and K. Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101. 10X.:22001.0582,, 2021.
[9] Cichosz. Truncating temporal differences: On the efficient implementation of (lambda) for reinforcement learning. Journal of Artificial Intelligence Research, 2:287-318, 1994.
[10] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation benchmark reinforcement learning. In International conference on machine learning, pages PMLR, 2020.
[11] K. W. Cobbe, J. Hilton, O. Klimov, and J. Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020-2027. PMLR, 2021.
[12] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Imple- mentation matters in deep rl: A case study ppo and trpo. In International conference learning representations, 2019.
[13] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learne architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
[14] M. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI fall symposium series, 2015.
[15] M. Hessel, I. Danihelka, F. Viola, A. Guez, S. Schmitt, L. Sifre, T. Weber, D. Silver, and
H. Van Hasselt. Muesli: Combining improvements in policy optimization. In International Conference on Machine Learning, pages 2214-4226. PMLR, 2021.
[16] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty- AAAI conference on artificial intelligence, 2018.
[17] G. Hinton, O. Vinyals. J. Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 02531, 2(7), 2015.
[18] D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. Van Hasselt, and D. Silver. Distributed prioritized experience replay. arXiv preprint WxXiv:1003.0093 2018
[19] Huang, R. F. J. Dossa, A. Raffin, A. Kanervisto, and W. Wang. The 37 implementa- tion details of proximal policy optimization. In ICLR Blog Track, 2022. https://iclr-blog- httpa//oo00200/////pmmeeeeeeeeee
[20] M. J. Kearns and P. Singh. Bias-variand error bounds for temporal difference updates. In COLT, pages 142-147. Citeseer, 2000.

---

[21] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 2014.
[22] M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
[23] S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06160 2018.
[24] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pages 1928-1937. PMI.R, 2016.
[25] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 518(7540):529-533 2015.
[26] F. Pardo, A. Tavakoli, V. Levdik, and P. Kormushev. Time limits in reinforcement learning, In International Conference on Machine Learning, pages 4045-4054. PMLR, 2018.
[27] R. Raileanu and R. Fergus. Decoupling value and policy for generalization in reinforcement learning. In International Conference on Machine Learning, pages 8787-8798. PMLR, 2021,
[28] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis. T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 (88(883))604-609, 2020.
[29] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learming, pages 1889-1897. PMLR, 2015.
[30] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0244 2015.
[31] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization 2017.
algorithms. arXiv preprint
[32] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L., Sifre, D. Kumaran, T. Graepel, al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1 140-1 1144, 2018.
[33] R. S. Sutton. Learning to predict by the methods temporal differences. Machine learning. 3(1):9-44, 1988.
[34] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026 5033. IEEE, 2012.
[35] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
[36] R. J. Williams. Simple statistical cradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-25 256, 1992.
[37] C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.0199: 2021.
12


Reviews:
[{"review_id": "knm5U764qDf", "reviewer": "Reviewer_2gBU", "summary": "The paper analyzes various design decisions made by PPO, a popular actor critic policy gradient method, and proposes various fixes to the existing algorithm/architecture to improve the performance. Specifically, they study the impact of the objective noise levels on the policy training and how it interacts with the various design decisions like the network architecture, return estimation, batchsize etc. They use the analysis to propose improvements to the algorithm and the architecture which results in improvements to the policy performance when tested on the Atari-5 benchmark.", "questions": "I think both the points in the Weaknesses sections are fixable and would appreciate the authors making amends along those directions.\n\n**Minor Comments**\n1. I think the following was a typo : Line 185 : policy -> value ", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper provides simple, well motivated fixes to PPO/PPG that demonstrate improved performance on the Atari Benchmark!\n2. The paper is very well written and easy to follow. I also really liked the motivating example in section 3.1 to demonstrate the impact of noise scale on the architecture.\n3. I liked the careful ablations in sections 6.1 and 6.2 analyzing the impact of the specific hyperparameters.\n**Weaknesses**\n1. Although the paper mentions that they leave the continuous control tasks for future work, I would still have liked some analysis of it in this paper itself to better understand the implications on those settings as well since PPO is widely used there.\n2. There have been various other papers in the field analyzing the various aspects of PPO over the past few years. I would have liked a more thorough review of the literature especially along those lines and see this work placed in the context of those papers.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AxIeU3lsGyb", "reviewer": "Reviewer_nzwx", "summary": "This paper proposes a dual network architecture extension of PPO. It considers three modifications: a separate neural network for value function and policy, calibration of bias-variance trade-off, and constrained distillation. The method is evaluated on a new Atari-5 environment and compared with a similar baseline Phasic Policy Gradient (PPG).\n", "questions": "How is the combined score in Figure 6 achieved? The performance improvement is visible only in the “Phoenix” environment and overlaps in other environments. Any aggregated metrics, such as the probability of improvement proposed in [1], might give a better understanding. How many seed run was used to generate Figure 6? It is mentioned that “multiple seeds run” in line 175.\n\nThe baseline PPG was heavily tested on the Procgen benchmark [2] for sample efficiency. Thus a comparison of the Procgen benchmark seems critical to assess the significance of the proposed method. Therefore, I suggest adding the experiment on Procgen to verify how the proposed method performs on existing and well-established benchmarks in addition to the newly proposed Atari-5.\n", "limitations": "Yes. Limitations and potential societal impacts are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength:\n- Tackle the important problem of improving the policy gradient method, which has wide applicability\n- Overall, well-written paper and easy to follow.\n- Show some improvement over baseline PPG on the Atari-5 benchmark.\n\n### Weakness (Details in Questions):\n- Lack of empirical evidence that demonstrates DNA’s advantage.\n- Missing experiment on Procgen benchmarks where the baseline PPG was evaluated.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Al6MZAjkI-2", "reviewer": "Reviewer_Knd8", "summary": "The paper identifies that the noise level between learning the value function and the policy in reinforcement learning can have an order of magnitude difference. The paper thus proposes to learn these two tasks independently followed by a constrained distillation phase. The paper also argue that the policy gradient noise level can be reduced by using a lower variance return estimate. On the other hand, a lower bias estimate should be used to reduce the value learning noise level.\n", "questions": "Can the authors also reconcile the results presented in the paper with this paper https://arxiv.org/abs/2101.05982? This paper seems to suggest that we should use a high update to data ratio, whereas the current paper seems to suggest that we should perform fewer updates. \n", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nThe paper demonstrates the issue of jointly learning two tasks with different noise level in a simple motivating example. The motivating example clearly illustrates destructive interference and provides a great starting point for the discussion in the remainder of the paper.\n\nThe paper introduces additional modification on top of PPO, motivated by the previously identified issues, that leads to significant performance Improvement.\n\nWeakness\n\nI would have liked to see a discussion on how the reader should interpret equation 4 and 5 in the paper. Also, How does the y-axis of Figure 2 relate to the left hand side of equations 4 and 5? This is not clearly explained in the paper if I am not mistaken. It would have been helpful if the authors include pseudocode on how the y-axis of Figure 2 is calculated.\n\nAlso, the noise scale as explained in equation 3 does not seem to take into account the directions of the true gradient. Is that true? If that is true then the quantity used to compute the noise scale seems a bit odd to me because shouldn't we take into account the direction of the true gradient and the direction of the noise vector as well? \n\nThe paper also introduces a new benchmark. Even if an anonymous copy is included, I would have liked to see a self-contained description in the main paper on why results on the new benchmark are meaningful or more meaningful compared to the previous version of the arcade learning environment benchmark.\n\nCan the authors also please elaborate on the discussion at the end of section 4.2 – why is it that because of bootstrapping, estimate used for V Target benefit from being low bias? \n\nI found the exposition in the experimental results section a bit unclear. If the paper is about noise level then why is it that the first experimental result presented has to do with the number of training epochs? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept