Simplifying Deep Temporal Difference Learning

Authors: Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, Mario Martin
Primary area: reinforcement learning
Keywords: Reinforcement Learning, TD, Theory, Q-learning, Parallelisation, Network Normalisation

Abstract:
$Q$-learning played a foundational role in the field reinforcement learning (RL).
However, TD algorithms with off-policy data, such as $Q$-learning, or nonlinear function approximation like deep neural networks require several additional tricks to stabilise training, primarily a large replay buffer and target networks. Unfortunately, the delayed updating of frozen network parameters in the target network harms the sample efficiency and, similarly, the large replay buffer introduces memory and implementation overheads. In this paper, we investigate whether it is possible to accelerate and simplify off-policy TD training while maintaining its stability. Our key theoretical result demonstrates for the first time that regularisation techniques such as LayerNorm can yield provably convergent TD algorithms without the need for a target network or replay buffer, even with off-policy data. Empirically, we find that online, parallelised sampling enabled by vectorised environments stabilises training without the need for a large replay buffer. Motivated by these findings, we propose PQN, our simplified deep online $Q$-Learning algorithm. Surprisingly, this simple algorithm is competitive with more complex methods like: Rainbow in Atari, PPO-RNN in Craftax, QMix in Smax, and can be up to 50x faster than traditional DQN without sacrificing sample efficiency. In an era where PPO has become the go-to RL algorithm, PQN reestablishes off-policy $Q$-learning as a viable alternative.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730338732333

thread_type:
official_review

<thread_object>
creator: reviewer_kann

<content>
summary:
This paper introduces Parallelised Q-Network (PQN), a streamlined deep online Q-Learning algorithm. PQN comprises two key components: a TD Learning objective without a target network, which instead applies layer normalization and L2 regularization, and a parallelized sampling approach that avoids the use of a replay buffer by leveraging vectorized environments. The authors provide theoretical analysis to support the claim that regularization can stabilize TD Learning in the absence of target networks. PQN demonstrates competitive performance across a wide range of environments, achieving results in significantly less wall-clock time.

strengths:
1. PQN is straightforward and easy to implement. It removes the need for a target network and simplifies TD Learning.
2. The paper includes theoretical analysis showing that regularization can help keep TD Learning stable without using target networks.
3. PQN achieves higher computational efficiency compared with baseline methods, with minimal impact in sample efficiency.

weaknesses:
1. It’s somewhat counterintuitive that PQN maintains sample efficiency while training only on online samples without a replay buffer. Additional explanation would help readers understand this aspect better.
2. The removal of the target network in TD Learning and the parallelized sampling are independent components of the algorithm, yet their individual contributions to overall performance are unclear. More controlled experiments, like the one in Figure 6.d, would clarify the impact of each component.
3. The parallelized sampling approach depends on vectorized environments, which feels more like an engineering choice than a novel contribution and is not feasible in many real-world applications. A significant portion of the wall-clock savings seems to come from this aspect. If my understanding is correct, DQN could also potentially eliminate the replay buffer and use a similar parallelized sampling approach. It would be informative to see how DQN performs under this setup.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730470447546

thread_type:
official_review

<thread_object>
creator: reviewer_jf9o

<content>
summary:
This paper analyses stability in temporal difference (TD) methods. The main contributions are theoretical proofs that (i) TD instability can be established using a Jacobian evaluated on the unit circle; and (ii) using the Layernorm regularisation technique can ensure stability. This then leads the authors to propose a deep Q-learning algorithm called PQN which is comptetive with the PPO approach to reinforcement learning.

strengths:
The paper is well written, well formatted and quite readable. The authors present the essence of their results very well and show that stability of TD algorithms reduces to checking that a Jacobian is negative definite on the unit circle. This is a nice succinct and somehow intuitive result. Given the complexity of the proof, it was good to see the summary presented so concisely. 

Other results are then presented after this, including some insight into the causes of instability and then the approach using the layernorm to obtain stabilisation. The presentation here was not as clear as the above, but still acceptable and still concise. 

The authors claim that their parallelised version of Q learning is motivated well and this seems to be backed up by experiments.

The overall implications of the authors' results are very significant: they have discovered and captured the root cause for TD instability, they have proposed an improvement which guarantees instability and they have showed that their new PQN performs exceptionally well on some examples.

weaknesses:
A criticism is that the proof of the main theoretical results is long (there are 20 pages of additional material) and I would say not particularly well organised. Before the authors give the proofs, in my view, it would be good for them to outline the main steps. I found the proofs hard to follow and as one goes through the proofs, there is a feeling of being somewhat adrift. In other words, the summary in the main paper is good; the actual proofs in the appendix are less well clear.

questions:
One question I have is why does the Jacobian have to be negative definite on the unit Circle? Why is simple negative definiteness not enough? Since any vector can be written as || u || = c || v || where c is a positive constant, it seems that simply negative definiteness of the Jacobian is required? It would be helpful to the reader if the authors could give more justification and/or insight into the reason negative definiteness on unit circle is required, or the authors may wish to reconsider their results and see whether the restriction can be removed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709884230

thread_type:
official_review

<thread_object>
creator: reviewer_gfms

<content>
summary:
This paper proposes simplifications to multiple components of the Deep Q-Network (DQN)/TD learning method to enable more efficient training on a single GPU, offering a potential DQN baseline for future research. The modifications include:
- Eliminating target network tricks: The authors theoretically demonstrate that combining layer normalization with L2 regularization leads to convergent temporal difference (TD) learning. They then conducted experiments to validate this empirically by removing the target network update tricks.
- Removing replay buffer for experience replay: The paper identifies the replay buffer as a memory bottleneck that limits single-GPU training. While directly removing the replay buffer impacts sample efficiency, the paper demonstrates that when combined with vectorized environments, the GPU-based training method achieves better wall-clock time efficiency.
- Batch-wise rollout using vectorized environments: The paper implements DQN training in a batch-wise manner, leveraging GPU parallelization (after removing the replay buffer). Instead of single actor rollout, the paper uses vectorized environments to generate rollouts in batch.

To validate their approach, the authors conduct comprehensive experiments on both theoretical/proof-of-concept environments (Baird's counterexample) and standard benchmarks (Atari and Crafter). Their results show that the proposed Parallelized Q-Network (PQN) achieves comparable performance to well-known baselines like PPO and Rainbow. Through ablation studies, they further demonstrate the importance of network normalization and justify the removal of the replay buffer.

strengths:
- This paper is well-written and easy to follow, with clear presentation of both theoretical derivations and experimental results.
- The paper's motivation is clear and compelling enough to me: it mainly provides a simplified Q-learning baseline that effectively leverages GPU parallelization and vectorized environments.
- The proposed experimental evaluation is relatively comprehensive: it covers multiple domains including proof-of-concept environments, standard single-agent benchmarks such as Atari and Crafter, and multi-agent scenarios. They also covers variants of the Q learning methods to support the claim better in general.

weaknesses:
There is no major weakness of this paper, but feel free to check the question section for minor questions.

questions:
- Including Baird's counterexample results in the main text would strengthen the paper in my opinion, by providing a clearer connection between the theoretical analysis and experimental validation.
- (Minor) The PPO baseline comparison in Figure 3 could be more consistent, though I understand the thinking to save compute. The paper could either compare both methods using 4e8 training frames in Figure 3(a), or include PPO results directly in Figure 3(b) across all Atari environments. Either approach would more effectively demonstrate PQN's competitiveness against established policy gradient methods, in terms of sample efficiency.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717832323

thread_type:
official_review

<thread_object>
creator: reviewer_lgpr

<content>
summary:
Modern deep-reinforcement learning resorts to techniques such as replay buffers and target networks to provide stability with nonlinear off-policy learning. However, learning becomes unstable without a replay buffer or target networks and can diverge. Recently, several works suggested using layer normalization or layer normalization in addition to l2 regularization to remedy this learning instability issue. This paper theoretically studies layer normalization’s role and identifies how layer normalization helps with stability and convergence. The paper also proposed a new method called PQN that uses layer normalization and parallelized environments to stabilize learning. The authors show the effectiveness of their method through a series of experiments on different domains of environments.

strengths:
The work on simplifying deep reinforcement learning and removing techniques that might not be necessary, like replay buffer and target networks, is undoubtedly fundamental to deep reinforcement research. This has vast implications for rethinking the widely existing deep RL approaches and can help in other important directions, like scaling RL with the number of parameters/samples. This paper provides a unique view that challenges existing beliefs on the importance of replay buffers and target networks. The approach is effective and efficient since it can be implemented parallelized on GPU, which outperforms other baselines with respect to wall clock time.

weaknesses:
- The theoretical part of the manuscript is largely incoherent. 
  - The current manuscript scatters many things in the theory parts. It lacks a proper flow of ideas when describing the theoretical results and their implications, which makes it difficult to follow. Currently, it reads as bullet points, listing findings quickly without proper linking between subsequent findings or results.
  - For example, the current theorems and lemmas are not well integrated with the text before and after them. They read as detached components, making reading unnecessarily harder.
  - Notations can be improved. I suggest following Sutton & Barto (2018).
  - Two things that are instrumental for the results based on Jacobian analysis: inequality 3 and inequality 4. More discussion is needed to understand these two conditions and their implications.
  - Off-policy instability and nonlinear instability require their own theorem statements and separate proofs (even if previous works have shown them). In addition, the TD stability criterion needs a theorem statement about contraction mapping. 
  - The connection for why l2 is needed was not clear. It was directly introduced after layer norm without proper linking.
- PQN solves a problem different from the baseline DQN, but this was never discussed
  - The authors emphasize that PQN does not use a replay buffer or target network as an advantage over other methods, which is great. However, a similar emphasis is needed for the fact that PQN requires parallel environments and probably may fail if a single environment were used (the setting of the other baselines). Additionally, PQN solves another orthogonal problem (parallelized worlds) to the original RL problem (single world).
  - Figure 1 is inaccurate. The replay buffer is part of the agent, not an external component. This needs to be fixed.
  - The difference between Distributed DQN and PQN is unclear from Figure 1, although both solve the same problem (parallelized worlds), especially the point on synchronism and GPU is not clear.
- The paper has several inaccuracies.
  - The authors claim that their PQN is based on Peng’s Q($\lambda$), but the actual algorithm does not use eligibility traces. Instead, the authors use Q-learning with $\lambda$-return. This needs to be corrected. Additionally, the equation (line 326) needs to be derived from the first principle to make the paper accessible to the unfamiliar reader.
  - Algorithm 1 is Q-learning with one-step targets. The authors mention that they use $\lambda$-return target, so Algorithm 1 needs to be replaced with the algorithm actually used (Algorithm 2 in Appendix C).
  - The authors claim they stabilize learning Baird’s counterexample and use Figure 7a to demonstrate that. However, in Figure 7a, the error increases from the starting point until it plateaus. I don’t think meaningful learning has happened since the error has increased instead of decreased. I see that with layer norm or layer norm + L2 regularization, the error doesn’t increase without bound but at the same time, the problem is not solved either.
- Unfair or unclear empirical evaluation
  - The authors compare algorithms that work with parallel environments against ones that do not, which requires careful experiments to compare them. 
  - In Figure 3, the authors need to write rainbow (200M) or DQN (200M) to understand what those horizontal lines represent clearly.
  - When the authors say that PQN was trained for 400M frames, does that include the parallel environments? For example, if 128 parallel environments are used (according to Table 5), does this mean 3.125M frames were collected from each environment, resulting in a total of 400M frames, or does it mean that 400M frames were collected from each environment, resulting in 128x400M=51200M overall frames? The first option gives a fair comparison, but the second option is biased towards PQN since significantly more experience is used. I would like the authors to clarify this point.
  - The authors used only 3 independent runs for Atari, relying on precedence.  This is a too low number to provide any statistical significance. Even if something was accepted before, it does not mean it is correct. I highly suggest the authors increase the number of independent runs to at least 10. This should be possible since both PQN and PPO are efficient (small clock time) and easy to run, according to the paper's claims.
  - The authors mentioned that DQN/Rainbow uses 50M updates compared to 700k updates for PQN. I think it is unclear how these numbers are obtained, especially for PQN.
  - In Figure 6a, PQN still learns well without divergence when no layer normalization is used. What is the reason? Why has no divergence happened?
  - Since the authors compare against DQN and Rainbow (methods that use a single environment), an ablation where a different number of environments are considered (e.g., n=1 and n=10) to understand the provided stability coming from parallelized environments. I think parallel environments make the gradient signal more reliable compared to the single environment case, which is more prone to noisy gradients. This may be instrumental for PQN; thus, an ablation is needed.

&nbsp;

**Minor issues:**

- line 9 in Algorithm 1 and line 10 in Algorithm 2 are incorrect. The negative sign should be a plus. Additionally, I think the use of $\texttt{StopGrad}$ can be eliminated with the TD error vector you gave in Eq. 2

&nbsp;
&nbsp;

Overall, I believe the ideas from this paper could serve as a good contribution to the community, but the current manuscript is not ready for publication and needs significant improvement. I'm willing to increase my score, given that the authors improved their manuscript's quality by 1) rewriting the theoretical part to make it coherent and more rigorous, 2) Fixing the inaccuracies, and 3) improving the empirical evaluation quality.

questions:
- Is PPO using the same number of parallel environments as PQN in all experiments? I couldn’t find this information in the paper. Could you share this information and add it to the paper’s revision?
- In line 428, the authors refer to a histogram in Appendix E, but there is no histogram. Do they mean the bar plot in Figure 12?
- Typically, layer norm is not applied to the post-activation but instead to the pre-activation. This is an important distinction. The theory still works with the preactivation layer norm as long as you don’t use activation functions that scale up the inputs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732367957623

thread_type:
public_comment

<thread_object>
creator: ~nico_bohlinger1

<content>
title:
Continuous action spaces and compatibility with SAC, DDPG, TD3

comment:
PQN is a great step to lift the boundary between off-policy and on-policy RL algorithms and research. The authors show how the removal of replay buffers and target networks and the addition of training batches collected online by leveraging parallelized environments simplify and improve upon DQN. All the suggested changes can also directly be applied to off-policy algorithms for continuous action spaces, like SAC, DDPG, TD3, etc. I want to suggest that the authors could use at least one of these algorithms to test the general applicability of their changes and benchmark on the typical continuous action space environments, e.g. Gym MuJoCo. This is important to determine the usefulness of the proposed changes for the whole field and also to make a fairer comparison with the often mentioned PPO and CrossQ.
I would be happy to discuss the applicability of PQN's changes to algorithms for continuous action spaces.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733155856979

thread_type:
public_comment

<thread_object>
creator: ~mark_towers1

<content>
comment:
I'm very impressed by this paper and wish it would be published as I believe that the RL community can learn a lot from the extensive theoretical and empirical results presented.

However, I believe the authors consistently incorrectly cite the number of network updates for Rainbow as 50 million (Table 3 and Line 442, for example). The confusion arises from the difference in frames and steps. Rainbow uses a frame skip of four (for 50 million steps) and then does a gradient step every four steps (not frames), meaning they run 12.5 million updates, not the listed 50 million. 
Additionally, the authors say PQN uses 700k network updates in Table 3 and L442, but I believe the value is actually ~780k, an error of >10%. Like Rainbow in Atari, PQN uses a frame skip of four with 128 environments and 32 rollout steps for roughly 12k batch updates (50 million / 128 environment / 32-step rollout). For these batches, PQN has two epochs of 32 mini-batches, making the number of network updates ~780k. Please correct me if you believe me wrong; otherwise, could the paper be updated with these corrected values? 
 
An additional detail is that on reviewing the open-souced code for the implementation by the author (thank you this is extremely helpful for future work), I found the `pqn_atari` implementation config had `episodic_life=True`. Could the authors clarify if this parameter was used for the experiments in Section 5.2 and Figure 4? This is weakly referenced in Figure 14 that this was disabled so I presume it is enabled in Figure 4 but was not included in the Atari Hyperparmeter table (Table 5). If so, I strongly believe that the authors should note this when discussing their Atari results, as this has an impact on an agent's training for Atari to ensure fair comparisons in Figure 4 and for future work. The impact of this parameter is even noted by EnvPool in their [documentation](https://envpool.readthedocs.io/en/latest/env/atari.html) to "improve value estimation". The parameter changes Atari's behaviour to provide termination (done) signals when an agent loses a life, not just at the end of an episode, improving value estimation. As a result, if the authors are using this for Figure 4 but it is not used (to my knowledge) in the prior work (Rainbow, Prioritised DDQN and DDQN), this potentially provides an unfair comparison. Therefore, a discussion or note on this parameter seems necessary, in my opinion, beyond a caption in Appendix D of the paper.

A side note is that Figure 14 should cite [Machado et al., 2018](https://arxiv.org/abs/1709.06009) rather than Casto et al., 2018 to my understanding.

I would also request that the authors publish PQN performance for 200 million (alongside 400 million), as this is the standard benchmark in the field and would allow future readers to better compare data with other papers (Tables 3 and 9).
</content>
</thread_object>

</thread>
<thread>
creation date:
1745264855825

thread_type:
public_comment

<thread_object>
creator: ~brett_daley1

<content>
title:
Paper contains copy and pasted material from Daley & Amato (2019)

comment:
I am the first author of [Reconciling λ-returns with Experience Replay](https://arxiv.org/pdf/1810.09967), published at NeurIPS 2019, which came up during the reviewer discussion [6].

I was not involved in the review process.
It only recently came to my attention that the recursive λ-return derivation in Appendix B.4 reproduces a large portion of our own derivation (Appendix D of our paper) without acknowledgment, including
- 8 equations (same notation),
- 4 English sentences (identical wording).

[See the attached screenshots for a comparison.](https://drive.google.com/drive/folders/172odF77RDlxLABC6br66F0_YBcXc48ms?usp=sharing)

Copying and pasting without attribution is plagiarism.
This could be fixed by either
- Explicitly stating that a substantial portion of the derivation has been copied from Daley & Amato (2019), or
- Removing the derivation altogether and referring readers to Appendix D of Daley & Amato (2019).

Note that although it appears you slightly changed the λ-return formula used in your paper, we discuss both formulas in our work (see footnote 4 of our paper).
Both formulas have been well known since the 1990s.
However, the derivation in our paper is original and must be acknowledged if it is used.

Please resolve this soon for both the camera-ready and arXiv versions of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper makes two contributions. First, a proof that TD learning converges when the network uses layer normalization and weight-decay. This is demonstrated in experiments that show that one need not use a target value network or a “replay buffer” (there was a lot of back and forth on what constitutes a replay buffer with the reviewers). Second, the authors use vectorized environments for batched-rollouts to speed up training. We would like to thank the reviewers and the authors for a healthy discussion that has led to improvements in the manuscript during the review process.

I recommend that this paper be accepted. I encourage the authors to tie up some of the loose ends (e.g., public comments on this forum, as well as comments by Reviewer LgPr) in the camera-ready manuscript.