Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a modification of the DQN algorithm by adding a proximal term that biases the learning to retain the Q algorithm in the proximity of the target network. The authors show that the proposed algorithm perform better compared to the original DQN algorithm on several Atari games. 
main_review: The authors point out the various challenges in stabilizing deep reinforcement learning. They argue that applying the idea of proximal iteration to DQN can improve the stability of the learning and the performance of the algorithm. In practice, this implies a small change in the weight update term of the DQN algorithm, which biases the weight changes towards the target network. 

The authors compare the resulting algorithm with the original DQN both the version from Google Dopamine and the authors' own reimplementation of it. The approach improves on the original DQN, but it does not matching the current state of the art. 

As a note: Formula 6 and algorithm 1 are different - formula 6 does not capture the fact that the \theta is updated only very rarely.
summary_of_the_review: The paper describes a relatively simple modification of the DQN algorithm using a proximal term. The approach improves the performance of DQN, but it does not improve on the current state of the art.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper applies proximal iteration, via L2 regularization from the weights of the target network, to deep Q-learning. Although the change in the algorithm is minimal, the authors demonstrate performance improvements across Atari environments as compared with baselines.
main_review: **Strong Points:**

The main strength of this method is in its relative simplicity relative to its empirical performance improvements. Incorporating an L2 weight regularization penalty into existing frameworks is relatively straightforward, as the authors emphasize in Algorithm 1 by modifying a single line. As shown in Figures 1 and S2, this simple change results in a fairly consistent and significant improvement in task performance.

In addition to the conceptual simplicity of the proposed method, its computational cost is also negligible, involving only a squared L2 norm between sets of weights.

While the proposed method introduces an additional hyperparameter, $c$, the authors note that it is relatively easy to tune, setting it to 0.2 for all experiments based on a hyperparameter sweep on four environments. Thus, it seems likely that this method will be fairly straightforward to apply to other environments and algorithmic setups.

The empirical analysis provides performance comparisons with relevant baselines over a substantial portion of the Atari 2600 benchmark, a standard set of environments for pixel-based deep RL with discrete actions. The authors directly compare with both DQN and DDQN, while also providing a comparison with the DQN implementation from the Dopamine codebase. One can conclude from this analysis that the proposed method generally improves performance in these settings. The authors also provide additional analyses, investigating the magnitude of updates, as well as sweeping over various update rates and techniques for DQN.

**Weak Points:**

The primary weak point of this paper is an overall lack of clarity regarding why the proposed method helps. I view this as equally as important, or more, as the empirical evaluation itself. That is, proposing a simple change to deep Q-learning methods may help improve current methods, but understanding why this change helps would improve or unlock a range of future methods, illuminating the root cause of the issue. Currently, the paper alludes to the ‘deadly triad,’ mentioning, but not explaining, “extreme volatility in the optimization procedure.” The authors go on to propose proximal iteration as a method for combatting such volatility, however, as the authors acknowledge, these intuitions are not necessarily applicable in the case of non-linear function approximators. Indeed, the deadly triad manifests in issues with estimating Q-values, e.g., optimism bias. Yet, with non-linear function approximators, proximal iteration in the space of weights provides no guarantees on the stability of Q-value outputs. To properly situate their method, the authors would need to explain a) what is unique about the “extreme volatility” that arises in the deadly triad, b) why it is appropriate to apply proximal iteration in the weight-space rather than the output space, e.g., Piche et al., 2021.


The proposed method is somewhat lacking in novelty, and the authors are missing a substantial number of citations of previous works that have investigated constraining parameters or outputs in value networks. Examples include natural gradient deep Q-learning (Knight & Lerner, 2018), PreQN (Achiam et al., 2019), KOVA (Di-Castro Shashua & Mannor, 2020), GAE (Schulman et al., 2015), etc. While the proposed method does not exactly match the setups in these previous works, the motivation and overall idea are highly similar. At the very least, a more in-depth discussion of related work is required. Further, given the similarity with these previous works, it would be helpful to compare with some other stabilization schemes.

The authors investigate their method entirely in the Atari domain. Accordingly, it’s unclear whether the hypothesized instabilities in Q-learning are more pronounced in these environments and whether the proposed method is generally applicable outside of this domain. While I do not see this as grounds for rejection, experiments in other domains would help to bolster the authors’ claims.

Likewise, because the authors only evaluate on Atari, their method is entirely evaluated on discrete control tasks, whereas Q-networks are also utilized in continuous control tasks. To truly demonstrate the generality of the problem and proposed solution, it would be helpful if the paper included at least some experiments with continuous control experiments, e.g, with actor-critic algorithms. The proposed method could also be applied for the purposes of state-value estimation.


- Piche et al., 2021. Beyond target networks: improving deep q-learning with functional regularization
- Knight & Lerner, 2018. Natural gradient deep q-learning
- Achiam et al., 2019. Towards characterizing divergence in deep q-learning
- Di-Castro Shashua & Mannor, 2020. Kalman meets Bellman: improving policy evaluation through value tracking
- Schulman et al., 2015. High-dimensional continuous control using generalized advantage estimation.


**Additional Comments:**

Some of the background in sections 2 and 3 feels overly pedagogical and/or not applicable to the setting used in the paper. For instance, it’s not clear what the discussion of the contraction mapping really adds to the paper. Similarly, much of the paper is devoted to discussing proximal methods in the context of strongly convex objectives, e.g., with linear functions, which is not the setting used in the paper. This space would be better utilized by characterizing the instabilities with Q-learning and explaining why proximal iteration in weight space is appropriate for tackling these instabilities.

Equation 6: I believe the last line shouldn’t be indented. Also, this feels somewhat redundant, given that it’s a simplified version of Algorithm 1.

Results: I would present the DDQN results in the main paper, perhaps right alongside the DQN results. It’s not clear how the 40 environments were selected from the total set of environments. The learning curves in Figure 2 are cherry-picked. It’s unclear what the update magnitudes tell us about the algorithms. These results need to be tied more specifically to the issues with deep Q-learning.

summary_of_the_review: My reasons are outlined in the weak points section above. While this paper provides a set of promising empirical results, the relative lack of explanation and analysis around why proximal iteration helps in deep Q-learning significantly detracts from the impact of this paper. I would like to see these aspects of the paper improved, perhaps drawing on previous works that have tackled this problem. Relatedly, given that other works have attempted to improve deep Q-learning via regularization, these works need to be cited and, ideally, compared against. Finally, it would help to expand the scope of the results section to include other environments or setups in which deep Q-learning is used, e.g., actor-critic algorithms, demonstrating the generality of the proposed method. In its current form, this paper will leave readers a) confused regarding what problems the proposed method is tackling, b) unaware of previous works in this area, and c) unsure about the method’s generality. For these reasons, I cannot recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission exploits *Proximal Iteration* to optimize DQN-type RL algorithms which ensures the online-network remains in the vicinity of the target network to stabilize the training process. This simple *trick* improves the performance of DQN and DDQN on a set of Atari-2600 environments.
main_review: **Strengths:**

This submission is well written and easy to read. The authors conduct extensive experiments to demonstrate the proposed regularization term in proximal iteration boost the performance of different DQN-type algorithms on a lot of Atari games.

**Weakness:**
1. My biggest concern is about the novelty of this submission. Although the regularization term in Equation 4 is effective for DQN-type algorithms, the authors did not give a convincing explanation to explain why proximal iteration works in this setting. In addition, the theoretical analysis in the main text (Remark 1 and 2) are all existing results, and the authors do not introduce new theoretical results to the community.
2. The authors propose to bias the online weights towards the target network, but no detailed explanation is given for this choice. A more natural option is to bias the online weights towards the old weights in the previous iteration, the authors should at least compare the performance of these two options.
3. In Equation 6, $\theta$ is updated every iteration, but in Algorithm 1, $\theta$ is updated only when numUpdates % period = 0.
4. The author just verified through experiments that DQNPro can be combined with DDQN. Since there are some contradictions between the functions of DQNPro and DDQN, is there a deeper explanation for the performance improvement?
5. Minor issue: It is better to redraw some figures in the experiment section using more professional software, Figures 2-5 look blurry.

summary_of_the_review: The authors use a simple trick to improve the performance of DQN-type algorithms on some Atari environments. Although the proposed method has achieved good results, the novelty is somewhat limited, and the reasons for the performance improvement are not explained clearly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper innovatively adds the proximal iteration to the update of the critic network, without the additional computational cost compared with the original SGD optimization method, it empirically demonstrates the advantage of this technique.
main_review: $\textbf{Strengths}$
The method proposed is simple and effective, although proximal iteration is common in RL algorithms, this is the first trial to apply it to the parameters of the critic networks. The authors theoretically analyze the advantage of proximal iteration. The experimental results are solid and the ablation study is carefully conducted. The overall writing and organization are excellent.

$\textbf{Weaknesses}$
The technical contribution is limited given what the authors show in the experiment part, both DQN and DDQN are old models, so I think the authors should try to address at least one of the following aspects.

(1) Apply the proposed technique to more advanced deep reinforcement learning models, like DDPG and SAC on continuous control problems.

(2) Analyze the source of improvement of other DQN variants, in terms of the magnitude of the updates to the target network similar to Figure 4, I think this will give a new perspective into the understanding of the critic network.
summary_of_the_review: This paper is innovative and well written, the experiment parts are very solid and the authors give a clear analysis. But I'm concerned about its technical contribution and conservatively rate it as around the borderline. I hope the authors could address or explain their thoughts about my concerns.
review_title: null

Meta-review: The paper applies proximal iteration to Q-learning, which significantly improves the performance of DQN. Reviewers agreed the paper is not ready for publication, for a couple reasons. DQN is quite far from current state-of-the-art. Improvements therefore need to be well-founded to be of broad interest. If the algorithm that is being improved is not competitive, there should be more general lessons that can be extracted from how and why the improvement works. Unfortunately, the reviewers felt that there was insufficient understanding of why proximal iteration helps.
Decision: Reject