Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Given that in Q-learning, the Bellman error (which we optimize) involves the target Q values with ever-changing parameters, two major approaches have been proposed to address the instability: (1) have a copy of a slightly old version of Q (i.e., periodically update the parameters to estimate target Q); (2) use a moving average of parameters. Learning is slow in both cases. For example for (1), suppose the Markov chain has N states and we update the target network once every H steps, then it would take NH steps for a sparse reward (right-most reward of the chain) to propagate to the beginning. Similarly, weight regularization in DQN is ineffective too. 

Therefore, the authors propose deep Q learning with functional regularization (FR DQN). See Eq. (5). 

Essentially, they replace the target network Q’ by the current network Q (and adding stop-grad); and they add the regularization term at the end, i.e., (Q - Q’)^2, so that the up-to-date parameter can be used in the Bellman error.

main_review: The approach is elegant and well-motivated.

Writing is clear.

The results in the four rooms task show good performance: compared to DQN and Polyak DQN, the proposed approach can recover the optimal value function quickly (given that the optimal value function in this task can be computed using tabular methods in this task). The proposed approach can match and sometimes get better than DQN / Polyak DQN on Atari tasks. 


Comments/concerns: 

More detailed analysis on how FR DQN performs on tasks with very sparse rewards would be interesting (perhaps also with a relatively large action space), given that this is one of the motivations. Will FR DQN actually perform better when there're sparse rewards? 

How does FR DQN compare to "increasing the update frequency of target nerworks"?

More principled / automatic ways of adjusting kappa in Eq. (5) would be interesting. The kappa value is likely a very important term. 

It’s not immediately clear to me why on some tasks, FR DQN results in a much larger standard deviation (seqquest).

It's not clear to me why FR DQN needs such a "target update period" shown in Table 1 in the appendix. 

Minor: page 4 typo ("sectionSection"), page 6 typo ("move between room-"). 
summary_of_the_review: The approach is well-motivated. The comparison to Polyak DQN is great. More experiments on complex tasks with relatively large action space, with sparse rewards will be great. A few minor concerns/questions in the main review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a functional regularization scheme to constrain updates to the Q-value estimates to get target networks removed. The paper is theoretically justified and puts the scheme into the context of delayed target networks and polyak-averaging (that both instead introduce a lag to the Q-values estimated by the neural network). The paper provides experiments on some Atari benchmarks against a DQN and DDQN that use polyak-averaging and delayed target network updates.
main_review: The paper is well-written and easy to follow along. I liked reading the paper as its motivation is clearly lined out from the beginning and the claims are also supported by a theoretical justification. Target networks often provide an easy solution to get an RL agent running but properly tuning their update frequency remains difficult in practice. This is also due to the fact that all their implications are largely still unknown. Functional regularization of the prediction of the network presents an easy solution, whose implication becomes clear through the provided theory. The proposed solution should also be (despite not being mentioned by the authors) much more computationally efficient than both approaches that use target networks and related approaches that try to remove the need for target networks. That being said, the authors could also highlight this a bit more.

From an experimental point of view, the used environments are sufficient. Atari serves well as a benchmarking environment and comparing the approach to DQN and DDQN also seems sufficient (first I wondered why the other improvements are not being used but those really should not affect each other, so it is better to keep it as simple as possible, DDQN makes sense; also, I wondered how results in DDPG should look like – however, the results should also hold here). Question: did you also use tuned DDQN hyperparameters from DQN Zoo for the experiments? This is not mentioned anywhere.

However, what I am missing a bit is an experimental benchmark against related approaches that have been proposed (e.g., Shao et al. (2020) and Kim et al. (2019) which have both been discussed as similar approaches in the paper). I don’t expect them to be much worse or much better either. The proposed functional regularization remains much more intuitive and tractable and also more computationally efficient (which at least for me would the killer-argument to used FR over other that trade performance in sample efficiency over ease of implementation).

Some minor comments:
-	p4: In practice, deep Q-learning methods are known to be unstable, [to] not approximate the true value function, and [to] sometimes even (soft-)diverge (Van Hasselt, 2010; Van Hasselt et al., 2016; 2018).
-	Sec. 4.1.2: for the experiments in Fig. 2(a): do other DQN-variants that use other target network update periods lie in between those two graphs (10 and 100)?
-	Sec. 4.1.2: \tau=0.05 initially outperforms \tau=0.01 […]. Does it reach the plateau in the end (looks like it will but not sure)
-	Sec. 4.2.1: “Complex DQL methods currently hold state-of-the-art results in this benchmark (Hessel et al., 2018). As previously mentioned, complex DQL methods show state-of-the-art results in this benchmark task.” – this seems redundant. 
-	Can you explain why in Sequest everything is different?
-	Fig 2, d) Success rate is quite high for different kappa (in [0.7, 1.0]), but still somewhat “sensible”. In the last part of Sec. 6 Discussion they claim “[…] the performance does not dramatically change for small changes in kappa […].”; Also: It would have been nice to see convergence speed for different values of kappa (as opposed to only success rate)
-	p4: Typo “sectionSection 3.2”

summary_of_the_review: The paper is easy to read and provides a significant contribution to a very widely known problem. The solution can easily be integrated in existing applications and frameworks and is hence of broad interest to the community. While I am missing a few experimental benchmarks against approaches that address the same problem but differently, in my opinion the paper provides sufficient experimental evidence to judge the advantages and limitations of the proposed approach.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new version of deep Q-learning, by avoiding the use of target networks in Bellman errors. Instead, the authors added a regularization term to enforce the current Q-value estimates close to its lagged version. Some analysis was provided to draw the connection with previous work on functional regularization and Polyak updating. Finally, the authors showed in experiments that the proposed method achieved better or similar performance, when compared with DQN and DDQN baselines.
main_review: Strength:
1. The idea of using up-to-date parameters in the target network looks interesting. It was believed that the target network needs to be periodically updated for stability, so this paper brings a new perspective on the target network in DQNs.

2. The authors showed empirically that the proposed method can outperform DQN baselines across a few different domains. The performance gap indeed shows the promise of the proposed method.

Weakness:
1. It's not convincing to me that the regularization term in Equation (5) can be interpreted as a KL divergence between two GPs. As the authors stated, "mean functions are the Q-values at different state-action pairs and whose covariance function is the identity". Given that Q_{\bar{\theta}} corresponds to a lagged version of Q_{\theta_t}, the assumption may not hold, since covariance functions are not necessarily the same and equal to the identity matrix.

2. As a part of the motivation, the authors mentioned a few times about the lack of propagating newly encountered rewards, due to the use of target networks. This statement is a bit problematic: Since rewards are randomly sampled from a replay buffer in DQNs, it's difficult to guarantee whether the used rewards can be newly encountered or not.  

3. The experiments did show the advantages over DQN and DDQN baselines. However, it lacks the comparison against a very related work as follows, which also removes the need for target networks in DQNs. A direct comparison could strength this work empirically.

     Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: removing the need for a target network in deep q-learning. In Proceedings of the Twenty Eighth International Joint Conference on Artificial Intelligence, 2019.
summary_of_the_review: This paper brings a new perspective for the use of target networks in DQNs, and the experiments show some promise. However, the interpretation on why it works has some flaws, and a more rigorous analysis could make it more convincing. Furthermore, the authors need to compare with the related work above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In Q-learning, a target network is often used to define the Bellman loss. The authors propose to estimate the Bellman loss with the “online” network and use the target network for functional regularization. In practice, functional regularization means minimizing the square difference between the target and the online network. Empirically, the authors show that this can improve performance over 7 Atari tasks and the four rooms environment.

main_review: Overall the paper is well written and easy to follow. The idea of using the target network for “functional regularization” is simple (in a good sense) and well-motivated. The novelty is somewhat limited, but for empirical papers I think performance is more important than novelty. The authors include code that strengthens their arguments. Given that the paper presents no theorems, I think the sections which mathematically compare the proposed methods to previous ones could be made a little smaller. At any rate, my main concern is the empirical evaluation.

## Four rooms

The four rooms environment is a simple toy environment. It is reasonable to use such an environment as a sanity check to e.g. show that some theoretical property holds. However, the proposed method lacks theoretical grounding in the form of theorems or guarantees and thus relies on empirical testing to demonstrate any virtues. I think that the four rooms environment is too simple for any empirical arguments, and that performance on it shows very little for an empirical method. 


## Atari

The first issue with the atari environment is the number of environments. The authors use 7 tasks considered by previous researchers and use 10 seeds for each of these. I think it would be better to use two seeds in each Atari environment (I think there are ~55).  This would not lead to statistically significant results on individual tasks, but I think it would lead to a better picture of the agents' performance once the authors average human-normalized scores across environments. Just using a few tasks raises the risk that the group of 7 tasks is cherry-picked (in the sense that there are many such groups to choose from) which undermines the results.

The second issue with the atari environment is the baselines. The DQN algorithm is over 5 years old at this time and has seen many improvements, so improving upon it does not advance the state-of-the-art. The authors state that “it is outside of the scope of this paper to combine FR DQN with each and every DQN improvement”. This is understandable, but what can be even more impactful is improving the performance of competitive methods. I would suggest that the authors consider ONE state-of-the-art baseline that is more competitive and show that their methods improve it. Suitable baselines might be spr (https://github.com/mila-iqia/spr) or rainbow (https://github.com/google/dopamine/tree/master/dopamine/agents/rainbow), both of which have excellent open-sourced code-bases. For such methods, the authors would also not need any parameter tuning.

summary_of_the_review: The paper presents a straightforward and well-motivated idea for using online q-values for calculating the bellman loss. The paper is well written and easy to follow. Since there are essentially no theoretical contributions, the empirical evaluation is important but unfortunately has several problems. The authors consider the toy environment of four rooms, which I believe is too simple. The authors also consider 7 Atari environments, I think it would be better to consider all environments from the Atari suite. Most importantly, for the Atari environments, the baselines considered are very simple, and improving upon them does not show that the state-of-the-art is being advanced.
review_title: null

Meta-review: This paper presents a simple, reasonable, alternative to target networks.  Given the effectiveness of target networks, and the fact that they are still somewhat poorly understood, this is a good topic for consideration.

It is unfortunate that the paper did not have more depth, in terms of analysis and/or analytical experiments that expose the properties of the suggested approach, and the mechanism still seems heuristic (and inspired by the success of target networks, and similar) more than principled.  That said, the proposed mechanism does seem somewhat effective (even if performance differences are not very pronounced), and is clean and simple to implement.

This version of the paper is rejected because we believe the paper could be a lot better than it currently is.  If the proposed regularisation mechanism is really as good as the authors argue, then it should be possible (and hopefully even easy) to demonstrate this clearly in more settings (e.g., in more algorithms).  Alternatively or additionally, the authors could consider digging deeper into the understanding of the method.  For instance, the paper often argues that target networks slow down learning, but (naively?) one could argue the exact same point (in general) for regularisation: this will trade off stability for speed.  It could be that the proposed mechanism is indeed a better way to achieve this trade off, but this is currently argued heuristically and not really proven (either theoretically, or with sufficient empirical evidence)
(For what it is worth, I personally did not find Section 3.2 particularly enlightening, because it is known these TD algorithms are not actually gradient algorithms, and hence considering 'losses' and 'gradients' in this way does not convince me we are getting at an actual deeper understanding of the dynamics of these algorithms.)

I wholeheartedly encourage the authors to take the comments and suggestions to heart and use these to improve the paper (as they have already started to do during this reviewing cycle), because I believe that there could be quite a good paper on this topic.  I hope the authors can convince themselves and their readers more convincingly that this idea is an actual, lasting contribution to the literature.  Ultimately, if they can, this will make the paper more impactful.  So although I appreciate this decision will come as a disappointment, I hope the authors also see this as an opportunity to make a larger research impact.

In particular, I would encourage considering: 1) comparing to our current theoretical understanding of target networks (see, e.g., [1]); 2) considering the effect of multi-step updates (shown in, e.g., [2] and [3] to be quite effective); and 3) considering whether the proposed approach (or a variation thereof) could be understood as a more fundamental idea: could this update be derived from first principles?

[1] Shangtong Zhang, Hengshuai Yao, Shimon Whiteson (2021). Breaking the Deadly Triad with a Target Network.

[2] Hessel et al. (2017). Rainbow: Combining Improvements in Deep Reinforcement Learning. 

[3] van Hasselt et al. (2018). Deep Reinforcement Learning and the Deadly Triad.
Decision: Reject