Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the convergence of a regularized Q learning algorithm and the features are learned from a nonlinear approximation. The features are updated much slower than the $Q$ function parameter. The authors prove that the proposed regularized Q-learning converges as long as the feature learning scheme converges and other conditions hold. Empirical results are provided to show that the proposed algorithm can converge while Q-learning diverges in some problems.

strength_and_weaknesses: I think the problem studied here is very interesting, especially the investigation of when Q learning diverges and how to remedy it. The experimental results are a good illustration of such a goal. However, the motivation for the particular regularization is not clear. There is no sufficient discussion on how the regularization is designed and what happens when the regularization deviates from 0. The theoretical analysis in this paper seems to be sloppy and does not address the convergence of the proposed algorithm as claimed.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and the presentation is clear except for some clarification on the motivation of the algorithm design. The theoretical novelty is limited given that the results are also much weaker though the authors try to solve a harder problem than existing work for Q-learning with linear function approximation.

summary_of_the_review: For Definition 1, it is unclear why there are two regularization parameters in the update. In particular, what is the role of $\xi$? It is clear that when $\xi=1$, the regularization disappears. But what happens with a very small or large $\xi$? Maybe the authors should motivate this regularization in the loss function before presenting it in the update rules. 

What is $\rho$ in Definition 1?

The result in Theorem 1 is somewhat weak since it only holds asymptotically. In practice, the convergence could be impacted heavily by the approximation of the feature learning at each step, which is now assumed to converge in Assumption 1. 

Therefore, the theoretical novelty is limited given that the results are also much weaker though the authors try to solve a harder problem than existing work for Q-learning with linear function approximation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new scheme of deep Q-learning by introducing final layer regularization and then updates it along faster time-scale than that of the non-linear features. Moreover, it provably converges even when the  features are non-stationary. A bound is also derived on the error introduced by regularization. 
strength_and_weaknesses: Strength:

1. The paper propose a novel scheme for deep Q-learning with non-stationary features and rigorously proved that it converges. 

2. Three settings are considered under which the features converge. 

3. Numerical results support the theoretical analysis. 

Weakness:

1. A lack of (empirical) verification of the obtained bound. 
2. Three settings with feature convergence are considered. How about other cases? It is suggested to add some discussions and/or limitations of the current results. 
clarity,_quality,_novelty_and_reproducibility: This paper is clearly written and the treatment of convergence of deep Q-learning with non-stationary features is novel. 
summary_of_the_review: Generally this is a good and rigorous paper on evaluating the convergence of deep Q-learning with non-stationary features. A new scheme is proposed and a following proof is given to show its convergence under some assumptions. My only concern is a lack of discussions of the generalization results for other settings apart from the three considered in current paper, i.e., a discussion of limitations and useful insights are suggested.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies Q-learning with neural network approximation and proposes a two-layer regularized Q-learning algorithm. The algorithm can be viewed as an extension of Q-learning with linear function approximation, where an additional layer is used to learn the features. The authors show asymptotic convergence of the proposed algorithm using the ODE approach. Numerical simulations were performed on a well-known divergent example of vanilla Q-learning, on which the proposed algorithm is stable.
strength_and_weaknesses: While the writing and the exposition are clear, there are many limitations of this work, as elaborated below.

(1) Compared to using linear function approximation, the two-layer neural network approximation enables the agent to also update the features. Therefore, one should expect the function approximation error to be smaller than simply using linear function approximation. In my point of view, this is the main advantage of using two-layer neural network, and is the motivation of this work. However, in view of Theorem 2, this advantage is not theoretically clear. In particular, it is not clear if the function approximation error $||Q^*-\Phi_{u^*}Q^*||$ is smaller compared to using linear function approximation where the features are fixed.

(2) This paper follows the regularization framework proposed in (Zhang et al. 2021) and (Lim et al. 2022). However, it was discussed in a recent work (Chen et al. 2022) Appendix E that the regularization technique proposed in (Zhang et al. 2021) is to stabilize Q-learning by changing the problem discount factor to a smaller one. This is the reason that their algorithm does not converge to $Q^*$ even in the tabular setting. Similar issue is present in (Lim et al. 2022) too. Since this paper follows that regularization framework, it has the same issue. To see this, the second and the third term on the right-hand side of Theorem 2 Eq. (4) do not vanish even in the tabular setting, suggesting that the MDP problem is no longer the original one. While this is a limitation in previous papers, since this paper follows the same framework, the validity of the algorithm design is unclear.

(3) As a follow-up concern to (2), since introducing the regularization parameter is equivalent to reducing the MDP discount factor, and it is well-known that small discount factor leads to convergence of Q-learning with linear function approximation, the numerical simulations are not surprising. Also, since DQN (Mnih et al., 2015) is known to be successful, and the authors propose a different DQN scheme, it may be a good idea to compare the empirical performance of the proposed algorithm to that of DQN (Mnih et al., 2015).

(4) Both the regularization parameters $\zeta$ and $\epsilon$, and the projection radius $\rho$ depend on the unknown parameter $\sigma$. Therefore, it is unclear how to implement this algorithm in practice.

(5) The technical contribution is incremental because (i) using regularization to overcome divergence is readily established in the literature, (ii) the convergence of the slower time-scale follows from existing literature on SGD, and (iii) the convergence of the faster time-scale follows from the ODE framework.



clarity,_quality,_novelty_and_reproducibility: The paper is well written, but the results are not novel and seem to be a combination of several existing papers (Zhang et al. 2021), (Lim et al. 2022), (Mertikopoulos et al. 2020), and (Borkar 2008).
summary_of_the_review: Since the designed algorithm (which can be viewed as an extension to (Zhang et al. 2021), (Lim et al. 2022)) suffers from the same limitations as in (Zhang et al. 2021), (Lim et al. 2022), and the technical contribution is incremental, I do not recommend this paper being accepted by ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewer seem to reach a consensus that the paper is not ready to be published in neurips. In particular, the reviewers raised following concerns. 

--- "The theoretical novelty is limited given that the results are also much weaker though the authors try to solve a harder problem than existing work for Q-learning with linear function approximation."

---"A lack of (empirical) verification of the obtained bound."

---"The technical contribution is incremental because (i) using regularization to overcome divergence is readily established in the literature, (ii) the convergence of the slower time-scale follows from existing literature on SGD, and (iii) the convergence of the faster time-scale follows from the ODE framework."

---"Three settings with feature convergence are considered. How about other cases? It is suggested to add some discussions and/or limitations of the current results."
justification_for_why_not_higher_score: The reviewer seem to reach a consensus that the paper is not ready to be published in neurips. In particular, the reviewers raised following concerns. 

--- "The theoretical novelty is limited given that the results are also much weaker though the authors try to solve a harder problem than existing work for Q-learning with linear function approximation."

---"A lack of (empirical) verification of the obtained bound."

---"The technical contribution is incremental because (i) using regularization to overcome divergence is readily established in the literature, (ii) the convergence of the slower time-scale follows from existing literature on SGD, and (iii) the convergence of the faster time-scale follows from the ODE framework."

---"Three settings with feature convergence are considered. How about other cases? It is suggested to add some discussions and/or limitations of the current results."
Decision: Reject