Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning

Abstract: We identify an implicit under-parameterization phenomenon in value-based deep RL methods that use bootstrapping: when value functions, approximated using deep neural networks, are trained with gradient descent using iterated regression onto target values generated by previous instances of the value network, more gradient updates decrease the expressivity of the current value network. We char- acterize this loss of expressivity via a drop in the rank of the learned value net- work features, and show that this typically corresponds to a performance drop. We demonstrate this phenomenon on Atari and Gym benchmarks, in both offline and online RL settings. We formally analyze this phenomenon and show that it results from a pathological interaction between bootstrapping and gradient-based optimization. We further show that mitigating implicit under-parameterization by controlling rank collapse can improve performance.

Reviews and author responses:

###Summary:

This paper identifies a type of implicit under-parameterization phenomenon in deep RL methods that use bootstrapping.  It is found that after an initial learning period, the effective rank of the feature matrix keeps decreasing. This implies that the representational power of the network is not fully utilized. The authors call it a type of implicit under-parameterization.  Moreover, the emergence of this under-parameterization strongly correlates with the poor performance.  Some preliminary theoretical analyses are provided to explain this phenomenon. 

###Pros:

The paper is well written, and I can follow the idea very smoothly. The implicit under-parameterization phenomenon seems very intriguing and useful for designing better bootstrapping-based deep RL methods.  The theoretical analyses are very illustrative though still preliminary. 

###Cons:

1. The analysis in Section 4.1 seems not correct to me. For kernel regression,  the implicit bias of gradient descent with an infinite number of iterations is to pick up the solutions with the smallest RKHS norm. This implies  $c=0$ in Eq. (1), which would make the subsequent analysis problematic.  However, if early stopping is applied, the GD solutions will be equal to the one given in Eq. (1) with $c>0$. The value of $c$ depends on how the early stopping is applied. Please refer to [1] for more details. 


###Other comments:

1. The analysis in Section 4.1  is very illustrative due to the use of the kernel model.  But all the experiments are done for neural networks. It would be much helpful if some extra experiments with kernel models can be added, for which we can directly compare the experimental results and the theoretical analysis. 




[1] Suggala, Arun, Adarsh Prasad, and Pradeep K. Ravikumar. "Connecting optimization and regularization paths." Advances in Neural Information Processing Systems. 2018.


### Post-rebuttal Comments
I thank the authors for the response and the efforts to update the drafts. I  think this submission made some original contributions. 

Post discussion review
-----
-----

# Summary

The authors present evidence that the approximate rank of the features is correlated with the learned policy's performance and that this rank shrinks when using bootstrapping. They provide empirical evidence in several RL settings and domains and present some theoretical arguments which explain this behavior in the context of kernel regression and deep linear networks. Finally, they propose a simple approach for mitigating the rank collapse and show that this improves the performance of the learned policy in some cases.

# Reason for score

The authors isolate an interesting phenomenon and present some compelling empirical evidence. This is interesting work and I have have no doubt that it is of sufficient quality for publications.

# Pros

* The main contributions of this work might help us better understand the effects of using bootstrapping with function approximation and gradient descent, a critical aspect of many RL methods. Using neural nets to learn (Q-)value functions on novel domains is still to this day a frustrating experience due to how unstable and unpredictable gradient descent + bootstrapping is. As a result, this subject of this work is quite important and likely of great interest to the field.
* The experiments are well designed and relevant to the main thesis. The empirical results are well presented and easy to understand.

# Cons

* After a very productive and enlightening discussion with the authors, the only noteworthy issue is that this paper contains too many contributions for the format making some of them hard to appreciate. A more focused in depth dive into a subset of the theoretical contributions might have been preferable and possibly provide more insight.

# Conclusion

I strongly support the acceptance of this submission. After discussion with the authors and resulting updates to the paper, I don't see any reason for rejecting this paper. All of the major concerns from my initial review have been addressed.


Initial review
-----
-----
# Summary

The authors present evidence that the approximate rank of the features is correlated with the learned policy's performance and that this rank shrinks when using bootstrapping. They provide empirical evidence in several RL settings and domains and present some theoretical arguments which explain this behavior in the context of kernel regression and deep linear networks. Finally, they propose a simple approach for mitigating the rank collapse and show that this improves the performance of the learned policy in some cases.

# Reason for score

Although the authors isolate an interesting phenomenon and present some compelling empirical evidence, I have a few concerns about the theoretical contributions which, hopefully, the authors can address or clarify any misunderstanding. This is interesting work and I am more than willing to adjust my review if the authors can assuage my concerns.

# Pros

* The main contributions of this work might help us better understand the effects of using bootstrapping with function approximation and gradient descent, a critical aspect of many RL methods. Using neural nets to learn (Q-)value functions on novel domains is still to this day a frustrating experience due to how unstable and unpredictable gradient descent + bootstrapping is. As a result, this subject of this work is quite important and likely of great interest to the field.
* The experiments are well designed and relevant to the main thesis. The empirical results are well presented and easy to understand.

# Cons

* This did not feel like an 8 page paper. This paper took a long time to review. With 18 pages of appendix, 9 of which are clarifications and proofs, what is left of the theoretical contributions in the main body of the paper doesn't provide much insight into the role/importance of the assumptions or into what makes each claim true.
* The proof for theorem 4.2 appears to make use of the assumption that $\varepsilon(s, a) = W_N \cdot \zeta [s; a]$ and $y_k = Q_{k-1} + \varepsilon$. This is not conveyed in the main body of the paper but seems to be a fairly strong assumption on the form of the bootstrapped targets.
* Similarly, I would argue that the premise that the bootstrapped targets will eventually be close to the previous, i.e., $y_k \approx Q_{k-1}$, is flawed. There is no guarantee that applying the Bellman operator will return a function that is inside your function class, even in the linear case. Furthermore, we know this phenomenon to be significant and motivated work on the projected Bellman error, a concept heavily used by the various variants of gradient temporal difference learning.
* In theorem 4.1, the assumption that $S$ is a normal matrix seems impractical and likely makes this result only applicable to very rare cases.
* In proposition 4.1. it isn't immediately apparent where in the proof the assumption that the loss $L$ is the TD loss is leveraged. If it isn't used, this would suggest that this is a general property of deep linear networks and wouldn't support the authors observations that the rank issues are specific to bootstrapping.

# Questions for the authors

1. Was anything done to "normalize" the results in figure 2 to account for the differing number of total updates as a result of different n? Can these observations be explained by the fact that more updates results in the parameters traveling further from their initial values? What happens when plotting the srank vs. # of updates in the setting? (These likely don't need 3 distinct answers)
2. Could the authors elaborate on why the normal matrix assumption might be reasonable, or, otherwise, explain why this doesn't make it a vacuous result?
3. What is the purpose of the "explaining implicit under-parameterization across fitting iterations" section? I think I am missing the insight this is trying to provide. Why would the parameters change at all if I reuse the results of the previous minimization as targets? What does the Bellman error refer to here and what does it mean to attain zero (or any value) TD error when the targets are just the Q-values?
4. The balanced assumption used with the deep linear networks seems critical for the proof. Is my assessment correct or could these results possibly hold without it? How does this assumption limit the applicability of the insight gained here to more practical neural networks?

# Misc comments and typos
* page 2, Yang et al. don't seem to use the term "effective rank", but do use the term "approximate rank".
* page 4, "we first remove the confounding **with** issues [...]"
* page 30, proof, it would help to explicit state the dimensions of $\zeta$. Is the $\top$ on $\zeta^\top$ a typo? Otherwise, why is it not used further down?  (no need to answer either way, just reporting on something that tripped me up)

Based on reviewers’ comments, we have clarified the theoretical section and added some new experiments. Changes in the revised paper are shown in blue and clarifications are highlighted in red. We summarize the major changes below: 

1. **Updated Writing in Section 4** and added high-level summaries for the proofs, justification, and the significance of the assumptions (normality of S in Theorem 4.1 *[Reviewers 1, 3]* , the value of c in Theorem 4.1 *[Reviewer 4]* , analysis close to a fixed point in Theorem 4.2 *[Reviewers 2, 3]*, balancedness *[Reviewers 1, 3]*) in more detail. 

2. **Re-written the paragraph “Explaining implicit under-parameterization across fitting iterations”** and changed the title to improve its exposition *[Reviewer 3]*. 

3. **Discussed extension of Theorem 4.1 for non-normal matrices** using a notion of effective rank based on the norm of eigenvalues (Appendix C). We have also added experiments on Atari and gridworlds to empirically justify normality by showing near equality of singular values and norm of eigenvalues (Appendix A.7) *[Reviewer 3]*.

4. **Clarified the consequence/implications of Theorem 4.2** that rank drop near a TD fixed point can prevent convergence to the fixed point and added an experiment (Figure 4) on gridworlds with deep linear nets to validate this consequence empirically *[Reviewers 2, 3]*.

5. **Added plots** for normalized results of rank vs the number of updates in Appendix A.8 *[Reviewer 3]* and zoomed-in runs for offline RL on Atari in Figure A.1 *[Reviewer 2]*.

6. **Added experiment** that demonstrates that rank collapse is non-existent (in fact, rank increased in our experiment with more training) with “true targets” (Appendix A.3 and Figure A.11a) *[Reviewer 3]*

7. **Supplemented Theorem 4.2** with discussion in Appendix D.2 that highlights how one-step rank drop compounds because of the sequential bootstrapping procedure. *[Reviewer 3]*

8. **Added clarification** and discussion for $\lambda_k$. *[Reviewer 3]*

9. **Added preliminary results** for adding Rainbow components to DQN with penalty (Appendix A.6.2) *[Reviewer 1]*

We have also corrected typos and notation issues. *[Reviewers 1, 3]*

We would appreciate if the reviewers can please take a look at these changes and let us know if they have any other concerns.

I would like to thank the authors for their reply and for addressing my concern regarding normality assumption in theorem 4.1. I also enjoyed the discussion between the authors and Reviewer3. I would recommend accepting the paper.

We thank the reviewer for the elaborate discussions and the score update. Per reviewer's suggestion, we have addressed their comment about $\lambda_k$ in the revised paper (around Equation 5) and added a discussion to Appendix D.1 in magenta color. 


Thank you for the interesting submission and follow up discussion. I've updated my score under the assumption that the $\lambda_k$ issue I've raised will be addressed before the end of this discussion stage. I will be adding an updated review later during the final reviewer/AC discussion period.

The authors have provided some convincing evidence that the phenomenon they've observed is in fact only present when bootstrapping is used. At this point, I am convinced that the the authors have identified a real cause and effect, where regressing on bootstrapped targets leads to a significant preference towards rank deficient features. I don't see any major issues left to address and I believe this paper is worth publishing.

## $\lambda_k$

Is this formally introduced in the paper? Section or theorem 4.2 should state the assumptions about $\lambda_k$ explicitly stated, e.g., what range values can it take, along with any assumptions required for its existence.

## Current thoughts about the strengths and weaknesses of the last draft

* The empirical results are thorough and very convincing. Even without any formal theory, there is enough there to support the main thesis and warrant publication.
* I feel like the theory could be distilled and its presentation possibly made more "efficient". To be clear, I think that in its current state, it is good enough for publication and shouldn't get in the way of acceptance.  I hope the authors will forgive this arguably unconstructive comment, but I can't quite precisely identify why I believe this to be the case. This could just be a consequence of how long the paper effectively is. Maybe some of it should be removed and published separately (e.g., kernel NN stuff or linear NN stuff)? Is this something the authors would be willing to entertain? I don't want to coerce the authors into presenting their work in way they don't agree with. The intent of this last suggestion is only to try and make this paper as good as possible, and it's okay if the authors disagree. (Also, note that before making a change like that, we should try and get some input from the other reviewers)


As a small tangent, in the future, the authors might want to consider making a longer journal version of this work so that each contribution can be given enough space to convey the intuition the authors try to explain here. I think they have isolated something interesting and I suspect there would be interest for a more "complete" presentation of this work.

Thanks for the prompt responses and for engaging in the discussion. We are glad to hear that the reviewer now has a positive assessment of the paper, and would appreciate it if they could update the score to reflect the discussion and their assessment.

Thank you for the constructive feedback. Please let us know if our response below adresses your concerns about the paper. We will be happy to clarify any other concerns/questions that may be remaining.

The remaining answers to specific questions are shown below:

**“The assumption that $\varepsilon(s, a) = W_N (k - 1, T) \zeta [s; a]$ should be in the main body of the paper.”**

Thanks for pointing this out. We’ve fixed it now. 

**Re: Suggala et al 2018 “... Could the authors clarify why conclusions about the behavior of (5) can be transferred to the original optimization formulation?”**

Our intent was to highlight that the solution obtained from the original optimization formulation will have similar properties to the solution of the effective-rank regularized objective (5). Intuitively, the justification for this is that larger singular values grow much faster than small singular values which dictates how the ratio of singular values evolves through time, thereby reducing effective rank. That said, the optimization path for the original problem may not correspond to the rank-regularized objective, a detail we now explicitly mention in the revision to remove any confusion. 

We hope that our responses help resolve the uncertainty surrounding the hypothesis of this paper. Please let us know if there are any other concerns remaining. 

We thank the reviewer for their prompt response and helpful comments.  To address the reviewer’s new concerns, we have updated the paper in purple (to demarcate from previous edits) and: 

(1) Added a new experiment that demonstrates that rank collapse is non-existent (in fact, rank increased in our experiment with more training) with “true targets” (**Appendix A.3** and **Figure A.11a**)

(2) Supplemented Theorem 4.2 with discussion in **Appendix D.2** that highlights how one-step rank drop compounds because of the sequential bootstrapping procedure.

We begin by responding to the reviewer’s high-level thoughts, and then specific comments.

**Linear NN induce solutions with lower rank, … At its core, the argument is the repetition of the optimization loop and that there are non-zero errors.** 
  
We want to clarify that simply doing repeated regression with non-zero errors -- when targets are not computed from the current Q-estimates -- is not sufficient to induce the large rank drop (or collapse) that we observed in value-based RL. Instead, the reason for large rank drop is that FQI uses low rank solutions (preferred by linear NNs), obtained in the previous iteration as “targets” to compute the current loss for training the Q-function, further compounding the rank drop.

We verified the above hypothesis by running the experiment suggested by the reviewer, where we perform repeated regression with true targets (which leads to non-zero errors) and demonstrate that obtained solutions do not show rank deficiency (Figure A.11a). We also added analysis in Appendix D.2 to highlight this compounding effect and updated Section 4.2.  Our bound suggests that the effective rank is affected by the compounding of TD error since previous Q-estimates are used as targets (Equation D.11). 

**(1) Theoretical results involving the Bellman or TD error... errors are propagated...doesn't seem to be the case**

To address the reviewer’s concern, we have supplemented Section 4.2 with a discussion in Appendix D.2 that shows how the recursive structure of dynamic programming can cause rank decrease in previous iterates to compound in future fitting iterations, assuming closedness under the Bellman update, a common assumption in fitted-Q analyses (Munos and Szepesvari, 2008; Chen and Jiang, 2019).

Intuitively, using a previous solution $Q_k$ to form the target $y_{k+1}$ propagates the rank versus TD error trade-off in (5) from one fitting iteration to the next, and therefore every subsequent iteration. Our analysis in Appendix D.2 shows that the rank drop after several iterations $srank_\delta(W_\phi(k)) - srank_\delta(W_\phi(0))$ depends (amongst other terms) on the sum of the fitting errors across iterations, weighted by a term that reflects the strength of rank-regularization for each iteration: $\sum_{j=1}^k \frac{||Q_j - y_j||}{\lambda_j}$, which is reminiscent of error propagation in approximate dynamic programming. As a result, fitting errors in past iterations compound additively, potentially forcing larger rank decreases as more iterations are completed. 

$$srank_\delta(W_\phi(k)) \leq srank_\delta (W_\phi(0)) + \sum_{j=1}^k c_j - \sum_{j=1}^k \frac{||Q_j - y_j||}{\lambda_j}$$

Our result also depends on  change in rank ($c_k$) due to the reward and dynamics transformations used to compute the target values at the $k^{th}$ iteration -- tighter control on this change at any iteration results in tighter lower bounds on rank drop. Without any more assumptions, it is hard to analyze $c_k$ theoretically. However, we discuss two scenarios for $c_k$:

(i)  Empirically, on Atari games (Figure D.2), we show that the rank of the target features due to dynamics transformation does not change much within any fitting iteration, implying that the compounding effect encourages lower ranks with more fitting iterations, as observed.

(ii) Theoretically, Theorem 4.2 analyzes the scenario when the current Q-function is close to the optimum, then $c_k$ is small (Equation D.16, in the proof of Theorem 4.2).

**(2) "The non-stationary policy causes the targets to change and, leads to non-zero errors. Would using true targets for current policy lead to similar rank deficient preference?”**

To test the reviewer’s hypothesis, we ran an experiment in the control setting for the Gridworld environment, regressing to the target computed using the true value function $Q^\pi$ for the current policy $\pi$ instead of using the bootstrap TD estimate. The true value function $Q^\pi$ is computed via tabular policy evaluation. The results, shown in Figure A.11a, demonstrate that regressing to true targets increases effective rank with more training. This is distinct from Figure 5(a) (Standard FQI), when effective rank deteriorates with bootstrapping. These results indicate that bootstrapping plays a more significant role in causing rank drop compared to non-stationarity. 

Remaining answers in the next post.

First, I want to commend the authors for a very informative response as well as quality changes to the paper. These edits make it noticeably easier to appreciate the main contributions and, in my eyes, have made this submission much stronger. The authors have addressed a good portion of my concerns and I think it possible that further discussion will help clarify what remains. The authors can consider any of my previous concerns not discussed below as addressed.

# Assumptions for Theorem 4.2

1. The assumption that $\varepsilon(s, a) = W_N (k - 1, T) \zeta [s; a]$ seems quite impactful and, as far as I can tell, it plays an important role in the proof. This assumption really should be in the main body of the paper.  To be clear, this specific point isn't about whether the assumption is reasonable or not, just that it since it constrains the settings where the proof holds, it should be part of the theorem statement.
2. It would make the result stronger if the authors could relax this assumption and base the proof only on the $|\varepsilon(s, a)| \ll |Q_{k−1}(s, a)|$ assumption found in the current theorem statement. Would it be possible to bound $|\varepsilon(s, a) - W_N (k - 1, T) \zeta [s; a]|$ in such a way that the rest of the proof could be reused, possibly with the current version as a special case if this would requires looser bounds?  I have no doubt that the authors have spent considerably more time thinking about this than I have so I look forward to hearing if they think this is feasible.

# Rank drop within a fitting iteration “compounds” due to bootstrapping

As I understand it, the comparison between the continuous optimization dynamics and the regularization formulation in Suggala et al. (2018) is insightful because the authors of that paper bound the difference between these two formulations. It is not obvious to me if such a bound exists for the srank in the context of this submission. Could the authors clarify why conclusions about the behavior of (5) can be transferred to the original optimization formulation?

# Elaborating on my (reviewer 3) high-level thoughts

I find the empirical results to be very convincing and the hypothesis put forth compelling. I think my biggest issue lays with how the theoretical results seems to suggest a different story. I'm concerned that the theory might be hinting at a different phenomenon than the one from the main claim, that is, that bootstrapping induces rank deficient features. Before elaborating on this, I will try to summarize my takeaway from the theoretical results from section 4.2 to allow the authors to clarify or correct any misconceptions I might have.

The results for the linear NN highlight an induced preference for solutions with lower rank, regardless of the loss, so long as the value of the loss for the current solution is non-zero. Since fitted-Q iteration (FQI) involves doing regressions several times on a non-zero loss, the behavior of drifting towards lower ranks is induced several times. At its core, the argument is based off of the repetition of the optimization loop and on the fact that there is non-zero errors.

While this could be related to bootstrapping as the authors argue, I two concerns: 1) the dynamics of the bellman error is never used, only the fact that there is a non-zero error is needed, 2) connections presented in this work all look at a mix of policy search and policy evaluation.

1. Theoretical results involving the Bellman or TD error typically capture, in one way or another, the fact that errors are propagated through the recursive structure of the error's definition. This doesn't seem to be the case in this work where only the dynamic programing nature of the algorithm considered is necessary to argue for non-zero errors. In of itself, this doesn't invalidate the result but it does raise some concerns that the hypothesis might not be the "right" one.
2. Regardless of bootstrapping, the non-stationary policy causes the targets to change and, as a consequence, leads to non-zero errors. This begs the question: would using the true targets for the current policy lead to similar rank deficient preference as observed here? Admittedly, this could be somewhat tricky to investigate since this behavior is likely sensitive to the properties of the environment and it is often difficult to generate the "true" targets in the typical benchmark domains. 

Together, these two issues make me uncertain that the hypothesis of this paper is correct, despite no obvious flaws with the provided empirical and theoretical results.


We thank the reviewer for their constructive and positive feedback on this paper. We have updated the paper to add a discussion on normality of $\mathbf{S}$ in Theorem 4.1 and Appendix C, clarified the assumptions in Section 4.2, and corrected typos in the paper. We answer the specific questions below. 

**How restrictive is the assumption that S is a normal matrix in theorem 4.1?** 
Normality is not a very restrictive assumption: In practice, we found that on gridworlds (Figure A.20) and five Atari games (Figure A.21), a different notion of effective rank computed using norms of eigenvalues of S is roughly equal to the effective rank, $srank_\delta(\mathbf{S})$ computed using singular values. This new notion of effective rank, denoted as $srank_{\delta, \lambda}(\mathbf{S})$, still enjoys an analogous version of Theorem 4.1 *for any* matrix $\mathbf{S}$, even if it is non-normal, as now discussed in Section 4.1. Thus, Theorem 4.1 does explain the phenomenon of rank collapse under not so restrictive assumptions. Furthermore, when $\mathbf{S}$ is approximately normal, a weakened version of Theorem 4.1 still holds, which we now elaborate on at the end of Appendix C.

**Is it possible to extend theoretical analysis to when Bellman optimality operator is used instead of Bellman operator?** 
For the analysis in Section 4.1, we suspect that we will need additional assumptions on the alignment of the eigenspaces of the $P^{\pi_i}$ matrices generated by each of the greedy policy iterates $\pi_i$ for the Q-function, which are used for the optimality backups. Our analysis in Section 4.2 which analyzes the rank drop near a fixed point still applies to the Bellman optimality backup. We have added a discussion stating that the extension to the Bellman optimality operator is certainly interesting for future work.
 
**Figure 3 (d) contains one red trajectory, for which srank does not collapse. Could you please comment what special properties this trajectory has that srank stays almost the same?**
Inspecting individual runs for this figure, we observe that the TD error is the smallest of all 5 runs for the run that does not suffer from rank collapse. Quantitatively, TD error for this run is *one-third* of that of the run that attains the next largest rank. For the other runs where rank collapses, TD error takes values in the range of $10\text{x}- 100\text{x}$ times the TD-error for the highest rank run. This is consistent with our results that there can be a tradeoff between training the Q-function and rank collapse.

**All except the last-layer weights share singular values (a.k.a. “balancedness”)." According to Appendix the stronger assumption $W_jW_j^T = W_{j+1}^TW_{j+1}$ is required**
The assumption in Equation D.4 is the balancedness assumption, and we now indicate in the revised main paper that we require the assumption on the weight matrices. This condition allows us to relate singular values of consecutive weights matrices, as was discussed previously for simplicity. We also note that this assumption is weaker than the balancedness assumption used in Arora et al. (2018; 2019), which will trivially give rank 1 features in our setting.

**Typos: Equation (C.12), Legend in Figure 3(a), $L_0$ in Equation D.4 - D.5** 
We have revised the paper to fix both these typos and the change to Equation C.12 is highlighted in red. This was a typo, thanks for pointing this out. We have fixed this typo by replacing $L_0$ with $L_k$ which stands for the TD error at fitting iteration $k$.

**Rainbow performance increased, while DQN performance decreased in online settings. What is the key underlying component that leads to different results?**
To provide a concrete answer, we have launched a set of experiments that ablate the key differences between Rainbow and DQN with the penalty:  a distributional DQN loss, prioritized experience replay,  n-step returns. This experiment will take around 2 weeks to run, so we will update the final manuscript with the new analysis. 

Our current hypothesis is that this is a consequence of the feedback loop between learning and data generation in online RL (Schaul et al, 2019 [1]). Namely, at the onset of training, DQN learns slower because of the added penalty, causing it to collect poorer data, which in turn makes the agent continue to learn slower (as compared to without the penalty). Rainbow typically learns quickly and collects more diverse data compared to DQN. Improving rank via the penalty can help fit this quickly changing data more effectively for Rainbow, thereby improving performance. Note that in offline RL, where the quality of data trained on is not tied to the intermediary agent performance, DQN does improve with the penalty (Figure 6).

[1] Schaul, Tom, et al. "Ray interference: a source of plateaus in deep reinforcement learning." arXiv preprint arXiv:1904.11455 (2019).

**Summary**: We thank the reviewer for constructive comments on the paper. We have added a discussion of the reference pointed out by the reviewer in Section 4.1 and clarified the concern about the correctness of the theorem. 

**“The analysis in Section 4.1 seems not correct to me ... The value of $c$ depends on how the early stopping is applied. Please refer to [1] for more details.”**
Thank you for suggesting the reference. The choice of $c$, which controls the strength of the regularizer, is directly based on the work from Mobahi et al. 2020, independent of RL and bootstrapping. As the reviewer pointed out the connection between optimization and regularization paths, the theory of self-distillation in Mobahi et al. 2020 or kernel regression in Section 4.1 with $c > 0$ explains the learning dynamics of gradient descent with early stopping in the presence of strongly convex or separable losses. With this choice of $c > 0$, our theorem in Section 4.1basically explains the presence of the rank collapse phenomenon when gradient descent with early stopping is used to optimize the TD error in Fitted Q-iteration.
We have added this connection to Section 4.1, and made it clear that the choice of $c$ is directly taken from self-distillation and a value of $c > 0$ is optimal which matches the practical setting where early stopping or only a few gradient steps are used for training. 


We answer the remaining questions in this post.

**“What is the purpose of the "explaining implicit under-parameterization across fitting iterations" section?”** 
The goal of this section is to show that the implicit rank drop effect from supervised learning is exacerbated when bootstrap targets are used as labels for the next iteration. To better clarify its goals, we have changed the title of this subsection to “Rank drop within fitting iteration compounds due to bootstrapping” and added more discussion.

**Normalize Figure 2 by updates (“What happens when plotting the srank vs. # of updates in the setting?”)**
We have added a new plot in Appendix A.8 (Figure A.22) that plots the ranks as a function of the number of gradient updates in Figure 2 on Atari domains, which still shows that a larger value of n (see n=4 vs n=8) leads to lower values of srank. This means that performing gradient-based training of bootstrapped objectives leads to a lower effective rank in the data-efficient setting (i.e., when n = number of environment steps per gradient updates is larger) as compared to a regular RL setting (n=1). 

**“Can these observations be explained by the fact that more updates result in the parameters traveling further from their initial values?”** 
We are not aware of prior work that connects rank regularization in deep learning to the distance travelled by the parameter update during training. If the reviewer has suggestions, we are happy to discuss those connections in our paper. 

**Summary**: We thank the reviewer for constructive comments. We have substantially updated the writing in Section 4.2, added high-level summaries for the proofs, and justify assumptions (normality of $\mathbf{S}$, analysis close to a fixed point) in more detail. We have re-written the paragraph “Explaining implicit under-parameterization across fitting iterations” to improve its exposition and we have added experiments to empirically justify normality and also plot normalized results for Figure 2. We will be happy to make more changes if the reviewer has any further suggestions. We respond to specific queries next:

**Assumptions for Theorem 4.2, “Bellman targets will be eventually close to the previous Q-values ..”** 
We have highlighted the assumptions in Section 4.2 and improved the theorem statement. The goal of Theorem 4.2 is to show that even when the Q-function is relatively closer to a fixed point of the Bellman equation, a drop in rank may happen and as we now show on a linear network in Figure 4, this can prevent convergence to this fixed point by virtue of pushing the Q-function faster from $Q^\*$. 

**“The proof for Theorem 4.2 makes a strong assumption on the form of the bootstrapped targets”** 
 Typically, an initialization close to the fixed point is a favorable condition for convergence to the optimal Q-function. If a method fails to converge to the optimum starting from this initialization, it is unlikely that the method will perform well more generally. We also emphasize that analyzing dynamics near a fixed point is a standard technique in non-convex analysis (e.g., linearizing the function near the optimum), analysis of ODEs (Borkar and Meyn, 2000), and optimal control (e.g., local analysis of the Jacobian eigenspectrum).

**Why is normality of S in Theorem 4.1 reasonable?**
 Without normality, a result analogous to Theorem 4.1 can be derived using an alternate notion of effective rank that utilizes eigenvalues instead of singular values: $$srank_{\delta, \lambda}(M_k) = \min~ \Big[ k: \frac{\sum_{i=1}^k |\lambda_{i}(M_k)|}{\sum_{i=1}^{d} |\lambda_i(M_k)|} \geq 1 - \delta \Big].$$The normality assumption allows us to associate this eigenvalue-based rank to the practical singular-value based effective rank. When the matrix $\mathbf{S}$ is not perfectly normal, but only approximately so, a weakened version of Theorem 4.1 holds (end of Appendix C). We have updated Section 4.1 and Appendix C to reflect this discussion about the normality assumption. 

**“S is a normal matrix seems impractical"**.
Since S cannot be computed in practice as it depends on the Green's kernel, we instead estimate a surrogate based on the feature matrix $\Phi$. We measure the eigenvalues and singular values of this surrogate matrix on Atari games and the gridworld domain and present these results in Figure A.20 and Figure A.21 respectively. We observe that the effective rank computed using norms of eigenvalues is roughly equal to the effective rank computed using singular values in all 5 games tested. Hence, practically, a drop in the effective rank computed using eigenvalue norms translates to a drop in effective rank defined using singular values. This observation, in commonly used Atari domains and the gridworld, justifies that assuming normality of $\mathbf{S}$ is not a restrictive assumption since eigenvalue effective rank always satisfies Theorem 4.1 and practically, this other notion of effective rank is similar to $\text{srank}_\delta$.

**Proposition 4.1 does not use any property specific to TD-loss (“This wouldn't support the observations that the rank issues are specific to bootstrapping”)**: Although Proposition 4.1 is generally applicable to deep linear nets trained with gradient descent, our main result is to show how the rank drop effect in this proposition is m due to bootstrapping in RL (Equation 5, Section 4.2).
Supervised learning with neural nets also exhibits a rank drop, however, this effect is more pronounced in RL, resulting in rank collapse. The next paragraph after Proposition 4.1, which we now clearly demarcate in the revised paper, shows that rank decrease due to supervised learning within one fitting iteration gets compounded when we utilize the labels generated from the previous instance of the network for training in future iterations. 

**Balancedness and neural networks (“limit the applicability of the insight gained here to more practical neural networks?”)**
While balancedness is an assumption that only applies to deep linear networks, and is thus limited, it is one of the common models for studying the learning dynamics of deep supervised learning (Arora et al. 2018, 2019, Du et al 2018, Gunasekar et al. 2017, 2018, 2019), and typically provide insights also applicable to practical neural networks.   

**How to deduce rank collapses in data-efficient RL + “..pseudo-optimal policy... surrogate for Q\*?”**
We have added a new comparison with surrogate Q* for Atari in Figure 2 and we already had an experiment for gridworlds in the submission where we compared the rank of the learned Q-function to the rank of the optimal Q\*. As shown by the dashed line in Figure 2, the rank of the value network with bootstrapping is far lower relative to this pseudo-optimal Q\*, which indicates that rank has indeed collapsed. 

**Other factors affecting performance, Seaquest in Figure 1, Asterix in Figure 1 rank drop not aligned with performance drop**
As observed by the reviewer, when the rank does not collapse, performance may still drop due to other factors, which we acknowledge and highlight in red in Section 3 in the revision.  Our sole claim is that in many data-efficient RL problems, deep Q-learning can induce very low-rank solutions that heavily alias value predictions at different states and exhibit poor performance.

We have also added zoomed-in plots for Atari games in Appendix A.1 (Figure A.1) showing that mostly the optimum for rank appears close enough to the optimum for performance beyond which both degrade validating our claim above. While not perfectly aligned in Seaquest with 4x data, it holds as a general pattern across most environments when more gradient steps are performed (Figure A.1). 

**“Perhaps the underlying problem is really that minimizing TD^2 is not an appropriate way ..”**
 We analyze TD^2 since it is the commonly used method for deep Q-learning and actor-critic algorithms, and our goal is to understand why these commonly used methods might fail. Our results on Rainbow in Figure A.17 also study the C51-style cross-entropy loss for deep Q-learning and we observe similar findings. While the problem at a high-level can be attributed to TD^2 not being the optimal way to find the fixed point, the goal of our work is to analyze and understand precisely why this is the case, specifically for neural networks in practice. Our results highlight one of the issues specific to minimizing TD^2 with neural nets and can aid better algorithm design.

**TD error remains small (Figure 3), returns are quite bad, how can one explain this + "If rank collapse entails that the TD error is not minimized well-enough....how can one explain this figure?”** 
The notion of “smallness” of TD-error depends on the problem and value function, which makes it possible to only relatively compare the different curves in Figure 3 (that is, comparing curves n=1 with n=4). It is unclear if TD error on figure 3 can be considered small, as a desired “small” value of TD error is unknown in Atari. We plot TD error on **log** scale and show that larger ranks (n=1) typically correspond to smaller TD error (n=1).


**“While we are trying to find the fixed point of the Bellman equation in RL, there is no such fixed point in kernel regression.”**
 It would be great if the reviewer can clarify this statement. Based on our understanding, both kernel regression and RL with neural network function approximation may not converge to a fixed point in a general scenario, however prior works like kernel LSTD (Xu et al. 2005, 2007) do show that kernel regression converges to a fixed point under certain conditions. In this sense, the convergence of kernel regression in an RL setting does not seem to be any worse than convergence for any other non-linear function approximator.  

**Name “implicit under-parameterization” (..large number of sentences where the word “implicit” is used in a vague sense.. )**
We used the word “implicit” heavily to signify that the under-parameterization effect is implicitly caused (no explicit loss encourages this) similar to the implicit regularization effect of gradient descent. Specifically, “Implicit under-parameterization” broadly identifies the issue that a deep Q-network behaves as a low-capacity model due to the implicit regularization effect of training with bootstrapped objectives. We are more than happy to consider other names for this phenomenon if the reviewer has any suggestions.

**Summary**: We thank the reviewer for their constructive feedback. First of all, we would like to clarify that we do not claim that rank collapse is the only source of issues in bootstrapping based deep Q-learning algorithms (e.g., in Seaquest), but we show a strong correlation between rank collapse and poor performance in both online and offline settings (16/21 settings in Appendix A.1). We have also addressed theoretical concerns relating to the assumption of near optimality in Theorem 4.2, and the proof of Theorem 4.1. We have added an experiment (Figure 4) showing the practical consequences of Theorem 4.2. Changes in the paper are shown in blue in Section 3, 4, and Appendix A, and clarifications are highlighted in red. We would be happy to make more changes if the reviewer has any further suggestions or questions.  We provide specific answers below. 

**“Why is Theorem 4.1 not a direct application of Theorem 5 in Mobahi et al. 2020?”**
The proof of Theorem 4.1 is more complex than Theorem 5 in Mobahi et al. 2020 due to the bootstrapping objective in RL, which is not the same as self-distillation. As a result, our conclusion in Theorem 4.1 is also different from this prior work since we show rank does not decrease at each step, but it decreases gradually as more fitting steps are performed. 

For instance, (1) while self-distillation admits a simple closed-form solution of the $k$-th iterate only in terms of A, a PSD matrix, the bootstrapping update requires reasoning also about additive reward terms and multiplicative terms from the transition dynamics, which complicates the analysis, and (2) since the transition matrix is not symmetric, we require additional machinery to analyze the magnitudes of the singular values through different iterations (for more details, see “Bound on Singular Value Ratio” Equations C.21-C.29).

**“The development in Sec 4.2 around eq. (5) argues that when $Q_k(s,a) = Q_{k+1}(s,a)$ for all pairs $(s,a)$ you get rank collapse; special situation.... value function at each (s,a) is essentially proportional to the rewards at that state-action pair ”:**
Equation 5 is not our main focus, but rather a preliminary sub-case to build intuition for the full bootstrapping setting. Our main result in this section is Theorem 4.2 that handles the full bootstrapping setting. We have made this more clear in the updated draft.

We request the reviewer to clarify why the Q-function needs to be proportional to the reward under the condition $Q_k(s, a) = Q_{k+1}(s, a)$. To the best of our understanding, this condition is satisfied when $Q_k$ is the true value function, $Q_k = Q^\pi$,  where $Q^\pi =  r + \gamma P^\pi Q^\pi$ which is not necessarily proportional to the reward function.

**Assumption in the proof of Theorem 4.2 (“in eq. (D.15) it is assumed that zeta is small enough... simply rescaling all the rewards... should result in rank collapse”)**
Unlike Theorem 4.1, which analyzes the rank drop effect over the course of the optimization, far from a fixed point, Theorem 4.2 instead shows that rank drop can happen even when we initialize the Q-values close to a fixed point (i.e., zeta is small), which is a favorable “good” initialization point for the Q-function. Even then, we find that convergence to this fixed point may be inhibited. We have added an experiment on a deep linear net in Figure 4 demonstrating that the distance from the fixed point increases when rank drop appears. We now highlight the goal of Theorem 4.2  in red at the beginning of Section 4.2. 

Based on the above reasoning, Equation D.12 states the assumption that the Q-function is initialized close to a fixed point and we analyze rank drop starting from this initialization. Specifically, we assume that $\zeta$ is small relative to the Q-function $(||\zeta|| << ||Q||)$. Rescaling rewards will not be sufficient to induce rank drop, because our argument only holds when $\zeta$ (i.e., neighborhood size around Q) is appropriately scaled as well. 

The remaining answers are in the next post.

The main contributions of the paper are the following ones:
 1. Identifying feature rank collapse problem in RL algorithms using bootstrapping and gradient descent optimization for value function estimation and pinning down this problem to these two factors.
 2. Theoretical analysis of rank collapse based on Neural Tangent Kernel framework and ideas from analysis of continuous-time differential equations. In particular, the authors showed that rank collapses near optimal point when fitting resembles self-distillation.
 3. The regularization term heuristic to prevent rank collapse.
 
Overall, the paper contains a very extensive experimental part, theoretical part and very-well motivated idea. However, the authors tried to put too much information into one paper, therefore sometimes it is difficult to follow. For example, Preposition 4.1 is difficult to follow, since a lot of interesting and important details are hidden in Appendix.
 
Some questions and issues.
 1. There is an assumption that S is a normal matrix in theorem 4.1. How restrictive is this assumption?
 2. Is it possible to extend theoretical analysis from policy evaluation settings to policy training settings. i.e. when Bellman optimality operator is used instead of Bellman operator?
 3. I would recommend adding a title to Figure 3 (a).
 4. Figure 3 (d) contains one red trajectory, for which srank does not collapse. Could you please comment what special properties this trajectory has that srank stays almost the same?
 5.  "Similar to Arora et al. (2018; 2019), we assume that all except the last-layer weights share singular values (a.k.a. “balancedness”)." According to Appendix the stronger assumption W_j*W_j^T = W_{j+1}^T*W_{j+1} is required.
 6. I assume lambda is missing in equation (C.12).
 7. The question regarding the equation between D.4 and D.5. I understand how the derivative was computed, but I am not sure that I understand what 0 means in dL_0(W_{N:1})/dW_{N:1}.
 8. I am a bit puzzled by the fact that Rainbow performance increased, while DQN performance decreased in online settings. What is the key underlying component that leads to different results? 


This paper discusses a phenomenon wherein the feature vectors of the learned value function in reinforcement learning (RL) lose their diversity as training progresses. The paper analyzes the rank of the final hidden layer in the model parameterizing the value function and shows experimentally that for offline-RL and online-RL setups on Atari and Gym benchmarks, this rank collapse occurs with a drop in the average return. The paper further develops two models for understanding this phenomenon, (i) where the value function is modeled using the neural tangent kernel, and (ii) where the value function is modeled using a deep linear network. The paper argues that bootstrapping results in reduction of the rank of the feature matrix as training progresses for these models. A regularization term that equalizes the singular values of the feature matrix is used to mitigate this rank collapse and experimental results on Atari benchmarks are shown with this regularizer.

The main claim of this paper is to identify the phenomenon of rank collapse of the feature matrix. I have concerns about the experimental findings of this paper and correctness of its theoretical claims, which are discussed below. I am willing to increase my score if the authors can convincingly argue otherwise. Broadly, I agree this is an interesting direction but current manuscript does not convince the reader that rank collapse is indeed the cause of degradation of performance.

Comments.

1. Figure 1 does not completely validate the claims on page 3. In Asterix, increasing the amount of data does not lead to rank collapse but the returns degrade significantly during training, why? In Seaquest, the returns (blue) have degraded essentially to zero even when the rank (blue) is at its maximum. This suggests that there are other factors which are causing the drop in performance instead of/in addition to the rank. The trends in Appendix A1 are similarly inconsistent, as is Figure 2 (Ant-v2). The implication “if low rank, then low returns” is reasonable to expect due to reduced capacity of the value function approximation. But how do the authors deduce from these experiments that “rank collapses in data-efficient RL” (first sentence of Section 3.1).
2. I have a similar concern about Fig. 3b (Seaquest). The rank for n=4 gradient steps/transition clearly collapses, yet the TD error remains small, and yet the returns are quite bad. If rank collapse entails that the TD error is not minimized well-enough, and that is the cause of the drop in returns, then how can one explain this figure? I suspect the discrepancy is because the TD error is used in Fig. 3b. Can you perhaps compute a pseudo-optimal policy using a good RL method (say Rainbow) for Seaquest and use its value function as the surrogate for Q*?
3. The narrative will benefit from being more precise. There is an egregiously large number of sentences where the word “implicit” (the paper uses this word 37 times in the first 8 pages) is used in a vague manner (see for instance Definition 1). Further, “implicit under-parametrization” a bad monicker, should the lottery ticket hypothesis be also called implicit under-parametrization?
4. Why is Theorem 4.1 here not a direct application of Theorem 5 of Mobahi et al., 2020? Further, the big intellectual gap in the argument is that while we are trying to find the fixed point of the Bellman equation in RL, there is no such fixed point in kernel regression. So while the argument that self-distillation during iterative TD^2-minimization may cause a loss of diversity of the feature space, it does not seem to the only reason, after all some examples in Fig 3 do not show rank collapse.
5. Perhaps the underlying problem is really that minimizing TD^2 is not an appropriate way to find the fixed point of the Bellman iteration when using function approximation. Indeed, if the TD error is small (Fig. 3b, n=4), there is nothing the network can do to improve the returns. TD error is small in this case in spite of the feature matrix having low rank; it indeed depends on the complexity of the value function.
6. The development in Sec 4.2 using the work of Arora et a., 2019 around eq. (5) argues that when Q_k(s,a) = Q_{k+1}(s,a) for all pairs (s,a) you get rank collapse; this is a very special situation where the value function at each (s,a) is essentially proportional to the rewards at that state-action pair. I tried to follow the proof of the argument for the botostrapped updates in Theorem 4.2 but to my understanding it hides this same issue, e.g., in eq. (D.15) it is assumed that zeta is small enough which is not true. By this argument simply rescaling all the rewards to have small magnitude should result in rank collapse.

Meta-review: Knowledgeable R3 found the paper very good (8). He/she found the authors' responses very informative and that edits made the paper much stronger. R2 expressed reservations about rank collapse being the cause of degradation of performance, but also indicated his/her willingness to increase the score if the authors can convincingly respond to his/her concerns. This concerned was shared by other reviewers, and there was an extensive discussion during the discussion period. R3 and R1 found the authors' responses very convincing. Fairly confident R1 found the paper good, appreciated the discussion, and recommends the paper to be accepted. R4 found the paper marginally above the acceptance threshold, however expressing a lower confidence in his/her assessment. In summary, the article contains extensive experiments, theory, and a well motivated idea, elucidating an intriguing phenomenon and useful for designing better bootstrapping-based deep RL methods. Although the reviewers expressed some reservations in their initial reviews, there was a lively discussion with quite positive final feedback. Weighing the ratings by confidence and participation in the discussion, I am recommending the paper for acceptance. I would like to encourage the authors to make efforts in making the presentation as clear as possible, having in mind the discussion and comments from the reviewers. 

Decision: poster-presentations