Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides empirical and theoretical evidence that value-based
methods that optimize TD errors with SGD have an implicit "regularizer" that
increases the dot-product of the representation at successive states.
The theoretical analysis shows that this can be offset by a term that
penalizes large dot products, which motivates the proposed regularizer
(DR3). The paper shows empirically that these large dot products are a
source of divergence and that training with the DR3 regularizer can
stabilize learning.

main_review: Strengths
---------

-   Interesting theoretical analysis, while somewhat compromised to make
    the algorithm tractable, still provides insight into the problem
    addressed. It might be good to further show evidence that the noise
    model is well-justified in the RL regime, which would justify the
    leap from theory to empirics.
-   Empirical analysis is comprehensive, clearly demonstrating the
    problem and algorithms stability when the problem is addressed
    (through the proposed regularizer).

Weaknesses
----------

-   While the empirical analysis is comprehensive, the performance of
    the algorithms with the DR3 regularizer in the experiments are not
    completely compelling. Although the mean performance seems better,
    the statistical significance is not there in Figure 3 and Table 2
    (and Figure 6, Seaquest). The results in Atari across all games are
    more impressive, but the aggregation and normalization across games
    can hide idiosyncrasies.

Detailed Comments
-----------------

-   Section 3, Experimental setup: For TD-Learning, it says an action is
    sampled. Shouldn't this be an expectation over the policy (and hence
    use every action, without sampling?). If all the actions are
    included in the target through an expectation, the problem of
    divergent action-values should be less pronounced if not absent. The
    problem seems to be that some action-values are left dangling (i.e.
    the action-values of actions that are not sampled). By including
    them all through an expectation, any problematic danging
    action-values should be corrected immediately. The problem remains
    for Q-learning, but further analysis needs to ablate maximization
    bias and other confounders.

-   Sec 3.1, Paragraph 2: What is meant by learned behavior policy?
    Shouldn't the behavior policy be given or fixed?

-   I think there is a leap in reasoning by connecting states aliasing
    and high dot-products. Dot products can increase between two vectors
    without any aliasing. If the magnitude of both vectors increase then
    the dot product will increase without any change in the angle
    between the vectors.

-   Increasing dissimilarity between successive states is intuitively
    reasonable, but this can also be a hindrance if the successive
    states are indeed similar. Is there any way to account for this?

-   Shouldn't a target network prevent the representations from being
    similar? The networks are different and hence the representations
    will be a few gradient steps away.

-   Section 2 Equation 1: While we do take the gradient of this
    expression, it is not correct to say that this is a proper
    optimization objective. It is not minimized in practice (and often
    diverges). In addition, methods that actually minimize this
    objective, such as residual-gradient, do not result in a good
    control policy.

-   Figure 2: The left and center plots communicate the issue clearly,
    except that the training loss plot on the right is not reflective of
    the problem. As you point out, bootstrapping algorithms do not
    minimize any objective function as the gradient does not belong to
    any objective function.
summary_of_the_review: Overall, the paper raises and simultaneously addresses an interesting
problem for RL algorithms that employ function approximation. The paper
is somewhat held back by the empirical analysis, specifically in the
lack of statistical significance in several of their experiments. I am
also not sure of the correctness of the intuition that the dot-products
result in aliasing in reinforcement learning. The paper is still above
the acceptance threshold, but I hope that my comments below contribute
to improving the paper.


Edit: After discussion with the author and reading the back-and-forth between the authors and reviewer 5pgu, I have increased my score from 5 -> 8.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses how the implicit regularization effect of SGD could be harmful in the offline deep RL setting, due to degenerate feature representations, aliasing the representations for state-action pairs that appear on either side of the Bellman backup. To address this issue, the paper proposes a simple explicit regularizer (DR3) that counteracts the undesirable effects of this implicit regularizer. When combined with existing offline RL methods, DR3 has strong performance and stability (Atari 2600 games, D4RL domains and robotic manipulation from images).
main_review: The paper is overall well-written and clear. The method proposed is simple, quite clearly motivated and the claims are quite well supported by the experiments. Here are a few elements/questions that might need some clarifications:
- the dot products can increase either because of an increased co-adaptation or because the magnitude of the vector of features considered increases. How can be sure that it is due to a co-adaptation and not because of an increase in the magnitude of the features?
- In Fig 3, 5% and 25% of the data used by Singh et al. (2020) are used to make these more challenging. Would it also be useful to provide the results for the case of 100% of the data? It seems that even when data is relatively abundant, it would be interesting to report the performance with and without the regularizer. The same comment applied for Figure where a plot with 100% of the data could also be of interest.
- The authors mention that they will release scores for individual runs as well as open-source their code. Is it possible to have access to that during this rebuttal period?

Minor remarks: 
- typo: "(...) and the the dot-product (...)"
summary_of_the_review: Paper that investigates a simple regularization technique for off-policy value function learning in deep RL. The idea is quite well motivated by showing a few insightful values through learning (not only the score). The empirical results are quite convincing, even though they could be more complete.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose to study the under-parametrization/co-adaptation effect (observed while optimizing over-parametrized networks with SGD) arising in TD learning based RL algorithms which looks correlated to performance drops. They prove an implicit regularization effect, following previous works focussing on supervised learning, that could explained the co-adaptation phenomenon. They derive an explicit regularizer and show its effects on plethora of experiments.
main_review: Overall the paper is well written, pedagogical, and the problem at stake is really interesting. But I find the presentation of experimental results misleading which makes me doubt about the performances of the proposed solution.

### Major revisions/questions/remarks
  * Roughly 3 papers out of 5 refer to non peered reviewed papers (arXiv)
  * Which algorithm is used to learn the Q-value in Figure 5? I am assuming it is CQL since the results match the ones for CQL in Kumar et al. 2021 
  * "it also alleviates rank collapse, with no apparent term that explicitly increases rank" (Section 5), and "This is potentially suprising, because no term in the practical DR3 regularizer explicitly aims to increase rank" (Appendix A.3):
    * The only srank's behavior exposed in this paper are the ones observed while learning the Q-network with the CQL algorithm. What about DQN, REM, and BRAC? Is the alleviation of rank collapse is specific to CQL+DR3?
    * Looking at the simplified loss the authors based their intuition on, composed of 3 (maybe 4?) terms:
      * 1. TD error: $||r + {\phi^\prime}^\top w$ - $\phi^\top w||^2$;
      * 2. CQL OOD samples regularization: ${\phi^\prime}^\top w$ (at least);
      * 3. Implicit regularization: $||\phi||^2 - \[\[{\phi^\prime}^\top\]\] \phi$;
      * 4. Potential L2 regularization on weight: $||w||^2$ (at least);
      * It looks like adding the explicit regularization is an explicit way to increase rank, even if this regularization is more computationally efficient than Kumar et al. 2020. Am I missing something?
    * Plotting the norms of $\phi$, $\phi^\prime$, and their cosine distance with and without DR3 could be quite informative
  * Comparing Figure 6 (resp. A.4) to Figure 5 (resp. A.3) is misleading the reader:
    * At first sight it looks like the performances are somehow related to the srank not shrinking
    * But at 50x62.5K steps the srank of CQL is higher than CQL+DR3 for Breakout for example.
    * Why the Average Return of Asterix is missing (Figure 6)?
    * Looking further at the results reported in Kumar et al. 2021, the authors are missing what seems like the interesting region w.r.t. their claims. It is only after more than 50x62.5K gradient steps that the performance drops appear. Why did the authors clipped the plots if the experiments were done for at least 200x62.5K gradient updates. This could help conclude about the training stability the authors claim reporting in the paper (e.g. p. 2 " giving rise to methods that train for longer, reach a better solution, and remain stable at this solution ").
    * Nevertheless I notice what looks like a significative improvement w.r.t. the average return
  * Same goes for Figure 4
    * The x-axis' scale for 1% Uniform Replay is different from the other ones, why? It looks like CQL+DR3 suffers a performance drop and that with more gradient step the performances with or without the explicit regularization would be similar.
    * Is that the case for the 1% Uniform Replay experiment at 200x62.5K gradient updates?
  * "we evaluate DR3 in conjunction with CQL on the harder D4RL domains (antmaze, kitchen domains)" (Section 6), "we use the harder AntMaze and Franka Kitchen domains for evaluating CQL, since these domains are challenging for CQL." (Appendix E.2)
    * W.r.t. which metric those domains are harder than the other ones? From Kumar et al. 2021: "These tasks are especially challenging, since they require composing parts of trajectories, precise long-horizon manipulation, and handling human-provided teleoperation data". In this case, does the potentially poor performances are correlated to the co-adaptive phenomenon? Reporting sranks after 2M gradient steps would help conclude.
    * While trying to put emphasis on "stability", what looks like the main focus of the paper, I was expecting the authors to focus on environments for which previous methods suffer from performance drop and the under-parametrization phenomenon such as Ant-v2, Hopper-v2, and Walker2d-v2 as shown in Kumar et al. 2021 or hopper-medium-v0 as shown in Kumar et al. 2020b
  * I do not understand why the effect of DR3 on D4RL is discussed using BRAC and not CQL nor REM. This would have had more impact since the paper mainly focus on the effect of DR3 w.r.t. CQL and REM. 
  * Overall a rigorous statistical significance test, to highlight significatively better results in the tables, is missing

### Minor revisions/questions/remarks
  * Does light colored regions on the plots always refer to 95% CI?
  * Does the under-parametrization phenomenon and the drop in performances are specific to over-parametrized NNs?
  * What optimizer the authors are using to optimize their models? This could be interesting since the theory developed in the paper assume an SGD optimization scheme.
  * It would be better if all the algorithms are referenced in the main paper, e.g. COG has only its attached reference in the appendix (?)
  * Writing "This protocol is standard in Atari" seems a bit far-fetched when the references point only to Google related papers
  * What does bold format mean in the paper tables?

### Some typos
  * $\theta$ missing for some gradients
  * p. 12 "A eoretical comparison"
  * p. 13 "in ou practical"
  * p. 16 "(Equation'4)""
  * p. 22 "atleast"
  * p. 24 "wprked"

Authors answering remarks and questions during the rebuttal period might lead to a higher grade. But, as is, important claims of the paper are not well-supported and the way the results are presented is suspicious.
summary_of_the_review: The paper looks theoretically sound some questions remain regarding the experimental efficacy of the proposed regularizer claimed by the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper first empirically notices the feature co-adaptation phenomenon which will be exacerbated in TD-learning with out-of-sample next actions. The authors theoretically show that the implicit regularization of (noisy) SGD encourages maximizing the inner product between the encoded features of current and next state-action pairs, which is responsible for the observed phenomenon. Thus, the authors propose a novel explicit regularizer for offline TD-learning to encourage smaller inner products. Extensive experiments on both video game environments and robotics demonstrate the effectiveness of the proposed regularizer, with various offline TD-learning algorithms as the testbeds.
main_review: Strengths:
1. The identified phenomenon is very critical: When the consecutive state-action pairs are encoded to have similar representations, the learned value functions, as well as corresponding policies, would perform poorly.
2. This paper provides theoretical evidence to explain the feature co-adaptation phenomenon, which meanwhile explains why it is severe for out-of-sample next actions.
3. The empirical evaluation strongly confirms the effectiveness of the proposed regularizer.
Weakness:
1. There are some typos, e.g., "stableif" in Theorem 3.1.
2. The organization can be improved further. It is not easy to follow this story, and there are many preliminaries about offline RL that are indispensable for readers to understand the discussions.
summary_of_the_review: This is a good paper which studies an important problem from both empirical and theoretical aspects and brings in sufficient novel stuff. I appreciate the contributions made by the authors and recommend this paper.
review_title: null

Meta-review: The paper proposes an interesting hypothesis about deep nets' generalization behavior inside RL methods: it suggests that the nets' implicit regularization favors a particular form of degeneracy, in which there is excessive aliasing of state-action pairs that tend to co-occur. It proposes a new regularizer to mitigate this problem. It evaluates the hypothesis and the regularizer empirically, and it provides suggestive derivations to motivate both.

The reviewers praised the comprehensive empirical analysis, the insights into learning, and the combination of empirical and theoretical evidence. The authors participated responsively and helpfully in the discussion period, and addressed any concerns raised by the reviewers.

This is a strong paper: it derives and motivates a novel hypothesis about an important problem, and analyzes this hypothesis both mathematically and experimentally.
Decision: Accept (Spotlight)