Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper discusses several practical considerations for conservative offline RL methods (CQL specifically). It provides metrics for practitioners to identify overfitting and underfitting of the algorithm without evaluating the policy with any online data. It also provides corresponding guidelines when overfitting or underfitting happens. The claims are supported by thorough experimental analysis on both simulated and real robots.

strengths_and_weaknesses: Strengths:
- The issue discussed in the paper is important and also very common in offline RL when applied to robotics. The paper provides a thorough discussion on it and thus provides a strong contribution to the field.
- The claims are well supported by the analysis in the experiment section.
- The paper is well organized and easy to understand.

Weaknesses:
- Scenario #2: The capacity-increasing measures are evaluated using the 35-object task. From the policy performance plot and the TD-error plot on the website, the improvement of performance is not clear to me. A more extensive study is needed to verify the hypothesis of underfitting. Maybe it would be interesting to see the same experiments for the 10-object and 20-object tasks. Since they actually have larger TD errors than 35-object from Figure 5, we might see a clearer improvement. 
- Line 172-175: “Thus, a low Q-value … on states in the training dataset.” Can you clarify this further?
- Figure on the right of Line 340-346: Missing figure number and description. The legend covers a part of the curves and should be adjusted.

Extended questions:
- How much does the CQL hyperparameter alpha affect the conclusions? For the overfitting case, would alpha affect the drop of the Q-value?
- For the robot experiment, how are the instability and underfitting related to overestimation?
- Line 415: “Our workflow is also specific to conservative offline RL methods, and particularly CQL.” It would be great if the authors can discuss this a bit more. To what extent do the conclusions hold for general offline RL or online RL methods?
summary_of_recommendation: The paper proposes metrics and guidelines for overfitting and underfitting issues of CQL. It provides a thorough analysis of the issues and solutions with experiments in simulation and in the real world with considerably large-scale and challenging manipulation tasks. The paper provides important insights for offline RL. It is both technically informative and enjoyable to read.
issues: Address the weaknesses mentioned above and provide more discussion on the extended questions.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Offline reinforcement learning is very important to applications where data can be only acquired offline (e.g., medical application); or when one wants to avoid the application of untrained policy to real systems (e.g, robotic applications).

In supervised problems, there is a clear definition of "overfitting" and "underfitting".  More in detail, overfitting happens usually when the functional space of the approximator is large and there is a scarcity of samples. The result is that the model will "fit" the noise in the data. Such an issue can be mitigated by shrinking the functional space, introducing regularization, by the provision of more samples (when possible), or by early stopping. Underfitting happens usually when the functional space of the approximator is too small, or the regularization term is too strong, and the approximator will exhibit a high bias. Underfitting can be usually avoided by introducing a larger functional space or by weakening the effect of the regularization.

Eary stopping is a technique used along with gradient optimization techniques. The idea is that in the early stages of the optimization, the functional approximator is probably underfitting the function, while in the last stage of the learning procedure, the function could be fitting the noise. Therefore, if one could estimate when the overfitting starts, could stop the optimization at that point to avoid both the problems.
The detection of the underfitting or overfitting can be done by measuring the loss on the validation set. When the loss on the validation set starts to increase (and the loss of the training set keeps decreasing), we can detect the overfitting.

Most of the reinforcement learning literature focuses on online problems. Therefore, there is not a clear training set and validation set. Furthermore, the overfitting can be easily mitigated by the provision of new samples. 

Contrarily, in offline reinforcement learning the training set is fixed.

This paper attempts to define overfitting and underfitting for offline reinforcement learning (focusing on conservative Q-Learning). Furthermore, since the most natural metrics would require interaction with the environment, the authors propose some heuristics to detect the underfitting or the overfitting without the interaction with the environment. Furthermore, they propose some techniques to overcome underfitting and overfitting.





strengths_and_weaknesses: __Strength__

The motivation of the paper is really strong. I believe that offline reinforcement learning is crucial in many applications. Furthermore, most of the literature on reinforcement learning does not mention the problem of overfitting or underfitting. The problem is even more important in offline reinforcement learning, as there is no possibility of providing more samples. 
Measuring these quantities without the possibility of interacting with the environment is also very problematic. 

For this reason, I believe that the attempt of the authors of providing a solution to this problem is very important for the community.

Even though, as I will write later, I am not fully convinced by the heuristics proposed in the paper, the experimental section is in my opinion strong enough to support the statements made. 


__Weakness__


A first weakness of the paper is to consider the specific case of CQL. I do believe that overfitting or underfitting is a much larger and more general problem than a specific algorithmic solution such as CQL. I regard the regularization term in CQL as an attempt to avoid overfitting. The "surrogate" objective introduced in CQL, is a modification of the real objective that includes a regularization term. Therefore one should still focus on the real objective. 

Another weakness of the paper is to look at the average $Q_\theta(s, a)$. I believe that a much better metric should be $J(\theta)$, which in terms of the algorithm can be estimated with $\mathbb{E}_{\mu_0}[V_\theta(s)]$. This is the metric that we want to optimize. Maximizing the average Q-function tells only how we are maximizing the surrogate objective. Refer to [1, 2, 3] to see why maximizing this objective is "wrong".

There are many forms of regularization. As said, I already believe that the regularization term in CQL is avoiding overfitting. Although I believe that VIB can be a good way to regularize the Q-function, I don't see why other simpler techniques have not been tested (L2, L1, ...).



[1] Section 5.1 of https://arxiv.org/pdf/1811.09013.pdf
[2] Section 3 paragraph "Training Mismatch" of http://proceedings.mlr.press/v97/fujimoto19a/fujimoto19a.pdf
[3] Section 4.1.2 of https://arxiv.org/pdf/2010.14771.pdf




summary_of_recommendation: I do recommend the acceptance of the paper, mostly because I think it is an important attempt to a problem that is really interesting for the community.
I am convinced that the paper is good, even though the proposed heuristics are relying on intuition, and I do not entirely believe that they are correct.

Update
==

I am satisfied with the author's answer and I am keeping my score unchanged. 
issues: 1) Metric 4.1.

In my opinion, the overfitting happens when the algorithm estimates a large return $\mathbb{E}_{\mu_0}[V_\theta(s)]$, while on the real environment the return starts actually to decrease.

I do understand that metric 4.1. detects sort-of when the maximization of the $Q$-function is already "over", and the regularization term starts to kick in. However, suppose that the network of the Q-function is _really_ small, and therefore overfitting cannot possibly happen. 
Still, I can imagine that in that case, the metric 4.1. can still happen (when eq.1 reaches "convergence", and when the optimizer must exploit the regularization terms to increase the term in eq 2).

2) Metric 4.2. 

Even though I generally accept the idea to train the dataset on two different functional approximators to detect the underfitting (if the one with higher capacity has a lower loss, then _probably_ the smaller one is underfitting); still I do not fully agree in the case of RL.
For example, can be that the smaller functional approximator is actually approximating the Q-function better, even though the empirical error is higher. In fact, it can happen that, if the Q-function has a higher magnitude, the magnitude of the error will also increase.

I know this effect, as I often noticed that at the beginning of the learning, the Bellman error tends to usually increase, as the magnitude of the Q-function also increases. That does not mean that the approximation is worst: actually, it is better, but the loss does not reflect it monotonically.

__side note__

I'd modify eq.2 with the following, for clarity, 

$$
\pi^* := \mathrm{arg}\max_\pi J_D(\pi) - \alpha D(\pi, \pi_\beta)
$$

as $a$ and $s$ do not appear on the right-hand side of the equation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Although current offline RL algorithms can learn from prior data without interacting with the environment, the learned policy has to be evaluated online. This paper proposes a workflow, including a set of protocols and metrics, to be used to adjust model capacity, regularization in offline model-free RL. The proposed workflow shows that it can successfully detect overfitting and underfitting in offline RL and thus can be used as guidelines to improve policy performance. 

strengths_and_weaknesses: Strengths: 
1. The motivation is strong. Policy evaluation in offline RL is an important research topic. Although there are some limitations, the proposed workflow can be used to detect overfitting and underfitting in offline RL.
2. The paper offers well-designed experiments, both on simulators and real robots, to demonstrate the effectiveness of the workflow.
3. The paper is well-written and easy to follow. 

Weaknesses: 
1. My main concern is the effectiveness of the proposed metrics and the policy selection strategy. 
    
    <a>  For metric 4.1 (overfitting), the decrease of the average data Q-value serves as the indicator of overfitting. But whether this metric is robust enough w.r.t other design/hyperparameter choices? For example, when choosing a larger \alpha, we may easily observe the decrease of Q-value. Also, when choosing an improper policy \mu(.|s), such as \mu(.|s)=behavior policy, Q-value may not decrease under overfitting.
    

    <b>  For guideline 4.1 (policy selection), the author suggests selecting the checkpoint that attains the highest average dataset Q-value before overfitting. But for example in Fig 9, the highest avg. Q-value is not corresponding to a good policy. 

2. My second concern is whether the proposed workflow is limited to CQL (or Q-value regularized algorithms). In CQL, the Q function is trained with equation (1), but there are also many other offline RL methods, such as TD3_BC (https://arxiv.org/abs/2106.06860) etc., which add the constraint on policy while keeping the Q function untouched. In these algorithms, I’m afraid the proposed metrics are not suitable, e.g., even under overfitting, the Q function maybe won’t decrease like in CQL.

3. The proposed workflow, in my understanding, can only be used to detect overfitting and underfitting. But for the rest hyperparameters, for example, the \mu(.|s), \alpha in eq. (1), we still need to tune them via online evaluation. 

summary_of_recommendation: The paper has several weaknesses as listed above, but considering the workflow can be successfully used to detect underfitting and overfitting as demonstrated in two different real-world robots and multiple simulated tasks, my recommendation is "Weak Accept".
issues: The issues are listed above. Please check the "Strengths And Weaknesses".
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Paper presents two metrics for detecting underfitting and overfitting in offline RL: 1) the magnitude of Q value 2) the TD-error vs. model size. The authors also provide suggestions to address these two issues in the light of the proposed metrics. Experiments on both simulated and real-world offline RL tasks demonstrate the effectiveness of the resulting model selection.
strengths_and_weaknesses: [Strengths]

[+] The paper is overall clear and well-written. I like the illustrative writing style and I believe this paper is mostly readable to those who are not familiar with offline RL and related topics. 

[+] The research problem seems to be incremental and it is indeed important -- model selection can be particularly difficult for offline RL as ideally no online eval should be allowed during training. The authors do bring intuitive and interesting perspectives to this problem.

[+] Although personally, I don't think this paper presents significantly novel ideas, the method here is well-motivated, reasonable, and technically sound. 

[+] The empirical evaluations are impressive. I like how the authors extensively test their method on both simulated and real-world domains and attain consistent improvement over baselines. 

[Weaknesses]

At this point, I cannot find any major issues with this paper. But I have to point out that the two metrics here are straightforward, especially for the underfitting one. The one on overfitting deserves more inspection as a decrease of Q value could also due to some other factors like weight decay. 

As a side note, is it possible to extend these two metrics to other offline RL algorithms? I feel the first one highly relies on CQL.
summary_of_recommendation: Given the overall technical quality, I recommend an acceptance at this point. But I do hope the authors can address some of my concerns in the rebuttal and revise their paper accordingly.
issues: 1. More inspection on the decrease of Q value. Is overfitting the only possible factor that leads to this effect?

2. Is it possible to apply these two metrics to other offline RL algorithms?
review_title: null

Meta-review: The paper studies an important topic of accessing overfitting and underfitting in CQL. 

All reviewers see merits in this work but also point out weaknesses and have concerns that I share.
I agree with the evaluation of the reviewers and invite the authors to answer the raised questions and indicate how the paper will be improved for the camera-ready version. 
In particular: (however, this does not mean the other points should not be addressed)
 - Scenario #2 regarding the object 35 task and alternative tasks
 - How much does the CQL hyperparameter alpha affect the conclusions? For the overfitting case, would alpha affect the drop of the Q-value?
 - The relation and applicability to other offline RL algorithms

The authors have addressed most of the concerns and have added several additional experiments. 
I recommend acceptance of the paper.
 
Decision: Accept (Oral)