A Workflow for Offline Model-Free Robotic Reinforcement Learning
Authors: Aviral Kumar
Anikait Singh
Stephen Tian
Chelsea Finn
Sergey Levine
Venue: CoRL2021 Oral (2021)
Keywords: workflow
offline RL
no online tuning

Abstract: Offline reinforcement learning (RL) enables learning control policies by utilizing only prior experience, without any online interaction. This can allow robots to acquire generalizable skills from large and diverse datasets, without any costly or unsafe online data collection. Despite recent algorithmic advances in offline RL, applying these methods to real-world problems has proven challenging. Although offline RL methods can learn from prior data, there is no clear and well-understood process for making various design choices, from model ar- architecture to algorithm hyperparameters, without actually evaluating the learned policies online. In this paper, our aim is to develop a practical workflow for using offline RL analogous to the relatively well-understood workflows for supervised learning problems. To this end, we devise a set of metrics and conditions that can be tracked over the course of offline training and can inform the practitioner about how the algorithm and model architecture should be adjusted to improve final performance. Our workflow is derived from a conceptual understanding of the behavior of conservative offline RL algorithms and cross-validation in supervised learning. We demonstrate the efficacy of this workflow in producing effective policies without any online tuning, both in several simulated robotic learning scenarios and for three tasks on two distinct real robots, focusing on learning manipulation skills with raw image observations with sparse binary rewards. Explanatory video and additional content can be found at https://sites.google.com/view/offline-rl-workflow 

A Workflow for Offline Model-Free Robotic Reinforcement Learning
Aviral Kumar",,, Anikait Singh*.,, Stephen Tian', Chelsea Finn", Sergey Levine
1 UC Berkeley, 2 Stanford University (" Equal Contribution)
aviralkdberkeley edu, asap7772@berkeeey. edu
Abstract: Offline reinforcement learning (RL) enables learning control policies by utilizing only prior experience, without any online interaction. This can allow robots to acquire generalizable skills from large and diverse datasets, without any costly unsafe online data collection. Despite recent algorithmic advances in offline RL., applying these methods to real-world problems has proven challeng- ing. Although offline RL methods can learn from prior data, there is no clear and well-understood process for making various design choices, from model ar- chitecture to algorithm hyperparameters, without actually evaluating the learned policies online. In this paper, our aim is to develop a practical workflow for using offline RL analogous to the relatively well-understood workflows for supervised learning problems. To this end, we devise set of metrics and conditions that can be tracked over the course of offline training, and can inform the practitioner about how the algorithm and model architecture should be adjusted to improve fi- nal performance. Our workflow derived from a conceptual understanding of the behavior of conservative offline RL algorithms and cross-validatio in supervised learning. We demonstrate the efficacy of this workflow in producing effective poli- cies without any online tuning, both in several simulated robotic learning scenarios and for three tasks on two distinct real robots, focusing on learning manipulation skills with raw image observations with sparse binary rewards. Explanatory video and additional content can be found at sites cates.google..oo gowgle. wwwpggl..comvview//ooooo http///www//wwwweeeeeee wtwp:///fflieee--wooreeeeeeeeeee workffff.ww.
Keywords: workflow, offline RL, offline tuning
Introduction
Overlitting
soum Delaeen
Offline reinforcement learning (RL) can in principle make it possible to convert existing large datasets of robotic experience into ef- fective policies, without the need for costly dangerous online interaction for each train- ing run. While offline RL algorithms have improved significantly [1, 3, 4, 5], apply- ing such methods to real-world robotic con- trol problems presents a number of major chal- lenges. In standard online RL, any interme- diate policy found during training executed in the environment to collect more experience, which naturally allows for an evaluation of the policy performance. This ability to evaluate in- termediate policies lets practitioners use "brute- force" to evaluate the effects of various design factors, such model capacity and expressivity, the number of training steps, and so forth, and facilitates comparatively straightforward tuning. In contrast, offline RL methods do not have access to real-world on-policy rollouts for evaluating the learned policy. Thus, in order for these methods to be truly practical for real- world applications, we not only require effective algorithms, but also an effective workflow: a set of protocols and metrics that can be used to reliably and consistently adjust model capacity. regularization, etc in offline RL to obtain policies with good performance, without requiring real-world rollouts for tuning.
Capaco
Pollcy selection
g..
Undertitingg
COLBRAC)
Fatamancc
Our proposed workflow aims to detect overfitting and underfitting, and provides guidelines for addressing these issues via policy selection, regulariza- tion, and architecture design. We evaluate this work- flow two real-world robotic systems and simulation domains, and we find
Figure 1:
effective.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

A number of prior works have studied model selection in offline RL by utilizing off off-policy eval- uation (OPE) methods [6] estimate policy performance. These methods can be based either on model or value learning [7, 8, 9, 10] or importance sampling [6. 11, 12, 13]. However, developing reliable OPE methods is itself an open problem, and modern OPE methods themselves suffer from hyperparameter selection challenges (see Fu et al. [14] for an empirical study). Moreover, accurate off-policy evaluation is likely not necessary to simply tune algorithms for best performance - we do not need& a precise estimate of how good our policy is, but rather workflow that enables us to best improve it by adjusting various algorithm hyperparameters.
In this paper, we devise a practical workflow for selecting regularizers, model architectures, and policy checkpoints for offline RL methods in robotic learning settings. We focus on a specific class of conservative offline RL algorithms [15, 2] that regularize the Q-function, but also show that our workflow can be effectively applied to policy constraint methods [16]. Our aim not to focus on complete off-policy evaluation or to devise a new approach for off-policy evaluation, but rather to adopt a strategy similar to the one in supervised learning. Analogously to how supervised learning practitioners can detect overfitting and underfitting by tracking training and validation losses, and then adjust hyperparameters based on these metrics, our workflow (see Figure I for a schematic) first defines and characterizes overfitting and underfitting, proposes metrics and conditions that users can track to determine an offline RL exhibits overfitting or underfitting, and then utilizes these metrics to inform design decisions pertaining neural net architectures, regularization, and early stopping. This protocol is intended to act as a "user's manual" for practitioner, with guidelines for how to modify algorithm parameters for best results without real-world evaluation rollouts.
The primary contribution of this paper is a simple yet effective workflow for robotic offline RL. We propose metrics and protocols to assist practitioners in selecting policy checkpoints, regulariza- tion parameters, and model architectures for conservative offline RL algorithms such as CQL [2] and BRAC [16]. We empirically verify the efficacy of our proposed workflow simulated robotic manipulation problems as well as three real-world robotic manipulation problems on two different robots, with diverse objects, pixel observations, and sparse binary reward supervision. Experimen- tally, we evaluate our method on two real-world robots (the Sawyer and WidowX robots), and one realistic simulated tasks. Our approach is effective in all of these cases, and on two tasks with the Sawyer robot that initially fail completely, our workflow improves the success rate to 70%.
2 Preliminaries, Background, and Definitions
The goal in RL is to optimize the infinite horizon discounted return R Ei= ("r ax), where r(s, a) represents the reward function evaluated at a state-action pair (s, a). We operate in the offline RL setting and are provided with a fixed dataset D {(ss a, r(s, a). s') s')), consisting of transition tuples obtained from rollouts under a behavior policy T s s). Our goal is to obtain the best possible policy by only training on this fixed offline dataset D. with no access to online rollouts. We focus on conservative offline RL algorithms that modify the Q-function to penalize distributional shift, with most experiments on CQL [2), though we also adapt our workflow to BRAC [16] in Appendix F.I.
Conservative Q-lcarning (CQL). The actor-critic formulation of CQL trains a Q-function (s. a) with a separate policy (ais), which maximizes the expected Q-value Ea-D, (Qo(s, a)] like other standard actor-critic deep RL. methods [17, 18, 19]. However, in addition to the standard TD error (TD(O) (0) (in blue below), CQL applies a regularizer (0) (in red below) to prevent overestima- tion of Q-values for out-of-distribution (OOD) actions. This term minimizes the Q-values under a distribution u(als), which is automatically chosen to pick actions a with high Q-values Qo a), and counterbalances this term by maximizing the values of the actions in the dataset:
min a (LEw ((En-)))a))(1))))) QQ a)] - Ex D [Qoss a))) + 2 EAa.'' 0
[Qo(s. a) - 18" Q(s, a))² (1)
where B"d ((,, a) is the Bellman backup operator with a delayed target Q-function, Q: 13th ((s a) = r(s, a) + " [Q(s', a')] In practice, CQL computes (aas)) using actions sampled from the policy π (a(s). More discussion of CQL is in Appendix B. In this paper, we will utilize CQL as a base algorithm that our workflow intends to tune, but we also extend it to BRAC.
Overfitting and underfitting in CQL. Conservative offline RL algorithms [2, 20] like CQL can be sensitive to design choices, including number of gradient steps for training [21, 22] and network capacity. These challenges are also present in supervised learning, but supervised learning methods benefit from a simple and powerful workflow that involves using training error and validation error to characterize overfitting and underfitting. A practitioner can then make tuning choices based on
2

---

these characterizations. To derive an analogous workflow for offline RL, we first ask: what do overfitting and underfitting actually mean for the case of conservative offline RL?
To define overfitting and underfitting generically for any conservative offline RL method, we con- sider an abstract optimization formulation for such methods [2]:
π. = arg max JD (T) - (T( TT)
(Conservative offline RL).
(T( denotes the average return of policy π in the empirical MDP induced by
Quantity
Supervised Learning
Conservative Oflline RI.
Test
Loss evalunted on test data, Dex
Performance of policy. J(x)
Train cevor
Loss 2 evaluated on train data, Pmin
Objective in Equations 2,
the transitions in the offline dataset D. and DTT, TT)) denotes a closeness con- straint to the behavior pol- icy, effectively applied by the offline RL method. Our definition of conservative offiine RL requires that this divergence be computed in expectation over the state visitation distribution of the learned policy π in the empirical MDP as discussed in Appendix F.I. For example, Equation translates utilizing Deou (p. q) := Σx p(x) (p(x) (xx - 1) in Equation 2 (see Theorem 3.5 in Kumar et al. [2] for a proof). The training loss is discussed in Equations and and the test loss is equal to the negative of the actual return (TT) of the learned policy. Analogously to supervised learning, we can use the notion of train and test error to define overfitting and underfitting in offline RL, as discussed in Table 1. However, note that the conditions summarized in Table 1 are not measurable completely offline. Precisely estimating if run of an offline RL method overfits or underfits requires evaluating the learned policy via interaction with the real-world environment. In Section 3, our goal will be to devise offiin metrics for characterizing overfitting that do not have this requirement. We will tailor our study specifically towards CQL, though we extend it to BRAC in Appendix F.I. A similar procedure could be devised for other offline RL methods, but we leave this for future work.
Overiitiing
((Dmi)) ) low, ((D,a) high, Deu is validation set drawn L.d. Dinin
Training objoctive Equation is ex- temely low, low value of J(x) Training objoctive Equation is ex- temely high. low value of J(z)
Underfiting high value of train enor ((Dmia)
Table 1: Summary of train error, error and our definitions of overfitting and underfitting in supervised learning and conservative offline RL methods. We will propose metrics to measure these phenomena in a purely offfine man- ner and recommend how to tune the underlying method accordingly.
3 Detecting Overfitting and Underfitting in Conservative Offline RL
In standard supervised learning, we can determine if method overfits or underfits by comparing the training loss to the same loss function evaluated on held-out validation dataset, which serves as a "proxy" test dataset. In contrast, the return of the learned policy J(T) in RL does not have direct proxy that can be computed offline. Thus, our goal is to identify offline metries and conditions that allow us to measure overfitting and underfitting in conservative offline RL, with a focus on CQL. We also adapt these conditions to BRAC [16], policy-constraint method in Appendix F22
Detecting overfitting in CQL. Our definition of overfitting (Table 1) corresponds to low value for the training loss (Equation 1), but poor actual policy performance (Tx. To detect this, we analyze the time series of the estimated Q-values averaged over the dataset samples (s, a, r, r,s") E DD over the course of training with a large number of gra- dient steps. A run is labeled as overfitting if we see that the expected dataset Q-value exhibits a non- monooonot trend: the average Q-values first increase and then decrease as shown in the figure on the right. Ad- ditionally, we would see that training loss in Equation 1 eventually becomes very low. Why do we see such trend in the average dataset Q-value? Since CQL. selectively penalizes the average Q- value under the distribution p(aass) s) supported on actions with large Q-values, we would expect the Q-values states from the dataset s ~ D and the learned a ((-ss) to be small since the policy is trained to maximize the Q-function as well. This in turn would lead to an eventual reduction in the average Q-value on dataset actions, Eg ( (s, a)]. This would be visible after sufficiently many steps of training, when values have propagated via Bellman backups in Equation giving rise to the non-monotonic trend. If such trend is observed, this raises two questions, as we discuss next.
Overfitting
Policy retum Datasel Q-value
Ratur Drmlus
Training stops
What does a low average Q-value Ex.a<D(Qot (s, a)] imply about J(T)) We show in Appendix A that, in principle, CQL training (Equation 1) should never learn Q-values smaller than the dataset Monte-Card Carlo return, and the Q- Q-values should increase unless the learned policy и better than π.- Intuitively, this is because the objective in Equation I aims to also maximize the average dataset
3

---

Q-value and thus the Q-values for the behavior policy are not underestimated in expectation. Now, if the policy optimizer finds policy that attains a smaller learned Q-value than the dataset return, the policy can always be updated further towards the behavior policy so as to raise the Q-value. Therefore, Q-values can only decrease when the policy found by CQL. is better than the behavior policy. We formalize this intuition in Appendix A in Theorem A.1. Thus, a low Q-value on (s, a) E D indicates that the Q-function predicts extremely small Q-values on actions sampled from u(als). Typically, this would mean the highest Q-value actions a at state s E D are those sampled from the offline dataset, drawn from the behavior policy. Thus, policy optimization, which aims to maximize the Q-value, would make π (als) closer to the behavior policy (a(a))) on s E D.
Which training checkpoint is likely to attain the best policy performance? Tracking overfitting in supervised learning is important for selecting the best best-performing checkpoint, before overfitting becomes severe. Analogously, we can compare the average dataset Q-value across different check- points within the same run to pick the best policy. Since CQL aims to increase the average dataset Q-value (Equation 1), we would expect Q-values to initially increase, until learning starts to overfit and the average dataset Q-value starts decreasing, We should therefore select the latest checkpoint that corresponds to a peak in the estimated dataset Q-value. A visual illustration of this idea is shown in the figure on the previous page, where the checkpoint marked by the green line is recommended to be chosen. In summary, (a) to detect overfitting we can track:
Metric 3.1 (Overfitting). A low average data Q-value Es,n- (Qo a)) that decreases with more gradient steps on Equation I indicates that the offline RL algorithm is overfilling
and (b) further, given a run that exhibits overfitting, our principle for policy selection is given by:
Guideline 3.1 (Policy selection). If a run overfits (per Metric 3.1), select the checkpoint that attains the highest average dataset Q-value before overfitting for deployment.
Finally, for actor-critic algorithms [18] that update the actor slower than the critic, the next policy checkpoint after the peak in the average dataset Q-value appears must be selected. In most of our experiments, we find that simply utilizing the policy checkpoint at the point of the peak in the Q- value also leads to good results making this rare concern, but in some cases, utilizing the next checkpoint after the Q-value peak performs better empirically.
Detecting underfitting in CQL. Next, we turn to devising a procedure to detect underfitting. As summarized in Table 1, underfitting occurs when the RL algorithm is unable minimize the training objective in Equation effectively. Therefore, large values for the TD error, the CQL regularizer, both imply underfitting. A large value for the CQL reg- ularizer, R0)), indicates an overestimation of Q-values relative to their true value [2] and thus, unlike the overfitting regime, we would not ex-
Underfitting
Relum Train Enor
Policy retum
TD error/ CQL regularzerr
Optimal retum
Training steps pect the average learned Q-value to decrease with more training. Thus, one approach to predict underfitting is track both the TD error, (T), and the CQL regularize, R(0), and check if the value of even one of these quantities is large. More discussion is provided in Appendix A.
How do we determine if the TD error and the CQL regularizer are "large"? In order to determine the error of a particular run is large, we can rerun the base CQL algorithm but with models of higher capacity, which does not necessarily correspond to the function approximator size, we will discuss in Section 4. For each model, we record the corresponding training errors and check the training TD error and CQL regularizer value are reduced with capacity increase. If increasing capacity leads to a reduction in the loss without exhibiting the overfitting signs described previously, then we are in an underfitting regime. Another approach to answer the question is to utilize the value of the TD error (f))(f) and the task horizon (1/(1-2)) to estimate the overall error in the learned Q-values against the actual Q-value, which is equal to LTD ())( (0)/(1 (1 >) [23] (see Appendix A). If this overall error spans the range of allowed Q-values on the task which could be inferred based on the structure of the reward function in the task then we can say that the algorithm is underfitting.
Metric 3.2 (Underfitting). Compute the values of the training TD error, LTD(O) and CQL regularizer, R(0) for the current run and another identical run with increased model capacity. If the training errors reduce with increasing model capacity, the original run was underfitting.

---

4 Addressing Overfitting and Underfitting in Conservative Offline RL
The typical workflow for supervised learning not only identifies overfitting and underfitting, but also guides the practitioner how to adjust their method so as to alleviate (e.g., by modifying regularization model capacity), thus improving performance. Can we devise similar guidelines to address overfitting and underfitting with conservative offline RL? Here, we discuss some ways to adjust regularization and model capacity to alleviate these phenomena.
Capacity-decreasing regularization for overfitting. As we observed in Section 3, the mechanism behind extremely low Q-values on the dataset is that CQL. training minimizes Q-values on actions sampled from p(ass). Two possible approaches to preventing over-minimization of these values are (1) applying regularization such as dropout [24] on Q-function layers, similar to supervised learning, and (2) enforcing that representations of the learned Q-function match a pre-specified target for all state-action tuples. For (2), can apply techniques such as variational information bottleneck (VIB) [25, 26] regularizer on the learned representations, ((s). Formally, let (s, a) denote a state- action pair. Instead of predicting deterministic ((s) E IRd (Figure 10), we modify the Q-network predict two distinct vectors, @m E Fd and (s)s) E Rd and sample d(s) randomly from a Gaussian centered at 6m with covariance Φg, i.c., 6(s) ~ ((mm(s) diag(øy (s)). VIB then regularizes (o(mm (s), ((xa(((((s)) be close to a prior distribution, .V (0,0 1):
min Ccou (0)+ [DKL (N(om (s), (oxgg()(s))) (),, (VIB regularizer),
0
Guideline 4.1. To address overfitting, recommend using some form of capacity-decreasin regularization on the Q-function, such as dropout or the VIB regularizer shown in Equation 3.
Capecity-increasiing techniques for underfitting. To address underfitting, we need to increase model capacity to improve optimization of the training objective. Analogous supervised learning. model capacity can be increased by using more expressive neural nets (e.g.. ResNets [27], trans- formers [28]) for representing the learned policy. We use ResNets in experiments (Figure 10). However, the RL setting presents an additional challenge with capacity: while larger models in principle have more capacity, recent work [29, 21, 22] has shown that utilizing larger networks to represent Q-functions does not always improve its capacity in practice, because TD-based RL meth- ods introduce an "implicit under- paremeetrrtttiaiiionn effect that can result in aliased (i.e., similar) internal representations for different state-action inputs, even for very large neural networks that can express the true Q-function effectively. To address this issue, these works apply "capacity- increasing" regularizer to Q-function training. For instance, we can use the DR3 regularizer [22], which penalizes the dot product of (ss) and (is') aliasing. This objective is given by:
for a transition (s, a, s') D, and hence reduces
min (oQu(() + B (s))) (ss')] 9
(DR3 regularizer [22]),
(4)
Guideline 4.2. address underfitting, we recommend using some sapacity-incrastion regu- larization on the Q-function and the policy either in conjunction or separately. Examples: (I) bigger policy networks (e.g., ResNets), (2) DR3 regularizer on the Q-network.
5 Evaluation of Our Workflow Metrics and Protocols in Simulation
Next, we empirically validate the workflow proposed in Sections 3 and 4 suite of simulated robotic manipu- lation domains that mimic real-robot scenarios, from im- age observations with sparse binary rewards. We will ex- amine how applying the workflow in Section 3 to detect overfitting or underfitting and then utilizing the strategies in Section 4 affects the performance of offline RL meth- ods. An improved performance would indicate the effi- cacy of our workflow in making successful design decisions without any online tuning.
Figure 2: Simulated domains [3] use.
Experimental setup. We the environments from Singh et al. [3] to design offline RL tasks and datasets that we use for our empirical analysis. We consider two tasks: (1) a pick and place task and (2) grasping object from drawer task. Examples of trajectories in both of these simulated domains are shown in Figure 2 and are detailed in Appendix D. Briefly, the pick and place task consists of 6-DoF WidowX robot in front of tray with object. The goal is to put the object inside the
5

---

tray. A non-zero reward of +1 is provided only when the object has been placed in the box. The offline dataset for this task consists of trajectories that grasp an object with a 35% success and other trajectories that place an object with a 40% success. Our second task is grasp- ing from drawer task where the WidowX robot is placed in front of drawer and multiple objects. The robot can open or close the drawer, grasp objects from inside the drawer or on the table, and place them anywhere in the
scene. The goal is to close the top drawer, then open the bottom drawer and take the object out. Only if the object has been taken out, a reward of +1 is obtained. The offline dataset consists of trajectories with 30-40% success rate for opening and closing a drawer and other trajectories with only 40% placing success. We use a 1.0 for CQL training in all experiments, which is directly taken from prior work [3], without any tuning. However, too low or too high a values will inhibit the effectiveness of regu- lar CQL and we first need to tune a as discussed in Ap- pendix G. More details are provided in Appendix D.
Grodae::
Figure 3: Policy performance (Top) and average dataset Q-values of CQL (bot- tom) with varying number of trajectories. Vertical bands indicate regions around the peak in average Qavuu and observe that these regions correspond to policies with good actual performance.
regual
Scenario #1: Variable amount of training data. Our first scenario consists of the simulated tasks discussed above with a variable number of trajectories in the training data (50, 100, 500, 1 10000). We run CQL and track metrics 3.1 and 3.2 in each case. Observe in Figure 3 (bottom) that with fewer tra- jectories, the average dataset Q-value F, [Q a)] first rises, and then drops. This matches the description of overfitting in Section 3. Observe in Figure 4 (left) that, at the same time, the value of the CQL regularizer is very low, which is not consistent with what we expect of underfitting. Thus, we can conclude that these conditions exhibit overfitting, es- pecially with 50 and 100 trajectories. The vertical dashed lines indicate the checkpoints that would be selected for evaluation per Guideline 3.1. We fur- ther visualize the performance of the chosen check- points against the actual return of each intermediate policy in Figure 3 (top). Note that this value is ob- tained by rolling out the learned policy, and would not be available in a realistic offline RL setting, but is provided only for analysis. Selecting the checkpoint based on Guideline 3.1 leads us to select a model with close to the peak performance over the training process, validating the efficacy of Guideline 3.1.
Lodt: CCU Regabrrzer
Right: Mitigaingg Querlittnn with
Paloefaaes Tanc
Grupta bves Sbotod Deang Took
-20
OGL VB8
Figure 4: Left: CQL regularizer attains low val- ues, especially with 50 and 100 trajectories in the pick and place task, Right: Using VIB mitigates overfitting, giving rise to stable trend in Q-values and better performance which does not degrade with more training steps.
Since we detected overfitting by following our work- flow, we now aim to address it by using the VIB regularizer in the setting with 100 trajectories. As shown in Figure 4 (right), applying this regularizer not only alleviates the drop in Q-values after many training steps, but allows us to pick later checkpoints in training which perform better than base CQL on both the tasks. This validates that overfitting, as detected via our workflow, can be effectively mitigated by decreasing capacity, in this case by using VIB. We evaluate dropout, e and le regularization schemes in Appendix J.
Scenario #2: Multiple training objects. Our second test scenario consists of the pick and place task, modified to include variable number of object types (1, 5, 10, 20, 35). Handling more objects requires higher capacity, since each object has different shape and appearance. In each case, CQL is provided with 5000 trajectories. Following our workflow from Section 3, we first compute the average dataset Q-value and the training TD error. We observe in Figure 5 that, unlike in Scenario #1, Q-values do not generally decrease when trained for many steps, suggesting that the Q-function To check for underfitting, we visualize the training TD error and find that, with 10, 20 and 35 objects, TD error magnitudes are in the range of [1.0, 2.0], which suggests a
is likely not overfitting.

---

overall Q-vauue error of [30.0, 60.0] since the task horizon is 30. On an absolute scale, this error magnitude is large: since the rewards are 0/1, the range of difference between actual Q-values for any two policies is at most 30, which suggests that the error magnitude in the runs in Figure 5 are high. Hence, we conclude that this scenario generally exhibits underfitting with more objects. Indeed this trend is reflected in the policy performance that we plot for analysis in Figure 5: note that the policy return decreases with an increased number of objects, and the policy performance initially increases and saturates at suboptimal value.
Avg. Returns vs. Num Objects
TD Err vs. Num Objects
G-vales vs Num Objects
Avg Returne Pick Place
Avtrago Retuuee
Ang Qumbacin D
Averago Remens
20-
10
OgL
oigi--llsbbe OSu Badder
0.007
6.IM Grochent Skergs
0.001
0.MM
0:75
C.3M
Grautient
Steges
Gradient
Stepp
0.0M
0.1M
0.2M
0:3M
Figure 5: Performance (left), TD error (middle) and average dataset Q- values (right) for the pick and place task with variable number of objects. Note that while the learned Q-values increase and stabilize, TD error values in scenarios with more than 10 objects are large (1.0-2.0). Correspondingly, the performance generally decreases as the number of objects increases.
Gradient Stepe
Figure 6: Correcting un- derfitting by applying our workflow for 35 objects.
To address underfitting in the multi-object case, we apply the proposed to the 35-object task (results for 10 and 20 object settings are in Appendix I). We use a more expres- sive ResNet architecture for the policy and the DR3 regularizer for the Q-function together. Observe in the figure on the right that this combination (shown in red) improves policy performance this setting (compared to green), which validates our workflow protocol for addressing underfitting.
measures
6 Tuning CQL for Real- Robotic Manipulation
Having evaluated the efficacy of our proposed work- flow in simulation, we now utilize our workflow tune CQL for real-world robotic manipulation. We
"acc nd
Oaon dowwrr
test in two setups that require the robot to learn from sparse binary rewards and image observations. The settings differ in robot platform, task specification, and dataset size. Additional results and robot videos are at the following website: https://siton google. wwt./wowoo/wiiieeeeeeeee
and
Figure 7: Real-world tasks. Successful rollouts of CQL tuned with our workflow from Sections 3 & 4. Top to bottom: Sawyer lid on pot, Sawyer drawer opening, WidowX pick-place task.
Sawyer manipulation tasks [30]. First, train a Sawyer robot in a tabletop setting to perform two tasks: (1) placing the lid onto a pot and (2) opening drawer. The robot must perform these tasks in the presence of visual distractor objects. as shown in Figure 7. We directly use the dataset of 100 trajectories for each task collected by Khazatsky et al. [30] for our experiments so as to mimic the real-world use case of leveraging existing data with offline RL. We use oour-dimesiional actions with 3D end-effector velocity control in æyz-space and 1D gripper open/close action. More details regarding the setup are provided in Appendix D.
We run default CQL on these tasks and track the average Q-value, TD error, and CQL reg- ularizer value. As shown in Figure 8, the av- erage Q-value does not decrease over training, and the TD error (and CQL regularizer shown in Appendix E.2) large. Per our discussion in Section 3, this indicates underfitting. Following our guidelines from Section we utilize more expressive ResNet policy (Figure 10), which in- creases the number of total convolutional layers from 3 to 9. We observe that this reduces the values of both the TD error Figure and CQL regularizer (Appendix E.2) on both tasks. We then evaluate the learned policy over 12 trials including ones that are unseen during training.
Avg Q-value vs Architecture
Error v8 Architecture
Avg. Quviue in P2
Dosesr CQL BadNet
Pie Cij
Pot OQL, MonNer
TD Epnor (yo(0)
4.60
Ree: CTJ
Pott rig Redive
Drance: CQL ReNNa
Erezer: OX)
23K 134
124.
I2k Gradient Steps
UCK
Grodlinn Stege
Figure 8: Average Q-value and TD error Sawyer tasks as model capacity increases. Q-values increase over training with lower capacity ruling out overfitting and increasing model capacity leads to reduction in TD error indicating the presence of underfitting.
conducted with different sets of distractor objects, While the policy trained using base CQL is unable
7

---

Real WidowX
Real Widooe
Policy selection Real Widcom X
Arg. Quainn in D
Avz Quoaktee D co io i
polce
Avg. Quodie
Success Fole
Mmmmmm
22
Success
Q-ilaa (VB)
586.
100K
200K 2000.
DIREL
30.6
100%
Gradiend stege
Gradient stoges
Gralls ol starpe
Figure 9: Q-values (left) and performance of CQL with (middle) and without (right) the variational infor- mation bottleneck correction for overfitting on the real-world widowX pick and place task. Since the Q-values start to decrease with more training, workflow detects that CQL is overfitting. Using our policy selection guideline (Guidcline 3.1) enables us to choose checkpoint 50 marked with the green vertical dashed line (right) which performs well. Further, addressing overfitting by applying the VIB regularizer stabilizes the Q-values (brown) which do not decrease unlike base CQL (blue) (left). Finally, applying the VIB regularizer improves performance and reduces sensitivity to policy selection (middle).
to successfully complete either task even once attaining score of 0/12 on both tasks, the run that uses ResNet attains a significantly better success rate of 9/12 on the put lid on pot task and 8/12 on the drawer opening task, equal to 70.8% % success rate average.
WidowX pick and place task. In our second setting, we tune CQL on pick and place task with a WidowX 250 robotic arm, shown in Figure 7. The dataset consists of 200 trajectories collected by running a noisy scripted policy (Appendix D) with 35% success. We run CQL on this task and track the average Q-values, which we find initially increase and then decrease (Figure 9 (left; labeled as "Q-values")), indicating overfitting. We then evaluate our policy selection scheme, which in this case suggests deploying checkpoint 50, the immediate checkpoint after the peak in Q-values. To see if this checkpoint is effective, we evaluate the performance of a few other policy checkpoints (for analysis only) and plot this performance trend in Figure (right) as dashed line. Observe that indeed the checkpoint found by our workflow attains the highest success rate (7/9) compared to other checkpoints, which only succeed < 4/9 times.
Since overfitting is detected, we now turn to ad- dressing overfitting by adding the VIB regular- izer (Equation 3) during training. As shown in Figure 9 (left), the Q-values obtained after the addition of this regularizer (shown in brown; la- beled "Q- "Q-values (VIB)") are now stable and do not decrease over the course of training and so can choose any policy for evaluation. We evaluate multiple policies, for visualization pur- poses only, in Figure 9 (middle), we find that all of them attain a >7/9 success, comparable or better than the base CQL algorithm (Figure 9 (right)). This indicates that addressing overfitting not only leads to some gains in performance but also greatly simplifies policy selection as all checkpoints perform similarly and well. Table 2 summarizes these results below, where the bold entries denote the checkpoints found by our policy selection rule. These results indicate the effectiveness of our workflow in tuning CQL by addressing overfitting and underfitting on multiple real robot platforms.
Real-world WidowX pick and place
Method Epoch
50
75
100
200
CQL CQL+ VIB
7/9
4/9
4/9
2/9
3/9
8/9
7/9
7/9
Performance of various policy checkpoints of CQL and CQL VIB the real WidowX pick and place task (bold entry denotes the checkpoint selected by our workflow). Note that when overfitting is cor- rected via VIB, multiple checkpoints perform well.
Table 2:
Discussion
While offline RL algorithms have improved significantly, applying these methods to real-world robotic domains is still challenging due to little guidance tuning them. In this paper, devise a workflow for algorithms such as CQL and BRAC, which consists of a set of metrics and condi- tions that can be tracked by a practitioner over the course of offline training to detect overfitting and underfitting, and recommendations to addresses the observed challenges. Applying our work- flow both in simulation and the real world shows strong performance benefits. While our proposed workflow is an initial step towards practical robotic offline RL and is based on our best conceptual understanding of certain offline RL algorithms, these guidelines are heuristic. To some extent this is unavoidable, since a workflow is a set of guidelines and recommendations, rather than a rigid algorithm. Regardless of how theoretically justified it is, in the end, its value is determined by its ability to produce good results. We believe the breadth of tasks considered, which consist of two dif- ferent real robots and multiple simulated tasks, indicates its broad applicability. However, deriving theoretical guarantees regarding workflows of this type is an important direction for future research.
8

---

Acknowledgements
We thank Ilya Kostrikov, Avi Singh, Ashvin Nair, Alexander Khazatsky, Albert Yu, Jedrzej Orbik, and Jonathan Yang for their help with setting up and debugging various aspects of the experimental setup as well as for providing us with offline datasets we could test our workflow on. We thank Dibya Ghosh, anonymous reviewers, and the area chair from CoRL for constructive feedback on an earlier version of this paper. AK thanks George Tucker and Rishabh Agarwal for valuable discussions. This research was funded by the DARPA Assued Autonomy Program and compute support from Google and Microsoft Azure.
References
[1] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.082 /2, 2021.
[2] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. arXiv preprint arXiv:2006.04779 2020.
[3] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500 2020.
[4] Y. Chebotar, K. Hausman, Y. Lu, Xiao, D. Kalashnikov, Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, and S. Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749, 2021.
[5] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR, 2018.
[6] D. Precup. Eligibility traces for off off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
[7] Kostrikov and O. Nachum. Statistical bootstrapping for uncertainty estimation off-policy evaluation. arXiv preprint a 3609, 2020.
[8] C. Paduraru. Off-policy evaluation in Markov decision processes. PhD thesis, Ph. Disserta- tion. McGill University, 2012.
[9] T. L.. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.00055, 2020.
[10] O. Nachum and B. Dai. Reinforcement learning via fenchel-meckfellarr duality. arXiv preprint arXiv:2001.0188 2020.
[11] P. Thomas, G. Theocharous, and M. Ghavamzadeh. High confidence policy improvement. In International Conference Machine Learning, pages 2380-2388, 2015.
[12] S. Thomas, G. Theocharous, and M. Ghavamzadeh. High-confidence off off-policy evaluation. In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
[13] N. Jiang and L. Li. Doubly robust off-po value evaluation for reinforcement learning. arXiv preprint arXiv:1511.0372. 2015.
[14] Fu, M. Norouzi, Nachum, Tucker, ziyu wang, A. Novikov, M. Yang, M. R. Zhang, Y. Chen, A. Kumar, C. Paduraru, S. Levine, and T. Paine. Benchmarks for deep off-po evaluation. In International Conference on Learning Representations, 2021. URL https: // //opnrevieew
[15] Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.

---

[16] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1XXIv:9111.11360. 2019.
[17] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (rXXiv:509.0225) 2015.
[18] S. Fujimoto, H. Van Hoof, and D. Meger. Addressing function approximation error in actor- critic methods. arXiv preprint
2018.
[19] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: /801.0/290, 2018.
[20] I. Kostrikov, J. Tompson, R. Fergus, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050, 2021.
[21] A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under- paramntttetttiintooonnn inhibits data-efficient deep reinforcement learning. In International Conference on Learning Repre- sentations, 2021. URL https:////pppeeeeiiiee htw.:opperreiieeeeeeetorrreeeeee
[22] A. Kumar, R. Agarwal, A. Courville, T. Ma, G. Tucker, and S. Levine. Value-based deep reinforcement learning requires explicit regularization. In RI. for Real Life Workshop & Overparrameterization:: Pitfalls and Opportunities Workshop, ICML, 2021. URL https: //rrive.google wtw.:eeooooo.....oooeeeeeeeeeeee cttm//Ellee///A/A/5555aaaa-----
[23] R. Munos. Error bounds for approximate policy iteration. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML'OS '3, page 560-567, AAAI Press, 2003. ISBN 1577351894
[24] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 (1):1929-1958, 2014.
[25] A. A. Alemi, I. Fischer, V. Dillon, and K. Murphy, Deep variational information bottleneck. arXiv preprini arXiv: 1612.00110 2016.
[26] A. Achille and S. Soatto. Emergence of invariance and disentanglement in deep representa- tions. The Journal of Machine Learning Research, 19(1):1947-199) 2018.
[27] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is you need. arXiv preprint arXiv: 1706::770.0000 1706.0367,,, 2017.
[29] D. Ghosh and M. G. Bellemare. Representations for stable off off-policy reinforcement learning. arXiv preprint arXiv:2007.05520 2020.
[30] A. Khazatsky, A. Nair, D. Jing, and S. Levine. What can i do here? learning new skills by imagining visual affordances. arXiv preprint 106.0266660067 2021.
[31] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog. E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning, pages 651-673, 2018.
[32] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. 2018.
[33] OpenAl. Learning dexterous in-hand manipulation. In arXiv preprint arXiv: arXiv:/808.0000 /808.00/77, 2018.
[34] H. van Hoof, T. Hermans, G. Neumann, and Peters. Learning robot in-hand manipulation with tactile features. 2015.
10

---

[35] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforeement learning and demonstra- tions. In RSS, 2018
[36] V. Kumar, A. Gupta, E. Todorov, and S. Levine. Learning dexterous manipulation policies from experience and imitation. CoRR, abs/161 1.05095, 2016.
[37] C. Schenck and D. Fox. Visual closed-loop control for pouring liquids. In International Conference Robotics and Automation (ICRA), 2017.
[38] A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In IROS, 2017.
[39] Matas, S. James, and A. J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference Robot Learning (CoRL), 2018.
[40] R. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman. Efficient adap- tation for end-to-end vision-based robotic manipulation. arXiv Att::2004.101900 2020.
[41] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zoina, Y. Ay- tar, D. Budden, M. Vecerik, et al. A framework for data-driven robotics. arXiv preprint 1909: 12200, 2019.
[42] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
[43] Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for Asiionbased robotic control. arXiv preprint 1xXiv:8812.0056.. 2018.
[44] A. Xie, Ebert, S. Levine, and C. Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight. Robotics: Science and Systems (RSS), 2019.
[45] Y. Hristov, A. Lascarides, and S. Ramamoorthy, Interpretable latent spaces for learning from demonstration. arXiv preprint arXiv:1807.06583. 2018.
[46] Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and Levine. Model-based visual planning with sulf-supervised functional distances. arXiv preprint 1.0Iv::2012.1337333 2020.
[47] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto. Visual imitation made easy. arXiv e-prints, pages arXiv-2008, 2008, 2020.
[48] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. arXiv preprint 1rXXi::005.06411 2021.
[49] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L.. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[50] A. Mandlekar, D. Xu, R. Martin-Martin, S. Savarese, and L. Fei-Fei. Learning to generalize across hong-horizon tasks from human demonstrations, 2020.
[51] S. Lange, Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learn- ing. pages 45-73. Springer, 2012.
[52] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- 2018
ration. arXiv preprint arXiv:
[53] A. Kumar, Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
[54] X. B. Peng, A. Kumar, G. Zhang and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117, 2019.
11

---

[55] N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Way off off-py batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprinl 19XXv0/000000450 2019.
[56] A. Nair, M. Dalal, A. Gupta, and S. Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09333 2020.
[57] R. Fakoor, J. Mueller, P. Chaudhari, and A. J. Smola. Continuous doubly constrained batch 2021.
reinforcement learning. arXiv preprint ar
[58] Yu, G. Thomas, L. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optmizatiio. arXiv preprint vrXiv:2005.13239 2020.
[59] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-basee offline rein- forcement learning. arXiv preprint arXiv:2005.. 05951, 2020.
[60] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent space models. Learning for Decision Making and Control (L4DC), 2021.
[61] D. Precup, R. S. Sutton, and S. Dasgupta. Off-policy temporal-difference learning with func- tion approximation. In ICML, pages 417-424, 2001.
[62] C. Voloshin, H. M. Le, N. Jiang, and Y. Yue. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv: 191..0654, 2019.
[63] O. Nachum, Y. Chow, Dai, and Li. Dualdice: Behavior-agnostie estimation discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pages 2315-2325. 2019.
[64] R. Qin, Gao, X. Zhang, Z. Xu, Huang, Z. Li, Zhang, and Y. Yu. Neorl: near real-world benchmark for offline reinforcement learning. arXiv preprint arXiv:2102.00714, 2021.
[65] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy- based policies. In International Conference on Machine Learning (ICML), 2017.
[66] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. wdvintage-weiighee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 190000791, 2019.
[67] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2 06860, 2021.
12


Reviews:
[{"review_id": "Zyt0GWar4KE", "reviewer": "Reviewer_bymo", "summary": "Although current offline RL algorithms can learn from prior data without interacting with the environment, the learned policy has to be evaluated online. This paper proposes a workflow, including a set of protocols and metrics, to be used to adjust model capacity, regularization in offline model-free RL. The proposed workflow shows that it can successfully detect overfitting and underfitting in offline RL and thus can be used as guidelines to improve policy performance. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The motivation is strong. Policy evaluation in offline RL is an important research topic. Although there are some limitations, the proposed workflow can be used to detect overfitting and underfitting in offline RL.\n2. The paper offers well-designed experiments, both on simulators and real robots, to demonstrate the effectiveness of the workflow.\n3. The paper is well-written and easy to follow. \n\nWeaknesses: \n1. My main concern is the effectiveness of the proposed metrics and the policy selection strategy. \n    \n    <a>  For metric 4.1 (overfitting), the decrease of the average data Q-value serves as the indicator of overfitting. But whether this metric is robust enough w.r.t other design/hyperparameter choices? For example, when choosing a larger \\alpha, we may easily observe the decrease of Q-value. Also, when choosing an improper policy \\mu(.|s), such as \\mu(.|s)=behavior policy, Q-value may not decrease under overfitting.\n    \n\n    <b>  For guideline 4.1 (policy selection), the author suggests selecting the checkpoint that attains the highest average dataset Q-value before overfitting. But for example in Fig 9, the highest avg. Q-value is not corresponding to a good policy. \n\n2. My second concern is whether the proposed workflow is limited to CQL (or Q-value regularized algorithms). In CQL, the Q function is trained with equation (1), but there are also many other offline RL methods, such as TD3_BC (https://arxiv.org/abs/2106.06860) etc., which add the constraint on policy while keeping the Q function untouched. In these algorithms, I’m afraid the proposed metrics are not suitable, e.g., even under overfitting, the Q function maybe won’t decrease like in CQL.\n\n3. The proposed workflow, in my understanding, can only be used to detect overfitting and underfitting. But for the rest hyperparameters, for example, the \\mu(.|s), \\alpha in eq. (1), we still need to tune them via online evaluation. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper has several weaknesses as listed above, but considering the workflow can be successfully used to detect underfitting and overfitting as demonstrated in two different real-world robots and multiple simulated tasks, my recommendation is \"Weak Accept\".", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "The issues are listed above. Please check the \"Strengths And Weaknesses\".", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "IhaW48quz-", "reviewer": "Reviewer_J1Lk", "summary": "Paper presents two metrics for detecting underfitting and overfitting in offline RL: 1) the magnitude of Q value 2) the TD-error vs. model size. The authors also provide suggestions to address these two issues in the light of the proposed metrics. Experiments on both simulated and real-world offline RL tasks demonstrate the effectiveness of the resulting model selection.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n[+] The paper is overall clear and well-written. I like the illustrative writing style and I believe this paper is mostly readable to those who are not familiar with offline RL and related topics. \n\n[+] The research problem seems to be incremental and it is indeed important -- model selection can be particularly difficult for offline RL as ideally no online eval should be allowed during training. The authors do bring intuitive and interesting perspectives to this problem.\n\n[+] Although personally, I don't think this paper presents significantly novel ideas, the method here is well-motivated, reasonable, and technically sound. \n\n[+] The empirical evaluations are impressive. I like how the authors extensively test their method on both simulated and real-world domains and attain consistent improvement over baselines. \n\n[Weaknesses]\n\nAt this point, I cannot find any major issues with this paper. But I have to point out that the two metrics here are straightforward, especially for the underfitting one. The one on overfitting deserves more inspection as a decrease of Q value could also due to some other factors like weight decay. \n\nAs a side note, is it possible to extend these two metrics to other offline RL algorithms? I feel the first one highly relies on CQL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Given the overall technical quality, I recommend an acceptance at this point. But I do hope the authors can address some of my concerns in the rebuttal and revise their paper accordingly.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "1. More inspection on the decrease of Q value. Is overfitting the only possible factor that leads to this effect?\n\n2. Is it possible to apply these two metrics to other offline RL algorithms?", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "Dw3q5FeyVht", "reviewer": "Reviewer_rBLy", "summary": "The paper discusses several practical considerations for conservative offline RL methods (CQL specifically). It provides metrics for practitioners to identify overfitting and underfitting of the algorithm without evaluating the policy with any online data. It also provides corresponding guidelines when overfitting or underfitting happens. The claims are supported by thorough experimental analysis on both simulated and real robots.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The issue discussed in the paper is important and also very common in offline RL when applied to robotics. The paper provides a thorough discussion on it and thus provides a strong contribution to the field.\n- The claims are well supported by the analysis in the experiment section.\n- The paper is well organized and easy to understand.\n\nWeaknesses:\n- Scenario #2: The capacity-increasing measures are evaluated using the 35-object task. From the policy performance plot and the TD-error plot on the website, the improvement of performance is not clear to me. A more extensive study is needed to verify the hypothesis of underfitting. Maybe it would be interesting to see the same experiments for the 10-object and 20-object tasks. Since they actually have larger TD errors than 35-object from Figure 5, we might see a clearer improvement. \n- Line 172-175: “Thus, a low Q-value … on states in the training dataset.” Can you clarify this further?\n- Figure on the right of Line 340-346: Missing figure number and description. The legend covers a part of the curves and should be adjusted.\n\nExtended questions:\n- How much does the CQL hyperparameter alpha affect the conclusions? For the overfitting case, would alpha affect the drop of the Q-value?\n- For the robot experiment, how are the instability and underfitting related to overestimation?\n- Line 415: “Our workflow is also specific to conservative offline RL methods, and particularly CQL.” It would be great if the authors can discuss this a bit more. To what extent do the conclusions hold for general offline RL or online RL methods?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes metrics and guidelines for overfitting and underfitting issues of CQL. It provides a thorough analysis of the issues and solutions with experiments in simulation and in the real world with considerably large-scale and challenging manipulation tasks. The paper provides important insights for offline RL. It is both technically informative and enjoyable to read.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Address the weaknesses mentioned above and provide more discussion on the extended questions.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "5ecmjUufRx", "reviewer": "Reviewer_D6Ls", "summary": "Offline reinforcement learning is very important to applications where data can be only acquired offline (e.g., medical application); or when one wants to avoid the application of untrained policy to real systems (e.g, robotic applications).\n\nIn supervised problems, there is a clear definition of \"overfitting\" and \"underfitting\".  More in detail, overfitting happens usually when the functional space of the approximator is large and there is a scarcity of samples. The result is that the model will \"fit\" the noise in the data. Such an issue can be mitigated by shrinking the functional space, introducing regularization, by the provision of more samples (when possible), or by early stopping. Underfitting happens usually when the functional space of the approximator is too small, or the regularization term is too strong, and the approximator will exhibit a high bias. Underfitting can be usually avoided by introducing a larger functional space or by weakening the effect of the regularization.\n\nEary stopping is a technique used along with gradient optimization techniques. The idea is that in the early stages of the optimization, the functional approximator is probably underfitting the function, while in the last stage of the learning procedure, the function could be fitting the noise. Therefore, if one could estimate when the overfitting starts, could stop the optimization at that point to avoid both the problems.\nThe detection of the underfitting or overfitting can be done by measuring the loss on the validation set. When the loss on the validation set starts to increase (and the loss of the training set keeps decreasing), we can detect the overfitting.\n\nMost of the reinforcement learning literature focuses on online problems. Therefore, there is not a clear training set and validation set. Furthermore, the overfitting can be easily mitigated by the provision of new samples. \n\nContrarily, in offline reinforcement learning the training set is fixed.\n\nThis paper attempts to define overfitting and underfitting for offline reinforcement learning (focusing on conservative Q-Learning). Furthermore, since the most natural metrics would require interaction with the environment, the authors propose some heuristics to detect the underfitting or the overfitting without the interaction with the environment. Furthermore, they propose some techniques to overcome underfitting and overfitting.\n\n\n\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strength__\n\nThe motivation of the paper is really strong. I believe that offline reinforcement learning is crucial in many applications. Furthermore, most of the literature on reinforcement learning does not mention the problem of overfitting or underfitting. The problem is even more important in offline reinforcement learning, as there is no possibility of providing more samples. \nMeasuring these quantities without the possibility of interacting with the environment is also very problematic. \n\nFor this reason, I believe that the attempt of the authors of providing a solution to this problem is very important for the community.\n\nEven though, as I will write later, I am not fully convinced by the heuristics proposed in the paper, the experimental section is in my opinion strong enough to support the statements made. \n\n\n__Weakness__\n\n\nA first weakness of the paper is to consider the specific case of CQL. I do believe that overfitting or underfitting is a much larger and more general problem than a specific algorithmic solution such as CQL. I regard the regularization term in CQL as an attempt to avoid overfitting. The \"surrogate\" objective introduced in CQL, is a modification of the real objective that includes a regularization term. Therefore one should still focus on the real objective. \n\nAnother weakness of the paper is to look at the average $Q_\\theta(s, a)$. I believe that a much better metric should be $J(\\theta)$, which in terms of the algorithm can be estimated with $\\mathbb{E}_{\\mu_0}[V_\\theta(s)]$. This is the metric that we want to optimize. Maximizing the average Q-function tells only how we are maximizing the surrogate objective. Refer to [1, 2, 3] to see why maximizing this objective is \"wrong\".\n\nThere are many forms of regularization. As said, I already believe that the regularization term in CQL is avoiding overfitting. Although I believe that VIB can be a good way to regularize the Q-function, I don't see why other simpler techniques have not been tested (L2, L1, ...).\n\n\n\n[1] Section 5.1 of https://arxiv.org/pdf/1811.09013.pdf\n[2] Section 3 paragraph \"Training Mismatch\" of http://proceedings.mlr.press/v97/fujimoto19a/fujimoto19a.pdf\n[3] Section 4.1.2 of https://arxiv.org/pdf/2010.14771.pdf\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I do recommend the acceptance of the paper, mostly because I think it is an important attempt to a problem that is really interesting for the community.\nI am convinced that the paper is good, even though the proposed heuristics are relying on intuition, and I do not entirely believe that they are correct.\n\nUpdate\n==\n\nI am satisfied with the author's answer and I am keeping my score unchanged. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1) Metric 4.1.\n\nIn my opinion, the overfitting happens when the algorithm estimates a large return $\\mathbb{E}_{\\mu_0}[V_\\theta(s)]$, while on the real environment the return starts actually to decrease.\n\nI do understand that metric 4.1. detects sort-of when the maximization of the $Q$-function is already \"over\", and the regularization term starts to kick in. However, suppose that the network of the Q-function is _really_ small, and therefore overfitting cannot possibly happen. \nStill, I can imagine that in that case, the metric 4.1. can still happen (when eq.1 reaches \"convergence\", and when the optimizer must exploit the regularization terms to increase the term in eq 2).\n\n2) Metric 4.2. \n\nEven though I generally accept the idea to train the dataset on two different functional approximators to detect the underfitting (if the one with higher capacity has a lower loss, then _probably_ the smaller one is underfitting); still I do not fully agree in the case of RL.\nFor example, can be that the smaller functional approximator is actually approximating the Q-function better, even though the empirical error is higher. In fact, it can happen that, if the Q-function has a higher magnitude, the magnitude of the error will also increase.\n\nI know this effect, as I often noticed that at the beginning of the learning, the Bellman error tends to usually increase, as the magnitude of the Q-function also increases. That does not mean that the approximation is worst: actually, it is better, but the loss does not reflect it monotonically.\n\n__side note__\n\nI'd modify eq.2 with the following, for clarity, \n\n$$\n\\pi^* := \\mathrm{arg}\\max_\\pi J_D(\\pi) - \\alpha D(\\pi, \\pi_\\beta)\n$$\n\nas $a$ and $s$ do not appear on the right-hand side of the equation.\n\n"}}]

Author rebuttal:

Meta-review: The paper studies an important topic of accessing overfitting and underfitting in CQL. 

All reviewers see merits in this work but also point out weaknesses and have concerns that I share.
I agree with the evaluation of the reviewers and invite the authors to answer the raised questions and indicate how the paper will be improved for the camera-ready version. 
In particular: (however, this does not mean the other points should not be addressed)
 - Scenario #2 regarding the object 35 task and alternative tasks
 - How much does the CQL hyperparameter alpha affect the conclusions? For the overfitting case, would alpha affect the drop of the Q-value?
 - The relation and applicability to other offline RL algorithms

The authors have addressed most of the concerns and have added several additional experiments. 
I recommend acceptance of the paper.
 
Decision: Accept (Oral)