Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrations

Authors: Bryan Chan, Anson Leung, James Bergstra
Primary area: applications to robotics, autonomy, planning
Keywords: Reinforcement learning, learning from demonstrations, image-based grasping

Abstract:
Offline-to-online reinforcement learning (O2O RL) aims to obtain a continually improving policy as it interacts with the environment, while ensuring the initial behaviour is satisficing.
   This satisficing behaviour is necessary for robotic manipulation where random exploration can be costly due to catastrophic failures and time.
   O2O RL is especially compelling when we can only obtain a scarce amount of (potentially suboptimal) demonstrations—a scenario where behavioural cloning (BC) is known to suffer from distribution shift.
   Previous works have outlined the challenges in applying O2O RL algorithms under the image-based environments.
   In this work, we propose a novel O2O RL algorithm that can learn in a real-life image-based robotic vacuum grasping task with a small number of demonstrations where BC fails majority of the time.
   The proposed algorithm replaces the target network in off-policy actor-critic algorithms with a regularization technique inspired by neural tangent kernel.
   We demonstrate that the proposed algorithm can reach above 90\% success rate in under two hours of interaction time, with only 50 human demonstrations, while BC and two commonly-used RL algorithms fail to achieve similar performance.

Decision: reject
Reviewer scores: 3, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729523313603

thread_type:
official_review

<thread_object>
creator: reviewer_awmg

<content>
summary:
This paper proposes a regularization method for CQL, improving efficiency in the offline-to-online RL setting. Inspired by the neural tangent kernel (NTK) literature, the paper replaces the target Q network with an NTK regularization term. Experiments are conducted on an image-based grasping task using a vacuum. Results show that the method outperforms BC and RL baselines after online RL and does not rely on a pre-trained image encoder.

strengths:
1. To my knowledge, the proposed NTK regularization for offline-to-online CQL is a novel approach that addresses the learning stability issues.
2. The paper demonstrates promising experimental settings on a real-world robotic grasping task, taking acceptable training time to improve the policy on the real robot.
3. The paper presents a detailed experimental analysis of the grasping task.

weaknesses:
1. My main concern is that the proposed method appears to be a general approach for offline-to-online RL rather than designed for robotic grasping. The motivation behind the proposed regularization term does not seem related to the task of robotic grasping. However, the majority of the paper is written as if it is focused on robotic grasping.

2. Because of this, I suggest that the experiments should include various RL benchmarks, such as D4RL, to demonstrate the general effectiveness of the proposed algorithm, rather than focusing solely on a single vacuum grasping task.

3. Regarding the literature on robotic grasping, the use of a vacuum to grasp flat objects, as demonstrated in the paper, represents a relatively simple setup. It remains unclear whether the method would be effective with parallel grippers or in grasping more complex objects. If hardware devices are not available, conducting simulation experiments on these settings could strengthen the experimental results.

questions:
Please see the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730307340043

thread_type:
official_review

<thread_object>
creator: reviewer_l1gj

<content>
summary:
The paper proposes a variation on conservative Q-learning (CQL) to use when training a robot grasping policy in a low data regime.

Drawing inspiraction from the Neural Tangent Kernel line of work, the Q-function is parameterized as $Q(s,a) = w^T \phi(s,a)$ where $w$ are the weights at the last layer and $\phi(s,a)$ is some (potentially deep net based) featurizer. If we froze the weights of $\phi(s,a)$, then the kernel $\kappa(s,a,s',a')$ would be exactly $\phi(s',a')^T\phi(s,a)$, and this controls the amount of generalization occurring between state action pairs.

In this work, we do not freeze $\phi(s,a)$, but instead add an auxiliary loss to CQL to keep $\phi(s',a')^T\phi(s,a)$ close to 0 to reduce overgeneralization (a key challenge in offline RL). This is done with an mean squared loss, $E_D[ (\phi(s, a_u)^T \phi(s', a_\pi'))^2]$, encouraging decorrelation between a uniformly sampled action $a_u$ and the current policy's action $a_\pi$.

This method is studied in a real-life grasping task.

strengths:
The paper does a good job of presenting the core change of their method, as well as motivating why it could help with overgeneralization. The model architectures used are well-defined

weaknesses:
The method doesn't seem to meaningfully build on prior work.

* RL and BC have been applied to image-based grasping before, including in the low demonstration regime.
* Image-based RL has on occasionally been done before as well (with SAC)

The absolute performance numbers relative to the BC baseline seem quite low. The paper is correct in observing that the NTK objective from Kumar et al 2022 is partly ill-formed: minimizing $E_D[ \phi(s, a_u)^T \phi(s', a_\pi')]$ allows for a negative kernel value, which could let learning in one Q-value cause unlearning in another Q-value. However, the Ma et al 2023 citation is described as also doing this, which is incorrect. The Ma et al 2023 paper minimizes the absolute value $|\phi(s, a_u)^T \phi(s', a_\pi')|$, i.e. it uses an L1 loss. This paper does technically do something different (L2 loss instead), but the results of doing so aren't great.

In short I'm not very convinced on novelty or effectiveness. This combined with using a single hard-to-reproduce robot environment make me believe the paper is not good enough.

questions:
Could the authors explain why they remove the target network when including the NTK regularizer? As far as I can tell, the target network is maintained in prior work that uses NTK regularization, I am not sure why it was removed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646785832

thread_type:
official_review

<thread_object>
creator: reviewer_cv3d

<content>
summary:
This paper presents a novel regularizer inspired by neural tangent kernel (NTK) literature that can decorrelate the latent representation of different state-action pairs, while maintaining reasonable Q-value estimates. This is presented as a novel offline-to-online reinforcement learning algorithm to learn image-based tasks with small number of demonstrations. The proposed algorithm replaces the target network in off-policy actor-critic algorithms and is shown to overperform alternative methods.

strengths:
The paper is structured fairly well and evaluations are carried out with experiments and comparisons that demonstrate the proposed algorithm.
The method presented in well framed in relevant literature and presented and formulated in a sound way.
The work is well motivated and its application is relevant to the field, and especially interesting for real-robot applications.
The experimental setup is explained well and plots and graphs are clear.
The paper analyses different aspects of the proposed method, eg latent feature, Q-estimates, performance against BC and SAC.

weaknesses:
The clarity of the paper can be improved in terms of readability and crispness of contributing statements. For example in Figure 3, it is not immediately clear from the figures why the three plots show performance of different algorithms, while the caption is very verbose and sometime vague (eg "various image encoder" could be more specific).
While it is nice to see comparisons with BC and SAC, it would be interesting to see performance of conservative Q learning as well, as this method is based on it.
Other experiments would have helped support the contribution, eg experiments on D4RL, robomimic, or other available data/tasks.

questions:
- In which cases does your method stop outperforming others? for example, is there a dependency on the number or quality of demonstrations?
- Can you clarify what aspect in particular of the NTK helps most? could other kernel be used/compared? if not, why?
- Did you test your algorithms on other baselines/tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700647676

thread_type:
official_review

<thread_object>
creator: reviewer_tct9

<content>
summary:
This paper introduces a novel offline-to-online reinforcement learning (O2O RL) approach tailored for robotic grasping with limited image-based demonstrations. The authors tackle the challenge of scarce data by developing a method, called Simplified Q, which integrates a regularization technique inspired by neural tangent kernel (NTK) theory, aiming to stabilize Q-value estimates and reduce the risk of Q-divergence. This method eliminates the need for a target network, which is traditionally used in actor-critic frameworks, allowing the agent to learn effectively with only 50 demonstrations and minimal online interaction. The approach is tested on a robotic grasping task with a vacuum-based manipulator and achieves a high success rate (>90%) in less than two hours, significantly outperforming behavioral cloning (BC) and other standard RL baselines. The paper demonstrates the algorithm’s robustness through ablation studies and comparative experiments, highlighting its potential for real-world applications where exploration is costly and data is limited​.

strengths:
The proposed technique demonstrates impressive sample efficiency, allowing reinforcement learning to succeed with a limited number of demonstrations.

The paper provides strong empirical support through real-world experiments on image-based robotic manipulation, highlighting the practical applicability of the approach.

weaknesses:
This work appears to reframe concepts that are well-explored in the reinforcement learning from demonstrations (RLfD) literature. The setup of offline-to-online reinforcement learning (O2O RL), which involves pretraining a policy with approximately 50 demonstrations and then continuing online RL (with optional further use of the demonstrations), as detailed on page 4, closely mirrors standard practices in RLfD. Many RLfD approaches similarly involve pretraining on demonstrations followed by online refinement of the policy using both interaction data and demonstration experiences, in both continuous control domains, e.g. [1][3][4][5], and discrete domains, e.g. [2].

To strengthen the contribution, it would be beneficial for the authors to provide a theoretical discussion on the relationship between O2O RL and RLfD. Additionally, incorporating experimental comparisons with established RLfD approaches would offer a more robust evaluation and ensure that the proposed method’s performance is rigorously benchmarked against strong baselines.

References:

[1] Vecerik, Mel, et al. "Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards." arXiv preprint arXiv:1707.08817 (2017).

[2] Hester, Todd, et al. "Deep q-learning from demonstrations." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

[3] Zha, Y., Guan, L., & Kambhampati, S. (2024, March). Learning from ambiguous demonstrations with self-explanation guided reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 9, pp. 10395-10403).

[4] Nair, Ashvin, et al. "Overcoming exploration in reinforcement learning with demonstrations." 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.

[5] Rajeswaran, Aravind, et al. "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations." arXiv preprint arXiv:1709.10087 (2017).

questions:
Do you have insights on the number of demonstration trajectories required for effective training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731940531694

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to the Reviewers

comment:
We thank all reviewers (TcT9, cV3D, L1Gj, and awMg) for their feedback. We are pleased to see that the results from the real-life robotic task are highly appreciated and interesting (reviewers TcT9, cV3D, awMg). We are also delighted to see that our analyses are well motivated and sound (reviewers TcT9, cV3D, awMg).

Here are a few common comments regarding this work that we will address altogether. All the updates in the manuscript are reflected in blue text.
1. We understand that all reviewers are concerned about the effectiveness of Simplified Q due to the single real-life robotic experiment. Therefore we have included a few additional results. We believe the reviewers will be receptive to the extra results and find our work more convincing than previously, and as a result reconsider their scores accordingly.
    - We have included two more RL algorithms that aim to stabilize Q-divergence, namely DR3 (Kumar et al., 2022) and SAC with LayerNorm (Yue et al., 2023). Furthermore, we have compared all algorithms across three seeds rather than a single seed, with the goal of providing more convincing statistical results. All results are presented in Figures 2 and 3 of the updated manuscript. Generally, the results remain consistent with our original claim where Simplified Q can outperform the compared algorithms with a similar number of total data. Even with the similarity with DR3, we have shown that Simplified Q is more sample efficient than DR3 and further experiences less P-stops over the training run.
    - We have compared BC, CQL, and Simplified Q on a simulated environment with low-dimensional observation, robomimic (Mandlekar et al., 2022). Specifically the algorithms are evaluated on lift, can, and square, with both proficient-human (PH) and multi-human datasets. We again compare all algorithms across three seeds. We directly add the regularizer on the CQL implementation and remove the target network—Simplified Q uses the same hyperparameters as the original CQL implementation for this experiment. The implementation can be found here: https://anonymous.4open.science/r/robomimic-3DB6/robomimic/. The results are presented in Table 1 of the updated manuscript. Similar to our findings in Figure 1, middle, offline RL algorithms are inferior to BC. However, we can still observe that Simplified Q can generally outperform, if not comparable to CQL, again corroborating our findings in the real-life experiment.
    - Finally, we conducted ablation studies and sensitivity analyses on the newly introduced $\beta$ in appendix C.2. We hope all reviewers can take into consideration the effort required for running a real-life robotic experiment—it is difficult to scale due to hardware limitations and require significant human time in case of unexpected situations, e.g., the robotic arm is unable to recover from a pose due to collision.
1. Reviewers TcT9, cV3D, and L1Gj have raised questions about the differences and benefits between Simplified Q and previous work (e.g. DR3 and methods from RLfD).
    - Generally, we believe O2ORL and RLfD are very similar—O2ORL usually imposes no assumptions on the offline dataset (aside from having reward signal) while RLfD usually assumes the offline dataset is near optimal (but may exclude reward signal). This means that in O2ORL using BC as part of the policy optimization objective is undesirable and may require extra care. Unlike methods like TD3-BC (Fujimoto and Gu, 2021) and CQN (Seo et al., 2024), Simplified Q does not leverage any form of BC in the learning objectives. We also note that RLPD (Ball et al., 2023) have demonstrated that SACfD is inferior to RLPD for a wider range of tasks.
    - Our work heavily leverages similar analyses done in previous studies, as pointed out in sections 3 and 5. However, our insight is that this regularizer can simplify Q-learning by removing the target network which reduces half of the Q-parameters and reduces the number of related hyperparameters into a single scalar. On the other hand, other works complicate their algorithms by introducing extra components and hyperparameters which make deployment to real-life systems difficult—one will have to tune more hyperparameters which is often infeasible in real-life applications. We also found that generally our hyperparameter is quite robust to a range of values between [0.1, 0.4] as shown in Table 2c.
    - As far as we know, we are one of the first to successfully apply O2ORL in real-life image-based robotic manipulation without any behavioural cloning nor hand-designed heuristics (e.g. predicates or hierarchies). This is promising as this is an important step towards enabling self-improvement on real-life robotic systems.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an offline-to-online reinforcement learning approach for image-based robotic grasping, introducing a simplified Q-learning method that replaces the target network with an NTK-inspired regularization technique. Despite some interesting elements, the reviewers identified several critical limitations.

The primary concern, raised by multiple reviewers, was the paper's unclear positioning and limited technical novelty. While the authors demonstrated good results on a robotic grasping task, Reviewer awMg noted that the proposed NTK regularization appeared to be a general RL approach rather than one specifically designed for robotics, yet lacked validation on standard RL benchmarks like D4RL. This fundamental disconnect between the method's generality and the paper's robotic focus remained unresolved in the rebuttal.

Reviewer L1Gj's detailed analysis identified the main technical contributions as relatively minor modifications to existing approaches - namely, pushing towards zero correlation rather than negative correlation (compared to DR3), and simplifying OOD policy handling. These incremental changes, while potentially useful, do not represent a significant advancement in the field.

While the authors made efforts during the rebuttal to address clarity issues raised by Reviewer cV3D and added additional baselines, Reviewer TcT9 highlighted a more fundamental problem with the paper's relationship to reinforcement learning from demonstrations (RLfD). The added baseline comparisons (DQNfD, SACfD) were not state-of-the-art, and the paper failed to properly position itself within the broader context of RLfD research.

This AC has carefully considered the authors' response to TcT9's review but find the criticism valid - the paper's positioning remains unclear and the technical contributions, while potentially useful, are too incremental. While the real-world results are promising, the lack of broader validation, limited technical novelty, and unclear relationship to existing literature make this work more suitable for refinement and resubmission.