FOSP: Fine-tuning Offline Safe Policy through World Models

Authors: Chenyang Cao, Yucheng Xin, Silang Wu, Longxiang He, Zichen Yan, Junbo Tan, Xueqian Wang
Primary area: reinforcement learning
Keywords: Safe RL, Offline-to-online RL, Robot Learning

Abstract:
Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize to unseen scenarios safely. In this paper, we aim to improve safety during the deployment of vision-based robotic tasks through online fine-tuning an offline pretrained policy. To facilitate effective fine-tuning, we introduce model-based RL, which is known for its data efficiency. Specifically, our method employs in-sample optimization to improve offline training efficiency while incorporating reachability guidance to ensure safety. After obtaining an offline safe policy, a safe policy expansion approach is leveraged for online fine-tuning. The performance of our method is validated on simulation benchmarks with five vision-only tasks and through real-world robot deployment using limited data. It demonstrates that our approach significantly improves the generalization of offline policies to unseen safety-constrained scenarios. To the best of our knowledge, this is the first work to explore offline-to-online RL for safe generalization tasks. The videos are available at https://sunlighted.github.io/fosp_web/.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730678447133

thread_type:
official_review

<thread_object>
creator: reviewer_uhwz

<content>
summary:
This paper studies the problem of offline-to-online finetuning of learned policies with safety constraints. This is, to my knowledge, the first work to study policy safety in this setting; prior work either focus on either offline or online RL exclusively. The key contribution of this paper is an algorithm, FOSP, that builds upon model-based RL algorithm Dreamer to enable fast and safe offline-to-online finetuning on tasks with visual observations, both in simulation and on a real robot. The algorithm works by first pretraining a world model on an offline dataset (using the in-sample trick from IQL), and then finetuning the model during the online phase using safe policy expansion to minimize safety violations. Experimental results indicate that FOSP is effective at reducing constraint violations while maintaining reasonably good task performance (i.e. rewards) compared to prior methods for safe RL. Vanilla Dreamer without regards for safety achieves consistently high reward (more than any other approach) but has significant safety violation.

strengths:
- The paper studies an interesting problem (offline-to-online safe RL) that to my knowledge has not been explored much (if at all) by prior work. I appreciate the substantial background and derivation of approach which will help readers appreciate the technical contributions more. I believe that there is adequate discussion of related work.
- The paper is generally well written and easy to follow. The paper has minor grammatical errors but they do not detract significantly from my understanding.
- The resulting algorithm appears to rather practical, and can be applied to tasks with visual observations in both simulation and on a real robot. A significant amount of prior work in safe RL does not consider visual observations nor real robot experiments.
- Baseline methods appear to be quite strong. I appreciate including vanilla Dreamer for comparison (it can be considered an ~upper bound in terms of task performance with no regard to safety), as well as SafeDreamer as a recent and highly related method.

weaknesses:
- The considered tasks are fairly toy. It would be useful to know how the method performs on more realistic problem settings. I believe that the real-world task is a nice step in that direction, but being simply a reaching task it does not have any of the additional complexity that usually comes with real-world tasks: object interaction, real-time control, dynamic tasks (environment might change even with an all-zero action step). Especially the latter two can have significant safety consequences.
- A substantial limitation of the method appears to be the lack of control or predictability in how safe the model really is during finetuning / at test-time when presented with OOD data, e.g. novel visual observations beyond scene re-configurations. I did not find any significant discussion of this in the paper, so it would be useful to clarify (verbally or with empirical evidence) when the proposed method can be expected to succeed / fail in a transfer setting; see my question below for more context.

questions:
I would appreciate if the authors can address (in whichever way they deem appropriate) my comments in the "weaknesses" section above, as well the following questions:

- How would the proposed method behave in a transfer task setting where the change is more visual in nature? E.g. the target object is now green and the obstacles are now red. I suspect that such novel settings would trick the method into substantial safety violations. It would be informative to either include experimental results or a written discussion on the types of transfer settings in which the method can be expected to maintain low safety violations.
- Do the authors plan to release code that reproduces their main experimental results? I did not find any mention of that in the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702530646

thread_type:
official_review

<thread_object>
creator: reviewer_e4wn

<content>
summary:
The paper proposed an offline-online safe RL framework based on safedreamer, integrating the concepts from RecoverRL. When feasible, the method optimizes rewards, and when infeasible, it focuses on cost optimization. Additionally, the reachability estimation function from RESPO is utilized to introduce cost into the optimization process. To mitigate the issue of inaccurate critic estimates during offline training, in-sample actions from Implicit Q-learning are employed in the Q function learning.

strengths:
1. The proposed safe model-based RL framework effectively addresses offline-online generalization tasks.
2. It demonstrates the capability to safely fine-tune in previously unseen safety-constrained scenarios during real-world deployment.

weaknesses:
1. In Figure 4, the results for DreamerV3 could be omitted, as they overshadow the cost performance of all baseline methods.
2. There are too few obstacles in the real-world environment, making it difficult to assess the agent's obstacle avoidance behavior. And the website does not provide any differences between the proposed algorithm and the baseline in the video demos.
3. In Figure 4, why do the rewards for FOSP in PointGoal2 and PointGoal1 not continue to rise? Additionally, SafeDreamer does not show an increase in reward for PointButton1. Does this indicate that the fine-tuning phase was ineffective? According to the paper's description, the offline data comprises a mixture of unsafe, safe, and random policies; thus, the policy trained on offline data should not be optimal, and performance should improve during the online fine-tuning phase.

questions:
1. I'm curious about the video demonstration of the performances of SafeDreamer and FOSP in the SafetyFadingEasy and Hard environments. Based on the experimental results you presented, FOSP seems to perform better than SafeDreamer. This environment tests the agent's memory, yet it appears that FOSP does not specifically address this aspect. What accounts for the performance improvement?
2. Why do you need to learn Q(s, a) when I recall that DreamerV3 only uses V(s)?
3. How did you adapt Recovery RL to the image-based setting, and how did you modify SafeDreamer to the offline setting?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707911245

thread_type:
official_review

<thread_object>
creator: reviewer_gtcr

<content>
summary:
This paper proposed a model-based offline-to-online safe RL algorithm, FOSP. The proposed framework first uses an offline dataset to learn a world model and a pre-trained policy. The pre-trained policy is then fine-tuned through online safe RL fine-tuning. The effectiveness of FOSP is demonstrated through experiments in the Safety gym and a real-world robot experiment.

strengths:
Extensive experiments and ablation studies were conducted, including a real-world robot experiment with high-dimensional visual observations. The proposed FOSP algorithm achieved good performance, especially in the real-world robot experiments, and outperformed baseline approaches.

weaknesses:
While the proposed algorithm performed well in extensive experiments, I found it hard to appreciate its technical contributions due to its complex algorithm design (e.g., many moving parts, iterations between offline and online learning) and confusing presentation. It is unclear to me if all the design choices are necessary and which components are novel or come from the literature. It would be good if the authors could clearly state their technical contributions and novelty (e.g., does the novelty mainly lie in a new combination of existing components in the literature?). It would also help if the authors could provide more insights on why FOSP would outperform baseline approaches, especially SafeDreamer.

questions:
1. Where does the behavior policy $\pi_b$ in Sec. 4.2 come from? Does it refer to the offline RL policy from Sec. 4.1?
2. Does the training step in Sec. 4.2 also occur offline? Why is it necessary to divide the offline learning stage into two steps?
3. The derivation in Sec. 4.2 is very hard to follow. What is the advantage of using the reachability estimation function from RESPO? It would be helpful to introduce RESPO in the Preliminaries Section. Is $1(s\in S_f)$ equivalent to $u^\pi(s)$? How is Eqn (14) derived from Eqn (13)? Why would introducing advantage functions simplify the constraints? 
4. The limitation and future work section should be moved to the main text.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714452406

thread_type:
official_review

<thread_object>
creator: reviewer_2vut

<content>
summary:
This paper presents FOSP, a novel safe reinforcement learning method. FOSP primarily addresses the issue of enhancing safety through offline training and online fine-tuning. Its main contributions include:

1. The introduction of an offline-to-online reinforcement learning framework.
2. Enhanced safety and performance balance during visual tasks through the integration of offline and online phases.
3. In real-world deployments, FOSP allows safe fine-tuning in unseen safety constraint scenarios.
4. The experiments validate the method's effectiveness in both simulated and real robotic tasks and demonstrate its safe generalization ability in new scenarios.

strengths:
1. The FOSP method introduces the concept of safe generalization within the offline-to-online reinforcement learning framework, combining world models with offline training and online fine-tuning to enhance safety and performance. This novel strategy for applying reinforcement learning in safety-critical scenarios is insightful.
2. The design of experiments is comprehensive while covering tasks in both simulated environments and validations in real robotic settings.
3. The experiments of FOSP in real robotic tasks demonstrate real-world value and safe generalization in unseen scenarios.
4. The paper clearly articulates the optimization objectives at each stage, aiding in understanding.

weaknesses:
1. While the experiments were validated in Safety-Gymnasium and real robotic environments, the number and complexity of tasks remain relatively limited.
2. The motivation for introducing the new offline to online setting in the paper has not been well communicated.

questions:
1. Although this method performs better than others, it also introduces a new design for online to offline transitions. What specific problem is this design intended to address? The reviewer hopes the authors can clarify the benefits of this setting. For instance, the reviewer notes that in Table 1, the performance of SafeDreamer (offline) is actually comparable to that of FOSP.
2. How is safety ensured in this method, particularly when dealing with unseen data and new environments? Specifically, how does the online fine-tuning process avoid safety hazards? For instance, how does the constraint violation during online fine-tuning compare to the converged offline policy?
3. The reachability estimation function is a key component of this paper. How is the feasible part determined through $ S_f(\boldsymbol{s}):=\{\boldsymbol{s}|V_\varphi^c(\boldsymbol{s})=0\} $? In particular, how does this hold when $ V_{\varphi}^c(\boldsymbol{s})=0 $ may not strictly hold in the fitted value function? 
4. How do the authors view the potential of this method for safe sim2real applications? That is, training under simulated environment data while ensuring safety and improving performance during real-world deployment.
5. The safety policy expansion mechanism stabilizes performance during the initial fine-tuning phase, but does it impose limitations on long-term fine-tuning? For example, what would the comparison of results be between the FOSP(online) version and FOSP in short-term versus long-term training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732096384366

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper revise instruction

comment:
We appreciate the suggestions from all the reviewers. The revised version of the paper has been submitted and the revisions are highlighted in red. We would like to write instructions to help reviewers quickly find the corresponding revision.

Main body:
- In Introduction, we add the motivation for introducing safe offline-to-online setting in this paper following advice from Reviewer 2vUt
- In Preliminaries, the introduction of Reachability Estimation Function is added according to Reviewer gtcR’s suggestion.
- In Method, we rewrite the reachability estimation function to make it correspond to the definition. Additionally, we clarify why the method introduces Augmented Lagrangian and advantage function.
- In Conclusion, we state our limitations and guide readers to find the details in Appendix.

Appendix:
- In Appendix H.3, we stress the motivation to transfer our method to Fading environment. 
- In Appendix H.3, we add the results of the average reward and cost for the transfer task from SafetyPointGoal1 to SafetyPointGoal2. It indicates that the converged offline policy might violate some unseen constraints in a new environment. And the fine-tuning process can help the agent quickly reduce the violations.
- We illustrate Figure 14, which is a version of Figure 3 with DreamerV3 removed. 
- We add the comparison of FOSP (online version) and FOSP to further demonstrate the effectiveness of fine-tuning to Appendix H.4. 
- We devise a transfer task with more visual variations in Appendix H.5. We analyze the reason why our method fails to handle the visual change task.
- We add an experiment on a more realistic environment Race in Appendix H.6. We show FOSP has better performance with lower cost even in more complex environments.
- In Limitations, we add the simplicity in the real-world experiment and the failure in novel visual observations. In Future Works, we give some promising solutions to provide the community with more insights into this field. In addition, we give some analysis in safe sim2real following Reviewer 2vUt’s suggestion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732096675653

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
We sincerely thank all reviewers for their constructive and thoughtful feedback. The reviewers agreed that the safe offline-to-online method FOSP we proposed is effective in solving the safe generalization problem. We would like to summarize the strengths as follows:

1. The method FOSP uses safe offline-to-online RL to solve safety-critical generalization tasks, which is **novel and insightful**.
2. The experimental design is **comprehensive**, effectively demonstrating that FOSP can safely fine-tune in new environments, showing its strong performance with sufficient baselines.
3. The experiments also include **real-world robotic trials**, making it one of the few works to deploy safe RL into the real world.
4. The paper includes **detailed derivations** of the proposed method, making readers easy to follow.

However, the reviewers had some concerns. We have made every effort to address their concerns by adding detailed explanations and supplementary experiments. The specific points are as follows:

- Reviewer 2vUt, E4Wn and UhwZ are concerned the experiments are **relatively simple**. We have supplemented the Race experiment to show our method can perform well in a more realistic environment. For the real-world experiment, our method has been preliminarily validated for real-world transfer. Some more complex tasks like dynamic obstacles require specific algorithmic designs such as the prediction module, which are not the core of the problem we aim to solve. Thus, we would leave them for future researchers to further explore and we have proposed some promising solutions in the Future Works.
- Reviewer UhwZ wanted to know the impact of **visual changes** on our method. We have conducted an experiment for significant visual changes in the simulation environment. It indicated that our method struggled to handle it. However, it is mainly attributed to the vision-only input of the model rather than our main design. Hence, we propose the addition of semantic information as a possible solution and write it into Future Works.
- Reviewer E4Wn felt confused with some results in **Figure 4** (Figure 3 in the new version). Therefore, we presented the detailed evaluation results in Figure 4 with tables to further demonstrate the effectiveness of the fine-tuning process.
- Reviewer gtcR found it difficult to understand the **technical contributions**. We first clearly clarified our thought process. Then, we used both theory and experiments to explain the importance of the components and why our method outperforms the baselines. Meanwhile, we add more detailed explanations to the paper to help readers better understand our method.
- Reviewer 2vUt thought the **motivation for the new setting** was not well communicated. We have explained the motivations and supplemented them in the paper.

Again, we greatly appreciate all the efforts the reviewers and AC made in providing their suggestions. The reviewers' comments are invaluable for us since they will help us improve our paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes FOSP, an offline-to-online safe RL framework that uses a model-based approach to fine-tune an offline-trained policy. By incorporating in-sample optimization and reachability guidance, FOSP aims to ensure safe performance even in unseen scenarios. Experiments in simulation (Safety-Gymnasium) and real-world tasks demonstrate that FOSP generalizes better and reduces constraint violations compared to SafeDreamer and other baselines.

Strengths:

-- Novel Framework: Presents a new offline-to-online approach to safe RL, integrating model-based updates and safety mechanisms for vision-based tasks.

-- Extensive Experiments: Validates performance in both simulated and real-world settings, underscoring the method’s practical relevance.

-- Real-World Deployment: Demonstrates successful application on real robots, showing promise for real-world safety-critical tasks.

Weaknesses:

-- Limited Task Complexity: Experiments focus on relatively simple tasks; the method’s performance on more complex or dynamic scenarios remains unclear.

-- Algorithm Clarity: The design involves multiple components and iteration steps, making it hard to separate novel elements from existing approaches.

-- Safety Guarantees: More discussion is needed on the method’s robustness to out-of-distribution data or dramatic environment changes.

After carefully reading the paper, the reviews and rebuttal discussions, the AC agrees with the reviewers on recommending to accept the paper.