FOSP: Fine-tuning Offline Safe Policy through World Models
Authors: Chenyang Cao
Yucheng Xin
Silang Wu
Longxiang He
Zichen Yan
Junbo Tan
Xueqian Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Safe RL
Offline-to-online RL
Robot Learning
Primary area: reinforcement learning
TL;DR: a safe model-based offline-to-online RL framework to solve safe generalization tasks

Abstract: Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize to unseen scenarios safely. In this paper, we aim to improve safety during the deployment of vision-based robotic tasks through online fine-tuning an offline pretrained policy. To facilitate effective fine-tuning, we introduce model-based RL, which is known for its data efficiency. Specifically, our method employs in-sample optimization to improve offline training efficiency while incorporating reachability guidance to ensure safety. After obtaining an offline safe policy, a safe policy expansion approach is leveraged for online fine-tuning. The performance of our method is validated on simulation benchmarks with five vision-only tasks and through real-world robot deployment using limited data. It demonstrates that our approach significantly improves the generalization of offline policies to unseen safety-constrained scenarios. To the best of our knowledge, this is the first work to explore offline-to-online RL for safe generalization tasks. The videos are available at https://sunlighted.github.io/fosp_web/.

Published as a conference paper at ICLR 2025
FOSP: FINE-TUNING OFFLINE THROUGH WORLD MODELS
SAFE
POLICY
Chenyang Cao', Yucheng Xin Silang Wu', Longxiang He', Zichen Yan", Junbo Tan"
Xueqian Wang 1*
Shenzhen International Graduate School, Tsinghua University,
2 College of Design and Engineering, National University of Singapore
ABSTRACT
Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these ap- proaches heavily rely on the dataset and struggle to generalize to unseen scenar- ios safely. In this paper, we aim to improve safety during the deployment of vision- based robotic tasks through online fine-tuning an offline pretrained pol- icy. To facilitate effective fine-tuning, we introduce model-based RI., which is known for its data efficiency. Specifically, our method employs in-sample op- timization to improve offline training efficiency while incorporating reachability guidance to ensure safety. After obtaining offline safe policy, safe policy expansion approach is leveraged for online fine-tuning. The performance of our method is validated on simulation benchmarks with five vision-only tasks and through real-world robot deployment using limited data. It demonstrates that our approach significantly improves the generalization of offline policies unseen safety-constrained scenarios. To the best of our knowledge, this is the first work explore offline-to-online RL for safe generalization tasks. The videos are avail- able at https:: /tttp:://suniightee github. 1o/ /ospppsp....... web/
INTRODUCTION
Offline Reinforcement Learning (RL) has been widely studied within the academic community (Wu al., 2019 Fujimoto et al., 2019 Kostrikov et al.] 2021 Kumar et al., 2020), using pre-collected datasets to learn policies without exploration (Levine et al. 2020). At the offline training stage, safety constraints can incorporated into policy learning, which focuses on learning to safe rather than learning safely. In this way, offline trained policy is promising to avoid constraint vi- olations during online deployment. However, offline datasets cannot fully simulate complicated real-world environments and will cause out-of-distributie (OOD) issues in the case of unseen data (Kumar et al. 2020). Such generalization tasks are significant challenges for offline learning and safe learning.
This paper aims to explore the design of a practical safe RL algorithm that can both leverage offline data and quickly adapt to novel environments while maintaining safety properties. A straightforward way is to train a safe RL policy on offline data. However, the offline trained policy is prone to dis- tribution shift issues, making hard apply to unseen environments due severe safety constraint violations. Hence, a safe fifline-to-oling training mechanism is necessary to improve generaliza- tion. We consider utilizing model-based methods to complete generalization as quickly as possible. Model-based RI. has proven to be a powerful tool to enhance sample efficiency by constructing world model of the environment (Hafner al. 2019a: Cang et al., 2021] Hafner et al.) [201bb] [020)] 2023). The world model reduces random explorations by predicting the outcomes of different ac- tions, leading to fast convergence speed during online fine-tuning after minimal interactions (Feng al. 2023). Moreover, it significantly improves action safety in high-dimensional tasks such as vision- based operations due to its prediction ability (As et al. 2022 Huang et al.) 2023). Thus, we leverage its advantages to address the safe generalization problem in real-world deployment.
$ *Corresoonding authors

---

Published as a conference paper at ICLR 2025
Offline pretrain
fine line-tuning
policy expansion
Generalization Tasks
offline regulation
(offrima Montine)
Dataset
((Gls)
((.aa)
(sls)
safe transfer
D
5,7,C
((w)
anlect
updste
Σ Excc
f2.6 dg. 82
Fing-tuning
Pe
Real World
World Model
Xp
Figure 1. Fine-tuning offline safe policy through world models. We propose a framework for offline pretraining and online fine-tuning the world model. We first pretrain the agent by the offline dataset and rollouts generated from world models. The grey section depicts the architecture of the world model: it first encodes an image observation into its latent state 8p, then, for each latent state, generates an action using the policy, as well predicts the reward, cost, and next state. In the offline-to-online phase, we employ policy expansion initialize a new policy for online fine-tuning. The pretrained Q-value is leveraged to construct a softmax probability distribution. Then, we select an action by this distribution for the agent to safely interact with the real world, generalizing it to novel tasks.
We consider a framework of pretraining an offline safe policy with a world model and then directly fine-tuning it during online interactions. However, such strategy usually causes degradation of pol- icy performance and a lack of safety after fine-tuning (Nair et al. (2020). The main reason lies in too many mixed constraints (both soft and hard), such as safety constraints and behavior regularization, making it difficult to ensure both optimality and safety. Meanwhile, the world model will introduce prediction errors due to the distribution shift during the fine- fine-tuning on unseen tasks, leading in- creasing safety risks. Therefore, we seek to handle mixed constraints and use the offline safe policy as guidance for online correction. Based on this key insight, we propose FOSP (Fine-tuning Offline Safe Policy through World Models), which enhances offline safe training and bridges model-based offline training with online fine-tuning without constraints violations, as shown in Figure T
Taking advantage of SafeDreamer (Huang et al. 2023), safe version of DreamerV3 (Halner al. 2023). we employ world models to optimize both offline and online RL, significantly improving data utilization efficiency. Specifically, during offline pretraining, optimization to update the Q-value conservatively in order to deal with the value overestimation. We also consider safe constraints with feasibility-guide method, while simultaneously using the reachability estimation function to address the mixed constraints involved in this problem. More- over, we adopt the safe policy expansion (Zhang et al. (202)) to bridge offline and online algorithms, avoiding performance drops during the initial stage and suboptimality with nearly zero violations. In this way, FOSP balances the trade-off between optimal performance and constraint violations, allowing for safe fine-tuning generalization tasks.
approach leverages in in-sample
To evaluate our method, we design experiments across numerous tasks and real-world robotic arms including offline training and online fine-tuning. For simulation tasks, experimental results show robust performance both offline and online fine-tuning phases and achieve nearly zero cost, outperforming prior RL algorithms. Furthermore, we deploy FOSP in real-world experiments, utilizing a Franka manipulator to perform trajectory planning tasks. It turns out that method's ability across tasks with different safety regions can be transferred unseen safety-critical scenarios through fhw-shot fine-tuning
(Ji 2023)
We summarize contributions follows: (1) To the best of our knowledge, FOSP is the first approach that tackles safe generalization tasks by offline-to-online RL. (2) It handles the trade-off between the performance and constraints satisfaction across offline and online training phases
2

---

Published as a conference paper at ICLR 2025
various vision-only tasks. (3) In real-world deployment, it does not need sim-to-real transfer and can be safely fine-tuned in unseen aafety-constrained scenarios. (4) It can solve tifline-to-online safe RL adaptation by only a few trials while maintaining safety.
2 RELATED WORKS
Offine-to-online RL Offline RL, which trains a policy by leveraging a large amount of exist-
ing data, is valuable to real-world scenarios (Wu et al.
2019 Fujimoto et al. 2019). Prior of-
fline RL works focus on addressing the distribution shift and value overestimation problem YYu
ct al. 2020 2021: Rafailov et al.] 2021: Kumar et al. 2020: Kidambi al.
2020 Fujimoto & Gul
2021 Recently, in-sample methods have demonstrated their robust performance by avoiding out-of-
distribution (OOD) state-action pairs (Kostrikov et al. 2021| Xu al. 2023 Garg al., 2023 Peng
jet al. 2019, Nair et al. 2020: He et al. 2024) Yang et al., 2024). Unlike training solely on offine
data, offline-to-onling RL involves further fine-tuning the offline policy online. And many offline
RL algorithms have extended to online fine-tuning (Kostrikov et al. [222]] Nair et al. 2020). But these methods have shown that the more effective the offline RL method is, the worse online fine- tuning performance is (Xiao et al. 2023). Some previous works have explored how to effectively fine-tune from offline pretrained policy, including balanced sampling
replay buffer (Lee et al.) 20226), policy calibrating (Nakamoto et al., 2024 Rudner et al. 2021), and parameter transferring (Xie et al. 2021 Rajeswaran et al. 2017). However, existing solutions are restricted by policy tran- sitions or the need to estimate the density of the online policy distribution (Zhao et al. 2022). PEX (Zhang et al. 2023) is an approach that bridges offline and online algorithms and is adaptive for fine-tuning different online methods with offline policy guidance. We extend it to safe algorithms that not only improve the performance but also ensure safety, for it is shown as an effective approach in offline-tooonline training.
Safe RL Safe RL focuses on optimizing objectives with different constraints (Altman 2021
Many methods are based on policy search algorithms that ensure adherence to hear-constraints iter- atively (Achiam et al. (217)) and Lagrangian methods that convert it to an unconstrained problem to solve (Chow et al.] 2018: Ding et al.] 2020; Tessler et al. 2018). Model-based RL methods utilize world models for action selection so that they ensure safety by background planning. Some recent works are trying to address challenges remaining in vision-input tasks (Sikchi al.) 2022] Hansen iet al. 2022, Hafner et al.) 2019b). LAMDBA (As et al.) (022) is method extending DreamerV (Hatner et al.) 2019a) to safe RL by using augmented Lagrangian. Safe-SLAC (Hogewind et al.,
2022) incorporating the Lagrangian method into SLAC (Lee et al. 2020) improves the computa- tion complexity of LAMBDA. SafeDreamer (Huang et al., a 2023) the latest work in safe model- based reinforcement learning. It enhances safety in online or background planning with DreamerV3 (Hafner et al., 2023), achieving state-of-the-art (SOTA) performance. However, prior works only aim to address issues in the online setting, which cannot avoid costs incurred during exploration in the real world. By contrast, some safe offline RL can learn policies from offline datasets by using OOD action detection (Cao et al. 2024; Xu et al.) 2022 Li et al. 2022). DICE-based theory (Lee) et al. 2022a), and sequence modeling (Lin et al. 2023 Liu et al. 2023). Nonetheless, they only utilize soft constraints which sometimes
not effectively decrease constraint violations. Some works consider the hard constraints by using control theory methods such as control barrier func- tion (CBF) (Choi et al., 2020) and HJ reachability (Zheng et al., 2023 Yu et al.) 2023 Ganai al.) 2024 Fisac et al.) 2201)) ). Our work draws from these works and unifies all the constraints with model-basee based background planning, continuing the CMDP paradigm that satisfies safety constraints in expectation.
3 PRELIMINARIES
World Model Our work extends DreamerV3 (Hafner al., 2023). a powerful and robust baseline basel RL using world model and actor-critic. It has the ability to model high-dimensional observations of the environment and increase the data efficiency of RL. The world model learns com- pressed image input representations and predicts future states and rewards sequences for decision- making. It is implemented as a Recurrent State-Space Model (RSSM) (Hafner et al.] 2019b). First, the model maps the high- dimensiion observations 2; stochastic states z;. And deterministic latent states h, are predicted by x, and actions a, By concatenating both of them, we get the model state 8t
(hr, 2.). Finally, we facilitate 8, to predict the next observations and rewards. Due
3

---

Published as a conference paper at ICLR 2025
to the safety considerations in the environment, add a cost decoder the original model. All components in the world model are as follows:
latent encoder: Zt ya(2z hr we))
observation decoder:
d; ~ po (xt
2t, ht)
deterministic state:
h fol 21-11 1, a
reward decoder:
rt p6 P T2
zt. ht)
stochastic state: 2. Pe( Zt he)
cost decoder:
C₁ pa (c
žz, hs)
The model is trained end-to-end by the ELBO or variational free energy of hidden Markov model
loss function with KL balancing:
mmodel (0) DE Σ
pa (x) - In po 8τ)
In (e 8z)
DKL.sg (z, h,, x. llpl + 3D
(h.))))))
(1)
The stop- cop-gradient operator sg(-) makes the representations more predictable in the imagination train- ing.
Safe Model-based RL Safe RL is frequently formulated Constrained Markov Decision Pro- cess (CMDP) M = (S,A, IP, C, p, 7) (Altman 2021) with discrete time steps . S T).
and A denote the state and action space. P( 8,8 a represents the transition probability from to under action R is reward space that maps S x A to I. And C is cost function set contain- ing cost functions c : S x A [0, Cmez]. px() : - [o, 1] is the initial state distribution and y E (0, 1) is the discount factor. Given a policy distribution Tæ. the cost state value function V and cost aation-state value function Qe are defined by V=(s) = EE 12%-o 7 ecc( 8r, at)], Q a) =
Ex~xg E/= c(c ac)] like the common MDP. We define the unite-horizon reward function and cost function as follows:
T re
J (xp)) ETa PPP100000
80
T c
( ETy po.
so
& d'
vie (1....,,)}
(3)
where i denotes different safety constraints we want to avoid and d' are cost thresholds. Therefore, our safe model-based RL problem can be formulated as follows:
max J (TT) s.t. (F(p)) d', Vie (1.....C).
Reachability Estimation Function RESPO (Ganai et al. 2024) first introduced the reachability estimation function to capture the probability of constraint violations. Suppose that the set S, is constraint violation set defined in the state space S, containing all states that violate the constraints. The reachability estimation function (REF) u S [0, 1] can be defined as:
= Eay=a max [(5g|=00 x) E S.)
(5)
s,ET
The value max,,er 1[(s)) Su, T) S.) defined on specific trajectory. equals 1 if existing violations and 0 otherwise.
METHODS
In this section, we introduce FOSP for fine-tuning safe actions with offline pretrained world models. According to Figure [ we first train an offline safe policy and subsequently fine-tune it online. In Section [4.1] we address the Q-value overestimation offline learning by utilizing in-sample optimization to train the critic network a,) In Section 4.2 to balance the trade-off between performance and constraint violations during offline training, we employ the reachability estimation function train the safe actor (s)) In Section 4.3] the policy is online fine-tuned by safe policy expansion, which helps mitigate the performance drop and enables further improvements. The whole algorithm can be found in Appendix
4

---

Published as a conference paper at ICLR 2025
4.1 IN-SAMPLE OPTIMIZATION FOR OFFLINE TRAINING
Value overestimation is an extensive challenge in offline RL due to the distribution shift between the state-action pairs from the dataset and those generated by the learned policy. This issue becomes even more pronounced in model-based offine RL algorithms, where both the policy and dynamics model pe are learned from the offline data. It not only amplifies value overestimation but also causes additional complications in estimating the latent dynamics and predicting future returns, further impacting the overall performance. To mitigate these challenges, we propose the in-sample optimization as solution.
First, we consider a sequence of data B (IIrT, d1:2 Pi:;; (1:r) sampled from the offline dataset. Then, the model generates latent states 8 90(81:r/hi:rr (IT)) which are used as initial states 80 to generate rollouts:
alir (=(aas(tr) s4,7'' Po(s)a" $1.T), Mer ~ po(r)())T), e/.r po(c((":.) (6)
where h represents the horizon of the generated sequence. We use Qo(s, a) as the critic network. Motivated by Implicit Q-learning (IQL) (Kostrikov et al., 202I). we only use in-sample actions as calculated components of A-return. Due to out-of-distribttion actions from (s). a high Q-value from Qo doesn't always mean the action will lead the agent desirable state. Thus, a separate value network (s) (a))) [Q (8, a)] only conditioned on states is proposed to mitigate the out-of-distribution issue. It approaches the action distribution expectation of Q-value and can be optimized by expectile regression:
(vp)) = ((p))=E,70)) Ey-D,Tv. | 1(Qu(6/,a)) - V2(6)) 00I(Q V=(
(7)
where x (0, is a constant. We use rollouts in equation 6 estimate returns by TD(A) and update the critic by equation 2
RM = ), R = r + 7(11 A)V) -ARR+))
(8)
(b(0) 1 1 HT
E Ey D.y
Σ Σ (Qo(s) a!:) R2)2
where H is the horizon of the model and h & H. is noted that the A-return in equation 2 is esti- mated solely through the generated rewards. However, during the early stages of offline training, the world model may not be able to generate accurate rewards. The real rewards from the offline dataset can better guide the critic's 's learning process. Therefore, we add equation [0 as regularization term to the critic loss to improve learning efficiency:
T Eieo(s) ax) - (rr + (o((s
(0)
ET
F
(10)
T
m Po
where a,, P2 are from the offline dataset and the final critic loss becomes
Lo(d) - E Cz(0),
(11)
4.2 REACHABILITY ESTIMATION FUNCTION AS SAFETY GUARANTEE
To ensure the safety of the policy, prior works consider adopting the relaxation of the Lagrangian multiplier method (Stooke et al. (2020). The optimization problem on Aquation{aaan be transformed into unconstrained problem with the Lagrangian multiplier A;:
max min (T >s (J{ (TT) d'), Vi (1.....).. A.30
(12)
However, the soft constraints will still lead to certain chance of violation, which is exacerbated in the offline setting due to the difficulty in estimating the cost values. Hence, we replace soft constraints with hard constraints by the feasibility guidance approach YYu al. 2022). The feasible states are included in the feasible set (s) == (s) = 0). Then we can divide the optimization problem into two parts: the feasible part and the infeasible part.
For the feasible part, we want the agent to maintain safety within the feasible set and maximize the reward return. For the infeasible part, we want the policy to violate the constraints little as
5

---

Published as a conference paper at ICLR 2025
possible and return to the feasible set. As the safe offline RL problem has another objective to optimize: DKL (Tollll mb) (Ty is the behavior policy), we can reformulate the problem as follows:
Feasible part: max E (s) 1{ s.t. V5(s) -0, DKL(TuIlTb) E
(13)
Infeasible part: max Es - (5)) 1 $/ s.t. DKL(Fel(Tb) & E
(14)
Note that the policy constraint DRL (T m.) e soft constraint, which complicates the feasible part problem due to coupled hard and soft constraints. Thus, we use the reachability estimation function u² (s) = - [max, { so. TT S. from RESPO (Ganai et al. 2024). where S.
and Sj are complementary. This function helps handle the mixed constraints by prioiitzing the hard constraints and also assists the agent reentering the feasible region even when ventures into the infeasible region. And the problem becomes:
max EN (8) (1 (1-"))) V ()
S.L. (+(()) = 0, (TI( [| & 6.
(15)
To simplify the constraints, Meanwhile, we leverage advantage functions A"(s, a) = Q (s. a) (s)s) and ( a) =
introduce Augmented Lagrangian with a proximal relaxation method.
0g(1 a) - (s) to assess the influence of actions on the value function according to Proposi- tion 1 The final problem can be formulated as follows:
max E. (AA(s, (VV( X 1² )) ()- (1- u a) s.t. DKL(TelITd < (16)
A&VC+ (VS)2
if i + f 0,
+ (v Xp. )
(25)2
(17)
otherwise
ar
where X and b are Lagrange multipliers. We can finally obtain closed- form solution of n.. which we can use to extract the optimal policy (Zheng et al. 2023). (For more details see Appendix A.2
1 z w Tr(als).
(als)
(18)
where w w= (1-1 m exp(311
uh exp(-32A a)).
(19)
and the policy loss is as follows:
Cx(1)= = - Σ (w log (ail8)) + 1 (a)]((()))) ( (11
u (20)
where 7), B1, B2 are hyperparameters, E is the entropy. For critic learning, we use the same way to train cost critic 23 (s, a) and cost value VO (s) as Section 4.1
4.3 SAFE POLICY EXPANSION FOR WORLD MODELS FINE-TUNING
We aim to improve performance and generalize the learned policy similar tasks while maintaining the safety learned from the offline policy through online fine-tuning. However, applying online safe RL algorithms for fine-tuning often leads to initial performance drops and constraint violations, probably caused by the distribution shift between the offline and online data (Lee et al.) 2022b). Conversely, directly fine-tuning the original offline RL algorithm typically exhibits worse online performance due to their conservative design (Kostrikov et al. 2021 Kumar et al. 2020). Therefore, we try to bridge offline-online RL. by safe policy expansion.
Suppose that T is the offline learned policy. Rather than directly fine-tuning it in the online stage, we optimize a new policy Twr by gradually learning from frozen prior policy Ty. Concretely, given a current state an action set from which actions are selected is composed of actions generated by
two policies: A = {a (s), agd ~ (s)) The actions are then probabilistically selected based on their potential utilization value (Zhang et al. 2023):
(x)(
[kw)[k] (0( exp(((ss a vk (1,
(21)

---

Published as a conference paper at ICLR 2025
Table 1. Oftline-to-online results. Reward return and cost return of methods in oflline-to-online fine-tuning (1M steps for offline and 0.5M steps for online fine-tuning). We report the mean value of independent runs with different seeds.
FOSP(Ours)
Recover-RL RL
DreamerV3
SafeDreamer
Tasks
Reward
Cost L
Reward
Cost
Reward
Cost
Reward [
Cost
PointGooll
18.7 -2.5
1,4 1.2
5.4-122.
17.4 -5588
82.3--8.11
12.7--19.2
1.5- -.02
PointButionn
14.7-18.1
9,5-2211
5.--6.3
10.3-2.8
15.1-20.88
167.7-157.99
8.9-10.77
10.2-+.55
PoimiPushl
4.0-1332
18.1
0.4-3.88
-0.8-0.5
2.36
30.9-+26.3
1.3-
-0.12
PointGoal2
8.1-13.55
7.5-0.23
0.5-+2.7
53.2-0.3
9.5-188.9
367.1-290.2
3.6-12.7
8.6-0.3
CarGoal2
10.1-14.5
1.6-0.007
4.6-+8.9
0.6-0.113
16.7-24.1
23L0--287.9
6.9--.88
0.7-00.99
Average
16.6
0.54
6.7
0.91
20.8
171.1
12.5
1.01
Table 2. Compare with online algorithms. We compare method with some online-only algo- rithms. SafemDeemer(+15 +planning) was trained online for 0.5M steps and CPO and PPO-Lagrangian trained for 10M training steps until they converged. We report the mean value of 5 independent runs with different seeds.
FOSP(Ours)
SafeDreeame((<plannnng)
CPO
PPO-Lag
Tasks
Reward
Cost
Reward
Cost
Reward
Cost
Reward
Cost
PointGoall
21.5
0.2
20.1
0.6
22.3
19.5
29
PointButton
18.1
2.1
12.4
17.1
5.3
PointPush1
13.2
0.1
0.7
3.7
2.9
PointGoal2
13.5
0.23
13.0
1.7
13.8
51
1.8
30
CarGiol2
14.5
0.07
11.2
0.34
15.5
52
7.8
28
Average
16.6
0.54
12.9
1.77
14.5
51.8
7.5
27.7
@(s, a) = Qo(s,a)
(8,
(22)
Q+
- 7 Tmars C+ LiT ? Cmar to nor- malize the Q-value, where Tmazy Cmaz represent the max reward and cost. Q(s, a) represents the balance between the normalized cumulative rewards and cumulative costs. Based on the probability function, the two policies can adaptively explore the environment safely according to their respec- tive probabilities, determined by the Q-values of their generated actions. We can ensure the stability of initial performance during the early fine-tuning stages, exploring with a better policy under the premise of minimal constraint violations. At the same time, we leverage online algorithms to en- hance performance further. According to Section 4.2 the weight of policy loss in the online stage derived from equation T6 without the behavior policy constraint can be written as:
where a temperature. We use upper bounds Q+= =
= (1 1114" - - 32,
(23)
This approach learns new policy through online updates while leveraging the afffine-trained policy as prior knowledge. Ultimately, this enables the policy to generalize to new tasks during online fine- tuning safely. In this way, the offline replay buffer can be directly converted into online replay buffer, allowing online exploration trajectories to be continuously added during online fine-tuning, thereby simplifying the complexity of balanced sampling (Lee et al. 2022b).
5 EXPERIMENTAL RESULTS
In this section, we evaluate our method across different agents and tasks within the Safety- Gymnasium simulation environment (Ji et al. 2023). as well as on real-world motion planning control tasks using a Franka robot. We aim to address the following issues: (1) How does FOSP perform in offline pretrain and online fine-tuning? (2) What role does each component of FOSP play in overall functionality?
(3) How does the offline dataset affect experimental results? (4) Can FOSP successfully handle unseen safety regions in the new scenarios?

---

Published as a conference paper at ICLR 2025
PorittyorrlButont
SafetyCarGaoa22
SafictPontGoon
SaletyPoniPush
Roward
200
0.30
340
0.02
Stop
Step
Stop
Step
Step
FOSP(Ours)
SateDreamer
CPC
SafaDreamer (plarning)
Recovery
DreamerV3
PPCLag
Figure 2. Online experimental results. Comparing FOSP to baselines across five image-based safety tasks at the online fine-tuning stage. The results for model-based algorithms are obtained after fine-tuning for 750,000 steps. The dashed lines represent the benchmark results for CPO and PPO-Lagrangian after 10 million training steps across all tasks. The SafeDreamer (planning) was trained online for 0.75 million steps. Reward: averaged episode reward return. Cost: averaged episode cost return. Cost Regret: averaged cost value throughout the training phase.
5.1 EXPERIMENTAL SETUP
Simulation Tasks We consider five tasks on
benchmark (Ji et al. 2023) envi- ronments. The agents need to navigate to predetermined goals without collision with hazards and vases. We measure the performance with three metrics: average undiscounted episode reward re- turn (Reward), average undiscounted episode cost return (Cost) and average cost value throughout the training phase (Cost Regret). To evaluate the advantage world models handling high- dimensional features, perspective of the agent, which is more representative of real-world environments. The standard of- fline datasets are sampled by three different behaviors: unsafe policy, safe policy and random policy. They are mixed in 1:1:1 ratio, with each part containing 200 trajectories. All the tasks are trained for five seeds. For more information see Appendix [
use 64x64 pixels RGB image as inputs obtained from the first-person
Baselines Prior works in offline safe RL using the model-free method typically perform poorly in vision-only tasks due to their slow convergence rate (Liu et al. 2023 Zheng et al., 2023). Therefore, we mainly compare FOSP with the following model-based baselines: Recovery RL (model-based version) Thananjeyan et al. 202I). a method first learns the constraint violation regions offline and then performs online policy training: SafeDreamer (Huang et al.] 2023), an online powerful algo- rithm in safe mbsel- reinforcement learning and Dreamer V3 (Hafner et al. 2023) a strong baseline in visual tasks but overlooks constraint violations. We also choose classical safe RL algo- rithms CPO (Achiam et al. (217)) and PPO-Lagrangian (Schulman et al.) 2017) to compare at the online stage, following the experimental protocol of Ray (2019).
Real Robot The real robot enviromment includes 7-DOF Franka Emika Panda robot as the con- trolled agent and static third-person Intel RealSense camera obtain images as inputs. The agent also receives additional inputs, specifically the pose information of the robot's end-effector. We design the task SafeReach that controls the robot reach a desired goal while avoiding collisions with predefined obstacles in its field of view. The rewards and costs sparse and detected manu- ally, determining whether the robot reaches the goal or violates the constraints. The dataset consists of demonstration data, violation data, and failure data. Furthermore, regions and different targets that are not contained in the offline dataset. These unseen tasks re- quire the agent to recognize objects that it has not encountered in the dataset and to generalize its
design some unseen safe
8

---

Published as a conference paper at ICLR 2025
Ablation Studies on Dataset Coverage
Ablation Studies Dataset Sizes
Table 3. Real-world unseen tasks. We record the success rate (SR, %) and the constraint viola- tion rate (CV, %) over 20 tests in three tasks while it will be labeled violation if it collides with obstacle. The robot has fine-tuned 40 gradient steps. See Appendix for more details.
Offine Reward
15.0
Offine Rpward
Offine
Offine Cast
15
Online
12.5
Online Reward
10.0
10
-
5.0
2.5
Smaller (300) Standard (600) Larger (900)
Figure 4. Dataset ablation studies. We evaluate ablations SafetyPointGoal2 with five seeds. Due to the large offline costs affecting the visual presentation, in the left figures. The x label in the left image represents safe data unsafe data. 1M steps for offline and 0.5M steps for online fine-tuning.
Before
During FT
After
Tasks
CV
SR
SR
reduced them by 10 times
40
40
30
35
45
60
20
60
40
40
30
20
decision-making abilities to these new scenarios. The tasks further illustrate the practicality of our approach.
5.2 MAIN RESULTS
Studies
We show our main offline pretrain and online fine-tuning results in Table to answer the question (I). During offline pretrain- ing, FOSP outperformed the similar safe algorithms Recover RL and SafeDreamer, excelling in decreasing costs and better balane- ing task performance with constraint avoidance. Furthermore, it has comparable slightly lower performance than DreamerV3 with nearly zero violations. (Appendix [H) The learning curves of on- line fine-tuning are shown in Figure It proves that our algorithm can also be fine-tuned to achieve better performance. Meanwhile, FOSP further reduces the costs during the online fine-tuning. We also compare our method with the online planning version of Safe- Dreamer (OSRP) (Huang et al., 2023) which can only be trained online. We notice that the SafeDreamer training exclusively online outperforms the offlie-to-online approach but has more constraint violations. This may be because the safety constraints learned dur- ing offline pretraining provide guidance for online fine-tuning, mak- ing the model more sensitive safety constraints. Taking this advantage, FOSP is able to bridge offline and online safety while maintaining its performance.
Clssk
Step
FOSP(Ours)
wlo policy expansion
win in-sampla optimization
wio reachabilty estimation function
win lagrangian penally
Figure Module abla- tion studies. We evaluate ablations SafetyPointGoal2 with means of five seeds. The vertical line divides the offline and online phases.
5.3 ABLATION STUDIES
Module Ablation In this section, we compare the effects of dif- ferent components of the model on its performance in Figure 3 to answer the question (2). Specifically, we provide the following ab- lations: (i) learning value function without in-sample optimization
(equation 7 in Section 4.1 (ii) directly fine-tuning the model without safe policy expansion (equa-
tion 21 in Section 4.3 (iII) update safe actor without reachability estimation function in Section 4.2
(iv) en the penalty without the Augmented Lagrangian and use the feasible value function (Fisac al. 2019). The results show that each component of method individually (i) improves offline performance, (ii) stabilizes the transition phase from offline to online, (iii) enhances overall perfor- mance while maintaining safety constraints, and (iv) ensures compliance with safety constraints.
Dataset Ablation We also seek to investigate how the coverage and size of offline datasets affect the algorithm's 's performance to answer the question (3). For the dataset coverage, it is well estab- lished that higher rewards in the dataset lead to better performance (Yu al. 2020). Therefore,

---

Published as a conference paper at ICLR 2025
focus examining the impact of cost by adjusting the proportion of safe and unsafe data. The results in Figure demonstrate that safe data tends to guide the final policy towards being overly aggressive, achieving high rewards but frequently violating constraints. On the other hand, unsafe data causes the policy to become overly conservative, hindering exploration to avoid constraint vi- olations. This behavior arises due errors in the dynamics model in model-based methods. When the dataset is imbalanced, the dynamics model trained through supervised learning struggles to ac- curately predict the costs associated with each state-action pair in the real environment. As a result, the actor-critic, which depends heavily on data generated by the dynamics model, fails to predict suboptimal policy. To examine the impact of dataset size, we scaled the standard dataset up and down proportionally. The final results indicate that while increasing the dataset size leads to the learning of better and safer policies. Furthermore, the gains during the online fine-tuning stage are relatively modest.
future state costs accurately, resulting
5.4 SAFE GENERALIZATION TASKS
Safe Ganeralization Tasks
In this section, we show how our method solves different tasks with unseen safety regions in the offline dataset and answer the question (4). We first evaluate it in the simulation, offline pretraining the agent using data collected in the SaferyPointGoll/, then place in the PaferyPointoall,, which has more types and greater number of unsafe regions, for online fine-tuning. Although these tasks are the same type, the limitations of the previous data still pose sig- nificant challenges to the effectiveness and safety of fine-tuning The experimental results in Figure [ demonstrate the advantages of FOSP in these safety generalization tasks, showing better perfor- mance and safer fine-tuning compared to SafeDreamer.
Reward
Cost
In the real-world environment, robots will encounter many safety constraints while performing various tasks. Our key challenge how to safely train the robots to adapt to unseen constraints by safe learning-based methods. Following this insight, we introduce obstacles and targets of different shapes and sizes in Reach tasks, where the robot will only encounter a limited number of these dur- ing offline pretraining. We first train the agent for 0.5 million steps using a precollected dataset, then deploy the trained agent to the real-world environment execute tasks and fine-tune it. During online fine-tuning, each time the robot finishes a trajectory, we add it the replay buffer and update model parameters. Our goal is make the robot use few trials to learn planning policies in envi- ronments with new obstacles safely as possible. Thus, we design three real-world transfer tasks and show their results in Table 3 For more details see Appendix
100300
202003
300000
RO00O 300000
Slep
FOSP/Ours)
SafeDreamer
Figure 5. Simulation gener- alization task. We compared the performance of models pretrained 1M steps on Safe- tyPointGoall using FOSP and SafeDreamer, evaluating their fine-tuning and generalization results after 0.5M steps on SafetyPointGoall.
In brief, FOSP can successfully finish these tasks safely, demon- strating its strong offline learning and adjustment capabilities. contrast, only learning from the offline dataset makes it difficult to generalize to new scenarios. Its strong adaptability in fine-tuning showcases the robustness and great potential of FOSP.
6 CONCLUSION
We introduce FOSP, novel method that integrates model- reinforcement learning with offline- to-online fine-tuning to enhance safety in robotic tasks. FOSP addresses the challenges of balancing performance and safety during both offline and online phases using the reachability estimation func- tion to unify different constraints and offline safe policies to guide online exploration. Our exper- imental results demonstrate that FOSP achieves robust performance in various safe planning tasks with vision imputs. Furthermore, we evaluate FOSP on unseen safety-critica tasks in both simulation and dynamic real-world environments, highlighting its practical value in safe few-shot fine-tuning. Overall, FOSP is the first approach to extend MBRL into a safe offline-to-online framework to solve the safe generalization problem, showing promising results in the Safety- Gymnasium benchmark.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No.62103225), the Natural Science Foundation of Guangdong Province (No.2024A1515010005 and the Natural Science Foundation of Shenzhen CACCY22020001111000000000 No. 1C111002220111111000000000
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models. arXiv preprint hal::2201.098022 2022.
Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer offline rl. arXiv preprint arXiv:2106.09119, 2021.
Chenyang Cao, Zichen Yan, Renhao Lu, Junbo Tan, and Xueqian Wang. Offline goal- conditioned reinforcement learning for safety-critical tasks with recovery policy, arXiv preprint arXiv:2403.01734, 2024.
Jason Choi, Fernando Castaneda, Claire Tomlin, and Koushil Sreenath. Reinforcement learning for safety-critical control under model uncertainty, using control Yyapunov functions and control barrier functions. arXiv preprint arXiv:2004.07584 2020.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained rein- forcement learning with percentile risk criteria. Journal of Machine Leurning Research, 18(167): 1-51, 2018
Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems, 33:8378-888 88988, 2020.
Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Fine- tuning offline world models in the real world. arXiv preprint arXiv: 10. 16029, 2023.
Jaime Fisac, Neil Lugovoy, Vicenç Rubies- Royo, Shromona Ghosh, and Claire Tomlin. Bridg- ing namilton-jacohb safety analysis and reinforcement learning. In 20/9 International Conference on Robotics and Automation (ICRA), pp. 8550-8556. IEEE, 2019.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-201 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02322 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1rXiv:1912.01603 2019a.

---

Published as a conference paper at ICLR 2025
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555-2565, PMLR, 2019b.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02198 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.044 04104, 2023.
Nicklas Hansen, Xiaolong Wang. and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arX1v:2203.04955, 2022.
Longxiang He, Li Shen, Junbo Tan, and Xueqian Wang. Aligniql: Policy alignment in implicit q-learning through constrained optimization, 2024.
Yannick Hogewind, Thiago D Simao, Tal Kachman, and Nils Jansen. Safe reinforcement learning from pixels using stochastic latent representation. In The Eleventh International Conference on Learning Representations, 2022.
Weidong Huang, Jiaming Ji, Borong Zhang, Chunhe Xia, and Yaodong Yang. Safe dreamerv3: Safe reinforcement learning with world models. arXiv preprint arXiv:230..07176. 2023.
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuchai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36, 2023.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-22 11-222,,,, 2020.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances Neural Information Pro- cessing Systems, 33:741-752, 2020. 2020.
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXhv:2204.0895) 2022a.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022b.
Sergey Levine. Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint 0rXiv:2005.01643, 2020.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. arXiv preprint arXiv:2205.11027, 2022.
Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Safe offline reinforcement learning with real-time budget constraints. In International Conference Machine Learning, pp. 21 2122-12155, PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Con- ference Machine Learning, pp. 21611-21630. PMLR, 2023.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.0993 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36, 2024.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Divantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 1rX00:1910000177,, 2019.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offine reinforcement learning from images with latent space models. In Learning for dynamics and control, pp. 1154-1168. PMLR, 2021.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 7709. 10087, 2017.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 1rXiv:1910.01708: 7(1):2, 2019.
Tim GJ Rudner, Cong Lu, Michael A Osborne, Yarin Gal, and Yee Teh. On pathologies in kl- regularized reinforcement learning from expert demonstrations, Advances in Neural Information Processing Systems, 34:28376-28389, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1eXiiv1707.063. 2017.
Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off off-policy with online planning. In Conference on Robot Learning, pp. 1622-1633. PMLR, 2022.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Chen Tessler, D Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: 1074, 2018.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6 (3):4915-4922, 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1xX.:1911.1136/1 2019.
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. arXiv preprint arXiv:2302. /4372, 2023.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. Advances neural information processing systems, 34:27395-27407 2021.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8753-8760, 2022.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint 2303 /5800, 2023.
13

---

Published as a conference paper at ICLR 2025
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti- anti-exploration with distributional random network distillation. In Forty-fite first International Conference on Machine Learning. 2024.
Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference Machine Learning, pp. 25636-25655. PMLR, 2022.
Dongjie Yu, Wenjun Zou, Yujie Yang, Haitong Ma, Shengbo Eben Li, Yuming Yin, Jianyu Chen, and Jingliang Duan. Safe model-based reinforcement learning with an uncertainty-awant reachability certificate. IEEE Transactions on Automation Science and Engineering. 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model- offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-144422 2020.
Tianhe Yu, Aviral Kumar, Rafacl Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforce- ment learning. arXiv preprint arXiv:2302.00935, 2023.
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behav- ior cloning regularization for stable offline-to-onling reinforcement learning. arXiv preprint arXiv:2210. 138444 2022.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Feasibility safe offline reinforcement leaming. The Twelfth International Confer- ence on Learning Representations, 2023.
14

---

Published as a conference paper at ICLR 2025
A THEORETICAL INTERPRETATIONS
A. DERIVATION OF THE FINAL OPTIMIZATION PROBLEM
This section analyzes the derivation from equation 13 equation equation I6 by using the Augmented Lagrangian and advantage function. First, we introduce Proposition 1, which refers to Zheng al. (023) (Theorem Peng et al. (2019) (Appendix A).
Proposition 1. The optimization objective of equation T3 is the necessary condition of
max Eg (s, a) 1 St}].
Proof. We start with the advantage function estimation in Kakade & Langford (0002) and Peng et al. (019). Suppose that (s) 7'p(s; s s|x)) is discounted state distribution with n, and p s s/x² ) is the probability of the state guided by for 1. steps. The discounted sum of advantage expectation under d ()) can be represented as follows:
Σ 7' A,, ax)
0o. (+))
= E re
Σ 7' (r(S), az) + Va(Bi))
= -E -s
Va((s)) + "rr(sa
2-0
-ggg (00) [Vi(su)] + d
Σ r'r(si, ai)
= -(() + [(T),
where y is the behavior policy used to sample data. So we get the objective representation J(T)
(x)d [a-r(ale)LA,(s1,.. az)] + (u). Then, we have the following derivation:
(e) a:)] = max (x) Ea-rtale)44 a:)l
max Bond Bantialo at)l + ((u)
= -max J(T)
= max B[]](
The last equation holds by the definition of (st).
Due to the symmetry max, [s]] and max, E[V](s)]. the same proof extends to max, [s)] as well. So we have proposition 2.
Proposition 2. The optimization objective of equation 14 is the necessary condition of max, E. 1(s S,1l.
The problem (equation equation converts to:
Feasible part: max E, A'( 1f8 8T1}. s.t. vo(()) =0, DRI.(TT & e, i
(24)
Infeasible part: "
maxE. (s) a) 1(sE S s.t. (RL(TT/|TE) Se.
(25)
However, is hard for us to get close form of Ty. Therefore, we begin by disregarding the con- straints of the behavior policy and focus on addressing the hard safety constraints. The Augmented Lagrangian is used to deal with the feasible part (As et al. 2022). We use the following relaxation:
max min 2,30
A((s, Sy) - Ao V5(s)) + 1 (Ap - 22)2
(26)
15

---

Published as a conference paper at ICLR 2025
where Ap is Lagrange multiplier and μ² is non-decreasing penalty term corresponding to gradient step k. By smoothing the left-hand side term in equation 24, the last term ensures that Xp stays near its previous estimate. Differentiating equation 24 with respect to XB leads to the following update rule for the Lagrange multiplier:
A
Ve
if 2 + 4 0,
x
(27)
0
otherwise
where the X, Xk+1 only update when Tς. satisfies the constraints. The feasible objective becomes the following form:
JT Xp. 1²*) = (' (8) Sy) 1 (Vg. Xp u²),
(28)
AoVe + 4 (V9)² if i + Ve 7 0,
+ (Vo Xp. (Vs u²)
(29)
otherwise
Then, utilize the reachability estimation function to connect the two parts of the problem like So the final problem becomes equation T6
15
A.2 EXTRACTION OF THE OPTIMAL POLICY
We introduce proposition from Nair ct al. (2020) to solve the soft constraints from offline training
to provide a solution of equation 16
Proposition 3. Suppose that the optimizing problem has the following form:
max Ea-
[ [A(s, s.t. (T
The optimal solution will satisfy:
π (a|s) oc exp(aA(ss ())mm)aa)))))))
Proof. The Lagrange function can be formulated as follows:
[ 11 Ba-= [Aa==((ss,))) a)] X( ((DKu(TTllTT) c),
Then we take the partial derivative with respect æ and set it to 0:
A(s, a) AlogTo(als + Alog Tu (als) =
So we have:
" (a|ss) (xp(aA(s.a))xx((15))
where Z is normalizing constant.
Therefore, we have a solution to equation 16
The solved Fe can be explicitly expressed by equa-
tion 20
B IMPLEMENTATION DETAILS
To implement the REF Ug. the theorem introduced by Ganai et al. (2024) (Theorem 1). The REF can be trained as follows:
u(s)) = max(1(s E Sy), ma((s+1)))
(30)
where You is discount parameter 0 < Yu that its learning rate is greater than that the critic and policy, but less than that of the Lagrangian multiplier updates.
convergence of u(s). Meanwhile. ensure
16

---

Published as a conference paper at ICLR 2025
C PSEUDO CODE
Algorithm FOSP: Fine-tuning Offline Safe Policy through World Models
Require:
Offline dataset D. policy Rg, critics Qai and cost critics Q2g value network Ve Vey reachability function network ug, world model pe. policy rollout length H, number offline training steps Nottine. number of online fine-tuning steps Nonline-
1: Initialize neural network parameters 0, 1), OK. Sk, (k = 1,
2: for 1, 2, 3, .... Nomine do
3: Sample batch of trajectories B D.
Compute model state 8, Po (1)(811))1)
5: Update using equation|
6:
Generate H- -tep latent policy rollouts using world model
7:
Compute target estimation R, by TD(A).
Update Veg VG using equation
9:
Update Qo 2gz using equation
10:
Update Te using equation 20
11:
Update ug using equation 30
12:
Initialize new policy Ter and Treeze offline policy कe transfer critics and values.
for = 2,2,3, 3,.. .Noatine do
14:
Switch a policy according to equation 21 and rollout the policy in the environment for an episode to collect new trajectory
15:
D D.
16:
for cach training step do
Sample a batch of trajectories from D.
18:
Update 0 using equation]] D
19:
Generate H-step latent policy rollouts using world model pe.
20:
Compute target estimation R, by TD(A).
21:
Update Vo V using equation 7
22:
Update Q Q0 using equation
23:
Update Ty using equation 23
24:
Update ug using equation 30
D EXPERIMENTAL DETAILS
D.1 REAL WORLD EXPERIMENTS
The experiments verify the effectiveness of the algorithm by controlling robotic arm to avoid ob- stacles and complete a Reach task to the target object, called SafeReach. As shown in Figure (7c),, the main hardware setup consists of three parts: a Franka Panda robotic arm, an experimental plat- form for placing obstacles and target objects, and an Intel RealSense D435 camera for perception. On the experimental platform, we used white lines to designate 25cm 25cm operational area as the task space for the robotic arm.
SafeReach For the task of reaching. the robot receives the image information acquired by the camera along with the current joint posture information as planning condi- tions determine the movement decision the step. As shown in Figure 7 the robotic arm needs to navigate within the designated movement space, bypassing the ob- stacles which are represented by the green rectangular blocks, touch the predefined target which represented by the red square block.
Original
Dataset Collection To train the model for the SafeR- each task, 200 motion trajectories were collected as dataset by teleoperating the robotic arm using a 3Dcon- nexion SpaceMouse. The dataset consists of 120 trajec- tories where the robotic arm did not touch any obstacles and successfully reached the target object; 30 trajectories
Figure 6. Unseen Tasks. We choose three unseen tasks to show the general- ization of our algorithm with different obstacles and goals.
17

---

Published as a conference paper at ICLR 2025
Simulation Tasks
Agent Type
Real Robot Retup
Figure 7. Simulation and real-world environment. Different tasks and agents in the simulation and real-world. (a) Simulation tasks: we consider four different tasks in the simulation: Push1, Goal1, Buttonl, Goal2. (from upper-left to lower-right) The preceding words indicate the task type and the following number represents the difficulty level. (b) Agent type: Car (upper) and Point (lower). (c) Real-robot setup: we use raw images inputs and enable the robotic arm to complete obstacle avoidance tasks safely.
where it did not touch any obstacles but also did not successfully reach the target; 40 trajectories where it touched obstacles but successfully reached the target; 10 trajectories where it touched ob- stacles and did not successfully reach the target. Each trajectory contains 80 data points at different time steps. Each data point includes state: 64 64 RGB image captured by the camera, 6- dimensional pose of the robotic arm; action: a 6-dimensional variation recording the changes in the robotic arm's 's pose; cost: 1-dimensional cost index indicating whether the robotic arm touched an obstacle; reward: 1-dimensional reward index indicating whether the object reached the target ob- ject. The labeling methods for the cost and reward index are shown in Figure 8 The cost assigned value of I when the robotic arm collides with an obstacle, and 0 otherwise. Similarly, the reward set to when the object reaches the target and 0 in all other cases. We defined 3 placement points for target objects and placement points for obstacles in the scenario. By restricting the placement positions of the target objects and obstacles, we ensure that the model can converge more efficiently on limited dataset.
Transfer Tasks To validate the generalization of the algorithm, we design series of transfer ex- periments. As illustrated in the upper side of Figure the original task involves 2 green rectangular obstacles and red cylindrical target object. We design three different transfer tasks, depicted on the lower side of Figure 6) The tasks include altering the shape of the obstacles, changing the shape of the target object, and increasing the number of obstacles. These experiments aim verify the algo- rithm's s generalization in avoiding obstacles of various shapes and quantities, well as in reaching target objects of different shapes.
D.2 SIMULATION ENVIRONMENT
Metrics In the simulation environment, we use three metrics to measure the performance:
Reward J, is the average episodic sum of rewards.
Tap
J, Cost Je is the average episodic sum of costs.
E T
Je E
CL.s
18

---

Published as a conference paper at ICLR 2025
initial Irame
intermediate frame
intermediate frame
intermediate frame
final frame
Figure 8. Dataset collection. Each row in the figure represents five frames the robotic arm's motion captured from the same single trajectory. From the moment the robotic arm starts touching an obstacle to the moment the obstacle's ss posture stops changing, the cost label the dataset is marked I, while in other cases it is marked as O, as shown in the red-framed part in the image. When the robotic arm touches the target object, the reward label in the dataset is marked as 1, as shown in the yellow-framed part in the image.
Cost Regret Pe means the average cost over the entirety of training.
Pe Σ Cr
T
where is the number of episodes, Top is the timestep of episodes and T is the total timestep.
SofetyGoal The agent's objective to reach a goal while avoiding obstacles in its environment. Each time the agent successfully reaches the designated goal, the environment randomly gener- ates a new one. The agent earns rewards for approaching or reaching the goal but incurs penalties when it encounters obstacles. These obstacles include immovable hazards and movable vases. The agent's 's observation state is represented by the images from its front and back. Two specific tasks are selected: the Point-Goal task and the Car-Goal task, which are performed by the Point and Car agents, respectively. The Point agent is robot that operates on 2D plane and is capable of rotating and moving both forward and backward. On the other hand, the Car agent, which is slightly more complex, features two independent parallel wheels and freely rolling rear wheel.
The agent's 's goal is to navigate around both stationary and moving obstacles in the environment press of several target buttons. Similar to the goal task, the agent is rewarded for
SafetyButton
19

---

Published as a conference paper at ICLR 2025
approaching pressing the target button. The Button task introduces dynamic obstacles that move quickly along set paths, making it more challenging than the Goal task due to the costs incurred from collisions with these moving obstacles.
SafetyPush The agent's goal is to push an object to reach a desired goal while avoiding hazards and vases in the environment. It will get rewards when the object successfully pushes the yellow object to the goal. The agent does not fully control the object's 's movement, making the task more difficult to handle. And there are also some line-of-sight obstruction problems in vision-only situa- tions.
Safety SafetyFading This environment is similar to the SafetyGoal, where the agent needs to reach goal while avoiding obstacles. However, over time, the goal and obstacles gradually become unde- tectable. The agent needs to gather much information about the environment as possible from the initial stage and form memory of the goal and obstacle locations to complete the task accord- ing its initial plan. This is undoubtedly more challenging than SafetyGoal and is even harder to accomplish for visual-only input agents.
D.3 SIMULATION DATASET
We collect the simulation standard dataset from three policies: random policy, safe policy and unsafe policy. Each policy collected 200 trajectories for each task, which were then combined into the final dataset. This approach ensures that dataset uniformly includes all types of data: low cost with low return, low cost with moderate return, high cost with low return, and high cost with high return. Thus, can make better trade-off in training different offline policies. The safe policy trained by SafeDreamer (Huang et al. 2023) and the unsafe policy is trained by DreamerV3 (Hafner et al.) 2023) for 1.5M steps on each task. The standard dataset distributions on five simulation tasks are illustrated in Figure 9 They show that unsafe policies tend to collect data with higher returns but significant cost uncertainty in these scenarios.
E DETAILS OF WORLD EXPERIMENTS
We illustrate the details of real-world experiments in Figure 10 The FOSP is first pretrained on an offline dataset for 0.5 million steps and deployed in the real world to exccute the unseen tasks. In each task, we recorded the robot's 's trails before and after fine-tuning over 40 trials. As we can see, the robot's ability to safely complete unseen tasks has significantly improved after fine-tuning. shows the strength of fine-tuning different tasks.
We also compare our method with SafeDreamer on these tasks after fine-tuning. Both methods pretrained for 500,000 steps and undergo 40 fine-tuning iterations. As depicted in Figure I1 SafeDreamer sometimes violates constraints even when the task is completed. In contrast, FOSP can learn with zero constraint violations and ultimately reach the goal.
F BASELINES
model-hased reinforcement learning method that outperforms specialized methods across over 150 diverse tasks. It has stable performance without adjusting hyperparameters in exploring farsighted policies from pixels and sparse rewards an open world. However, it overlooks the safety considerations in the environment, which brings high costs in safety-critical tasks.
D3eamerV3 DreamerV3 (Hffne et al. 2023)
uses RSSM as the world model, and the loss function similar to equation but lacks the cost head.
mmndel (0) = =E,wD
Σ 1=1
In po a 8t) In Iopo s2) DDz[ss((o(2 x:))lle (zel h+)
8KZlg0(Z k, ( (2 (zilh.))1
20

---

Published as a conference paper at ICLR 2025
PointGoell
Po ininGoa2222
40-
safe data undato cata
safa data
UnGate data
Cout Natums
Cost Retumn
2004
Reward Recumss
Reward Recurs
PointButton1
PointPush1
unsade cista
unsade ditaa
250
Cust Reauurt
Reward Beruuns
Reward Recums
CarGoal2
random safe
unsofo data
129
25
Figure 9. The dataset distribution on five simulation tasks. We plot the reward return and the cost return of every data trajectory. Gray dots represent random data, yellow dots represent unsafe data, and blue dots represent safe data.
And simple actor-critic framework is used in its decision-making part.
Actor: a₂ (ar())))
Critic: (o(Ro(s,) = ΣΣ v/r
Actor loss: L(V)= = Σ((Rs -V( mtte//mxx(115) log (ar(s)) + n/ll (a)()))))
Vo(or))/
Critic loss: L(Ø)
Σ log logps (R(ls))))
21

---

Published as a conference paper at ICLR 2025
Original task (offline):
Transfer taskl:
Before fine-tuning
After fine-tuning
Transfer task2: Before fine-tuning
After fine-tuning
Transfer task3: Before fine-tuning
After fine-tuning
Figure 10. Deployment different real-world unseen tasks. The trails from top to bottom show the original offline training task and three unseen fine-tuning tasks, with each fine-tuning task displaying both before and after fine-tuning trails. The green frames indicate successful task completion, while the orange frames represent failures.
In the offline pretraining phase, we removed the online planning component and used it only for online fine-tuning. achieve robust performance the offline phase, we train latent dynamics model ensemble and use the uncertainty estimation approach (Yu et al. 2020
( (1, a+)
re(s) a std( (log(r((=(h))))
where (8(8) is the reward predicted by the reward encoder.
SafeDreamer SafeDreamer (Huang et al. 2023) incorporates Lagrangian-based methods into world model planning processes and achieves nearly zero cost performance on various tasks. performs well in high-dimensional vision-only input safety-critical tasks, surpassing the prior works (As et al., 2022; Hogewind et al., 2022) and balaneing performance and safety. The framework uses the same world models based DreamerV3 V3 (Hafner al. (023). We use the BSRP-Lag version of the model, which utilizes the Lagrangian method in background safety-reward planning that avoids online planning. In the offline setting, train it by providing the offline dataset and removing the parts involving interaction with the environment. This version SafeDreamer also achieves the best performance of the three versions.
adds a cost head into world models like our method and uses the same loss equation I in model training. It constructs cost critic (R.(s)) like V R2 (s)) and utilize Augmented Lagrangian
22

---

Published as a conference paper at ICLR 2025
Original (offline):
Transfer task1:
Baseline
Our method
Transfer task2: Baseline
Our method
Transfer task3:
Baseline
method
Figure I1. Compare with baseline different real-world unseen tasks. In each task, com- pare our method with the fine-tuned SafeDreamer (baseline). The green frames indicate successful goal reaching, the red frames indicate constraint violations and the orange frames represent failures.
method to update the actor:
(())
Σ t=1
(st) + n/H
[Tg (a,
8.)] v (C* (f.))
A A
+ ah
A2 14
A, otherwise,
+
V (s)) 1p)1²) N
4
where 5 (*) b.
Recovery RL Recovery RL (Thananjeyan al., 202I
leverages offline data to learn the constraint
violation zones before learning and uses two policies to separate the goal of enhancing performance achieves nearly zero cost uncertain environments where safety
and satisfying the constraints. limits exploration. We use the model-based version of this baseline. First, train cost Q-value by the following MSE loss:
((Φ) (Qg(s;, ai) - (ce + (1 Br (
23

---

Published as a conference paper at ICLR 2025
Table Hyperparameters for FOSP
Module
Name
Symbol
Value
Number of latent
Ni
48
Classes per latent
C
48
Batch size
B
64
World Model
Batch length
T
16
Learning rate
lm
10
Coefficient of KL-divergence
B
0.1
Generation horizon
II
15
Penalty term
V
5 5-9
Augmented Lagrangian
Initial Penalty multiplier
110
1 1-6
Initial Lagrangian multiplier
X0
0.01
Discount horizon
0.997
Reward lambda
Ay 0.95
Cost lambda
Xe
0.95
Expectile
K
0.8
Actor Critic
AWR temperature
B f2
10
REF discount
Yu
0.99
PEX temperature
10
Actor entropy regularize
17
3 30-4
Learning rate
lac
3 10-5
REF Learning rate
4,
5 5010 5
Number of MLP layers
MLP
General
Number of MLP layer units
Nunits
512
Action repeat
Trepeat 4
Then we select actions from the safe set and the recovery set:
Tonak
if (t, ar) ((s, a) S A: (5 < Ec)
a,
Trecwwrry
if (8,, a.) E {(s, E S x A: (2 a) > cc):
where is threshold. We follow the model predictive control (MPC) as learned dynamic model fa and use a VAE-based model capture the high-dimensional information. And we utilize SAC Haarnoja al. 2018 learn Ttask-
PPO-Lagrangian PPO-Lagrangian uses the objective of clipped PPO (Schulman al. 2017 to optimize:
To (a(a)s)
To (aass)
[(I) ppo min(
A, Foh
(s, a), clip(
Celip: + <cip) A, (s, a));
To. [a
To. (a
We the PID Lagrangian Stooke 2020) method and obtain the loss of the PPO-Lagrangian:
((0) ppol
+ X (C(0) - XA Te a)).
CPO CPO (Achiam et al. 2017) uses a local policy search combined with trust region recovery to ensure that single-step policy updates follow direction that does not violate the constraints. It introduces this form to optimize the problem:
28 1
0+ Ok
11
b.
11
11-11
where H is the Hessian of KL-divergence.
G HYPERPARAMETERS
The experiments for FOSP were conducted in a Python 3.10 environment with JAX 0.4.26. Our setup included CUDA version 12.1, running Ubuntu 20.04. The hardware used comprised four
24

---

Published as a conference paper at ICLR 2025
SufesyPorntGalt
Sonttonnn
SafclyCarGoal2
PaletyPain
SafictyPinnPusht
100
242
WW
202
200
300
03
0.99
0.1
0.9
POSPPCTum))
Recovvy RL.
Sodee Dmnommrr
Dreemeev
Figure 12. Offline experimental results. Comparing FOSP to baselines across five image-based safety tasks. The results for all three algorithms are obtained after training for million steps. Reward: averaged episode reward return. Cost: averaged episode cost return. Cost Regret: averaged cost value throughout the training phase. As we can see, FOSP can maintain safety and achieve better performance during offline training.
SafetyPoinlGuult
Subuiinn
SufelyCarGodf2
SafelyPaniiaa12
SafeelyainnIushht
Reward
200000 10.00 020020
JUCISS WULSC BUEDO
.DADX IDOOO SEOOS
JDADS 1DO0 GOOO
Cost
20000 4000
200K3 DECK
DOCICE 000000
20000 40000 BOOK
20000 BXXX
0100:
1.00
Coed Dagget
038
1.00
0.00
020
0301
03P
0.000
1300
3h0p
Stop
Stop
Step
Step
FOSPIOurs)
Recovery
SateDreamer
Figure 13, Online experimental results without Dreamer These results are the same as Figure 2 We omit the curves of some baselines to clearly illustrate the results of other main methods.
GeForce RTX 4090 GPUs and Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz. And the experiments' hyperparameters setting is shown in Table
H ADDITIONAL EXPERIMENTAL RESULTS
H. OFFLINE TRAINING RESULTS
We present the training curves of FOSP during offline pretrain in simulation experiments in Figure The results show that with minimal fine-tuning, FOSP achieves better performance, outperform-
25

---

Published as a conference paper at ICLR 2025
Offline
Online fine-tuning
17.5
15
15.0
12.5
Reward
10.0
7.5
5.0
2.5
200000
400000
600000
800000
1e6
400
100
80
Cost
200
0.0
0.2
0.4
0.6
0.8
1.0
0
200000
400000
600000
800000
1e6
Cost Regret
0.25
0.20
0.010
0.15
0.10
0.005
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0
200000
400000
600000 800000
Step
1e6
Step
safe
safe:unsafe=1:1 safe:unsafe=3:1
safecunsafe=1:3
unsafe
Figure 14. Results on dataset ablation. We conducted experiments with five different dataset ratios, performing an ablation study the proportion of safe and unsafe data. Each model was trained offline for million steps and fine-tuned online for 0.8 million steps.
ing Dremerr. Although does not match the performance of DreamerV3, FOSP achieves nearly zero cost of the agent throughout the entire process. Note that all methods struggle to get higher re- wards in PafetyyointPush because the line-of-sight obstructions necessitate online exploration.
H.2 DATASET ABLATION RESULTS
The detailed training curves of dataset coverage ablation are shown in Figure 14 They all trained on the SanetyPointGoal22 for million steps during the offline stage and 0.8 million steps for online fine-tuning. The dataset size is limited to 600 trajectories. As we can see, due to the characteristics of model-based safe RL, high-cost, high-reward policy learned from purely safe dataset, while both cost and reward are low from purely unsafe dataset. This is because the uneven distribution of the dataset leads to biases in the training of the dynamics model. Since model-based reinforcement learning algorithms rely heavily on impact the final decision-making. the dataset only contains safe data, the dynamics model trained through supervised learning will mistakenly assume that the agent will always incur cost of 0 in any state, leading the agent to ignore dangerous areas (failing to learn the cost critic). Conversely, suppose the dataset only contains unsafe data. In that of unsafe states that hinder the agent from completing the task, ultimately causing the agent to lose the ability accomplish the task (overleam the cost critic).
rollouts for training, this bias can significantly
the model will generate large number
26

---

Published as a conference paper at ICLR 2025
Offline
Online fine-tuning
10.0
12
7.5
Reward
5.0
2.5
0.0
-2.5
0.0
0.2
0.4
0.6
0.8
1.0
200000
400000
600000
800000
36
30
250
25
Cost
200
20
15
100
10
0.0
0.2
0.4
0.6
0.8
1.0
200000
400000
600000
800000
1e8
0.008
0.15
Cost Cost Regret
0.006
0.10
0.004
0.05
0.002
0.00
0.000
0.0
02
0.4
0.6
0.8
1.0
200000
400000
600000
800000
Step
1e6
Step
saferrandom=1:1
unsafe:random=111
Figure 15. Dataset ablation with random data. We designed a series of comparative experiments involving mixtures safe/unsafe data and random data, Each model was trained offline for million steps and fine-tuned online for 0.8 million steps.
The dataset coverage further influences online performance. Due to offline training results, the policy pertrained on a safe dataset will have higher rewards with aggressive behaviors while the unsafe one will be more conservative on the initial stage of fine-tuning. It also affects the final performances. As result, we adopt compromise approach to train the policy (i.e. green curve), which achieves relatively higher rewards while maintaining zero cost.
We also illustrate the safe-random and unsafe- randoo-randomm mixed experiments to investigate the impact of random data in Figure 15 Compared to experiments on purely safe or purely unsafe datasets (Figure 14), adding random data can help alleviate errors in learning the world model to some extent. As shown in the training curves, experiments mixing safe data with random data resulted in lower costs during offline training than purely safe experiments. And mixing unsafe data with random data enhanced the model's exploration, getting relatively higher rewards, Consequently, need to add some random data to the dataset to get better performance.
The dataset size ablation studies shown in Figure 16 The results align with general expectations. The policies trained on larger datasets outperform those trained on smaller datasets. However, as the amount of data increases, the improvements become less significant.
27

---

Published as a conference paper at ICLR 2025
Offline Results
Online Results
10.0
14
Reward
8
200000
1e6
20
200
Cost
100
5
0.2
0.4
0.6
0.8
200000
400000
600000
166
0.006
Cost Cost Regret
0.08
0.004
0.04
0.002
0.00
0.000
0.0
0.2
0.4
0.6
0.8
1.0
200000
400000
600000
Step
Step
Standard
Smaller
Larger
Figure 16. Dataset ablation with different sizes. We illustrate the details of dataset size abla- tion studies. "Standard" represents standard size, which contains 600 trajectories. "Larger" and "Smaller" denote the larger one containing 900 trajectories and the smaller containing 300 tra- jectories. Each model was trained offline for million steps and fine-tuned online for 0.7 million steps.
H.3 SAFE GENERALIZATION RESULTS
As depicted in Figure 17 we devise more simulation experiments on safe generalization tasks. The upper one shows the performances during the offline phase. The following two figures show their generalization performance on the more challenging tasks, SafetyFadingEasy/ and SaferyFading- Hardl. During the fine-tuning process, the target will gradually disappear and the agent should quickly find a path to reach the goal. The results demonstrate that FOSP can get higher rewards and lower costs even though it deployed on tasks different from offline pretraining. Compared to SafeDreamer, it is significantly important that it can maintain near constraint violations during fine-tuning on new tasks.
We test the specific average reward and cost for the transfer task from SafetyPoin/Godl/ to Safe- pyPointGaa22 and present the results in Figure 18 may also face some costs caused by novel constraints. As shown in Figure [8] the initial cost in the new environment is slightly higher than the original but reduces quickly as fine-tuning pro- gresses. It indicates that the safe policy expansion mechanism helps the model adapt new safety
Although FOSP can outperform SafeDreamer,
28

---

Published as a conference paper at ICLR 2025
Offline Resuill
SadillyFadi FgFyyy
SedfelyFodlingHaard
20.0
17.
Reward
12
Reward
Reward
10.0
2.5
80
Cost
Coos
0.9
100300
Slop
Step
Step
FOSP()urs)
SafeDreamer
FOSP(Ours)
SafeDreamer
FOSP(Ours)
SafeDreamer
Figure 17. More experiments on safe generalization tasks. We evaluate our method on different safe generalization tasks. The upper one illustrates the offline pretrain performances on Safety- PointGoall. We then fine-tune the model on SafelyFadingEasy and SafetyFadingHard.. FOSP can consistently outperform the baseline while ensuring safe fine-tuning.
Transfer Task Reward
Transfer Task Cost
17.5
Cost Original
15.0
Reward
12.5
2
7.5
Reward
5.0
Original
20k
50k
100k
300k
500k
0
20k
50k
100k
300k
500k
Steps
Steps
Figure 18. Results average reward and cost for transfer task. The results evaluations on PafetyPointGoal to SafetyPointGool2 task. The x-axis represents the number of fine-tuning steps. The original means the model's performance the end of offline pretraining in the original environment.
constraints quickly. The fine-tuning process can further improve its safety performance, promoting better generalization.
H.4 COMPARISONS BETWEEN ONLINE TRAINING AND FINE-TUNING
To further validate the effectiveness of the fine-tuning process, we compare the results between FOSP directly trained online and FOSP with fine-tuning. We illustrate the results for different num- bers of steps in the 5 The results denote that the online version FOSP does not perform as well as the fine-tuning method, even after training for IM steps. Meanwhile, direct online training faces challenges in requiring large number of training steps to converge. In contrast, our framework allows the agent to leverage offline knowledge to enhance online performance effectively with rel- atively short fine-tuning steps. The results also suggest that the policy can continually improved during online fine-tuning.
29

---

Published as a conference paper at ICLR 2025
Table Online fine-tuning and direct online results. Reward return and return of FOSP in offline-to-oni to-online fine-tuning (IM steps for offline pretraining) and direct online training. We report the mean value of independent runs with different seeds.
Online fine-tuning
Direct online
Task
Metric
20k steps
50k steps
100k steps
300k steps
500K steps
500k steps
1000k steps
PointGooll
Reward
18.723
18.692
19.581
20.063
19.649
21.517
14.749
18.586
Cost
1.451
3.4
2.719
0.328
0.268
0.2
2.66
1.08
Reward
14,65
13,415
13.569
15.306
17.86
18.102
10.57
12.81
PointButton!
Cost
9.622
8.639
6.664
4,8661
4,256
2.188
6.88
2,76
PointPusht
Reward
7.253
7.916
10.373
13.281
1.397
Cost
18.117
1.232
1.497
1,089
0.507
0.157
2.49
2.05
Reward
13.556
PointGoul2
Cost
7,556
1.486
0.427
0.18
0.214
0.234
4.779
3,089
CarGoul2
Reward 10.146
9.063
9.816
10,275
12.378
14.512
8,749
10,144
Cost 16688
1.063
0.385
0.394
0.18
0.071
2.12
1,21
H.5
MORE VISUAL CHANGES ON TRANSFER TASK
To evaluate the performance of our method under significant visual variations, the model pre- praitrd on SafetyPointGoal fine-tuned in the SnteerPooinBBuddnnGaaall environment, with the results presented in Figure [9| The new environment has the same task as the original one but the visual inputs are different. The agent struggles to finish the task since it fails to recognize hazardous areas and goals in the new environment. Plus, the inaccurate visual demonstrations in the offline dataset significantly impact the performance during online fine-tuning.
Based on the simulation results, changes in the real world. Swapping the colors of obstacles and goals could be a straightforward setting (i.e., making obstacles red and goals green). However, some prior work has demonstrated that robots are sensitive to the color (Feng et al.] (023) and the robot is highly likely to mistakenly identify the obstacles the goal. Additionally, it violates our original intention of using colors to distinguish obstacles and goals. Hence, the robot will fail in this setting.
easily know that FOSP not deal with significant visual
80-
Reward
Cost
100000 200000 300000 40000G 300003
100000 200C00 302000 40CDOD SC0000
Step
Step
Figure 19. Fine-tuning on BunillyPninuullddnnCCaaalllL The model pretrained for IM steps on SafetyPointGiall is fine-tuned SafetyPointBuildingCCon While the task objectives remain the same between these two environments, the agent's visual inputs are entirely different. In Safety- PointBuildingGoal,, the agent must avoid red-marked areas and reach the parking area (P). These hazardous areas and goals are totally different from the original ones.
H.6 EXPERIMENTS ON RACE
We utilize more realistic environment, Race from al. (2023), where the agent receives 64x 3 image inputs, shown in Figure 20 An increase environmental complexity better highlights the reliability of the algorithm. We employ Level of the environment, which requires the agent to reach the goal position from distant starting point while ensuring it avoids straying into the grass and prevents collisions with roadside objects. The offline dataset collected by standard procedures mixture of safe, unsafe and random data.
As illustrated in Figure 21 FOSP has superior performance and lower constraint violations during the offline training phase. In the online fine-tuning stage, it further optimizes safety performance
30

---

Published as a conference paper at ICLR 2025
Figure 20. Race environment. The left subfigure shows the panoramic picture of the environment. The right subfigure is the first perspective of the agent.
Offline Reward
Offline Cost
120
Reward
Cost
20
0-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
1e6
Steps
Online Reward
Online Cost
30
25
Reward
20
Cost
15
10
100000
200000
300000
400000
500000
100000
200000
300000
400000
500000
Steps
Steps
FOSP
SafeDreamer
Figure 21. Race experiments results. We compare FOSP with SafeDreamer on Race2. Each model was trained offline for million steps and online for 0.5 million steps. FOSP has comparable rewards but lower costs than SafeDreamer.
while ensuring a stable improvement in task rewards. Conversely, SafeDreamer exhibits higher constraint violations during the offline phase, leading to unsafe behaviors the beginning of the online fine-tuning. Additionally, its safety design limitations make challenging to achieve nearly zero constraint violations.
31

---

Published as a conference paper at ICLR 2025
1 LIMITATION AND FUTURE WORK
I.1 LIMITATIONS
Safe constraints Despite achieving good results in many simulated environment experiments, the proposed framework still has some safety concerns. First, it inevitably requires a balanced sampled and sufficiently large dataset for offline policy pertraining, which can be undesirable for safety- critical applications. An uneven dataset, such as one lacking unsafe data, or insufficiently large dataset, can lead to deviations in model learning, thereby compromising its safety performance and generalization. Second, it may exceed the feasible region from time to time if the cost distribution has a long tail since CMDP only requires the policy to satisfy the expectation cost constraint. It might be considered that incorporating risk-constrained methods (Chow et al. 2018) solve problems with long tail cost distributions.
Real-world experiments In experiments with real robots, we find that the model performs poorly in tasks with occluded vision. Our test results remain unstable and are sensitive to learning rate adjustments. This indicates the need for significant effort in tuning the model's hyperparameters to achieve hetter performance. In addition, SafeReach task is quite preliminary. It is difficult to apply FOSP to more realistic scenarios like real-time control or dynamic obstacles predictive capabilities. The model also fails to handle novel visual observations beyond scene configurations during fine-tuning because struggles to interpret the meaning of different objects in new environments.
lacks certain
1.2 FUTURE WORKS
More complex real-world tasks To further decrease the violations, the safety component of our method can be improved during the offline-online stage. The SafeReach task is preliminary ex- periment in the real world and it is possible to expand our method to general robot tasks such as grasping, pulling, and pushing. For the tasks with novel visual observations, leveraging semantic information into the inputs is promising solution, which can help the agent attach meaningful interpretations to raw sensory inputs.
Improvement in real-world scenarios The real-world task performance of our method can be further improved. To address the issue of inaccurate position determination from a single perspective and line of sight occlusion, the multi-view RL shows its efficiency in catching better features from images. Increasing the dataset capacity and uniforming the dataset distribution can probably enhance the robustness of the performance and expand it to more complex scenarios.
Safe sim2real FOSP also shows potential in the field of safe sim-to-real transfer. Due to the limitations simulators in fully replicating real-world environments, robots may encounter safety challenges when deploying algorithms in the real world. Thus, it is important to fine-tune it in the real world with safety considerations. Mcanwhile, the use of world models will help speed up the fine-tuning process.
32


Reviews:
[{"review_id": "sZkTxPaMXV", "reviewer": "Reviewer_2vUt", "summary": "This paper presents FOSP, a novel safe reinforcement learning method. FOSP primarily addresses the issue of enhancing safety through offline training and online fine-tuning. Its main contributions include:\n\n1. The introduction of an offline-to-online reinforcement learning framework.\n2. Enhanced safety and performance balance during visual tasks through the integration of offline and online phases.\n3. In real-world deployments, FOSP allows safe fine-tuning in unseen safety constraint scenarios.\n4. The experiments validate the method's effectiveness in both simulated and real robotic tasks and demonstrate its safe generalization ability in new scenarios.", "questions": "1. Although this method performs better than others, it also introduces a new design for online to offline transitions. What specific problem is this design intended to address? The reviewer hopes the authors can clarify the benefits of this setting. For instance, the reviewer notes that in Table 1, the performance of SafeDreamer (offline) is actually comparable to that of FOSP.\n2. How is safety ensured in this method, particularly when dealing with unseen data and new environments? Specifically, how does the online fine-tuning process avoid safety hazards? For instance, how does the constraint violation during online fine-tuning compare to the converged offline policy?\n3. The reachability estimation function is a key component of this paper. How is the feasible part determined through $ S_f(\\boldsymbol{s}):=\\{\\boldsymbol{s}|V_\\varphi^c(\\boldsymbol{s})=0\\} $? In particular, how does this hold when $ V_{\\varphi}^c(\\boldsymbol{s})=0 $ may not strictly hold in the fitted value function? \n4. How do the authors view the potential of this method for safe sim2real applications? That is, training under simulated environment data while ensuring safety and improving performance during real-world deployment.\n5. The safety policy expansion mechanism stabilizes performance during the initial fine-tuning phase, but does it impose limitations on long-term fine-tuning? For example, what would the comparison of results be between the FOSP(online) version and FOSP in short-term versus long-term training?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The FOSP method introduces the concept of safe generalization within the offline-to-online reinforcement learning framework, combining world models with offline training and online fine-tuning to enhance safety and performance. This novel strategy for applying reinforcement learning in safety-critical scenarios is insightful.\n2. The design of experiments is comprehensive while covering tasks in both simulated environments and validations in real robotic settings.\n3. The experiments of FOSP in real robotic tasks demonstrate real-world value and safe generalization in unseen scenarios.\n4. The paper clearly articulates the optimization objectives at each stage, aiding in understanding.", "weaknesses": "1. While the experiments were validated in Safety-Gymnasium and real robotic environments, the number and complexity of tasks remain relatively limited.\n2. The motivation for introducing the new offline to online setting in the paper has not been well communicated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3HOz6zvmPc", "reviewer": "Reviewer_gtcR", "summary": "This paper proposed a model-based offline-to-online safe RL algorithm, FOSP. The proposed framework first uses an offline dataset to learn a world model and a pre-trained policy. The pre-trained policy is then fine-tuned through online safe RL fine-tuning. The effectiveness of FOSP is demonstrated through experiments in the Safety gym and a real-world robot experiment.", "questions": "1. Where does the behavior policy $\\pi_b$ in Sec. 4.2 come from? Does it refer to the offline RL policy from Sec. 4.1?\n2. Does the training step in Sec. 4.2 also occur offline? Why is it necessary to divide the offline learning stage into two steps?\n3. The derivation in Sec. 4.2 is very hard to follow. What is the advantage of using the reachability estimation function from RESPO? It would be helpful to introduce RESPO in the Preliminaries Section. Is $1(s\\in S_f)$ equivalent to $u^\\pi(s)$? How is Eqn (14) derived from Eqn (13)? Why would introducing advantage functions simplify the constraints? \n4. The limitation and future work section should be moved to the main text.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Extensive experiments and ablation studies were conducted, including a real-world robot experiment with high-dimensional visual observations. The proposed FOSP algorithm achieved good performance, especially in the real-world robot experiments, and outperformed baseline approaches.", "weaknesses": "While the proposed algorithm performed well in extensive experiments, I found it hard to appreciate its technical contributions due to its complex algorithm design (e.g., many moving parts, iterations between offline and online learning) and confusing presentation. It is unclear to me if all the design choices are necessary and which components are novel or come from the literature. It would be good if the authors could clearly state their technical contributions and novelty (e.g., does the novelty mainly lie in a new combination of existing components in the literature?). It would also help if the authors could provide more insights on why FOSP would outperform baseline approaches, especially SafeDreamer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEIrLYZmyd", "reviewer": "Reviewer_E4Wn", "summary": "The paper proposed an offline-online safe RL framework based on safedreamer, integrating the concepts from RecoverRL. When feasible, the method optimizes rewards, and when infeasible, it focuses on cost optimization. Additionally, the reachability estimation function from RESPO is utilized to introduce cost into the optimization process. To mitigate the issue of inaccurate critic estimates during offline training, in-sample actions from Implicit Q-learning are employed in the Q function learning.", "questions": "1. I'm curious about the video demonstration of the performances of SafeDreamer and FOSP in the SafetyFadingEasy and Hard environments. Based on the experimental results you presented, FOSP seems to perform better than SafeDreamer. This environment tests the agent's memory, yet it appears that FOSP does not specifically address this aspect. What accounts for the performance improvement?\n2. Why do you need to learn Q(s, a) when I recall that DreamerV3 only uses V(s)?\n3. How did you adapt Recovery RL to the image-based setting, and how did you modify SafeDreamer to the offline setting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed safe model-based RL framework effectively addresses offline-online generalization tasks.\n2. It demonstrates the capability to safely fine-tune in previously unseen safety-constrained scenarios during real-world deployment.", "weaknesses": "1. In Figure 4, the results for DreamerV3 could be omitted, as they overshadow the cost performance of all baseline methods.\n2. There are too few obstacles in the real-world environment, making it difficult to assess the agent's obstacle avoidance behavior. And the website does not provide any differences between the proposed algorithm and the baseline in the video demos.\n3. In Figure 4, why do the rewards for FOSP in PointGoal2 and PointGoal1 not continue to rise? Additionally, SafeDreamer does not show an increase in reward for PointButton1. Does this indicate that the fine-tuning phase was ineffective? According to the paper's description, the offline data comprises a mixture of unsafe, safe, and random policies; thus, the policy trained on offline data should not be optimal, and performance should improve during the online fine-tuning phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IrIjeJS7Kr", "reviewer": "Reviewer_UhwZ", "summary": "This paper studies the problem of offline-to-online finetuning of learned policies with safety constraints. This is, to my knowledge, the first work to study policy safety in this setting; prior work either focus on either offline or online RL exclusively. The key contribution of this paper is an algorithm, FOSP, that builds upon model-based RL algorithm Dreamer to enable fast and safe offline-to-online finetuning on tasks with visual observations, both in simulation and on a real robot. The algorithm works by first pretraining a world model on an offline dataset (using the in-sample trick from IQL), and then finetuning the model during the online phase using safe policy expansion to minimize safety violations. Experimental results indicate that FOSP is effective at reducing constraint violations while maintaining reasonably good task performance (i.e. rewards) compared to prior methods for safe RL. Vanilla Dreamer without regards for safety achieves consistently high reward (more than any other approach) but has significant safety violation.", "questions": "I would appreciate if the authors can address (in whichever way they deem appropriate) my comments in the \"weaknesses\" section above, as well the following questions:\n\n- How would the proposed method behave in a transfer task setting where the change is more visual in nature? E.g. the target object is now green and the obstacles are now red. I suspect that such novel settings would trick the method into substantial safety violations. It would be informative to either include experimental results or a written discussion on the types of transfer settings in which the method can be expected to maintain low safety violations.\n- Do the authors plan to release code that reproduces their main experimental results? I did not find any mention of that in the paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper studies an interesting problem (offline-to-online safe RL) that to my knowledge has not been explored much (if at all) by prior work. I appreciate the substantial background and derivation of approach which will help readers appreciate the technical contributions more. I believe that there is adequate discussion of related work.\n- The paper is generally well written and easy to follow. The paper has minor grammatical errors but they do not detract significantly from my understanding.\n- The resulting algorithm appears to rather practical, and can be applied to tasks with visual observations in both simulation and on a real robot. A significant amount of prior work in safe RL does not consider visual observations nor real robot experiments.\n- Baseline methods appear to be quite strong. I appreciate including vanilla Dreamer for comparison (it can be considered an ~upper bound in terms of task performance with no regard to safety), as well as SafeDreamer as a recent and highly related method.", "weaknesses": "- The considered tasks are fairly toy. It would be useful to know how the method performs on more realistic problem settings. I believe that the real-world task is a nice step in that direction, but being simply a reaching task it does not have any of the additional complexity that usually comes with real-world tasks: object interaction, real-time control, dynamic tasks (environment might change even with an all-zero action step). Especially the latter two can have significant safety consequences.\n- A substantial limitation of the method appears to be the lack of control or predictability in how safe the model really is during finetuning / at test-time when presented with OOD data, e.g. novel visual observations beyond scene re-configurations. I did not find any significant discussion of this in the paper, so it would be useful to clarify (verbally or with empirical evidence) when the proposed method can be expected to succeed / fail in a transfer setting; see my question below for more context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)