Robust Transfer of Safety-Constrained Reinforcement Learning Agents
Authors: Markel Zubia
Thiago D. Simão
Nils Jansen
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Safe Transfer
Adversarial Training
Robustness
Primary area: reinforcement learning
TL;DR: This paper trains agents under action disturbances for a safe and robust transfer between environments with different dynamics, preventing safety violations after the transfer.

Abstract: Reinforcement learning (RL) often relies on trial and error, which may cause undesirable outcomes. As a result, standard RL is inappropriate for safety-critical applications. To address this issue, one may train a safe agent in a controlled environment (where safety violations are allowed) and then transfer it to the real world (where safety violations may have disastrous consequences). Prior work has made this transfer safe as long as the new environment preserves the safety-related dynamics. However, in most practical applications, differences or shifts in dynamics between the two environments are inevitable, potentially leading to safety violations after the transfer. This work aims to guarantee safety even when the new environment has different (safety-related) dynamics. In other words, we aim to make the process of safe transfer robust. Our methodology (1) robustifies an agent in the controlled environment and (2) provably provides---under mild assumption---a safe transfer to new environments. The empirical evaluation shows that this method yields policies that are robust against changes in dynamics, demonstrating safety after transfer to a new environment.

Published as a conference paper at ICLR 2025
ROBUST TRANSFER OF SAFETY-CONSTRAINED REINFORCEMENT LEARNING AGENTS
Markel Zubia', Thiago D. Simão?, Nils Jansen'.3 RRhh Univesity Bochum, Germany
?iindhoven University of Technology. The Netherlands
Raabboud University Nijmegen, The Netherlands
markel.. zubi a rrub. de, t. simao@tue nl, n. jansen@rub.de de
ABSTRACT
Reinforcement learning (RL) often relies on trial and error, which may cause un- desirable outcomes. As result, standard RL is inappropriate for safety-critical applications. To address this issue, one may train a safe agent in a controlled environment (where safety violations are allowed) and then transfer it to the real world (where safety violations may have disastrous consequences). Prior work has made this transfer safe as long as the new environment preserves the safety- related dynamics. However, in most practical applications, differences or shifts in dynamies between the two environments are inevitable, potentially leading to safety violations after the transfer. This work aims to guarantee safety even when the new environment has different (safety-related) dynamics. In other words, we aim to make the process of safe transfer robust. Our methodology (I) robus- tifies an agent in the controlled environment and (2) provably provides-under mild assumption safe transfer to new environments. The empirical evaluation shows that this method yields policies that are robust against changes in dynamics, demonstrating safety after transfer new environment.
INTRODUCTION
prevalent strategy render reinforcement learning (RL, (Sutton Barto 2018) safe involves the use of transfer learning, where agents are initially trained in controlled training environment, such as a simulation or laboratory (Zhang et al.) 2020). Since unsafe interactions with the controlled environment pose real harm, these agents can be trained via standard safe reinforcement learning methods. The intention is to later transfer these trained agents to target environments where safety may be imperative (Gareía & Fernández 2015 Peng al. 2021).
While numerous works have successfully maintained safety after transferring from a controlled training environment to the target (Yang et al. 2023 Feng al.) 2023 Gimelfarb al.) 2021 Karimpanal et al.) 2020), their theoretical guarantees of safety rely on the strong assumption that selety-relevant dynamics remain unchanged between the environments. However, the target environment likely differs from the training environment. Therefore, robustness to changes or shifts in dynamics plays crucial role in realistic settings involving safety (Meng et al. 2023).
Problem statement. This work aims to train agent in a controlled environment (where safety violations are allowed) maintain safety throughout the process of transfer- learnin, even under the worst-case transition dynamics in the new environment.
Taking inspiration from the method developed by Yang et al. (003)) we focus on a setting where an agent is trained within a reward-free environment, called the source task (o). Here, the agent learns to navigate the environment safely without reward signal. To account for potential differences in dynamics, robustify this agent during the training in the source task by adding different kinds of action disturbances. These try to mimic changes in dynamics (different friction, mass, etc.) that would happen in an actual sim-to-real transfer setting, these changes in dynamics can be viewed as additional disturbances in the system (Bayar & Bernhard, 2008). More specifically, we randomly sampled action noise (Hollenstein et al. 2022), adversarial action noise Tessler al.

---

Published as a conference paper at ICLR 2025
disturbancc
guide
cource
transdez
student guide
GDknown
source (controlled environment)
target (real world)
Figure I: Training the guide under action disturbances in the source task and transferring teach the student in a target task with unknown dynamics.
2019), and entropy maximization (Ziebart et al. 2008 Eysenbach & Levine 2022). Subsequently, this trained agent is transferred to different environment, called the target task (0), where the reward is revealed. The trained agent, referred to as the guide, then used to train a new agent, the student, while making sure that no safety-violations occur during training, visualized in Figure
Contributions. The key contributions of this paper are: 1. Introduce a mathematical framework for modeling robustness in the context of safe transfer learning. Examine and compare how robustness during transfer is affected by entropy maximization, action noise, and adversarial ac- tion perturbations.
Propose an adaptation of an algorithm for safe transfer-leaminng developed by Yang et al. (2023) that accounts for uncertainties in the target environment. 4. Empirically and qualitatively analyze the proposed algorithm.
The empirical analysis shows that guides trained with action disturbances tend to have lower worst- case cumulative costs without compromising their overall performance. Furthermore, guides that are robust to changes in dynamics in the source task are better at transferring knowledge in challenging environments, with some achieving completely safe transfers, unlike non-robust ones.
RELATED WORK
Safety in RL. Traditionally, in RL, unsafe behavior is avoided through reward shaping by inte- grating the safety-related information into the reward signal (Laud & Delong 2003). However, such approaches demand significant engineering effort (Kamran al. [022 Roy et al. 2022). . To avoid reward engineering, safety can be imposed through constraints (Garcia & Fernández 2015). The
literature provides multiple types of constraints (Wachi al., (022)). For instance, mtte-wise safety involves establishing state-specific hard constraints (Zhao et al. 2023 Zhan et al., 2024). Shielding an example of a state-wise safe algorithm, where unsafe actions are blocked at runtime (Alshickh et al. 2018: Carr et al.) 2023). We focus on constraints in expectation, which provides a good trade-off between safety and performance (Altman al. 2019).
Robust MDPs. Robustness is an essential component in sequential decision-making tasks when the transition function is unknown (Suilen et al. 2022 Cubuktepe et al. 2021 Moos et al.) 2022] Gu et al.) 2025). Robust constrained Markov decision processes (RCMDP) consider
of po-
tential transition dynamics, called uncertainty set (Russel et al. 2021). An agent is robustified in this framework by iteratively exposing it the worst-case dynamics. Solving RCMDPs may in- volve computing the worst-case using linear programs (Russel et al. 2021). training an adversary to suggest such dynamics (Bossens 2024). applying perturbations the transition function (Goyal & Grand-Clement 2023). as well as combining homotopy continuation with bisection methods (Be- hzaddan al., 2021). In contrast to the RCMDP framework, we assume that the uncertainty set unknown during training to reflect realistic scenarios. We aim to train agents capable of generaliza- tion to complete the task in an environment with newly revealed dynamics.
Adversarial training. Adversarial training is common method of achieving robustness (Moos et al.. 2022). We may adversarially perturb observations by adding noise to sensors (Liu et al. Zhaagg et al. 202T), or the dynamics of the environment, for instance, by applying adversarial
2023
2

---

Published as a conference paper at ICLR 2025
physical forces to the agent (Li et al. 2023). We adopt perturbations to the agent's actions, as preliminary benchmarks showed that they consistently yielded the best results in our setting.
3
BACKGROUND
This section covers the theoretical foundations of our method.
Constrained Markov decision processes. constrained Markov decision process (CMDP, Alt- man 1999) is characterized by a tuple M (S, A, P. r, c, d, y), where S is the continuous state space, is the continuous action space, P: S x A + (S) is the transition probability function with DSstr(S) being the set of probability measures
the Borel sets of S, S A + IR is the reward function, c: x A + R is the cost function, d E R$ is the cost-budget, and y E [0,1) is the discount factor. A stationary stochastic policy is function a possibly stochastic action given the current state. We define the set of stationary stochastic poli- cies II. The expected cumulative reward of policy in state over finite horizon of length T is V (s) EE, Ei r(8t, dtt
S - Distr(A) (A) that suggests
80
and the expected cumulative cost is defined similarly
as CT = E,, Ei ((s,, d²) 80 The goal of the CMDP framework is to find an opti- mal policy x²,, that is, a policy that maximizes the expected cumulative reward while keeping the expected cumulative cost below d for all states S:
max (s) xell
(s) sdd.
The CMDP objective can be transformed with the Lagrangian relaxation method (Bertsekas) 1997).
max min (T(s) X(C (s) d).
Hell X
We also define the expected return as Q7 = E, Et 7 7rr((), (:)s =8, 8,000 = a) and the expected cost-return, Q is defined analogously.
Safe transfer-learning. In the transfer learning framework for RL, agent must leverage the knowledge gained from the source task (o) to learn the targer task (0) more efficiently (Zhu et al.) 2023 Said tasks are modeled by CMDPs in safety-critical applications, where the constraints model aspects concerning safety (Gareía & Fernández) 2015) Peng et al.) (021). Yang et al. (2023) propose reward-free source task M (8", A P 0, d' :7), where can train agent called the guide, with policy T,, that can safely navigate it without relying on a reward signal. This trained guide can then be used train the student agent, with policy σ in the target task po, p .d.7).
M =
Various transfer metrics exist for evaluating the extent which one agent benefits from leveraging the knowledge of another agent (Taylor & Stone [2009] Yang et al. (002)). One such metric is safety jump-start, which measures the difference cumulative costs during the initial epoch between an agent that uses prior knowledge and one learning from scratch. Similarly, A time to safety assesses the difference in time required to reach safe behavior. Analogously, return jump-start and time to optimum concern the reward signal.
ROBUST SAFE
LEARNING FRAMEWORK
This paper focuses on training guide in the source task and transferring it to a target task that has different dynamics, which are unknown during the training of the guide, to then train a student while avoiding safety violations. Our source task is modeled by reward-free CMDP M = (5", A Po, V. c², dd,7) 7) as already introduced in the previous section. Since the dynamics of the target task are not known, we choose a more robust model. In particular, we model the tar- get task as CMDP where the transition function has been replaced by set of possible transition dynamics functions, M = (S o _ "0, c d .7), where every po E U a dynamics function po; @ x R + Dis We refer to U as the uncertainty set. One possible instance of such an uncertainty set is to define intervals of probabilities on transitions, creating infinitely many potential dynamics of the target task.
3

---

Published as a conference paper at ICLR 2025
4.1 SAFETY GUARANTHES IN WORST-CASE ENVIRONMENTS
Since the intention is to transfer the policy of the guide, n², from M to M both environments should share the same action space:
Assumption 1. A = ~ A
Additionally, notice that the guide can only process observations from the source task's 's state space 8 Thus, to transfer the guide to the target task, need to be able to map the state space of the target task to the source space of the target task. In particular, we are interested in cases where this mapping preserves information about the dynamics concerning safety guarantee safe transfer. This concept is introduced by Li et al. (2006):
Definition Function a: S + St is a Qr-irrelvvant abstraction for .M- (S, A, P.r,c,d,)))) whenever Vs, s' SS A, VT E I1, o(s) os') = (F,M(s,a) 1 aa).
Similar model-invariant abstractions have been used in the CMDP setting to guarantee that safety constraints satisfied (Simão et al. 2022).. Now, we assume that such mapping exists from 5 to 8 and that this function preserves boundedness:
Assumption 2. There a Qf-irrelevant abstraction a: o:S 55 for that preserves boundedness.
Given arbitrary QC-irrelevant abstraction, we consider how this function can map any CMDP M to a new CMDP M the latter heing referred to as the abstracted task (Li et al. 2006). Refer to Appendix for an explicit construction.
To clarify, even though maps the states in the target task to states in the source task, it does not imply that the dynamics in the two tasks are similar, and it in fact does concern the source task's dynamics in any manner. More precisely, Assumption 2 iiplies that the states in the target task can be mapped the states in the source task, where the information about the targer task's 's dynamics is preserved, but the source task's 's dynamics are not referenced and could thus be vastly different from those of the target task.
Intuitively, strips away information that is not safety-relevant, to return new abstracted CMDP where the dynamics concerning safety are the same as in the initial CMDP. Therefore, it has been formally proven that an agent that is safe in the initial task is also safe in the abstracted task:
Lemma 1. (Abel et al. 2016) Leto: 5 S St be Q Qfiirrelevant, and tasks M = (S. A, P. c, ?) and M (ST A, ,ric v) constructed as in Appendix C Then, Vs E S. E Vm (s)
I1, O (o())
We have not yet imposed any restrictions on the uncertainty set, which means that the various dy- namies transition functions in this set could potentially be infinitely different from each other. There- fore, it is reasonable to disallow this by supposing that the uncertainty set is bounded:
Assumption 3. U is bounded, that is, Re E RR P' E U P P'I S.. &
In the following theorem, we prove that an agent that is robust against changes dynamics in the source task will retain part of this robustness when transferred to the target task:
Theorem 1. Given Assumptions
2 and
df < dd then there exists 5 E IR such that policy po 6, then is safe in MG for all P' Uo.
is safe in M9p for all P where
Proof. Since U is bounded (Assumption and preserves boundedness (Assumption) D. there is
max ||pt - | PeU'
where the abstracted dynamics pt is constructed as in Appendix
C Now, suppose that π is safe for
all P with | S 6, that is, a (s, a) & dº Then, for all P U have C (8,a) a) s d
by construction of 8. Thus, 2 ( (8,a) S d < d given Lemma and premise do < d O
This proof is visualized in Figure Theorem I justifies increasing ó in the source task with robus- tification methods before transferring to a target task with uncertain dynamics, which an essential component in the method we introduce in Section 5
4

---

Published as a conference paper at ICLR 2025
M
M
(Uo)
8
Figure 2: Visualization of the space of dynamics functions from Theorem
|
4.2 ROBUST TRANSFER METRICS
Key insight. While an agent that is robust in the source task will also be robust in the target task, as shown in Theorem [ this is not enough to be able to perform the transfer of knowledge successfully. To illustrate, notice that in many environments the "do nothing" policy is safe everywhere U even though it is clear that this policy provides no value to train an agent in the target task. Thus, sometimes an agent that prioritizes exploration while taking more risks may be more useful for training another agent in the target task, compared to an overly conservative one, resulting in an overall lower cumulative cost. This serves as motivation for introducing metrics that measure the usefulness of the guide's knowledge from the perspective of safety and robustness.
Let's say that algorithm uses the knowledge from the agent trained in the source task, n², to train a new agent n in the target task. To do so, the algorithm needs to sample K trajectories of length in M generating a history of environment interactions {pt have a history of probability distributions over the space of trajecto-
Since the algorithm
may be nondeterministic, ries, denoted by E Distr((S x A)" x S) where the Kleene operator (+)* indicates that we may have finite but arbitrarily long sequences of state and action pairs. This is expressed compactly as M)) {pihi=1..... and we denote learning from scratch as (0,
Various transfer metrics exist for evaluating which extent the target policy 7 benefits from leveraging the knowledge of T (Taylor & Stone 2009), some of which have been adapted to account for safety by Yang al. (2023 As these metries have only been defined informally in prior work, we next provide rigorous definitions.
Definition 2. Given ((T", M = {pt} and (0) = {pt}. , the safety jump-start is defined as
3 M (((T)- c(F')
T ~ pl, 7' ppl
c(T')
where c((s,, ma Ete T'c(,,
In other words, the safety jump-start measures the difference in excess cumulative costs during the initial trajectory or epoch between an agent that uses prior knowledge and one learning from scratch.
Definition 3. The A time to safety assesses the difference in time required to reach safe behavior: Given (= .M0-- = {pr} and (0, M)) [p]). the metric is defined as
C(P, n E ((T)) m((Ta))) T4 P.. pil. where m((r:)) = miny=1.....4 such that Vt' 2 t, c(T)) $ d
Definition 4. Analogously the safety jump-start and A time to safety, the return jump-star start and A time to optimum concern the reward signal, respectively denoted by B(M x²) and R(M m)).
The transier-lerming problem at hand to train the guide in a reward-free source task modeled by CMDP, M later it to learn student policy in the target task modeled by CMDP, M with uncertainty set U is important to bear in mind that M completely inaccessible during the training of the guide in M meaning that the guide cannot interact with 1 during training.
Formal statement of the problem. Given a reward- source CMDP M and a target CMDP M with uncertainty set U minimize the worst-case safety jump-start while keeping A time to
5

---

Published as a conference paper at ICLR 2025
optimum above threshold d,
min max 3(P,)) s.t. R(P,T) 2 dy. xell PEUO
Even though the problem statement only includes the safety jump-start and time to optimum of the guide, we interested in measuring the A time to safety and return jump-start as well. Definition 5, Given a policy E IT, the worst-case transition dynamics is
p+ arg max (P,
PEU
In practice, estimating the worst-case transition dynamics may require evaluating policy within various E U and picking the environment that yields the highest cost. Consequently, notice that we likely choose outliers through this process, causing the approximate worst-case safety jump-start to have high variance. In an attempt to mitigate this issue, rather than selecting only one environment, one can pick a portion p E [0, of the environments where the agent performs the worst. The following definition formalizes this concept.
Definition 6. For p E [0, 11, the p-tail of the safety jump-start is
JSP(M xx) = J(M). T)
there exists y E R such that
B [oo]
p. where B (P U (3MP y).
Additionally, (M, x) is the average performance of within the entirety of U While the problem statement does not include the p-tail and average performance, we will keep track of them in the empirical analysis.
ROBUST GUIDED SAFE EXPLORATION
Our method trains robust guide via action disturbances in the source task and transfers it to the target task to train the student safely.
5.1 TRAINING THE GUIDE
Since the source task reward-free CMDP, M (S,A, it likely that an agent trained in this environment would learn to "do nothing", as it would only need to satisfy the safety constraints. Therefore, we use distance bonus encourage the guide explore the environment,
T (st, az) EE ||st اا+1|1 Sc+1 P(s.,
where ||: SF - IR is norm and (-) is state abstraction function Yang al. 2023
To transfer the guide to uncertain target task, we must first robustify it during its training in source task. We employ three domain-agnostic robustification techniques.
Entropy maximization.
To maximize the entropy of the agent, we include the following term in the reward signal r (st, ax) = ax) + arr"'st, (st, at), where r2H( a.) log =(a)))))
Random action noise. Instead of sampling the action directly from the agent, some random noise is added: (1 (s)))s))) + o. (),
Adversarial perturbations. The guide is trained in a noisy action robust MDP Tessler et al., 2019 a (1 - a (s;(s) + (+t)s, where the guide trying to maximize the objec- tive rt r while satisfying the cost constraints, and the adversary is trying maximize the cumulative cost.
Prior work has established that perturbation size (a) provably makes the policy robust to some 8 (Feng et al.] 2020 Eysenbach & Levine 2022). Therefore, a, we are effectively computing a robust policy for a larger uncertainty set in the source task. Con- sequently, the agent will be safe in larger set of target tasks. As theoretical bound between
increase the value of

---

Published as a conference paper at ICLR 2025
and ô can only be shown in specific scenarios, adopt empirical approach in the next section and evaluate the robustness effect across different values a.
Note that Theorem I does not state that there always is robustly safe policy. If ó is too large, such policy may not exist. In practical terms, we fail to find a feasible policy for a given 6, our method still attempts to find conservative policy in the source target, which ensures reduction of safety violations in the target environment. Alternatively, we could search for the largest < 5 such that a feasible policy exists. Nevertheless, this policy can easily be found in the constrained RL setting. For instance, in problems exhibiting an action with zero cost, such as in the environments used in the empirical evaluation, the agent can choose this action, which ensures it will satisfy the constraints.
5.2 TRANSFERRING KNOWLEDGE TO THE STUDENT
To transfer the knowledge of the guide to the student, our approach is similar to (Yang et al. 2023) We want to ensure safety throughout by sampling from the student whenever the cost signal is zero, and otherwise we sample from the guide:
n" (st) = [ ( (s))
{ (
if there is a S where cy O, else.
is desirable let the student's policy imitate the guide's whenever the cumulative cost is above the safety threshold, while this is not necessary once the student behaves safely. We achieve this effect by adding a new term ri to the reward signal measuring the similarity between the two policies, weighted by the Lagrangian multiplier, A. When both policies nondeterministic, r measures the KL divergence between the two distributions. Otherwise, if either policy is deterministie, we compare the deterministic action with the mean of the nondeterministic policy's distribution.
6 EMPIRICAL ANALYSIS
We train one guide for each method of robustification (Section 5.1 answer the following questions:
Our empirical analysis aims to
1. How does each guide perform in the source task?
2. How does each guide perform in the target task's worst-case dynamics?
3. What is the safety jump-start of each guide in the target task?
4. How is the full training of the student affected by the guide's robustness?
We evaluate our method on benchmark environments created using a framework for safe rein- forcement learning called Safety-Gymmasiine (Ji et al., (023). In these environments, the RL agent controls robot that must reach the goal while avoiding the hazards. Appendix B provides more details.
The source tasks. There are three source tasks: (M1, MG. and Mg). These environments may contain static obstacles called hazards and dynamic obstacles called vases, which are always con- strained. Additionally, all three environments have constrained walls to discourage the agent from going out of bounds. The environments differ as follows: MM has hazard located in the center; M2 has hazards that change location every epoch, and Mg has 8 hazards and vases that change location every epoch.
The target tasks. source environments MG, MG, and Mg. except for two major differences.
The target environments MP. M and My similar to their respective
1. The task in the target environments is to reach a specific location, called the goal. There- fore, the observations in the target tasks have the additional measurements that concern the reward signal: a LIDAR in [0, 16 for detecting the goal, We have a mapping a: 5 - S which simply strips away the information concerning the reward signal. It is easy to show that a Q" irrelevance abstraction, satisfying Assumption
'The source code is available
wwwiiiff/saafe-eee and www.st-transtttttee

---

Published as a conference paper at ICLR 2025
Maximize Entropy
Adversarial Perturbation
Random Perturbation
Cost 500
env3
env2
20
Return
10
0.0
1e-08
0,001
10,0
0,06
0.12
0.17
0,23
0,29
0.35
0.06
0.12
0.17
0.23
0,29
0.33
Figure 3: The cost and return in the last epoch of each type of guide.
2. The transition dynamics function in the target tasks is uncertain. In our practical imple- mentation, this is achieved by modifying the physics parameters concerning the friction and mass the agent in the simulated environments. The friction refers to the force that opposes the motion of the robot, and higher friction value causes the agent to decelerate quicker. The mass of the agent affects its moment inertia, meaning that it impacts how quickly the agent can change its velocity.
One can define the function u: I2 + (S x A - (S)) that maps the friction and mass of the agent to the corresponding transition dynamics function. Thus, notice that preserves boundedness, which implies that U is bounded, satisfying Assumption 3
6.1 PERFORMANCE OF THE GUIDES THE SOURCE TASK
We train various guides with the three robustification techniques: random action noise, adversarial action perturbations, and entropy maximization. The guides are trained with different weights of the robustification methods to which values of these parameters yield the best outcomes. The ranges of these weights are based on the results of prior work (Tessler et al.) 2019 Liu et al. (023): the random noise is weighted by a (0.06, 0.12, 0.17, 17, 0.23, 0.29, 0.35 the adversarial perturbation weights are a E (0.06, 0.12, 0.17, 0.23, 0.29, 0.35]; and the entropy bonus is weighted by E {10 10-3 10 0.2 0.5, 1.0, 1.5, 10.0).
Figure 3 shows the cumulative returns and costs at the very last epoch of the three kinds of agents trained with the different values of the robustification weights. Most guides learn a safe policy that obtains a cumulative reward slightly below 20, except for the entropy- matropy-maximizing agent when 0.1, which does not learn a proper policy, presumably due the overly high entropy.
6.2 GUIDES' PERFORMANCE IN THE TARGET TASK'S 's WORST-CASE DYNAMICS
The uncertainty set (U)) consists of uncountably many dynamics transition functions, parameter- ized by the mass (m) and friction (77) of the agent, making it challenging to compute the worst-case dynamics. Therefore, we restrict the uncertainty set to a finite subset (U+) by discretizing the values of the parameters to m = m,,.... m, and rj = ........ AN. In our experiments, we use values for each parameter by letting m4 (0.5 - )m and = (0.5 + - ))² for = 1, 8.
where m and 17 correspond to the dynamics in the source task.
First, evaluate all guides within their respective source tasks, but instead of using the source task's dynamics (Pº), we evaluate them within the discretized uncertainty set of the target task, T This experiment provides insights into the relationship between the different robustification algorithms and the robustness they provide. Figure 4 shows the cumulative costs of a robust and non-robust agent evaluated in the dynamics functions of the discretized uncertainty set.
The robust guide can navigate the source task safely even when the dynamics are unfavorable, while the non-robust agent struggles maintain safe expected cost when the shift in dynamics is large.
8

---

Published as a conference paper at ICLR 2025
1.6
1.4
30
30
Friction
1.2-
Friction
1.2
1.0
0.8
0.8
-1
10
0.6
0.4
1.4 1.6
0.44
0.6
1.0 Mass
1.2
1.6
Mass
Figure 4: The expected cumulative costs in Mi of guide trained with no action noise (left) and guide with an action noise of a = 0.29, each dot representing a different dynamics function, with mass the r-axis and friction on the y-axis.
1.6
1.6
100
1.4
14
30
50
Friction
Friction
1.2
1.0
-10
-50
0.4
0.6
0.8
1.0 Mass
1.2
L.4
1.6
0.4
0.6
0.8
L.D
12
14
1.6
Mass
Figure 5: Expected cumulative cost in the source task (left) and the safety jump-start in the target task as percentage (right) of the agent trained with a = 0.29 within M₁. Low values of the safety jump-start are desirable, where a jump-start of 100 concerns a fully safe transfer.
Heatmaps like the ones shown in Figure 4 have been computed for all three robustification methods and different values of a and they can be seen in Appendix F Once these heatmaps are calculated, it is easy obtain the worst-case expected cost, well the p-tail and the average, shown in Tables 2 and It is crucial to note that the values shown in the tables are the expected costs of one batch guides, implying that conclusions cannot be drawn from the comparison the values of a during training with their respective costs, due to lack of statistical significance. Nevertheless, since the expected costs are computed quite accurately,
correct to claim that some guides are more robust than others, which will be useful in the next sections to measure how the robustness of the guide impacts the student's training in the target task.
6.3 SAFETY JUMP-START OF EACH GUIDE IN THE TARGET TASK'S WORST-CASE DYNAMICS
The safety jump-start assesses the difference in cumulative costs during the first epoch between the student learning from guide and one learning from scratch. Measuring the safety jump-start is computationally cheap compared to other metrics, as it requires training the student for only one epoch as opposed to fully training said student.
We transfer the knowledge of the guides to two kinds of students: nondeterministic student (SAC) and deterministic one (DDPG). One major problem with transferring the knowledge to determin- istic student is that it is much likelier to end up in unrecoverable states by selecting the same actions over and over, rendering the control-switch rescue method completely ineffective. To lessen this issue, the deterministic student is trained with relatively high random action noise of a 0.75.
Appendix G shows tables collecting the worst-case, 0.1-tail, and average safety jump-starts mea- sured from every guide to the two kinds of students. An interesting observation when comparing these with Tables and is that policies that appear robust in the source task may not necessarily have the ability effectively transfer their knowledge to student in their target task. An extreme

---

Published as a conference paper at ICLR 2025
Mass x0.50
350
Mass x0.93
300
Mass x 1.21
200
Mass x1.50
150
50
0.6
Friction multiplier
Figure 6: The expected cumulative costs in M2 for the average guide given each mass mass-friction pair.
20
800
Robust
10
Non-robust
600
Train from scratch
Retur
.10
400
Robust
20-
Non-robust
200
30
Train from scratch
100
200
300
400
500
100
200
400
500
Epoch
Epoch
Figure 7: Comparison the cumulative rewards and costs during training between the students trained with the robust guide (a 0.29) and the non-robust guide (Yang et al. 2023)
example of such phenomenon be in Figure where agent that is robust in the source task has an exceptionally poor safety jump-start in worst-case environments.
6.4 EFFECT OF THE GUIDE'S ROBUSTNESS ON THE STUDENT'S TRAINING
To determine how the transfer is affected by the robustness of the guide, we train the student with a robust guide and non-robust one. For the robust guide, we will use the agent trained with action noise in My where introduced by Yang et al. 2023). We then transfer the guides to the dynamics that appear most challenging based on the chart shown in Figure m 1.5m² and 17 0.5mf'
0.29. The non-robust guide is trained within M2 using the method
The reward and cost during training of the students trained with robust and non-robust guides are shown in Figure 7 Since the goal of the transfer is avoid safety violations within the target task, the transfer done with the robust guide seems ideal. Even though the training takes place in a task with unfavorable dynamics, the safety jump-start with the robust guide nears nears- -100% meaning that the behavior policy is safe from the very start. Moreover, the return jump-start of the student with the robust guide is much greater than that of the student with a non-robust guide.
7 CONCLUSIONS
We propose a method to transfer agents to environments with different or even worst-case dynamics while satisfying safety constraints. The empirical evaluation shows that, in general, agents trained with action disturbances have lower worst-case expected cumulative costs without sacrificing the expected return. In addition, observe that guides that are robust in the source task do not always have a favorable worst-case jump-start, which backs our theoretical insights. Lastly, the agents with low worst-case safety jump-start demonstrate better capability for transferring knowledge to the student unfavorable dynamics, compared to non-robust guides, where some robustified agents have been shown capable of achieving fully safe transfer.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like thank the anonymous reviewers for their valuable feedback. This research has been partially funded by the ERC Starting Grant 101077178 (DEUCE).
REFERENCES
David Abel, D. Ellis Hershkowitz, and Michael L. Littman. Near optimal behavior via approximate state abstraction. In ICML, pp. 2915-2923, 2016.
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topeu. Safe reinforcement learning via shielding. In AAAI, pp. 2669-2678, 2018.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Eitan Altman, Said Boularouk, and Didier Josselin. Constrained Markov decision processes with total expected cost criteria. In VALUETOOLS, pp. 191-192, 2019.
Tamer Başar and Pierre Bernhard. H-infinity optimal control and related minimax design problems: dynamic game approach. Springer Science & Business Media, 2008
Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for Lxc connstrnae s- rectangular robust MDPs. In NeurlPS, pp. 2998222599,, 2021.
Dimitri P. Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3): 334-334, 1997.
David M. Bossens. Robust lagrangian and adversarial policy gradient for robust constrained Markov decision processes. In IEEE CAI, pp. 1227-1239, 2024.
Steven Carr, Nils Jansen, Sebastian Junges, and Ufuk Topeu. Safe reinforcement learning via shield- ing under partial observability. In AAAI, pp. 14748-14756, 2023.
Murat Cubuktepe, Nils Jansen, Sebastian Junges, Ahmadreza Marandi, Marnix Suilen, and Ufuk Topcu. Robust finite-state controllers for uncertain POMDPs. In AAAI, pp. 11792-11881 11800, 2021.
Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In ICLR, 2022.
Fei Feng, Ruosong Wang, Wotao Yin, Simon S. Du, and Lin F. Yang. Provably efficient exploration for reinforcement learning using unsupervised learning. In NeurIPS, 2020.
Zeyu Feng, Bowen Zhang, Jianxin Bi, and Harold Soh. Safety-constrained policy transfer with successor features. In ICRA, pp. 7219-7225. IEEE, 2023.
Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16:1437-1480, 2015.
Michael Gimelfarb, André Barreto, Scott Sanner, and Chi-Guhn Lee. Risk-aware transfer in rein- forcement learning using successor features. In NeurIPS, pp. 17298-17310, 2021.
Vineet Goyal and Julien Grand-Clement. Robust Markov decision processes: Beyond rectangularity. Mathematics of Operations Research, 48(1):203-22) 22032222622222 2023.
Shangding Gu, Laixi Shi, Muning Wen, Ming Jin, Eric Mazumdar, Yuejie Chi, Adam Wierman, and Costas Spanos. Robust gymnasium: A unified modular benchmark for robust reinforcement learning. In ICLR, 2025.
Jakob J. Hollenstein, Sayantan Auddy, Matteo Saveriano, Erwan Renaudo, and Justus H. Piater. Action noise in off-policy deep reinforcement Iearning: Impact on exploration and performance. Trans. Mach. Learn. Res., 2022, 2022.
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong. Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark, 2023.
11

---

Published as a conference paper at ICLR 2025
Danial Kamran, Thiago D. Simão, Qisong Yang, Canmanie T. Ponnambalam, Johannes Fischer, Matthij T. J. Spaan, and Martin Lauer. A modern perspective on safe automated driving for different traffic dynamics using constrained reinforcement learning. In ITSC, pp. 4017-4023, 2022.
Thommen George Karimpanal, Santu Rana, Sunil Gupta, Truyen Tran, and Svetha Venkatesh. Learning transferable domain priors for safe exploration in reinforcement learning. In IJCNN, pp. 1-10, 2020.
Adam Laud and Gerald DeJong. The influence of reward the speed of reinforcement learning: An analysis of shaping. In ICML, pp. 440-447, 2003.
Lihong Li, Thomas J. Walsh, and Michael L. Littman. Towards unified theory of state abstraction for MDPs. In AI&M, 2006.
Zeyang Li, Chuxiong Hu, Shengbo Eben Li, Jia Cheng, and Yunan Wang. Robust safe reinforcement learning under adversarial disturbances. In CDC, pp. 334-341111 341. IEEE, 2023.
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang. Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. In ICLR, 2023.
Jinling Meng. Fei Zhu, Yangyang Ge, and Peiyao Zhao. Integrating safety constraints into adversar- ial training for robust deep reinforcement learning. Inf. Sci., 619:310-323, 2023.
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Mach. Learn. Knowl, Extr., 4(1):276-315, 2022.
Zhenghao Peng, Quanyi Li, Chunxiao Liu, and Bolei Zhou. Safe driving via expert guided policy optimization. In CoRL, volume 164, pp. 1554-1563, 2021.
Julien Roy, Roger Girgis, Joshua Romoff, Pierre-Luc Bacon, and Christopher J. Pal. Direct behavior specification via constrained reinforcement learning. In ICML, pp. 18828-18844 2022.
Reazul Hasan Russel, Mouhacine Benosman, Jeroen Van Baar, and Radu Corcodel. Lyapunov robust constrained-MDDs: Soft-constrained robustly stable policy optimization under model uncertainty. arXiv preprint ar #rXiv22108.02001 2021.
Thiago D. Simão, Nils Jansen, and Matthijs T. J. Spaan. AlwaysSafe: Reinforcement learning
without safety constraint violations during training. In AAMAS, pp. 1236-1235, 2021.
Marnix Suilen, Thiago D. Simão. David Parker, and Nils Jansen. Robust anytime learning of Markov decision processes. In NeurIPS, 2022.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, second edition: An Introduction. MIT Press, 2018
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. J. Mach. Learn. Res., 10:1633-1685, 2009.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. In ICML, volume 97, pp. 6215-6224, 2019.
Akifumi Wachi, Xun Shen, and Yanan Sui. survey of constraint formulations in safe reinforcement learning. In IJCAI, pp. 8262-8271, 2024.
Qisong Yang, Thiago D. Simão, Nils Jansen, Simon H. Tindemans, and Matthijs T. J. Spaan. Rein- forcement learning by guided safe exploration. In ECAI, pp. 2858-2865, 2023.
Simon Sinong Zhan, Yixuan Wang, Qingyuan Wu, Ruochen Jiao, Chao Huang, and Qi Zhu. State- wise safe reinforcement learning with pixel observations. In L4DC, pp- 1187-1201, 2024.
Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsich. Robust reinforcement learning state observations with learned optimal adversary. In ICLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, and Dinesh Jayaraman. Cautious adapta- tion for reinforcement learning in safety-critical settings. In ICML, pp. 11055-11065, 2020.
Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu. State-wise safe reinforcement learning: A survey, In IJCAI, pp. 6814-6822. ijcai.org, 2023.
Zhuangdi Zhu, Kaixiang Lin, Anil K. Jain, and Jiayu Zhou. Transfer learning deep reinforcement learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 45(1
2023.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI, pp. 1433-1438, 2008
13

---

Published as a conference paper at ICLR 2025
A HYPERPARAMETERS
The hyperparameters in our method are summarized Table modeled by a multilayer perceptron (MLP).
All actor and critic networks are
Parameter
Mi
M2
M3
Actor network size
[256, 256]
[256, 256]
[256, 256]
Critic network size
[256, 256]
[256, 256]
[256, 256]
Size of replay buffer
10
106
10
Batch size
256
256
256
Steps per epoch
2000
2000
2000
Number of epochs
106
106
10
Actor learning rate
5 10-6
5 10-6
5 10-6
Critic learning rate
10
10-3
10-3
Lambda learning rate Safety constraint
5 10
50
10-
5
8
25
Table 1: The hyperparameters used the experiments.
B THE ENVIRONMENTS
The actor is small robot called the poin/ that can move forwards, backwards, and steer left-to-right. The action space [-1, 1J2, where the first value for the throttle and the second one for steering. The observation space has the following components:
Pseudo-LIDAR for hazards in [0, 1
Pseudo-LIDAR for vases in [0, 1]
Velocimeter (-00, )3
Accelerometer in (-00, oo)
Gyroscope in (-00, 00)3
Magnetometer in (-o0, )3
The pseudo-LIDAR casts 16 rays in different angles. It is termed "pseudo" because, unlike with real LIDARs, the rays can go through objects. The other sensors measure in three dimensions: velocimeter for velocity (m/s), an accelerometer for acceleration (m/s²), a gyroscope for angular velocity (rad/s), and magnetometer for magnetic flux (Wb).
Renders of the tasks are shown in Figures
and 10
Figure 8: Renders of tasks Mi (left) and M (right).
14

---

Published as a conference paper at ICLR 2025
Figure 9: Renders of tasks M3 (left) and M2 (right).
Figure 10: Renders of tasks Mg (left) and M (right).
Mass x0.5
Mass x1.0
Mass x1.5
Friction x0.5
Friction
Friction
Figure 11: The trajectories of (-)) (1.0, 1.0) where the mass and damping multiplied by 0.5, 1.0, and 1.5. Both multipliers are 1.0 in the nominal dynamics.
15

---

Published as a conference paper at ICLR 2025
C STATE ABSTRACTION
Q'-irrelevant abstraction a: S - St maps CMDP to an abstracted CMDP (Li et al. 2006
Let M A.r. 7), and S [0, 1]. The corresponding abstracted task is M (S), A, Pt cr,, 7), where
Pl(s") |s) - EcEo-l(ot) (0) 1s,0).
rt(st,a) Enco 1(4) 1(1) (())(s)))
(()),a) -Eso--(0)) ((s)c(s,)) a),
with o (s)= (ss) = ز and Exco- w(s) L, for S
D CUMULATIVE COSTS THE UNCERTAINTY SETS
M
M3
Mg
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
0.00
805.1
780.2
316.7
862.6
856.2
268.6
536.9
457.9
89.3
0.06
843.2
816.6
221.5
353.3
221.4
41.2
635.0
574.5
130.6
0.12
873.5
868.2
438.1
842.8
792.9
179.2
89.7
77.2
34.2
0.17
780.8
683,9
150.5
692.8
399.0
110.4
725.0
569.0
95.8
0.23
2.3
1.5
0.2
91.3
77.7
21.5
286.2
268.8
126.5
0.29
3.2
1.1
0.1
95.7
59.7
18.1
859.0
754.0
168.2
0.35
134.5
100.0
11.5
271.6
162.9
58.6
833.6
809.0
190.1
Table 2: The worst-case, 0. 1tt-ia,,, and average expected costs of all the guides trained with random action noise within the three source tasks.
Mi
Mg
M
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
0.0
805.1
780.2
316.7
862.6
856.2
268.6
536.9
457.9
89.3
0.06
880.4
871.5
537.2
769.5
717.3
108.6
219.7
210.3
49.0
0.12
840.0
821.7
302.7
626.6
569.8
75.8
338.1
218.5
45.3
0.17
484.0
367.2
87.7
713.9
554.1
109.7
166.4
148.3
65.5
0.23
790.7
774.8
372.5
773.0
678.9
97.5
459.9
335.5
102.8
0.29
38.5
17.5
2.7
233.8
212.4
52.6
699.5
640.5
143.7
0.35
742.5
687.9
294.2
211.4
139.0
57.2
255.3
225.7
84.7
Table 3: The worst-case, 0.1-tail, and average expected costs of the guides trained with adver- sarial perturbations within the three source tasks.
Mi
M
M
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg-
0.0
147.4
82.9
9.7
668.7
563.6 423.1
108.2
892.7
881.7
262.4 186.4
10-3
594.2
139.0
13.2
551.1
59.3
877.6
871.0
Table 4: The worst-case, 0-1ita and average expected costs of all the guides trained with entropy maximization within the three source tasks.
E SAFETY JUMP-START TABLES
16

---

Published as a conference paper at ICLR 2025
Mi
M2
Mg
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
0.06
-60.4
-62.1
-86.6
-56.3
-63.0
-87.4
-48.9
-55.1
-82.9
0.12
-49.6
-54.3
-80.9
-31,4
-39.2
-84.1
-41.5
-53.6
-80.2
0.17
-64.3
-69.8
-87.9
-32.4
-51.2
-83.9
-36.6
-47.6
-80.8
0.23
-71.2
-80.3
-91.9
-34.8
-49.7
-85.2
-23.7
-39.1
-75.0
0.29
-62.8
-65.2
-86.5
-55.8
-59.5
-87.3
-15.6
-25.7
-76.4
0.35
-49.3
-61.8
-87.5
-5.8
-35.7
-79.6
-29.7
-38.6
-78.9
Table 5: Deterministic students with the random action noise guide.
Mi
M
Mg
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
0.06
179.7
134.9
-17.6
50.6
13.1
-63.0
18.8
2.7
-57.9
0.12
263.5
183.3
15.9
138.2
105.3
-47.3
64.5
40.4
-50.6
0.17
156.9
101.6
-20.3
123.1
53.2
-51.3
70.1
29.1
-53.1
0.23
46.7
18.7
-52.3
117.4
47.8
-56.1
68.5
48.0
-47.0
0.29
195.0
126.4
-18.8
86.2
26.8
-63.7
126.7
105.7
-41.1
0.35
205.1
103.8
-28.7
135.2
92.8
-41.1
116.2
63.2
-51.6
Table 6: Nondeterministic students with the random action noise guide.
Mi
M2
M3
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
0.06
-50.8
-57.0
-82.7
-44.9
-52.4
-84.1
-41.7
-49.0
-79.8
0.12
-56.6
-63.0
-87.2
-31.3
-39.9
-83.2
-39.7
-51.2
-81.3
0.17
-68.4
-72.1
-88.6
17.3
-28.9
-78.8
-41.8
-52.4
-80.8
0,23
-52.7
-64.4
-85.7
-19.9
-29.9
-80.5
-49.7
-58.6
-83.2
0.29
-62.0
-69.4
-87.8
-31.8
-49.4
-83.5
-52.3
-59.1
-82.9
0.35
-65.0
-67.6
-80.5
-61.0
-68.2
-88.4
-48.8
-52.4
-79.9
Table Deterministic students with the adversarially trained guide.
Mi
M
M
Worst
0-titail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
0.06
177.7
143.1
0.2
151.5
86.9
-50.1
61.7
30.8
-5.55
0.12
106.3
88.9
-30.8
142.1
102.6
-46.3
43.0
25.6
-54.8
0.17
73.6
53.4
-36.2
161.0
122.9
-36.3
21.2
13.4
-54.3
0.23
197.2
125.7
-14.2
135.2
99.9
-44.9
40.9
21.3
-55.6
0.29
188.7
101.0
-23.7
86.6
36.1
-55.4
24.1
1.3
-59.4
0.35
172.2
112.2
10.1
56.3
26.5
-63.5
74.1
39.4
-50.2
Table 8: Nondeterministic students with the adversarially trained guide.
Mi
M₁
Worst
0.
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
0.0
-69.3
-75.5
-90.8
-34.3
-58.5
-86.2
12.3
-5.0
-71.8
0.001
-44.0
-66.2
-89.4
-30.1
-54.9
-84.8
-0.7
-20.5
-73.6
Table Deterministic students with the entropy maximizing guide.
17

---

Published as a conference paper at ICLR 2025
Mi
M2
Mg
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
Worst
0.1-tail
Avg.
0.0
44.9
29.1
-46.3
80.3
45.3
-57.2
132.1
91.5
-00.1
0.001
165.2
77.6
-37.4
101.6
73.8
-51.4
137.8
108.3
-36.0
Table 10: Nondeterministic students with the entropy maximizing guide.
18

---

Published as a conference paper at ICLR 2025
F ROBUSTNESS HEATMAPS FROM THE SOURCE TASK
Avg. cost actroise, 9,00 task1
actnoise, 0.00 mlsk2
cost actnosso Baakkk
2.8
35
33
12
17
12+
45
3.0
29
A00
15
15
64
04-
044
04
04
Body
Dody
masi
Avg. actroise 0,06
actroise 0,05
of
3,06 task3
3.0
25
-5
1.2
1.2
25
3.0
22
3.0
23
2.0
20
15
15
2.8
0.
0.4
Body
Avg.
Aug. actnoise, maakk
Aug.
3.5
3.0
3.0
1.4
2.4
23
25
1.0
22
3.0
20
3.0
08-
0.
30
54
1.0
12
L4
12
1.4
0.6
1.6
Bady
Dody
Avg. cost actroise 0,17 task1
Ava. actnoise 0.17 3ask2
Avg. cost of accioiso 0,17 tesk3
3.5
40
3.0
3.0
25
3.1
25
25
25
22
3.0
6-3
0.
3.8
10
08
166
14
1.4
17
144
1.
Body
Body
Body mast
19

---

Published as a conference paper at ICLR 2025
Avg. cost actroise 0,23 taskl
Avg. cosl of actroise 0,23 Jask2
Avg. cost of accanise 0.23 Lesk3
14-
30
1.
12-
1.2
22
1.
2.0
15
15
15
0.8
044
1.6
04
1
1/
1.4
04-
04
0.6
Body man
Body mass
Body mass
zost actroise 0.29 task1
cost actnoise 0.29 mwsk2
Avg. cost acthois 3.29 twnks
3.0
3.0
39
30
14
25
22
3.0
20
.03
04
54
1.2
54
Bady
Dody
Avg. cost actroise 0,35 task1
Ava. cost actroise 0.35 cask2
Avg. cost of accaoiis tesk3
1.5
3.0
3.0
1
3.4
25
25
25
22
20
0.
18 0.
2.8
88
0.6
A/ 1-B Body mase
1:4
1/
1.4
1
Body
Body made
Figure 12: Average expected costs of all the guides trained with random action noise within the three source tasks, where each dot represents a different dynamics function.
Avg. cost advect, 0,00 basiC.
Avg. cost of advact 0,00 task2
Avg. cost advect 0,00 task3
3.5
40
3.0
1.0
25
1.4
3.1
30
25
25
25
22
3.0
6-3
0.
3.8
10
08
06
1.6
14
1.4
6.
17
1.4
1.
Body mase
Body
Body mast
20

---

Published as a conference paper at ICLR 2025
Ang. coul of adeact, 0,06 Lesk:
Avg. cost of adivact 0,06 tawk2
Avg. coul of edvect 0,06 tesk3
14
14
30
1.
12-
1.2
22
1.0
2.0
15
15
15
10
0.8
044
5/f
1.6
12
04
1.5
1/2
1.4
04-
54
0.6
Body maz
Body mass
Body mass
cost adeect 0,12 Pask:
Aug. cost of advoet 0.12 task?
Aug. cost advaet, 6,12 tasics
3.0
39
30
144
1.0
22
20
.033
200
04
54
1.2
54
Bady
Dody
advact 0,17 taski
cost of advect 0,17 tesk2
Avg, advect C,17 tesk
3.0
25
25
1.2
1.2
25
1.0
22
3.0
20
2.0
20
15
0.8
A/
0.
12
0.4
adooet beck?"
1.6
3.6
1.4
1.4
122
23
25
3.0
22
3.0
20
3.0
25
15
6
244
10
04
54
1.0
DUB
Dody
Dady
21

---

Published as a conference paper at ICLR 2025
Avg. coul of adeact, 0,29 Lesk:
Avg. cost of adivact 0,29 tewk2
Avg. coul of edvact C teak3
30
1.
12-
1.2
22
1.0
3.0
15
15
15
045
044
1.6
04
1.3
1/
1.4
04-
54
0.6
Body maz
Body mass
Body mass
Avg. cost actr Sitroisse 0,35 task1
Avg. cost of octnoise 0,35 cask2
Arg. cost of acaooise 5,35 task3
1.4
3.6
M
14
1.4
30
30
25
15
0.8
10
10
10
04
04-
24
44
0.6
A/
1.6 Body may
17
14
1.
0.
0.6
0.
12
144
1.6
04
06
6-
1-5
17
1
16
Body mags
Body mass
Figure 13: Average expected costs of all the guides trained with adversarial perturbations within the three source tasks, where each dot represents a different dynamics function.
Avg. rost of alphasar 0,0 bawk]
Avg. cost alphosec 0,0 tank?
cost alphasec O,C. taakk
10
30
1.2
12
1.2
32
3.0
15
15
15
0.5
04
04
1.3
04
Dady
Dady
22

---

Published as a conference paper at ICLR 2025
Aug. cosl of alphasac 0.002 taski
Avg. cosl of alphassc 0,001 tosk2
Aug. cosl alpi asac 0,001 task3
30
12
12
1.2
22
15
15
15
10
34-
0441
1.6 Body mane
04
1.3 Body mass
1.2
1.4
04
Body mass
Figure 14: Average expected costs of all the guides trained with entropy maximization within the three source tasks, where each dot represents a different dynamics function.
G SAFETY JUMP-START HEATMAPS
G.1 TRANSFERRING TO THE NONDETERMINISTIC STUDENT
Safety jump-stert
Safety jump-start
Safety jump
1.6
203
3.0
10C
3.0
103
23
25
1.0
3.0
3.0
08
-32
-75
-73
-75
-150
-100
64
1.0
1.4
0.6
Bady
Dady
Safeity jump- start actnoise 0,12 task1
Safety jump actnossa 0,12
Safety jump actnoise task3
3.5
300
3.0
100
3.0
100
75
3.1
30
25
25
25
3.0
2.0
6.3
0.
3.8
08
-75
-75
-75
1.6
14
144
1.6
1.4
1.6
Body
Body
Body mass
23

---

Published as a conference paper at ICLR 2025
Safely jump-slart % of ectroise 0,37 task]
Safety jump-slart % actnosse 0,37 Lesk2
Safely jump pt----ett % of actno 0,17 task3
25
50
1.2
12
1.2
1.0
3.0
-21
-25
-25
0.-
0441
0.6
5/6
1.6
-100
04
1.5 Body
12
-100
04-
04
0.6
-100
Body man
Body mass
Safety jump start actnoise 0.23 twak1
Safety jump-start actnorse 0.23 twsk2
Safety jump- actno 0.23 tasks
3.0
79
T3
14
25
1.0
C
.033
-2
22
04
-100
54
1.2
-100
54
-120
Bady
Dody
0,25 task]
Safety jump actnoiso
Safety
task3
100
3.0
100
75
77
1.2
1.2
25
1.0
3.0
2.0
-25
-25
0.8
-29
-200
04
1/
15
100
0.-
100
Body
0,35 task]
Safety jump actnoiso 0,35
Safety
task3
100
3.0
100
75
75
1.2
1.2
23
1.0
3.0
2.0
-25
-25
0.8
64
-150
54
-100
54
-126
Bady
Dody
Figure 15: Safety staptstarts of the nondeterministic students with the guides trained with random action noise within the three target tasks, where each dot represents different dynamics function.
24

---

Published as a conference paper at ICLR 2025
Sufityy jump-alart advact, 0,06 teaki
Safely jump-atart % of advvect 0,06 task2
Safway jump-atart % of edieact 0,06 Lesk3
25
50
1.2
12-
1.2
1.0
3.0
-21
-25
-25
0.-
044
5/6
1.6
-100
04
1.5 Body
1/2
-100
04-
54
0.6
-100
Body man
Body mass
Safety Semp-startt advact 9.12 beskc
Sefety jump-start % scvvact 0,12 task??
jump-start % of ncvvect 0,12 baskc
3.0
79
T3
14
25
1.0
3.-
.033
-23
-2
04
-100
54
-100
54
-120
Bady
Todyy
Safuty advact teski
Safety advvect 0,17 task2
start of advect 0,17
100
3.0
100
75
75
1.2
1.2
25
1.0
3.0
2.0
-25
-25
0.8
-29
5/
-100
04
12
100
0.4
100
Sefwly jumpushart
beskc
Sefety jump-start
bask?
Safecy jump-stert
1.6
3.6
2.4
12
23
25
3.0
3.0
3.0
688
0:8
244
52
30
-73
-73
-72
04
-120
-100
54
-100
1.0
12
Dody
Dady
25

---

Published as a conference paper at ICLR 2025
Safwtyy jump-atart advact, 0,29 teak!
Safely jump-atart % of advvect 0,29 task2
Safway jump-alart % of ediaect 0,29 Lesk3
25
50
1.2
12-
1.2
1.0
3.0
-21
-25
-25
04-
044
5/6
1.6
12
-100
04
1.5
1/2
1.4
-100
04-
54
0.6
17
-10
Body mas
Body mass
Body mass
Safety jump-start % of advact 0,35 teski
Safely jump-start % of addvact, 0,35 task2
Safecy jamp-stars % of advect 0,35 task3
102
3.6
100
1
14
1.4
30
30
25
0.8
-50
-50
-00
04
04-
24
74
0.6
5/
1.6 Body mays
17
14
1.6
-100
0.6
12
144
1.6
100
0.
166
6-
1-5
17
1
1.6
100
Body maps
Body mass
Figure 16: Safety jump-starts of the nondeterministic students with the guides trained with adver- sarial perturbations within the three target tasks, where each dot represents a different dynamics function.
Safety jump start % of aiphesac_C 0.0 Daskk1
Safeey jump start 7% of aliphssec O.O, tasktkk
Safety Jump start % of elphasac. 0,0, 0,7as33
102
2.8
100
3.4
14
1.4
50
50
25
25
25
DCF
0/
0/
-00
-30
-00
0.8
-22
24
0.6
3/5
1.8
12
1.4
3.0
04
06
0.8
1.3
12
1.4
1.
04
DB
08
1/
32
3.4
1.6
Body miz
Dady mass
Body maxe
26

---

Published as a conference paper at ICLR 2025
Safety jumpslart % of alphasec 0,001 Leak1
Safwty jump-start % of elph 0,003 teak2
Safwcy pes % of alphases 0,001 Lask3
25
50
1.2
12-
1.2
1.0
3.0
-21
-25
-25
04-
044
5/6
1.6 Body man
12
-100
04
1.3 Body mass
1:2
1.4
1100
04
0.6
-200
Body mass
Figure 17: Safety jump-starts of the nondeterministic students with the guides trained with entropy maximization within the three target tasks, where each dot represents different dynamics function.
G.2 TRANSFERRING TO THE DETERMINISTIC STUDENT
0,06 task1
Safely jump-start actnoise 0,06 task2
Safely jump-ste cmmp-tee-t
Lask3
100
3.6
100
75
75
1.2
1.2
25
3.0
0
1.
3.0
-25
-35
0.8
0
0.4
-120
04
-100
0.-
-100
Body
Safety jump-ste strpp--ttttttt
Safety jump-start
Safety jump-
3.6
203
3.0
10C
3.0
100
23
25
1.0
3.0
3.0
CCT
0.5
-32
-75
-73
-75
04
-150
64
-100
64
-100
1,0
12
L4
2.6
1.4
1.6
0.6
Bady
Dady
Safety jump- start actnoise 0,17 task]
Safety jump actnossa 0.17 task2
Safety jump actnoise task3
3.5
300
3.0
100
3.0
100
75
3.1
30
25
25
25
3.0
6-3
0.
3.8
08
-3
-75
-75
1/6
1-4
1.4
17
1.4
1.6
Body mase
Body
Body mass
27

---

Published as a conference paper at ICLR 2025
Safely jump-start % of ectroise 0,25 task]
Safety jump-slart % actnosse 0,23 Losk2
Safely jump pl----ter % of actno 0,23 taak3
25
50
1.2
12
1.2
1.0
3.0
-21
-25
-25
0.8
13
044
1.6
-100
04
1.3 Body
1/2
1.4
-100
04-
04
0.6
-100
Body man
Body mass
Safety jump start actnoise 0.29 taak1
Safety jump-start actnoose 0.29 twsk2
Safety jump- ste--tttttte actno 0,29 task:
3.0
102
3.0
79
T3
144
25
1.0
30-
-2
-2
04
-150
54
1.2
-100
54
-120
Bady
Dody
Eody
Safety jump-start % actnoise 0,35 task]
Safety jump-start actnoise 0,35 task2
Safety jump- stertttttttt actnoise task3
1.5
100
3.0
100
3.4
160
25
1
3.4
30
25
25
25
2.0
0.
0.
2.3
38
0.6
-0
1.6
1/2
1:4
0.6
1/
1.4
1.6
Body mase
Body
Body made
Figure 18: Safety jump-starts of the deterministic students with the guides trained with random action noise within the three target tasks, where each dot represents a different dynamics function.
Safety jump start advact 0,00 teski
Safety jump starrtttt % adveett 0,0G task2
Safeeyy jumo-stert 5% of advecct 0,06 task3
1.5
300
3.0
100
3.0
100
75
1
1.4
3.1
30
25
25
25
6-3
0.
3.8
88
06
-35
-75
-75
1.6
144
1:4
1.6
32
6.
17
1.4
1.6
Body moze
Body
Body mast
28

---

Published as a conference paper at ICLR 2025
Safwty jump-atert % advact, 0,12 teak
Safety jump-atart % of advvect 0,12 task2
Safway jamp-alart % of adiaect 0,12 Lesk3
25
50
1.2
12-
1.2
1.0
3.0
-21
-25
-25
0.-
0441
1.6
-100
04
1.3
1/2
1.4
-100
04-
04
0/6
-100
Body maz
Body mass
Body mass
Safety Semp- adveact 7.17 beskc
Sefety jump-start % scvvact 0,17 taskk?
Swfety jamp-start % of ncvvect 0.17 baskc
3.0
3.0
79
T3
14
25
1.0
3.-
-2
-2
04
-150
54
1.2
-100
54
-120
Eady
Dody
Safuty jump advact teski
Safety addvect 0,23 task2
startttttttt of advect 0,23
100
3.0
100
75
75
1.2
1.2
25
1.0
3.0
2.0
-25
-25
0.8
-29
044
A/
-100
04
1/
12
100
0.
100
Body
Sefwly jumpushart
beskl
Sefety jump-start achoact bask?
Safecy jump-stert
0,29
1.6
107
3.6
1.4
1.4
12
23
25
3.0
3.0
3.0
6
0:8
244
52
30
-73
-33
-72
04
-150
-100
54
-100
1.0
L4
1.4
DUB
1.6
Dody
Dady
29

---

Published as a conference paper at ICLR 2025
Safwtyy jump-atart advact, 0,35 teak
Safely jump-atart % of advvect 0,35 task2
Safway jump-alart % of ediaect 0,35 Lesk3
25
30
1.2
12-
1.2
3.0
-21
-25
-25
34-
0441
5/6
1.6 Body mane
12
-100
04
1.3 Body mass
1:2
1.4
1100
04
04
0.6
-100
Body mass
Figure 19: Safety jump-starts of the deterministic students with the guides trained with adversarial perturbations within the three target tasks, where each dot represents different dynamics function.
Safety jump start % of aiphesac 0,0 0.0.tskk
Safety jump start 74 of aljphasac, O.O, t.0ttak2
Safety Jump start of aiphasac. 0,0,0 0r7ask3
102
2.8
100
1.4
14
1.4
30
50
31
25
35
59
5/
-00
-50
-00
04-
72
-33
144
0.66
2.5
1.0
12
1.4
3.0
04
0.8
1.2
122
1.4
1.6
04
1.6
08
1.0
32
3.4
1.6
Body
Dodyy mass
Body maxe
Safety jump-sttee Hiphesac.0
Safety jump-start aiphesac 0,001
Safery jump-star: aiphaser, 0,000,
3.5
300
3.0
10C
3.0
100
75
1.4
13
12x
13
35
3.0
-25
5.4
-3
0.
-120
-100
04
-100
Body
Figure 20: Safety jump-starts of the deterministic students with the guides trained with entropy maximization within the three target tasks, where each dot represents different dynamics function.
30


Reviews:
[{"review_id": "iZn6oF6PtJ", "reviewer": "Reviewer_YHLY", "summary": "This work seeks to ensure robustly safe transfer learning when the dynamics of a new environment differ from the original environment. The proposed approach (1) strengthens an agent’s robustness within a controlled environment and (2) provides a provable guarantee under some assumptions of safe transfer to new environments. Empirical results show that this method maintains safety after transitioning to a new environment. The authors also propose several metric for safe robust transfer learning, which might be useful.", "questions": "1. How is the theorem 1 derived? More specifically, there is a hidden assumption that there exists a $\\pi$ that is safe for all $P$ with $\\left\\|P-P^{\\diamond}\\right\\| \\leq \\delta$? Solving a robustly safe policy is also a very challenging task, especially for a very large number $\\delta$.\n\n2.  Is $P$ a matrix or a nonlinear transition operator? is the state and action space finite or continuous?\n\n3. In the experimental section, why is the mapping from friction and mass to transition dynamics continuous and bounded? For example, in some cases, the friction changes from static friction to kinetic friction, and the change point is usually not continuous. \n\n4. What exactly are the safety criteria are you considering? in the CMDP problem formulation in line 118, you said the goal is $C(\\pi)\\leq d$, while later you convert it to the constraints on the $Q^c$ function. They are not equivalent and the $Q^c$ constraints are much more restrictive than the $C(\\pi)$ constraints.\n\nOther minor questions related to the notations:\n\n1. What is the superscript ${}^*$ means in line 244?\n\nI will consider raising my score if the question are properly answered.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The topic is interesting and significant, and the solution is overall reasonable, the presentation of empirical results is clear and easy to understand.\n\n2. The results are mostly technically correct and bring both theoretical novelty and practical insights.", "weaknesses": "1. The assumptions are not as mild as claimed by the authors, and there are also many hidden assumptions. For example, the paper requests the existence of a robustly safe policy for under constraints only on $\\|P-P'\\|$.\n\n2. The technical part is confusing. The authors first claim to solve CMDP but later focus on satisfying constraints on every state-action pair $Q^c(s, a)\\leq d$, which is a much more conservative objective. The definition of norm and abstract operator is also sloppy.\n\n3. The robustness heavily relies on the action disturbance. There are many other dynamics-perturbation-based robust MDP algorithm such as [1], the authors should consider additional algorithms.\n\n[1] Robust markov decision processes: Beyond rectangularity\nV Goyal, J Grand-Clement - Mathematics of Operations Research, 2023\n\n4. The humans knowledge is important in the proposed formulation and is not properly discussed. For example, the abstraction needs to be safety-irrelevant, but sometimes all the observations are related to safety, and sometimes one does not know which observation is safety-irrelevant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1XNgXpNjyb", "reviewer": "Reviewer_LbED", "summary": "The paper investigates robust transfer learning of safety-constrained RL problems. The contribution includes: (1) propose a robust transfer learning framework, (2) theoretical analysis on the safety guarantee in transfer across environments with different dynamics, (3) consolidate safe transfer learning metrics and devise methods to train robust agent.", "questions": "- Please refer to the bulleted items listed in the Weaknesses section. I'm particularly curious about how does the algorithm quantify and increase $\\delta$ to achieve acceptable level of safety.  \n\n- Also, in the last paragraph of Section 6.2, the authors mentioned that \"values shown in the tables are the expected costs of one batch of guides\". Is there any challenge in getting the expected costs from multiples batches?  \n\n- Is there any reason why the students are limited to off-policy RL algorithms: SAC and DDPG? For deterministic policy, why is DDPG tested over TD3?  \n\n- In Appendix F (Safety Jump-Start Heatmaps), non-deterministic policy seems to be less robust than deterministic policy. Could you discuss why? This seems counterintuitive to me, especially for a CMDP task.\n\n- I'm more than happy to discuss and please let me know if there's anything I misunderstood or missed out.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written in an easy-to-understand manner and I appreciate the visual illustration in the paper.  \n\n2. The paper did a good background study on robust transfer learning problems and combine methods & metrics from related literature to form their own transfer learning framework.  \n\n3. The paper clearly defines the assumptions, mapping, metrics used in their framework and elucidates under what circumstances is safety guaranteed for transfer across tasks with different dynamics.", "weaknesses": "1. The paper highlighted that what sets itself apart (from prior work) is that it does not assume uncertainty set is known. However, the uncertainty set is assumed to be bounded by $\\epsilon$. I am not too sure if this assumption can be easily verifiable for different set of tasks, especially when $\\epsilon$ is related to $\\delta$ in theorem 1: when the norm-distance is high, the more we need to \"robustify\" to have safety assurance. The bound on the norm-distance $\\epsilon$ does not seem to be easily quantified.  \n\n2. In theorem 1, the paper concludes that $\\delta$ should be increased (i.e. making the policy more robust to dynamics that may be dissimilar to the training environment) to have safety assurance. However, this $\\delta$ parameter does not appear in subsequent chapters. In particular, Chapter 5 (robust training) does not seem to reconnect with this $\\delta$ parameter and the theoretical safety guarantee. The theory and the actual training algorithm seems a little disjoint in this regard.  \n\n3. Expanding on item 2 above, how is the problem formulation in \"formal statement of the problem\" connected to the parameter $\\delta$? I am also not too sure how does the training algorithm weigh the noise / perturbations (in Entropy Maximization, Random Action Noise & Adversarial Perturbations) such that sufficient $\\delta$ can be reached, in order to achieve good safety performance.  \n\n4. The robust training framework depends on the abstraction function $\\sigma$. I am not too sure if this abstraction function can be specified easily in practice? Especially when this function needs to connect the dynamics, reward and cost functions (in source and target environments) in accordance to the formulae listed in Definition 2. It may seem straightforward to define $\\sigma$ in the empirical analysis section of this paper since source/target envs are largely similar (albeit with different physics parameter) but I guess in practice it may not be that easy.  \n\n5. Chapter 5.1 introduces state abstraction function and I guess this requires substantial feature engineering effort? Is this exploration bonus a bit short-sighted since it only computes the norm-distance between the current state and the next state?  \n\n6. Chapter 5.2 needs to be expanded: the paper mentions that a Lagrangian multiplier $\\lambda$ is used to weigh similarity between two policies. I can't figure out how would this enable \"student's policy imitate the guide's whenever the cumulative cost is above the safety threshold\". In addition, how does the algorithm decide the correct $\\lambda$ value to apply during training? These should be further discussed.  \n\n7. The safety gymnasium task analyzed in empirical analysis seems to be only Goal task. Having more tasks may strengthen the paper in this aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2LHVBakJ6H", "reviewer": "Reviewer_Vwu8", "summary": "This paper addresses a critical challenge in safe reinforcement learning: how to transfer an agent trained in a controlled environment to a real-world setting with different dynamics while maintaining safety guarantees. It propose a methodology that makes the transfer process robust against variations in dynamics between source and target environments.", "questions": "- In Section 4.1, you assume the existence of a Q-irrelevant abstraction σ that is continuous and bounded. Could you elaborate on how this abstraction can be constructed in practice for high-dimensional state spaces? A concrete example or algorithm would significantly strengthen the paper's practical applicability.\n\n- The bounded uncertainty set assumption (Assumption 3) seems crucial for your theoretical guarantees. Could you provide insights on how to estimate the bound ε in practice?\n\n- Your empirical evaluation uses N=8 values for discretizing the uncertainty set. Could you justify this choice of granularity? And discuss how discretization errors affect your worst-case guarantees?\n\n- The control switching mechanism π_b(s_t) might lead to chattering. Have you analyzed the frequency of policy switches during transfer?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The theoretical work is sound and well-constructed. Theorem 1, which establishes safety guarantees under transfer, is particularly well-developed and properly scoped. The proofs are thorough without being needlessly complex, though some of the assumptions (like the existence of Q-irrelevant abstractions) might prove limiting in practice. The authors demonstrate that their robustified agents can achieve substantial improvements in worst-case performance, with some configurations reaching complete safety transfer. The empirical validation is thorough and well-designed. The authors test their approach across multiple environments and provide meaningful comparisons between different robustification techniques. The ablation studies effectively isolate the contributions of individual components, though more complex environments could have strengthened the conclusions further. Overall, the paper makes a worthwhile contribution to the field of safe reinforcement learning.", "weaknesses": "The paper exhibits some theoretical limitations in its treatment of safety-preserving abstractions and uncertainty quantification. The foundational assumption of Q-irrelevant abstractions $\\sigma: S_\\odot \\rightarrow S_\\diamond$ being continuous and bounded (Assumption 2) might become problematic for hybrid or discontinuous systems, while the bounded uncertainty set assumption $\\exists\\varepsilon \\in \\mathbb{R}, \\forall P, P' \\in U_\\odot, \\|P - P'\\| \\leq \\varepsilon$ fails to capture fat-tailed uncertainty distributions common in real systems. The robustification approach through entropy maximization $r^\\diamond_t(s_t, a_t) = r^b_t(s_t, a_t) + \\alpha r^H_t(s_t, a_t)$ lacks guarantees about value function Lipschitz constants, potentially compromising stability under state-space perturbations. Moreover, the control switching mechanism $\\pi_b(s_t) = \\begin{cases} (\\pi^\\diamond \\circ \\sigma)(s_t) & \\text{if } c_{t-1} > 0 \\\\ \\pi_\\odot(s_t) & \\text{else} \\end{cases}$ introduces potential chattering issues near constraint boundaries, while the p-tail metric $M_{\\leq p}(M_\\odot, \\pi) = \\int_B M(M^\\odot_P, \\pi) dP$ requires additional measure-theoretic assumptions that remain unjustified.\n\nThe empirical validation methodology demonstrates concerning limitations in both scope and rigor. The discretization of the uncertainty set using only $N=8$ values for mass and friction parameters provides insufficient coverage of potential failure modes, while the safety jump-start metric $J(\\pi^\\diamond, M_\\odot) = \\mathbb{E}[\\frac{c(\\tau) - c(\\tau')}{c(\\tau')} | \\tau \\sim \\rho_1, \\tau' \\sim \\rho'_1]$ masks potentially dangerous temporal patterns through trajectory averaging. The environments tested $(M^\\diamond_1, M^\\diamond_2, M^\\diamond_3)$ share similar underlying physics, failing to challenge the framework with fundamentally different dynamics structures. The paper's treatment of model parameterization through simple mass and friction variations $m_i = (0.5 + \\frac{i-1}{7})m^\\diamond$ and $\\eta_i = (0.5 + \\frac{i-1}{7})\\eta^\\diamond$ inadequately captures the complexity of real-world dynamic variations, particularly in systems with coupled parameters or higher-order effects. Furthermore, the Lagrangian multiplier approach for policy imitation lacks theoretical convergence guarantees when combined with robustification techniques, potentially leading to unstable learning dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9nZuVWiueC", "reviewer": "Reviewer_LcYf", "summary": "This paper addresses a problem where a policy is trained in a controlled environment and then transferred  to the real environment. While safety violation is allowed in a controlled environment, unsafe actions may be catastrophic in the real environment. Unlike the previous work that allow policy transfers as long as the new environment preserves safety-related dynamics, this paper explicitly incorporate the dynamics differences or shifts. The authors propose a method to robustifies an agent in the controlled environment and then provably achieves a safe transfer to new environments. The empirical evaluation shows the effectiveness of their proposed method.", "questions": "- Q1: In Definition 2, what is the definition of $\\omega$? I know that the authors mention that the sum of $\\omega$ is equal to 1, but I think it not uniquely defined. \n- Q2: Is it possible to compare the authors' proposed method with Yang et al. (2023)? As the authors mention, this previous work seems to be the state-of-the-art method if I understand correctly.\n\nSince I think this paper is overall of high quality, I am happy to increase the score from the current one (i.e., 5: borderline reject) when my concerns are addressed during the rebuttal period.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors address a very important and interesting problem. I think such problems are fit to some real-world safety-critical problems.\n- The Related Work section is concise but well-written and easy to follow.\n- The Background section is also easy to follow and well-presented.\n- Assumptions are all reasonable and indeed mild.\n- I think Theorem 1 is good. The proof seems to be correct and easy to follow.\n- Definitions on safety jump-start, $\\Delta$ time to safety, and a safe transfer-learning metric are well-presented despite their complicated notions.\n- The resulting algorithm presented in Section 5 is technically sound and easy to understand.", "weaknesses": "- In Related work section, I think there is a missing citations:\n    - Zhang, Jesse, et al. \"Cautious adaptation for reinforcement learning in safety-critical settings.\" International Conference on Machine Learning. PMLR, 2020.\n- Experimental results seem to include only ablation study. I am wondering why Yang et al. (2023) is not compared with their proposed method. If it has been already included as a baseline method, please clarify in the experiment section.\n- I think the experimental settings have not fully written in the main paper and appendix. Given the source code is not attached, I feel the reproducibility of this paper is low.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)