Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty
Authors: Xu Wan
Chao Yang
Cheng Yang
Jie Song
Mingyang Sun
Venue: NeurIPS 2025 poster ()
Keywords: Robust Safe Reinforcement Learning; Risk-sensitive Decision-making
Primary area: reinforcement_learning
TL;DR: We propose Fuz-RL, a fuzzy logic-based framework that enhances robustness of safe reinforcement learning by measuring and balancing uncertainties in a non-additive way, achieving better safety and performance under uncertainty.

Abstract: Safe Reinforcement Learning (RL) is crucial for achieving high performance while ensuring safety in real-world applications. However, the complex interplay of multiple uncertainty sources in real environments poses significant challenges for interpretable risk assessment and robust decision-making. To address these challenges, we propose Fuz-RL, a fuzzy measure-guided robust framework for safe RL. 
Specifically, our framework develops a novel fuzzy Bellman operator for estimating robust value functions using Choquet integrals. Theoretically, we prove that solving the Fuz-RL problem (in Constrained Markov Decision Process (CMDP) form) is equivalent to solving distributionally robust safe RL problems (in robust CMDP form), effectively reformulating the min-max optimization problem into a tractable CMDP with Choquet-integrated value functions. Empirical analyses on safe-control-gym and safety-gymnasium scenarios demonstrate that Fuz-RL effectively integrates with existing safe RL baselines in a model-free manner, significantly improving both safety and control performance under various types of uncertainties in observation, action, and dynamics.

Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty
Xu Wan
Chao Yang Alibaba DAMO Academy xiuxin, www.lllalaabbaa.. com
Zhejiang University wanxužzju. edu, en
Cheng Yang
Jie Song
Mingyang Sun" Peking University smyēpku. edu. cn
Alibaba DAMO Academy charis panniiiaann@@llllbea ibaba- inc. com
Peking University
j je song@pku edu, cn
Abstract
Safe Reinforcement Learning (RL) is crucial for achieving high performance while ensuring safety real-world applications. However, the complex interplay of multiple uncertainty sources in real environments poses significant challenges for interpretable risk assessment and robust decision-making. To address these challenges, propose Fuz-RL, a fuzzy measure-guides guided robust framework for safe RL. Specifically, our framework develops novel fuzzy Bellman operator for estimating robust value functions using Choquet integrals. Theoretically, prove that solving the Fuz-RL problem (in Constrained Markov Decision Process (CMDP) form) is equivalent solving distributionally robust safe RL problems (in robust CMDP form), effectively reformulating the min-max optimization problem into tractable CMDP with Phoquet-integratee value functions. Empirical analyses on safe-control-gym and -ymia--gymmmiium scenarios demonstrate that Fuz-RL effectively integrates with existing safe RL baselines in model-free manner, significantly improving both safety and control performance under various types of uncertainties in observation, action, and dynamics. The code is available https://github.com cwwm///////////ee waunx/FuzRL..
Introduction
While safe reinforcement learning (RL) has achieved remarkable success safety-crucial decision- making tasks, deploying safe RL in real-world applications remains challenging due to multiple sources of uncertainty [5, 8, 42]. Recent methods using Lyapunov functions and reachability analysis provide theoretical safety guarantees for control tasks [4, 9, 45, 48, 44, 43], but focus primarily idealized, deterministic settings. These approaches struggle with the complex, coupled uncertainties of real-world systems, including sensor noise, actuator delays, and environmental variations.
Existing robust approaches to safe RL face several key limitations for real-world tasks. Traditional min-max techniques [49, 29, 40] focus on worst-case scenarios, resulting in overly conservative policies and computational intractability. While distributionally robust methods attempt to model uncertainty distributions, they typically assume simplified, independent uncertainties through KL- divergence constraints [39] or Gaussian perturbations [41], and treat different perturbations with equal importance. Risk-sensitive approaches using probability measures like conditional Value-at ttte--t-iie -t-k-- (VaR) [38], Wang transform [34], and Entropic VaR [31] enable uncertainty quantification through
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

coherent risk functionals, require careful parameter tuning and struggle to handle multiple noise sources effectively.
However, when multiple uncertainties are correlated and converge on a single system component, the resultant performance degradation often exhibits super-additive behavior. To handle such coupled uncertainties, fizzyy measure theory has shown promise in various decision-makin tasks, from robust motion planning [7, 53, 18] to adaptive control [27, 16], through its ability model non-additive effects and provide clear behavioral interpretations. While these successes demonstrate the potential of fuzzy measures for uncertainty quantification, extending this approach to constrained Markov decision process (CMDP) remains challenging, particularly in balancing performance objectives with safety constraints under uncertainty.
Motivated by this, we propose a novel Fuzzy-guided framework for Safe RL (Fuz-RI.) that unifies un- certainty quantification and enhances current safe RL's robustness through fuzzy theory. Specifically, our main contributions are:
(I) We introduce novel fuzzy Bellman operator that integrates Choquet integrals of fuzzy measure into value function to achieve robust value estimation under potential perturbations.
(2) We provide robustness equivalence for our Fuz-RL framework by demonstrating that solving Fuz-RL problem (a CMDP form) is equivalent distributionally robust safe RL problems (a robust CMDP form).
(3) By seamlessly integrating the Fuz-RI. framework into three safe RL methods, we conduct several robust assessments involving observation, action, and dynamics uncertainty for safe-control-gym tasks and safety-gymnasiug tasks. As expected, Fuz-RL significantly enhances the robustness of safe RL algorithms in multi-source uncertainty scenarios.
2 Related Work
Our work builds upon and connects two main research directions: robust approaches in safe RL and fuzzy-based uncertainty quantification in MDPs. We review relevant work in these areas and highlight the research gaps our work addresses.
Robust Approaches Safe RL. Uncertainties within the CMDP framework manifest in various forms, including state shifts [23], action disturbances [41], and dynamics uncertainty [33]. To address these challenges, distributionally robust optimization has been employed, where policies optimized against worst-case transition kernels within Wasserstein ambiguity set [39, 28]. An alternative direction leverages risk-sensitive measures to enhance robustness under safety constraints. For instance, Conditional Value-at-Ra at-Risk (CVaR) has been integrated into policy optimization to explicitly balance expected return and worst-case performance [47], while coherent distortion risk measures offer formal robustness guarantees in safe RL [34]. Other approaches focus on adversarial robustness or model-based safety. Some works combine robust model predictive control (MPC) with tube-based constraints to ensure recursive feasibility under uncertainty [51]. Gaussian Processes have also been used as safety oracles in model-based RL to probabilistically identify constraints [2]. Adversarially robust methods further address observation perturbations via state-adversarial MDPs and policy regularization [24, 25,52], 52].
Fuzzy logic provides interpretable framework for quantifying and managing uncertainty in complex systems. Zadeh's fuzzy sets theory [50] laid the foundation for uncertainty measures. Building on this, possibility theory [11] emerged as a significant fuzzy approach to uncertainty quantification, offering an alternative to probabilistic methods. Then, [26] introduced credibility theory, which combines fuzzy and probability measures to create self-dual measure. For RL community, fuzzy Q-learning [13] and possibilistic MDPs [36] incorporate fuzzy logic into MDPs. Furthermore, [17]. [21] and [19] developed various fuzzy RL approaches that provide enhanced interpretability and effectiveness compared to deep neural sowwork-based RL methods. [15] introduces m measure, which combined the possibility measure and necessity measure to balance optimism and pessimism in decision-making systems, which has shown promise in Shance-constrainee programming. Recent advancements have further expanded the application of fuzzy logic in uncertainty modeling. For instance, [20] developed fuzzy adaptive sliding mode
Fuzzy Measures in MDPs.

---

control method for robotic systems with uncertainties, [37] conducted a comprehensive review of uncertainty quantification applications for healthcare.
However, incorporating fuzzy logic for robustness enhancement in safe RI. remains unexplored. Inspired by the fuzzy-guided ma fuzzy measure[15], aim to achieve robust risk-aversion and reward-pursuin value estimation through fuzzy logic for safe RL.
Preliminaries
3.1 Robust CMDP
We consider formulating the safe RI problem as an infinite-horizon CMDP [3], which is defined by the tuple (S, A. p. r, 7, do), where S is the finite state space and A is the action space. p x AA x S - [0, 1] is the transition model, e S x A x S + R are the bounded reward function and cost function defining the objective and constraint, respectively, 7 [0. 1) is the discount factor, and do : S + [0, 1] is the initial state distribution. A policy π S - ((A) maps states to distributions over actions.
To address system uncertainty, we introduce robust formulation of CMDP. Following the concept of (s, reettanggl uncertainty sets [32], define the uncertainty set P as:
P = II Pa. P. S A(S)
8,0
where Pe represents the set of all possible transition probabilities over S for a given state-action pair (s, a). For Vs E S,a E A. we define:
P {p(-1s, a): d(p(-Is, Po(-|s, a)) S c}
where Pu is the nominal transition model, -) is distance metric, and e 0 defines the uncertainty radius.
The objective of the robust CMDP is to find a policy π that solves the following constrained optimization problem:
max min E (T.p)) pEP
Σ Y (r az)
s.t. max E (T.p))
Σ fc(ss)
SB
(3)
PEP
1=0
where 7 (T, p) denotes trajectories sampled according to so ~ do, at ~ x(110) and st+1 ~ p(-[st, dz), and B > is the safety budget constraint.
For computational tractability, we partition the uncertainty set Pa into K distinct levels:
K Dad
PP
P
P (p(1)s,)) - dlp(()s,)) S {ex}
where 0- = Eo <1 < <EK = e defines a sequence of increasing uncertainty thresholds.
3.2 Fuzzy Measures Fundamentals
Traditional probability measures treat uncertainties in purely additive manner, assuming independent effects from different uncertainties. However, in real control systems, as the distance from nominal dynamies increases, the compound effects of uncertainties often exhibit super-additive behavior. For example, when considering two uncertainty sources A and B, their joint impact on system performance may be greater than the sum of their individual effects:
m(A U B) > m(A) + m(B)
(5)
Moreover, as the system deviates further from the nominal model, the impact performance and safety constraints typically grows non-linearly.
To capture such non-additive effects, we first introduce the concept of fuzzy measure, which provides an interpretable way to assess the impacts of uncertainty by assigning non-additive weights to combinations of uncertainty levels. The formal definition is as follows:

---

Definition 3.1 (Fuzzy Measure [30], For each state-action pair (s. let I (P%,,, Ps, s.,....,, p. denote the collection uncertainty sets. A fuzzy measure function m: + [0, 1]. satisfying:
(1) (0) 0, m({.) (boundary conditions),
(2) Af B C 2g then m(.A) S mnB) (monotonicity).
Measuring uncertainty impacts for all possible subset combinations in PP....P is compu- lationally intractable, requires exponential number of samples. To address this computational challenge while preserving the ability to model super-additive effects, we adopt the X-fuzzy measure, which offers efficient parameterization of subset relationships:
Definition 3.2 (A-Fuzzy Measure [10]). A X-fuzzy measure satisfies, for all disjoint subsets A, B Ig with A B 0:
g
m(Au U B) = m(A) + m(B) + Am(AA m(B3,
(6)
where > -1 determines the degree interaction. When E (-1,0), the measure exhibits sub- additive behavior; when À E (0, xe), models super-additive effects among different uncertainties. When À 0, the X-fuzzy measure reduces to normalized additive measure, i.e., a probability measure.
The connection between X-fuzzy measures and robust optimization is established through the Choquet integral's pessimistic characterization:
Lemma 3.3 (Choquet Integral Representation [12]) For any bounded measurable function P + R and convex X-fuzzy measure m on I with X > 0:
(C)
f dm
min Ep(f], Pecore(m)
P3
where core(m) (P: P(A) 2 m(A), YA I&) is the set of probability measures P. dominating m.
Fuzzy Measure-based Robust Safe RL Framework
4.1 Theoretical Foundation of Fuz-RL
In this section, we connect fuzzy measure with robust CMDP by introducing the Fuzzy Bellman
operator.
Fuzzy Bellman Operator. Leveraging Lemma 3.3, we define the fuzzy Bellman operator that encodes worst-case scenarios through the Choquet integral in Definition 4.1:
Definition 4.1 (Fuzzy Bellman Operator). Let B(S) denote the space of bounded measurable value functions on S. The fuzzy Bellman operator FF B(S) B(S) defined
((V) = Eawn ((s,,) + (C) / Ex' (V dm(p)]. dmm))
where m(-) is the convex fuzzy measure.
Furthermore, demonstrate that the fuzzy Bellman operator maintains fundamental properties of the standard Bellman operator (y-contraction and convergence) when integrated with value functions as detailed in Theorem 4.2 and Theorem 4.3. Therefore, the fuzzy Bellman operator be seamlessly integrated into value functions, enabling the establishment robust value estimation with theoretical guarantees.
Theorem 4.2 (1-contraction of Fuzzy Bellman Operator). For any Vi, V2 B(S),
[[F(V)) - F(V))llo S >I|Vi - V2lloo.
Theorem 4.3 (Convergence of Fuzzy Bellman Operator). Let V E B(S) be initial value function and Va+1= =(V"). Then 8 converges to a unique fixed point V satisfying V =(V*) with geometric rate \| V* <>"1|V" Vo

---

Robust Equivalence. Applying Lemma 3.3 shows that the Choquet integral automatically encodes a robust perspective via the fuzzy measure m. Consequently, we can prove the following Theorem 4.4:
Theorem 4.4 (Equivalent Theorem). Let m be a convex X-fuzzy measure on Ig defined by Definition 3.2 such that: (1) core(m) S P for all (s,a), (2) arg minpeps Bg (V()) S core(m) for all (s, a). (3) arg pax (Vc())) S core(m) for all (8.a). Define the dual fuzzy measure m' m'A == - m(Zg A) all C.
Then the fuzzy robust safe RL problem (CMDP (oMDPform):
max J (T) s.t. (T)
(7)
where
(-)
(TN((TP)
xr(s;, at)
dm(p)
(8)
= (C) (TT((TT)))
Aolse ar)
dm'(p)
(9)
is equivalent to the distributionally robust safe RI. problem (robust CMDP form).
The detailed proofs of Theorem 4.2, Theorem 4.3 and Theorem 4.4 are provided in Appendix A.
4.2 Practical Implementation of Fuz-RL
Having established the fuzzy Bellman operator and its theoretical properties, we now describe how to implement Fuz-RL in practice. The pseudo-code of Fuz-RL detailed in Appendix B Algorithm
Estimation of Fuzzy Measures. To operationalize the fuzzy Bellman operator, we require an efficient method for estimating fuzzy measures m(.) that capture uncertainty impacts across different perturbation subsets. We adopt neural network-based approach that learns fuzzy measure densities directly from state representations.
For each state s, we employ a fuzzy network á parameterized as a multi-layer perceptron (MLP) with two hidden layers. The network takes the state vector as input and outputs fuzzy density parameters {gx 1K
(MLP
(10)
where (.) denotes the softmax activation function applied to the network output. To maintain numerical stability, we apply clamping constrain the fuzzy values within [c, 1 c] where 10-4, preventing degenerate solutions while satisfying the mathematical constraints of X-fuzzy measures.
Given the learned densities g (9),...,gk) the interaction parameter À is determined by solving the characteristic equation:
I[ [ (g++)))==14
(11)
using hybrid bisection-Newton method with gradient detachment to ensure numerical stability. Once À is obtained, the fuzzy measure for any index subset A (I, K} can be computed via the A-rule:
m(A) [1 + Aga) 1
(12)
X
To simulate the impact of uncertain system state transitions across different uncertainty levels, we employ a stratified perturbation sampling strategy. For each uncertainty level l E (1...., . K), we apply independent isotropic Gaussian perturbations:
8: Ek Dj. nA NN I),
(13)
where é represents the perturbation scale for uncertainty level k, typically set as 6 = chase to create hierarchy of perturbation intensities. For each uncertainty level, we generate M independent samples and compute ((&g) as the average of the value estimates across these samples to reduce estimation variance.
5

---

Estimation of Choquet Integrals. To approximate the Choquet integrals used in the fuzzy Bellman operator, we leverage the globally learned fuzzy measures. For the standard Choquet integral applied to reward value aggregation, we sort the perturbed value estimates in descending order: (()( 2 ()) 2 ... 2 (K))), where (i) denotes the index after sorting. The corresponding fuzzy measures are computed m, m()( (i + 1), .)..... (K))), representing the capacity of the tail sets. Following the discrete Choquet integral formulation, the robust reward value is approximated as:
K / E (V(')] dm(p) = Vv( - -m:++1)) pa s'rup
(c)
(14)
where mK+1 0 by convention. For the dual Choquet integral used in cost value aggregation with pessimistic estimation, we sort the cost values in ascending order: Vo(5(1)) z Vo(5(2)) S (o)) and compute:
(C) / E [V(s')] dp) Σ (m: - pp d'mp
(15)
where m, m .,.... represents the capacity of the head sets, computed using the same global fuzzy measure densities but with different subset selection to achieve pessimistic estimation for costs. The choice of descending order sorting for rewards and ascending order sorting for costs is theoretically grounded in the dual relationship between and its dual measure m..
Value Network Updates. The value networks are updated through temporal difference learning with Choquet-integrated targets:
[(0,) E, (s) (re + + Vo, (st+1)) 1) 2],
(16)
C(O.) = E, [(Vo. (s:) - (a + Vo. (s(+1)))
where Vo and Vo, denote the Choquet-integrated value estimates computed using Equations (14) and (15), respectively.
Fuzzy Network Updates. The fuzzy network parameters 6 are updated at lower frequency than the value networks ensure stable learning of the uncertainty structure. Given the fuzzy density parameters s(*++1) predicted from next states, the network minimizes the discrepancy between Choquet-integrated predictions and Monte Carlo targets:
L( = E,, (r+ + V($++1) - R.)² (ce tr Vo($++1)
(17)
where Re, Ct are Monte Carlo returns of reward r; and cost Ct-
Policy Network Updates. Given that fuzzy value estimation implicitly addresses robustness through Choquet integration over multiple perturbations, the robust CMDP problem is solved using a primal- dual approach:
max min E.-a- A,
V+(s) Ax (Vo(s) -)]
(18)
where V, and V. represent the robust value estimates obtained through Choquet integration. The policy parameters are optimized to maximize the Lagrangian objective, while the Lagrange multiplier is adjusted to enforce the safety constraint, with specific update rules determined by the underlying safe RL algorithm framework.
Experiments
5.1 Experiments Setup
To fully evaluate the robustness of Fuz-RL in multi-source uncertainties, we conduct experi- ments on four Safe-Cootrol-Cym [6] tasks: Cartpole-Stab, Cartpole-Track, Quadrotor-Stab. and Maattotor-Trccke as well as four safety-critical control tasks with larger state-action

---

spaces from Gymnasium [22]: Safety-PointGoall1rg Sarety-PPonntuttonn--000 Safety-Pointtirceel--vo and Safety- Potty--PontPush---00
Uncertainty Setting. During the test phase, we leverage different perturbations provided by the Cafe-Control-Gyym consider the following settings observation, action, and dynamics:
(I) Observation uncertainty. We introduce white noise following a normal distribution E -V 1) as observation perturbation, where e is an adjustable parameter used to set different perturbation intensities. During testing, we vary e from - in increments 0.1 for Safe-Control-Gynn tasks.
(2) Action uncertainty. We simulate action uncertainty through an impulse noise disturbance model. Specifically, the perturbed action ā is formulated a. aj d,, where de is defined as:
EM de = EM (t- tunn D)
[saan stur + D]
start + D
(19)
0
otherwise
where E [-0.1, is the magnitude coefficient, M = 10 is the amplification factor, tstart = 20 is the start step. D 80 the duration, and y = 0.9 is the decay rate.
(3) Dynamics uncertainty. We apply white noise following a normal distribution & e (0 1) to dynamics parameters-such pole length and quadrotor mass-where the variation of follows the same scheme as that used for observation perturbation.
(4) Multi- source uncertainty. To analyze the non-additive nature of uncertainty perturbations, simultaneously apply all three uncertainty settings a coupled configuration during both training and testing. For Safety-Gymnasinn training with & = 0.5. For Fuz-series algorithms, we uniformly adopt the training configuration with Ebase 0.1, K 10, and M 5.
tasks, apply relatively small environmental perturbations during
Since the Lafety-Gyymastt observation uncertainty with [0, 0.5] an example for testing.
benchmarks do not provide built-in uncertainty interfaces, we take
Baselines. We adopt three safe RI. algorithms the baseline, including the PPO-Lagrangian (PPOL.)
[35]. Conservative Update Policy (CUP) [46]. and CVaR- LAAR--Pxximal-PPPP Pooinaal--llly-O-PP Policc-Opttmizationn (CPPO) [47]. After integrating the proposed fuzzy-guided framework,
obtain the corresponding Fuz-PPOL, Fuz- CUP, and Fuz-CPPO algorithms. Besides, the current state-of-the-are robust safe RL, Risk-Averse Model Uncertainty (RAMU) [34], has also been migrated our test tasks. All codes of Fuz-RL implemented based on the SpinningUp [1].
goal
but
sircle
push
40
10
10
0
-5
50C
1000
0
500
1000
500
1000
0
500
1000
300
300
300
500
200
200
200
100
100
50C Spochs
1000
500
1000
500 Rрochs
1000
0
500 Spochs
1000
Spochs
PPO-Lag
Fuz-2PO-Lag
Figure 1: Training Dynamics of PPOLag and Fuz-PPOLag under multi-source uncertainty on Sifety-Gymmasium tasks. The perturbation intensity during training is set E 0.5.

---

2041
But-toc
Cirole
Pa8hh
100
30
0.0
-2.3
0.30
3.6
0.275
0.20
0.350
2.06
0.15
3.0
0.000
Spoilon K6)
spolloc
(6)
spsilon (C)
FPO 1ag
Lag
Figure 2: Test Comparison PPOLag and Fuz-PPOLag under multi-source uncertainty setting over 5 Episodes and 5 seeds on aafety-Gymmasium tasks. The cost limit is set to 0.1.
Table 1: Detailed evaluation of Safe RL, Fuz-RL, and RAMU on Safe-Control-Gym tasks with observation, action, dynamics uncertainty. Each value is reported mean + standard deviation for 10 episodes and 10 seeds. We shadow the highest AvgRet and lowest AvgRisk for each task.
Tisks
Methods
Observation Uncertainty
Action Uncertainty
Dynamies Uncertainty
AvgRet
AvgRisk -
AvgRet
AvgRisk
AvgRet 1
ArgRisk
PPOL.
45 + 28
0.40 + 0.14
99 19
0.27 + 0.14
87
0.27 + 0.09
0.30
CPPO
41 19
0.47 : 0.10
76 19
0,36
71.144
0.31 a 0.11
Caripule Stab
RAMU
35
0.00
86
0.20 0.01
PPOL
47 25 1
0.34 0,11 0.06
102 17 1
0.24 0.12 0.00
9 13 1
0.22 0.08 0.05
Fuz-CUP
40
0.0.3
131
0.05
Fuz-CPPO
59 + 25
0.32 + 0.10 0.15
98 2
0.26 0.13 0,10
82 1:5
0.28 0.09 0.03
PPOL
70 16
0.35 + 0.14
95 12
0,18
91
0.21 + 0-
CPPO
87 + 17
0.42 0.11
113 16
0.26
106 12
0.32 0.09
Carppoee Track
RAMU
67
0.10
79
1iz-PPO1.
20 21
0.38 0,16 0.03
120 14 1:25
0.18 + 0.99 0.000
112 14 1:21
0,22 + 0.09 t 0.01
1.0.02
Puz-CPPO
93 10 1
0.31 + 0.10 0.0.1
107 1.6
0.21 + 0.09 1.0.5
107 12 1
0.23 0.08 0.09
PPOL
164 + 18
0.13 = 0.06
58 5:555
0,52 0.19
142 34
0.28 + 0.11
117
0.14+
CPPO
131 14
0.09 = 0.02
54 $4.550
0,36 0.11
117 1: 36
0.17 1
Quadrotor
0.06 0.04
39
Stab
Faz- PPOL
161 + 22
0.07 0.05 0.06
67 5:8:58
0.43 + 0.19 |.00
156 + 28
0.13 0.09 0.15
L.0.7
Fuz-CPPO
156
0.07 + 0.03 0.02
87 2:
0.33 + 0.08
130 0 13
0.09 + 0.12 L 0.08
PPOL
218 1
0-48 0.04
104 79
0,81 1 0.12
203
0,58 0.12
0.12
CPPO
152 16
0.77 + 0.04
76 1:60
0.72 a 0.11
124 a 33
0.73 0.08
Qundrotor Track
+:50
Fuz- PuZ-PPOL
2
0.28 0.05 0.20
99 1 67
0.64 1
0.17
194 =
0,38 0.15 0.15 0.30
Fuz-CUP
0.0.4
Fuz- PP-CPPO
168 2 14 1
0.47 + 0.05 0.30
79 13
0.67 + 0.09
L. 0.0.5
151 151:222
0.59 0.11.
5.2 Robustness Assessment in Safe Control Tasks
For Safe- Cine-Cootro--ymm tasks, we trained the three safe RL baseline algorithms along with their corresponding Fuz-RL and RAMU variants under the same configuration. The specific parameter settings and more detailed results are presented in Appendix C.2. For Safety-Gymnasiun tasks, we use PPOLag lag and Fuz-PPOLag as representative examples for experimental evaluation, with training and testing dynamics shown in Figure 1 and Figure 2, respectively.

---

Level 1
Level_
---
Level_
Level
- Level. 25
162
Dynamic Uncertainty
162
Observation Uncertaintyy
162
Uncerteinty
1.25
1.25
1.2
AngRet
1.00
0.75
0.8
0.50
0.50
0.5
0.4
Avgali
0.4
0.5
0.3
0.2
0.2
-0.05
0.00
0.10
-0.05
0.10
-0.05
0.00
Figure 3: Ablation study of the uncertainty level K.
We evaluate the models from two features "AvgRer" and "AvgRisk", which represent the average episodic return and the proportion of constraint violations, respectively, for each task over 10 episodes across 10 seeds.
Comparison between Safe RL and Fuz-RL. As depicted in Table (1) and Appendix C.3, Fuz-RL demonstrates superior safety in 94.9% cases and robust performance in 88.9% % tasks across various uncertainty settings. Taking Fuz-CUP an example, it achieves 61.4% higher AvgRet and 16.7% lower AvgRisk than CUP in CartPole-Stab task. Moreover, Fuz-RL shows better uncertainty resistance with slower performance degradation than Safe RL. The variance reduction in AvgRer is 20.7%, 9.9%, and 8.6%, while in AvgRisk is 13.2%, 7.1 %6 and 22.6% respectively.
Comparison between Fuz-RL and RAMU. In the 36 Fuz-RL-based experiments listed in Table (1), Fuz-RI. surpasses RAMU in achieving higher AvgRer in 83.3% % of the tasks. Furthermore, Fuz- RL exhibits lower AvgRisk compared to RAMU in 52.8% of them. It is important to highlight that the lower average episodic risk of RAMU is achieved by compromising average episodic rewards, especially in cases of actions affected by impulse disturbances, as shown in the "Action Uncertainty" section of Table (1). Fuz-RL consistently outperforms RAMU in all "Action Uncertainty" tasks.
Ablation Studies of Fuz-RL. We conduct ablation studies to examine how different uncertainty levels K affect Fuz-RL's performance. As illustrated in Figure setting uncertainty level (K 1) proves insufficient and leads to high episodic risk, while excessive levels (K 25) complicate training and reduce rewards. The optimal performance emerges at intermediate levels (K 5 to K = 15). where agents achieve higher rewards while maintaining lower and more stable risk across all three uncertainty types.
Conclusion and Future Work
In this paper, we propose Fuz-RL, a novel robustness enhancement framework that seamlessly integrates fuzzy logic into safe reinforcement learning. We develop novel fuzzy Bellman operator incorporating Choquet integrals, enabling robust decision-making without solving computationally expensive min-max optimization problems. Theoretically, we establish the equivalence between our fuzzy robust safe RL formulation and distributionally robust safe RL. Extensive experiments on the safe-control-gyy -ymm and eafety-gymnasiur benchmarks demonstrate that Fuz-RL significantly outperforms state-of-the-ant safe and robust RI algorithms across various uncertainty types, achieving superior performance both reward optimization and safety constraint satisfaction under diverse perturbation scenarios.

---

While Fuz-RL demonstrates promising results, it faces limited scalability in high- sige--imensional state will focus on developing more efficient uncertainty modeling techniques and
spaces. Future work extending the framework to handle non-stationary uncertainty distributions through adaptive learning mechanisms.
Acknowledgements
This work was supported in part by the Smart Grid-National Science and Technology Major Project (2025/D0803600, 20257ID0803044), National Natural Science Foundation of China under Grants 72571007, and the Natural Science Foundation of Zhejiang Province under Grant LZ23F030009.
References
[1] Joshua Achiam. Spinning up in deep reinforcement learning.22088) URL. htps://gitthb. com/epenaisspinnnguup. 2018.
[2] Filippo Airaldi, Bart De Schutter, and Azita Dabiri. Learning safety in model-based rein- forcement learning using mpe and gaussian processes. IFAC-PapersOnLine, 56(2):5759-5764, 2023.
[3] Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
[4] Aaron D Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8):38113386 2016.
[5] Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy opti- mization via bayesian world models. arXiv preprint arXiv:2201.0980) 2022.
[6] Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe learning in robotics: From learning- based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 2021.
[7] Hyo-Nam Cho, Hyun-Ho Choi, and Yoon-Bae Kim. A risk assessment methodology for incorporating uncertainties using fuzzy concepts. Reliability Engineering & System Safety, 78(2):173-183, 2002.
[8] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1511, 2018.
[9] Yinlam Chow, Ofir Nachum, Edgar Duenez- Guzman, and Mohammad Ghavamzadeh. A bysed approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018.
[10] Dieter Denneberg. Non-additive measure and integral, volume 27. Springer Science & Business Media, 1994.
[11] Didier Dubois and Henri Prade. Possibility theory: approach computerized processing of uncertainty. Plenum Press, New York, 1988.
[12] Itzhak Gilboa and David Schmeidler. Additive representations of non-additive measures and the choquet integral. Annals of Operations Research, 52:43-65, 1994.
[13] Pierre Yves Glorennec and Lionel Jouffe. Fuzzy q-learning. In Proceedings of 6th international fuzzy systems conference, volume pages 659-662. IEEE, 1997.
[14] Michel Grabisch et al. Set functions, games and capacities decision making, volume 46. Springer, 2016.
[15] Peng Guo and Yue Wang. Fuzzy chance-constrained programming with linear combination of possibility measure and necessity measure. Applied Mathematical Modelling, 49:562-578. 2018.
10

---

[16] Yusuf Güven, Ata Köklü, and Tufan Kumbasar. Exploring zadeh's general type-2 fuzzy logic systems for uncertainty quantification. IEEE Transactions on Fuzzy Systems, 2024.
[17] Daniel Hein, Alexander Hentschel, Thomas Runkler, and Steffen Udluft. Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies. Engineering Applications of Artificial Intelligence, 65:87-98, 2017.
[18] Abdelfetah Hentout, Abderraouf Maoudj, and Mustapha Aouache. A review of the literature on fuzzy-logic approaches for collision-free path planning of manipulator robots. Artificial Intelligence Review, (000)33399-44444 2023.
[19] Jordan Hostetter, Mohamed Abdelshiheed, Tiffany Barnes, and Min Chi. self-organizing neuro-fuzzy q-network: Systematic design with offline hybrid learning. In Proceedings of the 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2023.
[20] Changchun Hua, Shaocheng Tong, Yongming Li, and Jun Cheng. Fuzzy adaptive sliding mode control for robotic systems with uncertainties. IEEE Transactions Fuzzy Systems, 3 1166):144-1185) 2023.
[21] Jianyu Huang, Plamen P Angelov, and Chengbin Yin. Interpretable policies for reinforcement learning by empirical fuzzy sets. Engineering Applications of Artificial Intelligence, 91:103559, 2020.
[22] Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[23] Ezgi Korkmaz. Deep reinforcement learning policies learn shared adversarial features across mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7229-7238, 2022.
[24] Zeyang Li, Chuxiong Hu, Yunan Wang, Yujie Yang, and Shengbo Eben Li. Safe reinforcement learning with dual robustness. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[25] Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial train- ing without attacking: Worst-case-awe aware robust reinforcement learning. Advances in neural information processing systems, 35:22547-22566 2022.
[26] Yong-Jin Liu and Baoding Liu. Fuzzy Radon and Choquet integrals. Fuzzy Sets and Systems, 2002.
[27] Zhi Liu, Fang Wang, Yun Zhang, and CL Philip Chen. Fuzzy adaptive quantized control for class of stochastic nonlinear uncertain systems. IEEE transactions on cybernetics, 46(2):524 534, 2015.
[28] Zijian Liu, Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, and Zhengyuan Zhou. Distributionally robust g-learning. In International Conference on Machine Learning, pages 13623-13643. PMLR, 2022.
[29] Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-
359, 2005.
[30] Toshiaki Murofushi, Michio Sugeno, et al. Fuzzy measures and fuzzy integrals. Fuzzy measures and integrals: theory and applications, 2000:3-41, 2000.
[31] Xinyi Ni and Lifeng Lai. Risk-sensitive reinforcement learning via entropic-var optimization. In 2022 56th Asilomar Conference on Signals, Systems, and Computers, pages 953-959. IEEE, 2022.
[32] Arna Nilim and Laurent El Ghaoui. Robust control markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.

---

[33] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826. PMLR, 2017.
[34] James Queeney and Mouhacine Benosman. Risk-averse model uncertainty for distributionally robust safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[35] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration deep rein- forcement learning. arXiv preprint arXiv:1910.01700 7(1):2, 2019.
[36] Régis Sabbadin. Possibilistic markov decision processes. Engineering Applications of Artificial Intelligence, 14(3):287-300, 2001.
[37] Silvia Seoni, Vicnesh Jahmunah, Massimo Salvi, Prabal Datta Barua, Filippo Molinari, and Rajendra Acharya. Application of uncertainty quantification to artificial intelligence in health- care: A review of last decade (2013-2023) Artificial Intelligence in Medicine, 140: 140:10 2023.
[38] Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distributional reinforcement learning. In Learning for Dynamics and Control, pages 958-968. PMLR, 2020.
[39] Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint
2019.
[40] Yoki Tanabe, Takayuki Yamaguchi. and Yasuhiro Shirai. Max-min reinforcement learning for robust policies. IEEE Access, 10: 105300000000380, 2022.
[41] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215 6224. PMLR, 2019.
[42] Akifumi Wachi, Wataru Hashimoto, Xun Shen, and Kazumune Hashimoto. Safe exploration in reinforcement learning: A generalized formulation and algorithms. Advances in Neural Information Processing Systems, 36, 2024.
[43] Xu Wan and Mingyang Sun. Adapsafe2: Prior-free safe-certified reinforcement learning for multi-area frequency control. IEEE Transactions on Power Systems, 2024.
[44] Xu Wan, Mingyang Sun, Boli Chen, Zhongda Chu, and Fei Teng. Adapsafe: adaptive and safe-certified deep reinforeement learning based frequency control for carbon-neutral power systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5294 5302,,500,111 2023.
[45] Yixuan Wang. Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning, pages 36593-36604, PMLR, 2023.
[46] Long Yang, Jiaming Ji, Juntao Dai, Yu Zhang, Pengfei Li, and Gang Pan. Cup: A conservative update policy algorithm for safe reinforcement learning. arXiv preprint arXiv:2202.0756. 2022.
[47] Chengyang Ying, Xinning Zhou, Dong Yan, and Jun Zhu. Towards safe reinforeement learning via constraining conditional value at risk. In ICML. 2021 Workshop on Adversarial Machine Learning, 2021.
[48] Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference Machine Learning, pages 25655. PMLR, 2022.
[49] Jiong Yu and Athina Petropulu. On max-min fair congestion control. IEEE Communications Letters, 16911):16961166 1698, 1996.
12

---

[50] Lotfi A Zadeh. Fuzzy sets. Information and Control, 8(3):338-33 -373, 1965.
[51] Mario Zanon and Sébastien Gros. Safe reinforcement learning using robust mpc. IEEE Trunsactions on Automatic Control, 66(8):3638-3652, 2020.
[52] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations state observations. Advances in neural information processing systems, 33:21024-21037, 2020. 2020.
[53] Tao Zhao, Haodong Li, and Songyi Dian. Multi-robot path planning based on improved artificial potential field and fuzzy inference system. Journal of Intelligent & Fuzzy Systems, 3965)77621-7637, 2020.
13

---

A
Appendix Theorems and Proofs
Theorem A.I (7-contraction of Fuzzy Bellman Operator). For any V1, V/ E B(S),
[F(() - =(V2)llo S >>lVV - V2lloo
Proof. For any two value functions V1 and V2, and any state s:
(s) =(V2)(s))
7(0) / Es'Np/Vi(5))
Eawa
Px
(C) / Egg Vi(s) (2( (dm(()))))))))))
BBaas
P3
1 11 -1-v4 (V) J dm(p)
Bann
(c0) L dm((p)
=>I[V] VelloBBann
> -V2llo
Here, the fact that (C) Spg dm(p) for all and a, m is normalized fuzzy measure. Taking the supremum over all states yields the result.
Theorem A.2 (Convergence of Fuzzy Bellman Operator). Let V B(S) be an initial value function and Vn+1 = F(V"). Then 4 converges to a unique fixed point V satisfying V FFV*) with geometric rate [V" Vo >"1|| V*lloo
Proof. By Theorem 4.2, F is a contraction mapping. The Banach Fixed Point Theorem guarantees the existence of unique fixed point V such that V F(V*). Moreover, for any initial V the sequence {V"
os = defined by ya+1 F(V") converges to V*:
I 1m S >/ Vo 0 Oas
00
This convergence follows directly from the contraction property:
||
V/ [F((") - (V*))
< >/l" Vo
S / - V*
< V - Vo + V -V'l
As + oo, both terms approach zero due to 7 < 1, proving the convergence.
Lemma A.3 (Core Duality of Convex Fuzzy Measures). Let m be a convex fuzzy measure on
I (Pa ( ...... Pa P.k. } with dual measure defined by:
m' (A) == 1-11-my AA) A),
VACI.
Then the cores satisfy core(m'' (m)) and for any bounded measurable function { ::P. - R:
(C) / f dm'
Ep Pecore(m)
14

---

Proof. Part 1: Core Equivalence. For any convex fuzzy measure m on I" its dual m' is concave.
By the duality theorem for balanced fuzzy measures [14], for any probability measure P on p.:
P E core(m)
= P P) 2 m(A) A.
PEA
P(
U P)21-m(Z A), VACI.
PETPVA
P( U P) VA
PEA
P E core(m').
Part 2: Maximum Representation. For the concave measure m', the Choquet integral attains its maximum over the core:
(C) f f dm'= =
max PEcore(m")
[p
Bp[f],
Pecore(m)
where the last equality follows from coro(m') = core(m).
Theorem A.4 (Equivalence Theorem). Given a robust CMDP:
max min E (x.Pp) x PEP
7 Ar(8, ae)
S.f.
maxE,
(5.
2 7c((s, a+)
$ B,
PEP
let m a convex X-fuzzy measure on Ig such that:
1. core(m) C Ps for all (8, a).
2. arg minpep; Ep (s') C core(m)) for all (s,
3. arg maxpeP: E. E [V((8)) C core(m) for all (s, a).
Define the dual fuzzy measure m' (A) = - m(I AA) for all A C Ig
Then the Fuz-RL RL problem:
max J (T( s.t. J=(T) s B,
where
(x) (C) ETM(TT))
7 r at)
dm(()
(20)
- (C) (AT
Σ xo(s0, ax)
dm'(p)
(21)
equivalent to the original robust CMDP.
Proof. Step 1: Core Inclusion and Extremal Coverage. By Equation 10 with softmax activation
and clamping to [e, - where 5 = 10-1, the X-fuzzy measure satisfies m((Pe = 9b E (0, for each uncertainty level & E [1...., K} via Equation 12.
Since each uncertainty level P covers ee-neighborhood of the nominal dynamics (Equation 4), and the softmax ensures all levels receive positive weights, extremal perturbations are guaranteed to
belong to core(m). This establishes conditions (2) and (3), ensuring core(m) C Pa and covering extremal points of Po.
15

---

Step 2: Duality of Fuzzy Measures. For the convex fuzzy measure m, its dual m' is concave. By Choquet duality [14]:
(c) f p
B, (c) / sdm't @Ecore(mm)
inf (Ecore(m)
sup E [.].
Step 3: Reward Objective Equivalence. For the reward function, by Lemma 3.3:
(T)= /
E (x
rrr(s;, at)
dm(p)
(a)
min
By (T.()
@ccore(1)) (m)
min E, (x.pp) pEP
Σ r'r(se, at)
where (a) uses the Choquet integral representation for convex measure (Lemma 3.3),
and (b) holds because condition (2) ensures that core(m) contains the extremal point
arg pin ( Et= 7'r(st, 2" dz)]-
Step Cost Constraint Equivalence. For the cost function, by Lemma A.3:
(=) (C) / (TA(T
xc(ss. ar)
dm'(p)
(c) max (Ecore(m)
(xp(s
7 Ac(se, an)
8
d max E pEP
(T-pp)))
Σ Cc((s
where (c) uses the Choquet integral representation for concave measure m' (Lemma A.3), and (d) holds because condition (3) ensures that core(m) contains the extremal point arg maxpep (x)(p,)) 7 d+)].
Step 5: Equivalence Conclusion. Combining Steps 3-4, the Fuz-RL problem:
max JF((T)
s.t. (T) B
x
equivalent to the original robust CMDP:
max minE PEp
minE,a(f.pp (F.pp)
r ar)
S.I.
max (x
1 20((8, ax)
< B,
PEP
as both objectives and constraints encode the same worst-case expectations over P.
B
Appendix/Algorithm
B.1
Optimization Details
For optimal policy optimization based on value estimation, we solve the constrained optimization problem using the Lagrangian method:
((x,))= -(E B)
(22)
16

---

Algorithm Fuzzy-Guided Robust Framework for Safe RL (Fuz-RL)
1: Input: actor 0x, critics 0,, tc, fuzzy density parameters g, uncertainty levels K, replay buffer D.
2: Initialize: 8, c. fr buffer D. to MaxEpoch do
3: for epoch
4: for = 1 to 7 do
5: Sample action de For each i E [1...., K), generate perturbed state 8, = + NO 1).
To. (-]8a), observe 814-1 and get rt, Ct-
6:
7: Store the tuple (St, at, Ft, Ct2 S1+1. { / in D.
8:
end for
9:
for each actor/critic network update step do
10:
Sample mini-batch from D.
Compute perturbed values {Ve, (5:)) and (3.)).
12:
Calculate Choquet integrals using Eqs. (14)-(15).
Update Vo,, Ve,, fuzzy density parameters using Eq. (16) and Eq. (17).
14:
Update To, using Eq. (18) with specific safe RL algorithm.
end for
16:
end for
17:
Output: Trained parameters og 0y Pcs g-
The optimal policy ** and the optimal Lagrangian multiplier X can be obtained by:
(TT X²) arg max min [(T, A)
(23)
For given state 8, the optimal action selection rule becomes:
n (als) arg max (5, (s.)(s, aEA
(24)
where the action-value functions
(7, = r(s, + 71, (V; (s. = c(s.a) a) + + (s))
(25)
(26)
Here, Vi and V are the unique fixed points guaranteed by Theorems and 4.3, representing the optimal robust value functions for reward and cost, respectively.
In practice, we compute the optimal policy iteratively by initializing a Lagrangian multiplier ( and then alternating between policy updates and multiplier updates. At each iteration k compute:
(R) arg max J (Tx (1) (x) (J B)
(27)
"
(k+1) = (*()) aJ (x) B}}+
(28)
where >> is step size and [x] max æ). This process continues until convergence, yielding the optimal safe policy n² that maximizes reward while satisfying the safety constraint.
C Appendix Experiment Setting and More Results
C.1 Environment description
C.1.1 Double Integrator
The following dynamics describe the double integrator:
Øz++
1
0
[0.055
ut++
0
V1
+
at
(29)
where a, E [-1,1]. The safety constraints are [o]]
< 2 and |0| < 2.
The reward function induced unsafe state is designed as follows:
17

---

r(x,0)) = max(4 (2(x -[1)=115)) (( 1.)))
max(5 (3(z+22) (3(++22)+3((+223)) +
(30)
max(5 (3(x-22) + +3 2.2)
max(4 (2(x + + 2(u 1,5)
C.1.2
Safe Control Gym
The safe-control-gyne benchmark comprises three dynamical systems: the Cartpole, and the ID and 2D Quadrotors, as shown Figure. 4. In our setting, we use CartPole and 2D QuadRotor as the base environments.
x
[x, x, 0, 0,0
= [z, 2]"
[x, 2, 2, 01
u =
[21
T2
T
T1
F
CartPole 1D Quadrotor
2D Quadrotor
Figure 4: Schematics, state and input vectors of the cart-pole, and the 1D and 2D quadrotor environ-
ments in safe-control-gym.
For the CartPole system, the system state includes position x and velocity v of the cart, angle 0, and angular velocity w of the pole. The control inputs [-1, IR and external disturbances E [-0.5, 0.5] C IR are horizontal forces applied on the cart. The safety constraints are 10 < 0.2, i.e., keeping the pole nearly upright. The constraint function is h(0) min (0 0.2, 0.2 6).
For the 2D QuadRotor system, the state of the system is given by s xr. å. 0, 0 T where (æx =) and (#, 2) are the translation position and velocity of the COM of the quadrotor in the xzzpllae, and 0 and are the pitch angle and the pitch angle rate, respectively. The input of the system is the thrusts T2] generated by pairs of motors,
each side of the body's y-axis. The safety constraints are - 0.5 > and 1.5 > 0, i.e., maintaining its vertical position z between [-0.5, 1.5]. The constraint function is fuz)) min [z 0.5, 1.5 -25. 2}.
For the reward function setup, we utilize a weighted sum the errors between the current state s, action and their reference values the reward for each step. The details of the weighting provided in Table 2.
Besides, each environment in Safe-Control-Gyy supports two control tasks: stabilization and trajectory tracking. For stabilization, safe-controlgym provides an equilibrium pair for the system, de wt For trajectory tracking, the benchmark includes trajectory generation module capable of generating circular, sinusoidal, lemniscate, or square trajectories. The module returns references Tref; Uge, Vi {O...., L}, where L is the number of control steps in episode.
18

---

C.2 Hyper-parameters
C.2.1 Hyper-parameters RL
In all the experiments, we have revised the benchmark algorithms and Fuz-RL employing the RL framework provided by Spinning Up. The complete hyperparameters used in the experiments are shown in Table 2.
Particularly, for the CPPO and Fuz-CPPO algorithms, the risk threshold / for adverse trajectories is to 100. In the of the PPOL and CUP algorithms, the initial value of the Lagrange coefficient set to 0.001, with an upper limit of 0.2 and a learning rate of 0.02. For the RAMU algorithm, the Wang transform is utilized with 7 0.75, which is applied both the objective and the constraint,
Table 2: Hyperparameter Settings of Fuz- RL Training and Testing
CartPole-Stab CartPole-Traie Track QuadRotor-Stal
QuadRotor-Track
rollout length
150
150
250
250
training epoch batch size
500
500
1000
1000
64
64
64
128
cost limit uncertainty level K optimization step
10
10
15
15
40
40
80
80
actor learning rate
0.0003
0.0003
0,0002
0.0002
critic learning rate
0.001
0.001
0.001
0.001
fuzzy learning target KI.
0.0003
0.0003
0.0003
0.0003
0.2
0.2
0.15
0.15
hidden_siin sizes
[64, 64]
[64, 64]
[256, 128] 0.1
[256, 128]
www_act_eiight
0.1
0.01
0.01
[
!
1
0.01]
0.011
rew_state_ rew_state_weight weight
i
1 1
0.01
1
0.01
0.01
00.01 0.01
Table 3: The observation, dynamics and action uncertainty settings of
tasks
System
Uncertainty Object
Type
Config
CartPole
QuadRotor
(x, (0,
(x, 2))
Observation
white noise
std:[-0.1, 0.1]
(z, 2 (0,0) 0)
pole length pole mass
quadrotor mass
Dynamics
white noise
std:[-0.1, 0.1]
quadrotor inertia
Force: [-1, 1]
Step offset: 20
Action
Impulse noise
horizontal forces
motors thrusts
Duration: 80
Decary rate: 0.9
C.3
More experiment results
C.3.1 Comparative Analysis of Fuzzy Operator and Min-Max Operator in Safe Reinforvement Learning
We first formally define three safety sets: the fundamental safety set S. represents permissible state constraints, the safe forward invariant set Sz (a subset of S.) guarantees persistent state containment within S, under nominal conditions, and the robust safe forward invariant set SR (a conservative subset of Sx) maintains state invariance under worst-case disturbances.
As shown in Fig. 5(a), Conventional min-max approaches through robust control barrier functions (RCBFs) strictly confine states within SR, where SR (yellow region) occupies only 23.6% of S.
19

---

%
DA44
(a)
8365
1000
2.54
1414
1414
L010
22
Rebea
(1)
Figure 5: (a) Hierarchical relationship of safety sets, (b) Cost space and (c) Reward space trajectory comparisons, with dashed lines indicating safety boundaries.
(gray region). This conservative strategy ensures absolute safety the cost of exploration capability, sacrificing access to 41.7% of high-reward regions.
Our proposed fuzzy robust method overcomes this limitation through dynamic weighting different uncertainty levels. The training curves in Fig. 6 demonstrate that in the double-integrator environment, Fuz-RL's value iteration algorithm achieves 2. 17x higher final returns compared the min-max approach under Level-15 configuration. The underlying mechanism enables adaptive safety margin adjustment, permitting safe exploration in S. \ SR regions during 97.4% of test episodes. Trajectory heatmaps in Fig. 5(b)-(c) reveal that while conventional methods (black trajectories) remain strictly confined within SR, and non-robust approaches (blue trajectories) risk 32.6% boundary violations, our fuzzy robust method (green trajectories) achieves optimal performance balance with 97% safety rate through dynamic fuzzy measure.
w/o robust
fuzzy robust(Level 10)
min-max robust
fuzzy robust(Level 15)
Average Return by Epoch
200
EpRet
100
Average Risk by Epoch
EpRisk
0.1
100
200
300
400
500
Epoch
Figure 6: Training curve comparison in double integrator environment, with shaded regions indicating standard deviation across random seeds.
C.3.2 Comparison between Safe RL and Fuz-RL
Similar to the Quadrotor- Track task, we set different levels of perturbations in the observation, dynam- ics, and action evaluate the performance the CartPole-Stab, CartPole-Track and Quadrotor tasks under the three benchmark safe RI. algorithms and the corresponding Fuz-RL, as shown in Figures 7, 8, and Each point the figures represents the average metrics from 10 episodes run for each of 10 different seeds.
20

---

PPOL
Fuz-PPOL
CUP
Fuz-CUP
CPPO
PUZ-CPPO
Dynamic Uncertainty
Observation Uncertainty
Action Uncertainty
1e2
1e2
1e2
1.0
1.00
1.0
AvgRet
0.75
0.5
0.50
0.5
0.25
-0.05
0.00
0.05
0.10
0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
0.75
0.4
AvgRisk
0.50
0.3
0.3
0.2
0.1
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
Figure 7: Average episodic rewards and average episodic risk three safe RL and Fuz-RL. under various uncertainty settings in Cartpole Stab task.
C.3.3 Comparison between Fuz-RL and RAMU
To compare with the current SOTA algorithms robust safe RL, this section showcases the per- formance comparison between RAMU and Fuz-RL under uncertainties in observation, action, and dynamics, as depicted in Figures 1, Figures 12, and Figures 13, respectively.
C.3.4 Validation Power System Frequency Control Task
The IEEE 39-bus system, standard power grid benchmark with 10 generators and 46 transmission lines, was used to validate Fuz-RL's performance frequency control tasks. The system state captures frequency deviations (Af), generator rotor angles (δ), mechanical power outputs (Pm)), and tie-line power flows (Pie). Control actions involve real-time adjustments of generator active power setpoints (Prer) and discrete load shedding commands (0-100% reduction). The primary objectives to maintain frequency within 55.88 Hz, 60.2 Hz] under stochastic load/renewable fluctuations while minimizing control costs (2 Pref Prominalll 112) and avoiding safety-critical violations such line overloads (> 120% capacity).
Robustness tests were conducted under three uncertainty scenarios:
Observation noise (a = 0.1 Hz Gaussian noise in frequency measurements),
Action noise (100ms delay + 5% bias in control signals),
Dynamics noise (010 parameter drift in generator inertia/damping).
Table 4: Performance on IEEE 39-Bus Frequency Control (AvgRet AvgRisk)
Case
Method
Observation Noise
Action Noise
Dynamic Noise
IEEE-39 Bus
PPOL
-5456.30 0.17
-6357.81 0.16
-7471.96 0.52 -7363.20 0.47
IEEE-39 Bus
Fuz-PPOL
-4822.03 / 0.14
-5789.19 0.13
Fuz-PPOL. demonstrates consistent improvements over PPOL. Under observation noise, Fuz-PPOL get 1 11.6% higher returns and 17.6% lower risk. For action noise, the AvgRet metic is improved by 8.9% with 18.8% risk reduction. Under dynamics perturbations, Fuz-PPOL narrows performance degradation while reducing safety violations by 9.6%.
21

---

PPOL
Fuz-PPOL
CUP
Fuz-CUP
--- CPPO
Fuz-CPPO
Dynamic Uncertainty
1e2
Observation Uncertainty
Action Uncertainty
1.25
1.25
1.25
1.00
AvgRet
1.00
0.75
1.00
0.75
0.50
0.75
0.50
-0.10
-0.05
0.00
0.05
0.10
0.100
-0.05
0.00
0.05
0.10
0.10
-0.05
0.00
0.05
0.10
0.6
0.4
AvgRisk
0.2
0.2
0.2
0.1
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
E
Figure 8: Average episodic rewards and average episodic risk of three safe RL and Fuz-RL under various uncertainty settings in Cartpole Track task.
PPOL
Fuz-PPOL
CUP
Fuz-CUP
CPPO
Fuz-CPPO
Dynamic Uncertainty
1e2 Observation Uncertainty
Action Uncertainty
1e2
1e2
2.0
2.0
AvgRet
1.5
1.5
1.0
-0.10
-0.05
0.00
0.05
0.10
0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
0.75
0.4
AvgRisk
0.50
0.1
0.0
0.00
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
E
Figure 9: Average episodic rewards and average episodic risk of three safe RL and Fuz-RL under various uncertainty settings in Quadrotor Stab task.
22

---

PPOL
Fuz-PPOL
CUP
Fuz-CUP
CPPO
Fuz-CPPO
Dynamic Uncertainty
Observation Uncertainty
Action Uncertainty
1e2
2.0
2
2.0
AvgRet
1.5
1.5
1.0
0
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
0.10
-0.05
0.00
0.05
0.10
0.75
0.75
AvgRisk
0.50
0.50
0.5
0.25
0.25
0.00
0.00
0.0
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
Figure 10: Average episodic rewards and average episodic risk of three safe RL and Fuz-RL under various uncertainty settings in Quadrotor Track task.
Fun-FFOL
362
102
162
102
1.2
1.0
3.8
0.4
1.4
0.2
0.0
5.10
-0.0
05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
5..00
-0.05
0.00
0.05
0.10
-0.30
-0.25
0.50
0.05
0.10
0.6
0.5
0.0
02
0.65
02
0.2
-3.10
-0.05
0.05
0.10
00.0
-0.05
-3.10
-0.05
-0.10
-0.05
0.00
0.00
0.10
Figure 11: Average episodic reward (top) and average episodic risk (bottom) of Fuz-RL and RAMU different scales' observation uncertainty settings. The horizontal axis represents the uncertainty level.
SuP-CPRO
RoOCUPP
FuP-PAO
RAMUI
Co nPole Stap
Ca thte Tracs
GuadRoto: Stan
GuadRooss Tack
2.0
2.0
1.0
3.5
1.0
1.0
0.6
0.5
0.6
0.6
0.3
0.4
0.2
8.2
-3.10
-0.05
3.00
0.05
0.10
-3.10
-0.05
2.00
5.05
0.10
-0.10
-0.05
0.00
0.05
0.10
-3.10
-0.05
5.00
0.05
0.10
Figure 12: Average episodic reward (top) and average episodic risk (bottom) of Fuz-RL and RAMU
different scales' action uncertainty settings. The horizontal axis represents the uncertainty level.
23

---

CuP
PLE-POL
IAAUU
Carthulc Stub
Car thile Trock
Quad lotor Stbb
Quaditotor Black
1.2
3.2
1.0
0.8
0.5
0.5
0.7
0.4
0.0
0.4
0.8
0.2
0.2
0.50
0.05
0.00
5.05
0.20
0.30
0.05
3.20
3.50
0.00
0.03
0.10
Figure 13: Average episodic reward (top) and average episodic risk (bottom) of Fuz-RL and RAMU in different scales' dynamics uncertainty settings. The horizontal axis represents the uncertainty level.
24

---

aa) observation uncertainty settings
Fun-CPPQ
A.P.C.P
Tup APOL
.--
RANU
Cartiole Stub
Cartiole Track
Queditoter Stub
Quetifotor
1.2
2.0
1.0
1.5
0.5
1.0
1.0
0.6
0.5
0.5
0.5
0.5
0.4
0.4
0.2
-3.10
-0.05
6.50
6.05
5.10
-0.00
-0.05
0.50
0.05
3.10
-0.10
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.20
(b) dynamics uncertainty settings
Pas PPOL
Pac CUP
Pac CPCC
BAYU
CartPole Stub
Carthide Tuck
Qualltotta Slab
Quall lutor Track
1.0
2,0
0.5
0.0
1.5
0.6
0.4
0.2
1.2
0.30
0.15
0.8
0.4
0.4
0.2
0.0
-0.20
-0.05
3.00
0.1b
0.30
0.30
-0.05
0.50
-1.10
-1.25
-0.00
0.50
1.05
0.30
(c) action uncertainty settings
1:2
Cartio Stah
1.4
102
Carthole
2.0
QuadRtttrr
162
QuadRotor Track
1.2
1.5
1.5
1.0
-0.16
-0.05
0.00
0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
0.10
-0.55
0.00
0.85
0.10
-0.10
-0.05
0.00
0.05
0.10
0.
0.1
01
0.0
0.00
0.05
0.10
0.00
0.10
0.10
0.05
0.20
Figure 14: Average episodic reward (top) and average episodic risk (bottom) of Fuz-RL and RAMU in different scales' observation, dynamics, and action uncertainty settings. The horizontal axis represents the uncertainty level.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly indicate the main contributions of our Fuz-RL scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Section describe the limitations of the proposed Fuz-RL.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

The assumptions of the theoretical result are clearly stated in the Theorem. 4.3, Theorem. 4.2 and Theorm. 4.4, Refer Appendix A for the complete proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a thorough explanation Section C.I of all our experiments' settings, and the more general setting and detailed results can be found in Appendix C.2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The whole code is provided in the supplemental material with sufficient instructions in the "README.na file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
hwtpbbiisssoooooioiioioo onPolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nnps. wttpp.ccpppiiiiieeeeeeeepppppppp cctpublic//aeecccccccccccccc /gwii/gs///////eeer /Ce///eeeubbiiic cettsubbmsssiiono ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix C.2 clearly states the implementation details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We evaluate our experimental results across 10 episodes with 10 different random seeds, reporting standard deviations as error bars in the figures and as variance metrics the tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We indicate the information compute resources in appendix C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tearr/////ups//// cc.
Answer:
[Yes]
Justification:
We strictly adhere to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential impacts are stated in the Conclusion, see Section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All codes of Fuz-RL are implemented based the SpinningUp [1] which is explicitly mentioned and properly respected in Section 5.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets attached in the supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in Fuz-RL does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy fttpp://neerips... http:///ewrpp..coooooeeeeeeeeeee /0tp:////e///oooeeeeeee///////// for what should or should not be described.
32


Reviews:
[{"review_id": "rHiGJXg9S9", "reviewer": "Reviewer_LhzB", "summary": "This paper provides an algorithm that uses the concept of fuzzy logic to construct a CMDP framework to solve safe reinforcement learning problems. Fuz-rl can manage uncertainty quantification in a principled manner to perform robustly in Safe RL. The authors demonstrate their fuzzy safe RL framework is equivalent to the distributionally robust safe RL framework, but does not need to explicitly solve a min-max which can otherwise be computationally expensive and result in conservative policies.", "questions": "- Does Fig 4 indicate that there is some optimal number of uncertainty levels? K seems like another parameters I must tune and the performance seems to be heavily sensitive on this hyperparameter.\n- Relatedly, in line 105, how do you determine $\\epsilon_0, … \\epsilon_K$? Are there more hyperparameters?\n- If min-max approach causes ``overly-conservative policies” and Them 5.3 suggests the fuz-rl is equivalent to the distrubtional robust min-max framework, then why would fuz-rl not have equally ``overly-conservative policies”?\n- Could you comment on the generally computation overhead the fuz-rl approach provides? I see the computational analysis, but how much more memory and time does this approach take in training as opposed to vanilla PPO/CUP/CPPO?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is well-motivated and apart from the math, the paper is generally clear.\n- The paper proposed a generally rigorous approach to uncertainty quantification that is more efficient than techniques like cvar.\n- While fuzzy rl is not necessarily new, the theoretical approach taken and the hurdles overcome for practical implementation in the paper seems novel.\n\nWeakness:\n- Table 1, the main result, is extremely confusing – there are up and down arrows indicating whether or not performance improved, bolded numbers indicating best results, but then there are also contrasting green-red colors. I’m not sure what is considered good results\n- The experiments are quite limited to basic environments: cartpole and quadrotor. What is the observation space dimension of the quadrotor env? Can the approach scale to more complex environments? I’m not suggesting necessarily to run more experiments on these environments per se, but rather the limited experiments suggest Fuz-RL has limited scalability.\n- perhaps it would be useful to show a trajectory that demonstrates where exactly fuz-rl can interpretable provide benefit as opposed to the baseline algorithms? simply saying the results are better in table 1 seems hard to understand -- also many times the avg risk only drop by an order of 0.05 which is not too significant considering the variance in each result are compartively large.", "was_revised": true, "final_justification": "My primary concern was about the complexity of the environments and tuning the new parameters (K/epsilon/etc). The authors have responded with experiments on more complex envs like halfcheetah as well as ablations on the parameters in question.", "rebuttal": "Thank you very much for your recognition of our **method's novelty and proof clarity**. We greatly appreciate your detailed analysis and we will systematically address each concern from the perspective of **experiments ablation, evaluation, and clarification.**\n\n> **Uncertainty Levels $K$ Ablations**\n\nWe fully agree that the number of uncertainty level partitions may affect Fuz-RL's performance, as this directly influences the number of perturbation values considered in robust $V$ and $V_c$ estimation. Therefore, we tested results for **K = 1, 5, 8, 12, 15, 20, 25**. \n- During the rebuttal testing period, we set uncertainty perturbations as white noise following a normal distribution $\\mathcal{N}(0, 0.4)$ applied to observations. Test results is shown below:\n\n  | Ablation | CartPole-Stab |  | Quadrotor-Stab |  |\n  |----------|---------------|--|----------------|--|\n  |          | AvgRet | AvgRisk | AvgRet | AvgRisk |\n  | K=5      | 45     | 0.28    | 138    | 0.05    |\n  | K=8      | **58**     | **0.17**    | 136    | **0.05**    |\n  | K=12     | 49     | 0.25    | 133    | 0.06    |\n  | K=15     | 49     | 0.24    | **140**    | 0.06    |\n  | K=20     | 45     | 0.28    | 133    | 0.06    |\n\n- We further analyzed why fewer or more uncertainty levels affect performance ( $K=1$ or $K=25$). From the value function loss during training, larger $K$ leads to slower value loss descent. We believe that excessive $K$ increases the difficulty of robust $V$ estimation for each state. Since $V$ uses TD update during training with inherent errors, overly large $K$ increases the convergence burden on $V$, while overly small $K$ degrades to safe RL algorithms without robust enhancement.\n\n> **Complex Environment Test**\n\nWe greatly appreciate your suggestion regarding large-scale applications, as it can better demonstrate Fuz-RL's capabilities.\n\nWe conducted supplementary experiments on **safety-gymnasium tasks** [1] and **power grid dispatching tasks** [2] to test performance in complex environments (relative to CartPole and Quadrotor-2D) and evaluate Fuz-RL variants. Meanwhile, we explain that the original manuscript chose safe-control-gym because this benchmark **provides standardized interfaces for the three uncertainty types we focus on**. As reviewer zMCJ pointed out, our evaluation covers a broader range of uncertainty types than most safe RL papers.\n\n- **Safety-Gymnasium Safe Velocity Tasks**\n\n  | Task | Algorithm | AvgRet/AvgRisk | AvgRet/AvgRisk |\n  |------|-----------|----------------|----------------|\n  | | | (5% noise) | (10% noise) |\n  | HalfCheetahVel-v1 | PPOL | 2495/**0.04** | 2015/0.08 |\n  | | **Fuz-PPOL** | **2449/0.04** | **2335/0.04** |\n  | Walk2dVel-v1 | PPOL | 2098/0.12 | 1248/0.24 |\n  | | **Fuz-PPOL** | **2219/0.10** | **1821/0.15** |\n\n  Since Safety-Gymnasium environments **do not provide uncertainty perturbation interfaces**, during the testing phase, we only experimented with observation noise by randomly perturbing all state dimension values by 5% and 10%. We used the environment's default x velocity and y velocity constraints as cost functions. The experiments further demonstrate that when state perturbations are significant, Fuz-PPOL performs significantly better than PPOL.\n\n\n-  **Power Grid Dispatching Task**\n  | Case | Method | Observation Noise | Action Noise |\n  |------|--------|------------------|--------------|\n  | IEEE-39 | PPOL | -5456.30 / 0.17 | -6357.81 / 0.16 |\n  | IEEE-39 | Fuz-PPOL | -4822.03 / 0.14 | -5789.19 / 0.13 |\n  | IEEE-39 | MLP-Fuz-PPOL | -5147.18 / 0.17 | -6317.94 / 0.15 |\n  | IEEE-39 | **Expert-Fuz-PPOL** | **-4389.08 / 0.08** | **-5645.96 / 0.13** |\n\n    - In Appendix C.3.6, we test on a more complex power system frequency control task, following the CMDP setup from the reference paper [2], where the state observation includes current grid system features (208 dimensions) and the action space consists of generator power adjustment signals (10 dimensions).\n\n    - During the rebuttal phase, we further test 2 variants: \n      - **MLP-Fuz-PPOL**: Fuz-PPOL with a MLP-generated (original is NFN-generated) uncertainty weight $m(\\cdot)$\n      - **Expert-Fuz-PPOL**: variant which uses expert-selected fuzzy rules to replace Equation (7) auto-generated fuzzy rule (the activation strengths are also calculated by the NFN, only the rules are expert-selected). \n\n    - Interestingly, we further discovered that incorporating expert knowledge into the rule selection process for the frequency control task significantly enhances Fuz-RL's effectiveness. This finding offers valuable insights for enhancing robust safety in other large-scale scenarios.\n\n> **Table 1 Explanation**\n\nWe sincerely apologize for any confusion caused by Table 1's presentation. We provide detailed explanation of each element's meaning:\n\n- **Color coding**: \n\n   Since Fuz- series models are improvements based on their base models (we replace the original Value estimation with NFN-weighted robust value estimation), red/green colors represent **improvements/degradation of Fuz- series relative to original models**: red indicates metric increase, green indicates metric decrease. For AvgRet, we desire metric increase; for AvgRisk, we desire metric decrease.\n\n- **Bold formatting**: \n\n  For each task under each uncertainty setting, we use bold to indicate the **best values** among the 7 compared algorithms  (maximum AvgRet, minimum AvgRisk).\n\n- **Values**:\n \n  Each AvgRet and AvgRisk metric value in Table 1 represents the **mean±std across 10 different uncertainty levels $ε$**. Therefore, Table 1 encompasses **840 test experiments** (4 tasks × 3 uncertainty types × (4 baselines + 3 Fuz-RL variants) × 10 uncertainty levels). For **details of the specific 10 different uncertainty levels $ε$**, we show results in Figure 3 and Appendix Figures 9-14.\n\n> **Selection of $ε_{0,...K}$**\n\n  We are happy to have the opportunity to explain the details regarding Fuz-RL. $ε_{0,...K}$ is obtained by **equally dividing $[0, ε]$ into $K$ parts**. During training, for the $k$-th perturbation level, we randomly sample a value from $\\[(ε_{k-1}+ε_{k})/2, (ε_{k}+ε_{k+1})/2]$ as the current perturbation $\\zeta_k$ and add it to $s'$, creating perturbed states $s'\\_{\\zeta_k}$. The fuzzy measure $m(s'\\_{\\zeta_k})$ is then computed by our NFN. Finally, we enhance robustness by replacing the original $V$ and $V_c$ estimations with fuzzy measure-weighted robust versions:\n\n  $V = ∑^N_{n=1} ∑^K_{k=1} m(s'\\_\\{ζ_k})V(s'\\_\\{ζ_k}) / N$\n \n  $V_c = ∑^N_{n=1} ∑^K_{k=1} m(s'\\_\\{ζ_k})V_c(s'\\_\\{ζ_k}) / N$\n\n> **Min-max Robust v.s Fuzzy Robust**\n\nAlthough Theorem 5.3 does prove the mathematical equivalence between Fuz-RL and distributionally robust safe RL, this equivalence does not mean they exhibit the same level of conservatism during practical deployment. Traditional min-max methods apply identical worst-case assumptions to **all state-action pairs**. In contrast, Fuz-RL adaptively adjusts the weights of fuzzy measures **based on current state features $z_r, z_c$**. As illustrated in Appendix Figure 6:\n  - When in safe states, fuzzy meassure weight is dominated by low disturbance levels, with its control strategy leaning more toward **reward-pursuit**;\n  - When approaching the constraint boundary, the weights of high disturbance levels increase significantly, making the control strategy more inclined toward **risk-aversion**.\n\n> **References**\n\n  [1] Ji J, Zhang B, Zhou J, et al. Safety gymnasium: A unified safe reinforcement learning benchmark[J]. Advances in Neural Information Processing Systems, 2023, 36: 18964-18993.\n\n  [2] Wan X, Sun M, Chen B, et al. AdapSafe: Adaptive and safe-certified deep reinforcement learning-based frequency control for carbon-neutral power systems[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(4): 5294-5302."}, {"review_id": "NRqkoozsxz", "reviewer": "Reviewer_EyJ3", "summary": "This paper investigates a highly practical problem: robust safe reinforcement learning under multi-source uncertainty. The authors propose a new fuzzy-guided robust framework, named Fuz-RL, and prove that solving the Fuz-RL problem — formulated as a constrained Markov decision process (CMDP) — is equivalent to solving a distributionally robust safe reinforcement learning problem formulated as a robust CMDP. Experimental results on safe-control-gym scenarios under various types of uncertainties in observation, action, and dynamics demonstrate the empirical effectiveness of the proposed method.", "questions": "Please refer to the section \"Weaknesses\" for detailed information.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths：\n1. This paper proves the equivalence between the fuzzy robust safe RL problem and the distributionally robust safe RL problem, and the proof appears to be logically sound and reasonable.\n2. The paper presents comprehensive experimental results demonstrating that the proposed algorithm achieves good robustness and safety. It also includes ablation studies and interpretability analysis of the fuzzy measure weights.\n3. I checked Theorem 5.3 in particular—they appear correct based on standard RL math, with straightforward algebra and clear steps. I didn’t find any obvious mistakes in the provided derivations.\n\n# Weaknesses：\n1. To improve clarity, it is suggested to restructure the paper, although the current presentation is not unacceptable. Specifically, it is recommended to move the content of Subsection 4.1 to Section 3, to make it clear that this part provides background and foundational concepts rather than the novel contributions of the paper. Additionally, Subsection 4.2 and Section 5 should be combined into a single section, serving as the presentation of the proposed method and theoretical analysis.\n2. The related work section should include more discussion on robust safe reinforcement learning approaches, such as those achieving robustness through robust model predictive control [1] or adversarial robustness [2]. The authors should also analyze how their proposed method differs from and improves upon these existing approaches.\n3. It would be helpful to include training curves that correspond to the performance metrics reported in Table 1.\n\n# References\n[1] Zanon M, Gros S. Safe reinforcement learning using robust MPC[J]. IEEE Transactions on Automatic Control, 2020, 66(8): 3638-3652.\n\n[2] Li Z, Hu C, Wang Y, et al. Safe reinforcement learning with dual robustness[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.", "was_revised": true, "final_justification": "This paper addresses a highly practical and challenging problem: robust safe reinforcement learning under multi-source uncertainty. The theoretical contributions and innovation of this work are substantial and commendable, offering a new perspective on modeling uncertainty in safety-critical decision-making.\n\nWhile the original manuscript has some weaknesses in paper organization and experimental validation, the authors have provided clear and constructive responses during the rebuttal phase. They have committed to structural improvements and included additional experimental results that effectively address the majority of the reviewers' concerns.\n\nOverall, this is a strong and valuable paper. Given the significance of the problem, the novelty of the framework, and the satisfactory author response, the paper is recommended for acceptance.", "rebuttal": "Thank you very much for your **approval of the novelty of our method and the clarity of our theorems and proofs**. We fully agree with your structural suggestions and will implement the following changes:\n\n> **Section 4 and Section 5**  \n\nThank you again for your suggestions. We will move **Subsection 4.1** (Fuzzy Measures Fundamentals) to **Section 3** as background material; Meanwhile, we will combine **Subsection 4.2** (Neuro-Fuzzy Network) and **Section 5** (Theoretical Framework) into a unified Section 4: Proposed Method and Theoretical Analysis.\n\n> **Section 2**  \n\nWe fully acknowledge the need for more comprehensive coverage of robust safe RL methods. We will expand Section 2 to include **robust model predictive control-guided safe RL approaches** [1, 2] and adversarial robustness methods [3, 4, 5].\n\n[1] Zanon M, Gros S. Safe reinforcement learning using robust MPC[J]. IEEE Transactions on Automatic Control, 2020, 66(8): 3638-3652.  \n\n[2] Airaldi F, De Schutter B, Dabiri A. Learning safety in model-based Reinforcement Learning using MPC and Gaussian Processes[J]. IFAC-PapersOnLine, 2023, 56(2): 5759-5764.\n\n[3] Li Z, Hu C, Wang Y, et al. Safe reinforcement learning with dual robustness[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.\n\n[4] Liang Y, Sun Y, Zheng R, et al. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning[J]. Advances in neural information processing systems, 2022, 35: 22547-22561.\n\n[5] Zhang H, Chen H, Xiao C, et al. Robust deep reinforcement learning against adversarial perturbations on state observations[J]. Advances in neural information processing systems, 2020, 33: 21024-21037.\n\n\n> **Section 6**  \n\nThank you for your suggestion. Due to the initial page limit, we only presented the results of over 800 test experiments we conducted. We are happy to add the training process/results in Section 6 of the updated manuscript."}, {"review_id": "QpuJ9wmyK5", "reviewer": "Reviewer_zMCJ", "summary": "The paper proposes a deep neuro-fuzzy control architecture for robust safe reinforcement learning (RL). It defines a fuzzy control problem, and proves its equivalence to the more commonly seen distributional robustness problem. Solving the fuzzy RL problem with the proposed framework incurs lower computational cost, and (generally) shows improved safety and cost performance compared to existing safe RL methods. Evaluation is carried out in low-dimensional continuous control tasks and a higher-dimensional grid generation control task.", "questions": "1. Have you tried more complex environments than the ones in your evaluation? Are there any reasons that the method would not work in more complex environments with larger state, action or observation spaces?\n2. Did you try different numbers of membership functions other than 3 ({Low, Medium, High})? What are the trade-offs of varying this?\n3. How is the number of fuzzy rules $N$ chosen? Is it 3x3, i.e. every combination of {Low, Medium, High} for $z_c$ and $z_r$? If so, this should be made explicit in the formulation.\n4. The metric \"AvgRisk\" is defined as the \"proportion of constraint violations\" (line 264). Is this the proportion of episodes or timesteps? Is it discounted? Does a proportion measure <0.5 imply that the constraint is satisfied in the median? Even if it does, it wouldn't tell us whether the expected cost is below the constraint.\n\nI would say that question (1) is the most important for instructing my evaluation score, as clarity-related points could be fixed by some restructuring or citing other relevant works.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. To my knowledge, this is the first work that achieves robust control of CMDPs using fuzzy logic. Using fuzzy logic to solve the robust CMDP problem setting is a novel idea, and the existence of a proof to show equivalence to traditional distributional robustness is interesting.\n2. The proofs are clearly laid out and I could not find any issues with them. However I have not previously worked with fuzzy logic, so those components of the proofs are not familiar to me and I may have missed something.\n3. Generally the notation and technical content flow is good, and the paper is easy to follow even for someone without a background in fuzzy logic. However I do have some issues with clarity and structure, which I will describe below.\n\n\n# Weaknesses\n\n## Setting/justification\n\nOverall there are some issues with the high-level justification and intuition for why fuzzy logic should be used for robust safe RL.\n\n1. Lines 31-34 are not very concrete, and don't cite anything to back up their claims. It's not clear what \"complex, non-additive interactions\" are, and why standard probability measures, or previous mentioned methods, cannot handle them. I'm also not clear what \"treat different perturbations with equal importance\" means (line 27).\n2. Section 4.1 is also a bit abstract and difficult to link to the CMDP/safe RL problem. It's not clear why/when super-additivity helps with uncertain dynamics, or where super-additivity originates in an uncertain system's behaviour.\n3. Figure 1 does not seem to be a good use of space, as it illustrates a simple-enough concept from equation (5).\n\n\n## Method clarity\n\nWhen introducing the method (line 104 onwards), it's not made very clear _why_ one would need to consider different levels of uncertainty within \\epsilon. From the rest of the paper, it seems like different levels of perturbation might have different impacts on cost and reward for some (s,a) pairs, but this could be spelled out earlier on. More text explanation would help explain why it is necessary to partition the uncertainty into different levels, as it only becomes clear later on in the paper.\n\nThe method explanation would be clarified by leading with a high-level description of how the method works. Currently, Section 4 launches directly into fuzzy measures and approximating these with a fuzzy NN. It's not clear until much later on how the fuzzy measure fits into a full solution method -- I only started fully understanding the approach when reading the fuzzy rule format in (7). Figure (2) is therefore a helpful diagram, but should appear earlier in the text in my opinion. It should also be referred to when describing the method, and should have more labelling to make clear what e.g. the x/y lines are in the fuzzification step.\n\n\n## Technical\n\n1. The specific choice of the number of fuzzy membership functions (Low, Medium, High) is not discussed or ablated.\n2. I cannot see any statement/discussion of the number of fuzzy rules $N$.\n3. I cannot see the feature extractor parameters anywhere in the formulation, whereas everything else is explicitly parameterised.\n4. The experiment metric \"AvgRisk\" is not very clearly defined (see question below)\n\n## Evaluation\n\nThe continuous control tasks in the evaluation are quite simple and low-dimensional compared to other recent safe RL papers. RAMU and OTP (Queeney, James, et al. \"Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guarantees.\" TMLR) evaluate on more complex domains (quadruped, walker). The library used in this work (Safe-Control-Gym) is now quite old, and more recent alternatives such as Safety-Gymnasium offer more complex environments.\nHowever, this paper's wide range of types of perturbations tested (observation white noise, control delays and impulses, dynamics model parameters and drift) is more thorough and wide ranging than most other safe RL papers I have read.\n\n\n## Small clarity / typo\n\n- Line 75 \"for RL community\" -> \"for the RL community\".\n- In Line 104, explicitly saying and italicising \"K distinct _uncertainty levels_\" would make it clearer that these are the uncertainty levels referred to later in Section 4.1.\n- Line 254 \"the SpinningUp\".", "was_revised": true, "final_justification": "Thanks for your response. Given the additional experiments you have run and analysis provided, the ablation of different parts of the method is now much more extensive than the original submitted version. I have no further concerns and am therefore raising my score to \"accept\".", "rebuttal": "**Thank you very much for your recognition of the originality and clarity of our proofs and main paper.** We greatly appreciate your detailed analysis and will systematically address each concern in the **experiments evaluation, background, and method sections**.\n\n> **Evaluation**\n\nWe greatly appreciate your suggestion regarding large-scale applications, as it can better demonstrate Fuz-RL's capabilities.\n\nWe conducted supplementary experiments on **safety-gymnasium tasks** and **power grid dispatching tasks** to test performance in complex environments (relative to CartPole and Quadrotor-2D) and evaluate Fuz-RL variants. Meanwhile, we explain that the original manuscript chose safe-control-gym because this benchmark **provides standardized interfaces for the multiple uncertainty types we focus on**. As you pointed out, our evaluation covers a broader range of uncertainty types than most safe RL papers.\n\n- **Safety-Gymnasium Safe Velocity Tasks**\n\n  | Task | Algorithm | AvgRet/AvgRisk | AvgRet/AvgRisk |\n  |------|-----------|----------------|----------------|\n  | | | (5% noise) | (10% noise) |\n  | HalfCheetahVel-v1 | PPOL | 2495/**0.04** | 2015/0.08 |\n  | | **Fuz-PPOL** | **2449/0.04** | **2335/0.04** |\n  | Walk2dVel-v1 | PPOL | 2098/0.12 | 1248/0.24 |\n  | | **Fuz-PPOL** | **2219/0.10** | **1821/0.15** |\n\n  Since Safety-Gymnasium environments **do not provide uncertainty perturbation interfaces**, during the testing phase, we only experimented with observation noise by randomly perturbing all state dimension values by 5% and 10%. We used the environment's default x velocity and y velocity constraints as cost functions. The experiments further demonstrate that when observation perturbations are significant, Fuz-PPOL performs significantly better than PPOL.\n\n\n-  **Power Grid Dispatching Task**\n  | Case | Method | Observation Noise | Action Noise |\n  |------|--------|------------------|--------------|\n  | IEEE-39 | PPOL | -5456.30 / 0.17 | -6357.81 / 0.16 |\n  | IEEE-39 | Fuz-PPOL | -4822.03 / 0.14 | -5789.19 / 0.13 |\n  | IEEE-39 | MLP-Fuz-PPOL | -5147.18 / 0.17 | -6317.94 / 0.15 |\n  | IEEE-39 | **Expert-Fuz-PPOL** | **-4389.08 / 0.08** | **-5645.96 / 0.13** |\n\n    - In Appendix C.3.6, we test on a more complex power system frequency control task, where the state observation includes current grid system features (**208 dimensions**) and the action space consists of generator power adjustment signals (**10 dimensions continuous action space**).\n\n    - During the rebuttal phase, we further test 2 variants: \n      - **MLP-Fuz-PPOL**: Fuz-PPOL with a MLP-generated (original is NFN-generated) uncertainty weight $m(\\cdot)$\n      - **Expert-Fuz-PPOL**: variant which uses expert-selected fuzzy rules to replace Equation (7) auto-generated fuzzy rule (the activation strengths are also calculated by the NFN, only the rules are expert-selected). \n\n    - Interestingly, we further discovered that incorporating expert knowledge into the rule selection process for the frequency control task significantly enhances Fuz-RL's effectiveness. This finding offers valuable insights for enhancing robust safety in other large-scale scenarios.\n\n> **Background/Justification**\n\n- **Concrete examples of \"complex, non-additive interactions\" (Lines 31-34)**:\n\n  Thank you for your suggestion. We acknowledge that our initial description was too abstract. Therefore, we provide concrete examples with appropriate citations:\n\n  *In robotic control systems, sensor noise can amplify model uncertainty effects non-linearly. For instance, when IMU measurement errors combine with actuator uncertainties (such as motor backlash or delay), the resulting control performance degradation often exhibits super-additive behavior where the combined effect exceeds the sum of individual uncertainties [1,2].*\n\n- **\"Equal importance\" limitation (Line 27)**:\n\n  Traditional distributionally robust methods typically use uniform uncertainty sets that treat all perturbations within the uncertainty set equally, regardless of their likelihood or impact severity. Our approach allows adaptive weighting of different perturbation magnitudes based on system state and uncertainty levels.\n\n- **Figure 1 improvement**:\n\n  We agree that Figure 1 could be more informative. We will replace it with a concrete control example showing how sensor noise + model uncertainty creates super-additive safety violations. Meanwhile, we further elaborate that super-additivity emerges naturally in control systems because:\n  - Small perturbations in one subsystem can amplify uncertainties in coupled subsystems\n  - Most real control systems exhibit nonlinear responses where combined uncertainties produce disproportionate effects  \n  - When systems approach constraint boundaries, the same uncertainty magnitude has increasingly severe consequences\n\n> **Method Clarity**\n\n- **Uncertainty level partitioning motivation**:\n\n  Thank you very much for your suggestion. We will refine this section to better demonstrate the motivation for **uncertainty level partitioning**. We will add the following explanation before Equation (4):\n\n  *Different uncertainty magnitudes have fundamentally different impacts on safety and performance. Small perturbations may be acceptable for exploration, while large perturbations require conservative responses. By partitioning the uncertainty set into levels, our fuzzy measure can adaptively weight these different regimes based on the current system state.\"*\n\n- **Figure 2 enhancement**:\n\n  We will show Figure 2 earlier when introducing the method in Section 4, and add explicit labels for the x/y axes in the fuzzification step, including uncertainty level indicators.\n\n> **Technical Issues**\n\n- **Number of fuzzy rules $N$**: \n\n  For our experiments: $N = 3^3 = 27$ rules (3 $j_c$ × 3 for $J_r$ × 3 for $j_i$). Besides, in new experiments added during the rebuttal phase (Expert-Fuz-PPOL), we used expert experience to select partial rules $N' < N$, achieving superior performance in both training convergence speed and final performance. We commit to including these findings in the updated version.\n\n- **Feature extractor parameters**: \n\n  We will add explicit parameterization: $θ_{mlp}: S × A → R^2$, where $θ_{mlp}$ represents the learnable MLP parameters for extracting $z_r$ and $z_c$ features.\n\n- **AvgRisk definition**: \n\n  We apologize for any confusion. We clarify that the AvgRisk represents the proportion of episodes with safety constraint violations, not episode cost. Since safe-control-gym counts the number of constraint violations in an episode, we use whether this count is non-zero as the safety criterion.\n\n> **References**\n\n[1] Ni P, Xia Y, Li J, et al. Using polynomial chaos expansion for uncertainty and sensitivity analysis of bridge structures[J]. Mechanical Systems and Signal Processing, 2019, 119: 293-311.\n\n[2] Rahman S. Uncertainty quantification under dependent random variables by a generalized polynomial dimensional decomposition[J]. Computer Methods in Applied Mechanics and Engineering, 2019, 344: 910-937."}, {"review_id": "ccpS7yepUK", "reviewer": "Reviewer_P3M8", "summary": "The paper proposes Fuz-RL, a framework for safe RL that leverages fuzzy-measures to address coupled uncertainties by modelling super-additive effects, an aspect often overlooked by existing safe RL algorithms.\nThe authors propose learning uncertainty measures at varying levels of uncertainty using a neuro-fuzzy network. These learned measures are then integrated into a fuzzy Bellman operator enabling the estimation of robust value functions under potential perturbations. The paper demonstrates that solving the Fuz-RL problem is equivalent to solving a distributionally robust safe RL formulation. \nThe framework is incorporated into model-free safe RL approaches, and its effectiveness is evaluated across a range of environments within the SafeControl Gym. The results highlight that Fuz-RL often outperforms state-of-the-art baselines in terms of both safety and performance, particularly under action, observation, and dynamics uncertainties.", "questions": "Q1: In equation 8) I believe the term sum_i \\alpha_{R_i} in both numerator and denominator is redundant. Wouldn't it cancel out with the numerator?\nQ2: How do you enforce that the resulting measures in equation 8) satisfy the additivity condition (2) in Definition 4.1?\nQ3: Given the clipping in equation 8), how do you ensure you fulfil condition 1) in Def. 4.1, i.e. m({1,2...,K}) =1? As far as I am concerned, this condition is also used in Theorem A.1 in Appendix (line 422) to prove the contraction property of the fuzzy bellman operator.\nQ4: In equation 16, how are R_t and C_t estimated? Via Montecarlo sampling? Don't they suffer from high variance?\nQ5: What's the impact of the hyperparameter \\lambda_Theta in equation 16? An analysis on its impact would be useful.\nQ6: How is the cost function defined in each environment? I couldn't find an algebraic definition of the cost term in the appendix, which is needed in order to learn V_c and for the neuro-fuzzy network update.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper proposes a novel approach that utilizes fuzzy measures to enhance robustness in model-free reinforcement learning (RL), aiming to balance performance with safety constraints under uncertainty. The authors demonstrate the capabilities of their framework across several benchmark environments.\n\nWhile the approach is novel and is proved to work well in practice, I find the proposed methodology unnecessarily complex, which raises concerns about the ease of implementation and the generalizability of the method to more complex settings. I add some specific concerns below:\n\n\n1) Complexity: The authors argue that existing risk-sensitive approaches require careful parameter tuning and struggle with multiple noise sources. While the later might be a valid concern, existing methods, for example [1], typically involve fewer parameters and are more straightforward to implement. Specifically, the complexity lies in the fuzzification process and the associated rule application. Why one needs to define 3 different fuzzy degrees (low, medium and high)? I wonder if it could potentially be replaced by a single neural network that directly outputs a continuous uncertainty degree. Although the authors argue that their approach avoids the need for multiple networks to handle uncertainty measures (line 197), the proposed approach requires: one feature extractor, 6 membership functions to obtain the fuzzy degrees (which according to Appendix C.2.1 are nontrivial to define and require a lot of hyperparameters), and K parameters for the rule's weight. Could the approach be made simpler?\n\n\n2) Justification for super-additivity: It remains unclear whether the environments used in the evaluation justify the need for a fuzzy uncertainty measure. Would a simpler, linear additive uncertainty model not suffice in such relatively simple (1D and 2D) settings? If the fuzzy approach provides a significant advantage, a justification would be appreciated.\n\n\n3) Definitions 4.1 and 4.2 are presented without any references. This may inadvertently suggest that these are novel contributions, when in fact they originate from prior work—e.g., Choquet (1953) and Hung et al. (2015) [2], if not earlier. Appropriate citations should be included to acknowledge this foundational work.\n\n4) Baselines: The comparative evaluation remains somewhat unclear. It is not evident how the proposed method is integrated with the presented baselines (CPPO or PPOL), which as far as I am concerned are already safe RL algorithms.  Why not integrate Fuz-RL with default PPO instead? This would provide a fairer assessment of the added value of Fuz-RL. Furthermore, the only comparison with RAMU is relegated to the appendix, and results suggest that the improvements offered by Fuz-RL over RAMU are not statistically significant in most of the environments.\nThere are other relevant baselines, particularly risk-averse distributional RL methods where risk measures are easily computed from learnt quantile value functions such as in [1] or similar, that would provide a more comprehensive evaluation of the proposed framework.\n\n[1] Urpí, Núria Armengol, Sebastian Curi, and Andreas Krause. \"Risk-Averse Offline Reinforcement Learning.\" International Conference on Learning Representations.\n[2] Hung T. Nguyen; Vladik Kreinovich; Joe Lorkowski; Saiful Abu.  Why Sugeno λ-measures.\" 2015 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). IEEE, 2015.", "was_revised": true, "final_justification": "The authors have addressed most of my original concerns through detailed clarifications and new experiments. The rebuttal resolved issues around implementation complexity, provided evidence of effectiveness in more complex environments, clarified parameter learning, and explained specific experimental choices. Theoretical definitions will now be properly cited.\n\nOne main issue remains: the practical necessity of super-additivity (fuzzy measures) over simpler uncertainty models is still not fully justified, especially given the evaluated environments.\n\nGiven the substantial improvements and clarifications, but with this key point still unresolved, I have raised my score to Borderline Accept. The paper represents clear progress and addresses most prior concerns, but a stronger motivation for super-additivity is still needed.", "rebuttal": "Thank you very much for your **recognition of the originality and clarity of our paper**. We greatly appreciate your detailed analysis and will systematically address each concern from the perspective of experimental setting, evaluation, and clarification.\n\n> **W4,Q4: Experimental Settings**\n\nThank you for your questions regarding our experimental design:\n\n- **W4: Baseline selection**: \n\n  Since our paper aims to improve the **robustness** of existing **safe** RL algorithms in uncertain environments, we chose safe RL algorithms as baselines rather than basic RL algorithms. As shown in Section 5.2, we enhance robustness by replacing the original $V$ and $V_c$ estimations with fuzzy measure-weighted robust versions:\n\n  $V = ∑^N_{n=1} ∑^K_{k=1} m(s'\\_\\{ζ_k})V(s'\\_\\{ζ_k}) / N$\n \n  $V_c = ∑^N_{n=1} ∑^K_{k=1} m(s'\\_\\{ζ_k})V_c(s'\\_\\{ζ_k}) / N$\n\n  During the training phase, each perturbation $\\zeta_k$ is random sampled from the uncertainty level interval $\\[(ε_{k-1}+ε_{k})/2, (ε_{k}+ε_{k+1})/2]$  and applied to the next state $s'$, creating perturbed states $s'\\_{\\zeta_k}$. The fuzzy measure $m(s'\\_{\\zeta_k})$ is then computed by our NFN.\n\n- **W4: RAMU comparison**: \n\n  - The detailed comparison with RAMU is presented in Table 1. It's important to note that  each AvgRet and AvgRisk metric value in Table 1 represents the **mean±std across 10 different uncertainty levels $ε$**. Therefore, Table 1 encompasses **840 test experiments** (4 tasks × 3 uncertainty types × (4 baselines + 3 Fuz-RL variants) × 10 uncertainty levels). \n\n  - For **specific 10 different uncertainty levels' details**, we show the results in Figure 3 and Appendix Figures 9-14, with RAMU comparisons shown in Figures 12-14. Results show that across all robust test settings, Fuz-RL surpasses RAMU in achieving **higher AvgRet in 83.3% of the tasks.**\n\n- **Q4: $R_t$ and $C_t$ estimation**: \n\n  Since our baseline algorithms (PPOL, CUP, CPPO) are all PPO variants, Fuz-RL's $R_t$ and $C_t$  adopt the **rewards-to-go** approach consistent with the baseline methods and use GAE to estimate the advantage.\n\n  $R_t = ∑_{τ=t}^T γ^{τ-t} r_τ$\n\n  $C_t = ∑_{τ=t}^T γ^{τ-t} c_τ$\n\n> **W1,W2: Complex Tasks Evaluation**\n\nWe greatly appreciate your suggestion regarding large-scale applications, as it can better demonstrate Fuz-RL's capabilities.\n\nWe conducted supplementary experiments on **safety-gymnasium tasks** [1] and **power grid dispatching tasks** [2] to test performance in complex environments (relative to CartPole and Quadrotor-2D) and evaluate Fuz-RL variants. Meanwhile, we explain that the original manuscript chose safe-control-gym because this benchmark **provides standardized interfaces for the three uncertainty types we focus on**. As reviewer zMCJ pointed out, our evaluation covers a broader range of uncertainty types than most safe RL papers.\n\n- **Safety-Gymnasium Safe Velocity Tasks**:\n\n  | Task | Algorithm | AvgRet/AvgRisk | AvgRet/AvgRisk |\n  |------|-----------|----------------|----------------|\n  | | | (5% noise) | (10% noise) |\n  | HalfCheetahVel-v1 | PPOL | 2495/**0.04** | 2015/0.08 |\n  | | **Fuz-PPOL** | **2449/0.04** | **2335/0.04** |\n  | Walk2dVel-v1 | PPOL | 2098/0.12 | 1248/0.24 |\n  | | **Fuz-PPOL** | **2219/0.10** | **1821/0.15** |\n\n  Since Safety-Gymnasium environments **do not provide uncertainty perturbation interfaces**, during the testing phase, we only experimented with observation noise by randomly perturbing all state dimension values by 5% and 10%. We used the environment's default x velocity and y velocity constraints as cost functions. The experiments further demonstrate that when state perturbations are significant, Fuz-PPOL performs significantly better than PPOL.\n\n\n-  **Power Grid Dispatching Task**:\n  | Case | Method | Observation Noise | Action Noise |\n  |------|--------|------------------|--------------|\n  | IEEE-39 | PPOL | -5456.30 / 0.17 | -6357.81 / 0.16 |\n  | IEEE-39 | Fuz-PPOL | -4822.03 / 0.14 | -5789.19 / 0.13 |\n  | IEEE-39 | MLP-Fuz-PPOL | -5147.18 / 0.17 | -6317.94 / 0.15 |\n  | IEEE-39 | **Expert-Fuz-PPOL** | **-4389.08 / 0.08** | **-5645.96 / 0.13** |\n\n    - In Appendix C.3.6, we test on a more complex power system frequency control task, following the CMDP setup from reference [1], where the state observation includes current grid system features (208 dimensions) and the action space consists of generator power adjustment signals (10 dimensions).\n\n    - During the rebuttal phase, we further test 2 variants: \n      - **MLP-Fuz-PPOL**: Fuz-PPOL with a MLP-generated (original is NFN-generated) uncertainty weight $m(\\cdot)$\n      - **Expert-Fuz-PPOL**: variant which uses expert-selected fuzzy rules to replace Equation (7) auto-generated fuzzy rule (the activation strengths are also calculated by the NFN, only the rules are expert-selected). \n\n    - Interestingly, we further discovered that incorporating expert knowledge into the rule selection process for the frequency control task significantly enhances Fuz-RL's effectiveness. This finding offers valuable insights for enhancing robust safety in other large-scale scenarios.\n\n  [1] AdapSafe: Adaptive and Safe-Certified Deep Reinforcement Learning-Based Frequency Control for Carbon-Neutral Power Systems\n\n> **Q1-3: Explanation of Equation (8)**\n\nThank you for your suggestions regarding Equation 8. \n  $$m(P^a_{s,k}) = \\min \\left(\\frac{1}{K}, \\frac{\\exp \\left(\\frac{\\sum_i \\alpha_{R_i} \\theta_k}{\\sum_i \\alpha_{R_i}}\\right)}{\\sum_{j} \\exp \\left(\\frac{\\sum_i \\alpha_{R_i} \\theta_j}{\\sum_i \\alpha_{R_i}}\\right)}\\right)$$\nTo avoid confusion, we provide the following clarification on Eq. 8 and how it satisfies Conditions 1-2 in Definition 4.1:\n\n- **Q1: Regarding the $\\sum_i α_{R_i}$ terms**: \n\n  The appearance of $\\sum_i α_{R_i}$ in both numerator and denominator is **not redundant**. Equation (8) represents a **softmax normalization** applied to each uncertainty level $k$ individually. Specifically, for each uncertainty level $k$, we compute:\n\n  $m(P^a_{s,k}) = \\text\\{min}(1/K, \\text\\{softmax}\\_k(∑\\_i α_{R_i} θ_k / ∑\\_i α_{R_i}))$\n\n  The $∑\\_i α_{R_i}$ term serves as a normalization factor that ensures the softmax operation is properly scaled by the total rule activation strength.\n\n- **Q2: Regarding Condition (2)**: \n\n  Equation 8 naturally guarantees that for each element $k ∈ \\\\{1, 2, ...K\\\\}, m(P^a_{s,k}) ≥ 0$. This automatically satisfies monotonicity: if $A ⊆ B$, then $m(P^a_{s,A}) ≤ m(P^a_{s,B})$.\n\n- **Q3: Regarding Condition (1)**: \n\n  We satisfy this through Equation (6):\n\n  $mλ(P^a_{s,A∪B}) = m(P^a_{s,A}) + m(P^a_{s,B}) + λ m(P^a_{s,A}) m(P^a_{s,B})$\n\n\n  Since $λ > 0$ and $m(\\cdot) > 0$, **there necessarily exists a $λ$ such that $m_λ(P^a_{s,{1,...,K}}) = 1$**. Importantly, in practice, our robust $V$ and $V_c$ estimation relies only on **individual $K$ elements' fuzzy measures** $m(s'_{ζ_i}) \\ i \\in \\\\{1, 2, ..., K \\\\}$ in Equation (13) and not on **subset measures**, so we do not need to compute $λ$ in real-time during training.\n\n>  **Q6: Cost Function Definitions**\n\nWe apologize for any confusion. The cost functions are defined in the appendix in textual form. For clarity, here are the algebraic definitions:\n\n- **CartPole tasks**: \n\n  Constraint violation when pole angle exceeds ±0.2 rad:\n\n  $c(θ) = max(0, |θ| - 0.2)$\n\n- **Quadrotor tasks**: \n\n  Constraint violation when altitude outside [0.5, 1.5]:\n\n    $c(z) = max(0, 0.5 - z) + max(0, z - 1.5)$\n\n  These cost functions return positive values when safety constraints are violated and zero otherwise, enabling the learning of $V_c$ for constraint satisfaction and providing necessary signals for NFN updates.\n\n> **Q5: The Impact of the Hyperparameter \\lambda_Theta** \n\n  $\\lambda_\\Theta$ serves to balance safety and performance in the NFN training objective:  \n\n- When $\\lambda_\\Theta \\to 0$: The focus shifts to reward maximization, which may come at the expense of safety.  \n- When $\\lambda_\\Theta \\to \\infty$: The emphasis is placed on satisfying safety constraints, potentially leading to excessive conservatism.  \n\n  After conducting experiments with $\\lambda_\\Theta = 0.5, 1.0, 2.0$, we found that $\\lambda_\\Theta = 1.0$ strikes a favorable balance across most tasks. We commit to adding more ablation experiments of $\\lambda_\\Theta$ for this part in the updated manuscript.\n\n> **References**\n\n  [1] Ji J, Zhang B, Zhou J, et al. Safety gymnasium: A unified safe reinforcement learning benchmark[J]. Advances in Neural Information Processing Systems, 2023, 36: 18964-18993.\n\n  [2] Wan X, Sun M, Chen B, et al. AdapSafe: Adaptive and safe-certified deep reinforcement learning-based frequency control for carbon-neutral power systems[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(4): 5294-5302."}]

Author rebuttal:

Meta-review: The paper proposes Fuz-RL, a safe RL framework that aggregates multi-source, non-additive uncertainties using an adaptive neuro-fuzzy network and a fuzzy Bellman operator. The theory shows its equivalence to a distributionally robust CMDP, yielding robustness without heavy min–max optimization. Empirically, plugging Fuz-RL into standard safe-RL learners improves return–risk trade-offs under observation/action/dynamics uncertainties across several control benchmarks.

**Strengths**
* **Principled & novel formulation:** first to fuse fuzzy measures with robust safe-RL; clear link to distributionally robust CMDP with preserved convergence guarantees.
* **Interpretable robustness:** state-adaptive, learnable weighting over uncertainty levels.
* **Practicality:** drop-in value estimation; avoids adversarial training.
* **Empirical gains:** consistent improvements on safety and/or returns across tasks; ablations and analysis added.
* **Clarity of core proofs:** contraction/equivalence arguments considered solid.

**Weaknesses & how the authors address them.**
* **Limited experimental scope / formatting issues:** authors add larger-scale tasks, clarified Table formatting, and report wall-clock overheads; results remain favorable.
* **Sensitivity to design choices (e.g., number of uncertainty levels K, ε-schedules):** authors provid K-ablations, sampling details, and guidance; performance is shown robust to reasonable ranges.
* **Organization & related work coverage (robust MPC, adversarially robust safe-RL):** authors commit restructuring and expanding related-work discussion; add training-curve diagnostics.

This work proposes a theoretically sound, interpretable, and practical approach to robust safe RL. Reviewers indicated their concerns were  mostly resolved and kept accept-leaning scores. As a result, I recommend accept.
Decision: Accept (poster)