Title: Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty
Venue: NeurIPS 2025
Accepted: yes
Keywords: Robust Safe Reinforcement Learning; Risk-sensitive Decision-making
Abstract: Safe Reinforcement Learning (RL) is crucial for achieving high performance while ensuring safety in real-world applications. However, the complex interplay of multiple uncertainty sources in real environments poses significant challenges for interpretable risk assessment and robust decision-making. To address these challenges, we propose Fuz-RL, a fuzzy measure-guided robust framework for safe RL. 
Specifically, our framework develops a novel fuzzy Bellman operator for estimating robust value functions using Choquet integrals. Theoretically, we prove that solving the Fuz-RL problem (in Constrained Markov Decision Process (CMDP) form) is equivalent to solving distributionally robust safe RL problems (in robust CMDP form), effectively reformulating the min-max optimization problem into a tractable CMDP with Choquet-integrated value functions. Empirical analyses on safe-control-gym and safety-gymnasium scenarios demonstrate that Fuz-RL effectively integrates with existing safe RL baselines in a model-free manner, significantly improving both safety and control performance under various types of uncertainties in observation, action, and dynamics.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Fuz-RL, a framework for safe RL that leverages fuzzy-measures to address coupled uncertainties by modelling super-additive effects, an aspect often overlooked by existing safe RL algorithms.\nThe authors propose learning uncertainty measures at varying levels of uncertainty using a neuro-fuzzy network. These learned measures are then integrated into a fuzzy Bellman operator enabling the estimation of robust value functions under potential perturbations. The paper demonstrates that solving the Fuz-RL problem is equivalent to solving a distributionally robust safe RL formulation. \nThe framework is incorporated into model-free safe RL approaches, and its effectiveness is evaluated across a range of environments within the SafeControl Gym. The results highlight that Fuz-RL often outperforms state-of-the-art baselines in terms of both safety and performance, particularly under action, observation, and dynamics uncertainties.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposes a novel approach that utilizes fuzzy measures to enhance robustness in model-free reinforcement learning (RL), aiming to balance performance with safety constraints under uncertainty. The authors demonstrate the capabilities of their framework across several benchmark environments.\n\nWhile the approach is novel and is proved to work well in practice, I find the proposed methodology unnecessarily complex, which raises concerns about the ease of implementation and the generalizability of the method to more complex settings. I add some specific concerns below:\n\n\n1) Complexity: The authors argue that existing risk-sensitive approaches require careful parameter tuning and struggle with multiple noise sources. While the later might be a valid concern, existing methods, for example [1], typically involve fewer parameters and are more straightforward to implement. Specifically, the complexity lies in the fuzzification process and the associated rule application. Why one needs to define 3 different fuzzy degrees (low, medium and high)? I wonder if it could potentially be replaced by a single neural network that directly outputs a continuous uncertainty degree. Although the authors argue that their approach avoids the need for multiple networks to handle uncertainty measures (line 197), the proposed approach requires: one feature extractor, 6 membership functions to obtain the fuzzy degrees (which according to Appendix C.2.1 are nontrivial to define and require a lot of hyperparameters), and K parameters for the rule's weight. Could the approach be made simpler?\n\n\n2) Justification for super-additivity: It remains unclear whether the environments used in the evaluation justify the need for a fuzzy uncertainty measure. Would a simpler, linear additive uncertainty model not suffice in such relatively simple (1D and 2D) settings? If the fuzzy approach provides a significant advantage, a justification would be appreciated.\n\n\n3) Definitions 4.1 and 4.2 are presented without any references. This may inadvertently suggest that these are novel contributions, when in fact they originate from prior work—e.g., Choquet (1953) and Hung et al. (2015) [2], if not earlier. Appropriate citations should be included to acknowledge this foundational work.\n\n4) Baselines: The comparative evaluation remains somewhat unclear. It is not evident how the proposed method is integrated with the presented baselines (CPPO or PPOL), which as far as I am concerned are already safe RL algorithms.  Why not integrate Fuz-RL with default PPO instead? This would provide a fairer assessment of the added value of Fuz-RL. Furthermore, the only comparison with RAMU is relegated to the appendix, and results suggest that the improvements offered by Fuz-RL over RAMU are not statistically significant in most of the environments.\nThere are other relevant baselines, particularly risk-averse distributional RL methods where risk measures are easily computed from learnt quantile value functions such as in [1] or similar, that would provide a more comprehensive evaluation of the proposed framework.\n\n[1] Urpí, Núria Armengol, Sebastian Curi, and Andreas Krause. \"Risk-Averse Offline Reinforcement Learning.\" International Conference on Learning Representations.\n[2] Hung T. Nguyen; Vladik Kreinovich; Joe Lorkowski; Saiful Abu.  Why Sugeno λ-measures.\" 2015 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). IEEE, 2015.\n\nQUESTIONS:\nQ1: In equation 8) I believe the term sum_i \\alpha_{R_i} in both numerator and denominator is redundant. Wouldn't it cancel out with the numerator?\nQ2: How do you enforce that the resulting measures in equation 8) satisfy the additivity condition (2) in Definition 4.1?\nQ3: Given the clipping in equation 8), how do you ensure you fulfil condition 1) in Def. 4.1, i.e. m({1,2...,K}) =1? As far as I am concerned, this condition is also used in Theorem A.1 in Appendix (line 422) to prove the contraction property of the fuzzy bellman operator.\nQ4: In equation 16, how are R_t and C_t estimated? Via Montecarlo sampling? Don't they suffer from high variance?\nQ5: What's the impact of the hyperparameter \\lambda_Theta in equation 16? An analysis on its impact would be useful.\nQ6: How is the cost function defined in each environment? I couldn't find an algebraic definition of the cost term in the appendix, which is needed in order to learn V_c and for the neuro-fuzzy network update.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a deep neuro-fuzzy control architecture for robust safe reinforcement learning (RL). It defines a fuzzy control problem, and proves its equivalence to the more commonly seen distributional robustness problem. Solving the fuzzy RL problem with the proposed framework incurs lower computational cost, and (generally) shows improved safety and cost performance compared to existing safe RL methods. Evaluation is carried out in low-dimensional continuous control tasks and a higher-dimensional grid generation control task.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. To my knowledge, this is the first work that achieves robust control of CMDPs using fuzzy logic. Using fuzzy logic to solve the robust CMDP problem setting is a novel idea, and the existence of a proof to show equivalence to traditional distributional robustness is interesting.\n2. The proofs are clearly laid out and I could not find any issues with them. However I have not previously worked with fuzzy logic, so those components of the proofs are not familiar to me and I may have missed something.\n3. Generally the notation and technical content flow is good, and the paper is easy to follow even for someone without a background in fuzzy logic. However I do have some issues with clarity and structure, which I will describe below.\n\n\n# Weaknesses\n\n## Setting/justification\n\nOverall there are some issues with the high-level justification and intuition for why fuzzy logic should be used for robust safe RL.\n\n1. Lines 31-34 are not very concrete, and don't cite anything to back up their claims. It's not clear what \"complex, non-additive interactions\" are, and why standard probability measures, or previous mentioned methods, cannot handle them. I'm also not clear what \"treat different perturbations with equal importance\" means (line 27).\n2. Section 4.1 is also a bit abstract and difficult to link to the CMDP/safe RL problem. It's not clear why/when super-additivity helps with uncertain dynamics, or where super-additivity originates in an uncertain system's behaviour.\n3. Figure 1 does not seem to be a good use of space, as it illustrates a simple-enough concept from equation (5).\n\n\n## Method clarity\n\nWhen introducing the method (line 104 onwards), it's not made very clear _why_ one would need to consider different levels of uncertainty within \\epsilon. From the rest of the paper, it seems like different levels of perturbation might have different impacts on cost and reward for some (s,a) pairs, but this could be spelled out earlier on. More text explanation would help explain why it is necessary to partition the uncertainty into different levels, as it only becomes clear later on in the paper.\n\nThe method explanation would be clarified by leading with a high-level description of how the method works. Currently, Section 4 launches directly into fuzzy measures and approximating these with a fuzzy NN. It's not clear until much later on how the fuzzy measure fits into a full solution method -- I only started fully understanding the approach when reading the fuzzy rule format in (7). Figure (2) is therefore a helpful diagram, but should appear earlier in the text in my opinion. It should also be referred to when describing the method, and should have more labelling to make clear what e.g. the x/y lines are in the fuzzification step.\n\n\n## Technical\n\n1. The specific choice of the number of fuzzy membership functions (Low, Medium, High) is not discussed or ablated.\n2. I cannot see any statement/discussion of the number of fuzzy rules $N$.\n3. I cannot see the feature extractor parameters anywhere in the formulation, whereas everything else is explicitly parameterised.\n4. The experiment metric \"AvgRisk\" is not very clearly defined (see question below)\n\n## Evaluation\n\nThe continuous control tasks in the evaluation are quite simple and low-dimensional compared to other recent safe RL papers. RAMU and OTP (Queeney, James, et al. \"Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guarantees.\" TMLR) evaluate on more complex domains (quadruped, walker). The library used in this work (Safe-Control-Gym) is now quite old, and more recent alternatives such as Safety-Gymnasium offer more complex environments.\nHowever, this paper's wide range of types of perturbations tested (observation white noise, control delays and impulses, dynamics model parameters and drift) is more thorough and wide ranging than most other safe RL papers I have read.\n\n\n## Small clarity / typo\n\n- Line 75 \"for RL community\" -> \"for the RL community\".\n- In Line 104, explicitly saying and italicising \"K distinct _uncertainty levels_\" would make it clearer that these are the uncertainty levels referred to later in Section 4.1.\n- Line 254 \"the SpinningUp\".\n\nQUESTIONS:\n1. Have you tried more complex environments than the ones in your evaluation? Are there any reasons that the method would not work in more complex environments with larger state, action or observation spaces?\n2. Did you try different numbers of membership functions other than 3 ({Low, Medium, High})? What are the trade-offs of varying this?\n3. How is the number of fuzzy rules $N$ chosen? Is it 3x3, i.e. every combination of {Low, Medium, High} for $z_c$ and $z_r$? If so, this should be made explicit in the formulation.\n4. The metric \"AvgRisk\" is defined as the \"proportion of constraint violations\" (line 264). Is this the proportion of episodes or timesteps? Is it discounted? Does a proportion measure <0.5 imply that the constraint is satisfied in the median? Even if it does, it wouldn't tell us whether the expected cost is below the constraint.\n\nI would say that question (1) is the most important for instructing my evaluation score, as clarity-related points could be fixed by some restructuring or citing other relevant works.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a highly practical problem: robust safe reinforcement learning under multi-source uncertainty. The authors propose a new fuzzy-guided robust framework, named Fuz-RL, and prove that solving the Fuz-RL problem — formulated as a constrained Markov decision process (CMDP) — is equivalent to solving a distributionally robust safe reinforcement learning problem formulated as a robust CMDP. Experimental results on safe-control-gym scenarios under various types of uncertainties in observation, action, and dynamics demonstrate the empirical effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths：\n1. This paper proves the equivalence between the fuzzy robust safe RL problem and the distributionally robust safe RL problem, and the proof appears to be logically sound and reasonable.\n2. The paper presents comprehensive experimental results demonstrating that the proposed algorithm achieves good robustness and safety. It also includes ablation studies and interpretability analysis of the fuzzy measure weights.\n3. I checked Theorem 5.3 in particular—they appear correct based on standard RL math, with straightforward algebra and clear steps. I didn’t find any obvious mistakes in the provided derivations.\n\n# Weaknesses：\n1. To improve clarity, it is suggested to restructure the paper, although the current presentation is not unacceptable. Specifically, it is recommended to move the content of Subsection 4.1 to Section 3, to make it clear that this part provides background and foundational concepts rather than the novel contributions of the paper. Additionally, Subsection 4.2 and Section 5 should be combined into a single section, serving as the presentation of the proposed method and theoretical analysis.\n2. The related work section should include more discussion on robust safe reinforcement learning approaches, such as those achieving robustness through robust model predictive control [1] or adversarial robustness [2]. The authors should also analyze how their proposed method differs from and improves upon these existing approaches.\n3. It would be helpful to include training curves that correspond to the performance metrics reported in Table 1.\n\n# References\n[1] Zanon M, Gros S. Safe reinforcement learning using robust MPC[J]. IEEE Transactions on Automatic Control, 2020, 66(8): 3638-3652.\n\n[2] Li Z, Hu C, Wang Y, et al. Safe reinforcement learning with dual robustness[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.\n\nQUESTIONS:\nPlease refer to the section \"Weaknesses\" for detailed information.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an algorithm that uses the concept of fuzzy logic to construct a CMDP framework to solve safe reinforcement learning problems. Fuz-rl can manage uncertainty quantification in a principled manner to perform robustly in Safe RL. The authors demonstrate their fuzzy safe RL framework is equivalent to the distributionally robust safe RL framework, but does not need to explicitly solve a min-max which can otherwise be computationally expensive and result in conservative policies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper is well-motivated and apart from the math, the paper is generally clear.\n- The paper proposed a generally rigorous approach to uncertainty quantification that is more efficient than techniques like cvar.\n- While fuzzy rl is not necessarily new, the theoretical approach taken and the hurdles overcome for practical implementation in the paper seems novel.\n\nWeakness:\n- Table 1, the main result, is extremely confusing – there are up and down arrows indicating whether or not performance improved, bolded numbers indicating best results, but then there are also contrasting green-red colors. I’m not sure what is considered good results\n- The experiments are quite limited to basic environments: cartpole and quadrotor. What is the observation space dimension of the quadrotor env? Can the approach scale to more complex environments? I’m not suggesting necessarily to run more experiments on these environments per se, but rather the limited experiments suggest Fuz-RL has limited scalability.\n- perhaps it would be useful to show a trajectory that demonstrates where exactly fuz-rl can interpretable provide benefit as opposed to the baseline algorithms? simply saying the results are better in table 1 seems hard to understand -- also many times the avg risk only drop by an order of 0.05 which is not too significant considering the variance in each result are compartively large.\n\nQUESTIONS:\n- Does Fig 4 indicate that there is some optimal number of uncertainty levels? K seems like another parameters I must tune and the performance seems to be heavily sensitive on this hyperparameter.\n- Relatedly, in line 105, how do you determine $\\epsilon_0, … \\epsilon_K$? Are there more hyperparameters?\n- If min-max approach causes ``overly-conservative policies” and Them 5.3 suggests the fuz-rl is equivalent to the distrubtional robust min-max framework, then why would fuz-rl not have equally ``overly-conservative policies”?\n- Could you comment on the generally computation overhead the fuz-rl approach provides? I see the computational analysis, but how much more memory and time does this approach take in training as opposed to vanilla PPO/CUP/CPPO?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}