Uncertainty-aware Constraint Inference in Inverse Constrained Reinforcement Learning
Authors: Sheng Xu
Guiliang Liu
Venue: ICLR 2024 poster (2024)
Keywords: Inverse Constrained Reinforcement Learning
Constrained Reinforcement Learning
Inverse Reinforcement Learning
Uncertainty Modeling
Primary area: reinforcement learning
TL;DR: We proposed Uncertainty-aware Inverse Constrained Reinforcement Learning (UAICRL), a novel ICRL framework that models both the aleatoric and epistemic uncertainties towards uncertainty-aware constraint inference.

Abstract: Aiming for safe control, Inverse Constrained Reinforcement Learning (ICRL) considers inferring the constraints respected by expert agents from their demonstrations and learning imitation policies that adhere to these constraints. While previous ICRL works often neglected underlying uncertainties during training, we contend that modeling these uncertainties is crucial for facilitating robust constraint inference. This insight leads to the development of an Uncertainty-aware Inverse Constrained Reinforcement Learning (UAICRL) algorithm. Specifically, 1) aleatoric uncertainty arises from the inherent stochasticity of environment dynamics, leading to constraint-violating behaviors in imitation policies. To address this, UAICRL constructs risk-sensitive constraints by incorporating distributional Bellman updates into the cumulative costs model. 2) Epistemic uncertainty, resulting from the model's limited knowledge of Out-of-Distribution (OoD) samples, affects the accuracy of step-wise cost predictions. To tackle this issue, UAICRL develops an information-theoretic quantification of the epistemic uncertainty and mitigates its impact through flow-based generative data augmentation. Empirical results demonstrate that UAICRL consistently outperforms other baselines in continuous and discrete environments with stochastic dynamics. The code is available at https://github.com/Jasonxu1225/UAICRL.

Published as a conference paper at ICLR 2024
UNCERTAINTY-AWARE CONSTRAINT INFERENCE IN IN- VERSE CONSTRAINED REINFORCEMENT LEARNING
Sheng Xu
Guiliang Liu"
School of Data Science
School of Data Science
The Chinese University of Hong Kong, Shenzhen shengxule @ llink. cuhk. edu.cn cn
The Chinese University of Hong Kong, Shenzhen 1 liugui i ange cuhk. edu.
ABSTRACT
Aiming for safe control, Inverse Constrained Reinforcement Learning (ICRL) considers inferring the constraints respected by expert agents from their demon- strations and learning imitation policies that adhere to these constraints. While previous ICRL works often neglected underlying uncertainties during training, we contend that modeling these uncertainties is crucial for facilitating robust con- straint inference. This insight leads the development of an Uncertainty-awan Inverse Constrained Reinforcement Learning (UAICRL) algorithm. Specifically, 1) alcatoric uncertainty arises from the inherent stochasticity of environment dynam- ics, leading to constraint-violatin behaviors in imitation policies. To address this, UAICRL constructs risk-sensitive constraints by incorporating distributional Bell- man updates into the cumulative costs model. 2) Epistemic uncertainty, resulting (OoD) samples, affects
from the model's s limited knowledge of the accuracy of step-wise cost predictions. To tackle this issue, UAICRL develops an cnoomation-thhoreee quantification of the epistemic uncertainty and mitigates its impact through flow-based generative data augmentation. Empirical results demonstrate that UAICRL consistently outperforms other baselines in continuous and discrete environments with stochastic dynamics. The code is available com www.////wwww.//./oooooooooo /AAICRL
INTRODUCTION
Reinforcement Learning (RL) is effective technique for solving sequential decision-making problems, typically focusing on maximizing cumulative rewards. However, recent studies (Liu al., 202T Satija al., 2020 Yang et al. 2023) have argued that, achieve safe control, the optimal policy must adhere to the underlying constraints in the environment. For instance, in an open-road environment, an autonomous driving policy must comply with traffic rules and social norms. Explicitly specifying these constraints is challenging. A more practical alternative is to infer the constraints from expert demonstrations by analyzing the behavioral patterns of expert agents.
To achieve this goal, Inverse Constrained Reinforcement Learning (ICRL) MMalik et al.) (021) extends the Maximum Entropy Inverse Reinforcement Learning (MEnt[RL) framework (Ziebart al., 2008) to infer constraints (rather than rewards) from expert demonstrations. ICRI alternates between Constrained Reinforcement Learning (CRL) and Inverse Constraint Inference (ICT) until the imitation policy can reproduce the expert demonstrations. During the process, traditional ICRL algorithms
(Scobee & Sastry] 2020 Malik et al., 2021 Gaurav et al. 2023 Liu Zhu 2022 Qiao et al, 2023) often assume deterministic training environments, without considering the influence of underlying uncertainties. Specifically, in the CRL phase, stochastic transition functions introduce aleatoric uncertainty, thereby influencing the policy update in CRL and may result in constraint-violating behaviors. In pursuit of safety control, human experts often exhibit risk-averse behaviors, so an imitating agent cannot accurately replicate expert demonstrations unless formulates a risk-sensitive policy. In the ICI phase, due to the finite size of the training data, epistemic uncertainty arises when game contexts lie outside the data distribution, leading inaccurate cost predictions.
To achieve uncertainty- constraint inference, Liu et al. (2033) Papadimitriou et al. 2023) proposed modeling the posterior distribution of step-wise constraints using variational inference and
"Corresoondigg author: Guiliang Liu, Seggullian@@cuh..oom edu.en

---

Published as a conference paper at ICLR 2024
Monte Carlo sampling. McPherson al. (2021); Baert et al. (2023) incorporated maximum causal entropy likelihood into ICRL. Although causal entropy and constraint distribution are sensitive to aleatoric and epistemic uncertainties respectively, none of the previous methods can handle both uncertainties.
In this paper, introduce the Uncertainty-aware Inverse Constrained Reinforcement Learning (UAICRL), a novel ICRL framework that models both the aleatoric and epistemic uncertainties for achieving robust constraint inference. Figure provides an overview of UAICRL. Specifically,
1) We design a risk-sensitive constraint by modeling the distribution of cumulative costs and distorting this distribution to represent risk measures. The predicted distribution is sensitive aleatoric uncertainty and we empirically justify these findings in our experiments. To enable efficient risk- sensitive control in continuous spaces, we propose the Distributional Lagrange Policy Optimization (DLPO), which incorporates distributional estimation and Lagrange mechanics into the classic Proximal Policy Optimization (PPO) (Schulman al. (017) for constrained policy optimization.
2) We introduce a mutual iottuaaiiooommtiin driven metric (Gabrié et al., 2018) to quantify epistemic uncertainty constraint inference and propose an information-theores ICI objective to mínimize the impact of epistemic uncertainty when updating the constraint function. A key technique to achieve this objective involves augmenting the training data using the proposed Flow-based Trajectory Generation (FTG) algorithm. FTG effectively generates diverse set of trajectories based on the dataset and musk-dependent rewards, thus reducing the influence of OoD state-action pairs in constraint prediction.
Empirical evaluations show that UAICRL consistently achieves higher feasible rewards and lower constraint violation rates in stochastic environments with both discrete and continuous state spaces, outperforming other ICRL baselines. For a comprehensive evaluation, conduct in in-depth studies to individually assess how effectively UAICRI. handles the aleatoric and epistemic uncertainty.
PROBLEM FORMULATION
Constrained Reinforcement Learning (CRL). To solve a CRL problem, the agent optimizes the control policy under a Constrained Markov Decision Processes (CMDPs) M which can be defined by tuple (S, A. PT. PR.P.rPP C. Mo. Y.T) where: 1) S and A denote the space of states and actions. 2) pT(s (s, a) and (r(ss, a) define the transition and reward probabilities. 3) Pe(c]s, a) and e denote the probability of cost and the associated bound. 4) /4n defines the initial state distribution. 5) y E (0, 1) is the discount factor and T defines the planning horizon (T = oo principle). The goal of the CRL policy ı is to maximize expected discounted rewards under the constraint:
T
arg max Er PT PR.
Σ (rr((s), 2m
ai) s.t. Ex.pr Po.
E>to(se Se
(I)
Note that traditional CRL problems often assume the constraint signals are directly observable, but in real-world problems, such constraint signals are not readily available, and we must infer these constraints from the environment by solving the following inverse problem.
Inverse Constraint Inference (ICI). ICRL algorithms typically assume that reward signals are observable and the goal is to infer the constraints. Inspired by Malik et al. (202T). we define & as Bernoulli feasibility variable that takes two values {Ø+ Φ-} such that p o( [s, (; w) (w denotes model parameters) quantifies to what extent performing action a in the state s is feasible while p(o [s, a; w) denotes the probability this movement is infeasible. For clarity, in the rest of this paper, we denote o+ by é. Accordingly, the step-wise cost is then defined as (s, a) = 1 - p(0)s, o; w). Under these definitions, the likelihood of generating the expert dataset De can be represented as:
N
2
1
II exp [(()(m))] MM (T(")) n=1
(Z
where 1) MM denotes the CMDP with constraint function cw. 2) denotes the number of trajectories in the expert dataset, 3) the normalizing term E EMa = exp [r(T)] 1M' (r)dr, and 4) the identifier MS (m)) can be defined as (m)) w) = M p()) (n) at ww). By substituting them to Equation we can update by the gradient of the log- likelihood function (Malik et al. 2021
N
V. log (p [))
Σ log(p(o)s (n) _a(t)) [w]]]
(n)
NE NE
V. Σ
loglp( (e;
2

---

Published as a conference paper at ICLR 2024
where f is sampled from the current imitation policy xM Intuitively, our goal is to differentiate the trajectories generated by expert policies and imitation policies that may violate the constraints. When the imitation policy under the learned constraint model cu matches the expert policy, this gradient goes to O, and the update stops.
Modeling Uncertainty in ICRL. An ICRL agent iteratively solves the CRL and ICI problems until the imitation policy can re- produce expert demonstrations. However, existing ICRL algorithms typically assume deterministic setting without consider- ing uncertainties, but we argue that the aleatoric and epistemic uncertainties can emerge and significantly affect the training and the convergence of ICRL. To address them, we propose the 1) Distributional La- grange Policy Optimization (DLPO) algorithm with a risk-sensitive constraint by modeling the distribution of the cumulative costs (Section B) and 2) Data- Data-augmented Constraint Inference with flow-based trajectory generation for constraint inference from limited demonstrations (Section Figure T shows the flowchart of UAICRL framework.
Forward Step
Samping Nominal Trajectories (+)
Inverse Stop
Distributiona Lagrange Policy Opimiiztion (Obj. 4]
Flow Matching (Ob, 10] Pa(()) = Feur(s)
T (model aleetorii uncerteinty)
Expert Trajectories 2, - (tc)
Flaw-based Trajectory Generation
Risk-sensitive Constrain
Pal _ y As
Dj and
Data-augmneried Comstraint Inference [Obj. Cu ci'
Distributional Bellman Equation (Obj. 5] (*)+ + C x2
Constraint Model
(model epistemic uncerlainly)
Figure I: The flowchart of UAICRL.
POLICY OPTIMIZATION WITH RISK- SES--SENITIVE CONSTRAINTS
To address the alcatoric uncertainty, risk-sensitive RL commonly incorporates the risk measure into the optimization objective, for example, by constructing arg max, Pa 1 I'O A:) where O refers to the objective, a E [0, 1] defines the confidence level and denotes the risk measure. This approach imposes significant costs on the policies that may lead to hazardous consequences, even the likelihood of such events is low (e.g., a pedestrian being hit by a car on the highway). Although safety is important, being too conservative can harm performance and hinder reward accumulation (Mihatsch & Neuneier 2002: Choi et al. 2021
Risk-Sensitive Constraints for Policy Optimization. In this work, we present a novel approach that disentangles the unified objective into two components: rewards and costs, where rewards indicate the desired outcome, while costs account for negative consequences. For example, in autonomous driving, rewards might correspond to how quickly car reaches its destination, while costs capture the need adhere traffic rules and safety constraints during driving. To drive risk-sensitive policy, the risk incurred by aleatoric uncertainty can be incorporated separately from reward optimization. We formulate the trade-off between rewards and costs as a constrained optimization problem:
arg max E..pp PR. Σ ( (7tr(s),))) + B (>([((((((sss)) s.t. Pex Σ CC((S,, se (4)
IT
where H[[((a (ag refers to the causal entropy (Ziebart et al. 2010) and C(-) denotes the random To specify the risk measure, we define the trajectory-generating probability as y² (7) = (=o)) IT=i0
variable of state-action cost |
(8++1 or), where aleatoric uncertainty accu- mulates in sequential decision-making due to the inherent stochasticity in the initial state distri- bution (0())) policy n, and transition function pT pT(8441 (sc, dt)- We define the corresponding risk envelope 2 {Sa r + [0, alll ((T)p" (r) = 1} be a compact, convex, and bounded set, based on which the risk measure can be induced by the distorted probability dis- tribution P Ç p" For example, the Conditional Value-at-Rie aissk (CVaR) can be defined as P. P.IE. CC,] SuPC, eug Byapp (T) Etoo". /C.).
Distributional Estimator for Cumulative Costs. The difficulty of constructing the distorted probability- based risk measure lies in the unknown distribution of the cumulative costs. To estimate the distribution, we define the variable of discounted cumulative costs as (s(s:) = Σi- -C, = 8;. Based on Luo et al. (2022). we use N supporting quantiles parameterized with the rational- quadratic function to represent Zğ. where 0 denotes model parameters. Then we can update Z through quantile regression (Bellemare et al. 2017).
Throughout this paper, use uppercase letters (e.g., C) to represent random variables and lowercase letters (e.g., c) to represent instances of variables. For brevity, we use C. shorthand of C(S,, At).
3

---

Published as a conference paper at ICLR 2024
During training, when the agent performs action ~ ((-1s) state the agent receives cost c ~ Pe (-|s, a) and moves to a future state s' ~ pr (-sssss. a). This stochastic process can be captured by the distributional Bellman equation (state-form) (Gerstenberg al. 2023):
Z8(s) = J JaEA
[ PT(s' s, Pe(c|s, (be,))
T(als)
() ) (s')) da ds' de (5)
J'ES
JOEC
where the pushfoward operation (be,)) # involves shifting and scaling the distribution by e cand y. For brevity, we denote the distributional Bellman equation as (s) C(s. 7Z")". We show that the predicted distribution is sensitive to the alcatoric uncertainty.
Proposition 1. (Liu et al. 2022). The key components for representing the aleatoric uncertainty can be captured by the distributional Bellman equation under the measure of entropy.
The detailed proof in Appendix A.I
Leveraging the aforementioned policy optimization objec-
tive (4) and distributional estimator (5) we design the Distributional Lagrange Policy Optimization (DLPO) algorithm to learn the policy under risk-sensitive constraints. The implementation details are shown in Algorithm 2 in Appendix
B.
4 CONSTRAINT INFERENCE WITH FLOW-BASED DATA AUGMENTATION
Epistemic uncertainty arises due to the limited training data and the model's lack of knowledge about Out-of-Distriuttion (OoD) data. A common measure of epistemic uncertainty is the mutual information (w: xe, D) (Smith & Gal. 2018; van Amersfoort et al., 2020), which quantifies the amount of information gained by the model w when it observes the true label y for a given input æ. The greater the uncertainty of the model regarding the data, the more additional information it can obtain once the true label y is observed.
Intuitively, epistemic uncertainty arises when the constraint model w is required predict the cost of [T; w)), which is generated by exploratory behaviors during policy updates. To reduce the effect of epistemic uncertainty, we need to minimize the mutual information I (w; L|T, D), which can be represented as follows:
a trajectory F that locates Ool) of training data (i.e., predict c(f) = 1 p p(øl7;
Proposition 2. Let D denote the training dataset consisting of expert trajectories {Te} and imitation trajectories { Let 7 denote an OoD trajectory. Let w denote the constraint model parameters and q(w) denote the dropout distribution. [wu; $|F, DD can be empirically represented by:
H[p( D)]
M Σ Hlp(0// F; wm)) where Win ~ ((w)
The proof is in Appendix A.2
Specifically, 1) Hf D)]
o0) measures the amount of
information required to describe the feasibility & an exploratory trajectory based on the given training dataset D. 2) H[p( Wm. defines the entropy of constraint model parameterized by wm. To reduce the epistemic uncertainty, we integrate them into our ICI objective as follows.
4.1 DUGA-NUGE INVERSE CONSTRAINT INFERENCE
To reduce the impact of epistemic uncertainty, we can augment the conventional ICT objective (Obj. 3 with the mutual information term I (w: / D). Since [ww; 1 D) is intractable, based on Proposition
2
we instead maximize the following objective with its empirical representation:
(7)
1
Σ
IE
D.
log(p(dlst wm)]] E.
Σ t=
log(p(ølée wm))]] a H( D)] H[p( 4 (7; wm)))
M
1=0
where a controls the trade-off between the likelihod and mutual information, and we use dropout layers (Srivastava al. 2014) for q(w) by following Smith & Gal (2018). Besides, the conditional entropy (), is independent of w. which reaches zero (its minimum) when p( F, D) = p(D), indicating that the dataset has already recorded the trajectory and its feasibility (i.e., (
D). To achieve the goal, we expand the dataset by generating trajectories {(OC, -C) obtaining the augmented expert and nominal dataset DC and DG By substituting them to objective we get the following data-aug constraint inference objective:
M
E
loglp(olsi, at; wm)]]
Ea
Σ loglp(0lse شe wm)]]
Halp (7;,))
(8)
M
We propose an approach that can generate diverse set of expert and nominal trajectories as follows.
4

---

Published as a conference paper at ICLR 2024
4.2 FLOW-BASED TRAJECTORY GENERATION
We propose a Flow- Trajectory Generation (FTG) algorithm to perform conditional generation for maximizing p( F Unlike the traditional trajectory generation with policy rollout, FTG learns the transition densities from the trajectory dataset D. Morcover, the generation is guided by the feasibility +- {Ø, $ of trajectories, rather than the geward-maximizing policy.
To learn FTG, we define the non-negative trajectory flow function F T R+ where T denotes the
trajectory set. Intuitively, Frr) quantifies the mass of particles (Bengio et al. 2021) passing through T, and denser particles indicate a higher probability of generating the trajectory. Under this setting, the
state flow F(s) is the integral of trajectory flows passing through this state: F(st) = Sra., F(T)dT.
The transition probabilities can be defined as (=($t=1(ss) ((()))11)) where F(st + St+1)
F(*+)
Ir=(_..ee-tte (T) embeds the flow passing through the action a, The trajectory generation
probability (T) IT PF (st+1(sz). In order to generate { , the generated trajectories r must correspond to specific db. This is achieved by a careful design of the reward function. The reward functions for generating the expert ( = Φ)) and nominal trajectories ($b x)) can be set
1, 1= A Te
11, t= T= 8 E and R(ss)) = [0, otherwisc
Re (s))
0, otherwise
To generate expert trajectories, we assign positive reward (+1) to the terminated state of the recorded expert trajectories Ta D. and Os to others. The same approach is applied for generating nominal trajectories. We train two flow functions Fe and F by utilizing (s,) and R(s)) respectively.
Learning Flow Functions. A common approach to updating the flow function F-) is flow matching: for a state 8, E S, the inflows (u:T(s.a)=s. F(s, a)da must match outflows Sae.A F(se, a)da. To
enable flow matching in continuous environments, we sample M actions independently and uniformly from the continuous action space A. These actions serve as an approximation of flows that can be utilized to train the flow network. The resulting loss function is approximated as Li et al. (003)):
(10)
M
M
(e(T)
Σ
{
log
C+
exp Fe am))) ]] log C AR(s)) +
Σ m=1
exp Fe am)
}
2
1
where: 1) denotes the parameters of the flow network F(-). 2) G(-) is retrieval neural network with (St+1, at) as the input and 8 as the output (i.e., to predict the parent of a state), which can be trained by MSE Loss based on given trajectories. 3) >M Fg (S,, am) is an approximation of flows with M sampled actions. e can balance the use of small and large flows, thereby preventing numerical problems when computing the logarithm of extremely small flows.
Trajectory Generation. FTG aims generate trajectories G (So, ao.-.. 8T, aT) by starting from the initial state so and iteratively sampling the action 0, based on the scale of the flow F(st, az). thereby transitioning to the next state sc+1- In this process, actions with larger flow will be sampled with higher probabilities. To label 79, we set & to & if the generation is based on the expert flows (i.e., Fe, learned with R2), and o for nominal ones. The detailed implementation of FTG is shown in Algorithm 3 in Appendix [.I] Leveraging the modeling capability of the generative model, FTG learns intricate feature patterns of input data, generating trajectories that resemble the training data in the latent space, even if they are absent in the original state-action space. The incorporation of this data significantly mitigates the impact of epistemic uncertainty during constraint inference.
Building upon the aforementioned ideas for modeling alcatoric and epistemic uncertainties, Algo- rithm presents the whole UAICRL algorithm.
RELATED WORKS
Learning constraints from demonstrations. Given the demonstrations, prior works mainly focused on inferring implicit constraints by determining the permissibility of actions under specific states. In discrete state-action spaces, Chou et al. (2020b); Park et al. (2020) learned constraint sets to differentiate feasible from infeasible state-action pairs. Scobee & Sastry (2020) proposing inferring the constraint set under the maximum entropy principle, and McPherson et al. (2021); Baert et al.) (2023) extended it to stochastic environments using maximum causal entropy (Ziebart et al. 2010). In continuous domains, the goal is infer boundaries between feasible and infeasible state-action pairs.
5

---

Published as a conference paper at ICLR 2024
Algorithm 1: Uncertainty-aware Inverse Constrained Reinforcement Learning (UAICRL)
Input: Expert dataset De = {Te}. augmented data size L. Initialize the cost model Cw. policy Tg,, expert flow F. and nominal flow P'g: Update F with De by minimizing the flow loss (10)
do
Update Ty based on the risk-sensitive CRL objective (4) and distributional estimator (5) Sample nominal trajectories D { with xe in the environment, and update F'g with D by
the flow loss (10);
Initialize the augmented expert and nominal datasets D = DD and Do = D: for i 1,2,................... do
Generate expert and nominal trajectories T.. and 40 with F5 and Fg;
Add the generated trajectories to datasets: D = D. U UT and DO = DG Ufc;
end
Update the cost model C. with the ICI objective 8 based on D. and DC:
while D do not match De:
Malik et al. (2021); Gaurav et al. (2023); Qiao et al. (003) used neural networks to approximate constraints. Some recent studies (Liu et al. 2023 Chou et al. 2020a Papadimitriou et al. 2023) ap- plied Bayesian Monte Carlo and variational inference to infer a posterior distribution of constraints in high-dimensional state space. These constraint distributions can only model the epistemic uncertainty.
Uncertainty-Awan Reinforcement Learning. Incorporating uncertainty awareness in RL algo- rithms is essential for efficient exploration and control, with numerous downstream applications (Hoel et al. 2023 Liu et al. [022] Wu et al. (023). Several existing works study defining uncertainty measures RL environments, primarily using dropout layers (Chen al., (017) ensemble models
(Lttjens et al.
2019; An et al. [02T), such as Bootstrapped DQN methods (Osband et al. 2016 2020) and their offline extensions (Kumar et al., 2019 An al. 2021p. Another
Da Silva et
approach to measuring uncertainty is through Distributional RL (Luo et al., 2022 Bellemare et al. et al., 2018 Mavrin al. 2019), which directly models the distribution of future
returns with distributional Bellman operator. Besides, Risk-sensitive RL (Mihatsch & Neuneier 2002, Chow et al., 2015 Prashanth et al.] 2022 Ni & Lai [022 Luo et al. 2023) is indeed a closely related research area, in which agents employ risk measures, such exponential utility, variance, and Value-at-Riss Risk, to develop risk-aware aware policies. In contrast to prior works that apply risk measures for rewards maximization, work builds upon CMDPs and emphasizes capturing the cost uncertainty.
EMPIRICAL EVALUATION
In this section, we empirically evaluate the effectiveness of UAICRL by answering the following questions: 1) How well does UAICRL perform in discrete (Section 6.1 and continuous (Section 6.2 environments under varying degrees of stochasticity? 2) Can UAICRL handle more complex, real- world highway driving scenarios (Section 6.3)? 3) How well do the key components of UAICRL (DLPO and FTG) address the aleatoric and epistemic uncertainty, respectively (Section 6.4b?
Comparison Methods. Our experi-
Table 1: Baseline methods for constraint inference.
ments include the following baselines:
Generative Adversarial Constraint
Method
Continues
Space
Constraint Optimization
Maximum
Entropy
Alcatoric Uncertainty
Uncertainty
Epistemic
Learning (GACL) extends Genera- tive Adversarial Imitation Learning
GACL B2CL ICRL
X
X
(GAIL) (Ho & Ermon] (016) and em- ploys a discriminator D(s, a) to dif- ferentiate validate state-action pairs from infeasible ones. The (s.
VICRL
UAICRE-NRS
UAICRI-NDA
is directly appended to the reward function as penalization. 2) Binary Classifier Constraint Learning (BC2L) employs a binary classifier as the constraint model without utilizing the Maximum Entropy (MEnt) framework. 3) Inverse Constrained Reinforcement Learning (ICRL) MMalik et al. 2021

---

Published as a conference paper at ICLR 2024
follows the MEnt framework with risk-neutral constraint. 4) Variational ICRL (VICRL) (Liu ct al. 2023) captures epistemic uncertainty in constraint inference using Beta distribution based on ICRL. Additionally, perform ablation studies where 5) UAICRL-NRS removes the risk-sensitive constraint from UAICRI. and 6) UAICRI--ND removes the data augmentation from UAICRI.. summaiiss these methods.
Experiment Settings. We conduct empirical evaluations utilizing an ICRL benchmark (Liu et al. 2023), and extend
include stochastic dynamics by incorporating noise into transitions. Fol- lowing Malik et al. (02T). evaluation metrics include: 1) Constraint Violation Rate measures the probability that policy violates constraint in a trajectory. 2) Feasible Cumulative Rewards calculate the total rewards obtained by the agent before violating any constraints.
6.1 DISCRETE ENVIRONMENT: STOCHASTIC GRIDWORLD
We construct three Gridworld envi- ronments with different constraints, where the agent's objective is to nav- igate from starting location to tar- get location while avoiding the added constraints, as shown in Figure 2 The environment exhibits a certain degree of stochasticity, where, with specific probability (p. = 0.01 and 0,001), the environment receives a random action instead of an agent's is action. To be compatible with the environment set- tings, we utilize the discretized implementation of UAICRL by following Liu et al. 22033 Malkin ct al. 2022). Appendix [B.2] shows the implementation details.
ausdewarte Setting
Gridworld Satting
Gridwarté
1
Figure 2: Three different Gridworld settings with blue, red, and black markers indicating the starting, target, and con- strained locations respectively.
Table shows the evaluation perfor- mance, where bolding denotes the best results (highest rewards or lowest violation rates) in each setting. Check Appendix D.I for training curves. We find that UAICRL generally outper- forms other methods with lower con- straint violation rates and higher re- wards, while GACL and ICRL can hardly work well. BC2L and VICRL achieve relatively satisfactory results in the first two settings but perform poorly in the third setting.
Table 2: Evaluation of different methods in three Gridworlds with the random rate ps 0.01 and 0.001.
1..t
[1:01]
Grdwoudd
Galdwoedd
Gralwerd
Gidwwcld
Gribwerld
Grddwald
Setting
Setting
Setting 3
Setting
Sening
Sciting
BCI.
0:451
0,716
011.25
07641
01601
0,P2
Rewards
Pousible
VICRL
GACL
ICRL
0.12
0.244
0.337
0.310
0.109
0.532
0.033
0,051
0,000
0.011
0.356
0.778
0.368
0.610
0.070
0,000
0,089
0,070
UAICRL
0.650
0.683
0.35y
0.797
0.739
0
BCLL
0.33
0.19
0.53
0.29
0.27
0.32
Constrint
GACL
0.43
0.29
0.76
0.67
0.11
0.54
Violation
ICRL
0.53
0.33
0.63
0.36
0.27
6.73
Ruse
VICRL
0.35
0.35
0.45
0.39
0.26
0.53
UAICRL
0.13
0,89
0.34
0.09
0.77
0.38
6.2 CONTINUOUS ENVIRONMENT: STOCHASTIC MuJoCo
We utilize five MuJoCo environments in the benchmark (Liu et al. 2023) and additionally incorporate Gaussian noise into the transition function as pr (st+1 at) a²) = f (st. ae) + (N,,, σ). Each experi- ment is repeated with four random seeds, over which the mean standard deviation (std) results are reported. More details are shown in Appendix C.2
dissplayss the training curves in stochastic MuloCo environments with noise a = 0.1 (check Ap- pendix D.2 for results with a = 0.01 and 0.001). We observe that UAICRL-NRS and UAICRL-NDA can generally outperform the baselines in most environments, which underscores the effectiveness of employing data augmentation for modeling epistemic uncertainty and utilizing distributional estimator for capturing aleatoric uncertainty. By modeling both uncertainties, UAICRL generally obtains high feasible rewards (bottom row Fig. B whilst having a low constraint violation rate (top row Fig. 3}. This demonstrates UAICRL's robustness and its superior performance compared other methods. Note that although the reward shaping method, GACI, exhibits the fewest constraint violations in the Blocked Ant environment, it struggles to optimize rewards and underperforms in other environments. This underperformance is primarily attributed to the estimation errors encountered during the early stages of training. Interestingly, we find that increased stochasticity does not necessarily result in poorer model performance. This is because larger noise levels are more readily detectable, leading models to exhibit sensitivity to the risk of constraint violations. For a more comprehensive analysis of these phenomena, we provide more detailed discussion in Appendix
7

---

Published as a conference paper at ICLR 2024
Blocked Half Cheetah
Blocked Ant
Blocked Walker
Blocked Swimmer
Biesed Pendulumn
60
Contoeet Consoent Vrdwton
130
E40
06
62
800
682
2.0
075
2099
3000
Exledd
Exiend
Sphccs
Blocked Half-Cheetah
Blocked
Blocked Walker
Blocked Swimmer
Blased Pendulumn
3000
Feeebbe Rrwarrds
1523
33
1003
300
82
2233
6X0
109)
BKC
2830 ED0 SID
XC30 1336 (603) BACS
2330
RE5 1030
1506
IIEES BOCG IRE30
BC2L
GACL
ICRL
VICRL
UAICRLNRRS
UAICRL-NDA
UAICRL
Figure 3: The constraint violation rate (top) and feasible rewards (bottom) in five MuloCo environ- ments during training with stochasticity of (0, 0.1).
6.3 REALISTIC ENVIRONMENT: STOCHASTIC HIGHWAY DRIVING
We conduct experiments realistic high-
HighD Velocity Constraint
HighD Velocity Constraint
dimensional Highway Driving (HighD) en- vironment Krajewski al.) 2018 Liu
Sonooua Son Fac
сt al. 2023), which defines a complex high- way driving task with constraints and re- quires the agent to drive safely the desti- nation by observing human drivers' demon- strations. In this paper, focus on the ve- locity constraint, which guarantees the ego drive at safe velocity. Additionally, we add Gaussian noise to the agent's action a, to introduce the control-leve stochastic- ity. We report both the constraint violation rate (left, Figure 4 and feasible rewards (right, Figure under noise (0, 0.1) (check Appendix DD33 for complete results with (000, 0.01) and (0,, 0.00T)). Additionally, Appendix(D44 (D.4/4..swwwww the results of recovering multiple kinds of constraints. We find that although GACL violates the fewest constraints, is too conservative to pursue rewards, which limits its practical application. Other baselines including BC2L, ICRL, and VICRL can obtain high rewards, but their constraint violation rates are still relatively high. Instead, UAICRL can achieve the highest rewards while maintaining satisfactorily low constraint violation rate, indicating its great potential to handle uncertainties even complex environments.
-20
1030
1000
Epirads
Eplodd
BCZL
GACL
ICRL
VICRL
UAICRL
Figure 4: Model performance in the HighD environment with velocity constraint under (O 0.1).
6.4 IN-DEPTH STUDY ON UNCERTAINTY MANAGEMENT PERFORMANCE
To comprehend the critical factors contributing to uncertainty awareness, we study the model's capabilities of representing aleatoric and epistemic uncertainties from the following perspectives:
"Can DLPO lead to more robust control der stochastic dynamics?" To address this question, we provide each agent with the ground- truth constraints in the environment. We focus solely on whether DLPO can learn constraint- satisfying behavior while being influenced by aleatoric uncertainty. We report both the con- straint violation rate (left, Fig. 5) and feasible rewards (right, Fig. 5 in the Half-Cheetah envi- ronment with stochasticity of (0, 0.1) (check Appendix D.2 for complete results). The com- pared methods include: 1) Neutrall and DLPO-Averse, which refer to the options of ap- plying expectation and CV.R().25) as the risk measures (p) in Objective 2) Classic PPO-Lagrange,
Blocked Half-Cheetah
Blocked Half-Cheetah
1400
Comeraint Violatio Ralie
Feseitii Rewards
1200
10000
000
0.
m0)
02
2000
4000
2300
4320
PPO-Lag
DUPO-Neutral
DLPO Awerse
Figure 5: Model performance in the Blocked Half-Cheetal environment with stochasticity of (O, 0.1) when given ground-truth constraints.
8

---

Published as a conference paper at ICLR 2024
following the implementation in Liu et al. 2023). Our findings indicate that by implementing the risk-sensitive constraint, the policy learns to respect the constraints while collecting rewards. Furthermore, in most cases, the DLPO-Averse method employing CVaR the risk measure results in fewer constraint violations compared to the risk-neutral approach. However, CVaR sometimes leads to overly conservative behavior, hindering the agent from efficiently pursuing rewards.
Based on the same motivation, we train agents in Gridworld environ-
PPOl.g
DLPO
Prodicudd Constraber Dleniburiee
6-
5-
5
ments under known constraints to bet- ter understand the advantages of the distributional estimator. Fig. 6 visu- alizes the trajectories generated by the PPO-Lagrange (left) and DLPO (right) in a Girdworld setting (check Fig. D.3 for results in remaining set- tings). We notice that DLPO can maintain appropriate distance from the constrained locations, effectively mitigating the impact of noise. Specifically, at s 4), PPO-Lag chooses to move left, whereas DLPO decides to move upper left, maintaining a larger gap to the constraints. We use red circles to highlight the critical locations, where the agents display different actions, and visualize the predicted cost distribution at these locations. We find that these distributions are sensitive to the choices of actions, consequently assigning larger expectations and variances to actions with higher risk of constraint violation (e.g., moving lower left at s44 4) in Fig. 6)
TM
F-
671
3
4
0
2
3
5
Figure 6: The trajectories generated by PPO-Lag and DLPO, along with the predicted cost distributions at the critical state denoted by the red circle.
"Can FTG facilitate more accurate constraint inference?" Despite the ablation studies in the main experi- ments (e.g.. compare UAICRL. and UAICRL-NDA), for better under- standing of the effectiveness of FTG, we visualize the inferred constraint from MEICRL (Malik et al., (021) and our UAICRL with FTG. Figure ? illustrates the recovered costs in a
MEICRL
UAICRL
5
3
3
2
2
1
0
2
Figure 7: The constraint map recovered by MEICRL and UAICRI, along with the trajectories generated by our FTG. for results in remaining settings). The generated trajectory is not included in the expert dataset due to its restricted size. Generating such trajectories enhances the diversity the dataset, mitigating the redundant constraints learned by MEICRL (refer to white circles). This finding suggests that when trained with additionally generated trajectories, the constraint model can predict step-wi wise costs more accurately, thereby reducing the epistemic uncertainty.
Gridworld setting (check Fig. D.4 4
6.5 LIMITATIONS
Theoretical Results. While study indeed encompasses theoretical findings, it still lacks rigorous theoretical analysis. Existing theoretical results on IRL (Metelli et al. [021]] Lindner et al. 2022
Metelli al. 2023) depended the construction of the feasible set of rewards, but defining the exact feasible set for the constraint is challenging due to its various types (e.g.. hard, soft or probabilistic constraints) and optimization techniques (e.g.. Lagrange methods). A rigorous theoretical study beyond the scope of our current work. Nevertheless, we affirm the importance of such understanding and suggest extending the theoretical outcomes from IRL. to ICRL as a future work.
Limited Risk Measure. For faimess and simplicity, our study predominantly employs the widely used CVaR method. Although the performance of other metrics like VaR and Entropic Value-at -iss (EVaR) has not been studied, they could be easily integrated into our framework for future exploration.
CONCLUSION
This paper presents UAICRL, a novel ICRL framework capable of modeling both aleatoric and epistemic uncertainties. This achieved through the use of a distributional estimator and a flow-based generator, enabling uncertainty-awar constraint inference. Empirical results show that UAICRL outperforms a range ICRL methods in both discrete and continuous environments, as well as in a real-world highway driving task. Looking forward, promising extension would be to incorporate robust optimization methods into our UAICRL framework. This would enable learning more robust constraints for safe control in environments with unpredictable noise.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The work is partly supported by Shenzhen Fundamental Research Program (General Program) under grant JCYJ202308071 14202005, No2022ZD01 6004, by Shenzhen Science and Technology Program 2DSY 1021 102111111 and by Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence.
14002000001 by the National Key R&D Program of China under grant
112111145025.
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty offline reinforcement learning with diversified g-ensemble. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Mattijs Baert, Pietro Mazzaglia, Sam Leroux, and Pieter Simoens. Maximum causal entropy inverse constrained reinforcement learning, arXiv preprint arXiv:2305.028577 2023.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International Conference on Machine Learning (ICML). pp. 449 458, 2017.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
Lu Chen, Xiang Zhou, Cheng Chang, Runzhe Yang, and Kai Yu. Agent-aware dropout dqn for safe and efficient on-line dialogue policy learning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2454-2464, 2464, 2017.
Jinyoung Choi, Christopher R. Dance, Jung-Eun Kim, Sculbin Hwang, and Kyungsik Park. Risk- conditioned distributional soft actor-critic for risk-sensitive navigation. In IEEE International Conference on Robotics and Automation (ICRA). pp. 8337-8344, 2021.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Uncertainty- aware constraint learning for adaptive safe motion planning from demonstrations. In Conference on Robot Learning (CoRL), pp. 1612- 1639, 2020a.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations. In Algorithumic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13, pp. 228-222--252222222222 2020b.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Felipe Leno Da Silva, Pablo Hemaandez-Leal, Bilal Kartal, and Matthew E. Taylor. Uncertainty- aware action advising for deep reinforcement learning agents. In AAAI Conference on Artificial Intelligence, pp. 5792-5799, 2020.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI Conference on Artificial Intelligence, pp. 2892-2901, 2018.
Maël Franceschetti, Coline Lacoux, Ryan Ohouens, Antonin Raffin, and Olivier Sigaud. Making reinforcement learning work on swimmer. arXiv preprint 07X.v:2208.07587, 2022.
Marylou Gabrié, Andre Manoel, Clément Luneau, Nicolas Macris, Florent Krzakala, Lenka Zde- borová, et al. Entropy and mutual information in models of deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018
Ashish Gaurav, Kasra Rezaee, Guiliang Liu, and Pascal Poupart. Learning soft constraints from constrained expert demonstrations. In International Conference Learning Representations (ICLR), 2023.
Julian Gerstenberg, Ralph Neininger, and Denis Spiegel. On solutions of the distributional bellman equation. Electronic Research Archive, 31(8):445-4444) 4483, 2023.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4565 -573,, 2016.
Carl-Johan Hoel, Krister Wolff, and Leo Laine. Ensemble quantile networks: Uncertainty awwrr reinforcement learning with applications in autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 24(6):6030-60) 6041, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
Robert Krajewski, Julian Bock, Laurent Kloeker, and Lutz Eckstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories german highways for validation of highly automated driving systems. In International Conference on Intelligent Transportation Systems (FTSC), pp.
2118-2125, 2018.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Yinchuan Li. Shuang Luo, Haozhi Wang, and Jianye Hao. Cilownets: Continuous control with generative flow networks. In International Conference on Learning Representations (ICLR), 2023,
David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Guiliang Liu, Yudong Luo, Oliver Schulte, and Pascal Poupart. Uncertainty-aware reinforcement learning for risk-sensitive player evaluation in sports game. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezace, and Pascal Poupart. Benchmarking constraint inference inverse reinforcement learning. International Conference on Learning Representations (ICLR), 2023.
Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi- agent systems. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints model-free reinforce- ment learning: A survey. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 4508 4515, 2021.
Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte, and Pascal Poupart. Distributional reinforce- ment learning with monotonic splines. International Conference on Learning Representations (ICLR), 2022.
Yudong Luo, Guiliang Liu, Pascal Poupart, and Yangchen Pan. An alternative to variance: Gini deviation for risk-averse policy gradient. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Bjöm Lütjens, Michael Everett, and Jonathan P How. Safe reinforcement learning with model uncertainty estimates. International Conference on Robotics and Automation (ICRA), pp- 8662-8668, 2019.
Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference Machine Learning (ICML), pp. 7390-7399, 2021.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International Conference on Machine Learning (ICML), pp. 4424 4434, 2019.
11

---

Published as a conference paper at ICLR 2024
David Livingston McPherson, Kaylene C. Stocking, and S. Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In IEEE Conference on Control Technology and Applications, (CCTA), pp. 1208- -218,, 2021.
Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti, and Marcello Restelli. Provably efficient learning of transferable rewards. In International Conference on Machine Learning (ICML), pp. 7665-7676, 2021.
Alberto Maria Metelli, Filippo Lazzati, and Marcello Restelli. Towards theoretical understanding of inverse reinforcement learning. In International Conference on Machine Learning (ICML), pp. 24555-24591, 2023.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine Learning, 49 (2-3):267-290, 2002.
Xinyi Ni and Lifeng Lai. Evar optimization for risk-sensitive reinforcement learning. IEEE Transac- tions on Information Theory, 2022.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems (NeurIPS). 2016.
Dimitris Papadimitriou, Usman Anwar, and Daniel S Brown. Bayesian methods for constraint inference in reinforcement learning. Transactions on Machine Learning Research, 2023.
Daehyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Conference on Robot Learning (CoRL), pp. 1005-1014, 2020.
LA Prashanth, Michael C Fu, et al. Risk-sensitive reinforcement learning via policy gradient search. Foundations and Trends in Machine Learning, 15(5):537-69) 2022.
Guanren Qiao, Guiliang Liu, Pascal Poupart, and Zhiqiang Xu. Multimmdal inverse constrained reinforcement learning from mixture of demonstrations. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Harsh Satija, Philip Amortila, and Joelle Pineau. Constrained markov decision processes via backward value functions. In International Conference on Machine Learning (ICML), pp. 8502-8511, 2020.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Confer- ence on Learning Representations (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint NXXV::1777.06447 2017.
Dexter R. R. Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. International Conference on Learning Representations (ICLR), 2020.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. In The Conference on Uncertainty in Artificial Intelligence (UAI), pp. 560-569, 2018
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):192--1958 2014.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International Conference Machine Learning (ICML), pp. 9690-9700, 2020.
Jingda Wu, Zhiyu Huang, and Chen Lv. Uncertainty model-based reinforcement learning: methodology and application in autonomous driving. IEEE Transactions on Intelligent Vehicles, (1):194-2033 203, 2023.
12

---

Published as a conference paper at ICLR 2024
Qisong Yang. Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Safety-constrained reinforcement learning with a distributional safety critic. Machine Learning, 112(3):859-85) 2023.
Brian D. Ziebart, Andrew L. Maas, Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 1433-1438, 2008,
Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. International Conference on Machine Learning (ICML), pp- 1255-1262, 2010.
13

---

Published as a conference paper at ICLR 2024
A PROOF
A.1 PROOF OF PROPOSITION 1
Let Z, C denote the vector-valued random variables of size |S], where (()) E= C'C,SS
and (s) Saae ({ a)da. Let Pd denote the state transition probability matrix under policy т, where Pays' Sac P (s'[s, a) dı (als). Assuming the Bellman consistency holds by
Z C" +yP Z, we show that the uncertainty of distributions Z under an entropy measure can be given by:
H(Z) H[C*] |S| log(1 7) log |det (d")l
(11)
where d' = (1 -olr--l -P=) [0. 1] |||s|(|8| is the induced matrix for distributions over states by following policy æ and transition Pr.
Proof:
H(Z)
[I -P-) OT
(12)
(b)
log det
(1 (1->P=) + +[O"
(13)
d
log det
+ 11
(14)
-1S| log(1
7) log det [d H[C
(15)
(a) holds by following the Bellman consistency z = C +yP= Z. To show the invertibility of
(1 (1-7P"), is sufficient to demonstrate that the matrix is full rank. Specifically, for any non-zero vector R S|:
[1(1 PP) x 0o
x p 1lo
N | x 1
| <c >||=|||>0
(1 7/|1201
IV
(b) holds by applying the properties of differential entropy and utilizing the invertibility of (7- P*).
(c) holds by defining d= = (1-
We aim to demonstrate that d E (0, |||]
represents the induced matrix for distributions over states when following policy म. Specifically, the (s)th row of d" corresponds to the distribution over states induced by following policy after starting with 8o s. This follows directly from the definition of d=
d =1= (1-7)
1=1
(1 -1-17 (1-7)[1 (7P') |
1- (TP*)
(1 7))
1- PxP
Proposition provides a decomposition of the entropy of Z into three components. 1) The first component is the entropy of the cost variables, which quantifies the uncertainty associated with the current costs. 2) The second component captures the uncertainty induced by the discount factor, which determines the degree to which the current uncertainty estimation should be influenced by the stochasticity of future rewards or transitions. 3) The third component a log-absolute determinant of the induced distribution matrix. which measures the extent to which the transition function Py and the policy π stretch or alter the initial state-action distribution. These components correspond to the key elements of aleatoric uncertainty.
14

---

Published as a conference paper at ICLR 2024
A.2
PROOF OF PROPOSITION 2
Proof:
The mutual information term can be factorized follows:
[[w; 8/7 D)
H[p(017, D) By(wDD) [[D([)T:W]]
(16)
(b) 11(p(1/F,2)) / ((o1D))/p(((1::w))
(17)
@ Hlp(# D)] / ([W))
(18)
( Hlp(*17, D)]
Σ H|p (T where wm ~ g(w)
(19)
M
(a) holds by following the meaning of mutual information (i.e., the amount of information gained by the model w if receiving a label & for a new trajectory 7, given the dataset 2).
(c) holds by using the variational inference to approximate the intractable posterior p(w] (D) with simpler approximating distribution (ww). For neural networks, the dropout distribution is commonly used.
(d) holds by sampling from the approximating distribution (e.g., the dropout distribution) with Monte Carlo method.
B IMPLEMENTATION DETAILS
B. MORE ALGORITHMM
We show the Distributional Lagrange Policy Optimization (DLPO) and Flow-based Trajectory Generation (FTG) in Algorithm and Algorithm respectively.
Algorithm 2: Distributional Lagrange Policy Optimization (DLPO)
Input: Lagrange multiplier A, risk measure p. GAE lambda Ag. rollout rounds B, update rounds K. policy Ty, reward value critic V and distributional cost value critic Z"; Initialize state 80 from CMDP and the roll-out dataset Droul:
for b = 1,2...., B do
Perform policy ты and collect trajectories = [so, do, rD, Co..... ST, aT, IT Cr):
Calculate reward advantages AT and return R2 via GAE (Schulman al. 2016)
Calculate cost advantages Ai = (7>g)) [c, + hppp (s,4) p(Z (s.))] Add samples to the dataset Drolll Drali (st, re, AF, R ct A7):
end for k = 1, 2,...,K.KK.. do
Sample data point Sx. aue, Tw. Ar, R Cxs A% from the dataset Droli;
Calculate the clipping loss ICLIP_
(a,l_18)) To..tt(P,, (A% X(A% c)), clip( To Tolanls.) (an) 1 5, () A(A2 - -))]
min
Update policy parameters v by minimizing the loss: CLLI - 87 | (a,1s,)]
Update the reward critic /" by minimizing the loss: LVF = (s. R Update the cost distribution Z" by distributional Bellman operator (Equation
end
Update the Lagrange multiplier X by minimizing the loss I >(E (c) d)
B.2 DISCRETIZED IMPLEMENTATION
We follow the Policy Iteration Lagrange algorithm (see Algorithm 2 in Liu et al. (2023) to solve discretized control problems. Based on simply incorporate distributional term into the value
15

---

Published as a conference paper at ICLR 2024
Algorithm 3: Flow-based Trajectory Generation (FTG)
Flow neural network F';, retrieval neural network Xx1 trajectory dataset D (Ti empty probability buffer P. empty generated trajectory buffer DC, generate rounds Z; I/ Flow matching
Input:
while not converge do
Sample minibatch B of trajectory data from D;
Update retrieval network Gx by minimizing the loss: M MSELoss(s: Gx (#t+1, at)):
Uniformly sample M actions {am m=1 from action space A for each state st in B;
Calculate inflows Fin log + E exp Fg (Gx (st, am), am)
M
Calculate outflows Fout log ( + AR ( + Em=1 exp Fg (se; am)
Update flow network Fg by flow matching (Eqn. 10 end
II Trajectory generation for K = .., do Initialize state 8o from CMDP, set = 0: Append 80 to the generated trajectory ,.G. while * terminal do
Uniformly sample K actions {a,),=
_- from action space A:
Generate the action probability buffer P by calculating edge flow for each action:
P FF (st, d.))
Sample aę ~ P and execute to move to the next state st+1;
Append $1+1 to the generated trajectory T.C:
end
Append generated trajectory TH to buffer DC:
end
matrix and modify the flow-based data augmentation into a discretized version following Malkin et al.) (2022.
B.3 EXPERIMENTAL SETTINGS
For training the ICRL models, we utilized a total of 8 NVIDIA GeForce RTX 3090 GPUs, each equipped with 24 GB of memory. The training process was conducted a single running node, utilizing& 8 CPUs per task. The random seeds in the MuJoCo and HighD environment are 123, 321, 456, and 654. With the allocated resources described above, running seed in the virtual and realistic environments typically required a duration of 2-4 and 3-5 hours, respectively. To optimize all of our networks, we employed the Adam optimization algorithm (Kingma & Ba 2015). The learning rate was updated using an exponential decay schedule parameterized by decay factor in every iteration. Given our primary focus on stochastic environments where absolute constraint satisfaction is challenging, we opt for small but positive constraint bound = =1e-8, We summarize the main hyperparameters in Teble B.1
C ENVIRONMENTAL DETAILS
C.I GRIDWORLD
In this paper, we establish map with dimensions of 7 x 7 units and construct three distinct settings, as illustrated Figure 2. At each time, the agent is permitted to navigate to any of the adjacent eight grids by moving step. Starting from the initial location, reward of I is granted if the agent successfully traverses to the target location while avoiding the imposed constraints, and a reward of 0 is assigned for any other situations. Moreover, degree of stochasticity. Specifically, there exists a random rate (p. = 0.01 and 0.001) with which the environment receives random action instead of the intended action executed by the agent.
modify the environment to simulate certain
16

---

Published as a conference paper at ICLR 2024
Table B.I: List of the utilized hyperparameters in UAICRL. To ensure equitable comparisons, we maintain consistency in the parameters of the same neural networks across different models.
Parameters
HalfCheetah
Ant
Walker
Swimmer
Pendulum
General
Expert Rollouts
10
50
50
50
50
Max Length
1000
500
500
500
100
Gamma
0.99
0.99
0.99
0.99
0.99
PPO
Steps
2048
2048
2048
2048
2048
Neward-GAEA
0.95
0.9
0.9
0.9
0.9
Cost-GAE-X
0.95
0.9
0.9
0.95
0.9
Policy Network π
64, 64
64, 64
64, 64
64, 64
64, 64
Reward Network V
64, 64
64, 64
64, 64
64, 64
64, 64
Cost Network Z5
256,
256, 256
256.
256,
256,
Learning Rate
3e-4
3e-5
le-4
3e-4
le-4
Target KI.
0.01
0.02
0.02
0.01
0.02
Lagrangian
Initial Value
0.05
0.1
1
0.1
Learning Rate
0.1
0.02
0.05
0.01
0.05
Constraint Function
Network C..
20
40, 40
64, 64
20
20 0.001
Learning Rate w
0.05
0.005
0.001
0.001
Backward Iterations
10
5
5
5
5
DIPO
Quantiles
64
64 CVaR
64 CVaR 0.95
64 CVaR 0.95
64
Risk Measure
CVaR 0,25
CVaR
Risk Level
0.95
0.25
FTG
Network Fg
256, 256
256, 256
256, 256
256, 256
256, 256
Learning Rate f
3e-4
3e-4
3e-4
3e-4
3e-4
Sample Action Size
10000
10000
10000
10000
10000
Sample Flows
50
50
C.2 MuJoCo
The five MuloCo robotics environments are built upon MuJoCo (see Figure C.1
Figure C.I: MuJoCo environments. From left right, the environments are Half-cheetah, Ant, Walker, Swimmer, and Inverted Pendulum, respectively.
We modify them by incorporating predefined constraints and adjusting reward terms for ICRL. To simulate the stochasticity in the environment dynamics, we incorporate Gaussian noise 1) ~ N(u,, a)) into the transition function at each step such that (st+1 st, az) = f (St, az) + 1). In this work, we utilize μ = with o 0.1, 0.01 and 0.001 to represent different scales noise. We provide more comprehensive description as follows:
this environment, the agent controls robot with two legs. The reward obtained by the agent is determined by the distance the robot walks between two time steps, and penalty based on the magnitude of the input action. The game continues until a maximum time step of 1000 is reached. Due to the fact that the robot is easier move backward than forward, we define a blocked region where the X-coordinate < 3. So the robot restricted to move within the region where the X-coordinate > 3.
Blocked Half-Cheetar
2) Blocked Ant. this environment, the agent controls a robot with four legs. The rewards obtained by the agent depend the distance of the robot from the origin, and healthy bonus for maintaining
17

---

Published as a conference paper at ICLR 2024
Table C.I:
The stochastic MuJoCo environments with constraints
Name
Obs. Dim.
Act. Dim.
Constraints
Stochasticity
Blocked Half-Cheetah
18
6
X X-Coordinate 2 -3
N(H o)
Blocked Ant
113
8
X- C-Coordinate 2 -3
N(u,
Blocked Walker
18
6
X- X-Coordinate 2-3
N(u,
Blocked Swimmer
10
2
X- X-Coordinate 0.5
N(H,
Biased Pendulum
4
X- X-CCordinate 2 -0.015
N(H,o)
balance.
The game terminates when maximum time step of 500 is reached. Similarly the Blocked Half-Cheetah environment, we establish a constraint that blocks the region with X-coordinate < -3. As result, the robot is only permitted to move within the region where the X-coordinate 2 -3.
3) Blocked Walker. In this environment, the agent controls a two-legged robot and learns how to make walk. The termination conditions for the game are either when the robot loses its balance or when it reaches the maximum time step of 500. The reward obtained by the agent is determined by the distance the robot walks between two time steps, along with penalty based on the magnitude of the input action. Similar to the above environments, we impose constraint that blocks the region where the X-coordinate < -3. Consequently, the robot is only allowed to move within the region where the X-coordinate -3
4) Blocked Swimmer. In this environment, the agent controls a robot with two rotors connecting three segments, and learns how to move. The reward obtained by the agent is determined by the distance the robot walks between the current and previous time steps, along with a penalty based on the magnitude of the input action. The game ends when the robot reaches the maximum time step of 500. In contrast to the aforementioned environments, the Swimmer robot is easier to move forward than to move backward. Therefore, we constrain the region where the X-coordinate > 0.5. Consequently, the robot is only allowed to move within the region where the X-coordinate S 0.5.
5) Biased Pendulum. In this environment, the agent controls pole to balance it cart. The game terminates either when the pole falls or when the maximum time step of 100 is reached. To increase difficulty, we assign higher rewards to the left locations, where the constraints are also imposed. Specifically, we introduce a constraint that blocks the region with X-coordinate < -0.015. At each step, a reward of 1 is provided if the X-coordinate 2 0, and a reward of 1 is given the X- X-coordinate I -0.01. The reward gradually decreases from 1 to 0.1 when the X-coordinate falls within (-0.01, 0). Consequently, the agent is challenged to resist the temptation of higher rewards and stay within safe regions.
The MuloCo environment settings in this work are summarized in Table
C.3 HIGHWAY DRIVING
Figure illustrates the Highway Driving environment. In this scenario, the ego car is displayed in blue while other cars are shown in red. The ego car has limited visibility and can only observe objects within its vicinity (marked in blue). The objective is navigate the car reach the destination point without going off-road, colliding with other cars, or violating time limits and other constraints (c.g.. velocity).
In this paper, we mainly focus on the velocity constraint, where we limit the speed of the ego car to no more than 40 m/s, to ensure the ego car can drive safe speed. Additionally, we add Gaussian noise to the agent's 's action at each time step to introduce the control-level stochasticity.
Clssrvation Rackas
Desinatoon
Figure C.2: The Highway Driving (HighD) environment.
18

---

Published as a conference paper at ICLR 2024
D MORE EXPERIMENTAL RESULTS
D.1 MORE RESULTS IN GRIDWORLD
Figure D. and Figure D.2 show the training curves in Gridworlds with the random rate pa 0.01 and 0.001, respectively.
Girdworld-3
1.0
BC2L
DC2L
1.0
DCZL GACL
GACL
GACL
VICRL
VICAL M
VICHL
0.6
UY
0.4
0.4
0.4
0.2
0.2
40
80
20
60
50
with
Girdworld
with
1.0
BCZLL
1.0
EC2L
0.7
BC2L
GACL
GAOL
GACL
Fassble Ranands
0.8
ICAL
VICRL
VICRL
VUA
VICAL
0.4
0.4
0.4
0.2
0.2
0.2
20
-0-
60
80
20
400
60
50
20
60
Figure D.1: The constraint violation rate (top) and feasible rewards (bottom) in three Gridworld settings with the random rate Ps = 0.01.
Girdworld with Probo.99P 999
Girdworld-2 with Prob0.999
Girdworld-3 with ProbO.999
1.0
BCZL
1.0
BC2L
1.0
BCZL
0.8
ICRL
0.8
ICAL
0.8
ICRL VICRL UAICAL
VICRL
UNCHL
UAIERL
UY
0.2
0.2
0.2
0.0
0.0
60
60
50
20
40
60
60
50
Eprsode
Episade
Epriade
Girdworid-1 with Probo.999
Girdworld with Probo .999
Girdworid-3 with Prob0.999
RCPL
1.0
BCPL
BC2L
N
0.8
Femibba fawarta
0.8
ICRL
0.5
KCRL
ICRL
VICRL
0.6
WAICRL
WAICRL
UAICRL
WVV
0.2
0.2
0.0
0.0
20
40
80
400
60
50
20
60
Eolcode
Episode
Eblcade
Figure D.2: The constraint violation rate (top) and feasible rewards (bottom) in three Gridworld settings with the random rate P. = 0.001.
19

---

Published as a conference paper at ICLR 2024
Figure D.3 illustrates the trajectories generated by PPO-Lag and DLPO in the last two Gridworld settings, along with the predicted constraint distributions.
PPO-Lag
Predicted Conséraint Distribution
Tra)
6
Otn Tiaj
right upper righ:
1th Traj
5
Tra)
right
4
401 Tra)
30 Thd
4
(ra)
3
3
2
1
0
1
2
3
4
PPO-Lag
DLPO
Predicted Constraint Distriburtion
6
fight
5
upper right
lower right
0t1 Ta
2
1th TX
2th T
1
2th
1t Thn
0
4m
1
2
3
5
6
Quantee
Figure D.3: Each row refers Gridworld scenario. We illustrate the trajectories generated by PPO-Lag and DLPO, along with the predicted cost distributions at the critical state denoted by red circles.
Figure D.4 illustrates the constraint map recovered by MEICRL and UAICRL in the last two Gridworld settings, along with the trajectories generated by the expert flow model. White circles denote the redundant constraints learned by MEICRL.
MEICRL
UAICRL
6
Expert
5
5
Genatated
4
3
3
2
0
0
1
2
4
5
0
2
3
5
2
MEICRL
UAICRL
Expert
5
4
3
2
1
3 4
0
2
3
5
0
2
4
6
Figure D.4: Each row refers to Gridworld scenario. We illustrate the constraint map recovered by MEICRL and UAICRL, along with the trajectories generated by the expert flow model.
D.2 MORE RESULTS IN MuJoCo
Table D.I shows the evaluation performance in MuloCo environments with (/ 0.1).
20

---

Published as a conference paper at ICLR 2024
Table D.I: Evaluation of different methods in the MuJoCo environments with the stochasticity of (0, 0.1). We report the mean + std results averaged over 4 random seeds.
Blocked
Blocked
Blocked Walker
Blocked
Biased
Half-Cheetah
Ant
Swimmer
Pendulum
BC2L
333.4x175.2
995.0:225.2
222. 1194.9
91.91-46.1
0.161720.1295
GACL
577.24135.4
387 205. 1
55.545.5
64.74:30.6
0.0572=0.0439
Feasible
ICRL
343.0+224.2
965.0+432.4 4
166.9449.0
91.0+42.8
0.1440=0.1
Rewards
VICRL
615.44299.9
641.8+297.4
91.5=25.3
105.5#29.1
0.1447=0.1139 1339
UAICRI
1714.4+222.0
1313.54533,2
199.8=106.
177.3+30.7
0.2551 1347
BC2L
0.3120.16
0.0a0.0
0.8910.05
0.91
Constraint
GACL
0.2940. 16
1.14
0.0+0.0
0.92+0.04
0.87=0.10
Violation
ICRL
0.67+0.21
0.52+0.18
0.0=0.0
0.90=0.05
0.93=0.05
VICRL
0,57+0.21
0,37=0.17
0.0=0.0
0.88+0.03
0.91=0.07
Rate
UAICRL
0.07.0.06
0.42a0.17
0.020.0
0.8620.03
0.52+0.20
Figure D.5 and Figure show the additional experimental results in MuloCo environments with constraint recovery under the stochasticity of A/ 0.01) and (0, 0.001), respectively.
Chestah
Walker
Swimmer
Donarved Vidaton
00
2.1
32
00
Eriede
Blocked Holf-Cheetah
Blocked Ant
Blocked Walker
Blocked Swimmer
Biased Pendulum
com
1003
Fexabb freartte
3003
8000
2003
32
2022
4230
2533 1200 MED
13300 35300
15300
2030 1200 M0E
3006
25300 89C3
00530
Spacee
BC2L
GACL
ICRL
VICRL
UAICRL-NRS
UAICRL-NDA
Figure D.5: The constraint violation rate (top) and feasible rewards (bottom) in MuJoCo environments with constraint recovery under the stochasticity (O, 0.01).
Blocked Half Cheetah
Blocked Ant
Blocked Walker
Blocked Swimmer
Blased Pendulum
Dambrrt Dannoee Viraten
32
3000
2002
6530
2000 4200 SXCD REER 3000
20022
20000
3000
2000 30E 0893
8800 18530
20003
405.0
5500
Blocked Half Cheetah
Blacked
Blocked Walker
Blocked
Blased Pendu
Faodtle Rawwerte
0803
2003
1003
2000
2003
4000
2000 4900
5000
10,00 28300
30300
2000 4900
500
3000
20000
4003
60030
UAICRLANRS
LAIICLL-NAA
UAICRL
Figure D.6: The constraint violation rate (top) and feasible rewards (bottom) in MuJoCo environments with constraint recovery under the stochasticity of A' 0.001).
Figure D.7 D.S. and D.9 D.9 show the complete results in MuJoCo environments when given ground constraints under the stochasticity of 0.1,0 0.01) and (0, 0.001), respectively.
21

---

Published as a conference paper at ICLR 2024
Blocked Hall-hhetthh
Blocked Ant
Blocced Welker
Blocked Sivinmer
Blased Pendulum
Gonabair Gontoai Stawiin hale
06
64
05
04
Splice
2233
1231
3226
2000
1907
Blocked Hale-Cheetah
Blacked
Blocked Walker
Blocked
Blased Pendulum
3000
Faostte Denwerts
1203
1003
2000
100
1590
3000
2003
4000
2000
400
3000
12300
20300
2000 4090
2000
20000 4003
00030
PPO-Lag
DEPO-Neutral
DLPO-Averse
Figure D.7: The constraint violation rate (top) and feasible rewards (bottom) in MuJoCo environments with the stochasticity of (0, 0.1) when given ground-trutt constraints.
Blocked Cheetah
Blacked
Blocked Walker
Blocked Swimmer
Blased Pendulum
Voneteent tboe
Ma
64
3.3
3.2
0.1
400g
6030
2099 4030
3003 2000
20065
200C
30000
290
6500 LC030
CO0O
Blocked Half-Cheetah
Blocked Ant
Blocked Walker
Blocked Swimmer
Biased Pendulum
3817
5000
Faseabl Rreeette
2503
3500
3000
1523
2000
PPO-Lag
DLPO-Neutral
DLPO-Averse
Figure D.8: The constraint violation rate (top) and feasible rewards (bottom) in MuJoCo environments with the stochasticity of ('0, 0.01) when given ground-trutt constraints.
Cheetah
Walker
Swimmer
Consrnent Vidaton far
03
Criede
Blocked Half-Cheetah
Blacked Ant
Blocked Walker
Blocked Swimmer
Blased Pendulum
Feoabl fhewart
2003
4300
2000
J30
3300
1003
3000
32
2522
4200
602
1030 4200 MEDD JEED 2005
12300 35300
10030
1533 C200 RED 3006
20300 0902
10030
PPO-Lag
DUPO-Neutral
DUPO-Averse
Figure D.9: The constraint violation rate (top) and feasible rewards (bottom) in MuloCo environments with the stochasticity of (10 0.001) when given ground-truth constraints.
22

---

Published as a conference paper at ICLR 2024
D.3 MORE RESULTS HIGHD
Figure D.10 shows the additional experimental results in the HighD environment with constraint recovery under the stochasticity of (0, 0.01) and (0, 0.001), respectively.
HighD
High0 Constraint
Speed Condtraite Vilecon Hate
0.8
Fessible Rewards
0.6
10
0.4
0.2
0.1
-0
1000
20000
3000
1000
2000
3000
Episode
Episode
Velecity Constraint AT0, 0.001)
HighD Velocity with
Speed Conpraint Vielaton Rate
20
0.8
Fess Rewards
10
0.6
0.4
0.3
0.1
-20
2000
2000
3000
1000
2000
3000
Episode
Episode
BC2L
GACL
ICAL
VICRL
UAICRL
Figure D.10: Model performance in the HighD environment with velocity constraint under under.N(00. 0.01)
(top) and (0, 0.001) (bottom).
D.4 MULTIPLE CONSTRAINT RECOVERY
To investigate the model performance under multiple constraint scenarios, we expand the HighD environment to incorporate both speed and distance constraints by generating an expert dataset, wherein the agent adheres to both constraints. Figure 1.10 shows the results under the stochasticity of NO, 0.1). We find that UAICRL still outperforms other baselines with the highest rewards while keeping low constraint violation rate of both constraints, demonstrating its ability of uncertainty awareness with respect to multiple constraints. addition, in comparison with the single constraint scenario (as shown in Figure the rewards accrued by the policies exhibit significant reduction, while the rate of constraint violations remains unaffected.
[.Ilshowws
HighD Velocity and Distance Constraint
High Velocity end Distance Constraint 0.
HighD Velocity and Distance Constraint
Pate
4000 -0
Suodt
1000
1000
2000 Epioode
4005
2002
Eskodo
Episcde
BC2L
GACL
ICRL
VICRL
UAICRL
Figure D.11: Model performance in the HighD environment with both the speed and distance constraints under (0,
23

---

Published as a conference paper at ICLR 2024
E MORE COMPREHENSIVE ANALYSIS
In this section, we provide comprehensive analysis of the experimental results to supplement the discussion presented in the main body of the paper (Section 6.2). The aim to delve deeper into the nuances of our experiments, offering a more detailed examination of the insights that could not be fully explored within the constraints of the main text. We summarize some meaningful findings and provide some explanations as follows:
(1) "The baseline methods exhibit relatively unsatisfactory results in the experiments." "
This is because this paper primarily investigates the robustness of various methods in the face of aleatoric and epistemic uncertainties. As such, we intentionally control the scale of noise and the number of demonstrations, detailed in experimental settings (Section C). The baseline methods partially or completely disregard these factors (refer to Table DT Specifically, 1) baseline methods that do not address stochastic transition dynamics fail to learn constraints that ensure the safety of the learned policy. This oversight can negatively influence control performance. 2) If epistemic uncertainty in
(OoD) data points is not taken into account, it could lead to inaccurate cost predictions, subsequently affecting policy updates and overall performance.
(2) "When comparing the results across different levels of stochasticity, larger noise does not neces- sarily result in poorer model performance."
The reason is that when noise levels are relatively low, its impact becomes challenging for the distributional value function to detect, resulting in a constraint that is less sensitive to the risk. Conversely, as the level of noise (i.e., stochasticity) increases, the variance of the estimated distribution becomes prominent, thereby making the constraint more responsive to the risk of unsafe behaviors. Our algorithms can thus induce stricter constraint functions and thus lower constraint violation rate.
(3) "The divergence trend observed in the UAICRL method, particularly in the Blocked Walker task." "
As shown in the results, we find that different algorithms display varying trends in their reward and constraint violation rates throughout the training curves. This divergence is most noticeable in the Blocked Walker environment. For instance, by the end of training, the rewards for UAICRL and UAICRL-NDA progressively increase, while those for other methods tend to remain unchanged or even decrease, To understand this phenomenon, it's important to note that both UAICRL and UAICRL-NDA implement risk-sensitive constraints. These constraints enable the agent to more effectively manage the aleatoric uncertainty resulting from noise, thereby facilitating the learning of risk-aware policies. This could result in considerable improvement in policy updates, particularly when the agent encounters critical situations or locations. For instance, these constraints enable the agent to initially learn maintain safe distance from hazardous zones. This strategy allows the agent to better understand how to accrue rewards without entering dangerous events.
(4) "Most methods struggle to learn a safe policy with low constraint violation rate in the Blocked Swimmer task." "
There are several factors contributing to the complexity of the Blocked Swimmer task: 1) The Swimmer enviromment itself is inherently challenging to solve by RL methods. Previous works report similar difficulties RL control (Franceschetti et al., 2022). 2) The mechanical dynamics of the Swimmer Robot make it casier for the robot to move forward than backward. The block region is also defined in front of the robot (where the X-coordinate > 0.5). The intrinsic robust dynamics may strongly compel the robot to proceed forward, leading to constraint violations even in deterministic settings (refer to Figure 2 in Liu et al. (2023)) 3) Thirdly, the stochastic transition dynamics further complicate the safe control, even when agents are aware of the ground-truth constraints (refer to PPO-Lag in Figure D.7j. The task becomes even more difficult when the agent is required to predict constraints from provided demonstrations. In conclusion, the Blocked Swimmer environment already presents a challenge for traditional constrained RL. The challenge is amplified when constraints need to be inferred (as in the ICRL setting) and when transition dynamics are stochastic.
(5) "The discrepancy in the performance of GACL, which performs well in the Block Ant task but not in other tasks in Fig. D.5 Dn5 and
The observed discrepancy in GACL's performance, where it works well in the Blocked Ant task but underperforms in other environments, can be attributed its approach to constraint learning. Unlike
24

---

Published as a conference paper at ICLR 2024
traditional constrained RL problems, GACL directly appends the discriminator Dog a) to the reward function as a penalty, resulting in a modified reward function r'(s, a) = r(s, a) + Do a). This modification allows us to investigate the impact of simple reward shaping constraint learning. We find GACI. performs well in the Blocked Ant since this environment has the highest dimension in all environments. In other environments with lower dimensions, the discriminator log (ss, a) is prone to overfitting the imperfect data early in training. The log- tog-probability penalty from this low-quality discriminator applies substantial penalty to many feasible regions, rendering the algorithm incapable of learning. However, in sigh-dimensional environments (like the Ant task), the discriminator is less likely overfit. As result, the impact of early mistakes in the discriminator is less significant, enabling continuous learning.
(6) "In some environments in Figure MMICRI.perfone similarly to UAICRL-NDA."" "
We observe that in environments with high-dimensional state-action space (HalfCheetah, Ant, and Walker), UAICRL significantly outperforms UAICRL-NDA with higher rewards and lower constraint violation rates. However, in environments with relatively lower state-action dimensions (Swimmer and Pendulum), the difference of performance between UAICRL and UAICRL-NDA is less significant. This is because, in the high-dimensional environment, training dataset of fixed size can only encompass a limited number of data points within the input space. In contrast, in simpler or lower- dimensional environments, the input space is smaller. As result, dataset of the same size cover a larger proportion of data points. This increased coverage reduces the chances of (OoD) data points and the impact of epistemic uncertainty. Consequently, the incorporation of the data augmentation component does not lead to a substantial enhancement in performance.
25


Reviews:
[{"review_id": "Sd9dZ4D4wr", "reviewer": "Reviewer_4NEv", "summary": "This paper presents UAICRL, a novel approach for addressing Inverse Constrained Reinforcement Learning (ICRL) by considering both aleatoric and epistemic uncertainties. UAICRL leverages a distributional critic in conjunction with a risk-measure to calculate the cost, so as to handle aleatoric uncertainty. In addition, it utilizes mutual information and flow-based trajectory generation techniques to reduce epistemic uncertainty. The experimental results demonstrated improved performance and included ablation studies on the use of risk-sensitive constraint and data augmentation.", "questions": "1. I'm wondering if the flow functions can be substituted with other conditional generative models, or if the flow matching objective is tightly coupled with UAICRL. For example, can one use a conditional diffusion model to replace the Flow-based Trajectory Generation (FTG) algorithm?\n2. While Table B.1 suggests that FTG can maintain consistent hyperparameters across various tasks, I wonder how the performance of UAICRL might be influenced by the selection of hyperparameters for the FTG network. This concern arises from the potential of FTG to either underfit or overfit, which could lead to generating out-of-distribution trajectories and potentially causing a decline in overall performance. Could you explain the process of tuning the hyperparameters for the FTG network?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Addresses both aleatoric and epistemic uncertainty, in contrast to previous methods that primarily focus on epistemic uncertainty.\n- Works with both continuous and discrete spaces, unlike most previous methods limited to discrete spaces.\n- Supports stochastic training environments, whereas earlier works frequently assume deterministic environments.", "weaknesses": "1. The ablation of the mutual information term in Eq. (7) results in a configuration where only the risk-sensitive constraint from Eq. (4) is utilized. This particular setup is not discussed in the paper.\n2. I'm concerned about expanding the dataset by generating trajectories based on a learned flow function. It is still possible for the flow function to generate out-of-distribution data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xFz0ae8j77", "reviewer": "Reviewer_8mMN", "summary": "This paper considers addressing the uncertainty issues in the inverse constrained RL problem. The authors propose to (1) replace cost critic by a distributional one in constrained RL to model the aleatoric uncertainty, and (2) use FTG to augment data to reduce epistemic uncertainty. The authors compare their method with previous inverse constrained RL baselines on different domains including gridworld, safety-mujoco and highway driving.", "questions": "- What are the target cost limits $\\epsilon$ for experiments in table 2, fig 3&5?\n- Why are some experiments early stopped when they obviously have not converged? E.g., in fig.3&4.\n- The authors run experiments on Mujoco tasks with different scales of stochasticity in env. But many baselines have much higher constraint violation rate with smaller noise, e.g., comparing fig.3, D3&D4. My intuition is that these methods should behave better with smaller noise as they cannot model such uncertainty. Could you explain it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well organized and easy to follow.\n- The proposed method addresses two types of uncertainties in constrained RL, which are overlooked by previous research.\n- The ablation studies validate the effectiveness of distributional critic when encountering aleatoric uncertainty.", "weaknesses": "- The effectiveness of data augmentation is a little questionable. \n    - In theory, FTG can augment the expert and nominal datasets but the last term in the objective of eq.(8) includes the OOD trajectories $\\bar{\\tau}$. So how do you generate $\\bar{\\tau}$?\n    - In practical experiment (fig 3), UAICRL actually performs similarly to UAICRL-NDA, which removes the data augmentation part. \n    - Although the authors give more illustrations in fig.7 (I suppose the top is for MEICRL and bottom is for UAICRL), I think it's not very clear. For example, I believe the authors should at least explain what the generated trajectory is, and which parts are OOD.\n\nminor issues:\n- In fig 4, the baseline should be \"GACL\" instead of \"GAIL\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vfeUXpJcPE", "reviewer": "Reviewer_nMsK", "summary": "This paper presents a thorough discussion on the topic of inverse safe reinforcement learning. The authors introduce the Uncertainty-Aware Constraint Inference Constrained Reinforcement Learning (UAICRL), a novel framework that takes into account both aleatoric and epistemic uncertainties to enhance uncertainty awareness in constraint inference. The authors conducted extensive experiments to demonstrate the superior performance of their method over several other ICRL methods in both continuous and discrete environments, highlighting its strong uncertainty-aware capabilities.", "questions": "(1) Could you please explain the reason behind the divergence trend observed in the UAICRL method in section 6.2, particularly in the Blocked Walker task? Additionally, would it be possible to provide results with an extended number of epochs? Were the curves smoothed in your analysis? \n\n(2) What factors contribute to the challenges posed by the Blocked Swimmer task? It seems that most methods struggle to learn a safe (low-cost-violation) policy for this specific task. \n\n(3) Could you elaborate on the factors leading to the relatively unsatisfactory results of the baseline methods in the tasks? Specifically, what could explain the discrepancy in the performance of GACL, which performs well in the Block Ant task but not in other tasks shown in Figures D.3 and D.4? \n\n(4) How would you describe the generalizability of your method to multi-cost settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Noteworthy topic: The study on Inverse Constrained Reinforcement Learning appears to address a critical issue, and the proposed method holds promising potential for real-world applications. \n\n(2) Extensive experimental validation: The authors' extensive experiments with a wide range of baseline methods and tasks to demonstrate the strength of their approach are commendable and impressive.", "weaknesses": "(1) Insufficient theoretical support: It is observed that the proposed method may benefit from further strengthening its theoretical foundations, as acknowledged by the authors. \n\n(2) Limited discussion and explanation of experiments: While the manuscript presents extensive experimental results, a more comprehensive discussion and elaboration of these findings would enhance the paper's overall quality. Moreover, a detailed examination of performance across various scales of randomness within the primary context could provide valuable insights, as noted in my question (1-3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWneML8sos", "reviewer": "Reviewer_zJng", "summary": "This paper is interested in Inverse Constrained Reinforcement Learning (ICRL), that is, simultaneously learning imitation policies whilst learning and adhering to the constraints respected by the expert.\nThe authors propose to incorporate the uncertainties arising from stochastic environments (aleatoric uncertainty), along with\nthe epistemic uncertainties arising from learning from limited data.\nTo this end, they propose to learn the cost model with distributional Bellman updates.\nThey then propose a flow-based generative data augmentation scheme to mitigate issues arising from epistemic uncertainty.\nThat is, the augmented trajectories should remain in regions of the learned model which can be predicted confidently (low epistemic uncertainty).\nThe method is tested in i) a discrete-action grid world environment and ii) five MuJoCo environments with additional noise.", "questions": "- What are the main questions your results section is trying to answer? Can you summarize them in a few bullet points?\n- Have you made the figures legible?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Although I am not well-read in Inverse Constrained Reinforcement Learning (ICRL), this paper appears to have highlighted\nan important problem: imitation policies should satisfy the learned constraints subject to both the uncertainty in the environment (aleatoric uncertainty) and the uncertainty arising from learning the constraint from limited data (epistemic uncertainty).\nCapturing the aleatoric uncertainty with distributional Bellman updates seems like a good idea and makes sense intuitively.\nWhilst I found the flow-based trajectory generation section hard to follow, I get the general idea and this seems like a sensible way to consider the epistemic uncertainty.\nI also liked that throughout the paper the authors provide intuition for the math with real-world examples. This was nice.", "weaknesses": "The paper's biggest weakness is its presentation and clarity.\nI was generally happy with the paper up until Section 6 (Empirical Evaluation).\nI found this section particularly hard to follow.\nI'm not entirely sure what the main points are that the authors are trying to show in this section.\nI would suggest the authors try to summarize the main questions and introduce them at the start of section 6.\nThis gives the reader an idea of what to expect in the section, which makes for an easier read.\n\n**Experiments**\nRemember, readers are stupid, you should hold their hand and walk them through your figures.\nFor example, what does robust and superior performance allude to in this sentence:\n\"When implementing both techniques, UAICRL demonstrates even more robust and superior performance compared to other methods.\"\nThis could be made a lot easier for the reader with something like:\n\"When implementing both techniques, UAICRL (pink) generally obtains high feasible rewards (top row Fig. 3) whilst having a low constraint violation rate (bottom row Fig. 3). This demonstrates that UAICRL is more robust and has superior performance compared to other methods.\"\n\n**Illegible figures**\nMost figures are illegible due to being too small and the font size being too low.\nThis needs to be fixed before publication.\n\n\n**Conclusion is very short..**\nThe conclusion is very short and feels rushed. Surely the authors have more to say here??\n\n**Bolding**\nWhat does the bolding in the tables show? Does it show statistical significance from a statistical test or something else? This should be clarified somewhere in the text.\n\n**Code**\nThere is no README file in the code supplement so it is not clear how to setup the environment or how to run the experiments.\nIt would be good to at least have a notebook to see how the code/method works in practice.\n\nIn my opinion, the paper highlights an important problem, has a good technical contribution and has results which support the claims.\nHowever, I do not think the paper can be published until:\n- The experiments section is made clearer\n- The figures are made legible\n- The conclusion is written properly\n\n**Minor corrections**:\n- The paper has many textual citations in parentheses. For example \"(Liu et al., 2023; Papadimitriou et al., 2023)\" in paragraph 3. You should use \\citet instead of \\citep to remove the parentheses.\n- In Section 2, what is $\\mathcal{M}^{c_{\\omega}}$? It's not defined anywhere.\n- Figure 1.\n  - The text is way too small.\n  - It's also not clear where to start reading from. I think you should start reading from $\\mathcal{D}_{e}$ so perhaps this should be mentioned in the caption.\n- Page 3 footnote is missing a full stop.\n- Section 4.2\n  - $\\mathcal{T}$ is never formally introduced.\n  - $F(\\cdot)$ is overloaded and confusing as it is shown as $F(\\tau)$, $F(s_{t})$ and $F(s,a)$. I think you should distinguish the state flow $F(s_{t})$ from the trajectory flow function.\n- Algorithm 1:\n  - Does \"sample nominal trajectories\" imply interacting with the environment? If so, I would explicitly state this.\n  - Is it right that the augmented data is discarded at each iteration?\n- Table 1 text is way too small\n- Figure 2 needs larger text\n- Figures 3/4/5/D.3/D.4/D.5/D.6\n  - Larger text\n  - It only needs one legend.\n  - The x-axis ticks are too close\n  - The plot titles don't all need to say \"with noise $\\mathcal{N}(0,0.1)$\"\n- Figure 6 is not clear.\n  - Each column refers to a grid world scenario so this should be on the figure and/or mentioned in the caption.\n  - The text is way too small\n  - What are the constraints locations???\n  - Where does the agent start/end?\n- Figure 7 is not clear.\n  - Each column refers to a grid world scenario so this should be on the figure and/or mentioned in the caption.\n  - What does each row represent? Is the top row ICRL and the bottom row UAICRL? This needs to be made clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)