Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
Authors: Haoming Jing
Yorie Nakahira
Venue: ICML 2025 poster (2025)
Keywords: Control System
Stochastic System
Safety
Primary area: theory->everything_else
TL;DR: We propose a causal-RL–driven probabilistic safety certificate that accounts for latent-variable–induced distribution shifts by enforcing an invariance condition in probability space, enabling persistent feasible actions and long-term risk control.

Abstract: Existing control techniques often assume access to complete dynamics or perfect simulators with fully observable states, which are necessary to verify whether the system remains within a safe set (forward invariance) or safe actions are persistently feasible at all times. However, many systems contain latent variables that make their dynamics partially unidentifiable or cause distribution shifts in the observed statistics between offline and online data, even when the underlying mechanistic dynamics are unchanged. Such “spurious” distribution shifts can break many techniques that use data to learn system models or safety certificates. To address this limitation, we propose a technique for designing probabilistic safety certificates for systems with latent variables. A key technical enabler is the formulation of invariance conditions in probability space, which can be constructed using observed statistics in the presence of distribution shifts due to latent variables. We use this invariance condition to construct a safety certificate that can be implemented efficiently in real-time control. The proposed safety certificate can persistently find feasible actions that control long-term risk to stay within tolerance. Stochastic safe control and (causal) reinforcement learning have been studied in isolation until now. To the best of our knowledge, the proposed work is the first to use causal reinforcement learning to quantify long-term risk for the design of safety certificates. This integration enables safety certificates to efficiently ensure long-term safety in the presence of latent variables. The effectiveness of the proposed safety certificate is demonstrated in numerical simulations.

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
Haoming Jing'
Yorie Nakahira
Abstract
1. Introduction
Existing control techniques often assume access to complete dynamics perfect simulators with fully observable states, which are necessary to verify whether the system remains within a safe set (forward invariance) or safe actions are per- sistently feasible at all times. However, many systems contain latent variables that make their dynamics partially unidentifiable cause distri- bution shifts in the observed statisties between offlin and online data, even when the underly- ing mechanistic dynamics are unchanged. Such "spurious" distribution shifts can break many tech- niques that use data to learn system models or safety certificates. To address this limitation, we propose a technique for designing probabilistic safety certificates for systems with latent variables. A key technical enabler is the formulation of in- variance conditions in probability space, which can be constructed using observed statistics in the presence of distribution shifts due to latent variables. We use this invariance condition to con- struct safety certificate that can be implemented efficiently in real-time control. The proposed safety certificate can persistently find feasible ac- tions that control long-term risk to stay within tolerance. Stochastic safe control and (causal) reinforcement learning have been studied in isola- tion until now. To the best of our knowledge, the proposed work is the first use causal reinforce- ment learning quantify long-term risk for the design of safety certificates. This integration en- ables safety certificates to efficiently ensure long- term safety in the presence of latent variables. The effectiveness of the proposed safety certificate is demonstrated in numerical simulations.
Autonomous control systems must operate safely even in the presence of latent variables. For instance, autonomous ground vehicles must anticipate objects suddenly emerg- ing from behind occlusions or pedestrians unexpectedly changing their intent to cross the road. In such scenarios, risk-critical variables may be unobservable. These latent variables can induce distribution shifts between offline and online settings in visible variables and render the partial models associated with the latent variables unidentifiable. Therefore, in the design of control policies, one must plicitly differentiate between statistical models (Schölkopf et al., 2021; Ye et al., 2024), which capture correlations in observed states, and mechanistic models, which encode the causal structure of the system. Designing safety certifi- cates directly from data without this distinction can lead to vulnerabilities from spurious distribution shifts-divergence between offline versus online statistics even when the un- derlying dynamics remain unchanged (see Section 2.2 for mathematical details). Moreover, the effects of latent vari- ables on observed states may remain subtle within a short time horizon, but by the time they become apparent, cor- rective action may no longer be feasible. For example, if a child suddenly emerges from behind large bus, the vehicle may not have sufficient time to stop safely. The presence of such irrecoverable states further complicates safety assur- ance. While myopic safety can often be efficiently enforced, it is insufficient for ensuring long-term safety. However, the complexity of safety assurance increases unfavorably with the time horizon, particularly in the presence of latent variables.
Motivated by these challenges, this paper explores the fol- lowing research question:
How can we efficiently assure long-term safety for stochastic systems in the presence of latent variables, which induce distribution shifts in offline vs. online statistics and partially unobservable dynamics?
'Electrical and Computer Engineering Department, Carnegie Mellon University. Pittsburgh, USA. Correspondence to: Yorie Nakahira <ynakahir@ andrew.cmu.edus.
Existing safety certificates often rely on complete system models or fully observable states to verify whether the sys- tem remains within a safe set and whether safe actions ex- continuously (Hsu et al., 2023; Wabersich et al., 2023).
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
These methods commonly determine actions that satisfy forward invariance conditions in the state space (Blanchini, 1999), which require full knowledge of system dynamics and state observability for evaluation. However, many real- world systems do not meet these requirements due to the presence of latent variables. While myopic controllers are more practical for real-time control with limited onboard resources (Ames al., 2016; 2019), they may fail to guar- antee long-term safety due to the presence of irrecoverable states and the accumulation of tail-risk events over time. On the other hand, achieving long-term in systems with latent variables- demands complex approaches to handle distribution shifts (e.g., reevaluation of long-term probabilities or retraining using new data), which impose additional challenges in real-time control.
is the unobservable latent variable, and H E 2 is the length of the episode. The mechanistic model is given by P P(X+11, Wx, W,, U2), which captures the transition of the underlying system dynamics.
We make the following assumption about the latent variable. Here, the notation I denotes statistical independence. Assumption 2.1. The sequence {We} satisfies
W (Wilret, fXtlret, (U+)rs
(I)
Due to Assumption 2.1, the transition kernel ad- mits the decomposition We+41,W+1|X4,, U,) ((),iX,)P(X++al Us, Wt). This condition gives Markovian property in the observable state X,, ie.
In this paper, we focus safety certificates for stochastic systems with latent variables, uncertainties with unbounded support, and actuation limits. We formulate invariance con- ditions in probability space in a way that accounts for latent variables and distribution shifts and can be computed using observed statistics. First, we show a relation between risk measures and marginalized value or Q-functions by employ- ing a modified Bellman equation-adapterr to account for latent variables. Next, we present the conditions on the control action that are sufficient to manage risk within a specified tolerance, based on invariance conditions in prob- ability space. Based on this relation, we then show action constraints can be obtained from observed statistics, even in the presence of distribution shifts between observed offline vs. online statistics. In particular, action constraints are con- structed to assure long-term safety with persistent feasibility, and this construction leverages probabilistic invariance and the inherent conditions satisfied by marginalized value or Q-functions. These conditions are then utilized to design a safety certificate that can be used by a myopic controller to guarantee long-term safety. This approach also allows the design of safety certificates easily exploit the large exist- ing body of literature from such domains: the risk measure can be evaluated using existing risk quantification meth- ods, and its equivalent form in marginalized value function and Q-function can be evaluated by causal reinforcement learning technique.
UI) = Prx (Ur)r51). (2)
Let D = D2. DNo denote the available offline data. Here, ND the size of the training dataset, and each individual data D,, 2 E {1.2..... .ND) contains the sequence of observable state (00,)_(0 .) == Xo.11 and control action Uo.H in an episode. The control ac- tions are generated by a behavioral policy Th, i.e., U, ~ T (U, Wt), which is assumed to be unknown. Accord- ingly, in dataset D, the observable state satisfies the follow- ing offline statisties:
Pottine (Xe+1 (x, Ui)
BW. (x)) (P(x+1 (x U, (U, Wo]
(T Wo)
(3)
On the other hand, the online statistics of the observable state X, is given by
Pomline (X X1, =
Ew.-P(W,IX,) (P(eerl U, Wol,
(4)
where the latent variable W, is marginalized. The unobserv- able nature of the latent variable also causes a mismatch between the online statistics (4) and the offline statistics (3) of the offline data D.
2. Problem Statement
Unlike the behavioral policy from the offline setting, in the online setting, a decision policy cannot depend on the latent variable because the latent variable W cannot be observed by an online controller. The online policy is designed to
2.1. System Model
We consider a confounded Markov decision process de- scribed by the tuple (X W, PP, H) Here, X, E X' is the observable (visible) state, U, E U is the action, We E W
'Throughout the paper, we use P to denote the mechanistic model, which is the transition keree including the complete state (observable and latent) in a Markov decision process or a con- founded Markov decision process, On the other hand, we use P to denote the statistical model, which is either the offline statisties (e.g., (3)) or the online statisties (e.g., (4)) that only involves the observable states.
'To avoid confusion with later context, we do not include re- ward when describing the process. It is trivial extension to incorporate reward and maximize the reward during control by adding the reward to the optimization objective in (55).
2

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
satisfy multifaceted design considerations. Some objectives, such as performance objectives, are captured in a nominal policy n", i.c., U,- ~ =")))..) The safety objective is ensured by a safety certificate. The safety certificate is represented by a mapping 5 : X x U x Z - IR, where a control action U is considered to be safe with respect to a state X at time 1 when the constraint S(X,, 1) 2 0 is not violated. Here, safety is characterized by an event C(X;) that can be evaluated by the state X,. For example, common definition for safety is that the state must remain in a safe set C. In this example, we have C(X.) = (X, E C). The long-term safety with respect to a certain control policy T at time is quantified by the long-term safe probability
model P(X++1, W441 Xt, Wr, U.) remains unchanged, the observed statistics can have distribution shifts between the offline vs. online settings: i.e., Pontine P (X++1 U+) A Pomino Pomino(Xx+1 (X++1|XX, U2). Such "spurious" distribution shifts can fail many data-driven techniques that do not explicitly differentiate observed statistics vs. mechanistic models, and prohibit the use of existing stochastic safe control techniques that require accurate transition dynamics or perfect simu- lators.* Similarly, due to such distribution shifts, one may obtain misleading estimates of risk probability and proba- bilistic reachability when they are learned solely from data (observed offline statistics). However, this vulnerability has received little attention in stochastic safe control.
Pa (C(,,) N (C())))(()))))))))))))) (--(x))))).-..)))))))() nC(XH))
(5)
2.3. Related Work
where the probability is calculated assuming the use of policy π in the closed loop system with the online statistics Pontine- Here, the policy π has the form π. : X' x Z + ( i.e., a mapping from æ E X' and time E Z to a distribution of control action in the action space U4 which we denote as A(4)3.
2.3.1. SAFE CONTROL
Many design techniques are developed for the safety certifi- cate in stochastic or deterministic dynamical systems. These techniques include barrier/Lyapunov functions (Nishimura & Hoshino, 2024; Wang et al., 2021a; Vahs et al., 2023; Jahanshahi al., 2020; Dean et al., 2021), barrier certifi- cates (Prajna et al., 2007; Ahmadi et al., 2020; 2018), and predictive safety filters (Wabersich & Zeilinger, 2018; 2023; 2021; Wabersich et al., 2021). For partially observed sys- tems with known dynamics, existing literature has proposed robust control barrier functions for deterministic systems with bounded estimation errors (Zhang et al., 2022; Dean et al., 2021; Wang & Xu, 2022: Qin et al., 2022b; Zhao & Yu, 2024), well as control barrier functions and bar- rier certificates constructed on belief space or estimated state for stochastic systems (Vahs et al.. 2023; Ahmadi et al., 2020; Carr et al., 2023; Jahanshahi et al., 2022; Clark, 2019; Dean et al., 2021; Jahanshahi et al., 2020; Ahmadi et al., 2018). When perfect simulators are available, bar- rier/Lyapunov functions can be designed through optimiza- tion problems that check the existence of functions satisfy- ing barrier/Lyapunov function conditions at all states using sampled data (Nejati & Zamani, 2023; Anand & Zamani, 2023; Dai et al., 2023; Qin et al., 2022a; Wang et al., 2023; Lindemaan et al., 2021; Xiao et al., 2023). These techniques are commonly built based forward invariance conditions in the state space" (Blanchini, 1999). Some techniques also construct safety conditions based on invariance conditions in the probability space (Wang et al., 2022; Jing & Nakahira, 2022) or its low dimensional representation (Hoshino et al., 2023) to ensure long-term safety. These conditions com- monly require complete transition dynamics and fully ob-
In this paper, our goal is to study the design technique for safety certificates that ensure not only that the safety of the immediate future state but also that states irrecoverably leading to risk (no feasible control action Ieads the system to safe future states) are avoided. Specifically, the proposed technique assures that the long-term safe probability condi- tioned on the initial observable state Xo is not smaller than a threshold 1 - for the entire episode, i.e.,
p ((((())))((()))))))))))))))))))) n C(X H) { 2 1-6, C,
Vt E 10. HJ.,
(6)
given certain initial conditions. Here, the probability is calculated assuming the use of online policy a for times HH
{0, ...,,......... t-1 and policy ı for times {t, in the closed loop system with the online statisties P Pantine- This is achieved by characterizing the action constraints S(X,, 2 0 that are sufficient to assure the safety objective (6) and are continuously feasible at all time t.
2.2. Technical Challenges
Due to the presence of latent variables, samples of com- plete state (Xe, W.h cannot be obtained, and thus the un- derlying transition dynamics P(X++1, W W., U,) is not fully identifiable. On the other hand, one cannot treat the observed statistics (statistics of the observed variable (X.).) as it the mechanistic model (underlying transi- tion dynamics). This because, even when the mechanistic
"For example, Figure I and 2 show that the safcty guarantee from these techniques can fail under such distribution shift.
SSee Appendix A for an example
'The system state stays within certain set if it originated from within the set. The information of the full system state and state dynamics is needed to check this condition.
In Section 3, we define the augmented state Y which captures time, so that the input to the policy π is Y.
3

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
servable states evaluate. It is also difficult to check such conditions using noisy data in stochastic systems or biased offline data resulting from the presence of latent variables.
that brings the state safety has to be made. Our method does not rely on such assumptions and gives persistent fea- sibility guarantee on the control policy. While we employ the method of Shi et al. 2024 for safe controller design, the proposed framework is expected to generalize to other causal reinforcement learning techniques and their respec- tive settings.
Existing work has also studied the techniques to handle dis- tribution shifts. A common approach is to avoid distribution shifts by constraining the system to stay within the states with known distribution. For example, Lyapunov density model is used constrain the state within the distribution sampled in the offline training data (Kang et al., 2022; Cas- taneda et al., 2023; Wu et al., 2023). Another approach assumes all possible transition distributions are known or can be samples, or policies or safety certificates for all pos- sible distributions can be sampled in advance. For instance, meta-learning is used to learn effective policies under differ- ent distributions (Guan al., 2024; Richards al., 2023). Our problems differ from the first approach in the sense that the distribution shifts (arising from latent variables) cannot be avoided. Our problem differ from the second approach in the sense that ours do not have access to offline samples for all possible statistics of observed variables, and data with latent variables are not accessible. In addition, these approaches focus on addressing distribution shifts caused by changes in the system dynamics, while our approach considers distribution shifts purely due to the presence of latent variables even if the system dynamics is unchanged.
3. Proposed Method
Before introducing the proposed method, we first define a function V : X x 2 - [0, 1] that captures the long-term terg safety probability with respect to policy и conditioned on state x::
vr (x, == 1et
x), (7)
where t S H. Here, the probability is evaluated with the as- sumption that the sequence X has statistics (4). We then define two auxiliary Markov decision processes (MDPs). The first MDP is described by the tuple (V.u, . Ponline: T; H), where Y, := [X] K,)" E X x Z := y' is the state, and U. E l is the control action. In this process, the sequence Xo.H has the online statistics (4) when C(X,) occurs and the transition Xe+1 = X, when C(X,) does not occur, i.e.?
Little safe control literature handles this type of distribution shift, and, to the best of our knowledge, this paper is the first one to do so.
Pontine (Xet1 (h,1(X,UUK, U.)
Poontine X141|X,, U2), 1 - X,),
C(X, occurs C(X,) doesn't occur,
(8)
2.3.2. CAUSAL REINFORCEMEN LEARNING
There has been extensive work in causal reinforcement learn- ing that aim to address the biasing (confounding) effect of the latent variables. Existing works have developed esti- mation methods for value function and/or Q-function in the context of confounded Markov decision process ( (Wang et al., 2021b; Chen et al., 2022; Bennett et al., 2021; Shi et al., 2024; Fu al., 2022: Xu et al., 2023) and partially observable Markov decision process (Bennett & Kallus, 2024; Miao et al., 2022; Shi et al., 2022). Many algorithms are developed for settings with different available informa- tion, such as the availability of backdoor/frontdoor adjust- ment variable (Wang et al., 2021b), proxy variables (Miao et al., 2022: Bennett & Kallus, 2024), and instrumental variables (Chen et al., 2022; Fu et al., 2022). While these approaches offer techniques manage latent variables in diverse settings, to the best of our knowledge, none have been applied to stochastic safe control problems with per- sistent feasibility guarantee. This paper bridges that gap by introducing a framework that integrates causal reinforce- ment learning into safety certificate design. Although there exist methods such as Srinivasan et al. 2020 that use Q- function to represent safety and integrate safe control with learning, critical assumptions about the existence of actions
where 5(.) is the Dirac delta function. We define Ko to be sequence that captures the remaining time in the episode. Its statistics satisfies Ke+1 = K₁ 1, i.e.
Protine K2) = 8(Ke+1 -
(9)
The transition is given by Pontine 1Y4zl/e, U.)
kernel
of the MDP
P Pontine (X++1) |x,, U.)Pooliiee ( Kt).
The reward
function r r. - {00,1 is defined by
r <(~ k)7) = 1(k 1[k=0))(C(()),
(10)
where 1{E} is the indicator function which takes the value if event & occurs and 0 otherwise. The second MDP is described by the tuple (V, 21, Portine r, H). where Y, := [X," K,IT E y is the state. In this process, the sequence Xo.H has the offline statistics (3) when C(X;) occurs and
'Here, with slight abuse of notation, we use Pontiee ( Xx, U2) = Poutine PX U.) represent that, when X. Xx. X.+1 has the same distribution as X when the statistics of Xo.1 is Ponine- We use this notation system in (11) as well.
4

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
the transition X X, when C(X,) does not occur, i.e.,
Throughout this paper, we use the subscript in the expecta- tion to denote the distribution the transition kernel where the expectation is taken over.
P Pottino (Mttno(Xxe (Xe+1|Xx,.U))
Pommme& (Xt-1 Uc) ((Xi) occurs 8 Xt),
(11)
Proposition 3.1. Consider the marginalized value function defined in (12) for P Pontine and the long-term safe probability defined in for P Ponline- We have
((2)) doesn't occur.
The is
transition
kernel of the MDP P Pottine 1e41lYy,Ug
given by
V"(["", kP (2 H --k k)
(16)
Pottine (Xe+1lX3 Ke. U.) (K++1 Kt), Pomtie [K441 K₂) = Ponline ( (K44 K.).
where
for all E X and k E Z.
The rest of Section 3 is organized as follows. In section 3.1, we show that certain value function defined for the MDP (V,U, P Ponline H) is equal to the function TI. In Sec- tion 3.2, we introduce safety certificate formulated based on the value function and show that the satisfaction of the safery certificate provably ensures the safety objective (6). In Section 3.3, we propose an equivalent condition to the safety certificate that can be evaluated using certain Q-function defined for the MDP (2, Pontine; T; H). We then show that there always exists a control action U2 E u such that this condition is satisfied. In Section 3.4, we show that, using offline dataset D, one can obtain offline dataset D that has sequences with statistics Pomtine which can be used to learn value function and/or Q-function defined for the MDP (y,24 Pontine, r, H) with existing causal reinforce- ment learning methods. We then propose an integrated safe control algorithm using one existing causal reinforcement learning method as an example.
The proof is given in Appendix B.
3.2. Safety Condition
Here, we present sufficient condition to satisfy the safety objective (6) using the value function representation. We consider the condition
L Pustic
IV" (Y++1)) -VV 2 0, (17)
where y, [x?,, H 11 10, H -1..
Theorem 3.2. Consider the marginalized value function
defined in (12) for Ponline and the long-term safe probability
defined in (7) for Pontine- If upaT (Xu. 0) > 1 and the condition (17) is satisfied at all times E {0, 1,...... -1,,
then the safety objective (6) for the system with transition kernel P and online statistics P Ponline holds.
Proof. We have
3.1. Value Function Representation for Long-term Safe Probability
We consider the value function representation inspired by (Hoshino & Nakahira, 2024). We define the marginal- ized value function V : y - [0, 1] and the marginalized Q-function Q : y x 4 - [0, 1] with respect to the MDP (,, Pontine, r, H):
-Ep, xx)
(o) (X,, ())o)
(18)
where Px) is the conditional distribution of X, given Xo assuming the sequence Xox has statistics Poaline and a policy á is used for times {0, 10..,..... ,t 1}. From Propo- sition 3.1, we have that VT([=", ,kJ²) V= H k). Therefore, to prove the theorem, it suffices prove the following statement: if
(= [1(²) := EEp Pretne
rf:))10 AlT, m]
(12)
V ("' HJ) V*(()) > 1
(19)
EE
r(:))1%0 A'"" m]
adise
and the condition (17) is satisfied at all times t 1, 1,
7=0
(13)
H - 1}. then
(Tx 11TT u) =
E (P. [V (8:)) 2 1-e
(20)
Ep Puine 7=0
Σ
(Y:))1%0 = -==", ,Uo = - (14)
holds for all t E 10, H}, where Ps (Yr))) is the con- ditional distribution of y, given Yo assuming the sequence yo. has statistics Pentine and policy Å is used for times 1, , -t-1) 1-11. 1}. Note that we consider the policy स to be the online policy here. Since the online policy produces a deterministic control action, we define the online policy as a mapping y - 24 so that we have 4 (V.) = U, for all
= BB
= =
A'" Uo = u, T]. (15)
Patiee
7=0
Here, we may sum to infinity because, given i- - RT.
we have r((-)) = 0 for all 7 > 1 due to definition (10).
5

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
E 10,1,- H 1). We show (20) holds for all times t 10. 1.... H using mathematical induction. We first show that (20) holds at time O. We have
Proof. For all times l {0,1,... .H 1). the modified
Bellman equation gives
Q (YY,, =r(Y)) + EIV (Y4+)) EIV" U.]
(28)
(21) holds due to (19). We then show that, given (20) holds at time t, it also holds at time t + 1. Taking conditional expectation over the conditional distribution P= (o) on both side of (17) yields
A (Ya) = (P. (V" Pol
(29)
since r(Y.)) = for all times / + H. By definitions (12)
and (14), we have
V" = E u 1Q" (8, U)]]] U),
(30)
Bs (811))) Ep Pame B. IV (F++1))4 8.
E (2 (VT (Y:)
(22)
Combining (29) and (30), we have
S(Xr,Ut.1 =Q"
From the law of total expectation, we have
- 11"
(31)
(8+11) -B, 1o) 1. ( (h4 (V=()))))) (M))))))
PP, (P: Yo) 16 p P 1 (Y(+)) Yn (.118)
=EIV -
(32)
(23)
(P 180) (1+1)))8).
(24)
To ensure the safety objective, there must always exist feasi- ble control action that doesn't violate the safety condition (27). Here, we present a provable guarantee for such persis- tent feasibility.
Therefore, we have
EA (T (V" 2 (M IV (9.)))) 2,
(25) Theorem 3.4. For all times t E (0, . - 1}, there
(26) always exists U, E U such that (27) holds.
which gives that (20) holds at time + 1.
Proof. Consider
3.3. Evaluation of Safety Condition and Persistent Feasibility Guarantee
u arg max Q" (Y,, u), wEM
(33)
Evaluating (17) can be difficult, since even if the marginal- ized optimal value function V is available in closed form, the term (P=.) "V." (14+1)]8,, Ui] cannot be evaluated since the distribution Ponline (Ye42) Pe, U2) is un- known. Here, we show a condition that guarantees the satis- faction of (17) and can be evaluated with only the marginal- ized Q-function Q":
we have
Q"(Y,, (1., Ed.
(34)
We also have
J (U = =u y)d = 1, E v.
(35)
8(X,, Uc,t) (" U,) E EU (%.
(P. U) 2 0.
(27)
Due to (34) and (35), we have
-Q f. T(U = u ( (36)
where y, = - E 10. H H 1). This formulation allows the Q-function obtained from causal reinforcement learning techniques to be used for evaluating the safety condition. We then show that the satisfaction of this safety condition implies the satisfaction of the safety condition (17).
= =1 Q'(Y, = u YY)ddu
(37)
J Q"(,,))T(U = u u.
(38)
Lemma 3.3. Consider the marginalized value function de- fined in (12) and the marginalized Q-function defined in (14). For all times t E {0, 1, .. 1}, if the control action U, E U satisfies (27), then it also satisfies (17).
E E
(39)
As U, = u² e U satisfies (27), there exists a control action in l4 that satisfies (27).
6

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
3.4. Proposed Algorithm
define the Q-function conditioned on the mediator
Before introducing the proposed algorithm, we first show that, even if the MDP (),,4, Poftine r, H) is an auxiliary process and does not have the corresponding physical sys- tem, we can obtain dataset D = {D1, D2. ...., DN } using the available dataset D. Here, each individual data D. con- tains the sequence of state ooH and control action U/ in an episode, and the sequences follow the statistics Pottine- We propose Algorithm 1 that generates DD using D. Using data D, one can estimate the value function and Q-function defined in (12) and (14) using existing causal reinforce- ment learning methods. Here, we introduce an example method that uses the mediator variable to learn unbiased Q-function (Shi et al., 2024). This method utilizes the front- door adjustment (Pearl, 2009) to counter the confounding effect. Note that the model and assumption introduced in this subsection are specific the application of the corre- sponding method only. The proposed method works with any causal safe control methods whose model satisfies As- sumption 2.1.
Qhe (1x", [k]", mmm))= == k E ((()))18 [0= 11)" Ub u, Mo m,
T=0
(40)
E
Σ r(())1 1100 MA,, UU-- - u, Mo - m, T..
mnine
T=0
(41)
The Bellman equation is given by
QM (Y,, Mi)
=x(Y)) + EB (8. IV (Y++4))))) Un, M.]
(42)
=r(Y)) + B (8 M (1+ta)) U,,,MM,, Me),
(43)
Algorithm 1 Generation of D using D
where (43) holds because Pontine (1441) Pontine(Pr+)
18,, Uc. M,)
1: Input: offline dataset D
Pottinn (Y,+ Mc) (Ponline( Ye, U2, Mc) can be consistently estimated from offline data (Pearl, 2009)). Due to (43), we estimate Qh, iteratively with data D by solving
2: D+@ 9
3: for in (1,2, ,N) do
4: (X).n U/ + D;
5:
Xo S
6:
Yo + [[8 H|T
No H-1
7:
for in 10.1,...,, H - 13 do
arg min QEo
(F(?)) - Q0),) + (((t+1)
8:
if (Xe, occurs then
1=
9:
8 + X else
(44)
10:
11:
X++1 + X, end if
at iteration L + 1, where Q is the class of functions of the form y x U x M - [0, 1]. and 1 is the estimation for the value function V in iteration l. To evaluate (44), one needs to evaluate V using Q%, and known offline statistics. The value function can be written as
12:
13:
YiLl + PXT, H - t 1|T
14:
end for
15:
D. - (Y 3
16:
D DU
17:
end for
18:
Return D
- (V' = =
(45)
We define an observable mediator variable M, E MS, con- sider the spaces X, U1 W and M to be discrete, and make the following assumption.
wEll
>' B Punnn 11 IV (Pe+1)
wEM/
A Assumption 3.5. The mediator M, intercepts every directed path from U, to S.+1. The observable state X, blocks all back-door paths from U2 to My. All back-door paths from M2 to X are blocked by U.).
+ +r(y) = y. U.U. u]-(U, = =ulfi= = =y)
(46)
ΣΣ (5')+r())))
wEll v' PonlineYY = s'1i/ = y, U. = u)= =u(Yi v),
Here, the definitions for directed path and back-door path follow the definitions in Pearl 2009, Chapter 3.3.2. We also
(47)
Since M. is observable, in this example, the sequence Mon E 21,2,-.., ,ND] is also in the offline dataset D.
where (46) is due to the Bellman equation. From the front-
door adjustment formula in Pearl 2009, Chapter 3.3.2, con-
7

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
ditioned on Y, we have
as possible, we use the following optimization problem to obtain the safe control action:
Pontine (Ye+1 = v'Uue = u. u. 0 =1)
Σ Σ Poftine (Y--1 - y/ 3/|Ue= = =u', - = =m, Y, y)
arg min J(U", u) wEld
(55)
mEM MEMMu W'El
s.t. S(X,, U, t,t) 2 0,
Postine (Ur = u u'(';r = y) (Me = m/U - U, Yz - y)
(48)
where U the control action obtained from the policy x" and J : U x U R is function that penalizes deviation from U".
given Assumption 3.5. Substituting into (47), we have
V (v)
Algorithm 2 Proposed control algorithm
Σ Σ
1: Require: offline dataset D
2: Obtain dataset D with dataset D and Algorithm
uEl Y'EY
E
Σ
Pottine (Ye+1 = =y'lUe M = m, Y, = v)
M₁
4: Initialize 2 E eQ
[ [+0
MEM W'EM
5: while not converged do
P Pontiee (Ut- = n/ IV1 = v) U) (M₁ mlU = u, Y2 v)
6:
(% < (44)
T(U = u| = y)
(49)
7: (+
8: end while
EΣΣΣ(NE (V)(')) ( +()))
9: QT + QN
MEM W'El well y'Ey
10: t-0
Pottine (8+11 = 1/ 1U, u', M₁ m, m, = 3)
11: while t < H do
Pomtiee (Ue = w/ I = y) Po Pf (M, = m mlUU- = u, Y, v)
12: observe state X,
13: U ~ H"" (-1x)
T <U, = = y))
(50)
14: Estimate Q" using (54) with Q7:
15:
U.+ (55)
Similar to (47), from Bellman equation (43), we can write Qh as
16:
execute control action Ug
17:
t t-t+1
Qy m) = Σ (V*())++r())
18:
end while
3'Ey
PomtinePP+ = u = u, M, = m, Y: = y).
(51)
4. Numeric Simulation
Substituting into (50), we have
We consider a setting that resembles simplified driving sce- nario with discrete state space. Let X, = [X,, X?I" E z² be the state of the system, where X represents the po- sition of the vehicle on 1-dimensional road, and X? represents the velocity of the vehicle. The control ac- tion U,- E {-3, -2. -1, 0, represents the acceleration or deceleration applied to the wheels. The latent variable W {0, 1, 2, 3} represents the slipperiness of the road, which can make the acceleration or deceleration applied to the wheels less effective. The system also has uncertainty E (-1, 1) x -1,0, The system transition is given by
V (²))
ΣE
O5r u', m) (U, = u/(²r= = y)
MEM w'Ellllee well
P Pontiie = mm = Yr =
=)= (U, = u = v),
(52)
whose RHS only includes Qh, and distributions in the of- fline statistics. It is also easy to obtain Q" using Qhe We have
N₂
("(
-Ep (M, (M, (2, (Pr, U,, Mo = y, U.U,- = u| (53)
X/41 =X + X²
(56)
-Ep mm (M. Un, M,) = yy, Ur = u]
X = max (0, x2
(54)
sign(U; + N:) max(0, IÜ, + NNN - Wt) + N2?.. (57)
because Pontine Y.) P Pottine [M,U, Yt). We intro- duce the proposed algorithm in Algorithm 2. To ensure the safety objective while preserving as much performance
The distributions of W, and N₂ are given in Appendix C. The safety requirement is that the vehicle obeys a varying
8

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
speed limit. Specifically,
((x,)=(1(X) mod 10 < <4) (X² < 3}}
( mod 102 4] { I 5)). (58)
043
The offline dataset can be considered as human driving dataset where the human observes the slipperiness of the road in their behavioral policy, but the slipperiness is not recorded by the sensor. Specifically, we consider a behav- ioral policy x² that applies heavier brakes when the road is more slippery. The detailed distribution for the behavioral policy is given in Appendix C. The nominal policy simply chooses actions in the action space with identical probability, i.e.. [X, = 0.2, YU, E {-3, -2, -1, 1,0,1), X, E z². We consider H = 10 and = 0.2. We run 100 simulations, where each simulation simulates 100 trajectories starting from Xo = [0, 0j", with the following 2 methods:
Sole: hitehheeeo Bty
640
0.25
0.7
Time
Figure I. Probability of safety at each time for both controllers with 95% confidence interval shown in the shady region.
1. The proposed method. The proposed method has ac- cess to an unbiased estimate for the Q-function Q", which can be estimated using causal reinforcement learning method such as Wang et al. 2021b and Shi et al. 2024.
Propooed Mectee
DTCLPP
ferm Tiduteer
0.32
2. The discrete-time control barrier function (DTCBF) proposed in Cosner et al. 2023. This method cannot uti- lize the Q-function, so the safety condition is evaluated using the distribution obtained from the offline dataset. The detailed parameters are given in Appendix C.
0.00
0.78-
0.7
For both methods, the control policy is to maximize the control action while ensuring the corresponding safety con- dition.
Figure 2. Long-term safety at each time for both controllers with 95% confidence interval shown in the shady region. The long-term safety is equal to pa (C(X) n C(X++1) n... C(.,) defined in (6).
The simulation results are illustrated in Figure and 2. The results show that the proposed controller, although having access to the latent variable W or the ground truth transition dynamics, can achieve a safety performance that satisfies the safety objective (6) with the Q-function. On the other hand, the discrete-tine time control barrier function cannot achieve the safety objective with the offiin statistics even if the control action satisfies the safety condition.
certificate to guarantee long term safety with persistent fea- sibility, even when the system's dynamics are only partially identifiable. Beyond the proposed algorithm, we expect the proposed framework can be extended to incorporate other causal inference techniques for the design of safety certificates.
5. Conclusion and future work
Furthermore, while many techniques are developed for prob- abilistic reachability, its standard definitions involving all states may not be computable from observed statistics with- out complete system models (see Section 2.2). On the other hand, in Theorem 3.2 and Theorem 3.4, such definitions of reachability may not be necessary for the purpose of safe control: For example, the liku function, learnable from observed statistics, can be used to guarantee safety and persistent feasibility. This suggests a future direction: to explore new measures of reachability and latent risk that are both learnable from observed data and sufficient for safety certification.
While there may be a common belief that distribution shifts only happen when system dynamics change, spurious distri- bution shifts can arise due to the use of observed statistics with latent variables. In this paper, we introduce a novel approach for learning safety certificates using only observed statistics with such distribution shifts.
To our knowledge, the proposed work is also the first to apply causal reinforcement learning in the design of safety certificates. The framework leverages a new notion of in- variance-prre -robabl--oobaailisstcc invariance- that enables a safety

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
Acknowledgment
Blanchini, F. Set invariance in control. Automatica, 35(11):
1747-1767, 1999.
This work is sponsored in part by Carnegie Mellon Uni- versity Security and Privacy Institute (CyLab), part by the PRESTO Grant Number JPMJPR2136 from Japan Sci- ence and Technology Agency, in part by National Science Foundation under Grant No. 2442948, and in part by the Department of the Navy, Office of Naval Research, under award number N00014-23-1-2252. The views expressed are those the authors and do not reflect the official policy or position of the US Navy, Department of Defense or the US Govermment.
Carr, S., Jansen, N., Junges, S., and Topeu, U. Safe rein- forcement learning via shielding under partial observabil- ity. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 14748-14756, 2023.
Castaneda, F., Nishimura, H., McAllister, R. T., Sreenath, K., and Gaidon, A. In-distribution barrier functions: Self- supervised policy filters that avoid out-of dit-o--distribution states. In Learning for Dynamics and Control Conference, pp. 286-299, PMLR, 2023.
Impact Statement
Chen, Y., Xu, L., Gulcehre, C., Le Paine, T., Gretton, A., De Freitas, N., and Doucet, A. On instrumental variable regression for deep offline policy evaluation. Journal of Machine Learning Research, 23(002): 1-:22):14440 1-40, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Clark, A. Control barrier functions for complete and incom- plete information stochastic systems. In 2019 American IEEE, 2019.
References
Control Conference (ACC), pp.
Ahmadi, M., Cubuktepe, M., Jansen, N., and Topeu, U. Ver- ification of uncertain pomdps using barrier certificates. In 2018 56th Annual Allerton Conference Communi- cation, Control, and Computing (Allerton), pp. 115-122. IEEE, 2018.
Cosner, R. K., Culbertson, P., Taylor, A. J., and Ames, A. D. Robust safety under stochastic uncertainty with discrete- time control barrier functions. In Bekris, K. E., Hauser, K., Herbert, S. L., and Yu, J. (eds.), Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, URL https:
2023. doi: 10. 200.15607/RSSS2023.... / ///oi.oorgg.o. / 560 /RS. 2023 XTX. 084.
Ahmadi, M., Jansen, N., Wu, B., and Topeu, U. Control theory meets pomdps: A hybrid systems approach. IEEE Transactions on Automatic Control, 66(11)55191-5204, 2020.
Dai, B., Krishnamurthy, P., and Khorrami, F. Learning a better control barrier function under uncertain dynamics. arXiv preprint arXiv:2310.047795 2023.
Ames, A. D., Xu, X., Grizzle, J. W., and Tabuada, P. Control barrier function based quadratic programs for safety crit- ical systems. IEEE Transactions on Automatic Control, 6266)3386 2016.
Dean, S., Taylor, A., Cosner, R., Recht, B., and Ames, A. Guaranteeing safety of learned perception modules via measurementtrobuss control barrier functions. In Confer- ence on Robot Learning, pp.
Ames, A. D.. Coogan, S., Egerstedt, M., Notomista, G., Sreenath, K., and Tabuada, P. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pp. 3420-3433 IEEE, 2019.
PMLR, 2021.
Fu, Z., Qi, Z.. Wang. Z.. Yang, Z., Xu, Y., and Kosorok, M. R. Offline reinforcement learning with instrumental variables in confounded markov decision processes. arXiv preprini DXXiv:2299008666 2022.
Anand, M. and Zamani, M. Formally verified neural net- work control barrier certificates for unknown systems. IFAC-PaperOOnLLiee, (112):2431-2436 2023.
Guan, C., Xue, R., Zhang, Z., Li, L., Li, Y.C.,, Yuan, L., and Yu, Y. Cost-aware offline safe meta reinforcement learning with robust in-distribution online task adaptation. In AAMAS, pp. 743-751, 2024.
Bennett, A. and Kallus, N. Proximal reinforcement learn- ing: Efficient off- off-policy evaluation in partially observed markov decision processes. Operations Research, 72(3): 1071-1086, 2024.
Hoshino. H. and Nakahira, Y. Physics-informed rl for max- imal safety probability estimation. In 2024 American Control Conference (ACC), pp. 3576-3583. IEEE, 2024.
Bennett, A., Kallus, N., Li, I.., and Mousavi, A. Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders. In International Conference on Artifi- cial Intelligence and Statistics, pp. 1999-2001, PMLR, 2021.
Hoshino, K., Wang, Z., and Nakahira, Y. Scalable long-term safety certificate for large-scale systems. IEEE Control Systems Letters, 7:1285-1290, 2023.
10

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
Hsu, K.-C., Hu, H., and Fisac, J. F. The safety filter: uni- fied view of safety-critical control in autonomous systems. Annual Review of Control, Robotics, and Autonomous Systems, 7, 2023.
Intelligent Robots and Systems (IROS), pp. 12903-12910. IEEE, 2022b.
Richards, S. M., Azizan, N., Slotine, J.-J., and Pavone, M. Control-oriented meta-learning. The International Journal of Robotics Research, 42(10):777-797, 2023.
Jahanshahi, N., Jagtap, P., and Zamani, M. Synthesis of stochastic systems with partial information via control barrier functions. PMAC-PaperSOnLLine 2342):244-24466, 2020.
Schölkopf, B., Locatello, F., Bauer, S., Ke, R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5):
Jahanshahi, N., Lavaei, A., and Zamani, M. Composi- tional construction of safety controllers for networks of continuous-space pomdps. IEEE Transactions on Control of Network Systems, 10(1):87-99) 2022.
612-634, 2021.
Shi, C.. Uchara, M.. Huang, J., and Jiang, N. A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In Inter- national Conference on Machine Learning, pp. 20057- 20094. PMLR, 2022.
Jing, H. and Nakahira, Y. Probabilistic safety certificate for multi-agem systems. In 2022 61th IEEE Conference on Decision and Control (CDC). IEFE, 2022.
Shi, C.,, Zhu, J., Shen, Y.. Luo, S., Zhu, H., and Song, R. Off- policy confidence interval estimation with confounded markov decision process. Journal of the American Statis- tical Association, 119(545):23--288 2024.
Kang, K., Gradu, P., Choi, J. J., Janner, M., Tomlin, C.,, and Levine, S. Lyapunov density models: Constrain- ing distribution shift in learning-based control. In Inter- national Conference on Machine Learning, pp. 10708- 10733. PMLR, 2022.
Srinivasan, K., Eysenbach, B., Ha, S., Tan, J., and Finn, C. Learning to be safe: Deep with a safety critic. arXiv preprint arXiv:2010. 14603, 2020.
Lindemann, I.,, Hu, H., Robey, A., Zhang, H.,, Dimarogonas, D., Tu, S., and Matni, N. Learning hybrid control barrier functions from data. In Conference on robot learning, pp. 1351- PMLR, 2021.
Vahs, M., Pek, C., and Tumova, J. Belief control barrier functions for risk-aware control. IEEE Robotics and Automation Letters, 2023.
Miao, R., Qi, Z., and Zhang, X. Off- Off-policy evaluation for episodic partially observable markov decision processes under non- parametric models. Advances in Neural Infor- mation Processing Systems, 35:593-606, 2022.
Wabersich, K. P. and Zeilinger, M. N. Linear model pre- dictive safety certification for learning- control. In 2018 IEEE Conference on Decision and Control (CDC), pp. 7130-7133 IEEE, 2018.
Nejati, A. and Zamani, M. Data-driven synthesis of safety controllers via multiple control barrier certificates. IEEE Control Systems Letters, 7:2497-2502, 2023.
Wabersich, K. P. and Zeilinger, M. N. A predictive safety filter for learning- basen control of constrained nonlinear dynamical systems. Automatica, 129: 109597 2021.
Nishimura, and Hoshino, K. Control barrier functions for stochastic systems and safety- control designs. IEEE Transactions on Automatic Control, 69(1 [1 1):8088- 8095, 2024.
Wabersich, K. and Zeilinger, M. N. Predictive control bar- rier functions: Enhanced safety mechanisms for learning- based control. IEEE Transactions on Automatic Con- trol, 68(5):2638-2651 2023. doi: 10.1109/TA..202.. 3175628.
Pearl, J. Causality. Cambridge university press, 2009.
Prajna, S., Jadbabaic, A., and Pappas, G. J. A framework for worst-case and stochastic safety verification using barrier certificates. IEEE Transactions Automatic Control, 52 (8):1415-1428, 2007.
Wabersich, K. P., Hewing, L., Carron, A., and Zeilinger, M. N. Probabilistic model predictive safety certification for learning- control. IEEE Transactions on Auto- matic Control, 67(1):176-188, 2021.
Qin, 7.,, Sun, D., and Fan, C. Sablas: Learning safe control for black-box dynamical systems. IEEE Robotics and Automation Letters, 7(2):1928-111) 19088-199,1101 1,,,-193,,,,2200 2022a.
Wabersich, K. P., Taylor, A. J., Choi, J. J., Sreenath, K., Tomlin, C. J., Ames, A. D., and Zeilinger, M. N. Data- driven safety filters: Hamilton-jacoe reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5):137~ 177, 2023.
Qin, Z., Weng, T.-W., and Gao, S. Quantifying safety of learning- shaning-based self-driving control using timost-bbrrice functions. In 2022 IEEE/RSJ International Conference on

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
Wang, C., Meng, Y., Smith, S. L.., and Liu, J. Safety-critical control of stochastic systems using stochastic control bar- rier functions. In 2021 60th IEEE Conference Decision and Control (CDC), pp. 5924-52252 1991. IEEE, 2021a.
Wang, C., Meng, Y., Liu, J., and Smith, S. Stochastic control barrier functions with bayesian inference for known stochastic differential equations. arXiv preprint arXiv:2312.127799 2023.
Wang, L., Yang, Z., and Wang, Z. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Sys- tems, 34:21164-21151 2021b.
Wang, Y. and Xu, X. beserver-based control barrier func- tions for safety critical systems. In 2022 American Con- trol Conference (ACC), pp. 709-714. IEEE, 2022.
Wang. Z., Jing, H., Kurniawan, C., Chern, A., and Nakahira, Y. Myopically verifiable probabilistic certificate for long- safety. In 2022 American Control Conference (ACC), pp. 4894 4900. IEEE, 2022.
Wu, J., Wang. Y., Asama, H., An, Q., and Yamashita, A. Risk- sensitive mobile robot navigation in crowded en- vironment via offline reinforcement learning. In 2023 IEEE/RSJ International Conference Intelligent Robots and Sysiems (IROS), pp. 7456-7462. IEEE, 2023.
Xiao, W., Wang, T.-H., Hasani, R., Chahine, M., Amini, A., Li, X., and Rus, D. Barriernet: Differentiable control barrier functions for learning of safe robot control. IEEE Transactions Robotics, 39(3):2289-2300 2023.
Xu, Y., Zhu, J., Shi, C., Luo, S., and Song, R. instrumen- tal variable approach to confounded off-policy evaluation. In International Conference Machine Learning, pp. -8848-38880. PMLR, 2023.
Ye, W., Zheng, G., Cao, X., Ma, Y., and Zhang, A. Spurious correlations in machine learning: A survey, arXiv preprint 127755 2024.
Zhang, Y., Walters, S., and Xu, X. Control barrier function meets interval analysis: Safety-critical control with mea- surement and actuation uncertainties. In 2022 American Control Conference (ACC), pp. 3814-3819. IEEE, 2022.
Zhao, C. and Yu, H. Robust safety for mixed- autonmy traffic with delays and disturbances. IEEE Transactions on Intelligent Transportation Systems, 2024.
12

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
A. Mismatch between Online Statistics and Offline Statisties: An Example
Consider a system with observable state X; E {0, }} and latent variable W, E {0, 1]. The control action is U, E {0, 1}.
The system is considered to safe when X; 0. Suppose that the state transition probabilities are given by
P(X++1 = 0 XX; = 0, W. = 0, U2 = 0) = 0.9
(59)
P(X++1 = 0| o|X; = O, W; 1,U; = 0) =
(60)
P(X++1 1 = O|X; 1, W, 1,U; =0) 0) =
(61)
P(Xc+1 = 0|X; O, W; 0,U; ===]] 1]
(62)
P(X,+1 = O|X; O, W. = 1,U, = 1) = 0.1
(63)
P(X,+1 = X, = 1, W. = =1,U = 1) = 0.
(64)
The value for W. is determined by
P(W, 0|X; 0) 0.5
(65)
P(W, 0|X; =) O.
(66)
Note that PP( 11--== =1 1 1( -).), and omit the case for X, 1, W. =0 because it is reachable from
other states. The behavioral policy x² is given by
n (U, X, 0, W. =0) 0.5
(67)
π (U, = X = 0, W/ = 1) = 1.
(68)
We find out the safe probability of the immediate next time step given U. = 1 for both the offline statistics and the online statistics. We have
Ew (W,.- = 0, W, 1)
W.).
PP Pottine (X++1 = 0, =0,
(69)
[PU = 0,
(70)
and
P Pontine (X1+1 0|X, U,U; =Ew -w,, 0|X, 0, W., U, 1)]
(71)
-0.55.
(72)
We observe that the safe probability evaluated from the online statisties is significantly lower than the safe probability evaluated from the offline statistics. This shows that if controller uses the offline statistics to perform safe control, the safe probabilities associated with some control actions will be significantly over-approximated.
B. Proof of Proposition 3.1
Proof. Consider a function - x Z (0,
10 defined as - = =x),
V (x, ==
(73)
where the probability is evaluated with the assumption that the sequence N has the statisties (8). We first show that
(x, = 1/" (=x (x,,t,.V X,t 10,1..... H..
(74)
We have
f fx 1(C(X) n ( = x)dX
1 (xr,t)
(75)
where P (X XX, = æ) is the conditional distribution of the sequence X given X; when the sequence has statistics (4) and a policy x is used. Similarly, we have
T = - ICCkk//no......---ncCCc =
z)dd....
(76)
13

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
where Pr (N [X, = x) is the conditional distribution of the sequence X X given X, when the sequence has statistics (8) and policy is used. Note that, when the sequences Xt.1r and N take the same value, Px P*(XE=I X; = x) + Pe (Xum XX, z) only if there exists time { ,,, such that C(X²) (and C(X;) since X, = xx,) does not occur. In such case, we have ((C(X) n C(X++1) n n..x))(xx))) = 1(C(X,) n ((((+))) n. C(X+)) 0. Therefore, we have (74). Next, we show that
V 11 k) =
-=(<= (['" Vo 10,1, HI.
(77)
We have
V" V='(2" ([=x",))²)) = y
Σ m(X? ((Ix%.KI1)) pa (Pox188=
= =Ix", [ )df
(78)
where pa (Yox - |x" k)) is the conditional distribution of the sequence You given 8o - [x" A when the sequence
has statistics Pontine Pontino(Y (Yi-1 1Y,, U.) and policy π is used. Since r(xx", k)") * only if A = 0, and Kr = when T given Kg k, we have
(T ,1") = .SS. (1X%)00²) (Yo 1oo
-1= Pr = )dYo.
(79)
Since the distribution of sequence Xork and the distribution of sequence Kork are independent, we have
V" ([x [²)) - SSr.. r(XT7 0)') (Xox x)dRu..
(80)
From (76), we have
1 (a (x, H -k)
k)
- ICC -
(81)
x)
(82)
Note that r(XX,
) > iff C(X,) occurs for 7 E {0, 1,... kk,, which gives r(188,007) 1(C((x)
n CCK,)) Therefore, we have (77).
C. Details in Simulation
The distribution of W. is given by
P(W, 0) PP(W, =1) 1)= 1
(83)
if X! mod 6 6 3 and
P(W; = 1) = P(W, = 2) = P(W, = 3)
(84)
if X! mod 6 < 3. The distribution of N₂ is given by
P(N7 = -1 - -) = 1) 3
(85)
and
P(N - -22 =P(N? = -1) = PPN? = 0) -) P(N =2)
(86)
14

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
The behavioral policy x6 is defined as follows. When W; 1, X mod 10 4, and X 2 2, the policy satisfies
(U, ,,,,, W₂) 0.5
(87)
" (U, = -2,,,, Wt) 0.4
(88)
N (Ur - -1 Wi) 0.05
(89)
T TU O|X,, Wx) 0.04
(90)
T TU, = 1|x, W2 = 0.01.
(91)
When W, 2 1, X; mod 10 2 4, and X? 2 4, the policy satisfies
N (U, = - 3 X,, Wi) 0.5
(92)
n² (Ug _2,,,, W₂) 0.4
(93)
"" (U, 1|,,, Wo) 0.05
(94)
t TU, = 0|X,, W)) = 0.04
(95)
T TUUU, 1|x,, W;) 0.01.
(96)
When W, 2 2, X mod 10 < 4, and X² 2 1, the policy satisfies
7² (U, -3,,, W;) 0.5
(97)
#² (U, -2,X,, Wo) 0.4
(98)
" (Ue = 1|,,,, W)) = 0.05
(99)
T TUU O|X,, W,) 0.04
(100)
T'U 1|X,, W/) 0.01.
(101)
When 2, X7 mod 10 4, and x² 2 3, the policy satisties
#" (U; X,, We) 0.5
(102)
n (Ue = 22, Wt) = 0.4
(103)
n (U, - -1,,,, Wx) 0.05
(104)
" "U, O,X,, WT) 0.04
(105)
t TU, = 1|X,, W₂) 0.01.
(106)
When W, 2 3, x? mod 10 < 4, and x? 2 2, the policy satisfies
* :', = X|,,, WT) 0.9
(107)
+ (U, 22,,,, W,) 0.05
(108)
a (U, 1|x,, W,) 0.03
(109)
+ T = 0|X,, Wx) 0.01
(110)
" *² 1[X,, W;) 0.01.
(111)
When W, 3, X! mod 10 2 4, and x? 2 4, the policy satisfies
*² (U, X3,,, We) 0.9
(112)
*² d -2,,,, W,) 0.05
(113)
n (U, = 1| Wc) 0.03
(114)
" ? 0|X,, W:) 0.01
(115)
" + 1|X,, Wx) 0.01.
(116)
15

---

Safety Certificate against Latent Variables with Partially Unidentifiable Dynamics
Otherwise, the policy satisfies
(U; -3 W.) (V, - W,) T'U 1 WT) (U; 0|X,, Wc) W;) = 0.2.
(117)
For the discrete-time control barrier function, we first represent the safety requirement using C(X;) - 1(X, E CJ, where fx E z² h(x)) 2 0}. and
tanh
4.5+
Σ
sin sin( -mmm-n((² + +0.5)) x²²
(118)
12IT
n,/1,3,5,7)
We use the safety condition
[[ ah((.) +
(119)
with a 0.01 and 8 -0.5, such that the condition (6) guaranteed for 0.2 due to Cosner al. 2023, equation (13).
16


Reviews:
[{"review_id": "mf9D0Xc6Gw", "reviewer": "Reviewer_GFiT", "summary": "This paper proposes a probabilistic safety certificate design methodology for systems with latent variables, aiming to address the safety verification challenges faced by traditional control methods in scenarios involving partial observability and distribution shifts. The core framework involves:  \n1. **Specialized MDP Formulation**: A tailored Markov Decision Process (MDP) is defined for both offline and online settings, where the value function of this MDP is proven to correspond to the safety probability.  \n2. **Safety Certificate via Value Function**: The safety certificate is derived from the value function, ensuring that adherence to this certificate guarantees long-term safety.  \n3. **Q-Function-Based Computation**: Due to the computational infeasibility of directly calculating the expected value function under online data, it is demonstrated that the safety certificate can alternatively be computed using a specific Q-function. The optimal policy derived from this Q-function inherently satisfies long-term safety constraints.  \n4. **Causal Reinforcement Learning Integration**: The paper further proves that the required Q-function can be efficiently estimated through causal reinforcement learning techniques. By leveraging this Q-function to adjust control actions, the methodology ensures that every executed action maintains safety guarantees.  \n\nThis approach bridges safety-critical control and causal inference, providing a theoretically grounded and computationally tractable solution for systems operating under latent variables and distribution shifts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your efforts and valuable feedback. Please find below our answers to your questions.\n\nQuestion regarding discrete action: \nEquation (42) and its neighbors explain the technique of Shi et al. 2024. N can be used to estimate $Q^\\pi$ in our problem. While this specific method is developed primarily for a discrete action space, the proposed framework is not restricted to any specific choice of confounded MDP technique. Our main theoretical results—Proposition 3.1 (relation between long-term safety and value-function), Theorem 3.2 (safety condition), and Theorem 3.2 (persistent feasibility) do not require discrete action spaces. Thus, other techniques to learn the Q-function in a continuous action space, combined with the above theoretical results, can be used to handle a continuous action space.\n\nRegarding the impact of the latent variable and limited data: \nUnobserved variables and limited data are two different factors that contribute to errors in Q-learning. Without proper adjustment, even with infinitely many data, the learned Q function or quantified risk will not converge to the true values due to the distribution shift arising from latent variables: this is the problem we study in this paper. This issue has not been studied in the context of safety certificates, and the proposed method is expected to have less distribution mismatch (due to the adjustment for latent variables) than existing approaches that treat the observed distribution as the true dynamics. \n\nThis issue of uncertain systems with limited data is another problem that the existing tools in the safety certificate cannot handle well. Many existing tools for safety certificates require knowledge of accurate system dynamics and ground truth distributions. When the system is uncertain and needs to be inferred from limited data, existing methods, ranging from system identification to value estimation to reachability analysis, would all face this problem. In our approach, this problem appears in the form of a potential distribution mismatch in the Q-function. In other stochastic safe control approaches (which typically require a characterization of the probability of forward invariance), this problem appears in the form of errors. Regarding the influence of limited data volume versus the influence of latent variables, we don’t have a quantitative sense of which one is more significant, as this is not within the scope of the paper. We agree with the reviewer that a natural and interesting next step is to account for this. We will add this aspect to the discussion. \n‘Essential References Not Discussed’: Thanks for pointing out this missing reference for us. We obtained the idea of modifying the MDP structure from https://arxiv.org/abs/2403.16391, so we cited this paper as the source. As the paper you pointed out has this idea and is earlier, we will cite it as well. \n\nResponse to ‘Questions For Authors’: The nominal policy is the policy with respect to which the long-term safe probability is evaluated, i.e., the $\\pi$ in the long-term safe probability defined in (5).\n\nQuestion in ‘Supplementary Material’: What contributes to the discrepancy most is the behavioral policy. Consider that the behavioral policy knows that when the latent variable W is 1, choosing action 1 will lead to an unsafe state. It will then avoid choosing 1, but will only avoid that when W is 1. When we sample from this behavioral policy, we never know the consequence of choosing action 1 because the behavioral policy avoids doing so when it is risky, regardless of how many samples we have. We don’t know when choosing action 1 is risky because we cannot observe the latent variable. In fact, if action 1 is completely safe when W is 0, we will have a dataset that the next state is always safe when action 1 is chosen. \n\nQuestions in ‘Claims and Evidence’, ‘Theoretical Claims’, ‘Experimental Design and Analyses’ 1 and 3: These questions are all about Section 3.4, so we answer them altogether. The entire Section 3.4 considers an example algorithm using the Q-function estimation method proposed in https://arxiv.org/abs/2202.10589. The assumptions about the discrete action space, the existence of the mediator and the existence of the unbiased estimator are all from that paper. The proposed method itself does not require these assumptions if the Q-function estimation method does not require them. Regarding the derivation of (42), please check Section 3.6 of the aforementioned paper, as (42) is an equation from that section. \n\nQuestion in ‘Experimental Design and Analyses’, 2: Since we have a Q-function estimation, the first term is given. If the action space is discrete, one can compute the second term using the given policy and its associated distribution of actions. If the action space is continuous, then the expectation is given in the form of an integral, which can be approximated using various methods including Monte-Carlo.", "claims_and_evidence": "Equation 42 is presented in a discrete form. Does this imply that the proposed method is only applicable to discrete action spaces? How can the approach be extended to continuous action spaces?", "methods_and_evaluation_criteria": "The paper relies on two critical assumptions:  \n1. The existence of a consistent estimator $\\ \\hat{P}_M(M_t | \\hat{Y}_t, U_t)$ for the distribution $P(M_t | X_t, U_t) $.  \n2. The existence of a consistent estimator $\\hat{P}_U(U_t | \\hat{Y}_t) $ based on the offline dataset $\\tilde{D} $, which estimates the empirical distribution of the control action $U_t $ given the observable state $\\hat{Y}_t $.  \n\nWhile these estimators are assumed to converge to the true distribution parameters as the sample size approaches infinity, practical limitations arise when the dataset is finite. In such cases, the state transition dynamics in the offline dataset may not align with those in the online environment.  \n\nIn most real-world scenarios, these assumptions are unlikely to hold. For instance, in autonomous driving, it is impossible to collect exhaustive human driving data or fully capture the environmental transition distributions. The key concern is: What happens when these assumptions are violated? The paper does not adequately address the robustness of the proposed method under such conditions, nor does it provide mitigation strategies for scenarios where the dataset is insufficient or non-representative.\"", "theoretical_claims": "1. The derivation of Equation 42 is not sufficiently detailed in the manuscript. Could the authors provide a step-by-step derivation to clarify how this equation is obtained? Additionally, the origin of the optimal action $ u^* $ is unclear. If $ u^* $ is part of the action space $u $, how is it specifically derived or selected?", "experimental_designs_or_analyses": "1. The manuscript introduces the estimators $\\hat{P}_U(U_t | \\hat{Y}_t) $ and $ \\hat{P}_M(M_t | \\hat{Y}_t, U_t) $, but their practical interpretation remains unclear. Could the authors provide a concrete explanation of these terms using the example from Section 4 (Numerical Simulation)?\n\n2. Equation 25 is presented without sufficient detail on its computation. Could the authors elaborate on how this equation is calculated, using the specific context of the numerical example in Section 4? A step-by-step explanation would greatly enhance the reproducibility of the results.\n\n3. The role of the mediator $ M_t $ is not clearly defined in the context of the proposed method. Could the authors clarify what $M_t$represents in the numerical example from Section 4? For instance, is it an observed variable, a latent variable, or a function of other system states? A detailed explanation would help readers better understand its significance in the framework.", "relation_to_broader_scientific_literature": "It could potentially be applied to imitation learning and other systems to learn safe policies.", "essential_references_not_discussed": "In the paper 'Safe Reinforcement Learning by Imagining the Near Future' (https://arxiv.org/abs/2202.07789), Equation 2 also demonstrates that modifying the MDP structure enables the Q-function to be used for safety evaluation.", "other_strengths_and_weaknesses": "I have carefully reviewed the majority of the mathematical proofs in this paper and found no significant issues. The theoretical foundations presented by the authors are rigorous and well-constructed.", "other_comments_or_suggestions": "Minor Correction: In line 181, the sequence $ \\bar{X}_{0:H} $ is described as having the 'online statistics '.this should be corrected to 'offline statistics,' as the context suggests the use of pre-collected data rather than real-time data.", "questions_for_authors": "The paper states: 'The nominal policy simply chooses actions in the action space with identical probability.' Could the authors clarify the role of the nominal policy here?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtrUePJgbS", "reviewer": "Reviewer_toZQ", "summary": "The paper addresses the challenge of ensuring long-term safety in control systems that have latent (unobserved) variables causing partially unidentifiable dynamics and distribution shifts between offline (training) and online (deployment) data​. Traditional safety assurance methods typically assume full knowledge of the system or perfect state observability, which is often unrealistic when latent factors are present​. In response, this work proposes a probabilistic safety certificate framework that operates under partial observability. The key idea is to formulate forward invariance conditions in probability space, rather than in state-space, so that safety can be quantified using observed state statistics even under distribution shifts induced by latent variables.\n\nThe methodology integrates concepts from stochastic safe control and causal reinforcement learning. The authors derive conditions (safety constraints) on control actions that guarantee a minimum probability of remaining in a safe set for the entire horizon (episode) despite latent uncertainties​. They establish a relationship between a risk measure (long-term safe probability) and a suitably defined marginalized value/Q-function via a modified Bellman equation​. By leveraging this relationship, the safety certificate enforces that at every time step there exists at least one feasible action that keeps the system’s risk below a specified tolerance (e.g. ensuring safety probability ≥ 1–ε)​. A real-time controller can then use these action constraints: essentially acting myopically (greedily) but only choosing actions that satisfy the long-term safety certificate. The approach is designed to be computationally efficient for online use​. Importantly, the framework uses causal RL techniques to estimate the necessary Q-functions from offline data despite latent confounders, marking (as claimed) the first integration of causal RL for quantifying long-term risk in safety certificate design​. The paper demonstrates the effectiveness of the proposed safety certificate through numerical simulations of a driving scenario with an unobserved “road slipperiness” factor, showing that the new method maintains safety where prior methods fail​", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your efforts and valuable feedback. Please find below our answers to your questions.\n\nQ1, Q2.\nThe required offline dataset and the estimation procedures for $Q^{\\pi}$ depend on the specific choice of causal RL techniques. For example, the Q-estimator in Shi et al. 2024 is persistent, and the authors demonstrated in simulation that the mean squared error decreases at the scale of Q(exp(1/MSE)). The techniques in Wang et al. 2021 are shown to have $\\sqrt{T}$ regret. \nIn general, when no model information is available, sufficient data is needed to gather information about the transition dynamics of all states and to learn the Q-function. When some physics information is available, the physics information can be used to reduce the requirement on data points around failure scenarios (such as in Hoshino & Nakahira, 2024).\nWe also would like to mention that the proposed framework and its theoretical results---Proposition 3.1 (conversion of long-term safe probability into value-function), Theorem 3.2 (safety condition), and Theorem 3.2 (persistent feasibility)--- hold irrespective of the specific choice of causal RL methods. In the paper, we have demonstrated how to use the RL method of paper 2, but one can replace line 14 in Algorithm 2 (which uses Shi et al. 2024) with other Q-estimators. Data-efficient causal RL methods to estimate $Q^{\\pi}$ are beyond the scope of this paper: Rather, we focus on how to design a persistently feasible safety certificate that can leverage external causal RL methods.\n\nPaper 1: Shi, Chengchun, et al. \"Off-policy confidence interval estimation with confounded markov decision process.\" Journal of the American Statistical Association 119.545 (2024): 273-284. \n\nPaper 2: Wang, Lingxiao, Zhuoran Yang, and Zhaoran Wang. \"Provably efficient causal reinforcement learning with confounded observational data.\" Advances in Neural Information Processing Systems 34 (2021): 21164-21175.\n\nQ3.\nOur future work is to account for the errors in the estimated $Q^{\\pi}$ function. One approach is to compute a confidence interval around the estimated Q-function and adjust the safety threshold downward to guard against potential optimism. This would involve characterizing or bounding the errors arising from distributional mismatch and accounting for such errors in the derivation of the safety condition, which would help ensure that the safety guarantee is maintained even when estimation errors occur. \n\nQ4. \nThe online evaluation reduces to finding a control action that satisfies the safety condition (Eq. (25)). For example, the evaluation can be done by solving an optimization problem in Eq. (43), or projecting a nominal control action $U^n$ on the constraint set $S(X_t,u,t)\\geq 0$. The function $S(X_t,u,t)$ can be computed offline and stored (e.g., in a neural network), and the control action $u$ is the only optimization variable. We expect the proposed method to not require significant computation online.\n\nQ5. \nWhile extension to an infinite time horizon or a non-episodic task is beyond the scope of this paper, the high-level approach of our method is likely to admit an extension to such settings. For such settings, one can first find an auxiliary stochastic process based on observed statistics that can be used to explicitly differentiate online vs. offline statistics (like Eqs. (8) and (11)). One can explore different definitions of safety and adjusted value/Q-functions that still allow for relating long-term safety with adjusted value/Q-functions and for obtaining a safety condition based on such functions, e.g., incorporating a discount factor for future risk.\n\nQ6. \nThe current setting (confounded MDP) is a special case of POMDP where one can use the Markov nature of the observed state to design fast algorithms for a safety certificate. For latent variables that exhibit finite memory or autocorrelation, one can augment the state space with the necessary memory such that the augmented state is Markov. For most general POMDPs without any additional structures (which is by itself a hard problem), belief state is an option. We expect that, for belief state, the high level ideas to construct an auxiliary stochastic process based on observed statistics to explicitly differentiate online vs offline statistics (like equation (8) and (11)), approaches to leverage existing Q-estimator to obtain long-term safety, and constructing safety certificate based on Q-functions, is likely to still hold.", "claims_and_evidence": "The authors make several clear claims: (i) that their safety certificate ensures long-term safety (forward invariance in probability) in systems with latent variables, (ii) that it can be computed from observed (possibly biased) data despite distribution shifts, (iii) that it can consistently find feasible safe actions in real time, and (iv) that combining causal reinforcement learning with safe control is a novel contribution enabling efficient risk quantification. Overall, these claims are well-supported by the content of the paper, though a few warrant scrutiny:\n\nFor Theorem 3.2: The paper provides a rigorous formulation of the safety objective as a probabilistic forward-invariance condition (inequality (6) in the text) and shows that if certain action constraints are satisfied at each step, the probability of remaining in the safe set up to the horizon is at least $1-\\epsilon$​. The main condition is never decreasing the safety probability at each timestep (inequality (15)), I am not sure how realistic this is, however given this assumption the proof follows convincingly. \n\nThe authors assert that their method can utilize offline data, even if the offline and online transition dynamics differ due to latent factors​. They support this by introducing a procedure (Algorithm 1) to transform the offline dataset into a form ($\\tilde D$) that matches the “online” statistical structure needed for their safety analysis​. The claim is plausible given known results in causal inference; indeed, they cite prior works on handling unobserved confounders in offline RL (e.g. using proxy or instrumental variables) to justify that an unbiased Q-function estimate for the true dynamics can be obtained​. While the paper does not deeply detail the statistical assumptions in the main text, it references established causal RL methods (Wang et al. 2021b; Shi et al. 2024) for obtaining those unbiased estimates​. This boosts the credibility of the claim. One potential concern is that the success of this step relies on the offline data being sufficiently rich; the paper assumes the existence of a causal learning solution without extensively discussing failure modes if the data are poor. Nonetheless, given the references and the provided algorithm, the claim appears reasonably supported for scenarios meeting their assumptions (e.g., the latent variables satisfy the stated conditional independence in Assumption 2.1).\n\nThe abstract claims the certificate “can continuously find feasible actions” and can be implemented efficiently online​. The theoretical analysis indeed shows that if the system starts in a safe condition, there will always exist some control input that keeps the safety condition satisfied going forward, this is the usual proof technique used in this scenario. As for real-time efficiency, the authors argue that the certificate boils down to evaluating a learned Q-function and checking a constraint inequality at each step, which is computationally light. They do not report actual computation times, but given the form of the controller (essentially a one-step lookahead with a precomputed function), the claim of real-time implementability is believable. This claim would be stronger if accompanied by complexity analysis or timing results; however, no evidence in the paper contradicts it, so it stands as a reasonable assertion.", "methods_and_evaluation_criteria": "Methods: The methodology is well-suited to the stated problem. The authors formulate the safety assurance problem as maintaining a high probability of safety over an entire horizon (an event of no failures)​, which is appropriate for “long-term” safety considerations. To tackle partial observability and distribution shift, they adopt a confounded MDP framework (a Markov Decision Process with an unobserved variable $W_t$ influencing transitions)​. The core technical approach — deriving safety conditions in probability space — directly addresses the difficulty that latent variables pose for traditional state-space invariance checks​. By using a modified Bellman equation and defining a marginalized value function $V^{\\pi}$ and Q-function $Q^{\\pi}$ that incorporate the probabilities of staying safe, they convert the safety requirement into conditions on these functions​. This is not a novel idea in and of itself, but it is a smart approach because it allows leveraging the rich toolbox of RL (especially off-policy evaluation) to compute safety-related quantities that would otherwise require knowing the latent dynamics.\n\nThe experimental evaluation, while limited in scope, uses appropriate metrics for the safety problem. The primary metrics are the probability of safety at each time step and the long-term safety probability (the probability of no failure up to time $t$. The baseline method chosen for comparison is appropriate: they use a discrete-time control barrier function (DTCBF) method from Cosner et al. 2023 as the benchmark. This baseline represents a state-of-the-art safety filter that does not account for latent-variable-induced distribution shifts (it uses offline estimates naively). Comparing against DTCBF highlights the exact problem the paper is trying to solve. However, I would argue more baseline comparisons  (e.g., a naive RL agent with a reward penalty for failures, or a robust control method that assumes worst-case dynamics) might further strengthen the evaluation. However, given the novelty of the problem setup, I can understand that suitable alternative methods are limited.\n\nThe only real limitation in evaluation is the narrow range of experiments – primarily one scenario is tested. Possibly additional scenarios, inlcuding highly-non linear scenarios should be tested.", "theoretical_claims": "The theoretical claims in the paper are logically sound and align with known principles in safe control and Markov decision processes. The central theoretical result is that if a certain safety certificate condition is satisfied for all times, then the safety objective (the probabilistic invariance condition (6)) is guaranteed​. This claim is very much in line with the concept of forward invariance: if one can ensure a condition holds at each step (and that condition is designed to enforce future safety), then by induction the system remains safe over the horizon.\n\nThe modified Bellman equation and the link between the risk measure and the marginalized Q-function are credible and have been explored in different settings before.", "experimental_designs_or_analyses": "The experimental design uses a simulated autonomous driving scenario to evaluate whether the proposed safety certificate performs as expected under latent-variable-induced distribution shift. The scenario is described in enough detail to understand the setup: the observable state $X_t$ seems to include the vehicle’s position and speed (since they mention $X_t^1$ mod 10 and $X_t^2$ which likely correspond to road segment and velocity)​. The latent variable $W_t$ is the road slipperiness, which affects the vehicle’s braking dynamics but is not directly observed by the controller​. The offline dataset is generated by a human driver (behavior policy) that does react to slipperiness (slows down more on slippery roads), although this information is hidden in the data​. This cleverly creates a confounded offline dataset, exactly matching the paper’s problem assumptions. The online controller, by contrast, does not observe $W_t$ and must rely on the learned safety certificate. Beyond my earlier reservations about the scope of the experimental evaluation, the experiments in the paper are well designed and the analysis appears correct.", "relation_to_broader_scientific_literature": "The paper positions itself at the intersection of safe control (especially forward invariance and barrier functions) and causal reinforcement learning (offline RL with unobserved confounders). The authors do a good job discussing relevant prior work in both areas and highlighting the gap that this paper fills.\n\nOn the safe control side, they cite classic and recent works on Lyapunov/barrier functions and safety filters for both deterministic and stochastic systems​. They acknowledge methods dealing with partial observation via robust or belief-space barrier functions​. Importantly, they point out that these existing methods generally assume either known dynamics or at least no systematic bias between the data used for design and the true deployment scenario​. \n\nOverall, the paper’s positioning in literature is well described. It properly credits a wide array of relevant prior work from both the control theory community and the RL/causal community. The combination of these threads itself highlights the novelty: none of the cited works alone addresses what the paper does at their intersection. The authors make it clear that stochastic safe control and causal RL have “been studied in isolation until now”.", "essential_references_not_discussed": "The authors have cited a wide range of relevant work, and no critical references appear to be missing for the topics of interest.", "other_strengths_and_weaknesses": "Strengths:\n\nOriginality: The integration of causal inference techniques with control-theoretic safety guarantees is highly original.\nMotivation: Ensuring safety in autonomous systems under real-world uncertainties is extremely important.\nWriting: I have no issued with high level writing in the paper it is easy to follow.\n\nWeakness:\n\nPoor scope of the empirical evaluation: As noted, the experiments are limited to one scenario and a relatively simple one at that.\n\nReliance on accurate Q-function estimation: A potential practical weakness is that the method hinges on obtaining an unbiased estimate of the marginalized Q-function under the true dynamics. The paper assumes this can be done using existing causal RL algorithms​, but in practice these algorithms may require a lot of data or careful tuning. If the Q-function estimate has error, the safety guarantee might not strictly hold (unless the method accounts for estimation error margins, which wasn’t clearly discussed). \n\nLack of discussion about limitations: The approach might be brittle if the Q-function is imperfect. The paper does not deeply discuss the effects of function approximation or finite-sample errors on the safety guarantee – presumably a small approximation error could violate the guarantee. This is not dealt with in the current submission and represents a potential weakness if one were to implement this in the real world.\n\nSome additional implicity assumptions not discussed in the paper: here are implicit assumptions: e.g., that the latent variables obey Assumption 2.1 (conditional independence and Markov property)​, that the offline data policy had enough randomness (so that one can infer the effect of actions independent of the confounder), and that the model class for Q-function is flexible enough. If any of these assumptions are violated (for example, if the latent variable has temporal correlations, or if the behavior policy is deterministic given $W$), the approach might need modifications.\n\nTechnical clarity: While the high-level writing is clear, the theoretical sections (Section 3 in particular) likely involve heavy notation and could be challenging to parse for readers not already familiar with causal MDPs.", "other_comments_or_suggestions": "Adding a paragraph (perhaps in the conclusion or end of the introduction) explicitly stating the assumptions (like the latent variable independence and the need for sufficient exploration in data) and the current limitations (e.g., requirement for episodic tasks, needing an offline dataset) would increase transparency.", "questions_for_authors": "1. How rich does the offline dataset $D$ need to be to successfully implement the proposed safety certificate? For instance, does $D$ need to cover near-failure scenarios or a wide range of latent variable conditions to learn an accurate Q-function? If the offline data is limited or collected from a narrow policy, how would that impact the safety guarantee?\n\n2 .Could you elaborate on how exactly the unbiased estimate of $Q^{\\pi}$ is obtained using causal RL?\n\n3. What happens if the learned $Q^{\\pi}$ function has some error? In practice, any data-driven estimate will have uncertainty. Does the framework allow incorporating a confidence bound (e.g., using a slightly smaller threshold to account for estimation error)? If the estimated Q is optimistic by mistake, could the safety guarantee be violated?\n\n4. How computationally intensive is the safety certificate checking and Q-function computation in practice?\n\n5. Your safety objective is defined for a finite horizon $H$. How would the approach extend to an infinite-horizon setting or an ongoing (non-episodic) task?\n\n6. The analysis assumes the latent variable has no memory (i.e., is i.i.d. conditioned on the current state). How would the safety certificate approach handle a latent variable that is stateful or autocorrelated (for example, a hidden system mode that persists over time)? Would the method require an extension to POMDP belief-state tracking, or is there a way to incorporate such correlations in the current framework?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WyAF8y63XX", "reviewer": "Reviewer_aHyV", "summary": "This paper proposes a safety certificate against latent/unobservable latent variables that cause a distribution shift between offline learning data and online execution. The paper aims to efficiently ensure long-term safety for stochastic systems. The safety certificate in probabilistic space can construct a forward invariable condition that can be implemented in real-time control. The authors validate the proposed approach on a naive 2-dimensional numerical example, again the control barrier function approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your efforts and valuable feedback. Please find below our response to your comments. \n\nReferences Not Discussed: \n\nWang et al. 2023\n\nWe will cite this paper. This paper studies the problem of training a safe policy, to be continuously used, when the data with complete state is used for training, and the control policy also has access to the complete state. Our problem differs from this paper as follows.\nFirst, since the latent variable cannot be observed, the training cannot use data with complete state, and the policy cannot take the full state as input. Estimation of system dynamics or value functions using observable data only will be biased due to the distribution shifts between online vs. offline statistics arising from latent variables, even if the available data is infinite.\nSecond, our problem characterizes the set of safe control actions, which is useful for scenarios such as when the system already has a nominal controller. Since any action in the set of safe control actions is allowed, our problem requires additional steps to derive persistent feasibility (section 3.3), in order to achieve long-term safety. On the other hand, when an optimal policy is continuously used, the feasibility of actions can be directly obtained from the optimization problem used to compute the optimal policy.\n\nPrajna et al. 2004, and response to Q4\n\nWe have cited the journal version of this paper (Prajna et al. 2007). Both barrier certificates and proposed methods use conditions, reminiscent of super-martingale, to control the evaluation of certain values that do not decrease over time. Barrier certificates can directly formulate this condition on the Barrier function since the complete system dynamics is known and all states are observed. On the other hand, we need to define two auxiliary stochastic processes and value-like functions for the observed statistics, find the relation between safety probability and value-like functions in order to use causal RL methods to adjust the impact of latent variables by borrowing tools from causal techniques. This is the first paper that derives how to convert the problem of assuring long-term safety probability, with auxiliary stochastic processes and new definition of value-like functions, into the form that can be handled by causal RL techniques. \n\nQ1, Q2. \nFrom Assumption 2.1, the distribution of latent variable $W_t$ can depend on the current observed variable $X_t$, but conditioned on this dependence, it doesn’t depend on the history of the variables.\nTake an example when an object may jump from occluded areas: $W_t$ can denote the presence of an occluded object. The likelihood depends on the ego-vehicle’s location (included in the observed state), but given this location, it doesn’t depend on the past trajectories of the ego-vehicle or other occluded objects along the past.\nThis formulation of behavioral policy allows the offline data to be based on a behavioral policy that is different from the online policy. The online policy can only use the observed state, while the behavioral policy is also allowed to depend on different sets of information/measurement than the online policy. \nThe models are studied as confounded MDP, and more examples can be found in papers such as Shi et al. 2024 and Wang et al. 2021.\n \nQ3. \nThe design cannot directly use the underlying decision process because we cannot evaluate the standard value/Q-functions which are functions of all states (latent variables and observed variables), but naively defining a value-like function using observed variables results in a wrong safety estimate due to the distribution shifts between offline vs. online statistics. Instead, we construct two auxiliary stochastic processes in Eq. 8 (for online statistics) and Eq. 11 (for offline statistics) from the underlying decision process, allowing us to explicitly handle the distribution shifts between offline vs. online statistics.\n\nQ5. \nIn order for the safety probability to match the Q-function, the reward needs to be defined as Eq. 10. Eq. 10 leads to $r(\\hat{Y}_t) = 0$\n\nQ6. \nWe apologize for the confusion. The method that learns Q-function using the mediator is proposed in the existing work (Shi et al. 2024), which is only an example to demonstrate how the proposed method can be integrated with existing methods that learn Q-function, and is not considered as our contribution. In addition, we would like to clarify that Assumption 3.5, together with the introduction of the mediator variable, is subject to this example only and not to the proposed method, and the proposed method works with any method that learns unbiased Q-function. Regarding the details of how the mediator works and how the Q-function is learned, due to space limitations here and in the paper, please refer to Section 3.6 of Shi et al. 2024 for more detailed explanations and derivations.", "claims_and_evidence": "Yes, the reviewer acknowledges that this paper and its authors make clear and convincing claims.", "methods_and_evaluation_criteria": "The evaluation criteria make sense. It includes the unobservable latent variable that causes distribution shifts, and the evaluations are with the long-term safety probability.", "theoretical_claims": "Yes, but not all the theoretical claims. The reviewer has checked Section 3.2, safety condition, Section 3.3 Evaluation of Safety Condition and Persistent Feasibility Guarantee, and didn't find any issue.", "experimental_designs_or_analyses": "The experiment is sound. However, the major concern is the experiment design is only for a 2-dimensional numerical system, which is not convincing.", "relation_to_broader_scientific_literature": "This paper is related to the safe control, safe reinforcement learning community. The safety certificate with the unobservable latent dynamics is interesting and important to the community.", "essential_references_not_discussed": "The safety certificate formulated in this paper (key contribution) looks very similar to the one in \nEnforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments, ICML 2023, and\nStochastic Safety Verification Using Barrier Certificates, CDC 2004\nwhere they used the super-martingale property to formulate a barrier certificate for a probabilistic safety guarantee. \n\nThe authors may have to discuss how their approach compares to the existing references.", "other_strengths_and_weaknesses": "As mentioned above, strengths include studying an important problem for the community (well-motivated), most of the techniques and theoretical contributions are sound (from the part I can understand). \n\nWeaknesses include writing clarity, some major missing discussion on the existing references, and weak experiments.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How strong is assumption 2.1? and what does this assumption mean? Is this assumption common in real-world applications? \n2. In eq3, why the behavioral policy can depend on W_t, while the online policy cannot? The reviewer cannot figure out the fundamental difference between them. \n3. Please elaborate more on Eq8, what's the intuition of its design and what's the purpose? \n4. Eq15 looks like the super-martingale property used in barrier certificates in stochastic systems, as listed in the above two references. \n5. Wht r(\\hat{Y}_t) = 0 (line 286)? \n6. Can you make the section after assumption 3.5 clearer (how the mediator works and how the Q function is obtained/learned)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a probabilistic safety certificate design methodology for systems with latent variables. They have a specialized MDP formulation and the safety certificate is derived from the value function. They use Q-function computation and show that it can be efficiently estimated through causal reinforcement learning techniques.

The reviewers found the contributions of this paper interesting. I recommend the authors to address the the reviewers' remaining concerns in the final version.
Decision: Accept (poster)