Title: Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

robustness

causality

spurious correlation
Abstract: Robustness has been extensively studied in reinforcement learning (RL) to handle various forms of uncertainty such as random perturbations, rare events, and malicious attacks. In this work, we consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. These spurious correlations are ubiquitous in real-world tasks, for instance, a self-driving car usually observes heavy traffic in the daytime and light traffic at night due to unobservable human activity. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Although motivated, enabling robustness against spurious correlation poses significant challenges since the uncertainty set, shaped by the unobserved confounder and causal structure, is difficult to characterize and identify. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge. To solve this issue, we propose Robust State-Confounded Markov Decision Processes (RSC-MDPs) and theoretically demonstrate its superiority in avoiding learning spurious correlations compared with other robust RL counterparts. We also design an empirical algorithm to learn the robust optimal policy for RSC-MDPs, which outperforms all baselines in eight realistic self-driving and manipulation tasks.

Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation

Abstract

Robustness has been extensively studied in reinforcement learning (RL) to handle various forms of uncertainty such as random perturbations, rare events, and malicious attacks. In this work, we consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. These spurious correlations are ubiquitous in real-world tasks, for instance, a self-driving car usually observes heavy traffic in the daytime and light traffic at night due to unobservable human activity. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Although motivated, enabling robustness against spurious correlation poses significant challenges since the uncertainty set, shaped by the unobserved confounder and causal structure, is difficult to characterize and identify. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge. To solve this issue, we propose Robust State-Confounded Markov Decision Processes (RSC-MDPs) and theoretically demonstrate its superiority in avoiding learning spurious correlations compared with other robust RL counterparts. We also design an empirical algorithm to learn the robust optimal policy for RSC-MDPs, which outperforms all baselines in eight realistic self-driving and manipulation tasks. Please refer to the website for more details.

1 Introduction

Reinforcement learning (RL), aiming to learn a policy to maximize cumulative reward through interactions, has been successfully applied to a wide range of tasks such as language generation [1], game playing [2], autonomous driving [3], etc. While standard RL has achieved remarkable success in simulated environments, a growing trend in RL is to address another critical concern robustness -with the hope that the learned policy still performs well when the deployed (test) environment deviates from the nominal one used for training [4]. Robustness is highly desirable since the performance of the learned policy could significantly deteriorate due to the uncertainty and variations of the test environment induced by random perturbation, rare events, or even malicious attacks [5, 6].

Despite various types of uncertainty that have been investigated in RL, this work focuses on the uncertainty of the environment with semantic meanings resulting from some unobserved underlying variables. Such environment uncertainty, denoted as structured uncertainty , is motivated by innumerable real-world applications but still receives little attention in sequential decision-making tasks [7]. To specify the phenomenon of structured uncertainty, let us consider a concrete example (illustrated in Figure 1) in a driving scenario, where a shift between training and test environments caused by an unobserved confounder can potentially lead to a severe safety issue. Specifically, the observations brightness and traffic density do not have cause and effect on each other but are

controlled by a confounder (i.e. sun and human activity ) that is usually unobserved 2 to the agent. During training, the agent could memorize the spurious state correlation between brightness and traffic density , i.e., the traffic is heavy during the daytime but light at night. However, such correlation could be problematic during testing when the value of the confounder deviates from the training one, e.g., the traffic becomes heavy at night due to special events ( human activity changes), as shown at the bottom of Figure 1. Consequently, the policy dominated by the spurious correlation in training fails on out-of-distribution samples (observations of heavy traffic at night) in the test scenarios.

Train

Test (fail)

HeavyTraffic

The failure of the driving example in Figure 1 is attributed to the widespread and harmful spurious correlation, namely, the learned policy is not robust to the structured uncertainty of the test environment caused by the unobserved confounder. However, ensuring robustness to structured uncertainty is challenging since the targeted uncertain region the structured uncertainty set of the environment - is carved by the unknown causal effect of the unobserved confounder, and thus hard to characterize. In contrast, prior works concerning robustness in RL [8] usually consider a homogeneous and structure-agnostic uncertainty set around the state [9, 6, 10], action [11, 12], or the training environment [13-15] measured by some heuristic functions [9, 15, 8] to account for unstructured random noise or small perturbations. Consequently, these prior works could not cope with the structured uncertainty since their uncertainty set is different from and cannot tightly cover the desired structured uncertainty set, which could be heterogeneous and allow for potentially large deviations between the training and test environments. Unobserved Confounder Sun Human Activity BrightTraffic ness Density Spurious Correlation

In this work, to address the structured uncertainty, we first propose a general RL formulation called State-confounded Markov decision processes (SC-MDPs), which model the possible causal effect of the unobserved confounder in an RL task from a causal perspective. SC-MDPs better explain the reason for semantic shifts in the state space than traditional MDPs. Then, we formulate the problem of seeking robustness to structured uncertainty as solving Robust SC-MDPs (RSC-MDPs), which optimizes the worst performance when the distribution of the unobserved confounder lies in some uncertainty set. The key contributions of this work are summarized as follows.

We propose a new type of robustness with respect to structured uncertainty to address spurious correlation in RL and provide a formal mathematical formulation called RSC-MDPs, which are well-motivated by ubiquitous real-world applications.

We theoretically justify the advantage of the proposed RSC-MDP framework against structured uncertainty over the prior formulation in robust RL without semantic information.

We implement an empirical algorithm to find the optimal policy of RSC-MDPs and show that it outperforms the baselines on eight real-world tasks in manipulation and self-driving.

2 Preliminary and Limitations of Robust RL

In this section, we first introduce the preliminary formulation of standard RL and then discuss a natural type of robustness that is widely considered in the RL literature and most related to this work - robust RL.

Standard Markov decision processes (MDPs). An episodic finite-horizon standard MDP is represented by M = { S , A , T, r, P } , where S ⊆ R n and A ⊆ R d A are the state and action spaces, respectively, with n/d A being the dimension of state/action. Here, T is the length of the horizon; P = { P t } 1 ≤ t ≤ T , where P t : S × A → ∆( S ) denotes the probability transition kernel at time step t , for all 1 ≤ t ≤ T ; and r = { r t } 1 ≤ t ≤ T denotes the reward function, where r t : S × A → [0 , 1] represents the deterministic immediate reward function. A policy (action selection rule) is denoted by π = { π t } 1 ≤ t ≤ T , namely, the policy at time step t is π t : S → ∆( A ) based on the current state s t as π t ( · | s t ) . To represent the long-term cumulative

reward, the value function V π,P t : S → R and Q-value function Q π,P t : S × A → R associated with policy π at step t are defined as V π,P t ( s ) = E π,P [ ∑ T k = t r k ( s k , a k ) | s k = s ] and Q π,P t ( s, a ) = E π,P [ ∑ T k = t r k ( s k , a k ) | s t = s, a t = a ] , where the expectation is taken over the sample trajectory { ( s t , a t ) } 1 ≤ t ≤ T generated following a t ∼ π t ( · | s t ) and s t +1 ∼ P t ( · | s t , a t ) .

Robust Markov decision processes (RMDPs). As a robust variant of standard MDPs motivated by distributionally robust optimization, RMDP is a natural formulation to promote robustness to the uncertainty of the transition probability kernel [13, 15], represented as M rob = { S , A , T, r, U σ ( P 0 ) } . Here, we reuse the definitions of S , A , T, r in standard MDPs, and denote U σ ( P 0 ) as an uncertainty set of probability transition kernels centered around a nominal transition kernel P 0 = { P 0 t } 1 ≤ t ≤ T measured by some 'distance' function ρ with radius σ . In particular, the uncertainty set obeying the ( s, a ) -rectangularity [16] can be defined over all ( s, a ) state-action pairs at each time step t as

where ⊗ denotes the Cartesian product. Here, P t,s,a := P t ( · | s, a ) ∈ ∆( S ) and P 0 t,s,a := P 0 t ( · | s, a ) ∈ ∆( S ) denote the transition kernel P t or P 0 t at each state-action pair ( s, a ) respectively. Consequently, the next state s t +1 follows s t +1 ∼ P t ( · | s t , a t ) for any P t ∈ U σ ( P 0 t,s t ,a t ) , namely, s t +1 can be generated from any transition kernel belonging to the uncertainty set U σ ( P 0 t,s t ,a t ) rather than a fixed one in standard MDPs. As a result, for any policy π , the corresponding robust value function and robust Q function are defined as

which characterize the cumulative reward in the worst case when the transition kernel is within the uncertainty set U σ ( P 0 ) . Using samples generated from the nominal transition kernel P 0 , the goal of RMDPs is to find an optimal robust policy that maximizes V π,σ 1 when t = 1 , i.e., perform optimally in the worst case when the transition kernel of the test environment lies in a prescribed uncertainty set U σ ( P 0 ) .

Lack of semantic information in RMDPs. In spite of the rich literature on robustness in RL, prior works usually hedge against the uncertainty induced by unstructured random noise or small perturbations, specified as a small and homogeneous uncertainty set around the nominal one. For instance, in RMDPs, people usually prescribe the uncertainty set of the transition kernel using a heuristic and simple function ρ with a relatively small σ . However, the unknown uncertainty in the real world could have a complicated and semantic structure that cannot be well-covered by a homogeneous ball regardless of the choice of the uncertainty radius σ , leading to either over conservative policy (when σ is large) or insufficient robustness (when σ is small). Altogether, we obtain the natural motivation of this work: How to formulate such structured uncertainty and ensure robustness against it?

3 Robust RL against Structured Uncertainty from a Causal Perspective

To describe structured uncertainty, we choose to study MDPs from a causal perspective with a basic concept called the structural causal model (SCM). Armed with the concept, we formulate State-confounded MDPs - a broader set of MDPs in the face of the unobserved confounder in the state space. Next, we provide the main formulation considered in this work - robust state-confounded MDPs, which promote robustness to structured uncertainty.

Structural causal model. We denote a structural causal model (SCM) [17] by a tuple { X,Y,F,P x } , where X is the set of exogenous (unobserved) variables, Y is the set of endogenous (observed) variables, and P x is the distribution of all the exogenous variables. Here, F is the set of structural functions capturing the causal relations between X and Y such that for each variable y i ∈ Y , f i ∈ F is defined as y i ← f i ( PA ( y i ) , x i ) , where x i ⊆ X and PA ( y i ) ⊆ Y \ y i denotes the parents of the node y i . We say that a pair of variables y i and y j are confounded by a variable C (confounder) if they are both caused by C , i.e., C ∈ PA ( y i ) and C ∈ PA ( y j ) . When two variables y i and y j do not have direct causality, they are still correlated if they are confounded, in which case this correlation is called spurious correlation.

(a)SC-MDP(Ours)

(b)StandardMDP

(C)POMDP/SA-MDP

(d) MDPUC

(e)Confounded MDP

confounding

Ct

Ot

Ct

Ct

spuriouscorrelation

backdoorpath

St

1

St

n

St'

1

S

n2!

t'

St

St'

St

St'

St

St'

St

St'

at

rt

at'

at

rt

at

rt

at

rt

at

rt

3.1 State-confounded MDPs (SC-MDPs)

We now present state-confounded MDPs (SC-MDPs), whose probabilistic graph is illustrated in Figure 2(a) with a comparison to standard MDPs in Figure 2(b). Besides the components in standard MDPs M = { S , A , T, r } , we introduce a set of unobserved confounder C s = { c t } 1 ≤ t ≤ T , where c t ∈ C denotes the confounder that is generated from some unknown but fixed distribution P c t at time step t , i.e., c t ∼ P c t ∈ ∆( C ) .

To characterize the causal effect of the confounder C s on the state dynamic, we resort to an SCM, where C s is the set of exogenous (unobserved) confounder and endogenous variables include all dimensions of states { s i t } 1 ≤ i ≤ n, 1 ≤ t ≤ T , and actions { a t } 1 ≤ t ≤ T . Specifically, the structural function F is considered as {P i t } 1 ≤ i ≤ n, 1 ≤ t ≤ T - the transition from the current state s t , action a t and the confounder c t to each dimension of the next state s i t +1 for all time steps, i.e., s i t +1 ∼ P i t ( · | s t , a t , c t ) . Notably, the specified SCM does not confound the reward, i.e., r t ( s t , a t ) does not depend on the confounder c t .

Armed with the above SCM, denoting P c := { P c t } , we can introduce state-confounded MDPs (SC-MDPs) represented by M sc = { S , A , T, r, C , {P i t } , P c } (Figure 2(a)). A policy is denoted as π = { π t } , where each π t results in an intervention (possibly stochastic) that sets a t ∼ π t ( · | s t ) at time step t regardless of the value of the confounder.

State-confounded value function and optimal policy. Given s t , the causal effect of a t on the next state s t +1 plays an important role in characterizing value function/Q-function. To ensure the identifiability of the causal effect, the confounder c t are assumed to obey the backdoor criterion [17, 18], leading to the following state-confounded value function (SC-value function) and stateconfounded Q-function (SC-Q function) [19]:

Remark 1 . Note that the proposed SC-MDPs serve as a general formulation for a broad family of RL problems that include standard MDPs as a special case. Specifically, any standard MDP M = { S , A , P, T, r } can be equivalently represented by at least one SC-MDP M sc = { S , A , T, r, C , {P i t } , P c } as long as E c t ∼ P c t [ P i t ( · | s t , a t , c t ) ] = [ P ( · | s t , a t ) ] i for all 1 ≤ i ≤ n, 1 ≤ t ≤ T .

3.2 Robust state-confounded MDPs (RSC-MDPs)

In this work, we consider robust state-confounded MDPs (RSC-MDPs) - a variant of SC-MDPs promoting the robustness to the uncertainty of the unobserved confounder distribution P c , denoted by M sc -rob = { S , A , T, r, C , {P i t } , U σ ( P c ) } . Here, the perturbed distribution of the unobserved confounder is assumed in an uncertainty set U σ ( P c ) centered around the nominal distribution P c

with radius σ measured by some 'distance' function ρ : ∆( C ) × ∆( C ) → R + , i.e.,

Consequently, the corresponding robust SC-value function and robust SC-Q function are defined as

representing the worst-case cumulative rewards when the confounder distribution lies in the uncertainty set U σ ( P c ) .

Then a natural question is: does there exist an optimal policy that maximizes the robust SC-value function ˜ V π,σ t for any RSC-MDP so that we can target to learn? To answer this, we introduce the following theorem that ensures the existence of the optimal policy for all RSC-MDPs. The proof can be found in Appendix C.1.

Theorem 1 (Existence of an optimal policy) . Let Π be the set of all non-stationary and stochastic policies. Consider any RSC-MDP, there exists at least one optimal policy π sc ,⋆ = { π sc ,⋆ t } 1 ≤ t ≤ T such that for all ( s, a ) ∈ S × A and 1 ≤ t ≤ T , one has

In addition, RSC-MDPs also possess benign properties similar to RMDPs such that for any policy π and the robust optimal policy π sc ,⋆ , the corresponding robust SC Bellman consistency equation and robust SC Bellman optimality equation are also satisfied (specified in Appendix C.3.3).

Goal. Based on all the definitions and analysis above, this work aims to find an optimal policy for RSC-MDPs that maximizes the robust SC-value function in (5), yielding optimal performance in the worst case when the unobserved confounder distribution falls into an uncertainty set U σ ( P c ) .

3.3 Advantages of RSC-MDPs over traditional RMDPs

The most relevant robust RL formulation to ours is RMDPs, which has been introduced in Section 2. Here, we provide a thorough comparison between RMDPs and our RSC-MDPs with theoretical justifications, and leave the comparisons and connections to other related formulations in Figure 2 and Appendix B.1 due to space limits.

To begin, at each time step t , RMDPs explicitly introduce uncertainty to the transition probability kernels, while our RSC-MDPs add uncertainty to the transition kernels in a latent (and hence more structured) manner via perturbing the unobserved confounder that partly determines the transition kernels. As an example, imagining the true uncertainty set encountered in the real world is illustrated as the blue region in Figure 3, which could have a complicated structure. Since the uncertainty set in RMDPs is homogeneous (illustrated by the green circles), one often faces the dilemma of being either too conservative (when σ is large) or too reckless (when σ is small). In contrast, the proposed

(a) RMDPs

(b)RSC-MDPs

ValidSpace

ValidSpace

small  large

small

Nominal po

Nominal po

Space △(S)

Space △(S)

RSC-MDPs - shown in Figure 3(b) - take advantage of the structured uncertainty set (illustrated by the orange region) enabled by the underlying SCM, which can potentially lead to much better estimation of the true uncertainty set. Specifically, the varying unobserved confounder induces diverse perturbation to different portions of the state through the structural causal function, enabling heterogeneous and structural uncertainty sets over the state space.

Theoretical guarantees of RSC-MDPs: advantages of structured uncertainty. To theoretically understand the advantages of the proposed robust formulation of RSC-MDPs with comparison to prior works, especially RMDPs, the following theorem verifies that RSC-MDPs enable additional robustness against semantic attack besides small model perturbation or noise considered in RMDPs. The proof is postponed to Appendix C.2.

In words, Theorem 2 reveals a fact about the proposed RSC-MDPs: RSC-MDPs could succeed in intense semantic attacks while RMDPs fail . As shown by (6), when fierce semantic shifts appear between the training and test scenarios - perturbing the unobserved confounder in a large uncertainty set U σ 2 ( P c ) , solving RSC-MDPs with π ⋆,σ 2 RSC succeeds in testing while π ⋆,σ 1 RMDP trained by solving RMDPs can fail catastrophically. The proof is achieved by constructing hard instances of RSC-MDPs that RMDPs could not cope with due to inherent limitations. Moreover, this advantage of RSC-MDPs is consistent with and verified by the empirical performance evaluation in Section 5.3 R1 .

4 An Empirical Algorithm to Solve RSC-MDPs: RSC-SAC

When addressing distributionally robust problems in RMDPs, the worst case is typically defined within a prescribed uncertainty set in a clear and implementation-friendly manner, allowing for iterative or analytical solutions. However, solving RSC-MDPs could be challenging as the structured uncertainty set is induced by the causal effect of perturbing the confounder. The precise characterization of this structured uncertainty set is difficult since neither the unobserved confounder nor the true causal graph of the observable variables is accessible, both of which are necessary for intervention or counterfactual reasoning. Therefore, we choose to approximate the causal effect of perturbing the confounder by learning from the data collected during training.

In this section, we propose an intuitive yet effective empirical approach named RSC-SAC for solving RSC-MDPs, which is outlined in Algorithm 1. We first estimate the effect of perturbing the distribution P c of the confounder to generate new states (Section 4.1). Then, we learn the structural causal model P i t to predict rewards and the next states given the perturbed states (Section 4.2). By combining these two components, we construct a data generator capable of simulating novel transitions ( s t , a t , r t , s t +1 ) from the structured uncertainty set. To learn the optimal policy, we construct the data buffer with a mixture of the original data and the generated data and then use the Soft Actor-Critic (SAC) algorithm [20] to optimize the policy.

4.1 Distribution of confounder

As we have no prior knowledge about the confounders, we choose to approximate the effect of perturbing them without explicitly estimating the distribution P c . We first randomly select a dimension i from the state s t to apply perturbation and then assign the dimension i of s t with a heuristic rule. We select the value from another sample s k that has the most different value from s t in dimension i and the most similar value to s t in the remaining dimensions. Formally, this process solves the following optimization problem to select sample k from a batch of K samples:

where s i t and s ¬ i t means dimension i of s t and other dimensions of s t except for i , respectively. Intuitively, permuting the dimension of two samples breaks the spurious correlation and remains the

Low

Close

Brightness(distraction)

Day

Ego

Night

Ego

High

Far

most semantic meaning of the state space. However, this permutation sometimes also breaks the true cause and effect between dimensions, leading to a performance drop. The trade-off between robustness and performance [21] is a long-standing dilemma in the robust optimization framework, which we will leave to future work.

4.2 Learning of structural causal model

With the perturbed state s t , we then learn an SCM to predict the next state and reward considering the effect of the action on the previous state. This model contains a causal graph to achieve better generalization to unseen state-action pairs. Specifically, we simultaneously learn the model parameter and discover the underlying causal graph in a fully differentiable way with (ˆ s t +1 , ˆ r t ) ∼ P θ ( s t , a t , G ϕ ) , where θ is the parameter of the neural network of the dynamic model and ϕ ∈ R ( n + d A ) × ( n +1) is the parameter to represent causal graph G between { s t , a t } and { s t +1 , r t } . This graph is represented by a binary adjacency matrix G , where 1 / 0 means the existence/absence of an edge. P θ has an encoder-decoder structure with matrix G as an intermediate linear transformation. The encoder takes state and action in and outputs features f e ∈ R ( n + d A ) × d f for each dimension, where d f is the dimension of the feature. Then, the causal graph is multiplied to generate the feature for the decoder f d = f T e G ∈ R d f × ( n +1) . The decoder takes in f d and outputs the next state and reward. The detailed architecture of this causal transition model can be found in Appendix D.1.

The objective for training this model consists of two parts, one is the supervision signal from collected data ∥ s t +1 -ˆ s t +1 ∥ 2 2 + ∥ r t -ˆ r t ∥ 2 2 , and the other is a penalty term λ ∥ G ∥ p with weight λ to encourage the sparsity of the matrix G . The penalty is important to break the spurious correlation between dimensions of state since it forces the model to eliminate unnecessary inputs for prediction.

5 Experiments and Evaluation

In this section, we first provide a benchmark consisting of eight environments with spurious correlations, which may be of independent interest to robust RL. Then we evaluate the proposed algorithm RSC-SAC with comparisons to prior robust algorithms in RL.

5.1 Tasks with spurious correlation

To the best of our knowledge, no existing benchmark addresses the issues of spurious correlation in the state space of RL. To bridge the gap, we design a benchmark consisting of eight novel tasks in self-driving and manipulation domains using the Carla [22] and Robosuite [23] platforms (shown in Figure 4). Tasks are designed to include spurious correlations in terms of human common sense, which is ubiquitous in decision-making applications and could cause safety issues. We categorize the tasks into distraction correlation and composition correlation according to the type of spurious correlation. We specify these two types of correlation below and leave the full descriptions of the tasks in Appendix D.2.

Distraction correlation is between task-relevant and task-irrelevant portions of the state. The task-irrelevant part could distract the policy model from learning important features and lead to a performance drop. A typical method to avoid distraction is background augmentation [24, 25]. We design four tasks with this category of correlation, i.e., Lift , Wipe , Brightness , and CarType .

Composition correlation is between two task-relevant portions of the state. This correlation usually exists in compositional generalization, where states are re-composed to form novel tasks during testing. Typical examples are multi-task RL [26, 27] and hierarchical RL [28, 29]. We design four tasks with this category of correlation, i.e., Stack , Door , Behavior , and Crossing .

Shifted Environments

Nominal Environments

CarType

Crossing

Stack

Lift

5.2 Baselines

Robustness in RL has been explored in terms of diverse uncertainty set over state, action, or transition kernels. Regarding this, we use a non-robust RL and four representative algorithms of robust RL as baselines, all of which are implemented on top of the SAC [20] algorithm. Non-robust RL (SAC): This serves as a basic baseline without considering any robustness during training; Solving robust MDP: Wegenerate the samples to cover the uncertainty set over the state space by adding perturbation around the nominal states that follows two distribution, i.e., uniform distribution (RMDP-U) and Gaussian distribution (RMDP-G). Solving SA-MDP: We compare ATLA [6], a strong algorithm that generates adversarial states using an optimal adversary within the uncertainty set. Invariant feature learning: We choose DBC [30] that learns invariant features using the bi-simulation metric [31] and [32] (Active) that actively sample uncertain transitions to reduce causal confusion. Counterfactual data augmentation: We select MoCoDA [33], which identifies local causality to switch components and generate counterfactual samples to cover the targeted uncertainty set. We adapt this algorithm using an approximate causal graph rather than the true causal graph.

5.3 Results and Analysis

To comprehensively evaluate the performance of the proposed method RSC-SAC , we conduct experiments to answer the following questions: Q1. Can RSC-SAC eliminate the harmful effect of spurious correlation in learned policy? Q2. Does the robustness of RSC-SAC only come from the sparsity of model? Q3. How does RSC-SAC perform in the nominal environments compared to non-robust algorithms? Q4. Which module is critical in our empirical algorithm? Q5. Is RSC-SAC robust to other types of uncertainty and model perturbation? Q6. How does RSC-SAC balance the tradeoff between performance and robustness? We analyze the results and answer these questions in the following.

R1. RSC-SAC is robust against spurious correlation. The testing results of our proposed method with comparisons to the baselines are presented in Table 1, where the rewards are normalized by the episode reward of SAC in the nominal environment. The results reveal that RSC-SAC significantly outperforms other baselines in shifted test environments, exhibiting comparable performance to that of vanilla SAC on the nominal environment in 5 out of 8 tasks. An interesting and even surprising finding, as shown in Table 1, is that although RMDP-G, RMDP-U, and ATLA are trained desired to be robust against small perturbations, their performance tends to drop more than non-robust SAC in some tasks. This indicates that using the samples generated from the traditional robust algorithms could harm the policy performance when the test environment is outside of the prescribed uncertainty set considered in the robust algorithms.

R2. Sparsity of the model is only one reason for the robustness of RSC-SAC . As existing literature shows [34], sparsity regularization benefits the elimination of spurious correlation and causal confusion. Therefore, we compare our method with a sparse version of SAC (SACSparse): we add an additional penalty α | W | 1 during the optimization, where W is the parameter of the first linear layer of the policy and value networks and α is the weight. The results of both Distraction and Composition are shown in Figure 6. We have two important findings based on the results: (1) The sparsity improves the robustness of SAC in the setting of distraction spurious correlation, which is consistent with the findings in [34]. (2) The sparsity does

Distraction

(nominal)

Distraction

(shifted)

Composition

(nominal)

Composition

(shifted)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Ours

SAC

SAC-

-1

SAC-

-10

SAC-

-100

SAC-

-1000

not help with the composition type of spurious correlation, which indicates that purely using sparsity regularization cannot explain the improvement of our RSC-SAC. In fact, the semantic perturbation in our method plays an important role in augmenting the composition generalization.

R3. RSC-SAC maintains great performance in the nominal environments. Previous literature [21] finds out that there usually exists a trade-off between the performance in the nominal environment and the robustness against uncertainty. To evaluate the performance of RSC-SAC in the nominal environment, we conduct experiments and summarize results in Table 2, which shows that RSC-SAC still performs well in the training environment. Additionally, the training curves are displayed in Figure 5, showing that RSC-SAC achieves similar rewards compared to non-robust SAC although converges slower than it.

R4. Both the distribution of confounder and the structural causal model are critical. To assess the impact of each module in our algorithm, we conduct three additional ablation studies (in Table 3), where we remove the causal graph G ϕ , the transition model P θ , and the distribution of the confounder P c respectively. The results demonstrate

that the learnable causal graph G ϕ is critical for the performance that enhances the prediction of

the next state and reward, thereby facilitating the generation of high-quality next states with current perturbed states. The transition model without G ϕ may still retain numerous spurious correlations, resulting in a performance drop similar to the one without P θ , which does not alter the next state and reward. In the third row of Table 3, the performance drop indicates that the confounder P c also plays a crucial role in preserving semantic meaning and avoiding policy training distractions.

R5. RSC-SAC is also robust to random perturbation. The final investigation aims to assess the generalizability of our method to cope with random perturbation that is widely considered in robust RL (RMDPs). Towards this, we evaluate the proposed algorithm in the test environments added with random noise under the Gaussian distribution with two varying scales in the Lift environment. In Table 4,

Lift-0 indicates the nominal training environment, while Lift-0.01 and Lift-0.1 represent the environments perturbed by the Gaussian noise with standard derivation 0 . 01 and 0 . 1 , respectively. The results indicate that our RSC-SAC achieves comparable robustness compared to RMDP-0.01 in both large and small perturbation settings and outperforms RMDP methods in the nominal training environment.

R6. RSC-SAC keeps good performance and robustness for a wide range of β . As shown in Figure 7, the proposed RSC-SAC performs well in both nominal and shifted settings - keeping good performance in the nominal setting and achieving robustness, for a large range of (20%-70%). When the ratio of perturbed data is very small (1%), RSC-SAC almost achieves the same results as vanilla SAC in nominal settings and there is no robustness in shifted set-

1%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Range of acceptable

Performance & Robustness

CarType (nominal)

CarType (shifted)

Crossing (nominal)

Crossing (shifted)

tings. As it increases (considering more robustness), the performance of RSC-SAC in the nominal setting gradually gets worse, while reversely gets better in the shifted settings (more robust). However, when the ratio is too large (>80%), the performances of RSC-SAC in both settings degrade a lot, since the policy is too conservative so that fails in all environments.

6 Conclusion and Limitation

This work focuses on robust reinforcement learning against spurious correlation in state space, which broadly exists in (sequential) decision-making tasks. We propose robust SC-MDPs as a general framework to break spurious correlations by perturbing the value of unobserved confounders. We not only theoretically show the advantages of the framework compared to existing robust works in RL, but also design an empirical algorithm to solve robust SC-MDPs by approximating the causal effect of the confounder perturbation. The experimental results demonstrate that our algorithm is robust to spurious correlation - outperforms the baselines when the value of the confounder in the test environment derivates from the training one. It is important to note that the empirical algorithm we propose is evaluated only for low-dimensional states.However, the entire framework can be extended in the future to accommodate high-dimensional states by leveraging powerful generative models with disentanglement capabilities [35] and state abstraction techniques [36].

Acknowledgments and Disclosure of Funding

The work of W. Ding is supported by the Qualcomm Innovation Fellowship. The work of L. Shi and Y. Chi is supported in part by the grants ONR N00014-19-1-2404, NSF CCF-2106778, DMS-2134080, and CNS-2148212. L. Shi is also gratefully supported by the Leo Finzi Memorial Fellowship, Wei Shen and Xuehong Zhang Presidential Fellowship, and Liang Ji-Dian Graduate Fellowship at Carnegie Mellon University.

References

OpenAI. Gpt-4 technical report, 2023.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 , 2016.

Wenhao Ding, Haohong Lin, Bo Li, and Ding Zhao. Generalizing goal-conditioned reinforcement learning with variational causal reasoning. arXiv preprint arXiv:2207.09081 , 2022.

A Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmarking reinforcement learning algorithms on real-world robots. In Conference on robot learning , pages 561-591. PMLR, 2018.

Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452 , 2021.

Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems , 32, 2019.

Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction , 4(1):276-315, 2022.

Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and ChoJui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems , 33:21024-21037, 2020.

Songyang Han, Sanbao Su, Sihong He, Shuo Han, Haizhao Yang, and Fei Miao. What is the solution for state adversarial multi-agent reinforcement learning? arXiv preprint arXiv:2212.02705 , 2022.

Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning , pages 6215-6224. PMLR, 2019.

Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, and Soumik Sarkar. Robustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC) , pages 3959-3964. IEEE, 2020.

Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2):257-280, 2005.

Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics. The Annals of Statistics , 50(6):3223-3248, 2022.

Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767 , 2022.

Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Judea Pearl. Causality . Cambridge university press, 2009.

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Systems , 34:21164-21175, 2021.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Mengdi Xu, Peide Huang, Yaru Niu, Visak Kumar, Jielin Qiu, Chao Fang, Kuan-Hui Lee, Xuewei Qi, Henry Lam, Bo Li, et al. Group distributionally robust reinforcement learning with hierarchical latent variables. In International Conference on Artificial Intelligence and Statistics , pages 2677-2703. PMLR, 2023.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning , pages 1-16. PMLR, 2017.

Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293 , 2020.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning , pages 5639-5650. PMLR, 2020.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 , 2021.

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094 , 2022.

Yuchen Lu, Yikang Shen, Siyuan Zhou, Aaron Courville, Joshua B Tenenbaum, and Chuang Gan. Learning task decomposition with ordered memory policy network. arXiv preprint arXiv:2103.10972 , 2021.

Minjong Yoo, Sangwoo Cho, and Honguk Woo. Skills regularized task decomposition for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems , 35:37432-37444, 2022.

Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III. Hierarchical imitation and reinforcement learning. In International conference on machine learning , pages 2917-2926. PMLR, 2018.

Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742 , 2020.

Kim G Larsen and Arne Skou. Bisimulation through probabilistic testing (preliminary report). In Proceedings of the 16th ACM SIGPLAN-SIGACT symposium on Principles of programming languages , pages 344-352, 1989.

Gunshi Gupta, Tim GJ Rudner, Rowan Thomas McAllister, Adrien Gaidon, and Yarin Gal. Can active sampling reduce causal confusion in offline reinforcement learning? In 2nd Conference on Causal Learning and Reasoning , 2023.

Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfactual data augmentation. arXiv preprint arXiv:2210.11287 , 2022.

Jongjin Park, Younggyo Seo, Chang Liu, Li Zhao, Tao Qin, Jinwoo Shin, and Tie-Yan Liu. Object-aware regularization for addressing causal confusion in imitation learning. Advances in Neural Information Processing Systems , 34:3029-3042, 2021.

Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning , pages 2649-2658. PMLR, 2018.

David Abel. A theory of state abstraction for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 9876-9877, 2019.

Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Conceptaware mitigation of spurious correlation. arXiv preprint arXiv:2305.00650 , 2023.

Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.02259 , 2015.

Huan Xu and Shie Mannor. Distributionally robust markov decision processes. Advances in Neural Information Processing Systems , 23, 2010.

Eric M Wolff, Ufuk Topcu, and Richard M Murray. Robust control of uncertain markov decision processes with temporal logic specifications. In 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) , pages 3372-3379. IEEE, 2012.

David L Kaufman and Andrew J Schaefer. Robust modified policy iteration. INFORMS Journal on Computing , 25(3):396-410, 2013.

Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast bellman updates for robust mdps. In International Conference on Machine Learning , pages 1979-1988. PMLR, 2018.

Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint arXiv:1902.08708 , 2019.

Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for l1-robust markov decision processes. Journal of Machine Learning Research , 22(275):1-46, 2021.

Vineet Goyal and Julien Grand-Clement. Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research , 2022.

Esther Derman and Shie Mannor. Distributional robustness and regularization in reinforcement learning. arXiv preprint arXiv:2003.02894 , 2020.

Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust mdps using function approximation. In International conference on machine learning , pages 181-189. PMLR, 2014.

Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning , pages 511-520. PMLR, 2021.

Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distributionally robust reinforcement learning. arXiv preprint arXiv:2303.02783 , 2023.

Jing Dong, Jingwei Li, Baoxiang Wang, and Jingzhao Zhang. Online policy optimization for robust mdp. arXiv preprint arXiv:2209.13841 , 2022.

Kishan Panaganti and Dileep Kalathil. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics , pages 9582-9602. PMLR, 2022.

Zhengqing Zhou, Qinxun Bai, Zhengyuan Zhou, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3331-3339. PMLR, 2021.

Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. A finite sample complexity bound for distributionally robust Q-learning. In International Conference on Artificial Intelligence and Statistics , pages 3370-3398. PMLR, 2023.

Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09659 , 2023.

Zijian Liu, Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, and Zhengyuan Zhou. Distributionally robust q -learning. In International Conference on Machine Learning , pages 13623-13643. PMLR, 2022.

Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. Sample complexity of variancereduced distributionally robust Q-learning. arXiv preprint arXiv:2305.18420 , 2023.

Zhipeng Liang, Xiaoteng Ma, Jose Blanchet, Jiheng Zhang, and Zhengyuan Zhou. Singletrajectory distributionally robust reinforcement learning. arXiv preprint arXiv:2301.11721 , 2023.

Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems , 34, 2021.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, Matthieu Geist, and Yuejie Chi. The curious price of distributional robustness in reinforcement learning with a generative model. arXiv preprint arXiv:2305.16589 , 2023.

Shyam Sundhar Ramesh, Pier Giuseppe Sessa, Yifan Hu, Andreas Krause, and Ilija Bogunovic. Distributionally robust model-based reinforcement learning with large state spaces. arXiv preprint arXiv:2309.02236 , 2023.

Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. Advances in neural information processing systems , 35:32211-32224, 2022.

Xiaoteng Ma, Zhipeng Liang, Jose Blanchet, Mingwen Liu, Li Xia, Jiheng Zhang, Qianchuan Zhao, and Zhengyuan Zhou. Distributionally robust offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2209.06620 , 2022.

Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. On the foundation of distributionally robust reinforcement learning. arXiv preprint arXiv:2311.09018 , 2023.

Yuxin Pan, Yize Chen, and Fangzhen Lin. Adjustable robust reinforcement learning for online 3d bin packing. arXiv preprint arXiv:2310.04323 , 2023.

You Qiaoben, Xinning Zhou, Chengyang Ying, and Jun Zhu. Strategically-timed stateobservation attacks on deep reinforcement learning agents. In ICML 2021 Workshop on Adversarial Machine Learning , 2021.

Ke Sun, Yi Liu, Yingnan Zhao, Hengshuai Yao, Shangling Jui, and Linglong Kong. Exploring the training robustness of distributional reinforcement learning against noisy state observations. arXiv preprint arXiv:2109.08776 , 2021.

Zikang Xiong, Joe Eappen, He Zhu, and Suresh Jagannathan. Defending observation attacks in deep reinforcement learning via detection and denoising. arXiv preprint arXiv:2206.07188 , 2022.

Zhihong Deng, Zuyue Fu, Lingxiao Wang, Zhuoran Yang, Chenjia Bai, Zhaoran Wang, and Jing Jiang. Score: Spurious correlation reduction for offline reinforcement learning. arXiv preprint arXiv:2110.12468 , 2021.

Chenjia Bai, Lingxiao Wang, Lei Han, Animesh Garg, Jianye Hao, Peng Liu, and Zhaoran Wang. Dynamic bottleneck for robust self-supervised exploration. Advances in Neural Information Processing Systems , 34:17007-17020, 2021.

Guy Tennenholtz, Assaf Hallak, Gal Dalal, Shie Mannor, Gal Chechik, and Uri Shalit. On covariate shift of latent confounders in imitation and reinforcement learning. arXiv preprint arXiv:2110.06539 , 2021.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Causaladv: Adversarial robustness through the lens of causality. arXiv preprint arXiv:2106.06196 , 2021.

Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2203.01517 , 2022.

Annie Xie, Shagun Sodhani, Chelsea Finn, Joelle Pineau, and Amy Zhang. Robust policy learning over multiple uncertainty sets. In International Conference on Machine Learning , pages 24414-24429. PMLR, 2022.

Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. Advances in Neural Information Processing Systems , 33:7968-7978, 2020.

Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 , 2020.

Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep Q-learning with convnets and vision transformers under data augmentation. Advances in neural information processing systems , 34:3680-3693, 2021.

Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems , 34:5402-5415, 2021.

Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 13611-13617. IEEE, 2021.

Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hernández-Lobato, Kun Zhang, and Bernhard Schölkopf. Sample-efficient reinforcement learning via counterfactual-based data augmentation. arXiv preprint arXiv:2012.09092 , 2020.

Shubhankar Agarwal and Sandeep P Chinchali. Synthesizing adversarial visual scenarios for model-based robotic control. In Conference on Robot Learning , pages 800-811. PMLR, 2023.

Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems , 33:3976-3990, 2020.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 23-30. IEEE, 2017.

Nataniel Ruiz, Samuel Schulter, and Manmohan Chandraker. Learning to simulate. arXiv preprint arXiv:1810.02513 , 2018.

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning , pages 1162-1176. PMLR, 2020.

Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-tosim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12627-12637, 2019.

Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8973-8979. IEEE, 2019.

Sergey Zakharov, Wadim Kehl, and Slobodan Ilic. Deceptionnet: Network-driven domain randomization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 532-541, 2019.

Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble-based methods for avoiding known dataset biases. arXiv preprint arXiv:1909.03683 , 2019.

Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the difference that makes a difference with counterfactually-augmented data. arXiv preprint arXiv:1909.12434 , 2019.

Meike Nauta, Ricky Walsh, Adam Dubowski, and Christin Seifert. Uncovering and correcting shortcut learning in machine learning models for skin cancer diagnosis. Diagnostics , 12(1):40, 2021.

Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Processing Systems , 33:19339-19352, 2020.

Seonguk Seo, Joon-Young Lee, and Bohyung Han. Unsupervised learning of debiased representations with pseudo-attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16742-16751, 2022.

Gregory Plumb, Marco Tulio Ribeiro, and Ameet Talwalkar. Finding and fixing spurious patterns with explanations. arXiv preprint arXiv:2106.02112 , 2021.

Misgina Tsighe Hagos, Kathleen M Curran, and Brian Mac Namee. Identifying spurious correlations and correcting them with an explanation-based learning. arXiv preprint arXiv:2211.08285 , 2022.

Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations. In International Conference on Machine Learning , pages 66-88. PMLR, 2022.

Andrea Bontempelli, Stefano Teso, Fausto Giunchiglia, and Andrea Passerini. Concept-level debugging of part-prototype networks. arXiv preprint arXiv:2205.15769 , 2022.

Mohammad Taha Bahadori and David E Heckerman. Debiasing concept-based explanations with causal analysis. arXiv preprint arXiv:2007.11500 , 2020.

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , 32, 2019.

Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research , 23(267):1-6, 2022.

Appendix

Table of Contents

A Broader Impact

Incorporating causality into reinforcement learning methods increases the interpretability of artificial intelligence, which helps humans understand the underlying mechanism of algorithms and check the source of failures. However, the learned causal transition model may contain human-readable private information about the environment, which could raise privacy issues. To mitigate this potential negative societal impact, the causal transition model needs to be encrypted and only accessible to algorithms and trustworthy users.

B Other Related Works

In this section, besides the most related formulation, robust RL introduced in Sec 3.3, we also introduce some other related RL problem formulations partially shown in Figure 3. Then, we limit our discussion to mainly two lines of work that are related to ours: (1) promoting robustness in RL; (2) concerning the spurious correlation issues in RL.

B.1 Related RL formulations

Robustness to noisy state: POMDPs and SA-MDPs. State-noisy MDPs refer to the RL problem that the agent can only access and choose the action based on a noisy observation rather than the true state at each step, including two existing types of problems: Partially observable MDPs (POMDPs) and state-adversarial MDPs (SA-MDPs), shown in Figure 3(b). In particular, at each step t , in POMDPs, the observation o t is generated by a fixed probability transition O ( · | s t ) (we refer to the case that o t only depends on the state s t but not action); for state-adversarial MDPs, the observation is an adversary ν ( s t ) against and thus determined by the conducted policy, leading to the worst performance by perturbing the state in a small set around itself. To defend the state perturbation, both POMDPs, and SA-MDPs are indeed robust to the noisy observation, or called agent-observed state, but not the real state that transitions to the environment and next steps. In contrast, our RSC-MDPs

propose the robustness to the real state shift that will directly transition to the next state in the environment, involving additional challenges induced by the appearance of out-of-distribution states.

Robustness to unobserved confounder: MDPUC and confounded MDPs. To address the misleading spurious correlations hidden in components of RL, people formulate RL problems as MDPs with some additional components - unobserved confounders. In particular, the Markov decision process with unobserved confounders (MDPUC) [37] serves as a general framework to concern all types of possible spurious correlations in RL problems - at each step, the state, action, and reward are all possibly influenced by some unobserved confounder, shown in Figure 2(d); confounded MDPs [19] mainly concerns the misleading correlation between the current action and the next state, illustrated in Figure 3(e). The proposed state-confounded MDPs (SC-MDPs) can be seen as a specified type of MDPUC that focuses on breaking the spurious correlation between different parts of the state space itself (different from confounded MDPs which consider the correlation between action and next state), motivated by various real-world applications in self-driving and control tasks. In addition, the proposed formulation is more flexible and can work in both online and offline RL settings.

Contexual MDPs (CDMPs). A contextual MDP (CMDP) [38] is basically a set of standard MDPs sharing the same state and action space but specified by different contexts within a context space. In particular, the transition kernel, reward, and action of a CMDP are all determined by a (possibly unknown) fixed context. The proposed robust state-confounded MDPs (RSC-MDPs) are similar to CMDPs if we cast the unobserved confounder as the context in CMDPs, while different in two aspects: (1) In a CMDP, the context is fixed throughout an episode, while the unobserved confounder in RSC-MDPs can vary as { c t } 1 ≤ t ≤ T ; (2) In the online setting, the goal of CMDP is to beat the optimal policy depending on the context, while RSC-MDPs seek to learn the optimal policy that does not depend on the confounder { c t } 1 ≤ t ≤ T .

B.2 Related literature of robustness in RL

Robust RL (robust MDPs). Concerning the robust issues in RL, a large portion of works focus on robust RL with explicit uncertainty of the transition kernel, which is well-posed and a natural way to consider the uncertainty of the environment [13, 39-48]. However, to define the uncertainty set for the environment, most existing works use task structure-agnostic and heuristic 'distance' such as R-contamination, KL divergence, χ 2 , and total variation [49, 50, 14, 51, 52, 15, 49, 5358, 49, 59, 48, 60-64] to measure the shift between the training and test transition kernel, leading to a homogeneous (almost structure-free) uncertainty set around the state space. In contrast, we consider a more general uncertainty set that enables the robustness to a task-dependent heterogeneous uncertainty set shaped by unobserved confounder and causal structure, in order to break the spurious correlation hidden in different parts of the state space.

Robustness in RL. Despite the remarkable success that standard RL has achieved, current RL algorithms are still limited since the agent is vulnerable if the deployed environment is subject to uncertainty and even structural changes. To address these challenges, a recent line of RL works begins to concern robustness to the uncertainty or changes over different components of MDPs state, action, reward, and transition kernel, where a review [8] can be referred to. Besides robust RL framework concerning the shift of the transition kernel and reward, to promote robustness in RL, there exist various works [11, 12] that consider the robustness to action uncertainty, i.e., the deployed action in the environment is distorted by an adversarial agent smoothly or circumstantially; some works [9, 6, 10, 65-67] investigate the robustness to the state uncertainty including but not limited to the introduced POMDPs and SA-MDPs in Appendix B.1, where the agent chooses the action based on observation - the perturbed state determined by some restricted noise or adversarial attack. The proposed RSC-MDPs can be regarded as addressing the state uncertainty since the shift of the unobserved confounder leads to state perturbation. In contrast, RSC-MDPs consider the out-of-distribution of the real state that will directly influence the subsequent transition in the environment, but not the observation in POMDPs and SA-MDPs that will not directly influence the environment.

B.3 Related literature of spurious correlation in RL

Confounder in RL. These works mainly focus on the confounder between action (treatment) and state (effect), which is a long-standing problem that exists in the causal inference area. However,

we find that the confounder may cause problems from another perspective, where the confounder is built upon different dimensions of the state variable. Some people focus on the confounder between action and state, which is common in offline settings since the dataset is fixed and intervention is not allowed. But in the online setting, actions are controlled by an agent and intervention is available to eliminate spurious correlation. [68] reduces the spurious correlation between action and state in the offline setting. [69] deal with environment-irrelevant white noise; possible shift + causal [70]. The confounder problem is usually easy to solve since agents can interact with the environment to do interventions. However, different from most existing settings, we find that even with the capability of intervention, the confounding between dimensions in states cannot be fully eliminated. Then the learned policy is heavily influenced if these confounders change during testing.

Invariant Feature learning. The problem of spurious correlation has attracted attention in the supervised learning area for a long time and many solutions are proposed to learn invariant features to eliminate spurious correlations. A general framework to remedy the ignorance of spurious correlation in empirical risk minimization (ERM) is invariant risk minimization (IRM) [71]. Other works tackle this problem with group distributional robustness [72], adversarial robustness [73], and contrastive learning [74]. These methods are also adapted to sequential settings. The idea of increasing the robustness of RL agents by training agents on multiple environments has been shown in previous works [75, 30, 30]. However, a shared assumption among these methods is that multiple environments with different values of confounder are accessible, which is not always true in the real world.

Counterfactual Data Augmentation in RL. One way to simulate multiple environments is data augmentation. However, most data augmentation works [24, 76, 25, 77-80] apply image transformation to raw inputs, which requires strong domain knowledge for image manipulation and cannot be applied to other types of inputs. In RL, the dynamic model and reward model follow certain causal structures, which allow counterfactual generation of new transitions based on the collected samples. This line of work, named counterfactual data augmentation, is very close to this work. Deep generative models [81] and adversarial examples [82] are considered for the generation to improve sample efficiency in model-based RL. CoDA [83] and MocoDA [33] leverage the concept of locally factored dynamics to randomly stitch components from different trajectories. However, the assumption of local causality may be limited.

Domain Randomization. If we are allowed to control the data generation process, e.g., the underlying mechanism of the simulator, we can apply the golden rule in causality - Randomized Controlled Trial (RCT). The well-known technique, domain randomization [84], exactly follows the idea of RCT, which randomly perturbs the internal state of the experiment in simulators. Later literature follows this direction and develops variants including randomization guided by downstream tasks in the target domain [85, 86], randomization to match real-world distributions [87, 88], and randomization to minimize data divergence [89]. However, it is usually impossible to randomly manipulate internal states in most situations in the real world. In addition, determining which variables to randomize is even harder given so many factors in complex systems.

Discovering Spurious Correlations Detecting spurious correlations helps models remove features that are harmful to generalization. Usually, domain knowledge is required to find such correlations [9092]. However, when prior knowledge is accessible, techniques such as clustering can also be used to reveal spurious attributes [37, 93, 94]. When human inspection is available, recent works [95-97] also use explainability techniques to find spurious correlations. Another area for discovery is conceptlevel and interactive debugging [98, 99], which leverage concepts or human feedback to perform debugging.

C Theoretical Analyses

C.1 Proof of Theorem 1

In this section, we verify the existence of an optimal policy of the proposed RSC-MDPs, involving additional components - confounder C s and the infimum optimization problems with comparisons to standard MDPs [100].

To begin with, we recall that the goal is to find a policy ˜ π = { ˜ π t } 1 ≤ t ≤ T ∈ Π such that for all ( s, a ) ∈ S × A :

which we called an optimal policy. Towards this, we start from the first claim in (8).

Step 1: Introducing additional notations. Before proceeding, we let { S t , A t , R t , C t } denote the random variables - state, action, reward, and confounder, at time step t for all 1 ≤ t ≤ T . Then invoking the Markov properties, we know that conditioned on current state s t , the future state, action, and reward are all independent from the previous s 1 , a 1 , r 1 , c 1 , · · · , s t -1 , a t -1 , r t -1 , c t -1 . In addition, we represent P t ∈ ∆( C ) as some distribution of confounder at time step t , for all 1 ≤ t ≤ T . For convenience, we introduce the following notation that is defined over time step t ≤ k ≤ T :

which represent some collections of variables from time step t to the end of the episode. In addition, recall that the transition kernel from time step t to t +1 is denoted as s i t +1 ∼ P i t ( · | s t , a t , c t ) for i ∈ 1 , 2 · · · , n . With slight abuse of notation, we denote s t +1 ∼ P t ( · | s t , a t , c t ) and abbreviate E s t +1 ∼P t ( · | s t ,a t ,c t ) [ · ] as E s t +1 [ · ] whenever it is clear.

Step 2: Establishing recursive relationship. Recall that the nominal distribution of the confounder is c t ∈ P c t at time step t . We choose ˜ π = { ˜ π t } which obeys: for all 1 ≤ t ≤ T ,

Armed with these definitions and notations, for any ( t, s ) ∈ { 1 , 2 , · · · , T } × S , one has

where (i) holds by the definitions in (5), (ii) is due to (3) and that ˜ V π,P t ( s ) only depends on P + t by the Markov property, (iii) follows from expressing the term of interest by moving one step ahead and E π t is taken with respect to a t ∼ π t ( · | S t = s ) .

To continue, we observe that the ˜ V ⋆,σ t ( s ) can be further controlled as follows:

where (i) holds by the operator inf P +( t +1) ∈U σ ( P c +( t +1) ) is independent from π t conditioned on a fixed distribution of s t +1 , (ii) arises from the Markov property such that the rewards { r k ( s k , a k ) } t +1 ≤ k ≤ T conditioned on ( S t , A t , R t , C t , S t +1 ) or S t +1 are the same, and the last equality follows from the definition of ˜ π in (10).

Step 3: Completing the proof by applying recursion.

Applying (11) recursively for t +1 , · · · T , we arrive at

Observing from (12) that

which directly verifies the first assertion in (8) ˜ V ˜ π,σ t ( s ) = ˜ V ⋆,σ t ( s ) for all s ∈ S . The second assertion in (8) can be achieved analogously. Until now, we verify that there exists at least a policy ˜ π that obeys (8), which we refer to as an optimal policy since its value is equal to or larger than any other non-stationary and stochastic policies over all states s ∈ S .

C.2 Proof of Theorem 2

We establish the proof by separating it into several key steps.

<latexi sh

1_b

64="3L9/5

+fA2z dF7

IG

8

rWXU

>

B

H

c

V

NS

EJ

q

0m oM

T

Y

R

u

D

Z

j

w

y

v

C

Q

O

K

n

k

P

<latexi sh

1_b

64="G

HmErk

LogvV5J

S

C/I

n0

M

>A

B9

c

D

F

3j

ZX

W

Y

q

U

+

2K

z

T

y

8ON

u

7

d

P

R

p

w

Q

f

1

[0,0]

[0,

1]

1

1

r = 1

r = 1

(a) Standard MDP kernel

[1,0]

[1,

1]

1

1

P

0

1

1

[0,0]

[0,

1]

1 -𝑐

!

r = 1

r = 0

𝑎 = 1

[1,0]

[1,

1]

1

1

1 - 𝑐

!

𝑐

!

𝑐

!

(b) SC-MDP kernel

1

r = 1

r = 0

r = 0

r = 0

𝑎 = 0

𝑎 = 1

𝑎 = 1

𝑎 = 0

P

Step 1: Constructing a hard instance M of standard MDP. In this section, we consider the following standard MDP instance M = { S , A , P 0 , T, r } where S = { [0 , 0] , [0 , 1] , [1 , 0] , [1 , 1] } is the state space consisting of four elements in dimension n = 2 , and A = { 0 , 1 } is the action space with only two options. The transition kernel P 0 = { P 0 t } 1 ≤ t ≤ T at different time steps 1 ≤ t ≤ T is defined as

which is illustrated in Fig. 8(a), and

Note that this transition kernel P 0 ensures that the next state transitioned from the state [0 , 0] is either [0 , 0] or [0 , 1] . The reward function is specified as follows: for all time steps 1 ≤ t ≤ T ,

Step 2: The equivalence between M and one SC-MDP. Then, we shall show that the constructed standard MDP M can be equivalently represented by one SC-MDP M sc = { S , A , T, r, C , {P i t } , P c } with C := { 0 , 1 } . The equivalence is defined as the sequential observations { s t , a t , r t } 1 ≤ t ≤ T induced by any policy and any initial state distribution in two Markov processes are identical. To specify, S , A , T, r are kept the same as M . Here, {P i t } shall be specified in a while, which determines the transition to each dimension of the next state conditioned on the current state, action, and confounder distribution for all time steps, i.e., s i t +1 ∼ E c t ∼ P c t [ P i t ( · | s t , a t , c t ) ] for any i -th dimension of the state ( i ∈ { 1 , 2 } ) and all time step 1 ≤ t ≤ T . For convenience, we denote P t := [ P 1 t , P 2 t ] ∈ ∆( S ) as the transition kernel towards the next state, namely, s t +1 ∼ E c t ∼ P c t [ P t ( · | s t , a t , c t )] .

Then we simply set the nominal distribution of the confounder as follows:

In addition, before introducing the transition kernel {P i t } of the SC-MDP M sc , we introduce an auxiliary transition kernel P sc = { P sc t } as follows:

and

It can be observed that P sc is similar to P 0 except for the transition in the state [0 , 0] .

Armed with this transition kernel P sc , the {P i t } of the SC-MDP M sc is set to obey

which is illustrated in Fig. 8(b), and

With them in mind, we are ready to verify that the marginalized transition from the current state and action to the next state in the SC-MDP M sc is identical to the one in MDP M : for all ( t, s t , a t , s t +1 ) ∈ { 1 , 2 , · · · , T } × S × A × S :

where the second equality holds by the definition of P c in (17), and the last equality holds by the definitions of P (see (20) and (21)).

In summary, we verified that the standard MDP M = { S , A , P 0 , T, r } is equal to the above specified SC-MDP M sc .

In this section, without loss of generality, we consider total variation as the 'distance' function ρ for the uncertainty sets of both RMDP M rob and RSC-MDP M sc -rob , i.e., for any probability vectors P ′ , P ∈ ∆( C ) (or P ′ , P ∈ ∆( S ) ), ρ ( P ′ , P ) := 1 2 ∥ P ′ -P ∥ 1 . Consequently, for any uncertainty level σ ∈ [0 , 1] , the uncertainty set U σ 1 ( P 0 ) of the RMDP (see (1)) and U σ 2 ( P c ) of the RSC-MDP M sc -rob (see (4)) are defined as follow, respectively:

Step 3: Defining corresponding RMDP and RSC-MDP. Equipped with the equivalent standard MDP M and SC-MDP M sc , we consider the robust variants of them respectively - a RMDP M rob = { S , A , U σ 1 ( P 0 ) , T, r } with some uncertainty level σ 1 , and the proposed RSC-MDP M sc -rob = { S , A , T, r, C , {P i t } , U σ 2 ( P c ) } with some uncertainty level σ 2 .

Step 4: Comparing between the performance of the optimal policy of RMDP M rob ( π ⋆,σ 1 RMDP ) and that of RSC-MDP M sc -rob ( π ⋆,σ 2 RSC ). To continue, we specify the robust optimal policy π ⋆,σ 1 RMDP associated with M rob and π ⋆,σ 2 RSC associated with M sc -rob and then compare their performance on RSC-MDP with some initial state distribution.

To begin, we introduce the following lemma about the robust optimal policy π ⋆,σ 1 RMDP associated with the RMDP M rob .

Lemma 1. For any σ 1 ∈ (0 , 1] , the robust optimal policy of M rob obeys

In addition, we characterize the robust SC-value functions of the RSC-MDP M sc -rob associated with any policy, combined with the optimal policy and its optimal robust SC-value functions, shown in the following lemma.

Lemma 2. Consider any σ 2 ∈ ( 1 2 , 1] and the RSC-MDP M sc -rob = { S , A , T, r, C , {P i t } , U σ 2 ( P c ) } . For any policy π , the corresponding robust SC-value functions satisfy

In addition, the optimal robust SC-value function and the robust optimal policy π ⋆,σ 2 RSC of the RMDP M sc -rob obeys:

Armed with above lemmas, applying Lemma 2 with policy π = π ⋆,σ 1 RMDP obeying [ π ⋆,σ 1 RMDP ] 1 (0 | s ) = 1 in Lemma 1, one has

where the inequality holds by the fact that the probability distribution P obeying P 1 (0) = 1 4 and P 1 (1) = 3 4 is inside the uncertainty set U σ 2 ( P c 1 ) (recall that σ 2 ∈ ( 1 2 , 1] and P c 1 (0) = 1 ).

Finally, combining (27) and (26) together, we complete the proof by showing that with the initial state distribution ϕ defined as ϕ ([0 , 0]) = 1 , we arrive at

where the last inequality holds by T ≥ 2 .

C.3 Proof of auxiliary results

C.3.1 Proof of Lemma 1

Step 1: specifying the minimum of the robust value functions over states. For any uncertainty set σ 1 ∈ (0 , 1] , we first characterize the robust value function of any policy π over different states. To start, we denote the minimum of the robust value function over states at each time step t as below:

where the last inequality holds that the reward function defined in (16) is always non-negative. Obviously, there exists at least one state s π min ,t that satisfies V π,σ 1 t ( s π min ,t ) = V π,σ 1 min ,t .

With this in mind, we shall verify that for any policy π ,

To achieve this, we use a recursive argument. First, the base case can be verified since when t +1 = T +1 , the value functions are all zeros at T +1 step, i.e., V π,σ 1 T +1 ( s ) = 0 for all s ∈ S . Then, the goal is to verify the following fact

with the assumption that V π,σ 1 t +1 ( s ) = 0 for any state s = { [0 , 1] , [1 , 0] } . It is easily observed that for any policy π , the robust value function when state s = { [0 , 1] , [1 , 0] } at any time step t obeys

where (i) holds by r t ( s, a ) = 0 for all s = { [0 , 1] , [1 , 0] } , the fact P 0 t ( s | s, a ) = 1 for s ∈ S (see (14) and (15)), and the definition of the uncertainty set U σ 1 ( P 0 ) in (23). Here (ii) follows from the recursive assumption V π,σ 1 t +1 ( s ) = 0 for any state s = { [0 , 1] , [1 , 0] } , and the last equality holds by V π,σ 1 min ,t +1 ≤ V π,σ 1 t +1 ([0 , 1]) (see (29)). Until now, we complete the proof for (31) and then verify (30).

Note that (30) direcly leads to

Step 2: Considering the robust value function at state [0 , 0] . Armed with the above facts, we are now ready to derive the robust value function for the state [0 , 0] .

When 2 ≤ t ≤ T , one has

where (i) holds by r t ([0 , 0] , a ) = 1 for all a ∈ { 0 , 1 } and the definition of P 0 (see (15)), and the last equality arises from (33) .

Applying (34) recursively for t, t +1 , · · · , T yields that

When t = 1 , the robust value function obeys:

where (i) holds by r 1 ([0 , 0] , a ) = 1 for all a ∈ { 0 , 1 } , (ii) follows from the definition of P 0 (see (14)), and the last equality arises from (30) and (33).

Step 3: the optimal policy π ⋆,σ 1 RMDP . Observing that V π,σ 1 1 ([0 , 0]) is increasing monotically as π 1 (0 | [0 , 0]) is larger, we directly have that π ⋆,σ 1 RMDP (0 | [0 , 0]) = 1 .

̸

C.3.2 Proof of Lemma 2

To begin with, for any uncertainty level σ 2 ∈ ( 1 2 , 1] and any policy π = { π t } , we consider the robust SC-value function ˜ V π,σ 2 t of the RSC-MDP M sc -rob .

Step 1: deriving ˜ V π,σ 2 t for 2 ≤ t ≤ T . Towards this, for any 2 ≤ t ≤ T and s ∈ S , one has where (i) follows from the state-confounded Bellman consistency equation in (47), (ii) holds by that the reward function r t and P t are all independent from the action (see (16) and (21)), and the last inequality holds by P t ( s ′ | s, a, c t ) = 1 ( s ′ = s ) is independent from c t (see (21)).

Applying the above fact recursively for t, t +1 , · · · , T leads to that for any s ∈ S ,

Considering that the action does not influence the state transition for t = 2 , 3 , · · · , T and all other states s = [0 , 0] , without loss of generality, we choose the robust optimal policy as

which directly yields (see reward r in (16))

Step 2: characterizing ˜ V π,σ 2 1 ([0 , 0]) for any policy π . In this section, we consider the value of ˜ V π,σ 2 1 on the state [0 , 0] . To proceed, one has

where (i) holds by robust state-confounded Bellman consistency equation in (47), (ii) follows from r 1 ([0 , 0] , a ) = 1 for all a ∈ { 0 , 1 } which is independent from c t . (iii) arises from the definition of P in (20), (iv) can be verified by plugging in the definitions from (14) and (18), and the penultimate equality holds by (40).

Step 3: characterizing the optimal robust SC-value functions. Before proceeding, we recall the fact that U σ ( P c 1 ) = { P ∈ ∆( C ) : 1 2 ∥ P -P c 1 ∥ 1 ≤ σ 2 } .

Observing from (41) that for any fixed π 1 (0 | [0 , 0]) , c 1 ( 1 -2 π 1 (0 | [0 , 0]) ) is monotonously increasing with c 1 when 1 -2 π 1 (0 | [0 , 0]) ≥ 0 and decreasing with c 1 otherwise, it is easily verified that the maximum of the following function

obeys

Then, note that the value of ˜ V π,σ 2 1 ([0 , 0]) only depends on π 1 ( · | [0 , 0]) which can be represent by π 1 (0 | [0 , 0]) . Plugging in (43) into (41) arrives at when π 1 (0 | [0 , 0]) ≥ 1 2 ,

where the penultimate equality holds by σ 2 > 1 2 and letting π 1 (0 | [0 , 0]) = 1 2 . Similarly, when π 1 (0 | [0 , 0]) < 1 2 ,

Consequently, combining (44) and (45), we conclude that

C.3.3 Auxiliary results of RSC-MDPs

It is easily verified that for any RSC-MDP M sc -rob = { S , A , T, r, C , {P i t } , U σ 2 ( P c ) } , any policy π and optimal policy π ⋆ satisfy the corresponding robust state-confounded Bellman consistency equation and Bellman optimality equation shown below, respectively:

D Experiment Details

D.1 Architecture of the structural causal model

position

position

embedding

embedding

?s

1

Encoder

feature

~GΦ

feature

State

Decoder

st

Encoder

Next

State

0

Encoder

matrix

Decoder

multiplication

Action

Decoder

r't

Reward

at

dA

Encoder

R(n+dA)xf

R(n+1)xf

We plot the architecture of the structural causal model we used in our method in Figure 9. In normal neural networks, the input is treated as a whole to pass through linear layers or convolution layers. However, this structure blends all information in the input, making the causal graph useless to separate cause and effect. Thus, in our model, we design an encoder that is shared across all dimensions of the input. Since different dimensions could have exactly the same values, we add a learnable position

embedding to the input of the encoder. In summary, the input dimension of the encoder is 1 + d pos , where d pos is the dimension of the position embedding.

After the encoder, we obtain a set of independent features for each dimension of the input. We now multiply the features with a learnable binary causal graph G . The element ( i, j ) of the graph is sampled from a Gumbel-Softmax distribution with parameter ϕ i,j to ensure the loss function is differentiable w.r.t ϕ .

The multiplication of the causal graph and the input feature creates a linear combination of the input feature with respect to the causal graph. The obtained features are then passed through a decoder to predict the next state and reward. Again, the decoder is shared across all dimensions to avoid information leaking between dimensions. Position embedding is included in the input to the decoder and the output dimension of the decoder is 1 .

D.2 Details of Tasks

We design four self-driving tasks in the Carla simulator [22] and four manipulation tasks in the Robosuite platform [23]. All of these realistic tasks contain strong spurious correlations that are explicit to humans. We provide detailed descriptions of all these environments in the following.

Brightness

Behavior

Crossing

CarType

Day

Wait

Red light

Slow

Ego

go

Ego

Ego

Overtake

Night

Cross

Fast

Leading

Green light

Ego

Ego

Ego

Brightness. The nominal environments are shown in the 1 th column of Figure 10, where the brightness and the traffic density are correlated. When the ego vehicle drives in the daytime, there are many surrounding vehicles (first row). When the ego vehicle drives in the evening, there is no surrounding vehicle (second row). The shifted environment swaps the brightness and traffic density in the nominal environment, i.e., many surrounding vehicles in the evening and no surrounding vehicles in the daytime.

Behavior. The nominal environments are shown in the 2 nd column of Figure 10, where the other vehicle has aggressive driving behavior. When the ego vehicle is in front of the other vehicle, the other vehicle always accelerates and overtakes the ego vehicle in the left lane. When the ego vehicle is behind the other vehicle, the other vehicle will always accelerate. In the shifted environment, the behavior of the other vehicle is conservative, i.e., the other vehicle always decelerates to block the ego vehicle.

Crossing. The nominal environments are shown in the 3 rd column of Figure 10, where the pedestrian follows the traffic rule and only crosses the road when the traffic light is green. In the shifted environment, the pedestrian disobeys the traffic rules and crosses the road when the traffic light is red.

CarType. The nominal environments are shown in the 4 th column of Figure 10, where the type of vehicle and the speed of the vehicle are correlated. When the vehicle is a truck, the speed is low and

when the vehicle is a motorcycle, the speed is high. In the shifted environment, the truck drives very fast and the motorcycle drives very slow.

Lift

Stack

Wipe

Door

Right

Low

Left

Right

Green

Right

Diagonal

→Right

High

Left

Left

Red

Anti-

Right

Left

diagonal

Left

Lift. The nominal environments are shown in the 1 th column of Figure 11, where the position of the cube and the color of the cube are correlated. When the cube is in the left part of the table, the color of the cube is green, when the cube is in the right part of the table, the color of the cube is red. The shifted environment swaps the color and position of the cube in the nominal environment, i.e., the cube is green when it is in the right part and the cube is red when it is in the left part.

Stack. The nominal environments are shown in the 2 nd column of Figure 11, where the position of the red cube and green plate are correlated. When the cube is in the left part of the table, the plate is also in the left part; when the cube is in the right part of the table, the plate is also in the right part. In the shifted environment, the relative position of the cube and the plate changes, i.e., When the cube is in the left part of the table, the plate is in the right part; when the cube is in the right part of the table, the plate is in the left part.

Wipe. The nominal environments are shown in the 3 rd column of Figure 11, where the shape of the dirty region is correlated to the position of the cube. When the dirty region is diagonal, the cube is on the right-hand side of the robot arm. When the dirty region is anti-diagonal, the cube is on the left-hand side of the robot arm. In the shifted environment, the correlation changes, i.e., when the dirty region is diagonal, the cube is on the left-hand side of the robot arm; when the dirty region is anti-diagonal, the cube is on the right-hand side of the robot arm.

Door. The nominal environments are shown in the 4 th column of Figure 11, where the height of the handle and the position of the door are correlated. When the door is closed to the robot arm, the handle is in a low position. When the door is far from the robot arm, the handle is in a high position. In the shifted environment, the correlation changes, i.e., when the door is closed to the robot arm, the handle is in a high position; when the door is far from the robot arm, the handle is in a low position.

D.3 Example of Generated Data by Perturbations

We show an example of generated trajectories in the Lift task to demonstrate the reason why our method obtains robustness against spurious correlation. In Figure 12 (a), we show the collected trajectories from the data buffer. Since the green block is always generated on the left side of the table, the trajectories of the green block mainly appear on the left side of the table. In Figure 12 (b), we generate new trajectories from a trained transition model and we observe that the distribution of trajectories follows the collected data. In Figure 12 (c), we directly perturbed the dimensions of the state and used the same transition model to generate new trajectories. We find that the generated trajectories blend the color but fail to maintain the spatial distribution of the original data. In Figure 12

(d), we use the causal-based transition model to generate new trajectories and we find that the results not only follow the spatial distribution but also blend the color.

The results shown in Figure 12 illustrate that the data generated by our method eliminates the spurious correlation between the color and position of the block, therefore, enabling the policy model to generalize to the shifted environment.

Nominal (training)environment

Shifted(testing)environment

(a) Trajectory from nominal env

(b) Generated trajectory w/o perturbation

(c) w/ causal graph w/ perturbation

0.4

0.4

0.4

0.4

Ours

0.3

E'0

0.3

0.3

0.2

0.2

0.2

0.2

y-axisof table

0.1

0.1

0.1

0.1

0.0

0.0

0.0

0.1

0.1

0.1

0.1

0.2

0.2

0.2

0.2

trajectory of

trajectory of

0.3

greencube

red cube

E0=

0.3

0.3

0.4

0.3

0.2

0.1

x-axis of table

0.0

0.1

0.2

0.3

0.4

0.3

0.2

0.1

x-axis of table

0.0

0.1

0.2

E'0

0.4

0.3

0.2

0.1

x-axis of table

0.0

0.1

0.2

E'0

0.4

E'0-

0.2

0.1

x-axis of table

0.0

0.1

0.2

0.3

D.4 Computation Resources

Our algorithm is implemented on top of the Tianshou [101] package. All of our experiments are conducted on a machine with an Intel i9-9900K CPU@3.60GHz (16 core) CPU, an NVIDIA GeForce GTX 1080Ti GPU, and 64GB memory.

D.5 Hyperparameters

We summarize all hyper-parameters used in the Carla experiments (Table 5) and Robosuite experiments (Table 6).

D.6 Discovered Causal Graph in SCM

To show the performance of our learned SCM, we plot the estimated causal graphs of all experiments in Figure 13, Figure 14, Figure 15, Figure 16, and Figure 17.

0.00

0.00

0.00

0.00

0.00

0.00

000

other speed

other y

0.00 0.00

0.000.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.00

00'1

1.00

1.00

1.00

0.00

00

0.00

1.00

other y

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

1.00

1.00

1.00

other theta

other x

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

1.00

other speed

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

other speed

other y

0.00 0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.000.00

0.00

0.00

1.00

1.00

0.00

0.00

00'0

1.00

1.00

0.00

1.00

00'0

1.00

1.00

1.00

00:0

00

001

other theta

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

other theta

0.00

0.00

0.00

0.00

0.00

00'0

collision

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

other x

0.00

0.00

0.00

0.00

0.00

0.00

1.00

1.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

1.00

other y

0.00

0.00

0.00

0.00

0.00

0.00

000

1.00

weather

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

other speed

0.00

0.00

0.00

0.00

0.00

0.00

0.00

00'0

0.00

.00

other theta

0.000.00

0.00

000

0.00

0.00

000

000

0.00

000

1.00

goal x

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

olision

sunlight

0.00

0.00

0.00

0.00

0.00

2.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

1.00

.00

0.00

1.00

0.00

0.00

0.00

100

goal y

0.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

1.00

1.00

1.00

goal x

goal y

1.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

00

delta X

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

della x

delta y

0.00

0.00

delta y

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

1.00

oBe

o6a

peeds

f jauo

poods

Crossing

CarType

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

o5o

0.00

1.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

ego speed

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

ego speed

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

other x

0.00

1.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

1.00

other X

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

othery

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

other y

0.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

1.00

pads ogo

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

other speed

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

othertheta

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

other theta

000

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

colision

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

collision

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

trafi ight

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

1.00

car type

0.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

1.00

goal x

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

goal x

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

goal y

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

goal y

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

delta x

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

delta x

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

delta y

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

delta y

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

x o6o

o69

poods o6o

X Jao

人 o

speed

oba

speed

X go

人 sougo

collision

od e

x j606

[o]sod aqno

1.00

cube pos[1]

0.00

cube pos[2]

[ojenb eqno

1.00

cube quat[1]

cube quat[2]

[eenb aqno

0.00 0.00 0.00

gripper-cube pos[0]

gripper-cube pos[1]

gripper-cube pos[2]

1.00

cube color[0]

0.001

1.00

1.00

cube color[1]

cube color[(2]

joint pos cos[0]

joint pos cos[1]

0.000.000.000.000.000.000.000.000.00.000.000.000.000.001.00.000.000.000.000.000.00

1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

joint pos cos[2]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0

00000000000000000000000000000000000000000000000000000:00000000000000000000000000000000000000

joint pos cos[3]

000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

joint pos cos[4]

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

joint pos cos[5]

0000000000000000000000000000000000000000000000000000000000000000000000

joint pos cos[6]

0.100.0.0...0.......0.......0..0.0.0.0......00.....000....0

001000000000001

joint pos sin[o]

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

joint pos sin[1]

00000000000000000000000000000000000

joint pos sin[2]

0.00000.00000.000.00.000.00.00.0.0.1.01...00.000000.0.000.00000.0.00.0.00.000000

joint pos sin[3]

0.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.0.00.00.1.00.00.00.00.000.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.1.00

joint pos sin[4]

0000000000000000.00000000000

[uis sod o

000.00000..0000..0.1.0.0.0..000.0000.00.000.0..1..01..0..0

joint pos sin[6]

0.0.000.0.0000.00000.00000.0000000.0000.0000000.00.00000.0000.000.00.00000.00000.0000.00.00000000.00.0.0

joint vel[1]

000000000000000000000000000000

0.00.00.000.00.00.000.00.00.00.00.00.00.00.00.00.00.0.00.00.0.00.00.00.00.00.00.00.00.00.00.000.000.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.000.0

joint vel[3]

00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

1.00 1.00 1.00 1.00 1.00 1.00 1.00

0.0.00.00.0.00.00.0.0.00.00.00.000.000.00.000.000.000.000.000.00.000.00.000.000.00.00.00000000.0.0.0.00.00.00.000.00.00.000.00.000.000.00.0000000.000.0000.00.0

joint vel[5]

0.000.000.000.000.000.000.000.000.00.000.000.000.000.000.000.000.000.00.000.000.000.000.000.000.000.000.000.001.000.001.000.001.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

1.00 1.00 1.00 1.00 1.00 1.00 1.00

joint vel[6]

00000000000000000000000000000000000000000000000000000000000000000000000000000

1.00 1.00 1.00 1.00 1.00 1.00 1.00

eef pos[0]

0000.0.0000.00.00.0.00000.0000000000000.00.0.01.01.

1.00

eef pos[1]

0.01.00.0.00000000000.0000.00.00000000000.00.00.0000.00.0.0.000.0.0000..000.00.000.00.0.00.00.00000000000.

1.00

eef pos[2]

0.00000.000000.0000.00.0000.00.000000.000000.0000000.00000.00.00.0.00.00.00.00.0.00.0000.00.0000.00.0000.00.000.

1.00 1.000.00

1.00 1.00 1.00 1.00

eef quat[(o]

0.000.000.00.000.000.000.00.00.00.00.0.00.0.00.00.00.00.00.00.00.000.00000000.00.000.000.000.000.000.00.000.000.00.00.00.0.0.0.00.00.00.00.00.00.000.00.0000.00.000.000.000

eef quat[1]

0.0000.000000.0000.00000000.000.000.000.000.0000.000.00.00.0.0.00.00.0000.

eef quat[2]

0.00.00.0.0.0.0.0.0.00.00.00.00.00.00000000.00.000.000.00.000.00000.0..0.00.0.0.0.0.00.00.00.000.000000.00.000.000.00.000.000.000.0.0.

0.0.00.0.0.0.0.0.0.00.00.00.000.000.00000000.00000000.00000.0000000.0.000.0000.0.0.0.00.00.00.000.00.000000.000000.0000000000000.00000

0.00.00.00.00.00.00.00.000.00.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00.00.00.00.00.00.00.00.00.00.000.00.000.00.00.000.000.01.00.01.00.000.

1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper qpos[1]

0.00 0.000.000.000.000.000.00 0.000.00 0.000.000.00 0.00 0.000.000.000.000.000.00 0.000.00 0.000.00 0.000.000.00 0.000.000.000.000.000.00 0.00 0.000.000.000.00 0.00 0.000.000.000.001

1.000

000000000000

00000000000000

gripper qpos[2]

0.00.00.0.0.0.0.0.0.00.00.00.000.000.00000000.00.000.000.000000.0000000.0.0.00.0.0.0.0.00.00.000.000.0000000000.000.00000000000.000.0.

gripper qpos[3]

0.00 0.000.000.000.00 0.000.00 0.000.00 0.00 0.00 0.00 0.00 0.000.000.000.000.000.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.000.000.000.000.00 0.00 0.000.00 0.000.00 0.00 0.000.00 0.000.000.000.00

1.000.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper qpos[4]

0.00 0.000.000.00 0.000.000.00 0.00.000.000.000.000.00 0.00.000.00 0.00.000.000.000.000.000.00.000.000.00.000.000.00.000.000.00.000.000.00.000.000.00.000.000.00.000.000.00.001.00

0.00

1.00 1.001.00 1.00 1.001.00 1.00

gripper qpos[5]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0

0.00 0.000.000.000.000.000.00

gripper qvel[0]

0.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00.00 0.00 0.00 0.00 0.001

1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper qvel[1]

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.001.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper qvel[2]

00000:0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper qvel[3]

00000:000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000:000000000:0000000000000000000000000000000000

1.00 1.00 1.00 1.00 1.00 1.00 1.00

1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper qvel[5]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper pos[0]

1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper pos[1]

1.00

gripper pos[2]

1.00 1.00 1.00 1.00 1.00 1.00 1.00

gripper [0]

000000000000

pos[2]

[llenba

quat[3]

[ojsod

r-cube

[L]sod a

[z]sod e

color[1]

[ojsoos

cos[4]

cos(6]

sin[1]

tvel[1]

[]sod

[ejenb ,

Jeddu6

sodb

[qpos[2]

Jaddu

[e]sodb

[d]sodb

[s]sodb

rqvello]

rqvel[1]

Jeddu6

rqvel[3]

rqvel4]

jeddub

[qvel[5]

jeddub

b

[ojsod aqn

1.000

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

cube A pos[1]

0.000.00

1.00

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

1.001.001.000

1.00

oenbaqn

1.001.001.00.00

1.00

0.00

1.00

cube A quat[1]

0.000.000.000.00

1.001.001.00

1.00

0.00

1.00

cube A quat[2]

0.000.000.000.000.00

1.00

00..........................................

000000

1.00

1.00

ojsod g aqno

cube B pos[1]

1.00

cube B pos[2]

cube B quat(oj

engagn

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

cube Bquat[2]

gripper-cubeA pos[1]

gripper-cubeApos[0]

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000

1.00

1.00

gripper-cubeA pos[2]

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.001.000

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.001.000.000.001.000

0.00

.00

gripper-cubeB pos[0]

1.00

gripper-cubeBpos[1]

00.000.000.000.000.000.000.000.000.000.000.000.000.000

1.00

gripper-cubeBpos[2]

1.00

joint pos cos[0]

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00.000.000.000.000.000.000.000.000.000

joint pos cos[1]

.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.001.000.001.000

0.000.000.000.001.00.001.000.001.00.000.000.001.00.000.000.000.000.000.000.000.000.000.001.001.001.001001.001.001.00

joint pos cos[2]

joint pos cos[3]

0.000.000.000.001.001.001.00

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.001.001.001.00

joint pos cos[4]

.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.001.000.000.000.000.000

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00.000.000.00.000.000.000.00.000.000.o0

0.000.00000.000.000.000.000.00.000.000000.000.000.000.000.000.000.01.000

0.000.000.000.000.001.000.000.000.000.00

1.000.000.000.000.00

01.000.001.000.000.000.001.000.000.000.00.000.000.00000.000.000.001.001.001

joint pos cos[6]

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000

0.000.000.000.000.000.000.000.000.000.000

joint pos sin[o]

000.000.000.000.01.000.000.00.000.000.000.00000.000.000.000.001.00

0.000.000.000.000.0

joint pos sin[1]

joint pos sin[2]

0.000.000.000.000.000.00.000.000.000.000.000.000.000.000.000.000.000.00.000.000.000.000.00.000.00.0

[els sod

.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

1.000.000.000

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.0

00000000

joint pos sin[4]

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000

1.00

1.001.001.00

joint pos sin[5]

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

1.001.001.00

1.001.001.001.0

joint pos sin[6]

joint vel2]

joint vel[1]

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.0

000.0

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

1.00

1.00

01.00.001.000.000.00.000.000.000.00.00.000.000.000.000.000.000.000.001.001.001.001.001.001.001.00

1.00

0.000.000.000.000.000.000.000.000.000.000.000.000.000.00

1.001.00 1.001.001.00 1.001.00

joint vel[3]

1.000.001.00

1.000.000.000.00.000.000.000.000.000.000.000.00.000.00

1.001.001.00

1.000.000.00

0.000.000.000.000.00.00.000.000.00.00.000.000.0000.000.000.000.0.00.000.000.00.00.000.000.0.00.000.000.00.00.000.000.00.00.00

0.000.000.000.000.000.000.00.000.000.000.00.00.000.00

1.001.001.001.001.001.001.00

joint vel[6]

1.00

1.001.001.001.001.001.0.00

[o]sod jea

eef pos[1]

1.001.00

01.00

eef pos[2]

oenbjea

0.000.000.000.000.0.00.00.000.00.0.00.000.0.00.00.000.00.00.00.000.000.0.00.000.000.00.00.00.000.0.00.00.000.00.00.00.000.000.00.00.000.000.00.00.000.000.0.00.000.000.00.00.00.000.0.00.00.000.00..

eef quat[1]

eef quat[2]

enbea

000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

1.000.000.000.00

01.001.001.001.001.001.001.00

gripper qpos[1]

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00

1.00

.000.000.00

1.001.001.001.001.001.001.00

gripperqpos[2]

[elsodb jaddus

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.0

000.000.000.000.000.000.000.000.000.000.00

1.00

1.001.001.001.001.001.001.00

1.001.001.00.001.000.001.00

gripper qpos[4]

gripper qpos[5]

0.000.0.00.000.000.000.00.000.000.000.00.00.000.000.00.00.000.000.0.00.000.000.000.00.000.000.00.0.00.000.000.00.00.000.00.00.00.000.000.00.00.000.000.0.00.000.000.000.00.000.000..000.

L00

1.001.001.001.001.001.001.00

gripper qvel[o]

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.001

01.00.000.000.000.000.000.000.00.00.000.000.00000.000.000.00.000.000.000

gripper qvel[1]

gripper qvel[2]

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000

1.001.00 1.00 1.001.001.001.00

gripper qvel[3]

1.001.001.001.001.001.001.00

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00.00

1.001.001.001.001.001.001.00

gripper qvel[5]

000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.

gripperpos[0]

gripperpos[1]

1.001.001.001.001.001.001.00

gripper pos[2]

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.0

000.000.000.000.000.000.000.000.000.000.00

1.001.001.00.001.00.000.000.000.00.000.000.000.00.00.

1.001.001.001.001.001.001.00

gripper [0]

00.000.000.000.000.000.000.001.001.000.00

1.001.001.001.001.01.0.0

qenb

henb

0

2

5

8

#

#

#

jenb

sodb

sodb

sodb

A

B

B

B

B

B

B

T-cubeA

beB

3

jaddu

ripper

品

1.00

hinge qpos

0.00

1.00

1.00 1.00

0000000000000

1.001

.00 1.00 1.00 1

1.00 1.00 1.00

joint pos cos[0]

000000

1.000.00 0.00

joint pos cos[1]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0

00 0.00 0.00 0.00 0.00 0.00 0

joint pos cos[2]

1.00

joint pos cos[3]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

00 0.00 0.00 0.00

1.00 1.00 1.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

joint pos cos[4]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0

0.001.000.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00 1.00

0.00 0.00 0.00 0.00 0.00

joint pos cos[5]

1.00 1.00

joint pos cos[6]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

joint pos sin[0]

joint pos sin[1]

00 0.00 0.00

1.00 1.00 1.00 1.00 1.00 1.00 1.00

joint pos sin[2]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00

0.00

000000

0.00

1.00

joint pos sin[3]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

000000

000000

joint pos sin[4]

100 0.000.000.0

1.00

1.00 1.00

1.00

1.00

joint pos sin[5]

joint pos sin[6]

0.00

1.00

0.00 0.00

1.00

joint vel[1]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0

0.00

joint vel[3]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0

joint vel[5]

joint vel[6]

0.00 0.00 0.00 0.00 0.00 0.001.000.00 0.001.000.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00.00 0.00 0.00 0.00 0.00 0.00 0.0

0000000000000

eef pos[1]

eef pos[2]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.001.000.00 0.001.00

[oenbjea

eef quat[1]

eef quat[2]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00

0.00 0.00 0.00 0

1.00 0.00 0.00 0.001.000

000

gripper qpos[1]

[o]sodb jeddub

.00 0.00 0.00

00 0.00 0.00

gripper qpos[2]

gripper qpos[3]

gripper qpos[4]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00 1.00

0.00 0.00

1.00

000 0000000000000000000000000000000000

1.00

gripper qpos[5]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00

0.00

gripper qvel[1]

00 1.00 1

gripper qvel[2]

gripper qvel[3]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0

0.00

0.00

gripper qvel[5]

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

joint vel[o]

1.000.00

joint vel[1]

000000

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.000.000.000.000.000.000.000.000.000.00.00.0.000.000.000.00.00.000.000.00.000.000.000.000.000.000.00.01.000.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00

joint vel[5]

joint vel[6]

1.00 0.00 0.00

000

gripper [0]

1.00

pos

sod

pos[0]

pos[1]

[pos[2]

[ojsod

[L]sod

[]sod

sodb afuu

cos[0]

Scos[1]

cos[2]

[]soo

cos[4]

cos[5]

cos[6]

s sin[0]

sin[1]

5

t vel4]

vel(6]

[o]sod

pos[1]

[z]sod

[olenb

[]jenb

[henb

[elenb

[o]sodb.

[L]sodb ]

[ejsodb jaddun

rqpos[4]

rqvel[o]

rqvel[1]

rqvel[2]

rqvel[3]

qvel[4]

rqvel[5]

rewarc

handle

handle

gripper-door

gripper-door

gripper-door

gripper-handle

gripper-handle

joint

Jeddu6

e

join

■

■

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to make RL robust to distribution shifts by limiting their reliance on spurious correlations between state features.  This is achieved through a new RL algorithm designed in a variant of Robust MDPs extended to include a more structured uncertainty representation.\n\nSTRENGTHS:\n* This paper addresses a serious underlying flaw in the standard approaches for Robust MDPs, in that the uncertainty set is usually artificial an no alined to the ways we need our policies to be robust.\n\nWEAKNESSES:\nThe language is often confusing, for instance the term \"spurious correlations\" is often used to refer specifically to spurious correlations between state variables.  This is very confusing as it overloads the term, and  makes it hard to refer to other spurious correlations (such as a spurious correlation between the agents actions and the reward or another agents actions).  Better to say \"Spurious state correlations\".\n\nOn a similar note \"semantic uncertainty\" is a confusing term that wasn't explained until deep into the paper.  It sounds like uncertainty **about**  semantics (which is how it has been used in the literature), but it is meant to be uncertainty that is not just a norm-ball perturbation over the transition function.  A better term would be \"structured\" uncertainty.\n\n* Other  more minor confusing wordings make the paper more difficult to read than it needs to be:\n\t* I don't know what it means for portions of the state to \"not have causality\"\n\t* I don't know what it means for an uncertainty set to be \"shaped by the unobserved confounder and sequential structure of RL\"\n\t* I don't know what it means for some approach to be \"superior in breaking spurious correlations\"\n\t* There is a typo in : \"Despite various types of uncertainty have been investigated in RL\"\n\n\nIn addition it's unclear how sensitive the approach is to the degree of robustness.  It's a famous issue of RMDPs that the degree of robustness has to be carefully selected, too large and you have  an overly conservative policy, and too small you get no robustness.  Giving that part of the argument for this approach is alleviating that problem (as shown in Figure 3), a sensitivity analysis here would greatly improve the paper, as a main draw away from RMDPs is that they are very sensitive to the robustness parameter.\n\nQUESTIONS:\nHow sensitive is the approach to changes in B% and K?\n\nLIMITATIONS:\nIt is important to discuss somewhere how hyper-parameters (such as k and B%) were chosen, and could be chosen in a new domain.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a state-confounded (SC-)  and a robust-state confounded (RSC-) MDP formulation to account for setups where a confounder satisfying the backdoor criterion confounds the states. The RSC-MDP setup assumes that the confounder lies in an uncertainty set which is part of the MDP parameters, and the aim is to learn a value function that is the lower bound of the different value functions arising out of different instances of the RSC-MDP at different values of the confounder within its uncertainty set. They propose an empirical algorithm  to approximate the effect of perturbing the confounder by perturbing single dimensions of the states and setting them to values present in ‘otherwise nearby’ states (defined by distance between the two states in dimensions other than the dimension being edited). The paper claims that this indirectly estimates the effects of perturbing the confounder. They learn a  graph-based model that is supposed to be a causal model, to generate new transitions (given a state and action s_t, a_t, generate s_t+1, r_t) based on the perturbed states and (original) action pairs. They augment SAC by randomly replacing part of the sampled batches with the generated transitions and learn as usual, and show that this recipe is robust to the spurious correlation in the nominal envs they define. All experiments are in the state space (non-vision inputs).\n\nSTRENGTHS:\n1. The paper studies an important problem and proposes a set of  benchmarks with handcrafted spurious correlations in a CARLA environment and in a robosuite environment which could be useful for more future works to study spurious correlations in RL. \n2. The RSC-MDP formulation could also be useful, although I’m not sure how much the notation and proofs scale to an arbitrary number of confounding variables.\n\nWEAKNESSES:\n1. The paper structure is somewhat confusing to me: The MDP formulations and the empirical algorithm don’t seem to have much of a connection and seems like both were developed independently - please correct me if I’m missing something here. It would be great to further clarify exactly how the algorithm is helping to solve the robust SC-MDP.\n\n2. The notion of semantic uncertainty is not very precise or clear (I don’t think there’s any references either), and could be removed in my opinion since the RSC-MDP formulation can simply use the “uncertainty set” phrasing.\n\n3. The appendix suggests that the training environments have perfect spurious correlations: the correlation isn’t broken in even a few instances of the training environments, so for eg. there would never be a (non-generated) transition in the SAC buffer that would have the brightness value low in its state space value, in the case that the training envs are following the correlation regime where brightness and traffic are both high during the day. It’s unclear to me then, why the causal model would ever generate a state value that doesn’t also have the brightness value set to high (same logic for the perturbation procedure). Am I missing something else here? Or are the training envs set up such that an agent will train on two kinds of envs at once - night time with less traffic and day time with more traffic, and so it s possible to see samples with the brightness value set to both low and high across otherwise nearby states?\nIt would be great to clarify what exactly is the train and test env distribution and what is expected of the graphical causal model - is it to generate counterfactual transitions (counterfactual to what?) and conclusively show that it is actually doing that by visualizing or plotting some property of the samples it is generating. It's possible that I have misinterpreted something here, and I'm willing to raise my score if the authors' response clarifies some of these questions. \n\n4. I would also suggest mentioning relevant related work explicitly aimed at resolving causal confusion in online [1] and offline RL [2] (and comparing to the closest version if applicable).\n\n5. It would be great to include the training curves for the results in Table 1 as well, since that gives insight into whether other methods simply converge later to a similar highest reward, or whether they are entirely limited in their ability to converge to a similarly high reward as the best performing method. \n\n[1] Resolving Causal Confusion in Reinforcement Learning via Robust Exploration . Clare Lyle, Amy Zhang, Minqi Jiang, Joelle Pineau, Yarin Gal. ICLR Self-Supervised RL Workshop 2021.\n\n[2] Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning? Gunshi Gupta, Tim G. J. Rudner, Rowan Thomas McAllister, Adrien Gaidon, Yarin Gal, NeurIPS Offline RL Workshop 2022, CleaR 2023\n\nQUESTIONS:\n1. How is the experiment to answer R3 designed? It's unclear how to test the w/o P^{c} case since the confounder isn't known anyway? \n\n2. As stated previously, it would help to get more insights into how novel the generated transitions from the causal model really are.\n\n3. Have you observed or quantified the robustness-performance tradeoff mentioned in line 263, which will lead to a performance drop in some envs? I expect the perturbation procedure of states to only work for specific kinds of envs, so it would be good to show the failure cases as well.\n\nLIMITATIONS:\nGiven how heuristic the technique of generating perturbed states is (swapping single dimensions between otherwise nearby states), the paper should properly discuss the many challenges of scaling this approach to a high-dimensional state space like that in visual domains.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the spurious correlation challenge that arises in RL. Such correlation is typically useless to decision-making but may be learned by agents, which leads to failure in applying to unknown test cases. To this end, the authors proposed a novel RSC-MDP framework that models the spurious correlation challenge with an unobserved confounder. The authors then justify that previous robust MDP methods may fail on the RSC-MDP framework due to inaccurate uncertainty modeling. The authors proposed a heuristic method to handle the RSC-MDP challenge authors then conducted extensive experiments to compare the proposed method and SOTA baselines for robust RL. Experiments justified that the proposed method has better performance for the proposed spurious correlation attacks.\n\nSTRENGTHS:\n1. The paper is very well written, with a clear explanation of motivation and well organized problem setup and experimental design.\n2. The authors provide theoretical justification that the proposed robustness is milder and more accurate than the traditional robust MDP.\n3. The authors provide novel experimental design to assess the effect of spurious correlation in the state space of RL. The proposed baseline has the potential to be adopted by future study.\n\nWEAKNESSES:\n1. In the proposed driving example, the confounder issue spans over multiple steps of interactions, whereas the latent variable c introduced in RSC-MDP only affects one step, and each step is affected independently (i.e., $c_t$ is independently generated for all $t$). In reality, the brightness and traffic density do not normally get perturbed per each step of the transition. Do the proposed RSC-MDPs cover the targeted spurious correlation problem?\n\n2. Seems that the confounding within stat is not affecting the distribution of transition; that said, the observation still follows the distribution of trajectories after marginalizing the confounder $c$. To me, the spurious correlation challenge of RSC-MDPs is more of a robustness challenge than a confounding challenge to me, where the offline trajectory distribution is typically different from the interventional transition dynamics. How does the spurious correlation within each state affect the learning of transition P(s_{t+1} | s_t, a_t), hence affecting regular RL algorithms?\n\n3. Could authors provide more reasoning on the proposed heuristic perturbation in approximating the true underlying perturbation of latent variables? Intuitively it seems to align well with the motivative driving example, but it seems that the perturbation proposed in equation (6) introduces extra correlations within states. In particular, sample $s^i_k$ selected to replace $s^i_t$ is based on a rule that explicitly involves $s^j_t$ for $j\\neq i$. Do we worry that such introduced correlation ruins training for similar reasons of the targeted spurious correlation? \n\nMisc:\n\nline 81: discounted infinite-horizon -> finite horizon?\n\nline 220: $\\sigma_2 \\in (3/4, 1]$ according to the proof.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how to develop more robust reinforcement learning algorithms when spurious correlation exists in the observation space. The paper studies the problem from a causal perspective and present robust state confounded MDP as the problem formulation. The paper proposes an algorithm to solve this problem formulation by learning the structural causal model. The paper demonstrate experimentally that the proposed method is more robust to shift in the environment compared to baselines.\n\nSTRENGTHS:\nThe paper is mostly extremely well written and easy to follow. The structure of the paper makes sense, which first discusses the formulation of the problem in the form of robust state confounded MDP and comparison to existing formulation. The algorithm the paper proposes is intuitive and the improvements over the baseline in the experiment section are clear.\n\nWEAKNESSES:\nThe discussion around related work focuses on a few most relevant papers and does not address the broader literature if I am not mistaken. The problem of spurious correlation is known within the community and various prior works [1, 2]. Please see [2] for example, which also learns the dynamics model to alleviate the issue of spurious correlation. These prior works tackle this problem even though they did not propose the same robust state confounded MDP that the paper proposes. I would have liked to see a more thorough comparison with the broader literature in a related work section.\n\nOn page 3, the paper writes “people usually prescribe the uncertainty set of the transition kernel using a  heuristic and simple function rho with a relatively small sigma”. However, because of the lack of a related work session, I do not see evidence for this statement.\n\n[1] https://ben-eysenbach.github.io/rpc/\n[2] https://arxiv.org/abs/1909.11373\n\nQUESTIONS:\nPlease see my questions about related work\n\nLIMITATIONS:\nPlease see my questions about related work.\n\nI would have like to see the limitations that the method currently only work for state-based policies more prominently displayed and discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}