Achievable distributional robustness when the robust risk is only partially identified

Authors: julia  kostin
nicola  gnecco
fanny  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:18:56

Abstract: In safety-critical applications, machine learning models should generalize well under worst-case distribution shifts, that is, have a small robust risk. Invariance-based algorithms can provably take advantage of structural assumptions on the shifts when the training distributions are heterogeneous enough to identify the robust risk. However, in practice, such identifiability conditions are rarely satisfied – a scenario so far underexplored in the theoretical literature. In this paper, we aim to fill the gap and propose to study the more general setting of partially identifiable robustness. In particular, we define a new risk measure, the identifiable robust risk, and its corresponding (population) minimax quantity that is an algorithm-independent measure for the best achievable robustness under partial identifiability. We introduce these concepts broadly, and then study them within the framework of linear structural causal models for concreteness of the presentation. We use the introduced minimax quantity to show how previous approaches provably achieve suboptimal robustness in the partially identifiable case. We confirm our findings through empirical simulations and real-world experiments and demonstrate how the test error of existing robustness methods grows increasingly suboptimal as the proportion of previously unseen test directions increases.

Main paper: 
 introduction: 
 The success of machine learning methods typically relies on the assumption that the training and test data follow the same distribution. However, this assumption is often violated in practice. For instance, this can happen if the test data are collected at a later time or using a different measuring device. Without further assumptions on the test distribution, generalization under distribution shift is impossible. However, practitioners often have partial information about the set of possible "shifts" that may occur during test time, inducing a set of feasible test distributions that the model should generalize to. We refer to the resulting set as the robustness set. With R(β; P) denoting the population risk of a model parameterized by β for distribution P, the robust risk can be written asEQUATIONwhere P rob (θ ⋆ ) corresponds to the robustness set that we assume to be fully characterized by some true parameter θ ⋆ . In safety-critical applications, the goal is often to find a minimizer of the robust Framework accounts for bounded shifts partial identifiability of causal parameters partial identifiability of robustness set DRO [7, 16, 53, 35, 47] ✓ -✗Infinite robustness [38, 18, 33, 42, 6, 2, 51, 57, 29, 1] ✗ ✗ ✗Finite robustness [45, 26, 14, 28, 49] ✓ ✓ ✗Partially id. robustness (this work) ✓ ✓ ✓ risk, i.e. a robust prediction model that shows the best performance on the worst-case distribution out of the robustness set.A number of subfields in machine learning and optimization have addressed this problem. For example, in distributionally robust optimization (DRO) [7, 16] , the parameter θ ⋆ may be the training distribution P and the robustness set the neighborhood of P in some probability distance metric [30, 35, 22, 15] . Relatedly, adversarial robustness [23, 32] studies the risk on worst-case transformations of examples drawn from some distribution P and can be seen as equivalent to distribution shift robustness [53] . DRO-type methods minimize the worst-case robustness against arbitrary distribution shifts in the neighborhood without structural assumptions. Although being assumption-agnostic can be viewed as a strength, it also has its caveat: even when available, prior knowledge about the structure of expected test shifts cannot be incorporated. In such cases, the robust model's prediction might be overly conservative, resulting in suboptimal performance when the test shifts are in fact more benign. [47] .In many practical scenarios, data from heterogeneous sources is available at training time -for example, data from different geographic locations or time ranges. Due to the lack of modeling assumptions, multiple environments in the DRO setting cannot, in general, be leveraged to achieve better robustness in a given robustness set -in those contexts, the presence of multiple environments is usually argued to enable robustness against a larger robustness set. Instead, domain experts can anticipate which aspects of the joint probability distribution of (X, Y ) are more likely to shift. Such prior structural information can, for example, be incorporated through the framework of structural causal models (SCMs), via the approach of causality-oriented robustness [34, 11] . Importantly, the literature in this area has so far focused on settings when the desired robust objective R rob is identifiable, i.e. computable from training data. Traditional causal learning and invariancebased methods aim to fully identify some underlying causal parameter of the SCM for robustness against all (potentially infinite) interventions [38, 42, 6, 29] . However, the training data is often not heterogeneous enough to fully identify the causal parameter. Thus, another line of work [45, 50, 28] focuses on the scenario when the causal parameter is not necessarily identifiable, but the test shifts only occur in training directions, rendering the robust risk (1) identifiable. We provide an overview in Table 1 and an additional discussion of related work in Appendix A.In practice, causality-and invariance-based methods often result in wrong representations of the data [27, 44] and end up performing similarly to empirical risk minimization (ERM) that ignores the multi-environment information [3, 24, 43] . Many possible explanations for this observation have been proposed in the literature. In our work, we focus on the non-identifiability failure scenario. In particular, we extend the discussion of invariance-based methods to include the partially identifiable setting, where not only the causal parameter, but the robust risk (1) is not determinable using training data either 1 . Specifically, we aim to discuss the following question:What is the optimal worst-case performance any model can have for given structural relationships between test and training data and how do existing methods comparatively perform in such settings?When the robust risk is not identifiable from training data, we obtain a whole set of possible objectives that includes the true robust risk. In this case, we are interested in the best achievable robustness for any algorithm that we capture in a quantity called the identifiable robust risk: (2)Note that R rob,ID (β) is well-defined even when the standard robust risk is not identifiable -it takes the supremum over the robust risks induced by all possible true model parameters θ ⋆ that are consistent with the given set of training data distributions. Furthermore, the minimal value of the identifiable robust risk corresponds to the optimal worst-case performance in the partially identifiable setting. Spiritually, this minimax population quantity is reminiscent of the algorithm-independent limits in classical statistical learning theory [58] . 2 Even though our partial identifiability framework can be evaluated for arbitrary modeling assumptions on the distribution shift (such as covariate/label shift, DRO, etc.), we present it in a concrete setting for clarity of the exposition. Specifically, we discuss linear structural causal models (SCMs) with unobserved confounding (cf. Section 2), similar to the setting of IV (instrumental variables) and anchor regression [45, 46] .The identifiable robust risk (2) not only represents a notion of algorithm-independent optimality for any combination of training and test shifts. In the linear SCM setting in Section 2, we also show theoretically and empirically that the ranking and optimality of different robustness methods change drastically in identifiable vs. partially identifiable settings. The same can be observed in experiments on real-world data. Our experimental results strongly indicate that evaluation and benchmarking on partially identifiable settings are important for determining the effectiveness of robustness methods. Finally, while the identifiable robust predictor is only provably optimal for the linear SCM, experiments on real-world data in Section 3.3 suggest that our estimator may significantly improve upon other invariance-based methods in more realistic scenarios. 
setting: 
 In this section, we first introduce the linear causal model setting and describe our structural assumptions on the training and test distributions. Then, we introduce our framework for distributional robustness that allows for partial identifiability and define the identifiable robust risk, the worst-case robust risk among the possible robust risks induced by the training distributions.Data distribution. We are given multiple training environments indexed by e ∈ E train , where E train is a countable environment index set. For each training environment e, we observe data (X e , Y e ) ∼ P X,Y e consisting of input covariates X e ∈ R d and the target variable Y e ∈ R, which are generated by the linear structural causal model (SCM) (3) and its corresponding causal graph, depicted in Figure 1 . Throughout the paper, we assume that we observe the collection of training distributions P θ ⋆ ,Etrain = {P X,Y θ ⋆ ,e } e∈Etrain , omitting θ ⋆ when it is clear from the context. We discuss the corresponding finite sample setting in Appendix D.We assume that the true unobserved parameters θ ⋆ := (β ⋆ , Σ ⋆ ) ∈ Θ, where Θ ⊂ R d+(d+1)(d+1) , are invariant across environments. The joint covariance Σ ⋆ of the noise (η, ξ) can be written in blockform as Σ ⋆ = Σ ⋆ η Σ ⋆ η,ξ Σ ⋆ η,ξ ⊤ (σ ⋆ ξ )2 . We allow the presence of latent confounders between X e and Y e , and hence the case where Σ ⋆ η,ξ ̸ = 0. Note that the confounded noise setting is, in general, more challenging than the independent noise setting, since, given any number of environments, common estimators such as the linear regression estimator are biased away from the causal parameter β ⋆ . general, the environment shifts A e can be degenerate, i.e. the covariances Σ e are not assumed to be full rank. For simplicity of presentation, we further assume that E train contains a reference environment e = 0 satisfying µ 0 = 0 and Σ 0 = 0. In Appendix B, we discuss how our results apply if this condition is not met. Our additive shift structure implies that the joint distribution P X,Y,A e = P A e × P X,Y |A changes in each environment. However, we do not allow for direct interventions on Y or the latent confounders, that is P X,Y |A remains invariant. Note that our distribution shift setting is more general than covariate shift: due to unobserved confounding, the conditional distribution P ∼ N (0, M Σ A M ⊤ ), where M ∈ R d×q , Σ A ∈ R q×q ,yields the setup of continuous anchor regression [45] 3 . Discrete anchor regression [45] corresponds to a discrete environment index set E train = [m], m ∈ N, and deterministic mean shifts A e = µ e ∈ R d . The more general additive shift setting in [49] corresponds to E train = [m] and A e ∼ N (µ e , Σ e ). Note that in the above works, the environment index is modeled as a random variable E ∼ P E through which one assigns weights to different training environments. The results in this paper only depend on the support of P E , that is, whether an environment was seen or not. Thus, the population-level guarantees -the focus of this paper -are the same for any distributions with the same support on P E . EQUATIONwhere M test ⪰ 0. If the test distribution of X is given (as in the domain adaptation setting), one can directly estimate the shift of the test environment and setM test := E [A test A test ⊤ ] 4 .In the following, we consider the distributional robustness setting in which partial knowledge about test shifts is given in form of their maximum strength γ and general direction M ⊆ R d . We can then formalize this partial knowledge by setting M test = γΠ M , where γ > 0 and Π M is an orthogonal projection onto the subspace M. 5Given the test shift directions M and strength γ, our goal is to find an estimator using the training data that has a small risk over the entire set of shifted test distributions, called the robustness set, that we define asEQUATIONFor a given robustness set P θ ⋆ (γΠ M ), we define the robust riskEQUATIONand the corresponding robust predictor β rob := arg min β∈R d R rob (β; θ ⋆ , γΠ M ), where R(β; P) :=E P [(Y -β ⊤ X) 2 ]denotes the risk w.r.t. to the squared loss. The robust risk and the robust predictor can be explicitly computed as a function of the true model parameters θ ⋆ = (β ⋆ , Σ ⋆ ) and the test shift bound γΠ M :EQUATIONIn practice, neither the model parameters θ ⋆ nor the robust risk R rob itself can generally be fully identified. Instead, in the next sections we show that the robust prediction model can usually only be set-identified, except for specific combinations of the training and test shifts.In this section, we formally introduce the identifiable robust risk and related notions that allow us to characterize model robustness in the case when the robust predictor cannot be identified. We start with the notion of observational equivalence [17] : Definition 1 (Observational equivalence). We call model parameters θ 1 = (β 1 , Σ 1 ) and θ 2 = (β 2 , Σ 2 ) observationally equivalent with respect to a set of shift distributions {P A e : e ∈ E train } 6 if they induce the same set P θ,Etrain of training distributions over the observed variables (X e , Y e ) as described in Section 2.1, i.e.For all e ∈ E train : P X,Y θ1,e = P X,Y θ2,e . By observing P θ ⋆ ,Etrain , we can identify the model parameters up to the observationally equivalent set defined as Θ eq := {θ = (β, Σ) ∈ Θ : P θ,Etrain = P θ ⋆ ,Etrain }.When the observationally equivalent set is not a singleton, prior work only considers scenarios where the robustness set (5) and hence also the robust prediction model are still identifiable (see Equation (7) ). This scenario is shown in Figure 2a and discussed again in Section 3.2. In general, however, the observation of multiple training environments P θ ⋆ ,Etrain neither identifies the model parameters nor the robustness set or robust risk, which is the partially identified setting that we focus on. Instead, we can only compute a superset of the robustness setP Θeq (γΠ M ) := θ∈Θeq P θ (γΠ M ) ⊃ P θ ⋆ (γΠ M )and correspondingly, a set of robust risks {R rob (β; θ, γΠ M ) : θ ∈ Θ eq } and robust predictors B rob Θeq := {β rob θ : θ ∈ Θ eq }. In this case, we would still like to achieve the "best-possible" robustness, which is intuitively test shift robustness for the "hardest-possible" parameters that could have induced the observed training distributions. Definition 2 (Identifiable robust risk and the minimax quantity). Consider the data model (3). The identifiable robust risk is defined asEQUATIONWe will denote its minimizer as β rob,ID and refer to it as the identifiable robust predictor. The optimal robustness on test shifts bounded by γΠ M given training data P θ ⋆ ,Etrain is described by the minimax quantity EQUATIONv L p v d f / / n A X d q W y 4 E 7 x p j / O 1 4 = " > A A A C Q X i c d Z B N i x N B E I Z 7 d v 1 Y 4 1 d 2 P X p p D E I W Z O j J J u M e F 7 1 4 E a K Y 3 U A m h J p O J W m 2 e 2 b o r p E N w / w 1 L / 4 D b 9 6 9 e N h F v H q x k 4 1 B R Q s a X t 6 n i q p + 0 0 I r R 0 J 8 D n Z 2 b 9 y 8 d X v v T u P u v f s P H j b 3 D 0 5 d X l q J A 5 n r 3 A 5 T c K h V h g N S p H F Y W A S T a j x L z 1 + u + N l 7 t E 7 l 2 T t a F j g 2 M M / U T E k g b 0 2 a w 8 Q A L S T o 6 m 0 9 q R L C C 6 p s n t Y 1 b 2 9 J f 0 0 W S F C 3 k z k Y A 8 / 4 F r 7 e j s 0 x q + t D j 1 L f e T h p t k Q o 4 r g T R 1 y E R 8 e 9 X l d 4 0 Y t F J + r y K B T r a r F N 9 S f N T 8 k 0 l 6 X B j K Q G 5 0 a R K G h c g S U l N d a N p H R Y g D y H O Y 6 8 z M C g G 1 f r B G r + 1 D t T P s u t f x n x t f v 7 R A X G u a V J f e f q c P c 3 W 5 n / Y q O S Z s f j S m V F S Z j J 6 0 W z U n P K + S p O P l U W J e m l F y C t 8 r d y u Q A L k n z o D R / C r 5 / y / 4 v T T h j F Y e 9 N t 3 X y Y h P H H n v M n r A 2 i 9 h z d s J e s T 4 b M M k + s C / s k l 0 F H 4 O v w b f g + 3 X r T r C Z e c T + q O D H T 7 U o s r U = < / l a t e x i t > R rob (P ✓ ( , M gen ), )< l a t e x i t s h a 1 _ b a s e 6 4 = " 3L D 5 O i 4 m g C Q p Y 2 X i U V 0 2 g z G M 2 Q Q = " > A A A C Q n i c d Z B N i x N B E I Z 7 1 q 8 1 f k U 9 e m k M Q h Z k 6 I n J u M d F L 1 6 E K G Z 3 J R N C T a e S N N v d M 3 T X i G G Y 3 + b F X + D N H + D F g y J e P d j J x q C i B Q 0 v 7 1 N F V b 9 5 q Z U n I T 5 G e x c u X r p 8 Z f 9 q 6 9 r 1 G z d v t W / f O f Z F 5 S S O Z K E L d 5 q D R6 0 s j k i R x t P S I Z h c 4 0 l + 9 n T N T 9 6 g 8 6 q w r 2 h V 4 s T A w q q 5 k kD B m r Z f Z w Z o K U H X L 5 t p n R G + p d o V e d P w 7 o 4 M N 2 S J B E 0 3 W 4 A x 8 J D v 4 P P d m E e 0 T X M Q W B 5 a D 6 b t j o h F m v b S h I v 4 0 e F g 0 B d B D F L R S / o 8 i c W m O m x b w 2 n 7 Q z Y r Z G X Q k t T g / T g R J U 1 q c K S k x q a V V R 5 L k G e w w H G Q F g z 6 S b 2 J o O E P g j P j 8 8 K F Z 4 l v 3 N 8 n a j D e r 0 w e O t e X + 7 / Z 2 v w X G 1 c 0 P 5 z U y p Y V o Z X n i + a V 5 l T w d Z 5 8 p h x K 0 q s g Q D o V b u V y C Q 4 k h d R b I Y R f P + X / F 8 e 9 O E n j w Y t + 5 + j J N o 5 9 d o / d Z 1 2 W s M f s i D 1 j Q z Z i k r 1 j n 9 g X 9 j V 6 H 3 2 O v k X f z 1 v 3 o u 3 M X f Z H R T 9 + A q n i s z A = < / l a t e x i t > R rob (P ✓ ( , M seen ), ) 𝜃 ⋆ 𝛩 !" < l a t e x i t s h a 1 _ b a s e 6 4 = " X G m 7 o I r s X r k y p X e X n Z 4 v d A o d M t I = " > A A A C N 3 i c d V B N i x N B E O 1 Z P 3 a N X 1 G P X h q D s I I M P T G Z X d j L o h c v S h S z u 5 A J o a Z T S Z r t 7 h m 6 a 2 T D M P / K i 3 / D m 1 4 8 K O L V f 2 A n m x U V L W j 6 8 V 4 9 q u r l p V a e h P g Y b V 2 6 f O X q 9 s 6 1 1 v U b N 2 / d b t + 5 e + S L y k k c y k I X 7 i Q H j 1 p Z H J I i j S e l Q z C 5 x u P 8 9 N l K P 3 6 L z q v C v q F l i W M D c 6 t m S g I F a t J + m R m g h Q R d v 2 4 m d U Z 4 R r U r 8 q b h u 1 m O B A c 8 o 0 X 4 H 2 d z M A Z 4 N l C h 7 c L z 4 p d n j r Z p m k e T d k f E I k 2 7 a c J F / G S / 3 + + J A P q p 6 C Y 9 n s R i X R 2 2 q c G k / S G b F r I y a E l q 8 H 6 U i J L G N T h S U m P T y i q P J c h T m O M o Q A s G / b h e 3 9 3 w h 4 G Z 8 l n h w r P E 1 + z v j h q M 9 0 u T h 8 7 V x v 5 v b U X + S x t V N N s f 1 8 q W F a G V 5 4 N m l e Z U 8 F W I f K o c S t L L A E A 6 F X b l c g E O J I W o W y G E i 0 v 5 / 8 FR N 0 7 S u P + q 1 z l 8 u o l j h 9 1 n D 9 g u S 9 g e O 2 T P 2 Y A N m W T v 2 C f 2 h X 2 N 3 k e f o 2 / R 9 / P W r W j j u c f + q O j H T 0 p 5 r q Y = < / l a t e x i t > R rob ( ; ✓, ⇧ Mgen )< l a t e x i t s h a 1 _ b a s e 6 4 = " C 5 L m C G r m m u t F N R p K N P 4 O E D v u l H M = " > A A A C O H i c d V B N i x N B E O 1 Z P 3 a N X 1 G P X h q D s I I M P T G Z X d j L o h c v Y h S z u 5 A J o a Z T S Z r t 7 h m 6 a 2 T D M D / L i z / D m 3 j x o I h X f 4 G d b F Z U t K D p x 3 v 1 q K q X l 1 p 5 E u J j t H X p 8 p W r 2 z v X W t d v 3 L x 1 u 3 3 n 7 p E v K i d x K A t d u J M c P G p l c U i K N J 6 U D s H k G o / z 0 2 c r / f g t O q 8 K + 4 a W J Y 4 N z K 2 a K Q k U q E n 7 Z W a A F h J 0 / b q Z 1 B n h G d W u y J u G 7 2 Y 5 E h z w j B b h f 5 z N w R j g 2 U C F t g v P i 1 8 e j 2 i b p n k 0 a X d E L N K 0 m y Z c x E / 2 + / 2 e C K C f i m 7 S 4 0 k s 1 t V h m x p M 2 h + y a S E r g 5 a k B u 9 H i S h p X I M j J T U 2 r a z y W I I 8 h T m O A r R g 0 I / r 9 e E N f x i Y K Z 8 V L j x L f M 3 + 7 q j B e L 8 0 e e h c r e z / 1 l b k v 7 R R R b P 9 c a 1 s W R F a e T 5 o V m l O B V + l y K f K o S S 9 D A C k U 2 F X L h f g Q F L I u h V C u L i U / xv L p v d f / / n A X d q W y E x p j / O = " > A A A C Q X i c d Z B N i x N B E I Z d v Y d P X p p D E I W Z O j J J u M e F E a K Y U A m h J p O J W m e b o r p E N w / w L / D b e N h F v H q x k B R Q s a X t n i q p + I r R J D n Z b y d X v v T u P u v f s P H j b D d X l q J A n r A T c K h V h g N S p H F Y W A S T a j x L z + u + N l t E l T t a F j g M M / U T E k g b a w Q A L S T o m q R L C C p s n t Y b J f W S F C k z k Y A / F r e j s x q + t D j L f e T h p t k Q o r g T R y E R e X l d Y t F J + r y K B T r a r F N S f N T k l X B j K Q G a R K G h c g S U l N d a N p H R Y g D y H O Y z M C g G f r B G r + D t T P s u t f x n x t f v R A X G u a V J f e f q c P c W n / Y q O S Z s f j S m V F S Z j J W z U n P K + S p O P l U W J e m l F y C t r d y u Q A L k n z o D R / C r / y / v T T h j F Y e N t X y Y h P H H n v M n r A i h z d s J e s T b M M k + s C / s k l F H O v w b f g + X r T r C Z e c T + q O D H T U o s r U = < / l a t e x i t > R rob (P ✓ ( , M gen ), )< l a t e x i t s h a 1 _ b a s e 6 4 = " 3L D 5 O i 4 m g C Q p Y 2 X i U V 0 2 g z G M 2 Q Q = " > A A A C Q n i c d Z B N i x N B E I Z 7 1 q 8 1 f k U 9 e m k M Q h Z k 6 I n J u M d F L 1 6 E K G Z 3 J R N C T a e S N N v d M 3 T X i G G Y 3 + b F X + D N H + D F g y J e P d j J x q C i B Q 0 v 7 1 N F V b 9 5 q Z U n I T 5 G e x c u X r p 8 Z f 9 q 6 9 r 1 G z d v t W / f O f Z F 5 S S O Z K E L d 5 q D R6 0 s j k i R x t P S I Z h c 4 0 l + 9 n T N T 9 6 g 8 6 q w r 2 h V 4 s T A w q q 5 k k The definition of the identifiable robust risk reflects the absence of knowledge of the model parameters in test shift directions that were not observed during training. In words, the most robust parameter choice β rob,ID is the one that minimizes the worst-case robust risk for parameters in the observationally equivalent set . In the next sections, we compute these quantities explicitly for the setting of Section 2. This will allow us to compare the best achievable robustness in the partially identified case with the guarantees of prior methods in this setting.D B m r Z f Z w Z o K U H X L 5 t p n R G + p d o V e d P w 7 o 4 M N 2 S J B E 0 3 W 4 A x 8 J D v 4 P P d m E e 0 T X M Q W B 5 a D 6 b t j o h F m v b S h I v 4 0 e F g 0 B d B D F L R S / o 8 i c W m O m x b w 2 n 7 Q z Y r Z G X Q k t T g / T g R J U 1 q c K S k x q a V V R 5 L k G e w w H G Q F g z 6 S b 2 J o O E P g j P j 8 8 K F Z 4 l v 3 N 8 n a j D e r 0 w e O t e X + 7 / Z 2 v w X G 1 c 0 P 5 z U y p Y V o Z X n i + a V 5 l T w d Z 5 8 p h x K 0 q s g Q D o V b u V y C Q 4 k h d R b I Y R f P + X / F 8 e 9 O E n j w Y t + 5 + j J N o 5 9 d o / d Z 1 2 W s M f s i D 1 j Q z Z i k r 1 j n 9 g X 9 j V 6 H 3 2 O v k X f z 1 v 3 o u 3 M X f Z H R T 9 + A q n i s z A = < / l a t e x i t > R rob (P ✓ ( , M seen ), ) 𝜃 ⋆ 𝛩 !" < l a t e x i t s h a _ b a s e = " X G m o I r s X r k y p X e X n Z v d A o d M t I = " > A A A C N i c d V B N i x N B E O Z P a N X G P X h q D s I I M P T G Z X d j L o h c v S h S z u A J o a Z T S Z r t h m a T D M P / K i / D m K O L V f A n m x U V L W j V q u r l p V a e h P g Y b V f O X q s v U b N / d b t + e + S L y k k c y k I X i Q H j p Z H J I i j S e l Q z C x u P N l K P L z q v C v q F l i W M D c t m S g I F a t J + m R m g h Q R d v m d U Z R r U r q b h u m O B A c o X H d z M A Z N l C h c L z p d n j r Z p m k e T d k f E I k a c J F / G S / + + J A P q p C Y n s R i X R q c G k / S G b F r I y a E l q H U i J L G N T h S U m P T y i q P J c h T m O M o Q A s G / b h e w h G Z l n h w r P E + z v j h q M u T h V x v v b U X + S x t V N N s f q W F a G V N m l e Z U F W I f K o c S t L L A E A F X b l c g E O J I W o W y G E i v / F R N S u P + q z l u o l j h n D g u S g e O T P Y A N m W T v C f h X N k e f o / R / P W r W j j u c f + q O j H T p r q Y = < / l a t e x i t > R rob ( ; ✓, ⇧ Mgen ) < l a t e x i t s h a 1 _ b a s e 6 4 = " C 5 L m C G r m m u t F N R p K N P 4 O E D v u l H M = " > A A A C O H i c d V B N i x N B E O 1 Z P 3 a N X 1 G P X h q D s I I M P T G Z X d j L o h c v Y h S z u 5 A J o a Z T S Z r t 7 h m 6 a 2 T D M D / L i z / D m 3 j x o I h X f 4 G d b F Z U t K D p x 3 v 1 q K q X l 1 p 5 E u J j t H X p 8 p W r 2 z v X W t d v 3 L x 1 u 3 3 n 7 p E v K i d x K A t d u J M c P G p l c U i K N J 6 U D s H k G o / z 0 2 c r / f g t O q 8 K + 4 a W J Y 4 N z K 2 a K Q k U q E n 7 Z W a A F h J 0 / b q Z 1 B n h G d W u y J u G 7 2 Y 5 E h z w j B b h f 5 z N w R j g 2 U C F t g v P i 1 8 e j 2 i b p n k 0 a X d E L N K 0 m y Z c x E / 2 + / 2 e C K C f i m 7 S 4 0 k s 1 t V h m x p M 2 h + y a S E r g 5 a k B u 9 H i S h p X I M j J T U 2 r a z y W I I 8 h T m O A r R g 0 I / r 9 e E N f x i Y K Z 8 V L j x L f M 3 + 7 q j B e L 8 0 e e h c r e z / 1 l b k v 7 R R R b P 9 c a 1 s W R F a e T 5 o V m l O B V + l y K f K o S S 9 D A C k U 2 F X L h f g Q F L I u h V C u L i U / x 8 c d e M k j f u v e p 3 D p 5 s 4 d t h 9 9 o D t s o T t s U P 2 n A 3 Y k E n 2 j n 1 i X 9 j X 6 H 3 0 O f o W f T 9 v 3 Y o 2 n n v s j 4 p + / A Q 6 T a 8 h < / l a t e x i t > R rob ( ; ✓, ⇧ Mseen ) 𝜃 𝜃 (b) Partially identifiable setting 
main results for partially identified robustness: 
 We now compute the identifiable robust risk ( 8) and derive a lower bound for the minimax quantity (9) in the additive shift setting. We then compare the identifiable robust risk of existing robustness methods and ordinary least squares (OLS) with the minimizer of the identifiable robust risk both theoretically and empirically.The degree to which the model parameters θ ⋆ in the linear SCM setting (3) can be identified depends on the number of environments and the total rank of the additive shifts. This is well-studied, for instance, in the instrumental variable (IV) regression literature [4, 9] . In particular, in the setting of Section 2.1, the causal parameter β ⋆ can only be identified along the mean and variance shifts of the covariates across the training data. Therefore, if not enough shift directions are observed, it is merely set-identifiable. In the following, we show how set-identifiability of the model parameters translates into set-identifiability of the robust prediction model (7) . More formally, we denote by S the subspace consisting of all additive shift directions seen during training:EQUATION)The definition of the space S induces the following orthogonal decompositions of the causal parameter and test shift directions M:EQUATIONEQUATIONwhere S and R are matrices with orthonormal columns such that range S ⊂ S, range R ⊂ S ⊥ and range S, range R are the smallest subspaces satisfying Equation (11) 7 . The matrix S corresponds to test shift directions along the model can be identified. Conversely, R corresponds to test shift directions, along which the model is non-identified. The vector β S is the identifiable part of the causal parameter. It uniquely defines a set of identified model parameters that readsθ S := (β S , Σ S η , Σ S η,ξ , (σ S ξ ) 2 ) = (β S , Σ ⋆ η , Σ ⋆ η,ξ + Σ ⋆ η β S ⊥ , (σ ⋆ ξ ) 2 + 2⟨Σ ⋆ η,ξ , β S ⊥ ⟩ + ⟨β S ⊥ , Σ ⋆ η β S ⊥ ⟩)and can be computed from the training distributions. In the next proposition, we show that the model parameters and robust predictor can be identified up to a set around θ S , which can be interpreted as the set's geometric center. From the characterization of this set, it directly follows that the robust predictor is only identifiable if the test shifts are in the direction of the training shifts, i.e. M ⊂ S. Proposition 1 (Identifiability of model parameters and robust predictor). Suppose that the set of training and test distributions is generated according to Section 2.1, with some model parameter θ ∈ Θ. Then, it holds that (a) the model parameters generating the training distribution (3) can be identified up to the following observationally equivalent set :EQUATION)(b) the robust predictor β rob as defined in Equation ( 7) is identified up to the setEQUATION)The proof of Proposition 1 is provided in Appendix F.1. Proposition 1 implies two well-known settings: If we observe a rich enough set P Etrain of training environments such that S = R d , the model parameters are uniquely identified, corresponding to the setting of full-rank instruments [4] . However, even in the under-identified case S ̸ = R d , if the test shift directions M are contained in the space S of training-time shifts, i.e. R = 0, the robust prediction model is identifiable from training data regardless of the identifiability of the model parameters. This is the setting considered e.g. in anchor regression [45] and discussed again in Section 3.2 and Appendix C.So far, we have described how the identifiability of the robust prediction model depends on the structure of both the training environments (via the space S) and the test environments (via M). We now aim to compute the smallest achievable robust loss for the general partially identifiable setting, which allows for R ̸ = 0. In particular, we provide a lower bound on the best-possible achievable distributional robustness formalized by the minimax quantity (9) . First observe that without further assumptions on the parameter space Θ, the observationally equivalent set is unbounded, and the identifiable robust risk (8) can be infinite. The following assumption allows us to provide a finegrained analysis of robustness in a partially identified setting. Assumption 3.1 (Boundedness of the causal parameter). There exists a constant C > 0 such that any causal parameter β generating the SCM (3) is norm-bounded by C, i.e. ∥β∥ 2 ≤ C and henceΘ = B d (C) × R (d+1)×(d+1) .Furthermore, two key quantities that appear in the bounds are S tot = R d -range R, the space of directions which are either identified or unperturbed during test time, and C ker = C 2 -∥β S ∥ 2 , the maximum norm of the non-identified part of the causal parameter β ⋆ . Finally, recall that the reference distribution P X,Y θ ⋆ ,0 is observed and hence identifiable. Theorem 3.1. Assume that the training and test data follow the data-generating mechanism in Section 2.1 with test time shifts decomposed as in Equation (11) for some semi-orthogonal matrices S, R with range S ⊂ S, range R ⊂ S ⊥ . Further, let Assumption 3.1 hold with parameter C. The identifiable robust risk (8) is then given byEQUATION) Further, we obtain the following lower bound for the minimax quantity as defined in Equation (9):M(Θ eq , γΠ M ) = γI R̸ =0 C 2 ker + min R ⊤ β=0 R rob (β; θ S , γSS ⊤ ), if γ ≥ γ th ; ≥ γI R̸ =0 C 2 ker + min β∈R d R rob (β; θ S , γSS ⊤ ), else,whereEQUATION)We prove Theorem 3.1 in Appendix F.2. In the case of no new test shifts, i.e., R = 0, is discussed in prior work [45, 49] , as the strength γ of the shift grows, the identifiable robust risk saturates. On the other hand, if R ̸ = 0, i.e., the test shift contains new directions w.r.t. to the training data, the best achievable robustness M(Θ eq , γΠ M ) grows linearly with γ, and thus no infinite robustness is possible. We highlight that the minimax quantity is attained by the identifiable robust predictorβ rob,ID = arg min β∈R d R rob,ID (β; Θ eq , γΠ M ),and for γ ≥ γ th , the lower bound corresponds to its identifiable robust risk and thus is tight. Moreover, γ ≥ γ th , β rob,ID can be explicitly computed from the training distributions (cf. Appendix F.2) and is orthogonal to the space range R of non-identifiable test shift directions. In other words, for large shifts γ in non-identified directions, the optimal robust model would "abstain" from prediction in those directions. For smaller γ, β rob,ID gradually utilizes less information in the non-identified directions, thus interpolating between maximum predictive power (OLS) and robustness w.r.t. new directions (abstaining). Note that the model β rob,ID is a population quantity that is identifiable from the collection of training distributions. When only finite samples are available, we discuss in Appendix D how we can still estimate the minimax quantity by minimizing an empirical loss function (17) that can be computed from multi-environment data. Additionally, in Appendix D, we provide details on the computation of the empirical identifiable robust risk and the corresponding estimator.We now evaluate existing finite robustness methods in our partial identifiability framework and discuss in which scenarios they are far from the best achievable robustness. A spiritually similar systematic comparison of domain adaptation methods is presented in [12] , however, in our setting, the robust risk is not identifiable from data. We impose a probability distribution on the environment variable E ∈ E train s.t. P[E = e] = w e , which allows us to compare to the anchor regression framework and similar, where the environment weights are required to obtain an estimate of M test .In our discussion, we focus on discrete anchor regression [45] and pooled OLS estimators 8 . In discrete anchor regression, for each environment e, we observe data (X e , Y e ) following the SCM X e = µ e + η; Y e = β ⋆⊤ X e + ξ, where µ e ∈ R d are mean shifts and the noise is distributed like in Equation ( 3). The discrete anchor regression estimator minimizes the following robust risk:β anchor = arg min β∈R d R rob (β; θ ⋆ , γM anchor ),where M anchor = e∈Etrain w e µ e µ ⊤ e . The pooled ordinary least squares (OLS) estimator β OLS corresponds to β anchor with γ = 1. We observe that the test shifts bounded by γM anchor are fully contained in the space of identified directions S, since S = range ∪ e∈Etrain µ e µ ⊤ e = range M anchor . Thus, according to Proposition 1, the robust risk and robust predictor β anchor are identifiable for all γ > 0. We now evaluate the robustness performance of β anchor and β OLS with respect to the more general shifts bounded by M new := γM anchor + γ ′ RR ⊤ , thus consisting of training-identified shifts M anchor and a possibly smaller share of previously unseen shifts in range R ⊂ S ⊥ . With respect to M new , the robust risk is only partially identified, and identifiable robust risk (8) given byR rob,ID (β; Θ eq , M new ) = γ ′ (C ker + ∥R ⊤ β∥ 2 ) 2 + R rob (β; θ ⋆ , γM anchor ).We evaluate how the identifiable robust risks (14) of both previous methods depend on the strength γ ′ of the previously unseen shift:R rob,ID (β anchor ; Θ eq , M new )/γ ′ = (C ker + ∥RR ⊤ (Σ ⋆ η + γM anchor ) -1 Σ S η,ξ ∥) 2 + o(γ ′ ); R rob,ID (β OLS ; Θ eq , M new )/γ ′ = (C ker + ∥RR ⊤ (Σ ⋆ η + M anchor ) -1 Σ S η,ξ ∥) 2 + o(γ ′ ).In contrast, for the best achievable robustness in the anchor setting 9 it holds Thus, the anchor regression estimator is optimal in the limit of small unseen shifts but significantly deviates from the best achievable robustness for smaller shifts. This is due to the fact that the term ∥RR ⊤ (Σ ⋆ η + γM anchor ) -1 Σ S η,ξ ∥ only goes to zero as γ → ∞ (yielding the minimax risk) if M anchor is full-rank, otherwise, it is strictly bounded from below as Σ ⋆ η is full-rank. Moreover, under some conditions on the covariance matrix (e.g., if Σ ⋆ η is block-diagonalizable w.r.t. S and S ⊥ ), pooled OLS and the anchor estimator achieve the same rate in γ ′ , showcasing how finite robustness methods can perform similarly to empirical risk minimization if the assumptions on the robustness set are not met. We provide additional comparisons in Appendix C.M(Θ eq , M new ) γ ′ = C 2 ker + o(γ ′ ), if γ ′ ≥ γ th ; lim γ ′ →0 M(Θ eq , M new ) γ ′ = (C ker + ∥RR ⊤ (Σ ⋆ η + γM anchor ) -1 Σ S η,ξ ∥) 2 .In this section, we provide empirical evidence of our theoretical conclusions in Section 3.1 and Section 3.2. In particular, we compare the prediction performance of multiple existing robustness methods to the minimax lower bound, estimated by the identifiable robust predictor -including partially idenfitiable settings in which the test data contains shifts in previously unseen directions. We observe that both in a synthetic adversarial setting, empirical risk minimization and invariancebased robustness methods have significantly sub-optimal test loss in the partially identified setting, confirming our theoretical predictions in Section 3.2.Furthermore, we observe that even though the minimizer of identifiable robust risk is optimal only for the linear causal setting in Section 2.1, it surprisingly outperforms existing methods in a real-world experiment.Experiments on synthetic Gaussian data We simulate Gaussian covariates according to Equation (3) with multiple environments differing by linearly independent randomly selected mean shifts. Given a fixed confounding model represented by a noise covariance Σ and fixed directions S of training mean shifts, we "evaluate" the identifiable robust risk by first picking the most adversarial β ⋆ for fixed Σ and S, and then computing its robust risk (6) . We describe the full details of the data generation and loss evaluation in Appendix E. We consider two shift scenarios: in the first one, corresponding to the identifiable case in Figure 2a , the test environment is only perturbed by bounded shifts in training directions, as considered in prior work [45, 49] . In the second scenario, corresponding to the non-identifiable case Figure 2b , the test environment is perturbed by a mixture of training shifts and shifts in previously unobserved directions. We compute the empirical minimizers βOLS , βanchor and βrob,ID of the OLS, anchor regression and identifiable robust losses, respectively, and compare their test MSE (mean squared error) in Figure 3 . In the first (identifiable) setting -Figure 3 (left) -the robust risk is asymptotically constant across γ for both robust methods, while the error for the vanilla ERM or OLS estimator increases linearly. In contrast, in the second, partially identified, setting -Figure 3 (right) -all estimators exhibit linearly increasing test errors; however the slopes of the anchor and OLS estimator are much steeper and lead to larger errors than the empirical minimizer of ( 14) that closely matches the analytic theoretical lower bound.We consider the single-cell gene expression dataset from [41] , which consists of single-cell observations over d = 622 genes collected from both observational and several interventional environments. Following [48] , we only select the 28 genes that are active in the observational environment. For each gene j = 1, . . . , 28 is the target variable and the covariates are the three genes most strongly correlated with Y . For each D j , we perform three experiments -every experiment uses a different interventional environment besides the observational data as training data (an illustration of the data structure can be found in Figure 5 ). We then separately compute the mean-squared error on subsets of samples from all three interventional environments (including held-out samples from the interventional environment used for training). As a proxy for shift strength γ > 0, for each test environment, we pick the s × 100% of data points closest to the observational mean. More details on this process can be found in Appendix E. We describe the computation of the identifiable robust estimator in Appendix D. In Figure 4 , we show the test MSE of various OOD methods and the identifiable robust estimator as a function of s, presented in four different scenarios: no unseen shifts (left), some proportion of unseen shifts (middle panels) and 100% unseen shift directions (right). We compare the performance of anchor regression [45] , invariant causal prediction (ICP) [38] , Distributional Robustness via Invariant Gradients (DRIG) [49] , and OLS with our estimated lower bound Rob-ID. We observe that the performance ranking of the robustness methods significantly varies with the proportion of new test shift directions. When no new shift information is present, anchor regression and DRIG are optimal. However, as soon as some unseen directions are present, their performance becomes inferior to OLS/ERM and the gap to the minimizer of the identifiable robust risk (in the setting in Section 2) grows with the proportion of unseen shifts. While the MSE of the previous invariant methods increases drastically with the strength of the test shift, the test loss of the identifiable robust predictor remains relatively stable. 
conclusion and future directions: 
 This paper introduces the identifiable robust risk that is well-defined even in settings where the robust risk is not computable from training distributions. When the robustness set is identifiable (such as anchor regression-related methods [45, 49] ), the identifiable robust risk reduces to the conventional robust risk. In this paper, we instantiate our general framework for linear structural causal models with additive shifts. We compute tight lower bounds for this setting and show how existing invariancebased methods are suboptimal. Further, we demonstrate how i) the benefits of invariance-based methods strongly decrease in the partially identifiable setting; and ii) this suboptimality increases with perturbation strength and proportion of previously unobserved test shifts.The main limitation of our paper is its reliance on a linear causal setting to explicitly compute the observationally equivalent set and estimate the minimax quantity. However, we expect that the results and intuition developed in this paper can be extended to linear shifts in a lower-dimensional latent space via a suitable parametric or non-linear map [55, 10] . Important future directions include extending our results to more general causal graphs, non-linear relationships between covariates, non-additive shifts and the classification setting. Further, a potential use of our work is in the field of active intervention selection (e.g, [60, 21] ). By computing the most adversarial model parameter for a given estimator, e.g., OLS, we can obtain an intervention which minimizes the identifiable robust risk of the estimator on the next unseen shift. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Even if our work addresses the theoretical limits of distributional robustness, we mention in the abstract and in Section 1 that the topic of distributional robustness is central to safety-critical applications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The work develops a theoretical framework and considers synthetic experiments. Therefore, explicit safeguards do not seem applicable at this stage.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In the numerical experiment in Section 3.3, we cite the existing work that we compare to our framework and the dataset used. In running the numerical experiment, we reimplemented all the methods (including existing ones) for ease of comparison.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To put our work into context, first, we discuss relevant distributional robustness literature organized according to structural assumptions on the desired robustness set. Second, we summarize existing views on partial identifiability in the causality and econometrics literature and how our findings connect to their perspective.No structural assumptions on the shift. DRO: Distributionally robust optimization (DRO) tackles the problem of domain generalization when the robustness set is a ball around the training distribution w.r.t. some probability distance measure, e.g., Wasserstein distance [53, 35] or f -divergences [7, 16] . Considering all test distributions in a discrepancy ball can lead to overly conservative predictions, and therefore, alternatives have been proposed in, e.g., the Group DRO literature [47, 20, 31] . However, these methods cannot protect against perturbations larger than the ones seen during training time and do not provide a clear interpretation of the perturbations class [49] .Structural assumptions on the shift. Robustness from the lens of causality takes a step further, by assuming a structural causal model [37] generating the observed data (X, Y ). Infinite robustness methods: The motivation of causal methods for robustness is that the causal function is worst-case optimal to predict the response under interventions of arbitrary direction and strength on the covariates [34, 11] . For this reason, causal models achieve what we call infinite robustness. Depending on the assumptions of the SCM, there are different ways to achieve infinite robustness. When there are no latent confounders, several works [38, 18, 33, 42, 2, 6, 51, 57, 29, 1] aim to identify the causal parents and achieve infinite robustness by exploiting the heterogeneity across training environments.In the presence of latent confounders, it is possible to achieve infinite robustness by identifying the causal function with, e.g., the instrumental variable method [5, 25, 52, 8, 36] . There are different limitations to infinitely robust methods. First, the identifiability conditions of the causal parents and/or causal function are often challenging to verify in practice. Second, ERM can outperform these methods when the interventions (read shifts) at test time are not arbitrarily strong or act directly on the response or latent variable [3, 24] . Finite robustness methods: In real data, shifts of arbitrary direction and strength in the covariates are unrealistic. Thus, different methods [45, 26, 28, 49, 14] trade off robustness against predictive power to achieve what we call finite robustness. The main idea of finite robustness methods is to learn a function that is as predictive as possible while protecting against shifts up to some strength in the directions that are observed during training time. These methods, however, only provide robustness guarantees that depend on the heterogeneity of the training data and do not offer insights into the limits of algorithm-independent robustness under shifts in new directions.The problem of identification is at the center of the causal and econometric literature [39, 4] . It studies the conditions under which the (population) training distribution uniquely determines the causal parameters of the underlying SCM. Often, the training distribution only offers partial information about the causal parameters and, therefore, determines a set of observational equivalent parameters. This setting is known as partial or set identification and is used in causality and econometrics to learn intervals within which the true causal parameter lies [54] . In this work, we borrow the notion of partial identification to study the problem of distributional robustness when the robustness set itself is only partially identified.We discuss how our setting changes when we relax the assumptions on the existence of the reference environment. We consider the data-generating process in Equation (3) , whereE train = [m], m ∈ N.If no environment e exists with µ e = 0 and Σ e = 0, we first pick an arbitrary distribution P X,Y ref as the reference environment 10 . We denote Σ ′ η := Σ ⋆ η + Σ ref .First, we show we can express the space S of training additive shift directions defined in Equation (10) in the general case. We center all distributions by µ ref , so that E [X e ] = µ e -µ ref for all e ∈ E train .With respect to the arbitrary reference environment, we now defineS := range e∈Etrain Σ e -Σ ref + (µ e -µ ref )(µ e -µ ref ) ⊤ ⊂ R d .We now consider test shifts with respect to the environment P X,Y ref 11 . We define the test shift upper bound γΠ M . Again, we can decompose the upper bound as γΠ M = γSS ⊤ + γRR ⊤ , where SS ⊤ and RR ⊤ are orthogonal projections onto S ∩ M and S ⊥ ∩ M, respectively. Again, we can decompose the causal parameter β ⋆ as β ⋆ = β S + β S ⊥ . The projection β S of the causal parameter onto the relative training shifts induces the following observationally equivalent parameters corresponding to the reference distribution:θ S := (β S , Σ ′ η , Σ S η,ξ , (σ S ξ ) 2 ) = (β S , Σ ′ η , Σ ⋆ η,ξ + Σ ′ η β S ⊥ , (σ ⋆ ξ ) 2 + 2⟨Σ ⋆ η,ξ , β S ⊥ ⟩ + ⟨β S ⊥ , Σ ′ η β S ⊥ ⟩).Again, θ S can be identified from the training distributions and is referred to as the identified model parameters. The following adapted version of Proposition 1 shows that assuming shifts on P X,Y ref , the robust prediction model is only identifiable if the test shifts are in the direction of the relative training shifts: Proposition 2 (Identifiability of reference distribution parameters and robust prediction model). Suppose that the set of training and test distributions is generated according to Equations (3) and (4). Then, θ S is observationally equivalent to θ ⋆ and computable from training distributions. Furthermore, it holds that (a) the model parameters generating the reference distribution can be identified up to the following observationally equivalent set :Θ eq = {β S + α, Σ ′ η , Σ S η,ξ -Σ ′ η α, (σ S ξ ) 2 -2α ⊤ Σ S η,ξ + α ⊤ Σ ′ η α : α ∈ S ⊥ } ∋ θ ⋆ (b) the robust prediction model β rob as defined in Equation ( 7) is identified up to the setβ S + (γΠ M + Σ ′ η ) -1 Σ S η,ξ + {(γΠ M + Σ ′ η ) -1 α : α ∈ range R} ∋ β robThe proof is analogous to Appendix F.1. A version of Theorem 3.1 for perturbations on the reference environment follows accordingly.In the continuous anchor regression setting, during training we observe the distribution according to the SCM X = M A + η; Y = β ⋆⊤ X + ξ, where A ∼ N (0, Σ A ) is an observed q-dimensional anchor variable and M ∈ R d×q is a known matrix. Note that in this setting, we do not have a reference environment, but, since the anchor variable is observed, the distribution of the additive shift M A is known. The test shifts are assumed to be bounded by M test = γM Σ A M ⊤ . Since range M test ⊂ S = range M , no new directions are observed during test time, in other words, R = 0. Thus, both the corresponding robust loss and the anchor regression estimator can be determined from training data. It holds thatβ anchor = arg min β∈R d R rob (β; θ ⋆ , γM Σ A M ⊤ ).Again, the pooled OLS estimator corresponds to β anchor with γ = 1. Similar to the discrete anchor case, in case the test shifts are given by M new = γM Σ A M ⊤ + γ ′ RR ⊤ , the identifiable robust risk ( 8) is given byR rob,ID (β; Θ eq , M new ) = γ ′ (C ker + ∥R ⊤ β∥ 2 ) 2 + R rob (β; θ ⋆ , γM Σ A M ⊤ )and for the best achievable robustness of the anchor estimator it holdsR rob,ID (β anchor , Θ eq ; M new )/γ ′ = (C ker + ∥RR ⊤ (Σ ⋆ η + γM Σ A M ⊤ ) -1 Σ S η,ξ ∥) 2 + o(γ ′ ); lim γ ′ →0 R rob,ID (β anchor , Θ eq ; M new )/γ ′ = lim γ ′ →0 M(Θ eq , M new ) γ ′ .The above results follow by plugging M new with M := M anchor into the proof of Theorem 3.1 in Appendix F.2.C.2 Distributionally robust invariant gradients (DRIG) [49] DRIG [49] uses the framework of Gaussian additive shifts A e ∼ N (µ e , Σ e ). For each environment e, we observe data (X e , Y e ) distributed according to the SCM X e = A e + η; Y e = β ⋆⊤ X e + ξ, where the noise is distributed like in Equation ( 3). DRIG consider more a more general intervention setting, additionally allowing additive shifts of Y and hidden confounders H. However, their identifiability results can only be shown for the case of interventions on X, and since identifiability of the causal parameter is a crucial part of our analysis, we only consider shifts on the covariates. DRIG assumes existence of a reference environment e = 0 with µ 0 = 0 and for which it is required that the second moment of the reference environment is dominated by the second moment of the training mixture:Σ 0 ⪯ e∈[m]w e (Σ e + µ e µ ⊤ e ).This assumption allows [49] to derive the DRIG estimator which is robust against test shifts upper bounded by M DRIG := γ e∈[m] w e (Σ e -Σ 0 + µ e µ ⊤ e ). The following lemma allows us to make further statements about M DRIG : Lemma C.1. Let A and B be positive semidefinite matrices such that B ⪯ A. Then it holds that range B ⊂ range A. Hence, the robustness directions achievable by DRIG in the "dominated reference environment" setting are the same as the ones under the assumption Σ 0 = 0. Again, we observe that the test shifts bounded by γM DRIG are fully contained in the space of identified directions S. If the test shifts are instead bounded by M new := γM DRIG + γ ′ RR ⊤ , including some unseen directions range R ⊂ S ⊥ , the robust risk in the DRIG setting is only partially identified. The identifiable robust risk ( 8) is given byR rob,ID (β; Θ eq , M new ) = γ ′ (C ker + ∥R ⊤ β∥ 2 ) 2 + R rob (β; θ ⋆ , γM DRIG ),and again, the DRIG estimator is optimal for infinitesimal shifts γ ′ and suboptimal for larger γ ′ :R rob,ID (β DRIG ; Θ eq , M new )/γ ′ = (C ker + ∥RR ⊤ (Σ ⋆ η + γM DRIG ) -1 Σ S η,ξ ∥) 2 + o(γ ′ ); M(Θ eq , M new ) γ ′ = C 2 ker , if γ ′ ≥ γ th ; lim γ ′ →0 M(Θ eq , M new ) γ ′ = (C ker + ∥RR ⊤ (Σ ⋆ η + γM DRIG ) -1 Σ S η,ξ ∥) 2 .The above results follow by plugging M new with M := M DRIG into the proof of Theorem 3.1 in Appendix F.2.In this section, we discuss how to compute the identifiable robust loss and its minimizer from finitesample multi-environment training data. We first describe the finite-sample setting and provide a high-level algorithm. We then discuss some parts of the algorithm in more detail. Finally, we show that under certain assumptions, the empirical identifiable robust loss is consistent. 6: Return: identifiable robust predictor and the estimated minimax "hardness" of the problem:βrob,ID ← arg min β∈R d L n (β; βS , Ŝ Ŝ⊤ , R R⊤ ); M(D, γ, M ) ← min β∈R d L n (β; βS , Ŝ Ŝ⊤ , R R⊤ ).Training data. We observe data from m + 1 training environments indexed by E ∈ E train = {0, ..., m}, where E = 0 represents the reference environment. We impose a discrete probability distribution P E on the training environment E ∈ E train , resulting in the joint distribution (X, Y, E) ∼ P X,Y |E × P E . For each environment E = e, we observe the samples D e := {(X e,i , Y e,i )} ne i=1 , where (X e,i , Y e,i ) are independent copies of (X e , Y e ) ∼ P X,Y |E=e . Then, the resulting dataset is D := ∪ e∈Etrain D e with n := n 0 + • • • + n m . Furthermore, for each environment E = e, we define the weights w e := n e /n. Computation of the identifiable robust loss. In Algorithm 1, we present a high-level scheme for computing the identifiable robust loss from multi-environment data, which consists of multiple steps. First, nuisance parameters related to the training and test shift directions are estimated, which we describe in more detail below. Afterwards, the three terms of the loss are computed: the (squared) loss L ref (β; D 0 ) on the reference environment is computed asL ref (β; D 0 ) = n0 i=1 (Y 0,i -β ⊤ X 0,i ) 2 .The invariance penalty term L inv (β; βS , Ŝ Ŝ⊤ , γ) (which increasingly aligns any estimator β in the direction of the estimated invariant causal predictor βS as γ → ∞) can be computed as following in the linear SCM setting:L inv (β; βS , Ŝ Ŝ⊤ , γ) = γ∥ Ŝ Ŝ⊤ (β -β S )∥ 2 2 .Finally, the non-identifiability penalty term L id (β; Ĉker , R R⊤ , γ) can be computed as follows:L id (β; Ĉker , R R⊤ , γ) ← γ(C ker + ∥ R R⊤ β∥ 2 ) 2 .The non-identifiability term, with increasing γ, penalizes any predictor β towards zero on the subspace R of non-identified test shift directions. In total, the identifiable robust loss (in the linear SCM setting) equalsL n (β; βS , Ŝ Ŝ⊤ , R R⊤ ) = n0 i=1 (Y 0,i -β ⊤ X 0,i ) 2 + γ∥ Ŝ Ŝ⊤ (β -β S )∥ 2 2 + γ(C ker + ∥ R R⊤ β∥ 2 ) 2 ,where we suppress dependence on C and γ and only leave the dependence on the nuisance parameters.Choice/Estimation of nuisance parameters. We now provide more details on the empirical estimation of the nuisance parameters Ŝ, Ŝ, R, and βS .• The constant C corresponds to the upper bound on the norm of the true causal parameter β ⋆ . Thus, the practitioner chooses C in advance to ensure that (with high probability)∥β ⋆ ∥ 2 ≤ C.• The training shift directions Ŝ can be computed viaEQUATIONwhere for e ∈ E train , the matrix Cov(X e ) is the empirical covariance matrix estimated within the training environment E = e, and µ e ∈ R d is the empirical mean of the covariates within the training environment E = e. Additionally, we compute the orthogonal complement Ŝ⊥ (D) of the space Ŝ(D) 12 .• The decomposition of the test shift directions M into identified and non-identified shift directions (and their corresponding projection matrices) can be computed as follows. Let Π Ŝ and Π Ŝ⊥ denote the projection matrices on Ŝ(D) and Ŝ⊥ (D), respectively. Consider the singular value decompositionsΠ Ŝ M = U Ŝ Σ Ŝ V ⊤ Ŝ and Π Ŝ⊥ M = U Ŝ⊥ Σ Ŝ⊥ V ⊤ Ŝ⊥ Then, define Ŝ = U Ŝ , R = U Ŝ⊥ .The subspaces range(Π Ŝ M ) and range(Π Ŝ⊥ M ) are minimal subspaces contained in Ŝ and Ŝ⊥ , respectively, such that range(M ) ⊂ range(Π Ŝ M ) ⊕ range(Π Ŝ⊥ M ). The matrices Ŝ Ŝ⊤ and R R⊤ are their corresponding projection matrices.• The identified causal parameter βS (approximately) equals the true causal parameter β ⋆ on the space of training shift directions Ŝ. As conjectured in the anchor regression literature [45, 49, 26 ] (see, for example, the discussion right after Theorem 3.4 in [26] and Appendix H.3 therein) for γ → ∞, the estimators β γ anchor and β γ DRIG converge to the causal parameter β ⋆ on S. Thus, the identified causal parameter can be estimated asβS := Π Ŝ β ∞ anchor or βS := Π Ŝ β ∞ DRIGfor the setting of mean or mean+variance shifts, respectively.For any estimator β ∈ R d and given the estimated nuisance parameters φ := ( Ŝ Ŝ⊤ , R R⊤ , βS ), we define the sample identifiable robust risk asEQUATION)Therefore,EQUATIONEQUATION)We now want to prove convergence the four terms in ( 21) and (22) . For this, we use the following statements proved in Appendix D.4. Lemma D.2. Suppose B ⊆ R d is a compact set. Moreover, assume that the covariance matrixEQUATION)EQUATION)The two terms in ( 21) converge to 0 by Lemma D.2 and Lemma D.3, respectively. The first term in ( 22) equals 0 since β minimizes β → L n (β, φ). Finally, we observe thatEQUATIONsince we have that L(β, φ 0 ) = E [(Y 0 -β ⊤ X 0 ) 2 ] + γ∥SS ⊤ (β S -β)∥ 2 2 + γ C -∥β S ∥ 2 2 + ∥RR ⊤ β∥ 2 2.Since E [X 0 X ⊤ 0 ] ≻ 0, the first term is strongly convex in β. Moreover, the second and third terms are convex in β. Therefore, L(β, φ 0 ) is strongly convex in β. Since L(β, φ 0 ) is also continuous in β, it follows that there exists a unique global minimum. Let β 0 denote the global minimizer of L(β, φ 0 ). By the fact that L(β 0 , φ 0 ) is a global minimum, and by definition of strong convexity, there exists a positive constant m > 0 such that, for all β ∈ B,EQUATIONFix δ > 0. Then, by (26) , for all β ∈ B such that ∥β -β 0 ∥ 2 > δ it holds that L(β, φ 0 ) ≥ L(β 0 , φ 0 ) + mδ 2 2 > L(β 0 , φ 0 ).Since the inequality holds for all β ∈ B such that ∥β -β 0 ∥ 2 > δ, we conclude thatinf{L(β, φ 0 ) : ∥β -β 0 ∥ 2 > δ} > L(β 0 , φ 0 ).Since δ > 0 was arbitrary, the claim follows.Recall that for anyβ ∈ B L(β, φ 0 ) = E [g β,φ0 (X 0 , Y 0 )], L n (β, φ 0 ) = 1 n 0 i∈D0 g β,φ0 (X 0,i , Y 0,i ).To show the result, we must establish that the class of functions {g β,φ0 : β ∈ B} is Glivenko-Cantelli.From [56] , a set of sufficient conditions for being a Glivenko-Cantelli class is that (i) B is compact, (ii) β → g β,φ0 (x, y) is continuous for every (x, y), and (iii) β → g β,φ0 is dominated by an integrable function. By assumption, (i) holds. Moreover, by (19) , it follows that β → g β,φ0 is continuous for all (x, y) and thus (ii) holds. We now show that (iii) holds. Since B is compact we have that sup β∈B ∥β∥ 2 = C 1 < ∞. For fixed γ > 0, and all (x, y), we have thatEQUATIONwhere K < ∞ is a finite constant not depending on (x, y). Furthermore, we have thatEQUATIONsinceE [Y 2 ] < ∞ and E [X 0 X ⊤ 0 ]has bounded eigenvalues by assumption. From ( 27) and ( 28), it follows that (iii) holds.For fixed γ > 0, we have thatEQUATIONEQUATIONWe can upper bound (29) as follows,EQUATIONEQUATIONwhere (31) follows from the Cauchy-Schwarz inequality and that ∥A∥ 2 ≤ ∥A∥ F , the constants (32) follow from compactness of B, and the convergence in probability follows from Assumption D.1. Furthermore, we can upper bound (30) as follows,C 1 , C 2 < ∞ insup β∈B C -∥ βS ∥ 2 2 + ∥ R R⊤ β∥ 2 2 - C -∥β S ∥ 2 2 + ∥RR ⊤ β∥ 2 2 = sup β∈B C -∥ βS ∥ 2 2 + ∥ R R⊤ β∥ 2 2 + 2 C -∥ βS ∥ 2 2 ∥ R R⊤ β∥ 2 -C + ∥β S ∥ 2 2 -∥RR ⊤ β∥ 2 2 -2 C -∥β S ∥ 2 2 ∥RR ⊤ β∥ 2 ≤ sup β∈B ∥ βS ∥ 2 2 -∥β S ∥ 2 2 + sup β∈B β ⊤ ( R R⊤ -RR ⊤ )β + 2 sup β∈B C -∥ βS ∥ 2 2 ∥ R R⊤ β∥ 2 -C -∥β S ∥ 2 2 ∥RR ⊤ β∥ 2 = (I) + (II) + (III).By Assumption D.1, (I) converges in probability to zero. Regarding (II), we havesup β∈B β ⊤ ( R R⊤ -RR ⊤ )β ≤ sup β∈B ∥β∥ 2 2 ∥ R R⊤ -RR ⊤ ∥ F P → 0,where the inequality follows from Cauchy-Schwarz and that ∥A∥ 2 ≤ ∥A∥ F , and the convergence in probability follows from Assumption D.1 along with the compactness of B. It remains to upper bound (III). We have thatEQUATIONEQUATIONThe inequality in (33) follows from the compactness of B, the fact that R R⊤ has bounded eigenvalues, and that | √ x -√ y| ≤ |x -y| 1/2 for all x, y ≥ 0. The inequality in (34) follows from Cauchy-Schwarz and that ∥A∥ 2 ≤ ∥A∥ F . The convergence in probability follows form Assumption D.1 and the compactness of B.In this section, we provide more details of the data generation for our synthetic finite-sample experiments as well as data processing for the real-world data experiments.For the synthetic experiments, we generate a random SCM which satisfies our assumptions. For d = 15, we randomly sample the joint covariance Σ ⋆ of (X, Y ), fixing its total variance and the eigenvalues. We consider 7 environments including the reference environment, and for each environment except the reference, we randomly generate mean shifts of fixed norm. Since we have 6 non-zero random Gaussian mean shifts, it holds a.s. that dim S = 6. We then randomly generate an "initial guess" for β ⋆ ∈ R d of fixed norm C = 10. Now, with respect to the space S of the identifiable directions induced by the mean shifts, we choose the most "adversarial" causal parameter β ⋆ adv which is equal to β ⋆ on S, but on S ⊥ has the opposite direction of the noise OLS estimator Σ ⋆ η -1 Σ ⋆ η,ξ . We ensure that ∥β ⋆ adv ∥ 2 = C. Note that under the observed shifts, β ⋆ and β ⋆ adv are observationally equivalent. We complete β ⋆ adv to the set θ adv of observationally equivalent model parameters and generate the multi-environment training data according to θ adv and the collection of mean shifts.For Figure 3 (left), we define the test shift upper bound as M anchor = γSS ⊤ , where S is taken to be a two-dimensional subspace of S. We vary γ from 0 to 10, and for each γ, we compute the oracle anchor regression estimator by minimizing the discrete anchor regression loss. Additionally, we compute the pooled OLS estimator and the identifiable robust predictor β rob,ID as described in Appendix D. Finally, we generate test data with a Gaussian additive shift A test ∼ N (0, M anchor ). We evaluate the loss of β OLS , β anchor and β rob,ID on this test environment and include the population lower bound.For Figure 3 (right), we define the test shift upper bound as M new = γSS ⊤ + γ ′ RR ⊤ , where R is a 2-dimensional subspace of the space S ⊥ and we set γ ′ = 0.05γ to showcase the effect of small unseen shifts compared to large identified shifts. We vary γ from 0 to 40 and for each γ, compute the oracle anchor regression estimator by minimizing the discrete anchor regression loss. Additionally, we compute the pooled OLS estimator and the identifiable robust predictor β rob,ID as described in Appendix D, for which we use knowledge of spaces S and R and prior knowledge of M new . Finally, we generate test data with a Gaussian additive shift A test ∼ N (0, M new ). We evaluate the loss of β OLS , β anchor and β rob,ID on this test environment, plot the resulting test losses for different estimators and include the population lower bound. We consider the K562 dataset from [41] and perform the preprocessing as done in [13] . The resulting dataset consists of n = 162, 751 single-cell observations over d = 622 genes collected from observational and several interventional environments. The interventional environments arise by knocking down a single gene at a time using the CRISPR interference method [40] . Following [48] , we select only always-active genes in the observational setting, resulting in a smaller dataset of 28 genes. For each gene j = 1, . . . , 28, we set Y := X j as the target variable and select the three genes X k1 , . . . , X k3 most strongly correlated with Y (using Lasso), resulting in a dataset with columns Y, X k1 , . . . , X k3 . Given this dataset, we construct the training and test environments as follows. Let O denote the 10,691 observations collected from the observational environment, and let I i denote the observations collected from the interventional environment where the gene k i was knocked down. We will denote by I i,s the s-th quantile of datapoints in I i w.r.t. to the expression value of the gene k i . These are the s × 100% of datapoints with the weakest shift compared to the observational mean of the gene k i , and thus the parameter s ∈ [0, 1] is a proxy for the strength of the shift. Furthermore, denote by I * i,s a random sample of I i,s of a certain size. For each i ∈ {1, 2, 3}, we train the methods on D train . . , 1} we evaluate the models on the test samples from the three interventional environments. An example of the test data for different shift strengths s and a previously unseen direction is shown in Figure 5(b-c ). Figure 4 shows the test MSE performance as a function of perturbation strength. We compare our method Rob-ID, defined as the minimizer of the empirical identifiable robust risk (17) , with anchor regression [45] , invariant causal prediction (ICP) [38] , Distributional Robustness via Invariant Gradients (DRIG) [49] , and OLS (corresponding to vanilla ERM). We use the following parameters for Rob-ID: γ = 50, C ker = 1.0, and M = Id. For anchor regression and DRIG, we select γ = 50. For ICP, we set the significance level for the invariance tests to α = 0.05.These numerical experiments are computationally light and can be run in ≈ 5 minutes on a personal laptop. 13For every environment e ∈ E train , we observe the first moments E (X e ) and E (Y e ), and second moments E (X e X ⊤ e ), E (Y 2 e ) and E (X e Y e ). Since it holds by assumption that µ 0 = 0 and Σ 0 = 0, we have that E (X 0 X ⊤ 0 ) = Σ ⋆ η , and so we can identify Σ ⋆ η uniquely. Furthermore, it holds thatEQUATIONEQUATIONBy taking the difference between Equation (36) and Equation (35) , we can identify (Σ e + µ e µ ⊤ e )β ⋆ . Thus, the causal parameter β ⋆ is identifiable on the subspace S defined in Equation (10) and is not identifiable on its orthogonal complement S ⊥ . Thus, for any for vector α ∈ S ⊥ , the vector β = β ⋆ + α is consistent with the data-generating process. It remains to compute the covariance parameters induced by an arbitrary β := β ⋆ + α, for α ∈ S ⊥ . For every environment e ∈ E train , the second mixed moment between X e and Y e has to satisfy the following equality E (X e Y e ) = (Σ e + µ e µ ⊤ e + Σ ⋆ η )β ⋆ + Σ ⋆ η,ξ = (Σ e + µ e µ ⊤ e + Σ ⋆ η ) β + Ση,ξ , from which it follows that Ση,ξ := Σ ⋆ η,ξ -Σ ⋆ η α. By computing E (Y 2 e ) and inserting β = β ⋆ + α and Ση,ξ , we similarly obtainσ2 ξ := (σ ⋆ ξ ) 2 -2α ⊤ Σ ⋆ η,ξ + α ⊤ Σ ⋆ η α. Thus, we obtain the following set of observationally equivalent model parameters consistent with P θ ⋆ ,Etrain :Θ eq = {β ⋆ + α, Σ ⋆ η , Σ ⋆ η,ξ -Σ ⋆ η α, (σ ⋆ ξ ) 2 -2α ⊤ Σ ⋆ η,ξ + α ⊤ Σ ⋆ η α : α ∈ S ⊥ }. Since the observationally equivalent set is identifiable from the training distribution, but model parameters β ⋆ , Σ ⋆ η,ξ , (σ ⋆ ξ )2 are not, it is helpful to re-express the observationally equivalent set through identifiable quantities. For this, we note that the "identified causal predictor" β S = β ⋆ -β S ⊥ induces an observationally equivalent model given byθ S := (β S , Σ S η , Σ S η,ξ , (σ S ξ ) 2 ) = (β S , Σ ⋆ η , Σ ⋆ η,ξ + Σ ⋆ η β S ⊥ , (σ ⋆ ξ ) 2 + 2⟨Σ ⋆ η,ξ , β S ⊥ ⟩ + ⟨β S ⊥ , Σ ⋆ η β S ⊥ ⟩).From this reparameterization, we infer the final form of the observationally equivalent set :Θ eq = {β S + α, Σ ′ η , Σ S η,ξ -Σ ′ η α, (σ S ξ ) 2 -2α ⊤ Σ S η,ξ + α ⊤ Σ ′ η α : α ∈ S ⊥ } ∋ θ ⋆Therefore, Equation ( 12) follows. To find the robust predictor β rob , we write down the robust loss with respect to M test and any θ α from the observationally equivalent set :R rob (β; θ α , M test ) = (β S + α -β) ⊤ (M test + Σ ⋆ η )(β S + α -β) + 2(β S + α -β) ⊤ (Σ ⋆ η,ξ -Σ ⋆ η α) + (σ S ξ ) 2 -2α ⊤ Σ S η,ξ + α ⊤ Σ ⋆ η α.inserting α ∈ S ⊥ and rearranging, Equation ( 13) follows.F.2 Proof of Theorem 3.1We structure the proof as follows: first, we quantify the non-identifiability of the robust risk by explicitly computing its supremum over the observationally equivalent set of the observationally equivalent model parameters (referred to as the identifiable robust risk). Second, we derive a lower bound for the identifiable robust risk by considering two cases depending on how a predictor β interacts with the possible test shifts M test . In this proof, we use more general notation, with the test shifts bounded by a PSD matrix M test ⪯ γM + γ ′ RR ⊤ , which rangeM ⊂ S and rangeR ⊂ S ⊥ . The statement of the theorem follows by setting γ = γ ′ . However, we believe that the more refined statement is useful, e.g., when one expects strong shifts in training directions and only weak "new" shifts.Computation of the identifiable robust risk. For any model-generating parameter θ = (β, Σ) it holds that the robust risk of the model Equation ( 3) under test shifts M test ⪰ 0 is given byR rob (β; θ, M test ) = (β -β) ⊤ (M test + Σ ⋆ η )(β -β) + 2(β -β) ⊤ Σ η,ξ + (σ ξ ) 2 .We recall that the observationally equivalent set of model parameters after observing the multienvironment training data Equation ( 3) is given byEQUATIONwhere S is the span of identified directions defined in Equation (10) . Moreover, we recall that by Assumption 3.1, for any causal parameter β it should hold that ∥β∥ 2 = ∥β S + α∥ 2 ≤ C, which translates into the following constraint for the parameter α:∥α∥ 2 ≤ C 2 -∥β S ∥ 2 2 =: C ker .Inserting Equation (37) in Equation ( 8), we obtainR rob,ID (β; Θ eq , M test ) = sup α∈S ⊥ , ∥α∥2≤C ker R rob (β; θ α , M test ),where θ α is a short notation for(β S + α, Σ ⋆ η , Σ S η,ξ -Σ ⋆ η α, (σ S ξ ) 2 -2α ⊤ Σ S η,ξ + α ⊤ Σ ⋆ η α).We now compute the supremum explicitly in case M test has the form M test = γM + γ ′ RR ⊤ , where M is a PSD matrix with range M ⊆ S and R is a semi-orthogonal matrix with range R ⊆ S ⊥ . Note that this assumption includes both the setting of Theorem 3.1 and the setting of finite robustness methods in Section 3.2. For any α ∈ S ⊥ , we write down the robust loss asR rob (β; θ α , M test ) = (β S -β) ⊤ (M test + Σ ⋆ η )(β S -β) + 2(β S -β) ⊤ Σ S η,ξ + (σ S ξ ) 2 + α ⊤ M test α + 2α ⊤ M test (β S -β) = R rob (β; θ S , M test ) + α ⊤ M test α + 2α ⊤ M test (β S -β).The first term is the robust loss of β under test shift M test and the identified model-generating parameter θ S , thus it does not depend on α. By the structure of M test , we obtain thatf (α) := α ⊤ M test α + 2α ⊤ M test (β S -β) = γ ′ α ⊤ RR ⊤ α -α ⊤ RR ⊤ β.If R = 0, i.e., the test shifts consist only of the identified directions, we have f (α) = 0, independently of α, and thusR rob,ID (β; Θ eq , M test ) = R rob (β; θ S , M test ).This implies the first statement of the theorem.We now consider the case where R ̸ = 0, i.e., RR ⊤ is a non-degenerate projection. Our goal is to maximize f (α) subject to constraints α ∈ S ⊥ , ∥α∥ 2 ≤ C ker . Let R be an orthonormal extension of R such that range (R| R) = S ⊥ . Then, we can parameterize α ∈ S ⊥ as α = (R| R)( w w ) and the corresponding Lagrangian readsL(α, λ) = γ ′ α ⊤ RR ⊤ α -α ⊤ RR ⊤ β + λ(C 2 ker -∥α∥ 2 2 ) = γ ′ ∥w∥ 2 2 -w ⊤ R ⊤ β + λ(C 2 ker -∥(w, w)∥ 2 2 ).Differentiating with respect to w, w yieldsw = γ ′ γ ′ -λ R ⊤ β; w = 0.After differentiating w.r.t. λ, we obtain γ ′ γ ′ -λ = ± C ker ∥R ⊤ β∥2 . By inserting in the objective function and comparing, we obtain the value of the identifiable robust risk:EQUATIONPutting together the cases R = 0 and R ̸ = 0, and plugging inM test = γSS ⊤ + γ ′ RR ⊤ , we obtain R rob,ID (β; Θ eq , M test ) = γ ′ I R̸ =0 (C ker + ∥R ⊤ β∥ 2 ) 2 + R rob (β; θ S , γM ) = γ ′ I R̸ =0 (C ker + ∥R ⊤ β∥ 2 ) 2 + γ∥S ⊤ (β S -β)∥ 2 2 + R 0 (β S , β),where R rob (β; θ S , γM ) is the robust risk of the estimator β w.r.t. the "identified" test shift γM and the identified model parameter θ S , whereas R 0 (θ S , β) is the risk of β on the reference environment e = 0.Derivation of the lower bound for the identifiable robust risk. Now that we have explicitly computed the identifiable robust risk, we devote ourselves to the computation of the lower bound for its best possible valueinf β∈R d R rob,ID (β; Θ eq , M test ).In this part, we will only consider the case R ̸ = 0, since the case R = 0 corresponds to the (discrete) anchor regression-like setting, where both the robust risk and its minimizer are uniquely identifiable, and computable from training data. We will distinguish between two cases.Case 1: ∥R ⊤ β∥ 2 = 0. In this case, β is fully located in the orthogonal complement of R, which consists of S and R. We will denote (the basis of) this subspace by S tot = S ⊕ R. Thus, S tot is the "total" stable subspace consisting of identified directions in S and non-identified, but unperturbed directions R. We will parameterize β as β = S tot w. Thus, we are looking to solve the optimization problemβ rob,ID = arg min w (β S -S tot w) ⊤ (γSS ⊤ + Σ ⋆ η )(β S -S tot w) + 2(β S -S tot w) ⊤ Σ S η,ξ + (σ S ξ ) 2 .Setting the gradient to zero yields the asymptotic identifiable robust estimatorEQUATIONwhich corresponds to the loss value ofR rob,ID (β rob,ID ; Θ eq , M test ) = γ ′ C 2 ker + (σ S ξ ) 2 -2Σ S η,ξ ⊤ S tot [S ⊤ tot (γSS ⊤ + Σ η )S tot ] -1 S ⊤ tot Σ S η,ξ .Case 2:∥R ⊤ β∥ 2 ̸ = 0. Since for ∥R ⊤ β∥ 2 ̸ = 0, the objective function is differentiable, we compute its gradient to be∇R rob,ID (β; Θ eq , M test ) = 2γ ′ RR ⊤ β/∥RR ⊤ β∥ + 2γ ′ RR ⊤ β + ∇R rob (β; θ S , γM ) = 2γ ′ RR ⊤ β/∥RR ⊤ β∥ + 2γ ′ RR ⊤ β + 2(Σ ⋆ η + γM )(β -β S ) -2Σ S η,ξ. This equation is, in general, not solvable w.r.t. β in closed form. Instead, we provide the limit of the optimal value of the function when the strength of the unseen shifts is small, i.e. γ ′ → 0. We know that for γ ′ = 0, the minimizer of the identifiable robust risk is given by the anchor estimatorβ anchor = β S + (Σ ⋆ η + γM ) -1 Σ S η,ξ .Thus, we lower bound the term 2γ ′ C ker ∥R ⊤ β∥ by the scalar product 2γ ′ C ker ⟨R ⊤ β, R ⊤ β anchor ⟩/∥β anchor ∥ and expect it to be tight for small γ ′ . After inserting this lower bound in Equation ( 38) we obtain the minimizer of the lower bound of formβ LB = β S + (Σ ⋆ η + γM + γ ′ RR ⊤ ) -1 (Σ S η,ξ -γ ′ C ker RR ⊤ (Σ ⋆ η + γM ) -1 Σ S η,ξ ).We can now lower bound ∥RR ⊤ β LB ∥ as∥RR ⊤ β LB ∥ ≥ ∥RR ⊤ (Σ ⋆ η + γM ) -1 Σ ⋆ η,ξ ∥ -o(γ ′ ),from which the rate for small γ ′ follows. If we set γ = γ ′ and M = SS ⊤ , the claim (15) of the theorem follows. For Section 3.2, the lower bound directly implies optimality of the identifiable robust risk of the anchor estimator when the strength of the unseen shifts γ ′ is small. Additionally. if γ = 0, i.e. only unseen test shifts occur, we conclude that the OLS and anchor estimators have the same rates.Lower bound γ th for γ ′ . Finally, we want to derive a lower bound on the shift strength γ ′ such that for all γ ′ ≥ γ th Case 1 of our proof is valid, i.e. it holds that β rob,ID is given by the closed form "abstaining" estimator (39) . For this, we find γ th such that for all γ ′ ≥ γ th zero is contained in the subdifferential of R rob,ID (β rob,ID ; Θ eq , M test ) at β rob,ID . Then the KKT conditions are met, and β rob,ID is the unique minimizer of the identifiable robust risk due to strong convexity of the objective. We compute the subdifferential to be S = γ ′ C ker {RR ⊤ β : ∥β∥ 2 ≤ 1} + ∇R rob (β rob,ID ; θ S , γM ).Since β rob,ID is the minimizer of R rob (β; θ S , γM ) under the constraint R ⊤ β = 0, the gradient is zero in R ⊥ and it remains to show that∥RR ⊤ ∇R rob (β rob,ID ; θ S , γM )∥ ≤ γ ′ C ker , or γ ′ ≥ ∥RR ⊤ ∇R rob (β rob,ID ; θ S , γM )∥/C ker .Via an upper bound on the projected gradient, we derive the stricter conditionγ ′ ≥ ∥RR ⊤ Σ S η,ξ ∥(1 + κ(Σ ⋆ η )) C ker ,where κ(Σ ⋆ η ) is the condition number of the covariance matrix.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The numerical experiment described in Section 3.2 is computationally very light and can be run on a personal laptop in a few minutes. We describe this in Appendix E.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: We have read the NeurIPS Code of Ethics and confirm that our work conforms to it in all aspects.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Here, we mean partial identifiability of the robust risk, which is reminiscent of outputting uncertainty sets for a quantity of interest in the field of partial identification[54,19].In particular, extending(2) to its finite-sample counterpart would introduce a more natural extension of the classical minimax risk statistical learning theory. In this work, we focus on identifiability aspects instead of statistical rates.Note that in the anchor regression setup, the environment shift A, called anchor, is also observed, thus the training data consist of (A, X, Y ).Even though we do not observe Atest, the structural assumptions on Atest correspond to assumptions onE [X test X test⊤ ] -E [X0X ⊤ 0 ].When more refined information on the test shifts is given by a general PSD matrix Mtest ∈ R d×d , we can replace it by the upper bound Mtest ⪯ λmax(Mtest)Π range(M test ) and apply our results on the upper bound.In general, the distribution of Ae is unknown, since Ae is unobserved. In our setting, P A e can be identified because of the reference environment. Otherwise, one proceeds with relative shifts as described in Appendix B.The choice of S and R is not unique, but the subspaces range S, range R, which matter for our results, are.In Appendix C, we show that analogous results hold for continuous anchor regression and the method of distributionally robust invariant gradients (DRIG)[49].Here, we only vary γ ′ , whereas γ is fixed.In practice, it is useful to pick a distribution with the smallest covariance, i.e. tr Cov(X ref ) ≤ tr Cov(Xe) for all e.In other words, we require that the test distribution is a shifted version of the (arbitrarily) chosen reference distribution.In general, S(D) is a proper subspace of R d and the RHS of (16) corresponds to a sum of low-rank second moments. This can be consistently estimated if, for instance, the rank of each shift is known (e.g. in the mean shift setting), or the covariances have a spiked structure, allowing to cut off small eigenvalues.We use a 2020 13-inch MacBook Pro with a 1.4 GHz Quad-Core Intel Core i5 processor, 8 GB of RAM, and Intel Iris Plus Graphics 645 with 1536 MB of graphics memory. 


Decision: The paper received positive reviews, with the reviewers appreciating the novelty of the results and the quality of the presentation. Consequently, I recommend acceptance of this paper. When preparing the camera-ready revision, please pay close attention to the reviewers’ feedback and include the additional explanations provided during the rebuttal period.

Reviews: review: paper_summary: paper_summary: The paper studies a linear Structural Causal Model (SCM) for prediction under distribution shifts due to an an additive term to covariates that changes at test time, and an unobserved confounder between the covariates and the label. The key difference between the proposed analysis and those presented in other works (e.g. those in anchor regression, invariant causal prediction etc.) is that the shift is bounded in its strength. Another difference is that the optimal robust predictor w.r.t the entire uncertainty set is not identifiable. Hence, instead of the optimal robust error, the paper studies the optimal error that can be achieved from observable data. This corresponds to the robust error over an uncertainty set that is generated by all SCM parameters that could have produced the training data, which is in general a larger set than the uncertainty set we are interested in (i.e. of bounded additive shifts to the covariates).

Once the problems is set up, a parametric form for the set of parameters that can generate the training data is derived, along with the corresponding set of possible robust predictors. Then under a mild assumption on boundedness of the ground truth regression parameters, the following are derived: 1) a closed form is derived for the robust loss in the case where test distributions only induce shifts in directions that have been observed at training. The loss grows linearly with the strength of the shift; 2) a lower bound on the achievable risk from observed data, which is tight (and thus can be learned from observed data) for large enough shifts.

Further analysis and simulations with Gaussian data are done to compare the risk of the derived estimator with that of ERM and anchor regression (which does not exploit the boundedness of shifts to achieve better prediction). The results verify that the risk obtained by the proposed empirical estimator are close the lower bound given in the theorem, and outperform the two baselines as shifts grow large.
strength_weakness: strength_weakness: * As mentioned above and the authors mention in describing the limitations of their work, it is restricted to linear models. Intuitively, a considerable challenge in learning robust models is to identify the ``directions” that shift between domain. Under linear models it is rather straightforward, and the method proposed in the paper depends on linearity of the model in order to find these directions. This is unlike some other methods in the domain generalization literature, where certain formal results are provided on linear models but the methods can easily be tested in the non-linear case.
*  Even within the realm of linear models, I am not throughly convinced that the method is useful in real world problems. To make this more convincing, it might have been nice to run simulations over non-synthetic data and some more variants of methods. E.g. using several methods that learn invariant models when hyper-parameter tuning is performed with the objective proposed in this work may be of interest. That is to see whether boundedness of the shift should be taken into account during training, or is it enough to simply use it for model selection. Yet the most significant drawback is still of course, the limitation to linear models.

Some other/smaller comments:
* In line 60 prior work is cited to claim "that even minor violations of the identifiability assumptions can cause invariance based methods to perform equally or worse than ERM". However, I am not sure that the failures portrayed in these works are strictly due to identifiability violations. At least not the ones alluded to in this paper, which is the lack of heterogeneity in the training environments. In Kamath et al. 21, the predictor is identifiable but the problem is specifically in the IRMv1 objective, see also [1] who point out an objective that solves this issue. In Rosenfeld et al. 20, only one failure is due to not having enough environments (and arguably, this is since they consider the uncertainty set that includes shifts in all directions. I will also touch on this in the next minor comment), and the second one is due to non-linearity.
* Regarding analysis under lack of heterogeneity or unidentifiable optimal robust predictor: I think that the part of the analysis that touches upon the insufficient heterogeneity of the environments in order to identify the robustness set/robust predictor can be slightly reframed. If I am not mistaken, even in works that give results about identifiable robustness sets (e.g. when the number of environments is linear in the number of shifting features, or stronger ones like [2]), it is most likely possible to draw guarantees about robust risks, but only with respect to a smaller uncertainty set which is restricted to dimensions that shifted between the training environments. That is since the methods still enforce constraints which restrict weights in certain directions. While it is true that most of these prior works did not consider unidentifiable uncertainty sets, it might be related to choice of presentation, and not strictly because the methods are technically limited in that sense. Further, in lines 168-169 it is claimed that “prior work only considers scenarios where the robustness set and hence also the robust prediction model are still identifiable”. I am not sure this is entirely true. There are different formalisms that were considered for spurious correlations, such as those in [3]. In their setting, when the association is not what they call “purely spurious”, then the robust predictor cannot necessarily be recovered. Also [4] study a similar setting where due to similar reasons no guarantee can be given on identifying the robust predictor, but only a lower bound on the error is given.
* It might be worth mentioning in lines 112-113 that in case $H$ in figure 1 is a selection variable (I assume it can be since both edges are bi-directional) then the shift reduces to covariate shift, if I am not mistaken.
* Personally, I found the notation in the introduction which uses $\mathrm{shift}\in{\mathrm{shift set}}$ etc. somewhat redundant and confusing. The more formal notation in section 2 onwards was much easier to understand, so in my opinion it might be worth considering to go straight ahead into that notation.

[1] Wald Y, Feder A, Greenfeld D, Shalit U. On calibration and out-of-domain generalization. Advances in neural information processing systems. 2021 Dec 6;34:2215-27.

[2] Chen Y, Rosenfeld E, Sellke M, Ma T, Risteski A. Iterative feature matching: Toward provable domain generalization with logarithmic environments. Advances in Neural Information Processing Systems. 2022 Dec 6;35:1725-36.

[3] Veitch V, D'Amour A, Yadlowsky S, Eisenstein J. Counterfactual invariance to spurious correlations in text classification. Advances in neural information processing systems. 2021 Dec 6;34:16196-208.

[4] Puli AM, Zhang LH, Oermann EK, Ranganath R. Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlations. In International Conference on Learning Representations.
questions: questions: * It could’ve been nice to derive upper bounds on the robust loss in addition to the lower bounds proved in the theorem. That is, assuming these bounds can be calculated from observable data, I’d imagine that most practitioners would be more interested in an upper bound as it gives a strong guarantee on the risk worst possible risk. Do you think this is a reasonable goal, and do you perhaps have any insights on this? 
* Could you perhaps discuss the differences between the empirical optimization problem you derived in the paper and those derived in prior work? For instance, looking at eq. 19 in appendix D, it seems like there is a term that penalizes $\|\| \hat{S}^\top(\hat{\beta}^{\mathcal{S}} - \hat{\beta}) \|\|$ which I read as the weights in the “invariant directions” should be the same across the environments. This seems quite similar to ICP, IRM etc. Then the other term limits the projection on the shifting directions, but it might be useful for readers to have a short discussion on the conceptual similarities between this and penalized versions of other methods. Also, there might be a typo in that equation, $\mathrm{arg}\min_{\beta\in{\mathbb{R}^d}}$ should be $\mathrm{arg}\min_{\hat{\beta}\in{\mathbb{R}^d}}$ instead.
limitations: limitations: The authors have adequately discussed limitations.
paper_summary: paper_summary: This paper proposes a new framework for distributionally robustness under the linear causal setting. Specifically, the authors minimize the so-called identifiable robust risk, which corresponds to the maximum of the robust risk for parameters in the observationally equivalent set. Under such partially identifiable robustness framework, they discuss the lower bound of the risk and show that some estimated identifiable robust predictor can achieve that lower bound risk, where the corresponding empirical version can approximate the lower bound risk well. They also validate the better performance with finite sample numerical experiments.
strength_weakness: strength_weakness: -	Some paragraph in the training and testing data part can be reorganized as some formal assumptions, such as the assumption of M_{test} and linear assumption as well as the discussion in Section 3.2 in terms of the definition of S and M. This gives a clear picture on what some key conditions (and possible relaxations) are in this paper.
-	There are a few missing details (see Questions as follows), which I hope the authors can explain them a bit more.
questions: questions: -	Can the authors highlight a bit more on the (approximated) real-world example when the predictor is partially identifiable? Since it may be a bit unfamiliar to the general DRO audience for these notations and their relevance in practice.
-	Can the authors elaborate more on how the empirical estimations for the space of the training shift direction $\hat S$ and $\hat R$ are computed, as compared with the current version in Appendix D? From my preliminary understanding, this space determination is important for subsequent estimation.
-	The potential utility of **active intervention selection** is also interesting, and it seems aligned with some recent relevant work on distribution shifts. For example, in causal explanation [1] and incorporating specific features in improving distributional robustness [2]. I am wondering if the authors can provide more details on how their own intervention is and the connections with these existing literature, which can be potentially incorporated in the main body and appendix.

[1] Quintas-Martinez, Victor, et al. "Multiply-Robust Causal Change Attribution." arXiv preprint arXiv:2404.08839 (2024).

[2] Liu, Jiashuo, et al. "On the Need for a Modeling Language Describing Distribution Shifts: Illustrations on Tabular Datasets." arXiv preprint arXiv:2307.05284 (2023).
limitations: limitations: The authors discuss the main limitation of this paper, which I think is reasonable compared with existing literature.
paper_summary: paper_summary: This paper investigates the optimal minimax risk of a robust predictor when the robustness set is partially observable, under a structural causal model with hidden confounders. By decomposing the test covariance matrix of latent parameters into a component spanned by the training distributions and the orthogonal component, it is shown that the robust predictor is only identifiable if the test shifts are in the direction of the training shifts. While for unobserved test shifts, the best achievable minimax risk grows linearly with shift scale. The theory is applied to show sub-optimality of OLS and anchor regression with partially observable test shifts.
strength_weakness: strength_weakness: The structural causal model in Eq 2 and the resulting explicit solution in Eq 6 shows that the model is biased even without distribution shifts. To see this, take $\gamma =0$ in Eq 6 and the predictor does not reduce to $\beta^\star$. The model is unbiased only when the cross covariance between $\eta$ and $\xi$ vanishes, which implies no hidden confounder and only covariate shift. In contrast, the classic approach for invariant causal prediction [1] produces unbiased estimation beyond covariate shift.

[1] Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint arXiv:1907.02893.
questions: questions: 1. Typo: L137, the sub-space.
2. In Fig 1, bidirectional edges don't make sense for causal graphs, and it's not explained either. Moreover, is there exact equivalence between Fig 1 and the SCM in Eq 2?
limitations: limitations: The author has addressed limitations with regarding to the model, e.g., linear structure and additive noises.
paper_summary: paper_summary: This paper proposed a general framework of partially identifiable robustness to evaluate robustess in scenarios where the training distributions are not heterogeneous enough to identify the robust risk. They define 'the identifiable robust risk' and its correspondig minimax quantity. They show previous approaches achieve suboptimal robustness in this scenario.  Finally, they propose the empirical minimizer of the identifiable robust risk and show that it outperforms existing methods in finite-sample experiments.
strength_weakness: strength_weakness: I am not an expert in this field so I cannot point out many weaknesses. Maybe one thing is the lack of motivation from real-world application, such as in what applications or when the proposed framework is usefull in practice?
questions: questions: 1. On line 135-137, the $M_{\text{test}}$ is defined as  $M_{\text{test}}=\gamma\Pi_{M}$. What's the reasoning behind this choice?
limitations: limitations: yes

Comments: comment: Thanks for the reply. It addressed my questions.
Thank you for addressing the comments and questions, I also appreciate the additional empirical results.

I see the point in most of the replies, perhaps one small comment is that [3, 4] indeed provide some binary negative statements, but [4] also give a form of a guarantee (an estimator that has better-than-random worst case performance, and is optimal in a restricted manner). I agree that it is a different flavor of guarantee and analysis than the one given in this paper. Hence my comment is just meant to suggest a slight adjustment to the framing of the contribution, not to devalue it in any way.

I have raised my score following the in-depth response.
I acknowledge and thank the authors for their detailed response. Therefore, I keep my evaluation towards the paper.

Specificially, I am quite interested in the active intervention part authors mention and agree with what the authors say in the rebuttal with respect to its connection with the active learning side. Therefore, I am eager to see their revised version. Btw, for [3] (i.e. https://arxiv.org/pdf/2307.05284), I am pointing out their intervention part (i.e., Section 5 in their latest arXiv version instead of the conference version), which also discusses some potential feature-based interventions and can be of separate utility to the authors.
I acknowledge and thank the author for the response. In my review, my concern over the usefulness of the structural causal model is addressed. I recognize that the non-existence of an unbiased causal effect estimator for infinite robustness is expected where bounded distribution shift is considered, which is also part of the contribution. Therefore, I raise my score from 6 (weak accept) to 7 (accept).