Out-of-Variable Generalisation for Discriminative Models
Authors: Siyuan Guo
Jonas Bernhard Wildberger
Bernhard Schölkopf
Venue: ICLR 2024 poster (2024)
Keywords: Out-of-Variable Generalization
Causality
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: The ability of an agent to do well in new environments is a critical aspect of intelligence. In machine learning, this ability is known as $\textit{strong}$ or $\textit{out-of-distribution}$ generalization. However, merely considering differences in distributions is inadequate for fully capturing differences between learning environments. In the present paper, we investigate $\textit{out-of-variable}$ generalization, which pertains to an agent's generalization capabilities concerning environments with variables that were never jointly observed before. This skill closely reflects the process of animate learning: we, too, explore Nature by probing, observing, and measuring proper $\textit{subsets}$ of variables at any given time. Mathematically, $\textit{oov}$ generalization requires the efficient re-use of past marginal information, i.e., information over subsets of previously observed variables. We study this problem, focusing on prediction tasks across environments that contain overlapping, yet distinct, sets of causes. We show that after fitting a classifier, the residual distribution in one environment reveals the partial derivative of the true generating function with respect to the unobserved causal parent in that environment. We leverage this information and propose a method that exhibits non-trivial out-of-variable generalization performance when facing an overlapping, yet distinct, set of causal predictors. Code: https://github.com/syguo96/Out-of-Variable-Generalization

Published as a conference paper at ICLR 2024
GENERALIZATION FOR
DISCRIMINATIVE MODELS
Siyuan Guo
Jonas Wildberger
Bernhard Schölkopf
ABSTRACT
The ability of an agent to do well in new environments is critical aspect of intel- ligence. In machine learning, this ability is known as strong or oftto--distribuution generalization. However, merely considering differences in distributions is inadequate for fully capturing differences between learning environments. In the present paper, investigate variable generalization, which pertains to agent's generalization capabilities concerning environments with variables that were never jointly observed before. This skill closely reflects the process of animate learning: we, too, explore Nature by probing, observing, and measuring proper subsets of variables at any given time. Mathematically, oov generalization requires the efficient re-use of past marginal information, i.e., information over subsets previously observed variables. We study this problem. focusing on prediction tasks across environments that contain overlapping, yet distinct, sets of causes. We show that after fitting a classifier, the residual distribution in one environment reveals the partial derivative of the true generating function with respect to the unobserved causal parent in that environment. We leverage this information and propose a method that exhibits non-trivial vuriil-v generalization performance when facing an overlapping, yet distinct, set of causal predictors. Code: https:/ / / thub. com/ http//yyuuu/// 96/ Outttofttffff Var Varias iiail Genera zat ionioo
INTRODUCTION
Much of modern machine learning can be viewed as large-scale pattern recognition on suitably collected independent and identically distributed (i.i.d)) data. Its success builds generalizing from one observation to the next, sampled from the same distribution. Animate intelligence differs from this in its ability to generalize from one problem to another. The machine learning community studies the latter under the term out-of-distribution (OOD) generalization (Shen et al., 2021; Parascandolo al., 2021; Ahuja et al., 2021: Krueger al., 2021; Zhang et al., 202 b:a;;;a; Schölkopf, 2022), where training and test data differ in their distributions. However, differences in distributions do not fully capture differences in environments. In the present work, we investigate generalization across environments where different sets of variables are observed, referring to the problem as out-of-variable (OOV) generalization. While in practice we would expect that many real-world situations exhibit both aspects of OOD and OOV, we note that the OOV problem can occur even there is no shift in the underlying distribution. In the present paper, we will focus on this setting.
OOV generalization aims transfer knowledge learnt from set of source environments to target environment that contains variables never jointly present in any of the sources, or even not present at all. OOV is ubiquitous problem in inference. Scientific discovery synthesizes information and generalizes both Medicine is field where machine learning is thought have great potential since it can learn from millions of patients while a doctor may only see few thousand during their lifetime. However, we face strong limitations in guaranteeing dataset consistency. To begin with, patients have unique circumstances, and some diseases/sympton are rare. More generally, medical datasets come with different variable sets diagnostic measurements greatly vary across patients (serum, various forms of imaging, genomics, proteomics, immunoassays, etc.). A good doctor, however, will be able to generalize across patients even the measured variables are not identical. In practice, data scientists
and out-of-variable (Seneviratne et al., 2018; Hey et al., 2009).
" "Correspondence to: siyuan. wwaaunnguoottao www.guuuuuuungeeee .mp.biinnnn.g..........m mpg.ppg..... .de 1 University of Cambridge, United Kingdom 1 Maa Planck Institute for Intelligent Systems, Tübingen, Germany

---

Published as a conference paper at ICLR 2024
end up using imputation or simply ignoring aurely-measured features. To realize the potential of AI in medicine, we need to understand the OOV problem.
To provide context, we briefly discuss pertinent research threads.
Missing data Rubin (1976) refers to when covariates are missing for individual data points. Most approaches (Donner, 1982; Kim and Curry, 1977) either omit data that contain missing values or perform imputation. Our problem differs in that some variables are missing in entire environments.
Transfer learning studies how to re-use previous knowledge for future tasks. Recent work focused on transferring re-usable features (Long et al., 2015; Oquab et al., 2014; Tzeng et al., 2015) or model parameters (Dodge et al., 2020; Sermanet et al., 2013; Hoffman al., 2014; Cortes et al., 2019; Wenzel et al., 2022) of discriminative models. Deep learning based approaches (Meyerson and Miikkulainen, 2020; Reed et al., 2022) embed variable relationships as proximity in the latent spaces, Our work presents a theoretical study on OOV generalization showing that without additional assumptions, the discriminative OOV problem not solvable: marginal consistency between source and target discriminative models does not uniquely determine a solution.
Marginal problems in the statistical (Vorob' ev. 1962: Sklar, 1996) and causal literature (Mejia et al., 2021; Gresele et al., 2022; Janzing, 2018; Janzing and Schölkopf, 2010; Evans and Didelez, 2021; Robins, 1999), on the other hand, study how to merge marginal information from different sources. Concrete methods may involve searching for joint distribution that is consistent with marginal observations. The elegant work of Mejia et al. (2021) uses the maximum entropy principle to infer joint distributions compatible with observed marginal datasets; Gresele et al. (2022) study the existence of consistent causal models; Janzing (2018) aims to learn useful causal models that can predict properties of previously unobserved variable sets. Note that inferring the joint distribution for prediction tasks may be inefficient, in line with Vapnik's 's principle (Vapnik, 1999): given some task, one should avoid solving a more general problem as intermediate step. Our work takes different approach, showing that learning from residual error distribution is sufficient to achieve identifiability in nontrivial discriminative OOV scenarios.
Causality has been argued to be related to the issue of generalization across domains (Zhang et al., 2015; Schölkopf et al., 2011; I; Arjovsky et al., 2019; Pearl and Bareinboim, 2022). Distribution shifts between domains can be modelled as sparse causal mechanism shifts (Bengio et al.. 2019; Schölkopf, 2022; Perry et al., 2022), and the correct causal structure may help efficient modular adaptation (Parascandolo et al., 2018; Goyal et al., 2020). Other work considers domain differences as shifts in spurious correlations or aims to learn invariant causal information, robust across environments (Schölkopf et al., 2011: Peters et al., 2016; Rojas-Carulla et al., 2018; Heinze-Deml et al., 2018; Arjovsky et al., 2019; Jiang and Veitch, 2022; Krueger et al., 2021; Parascandolo et al., 2021; Ahuja al., 2018; Pfister et al., 2019; Rojas-Carully
et al., 2021; Lu al., 2021; Ceinze-Demi 2018). Causality approaches also include the transfer of causal effects across different experimental conditions (Pearl and Bareinboim, 2022; Bareinboim and Pearl, 2013; 2016; Degtiar and Rose, 2023). In the present paper, we highlight another connection between causality and generalization, where domain differences are due distinct sets of variables contained within them, and causal assumptions allow us to generalize even to variables that we have never seen during training.
al.,
Out-of-variable generalization studies the efficient re-use of marginal observations. We do not solve this problem, neither do we present a robust algorithm for real-world settings. We do present a cooncep.- proposing a setting and a predictor provably capable of leveraging additional information beyond what is typically used by discriminative models. We do so without the need for inferring joint distributions. Our main contributions are:
We contextualize ($ 1) and study OOV generalization ($ 2).
We investigate challenges for common approaches (e.g., transferring reusable features or model parameters) to solve discriminative OOV problems ($ 3.2). We show (Theorem 1) marginal consistency condition alone does not permit identification of the target predictive function for common OOV scenarios.
We study the identification problem in OOV scenarios when source and target covariates have dependent ($ 3.3.1) and independent ($ 3.3.2) structures. We find that the moments of the error distribution in the source domain reveal the partial derivative of the true generating function with respect to the unobserved causal parents ($ 3.3.4).
2

---

Published as a conference paper at ICLR 2024
(b) Proposed
(c) Oracle
X1
X
X1
X2
X
(a)
(d) Marginal
(e) MeanImputed
Figure 1: Example of an OOV scenario: (a) the blue box includes observed variables in the source domain, and the orange box those in the target domain. A directed edge represents a causal relation- ship. With Y not observed in the target domain, the goal is to predict Y in the target domain using the source domain. (b)-(e) shows an example of contour lines of various methods' prediction on E|Y| X2, X3)- Our proposed predictor (b) results in close match with the true expectation (c), an oracle solution trained as if we have sufficient data and observe all variables of interest. In contrast, marginal and mean imputed predictors (d, e) deviate far from the true expectation (for details, cf. 4).
We then propose an OOV predictor and evaluate its performance experimentally ($ 4), showing that our approach achieves non-trivial degree of OOV transfer.
Fig. I provides toy example of our problem. At first glance, it would seem all but impossible to have any transfer from the source environment (blue box) to the target environment (orange box) about information that under certain causal and functional assumptions, there is a previously overlooked source of information in this OOV setting, making it possible after all.
unobserved variable in the source. The goal of the present paper is to show
OUT-OF- VARIABLE GENERALIZATION
Denote by X a random variable with values x, P a probability distribution with density p. Consider an acyclic structural causal model (SCM) M consisting of collection random variables and structural assignments (Pearl, 2009)
X; = (PA, U.), i 1 1,
where PA: are the parents or direct causes of X, and U/ are jointly independent noise variables. Given an SCM M, one can define its corresponding directed acyclic graph (DAG) where the incoming edges for each node are given by its parent set. A joint distribution generated from some SCM M with DAG G allows the Markoy factorization
22
p(r1..... ............. Fn) II
ppa pay) 1=1
(2)
where pa are parents of X; in G. The factors ("mechanisms") in (2) are postulated to be independent: Principle (Independent Causal Mechanisms (ICM) (Peters et al., 2017)). A chunge in one mech- anism p(z: ) does not inform (Guo al., 2022; Janzing and Schölkopf. 2010) or influence (Schölkopf et al., 201 I) any of the other mechanisms p(æ, ppa;) (i + j).
Before defining 00V generalization, we begin by motivating the problem. Probabilistic representa- tions (such as the Markov factorization Eq. 2) have been argued to offer advantages for probabilistic inference (Koller and Friedman, 2010) and interpretability. We highlight that in addition, the Markov factorization frees us from the need of observing all variables of interest at the same time:
Observation (Estimating the joint via causal modules). Suppose that we have knowledge the causal DAG and would like estimate the joint density p. Provided that for each variable Xt. observe an environment containing X; and its causal parents, we can recover the joint density by multiplying (according Eq. 2) the conditionals p(a; paa) estimated separately in the environments.
3

---

Published as a conference paper at ICLR 2024
This phenomenon also occurs in undirected probabilistic graphical models, where the joint density p(Z1,.... ,x) = IeEeliqus (c( is recoverable given potentials learnt from environments that contain the variables appearing in each clique. The above are the simplest cases of OOV generalization, yet they already illustrate that causal assumptions can help. We will study more subtle case below.
To this end, we model an environment & = (D, as a domain D and a task T. The domain contains variable space X' == (X1, X2.... and its joint probability distribution P(X). Given a domain, task contains a target variable space y := Y, and a predictor f : X V. To differentiate between components belonging to the source and target environments, subscripts and are used.
Definition 1 (OOV Generalization). OOV uncertainty arises when the variable space of the tar- environment is not contained in any of the variable spaces of the source environments, i.e., dss (2, & Ve). If a method for estimating a quantity in the target environment (e.g. a predictor fi) improves by utilizing data from the source environments, we say it generalizes 00V.
While there nothing causal about OOV generalization. we use SCMs since it turns out that they allow the formulation of assumptions and methods that provably exhibit OOV generalization.
3 RESIDUAL GENERALIZZATIO UNDER CAUSAL ASSUMPTIONS
3.1 PROBLEM FORMULATION
For simplicity, we consider a univariate setting, referring Appendix C.I for multivariate extension. Consider an SCM with additive noise (Hoyer et al., 2009)
Y := <( X2, X3) +
with function o. jointly independent causes X,, and (0 o²). Assume that do not have access to an environment jointly containing all variables (X1, X2, Xa, Y). Instead, we have (Fig. la):
A source enviromment with jointly observed (X1, Y). and
A target environment with jointly observed (X2, X3), and unobserved Y.
Our goal is to predict Y given X3.
This OOV scenario posits two challenges: without joint observations of (X2, X3, Y), cannot train or fine-tune a discriminative model the target environment; further, due to the independence among the covariates, it is impossible to infer X3 from the covariates observed in the source environment. Fig. 2a shows a visualization of the problem.
To ground the problem in the real world, consider two medical labs collecting different sets of variables. Lab A collects X; lifestyle factors and X2 blood test; Lab B, in addition to X2, collects X3 genomics. Lab A hospital-based and can measure diseases Y. whereas is research lab. The OOV problem asks: given model trained to predict Y on Lab A's data, how should Lab B use this model for its own dataset that differs in the set of input variables?
3.2 CHALLENGES DISCRIMINATIVE OOV GENERALIZATION
Transfer learning often transfers reusable features or model parameters of a discriminative model fitted the source environment. This approach has inherent limitations when it comes OOV scenarios: lacking the outcome variable Y, we cannot fine-tune in the target environment; further the common features, in our case, are the common variables X₂ shared between environments. naive approach is then to predict the target sample using the model restricted to X2, i.e., the marginal predictor, cf. our experiments (S 4). However, such a predictor yields constant prediction irrespective of changing X3-
To leverage the shared variables further, we study the properties that the optimal target predictor expected to satisfy to restrict the set of potential target predictors. To see this concretely: suppose data sufficient and one can observe all variables of interest, training discriminative models on each environment yields the optimal predictive functions that minimize the mean squared error loss
f. x22 = Ex, Y
x1
44)
xz) Ex. Y 12, 23
(5)
4

---

Published as a conference paper at ICLR 2024
(a) Marginal to Marginal
(b) Marginal to Joint
(c) Merge datasets
Figure 2: Examples of OOV scenarios where marginal consistency condition alone (6) does not permit the identification of the optimal predictive function in the corresponding target domain.
for the source and target environment, respectively. With the discriminative model fa fitted in the source environment, its residual distribution the distribution of differences between the observed value and the prediction, Y f&( 1,1,1.X9) X1, X2. Note that the optimal predictive functions fas fe automatically satisfy the marginal consistency condition (see Appendix A): for any 22, we have
Ex,
Ex, (fe X3)]
Suppose we have trained the optimal predictor in the source environment. The marginal consistency condition (6) enforcing consistency over the shared variables, then restricts the solution space for the predictor in the target environment. However, Theorem shows that this restriction does not uniquely determine the target predictor, i.e., it does not permit identification of the optimal predictive function in the target environment for all the scenarios shown in Fig. 2. See Appendix C.2 for the multivariate version, and proofs.
Theorem 1. Consider the OOV scenarios Fig. 2, each governed by the SCM described $ 3.1. Suppose that the variables considered in Fig. 2a and Fig. 2b are real-valued and the variables X and X3 in Fig. 2c are binary. We assume that for all i, the marginal density pi(x;) is known, and denote its support set S {I IR pc(x) > 0}. Suppose that for all there exist distinct points v, x' E Sy. Then, for any pair Ja St satisfying marginal consistency (6) and for any R > O, there exists another function Sí with [[Se Jílla 2 R that also satisfies marginal consistency.
3.3 IDENTIFICATION IN OOV GENERALIZATION
We now study when identifiability of the optimal target predictive function can be achieved. To start with consider a different setting than our setup (Fig. 1), where covariates between source and target environments are independent and causes of the outcome variable contained in the target environment.
3.3.1 WITH DEPENDENT COVARIATES
Theorem Consider a target variable Y and its direct cause PAy. Suppose that we observe:
source environment contains variables (2, Y); training a discriminative model on this environment yields fanction [1(2) = B[ Z).
target environment contains variable PAy
Suppose Y == o(PAy) + EY, = (PAy + ez where g is known and invertible with 6.9-* uniformly continuous. Then in the limit of [Z]] + 0, the composition of the discriminative models in source environments also approaches the optimal predictor, i.e., Ypay fs g(pay) + <(pay).
Appendix C.3 details its multivariate statement and proof and Fig. 5 in Appendix shows an example of such scenario. Informally, Theorem states that one can identify the optimal target predictive function from the learnt source function in our setup if the dependence structure between the source and target covariates known and satisfies the above assumptions. However, real-world applications, the dependence structure between the source and target covariates may not be known even exist. To further understand this OOV problem, we next study a more challenging scenario when all covariates independent from each other. and demonstrate seemingly surprising result, that under certain assumptions, the optimal target predictive function is identifiable without the knowledge of the dependence structure among covariates.
5

---

Published as a conference paper at ICLR 2024
3.3.2 WITH INDEPENDENT COVARIATES
With theoretical results on the limitations of current approaches in transferring with discriminative model for OOV scenarios, we present a practical method for the base case illustrated in Fig. Ia and detail its underlying assumptions.
Simple Additive Model One solution to tackle the problem in Fig. la is to train separate discrimina- tive models for each observed variable. For example, given the source environment, we learn function mappings on (X1, Y) and (X2, Y) as f2. When facing a different set of variables, e.g., (X2, X3). we could directly re-use the learnt fz. With additional collection of Y in the target domain, we then train model on (X3, Y). The method offers degree of compositional flexibility and circumvents the need to jointly observe variables of interest, e.g., (X2, X3, Y). However, such a method first requires the collection of variable Y in the target domain and assumes that the generating function of Y has only linear relationships with its causes (i.e., there is no interaction term like X,X,, * j). A detailed description of the model and its underlying assumptions can be found in Appendix B.
Below, we propose a method to transfer in OOV scenarios that 1) does not require us to observe Y in the target domain, and 2) relaxes the linearity assumption. Note the main idea is general work for all scenarios in Fig. 2. We illustrate our method via an example ($ 3.3.3) with details $ 3.3.4.
3.3.3 MOTIVATING EXAMPLE
Consider the problem described $ 3.1 in the case where & is a polynomial:
(7) Let X, have mean pii variance a, for all Given sufficient data and the observation of variable Y in the target environment, we train discriminative models in each environment, yielding the optimal predictive functions that minimize the mean squared error as:
:= X1 02 + 0x + x4X1X2 + as X1 + og X2X3 an X7X1X2X3 +
(21, 22) (as/1s) (01 + asp3)) + (o2 + 06/4))22 (a4 + 07H3) Se fe(22,23) v2) = ((1/4) + (og + os/1)=3 + (a2 + 04/41)22 + (as + 07/41 7223
(8)
We first illustrate fine-tuning, this example, cannot identify target predictive function. Note coefficients {a;} are model parameters. We observe the coefficients for the common term 32 share some constituents between fa and ft in (8). One can thus expect, during fine-tuning, the coefficients may adapt quickly. However, it is clear that one cannot uniquely determine the coefficients of fo without observing from the target environment, since the system of equations is under-determined with eight unknown coefficients and four estimated values even in the above polynomial case.
'No noise' regime First assume that there is no noise, i.e. c = 0. Although we do not observe the cause X3. nevertheless have information about it in the source environment: The unobserved variable act as a noise term, and the residual distribution in the source environment carries a footprint of it. We will below that subject to suitable assumptions, this idea carries over to the noisy case.
'With noise' regime Now consider additional additive noise. We will see that the idea outlined above carries over under suitable assumptions. The third moment from the residual distribution the source environment takes the following form:
E [Y - (x1, æ1, x2] = (as + ag5T1 + ag + 2721 [[ - 13)91
(9)
We observe that the term in parentheses coincides exactly with the partial derivative, i.e.,
O
= 03 + Q551 + ag + 0777122
(10)
ax,
Under @ in (7) as polynomial, we know terms in the source environment with non-zero coefficients are g(I11, .22 = [1, 11, T2, 22]]]. One can then fit a linear model with features in g on the source environment and estimate the coefficients. The resulting predictor is fa (= 22)) = 8 9()1,2)), x2), where 31 = a3/43, 32 = a2 05/43, 3g Q2 + 06/3, 31 = a4 + 07//3. It is clear that learning alone cannot uniquely determine the coefficients a; This intuition is supported by Theorem 1. To illustrate the main idea of our method, consider the error in the source environment after fitting linear predictive model fa: Y f2(11 x2). Let W be some transformation of the error, where

---

Published as a conference paper at ICLR 2024
W ff 22)) and Ky (X3 [s)' estimated by observed X3 samples in the target environment. Fit against (0" (011 x2)) and estimate the coefficients 8, as shown in (9), enables the estimation of the coefficients 03 at5, ag a7. Combined with the estimated coefficients 3, can uniquely determine the coefficients of without the need to observe Y from the target environment.
Discussion The intuition behind this seemingly surprising result is rather straightforward X3 though unobserved in the source, is a generating factor of Y. Its information not only contained in the marginalized mean but also in the residual distribution of the error after fitting a discriminative model.
3.3.4 OUT-OF-VARIABLE LEARNING
This phenomenon is extendable to more general settings. Theorem 3 shows that the moments in the residuals still provide additional information about the partial derivative of the function é w.r.t X3. for general nonlinear smooth functions. Appendix C.4 shows its multivariate statement and the proof. Theorem 3. Consider the problem setup $ 3.1 and assume the function is everywhere twice differ- entiable with respect to X3. Suppose from the source environment we learn a function [=(11, x2) = E|Y Fı, Ø2). Using first-order Taylor approximation on the function Φ : 21 X 32 A' + for fixed 21, 22, the moments of the residual distribution in the source environment take the form
n
B[(Y 22))" [2,
Σ k=0
7
11
E
00
B[(X3 - 10a3-1a)0-3 11a)==k) (11)
0Xg 3
For = 3, this reduces to
0Φ
ff x2] =
EI((3s - [4a)* + BB
(12)
0X3 11.22.188
Theorem shows that the moments of the residual distribution include contribution from both the moments of the noise variable and the propagated effects caused by variables unique to the target environment. When n = 3, most terms that involve the undesired noise variable disappear.
Corollary For OOV scenarios described in $ 3.1, learning from the moment of the error distribution allows exact identification of when 2221, Ep.g c,h(x1, P22) where p.q {0, 1} and C E R, vi and h can be any function.
To Corollary 4 in action, recall when é is in (7), our solution is analytically exact, as shown in (10). Theorem 3 and Corollary 4 demonstrate that in this challenging OOV scenario where existing transfer learning methods fail to apply (cf. Theorem 1), learning from the residual distribution offers exact identification for a certain class of generating functions.
Next, we build a practical predictor that utilizes the above theoretical insights and present experimental results to evaluate OOV learning performance. To start with, note the target predictive function can be Monte Carlo approximated if the true function & known:
fe(22 = / ((I1, x2, (s) dr R
Σ 11, where I1,; ~ p(x1)
Assume smooth, by first-order Taylor approximation evaluated at (I1, T2, ua), rewrite as:
do
X2, (( 22, p3)
(I3 - maa O -2a)
(13)
0X,
21.22.2
Taking expectations of X3 on both sides of Eq. 13, we see that fe (x1, Re Φ(x1, #2 u3). Theorem 3 states that estimate the partial derivative term from the third moment of the error distribution. We thus propose MomentLearn, an OOV estimate fo for the target predictive function:
fr 13)
(2) + ho( (2) - pa), with ø1,5 ~ p(x1); n
(14)
where he is MLP parameterized by 0, modelling the partial derivative by regressing the 3rd moment of the residual distribution from the source. Note the proposed predictor is strictly better than naīvely marginalizing fa on the shared variables. The proposed estimate also satisfies the marginal consistency condition as the second term in (14) vanishes when taking expectations. See Algorithm in Appendix D.3 for a detailed procedure.
7

---

Published as a conference paper at ICLR 2024
Σ, a,X,
Eies X,
+ +E,
Oracle
0.31 0.15
0.26 0.44
0.57 0.32
Momentl.carn
0.32 + 0.15
0.31 + 0.48
0.71 + 0.37
MeanImputed
0)44 + 0.21
0.48 + 0.45
1.50 + 1.03
Marginal
0.52 0.45
0.75
1.75 1.27
(P]( 03
19Pa(X1, X2) X3
+GPpsX1, X2)
Oracle
0.06 0.04
0.08 0.06
0.18 0.14
MomentLearn
0.06 + 0.04
0.10 = 0.06
0.67 + 0.59
MeanImputed
0.37+ 0.26
0.32 + 0.25
1.31 + 0.95
Marginal
0.33 0.14
0.41 0.37
1.46 1.21
Table 1: Our method's ("Momentl_earn" OOV prediction performance in the target environment, compared to the "Marginal" baseline, the predictor that imputes missing variable with mean ("Mean Imputed") and the solution that has access to the full joint observations on the target domain ("Oracle"). Shown are mean and standard deviations of the MSE loss between the predicted and observed target values. GP;(-) denotes function sampled from a Gaussian Process with zero mean and Gaussian kernel. MomentLeam performs as expected by our theoretical results and even exhibits degree of robustness to function classes that are not covered by Theorem 3.
EXPERIMENTS
We perform both synthetic and real-world (Appendix D.4) experiments to evaluate our algorithm's OOV learning performance. OOV in this context means that X3 is not observed in the source, and we do not observe Y in the target environment. We generate synthetic data according to $ 3.1 for a range of function classes. The inputs X E TR3 are independently generated from Gamma distribution. Variable Y is function the inputs, and the observed values are generated with noise, You. = Y +, where e (O0 o²), 0.1. We benchmark our method's performance against several baselines. As measure of performance against an oracle solution, we compare with the predictor trained from scratch if we jointly observe all variables in the target environment on large data sets. To highlight the need to predict beyond marginalizing learnt models on the common variable, we compare with the marginal predictor. To benchmark against mean imputation method for missing data problem. we compare with the mean imputed predictor. See Appendix D. for implementation details and Appendix 1 on robustness analysis with respect to hyperparameters (D.2), varying noise scales (D.5) and heavy-tailed noise distribution (D.6).
Prediction Performance To evaluate our method's performance, we compare contour plots of prediction on the target variable Y given covariates X2, Xa in the target environment. Its functional relationship is as described in Eq. 7. Fig. (b)-(e) shows that our method's solution is almost identical to the oracle solution, whereas the marginal and mean imputed predictor deviate far from the oracle.
Systematic Analysis To systematically analyse the robustness of our method w.r.t different function classes, compare our method's prediction against the marginal, mean imputed and oracle predictor on increasingly more complex functions. We start with base functions with linear additive terms, i.e., consider functions that additionally fX,,, X2, X3) E; o;X;, In addition base functions, incorporate linear interaction terms, i.c., (2(X,, X2, Xz) = (X,,, X2, Xz) + +Zig 8ijxXX, For the final function class, we incorporate additional square terms fa(X,, X₂) = X2, >. X.?. We use 10k data in the source environment and randomly sample 5 functions in each function class and average the results after hyperparameter sweep. Table I records the mean and standarad deviation of the MSE loss between the predicted and observed target values for different methods. We observe the proposed MomentLearn performs comparatively with the oracle predictor and consistently outperforms both the marginal and mean imputed predictors in its accuracy and relia- bility even for the function class fs, which the method is not guaranteed identify. We further evalu- ate more general functions, where we sample functions randomly generated by Gaussian processes with Gaussian kernel. Just above, observe similar result which supports our theoretical results.
8

---

Published as a conference paper at ICLR 2024
Pronood Marg
Pricool
Par ca
LAKE
107
10
32'
plsemphe
(a) Polynomial
(b) Nonlinear
Figure 3: Shown are mean of the relative loss (and its 95% confidence interval) for varying numbr of joint samples observed in the target domain. MomentLearn outperforms the joint predictor in the few-sample region and is always preferred over the marginal predictor.
Sample Efficiency To evaluate our method's comparative advantage in terms of sample efficiency had we observed Y in the target domain, we consider few-shot learning setting. Suppose we observe few samples with joint variables (X2, X3, Y) the target environment. We generate the target variable Y as a function of the inputs, where the function takes either polynomial or nonlinear form:
Ynoty a Poly(())) Y Ynontinear v(aa (VaTX) where X standardized covariates, Poly (X) polynomial features as in Eq. 7, the coefficients a ~ (V I) and o2 denotes elementwise square operation. Fig.3 shows the mean of the relative loss (and its 95% confidence interval) over 5 runs, log(loss/loss, where loss is the OOV loss of the respective method trained on 100k points from the source environment, and loss, is the loss of the joint predictor trained with varying numbers of joint observations in the target environment. When the relative loss is zero (dashed black line), the method is on par with the oracle predictor. When the relative loss below zero, the respective method achieves a lower loss than the joint predictor trained on actual joint samples. We see that the proposed method always outperforms the marginal and mean imputed predictors. Both predictors never outperform training from scratch, irrespective of the number of joint samples observed. The proposed method beats the joint predictor until about 100 joint points are used.
CONCLUSION
We used Fig. to suggest that it would seem hard to enable transfer from the source environment (blue box) to the target environment (orange box). We supported this intuition by Theorem 1. However, we also showed that under certain assumptions (e.g., the variables follow causal graph with additive noise, and the functions in the data generation process smooth), there valid source of information in this OOV scenario, enabling exact identification in certain function classes. We also proposed an algorithm to utilize this information and showed experiments in which the algorithm exhibited a degree of robustness with respect to violation of the theoretical conditions.
We only considered the error of a target predictive function as a performance condition in Def. 1. We note that in the field of OOD generalization, larger variety of settings has been considered (Wildberger et al., 2023). We briefly discuss the extension of OOV in nulti-ennvronments (E.1), its robustness (F.2.1) 1) and its potential applications (F.3). Clearly, real-world problems require systems to achieve both OOD and OOV generalization. AI in medicine, for example, requires us to tackle the OOV problem for rare discase prediction and sample-efficient generalization.
We are far from being able to claim robust methods for real-world practical problems - the present contribution lies mainly in opening up avenues for future research. Some of them are of a conceptual nature, and some connected to the limitations of the present approach. We consider this work conceptually novel, exploring how generalization is intricately related to observability of variables and their (causal) relationships. We hope our work inspires further studies to explore and develop methodologies that apply to different O0V problems.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
B.S. would like to acknowledge a number of discussions with Dominik Janzing during the last decades that helped him understand the role of additional variables in causal modeling. S.G. would like to thank Vincent Berenz for his valuable contribution in systematizing the codeing framework for broader usage.
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, -han-Chrrsttppee Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, 34:3438-3455 2021.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: (tt.::9970009999 2019.
Elias Bareinboim and Judea Pearl. general algorithm for deciding transportability of experimental results. Journal of causal Inference, 1(I):107-134, 2013.
Elias Barcinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(27):735--335, 2016.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta- teca--raansfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv: 1901. 10912, 2019.
Corinna Cortes, Mehryar Mohri, and Andrés Muñoz Medina. Adaptation based on generalized discrepancy. Journal of Machine Learning Research, 20(1):1-30, 2019. URL http:// //mlr. jjmlr. org/ hpppppp/s2//11155 92 html.
Irina Degtiar and Sherri Rose. A review of generalizability and transportability. Annual Review of Statistics and its Application, 00.501-524, 2023.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith.
Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06. 06305, 2020.
Allan Donner. The relative effectiveness of procedures commonly used in multiple regression analysis for dealing with missing values. The American Statistician, 36(4):378-385 1982.
Robin J Evans and Vanessa Didelez. Parameterizing and simulating from causal models. arXiv preprint ar arXiv:2109.084 0.699, 2021.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In International Conference on Learning Representations, 2020.
Luigi Gresele, Julius von Kügelgen, Jonas M. Kübbler, Elke Kirschbaum, Bernhard Schölkopf, and Dominik Janzing. Causal inference through the structural causal marginal problem. 2022. URL http:/ /ttp///axvvvvv.////v//ee///.aav. orgg/bs//////ooooooooo....o....r 220222222222 01300.
Siyuan Guo, Viktor Tóth, Bernhard Schölkopf, and Ferenc Huszár. Causal de finetti: On the identification of invariant causal structure exchangeable data. arXiv preprint (5X.:22033.15566 2022.
Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlincar models. Journal of Causal Inference, 2018
Tony Hey, Stewart Tansley, and Kristin Tolle, editors. The Fourth Paradigm: Data-Intensin Scientific Discovery. Microsoft Research, Redmond, Washington, 2009. ISBN 978-0-9825442. 0-4. URL httpa: www. mi microfoft. com/ Fourth-paradigg--de wwwtmmmaaaaatttnninaninnenn intensi sci wtt.:eeeeiiiiiii-eeeeeeeeoo i scoverrry
10

---

Published as a conference paper at ICLR 2024
Judy Hoffman, Sergio Guadarrama, Eric S Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko. Lsda: Large scale detection through adaptation. Advances in neural information processing systems, 27, 2014.
P. O. Hoyer, D. Janzing, J.
M. Mooij, J. Peters, and B. Schölkopf. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems 2/ (NIPS), pages 689-696, 2009.
Dominik Janzing. Merging joint distributions via causal model classes with low ve dimension, 2018.
URL httph::////// /tte://axxx waaaaaiivvv.......a/aaa org/ abs / /804. 03206.
Dominik Janzing and Bernhard Schölkopf. Causal inference using the algorithmic Markov condition. IEEE Transactions on Information Theory, 56(10):5188-5194 2010.
Yibo Jiang and Victor Veitch. Invariant and transportable representations for anti-causal domain shifts. arXiv preprint arXiv:2207.01603, 2022.
Jae-On Kim and James Curry. The treatment of missing data in multivariate analysis. Sociological Methods & Research, 6(2):215-240, 1977.
Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. 2010.
David Krueger, Ethan Caballero, Joern-Henria Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. distrrbiitt generalization via risk extrap- olation (rex). In International Conference Machine Learning, pages 5815-5826. PMLR, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference machine learning, pages 97-105. PMLR, 2015.
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant causal representation learning for out-of- -f-distributton generalization. In International Conference on Learning Representations, 2021.
Sergio Hernan Garrido Mejia, Elke Kirschbaum, and Dominik Janzing. Obtaining causal information by merging datasets with maxent. 7 2021. URL http: org/ abs/abas/1100 07 640.
Elliot Meyerson and Risto Müikkulainen. The traveling observer model: Multi-task learning through spatial variable embeddings. ArXiv, abs/2010.02354, 2020. URL https: /api. semant escholaall ar org/ :2tp:///ooouus::22440000000
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1717-1724, 2014.
G. Parascandolo, N. Kilbertus, M. Rojas- Carulla, and B. Schölkopf. Learning independent causal mechanisms. In Proceedings of the 35th International Conference Machine Learning, PMLR 80:4036-4044, 2018.
Giambatiista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2021. URL. https: //tp::://////eeveeeeeeee net. forum d=hbl wwt.shhhlssss.....o.. SDDSLVV.
Judea Pearl. Causality. Cambridge University Press, 2009.
Judea Pearl and Elias Bareinboim. External validity: From do-calculus to transportability across populations. In Probabilistic and causal inference: The works of Judea Pearl, pages 451-482. 2022.
Ronan Perry, Julius von Kügelgeen, and Bernhard Schölkopf. Causal discovery in heteroge- neous environments under the sparse mechanism shift hypothesis. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural In- formation Processing Systems, volume 35, pages 10904-10917. Curran Associates, Inc., 2022. URL
cttpppppp.cppppp..epeeeeepppppee httpp/appprrrreeeeeeeeeeeeeeeeee files/ (ttppeer__ieeepppp//2222/ pdd.

---

Published as a conference paper at ICLR 2024
J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society, Series B:947-1012, 2016.
Niklas Pfister, Peter Bühlmann, and Jonas Peters. Invariant causal prediction for sequential data. Journal of the American Statistical Association, 11 114(527):1264-1277 2019.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth- marthmmroon, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Rccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. ISSN 2835-8856 URL https: http::///ppeeeeeieeee nww. /frrt/fmmm? id-likKOkHjvj. Featured Certification, Outstanding Certification.
James M Robins. Association, causation, and marginal structural models. Synthese, 121(1/2): 151-179, 1999.
Mateo Mojas-Carully Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309 ((()):309-1133) 1342, 2018.
Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
B. Schölkopf, D. Janzing, J. Peters, and K. Zhang. Robust learning via cause-effect models. 2011.
URL https: pnttts://arxiie wwtp/vaaxvvv./////.e///ae org. www.ooo////////.ooo./........est 1112.2732
Bernhard Schölkopf. Causality for machine learning. In Probabilistic and Causal Inference: The
Works of Judea Pearl, pages 765-804. 2022.
Martin G. Seneviratne, Michael G. Kahn, and Tina Hernandez-Bousaar.. Merging heterogeneous clin- ical data to enable knowledge discovery. Pacific Symposium on Biocomputing. Pacific Symposium Biocomputiing. 24:439 443, 2018.
Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint 1rXiv:1312.664 2013.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards ouitof-distribution generalization: survey. arXiv preprint arXiv:2108. /36244 2021.
Abe Sklar. Random variables, distribution functions, and copulas: personal look backward and forward. In IMS Lecture Notes Monograph Series. Distributions with Fixed Marginals and Related Topics, volume 28, 1996.
Michael Spivak. Calculus. Publish Perish, fourth edition, 2008
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on computer vision, pages 4068-4076. 2015.
Vladimir Vapnik. The nature of statistical learning theory. Springer, 1999.
N. N. Vorob'er ev. Consistent families of measures and their extensions. Theory of Probability & its Applications, 1(2):147-163, 1962. doi: 10.1147/1107014 URL https: //doi. //tp:////..............o org/10.
1137/11070
Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel,, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, Bernhard Schölkopf, and Francesco Locatello. Assaying out-of-distribution generalization in transfer learning. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Ad- vances in Neural Information Processing Systems, volume 35, pages 7181-7198. Curran Asso- ciates, Inc., 2022. URL UKLiattpps//preceeiiss..eeeeu 2022 2tt.ee222555522592222225552533 20002///f/aaccsS00000aaaaaaaaaee
In S. Koyejo,
files/paper httpa//eppppppeeeeeeeeeeeeeeeeee
12

---

Published as a conference paper at ICLR 2024
Jonas Wildberger, Siyuan Guo, Arnab Bhattacharyya, and Bernhard Schölkopf. On the interventional Kullback-Leibe divergence. Causal Learning and Reasoning (CLeaR). arXiv:23022005380 2023.
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key out-of-distribttion generalization? International Conference on Machine Learning, pages 12356-12367. PMLR, 2021a.
K. Zhang, M. Gong, and B. Schölkopf. Multi-source domain adaptation: A causal view. In Proceedings of the Twenty-Nith AAAI Conference on Artificial Intelligence, pages 3150-3157. AAAI Press, 2015, URL http:///ve paper/view 10052/99999900 /9994.
/cw///////////////////////e/ index, http:ppIpp/......./ /
Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen. Deep stable learning for ofitt--dist generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5372-5382, 2021b.
13


Reviews:
[{"review_id": "gmtb8dbv8B", "reviewer": "Reviewer_xwQY", "summary": "The paper investigates out-of-variable generalization, namely the ability for a predictive model to generalize to target domains in which the agent has never seen the joint variables in the target domain in a single source domain before. Under certain assumptions as well as when these assumptions don't fully hold, the paper shows that the error residual distribution in an environment provides information on the unobserved causal parent variable in this environment, and they use this information to derive an algorithm that performs OOV generalization with source and target domains that have overlapping sets of causal factors.", "questions": "- The abstract states \"merely considering differences in data distributions is inadequate for fully capturing differences between learning environments.\" Doesn't out-of-variable technically fall under out-of-distribution, so shouldn't this be adequate? Perhaps more specificity is needed here.\n- The abstract states \"Mathematically, out-of-variable generalization requires the efficient re-use of past marginal information...\" Why does it require efficient re-use? Could it work with \"non-efficient\" or inefficient re-use?\n- On page 2, should \"modal\" be \"model?\"\n- On page 6, do you mean \"parentheses\" instead of \"brackets\" between Eq (9) and Eq (10)?\n- Why is the joint predictor considered an oracle predictor if MomentLearn outperforms it?\n- Could you explain why MomentLearn is reliably more sample efficient than the oracle predictor for \"few\"-shot prediction?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**\n- As far as I know, though the problem the paper addresses is well-known as a significant problem, the paper provides several theoretical results, mathematical derivations, and supports these with simple empirical results that are novel.\n\n**Quality**\n- The quality of the paper is high. It addresses a high-value problem in a principled fashion, shows how certain assumptions help obtain certain results and how and in which cases these assumptions can be bypasses while maintain approximately accurate results, and evaluates these cases in terms of loss accuracy as well as sample complexity of its approach versus baseline approaches.\n- The paper openly highlights limitations in its work, such as assumptions made for theorems to hold, and proposes prospective future work in multiple avenues. This refreshingly is (1) included at all and (2) doesn't seem like a mere afterthought.\n\n**Clarity**\n- The paper is mostly clear in its explanation of motivation, preliminaries, approach, baseline usage, results, and limitations.\n- The paper does a great job providing simple, clear real-world examples to elucidate the problem and applications of the various theorems included in multiple cases.\n\n**Significance**\n- The significance of the problem the paper addresses is high and the problem is ubiquitous. The approach is promising and can be applied in many real-world settings through Monte-Carlo sampling or similar methods. The paper shows that their approach can perform relatively well in \"few\"-shot settings though this depends on the number of variables involved and the complexity of the problem.\n\nFrom what I can tell, this is excellent work that I hope motivates further addressing this *out-of-variable* generalization problem by the research and applied AI community. My only reservation is my limited knowledge on the understanding of and state-of-the-art theoretical and applied approaches addressing this problem.", "weaknesses": "- Referring to Figure 1, in the first paragraph in page 3, the claim \"it would seem all but impossible...(orange box)\" could be better explained.\n- In Figure 1, it is unclear whether \"With $Y$ not observed in the target domain\" is an assumption made or is somehow indicated in the diagram or earlier in the paper. Eventually I realized that it's an assumption made, but the illustration Figure 1a alone isn't enough to show this assumption. This ambiguity may clear for some or compound for some later in Section 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bF4y5DIQ4i", "reviewer": "Reviewer_woi7", "summary": "This paper describes the out-of-variable OOV problem, which in its simplest form, aims to learn a predictor Y = f_t(X2, X3) given an OOV predictor Y = f_s(X1, X2) and a dataset (X2, X3), but without any instance of (X2, X3, Y). The authors describe the setting in which this is possible and develops an algorithm. The key observation is that the third moment of the residue Y - f_s(X1,X2) contains information about X3 that is least polluted by the noise.", "questions": "Intro:\n- It seems OOV fits very well the frame of missing-not-at-random and covariate-dependent missingness. Could the authors comment on that?\n\nSection 2:\n- Theorem 2 is slightly confusing for me at first glance because I thought PA_Y by definition includes all parents of Y (so x1,x2, x3 in the example) and not just those in the target environment (x2, x3). It may be helpful to clarify.\n\nSection 3:\nAs I am trying to get a sense of the restriction and applicability of the approach, I was wondering the following questions: \n- How does the method fair with the oracle as the magnitude of the noise increases? \n- What if the noise is not gaussian but more heavy tailed? \n- Does the performance degrade or improve with increasing number of variables? \n- I assume Theorem 3 does not apply to discrete variables because of the violation of differentiability; is that right?\n\nSection 4:\n- Can include missing-not-at-random imputation and covariate-missing imputation as two more baseline models (a search in Google scholar using the two key phrases yields some methods).\n- It would be really interesting if the authors could find some real-world datasets, create source and target environments by sub-setting the columns, and see how the method performs.\n- Figure 3: I don’t quite understand the figure. It would be helpful to define OOV loss, be explicit about the number of samples on the y-axis being (x2,x3,y) or (x1,x2,y) or something else. I also don’t understand why relative loss is zero means the method is on par with the oracle predictor. Why not just show how the fine-tuning error compares with oracle training, which seems easier to interpret? Anyway, I am overall a bit confused about the figure, so my questions may not make sense.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The key observation/discovery is clever, and the algorithm is straight-forward to use.\n- The writing is clear, clean, and well-referenced. The examples also made things concrete and easy to follow.\n- The rigor and simplicity of the work can act as a foundation to build OOV research.", "weaknesses": "- The main weakness is the applicability of the method. The authors only showed results for proof-of-concept, not for real-world usage. \n- It is unclear how one could identify whether the assumptions are satisfied given a dataset.\n- It is unclear how bad the predictor would be if the assumptions are not satisfied.\n- It is not yet clear what realistic problem can be well modeled by OOV generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fZAon7Cssu", "reviewer": "Reviewer_s9Ga", "summary": "This work investigates out-of-variable (OOV) generalization, which is a sub-problem to OOD generalization, and refers to scenarios where an agent needs to generalize to environments containing variables that were never jointly observed before. The paper shows that if the source and target environments contain some overlapping variables (and under certain conditions), information from the predictor in the source environment can improve predictions in the target environment. More specifically,  the moments of the residual distribution from the optimal classifier in the source environment can be used to calculate the generating function with respect to the unobserved variable in the target domain.\n\nBased on this observation, the paper proposes a practical algorithm for OOV prediction, evaluates its performance, and compares it against the marginal predictor and imputed predictor, as well as an Oracle predictor.", "questions": "I would like to ask the following questions:\n\n1. For future work, is there a more complicated/realistic dataset to validate the algorithm?\n2. Is it possible to compare the algorithm to state-of-the-art marginal or causal methods such as Mejia et al. (2021) or Janzing (2018)? To validate if Vapnik’s principle holds and whether the proposed approach indeed improves results due to solving a less general problem.\n3. Theorem 3 connects all moments of the residual distribution to the partial derivatives with respect to the unique variable of the target environment. If additional moments were to be calculated as part of the proposed algorithm, would it improve results (for the general function case)? \n4. In general, since the paper's main claim is that in the real world, it is likely to encounter both aspects of OOD and OOV - How simple is it to combine state-of-the-art  OOD methods with the proposed approach? I cannot imagine at the moment a straightforward way to do that.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper proposes a new and important problem-setting - OOV generalization, which can occur in real-world situations, on its own or alongside OOD aspects. The work also provides an extensive study of the identification problems of various variants of OOV scenarios, including theoretical proofs and examples. \n\nIn addition, the paper proposes a practical algorithm to solve several OOV scenarios that achieves non-trivial OOV transfer on synthetic data.\n\nThe ideas presented in the paper are novel and the conclusion that information from source domains can be used for prediction in the target domain in this setting is important, and can potentially have a broad impact on future research in the field.", "weaknesses": "The main limitation of the paper is that the proposed approach was tested on only synthetic data, and was not validated using more challenging datasets. \n\nIn addition, the extension of OOV in multi-environments is mentioned mainly in the appendix and the algorithm was not tested empirically for that extension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EOWYeM4s8B", "reviewer": "Reviewer_qFZD", "summary": "The paper introduces out-of-variable (OOV) generalization, which is an the ability to generalize in environments with variables that have never been jointly observed before. OOV is an issue in settings where different variables (e.g. diagnostic tests) are available for different environments (e.g. different patients). The paper investigates challenges for common approaches when faced with the OOV problem, and proposes an OOV predictor that leverage moments of the error distribution. The work contributes to theoretical understandings of OOV and offers a proof-of-concept for a predictor capable of non-trivial OOV transfer.", "questions": "Please see weaknesses above.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper formally studies a new perspective on generalization.\n- The methods employed in the paper are sound.", "weaknesses": "- The paper does not demonstrate the practical applicability of the concept of OOV generalization, and the setting feels a bit contrived. Also it seems like OOV generalization can be thought of just a case of OOD generalization--if we think about all the variables together as the input, the OOV generalization is just a case of OOD generalization (e.g. covariate shift) where some inputs have clear signal from some features and other inputs have clear signal from other features. \n- It would be helpful to include more intuitive discussion throughout the paper providing more analysis on the sections. For example, more discussion on the assumptions of the settings/theorems would be helpful, and it's not clear exactly under what assumptions the proposed predictor is appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)