Causal vs. Anticausal merging of predictors
Authors: Sergio Hernan Garrido Mejia
Patrick Blöbaum
Bernhard Schölkopf
Dominik Janzing
Venue: NeurIPS 2024 poster (2024)
Keywords: Causality
Merging of predictors
Causal vs Anticausal
Maximum Entropy
Primary area: causal_inference
TL;DR: We study the asymmetries produced in the merging of predictors whenever we have causal information.

Abstract: We study the differences arising from merging predictors in the causal and anticausal directions using the same data.
In particular we study the asymmetries that arise in a simple model where we merge the predictors using one binary variable as target and two continuous variables as predictors.
We use Causal Maximum Entropy (CMAXENT) as inductive bias to merge the predictors, however, we expect similar differences to hold also when we use other merging methods that take into account asymmetries between cause and effect.
We show that if we observe all bivariate distributions, the CMAXENT solution reduces to a logistic regression in the causal direction and Linear Discriminant Analysis (LDA) in the anticausal direction.
Furthermore, we study how the decision boundaries of these two solutions differ whenever we observe only some of the bivariate distributions implications for Out-Of-Variable (OOV) generalisation.

Causal vs. Anticausal merging of predictors
Sergio Hernan Garrido Mejia
Patrick Blöbaum
Max Planck Institute for Intelligent Systems
Amazon Tübingen, Germany
Amazon Tübingen, Germany shgn@tuebingen.mpg de
Bernhard Schölkopf Max Planck Institute for Intelligent Systems Tübingen, Germany
Dominik Janzing
Amazon Tübingen, Germany
Abstract
We study the differences arising from merging predictors the causal and anti- causal directions using the same data. In particular we study the asymmetries that arise in a simple model where we merge the predictors using one binary variable target and two continuous variables as predictors. We use Causal Maximum Entropy (CMAXENT) as inductive bias to merge the predictors, however, we expect similar differences to hold also when take into account asymmetries between cause and effect. We show that if we observe all bivariate distributions, the CMAXENT solution reduces logistic regression in the causal direction and Linear Discriminant Analysis (LDA) in the anticausal direction. Furthermore, study how the decision boundaries of these two solutions differ whenever we observe only some of the bivariate distributions implications for Out-Of-Variable (OOV) generalisation.
other merging methods that
Introduction
A common problem in machine learning and statistics consists of estimating or combining models or (expert opinions) of a target variable of interest into a single, hopefully better, model (14]. There are several reasons of why this problem is important. For example, experts might have access to different data when creating their models, but might not have access the data available other experts, while there might be a modeller who can access the expert's opinions and put them together into a single model. Furthermore, experts might specialise certain areas of the support of the input space, that modeller with access to the expert's opinions could potentially produce single model exploiting the strengths each modeller. This problem is commonly known as "mixture of experts", "expert aggregation", "merging of experts" or "expert pooling" 440 22
31 30).
The merging of experts problem is usually ill-defined, in the sense that there are multiple joint models (that is, models that include all covariates) that after marginalisation would render the same prediction as the individual experts (that is, those which include only some of the covariates). This ill-definedness of the problem requires strong inductive biases. One way to provide this inductive bias principled way through the Maximum Entropy (MAXENT) principle 200. In brief, MAXENT suggests finding the distribution with maximum Shannon entropy subject to moment constraints. This turns out to be the same as choosing the distribution closest the uniform distribution having the same moments as those given by the constraints. In Section 2.2 we introduce MAXENT and Causal MAXENT (CMAXENT) more detail, the latter being an extension that allows to include causal information when available [77]
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Most of the research on the merging of predictors focuses on finding a meta meta-predicee predictor that uses the given models and best fits to the data [41), whereas the focus of the present article is understanding the implications of causal assumptions in the merging of experts instead of aiming for models with best performance. Furthermore, most of this research focuses on predictors that use the same predicting variables for each of the models. To the best of our knowledge, the only exception is the random subspace method, notable for being the basis of random forests [1]
What if in addition to the different models, a researcher has some causal knowledge of the underlying system? For example, they could know whether a variable or set of variables used in a particular model are causes or effects of the target variable, potentially changing the resulting predictor of interest. Causal knowledge produces asymmetries that have been exploited in the past to understand some common machine learning tasks like transfer learning, semi-supervised learning or distribution
shift 35 39
21
In the present work, we investigate how including causal knowledge produces asymmetric results when merging predictors. In particular, we are interested in how solutions the CMAXENT principle [17] differ when we assume different causal relations for the same data. That is, we are interested in the asymmetries produced by causal assumptions on CMAXENT inferences. In particular, we will study the differences in the solutions when we assume the causal data generation process (so that the covariates or predictors are causal parents of our target variable) in contrast with the anticausal generation process (so that the predictors are causal children of our target variable). We are going to study these asymmetries the case where we do not observe all the variables jointly; one of the differentiating characteristics of this research with respect to other merging of predictors work.
Including the right causal assumptions when merging predictors is relevant, for instance, in the medical domain. Suppose we are interested in the presence or absence of a disease, and we have models from hospitals and labs relating risk factors and symptoms to the disease we are interested in. Combining the predictors would be valuable to predict the disease but it also requires to include the right causal assumptions, if the direction matters for the merging of predictors: risk factors cause diseases and diseases cause symptoms. The literature of merging of predictors has focused on important aspects of the resulting models like generalisation bounds or speed of estimation but the relation to causality has remained largely unexplored.
Previous approaches have considered the problem of merging of experts using MAXENT 25 However, such research considers the problem from a purely statistical perspective and does not study the ramifications of different causal assumptions. In fact, the way they study the aggregation problem is by merging the probability of the outcome given by each expert without regarding how these probabilities were produced.
The main contributions of this article can be summarised as follows
We study the differences in causal and anticausal merging of predictors whenever the inductive bias used to merge the predictors allows causal information to be included.
In particular, we find that CMAXENT with a binary target and continuous covariates, reduces to logistic regression and LDA, two classic classification algorithms, when merging predictors in causal and anticausal directions, respectively.
Furthermore, we study the implications of these asymmetries Out Of Variable (OOV) generalisation whenever we do not observe all the first and second moments as constraints in the CMAXENT problem.
The remainder of the paper is organised as follows. In Section 2 we introduce basic notation and give brief overview of MAXENT and CMAXENT. Then, in Section we present the optimisation problem in the causal and anticausal direction and give the explicit solutions of the problems, thereby connecting the solutions of CMAXENT prove that the decision boundary in the causal and anticausal directions, with full knowledge of the moments (as defined in the section itself) renders equal slopes of the predictors. In Section 4 we weaken the assumption of full knowledge of the moments and instead assume knowledge of a subset of the moments in Section 3 Partial knowledge of the moments have implications for Out Of Variable (OOV) generalisation and resulting in differences in decision boundaries. We close with Section 5 with some discussion and concluding remarks. All the proofs are left to the appendix for the sake of brevity and clarity of the main text.
well-known classification algorithms. In addition, we

---

2 Notation and preliminaries
2.1 Notation
Let Y be a binary random variable taking values in = (-1,,11 and x X2) be pair of continuous variables, so that z; R. Let f y x IR² - be a measurable function, a measure on ) x R2, and p the density of the distribution of random variable with respect to the Lebesgue measure in the case real valued random variables, and with respect to the counting measure the case of discrete random variables. To be precise, p(Y, X) is density with respect to the product of the Lebesgue measure and the counting measure. We denote Ep (Y, Y X)] the expectation of f with respect to p. We restrict ourselves the scenario with two continuous variables and one binary outcome given that we can already observe asymmetries in the merging of experts, and can visualise such asymmetries without having project such space into 2 dimensions. The results here can be easily generalised into a discrete outcome variable (and indeed we do, in Corollary 77 Throughout the article we will care about finding a predictor of Y using X covariates. That is, we interested in the density p(Y X).
2.2 Maximum Entropy and Causal Maximum Entropy
The Maximum Entropy (MAXENT) principle was born in the statistical mechanics literature as a way to find a distribution consistent with set of expectation constraints [20). That is, given observed sample averages f N Ei=1 xz) find the density pYY,X) that the expectations with respect to p(,,x X) are equal to those observed.
Notice that MAXENT does not attempt to find the 'true' distribution of the data, but instead the distribution closest to the uniform distribution so that the expectation constraints are satisfied. We will examples of such optimisation problems in subsequent sections. Using the Lagrange multiplier formalism for constrained optimisation, one can prove that the solution to the MAXENT problem belongs to the exponential family. The MAXENT distribution and its properties have been studied widely, see Grünwald and Dawid [0] and Wainwright et al. [38] and references therein.
In Causal MAXENT (CMAXENT, Janzing [T7). the optimisation is performed in an assumed causal order; that is, we first find the MAXENT distribution of causes and then the Maximum Conditional Entropy of the effects given the inferred distribution of the causes. As argued in (36] this typically results in distributions that are more plausible for the respective causal direction. One can think of
CMAXENT as usual MAXENT with the distribution of the cause as additional constraint, where the latter has been obtained via separate entropy maximization.
3 Known predictor covariances
We will begin by studying the solution the CMAXENT problem when we observe all the bivariate distributions and summarise them with first and second moments. The restriction to first and second moments has several reasons: First, these simple constraints are already sufficient to explain the interesting asymmetries between causal and anticausal. Second, including higher order moments makes the problem computationally harder and increases the risk of overfitting on noisy finite sample results. Last, including more moments decreases the asymmetries between the causal directions. Mathematically, we have the following (estimated) expectations and their respective sample averages:
[
Y =
x = Φ =
x = x
T1
00]
E
Σx =
87
812
(2)
T2
81,2
82
where we assumed the mean of X is zero
3.1 The causal direction
Consider the causal graph Figure Ta and the expectations given in Equations and 2 As
mentioned on Section 2.2 CMAXENT suggests finding the density p(X) with maximum entropy
3

---

X1
(a) Graph in the causal direction
(b) Graph in the anticausal direction
Figure I: Causal graphs analysed throughout the article
consistent with the first and second moments of X, and then finding the density p(Y X) with maximum entropy, with the estimated p(X) and the moments that involve Y as constraints.
These steps can be summarised in the following optimisation problems. First, for p(X),
H(X) f p(x) log p(x) dx
max p(x)
jga
s.t.
E[X,] = T.s with i E {1,2]
[[X?] sy with i E (1,2)
(3)
E [2] = -1.2
p(x) dx 1.
R22
On the other hand, the Maximum Conditional Entropy optimisation problem is as follows
max ply x)
H(Y x)
122 Σ
plu x))((x log p(v x) dx
B[Y Φi, with i E (1,2)
(4)
B[y] =
Σ p xx) = 1, for each X.
Where p(X) is the one we found by solving Equation (3
Proposition (Resulting predictor in the causal direction). Using the Lagrange multiplier formalism for the optimisation problems in Equations (3) and (4) we obtain: (i) a multivariate Gaussian distribution for P(X), and (üi the density of conditioned on X given by
Px I1, x2) exp (Aoy A1yz1 + A2y + xxx1
(5)
a(11,222) log Σ exp (Now + \1yE )2y(22)
(6)
where axx)) a normalising constant. The density can be written as
(y = 1 31, .2)
= - + tanh(An + Ay + A222)).
(7)
The proof of this result can be found in 119 Section 3.1 and 3.2].
Remark 2 Notice that Equation (7). our predictor interest, is just a rescaled version of sigmoid function. That is, we can estimate pYY X) with logistic regression. A similar observation was done g in the context of using an exponential loss boosting. This relation was further explored in 24, where more direct relation to maximum likelihood and exponential families was established. In Section we discuss how these results in the statistical literature can interpreted as making causal assumptions about the relation between the predictor and target variables.

---

3.2 The anticausal direction
Now consider the graph in Figure In this scenario, covariates of our predictor of interest are the effects of our target variable. Following the CMAXENT principle, we first find the density p(Y) with maximum entropy and is consistent with first moment of Y and then find the density p(X Y) with maximum conditional entropy consistent with the moments that involve X and p(Y) found in the previous step. After this two-step process, we are left with the joint density p(Y, x() from which we can derive predictor of Y, p(Y X) using Bayes' Theorem (Section 3.3). The whole procedure can be summarised with the following optimisation problems. For the cause, we have
H(Y) = Σ p() log p(y)
max p(v)
s.t.
E| q
(8)
Σ p(v) = 1. "/
And for the effects,
H(X Y) = / Σ p(x y)p(y) log 1)dx
max p(x
JR²
X| 6., with i E {1,2}
E[X,] = Ti, with E [1,2]
E[x²] = 8 with {1,2)
E X2] $1.2
p y)dx = 1, for each R
Proposition 3 (Resulting predictor in the anticausal direction). Using the Lagrange multiplier
formalism for the optimisation problems in Equations (8) and we obtain a Bernoulli distribution for Y with ply = 1) = 9. and (x y) given by
px(x v) = exp(> Aiyar + A2y Aza1 + A422
(10)
+ A
+ AB2
+ A7 + B(y)l
= exp Σ Xchu(x v) B(y)
(11)
B(v) = log
exp
Σ )nh
dx.
(12)
JR3
where byg are the different functions for which we have the sample averages. The density px Y) is mixture of multivariate Gaussian distributions. Both components Px(X y -1) and [x(X = 1) have the same covariance matrix.
For the following sections, we introduce the following notation for the expectations of the mixture of Gaussians.
|g.1 [
v p4
EXX
Y| = Σx-y
(13)
In addition, will include the subscripts "causal" and "anticausal" where it might be ambiguous (e.g., Ex|Y.cansal represents the conditional covariance in the causal scenario and antiaaaic in the anticausal scenario). As mentioned in Propostion 3 the conditional covariance Σxly is the same for both values of y. However, we keep the conditional notation to distinguish it from the marginal covariance of X, Σx introduced in Equation (2). In Appendix A we derive the conditional expectations in Equation (13) and the marginal expectations used as constraints.
Remark Even though the causal graph in the anticausal direction implies that the conditional covariance Σxy is diagonal, the CMAXENT solution does not result in diagonal conditional
5

---

covariance. This is true because of the constraints in Equations T and 2 and the law of total covariance. Note that the CMAXENT distribution not necessarily Markov relative to the given DAG. As shown in [77] Section 5]. CMAXENT only provides the best guess and may therefore mix over different Markovian distributions such that the result no longer Markovian.
This relation between the marginal and conditional expectations will be essential in the subsequent sections where we explore the difference in the decision boundaries of the two resulting predictors of Y.
3.3 The predictor of Y in the anticausal direction
Recall that our main goal is to produce predictor of Y as function of the covariates X. In Section 3.1 3.1 we obtain the predictor of Y directly as a result of the CMAXENT principle, given that the predictor is already in the direction of the causal mechanism. On the other hand, in Section 3.2 we have to derive the predictor of Y using the found conditional distributions and Bayes' rule. The main result of this section is that with the constraints direction is equivalent to Linear Discriminant Analysis [14] Section 4.3]. Furthermore, we generalise this result to Quadratic Discriminant Analysis, and to exponential family version of discriminant analysis.
have used, CMAXENT in the anticausal
Theorem 5 (Predictor of Y using Bayes' rule). Using the results from Propostion the density Px(Y = X) is the ratio of the product of the Gaussian component with px (Y y) and the
mixture of Gaussians resulting from Propostion 3 Minimising the expected 0-1 loss, the optimal
decision rule arising from this density is equivalent to Linear Discriminant Analysis (LDA).
Corollary 6 (Quadratic Discriminant Analysis (QDA)). Quadratic Discrúminant Analysis can be interpreted as CMAXENT in the anticausal direction. This is achieved by replacing
E[X²| = = with i E {1,2]. and [[ = 31,2.
(14)
in Equation 9 with the following constraints:
E E v) = bh with i (1,2), and E[ X2 v) = 81,2,y
(15)
We will now extend this idea, where instead of modelling (X) as a mixture of Multivariate Gaussians (with equal covariance in LDA unequal covariance in QDA), p(X) now becomes a mixture of distributions, each coming from an exponential family of distributions corresponding to a more general set of constraints.
Corollary 7 (Exponential family discriminant analysis). Let Js be an arbitrary measurable function and f its corresponding sample average. In the general case where Y a discrete variable and we have d covariates X in the anticausal direction, the CMAXENT problem with constraints of the form:
(Xf
| yl ul==ft f.yi
(16)
results (x Y) being a mixture of exponential family distributions which then can be inverted (using Bayes' rule) to predictor of Y.
where Fryy are the sample averages of for specific y as in Section 2.2
Remark 8 In the previous corollary, the functions fi can be constant on any of the variables in X.
This idea has been extended to use kernels as way map X into more complex feature spaces. The resulting algorithm is called Kernel Fisher discriminant analysis 26 32
3.4 The geometry of the decision boundaries
Hastie et al. [4] Chapter 4.4.5] conclude that the log- posterior odds of the logistic regression and LDA are both linear x, but with different parameters defining the linear relation. In this section, we revisit these results in more detail and explore whether the CMAXENT solution in causal direction differs from the solution in anticausal direction. From a statistical decision theory perspective, the bog-posterior odds correspond to the Maximum A Posteriori (MAP) rule, the optimal decision boundary of classifier when minimising the expected 0-1 loss [3, Ch. 4.3.3).
Proposition 9 (Normal vector to the decision boundaries in causal and anticausal direction). Under the 0-1 loss, the normal vector to the decision boundary of the CMAXENT predictor is proportional

---

1. Ex consall in the causal direction.
2. Σ x|y anticamaal @ in the anticausal direction.
We will now prove that in the case where we know all the expectations in Equations T and ( the slope of the decision boundaries in causal and anticausal direction are the same.
Theorem 10 (Slope of the decision boundary is the same in causal and anticausal direction). Using the constraints in Equations T and (2). the slope of PY X) inferred using CMAXENT is the same in causal and anticausal direction.
Although it might seem from the above result that there is no asymmetry between the logistic regression and LDA even when include causal information, this is not entirely true. To begin with, the decision boundaries may be unequal although they are parallel, but more importantly learning the parameters of certain model might be easier. In the next section explore the differences that persist even under the light of Theorem T0
3.5
What are the differences?
In the previous sections we found that the slopes of the decision boundary of CMAXENT in both the causal and anticausal direction are linear and agree, whenever we have the first and second moments as in Equations ( and (2. This implies that, if the test data will come from the same distribution as the training data, either algorithm will work equally well. Previous research has studied the advantages and disadvantages [14] 33 Chapter 4.4.5] of each method and their properties such as asymptotic relative efficiency 77 parameter bias [3. asymptotic error under label noise 4 and
online learning performance M. All of these analyses base their results the fact that the logistic regression does not make an assumption on how the covariates X are distributed, whereas LDA does.
An alternative way of viewing this distinction is through the lens of generative and discriminative models. LDA is a generative model since it models both covariates and target variable and logistic regression only models the target function of the input. Ng and Jordan (28) analyse the difference in efficiency between the Naive Bayes algorithm (a generative model similar to LDA) and logistic regression, and find that both models have regimes in which they perform better than the other. Using the same models, Blöbaum et al. [5] and data from (35], find empirically that generative models perform better in anticausal than in causal direction.
4 Partially known covariances
In this section we explore variations of the solution of the CMAXENT solution in causal and anticausal direction when some of the sample averages are not known. In Section 4. we explore the case where the covariance between a particular predictor and the target is not known, and in Section (4.2 the case where we do not know the covariance between the predictors. In both cases we will see that the models we can infer (that is, p(Y X)) with CMAXENT will depend on the underlying causal assumptions.
4.1 Unknown predictor-target covariance
Without loss of generality, suppose we do not have the sample covariance between X, and Y, that is, do not know Φe2 in Equation (I).
In the causal direction, the CMAXENT solution of the distribution the causes X will still be a multivariate normal distribution with expectations given by the constraints relating X. The conditional density of the effects is the logistic-like regression of Equation (7). however, Xg will be O, as this is the parameter corresponding to the covariance between X₂ and Y. In other words, X₂ becomes irrelevant in the estimation of our target predictor.
In the anticausal direction, the distribution of the cause Y is unchanged because P(Y) is determined by the constraints and thus does not depend on Φ2. However, using the fact that the Gaussian distribution maximises the entropy over all distributions with the same variance (37 Theorem 8.6.5.], we can derive a bound on d2. We use the entropy of the Gaussian distribution because we do not know a closed form expression for the conditional covariance of p(Y X) given by Theorem
7

---

Propasition 11 (Bounds on unknown covariance between predictor and target). Assuming the causal graph in Figure and we do not know Φ2 an upper bound for Φ2 is given by:
q
9) 1)21,
(17)
q 3) of
The bound in Propostion 11 is found by differentiating the determinant of the conditional covariance (to which the differential entropy of the multivariate Gaussian is proportional to) with respect to the unknown covariance, Φ2 in this case, and finding the value for d2 for which this derivative is O.
The implication of the previous result is that even when we have not observed any joint data between X2 and Y we can still build model of Y that depends on X2. long as we can assume the data generation process is anticausal. We can consider this an instance of Out Of Variable (OOV) generalisation studied in [16] [12,, where we can exploit causal information and partial data to make models including variables that were never observed jointly with the target.
4.2 Unknown predictor covariance
Now suppose we observe all the sample averages in Equations T and 2 but we do not observe 81,2-
In the causal direction this implies that the multivariate Gaussian distribution resulting from the MAXENT problem on X diagonal, that is, X are marginally independent. The exponential form of (Y X)) does not change, exponential family do change, as the density of X changed so that the resulting (x X) needs to adapt in order to match Φb
still observed q and Φ, nevertheless, the parameters of the
Now we will explore the anticausal case. We will proceed in Sections 3.1 and 3.2 First we find P(Y) by maximising the entropy subject to the empirical average of Y. which is trivial because p(Y) already determined by its moments, and then we find p(X Y) subject to all the moments in Equations (T) and (2) with the exception of 81,2. We obtain the following result from solving the problem
CMAXENT optimisation MMAXXENoptimisation
Proposition 12 (Diagonal conditional covariance in the anticausal direction with unknown predictor covariance). The density p(X Y) that maximises the conditional entropy subject to the following constraints:
+1 S22
[0] 0
= E
= E|X
=
[[X]]= =
(18)
and pYY inferred on the first step of CMAXENT, independent after choosing a value of y: that is, X is conditionally independent given Y.
Remark 13 This result is reassuring given that under these moment constraints, p(X y) turns out to be Markov relative to the DAG in the anticausal direction. Contrary to Remark where we concluded that CMAXENT is not always Markov relative to DAG.
In Appendix E we derive the slopes of the decision boundaries in the causal and anticausal direction when we do not know the covariance between the predictors. We also find necessary and sufficient conditions for which the slopes are the same. From this simple example, we have learned the following: in causal direction, our inductive bias tells us that the covariates are not correlated and hence, the decision boundary depends only on the marginal variance of each X; and the covariance between Y and X. In the anticausal direction, CMAXENT infers X, and X, to be marginally correlated because they need to conditionally independent (this fact can proved using the law of total covariance). Hence, the marginal covariance of X, Σx is different in both scenarios. This is something we did not observe in the case with full information (Section
In addition, we derive the expressions of the decision boundaries the causal and anticausal direction (see Appendix [ That is, as proved in Propostion 9 we have that the decision boundaries of the predictors in causal and anticausal direction will differ with the same moments, but different causal assumptions. Figure 2 we showcase this phenomenon with synthetic data.
5 Discussion
In this article we have studied the differences arising from merging predictors in the causal and anticausal directions. In particular, have studied simple case with binary target variable and
8

---

Causal (Logistic)
Anticausal (LDA)
4
4
2
2
0
0
-2
-2
-4
-4
-2.5 0.0 2.5
2.5 0.0 2.5
Figure 2: Decision boundaries of the solution of CMAXENT the causal (left) and anticausal (right) direction when we do not have the covariance between the predictor variables 81,2-
two continuous variables. Although in this simple example we have already found connections with classical classification algorithms, and differences in the solutions in causal and anticausal direction, the example can be easily extended to more covariates (where the resulting distribution of the covariates would be d-dimensional Gaussian instead bivarate Gaussian), discrete target variable (as in Corollary 7 and a causal graph that contains both parents and children as predictors.
As stated the end of Section 3.1 the relation between merging of experts and logistic regression has been explored in the past. Friedman et al. [8] interpret the solution to the AdaBoost procedureas additive logistic regression. They arrive this interpretation starting from exponential loss function. They then propose likelihood based estimator of the AdaBoost procedure. Thus, since our results align with those in Friedman al. [8], give yet another interpretation of AdaBoost the solution of the merging of experts in causal direction using the CMAXENT principle.
Even though we have used CMAXENT inductive bias merge the predictors throughout the article, we believe that the asymmetries we found here (in particular, the geometry the decision boundaries) would hold when using any other inductive bias that allows causal information to be included. Whatever method we use to merge predictors, the following asymmetry seems natural: In anticausal direction try explain correlations between X, both components. In causal direction, correlations between X, and X2 do not tell us anything about the relation between X and Y, following the principle of Independent Mechanisms (see 29 for overview and 11 for a recent Bayesian view).
result of Y influeneing
The previous observation is useful in straightforward scenarios where different sources for a supervised learning task, say datasets with overlapping variables, or datasets produced from different experimental conditions (also called environments); but also in cases where the merging of data is more subtle, for example, in federated learning where the notion of horizontal and vertical federated learning [42) coincides precisely with the data sources described above but where causality is underexplored.
merging data from
Acknowledgments and Disclosure of Funding
We thank William R. Orchard and Yuchen Zhu for their valuable comments in a previous version of this article.

---

References
[1] A. Banerjee. An analysis of logistic models: Exponential family connections and online performance. In Proceedings of the 2007 SIAM International Conference Data Mining, pages 205-215. SIAM, 2007.
[2] M. S. Bartlett. An inverse matrix adjustment arising in discriminant analysis. The Annals of Mathematical Statistics, 22(1):107-111, 1951.
[3] J. O. Berger. Statistical decision theory and Bayesian analysis. Springer Science & Business Media, 2013.
[4] Y. Bi and D. R. Jeske. The efficiency of logistic regression compared to normal diseriminant analysis under clndiiiona classification noise. Journal of Multivariate Analysis, 101(7): 1622-1637, 2010.
[5] Blöbaum, S. Shimizu, and T. Washio. Discriminative and generative models causal and anticausal settings. In Advanced Methodologies for Bayesian Networks: Second International Workshop, AMBN 2015. Yokohama, Japan, November 16-18. 2015. Proceedings 2, pages 209-221. Springer, 2015.
[6] L. Breiman. Bagging predictors. Machine learning, 24:123-140, 1996.
[7] B. Efron. The efficiency of logistic regression compared to normal discriminant analysis. Journal of the American Statistical Association, 70(352):822-898, 1975.
[8] Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statistics, 28(2): 337-407, 2000.
[9] B. Ghojogh, F. Karray, and M. Crowley. Fisher and kernel fisher discriminant analysis: Tutorial. arXiv preprint arXiv: 190660933, 2019.
[10] P. D. Grünwald and A. P. Dawid. Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. Annals of Statistics, pages 1367-1433, 2004.
[11] S. Guo, V. Toth, B. Schölkopf, and F. Huszar. Causal de Finetti: On the identification of invari- ant causal structure in exchangeable data. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, vol- ume 36, pages 36463-36475. Curran Associates, Inc., 2023.
[12] S. Guo, Wildberger, and B. Schölkopf. Out-ol-variable generalization for discriminative models. In The Twelfth International Conference Learning Representations (ICLR), May 2024.
[13] M. Halperin, W. C. Blackwelder, and J. I. Verter. Estimation of the multivariate logistic risk function: comparison of the discriminant function and maximum likelihood approaches. Journal of chronic diseases, 24(2-3):125-158 1971.
[14] T. Hastie, R. Tibshirani, J. H. Friedman, and H. Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[15] T. K. Ho. The random subspace method for constructing decision forests. IEEE transactions on pattern analysis and machine intelligence, 20(8):832-88) -844, 1998.
[16] D. Janzing. Merging joint distributions via causal model classes with low VC dimension. arXiv preprint arXiv: arXiv:1804.03206, 2018.
[17] D. Janzing. Causal versions of maximum entropy and principle of insufficient reason. Journal of Causal Inference, 91):285-301, 2021.
[18] D. Janzing and B. Schölkopf. Semi-supervised interpolation in an anticausal learning scenario. Journal of Machine Learning Research, 16: 16:1923-1948, 2015. URL. papers/v16/
org/
http://vv//annnnnna.... aaniing5a..nn
10

---

[19] D. Janzing, X. Sun, and B. Schölkopf. Distinguishing cause and effect via second order exponential models. arXiv preprint arXiv:0910.5561, 2009.
[20] E. T. Jaynes. Information theory and statistical mechanics. Physical review, 106(4))620, 1957.
[21] Z. Jin, J. von Kügelgen, J. Ni, T. Vaidhya, A. Kaushal, M. Sachan, and B. Schoelkopf. Causal direction of data collection matters: Implications of causal and anticausal learning for nip. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1499-9913, 2021.
[22] M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[23] S. L. Lauritzen. Graphical models, volume 17. Clarendon Press, 1996.
[24] G. Lebanon and J. Lafferty. Boosting and maximum likelihood for exponential models, Advances in neural information processing systems, 14, 2001.
[25] W. B. Levy and H. Deliç. Maximum entropy aggregation of individual opinions. IEEE transactions on systems, man, and cybernetics, 24(4):606-61) 613, 1994.
[26] S. Mika, G. Ratsch, J. Weston, B. Schölkopf, and K.-R. Müller Fisher discriminant analysis with kernels. In Neural networks for signal processing IX: Proceedings of the 1999 IEEE signal processing society workshop (eat. no. 98th8468), pages 41-48. leee, 1999.
[27] I. J. Myung, S. Ramamoorti, and A. D. Bailey Jr. Maximum entropy aggregation of expert predictions. Management Science, 42410):1420-1436 1996.
[28] A. Ng and M. Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001.
[29] J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithmss The MIT Press, 2017.
[30] D. Poole and A. E. Raftery. Inference for deterministic simulation models: the bayesian melding approach. Journal of the American Statistical Association, (24452)):244-1255 2000,
[31] A. E. Raftery, Madigan, and A. Hoeting. Bayesian model averaging for linear regression models. Journal of the American Statistical Association, 12(43)):19--191 1997.
[32] V. Roth and V. Steinhage. Nonlinear discriminant analysis using kernel functions. Advances in neural information processing systems, 12, 1999.
[33] Y. D. Rubinstein, T. Hastie, et al. Discriminative vs informative learning. In KDD, volume pages 49-53, 1997.
[34] M. Saerens and F. Fouss. Yet another method for combining classifiers outputs: maximum entropy approach. In International Workshop on Multiple Classifier Systems, pages 82-91. Springer, 2004.
[35] B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij. On causal and anticausal learning. In 29th International Conference Machine Learning (ICMI. 2012), pages 1255-1262. International Machine Learning Society, 2012.
[36] X. Sun, D. Janzing, and B. Schölkopf. Causal inference by choosing graphs with most plausible markov kernels. In Ninth International Symposium on Artificial Intelligence and Mathematics (AlMath 2006), pages 1-11, 11, 2006.
[37] Thomas M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley & Sons, Nashville, TN, 2 edition, June 2006.
[38] M. J. Wainwright, M. I. Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning, 1(1-2):1-305, 2008.

---

[39] S. Weichwald, B. Schölkopf, T. Ball, and M. Grosse Grosse-Wentrup. Wentrup. Causal and anti- learning in pattern recognition for neuroimaging. In 2014 International Workshop on Pattern Recognition in Neuroimaging, pages 1-4. IEFE, 2014.
[40] D. H. Wolpert. Stacked generalization. Neural networks, 5(2):241-259) 1992.
[41] Y. Yao, L. M. Carvalho, and D. Mesquita. Locking and quacking: Stacking bayesian models predictions by log- oog-pooling and superposition. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
[42] C. Zhang, Y. Xie, H. Bai, B. Yu, W. Li, and Y. Gao. A survey on federated learning. Knowledge- Based Systems, 216:106775, 2021.
12

---

A Relation between the expectations of the Mixture of Gaussians and the known marginal expectations
In Propostion 3 we proved that the distribution resulting from the constraints in Equations and and the anticausal optimisation problem Equation P result a mixture of Gaussian distributions. Now we are going to explore the relation between the moments of the resulting distribution and the constraints used in the MAXENT optimisation problem.
We have the following expectations under Gaussian mixture model
E
XX
qui (1 p-11
(19)
X
414 (1 11--111 q)
(20)
E
XX
=xx
(21)
=B [Var(X
Y]] + Var(E [ Y]).
(22)
Where Equation 22
follows from the law of total covariance. We have
E Var X Y) =xEi + (1-
(23)
=xir,
(24)
Var(E
[X| Y1) =][ |}3 - Y]j²
(25)
[
X Y] =qaip + -1--9)-1111
(26)
E E X Y] =(gu) + 10100/111110011111111111111 (1-9))-1)(a44 + (1 (11-))==))
(27)
So that
E
Xx
= Ex + qurr (1 11
(28)
(qu) (1 (1-9) (1 (1- T)
=Ei, + (1 -1- 1 I + (1- - g)qu- 11 - 9)qu-144 - (1 q)qui 1_
(29)
ExiY + (1 - -9)glmm1 IT + --1 11 m
(30)
Exix + (1 - 199((4 - (p1 TT
(31)
Recall that the empirical averages used constraints in the maximum entropy optimisation problem are coincide with the expectations under the resulting exponential family distribution. Then, using the equations above and the constraints, the means of the multivariate Gaussian distribution are
X + Φ f1
(32)
2q
X Φ
#-1
(33)
2(1 9)
13

---

And the conditional covariance, which is the same for both components, is
s $1,2
E Exyy
81,
82
[x + (8 ( + T)_
x) Φ)
( 4
(34)
(1 9)²
2²q²
(x (x - T (x (x + Φ)
2 4)
2 9))
$
$12
Expr= =
81,2
82
xx + x T + Φx + 6Φ T
xx x60 - -xx + $06
91 2)
+
(35)
2 (1- 9)2
2²
XX
1Φ
68 T + OΦ
-X xΦ1 + +0x + ΦΦ
2 9)
22 9)
$122
Σxyy
81,2
82
{(1 9)
XX (9² + (1- -)- 9) 9(1 -)))
22 9)
(36)
x66 (4²
2) + q(1 q) q(1 -Q)
+ +xx (1² (0--(1-1 (1- 7)²
- - + - -111)))
+ 00 (0 + (1-g) + (( + + 4))
$7 $12
Σx1x
$1,2 $2
(37)
(49² 4q + 1) xΦ (2q 1) Φx T 1) + ΦΦ 1
22q(11 q)
87 812
1
Σx
(24 1)x 21{22 1)x + T
(38)
81,2
82
22
4)
enumerate the individual elements:
Ex)x =
18 (2q-1)223; 2(24 - 1)T:dh - 021
(39)
22q(1 9)
1
Expx(11))
[5,22 (2q 1)2 (2q 1) (2q 1) 2102). (40)
22g - 9)
B Predictor in the anticausal direction
Theorem 5 (Predictor of using Bayes' rule). Using the results from Propostion 3 the density Px(Y X) is the ratio of the product of the Gaussian component with (Y y) and the mixture of Gaussians resulting from Propostion 3 Minimising the expected 0-1 loss, the optimal decision rule arising from this density is equivalent Linear Discriminant Analysis (LDA).
14

---

Proof. We prove this for the case 1. the case for -1 can be derived in an analogous way.
The result follows from the application of Bayes' rule:
p(x 1)
ply x)
(41)
p(x)
q gexp (-3 (x m1)) " Ex1x -mi)
(- 11) Exiy (x -mi) + (1 ②) exp (--(x -1-1) _
ExIy (x
p-1))
gexp
(42)
(43)
1+ [1-4] exp
12x E (1) 11-11 + pl Exiy 141 1 Eixjt
(44)
1 +
(1-4)
exp
(x (x-12
Σx - (x -
C Derivation of the decision boundary
In the following two sections we give the proof of Propostion the causal and anticausal direction separately. First, we restate the proposition
Proposition 9 (Normal vector to the decision boundaries in causal and anticausal direction). Under the 0-1 loss, the normal vector the decision boundary the CMAXENT predictor is proportional
to
Excooma in the causal direction.
2. Xxyy anticaaaal & in the anticausal direction.
As mentioned on the proposition, we frame these results within the statistical decision theory framework (3], choosing particular loss function L(h(x), y). where l(x) is the predictor of y want to evaluate. We consider the 0-1 loss function. That is, L(h(x). y) 1 if h(x) = y. and
otherwise. The optimal decision rule for this loss is the well-known Maximum A Posteriori (MAP) rule from which we can derive our decision boundary.
C.1 Proof of Propostion in the causal direction
In the causal direction, the Maximum A Posteriori (MAP) rule, results in the decision boundary given by the following equation
p(y x) ply -1 x)
(45)
(1 tanh(Ao + λιI + \222)) = (1 tanh( (oohi--o-- A1T1 A222))
(46)
2
Ao + \ι + Agaz = - -o d1 -A222
(47)
Ao + A1
A222 =
(48)
In words, the decision boundary in the causal direction linear function of the covariates. Using this result, we proceed to prove the relation between the marginal covariance matrix and the normal to the decision boundary in Item of Propostion 9
We want to prove X x Σx Ex,y
Σx Φ
First, define the random variable Z == X1 Ag We can write p(y 1|x) entirely as function of Z, thus X 1l Y Z.
To continue with the proof, we consider the Hilbert space of centered random variables with ba- sis given by span (X) and covariance as inner product. Following this geometric interpretation, define W3 X, Z, where Z is the projection of X, onto the span of Z. That is
15

---

a; = Cov(X, 2| 2|Var(Z)=1 We have that W = [W,, W2) E span (X), so that Cov Z. W = 0.
As a result, W 11 because all variables in the span of X are Gaussian. Together with W 1T Y Z. this implies via the semi-graphoid axioms 23 that W II Z), so that W 11 Y and thus Cov[W,Y] 0.
Taking the inner product with Y on both sides of X; = ajZ + W, gives us Cov[X,, Y] a ayCov[2 [Cov[2,Y] Y] + Cov[W Y = a,Cov Z, = Cov[X,, Z)) -ConzZ))>CCZ Cov[Z, Y7. This is valid for j = 1,2, hence Σx,y = Ex.70z Egyy- By definition, Σx.z = AEx, so that Σx,y = -2 AE 7. ozy and finally (Exaz - oz,y) - = X, as required.
C.2 Proof of Propostion 9 in the anticausal direction
The normal to the decision boundary using the MAP rule in anticausal direction is derived in similar way to Appendix C.I In particular, the normal is given by the points of x where we are indifferent between choosing y 1 and y -1. To find such vector, solve for x in
p = 1 x) p(y -1 x)
(49)
p(x = 13 = 1) = p(x y = 11)p(y = -1).
(50)
In the second line of the above equation, we used p\Y = x)) x p(x Y = =y) = =y).
We have
q gexp (-2(x (x- - m1) ">xxx(x>>>>>>> (x11) = =(1 - ②) exp - - p--1) (x-
m11)
Ex (x- - mm-1) (51)
log 9 (1-g)== -
- (2x Σxiy (11 1111 + u Exly M1 - μ. Ex 1 1 1-1). 1). (52)
The above equation linear in x, giving us a linear decision rule, and we would choose Y 1
1
1111 - -/4 Σx|y 14 log x >xxxyy (14 1-1) > z -- x x
(53)
こタ (m-1- - m1) Σxly (M-1 + M1) - log
q
(54)
Remark 14 As mentioned in Theorem [ the decision rule in Equation (33 is known as the Gaussian discriminant analysis [14]. This is special case of Linear Discriminant Analysis. The family of LDA algorithms also contains Naive Bayes (if all the x are conditionally independent) and Quadratic Discriminant Analysis (QDA) (if the covariance matrices for each y are not equal, giving a curved decision rule).
Now we will prove that if we have use all the moments in Equations 1 and constraints, the slope the two decision boundaries are the same.
Theorem 10 (Slope of the decision boundary is the same in causal and anticausal direction). Using the constraints in Equations T and (2), the slope of px | X) inferred using CMAXENT the same in causal and anticausal direction.
Proof. In Propostion 9 we proved that in the causal direction, the normal vector to the decision boundary in the causal direction Σx Φ. Furthermore, using the law of total covariance (and the assumption that 0). we can write Ex Σx + cΦp T, where c 1/ - 9)) (see
Equation (88 ). Using the Sherman- Morrison formula 2
we can write
Σx =(Exiy + +op0 )-
(55)
CΣ xxI1oØ Σx
(56)
1 cΦ TXx
Applying this operator db and noticing that Φ Σx1y Φ a scalar, we obtain
Σ = x $ + x 1
(57)
where k= = T
Σx Φ)/(1 oΦ T "]E Thus Σ 1 Exiy db, as required.
16

---

D
Missing covariance between the outcome variable and one of the covariates
62. Can CMAXENT say anything about p x)? The answer is positive under the assumption that and X2 are correlated. To see this, we will use the following result in information theory: The entropy of distribution with given first and second moments is always less than the entropy of multivariate Gaussian given the same first and second moments [7 Theorem 8.6.5]. Hence, we analytically compute the maximum entropy solution of Ø2 via the entropy of the multivariate Gaussian as an upper bound on the entropy of X given Y.
Suppose do not observe E[YX; X2
To do this, will use the results of Appendix A where we found an expression of Σx Y function of Φ2 Since Φ2 appears on several elements of the Σxiy, we first compute the determinant of Expy, differentiate with respect 62 and equate to 0 to find the optimal Φ2.
For reference, the differential entropy of multivariate Gaussian of l dimensions and covariance matrix Σ is
k h: H()
+ log(2m) (m)) log (et log(det( E))
(58)
2
2
First we compute the determinant of Σxyy
det(Ex =3733 - - s?2(24 1)8202 $702
2q(1- -g)
1)24782
(2q 1) + 2(24 - 1)3 + (2g 1)2
20(1 -q)
(2q - q))2
2(2g 1)2
+ 2(29 1) 23 2 22q - 1) +2(2g 1)
20(1 - 9)
(2g - 4))²
0733
(2q - 1)ră0l + 2(2g 1)820700 + 0700
+
2q(1 - 9)
(2q - 9)
(1 1) 3152 + 31,2(2q - 1) + 31,2 1) + 21,21022
$1,2 +
24(1
(2q - 1)
(2g 1)^2333
(29 1) 12 - (2q - 1)
2q(1 9)
(2q 4)
(2q 104-16 -0)-)221522010 62
(2q - 1 1) $288.
(2q - 1)$ J cb2 - (2q - 1)2
(2g 9)
29(1 ④)
(2q(1 9))²
(2q 2) 01 02 - (2q 11
(2q - 1) 11220131.2
(2 9))2
29( 1 9)
2 (2g 1)32
(2q - 12
(2g x3200
(2g - 1 0220220
(2q q))²
(2q 9))²
Φ1 0102
(2q 1) 12
(2g 1)
(2q 1) 2
2q(1 4)
(2q(1 9))2
(2q(1 - 9))
(59)
17

---

Now we differentiate det (Ex(r) with respect to Φ2. equate to 0 and solve for Φ2
det ExIr
6/2(24 0)
25902
2(2q 1)
2(24 1)2
+
002
2g(1 - q)
20(1 - q)
(2q( - q))²
(2q - q))²
224 11201
22
11101122
2(2g 1/220
(2(2g - 1))2 + 209
(2g - 9))²
(2q 9))²
-
(2 1)
1
(2g - 1892
(24 - 1
+
2g(1 4)
2g(1 - 4)
(2(2g 1))2
(2(24 - 1))2
(2q 1) 81.2
(2g 1)8 10
2(2g 1)209022
(2g - 1)2 12th1
2g(1 q)
(2q q))²
(2q q))²
(2g(1 - q))²
2(2g - 1) 02
(2q - 12200
(2q - 1)
(2q q))
(2g qg)2
(2q 9
(60)
Equating the above derivative to O, we obtain
2 221-28fa(111 () + 2(2g 2 + 20² 2(2g 1) 2(24 1)2101)
-
sf2 (2 1) 2(24
- 1)
-222-1))2
2(24 1)ă207 51,224 (2 1) 21(1 2)
+ (2q 1) + (24 104-1136 10--))2,2221 (2q 1)2131.2
+ (2g 1) + (2q 1 8201 +
(2q 1 + (24 1) 1207
(61)
Which can be simplified to
Φ² 209 + 1 - 11 28/9(1 - 9)
(62)
[2 - (9 - 1)5752
- (2q - 1) 29(1 (2 - 1) 2y(1 - 11,201 (24 - 1) + (2g - 1)2g(1 - 4)}.
Although we have derived here the general case where the sample means are not zero, we will continue the analysis by coming back to such assumption. That is, क = I2 = 0, giving us the following expression for do which is easier to interpret
g(1 - 21,
Φ
(63)
q(1 9)8 of
In the numerator, we have that as the covariance between X₃ and X2 increases, the MAXENT covari- ance between X2 and Y increases too. Furthermore, we see that the denominator is always greater than by the Cauchy-Schward inequality of random variables, Cov(X, Y)² S Var(X,)Var given that q(1 q) is the variance of Y, 57 is the sample variance of X1, and Φ1 is the sample covariance between X1 and Y.
E Derivation of the decision boundary with unknown predictor covariance
Causal. In the causal case, have that the distribution of the causes is multivariate Gaussian with diagonal covariance matrix, and the conditional distribution of the target variable given the covariates is the same in Equation 7
As result, we have that the decision boundary is still proportional
Φ1
Σx Sxaasal
(64)
82 02
as derived in section Appendix
C.I
18

---

Anticausal. In the anticausal direction, we have that the target variable follows a Bernoulli distribu- tion, and the conditional distribution of the covariates given the target variable is again mixture of Gaussians with diagonal conditional covariance matrix. We first prove Propostion 12
Proposition 12 (Diagonal conditional covariance anticausal direction with unknown predictor covariance). The density p(X Y) that maximises the conditional entropy subject to the following constraints:
0 0
XXY
E
$
E = 52
(18)
and pYY) inferred on the first step of CMAXENT, is independent after choosing a value of y: that is, X is conditionally independent given Y.
Proof. The solution to the constrained optimisation problem has the same form as in Equation (0))
without the cross term:
(x v) = exp(A1yz] + A2y + >3 + 2442 + >şa + Aه= + B(y)]-
(65)
Conditioning any specific value of Y, gives us uncorrelated multivariate Gaussian, required.
Using Equation 31
we can express the conditional covariance as
Σx r =Ex
(1 9)9(μ1 - p ( - 1-11
(66)
7 1b
(#1, - 111,1) - a(1 - 4) (412 - -1 (11,1 - -1-11)
(11,1 - -1111) --
=
2b $2
(M1. - -1-1222
(67)
Since we know that Exyy is diagonal, then & = q( 9)1,1 - (111) p1122 From Equations (32) and (33). conclude that q ()1, p=1,1)² of Wit this, find an
expression of Σxly as a function of the constraints
Σxpr
(68)
On Appendix C2 (see also Hastie et al. 14 Sec. 4.4.5)) we proved that the slope of the decision boundary in the anticausal direction proportional
(69)
and using Equations
32 and
33
we have that the slope of the decision boundary is proportional to
[3 03 -101]
201
Exly
(70)
(5 05 02
natural question arises: when are these slopes the same? in other words, when are Equations and (70) linearly dependent? This question can be answered be equating
664
(57
(71)
1
to
3
22
2
22
(72)
We have
(5f -1
88
(73)
8
(5) 00)1 2000
(s
(74)
(52 03)01 1202
(75)
(57 070025001
(52 02)52
(76)
( 255
19

---

X1
Y
X3
XA
Figure 3: Graph in the causal and anticausal direction
Derivation of the target predictor when merging predictors in causal and anticausal direction
In this section we explore the predictor resulting from merging predictors including causes and predictors including effects of the target variable. We will assume the causal graph in Figure 3 We will use the first and second moments of each variable, the covariance between each X; and Y, the covariance between X, and X2, and between X3 and X4, as constraints.
Using CMAXENT, we first find the density p(X1, X2) with maximum entropy subject to the moment constraints; then p(Y X2) with maximum entropy subject the moment constraints (including X2),,xX), found in the previous step); and finally the density p(X3, X44 Y) that maximises the entropy subject the moment constraints (again, including the found p(Y)).
possible to see that these process will result in the same predictors as Section 3 That is, we find that p(X1, X2) is multivariate Gaussian, p(Y X2) a logistic-like regression, and P(X,, X4 Y) a Mixture of Bivariate Gaussians.
These distributions provide us with enough information to find the joint distribution of all our variables P(Y, X1, X2, X3, X4). with which we can derive our predictor of interest. We have
p(21, U2,
22, 33; 344 ())(
ply Fi, 22, 23, xa)
(77)
p( 22, 33. T4)
y)p( 24 ())()) p
(78)
p( T3,
p(y
æ1 32 (2(1 12, v)p(v)
(79)
02,
p(y
21. 23),, (v)
(80)
23,,,22,33,441
p(y æ1, 22) v)
(81)
Ey p(11, 24 y)p(y)
p(y Vy (2
11)
(82)
Σ, P(y 31, 21, 22, 22 x4 y)
Where the second inequality follows from the conditional independence between (X,, X2) and {X,, given Y, and the third inequality follows from Bayes' rule.
Equation (82) gives us the desired predictor. Notice that we found all of the elements needed to compute p x) on Section 3
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All the theoretical claims in the abstract and introduction are part of the main article.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Since this is a theoretical paper, most of the limitations come from the as- sumptions made to obtain the theoretical results. addition we discuss some potential computational problems, when applying these results in real world problems. These limita- tions are discussed in the main article, in any case.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: This is the bulk of the paper. We provide clear assumptions and proofs of every result (except corollaries) on the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [NA].
Justification:
The paper does not contain any empirical results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [NA].
Justification:
The paper does not include any data or code, besides a very small toy example.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.ca
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA].
Justification: As above.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA].
Justification: As above.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA].
Justification: As above.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Due its theoretical nature, the paper adheres to NcurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We briefly discussed that practitioners should take some these results into account but there is no broad impact in the potential application of the ideas outlined in the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no major risks posed by the paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use any licensed assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "ZrLsb4P4DM", "reviewer": "Reviewer_Wx67", "summary": "This paper studies the differences and properties that emerge when one uses causal, anticausal features for prediction.", "questions": "**Q1.** Do your results hold with incomplete causal sets?\n\n**Q2.** Are there any connections between your observations and robustness to distribution shifts?", "limitations": "The setting considered is too theoretically and empirically simple to be convincing about real-world tasks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1.** This work makes several interesting observations of causal and anticausal predictors under their parametric assumptions.\n\n**S2.** This work suggests some potential considerations for practitioners dealing with feature sets that contain both types of information.", "weaknesses": "**W1.** The primary weakness of this work is that the connections are underexplored empirically and in more complicated settings, e.g., higher dimensions and discrete data.\n\n**W2.** While I do not have an issue with the simplifications you have made to make the connections clear, the lack of more general results combined with a lack of real-world datasets that exhibit properties resembling the observations from your analysis limit the impact of this work is insufficient for the venue.\n\n**W3.** Some of the observations merely confirm properties already known, e.g., the asymmetries on causal and anticausal directions [1-2].\n\n[1] Schölkopf, Bernhard, et al. \"On causal and anticausal learning.\" arXiv preprint arXiv:1206.6471 (2012).\n\n[2] Janzing, D., and B. Schölkopf. \"Causal inference using the algorithmic Markov condition. to appear in IEEE Transactions on Information Theory.\" See also http://arxiv. org/abs/0804.3678 (2008).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We would like to clarify certain points related to the weaknesses and respond to their questions as best we can.\n\nW1 (about high dimensionality): in the examples we studied, we used two covariates but the results would be similar if we had predictors of more covariates. That is, if we had $D$ covariates we would obtain a transformed logistic regression in the form of Equation 7, but instead all the variables for which we have $Cov(Y,X_i)$ would appear on the right-hand side of the equation. Likewise, for the anticausal direction we would obtain the LDA algorithm where the Gaussians are $D$-dimensional as opposed to bivariate Gaussians. We used only two covariates to be able to visualise the differences in the geometry of the decision boundary.\n\nFurthermore, we recently got an interesting insight about the results in high dimensions. Intuitively, the result is that using the same data as constraints, the resulting model in the anticausal direction gives more weight, relative to the causal direction, to those variables that have higher covariance with the target variable.\n\nW1 (about discrete data): there are two options here:\n\n- First, if the target variable is discrete, in which case our results remain unchanged. In fact, Theorem 7 is written for a discrete (and not bivariate) target variable, and thus a generalisation of the previous results. \n- Second, the covariates are discrete. If the reviewer is referring to this case, we would agree that the study of discrete covariates requires a separate analysis. However, to showcase the asymmetries between causal and anticausal merging we decided to resort to cases that admit an analytical solution, which result in differences that are easier to interpret.\n\nW2: (about a potential application of the results) Merging predictors will get increasingly relevant for instance in the medical domain. If $Y$ is the presence/absence of a disease, the causal scenario is given when $X_1,\\dots,X_n$ describe risk factors, while $X_1,\\dots,X_n$ being symptoms would be anticausal. Research results from different labs may provide predictors of $Y$ from different sets of risk factors and/or different sets of symptoms. Combining them into a joint predictor which uses the union of all features would be highly valuable, but it also requires to include the right causal assumptions: risk factors cause diseases cause symptoms.\n\nW3: We know the content of references [1,2] in detail. Neither of them describe a merging scenario. The paper [1] first studied the implication of causal asymmetries (causal vs. anticausal) for machine learning, for the case of prediction (but not merging). This indeed inspired us, as mentioned in the introduction. The relation to [2] is much more remote: that paper develops a theory of causality that uses Kolmogorov complexity rather than statistical (Shannon) information to study conditional independence properties of causal graphs. It is fundamental for the justification of independent causal mechanisms, which in turn is related to machine learning, but we felt that we did not have to cite it. If the reviewer feels otherwise, we are happy to change this, and include a discussion.\n\nQ1: We are not completely sure by what the reviewer means with incomplete datasets. Assuming the reviewer is asking about the case where we do not have predictors for all causal parents, $Y$ (that is, in the causal scenario there may be additional causes). If we do not have predictors for each causal parent of $Y$, then we can still combine the predictors into a single model, however we are not going to obtain an approximation of the causal mechanism from the causal parents of $Y$ to $Y$ (recall the causal mechanism is simply $P(Y\\mid PA(Y))$). As a consequence, we might not be able to compute some causal quantities like interventional distributions without any further assumptions. Of course if the goal of merging the experts is purely prediction there is no consequence of not having all the causal parents, however, as we have shown on the paper the geometry of the decision boundary (regardless of whether one wants to predict or perform causal tasks) -will- change depending on the causal assumptions.\n\nQ2: This is an interesting question and it relates to our interpretation of Q1. In particular (in the causal scenario), if we have the sample averages of the cross-moments between the causal parents of $Y$ and $Y$, then CMAXENT gives us an approximation of the causal mechanism of $Y$ that is robust to distribution shifts. To be more precise, Grunwald and Dawid (2004), and Farnia and Tse (2016) prove that MAXENT and conditional MAXENT are robust Bayes in the family of distributions with the predetermined constraints. That is, it is the distribution chosen from the set of distributions that has the same expectations as those given as constraints, so that the expected value of the log loss is minimised in expectation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SRvrfSxrGa", "reviewer": "Reviewer_29XC", "summary": "This paper studies the problem of learning a mixture of experts (predictors) where individual predictors have been learned with different causal constraints. It studies different asymmetries that arise when we merge different predictors using the Causal Maximum Entropy (CMAXENT) objective. It goes on to show that different data-generating processes lead CMAXNET to reduce to different objectives under some restrictive setting. Next, they show how the learnt predictors will have different decision boundaries under different data moment restrictions.", "questions": "**Questions**:\n1. By merging the predictor, I understand learning a combined model using both the causal and anti-causal predictor. Is my notion of \"merging\" predictors correct, or am I missing something?\n\n\n**Typos**:\n1. Line 235: I think you mean X2 will be irrelevant in the estimation of the target predictor.", "limitations": "Yes, they are added to the checklist.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. \n2.  The contribution of this paper, however, restricted to a simple setup, is novel. The author shows that under different assumptions on the data-generating process, the CMAXENT objective will yield different predictors and establish necessary and sufficient conditions under which the predictors are different.", "weaknesses": "1. The connection with the OOV generalization literature is not discussed properly. In particular, it would be interesting to see how this paper's observation relates to the paper \"On causal and anti-causative learning\" (ICML 2012) and the guarantees they have for generalization to the distribution shift.\n\n2. In the introduction and abstract, the authors mention that they study the problem of merging the two predictors, i.e., one predictor trained to assume an anti-causal data generating process (DGP) and another assuming causal. Next, in Sections 3 and 4, the authors show the closed form of each predictor separately under different DGPs. However, little is said about the final \"combined\" predictor and its generalization properties. See Question 1 for more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading of our paper. We also thank the reviewer for pointing that the soundness and the contribution of the paper are good and the presentation is excellent. We will start giving a short comment on the reviewer's summary and start answering to the weaknesses and questions.\n\nComment about the reviewer’s summary: When we study the difference in the decision boundaries, we study the case where the moments are the same and not different. This is precisely one of the most surprising results of the paper.\n\nWeaknesses:\nWe appreciate the reviewer's comment about the connection between the literature; in the revised version of the paper we will be more explicit about the connections between “On causal and anticausal learning” and some of the more recent OOV literature. \n\nIn particular on the paper “On causal and anticausal learning” they study common Machine Learning tasks (distribution shift, semi-supervised learning, transfer learning) under the light of causal assumptions. They answer questions of the type: can we perform this task if the data comes from this assumed -causal- process? In this paper we do the same but with the task of merging of experts. Here we ask: What are the differences of the resulting models, if we assume the predictors are from a causal process versus an anticausal one? We answer this question using CMAXENT because CMAXENT 1) allows merging of predictors, and 2) allows to include causal information. Nevertheless, we hypothesise the resulting asymmetry would hold also for any other model that has these two properties.\n\nIn relation to OOV, we give an example of a DGP and a model so that we can do OOV. The OOV literature is fairly recent, in fact “on causal and anticausal learning” does not study an OOV scenario. We consider OOD and OOV \"dual\" modes of generalization but their precise link is yet to be understood. This is not part of the current papers technical contribution. \n\nWe can include an extended discussion of the above in the revised version of the paper.\n\nThis is an interesting question and we indeed have the results for the combined DGP but did not include them in the submission to keep it somewhat decluttered. Since the reviewer believes in the value of the combined DGP we can include it in the final version. As a summary of the results, the resulting conditional distribution of $Y$ given $X$ is the product of the two conditional distributions found in Proposition 1 and Theorem 5. This is expected, as given $Y$, the parents and effects of $Y$ are independent and so their joint distribution is the product of their distributions.\n\nQuestions:\nWe will improve the introduction and explanation of certain terms in the manuscript, this is valuable feedback. We do understand merging of experts as the task of putting predictors together into a single model. However, the results show that it matters whether you assume that the predictors are produced with causal or anticausal variables. If you can make these assumptions, then a different question arises: What do we want the combined model for? If we need the model to estimate causal effects then we conclude we should not use the assumed anticausal covariates, if it is about prediction, we should merge all the models available.\n\nTypo:\nThank you for reading the paper in such a level of detail. We will correct the typo and double check if we missed more.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ETdOSCzLFg", "reviewer": "Reviewer_546c", "summary": "The authors give a treatment of the mixture of experts problems using the idea of maxent; they use this as a tool to discuss how to merge causal and anti-causal inferences on the same data, in part as a way to assess the quality of the data being analyzed.", "questions": "Would the authors be willing to expand their literature review to encompass more of the literature relevant to what is being called the \"mixed of experts\" issue?\n\nWould the authors be willing to include an empirical example, if not in the main test, then in the Supplement, with a pointer from the main text? It would be very helpful to know whether these ideas about causal/anti-causal merging are helpful empirically or in simulation.\n\nWould the authors be willing to provide software to allow Neurips readers to easily evaluate the ideas in this paper?", "limitations": "This is a theoretical paper, entirely, so does not benefit from worked examples, so it is difficult to judge impact. I did not see any discussion of societal impact. I also did not see any discussion of promises of software or data to help users assess reproducibility of any results (as empirical results are not assessed).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The discussion of the differences and merging of causal and anti-causal analyses was strong and appreciated.", "weaknesses": "The framing of the paper, I felt, missed a lot of literature and possible approaches to the issue being addressed. That is, the paper is framed as a discussion of merging of expert models (which can be an important problem), and maxent is proposed as a method for doing this. But the merging of experts problem is itself framed as a problem of inferring causal graphs where each expert has access to only part of the data. The problem of overlapping dataset has been extensively studied, but no reference is made to that literature, or to any ideas that literature has proposed which may compete with the maxent proposal here. More recently (actually not that recently) the discussion has taken a turn into discussions of privacy, for contexts, e.g., where different hospitals have access to their own dataset but may want collaborate on building causal model without risking making available by inference the identities of their respective patients--i.e., the so-called \"federated learning\" problem. This has been extensively studied also in the literature to date with many proposals given for how to address it. I think this paper could benefit from a literature review of this sort to place the proposed ideas in context, with comparisons made to alternative methods, or at least reasons not to compare to particular methods that make sense.\n\nAlso, the paper is mainly theoretical but could have benefitted from discussion of an empirical or simulation example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our paper in detail and appreciate that they ranked the soundness and contribution of the paper as excellent. We would like to answer the reviewer’s concerns outlined in the weakness section, clarify some points that might have not been clear in the article and answer the reviewer’s questions.\n\nWeaknesses:\n\nAbout literature and privacy: We appreciate the reviewer’s comment on the literature. However, it is almost always the case that one can include more related work. We will include more work on overlapping datasets and admit that our own point of view is biased by the causal setting. We would also like to invite the reviewer to point us to some relevant papers about privacy and federated learning, which we will be happy to study and reference. We were under the impression that federated learning usually does not study the setting where the variable sets differ (i.e., the merging problem), but following your comment, we have been able to find some papers that study “vertical federated learning”, which we shall discuss. Our impression is that this area focuses more on practical applications and less interested in theoretical statements (and they do not connect the problem to the marginal problem of statistics), but it will be interesting to make those connections.\n\nAbout inferring causal graphs: The reviewer mentions in the weaknesses section that we are framing the merging of experts’ problem as one of inferring causal graphs. This is not entirely correct, and we would like to clarify it here (and we will do it in the revised version of the text as well). What we want is to come up with a single predictor of our target variable $Y$, given a combination of predictors. We have the information of the causal graph, so we don’t need to infer it. We explore the differences in the solutions of the merging of experts under different causal assumptions (in our case, whether the variables are causes or effects of our target variable).\n\nQuestions:\nAbout literature expansion: We will expand the literature with the overlapping datasets literature, and we are happy to expand it with privacy and federated learning related research.\n\nAbout an example: This is a fair point. However, we believe that the greatest merit of the paper are its theoretical contributions and not a potential empirical example. Indeed, Figure 2 is such an example for one particular data generation process, where we can clearly see the difference in the slope of the decision boundary for data with the same moments, under different CMAXENT solutions. If the reviewer is curious about the results of a particular example, we would be happy to run those tests and include them in the final version of the paper. \n\nFor a potential real world example (which we can include on the main paper) in the medical domain. If $Y$ is the presence/absence of a disease, the causal scenario is given when $X_1,\\dots,X_n$ describe risk factors, while $X_1,\\dots,X_n$ being symptoms would be anticausal. Research results from different labs may provide predictors of $Y$ from different sets of risk factors and/or different sets of symptoms. Combining them into a joint predictor which uses the union of all features would be highly valuable, but it also requires to include the right causal assumptions: risk factors cause diseases cause symptoms.\n\nAbout code: We are happy to share the code for our paper upon acceptance. However, we believe that most of the value of our paper comes from the theoretical insights we are giving on the problem of merging of experts when different causal assumptions are made. In fact, the code to produce Figure 2 consists of a very basic data generation process and a common python library to estimate the decision boundaries using the logistic regression and linear discriminant analysis, as Remark 2 and Theorem 5 of our paper point out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mAAZaZycXC", "reviewer": "Reviewer_vvvY", "summary": "The paper explores the potential differences in predictor merging when approached from causal versus anti-causal directions. The results from MAXENT and CMAXENT indicate that in the causal direction, the solution converges to logistic regression, whereas in the anti-causal direction, it converges to Linear Discriminant Analysis (LDA). The study also examines how the decision boundaries of these two solutions vary when only partial bivariate distributions are observed, highlighting implications for Semi-Supervised Learning (SSL) and Out-Of-Variable (OOV) generalization.", "questions": "1. **Noise**: What if we have observation noise (which is very common in MRI dataset)? Will the results still hold?\n2. **Co-variance shift**: Not sure whether I am missing anything, but, what if there is a co-variance shift?\n3. A curiosity is whether all the theoretical results hold if the distributions are not Gaussian?\n4. From a signal processing point of view can we get results for non-causal setups?", "limitations": "The major limitation of the paper is the experimentations that assumes the data distribution to be gaussian. The paper should include non-causal results and its interactions with causal and anticausal models.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper investigates the differences that arise in predictor merging from causal and anti-causal perspectives. It demonstrates through MAXENT and CMAXENT that the causal direction results in logistic regression, while the anti-causal direction leads to Linear Discriminant Analysis (LDA). Additionally, the paper analyzes how the decision boundaries of these two methods change when only some bivariate distributions are observed, discussing the implications for Semi-Supervised Learning (SSL) and Out-Of-Variable (OOV) generalization.", "weaknesses": "1. **Small scale dataset**: The main weakness is the small scale of the data and models studied in the paper. I believe the challenge of reducing computational cost with mixture-of-expert models is more relevant to larger models. The authors however only presented results on small bivariate distributions. Experiment results with larger models are appreciated. If experiments with larger models are not feasible, I hope authors can discuss potential limitations of the study under those larger-scale/multivariate scenarios. Do you expect the findings in Eqn 1&2 change in larger-scale/multivariate setups?\n\n2. **Lacks comparison**: This paper lacks sufficient comparisons with other papers. Can you explain what are the differences and advantages of the proposed method, compared to the pi-Tuning method proposed in [1] (Section 3.2 and Section 3.3)?\n\n3. **Contributions are obvious from the observations given in MAXENT and CMAXENT**: The overall paper seems like a consolidation of few previous papers. \n\n4. **Non-causal**: The paper has studied the causal and anti-causal setups but from the signal processing point of view pleas study the non-causal settings.  \n\n5. **Results under the availability of Noise and biases**: The paper has shown result in a toy example which is good for overall pipeline understanding, but not sufficient to understand it from ML perspective. For example: what if the random variable leverages some amount of noise and have biases that imposes skewness in the distribution? \n\n\nReferences:\n[1] Wu, Chengyue, et al. \"pi-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation.\" International Conference on Machine Learning. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We will first give some general comments, then answer each one of the points in the weaknesses and last answer the questions.\n\nWe noticed the strengths of the paper are the same as the summary. Is this correct?\n\nWeaknesses:\n1. The question about scaling is an important one. We believe that the results of Equations 7 and 10 (and Theorem 5) would remain unchanged whenever we have $D$ experts to merge. We studied the bivariate case because it allowed us to easily analyse and visualise the implications of our theoretical results. Furthermore, we recently got an interesting insight about the results in high dimensions. Intuitively, the result is that using the same data as constraints, the resulting model in the anticausal direction gives more weight, relative to the causal direction, to those variables that have higher covariance with the target variable.\n\n2. We are unaware of other papers that look at merging of predictors in the case of causal and anticausal learning. If the reviewer knows one such analysis we would be happy to compare it to ours. We read [1] in detail but we found that beyond the use of the term pooling of experts there is no strong relation between [1] and our work. Our work studies the asymmetries between merged predictors whenever we can make causal assumptions, whereas [1] proposes a method to do merging of experts in the setup of foundation models with relation to transfer learning, which we do not talk about in our paper. There is no causality involved in [1].\n\n3. We disagree with the reviewer with this statement. CMAXENT has been studied for the causal discovery and merging data scenario Garrido Mejia, et.al. (2022). What we study here is theoretical asymmetries from merging of experts with different causal assumptions. These are different tasks and we chose CMAXENT because it allows us to merge data, and allows us to include causal information on the final model. Furthermore, we analyse the geometry of the decision boundaries of different models using the same data. This is the only study we know that does such analysis a we find it to be of high value for the NeurIPS community.\n\n4. We do not see how the analysis of non-causal scenarios is relevant to the purpose of our paper, which is studying the differences arising from causal assumptions when merging experts. There is previous research, referenced on our paper, that studies merging of experts using MAXENT without any causal assumptions (see references [21], [23] and [30] on our paper for some examples).\nThis is an interesting question but, as the previous one, out of scope for the current work. Furthermore, there is research on the results of MAXENT under data noise, whereas there is very little research on merging of experts and causality.\n\nQuestions:\n1. Although this is an interesting question, we find it to be out of scope as we mentioned above. In this paper we are interested mainly in the asymmetries arising from the differences between causal questions on merging of experts.\n\n2. As usual in causality, if there is covariance shift one needs to understand whether the model is causal or anticausal, as a distribution shift can be interpreted as an intervention under the causal lens. If the model is causal, then the it is robust against distribution shifts, if the model is anticausal, then the model will not be robust against distribution shifts.\n\n3. We do not assume that the distribution of any covariate is Gaussian. This is a result of MAXENT under mean and (co)variance constraints. Several common distributions are the result of a MAXENT problem under moment constraints: Bernoulli, Exponential, Gaussian, etc. An excellent resource to understand the relation between common distributions and Maximum Entropy is Wainwright and Jordan (2008).\n\n4. If the reviewer is interested in merging of experts without making any causal assumptions they can check references [21], [23] and [30] on the main document and more recently:\n- Vetter, J., Moss, G., Schröder, C., Gao, R., & Macke, J. H. (2024). Sourcerer: Sample-based maximum entropy source distribution estimation. arXiv preprint arXiv:2402.07808.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for reading our paper and the interesting questions they asked. We also appreciate some of the reviewers considering the paper has a valuable contribution to the community, being sound and well presented. We invite the reviewers to increase the score if they feel their questions were satisfactorily answered.

In the revised version of the paper we will include an extended discussion about our results in higher dimensions, some potential applications of our theoretical insights and we will extended our literature review to include some references on overlapping datasets, and some interesting papers on federated learning where they have a similar setup as ours (though without the causal considerations).

Meta-review: **Summary.** This paper considers the problem of combining multiple predictors based on different causal relationships using Causal Maximum Entropy (CMAXENT), namely causal and anticausal predictors. Though the models and settings they analyze are simple and restrictive, their results demonstrate that CMAXENT reduces to standard model classes, logistic regression in the causal direction, and linear discriminant analysis in the anticausal direction. They also study the effect only partially observing the data distribution on the decision boundaries of these models.

**Strengths.** 
* The paper is written clearly, and the ideas are communicated effectively.
* Though the analysis only considers simple settings, the ideas and observations about the solutions to CMAXENT are novel and motivate an interesting direction for further research.
* The analysis is well-executed

**Weaknesses**
* The settings considered are limited. For instance, the analysis does not include categorical covariates (Reveiwer  Wx67 ) or distribution shifts (Reviewer vvvY).
* The authors should take care to address comments raised by the reviewers -- particularly the following: (i) a discussion about how their results are related to the utility and properties of the final merged predictor; (ii) motivate applications where the observations if this work is prescriptive with more attention; (iii) a potential weakness of this work is reduction OOD-generalization properties due to the use of anticausal features; multiple reviewers asked about the relationship between this work and the ideas in “On causal and anticausal learning.” Discussing this work and/or other related work would allow the authors to discuss this point. (iv) Expand the literature review with reviewer suggestions. (v) Discuss high-dimensional covariates.
* Related to the previous point (i), very little is said about the final predictor; should it be better or worse than the causal / anticausal predictor alone, and when? Some empirical demonstration of this would go a long way.

**Discussion**
The primary criticism from the reviewers was the extent of the theoretical contributions. The authors clarified the questions and limitations raised by the reviewers and agreed to address their discussion points (some enumerated below). Still, one reviewer recommended rejection due to the limited scope of the results; however, other reviewers found the paper's insights interesting enough to merit acceptance despite the limited settings explored -- though there were no strong supporters of acceptance.

**Recommendation**
Overall,  the ideas and observations in this work are interesting and potentially motivate more work in this important direction, e.g., merging multiple model decisions or Human/AI predictions. The reviewers recommend acceptance with the expectation that the authors will address the abovementioned weaknesses.
Decision: Accept (poster)