Interpolation and Regularization for Causal Learning
Authors: Leena Chennuru Vankadara
Luca Rendsburg
Ulrike von Luxburg
Debarghya Ghoshdastidar
Venue: NeurIPS 2022 Accept (2022)
Keywords: Causality
Interpolation
High-dimensional linear regression

Abstract: Recent work shows that in complex model classes, interpolators can achieve statistical generalization and even be optimal for statistical learning. However, despite increasing interest in learning models with good causal properties, there is no understanding of whether such interpolators can also achieve *causal generalization*. To address this gap, we study causal learning from observational data through the lens of interpolation and its counterpart---regularization. Under a simple linear causal model, we derive precise asymptotics for the causal risk of the min-norm interpolator and ridge regressors in the high-dimensional regime. 

We find a large range of behavior that can be precisely characterized by a new measure of *confounding strength*. When confounding strength is positive, which holds under independent causal mechanisms---a standard assumption in causal learning---we find that interpolators cannot be optimal. Indeed, causal learning requires stronger regularization than statistical learning. Beyond this assumption, when confounding is negative, we observe a phenomenon of self-induced regularization due to positive alignment between statistical and causal signals. Here, causal learning requires weaker regularization than statistical learning, interpolators can be optimal, and optimal regularization can even be negative.

Interpolation and Regularization for Causal Learning
Leena Chennuru Vankadara* University of Tübingen, Tübingen Al Center
Luca Rendsburg* University of Tübingen, Tübingen Al Center
Urrike von Luxburg University of Tübingen, Tübingen Al Center
Debarghya Ghoshdastidar
Technical University of Munich
Abstract
Recent work shows that in complex model classes, interpolators can achieve statis- tical generalization and even be optimal for statistical learning. However, despite increasing interest in learning models with good causal properties, there is no understanding of whether such interpolators can also achieve causal generalization. To address this gap, study causal learning from observational data through the lens of interpolation and its counterpart- peattrrpatt-rguuaariaatiii causal model, we derive precise asymptotics for the causal risk of the min-norm interpolator and ridge regressors in the high-dimensional regime. We find a large range of behavior that be precisely characterized by a new measure of con- founding strength. When confounding strength is positive, which holds under independent causal mechanisms find that interpolators cannot be optimal. Indeed, causal learning requires stronger regularization than statistical learning. Beyond this assumption, when confounding is negative, we observe phenomenon of self-induced regularization due to positive alignment between statistical and causal signals. Here, causal learning requires weaker regularization than statistical learning, interpolators can be optimal, and optimal regularization can even be negative.
Under a simple linear
standard assumption in causal learning
Introduction
We consider the problem of learning the causal influence of multivariate covariates x IRd on scalar target variable IR purely from observational data and under the presence of hidden confounders. Formally, given finite samples {(x,, 9;) drawn independently and identically (i.i.d) from the joint observational distribution p(x, y) p(# (v)la)) the goal of causal learning is to predict the effects on the target variable under interventions on the covariates æ. In other words, the goal is to learn predictive model that minimizes the expected loss on random draw from the interventional distribution Pao y) = p(x) (z))p())d((z)) which can be different from the observational distribution.
Recently. Janzing (2019) established close analogy between statistical and causal learning (albeit under highly constructed confounded model). As a consequence, Janzing (2019) suggested that standard statistical learning-theoreti techniques (such as norm-based regularization) may also help learn good causal models. However, the classical statistical principles of vas-vvaiancee trade-off have been challenged in recent years by highly complex classes of models that are trained to interpolate the data and yet achieve remarkable generalization properties across a broad range of problem domains (Zhang et al., 2021). A large volume of recent work suggests that interpolation can be compatible with and may even be necessary to achieve optimal statistical generalization the high-dimensional regime (Belkin et al., 2018; Belkin al., 2019a; Liang et al., 2020; Feldman, 2020). Despite the surge in interest, causal properties of such interpolators have not yet been explored. In this work, we
denotes equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

consider a simple linear causal model in the high-dimensional regime (n, d + oc, 30,d/n E (()) and ask: can interpolators achieve good causal generalization?
1.1 Motivation and Related Work
Resemblance between statistical and causal generalization Causal learning can be regarded as an instance of the general problem of learning under distribution shifts, where the training (observational) distribution is shifted from the test (interventional) distribution. In the framework of out-of-distribution generalization, an interesting proposition for causal learning arises from the following high-level idea. Observing small sample sizes may induce a similar bias as distribution shifts. Therefore, techniques for learning models with good out-of-sample generalization (such as regularization) may also help to learn models with good vice-versa. The literature provides plentiful evidence to support this general principle for different classes of distribution shifts. For instance, under broad class of distribution shifts, distributionally robust optimization is equivalent to norm- based regularization (Xu et al., 2009; Shafieezadeh Abadeh et al., 2015; Gao et al., 2017; Shafieezadeh-A -badhh et al., 2019; Blanchet et al., 2019; Kuhn et al., 2019). Analogously, distributionally robust optimization techniques are also employed for statistical learning under limited samples (Zhu al., 2020). Particularly relevant to our work is Janzing (2019), which formally establishes a close analogy between "generalizing from empirical to observational distributions" and "generalizing from observational to interventional distributions" under highly constructed confounding model. This analogy suggests that standard norm-based regularization such as lasso or ridge, typically used for statistical learning, may also help learn better causal models.
generalization and
Interpolation can be compatible with statistical learning Explicit norm- based regularization techniques were initially motivated by the classical learning theory principle of bias- variance trade- off, which is characterized by a U-shaped generalization curve. This principle recommends to avoid interpolation and instead suggests to balance data fitting with the complexity of the hypothesis class. Recently, however, these classical principles have been challenged by deep learning models. Despite being highly complex with the ability fit even random labels and often trained interpolate the training data, they achieve of-teof-thheas out-of-sample generalization across many domains (Zhang et al., 2021). 1). A partial explanation is provided by the double- descent phenomenon (Belkin et al., 2019b; Belkin, 2021). Extending the generalization curve beyond the interpolation threshold reveals two regimes: the classical U-curve in the underparumeterized regime and a decreasing curve the overparameteritee regime. This behaviour is not limited to deep neural networks, but extends to other settings such as random feature models and random forests (Belkin et al., 2019b; Hastic et al., 2022; Mei al., 2021). Follow-up work suggests that in the overparameterized regime, interpolators can indeed achieve low statistical risk (Belkin et al., 2019a; Liang et al., 2020; Bartlett et al., 2020; Tsigler et al., 2020; Muthukumar et al., 2020).
Is interpolation compatible with causal learning? On account of the parallels between statistical (out-of-sample) and causal (out-of-distribution) learning, is therefore natural ask: can interpo- lators also learn good causal models? One line of empirical work suggests that naively applying distributionally robust learning techniques such as importance reweighting or distributionally robust optimization approaches (which are equivalent to certain forms of regularization) offers vanishing benefits over empirical risk minimization in overparameterized model classes (Byrd et al., 2019; Sagawa et al., 2020; Gulrajani et al., 2021). However, there is also empirical evidence that suggests that augmenting such techniques with additional explicit norm-based regularization may help to learn distributionally robust models in the overparameterized regime (Sagawa et al., 2020; Donhauser et al., 2021). In the context of causal learning, it has been suggested that explicit regularization can be beneficial and might even need to be stronger than for statistical learning (Janzing, 2019; Vankadara et al., 2021). Existing work, therefore, remains unclear about the role of explicit regularization in causal learning, or correspondingly. whether interpolation is compatible with causal learning. In this work, we take a theoretical approach to systematically address these questions.
1.2 Our Contributions
We provide first analysis of causal generalization from observational data in the modern, overparam- eterized and interpolating regime under a simple linear causal model. Specifically, we consider the interpolating minimum l norm least-squares estimator and the family of regularized ridge regression

---

estimators in the proportional asymptotic regime. We seek answers to the following questions: is there a regime where the optimal causal regularization parameter is O, that can we observe benign causal overfitting? Furthermore, if the optimal causal regularization parameter is positive, how strongly do we need to regularize? How does the optimal causal regularization compare to the optimal statistical regularization? While our analysis is exhaustive, we emphasize the results under the assumption of independent causal mechanisms (Janzing et al., 2010), a standard assumption in causal learning.
Precise asymptotics of the causal risk (Section 3). We provide precise asymptotics of the causal risk of the ridge regression estimator as well as the minimum l2 norm interpolator in the high-dimensional setting: n, d o0, d. 7 E (0, oc). Our results confirm that, similar to the statistical setting, the causal generalization curve the min-nome estimator exhibits the double-descent phenomenon. This is because the variance term diverges the interpolation threshold and is decreasing in the overparameterized regime (y > 1).
measure of confounding strength ( (Section 2.1). We introduce a new measure of confound- ing strength that quantifies the relative contribution of the "confounding signal" to the "causal signal". can be interpreted as the strength of the distribution shift between the observational and interventional distributions. While < can take any real value in general, it is restricted to [0, 1] under the assumption of independent causal mechanisms. There it induces a strict, ordering of all causal models that entail the same observational distribution.
Benign causal overfitting (Section 4). When the causal signal dominates the statistical signal (S < O), we observe a phenomenon of self-induced regularization due to the confounding signal. As a consequence, the optimal causal regularization can be O or negative even if the optimal statistical regularization strictly positive. Under the assumption of independent causal mechanisms, however, we show that there is no benign causal overfitting. This is in contrast to benign statistical overfitting, which can occur in the highly underparameterized regime (7 - 0).
Optimal causal vs. statistical regularization (Section 5). We show that causal learning requires weaker regularization than statistical learning when the confounding strength C is negative. However, when > and in particular under the principle of independent causal mechanisms, we show that causal learning requires stronger regularization than statistical learning. More specifically, the optimal causal regularization strictly increasing in confounding strength.
2
Problem Setup
Figure 1: (a) Graphical model of the causal model defined in (1). (b) The usual statistical model. In both figures, observed random variables are shaded and unob- served variables are white.
M
α
3
3
x
x
(a)
(b)
We consider linear causal model with parameters M E Bdxi E RR,3 E Rd with / d and ²² > 0 described via the structural equations
N~~ (1) *~N100, x x= y= -= a
(I)
The covariates
dd and the target y E are confounded through E R', which follows a standard normal distribution. This structure implics that Ez = 0 and the covariance of x is Σ = Cov z M M A graphical representation of this causal model is given in Figure la. The observational joint distribution of this causal model is given by p(x, y) = pl (())))))))) where & ~ (0, Σ) and y/ı: ~ N((=" 3, 52). Here, the statistical parameter / == B + T consists of the causal parameter and confounding parameter = M+T a, where . -M is shorthand for (M+)² and M+ denotes the Moore-Penrose inverse of M. The statistical noise is given by G² 0² o2 ||||22
[[T'] 5, where ||> x" z'' denotes the generalized norm. Note that the observational distribution alone cannot distinguish the causal model from the one in Figure lb. The
NNot that ||a]|2 ||T'
||a|l M M where 1 M M is the orthogonal projection onto ker M.
3

---

goal of statistical learning is to predict after observing which is captured by the conditional distribution p(y]z). In contrast, the goal of causal learning to predict y after manipulating or intervening on This is formally captured by Pearl's do-calculus (Pearl, 2009), which describes interventions on random variables as a shift in the joint distribution. Intervening on with the value Co. denoted as do(r r ro), removes all arrows to and sets Ho- In our causal model (1), the intervention do(x = zo) removes the arrow z + x and yields the updated structural causal equations
z ~ (t, E &~,,00,0") I ===0, y = = + a
The corresponding distribution of y after intervening on x is therefore given by y|do(z = xo) ~ Naa 3, o" + [[] Since arbitrary interventions can introduce arbitrary distribution shifts, consider the natural class of interventions drawn from the observational marginal distribution on æ. This yields the interventional joint distribution (ao y) p(z) (yyy))((r) with the slight abuse of notation do(x) in which the random variable x and its value coincide.
Causal learning from observational data Assume we are given i.i.d. samples ((Ts, ]= from the observational joint distribution p(r, which we collect in X E Rnxd and Y E R". The usual statistical learning aims for the observational conditional p(y]ar), which means that train and test distributions coincide. Causal learning aims for the interventional conditional p(y do(x)), ) distribution shift problem for which train and test distributions differ. We define the corresponding causal risk RC and statistical risk R of any linear regressor 3 E Rd under the squared loss as R = Ey (x" ---- y)² and R²) = (1 (0--)) ")²
The following proposition (proven in Appendix A) characterizes the risks under the model (1).
Proposition 2.1 (Causal and Statistical Risk). For any / E BRd, the risks defined in Eq. (2) satisfy ( = ||3 913 0²
||T and R$(3) (B) || 311 +2
Therefore, 3 the optimal causal parameter and 3 is the optimal statistical parameter. In the following, simply refer to them causal and statistical parameters.
2.1 A New Measure of Confounding Strength
Since the interventional distribution generally differs from the observational distribution, we require a measure that quantifies how this shift influences causal learning from observational data.
Signal-to-noise ratios (SNRs) Before define measure of confounding strength, first define the statistical and causal signal-to-noise ratios, which help to intuitively understand our confounding strength measure. Recall that every causal model entails a statistical model since the causal parameter 33 and the confounding parameter T jointly specify the statistical parameter The statistical SNR is defined as usual by SNRs == ||3]|2/0² For the causal SNR, a
3 + r. r. natural notion would be [], + [[]]]] if the learning algorithm had access data from the interventional distribution y do(x) ~ Nx 1,-3,,02 + ;;; but since are constrained to data from the observational conditional y|æ [Nx=" the corresponding causal SNR, which quantifies the hardness of the learning problem. needs to take this into consideration. Accordingly, we consider the causal SNR as the ratio of the alignment between the statistical and causal parameters and the variance of the observational conditional. Formally, we define it SNRc = (B, B)²02, In what follows, we therefore often refer to (B, B) as the causal signal and || as the statistical signal. Correspondingly, we refer to (3 - 3. 3) = (I', (3) as the confounding signal, which is the alignment between the confounding parameter and the statistical parameter
Confounding strength Regression on observational data implicitly assumes that the interventional distribution coincides with the observational distribution, while it can be shifted in general. To quantify the impact of this distribution shift on the corresponding causal risk, we introduce new confounding strength measure Ç. It measures the relative contribution of the confounding signal to the statistical signal and is defined by
3) (T,
(T 3)
(3)
(8,3)
1|

---

Other notions of confounding strength are possible, but we will see later that this definition is well-suited to capture the shift strength for causal learning from observational data. Without further restrictions, can take any value in R. This measure divides the causal models into the following three regimes, depending on the relationship between causal and statistical signal:
z 1: the causal signal (8, (8) is non-positive, which implies that causal and statistical parameters are orthogonal or negatively aligned. Statistical learning is adversarial to causal learning.
0 < Ç < 1: causal and statistical parameters are positively aligned but the causal signal is weaker than the statistical signal || for example /3 = 3/22.
< O: the causal signal dominates the statistical signal, for example B =33.
The SNRs are related to the confounding strength measure via SNRC (1 SNRs. In particular, the causal signal decreases as the confounding strength increases.
The regime 0 < Ç 1 is practically most relevant Causal learning often requires strong as- sumptions because causal models cannot be uniquely identified by their observational distribution. A standard assumption is the principle of independent causal mechanisms (ICM) (Janzing et al., 2010; Lemeire et al., 2013; Peters al., 2017), which informally asserts that causal mechanisms share no information. In our causal model (1), corresponding assumption could be that the causal mechanisms a and / are drawn from rotationallyy invariant distributions. This implies that (8,1) as d + oc, which in turn falls in the regime 0 I Ç S 1. While our following analysis covers all possible causal models, we pay special attention to this regime because it might be of highest practical relevance. Note that for (8. r) = O, our measure of confounding strength coincides with the measure ç' = ||]]]]/ /([ + [|,]| introduced by Janzing et al. (2017). measures the relative contribution of causal and confounding signal in terms of lengths rather than inner products.
3 Causal and Statistical Risk of High- Higg-DDienssionl Regression Models
Causal learning is extremely challenging, because it requires scarcely available interventional data or has to rely on other information such as exogenous (Rothenhäusler et al., 2021) or instrumental variables (Angrist et al., 1991). In our setting where only observational data are available, causal learning requires additional model assumptions. One such approach has been followed by the Concorr method (Janzing, 2019) which leverages the ICM assumption to make improved choice of regularization parameter under a linear regression model. To fully characterize the effect of regularization on causal generalization, we consider two estimators for learning causal models from observational data (X, Y) E Rnnd x IR:: the min-norm interpolator and ridge regressors. The min-norm interpolator the minimum l2 norm solution to the least squares regression problem
Bo(X, Y) arg min [| : B arg minl|Y x²
BeRd
closed form is given by Y)(X, = (X x) - For À > 0, the ridge regressor solves
- -|Y Y x 2 + 11311
3x(X == arg min
(5)
BER* π
which has the explicit solution B (X,Y)
The min-norm interpolator can be obtained as a limiting case from the ridge regression solution via (X, Y) lim 8x(X, Y). Whenever it is clear from the context, we drop the dependence of the predictors on X and Y.
x
Before proceeding with the analysis, we motivate the idea that appropriate regularization can help to learn causal models from purely observational data. To this end, we compare regularization chosen by statistical cross validation to regularization based on an interventional validation set in Figure 2. Since cross validation implicitly assumes that there is no confounding, it close to Bayes optimal for
Ç = 0 when n >> d. However, as confounding increases, it falls behind regularization based on the interventional validation set. The latter even yields Bayes optimal risk again in the purely confounded setting < = 1, where the lack of causal signal (3 = 0) is encoded by infinite regularization. While we might not have access to an interventional validation set in practice, our theory will show that knowledge of confounding strength is sufficient for choosing appropriate regularization. Finally, we want to caution that even though regularization can help, it does not remove the hardness of causal learning. Reliable causal inference still requires stronger assumptions additional data.
5

---

Figure 2: Causal excess risk of ridge predictors based on 30, 000 samples from the obser- vational distribution. Regularization is chosen either by cross validation based tion set from the interventional distribution of same size. Each model has fixed dimensions d = 300, I = 350 and SNRs 5, but differ- underlying confounding strengths under the constraint (B,1) = O. The benefits of optimal regularization over cross validation increase with confounding strength.
Interventional validation
Cross validation Null prodictor
Causal excess risk ||@
valida-
2
8.6
0.2
0.4
0.6
0.8
1.0
Confounding strength <
3.1 Precise Asymptotics of the Causal and Statistical Risks
In this section, we provide precise asymptotics for the causal and statistical risks of the min-norm in- terpolator and ridge regression solutions in the high-dimensional regime. This regime is characterized by both n, d + oc such that d/n + E (0, oo), where is called the overparameterization ratio. We distinguish between the underparameterized regime (7 1) and the overparameterized regime (y 1). All proofs for this section are deferred Appendix B. Since the predictors B B(x, Y) are random variables in the training data X and Y, so is their corresponding causal risk. We consider the expectation of this risk under Y conditioned on X. According to Proposition 2.1, it given by R& (B) = Eyjx Rº (B) = y xx ||3 - 311 + x² + ||||| Due to its simple form, similar to the usual statistical risk, the causal excess risk can be decomposed into bias and variance:
By x 3 - B11 = 3 - 312 Ey - xx
(6)
(Ax)(A))
((A)
The next theorem is one of our main results. gives a closed-form expression for the limiting causal bias and variance of the min-norm interpolator and ridge regression estimators. We make the simplifying assumption of isotropic covariance Σ La. The proof relies on recent techniques from random matrix theory. It employs arguments similar to Dicker (2016), Dobriban et al. (2018), and Hastie al. (2022) and can correspondingly be extended to arbitrary covariances under boundedness assumptions on the spectrum. We leave such extensions for future work and focus on thoroughly understanding the isotropic causal model, because it already exhibits rather rich behavior.
Theorem 3.1 (Limiting Causal Bias- Variance Decomposition for the Ridge Estimator). Ler
|| r2, ||T|² = w², (T, B) = n7, and fix 0². Then as n. d > Do such that d/n + 7 E (0, 00), holds almost surely in X for every X > 0 that
(Bx) + B = + +2
2(w2 + n)mm(->) and
(B(a)) + VT.
= =0²7(m((-)) - >m'(-))))
(8)
where m()) ((1 - -1- - -32))22)(277)) and F2 =2 + w2 2m. Therefore R& (3x) + RS = B +V 52 +w²
w² The corresponding limiting quantities for the min-norm interpolator can be obtained by taking the limit + 0+ in (7) and (8).
From these limiting expressions we can see that the causal risk curve of the min-norm interpolator exhibits the double descent phenomenon: it diverges at the interpolation threshold y 1 due to the variance term and decreases again for 7 > 1. A corresponding visualization is given in Figure 4. Explicit regularization dampens the divergence of the variance term. While we are primarily interested in the causal risk, the corresponding statistical risk serves as a natural baseline. An analogue set of results for the statistical risk is given in Appendix C. These results have already been derived by Hastie et al. (2022) and can also be recovered as a special case of our causal results: for fixed statistical parameters |3 and 2² the statistical risk coincides with the causal risk of an unconfounded causal model defined with 3 = 3, o² の² and a = O. In particular, the corresponding statistical limiting expressions are the same in Theorem 3.1 after setting n = 2 = 0.
Optimal statistical and causal regularization By directly optimizing the closed form expressions for limiting causal and statistical risks we can find the optimal causal and statistical regularization.

---

6
Causal risk R
10
=
Risk
= 0.5
Bias
4
S - --1
Variance
2
0
2
3
4
2
Overparameterization ratio
O Overparameterization ratio y
Figure 3: Limiting causal excess risk Ry (without the constant 8² + of the min-norm interpolator for different causal signal strengths S. Dashed lines are the corresponding null-risks w², which are outperformed more often as S increases, For < 1, all three curves coincide.
Figure 4: Limiting bias-variance decomposition and causal excess risk of the min-norm interpola- tor (black) and optimally regularized ridge regres- sion (red). Crosses indicate unite-sample risks of
dd/y samples with d 300. The finite risks are well-ppedicted by their theoretical limit.
For any 7 (0, the optimal statistical regularization X5(7) = arg inf xe R can be expressed in closed- as Xg(7) = SNRs - y. The closed-form expression for the optimal causal regularization parameter X²-(7) = arg inf (x,)) RS is a root of 4th order polynomial and as such considerably intricate. For readability, we not include it here. We investigate the behavior of the optimal causal and statistical regularization in Section 4 and 5.
3.2 Basic Behavior the Limiting Risk
We start to analyze the results by assessing the basic behavior of the limiting causal risk. The causal risk of the null estimator 3 serves natural baseline to evaluate the performance of the the min-norm interpolator and the ridge regression estimators.
Regimes of the min-norm interpolator Theorem 3.1 characterizes the limiting causal risk of the min-norm interpolator. Its behavior is controlled by the causal signal-to-non to-ioee ratio, which we defined as SNRc (1 - -) SNRs. However, as we will later see, the causal risk of the min-norm interpolator can be lower than null risk when 0.5. To distinguish the regimes of the min- -orm interpolator, is therefore convenient to consider the closely related quantity S = (1 - 2() SNRs. It distinguishes between three different regimes (visualized in Figure 3).
For S 1, the causal signal dominates the noise and the min-norm interpolator can perform better than null risk in both under- and overparameterized regime.
For S < the causal signal weaker than the noise. Only the underparameterizedd regime can beat the null risk, whereas the overparameterized regime is always worse.
The previous two cases resemble the behavior of the statistical risk in the corresponding regimes of the statistical SNR. Contrary to the statistical risk, however, the causal risk admits a third regime S < 0. In this case, the min-norm interpolator always performs worse than null risk. Here, the causal signal (3, 3) is dominated by the confounding signal (T, 8), and interpolating the observational data overfits to the confounding.
Bias and variance The bias- decomposition of the causal risk given in Theorem 3.1 is visualized in Figure 4 for the min-norm interpolator and the optimally ridge-regularized regressor. The figure also shows the causal risk based on finite samples from the model, which is in high agreement with our asymptotic results. We compare the causal risk to the corresponding statistical risk. First note that the causal and statistical variance terms coincide exactly for both the min-norm interpolator and ridge regressors. This is because the variance term of the squared loss depends only on the variance in the training data, but not on the target parameter B. Since the training data are the same for both causal and statistical learning, the variance terms trivially coincide.
7

---

For the min-norm interpolator. as in the statistical case, the variance term is responsible for the double-descent behavior of the causal risk curve because explodes at the interpolation threshold 1 and decreases in the overparameterized regime y > 1. In the statistical setting, the bias strictly increases the overparameterized regime and, as a consequence, the best risk is always achieved in the underparameterized setting. In contrast, the causal bias of the min-norm interpolator can be decreasing in the overparameterized regime and therefore the optimal causal risk can be achieved in the highly overparameterized regime 7 oo. However, this only happens in the regime S < 0 where the risk of the min-norm interpolator is always worse than null risk.
Figure 4 shows the causal risk of the optimally regularized ridge regression estimator which trivially always below that of the min nin-norm risk. Similar the statistical setting, the corresponding generalization curve does not exhibit the double descent phenomenon. There are qualitatively different reasons for why regularization helps in statistical and causal learning. For both statistical and causal learning, regularization decreases the shared variance, which corresponds to the finite- sample error. However, while the statistical bias always increases with regularization, the causal bias can actually decrease. This implies that regularization not only helps with the finite-sample error, but can also reduce the error due to confounding.
Higher confounding implies higher causal risk for all X So far, we have investigated the causal risk under single causal model. Now we can compare different causal models using the confounding strength measure introduced in Section 2.1. The next proposition shows that governs the hardness of causal learning from observational data. Specifically, the causal risk of the ridge regression for any A E (0, 20) increases as the causal model becomes more confounded. A proof is given in Appendix D.
Proposition 3.2 (Causal Risk Increases with Confounding Strength). Consider the family of causal models parameterized as in (1) that entail the same observational distribution. Let C₁ and Cz be two such causal models with confounding strengths and (2 and alignments qn and 1/2 (defined in Theorem 3. 3. respectively. Then for all X, E (0, oo),
S1 > C2.
m S 72 = R Rx
In particular, for any fixed n, the measure of confounding strength ( establishes a strict ordering of causal models. This includes the ICM under which O.
4 Benign Causal Overfitting
large number of recent works suggest that minimum- interpolators can be optimal for statistical generalization (Belkin et al., 2018; Belkin et al., 2019a; Muthukumar et al., 2020). This phenomenon is often referred to as benign overfitting. Morcover, the optimal statistical generalization may even be achieved for negative regularization À < 0 (Kobak et al., 2020; Bartlett et al., 2020; Tsigler et al., 2020). is unclear, however, if such interpolators, which have implicit small-norm biases, can also be optimal when there is shift between the training and test distributions. In particular, we ask: the optimal causal regularization be 0 or even negative, that is, do we observe benign causal overfitting? ? To show that the optimal regularization can be negative, causal risk at 0 is positive. We summarize our key findings in Theorem 4.1.
simply show that the derivative of the
Theorem 4.1 (Optimal Regularization can be Negative). For any causal model parameterized as in (I), the following cases distinguish between whether the min-norm interpolator is optimal or not.
1. For negative confounding strength Ç O the optimal causal regularization X be 0 negative. A necessary and sufficient condition for X S depends on the difference in causal and statistical signal-ta -o-iitt ratios and is given by
? mmax (1.7) SNRC SNRS
(1-
For positive confounding strength Ç 0 the optimal causal regularization is positive X and RC > RR. hence regularization is beneficial. This includes the ICM.
In the highly overparameterized regime (> + oo), the benefit of explicit regularization vanishes and both the causal and statistical risks of the ridge regression estimator converge their corresponding
8

---

null risks, independently of the regularization. We do not refer to this as benign overfitting. However, we can observe benign causal overfitting when the causal SNR is larger than the statistical SNR (C <: 0), which happens when causal and statistical parameter are strongly aligned. This implies that the norm of the statistical parameter is smaller than the norm of the causal parameter. Consequentially, statistical regressors are implicitly biased towards solutions of smaller norm and causal learning exhibits self-induced regularization. Compare this to benign statistical overfitting, which happens for certain alignments between the regression parameter 3 and the covariance matrix Σ. In our isotropic setting Σ Ia. we can therefore never observe benign statistical overfitting. but we can observe benign causal overfitting This phenomenon occurs in both the underparameterized as well as the overparameterized regime. The range of y for which the optimal causal regularization is negative increases with the dominance of the causal signal over the statistical signal. As y approaches the interpolation threshold, it becomes harder for the optimal causal regularization to be negative. When the causal SNR is smaller than the statistical SNR (ç > 0) and in particular under the ICM (0 < S 1), the optimal causal regularization is strictly positive and the benefit of explicit regularization does not vanish. This can be the case even when the optimal statistical regularization vanishes. To see this consider the statistical risk in the highly underparameterizee regime + O. In this regime, the benefit of explicit regularization vanishes and the min-norm interpolator indeed achieves the optimal statistical risk. The optimal causal regularization is given explicitly by 22 = C/(1 () for 0 < Ç S 1 and X2- = oo for Ç > 1. This is strictly positive and increasing in the confounding strength Ç. and in fact diverges as ( approaches (see Theorem 5.2).
On Optimal Regularization
In this section, we investigate two key questions which are natural in the context of our work. How does the optimal causal regularization X compare to the optimal statistical regularization Xg? What is the dependence of the optimal causal regularization X2 on the confounding strength Ç?
Optimal statistical vs. causal regularization When the training and test distributions coincide, approaches such as cross-validation or information criteria (for example AIC or BIC) can be used to estimate the regularization parameter for optimal sample generalization. However, choosing the correct regularization parameter for causal learning can be challenging without interventional data. To understand the optimal causal regularization, it is natural to compare to the optimal statistical regularization, which can usually be estimated from data. Interestingly, our analysis reveals that when confounding strength is positive Ç > 0 and in particular under the ICM one needs to regularize more strongly for causal generalization than for statistical generalization. However, when the confounding strength is negative, that is, when the causal signal dominates the statistical signal, the optimal causal regularization X2: can actually be smaller than the optimal statistical regularization A²x We formally present this result in Theorem 5.1.
Theorem 5.1 (Optimal Statistical vs. Causal Regularization). For any causal model parameter- ized as in (1), the condition Ç = O defines phase transition for the optimal regularization via
X'g. <=0
X Xg, and <>0 = XC Xg.
In particular under the ICM, the optimal causal regularization X is always strictly larger than the optimal statistical regularization X'g, unless Ç 0, in which case they coincide.
Dependence on confounding strength < The problem of causal learning from observational data is a distribution shift problem where the distribution of the training data is shifted from that of the test distribution. As discussed earlier in Proposition 3.2, the confounding strength measure ( quantifies expect the optimal causal regularization to
the strength of this distribution shift. Therefore, increase with confounding strength. Theorem 5.2 indeed confirms this intuition.
Theorem 5.2 (Increasing Confounding Strength Requires Stronger Regularization). Consider the family of causal models parameterized as in (1) that entail the same observational distribution. The optimal causal regularization X6 only depends on the confounding strength ( and N is an increasing function in Ç. More specifically, using e = - -NNRs " maxxx (1, (1,7)/11-763
e e<<<1 - No 00) with a 0,
- Offf S o and X = for 1

---

6 Summary and Extensions
We characterize the role of explicit regularization for causal learning from observational data by computing the asymptotic risk of ridge-regularized regressors and the min-norm interpolator (Theorem 3.1). Under the principle of independent causal mechanisms (ICM), we find that causal learning requires stronger regularization than statistical learning (Theorem 5.1). A practical implication is that the regularization parameter for causal learning should be chosen larger than what suggested by cross- vassdaanii.tion. We can precisely state how much larger based on an estimate of confounding strength (Janzing et al., 2017: Janzing et al., 2018). Beyond ICM, we show that strong alignments between causal and statistical parameters can cause self-induced regularization and lead to benign causal overfitting (Theorem 4. One could consider generalizing our assumptions: arbitrary covariances, shifts in the marginal distributions of covariates, soft interventions, more complex hypothesis classes, or non-linear causal relationships. Since the linear model already exhibits rich behavior, we focus in this paper understanding the simple setting. Below, we briefly discuss extensions of our analysis to causal learning under soft interventions, non-linearity, and -anssssiianttt.
Soft interventions It is not always appropriate to consider causal learning under hard interventions. Instead, it often of interest to consider soft interventions. In these settings, the qualitative statements derived from analysis still hold. To illustrate this, we consider the class of shift interventions where the structural dependence of the covariates z is not destroyed as in the case of hard interventions but the observed covariates are merely perturbed (i.e., interventions of the form do(x == x + 1/)). Then it turns out that Causal risk = Causal riskpard + Statistical risk. From our results, it then follows that under ICM, A statistical I zoft S Acaucal This also supports our intuition since under soft interventions, we typically aim to achieve a tradeoff between statistical and causal predictability. We include complete analysis under shift interventions in Appendix F.
Extensions to non-linear models It is feasible to extend the analysis to structural causal models that arise in a reproducing kernel Hilbert space corresponding to positive definite kernel (i.e, where the best statistical model f and the best causal model f are functions in some RKHS). There are two major technical challenges to deriving the theoretical analysis in such non non-linear settings. Both are beyond what can be done in this paper and are left for future work, but we briefly outline them below.
1. Extend the definition of confounding strength ( beyond the linear setting. Since such definition is non-trivial already in the linear setting, is challenging to meaningfully generalize this to the non-linear setting. However, under non-linear causal models in the RKHS, we can naturally extend this definition by replacing the Euclidean norms with functional norms in the RKHS. Generalizing the analysis beyond this setting would require further careful consideration.
2. Derive limiting expressions for causal risk of regularized regressors in a non-linear hythth- esis class. In the case of kernel regression, this would still be feasible via recent random matrix theory results [27]. By optimizing the limiting expressions with respect to the regularization parameter, one can obtain the parameter that achieves the optimal causal risk and subsequently identify the relationship between optimal causal regularization and confounding strength.
Beyond Gaussianity The analysis can be extended beyond the Gaussian setting by considering random variables generated by finite mixtures of Gaussians. Due to the universality phenomenon in the high-dimensional limit, we believe that our limiting expressions (and the qualitative messages derived henceforth) would be rather robust to shifts in the marginal distribution as long as moments of order (4 + 8) for some corresponding results can be found in Appendix G. They show that for distributions with finite 4th moments, the finite saiie- risks of the min-norm interpolator and causally optimally regularized ridge regressor closely match the theoretically derived asymptotic risks.
are bounded. We conducted experiments to verify this claim and the
Acknowledgments and Disclosure of Funding
This work has been supported by the German Research Foundation through the Cluster of Excellence "Machine Learning New Perspectives for Science" (EXC 2064/1:390727645 and the BMBF Tübingen AT Center (FKZ: 01 (IIS1803AA) The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Leena C Vankadara and Luca Rendsburg.
10

---

References
Angrist, Joshua D and Alan B Keueger (1991). "Does compulsory school attendance affect schooling and earnings?" The Quarterly Journal of Economics.
Bai, Zhidong and Jack W Silverstein (2010). Spectral analysis of large dimensional random matrices. Springer.
Bartlett, Peter L et al. (2020). "Benign overfitting in linear regression".. Proceedings of the National
Academy of Sciences.
Belkin, Mikhail (2021). "Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation". Acta Numerica.
Belkin, Mikhail, Siyuan Ma, and Soumik Mandal (2018). "To understand deep learning we need to understand kernel learning". International Conference on Machine Learning (ICML).
Belkin, Mikhail, Alexander Rakhlin, and Alexandre B Tsybakov (2019a). "Does data interpolation contradict statistical optimality?" International Conference on Artificial Intelligence and Statistics (AISTATS).
Belkin, Mikhail et al. (2019b). "Reconciling modern machine-learning practice and the classical bias-variance trade-off'' Proceedings of the National Academy of Sciences.
Blanchet, Jose, Yang Kang, and Karthyek Murthy (2019). "Robust Wasserstein profile inference and applications to machine learning". Journul of Applied Byrd, Jonathon and Zachary Lipton (2019). "What is the effect of importance weighting in deep learning?" International Conference on Machine Learning (ICML). Dicker, Lee H (2016). "Ridge regression and asymptotic minimax estimation over spheres of growing dimension", Bernoulli.
Probability.
Dobriban, Edgar and Stefan Wager (2018). fligh-dimenssonal asymptotics of prediction: Ridge regression and classification". The Annals of Statistics.
Donhauser, Konstantin et al. (2021). "Interpolation can hurt robust generalization even when there is no noise". Advances in Neural Information Processing Systems (NeurIPS).
Feldman, Vitaly (2020). "Does learning require memorization? a short tale about a long tail". Sympo- sium on Theory of Computing (STOC).
Gao, Rui, Xi Chen, and Anton Kleywegt (2017). "Distributional robustness and regularization in statistical learning". arXiv preprint 172.0:171.
Gulrajani. Ishaan and David Lopez-Paz (2021). "In Search of Lost Domain Generalization". Interna- tional Conference on Learning Representations (ICLR).
Hachem, Walid, Philippe Loubaton, and Jamal Najim (2007). "Deterministic equivalents for certain functionals of large random matrices". The Annals of Applied Probability.
Hastie, Trevor et al. (2022). "Surprises in high nigh-dimensional ridgeless least squares interpolation". The Annals of Statistics.
Janzing, Dominik (2019). "Causal Regularization". Advances in Neural Information Processing Systems (NeurIPS).
Janzing, Dominik and Bernhard Schölkopf (2010). "Causal Inference Using the Algorithmic Markov Condition". IEEE Transactions on Information Theory.
(2017). "Detecting Confounding in Multivariate Linear Models via Spectral Analysis". Jourral of Causal Inference.
(20188.. "Detecting non-causal artifacts in multivariate linear regression models". International Conference on Machine Learning (ICML).
Kobak, Dmitry, Jonathan Lomond, and Benoit Sanchez (2020). "The Optimal Ridge Penalty for Real- world High-dimensional Data Can Be Zero or Negative due to the Implicit Ridge Regularization." " Journal of Machine Learning Research (JMLR).
Kuhn, Daniel et al. (2019). "Wasserstein distributionally robust optimization: Theory and applications in machine learning". Operations research & management science in the age of analytics. Lemeire, Jan and Dominik Janzing (2013). "Replacing Causal Faithfulness with Algorithmic Inde- pendence of Conditionals". Minds and Machines. Liang, Tengyuan and Alexander Rakhlin (2020). "Just interpolate: Kernel "ridgeless" regression can generalize". The Annals of Statistics.
Marěenko, Vladimir A and Leonid Andreevich Pastur (1967). "Distribution of eigenvalues for some sets of random matrices". Mathematics of the SSSR-Skornik. Mei, Song and Andrea Montanari (2021). "The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve". Communications on Pure and Applied Mathematics.
11

---

Muthukumar, Vidya et al. (2020). "Harmless interpolation of noisy data in regression". IEEE Journal on Selected Areas in Information Theory.
Judea (2009). "Causal inference in statistics: An overview". Statistics surveys.
Peters, Jonas, Dominik Janzing, and Bernhard Schölkopf (2017). Elements of causal inference: foundations and learning algorithms. The MIT' Press.
Rothenhäusler, Dominik et al. (2021). "Anchor regression: Heterogeneous data meet causality". Journal of the Royal Statistical Society: Series B (Statistical Methodology).
Rubio, Francisco and Xavier Mestre (2011). "Spectral convergence for a general class of random matrices". Statistics & probability letters. Sagawa, Shiori et al. (2020). "An investigation of why overparamettrization exacerbates spurious correlations". International Conference on Machine Learning (ICML).
Shafieezadeh Abadeh, Soroosh, Peyman M Mohajerin Esfahani, and Daniel Kuhn (2015). "Dis- tributionally robust logistic regression". Advances in Neural Information Processing Systems (NeurIPS).
Shafieezadeh-Aaadeh. Soroosh, Daniel Kuhn, and Peyman Mohajerin Esfahani (2019). "Regulariza- tion via mass transportation". Journal of Machine Learning Research (JMLR). Silverstein, Jack W (1995). "Strong convergence of the empirical distribution of eigenvalues of large dimensional random matrices". Journal of Multivariate Analysis. Tsigler, Alexander and Peter L Bartlett (2020). "Benign overfitting ridge regression". arXiv preprint
arXiv:2009. 14286
Vankadara, Leena Chennuru al. (2021). "Causal Forecasting: Generalization Bounds for Autore- gressive Models". arXiv preprint arXiv:2 I 1.09831.
Xu, Huan, Constantine Caramanis, and Shie Mannor (2009). "Robustness and Regularization of Support Vector Machines." " Journal of Machine Learning Research (JMLR).
Zhang, Chiyuan al. (2021). "Understanding deep learning (still) requires rethinking generalization". Communications of the ACM.
Zhu, Shixiang et al. (2020). "Distributionally Robust Weighted k-Nearest Neighbors". arXiv preprint 040004.006.4404
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] While our analysis is exhaustive, we paid special attention to the additional ICM assumption described the end of Section 2. Further possible generalizations of our model are discussed in Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Right before Section 3.1, we cautioned against using our results on optimal causal regularization as an excuse to dismiss the hardness of causal learning.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs are deferred to the appendix and referenced correspondingly the main paper.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [N/A]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (c.g., code, data, models) curating/releasing new assets...
12

---

(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "p3Lle5H3gVg", "reviewer": "Reviewer_XX7K", "summary": "The paper gives a theoretical analysis about the relationship between interpolation and causal generalization. The paper proposes a new measurement of confounding strength \\zeta, and discusses under different cases of \\zeta, how the optimal strength of the regularization varies. ", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not expert of the causal learning and theoretical statistical learning. My evaluation is only based on a general view of the paper, not diving into the details (proof, etc). \nStrength:\n1. the problem seems important and open\n2. the theoretical results see solid and very interesting\n\nWeakness:\n1. This is a pure theory paper. While it is unnecessary, demonstrating the results on some real-world datasets/applications will be  promising and even more interesting. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "8MzdXeAYXHZ", "reviewer": "Reviewer_hNPk", "summary": "The causal risk (under do operator) of ridge regression estimator and min-norm interpolator have been studied. Intriguing and deep relationship between causal signal strength and regularization for ridge regression is studied. ", "questions": "(1) It seems $\\||\\Gamma\\||^2_{\\Sigma}$ in line 122 on p.3 should’ve been $\\||\\Gamma\\||_{\\Sigma^+}^2$ based on Proposition A.1, in which the variance is $I-A^\\top (A A^\\top)^+ A$?\n\n(2) Should the variance $y|do(x=x_0)$ be $\\sigma_{\\tilde \\varepsilon}^2+\\||\\Gamma\\||^2_{\\Sigma^+}$ in line 132, 154, 155 on p.4, because $do(x=x_0)$ eliminates the dependence of $z$ on $x$ so $z^\\top\\alpha$ is also regarded as an error term?\n\n(3) Perhaps it would be great to add something to clarify that the $E_{Y|X}$ in $R_X^C(\\hat\\beta)=E_{Y|X}[R^C(\\hat\\beta)]$ is taken with respect to $\\hat\\beta$ in line 214 on p.6.\n\n(4) For line 227 and 228 in Theorem 3.1, to extend the results for ridge regression in Eq. (7) and (8) to min-norm estimator, I think one also needs $d>n$ or $d/n\\to\\gamma\\in(1,\\infty)$ in addition to $\\lambda\\to 0$? \n", "limitations": "None.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The study of causal risk for linear interpolator is timely and impactful\n+ Theoretical analysis is solid", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5Vr1mZxa-7Z", "reviewer": "Reviewer_ST5R", "summary": "This paper studies the problem of learning causal effects with the existence of hidden confounders under a simple linear causal model. The authors provide an analysis of this learning problem in observational settings from the perspective of interpolation and regularization. They drive precise asymptotic risk of the min-norm interpolator and the ridge-regularized regressors. They further propose a new measure of confounding strength to characterise rich behaviours of causal learning under the ICM principle. ", "questions": "+ In the simple linear causal model, why is the dimension of $x$ assumed to be smaller than that of $z$? \n\n+ If the variables considered in the paper are non-Gaussian, will it change the theoretical results? If so, how?\n\n+ Throughout the paper, only the hard intervention is taken into account. What will happen to the soft intervention?\n\n+ How will these theoretical results change in the nonlinear setting? As we know, under the nonlinear causal model, it is infeasible to represent the causal effect using only one parameter (e.g., $\\beta$). In this case, I can imagine the theoretical results would be quite different. ", "limitations": "The authors discussed the limitations but not the potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The problem studied in this paper is of importance to the community.\n+ convincing theoretical analysis and easy to follow\n\nWeaknesses:\n+ The presentation of some paragraphs should be improved, since they involve a few unexplained terminology (e.g., U-shaped generalization, double-descent phenomenon, benign causal overfitting, etc), which is unfriendly to those unfamiliar with learning theory. I suggest that the authors add a bit explanations to them to improve readability.   \n\n+ The work only focuses on the simple linear causal models with Gaussian variables. Hence, their theoretical results are also limited to the linear setting. However, this is not the case in practice. \n\n+ Empirical evaluation is limiting. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-_JDGNz7gz", "reviewer": "Reviewer_ryke", "summary": "This paper investigates the risk for a (linear causal) parameter estimator in terms of interpolation and regularization. The authors propose several notions to study the difference between causal and statistical learning behavior and showed that causal risk curve exhibits a double-descent phenomenon. Further authors proved that, under a certain condition, a causal risk can be decreased with a stronger regularization than usual for learning statistical parameters.\n", "questions": "My only question is whether learning a statistical parameter with regularization (or through interpolation in overparameter regime) can be referred to as causal learning.\n\nThanks for an intriguing paper.", "limitations": "- I would like to see how confounding measure can be estimated, which is then one can practically learn causal parameter. This seems wrong... \n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\n- Highly interesting questions and mathematically rigorous analysis.\n- Concepts introduced and theoretical results can be beneficial to further study “causal learning” of parameters in the sense of modern “statistical learning”.\n\n\nWeaknesses\n- Regardless of (strong/weak) regularization, it is hard to accept (in my causal mind) that a regularization “enables” causal learning. Nudging parameters toward a little bit doesn’t seem qualified for the term “causal learning” (e.g., figure 2 is drawn without omega square = squared norm of confounding strength)\n- Whether to use strong regularization or not depends on confounding strength measure (and SNR_S), where confounding strength measure would not be typically measured. (I didn’t check out [20, 21] where the authors claimed that they can estimate the confounding strength.) If Gamma can be measured, empirical validation would be desired.\n\nSorry if there is any misunderstanding. I will correct if the authors can point out any.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept