Towards Understanding Extrapolation: a Causal Lens
Authors: Lingjing Kong
Guangyi Chen
Petar Stojanov
Haoxuan Li
Eric P. Xing
Kun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: generalization
extrapolation
identification
adaptation
Primary area: causal_inference
TL;DR: We investigate the theoretical aspects of extrapolation under a causal model formulation.

Abstract: Canonical work handling distribution shifts typically necessitates an entire target distribution that lands inside the training distribution.
However, practical scenarios often involve only a handful target samples, potentially lying outside the training support, which requires the capability of extrapolation.
In this work, we aim to provide a theoretical understanding of when extrapolation is possible and offer principled methods to achieve it without requiring an on-support target distribution.
To this end, we formulate the extrapolation problem with a latent-variable model that embodies the minimal change principle in causal mechanisms.
Under this formulation, we cast the extrapolation problem into a latent-variable identification problem.
We provide realistic conditions on shift properties and the estimation objectives that lead to identification even when only one off-support target sample is available, tackling the most challenging scenarios.
Our theory reveals the intricate interplay between the underlying manifold's smoothness and the shift properties.
We showcase how our theoretical results inform the design of practical adaptation algorithms. Through experiments on both synthetic and real-world data, we validate our theoretical findings and their practical implications.

Towards Understanding Extrapolation: a Causal Lens
Lingjing Kong!*
Guangyi Chen 1.2+
Petar Stojanov
Haoxuan Li² Eric P. Xingl
Kun Zhang
Carnegie Mellon University
Mohamed bin Zayed University of Artificial Intelligence 3 Broad Institute of MIT and Harvard, Cancer Program, Eric and Wendy Schmidt Center
Abstract
Canonical work handling distribution shifts typically necessitates an entire target distribution that lands inside the training distribution. However, practical scenarios often involve only handful of target samples, potentially lying outside the training support, which requires the capability of extrapolation. In this work, we aim provide theoretical understanding of when extrapolation is possible and offer principled methods to achieve without requiring an on-support target distribution. To this end, we formulate the extrapolation problem with latent-variabl model that embodies the minimal change principle in causal mechanisms, Under this formulation, we cast the extrapolation problem into latent- variable identification problem. provide realistic conditions on shift properties and the estimation objectives that lead to identification even when only ouf-su target sample is available, tackling the most challenging scenarios. Our theory reveals the intricate interplay between the underlying manifold's smoothness and the shift properties. We showcase how our theoretical results inform the design of practical adaptation algorithms. Through experiments both synthetic and real- world data, validate our theoretical findings and their practical implications.
Introduction
Extrapolation necessitates the capability of generalizing beyond the training distribution sup- port, which essential for the robust deployment machine learning models in real-world scenarios. Specifically, given access to a source distribution Dere p(Xaros Ysrc) with support Xrte := (x)p()ss(()) and one or few out-of-support samples Xıge & Xav. the goal of extrapolation is to predict the target label Yigt. For example, if the training distribution includes dog images, we aim to accurately classify dogs under unseen camera angles, lighting conditions, and backgrounds. While intuitive for humans, machine learning models can be brittle to minor distribution shifts [1
Addressing distribution shifts has garnered significant attention from the community. Unsupervised domain adaptation under covariate shifts addresses the shift of the marginal distribution p(x) across domains. However, canonical techniques such as importance sampling and re- re-weighting [5/9] are predicated on the assumptions of overlapping supports Supp(pnr( C Supp(Pare(x)) and the availability the entire target marginal distribution (x). Similarly, domain generalization 110 assumes access to multiple source distributions Pare (x, y) whose supports jointly cover the target distribution. In addition to these methods, test-time adaptation (TTA) [3 is particularly relevant to our discussion of extrapolation. TTA addresses distriiiitiiiinonnn test samples the individual sample level. Canonical methods include updating the source model with entropy- baaee or self- supervised losses on target samples. However, most TTA research focuses empirical aspects, with limited theoretical formalization [1] Most related to our work, Kong et al. [88 and Li et al.
Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

x = s),
x = g(C,s)
x (C,
Larger shifts
Larger shifts
(c,(5) (5)
?
9c,
Be,
Beg
Bc,(8)
(s,(s)
Support region
Extrapolation region
Support region
Extrapolation
Unidentifiable region
Support
Extrapolation
region
region
region
(a) Extrapolation task.
(b) Dense shift (Theorem 4.2)
(c) Sparse shift (Theorem
Figure/ 1: Illustration of extrapolation and our theoretical conditions. The horizontal axis represents the changing variable s, ranging from the source support to out-of-support regions. The vertical axis represents the observed data x living on the manifolds indexed by different values of the invariant variable c. Figure (a) demonstrates that given a point out of support it is unclear which class manifolds it belongs to. Figure (b) illustrates the dense shift condition (Theorem 4.2) where potentially changes all pixels in the images, such as the camera angle in the example. In this case, we can identify the invariant variable under a moderate amount of shift until the shift becomes excessive. For instance, the back view of the cat in the figure could be confused with other animals. Figure (c) illustrates the sparse shift condition (Theorem [4.4 where s influences a limited number of pixels, such as the background in the example. In contrast to the dense shift, we can identify under the sparse shift regardless of its severity. In the figure, there no ambiguity of the class "cow" even though the background has changed to the moon.
19 propose theoretical frameworks to characterize distribution shifts and explore conditions for identifying latent changing factors. However, these frameworks assume access to multiple source distributions with overlapping supports, which are not directly applicable to the extrapolation problem considered in this work, where we have potentially only one out-of-support target sample Xigt.
Since the target Ytgt can be arbitrarily out of the source support (Figure Tap, extrapolation is ill- posed without proper assumptions the relationship between the source and the target. In this work, we formulate a latent-variable model that encodes a minimal change principle to address this ill-posednees. Specifically, we assume that latent variable determines such that == g(z). The minimal change principle entails the following two assumptions on the generating process. 1) The out-of-support nature of Xtgt stems from only a subspace of z, denoted s. while the complement partition e for the target sample Xtgt is within the training support, i.e., 2 == [c, s). Segt & San. and Cigs C.re. 2) The changing variable s only controls non- semantic attributes in x and thus doesn't 't influence the label i.e., y := (c). This formulation attributes the seemingly complex shifts in the pixel space of x simple intrinsic changes (in the sense that this change involves only s) in the latent space, allowing us to reason about the transfer via the invariant variable c. Under our formulation, extrapolation amounts to identifying invariant latent variables c, with which a model f + trained on the labeled source dataset can be directly applied to the target sample.
In light of this formulation, we investigate the identifiability conditions of the invariant variable c. We propose two sets of conditions addressing two regimes of the influence from s on x. We refer to the case where all dimensions x; (e.g., all pixels in an image) could be influenced by the changing variable s as the dense influence and the case where only limited subset of dimensions ny is affected as the sparse influence. Specifically, our first condition (Dense Influence, Theorem 4.2 states that if is's 's influence is dense, then assuming that e takes on only finite values, extrapolation requires the manifold associated with each value of с (i.c., -(c,-) over s) to be adequately separable, and the target changing variable Suge to be close to the source support Sae- Intuitively, if images from two classes (two values of c) are sufficiently distinguishable, such as cats and dogs, we can still recognize the class of the target sample Xegt, even if it has undergone moderate unseen shifts on all pixels like camera angles and positions controlled by (Figure (bb). Our second condition (Sparse Influence, Theorem 4.4) states that if s's influence is sufficiently sparse, then extrapolation can occur regardless of the distance of Sigt to the source support San. Intuitively, we can recognize the class "cow" even only the background is changed, regardless of the severity (Figure Ic).

---

Our theory provides insights into the interaction between the underlying manifold's smoothness, the out-of-support distance, and the nature of the shift. We conduct synthetic data experiments to validate our theoretical results. Moreover, we discuss the relationship between our results and TTA approaches. In particular, we apply our theoretical insights to improve autoencoder- MAE-TTT [20] and observe noticeable improvements. Additionally, we demonstrate that basic principles (sparsity constraints) from our framework can benefit the state-of-the-ar TTA approach TeSLA [2]]. Our empirical results not only show the practical viability of our theory but also pave the way for further advancements in the field.
In summary, our contributions are threefold:
We formulate the extrapolation task as a batent-variable identification problem. Our latent-variable model encodes complex changes in observed variables to a partition of latent variables, allowing us to reason about transferability from the source to the target through latent variable identification.
We provide identification guarantees for the proposed latent-variable model, including shifts of distinct properties (dense vs. sparse) and corresponding conditions on the generating process. Our theory provides an essential understanding of when latent-variable identification is possible without accessing an entire target distribution and assuming overlapping supports as in prior work [T8 19
Inspired by our theory, we propose to add likelihood maximization term to autoencoder-based MAE-TTT [20] to facilitate the alignment between the target sample and the source distribution. In addition, we propose sparsity constraints to enhance the state-of-the-an TTA approach TeSLA (21). We validate our proposals with empirical evidence.
2 Related Work
generalization has attracted significant attention in recent years. Unlike our work, the bulk of the work devoted generalizing to target distributions on the same support as the source distribution [22] 23 g Recent work (24 investigates extrapolation in the form of compositional generalization by resorting to structured generating functions (e.g., additive, slot-wise). Another line of work [28 30 studies extrapolation in regression problems and does not consider the latent representation. Saengkyongam al. [3] leverage latent variable model and linear relations between the interventional variable and the latent variable to handle extrapolation. In this work, we formulate extrapolation as latent variable identification problem. Unlike the conditions in prior work, our conditions do not constrain the form of the generating function and are more compatible with deep learning models and tasks.
Extrapolation.
Latent-variale variable identification for transfer learning. In the latent-variable model literature, one often assumes latent variables z generate the observed data x (c.g.. images, text) through generating function. However, the nonlinearity of deep learning models requires the generating function to be nonlinear, which has posed major technical difficulty in recovering the original latent variable [32] To overcome this setback, a line of work (3 assumes the availability of an auxiliary label u for each sample x and under different u values, each component 2; of experiences sufficiently large shift in its distribution. Since this framework assumes all latent components' distributions vary over distributions indexed by u, does not assume the existence of some shared, invariant information across distributions, which is often the case for transfer learning tasks. To address this issue, recent work [18 [99 introduce a partition of z into an invariable variable e and an changing variable (i.c., [. s)) such that c's distribution remains constant over distributions. They show both e and can be identified and one can directly utilize the invariant variable c for domain adaptation. However, their techniques crucially rely the variability of the changing variable s, mandating the availability of multiple sufficiently disparate distributions (including the target) and their overlapping supports. These constraints make them unsuitable for the extrapolation problem. In comparison, our theoretical results give identification of the invariant variable e (the on-support variable in the extrapolation context) with only one source distribution Pare (x) and as few as one out-off support target sample Xigt through mild assumptions on the generating function, directly tackling the extrapolation problem.
Please refer to Section Al for more related work.
3

---

3 Extrapolation and Latent-Variable Identification
Given the labeled source distribution p(xare: srre). our goal is to make predictions on a target sample Xigt outside the source support (Xegt & Asre). While more target samples would provide better information about the distribution shift, practice, we often have only a handful of samples to work with. Therefore, we focus on the challenging scenario where only one target sample Xtgt is available.
Making reliable predictions on out-of-support samples Xigt is infeasible without additional structure. Real-world problems where humans successfully extrapolate often follow minimal change principle: they involve sparse, non-semantic intrinsic shifts despite complex raw data changes. For example, person who has only seen a cow on a pasture can recognize the same cow on a beach, even the background pixels change significantly. Here, the cow corresponds to the part of the latent variable that remains within the support of the source data, which we call the invariant variable с (Cage E C while the background change corresponds the complement that drifts off the source support, which we call the changing variable s (Stgt & S.r). Clearly, extrapolation is impossible if the intrinsic shift dense (i.e., all dimensions change, z = s) or semantic (i.e., y is function of s). For instance, if the variable s also alters the cow's 's appearance drastically, making it unrecognizable, extrapolation fails. We define the data-genea process to encapsulate this minimal change principle, follows:
c p(c),, >=((sc); x = g(z). y By (c).
The data-generating process. The invariant latent variable e and the changing latent variable 8 jointly generate the observed variable æ. The dashed line indicates potential statistical dependence.
Figure 2:
In this process, the latent space E Z Rd comprises two subspaces: the invariant variable
c E C C Rd and the changing variable s E S C Rd. We define 2 := Zsre U {Zuge} as the source support augmented with the target sample Zegt and similarly X' and S. The invariant variable с encodes shared information between the source distribution p(Xare) and the out-of-support target sample Xigt, while the changing variable s describes the shift from the source support X Hence, Cigt E Cusc and Sugs & Sae- The variables z == [c, s] jointly generate the observed variable x E X C dx through an invertible generating function g : Rd - dxd Furthermore, we assume that the label y originates from the invariant variable c. This assumption reflects the reality that factors such as camera angles and lighting do not affect the object's class in an image.
Our variable model adheres to the minimal change principle in two key ways: (1) the target sample's 's out-of cal--ff-uupppp supprrt nature arises from only a subset of latent variables s, and (2) these changing variables are non non-semantic, thus not influencing the label y.
Extrapolation and identifiability. Under this framework, extrapolation is possible we can identify the true invariant variable in both the source distribution Parc and the target data Xigt. This allows us to learn a classifier fals e ++ y on the labeled source distribution Pare (xx, y). Since the target sample's 's invariant variable falls within the source support (Ctgt E Care), this classifier fels can be directly applied to the target sample Cigt. Thus, the task of extrapolation reduces to identifying the invariant variable e in both the source distribution p(Xare) and the target sample Xigt. In Section 4 we explore the conditions for identifying the invariant variable c.
Given the above reasoning, we define identifiability in Definition 3.1 (i.e., block-wise identifiabil- ity 7 24 which suffices for extrapolation.
Definition 3.1 (Identifiability of the Invariant Variable c). For any and X2, their true invariant variables C1. C2 are equal if and only the estimates é₁, é2 are equal: e₁ C2
4 Identification Guarantees for Extrapolation
In this section, we provide two sets of conditions on which can identify the invariant variable and discuss the intuition and implications.
As discussed in Section 3, we need to identify the target sample Xtgt with source samples x that share the same invariant variable values with the target sample, i.e., Care = Ctgt- This enables us to obtain the label of Xig: by assigning the label of such X The shift between the source distribution

---

P(Xere) and the target sample Xigt originates from the out- -upport nature of the changing variable Sigt, Lee, Sigt & Sare it is crucial to impose proper assumptions on the influence of s on x so that x retains sufficient footprints of e for identification beyond the source support.
We denote the Jacobian matrix of the generating function 9 J g and x's dimensions under the influence of s as [z) := {i E [dx] : 3j E {de + l..... ( s.t., [J (z)]e, # 0}. We note that the set Ig (z) is a function of z, since the influenced dimensions may vary over z. Intuitively, if influences x in a dense manner, i.e., large [z)(z) for potentially all dimensions 2;, there may not be dimensions of x serving as clear signatures of c, thereby increasing the difficulty of identify c. Additionally, the degree to which the changing variable s is out- out-of-support plays a critical role the further the target changing variable- the further the target changing variable Segt deviates from the source distribution support Sare. the more severe and unpredictable the shift becomes, making it harder to retrieve c. In the following, we formalize conditions on the influence of from these two perspectives, revealing an interesting trade-off and interaction between these factors.
Notations. The true generating process involves c, s, distributions p. and g (Equation we define their statistical estimates with ć, 8. p. and 9 through the objectives we will introduce We assume that the estimation process respects the conditions of the corresponding true-generating process.
4.1 Dense-shift Conditions
We begin by investigating scenarios where there are no constraints on the number of dimensions of x (i.e., the number of pixels) influenced by the changing variable s, i.e., potentially large [z)], which we term as dense shifts. For images, these shifts encompass global transformations such as changes in camera angles and lighting conditions that could potentially affect all pixel values (Figure Tb
Understanding the problem. As dense shifts could influence all the dimensions of x, every dimension could be out of the source support and there might not be dimensions of x that solely contain the information of c. Consequently, relying on any subset of x dimensions to infer the original becomes untenable. For instance, consider a scenario where the source distribution contains frontal- view images of a cat, while the target sample portrays the same cat from a side view (Figure Ib]. The model cannot recognize these two images the same cat (the same c) by matching a specific part of the side view, say the cat's nose, to samples in the source distribution because this cat's nose only shows up as a front view and can be vastly different in terms of the pixel region and values. The model cannot match specific features such as the cat's nose, between the side target and the source distribution, as the pixel region and values for the nose drastically differ.
Our approach. For the reasons above, we need to constrain such dense changes so that even when all dimensions are affected, the target sample adheres to some intrinsic structure determined by the underlying Cigt and remains distinguishable from samples of с * Cegt In many real-world distributions, we can interpret
the embedding vector of classes or other categories, with each value indexing a manifold ((C,)) -) over s. If manifolds are smooth and sufficiently separable from each other, they should exhibit limited variations in the adjacent region to the training support, avoiding confusion between distinct categories. For example, there exists a noticeable distinction between cats and lions, such that moderate illumination changes would not cause confusion until illumination significantly obscures distinguishing features. In the following, we formalize these structures by assuming a finite cardinality of e and constraining the distance of Sigt to the support Sare.
A Additional notations. We denote with Ja an upper bound of the Jacobian spectrum norm: J| (z)
Ju the support. In Appendix A2 we show Ju due Assumption 4.1 [i and Assumption 4.1
i
We denote with D(c1, C2) the Eg distance between two manifolds on the support boundary: D(c1, 02) inf iifss= E [lg(c), $1) - g(C2, s2)ll. where we denote the boundary of source support with Bd We denote with D Sare) the minimal &g distance between and the source support Sare i.c., D(s,Sacc == inf ||s - Sarell.
Assumption 4.1 (Identification Conditions under Global Shifts).
i [Smoothness & Invertibility]: The generating function g in Equation 7 is a smooth invertible function with a smooth inverse everywhere
ä |Compactnesss]: The source data space Xare C BRd is closed and bounded.
"We slightly abuse the notation p to denote density functions for continuous variables or delta functions for discrete variables.
5

---

ii [Discreteness]: The invariant variable takes on values from a finite set: C = {C+ [K)(
iv (Continuity]: The probability density function p(s]c) is continuous over E Sares for all C.
[Out-of-supoort Distance]: The target sample's out-support components Segt 's distance from the source support Sane is constrained: infaes... ||Stgt - s|| < minegcN Dicrate)
2Ju
Discussion on the conditions. As discussed above, the main conditions revolve two key factors: the discrete structure of the invariant distribution of p(c) in Assumption 4.1 and the off- off-support distance of the changing variable s in Assumption 4.1 The discrete structure of p(c) is applicable to many real-world scenarios, especially classification tasks where the semantic invariant information often manifests as discrete class labels or other categorical distinctions. While this assumption is typically valid, it can be extended to encompass continuous dimensions in the invariant variable c := [ce, cu] where Co and с₁ stand for the continuous and discrete dimensions, respectively. In such cases, we can group the continuous dimensions c, with the changing variable and the same proof would give rise to the identification of the discrete part Ca. which suffices for classification tasks. The off-support distance condition involves the smoothness of the generating function g. where smoother generating function allows more leeway for the target changing variable Sigt to deviate. When s controls the camera angle, may be able recognize slightly sided view of cats after seeing front views in the source until the deviates too far and all images become back views, potentially leading to confusion with other animals (Figure lbp. Assumption 4. Jensures that the generating process preserves the latent information, which is widely adopted in the literature [18] 19] 35
36
33 38)
Specifically, this guarantees that manifolds indexed by distinct values of are separate from each other, maintaining strictly positive distances between them. Assumption 4. j are technical conditions
mirroring realities that pixels values are bounded and the changing variable s often represent attributes that vary gradually across its support (e.g., lighting and angles).
Theorem 4.2 (Extrapolation under Dense Shifts). Assuming a generating process in Equation we estimate the distribution with model (ğ. p(c), p($)) with the objective:
p(Cig)) Subject to: p(x) = p(x), Vx E Karei Sigt E arg inf D(s, Sare)-
(2)
Under Assumption 4.1 the estimated model can attain the identifiability in Definition 3.1
Proof sketch. We estimate the generative process through maximum likelihood estimation on the we can establish the identification of c on the support (38], i.c., é₁ = è₂ > C1 C2 for X1,X2 X2 E Xare. This implics that all samples Nare on a given source manifold g(c.-) share identical values of è. In the objective, we maximize the likelihood p((ttt) drive Öigt to match one of the discrete values of é with nontrivial probability mass. Given the identification of on the support, this equates to assigning Xigs to manifold g(c"; with c E C. Our task now switches to ensuring that this the correct manifold for Xigt. i.e., c Cigt. To accomplish this, we select the estimated model that uses the minimal off- support distance on å (i.e., D(s Sare)) to explain the off-support nature of Xigt- This also embodies the minimal change principle. This and Assumption 4 guarantee that only the correct manifold (g(Crgt.") -)) can effective capture Xegt, thereby facilitating the desired identification. In practice, these constraints can be implemented through Lagrange multipliers. Full proof is given in Appendix A2
source distribution p(x) p(x). Under Assumption 4.1
4.2 Sparse-shift Conditions
We now examine cases where the changing variable s influences only a subset of dimensions of x, i.e., a limited [z)]. which we refer to as sparse shifts. For image distributions, these shifts include local corruptions or background changes that do not alter foreground objects (Figure
Additional notations. We define the index set (z) under the influence of c and the indices under the the exclusive influence of [o)s(z) == [z( Iz(z).
Understanding the problem. In contrast to the dense-shift scenario, here we have a non-trivial subset of dimensions [x]x that are unaffected by the changing variable s. Consequently, if these dimensions carry sufficient information about c, we can exploit them to directly recover the true regardless of the distance xl|z(2) deviates from the support. contrast, the dense-shift scenario, need to constrain the out-of-support distance of and assume the discreteness of c. Consider

---

scenario where fixed e represents a specific cow and s controls only the background. Despite the variation in the target background (e.g., desert or space), we can effectively match the cow in the target image to the correct source images (see Figure [c.. While this may seem intuitive for humans, it nontrivial for machine learning models to automatically recognize the region [x]zo,.(2) especially given its potential variation across z.
Our approach. For image classification, (x) corresponds to foreground objects (or a portion) unaffected under sparse changes induced by s (c.g., background changes). Humans can recognize this region because the pixels within it are strongly correlated (e.g., cow features). This observation motivates us to formalize such dependence structures in natural data to enable automatic identification. Assumption 4.3 (Identification Conditions under Local Shifts).
i (Smoothness & Invertibility:: The generating function g in Equation ii invertible and differen- fiable, and its inverse is also differentiable.
ii UIvariant Variable Informativenesss: The dimensions under c's exclusive influence is uniquely determined: for a fixed с E C, (x(TTe.(e.ss) * [xlTeeste" sn) for any c* + fc, $1 E S, and $2 E S.
ii [Sparse Influence]: At any z 2, the changing variable influences at most dg dimensions of x, ie.. [z) S dg. Alternatively, the two variables с and s do not intersect on their influenced dimensions Iz(z) (Z) 0.
iv [Mechanistic Dependence]: For all z. any nontrivial partition Pi, P1,P2 of the dimensions Ic (z) yields dependence between the sub-matrices of the Jacobian J, (z): rank([], (z)((]Izo (o) (z) < [z) p. (z)) + rank([J (z)( (z)).. 9 [J
rank([J,
Discussion on the conditions. Assumption 4.3 ffi stipulates that the influence of s is sparse, either in terms of dimension counts or in its intersection with the influence from c. It is noteworthy that while the influence is sparse, its location can vary over images, indicated by the dependence of Iz on Consequently, it can capture diverse image corruptions and background changes. Assumption 4.3 ensures that [x].... is sufficiently informative about c. For instance. it precludes scenarios where a sparse corruption alters the top stroke of "7" to resemble "1", rendering the uncorrupted region fundamentally unidentifiable. Assumption 4.3 iv enforces the dependence alluded in our previous discussion: the unaffected dimensions [x exhibit mechanistic dependence across them, characterized by the Jacobian rank (39]. Thus, generating separate parts of an object necessitates more capacity than generating the entire object, as the dependence across the two parts can inform each other's generation. This inherent dependence enables the identification of the unaffected region.
Theorem 4.4 (Extrapolation under Sparse Shifts). Assuming generating process in Equation estimate the distribution with model (g. p(č). p($)) with the objective:
sup p(Cugt), Subject to: p(x) p(x), Vx E Xare-
Under Assumption the estimated model can altain the identifiability in Definition 3.1
Proof sketch. Maximizing the likelihood p(Gugs) assigns value č* E C to Õigt Building motivation, we leverage mechanistic dependence (Assumption 4.3 iv to identify the unaffected dimension indices Lz))(z)) [dx] with our estimated model. In other words, we have [z)
Zers(2 (2). Consequently, the unaffected dimensions in the estimated variable equal their counterparts in the true model: [ ) =
Furthermore, Assumption 4.3 stipulates that the dimensions in the target sample (Kigelzera
cannot be attained by other o Cigt, so
we have established that é* corresponds to the correct value Cigt- Full proof is in Appendix A3
It's worth noting that unlike the global shift case (Theorem 4.2 here we do not place constraint on the sup-oo-support-nness of Segt, a point we empirically verify in Section 6.3
4.3 Implications for Practical Algorithms
Generative adaptation. Our theoretical framework, inherently a generative model, can be imple- mented through auto-encoding over the source distribution and the target. Akin to our estimation framework, MAF-TTT (20) trains a masked auto-encoding model (fane and fâce) on the source distribution and adapts to target samples through the auto-encoding objective. Consequently, we have
7

---

Table 1: Synthetic data test accuracy under both dense and sparse shifts across a range distances.
Shifts
Dense
Sparse
Distance
12.0
18.0
24.0
30.0
18.0
24.0
30.0
36.0
Only Source
0.59
0.55
0.45
0.45
0.54
0.54
0.56
0.52
IMSDA 18
0.46
0.48
0.48
0.50
0.50
0.36
0.40
0.54
Ours
0.78
0.69
0.72
0.72
0.72
0.72
0.76
0.70
Saor (Jene (x) for x E Xare U (x which approximates the distribution aspect of our estimation objectives Equation 2nnd Equation S
Despite the resemblance the reconstruction objective, MAE-TTT does not explicitly perform the representation aligmment as our objectives - classifier fels is only trained on the labeled source distribution, which takes in fene's 's output 2 and produces logit values. In addition, our objectives entail maximizing the target likelihood p ((Cugt to align Eigi the source support Cure. As large logit values indicate the sample is close to distribution modes [40] [1] and Sene is enforced invertible through auto-encoding,
can interpret minimizing the entropy of (atg)) as filtering 2itt obtain Crgt and driving it towards the modes of p(c). Therefore, we implement the entropy minimization loss inf - log Eg, (zta(Zug)) (Zegt) y log(.fel ( (Zegt)) )y) as surrogate for maximizing p(c). We show that this significantly boosts the performance of MAE-TTT in Section 6.1
Regularization. While our objectives simultaneously involve the source distribution and the target sample, the source distribution may not be accessible during adaptation. Aggressive updates on the target sample may distort the source information stored the model and ultimately impair the performance. To address this, propose to impose regularization during adaptation to enforce minimal changes and preserve the source information. In Section 6.2 we instantiate this with low-rank updates and sparsity constraints, showcasing the resultant benefits.
the nource-pretrainge backbone
Synthetic Data Experiments
In this section, we conduct synthetic data experiments on classification to directly validate the theoretical results in Section We present additional experiments on regression in Section A4.2]
Experimental setup. We generated the synthetic data following the generative process in Equation] T with de 4 and dg 2. We focus binary classification and sample class embeddings e₁ and C2 from (0, I,) and (V22,, Ie) respectively. We sample Sare from a truncated Gaussian centered the origin and sample Sigt at multiple distances from the origin. For the dense-shift case, we concatenate e and s and feed them to well-conditioned 4-layer multi-layer perceptron (MLP) with ReLU activation to obtain x. For the sparse-shift case, we pass vector. We duplicate dimensions of this vector and add to it. The final x is the concatenation of the 4-d vector and the 2-d vector. We sample 10k points for the source distribution and target sample for each run. We perform 50 runs for each configuration and compute the accuracy on the target samples. More details can found in Appendix A4
a 4-layer MLP obtain 4-d
Results and discussions. We compared our method with iMSDA 18 and a model trained only on source data. The results in both dense and sparse shift settings are summarized in Table Our method consistently outperforms both baseline methods (nearly random guesses) by large margin on all sub-settings, validating our theoretical results. The results on iMSDA suggest that directly applying domain-adaptation methods to the extrapolation task may result in negative effects for lack of the target distribution their training.
Real-world Data Experiments
We provide real-world experiments validate our theoretical insights for practical algorithms (Section 4.3) and theoretical results (Section 4.2) More results can be found in Appendix A5
TThe code is provided here
8

---

Table Comparison SOTA TTA Methods on CIEARIO-C, CIEARI00-C, and ImageNet-C. Average
error rates over 1S lest corruptions are reported. Baseline results are from Tomar et al. [2T]. Values are (means + standard deviations) over three random seeds. indicates our reproductions.
Method
CIFAR1O-C
CIFAR100-C
ImageNet-CC
Source 21
29.1
60.4
81.8
BN
42
15.6
43.7
67.7
TENT 15
14.1
39.0
57.4
SHOT 43
13.9
39.2
68.7
TTT+ [1]
15.8
44.4
59.3
TTAC 44
13.4
41.7
58.7
TeSLA-s 21
12.1
37.3
53.1
TeSLA-s+SC
11.7 0.01 +
37.0 + 0.06
50.9 = 0.15
TeSLA* 21
12.5 0.04
38.2 + 0.03
55.0 + 0.17
TeSLA+SC
12.1 0.11
38.0 + 0.13 |
54.5 = 0.12
6.1 Generative Adaptation with Entropy Minimization
As discussed in the first implication in Section 4.3 we incorporate an entropy-minimization loss to MAE-TTT and compare with the original MAE-TT.
Experimental setup. We conduct experiments on ImageNet-C 45 and ImageNet100-C [46] with 15 different types of corruption. For the baseline, utilize the publicly available code of MAB-TTT. In our approach, we do not directly integrate the entropy-minimizatton loss into the MAE-TTT framework. This is because the training process of self-supervised MAE relies masked images, whereas entropy-minimizae introduce additional training steps with unmasked images and apply the gutropy-minimizann during these steps. Specifically, the training process for each test-time iteration is split into two stages. We first follow the MAE-TTT approach by inputting masked images and training the model using reconstruction loss. In this stage, only the encoder is updated. Then, we input full images (32 in batch) and optimize the model with the entropy minimization loss following SHOT 43 In this stage, both the encoder and classifier are optimized. The learning rates for both stages are set the same.
requires the classification of the entire image. To address this, we loss
Table 3; Test accuracy (%) on ImageNet-C: C. The baseline results are from Gandelsman et al. 20
Ace (%6
brigh
cont
dcfoc
clast
fog
frost
gauss
glass
impul
jpeg
mota
pixel
sbot
snow
zoom
Avg
Joint Train
62.3
4.5
26.7
39,9
25.7
30.0
5.8
16.3
5.8
45.3
30.9
45.9
7.1
25.1
31.8
26.88
Fine-Tune
67.5
7.8
33.9
32.4
36.4
38.2
22.0
15.7
23.9
51.2
37.4
51.9
23.7
37.6
37.1
34.45
VIT Probe
68.3
6.4
24.2
31.6
38.6
38.4
17.4
18.4
18.2
51.2
32.2
49.7
18.2
35.9
32.2
32.06
TTT-MEE
69.1
34.4
50.7
44.7
50.7
30.5
36.9
32.4
63.0
41.9
63.0
33.0
42.8
45.9
45.92
Ours
73.8
14.0
33.6
69.0
47.8
64.6
38.6
42.2
36.6
68.4
32.4
67.4
41.2
51.2
35.4
47.77
Comparison with baselines. Table 3 we compare our method with the baseline MAE-TTT 20] and other baselines therein. We can observe that our algorithm largely boosts the performance of the MAE-TTT baseline over most corruption types. This corroborates our theoretical insights and showcases its practical value.
Table 4:
Understanding entropy-min minimizaaionn steps on ImageNet100-C. Values are classification accuracy (mean and standard deviation) over three random seeds.
Using entropy-minimization Step
Source MAE-TTT
2 Steps
Steps
Ace 50.29
59.12 + 0.5
63.99 0.5
65.09 + 0.2 65.01 0.39
Understanding entropy-minimizztion steps. T the results of entropy-minimine with different training steps. The results indicate that the additional entropy-minimizen steps significantly enhance the performance of the MAF-TTT framework, demonstrating the synergy between auto-encoding and hatropy-minmmizatee indicated in our theoretical framework.

---

6.2 Sparsity Regularization
As suggested by the second implication Section 4.3 we integrate sparsity constraints into the state- of-the-an TTA method, TeSLA/TeSLA-s (2T). Although theoretical results rely generative
model, we demonstrate that our implications are also applicable to discriminative models.
Experimental setup. We conduct experiments the CIFAR1O-C, CIFAR100-C, and ImageNet-C datasets [45), following the protocols outlined for TeSLA and TeSLA-s [21], with and without training data information. In the pre-train stage, we apply the ResNet50 47 as the backbone network and follow prior work [4] 44 to pre-train on the clean CIFAR IO, CIFARI00, and ImageNet training sets, with joint contrastive and classification losses. In the test-time adaptation process, adopt the sequential TTA protocol as outlined in TTAC [4 and TeSLA [2I] This protocol prohibits the change of training objectives throughout the test phase. To encourage sparsity, we add low-rank adaptation (LoRA) modules 48 to the backbone network, which limits the adaptation to low intrinsic dimensions. Beyond LoRA, we further implement a masking layer with corresponding sparsity constraint (&1 loss) to filter out redundant changes. More details can be found in Appendix AS
Results analysis. The average error rates under 15 corruption types for all CIFARIO-C, CIFARIO0- C, and ImageNet-C datasets are summarized Table We can observe that sparsity constraints consistently improve performance over the current SOTA method, TeSLA/TeSLA-s, across all three datasets. The lightweight nature of the sparsity constraint and its consistent performance enhancements make it a valuable addition. This demonstrates the potential of sparsity constraints as versatile, plug-and and-play module for enhancing existing TTA methods.
6.3
Shift Scope and Severity
To investigate the trade-off between the shift scope (dense vs. sparse) and severity, we sim- ulate different levels of corruption severity and corrupted region sizes and evaluate a classical TTA method TENT [15] these configurations. Following [45] we inject impulse noise to the CIFARIO dataset, with noise levels ranging from 1 to 10 to simulate various severity levels. To control the shift's scope, we crop regions of var- ious sizes and introduce corruption only to this region. Figure 3 displays classification error curves under various shift severity levels and re- gion sizes. We can observe that classification errors rise with increasing noise levels and region sizes. Notably, for large block sizes (dense shifts), the performance dramatically declines and even collapses as the severity level rises, whereas the performance remains almost constant over all severity levels in the sparse shift regime, verifying the theoretical conditions for Theorem 4.2 and Theorem 4.4
0.0
Lewel
Lovel
Closstication
0.4
Lewel 10
0.
12
24
20
Block Size
Figure 3: TTA classification errors under different levels of shift severity levels and scopes.
Conclusion and Limitations
In this work, we characterize extrapolation with a latent-variable model that encodes a minimal change principle. Within this framework, we establish clear conditions under which extrapolation becomes not only feasible but also guaranteed, even for complex nonlinear models in deep learning. Our conditions reveal the intricate interplay among the generating function's smoothness, the out-of- support degree, and the influence of the shift. These theoretical results provide valuable implications for the design of practical test time adaptation methods, which we validate empirically.
Limitations: On the theory aspect, the Jacobian norm utilized in Theorem 4.2 only considers the global smoothness of the generating function and thus may be too stringent if the function is much more well-behaave/smoothh over the extrapolation region of concern. Therefore, one may consider refined local condition relax this condition. On the empirical side, our theoretical framework entails learning an explicit representation space. Existing methods without such a structure may still benefit from our framework but lesser extent. Also, our framework involves several loss terms including reconstruction, classification. and the likelihood of the target invariant variable. A careful re- weighting of these terms may be needed during training.
10

---

Acknowledgments. We thank the anonymous reviewers for their valuable insights and recommen- dations, which have greatly improved our work. The work of L. Kong supported part by NSF DMS-2134088 through an award to Y. Chi. This material is based upon work supported by NSF Award No. 2229881, Al Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract RO1 HL 15805, and grants from Salesforce, Apple Inc., Quris AI, and Florin Court Capital. P. Stojanov was supported in part by the National Cancer Institute (NCI) grant number: K99CA277583-011 and funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard. This research has been graciously funded by the National Science Foundation (NSF) CNS241408S NSF BCS2040381, NSF IIS2 ISS212395,, NSF IIS II$195553,, NSF IIS212395;; NSF IIS231 1990; the National Institutes of Health (NIH) RO1GM140467. the National Geospatial Intelligence Agency (NGA) HM04772000002; the Semiconductor Research Corporation (SRC) AIHW award 2024AH3210; the National Institute of General Medical Sciences and the Defense Advanced Research Projects Agency (DARPA) ECOLE HR001 12390063. Any opinions, findings, and conclusions or recommendations expressed this publication those of the author(s) and do necessarily reflect the views of the National Science Foundation, the National Institutes of Health, the National Geospatial Intelligence Agency, the Semiconductor Research Corporation, the National Institute of General Medical Sciences, and the Defense Advanced Research Projects Agency.
(NIGMS) R01GM1
References
[1] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18599 2020.
[2] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[3] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[4] Ishaan Gulrajani and David Lopez-Paz. In search lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
[5] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2)-2272244, 2000.
[6] Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in Neural Information Processing Systems, 19, 2006.
[7] Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul Von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Stutistical Mathematics, 60:699-746, 2008
[8] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pages 819- 827, 2013.
[9] Petar Stojanov, Mingming Gong, Jaime Carbonell, and Kun Zhang. Low-dimensional density ratio estimation for covariate shift correction. In The 22nd international conference on artificial intelligence and statistics, pages 3449-3458. PMLR, 2019.
[10] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimiza- tion. In International Conference on Learning Representations, volume 9, 2021.
[11] Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint 100000191,.00804, 2019.
11

---

[12] Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. Journal of machine learning research, 22 (2):1-55, 2021.
[13] Yu Sun, Xiaolong Wang. Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference Machine Learning, pages 7229-9248, PMLR, 2020.
[14) Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Ttt++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34, 2021.
[15] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint WXXv:2006.10722 2020.
[16] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Phillipe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprinl arXiv:2001.0 2020.
[17] Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey test-time adaptation under distribution shifts. arXiv preprint M3X.x233333.1363 2023.
[18] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International Conference on Machine Learning, pages 1145-114477 PMLR, 2022.
[19] Zijian Li, Ruichu Cai, Guangyi Chen, Boyang Sun, Zhifeng Hao, and Kun Zhang. Subspace identification for multi-source domain adaptation. Advances in Neural Information Processing Systems, 36, 2023.
[20] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 255-29374-29885 2022.
[21] Devavrat Tomar, Guillaume Vray, Behzad Bozorgtabar, and Jean-Philippe Thiran. Tesla: Test- time self-learning with automatic adversarial augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20341-20350, 2023.
[22] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79: 151-175, 2010.
[23] Masashi Sugiyama, Matthias Krauledat, and Roburt Müller. Covariate shift adaptation by importance weighted eross validation. Journal of Machine Learning Research, 8(5), 2007.
[24] Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian-product extrapolation. Advances in Neural Information Processing Systems, 36, 2023.
[25] Thaddaus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Com- positional generalization from first principles. Advances in Neural Information Processing Systems, 36, 2023.
[26] Thaddäus Wiedemer, Jack Brady, Alexander Panfilov, Attila Juhos. Matthias Bethge, and Wieland Brendel. Provable compositional generalization for object-centric learning. arXiv preprint arX1v:2310.05327, 2023.
[27] Linfeng Zhao, Lingzhi Kong, Robin Walters, and Lawson LS Wong. Toward compositional generalization in object-oriented world modeling. In International Conference Machine Learning, pages 26841-26864. PMLR, 2022.
[28) Aviv Netanyahu, Abhishek Gupta, Max Simchowitz, Kaiqing Zhang, and Pulkit Agrawal. Learning to extrapolate: A transductive approach. In The Eleventh International Conference on Learning Representations, 2022.
12

---

[29] Xinwei Shen and Nicolai Meinshausen. Engression: Extrapolation for nonlinear regression? In 2023 IMS International Conference on Statistics and Data Science (ICSDS), page 232, 2023.
[30] Kefan Dong and Tengyu Ma. First steps toward understanding the extrapolation of nonlinear models to unseen domains. The Eleventh International Conference on Learning Representa- lions; 2022.
[31] Sorawit Saengkyongam, Elan Rosenfeld, Pradeep Kumar Ravikumar, Niklas Pfister, and Jonas Peters. Identifying representations for intervention extrapolation. In The Twelfth International Conference on Learning Representations, 2023.
[32] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411 1-4000 2000.
[33] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217 PMLR, 2020.
[34] Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica, 2020.
[35] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
[36] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[37] Julius von Kugelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style, 2021.
[38] Lingjnn Kong, Guangyi Chen, Biwei Huang, Eric P Xing, Yuejie Chi, and Kun Zhang. Learning discrete concepts latent bierarchical models. arXiv preprint MXXiv:2406.005/9 2024.
[39] Jack Brady, Roland S Zimmermann, Yash Sharma, Bernhard Schölkopf. Julius Von Kügelgen, and Wieland Brendel. Provably learning object-centric representations. In International Conference Machine Learning, pages 3038-3062. PMLR, 2023.
[40] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In International Conference Learning Representations, 2018.
[41] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
[42] Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan. and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
[43] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. PMLR. 2020.
[44] Yongyi Su, Xun Xu, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering. Advances in Neural Information Processing Systems, 15:17533-15555 2022.
[45] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1903. /2261, 2019.
[46] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. European conference on computer vision, pages 776-794. Springer, 2020.
13

---

[47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[48] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[49] Jun-Kun Wang and Andre Wibisono. Towards understanding gd with hard and conjugate pseudo-labels for test-time adaptation. In The Eleventh International Conference on Learning Representations, 2023.
[50] Sachin Goyal, Mingjie Sun, Aditi Raghunathan, and J Zico Kolter. Test time adaptation via conjugate pseudo-labels. Advances in Neural Information Processing Systems, 35:6204-6218, 2022.
[51] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Tin: A domain-shiff aware batch normalization in test-time adaptation. In The Eleventh International Conference on Learning Representations, 2023.
[52] Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees GM Snoek. Learning to generalize across domains on single test samples. arXiv preprint arXiv: 08045, 2022.
[53] Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11920-11929, 2023.
[54] Mihir Prabhudesai, Tsung- Wei Ke, Alex Li, Deepak Pathak, and Katerina Fragkiadaki. Test- time adaptation of discriminative models via diffusion generative feedback. Advances in Neural Information Processing Systems, 36, 2023.
[55] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information Processing Systems, 36, 2023.
[56] Muhammad Jehanzeb Mirza, Pol Jané Soneira, Wei Lin, Mateusz Kozinski, Horst Possegger, and Horst Bischof. Actmad: Activation matching to align distributions for pest-time-trainnng In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2023.
[57] Yushu Li, Xun Xu, Yongyi Su, and Kui Jia. On the robustness of open-world test-time training: Self-training with dynamic prototype expansion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11 1166-18866, 2023.
[58] Shuai Wang. Daoan Zhang, Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20050-20064 2023.
[59] Minguk Jang, Sae- Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighbor information. arXiv preprint 20X1v:2207.10792, 2022.
[60] Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24172-24182 2023.
[61] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International confer- ence on machine learning, pages 16888-16905. PMLR, 2022.
[62] Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: Degradation-free fully test-time adaptation. In The Eleventh International Conference Learning Representations, 2023.
[63] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations, 2023.
14

---

[64] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
[65] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 295-305, 2022.
[66] Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In International Conference on Machine Learning, pages 42058-42080, PMLR, 2023.
[67] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[68] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2014.
[69] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[70] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprini arXiv:2003.044 04297, 2020.
[71] Yongcan Yu, Lijun Sheng, Ran He, and Jian Liang. Benchmarking test-time adaptation against distribution shifts in image classification. arXiv preprint arXiv:2307.03113, 2023.
15

---

Appendix for
"Towards Understanding Extrapolation: a Causal Lens" "
Table of Contents
A1 Related Work
17
A2 Proof for Theorem 4.2
18
A3 Proof for Thoorem 4.4
19
A4 Synthetic Data Experiments
21
A4.1 Implementation Details
A4.2 Regression Task Evaluation
21
A4.2.1 Implementation
21
A4.2.2 Results and Analysis
A5 Real-world Data Experimental Details
22
A5.1 Datasets
22
A5.2 Generative Adaptation with Entropy Minimization
22
A5.3 Sparsity Regularization
22
A5.4 Recoverability of the Invariant Variable
22
A5.5 Additional quantitative results.
23
16

---

A1 Related Work
In this section, we discuss some related topics including extrapolation, latent-variable identification, and test-time adaptation.
Extrapolation. Out-of-distribution generalization has attracted significant attention in recent years. Unlike our work, the bulk of the work is devoted to generalizing to target distributions on the same support as the source distribution [22
23 g Recent work (24 investigates extrapolation in the form of compositional generalization by resorting to structured generating functions (e.g., additive, slot-wise). Another line of work (28 30 studies extrapolation in regression problems and does not consider the latent representation. Saengkyongam et al. [31] leverage a latent variable model and assumes a linear relation between the intervention variable and the latent variable to handle extrapolation. In this work, we formulate extrapolation as a latent variable identification problem. Unlike the semi parimearaettric conditions in prior work, our conditions do not constrain the form of the generating function and are more compatible with deep learning models and tasks. We demonstrate that our conditions naturally lead to implications benefiting practical deep-learning algorithms.
Latent-variable identification for transfer learning. Identifying latent variables in a causal model has become one canonical paradigm to formalize and understand representation learning in the deep learning regime. Typically, one would assume some latent variables z generate the observed data x (e.g., images, text) through a generating function. However, the nonlinearity of deep learning models requires the generating function to be nonlinear, which has posed major technical difficulty in recovering the original latent variable [32]. To overcome this setback, line of work [33 3t assumes the availability of an auxiliary label for each sample and under different u values, each component 2 of z experiences sufficiently large shift in its distribution. This condition leads to component- wise identification of z, i.c., each estimate 2 is equivalent to Zn(i) up to invertible mapping for a permutation function π [dg] + [dz]- Since this framework assumes all latent components' distributions vary over distributions indexed by u, it doesn't assume the existence of some shared, invariant information across distributions, which is often the case for transfer learning tasks, To address this issue, recent work [88 19] introduce a partition of z into an invariable variable e and an changing variable (i.e., == s)) such that c's distribution remains constant over distributions. They show both с and s can be identified and one can directly utilize the invariant variable c for domain adaptation. However, their techniques crucially rely on the variability of the changing variable 8, mandating the availability of multiple sufficiently disparate distributions (including the target) and their overlapping supports. These constraints make them unsuitable for the extrapolation problem. In comparison, our theoretical results give identification of the invariant variable c (the on-support variable in the extrapolation context) with only one source distribution Pare (x) and as few one out-off support target sample Xtgt through mild assumptions on the generating function, which directly tackles the extrapolation problem.
Test-time adaptation. Test-time Adaptation (TTA) aims at adapting models trained on a source domain to align with the target domain during testing 55). It is broadly classified based on whether the training objective is modified. Test-time Training (TTT) methods [33 [4] 36 57], including TTT (13) and TTT++ [14], proficiently adjust models target domains by implementing similar self-supervised learning strategies on both training and testing data. In contrast, Sequential Test-Time Adaptation [15] (5 55 63) (sTTA) garners significant interest due its practicality, notably its one pa-s sequential inference and no training objective access. Research in sTTA primarily concentrates on two facets: the selection of model parameters for adaptation and the refinement of pseudo-labeling techniques for enhanced efficiency. For instance, TENT (15) fine-tunes the Batch Normalization (BN) layers by minimizing entropy, SHOT [6] adjusts the backbone network while maintaining a static classifier, and T3A (64] updates the classifier prototype. Moreover, burgeoning
line of research (65] 21 44 (50 15 [6] focuses on deriving more robust self-training signals through improved pscudo labeling strategies. For example. TTAC 44 employs clustering techniques to extract more accurate pseudo labels. Despite the prominent recent development, these algorithms tend to be brittle and sensitive to hyper-paramete tuning (66) and limited in theoretical understanding [17]. Our work offers formalization and understanding to fill this gap. We show that insights inferred from our theory can indeed benefit existing TTA algorithms, which hopefully will serve as the first step to bridge the theory and practice for TTA algorithms.
17

---

A2 Proof for Theorem 4.2
Assumption 4.1 (Identification Conditions under Global Shifts).
i [Smoothness & Invertibility]: The generating function g in Equation Z is smooth invertible function with a smooth inverse everywhere.
i [Compactness]: The source data space Aare C. RR is closed and bounded.
ii [Discreieness]: The invariant variable с takes on values from a finite set: C [cC
iv [Continuity]: The probability density function p(s|c) is continuous over E Sare all E C.
[Ou-oof-suport Distance]: The target sample's out-support components Segt 's distance from the source support Sye is constrained: infaes... ||Sigt - s|| S minercy (erm)) D(
2J,
We first present Lemma Al from Kong et al. 38 which establishes the discrete information the source support and serves as the starting point in the proof of Theorem 4.2
Lemma A1 (Source discrete subspace identification (38)- Assuming a generating process in Equa-
tion 7 we estimate the distribution with model (ğ. p(c), ((S)). Under Assumption 4.1 it it
follows that the estimated variable takes on values from (cx) where each value corresponds
uniquely to one value of the true variable c, i.e., c C₁
Theorem 4.2 (Extrapolation under Dense Shifts), Assuming a generating process in Equation estimate the distribution with model (g. p(c). p($)) with the objective:
sup p(Cttt)
Subject to: p(x) = p(x), Vx E Xarci Sigt E Cargini D Sare).
(2)
Under Assumption 4.1
the estimated model can attain the identifiability in Definition 3.1
Proof for Theorem 4.2 Lemma A1 shows that the discrete invariant variable is identifiable on the source distribution.
In the following, too far away from the source support Sac. Suppose that Xegt resides on both manifolds g(cr, and g' (cw where A K.. The generating function o' belongs the generating function class and behaves exactly the same as on the source support, i.e., g' = over C x Sac- We define the minimal distance D(c,, cw') between the two manifolds on support boundaries, i.c..
show that the target's invariant variable Crzt is identifiable if Sigt does not drift
S2) > O. Since Xtgt lives on both manifolds g(Aa,)) .) and g'(cur,), we can express as Xigt = g(c,, Segt) = g' , We define Sace E arg millse... ||s - Stgill and Saro E arg minges... ||s s'gell as two closest points on the source support to Sigt and s'ys respectively. It follows that
D(ck, Cw) ) := min EddE)((..)) [[9(c,,
Xegt g(Ck: Surc) = Ja(cer))Sarn ([Ssttttssssss h)ddt) h;
(4)
Xegt g'(cw, S'mc) = Xigt g(Ck', strre) = (fo Joce + tt. h') h',
where h = Stgt Sacc and h s'gt syrc-
18

---

It follows from Equation 4
g(ck,, Ssrc) = (oo I Io(err) (Sare t. .i)dt) h (So Intour ( + h')dtt h';
g(cw' , sare)
VV Jot aoee, (Sare + +thh)tt) h 1--(('rcew Jatown + t')')dt h' 2 D(ck, ck));
I..
1 J9(cn,)) ( + tt. th)dt) h + +[(
J Jo(e + tt h'jdt h' 2 D(ck,co)
[[b'I)) D(ck: cx));
D(cR, Cx.)
max ([bb], [h]]
2Ju
(5)
2J, Assumption 4.1 states that ||b|| D(c for the true generating function g. Therefore, Xigt
only be explained by one manifold, which denote as g(Cigt:) -).
Finally, we show that the objective Equation M guaranteess that the solution uigt corresponds to the true Cigi- We suppose that Ctgt C₂ which corresponds to è₁ for specific k [K]. First, note that Cigt could only take values from {êx ke[[]] due to the constraint sup p(ča). Also, the correct take the true generating function Thus, for another plausible solution õµ² * č₁. we would have
solution e₁ is always a feasible solution to the objective Equation 2 since
D(ck, Cx.)
max
2.J,
where the definitions are analogous to those in Equation 5 and decorated with to indicate the difference. Due to the
term min, g.s.e ||Sigt Sarell. the distance for the
correct solution èę is upper-bounded by fi D(és, since this is attainable when the estimated 2J,
generating function is the true function, i.e., g Equation that the alternative solution which the distance-minimizing regularization would exclude. Therefore, we have shown that the estimated Čigt corresponds to the correct Cy.
e would always yield hn' >
(gr)
2.,
A3 Proof for Theorem 4.4
Assumption 4.3 (Identification Conditions under Local Shifts).
i [Smoothness & Invertibility:: The generating function g in Equation iis invertible and differen- tiable, and its inverse is also differentiable.
(Invariant Variable Informativeness): The dimensions under c's exclusive influence uniquely determined: for fixed C, [x]_Tec((..5) # (xleee= #2) for any c* + c. S E S, and E S.
ii (Sparse Influence]: At any z Z, the changing variable s influences at most dg dimensions of x, i.e. [Ia(Z] < dz. Alternatively, the two variables e and do not intersect on their influenced dimensions
0.
iv [Mechanistic Dependence]: For all z. any nontrivial partition Pt, P2 of the dimensions Ier (z) yields dependence between the sub-matrices of the Jacobian J (z): rank([], (z)]
(z))
rank([], (z)]p (x)) + rank([], (z)) (z))p;())). (z))-
Theorem 4.4 (Extrapolation under Sparse Shifts). Assuming generating process in Equation we estimate the distribution with model (g. ((C), ((S)) with the objective:
sup p(Crgs), Subject to: f(x) p(x), Vx E X
(3)
19

---

de. Under Assumption 4.3 the estimated model can attain the identifiability in Definition 3.1
Lemma A1 (Brady et al. (39] ). Let 9.9 Rd + IRdx be smooth and invertible. Then, for any z E R S C R.,, rank J [z) s) = rank [J where 2 := 9 g(z).
Proof. The proof consists of two steps. In the first step, we show the identi ficationaaion of the index set Ie:=(z) c [dx] over which x receives only c' influence. That is. 9 maps the estimated invariant variable è to dimensions generated by the true invariant variable alone. In the second step, show that the objective sup p ((Cgg) assigns to the estimated invariant variable for the target sample Cegt the dusticidist estimated value Care that is also generated by Ctgt- Recall the notation 2 = s]. We denote the latent source support as Zare and the set augmented with the target sample as 2 = Zare U {
Step We first show that 9 cannot map the estimated invariant variable è to x dimensions generated by both the invariant variable and the changing variable s.
We show this by contradiction. Suppose that 3z" E Z, such that Leta ( (z)) + 0 and I,), (2') 0
We partition (z") into two disjoint sets Ic)s.1 == {i E Icia(zz)) (z))) E (2)) and Tela.22 = {i E (z))((")) Ji & (2')) base on the overlap with (2") By the supposed condition Equa- tion 7 it follows that Ie)x,11 # 0 and Ig.1 # 0.
We now show that Ieix,2 is nonempty by contradiction. Suppose that Ietx, is empty. It follows that Ieis = Ieix,2 C Zeys- By definition, we know that Ig.1 C Ieyx- Thus, A == Icls UI,,1 C Lers. This inclusion implies that
rank (2 (2)A)) = (J S de.
(8)
Since Iet, nIg.1 = 0, [,, Ie +1:d, = O, and each row of (zz is nonzero by definition, we have that
rank (z,(zz)) [A.)) > rank((J,( (I)))))))))))) de:
(9)
which contradicts Equation 8 Therefore, Zeis. is nonempty. Since we have rank (J, (z" )zz;z)) and (I 22) forms a pair of nonempty partition, Assumption 4.3 implies that
de
rank((J, (2 (2))( Je))))) + rank([], (((")))===a))) )..)...: rank [J (ee..))) de.
(10)
As the g's and g's Jacobian matrix ranks are related (Lemma A1). Equation 10 implies that rank((J )Ize....: + rank([] [et.aa)) > de.
(11)
2 n n Lota,1 by definition and [Jg(2") Iz de has full-row rank (Assumption 4.3 miil it follows that
Since Ioys.1 C Caas. and Ia
rank( (2(22) (2)(())))))))) I......)) rank (J, [2]((2)) lIze...:) + rank([]g(e (ī²)²) Ie > de.
(12)
However, Lemma Al implies that rank([J, Iz....)) = rank((]
(13)
Thus, we have arrived at a contradiction. We have shown that I maps the estimated invariant variable è to x dimensions generated by the true invariant variable alone, i.e., (2)) C (z). Moreover, an index i' from the S's region [2) also belonged to (z)=(z), i.e., i' ix(2) n Iet= then we would have [J)()) follows that we can identify the index set exclusively influenced by over z E Z:
= rank [J, (2) (e)).)) > (Jn)((J
de. Thus,
(z) = (2).
(14)
Step 2. By definition, each value of è determines the region (x)s,,)(.s)) Due to Equation 14 we have
[x] ()s. = [x (c..s) where e can only take on a unique value according to Assumption 4.3
Therefore, there exists an one-to-one mapping he from to over Z. Further, since the maximum likelihood estimation is performed over the entire source distribution p(x) (Equation 33. the image of
he equal to the entire C, thus is onto. Thus, have shown that there exists an invertible mapping
ha : C + e valid over 2 (including the target sample), resulting from our objective (Equation
20

---

Table A1: Synthetic data results on regression (MSE) under both dense and sparse shifts across various distances.
Shifts
Dense
Sparse
Distance
18
24
30
18
24
30
Only Source
11.64
2.44
3.26
1.84
3.32
5.84
Ours
1.40
1.60
1.68
1.15
1.48
1.60
A4
Synthetic Data Experiments
A4.1 Implementation Details
We employ a variational auto-encoder [67 whose encoder and decoder are both 4-layer MLP with 32 dimensions and leaky ReLu (a 0.2). Following Equation 2 and Equation 3 implement reconstruction loss, KL loss on source distribution, the likelihood loss on the target sample, and classification loss on the source data. For the dense case, we implement an additional distance loss to minimize the lg distance of Sigt to the center the source support (which is the origin in our case). The source-only baseline is trained only with classification loss. The iMSDA implementation is adopted directly from the source code of Kong et al. [18]. We train all methods with Adam [68] and learning rate 2e 3 for 25 epochs. We fix the loss weights Acts Arecons 0.1, Atgt. Atgt_ likelihood 0.1, and A --_iittance = 0.01 (for dense shifts) overall distance configurations. We only tune λKL from the interval {1c 1e 2, le 3). We synthetic data experiments one Nvidia L40 GPU and each run consumes less than 2 minutes.
A4.2 Regression Task Evaluation
In addition the classification experiments, we evaluate our model on regression in this section.
A4.2.1
Implementation
Data generation. The regression target is generated from uniform distribution UO, 4). We sample 4 latent invariant variables с from a normal distribution Ny,, Two changing variables in the source domain 8are are sampled from truncated Gaussian centered at the origin. In the target domain, changing variables Sigt are sampled at multiple distances (e.g.. {18, 24, 36}) from the origin. For dense shifts, observations MLP with ReLU activation. For sparse shifts, only two out of six dimensions of x are influenced by the changing variable We generate 10k samples for training and 50 target samples for testing (one target sample accessed per run).
generated by concatenating and and feeding them a 4-yeyer
Model. We make two modifications the classification model in Section 5 First, substitute the classification head with a regression head (the last linear layer). Second, we replace the cross-entropy loss with MSE loss. We fix the loss weights of MSE loss and KL loss at 0.1 and 0.01 for all settings, respectively, and keep all other hyper-parameters the same as the classification task. We use MSE the evaluation metric.
A4.2.2 Results and Analysis
Table Afble|AJdssslayy the evaluation results. We can observe that our method consistently outperforms the baseline and maintains its performance over a wide range of shift distances. contrast, the baseline that directly uses all the feature dimensions degrades drastically when the shift becomes severe. This indicates that our approach can indeed identify the invariant part of the latent representation, validating our theoretical results.
21

---

Table A2: Hyperparameters for minimal change constraint in our experiments.
Dataset
r
ratio;
ratio,
ImageNet
4
1 x 10-3
CIFAR 100
16
2
1 x 10-1
CIFARIO
64
1 x 10-6
A5
Real-world Data Experimental Details
A5.1
Datasets
The datasets used in our experiments include CIFARIO-C, CIFAR100-C, ImageNet-C 45 and ImageNet100-C. CIFAR10 10-C and CIFAR are extended versions of the CIFAR datasets [69 designed to evaluate model robustness against visual corruptions, featuring 10 and 100 classes respectively, each with 50,000 clean training samples and 10,000 corrupted test samples. ImageNet-C. on the other hand, scales this concept up with 1,000 classes, providing 50,000 test samples of each of 15 corruption types. ImageNet- 100 [66] a subset of ImageNet with 100 classes. our experiments. we build ImageNet100-CC by selecting 100 classes reported in Tian et al. [46] from ImageNet-C 45 with 15 different types of corruption.
AS.2 Generative Adaptation with Entropy Minimization
When applying entropy minimization in the MAE-TTT framework (2),. we did not directly integrate the entropy-minimization loss. The self-supervised MAE training process relies on masked images, whereas entropy minimization requires classifying the entire image. To address this, we introduced additional training steps using unmasked images and applied the entropy-minimizztion loss during these steps. Specifically, the training process for each test-time iteration is split into two stages: 1) Stage One: We follow the MAE-TTT approach by inputting masked images and training the model using reconstruction loss. In this stage, only the encoder is updated. 2) Stage Two: We input full images (32 in batch) and optimize
model with the entropy minimization loss following SHOT 43 In this stage, both the encoder and classifier are optimized. The learning rates for both stages are set to be the same. The experiments are conducted with the PyTorch 1.11.0 framework. CUDA 12.0 with 4 NVIDIA A100 GPUs.
A5.3 Sparsity Regularization
Here, provide the implementation details of our modification to add sparsity regularization. In the pre-train stage, we apply the ResNet50 [47) as the backbone network and follow [4 44] to pre-train it the clean CIFARIO, CIFARIO0 and ImageNet training sets, with joint contrastive and classification losses. In the test-time adaptation process, we adopt the sequential TTA protocol as outlined in TTAC throughout the test phase. Moreover, all testing data be processed in a sequential manner (one-pass), ensuring each data point passed through the adaptation process exactly once.
and TeSLA (2I] This protocol prohibits the change of training objectives
Our method built upon TeSLA 21 and follows most of its hyperparameters.. Thus, we only involved, including the low-rank dimension r, the ratio of lrerra and the ratio of sparsity loss ratio,. The
discuss the extra hyperparameters learning rate factor for soft frozen ratio; Inucckoone details are shown in Table A2 It was observed that the constraints minimal change need to be more stringent as the complexity of the data increases, The experiments are conducted with the PyTorch 1.10.0 framework, CUDA 11.7 with an NVIDIA A100 GPU.
A5.4 Recoverability of the Invariant Variable
We assess the impact of the recoverability of the invariant variable on Test-Time Adaptation (TTA) methods. To do this, we compare the performance TTA methods using supervised and unsupervised pre-trained models that have similar ImageNet classification accuracy. Our goal to validate whether invariant variables learned from annotated labels can improve test-time adaptation. We assume that
22

---

Table A3: ImageNet-C evaluation of TTA algorithms with supervised and moco pre-trains.
TTA Algorithm
TENT 15
EATA 61
BN-adapt 7
SAR
68
Supervised pre-train
38.13
42.77
41.47 29.13
Moco pre-train
8.94
1.25
20.13
12.23
Table A4: Detailed corruption-wist results on CIFARIO-C, CIFARIOO-C, and ImageNet-C. We report the rates (%) on 15 testing corruptions.
Datase
Method
Gaus.
Shot
Impu.
Defo.
Glas.
Moti.
Zoum
Snow
Fros.
Fog
Cont.
Blas.
Pixc.
Avg.
17.2
20.2
SHOT
15.3
23.6
9.0
23.4
14.0
12.4
16.1
7.5
17.4
12.5
13.1
CIFAR1O-C
TIT+
18.0
17.1
30.8
10.4
29.9
13.0
9.9
14.8
14.1
15.8
7.0
7.8
19.3
12.7
16.4
15.8
TEAC
17.9
15.8
22.5
8.5
23.5
11.2
7.6
11.9
12.9
13.3
6.9
7.6
17.3
12.3
12.6
13:4
TeSLA
13.3
12.5
20.8
8.8
21.1
11.8
7.3
12.6
11.2
15.6
7.6
7.6
16.2
9.7
11.6
12.5
ReSLA+MC
14.4
7.2
15.4
9.2
41.2
50.8
20.3
41.9
30.1
SHOTT
44.1
41.8
53.3
31.5
50.6
36.0
29.6
40..7
40.1
41.9
29.5
33.6
44.0
34.9
36.6
39.2
CIFAR100-C
TTT++
50.2
47.7
66.1
35.8
61.0
38.7
35.0
44.6
43.8
48.6
28.8
30.8
49.9
39.2
45.5
44.4
TTAC
47.7
45.7
58.1
32.5
55.3
36.6
31.2
40.3
40.8
44.7
30.0
79.3
47.1
37.8
58.3
41.7
TeSLA
38.9
51.5
32.2
69.1
36.9
29.7
40.4
37.4
$6.0
29.3
30.7
42.7
32.9
34.6
38.2
68.7
69.1
72.5
73.3
$3.0
42.7
37.6
74.5
45.5
41.6
47.8
SHOT
77.0
74.5
76.4
81.2
79.3
72.5
61.7
65.7
66.3
55.6
56.0
92.7
57.1
56.3
58.2
68.7
ImageNet-C
TIAC
71.5
67.7
70.3
$1.2
77.3
64.0
$4.4
51.1
56.9
45.4
32.6
79.1
46.0
43.7
48.6
59.3
TTT++
69.4
66.0
69.7
84.2
81.7
65.2
53.2
49.3
56.2
44.4
32.8
75.7
43.9
41.6
46.9
58.7
IeSLA
62.9
49.1
56.6
33.7
43.3
annotated labels help learn better invariant variables c, which play an important role in solving extrapolation problems.
For this detailed analysis, employ ResNet50 models pre-trained in both supervised and self- supervised manners, such as MoCo [70]. For the MoCo model, we apply the linear probe to fit the classifier. For a fair comparison, we select checkpoints from both pre-training methods that have similar ImageNet accuracies, approximately 69.7%. We follow the open-source TTA Benchmark [7]] to evaluate both models using different downstream TTA methods. ImageNet-C is used as the evaluation dataset, and all hyperparameters are set to their default values.
The performance of the different pre-trained models is summarized in Table A3 Methods using supervised pre-training outperform those using unsupervised pre-training by significant margin, indicating that the invariant variables learned from annotated labels play a crucial role in enhancing test-time adaptation.
A5.5 Additional quantitative results.
In Table A4 we provide the detailed performance with corruption-wis classification error rates on all CIFARIO-C, CIFARI00-C, and ImageNet-C datasets. Specifically, we report results under seed on all 15 testing corruptions including Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic Transformation. Pixelate, and JPEG Compression.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We emphasize our contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
All proofs are given in Appendix A2 and Appendix A3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Implementation details are given Section 5 Section Appendix AS
6
Appendix
A4 and
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We provide our Github link in the main paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
seipaiisssoooooooyyo. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details are given in Section 5 Section 6 Appendix A5
Appendix A4 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All our real-world experiments are over at least three random seeds. Our synthetic data experiments are over 50 random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We disclose our compute resources in Appendix A4 and Appendix A5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp//pwe...aeo..oco/oo//////aa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is on the understanding the fundamental aspect of machine learning, posing no direct societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the employed codebase in Appendix A4 and Appendix
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "ePLPAYTh0S", "reviewer": "Reviewer_zpbG", "summary": "* With regard to extrapolability in classification problems, they introduce a reasonable assumption that the latent factor generating the distribution shift only affects the input x but does not affect the label y is introduced and the identifiability of the latent factor is proved under additional reasonable assumptions.\n* The interaction between the smoothness of the generating function, the distance out of support, and the nature of the shift (is the shift restricted to some pixels of the image?) is clarified.\n* They discuss the relationship with test-time adaptation, extend using sparsity constraint, and verify its empirical performance.", "questions": "Q1. Is the reported error bar in Table 2 standard error or standard deviation?\nQ2. Does it matter if the marginal distribution of the label $p(y)$ changes in the test phase?", "limitations": "Most of the important results are for the image classification problem, where the class-conditioned distribution p(x|y) is separated for each y and x has much information, and thus only unchanged pixels have enough information of $y$ under the sparse change assumption. It seems not to be simply extended to numerical predictions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* The relation between extrapolation and causality is a hot research topic.\n* The motivation behind the assumption is clear that the class label is related only to the inviant factors: \"factors such as camera angles and lighting do not affect the object’s class in an image.\"\n* Under the reasonable assumption, an interesting theoretical result on the identifiability of the latent invariant factors is shown.\n* The existing test-time adaptation is shown to be related to this theory and extended based on the implication; the sparsity is added in the adaptation, which shows superior empirical performance.", "weaknesses": "* Methodological improvement and its empirical superiority to the existing test-time adaptation itself is marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging words and valuable feedback! Below, we address your questions and indicate the changes we’ve made thanks to your suggestion.\n\n> W1: “Methodological improvement and its empirical superiority to the existing test-time adaptation itself is marginal.”\n\nThank you for your feedback. Please kindly note that our main empirical results (Table 3) demonstrate rather significant performance gain over the baseline ($+4.87 \\\\%$). This is the case where the base method (MAE-TTT) is a representation-based approach and thus conforms to our theoretical model (please see Section 3.3), thus directly substantiating our theoretical insights. \n\nTable 2 shows that our theoretical insights can consistently benefit a broader class of TTA approaches, even if they don’t exactly conform to our framework. Although some gains are marginal, we believe that these results demonstrate the generality of our theoretical insight.\n\n>Q1:  “Is the reported error bar in Table 2 standard error or standard deviation?”\n\nIt is the standard deviation over three random seeds. Thanks to your reminder, we’ve already included this in the figure caption in our revised manuscript.\n\n>Q2. “Does it matter if the marginal distribution of the label $p(y)$ changes in the test phase?”\n\nThank you for the interesting question. The marginal shift of $p(y)$ would not affect the model performance, under proper assumptions of the shift (please note assumptions are generally necessary to avoid arbitrary shifts).  \n\nTo be specific, under our graphical formulation in which $y$ is a child of $\\mathbf{c}$,  the shift of $p(y)$ corresponds to that of the support-invariant variable $p(\\mathbf{c})$. If the shift doesn’t change the support of $p(\\mathbf{c})$ but only its density on the shared support, our guarantee will still hold true with sufficient samples. This is because this conditional distribution $ p( y | \\mathbf{c} )$ can still be learned and transferable as in our case.\n\n> Q3: “Most of the important results are for the image classification problem…It seems not to be simply extended to numerical predictions.”\n\nThank you for noting this. As our theory doesn’t place specific assumptions on the conditional distribution $ p( y | \\mathbf{c} ) $, the framework is rather flexible to accommodate regression problems. \n\nSpecifically, the procedure to learn the disentangled invariant variable $ \\hat{\\mathbf{c}} $ through objectives (3) or (4) is totally unsupervised and thus agnostic to $y$’s distribution. After learning $ \\hat{\\mathbf{c}}$, we can choose to train a regressor on pairs $ (\\hat{\\mathbf{c}}, y) $ from the source distribution. \n\nThanks to your question, we have included in our manuscript the following synthetic experiments on regression tasks. \n\n**Data Generation:** The regression target $y$ is generated from a uniform distribution $U(0,4)$. We sample 4 latent invariant variables $\\mathbf{c}$ from a normal distribution $N(y, I_c)$. Two changing variables in the source domain $\\mathbf{s} _{\\mathrm{src}}$ are sampled from a truncated Gaussian centered at the origin. In the target domain, changing variables $\\mathbf{s} _{\\mathrm{tgt}}$ are sampled at multiple distances (e.g., $\\{18, 24, 36\\}$) from the origin. Observations $\\mathbf{x}$ are generated by concatenating $\\mathbf{c}$ and $\\mathbf{s}$ and feeding them to a 4-layer MLP with ReLU activation. We generate 10k samples for training and 50 target samples for testing (one target sample accessed per run).\n\n**Model:** We make two modifications on the classification model in the paper. First, we change the classification head to a regression head (the last linear layer). Second, we replace the cross-entropy loss with MSE loss. We fix the loss weights of MSE loss and KL loss at 0.1 and 0.01 for all settings, respectively, and keep all other hyper-parameters the same as in the classification task. We use MSE as the evaluation metric.\n\n\n**Results:** The results are summarized below, which indicate that the proposed method can be extended to the regression setting.\n\n| Dense Shift Distance | 18       | 24       | 30       |\n|---------|----------|----------|----------|\n| Baseline| 1.6400   | 2.4430   | 3.2627   | \n| Our Method | 1.4006   | 1.6039   |1.6812   |\n\n\nWe are also working on more settings and will update you once the results are available.\n\n---\n\nPlease let us know if we have properly addressed your questions and we are more than happy to discuss more!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s8Pb7QdYZD", "reviewer": "Reviewer_HS3a", "summary": "The authors discuss the problem of domain adaptation or distribution shift in the case when only a single point in the shifted domain is available rather than the full distribution. The authors approach such a problem from the perspective of a latent variable model, which assumes that the observations are generated from two latent variables: one is invariant under domain shifts (and carries informations about the classification label) and the other one changes under domain shifts but is irrelevant for classification. However, the generator function mixes these two latent variables making the classification under domain shifts difficult. The authors theoretically discuss the identifiability of the invariant latent variables in two scenarios, when the irrelevant for classification latent variables affects all dimensions in the input or only a subset of them. These theoretical results are validated in experiments on synthetic and real-world data.", "questions": "- It is very interesting that you derive a bound of how far the target point can be from the source manifold in Assumption 3.1-v. I wonder how tight do you think this bound is?\n- Could you provide a bit more intuition why Assumption 3.3.-iv allows us to \"identify the unaffected dimension indices [...] with our estimated model\", I am not sure I understood this point?\n- Did I understand correctly that in practice the estimation algorithms (as described in Sec. 3.3.) reduce to training an auto-encoder to match the data distribution, and a classifier in the latent space? So the key idea is that \"raising\" the classification to the latent space allows us to separate invariant latents from the nuisance variables?", "limitations": "The limitations are adequately addressed", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Interesting and practically relevant problem \n+ The paper is clearly written and easy to follow\n+ Theoretical results formalising the intuition of invariant and nuisance latent variables", "weaknesses": "- I am not sure I completely understood the connection between the theory and the actual learning objective. Specifically, I would appreciate if the authors could elaborate a bit more on how the Eq. (2) and (3) are connected to the learning objective in Sec. 3.3 (also see the Questions part of the review)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable questions! We address your questions point-to-point in the following.\n\n> W1 & Q3: “... how the Eq. (2) and (3) are connected to the learning objective in Sec. 3.3”, “Did I understand correctly that in practice the estimation algorithms (as described in Sec. 3.3.) reduce to training an auto-encoder to match the data distribution, and a classifier in the latent space? So the key idea is that \"raising\" the classification to the latent space allows us to separate invariant latents from the nuisance variables?”\n\nThank you for the great question! You are absolutely right about the general implementation framework. The matching distribution constraint in (2) (3) entails learning a generative model with a representation space $\\hat{\\mathbf{z}}$ and maximizing likelihood of the target invariants $\\hat{p} ( \\hat{\\mathbf{c}}_{\\mathrm{tgt}} )$ enables us to separate the invariant latents $\\mathbf{c}$ from the nuisance variables $\\mathbf{s}$, as you pointed out precisely. Given this disentangled representation, we can train a classifier on the invariant latents from the source distribution, which can be directly applied to the target distribution. Please let us know if you’d like elaboration – thanks!\n\n\n\n\n> Q1: “It is very interesting that you derive a bound of how far the target point can be from the source manifold in Assumption 3.1-v. I wonder how tight do you think this bound is?”\n\nWe appreciate your insightful question. This bound is tight in a sense that the equality can be attained in worst-case scenarios. This is because beyond the threshold in the bound, a manifold characterized by a distinct invariant variable  $ \\mathbf{c} \\neq \\mathbf{c} _{\\mathrm{tgt}} $ may also explain the target sample $\\mathbf{x} _{\\mathrm{tgt}}$. This ambiguity would thwart our attempt to uniquely determine the invariant latent of $ \\mathbf{x} _{\\mathrm{tgt}} $.\nTo aid intuition, let’s look at Figure 1b. The threshold in the bound exactly characterizes the starting point of the “unidentifiable region”. In the worst case when the target sample $\\mathbf{x} _{\\mathrm{tgt}}$ lands in the doubly shaded area, we cannot uniquely identify which manifold it belongs to without further assumptions/knowledge.\n\n\n> Q2: “Could you provide a bit more intuition why Assumption 3.3.-iv allows us to \"identify the unaffected dimension indices [...] with our estimated model\", I am not sure I understood this point?”\n\nThank you for the question! Assumption 3.3 iv enforces that the unaffected pixels exhibit certain dependence among them, in the sense that if we divide this region into any two partitions and generate these two regions separately, we would need more “capacity” (i.e., Jacobian ranks) than generating this region jointly, because in the latter case some information would be shared between the partitions. Further note that Assumption 3.3 iii coerces the affected region to be either small or disjoint from the unaffected region. Thus, the cross-region dependence (between the unaffected and the affected regions) is small. Intuitively, this clear contrast offers us the signal to disentangle these two regions. \n\nFor instance, in Figure 1c, the pixels within the region of the cow are highly dependent, whereas the cow region and the background region are significantly less so. Thus, this contrast aids our humans to distinguish the two regions rather easily.\n\n---\nPlease let us know if you’d like further illustration, thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EGII3e4KET", "reviewer": "Reviewer_gbwX", "summary": "The paper addresses the problem of out-of-support extrapolation and presents identification results within a latent variable framework. This framework resembles invariant learning, where one subset of latent variables directly causes the labels, while another subset, termed style latents, undergoes a distribution shift at test time. The authors provide identification results for scenarios where this distribution shift affects the entire image (global shift case) and where it affects only specific regions of the image (local shift case). Additionally, they connect their identification results to the entropy maximization objective used in test time adaptation (TTA) algorithms. These theoretical aspects are verified through experiments on popular TTA benchmarks, such as CIFAR-C and ImageNet-C.", "questions": "Please refer to the weaknesses section in my review above for more details regarding my concerns and queries.\n\n* Why does the identification analysis apply for TTA when its learning objective is different from the learning objective considered in the identification analysis?\n\n* How do the identification criteria considered in this study relate to standard identification criteria used in causal representation learning/ non-linear ICA [1, 2, 3]? Does satisfying the proposed criteria imply we learn meaningful content variables such that they are related to the true content variables?\n\n* Following the results in Table 2 regarding the effect of adding sparsity regularization, it is unclear how TeSLA + SC provides an improvement over TeSLA, as the difference in mean performance between the two approaches is within the standard error. The authors should consider revising line 357 to accurately reflect this observation.", "limitations": "Yes, the authors have properly addressed the limitations and impact of their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The identification results for out-of-support extrapolation presented in the paper appear to be novel, to the best of my knowledge.\n\n* The paper is overall well-written, with clearly explained assumptions required for the identification results, and detailed descriptions of the experimental setup and results.", "weaknesses": "* My main concern is with the technical soundness of the paper, particularly regarding certain claims and connections. I believe the connection between the identification results and the test-time adaptation (TTA) objective is problematic.  Specifically, Theorems 3.2 and 3.4 require that the learned distribution of the data ($\\hat{p}(x)$) matches the true distribution ($p(x)$). However, the TTA methods are not generative models, and therefore they do not estimate the distribution of the data. However, TTA methods are not generative models and do not estimate data distributions, so they cannot enforce the constraint $p(x) = \\hat{p}(x)$.  Therefore, it is unclear how the identification results provide insights into TTA methods. Identification concerns the existence of a unique solution for perfect estimation under the learning objective, but the learning objectives for the identification analysis and TTA methods are different. While the authors discuss this in Section 3.3, however, the issue is deeper,  they do not address the density estimation constraint and its connection to test-time adaptation comprehensively.\n\n* The definition of identification considered by the authors (Definition 2.1) is problematic because it does not relate the estimated latents ($\\hat{c}$) to the true latents ($c$). This implies that the estimated latents could be an arbitrary function of the true latents, which is concerning as it makes it unclear why such identifiability would be desirable. The identification criteria state that the estimated content latents are the same whenever the true content latent variables are the same. However, this does not necessarily imply a meaningful relationship between the estimated and true latents. For instance, block identification [1] ensures that the estimated content latents are a function of only the true content latents, establishing a clear relationship. If we cannot relate the estimated latents to the true latents, it is unclear whether the inferred content latents ($\\hat{c}$) are genuinely informative about the true content latents ($c$). This lack of a meaningful connection undermines the practical value of the identifiability results.\n\nReferences:\n\n[1] Lachapelle, Sébastien, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. \"Additive decoders for latent variables identification and cartesian-product extrapolation.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Khemakhem, Ilyes, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. \"Variational autoencoders and nonlinear ica: A unifying framework.\" In International conference on artificial intelligence and statistics, pp. 2207-2217. PMLR, 2020.\n\n[3] Zimmermann, Roland S., Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. \"Contrastive learning inverts the data generating process.\" In International Conference on Machine Learning, pp. 12979-12990. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and the valuable time you have dedicated to our work.  \n\nIt seems that the reviewer might have seen a previous version of this manuscript. In that case, please let us kindly highlight that this submission is significantly different: in the earlier instance, we tried justifying TTA with our theory, which we later realized was not optimal. \n\nIn this submission, the main contribution is to provide a fundamental understanding of extrapolation and discuss principled ways to address it, independent of existing TTA algorithms. To support it, we substantially rewrote the introduction, problem formulation, and analysis of the identification theory. Further, we implemented our framework with synthetic experiments in Sec 4 to rigorously validate our theory. Additionally, for real-world experiments, we base our implementation on the autoencoder-based MAE-TTT, whose reconstruction loss facilitates matching the estimated distribution $\\hat{p}(\\mathbf{x})$ and the true marginal distribution match $p(\\mathbf{x})$ in our objective. \n\nThus, we feel the review comments mainly apply to the earlier version, not this one. We apologize for any confusion caused by the multi-version problem may have caused and appreciate your further feedback.\n\n>W1& Q1: “...the connection between the identification results and the test-time adaptation (TTA) objective is problematic..”\n\nThank you for raising this concern. Please kindly note that we do not claim to justify or explain existing TTA methods with our identification theory. Instead, we discuss the relationship between our theoretical framework on extrapolation and TTA methods to identify gaps (lines 281-286) and explore how our ideas may benefit them. Our experiments show that incorporating our insights (aligning the invariant variable in Sec 5.1 and sparsity in Sec 5.2) can improve TTA methods.\n\nIn Sec 3.2 and the main experiments in Sec 5.1, we focus on the autoencoder-based model MAE-TTT, where the reconstruction loss facilitates matching the estimated distribution $\\hat{p}(\\mathbf{x})$ with the true marginal distribution $p(\\mathbf{x})$, especially in the extreme case where the reconstruction is perfect.\nEven with our modifications, TTA algorithms may not fully adhere to the theoretical framework. Nonetheless, we hope this serves as a meaningful step towards developing more principled extrapolation methods for real-world datasets.\n\n>W2 & Q2: “The definition of identification considered by the authors (Definition 2.1) is problematic.”\n\nThank you for the question. Our identification notion (Definition 2.1) implies the block-wise identifiability [1] – there exists an invertible map from the true invariant variable $\\mathbf{c}$ to the estimated invariant variable $\\hat{\\mathbf{c}}$. In light of your feedback, we’ve added the following text to line 120 to make it clearer: “which is equivalent to the blockwise identifiability in prior work [1].”\n\nWe give a proof that Definition 2.1 implies block-wise identifiability. \n\nAs both the generating function $g$ and the estimated one $\\hat{g}$ are invertible (Assumption 3.1), we have an invertible map $ h: ( \\mathbf{c}, \\mathbf{s} ) \\mapsto ( \\hat{\\mathbf{c}}, \\hat{\\mathbf{s}} ) $.\n\nWe first show that Definition 2.1 implies that $\\hat{ \\mathbf{c}}$ depends only on $ \\mathbf{c}$. Suppose, for contradiction, that $\\hat{ \\mathbf{c}}$ depends on both $ \\mathbf{c}$ and $ \\mathbf{s}$. \nThen there would exist $  \\mathbf{c} _{o}$, $ \\mathbf{s} _{1}$, and $ \\mathbf{s} _{2}$ $( \\mathbf{s} _{1} \\neq  \\mathbf{s} _{2}$) such that: $h( \\mathbf{c} _{o},  \\mathbf{s} _{1}) = (\\hat{ \\mathbf{c}} _{1}, \\hat{ \\mathbf{s}} _{1})$ and $h( \\mathbf{c} _{o},  \\mathbf{s} _{2}) = (\\hat{ \\mathbf{c}} _{2}, \\hat{ \\mathbf{s}} _{2})$ where $\\hat{ \\mathbf{c}} _{1} \\neq \\hat{ \\mathbf{c}} _{2}$. \n\nHowever, this contradicts Definition 2.1, because we have $ \\mathbf{c} _{o} =  \\mathbf{c} _{o} $, but $\\hat{\\mathbf{c}} _{1} \\neq \\hat{\\mathbf{c}} _{2}$. Therefore, the proposition must be false, and $\\hat{ \\mathbf{c}}$ must depend only on $ \\mathbf{c}$.\n\nNow we define $h_{c}:  \\mathbf{c} \\mapsto \\hat{ \\mathbf{c}}$ as follows: for any $ \\mathbf{c}$, choose any $ \\mathbf{s}$ and let $h_{c}( \\mathbf{c}) = \\hat{ \\mathbf{c}}$, where $(\\hat{ \\mathbf{c}}, \\hat{ \\mathbf{s}}) = h( \\mathbf{c},  \\mathbf{s})$. This is well-defined because we've proven that $\\hat{ \\mathbf{c}}$ only depends on $ \\mathbf{c}$, so the choice of $ \\mathbf{s}$ doesn't matter.\n\nWe finally show that $h_{c}$ is invertible by noting the following. \n\n$h_{c}$ is injective: Let $ \\mathbf{c} _{1},  \\mathbf{c} _{2}$ be in the domain of $h _{c}$. If $h _{c}( \\mathbf{c} _{1}) = h _{c}( \\mathbf{c} _{2})$, then $\\hat{ \\mathbf{c}} _{1} = \\hat{ \\mathbf{c}} _{2}$ by definition.  By Definition 2.1, this implies $ \\mathbf{c} _{1} =  \\mathbf{c} _{2} $. Therefore, $h _{c}$ is injective.\n\n$h_{c}$ is surjective: Let $\\hat{ \\mathbf{c}}$ be in the image of $h_{c}$. Since $h$ is invertible, there exists $( \\mathbf{c},  \\mathbf{s})$ such that $h( \\mathbf{c},  \\mathbf{s}) = (\\hat{ \\mathbf{c}}, \\hat{ \\mathbf{s}})$ for some $\\hat{ \\mathbf{s}}$. By definition of $h_{c}$, $h_{c}( \\mathbf{c}) = \\hat{ \\mathbf{c}}$. Therefore, $h_{c}$ is subjective.\n\nPlease let us know if we have fully addressed your concerns and we’d be delighted to discuss further.\n\n>Q3: “it is unclear how TeSLA + SC provides an improvement over TeSLA, as the difference in mean performance between the two approaches is within the standard error.”\n\nThanks for the comment. We were wondering if there was a misreading – the improvements of TeSLA+SC over TeSLA are $0.4$. $0.2$, and $0.5$, all larger than the std $0.1$.\nAs we acknowledge in lines 357-359, these improvements, though modest, are consistent and thus demonstrate the applicability of our approach beyond autoencoding TTA approaches like MAE-TTT.  \n\n---\n\nWe are eager to hear your feedback. We’d deeply appreciate it if you could let us know whether your concerns have been addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "toxlsrLs0v", "reviewer": "Reviewer_5kN3", "summary": "This work addresses the challenge of extrapolation in scenarios where only a few target samples are available. It aims to provide a theoretical understanding and methods for effective extrapolation without needing a target distribution within the training support. The approach involves a latent-variable model based on the minimal change principle in causal mechanisms. The study identifies conditions under which identification is possible even with a single off-support target sample. Experiments with synthetic and real-world data validate the theoretical and practical aspects of the findings.", "questions": "How would you apply your method for regression problems? Do you think it would perform well?", "limitations": "See weakness 4.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper gives relatable motivating examples making it approachable for readers.\n2. The paper is well-written, with a few minor typos. \n3. The work tackles a very relevant problem in today's world.", "weaknesses": "1. Table 2 appears to omit results from TeSLA-s, which outperforms their TeSLA+SC. Thus their claims of SOTA are in question. Table 1 of:\n     * Devavrat Tomar, Guillaume Vray, Behzad Bozorgtabar, and Jean-Philippe Thiran. Tesla: Test-time self-learning with automatic adversarial augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20341–20350, 2023.\n\n2. The paper lacks expected sections. There is no related work, making it difficult to contextualize their work within the field. Additionally, some of the introduction could be better placed in a background section, specifically lines 41-67.\n\n3. The authors do not provide any source code.\n\n4. More elaboration on the limitations would enhance the paper. The authors seem to have overlooked reflecting sufficiently on potential limitations of their method. For instance, since everything operates in latent space, the method would only be applicable to models that possess a latent representation.\n\n5. Minor typos:\n     * What do the numbers in Table 1 represent? Accuracy? AUROC?\n     * Line 127: Should it not be, \"... values c as x_{src}\"?\n     * Line 127: Missing period.\n     * Line 321: It would be helpful to provide a link to the code for MAE-TTT.\n     * Line 356: Missing period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and the time dedicated to reviewing our work. We address your concerns and questions as follows.\n\n>W1: “Table 2 appears to omit results from TeSLA-s.”\n\nThank you for the comment. Thanks to your remark, we perform additional experiments to apply our sparsity constraint (SC) module to TeSLA-s and present the results as follows. We repeat our experiments over 3 random seeds and adopt the hyper-parameters from TeSLA+SC without any tuning.\n\n| Model   | CIFAR10-C     | CIFAR100-C    | ImageNet-C    |\n|---------|---------------|---------------|---------------|\n| Tesla-s | 12.1          | 37.3          | 53.1          |\n| Tesla-s+SC    | **11.7 ± 0.01** | **37.0 ± 0.06** | **50.9 ± 0.15** |\n\nWe can observe that our module can consistently benefit existing methods (especially on the hardest dataset ImageNet-C), thus showcasing the applicability of our theoretical insights.\nWe have included this result into Table 2 in our revised manuscript -- thank you for pointing us to this!\n\nWe didn't include TeSLA-s in the initial draft because TeSLA-s relies on additional access to the source dataset during the adaptation (please also see this in Table 1 of Tomar et al.), which may not be as realistic and challenging as the source-free setting for TeSLA and TeSLA+SC.\n\n>W2: “The paper lacks expected sections.”\n\nThank you for the constructive suggestion to help us improve the readability of our paper! In the submission, we deferred the related work section to Appendix A1. In light of your suggestion, we have condensed and placed it as Section 2 in our revision (original line 84 - 110) to aid contextualization (we reorganized spacing in Section 4 & 5 and moved Section 5.3 to the appendix to make space).\nFurther, given your suggestion, we have condensed and merged formalisms (especially in lines 46-53 and 56-60) into the beginning of Section 2 in the revised version to increase the readability.\n\n>W3: source code.\n\nThanks for the comment. We now provide source code for TeSLA+SC and have already passed them to the AC, following NeurIPS guidelines. Please let us know if you have any questions regarding the code, thank you!\n\n>W4: “More elaboration on the limitations.”\n\nThank you for raising this concern. In light of your suggestion, we have expanded the limitation section in our revised paper. The added content is as follows.\n\n“On the theory aspect, the Jacobian norm utilized in Theorem 3.1 only considers the global smoothness of the generating function and thus may be too stringent if the function is much more well-behaved/smooth over the extrapolation region of concern. Therefore, one may consider a refined local condition to relax this condition.\nOn the empirical side, our theoretical framework entails learning an explicit representation space. Existing methods without such a structure may still benefit from our framework but to a lesser extent. For instance, TeSLA (Table 2) receives a smaller boost compared to MAE-TTT (Table 3).\nAlso, our framework involves several loss terms including reconstruction, classification, and the likelihood of the target invariant variable. A careful re-weighting over these terms may be needed during training.”\n\n>W5: \"minor typos\".\n\nThank you so much for pointing them out! 1) Yes, it is classification accuracy -- we have added it to Table 1 caption. 2,3) We've reworded the sentence as \"we need to identify the target sample $ \\mathbf{x} _{\\mathrm{tgt}} $ with source samples  $ \\mathbf{x} _{\\mathrm{src}} $ that share the same invariant variable values with the target sample, i.e., $ \\mathbf{c} _{\\mathrm{src}} = \\mathbf{c} _{\\mathrm{tgt}} $.\" 4,5) Thanks -- added both to the revision!\n\n>Q1: “regression problems.”\n\nThank you for noting this. As our theory doesn’t place specific assumptions on the conditional distribution $ p( y | \\mathbf{c} ) $, the framework is rather flexible to accommodate regression problems. \n\nSpecifically, the procedure to learn the disentangled invariant variable $ \\hat{\\mathbf{c}} $ through objectives (3) or (4) is totally unsupervised and thus agnostic to $y$’s distribution. After learning $ \\hat{\\mathbf{c}}$, we can choose to train a regressor on pairs $ (\\hat{\\mathbf{c}}, y) $ from the source distribution. \n\nThanks to your question, we have included the following synthetic experiments on regression.\n\n**Data Generation:** The regression target $y$ is generated from a uniform distribution $U(0,4)$. We sample 4 latent invariant variables $\\mathbf{c}$ from a normal distribution $N(y, I_c)$. Two changing variables in the source domain $\\mathbf{s} _{\\mathrm{src}}$ are sampled from a truncated Gaussian centered at the origin. In the target domain, changing variables $\\mathbf{s} _{\\mathrm{tgt}}$ are sampled at multiple distances (e.g., $\\{18, 24, 36\\}$) from the origin. Observations $\\mathbf{x}$ are generated by concatenating $\\mathbf{c}$ and $\\mathbf{s}$ and feeding them to a 4-layer MLP with ReLU activation. We generate 10k samples for training and 50 target samples for testing (one target sample accessed per run).\n\n**Model:** We make two modifications on the classification model in the paper. First, we change the classification head to a regression head (the last linear layer). Second, we replace the cross-entropy loss with MSE loss. We fix the loss weights of MSE loss and KL loss at 0.1 and 0.01 for all settings, respectively, and keep all other hyper-parameters the same as in the classification task. We use MSE as the evaluation metric.\n\n**Results:** The results are summarized below, which indicate that the proposed method can be extended to the regression setting.\n\n| Dense Shift Distance | 18       | 24       | 30       |\n|---------|----------|----------|----------|\n| Baseline| 1.6400   | 2.4430   | 3.2627   | \n| Our Method | 1.4006   | 1.6039   |1.6812   |\n\nWe are also working on more settings and will update you once the results are available.\n\n---\nPlease let us know if you have further questions -- thank you so much!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all reviewers for their efforts and helpful comments regarding our paper. We are encouraged that Reviewers **zpbG**, **HS3a**, and **5kN3** find our problem setup relevant and interesting, Reviewers **zpbG** and **gbwX** find our theory interesting and novel, and all Reviewers find our paper clearly written. 


Below is a summary of our responses:

* To **Reviewer zpbG**: We have provided further details and introduced a new experiment focusing on the regression task.
* To **Reviewer HS3a**: We have detailed the connections between our theoretical framework and the practical learning objectives and assumptions.
* To **Reviewer gbwX**: We have clarified the key contributions of our formulation and theory.
* To **Reviewer 5kN3**: We have included new experiments using TeSLA-s as a baseline and in a regression setting. We have also shared the source code for TeSLA+SC and TeSLA-s+SC, which has been forwarded to ACs. Further, we have expanded the limitation section, moved the related works from the Appendix to Section 2, and corrected the typos.


Please review our detailed responses to each point raised. We hope that our revisions and clarifications satisfactorily address the concerns raised. We thank you again for your valuable time and expertise.

Meta-review: After reviewing the paper, the feedback from the reviewers, and the authors' thorough rebuttals, I recommend accepting the paper. The authors have effectively addressed the concerns raised, including those from Reviewer 5kN3 (I read both the review and rebuttal). The work tackles an important problem and presents a solid theoretical foundation with empirical validation. Given these strengths, I believe the paper merits acceptance.
Decision: Accept (poster)