Introduction
Extrapolation necessitates the capability of generalizing beyond the training distribution support, which is essential for the robust deployment of machine learning models in real-world scenarios.
Specifically, given access to a source distribution \mathcal{D}_{\text{src}}:=p(\mathbf{x}_{\mathrm{src}},\mathbf{y}_{\mathrm{src}}) with support \mathcal{X}_{\text{src}}:=\text{Supp}(p_{\text{src}}(\mathbf{x})) and one or a few out-of-support samples \mathbf{x}_{\mathrm{tgt}}\notin\mathcal{X}_{\text{src}} , the goal of extrapolation is to predict the target label \mathbf{y}_{\mathrm{tgt}} .
For example, if the training distribution includes dog images, we aim to accurately classify dogs under unseen camera angles, lighting conditions, and backgrounds. While intuitive for humans, machine learning models can be brittle to minor distribution shifts [ 1 , 2 , 3 , 4 ] .

Addressing distribution shifts has garnered significant attention from the community. Unsupervised domain adaptation under covariate shifts addresses the shift of the marginal distribution p(\mathbf{x}) across domains. However, canonical techniques such as importance sampling and re-weighting [ 5 , 6 , 7 , 8 , 9 ] are predicated on the assumptions of overlapping supports \text{Supp}(p_{\text{tgt}}(\mathbf{x}))\subset\text{Supp}(p_{\text{src}}(\mathbf{x})) and the availability of the entire target marginal distribution p_{\text{tgt}}(\mathbf{x}) . Similarly, domain generalization [ 10 , 11 , 12 ] assumes access to multiple source distributions p_{\text{src}}(\mathbf{x},\mathbf{y}) whose supports jointly cover the target distribution.
In addition to these methods, test-time adaptation (TTA) [ 13 , 14 , 15 , 16 ] is particularly relevant to our discussion of extrapolation.
TTA addresses out-of-distribution test samples at the individual sample level.
Canonical methods include updating the source model with entropy-based or self-supervised losses on target samples.
However, most TTA research focuses on empirical aspects, with limited theoretical formalization [ 17 ] .
Most related to our work, Kong et al. [ 18 ] and Li et al. [ 19 ] propose theoretical frameworks to characterize distribution shifts and explore conditions for identifying latent changing factors.
However, these frameworks assume access to multiple source distributions with overlapping supports, which are not directly applicable to the extrapolation problem considered in this work, where we have potentially only one out-of-support target sample \mathbf{x}_{\mathrm{tgt}} .

Since the target \mathbf{y}_{\mathrm{tgt}} can be arbitrarily out of the source support (Figure 1(a) ), extrapolation is ill-posed without proper assumptions on the relationship between the source and the target.
In this work, we formulate a latent-variable model that encodes a minimal change principle to address this ill-posedness. Specifically, we assume that a latent variable \mathbf{z} determines \mathbf{x} such that \mathbf{x}:=g(\mathbf{z}) .
The minimal change principle entails the following two assumptions on the generating process.
1) The out-of-support nature of \mathbf{x}_{\mathrm{tgt}} stems from only a subspace of \mathbf{z} , denoted as \mathbf{s} , while the complement partition \mathbf{c} for the target sample \mathbf{x}_{\mathrm{tgt}} is within the training support, i.e., \mathbf{z}:=[\mathbf{c},\mathbf{s}] , \mathbf{s}_{\mathrm{tgt}}\notin\mathcal{S}_{\text{src}} , and \mathbf{c}_{\mathrm{tgt}}\in\mathcal{C}_{\text{src}} .
2) The changing variable \mathbf{s} only controls non-semantic attributes in \mathbf{x} and thus doesn’t influence the label \mathbf{y} , i.e., \mathbf{y}:=g_{y}(\mathbf{c}) .
This formulation attributes the seemingly complex shifts in the pixel space of \mathbf{x} to simple intrinsic changes (in the sense that this change involves only \mathbf{s} ) in the latent space, allowing us to reason about the transfer via the invariant variable \mathbf{c} . Under our formulation, extrapolation amounts to identifying invariant latent variables \mathbf{c} , with which a model f:\mathbf{c}\mapsto\mathbf{y} trained on the labeled source dataset can be directly applied to the target sample.

In light of this formulation, we investigate the identifiability conditions of the invariant variable \mathbf{c} . We propose two sets of conditions addressing two regimes of the influence from \mathbf{s} on \mathbf{x} .
We refer to the case where all dimensions \mathbf{x}_{i} (e.g., all pixels in an image) could be influenced by the changing variable \mathbf{s} as the dense influence and the case where only a limited subset of dimensions x_{i} is affected as the sparse influence.
Specifically, our first condition (Dense Influence, Theorem 4.2 ) states that if \mathbf{s} ’s influence is dense, then assuming that \mathbf{c} takes on only finite values, extrapolation requires the manifold associated with each value of \mathbf{c} (i.e., g(\mathbf{c},\cdot) over \mathbf{s} ) to be adequately separable, and the target changing variable \mathbf{s}_{\mathrm{tgt}} to be close to the source support \mathcal{S}_{\text{src}} .
Intuitively, if images from two classes (two values of \mathbf{c} ) are sufficiently distinguishable, such as cats and dogs, we can still recognize the class of the target sample \mathbf{x}_{\mathrm{tgt}} , even if it has undergone moderate unseen shifts on all pixels like camera angles and positions controlled by \mathbf{s} (Figure 1(b) ). Our second condition (Sparse Influence, Theorem 4.4 ) states that if \mathbf{s} ’s influence is sufficiently sparse, then extrapolation can occur regardless of the distance of \mathbf{s}_{\mathrm{tgt}} to the source support \mathcal{S}_{\text{src}} . Intuitively, we can recognize the class "cow" even if only the background is changed, regardless of the severity (Figure 1(c) ).

Our theory provides insights into the interaction between the underlying manifold’s smoothness, the out-of-support distance, and the nature of the shift. We conduct synthetic data experiments to validate our theoretical results. Moreover, we discuss the relationship between our results and TTA approaches.
In particular, we apply our theoretical insights to improve autoencoder-based MAE-TTT [ 20 ] and observe noticeable improvements. Additionally, we demonstrate that basic principles (sparsity constraints) from our framework can benefit the state-of-the-art TTA approach TeSLA [ 21 ] .
Our empirical results not only show the practical viability of our theory but also pave the way for further advancements in the field.

In summary, our contributions are threefold:

We formulate the extrapolation task as a latent-variable identification problem. Our latent-variable model encodes complex changes in observed variables to a partition of latent variables, allowing us to reason about transferability from the source to the target through latent variable identification.

We provide identification guarantees for the proposed latent-variable model, including shifts of distinct properties (dense vs. sparse) and corresponding conditions on the generating process. Our theory provides an essential understanding of when latent-variable identification is possible without accessing an entire target distribution and assuming overlapping supports as in prior work [ 18 , 19 ] .

Inspired by our theory, we propose to add a likelihood maximization term to autoencoder-based MAE-TTT [ 20 ] to facilitate the alignment between the target sample and the source distribution. In addition, we propose sparsity constraints to enhance the state-of-the-art TTA approach TeSLA [ 21 ] . We validate our proposals with empirical evidence.