RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Identification Guarantees for Extrapolation
In this section, we provide two sets of conditions on which one can identify the invariant variable \mathbf{c} and discuss the intuition and implications.

As discussed in Section 3 , we need to identify the target sample \mathbf{x}_{\mathrm{tgt}} with source samples \mathbf{x}_{\mathrm{src}} that share the same invariant variable values with the target sample, i.e., \mathbf{c}_{\mathrm{src}}=\mathbf{c}_{\mathrm{tgt}} .
This enables us to obtain the label of \mathbf{x}_{\mathrm{tgt}} by assigning the label of such \mathbf{x}_{\mathrm{src}} .
The shift between the source distribution p(\mathbf{x}_{\mathrm{src}}) and the target sample \mathbf{x}_{\mathrm{tgt}} originates from the out-of-support nature of the changing variable \mathbf{s}_{\mathrm{tgt}} , i.e., \mathbf{s}_{\mathrm{tgt}}\notin\mathcal{S}_{\mathrm{src}} , it is crucial to impose proper assumptions on the influence of \mathbf{s} on \mathbf{x} so that \mathbf{x} retains sufficient footprints of \mathbf{c} for identification beyond the source support.

We denote the Jacobian matrix of the generating function g as \mathbf{J}_{g}(\mathbf{z}) and \mathbf{x} ’s dimensions under the influence of \mathbf{s} as \mathcal{I}_{\mathbf{s}}(\mathbf{z}):=\{i\in[d_{\mathbf{x}}]:\exists j\in\{d_{\mathbf{c}}+1,\dots,d_{\mathbf{z}}\},\text{s.t.},\,[\mathbf{J}_{g}(\mathbf{z})]_{i,j}\neq 0\} .
We note that the set \mathcal{I}_{\mathbf{s}}(\mathbf{z}) is a function of \mathbf{z} , since the influenced dimensions may vary over \mathbf{z} .
Intuitively, if \mathbf{s} influences \mathbf{x} in a dense manner, i.e., large \left\lvert\mathcal{I}_{\mathbf{s}}(\mathbf{z})\right\rvert for potentially all dimensions x_{i} , there may not be dimensions of \mathbf{x} serving as clear signatures of \mathbf{c} , thereby increasing the difficulty of identify \mathbf{c} .
Additionally, the degree to which the changing variable \mathbf{s} is out-of-support plays a critical role – the further the target changing variable– the further the target changing variable \mathbf{s}_{\mathrm{tgt}} deviates from the source distribution support \mathcal{S}_{\mathrm{src}} , the more severe and unpredictable the shift becomes, making it harder to retrieve \mathbf{c} .
In the following, we formalize conditions on the influence of \mathbf{s} from these two perspectives, revealing an interesting trade-off and interaction between these factors.
Dense-shift Conditions
We begin by investigating scenarios where there are no constraints on the number of dimensions of \mathbf{x} (i.e., the number of pixels) influenced by the changing variable \mathbf{s} , i.e., potentially large \left\lvert\mathcal{I}_{\mathbf{s}}(\mathbf{z})\right\rvert , which we term as dense shifts.
For images, these shifts encompass global transformations such as changes in camera angles and lighting conditions that could potentially affect all pixel values (Figure 1(b) ).
Sparse-shift Conditions
We now examine cases where the changing variable \mathbf{s} influences only a subset of dimensions of \mathbf{x} , i.e., a limited \left\lvert\mathcal{I}_{\mathbf{s}}(\mathbf{z})\right\rvert , which we refer to as sparse shifts.
For image distributions, these shifts include local corruptions or background changes that do not alter foreground objects (Figure 1(c) ).
Synthetic Data Experiments
In this section, we conduct synthetic data experiments on classification to directly validate the theoretical results in Section 4 .
We present additional experiments on regression in Section A4.2 .
Real-world Data Experiments
We provide real-world experiments to validate our theoretical insights for practical algorithms (Section 4.3 ) and theoretical results (Section 4.2 ).
More results can be found in Appendix A5 . 2 2 2 The code is provided here .
Generative Adaptation with Entropy Minimization
As discussed in the first implication in Section 4.3 , we incorporate an entropy-minimization loss to MAE-TTT and compare it with the original MAE-TTT.
Sparsity Regularization
As suggested by the second implication in Section 4.3 , we integrate sparsity constraints into the state-of-the-art TTA method, TeSLA/TeSLA-s [ 21 ] .
Although our theoretical results rely on a generative model, we demonstrate that our implications are also applicable to discriminative models.
Shift Scope and Severity
To investigate the trade-off between the shift scope (dense vs. sparse) and severity, we simulate different levels of corruption severity and corrupted region sizes and evaluate a classical TTA method TENT [ 15 ] on these configurations.
Following [ 45 ] , we inject impulse noise to the CIFAR10 dataset, with noise levels ranging from 1 to 10 to simulate various severity levels.
To control the shift’s scope, we crop regions of various sizes and introduce corruption only to this region.
Figure 3 displays classification error curves under various shift severity levels and region sizes.
We can observe that classification errors rise with increasing noise levels and region sizes.
Notably, for large block sizes (dense shifts), the performance dramatically declines and even collapses as the severity level rises, whereas the performance remains almost constant over all severity levels in the sparse shift regime, verifying the theoretical conditions for Theorem 4.2 and Theorem 4.4 .
Conclusion and Limitations
In this work, we characterize extrapolation with a latent-variable model that encodes a minimal change principle.
Within this framework, we establish clear conditions under which extrapolation becomes not only feasible but also guaranteed, even for complex nonlinear models in deep learning.
Our conditions reveal the intricate interplay among the generating function’s smoothness, the out-of-support degree, and the influence of the shift.
These theoretical results provide valuable implications for the design of practical test time adaptation methods, which we validate empirically.

Limitations :
On the theory aspect, the Jacobian norm utilized in Theorem 4.2 only considers the global smoothness of the generating function and thus may be too stringent if the function is much more well-behaved/smooth over the extrapolation region of concern. Therefore, one may consider a refined local condition to relax this condition. On the empirical side, our theoretical framework entails learning an explicit representation space. Existing methods without such a structure may still benefit from our framework but to a lesser extent.
Also, our framework involves several loss terms including reconstruction, classification, and the likelihood of the target invariant variable. A careful re-weighting of these terms may be needed during training.
A4.1 Implementation Details
We employ a variational auto-encoder [ 67 ] whose encoder and decoder are both 4-layer MLP with 32 dimensions and leaky ReLu ( \alpha=0.2 ).
Following Equation 2 and Equation 3 , we implement reconstruction loss, KL loss on the source distribution, the likelihood loss on the target sample, and a classification loss on the source data.
For the dense case, we implement an additional distance loss to minimize the \ell_{2} distance of \hat{\mathbf{s}}_{\mathrm{tgt}} to the center of the source support (which is the origin in our case).
The source-only baseline is trained only with classification loss.
The iMSDA implementation is adopted directly from the source code of Kong et al. [ 18 ] .
We train all methods with Adam [ 68 ] and learning rate 2e-3 for 25 epochs.
We fix the loss weights \lambda_{\mathrm{cls}}=1 , \lambda_{\mathrm{recons}}=0.1 , \lambda_{\mathrm{tgt\_likelihood}}=0.1 , and \lambda_{\mathrm{s\_distance}}=0.01 (for dense shifts) overall distance configurations.
We only tune \lambda_{\mathrm{KL}} from the interval \{1e-1,1e-2,1e-3\} .
We run synthetic data experiments on one Nvidia L40 GPU and each run consumes less than 2 minutes.
A4.2 Regression Task Evaluation
In addition to the classification experiments, we evaluate our model on regression in this section.
A4.2.2 Results and Analysis
Table A1 displays the evaluation results.
We can observe that our method consistently outperforms the baseline and maintains its performance over a wide range of shift distances. In contrast, the baseline that directly uses all the feature dimensions degrades drastically when the shift becomes severe.
This indicates that our approach can indeed identify the invariant part of the latent representation, validating our theoretical results.
A5.1 Datasets
The datasets used in our experiments include CIFAR10-C, CIFAR100-C, ImageNet-C [ 45 ] , and ImageNet100-C. CIFAR10-C and CIFAR100-C are extended versions of the CIFAR datasets [ 69 ] designed to evaluate model robustness against visual corruptions, featuring 10 and 100 classes respectively, each with 50,000 clean training samples and 10,000 corrupted test samples. ImageNet-C, on the other hand, scales this concept up with 1,000 classes, providing 50,000 test samples of each of 15 corruption types.
ImageNet-100 [ 46 ] is a subset of ImageNet with 100 classes. In our experiments, we build ImageNet100-C by selecting 100 classes reported in Tian et al. [ 46 ] from ImageNet-C [ 45 ] with 15 different types of corruption.
A5.2 Generative Adaptation with Entropy Minimization
When applying entropy minimization in the MAE-TTT framework [ 20 ] , we did not directly integrate the entropy-minimization loss. The self-supervised MAE training process relies on masked images, whereas entropy minimization requires classifying the entire image. To address this, we introduced additional training steps using unmasked images and applied the entropy-minimization loss during these steps.
Specifically, the training process for each test-time iteration is split into two stages:
1) Stage One: We follow the MAE-TTT approach by inputting masked images and training the model using reconstruction loss. In this stage, only the encoder is updated.
2) Stage Two: We input full images (32 in a batch) and optimize the model with the entropy minimization loss following SHOT [ 43 ] . In this stage, both the encoder and classifier are optimized.
The learning rates for both stages are set to be the same.
The experiments are conducted with the PyTorch 1.11.0 framework, CUDA 12.0 with 4 NVIDIA A100
GPUs.
A5.3 Sparsity Regularization
Here, we provide the implementation details of our modification to add sparsity regularization.
In the pre-train stage, we apply the ResNet50 [ 47 ] as the backbone network and follow [ 14 , 44 ] to pre-train it on the clean CIFAR10, CIFAR100, and ImageNet training sets, with joint contrastive and classification losses.
In the test-time adaptation process, we adopt the sequential TTA protocol as outlined in TTAC [ 44 ] and TeSLA [ 21 ] . This protocol prohibits the change of training objectives throughout the test phase. Moreover, all testing data be processed in a sequential manner (one-pass), ensuring each data point is passed through the adaptation process exactly once.

Our method is built upon TeSLA [ 21 ] , and follows most of its hyperparameters. Thus, we only discuss the extra hyperparameters we involved, including the low-rank dimension r , the ratio of learning rate factor for soft frozen ratio_{l}=\frac{\text{lr}_{lora}}{\text{lr}_{backbone}} , and the ratio of sparsity loss ratio_{s} . The details are shown in Table A2 .
It was observed that the constraints on minimal change need to be more stringent as the complexity of the data increases.
The experiments are conducted with the PyTorch 1.13.0 framework, CUDA 11.7 with an NVIDIA A100 GPU.
A5.4 Recoverability of the Invariant Variable
We assess the impact of the recoverability of the invariant variable on Test-Time Adaptation (TTA) methods. To do this, we compare the performance of TTA methods using supervised and unsupervised pre-trained models that have similar ImageNet classification accuracy. Our goal is to validate whether invariant variables learned from annotated labels can improve test-time adaptation. We assume that annotated labels can help learn better invariant variables \mathbf{c} , which play an important role in solving extrapolation problems.

For this detailed analysis, we employ ResNet50 models pre-trained in both supervised and self-supervised manners, such as MoCo [ 70 ] .
For the MoCo model, we apply the linear probe to fit the classifier.
For a fair comparison, we select checkpoints from both pre-training methods that have similar ImageNet accuracies, approximately 69.7 \% . We follow the open-source TTA Benchmark [ 71 ] to evaluate both models using different downstream TTA methods. ImageNet-C is used as the evaluation dataset, and all hyperparameters are set to their default values.

The performance of the different pre-trained models is summarized in Table A3 . Methods using supervised pre-training outperform those using unsupervised pre-training by a significant margin, indicating that the invariant variables learned from annotated labels play a crucial role in enhancing test-time adaptation.
A5.5 Additional quantitative results.
In Table A4 , we provide the detailed performance with
corruption-wise classification error rates on all CIFAR10-C,
CIFAR100-C, and ImageNet-C datasets. Specifically, we report results under seed 0 on all 15 testing corruptions including Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic Transformation, Pixelate, and JPEG Compression.
(a) Extrapolation task.; (b) Dense shift (Theorem 4.2 ).; (c) Sparse shift (Theorem 4.4 ).; Illustration of extrapolation and our theoretical conditions. The horizontal axis represents the changing variable \mathbf{s} , ranging from the source support to out-of-support regions.
The vertical axis represents the observed data \mathbf{x} living on the manifolds indexed by different values of the invariant variable \mathbf{c} .
Figure (a) demonstrates that given a point out of support it is unclear which class manifolds it belongs to.
Figure (b) illustrates the dense shift condition (Theorem 4.2 ) where \mathbf{s} potentially changes all pixels in the images, such as the camera angle in the example.
In this case, we can identify the invariant variable \mathbf{c} under a moderate amount of shift until the shift becomes excessive. For instance, the back view of the cat in the figure could be confused with other animals.
Figure (c) illustrates the sparse shift condition (Theorem 4.4 ) where \mathbf{s} influences a limited number of pixels, such as the background in the example.
In contrast to the dense shift, we can identify \mathbf{c} under the sparse shift regardless of its severity.
In the figure, there is no ambiguity of the class “cow” even though the background has changed to the moon.
Figure 2 : The data-generating process. The invariant latent variable c and the changing latent variable s jointly generate the observed variable x .
The dashed line indicates potential statistical dependence.; The data-generating process. The invariant latent variable c and the changing latent variable s jointly generate the observed variable x .
The dashed line indicates potential statistical dependence.
Figure 3 : TTA classification errors under different levels of shift severity levels and scopes.; TTA classification errors under different levels of shift severity levels and scopes.
Synthetic data test accuracy under both dense and sparse shifts across a range of distances.
Comparison of SOTA TTA Methods on CIFAR10-C, CIFAR100-C, and ImageNet-C. Average error rates over 15 test corruptions are reported. Baseline results are from Tomar et al. [ 21 ] . Values are (means \pm standard deviations) over three random seeds. ∗ indicates our reproductions.
Test accuracy (%) on ImageNet-C. The baseline results are from Gandelsman et al. [ 20 ] .
Understanding entropy-minimization steps on ImageNet100-C . Values are classification accuracy (mean and standard deviation) over three random seeds.
Synthetic data results on regression (MSE) under both dense and sparse shifts across various distances.
Hyperparameters for minimal change constraint in our experiments.
ImageNet-C evaluation of TTA algorithms with supervised and moco pre-trains .
Detailed corruption-wise results on CIFAR10-C, CIFAR100-C, and ImageNet-C . We report the error rates (%) on 15 testing corruptions.