Test-Time Adaptation by Causal Trimming
Authors: Yingnan Liu
Rui Qiao
Mong-Li Lee
Wynne Hsu
Venue: NeurIPS 2025 poster ()
Keywords: test-time adaptation; distribution shift; causality; representation;
Primary area: general_machine_learning
TL;DR: We propose a test-time adaptation method to identify and remove non-causal features.

Abstract: Test-time adaptation aims to improve model robustness under distribution shifts by adapting models with access to unlabeled target samples. A primary cause of performance degradation under such shifts is the model’s reliance on features that lack a direct causal relationship with the prediction target. We introduce Test-time Adaptation by Causal Trimming (TACT), a method that identifies and removes non-causal components from representations for test distributions. TACT applies data augmentations that preserve causal features while varying non-causal ones. By analyzing the changes in the representations using Principal Component Analysis, TACT identifies the highest variance directions associated with non-causal features. It trims the representations by removing their projections on the identified directions, and uses the trimmed representations for the predictions. During adaptation, TACT continuously tracks and refines these directions to get a better estimate of non-causal features. We theoretically analyze the effectiveness of this approach and empirically validate TACT on real-world out-of-distribution benchmarks. TACT consistently outperforms state-of-the-art methods by a significant margin.

Adaptation by Causal Trimming
Yingnan Liu
Rui Qiao!.3 Mong Li Lee
Wynne Hsu 1.2
Sccoool of Computing, National University of Singapore Institute of Data Science, National University of Singapore *Singapore-MIT Alliance for Research and Technology singgan rui qiaaoiiauuuu wuw..uuuuuuuuu.u.eueeuuee...e... edu, {dcsleeml, dcshsuv) eduueeuu
Abstract
Test-time adaptation aims to improve model robustness under distribution shifts by adapting models with access to unlabeled target samples. A primary cause of performance degradation under such shifts the model's reliance features that lack direct causal relationship with the prediction target. We introduce Test-time Adaptation by Causal Trimming (TACT), method that identifies and removes non-causal components from representations for test distributions. TACT applies data augmentations that preserve causal features while varying non-causal ones. By analyzing the changes in the representations using Principal Component Analysis, TACT identifies the highest variance directions associated with non-causal features. It trims the representations by removing their projections on the identified direc- tions, and uses the trimmed representations for predictions. During adaptation, TACT continuously tracks and refines these directions to get better estimate of non-causal features. We theoretically analyze the effectiveness of this approach and empirically validate TACT real-workd out-of- distribbiittion benchmarks. TACT consistently outperforms of-eeooftthearr methods by significant margin. Our code is available at https: / github. com NancyQuris/TACe
1 Introduction
Machine learning models often exhibit significant performance degradation when evaluated on data drawn from distribution that differs from their training data distribution [13]. To address this challenge, test-time adaptation (TTA) has emerged as promising approach. TTA methods adapt pretrained model to the test distribution dynamically, using the incoming test data to enhance predictive performance without requiring access to the original training data [19, 51, 56). Despite recent advances, many existing TTA methods rely heavily predicted labels generated by the model itself to guide the adaptation process [12, 37, 38, 51]. However, the effectiveness of these methods hinges critically on the quality of the predictions. When the model's predictions are influenced by non-causal features that do not have a direct causal relationship with the prediction target [26, 54], the predicted label may be unreliable, leading sub-optimal adaptation outcomes [29, 47).
Unlike causal features that have stable associations with the semantic structure of the prediction task [27]. non-causal features exhibit inconsistent spurious correlations with the prediction target across training and test distributions [55]. Over-reliance on non-cause features is a key factor in model performance degradation under distribution shift. While DeYO [29] recognizes this issue, it does not explicitly mitigate reliance on non-causal features. Instead, updates the model using predictions that leverage causal features only, relying on gradual adaptation reinforce causal features time. Consequently, early predictions may still be influenced by non-causal signals, requiring many adaptation steps to suppress their effects.
Given the above limitations, we propose to actively reduce non-causal features. Prior studies have shown that feature representations learned through standard training encode mixture of causal and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

non-causal features and that the causal part is often learned sufficiently well for accurate prediction [20, 27]. Motivated by this, we propose a Test-time Adaptation by Causal Trimming (TACT) framework that seeks to improve adaptation performance by isolating and removing non-causal components from the representations of samples from test distributions. Our framework aims to achieve more reliable predictions in the presence of distribution shift by reducing the model's s dependence on unstable, non-causal features. To identify non-causal features in representations, we analyze how these representations change when we apply targeted perturbations to the input data. Specifically, we perform input augmentations that preserve the underlying causal contents while introducing variability in other, non-causal aspects of the input [9, 11, 18, 31, 32]. These augmentations produce multiple test-time samples that share the same causal semantics but differ in spurious incidental attributes. By examining how the feature representations of these samples vary, we can disentangle causal and non-causal components.
We operationalize this by applying Principal Component Analysis (PCA) to the set of augmented representations and identify the di- rection of greatest variance. We interpret this dominant direction as being aligned with the non-causal features, under the assump- tion that causal content remains stable across augmentations, while
DeYO
TACT
No adapt
Betch accuraty (%)
20
non-causal attributes vary. This approach is inspired by prior work showing that high-level semantic factors are often linearly encoded in the learned representation space [1, 39, 46]. Building on the in- sight that linear manipulations in representation space can produce meaningful changes in semantic content [40, 50], we propose to reduce the influence of non-causal features by subtracting the projec- tion of a test sample's 's representation along the identified non-causal direction. Since the prototypes used for prediction, defined as tem- plate representations corresponding to the weights for each class in the linear classifier, are influenced by non-causal features, we apply the same operation to them using the identified non non-causal direction. During adaptation, we maintain a moving average of the updated prototypes to mitigate noise effects. Compared to Do,O, TACT can immediately produce predictions that are less affected by non-causal features, eliminating the need for iterative updates to achieve reliable results (see Figure 1). We provide theoretical analysis to establish the conditions under which TACT can improve prediction accuracy under distribution shift. Empirically, we evaluate TACT on five real- world out-of-distribution datasets, demonstrating its effectiveness and superiority over state-of -ftte-rrt TTA methods.
65
60
50 Adaptation
100
Figure 1: Batch accuracy on Camelyon17 dataset of the first 100 adaptation steps.
2 Related Work
Existing TTA methods can be broadly categorized into backpropagation-free and backpropagation based methods. Backpropagation free methods modify model outputs or intermediate representations without gradient-based optimization. These include modifiable prompts [36], re-normalized represen- tations [44], updated prototypes [19, 57], and maximum likelihood estimation [4]. Backpropagationn based methods update the model with the gradient of objective functions such as entropy minimization [12, 37, 38, 51] and self-training with pseudo-labels [17, 25, 47, 52]. Entropy Minimization encour- ages more confident predictions by reducing the entropy of model predictions during adaptation. Self-training employs cross entropy [5, 17, 30, 47] and knowledge distillation [25, 52, 53] using model predictions as pseudo-labels. Regularization measures such as information maximization [30], representation statistics alignment [23, 58), and consistency regularization [35, 56] for invariant prediction under augmentations have been proposed to regularize the adaptation.
A key challenge in test-time adaptation is obtaining reliable pscudo-labels to guide model updates. One line of work assumes that correct predictions tend exhibit low entropy, and update the model using only nigh-confidence samples with low-entropy predictions [19, 37, 38, 57]. However, DeYO [29] shows that spurious correlations can also result in low entropy predictions and proposes a causal intervention technique to identify predictions that are more likely based on causal features, using them selectively for model updates. Another line of work refines pseudo-labels by incorporating updated prototype and neighborhood information [5. 17, 47, 53). AdaContrast [5] uses soft voting among nearest neighbors. TSD [53] relies on updated prototypes and spatial local clustering. TAST [21] employs neighbourhood information in self-training PROGRAM [47] considers both prototype

---

and neighbour-bet pseudo-labee to enhance label quality. PASLE [17] progressively refines the pseudo-labels of uncertain predictions using updated prototypes.
All the above methods, except for DeYO, do not consider the effect of non-causal features on model prediction. Although Yo finds that non-causal features would make entropy unreliable metric to reflect prediction correctness, it does not adjust model predictions. TACT adjusts model predictions by reducing non-causal features, and our adjusted prediction can be used as a more reliable pseudo-label.
3 Preliminaries
We consider the problem of adapting a well-trained model to test-time distributions that differ from the training distribution. Our goal is to improve the model's performance on these shifted distributions with unlabeled test samples. Following prior work (18, 48), we model the dis- tribution shift using a structural causal model that captures the underlying fata-generating process, as illustrated in Figure
X
X
Y
x
We model the observed sample and its label Y as being generated from causal factors X and non-causal factors XNc. Only X is causally related to Y, while is related to both X and XNC- The correlation between X and Y is stable, i.e., the conditional distribution P(Y xx) remains unchanged at test time. We also assume that the distribution of causal factors P(Xc) remains invariant across the training and test datasets, whereas distribution shifts arise from changes in P(X NC). The model would have stable performance across distributions the prediction is based solely on features related to the causal factors Xc. In contrast, reliance on non-causal factors XNC can lead to unreliable performance under distribution shifts.
Structural causal model of the data- generating process.
Figure 2:
We consider a c-class classification task, where the model f f= h o g used for adaptation is composed of a feature extractor and a classifier h. The feature extractor g maps input sample to d- dimensional vector E Rd as the representation. The classifier h maintains a set of class prototypes {gi (g...... }e Rd, where each prototype qi serves as a template representation for class i. Predictions are made by computing the similarity between the input representation and each class prototype using the dot product qG, referred the logit of class i. A softmax function is then applied to the logits to obtain the probability distribution over the classes.
4 Proposed Method
The works in [20, 27] show that models often capable of learning causal features, when their predictions are predominantly driven by non-causal features with spurious correlations. However, the predictive influence of these causal features is fiequeenly obscured or suppressed due to the heavily weighted non-causal components in the learned representations. Based on this observation, we propose TACT, method that identifies non-causal features and reduces their influence by causally trimming the learned representations. We hypothesize that non-causal features are embedded in representations along specific direction. Such direction is of the maximum variance when the non-causal features change. To suppress their influence. we subtract the projection of both the input representation and class prototypes onto this identified direction. This operation attenuates the non-causal information present in both elements. Since class prototypes serve as canonical representations for each class, and the non-causal direction estimated from a single test sample may be noisy, we maintain moving average of the updated prototypes throughout test-time adaptation. At inference, predictions are made by measuring the similarity between the adapted representation and the moving average of the updated prototypes, thereby reducing intluence of non-causal features.
4.1 Non-Cause Feature Identification
Given sample at test time, if we have access to additional samples generated with the same causal factors but different non-causal factors, we can compare their representations to infer the influence of non-causal factors. Changes in the representations across these samples can be attributed to variations

---

in these non-causal factors. By systematically analyzing these representational differences, we can isolate and identify the components of the representation that correspond to non-causal features.
To simulate variations in the data-generating process, we apply data augmentation to target non-causal features [9, 11, 18, 31, 32]. For test sample x; generate n augmented samples { that preserve the causal feature while varying the non-causal factors. We collect the representations of these samples in matrix and 2 are those of the augmented samples.
[z, 81, ...: =n] where 2 the representation of the original sample
We interpret non-causal features as corresponding to specific, disentangled directions in the repre- sentation space, consistent with prior work that indicates high-level semantic concepts are linearly encoded as vector directions in learned representations [1, 39, 46]. For instance, the vector difference between "woman" and "man" would resemble that between "queen" and "king" [33], both aligning to the direction representing gender. Along this direction, specifie instances of the gender concept, such as "male" and "female", take different magnitudes.
Given representations of samples that differ only in their non-causal factors, the direction along which the representations change the most is expected to capture the non-causal features. This dominant direction can be identified via Principal Component Analysis (PCA) which analyzes the covariance matrix of the representations to extract the principal components. Principal components are vectors along which the representations' projections exhibit maximum variances. We first compute the mean of the representations as: n+I** 2 + n+1 Ei= 24 Using this mean, we construct matrix -12
n
Z= [2, 2, T that has the same size the representation matrix Z. The covariance matrix of the representations is then given by Σz (Z- Z) (Z Z). The eigenvectors of Σz correspond to the principal components, and their eigenvalues quantify the variance along these components [22]. Since Σz is real symmetric matrix, its eigenvectors form an orthonormal basis in R2 [16]. Using spectral decomposition, we express the covariance matrix as: Σg = QAQ T, where Q [e1, 62; ea) is an orthogonal matrix whose columns are the orthonormal eigenvectors, and A is diagonal matrix containing the eigenvalues of Σz. Here, e; denotes the direction along which the variance of the projected representations is the 7th largest.
4.2 Causal Trimming to Reduce Non-Causal Feature
Prior work has demonstrated that applying linear transformations to representations can manipulate the semantics they encode [40, 50]. Since the principal components )d form orthonormal basis in Rd any representation z can be expressed as a linear combination of these components. To reduce the influence of non-causal features, we propose to trim the representation by removing its components along the top-m principal components:
m
8= 2
(2 e,)e;
(I)
1=1
Since each e; is a vector of unit length, the term (2 - e;)c; is the projection of z onto e; whose magnitude is given by the dot product (a ee). By subtracting these terms, we obtain an updated representation & which is composed of components only formed by {e.] If causal features are invariant under data augmentations and their corresponding semantic directions are orthogonal to those of the removed directions, causal information present in is preserved in the trimmed representation I.
4.3 Model Adaptation
In prototype-based classifier, each class prototype z serves as a template representation learned by the classifier h, summarizing the representations of samples belonging to class j. However, if the learned representations encode non-causal features, the prototypes will be influenced by these features. To mitigate this issue, we apply the same causal trimming to the class prototypes. Specifically, let gj be the prototype of class j. Given the top-m principal components (ca 1. that used to trim the test sample representation 2, we obtain the trimmed prototype q for each class j E {1,2,.... cf as:
4 = 1
(g - e))e;

---

Since the identified non-causal directions may vary across samples due to noise or context-specific factors, we compute batch-wise average trimmed prototypes obtain a more stable estimate. To track the estimate across batches during adaptation, we maintain a moving average of the trimmed prototypes during test-time adaptation. Suppose we obtain trimmed prototypes df" for each class at batch i, the moving- average Φ is updated by i = و + 1 This moving average serves as a more robust estimate of the causally refined prototypes. effectively smoothing out sample-specific variance. The prediction made by the average of the trimmed prototypes the same as that of an ensemble over the logits produced by individual trimmed prototypes, resulting in more stable predictions. At test time, for given input sample z, we compute the causally trimmed representation è and compare it to the
trimmed prototypes 0z. The logit for class j is given by the dot product 2 .z. and the final predicted label y is given by:
exp (2 (5)
y arg max
exp (3 (.)
5 Theoretical Analysis
We present the conditions under which TACT would correct a wrong prediction and maintain correct prediction, We consider binary classification Y E [+1,-1] The two prototypes learned by the binary classifier h represented {4+1, 9-1). We drop the bias term for simplicity. Meanwhile, we assume the existence of causal prototypes p which always make correct predictions on the learned representations and do not leverage non-causal features. To simplify the analysis, we consider the decision boundary vectors Aq q+1 - 9-1 and Ap = p+1 Pp11- We analyze the representation = of instance with label y. Given the principal components {er) computed from
2 and its augmented variants, we write z as Σi= a,Ci, where a; is the magnitude of 2's projection on eş. Similarly, we define the learned decision boundary Âq's projection magnitude as { We write the projection magnitude of causal decision boundary Ap as { to view Ap as a transformation from Aq by a projection magnitude l7 on the direction of each principal component.
We can obtain by trimming the top-m principal components (PCs) for z. Proposition 1 shows the conditions under which TACT can correct a wrong prediction.
Proposition 1. For any 2 that is misclassified by the learned decision boundary Aq. the misclassi- fication can be corrected by using the representation obtained after removing the top-m principal components, if both of the following two conditions are satisfied:
n
d
OA < 0 and y
Σ i=m+1
a.7 > 0
(4)
0
0
(5)
i=+++
Appendix A. provides the formal proof. Equation (4) captures the case in which a prediction based solely on the top-m PCs leads to an incorrect outcome, whereas prediction based on the remaining PCs yields the correct result. Equation (5) requires the absolute value of the prediction score derived from the top-m PCs must be greater than that from the remaining PCs. Together, these conditions in Proposition 1 suggests that a wrong prediction can be corrected by TACT when the top-m PCs are solely responsible for the wrong prediction, and the prediction made by the top-m PCs weighs more than the prediction made by the remaining PCs.
In Proposition 2, we establish the conditions under which the trimmed representation é retains sufficient causal information preserve the correct prediction by the causal decision boundary Ap.
Proposition 2 (Causal Preservation). For any original representation z, the trimmed representation 2 preserves the correct prediction under the causal decision boundary Ap if any one of the following
5

---

conditions holds:
Σ
71007
77a47
(6)
m Σ 77477 < y Σ 7,47%
The proof is provided in Appendix A.2. Equation (6) characterizes three cases: (a) the top-m PCs have no contribution to the causal prediction; (b) the top-m PCs has a negative influence on the causal prediction and thus their removal is beneficial; (c) the top-m PCs has a positive contribution, but the representation forms by all PCs contribute even more strongly. When the top-m PCs have no contribution the causal predictions, they are considered non-causa features. In other words, the removed component 2 does not contain causal information. When the top-m PCs contain causal information, m should be selected such that the top-m PCs contribute less to the prediction compared to all the PCs, ensuring that the trimmed representation E remains causally informative. In other words, sufficient causal features need to be preserved after causal trimming.
Finally, in Proposition 3, we identify the conditions under which causal trimming would have no negative impact on the prediction of samples that already correctly classified.
Proposition 3. Suppose is correctly classified by the learned decision boundary Ag. The trimmed representation obtained via TACT will still be classified correctly if either of the conditions holds:
1. y(= a) S O, or
2. y(= =) 0, and Equation (7) holds, assuming already satisfies the Causal Preserva- tion condition (Proposition 2).
sign
Σ j=m++
TR047
sign
Σ
0477
\==++1
The proof can be found in Appendix A.3. Equation (7) indicates that when classification relies only on the representations formed by the remaining PCs, the learned decision boundary makes the same prediction as the causal decision boundary. Proposition also shows that if correct prediction is made by the learned decision boundary. TACT will preserve this correstness as long as the removed part å contributes negatively does not contribute the prediction. On the other hand, when the trimmed representation & contains sufficient causal information as established Proposition the learned decision boundary is required to align directionally with the causal decision boundary defined by the remaining PCs.
6 Performance Study
We study the test-time adaptation performance under real-world distribution shifts, using datasets from multiple modalities, including image, audio, and text. Compared to prior works that pri- marily benchmark on image data, our comprehensive experiments offer broader insights into the generalizability of TACT and other TTA methods.
Datasets. We summarize the datasets used our experiments below:
Birdcalls [15, 24, 34], curated by [9], is an audio classification dataset to identify bird species from clips recorded in diverse environments. Each clip is converted into Mel spectrogram for classification. Distribution shifts stem from variations in microphone gain settings, habitat acoustics (e.g. other animal sounds), and bird population. The test set includes 724 audio clips. Camelyon17 [2], sourced from from the Wilds benchmark [28], is medical imaging dataset for binary classification of tumor versus normal tissue images. The distribution shift arises from variations in slide staining protocols, patient demographies, and scanner equipment. The test set consists 85,054 images.
CivilComments [3], from the Wilds benchmark [28], is natural language dataset comprising user-submitted text comments. The task is to classify whether comment is toxic or non-toxic.

---

The toxicity is spuriously associated with the mention of certain demographics in the training
data. The test set contains 133.782 comments.
ImageNet-R [14] contains 30,000 images of objects from 200 ImageNet [42] classes. The images consist of various renditions, resulting in visual domain shifts from the original dataset. ImageNet-V2 V2 [41] is collected years after the original ImageNet using the same methodology. and includes 10,000 images across 1,000 original classes. It represents natural temporal shift.
Non-causal feature identification for TACT. We applied the following data augmentations to identify non-causal features in each dataset: For Birdcalls, we follow [9] that investigates augmentations that randomize features independent of labels but dependent on distributions. Here, random color jitter applied to the Mel spectrograms to simulate changes in microphone gain settings. For Camelyon 17, we use stain color jitter [49] as suggested in [9] to mimics variations in histopathological slide staining. For CivilComments, we randomly prepend
append short demographic-reeereee sentences to the original text. The full list of sentences is provided in Appendix B. For ImageNet-R and ImageNet-V2 V2, where the sources of distribution shift are unknown, we experiment with general- purpose image augmentations. Specifically, we apply AutoAugment [6] with ImageNet policy and RandomAugment [7]. Both methods apply series of transformations to the images. A detailed discussion on augmentation design and selection in practice is presented Appendix C.
Baselines. Since TACT is backpropagaaion--fre approach, we compare TACT with the following state-of-the-art (SOTA) TTA backpropagation-Fe free algorithms:
T3A [19] adapts the classifier by updating class prototypes using confident test-time represen- tations.
LAME [4] adjusts model output probabilities via Laplacian-adjusted maximum likelihood
estimation.
FOA [36] introduces an adaptable prompt at model input to match the representation statistics of test and train data.
We also implement variant called TACT-adapt, where predictions from TACT are used to guide gradient-based model updates with cross entropy loss Lcg. We employ the information maximization loss LIM proposed in SHOT [30] as regularization. We optimize the model using the objective:
f == Lcr (Ú. TTACT) + X (ū). is the model's prediction, and yTACT is TACT's prediction. À is the hyperparameter balancing the two terms.
We compare TACT-adapt with the following SOTA backpropagation-baseed methods:
SHOT [30] adapts the feature extractor using information maximization and cross entropy loss on confident prediction.
Tent [51] performs entropy minimization update the affine parameters of normalization
layers at test time.
SAR [38] builds upon Tent by incorporating charpness-awar minimization and model reset to mitigate overfitting to noisy samples.
DeYO [29] identify confident samples that leverage causal features only by image augmenta-
tions that destroy shapes and using confidence- confidence-reweighheed reweighttd entropy minimization to update the affine parameters.
TAST [21] adapts trainable module top of the trained feature extractor via self- self-training with nearest neighbor information.
TSD [53] enhances feature representations through self eelf-disillattin and local clustering, ensur- ing alignment and uniformity while filtering noisy labels.
PASLE [17] refines uncertain pseudo-label progressively using selective label enhancement with candidate label sets and cusssffer-consisstee loss.
Model architecture. We study TACT on munsformer-based architectures, which are increasingly used in practice but remain relatively underexplored in TTA. Specifically, we use ViT-B/32 [8] as the backbone for Birdcalls, Camelyon17. 17, ImageNet-R, and ImageNet- and DistilBERT [43] for CivilComments. Appendix D.I provides more details model studied.
Hyperparameters and model selection. We use a test batch size of 64 [29, 36). There are two hyprpaarameters in TACT, the number of augmentation
and the number of removed principal components m. We search n E [2-, 22,22,, ...,, ,,,,, m E [1, 16) and m is an integer. For TACT- adapt, we search À E [1,5} x {0.1, 1, 10, 1003. The rest hyperparameters follow the search space
7

---

Table 1: Test-time adaptation performance (%). We group the methods into backpropagation (BP-free) and backpropagation-batt based (BP-based) The best performance of each dataset is in bold.
Method
Birdcalls
Camelyon17
CivilComments
ImageNet-R
ImageNet-V2
No TTA
22.74
62.31
55.38
41.83
62.97
T3A
1.33
69,96:1:.98
56.43310.00
11.78:10.12
62.93:10.02
BP-
LAME
23.66++1011
62.38:+0003
56.24:10.10
41.77:40.01
63.00:+0.02
free
FOA
1.81
58.36:+0.77
41.46-+0.16
62.76++0.08
TACT
1..14=1..69
70.17=00.05
7180110.35
43.59:10.02
63.33+0.10
SHOT
26.82:15.14
80.2811..61
13.93
48.79110.08
63.32:0009
Tent
23.16+00.42
62.29+0.01
55.08+00.0
42.08+0.05
63.09+00.33
SAR
23.16+00.42
62.30+00.0
55.08
42.58+0.11
62.97+0001
BP-
De
23.29-+0.39
19.64+1.47
46.87+00.08
62.96+0.01
based
TAST
2..08
83.01
56.56.10.20
41.09.10.08
62.84110.07
TSD
1..7
67.3311.74
55.38
41.76.10.01
62.9810.01
PASLE
27.35.11.79
60.66.10.04
55.77
46.08
63.15100.04
TACT-adapt
31.25++3.59
83.70=1.10
7.19
48.81+00.05
63.44::0007
of SHOT. For all baseline methods, we perform hyperparameter tuning within the search spaces specified in their respective papers. The detailed configurations and search procedures are provided Appendix D.2. Following the protocol recommended [59]. we employ oracle selection choose the best rest-performing hyperparameters, ensuring fair and consistent evaluation across all methods.
6.1 Test-time Adaptation Performance
Following the evaluation protocol of each dataset, we use macro F1 for Birdcalls, accuracy for Camelyon17. ImageNet-R and ImageNet- V2, and worst-group accuracy for CivilComment, whose data are grouped by demographic attributes and toxicity. Due the high variability observed in Birdcalls, each experiment is repeated ten times, whereas experiments on the remaining datasets are conducted three times. The mean and standard deviation are summarized in Table 1.
We see that TACT consistently outperforms existing backpropagation free methods on all the datasets, with substantial gains of 4% on Birdcalls, 15% on CivilComments, and 1.7% on ImageNet- R. Further, TACT-adapt achieves the best overall performance across all datasets, outperforming both backpropagation-fiee freee and backpropagation baadd baselines. These results suggest that non-causal features are a major source of performance degradation under distribution shift, and that removing them improves predictive reliability. It also confirms the value of TACT not only as standalone method but also as a reliable supervisory signal for test-time learning.
We note that TACT performs well when causal features are approximately invariant under augmenta- tion. For ImageNet-R and ImageNet-V2, Autonugment [6] and RandomAugment [7] maintain the key causal features, which are object structure and shape [10, 29]. Other causal features that could be helpful in inferring objects, such as color when inferring strawberries. are altered. In addition. the models we perform adaptation on do not have their representation space explicitly constrained such that causal and non-causal features are linearly encoded, disentangled, or orthogonal. Yet, the approximate separation of causal and non-causal features by PCA yields consistent performance gains, suggesting the robustness of TACT.
6.2 Visualization of Predictions after Causal Trimming
To gain insight into the predictions made after causal trimming, we employ GradCAM [45] to visualize the focus of the original predictions and those made by TACT samples from ImageNet-R. GradCAM identifies which parts of an input image contribute most to a prediction by computing the gradients of the predicted class score with respect the embeddings of the image patches. The resulting heatmaps are overlaid on the input images, where brighter regions indicate higher importance for the prediction.
The visualization results are presented in Figure 3. Compared to the original predictions, TACT places less emphasis non-causal information, such as background elements. For instance, in the

---

Input
GmaCAM
TACT-GridCAM
Inpot
TACT-GraCCAM
ground
ground diver, prediction: grey whole
ground truch: white shark
ground truth: strawberry: prodiction: bathrub
ground truth: Jorikeet
ground truth: space shuttle; prediction: harp
ground Iruth: meerkal
ground truth: pug. dog; prediction: hatchet
(a) correct predictions
(b) wrong predictions corrected by TACT
Figure 3: GradCAM visualizations of the original predictions and TACT's predictions.
snowbird sample, TACT disregards irrelevant features like the surrounding branches. Similarly, in the white shark example, TACT restricts the focus to the object itself, unlike the original prediction that diffuses significant attention across the background. The sea background in the scuba diver example, and the dot texture in the background of the strawberry example, are likely to be spuriously correlated with certain prediction classes. These features are de-emphasized by TACT, contributing more accurate prediction.
Furthermore, TACT enhances attention object's defining characteristics. This clearly demonstrated in the lorikeet example, where the beak becomes the key focus, and the meerkat example, where attention is concentrated on the banded pattern and body. Moreover, in cases where the original prediction neglects causal features, as shown in the space shuttle and pug-dog example, TACT redirect the emphasis the actual salient features, such as the nose cone of the space shuttle and the face of the pug-dog, resulting in improved prediction performance.
causal features, leading sharper focus
6.3 Effect of Hyperparameters
The performance of TACT depends two key hyperparameters: the number of augmentations and the number of removed principal components m. These parameters govern the accuracy of non-causal direction estimation and the extent of causal trimming, respectively. Figure 4 shows the performance under different numbers of augmentations and removed principal components for the Camelyon17, CivilComments and ImageNet-R datasets.
Since representations from augmented samples are used to compute the covariance matrix from which the directions of maximum variances are identified, larger number of augmentations generally leads to more stable and accurate identification of non-causal directions, Empirically, we find that values of {128, 256, 512} provides sufficient performance, while small values of often fail to

---

Camelyon17
CivlComments
imagelietRR
70
01.
Nerfomartee (86)
63.5
1004
60
65.0
02.5
89.3
0.50
57.9
0.5
15.0
45
36.2
4.75
nurbber
Figure Performance across number of augmentation and number of removed principal components.
adequately capture the variance needed for accurate principal component estimation. The number of removed principal components m, should be carefully selected to ensure effective reduction of non-causal features while retaining sufficient causal features as suggested by our theoretical analysis. In practice, removing the top principal component, which typically captures the dominant non-causal variation, often suffices. However, for datasets with more complex layered distribution shifts, such as ImageNet-R, R, removing more principal components would further boost the performance.
6.4 Ablation Study
We conduct two sets of ablation experiments. In the first experiment, we isolate the impact of representation trimming, without prototype averaging, In the second experiment, we assess whether prototype averaging alone can sufficiently filter out non-causal features.
Table 2 shows the results. We observe that trimming the representation & yields better performance than adaptation, confirming that removing components aligned with non-causal directions in representations is beneficial. While using only the averaged trimmed prototypes I also improves performance over no adaptation, the gains generally less significant than when trimmed repre- sentations are employed. This suggests that relying solely on the averaged trimmed prototypes is insufficient for effectively reducing non-causal features. The best performance achieved when both the trimmed representation and the averaged trimmed prototypes are used in conjunction. indicating that mitigating non-causal features in both representations and prototypes is crucial.
Table 2: Results of ablation study.
trim
trim
average
Birdcalls
Camelyon17
CivilComments
ImageNet-R
ImageNet-V V2
No TTA
22.74
62.31
55.38
41.83
62.97
25.91
69.43-10.11
67.84:10.37
43.21:10.33
63.2411010
/
27.36-10.23
64.74+0.05
62.41110.08
42.24:10.00
63.03::0001
/
31.14+1..99
70.17=0005
71.80-10.35
43.59
63.33-..10
7 Conclusion and Future Work
We present TACT, a test-time adaptation method that reduces model reliance on non-causal features for test representations. TACT identifies non-causal components in the representation space by analyzing samples with identical causal features but varying non-causal features. The directions of maximum variance among the representations are treated as the non-causal directions. To adapt the model, we subtract the projection of the representation and class prototypes onto this non-causal direction. We keep track of the identified directions and utilize the average of the trimmed class prototypes for improved prediction. We analyze the theoretical conditions for TACT to enhance predictive performance. Extensive experiments on five real-world -ii-tf-distribution datasets demon- strate the effectiveness and generalizability of our approach. While TACT demonstrates strong performance, requires prior knowledge of the data to select augmentations that vary non-causal features without altering causal ones. Future work should explore identifying non-causal features when such knowledge is unavailable, and better methods to find non-causal features beyond PCA's orthogonality constraint.
10

---

Acknowledgement
We thank Dr Fusheng Liu for the helpful discussions. We appreciate the anonymous reviewers and AC for the constructive and valuable feedback.
References
[1] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. latent vari- able model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
[2] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Mcyke Hermsen, Babak Ehteshami Bejnordi. Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection individual metastases classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018.
[3] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, 2019.
[4] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8344-8353, 2022.
[5] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 295-305, 2022.
[6] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 113-123, 2019.
[7] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[9] Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsumori Hashimoto, and Percy Liang, Oft-of-dommin robustness via targeted augmentations. In International Conference on Machine Learning, pages 10800-10834. PMLR. 2023.
[10] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
[11] Karan Goel, Albert Gu, Yixuan Li, and Christopher Re. Model patching: Closing the sub- group performance gap with data augmentation. In International Conference on Learning Representations, 2021.
[12] Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. SoTTA: Ro- bust test-time adaptation on noisy data streams. In Advances in Neural Information Processing Systems, 2023.
[13] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. International Conference Learning Representations, 2021.

---

[14] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli. Mike Guo, et al. The many faces of robustness: A critical analysis of out-of distribiitiittionnnn generalization. In Proceedings of the IEEF/CVF international conference on computer vision, pages 8340-8349, 2021.
[15] W. Alexander Hopping, Stefan Kahl, and Holger Klinck. A collection of fully-annotated sound- scape recordings from the southwestern amazon basin. URL 2022.
[16] Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
[17] Yihao Hu, Congyu Qiao, Xin Geng, and Ning Xu. Selective label enhancement learning for test-time adaptation. In International Conference on Learning Representations, 2025.
[18] Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In International Conference Learning Representations, 2023.
[19] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In Advances in Neural Information Processing Systems, volume 34, pages 2427-2440, 2021.
[20] Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson On feature learning in the presence spurious correlations. In Advances in Neural Information Processing Systems, 2022.
[21] Minguk Jang, Sae-Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighbor information. In International Conference on Learning Representations, 2023.
[22] I.T. Jolliffe. Principal Component Analysis. Springer Series in Statistics. Springer, 2002.
[23]
Sanghun Jung, Jungsoo Lee, Nanhee Kim, Amirreza Shaban, Byron Boots, and Jaegul Choo. Cafa: Class-aware feature alignment for test-time adaptation. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 19060-19071, 2023.
[24] Stefan Kahl, Russell Charif, and Holger Klinck. A collection of fully annotated soundscape recordings from the northeastern united states. URL https://zeendo.. org/records/701 https://ccoodoorrrrroo///////// 2022.
[25] Juwon Kang, Nayeong Kim, Donghyeon Kwon, Jungseul Ok, and Suha Kwak. Leveraging proxy of training data for test-time adaptation. In International Conference on Machine Learning. pages 15737-15752. PMLR, 2023.
[26] Jivat Neet Kaur, Emre Kiciman, and Amit Sharma. Modeling the data tata-generating process is necessary for out-t--distributiin generalization. In International Conference on Learning Representations, 2023.
[27] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson Last layer re-training is sufficient for robustness to spurious correlations. In International Conference on Learning Representations, 2023.
(28) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[29] Jonghyun Lee, Dahuin Jung, Sachyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In International Conference on Learning Representations, 2024.
[30] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. International Conference on Machine Learning, pages 6028-6039. PMLR, 2020.
12

---

[31] Fangrui Lv. Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain generalization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8046-8056, 2022.
[32] Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, pages 7313-7324. PMLR, 2021.
[33] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association computational linguistics: Human language technologies, pages 746-751, 2013.
[34] Amanda Navine, Stefan Kahl, Ann Tanimoto-Johnson, Holger Klinck, and Patrick Hart. collection of fully-annotated soundscape recordings from the island of hawai'i. URL https://doi.org/10.52// htt../2//222ooo.7777999999999999 2022.
[35] A. Tuan Nguyen, Thanh Nguyen-Tang. Ser-Nam Lim, and Philip H.S. Torr. Tipi: Test time adaptation with transformation invariance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 84162-24171, June 2023.
[36] Shuaicheng Niu, Chunyan Miao, Guohao Chen, Pengcheng Wu, and Peilin Zhao Test-time model adaptation with only forward passes. In International Conference on Machine Learning, 2024.
[37] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Confer- Machine Learning, pages 16888-16905. PMLR, 2022.
[38] Shuaicheng NIu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations, 2023.
[39] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, pages 59643-39666 PMLR, 2024.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PmL.R, 2021.
pages
[41] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference machine learning, pages 5389-5400. PMLR, 2019.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 1115:222111222., 2015.
[43] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.01108, 2019.
[44] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Processing Systems, volume 33, pages 1153--1151, 2020.
[45] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via bradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626 2017.
13

---

[46] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 9243-9252, 2020.
[47] Haopeng Sun, Lumin Xu, Sheng Jin, Ping Luo, Chen Qian, and Wentao Liu. PROGRAM, PROtotype GRAph model based pseudo-label learning for test-time adaptation. In International Conference Learning Representations, 2024.
[48] Xinwei Sun, Botong Wu, Xiangyu Zheng, Chang Liu, Wei Chen, Tao Qin, and Tie- Yan Liu. Recovering latent causal factor for generalization to distributional shifts. In Advances in Neural Information Processing Systems, volume 34, pages 16846-16859, 2021.
[49] David Tellez, Maschenka Balkenhol, Irene Otte-Höller, Rob Van De Loo, Rob Vogels, Peter Bult, Carla Wauters, Willem Vreuls, Suzanne Mol, Nico Karssemeijer, et al. Whole-slide mitosis detection in h&e breast histology using phh3 a reference to train distilled stain-invarian convolutional networks. IEEE transactions on medical imaging, 37(9):2126-2) 2136, 2018.
[50] Paul Upchurch, Jacob Gardner, Gcoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7064-7073, 2017. 2017.
[51] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[52] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[53] Shuai Wang, Daoan Zhang, Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 20060-20060, 2023.
[54] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre- -llvis Rebuffi, Ira Ktena, Krishna- murthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2022.
[55] Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-benchs Quantifying and understanding two dimensions of -iitr--distiiuttonn generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7947-7958, 2022.
[56] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems, volume 35, pages 38629-38642, 2022.
[57] Yifan Zhang, Xue Wang, Kexin Jin, Kun Yuan, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Adanpe: Exploring non parammemetiicc classifier for test-time adaptation. In International Conference on Machine Learning, pages 41647-41674 PMLR, 2023.
[58] Zhen-Yo Yu Zhang, Zhiyu Xie, Huaxiu Yao, and Masashi Sugiyama. Test-time adaptation in non-stationary environments via adaptive representation alignment. In Advances in Neural Information Processing Systems, volume 37, pages 94607-94632, 2024.
[59] Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In International Conference on Machine Learning, pages 42058 42080. PMLR, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the claims and contributions in the abstract and introduction, which are further justified in Sections 4, and 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: discuss the limitations in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

We discuss the assumptions of the Propositions in Section 5, and provide the proof in Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided experimental configurations in Section and Appendix for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the code and the used data in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wttpccccpppc/eeeeeeee.eec...lele sguides/CodeSiiee ctmbbmmmssiooooccyyy...... for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have provided experimental configurations Section 6 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have included error bars for both the main experiments and the ablation study in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources used in the Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httt://p.....pbleeccccccccceeeee
Answer: [Yes]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential socictal consequences of our work, none which we feel must be specifically highlighted here.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper uses existing public datasets and develops a method trained on these datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the datasets in Section 6 and baseline related works in Section 2 and Section 6.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddataees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the code and datasets used in the supplemental materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
The core method development in this research does not involve LLMs as any important, original, non-standard components, except that ChatGPT to generate the augmentations for the textual data in CivilComments dataset, as described Appendix B.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttppp/////erss.../. for what should or should not be described.
21

---

A Details of Theoretical Analysis
A.1 Conditions for TACT to correct a wrong prediction
We first restate Proposition follows:
Proposition For any z that misclassified by the learned decision boundary Dq, the misclassi- fication can be corrected by using the representation obtained after removing the top-m principal components, if both of the following two conditions are satisfied:
an 0 and y
Σ 1=m+1
am > 0
(4)
0.7
Σ
0%7
(5)
i fimm+
Proof. As the learned decision boundary Aq cannot classify correctly, we have:
yz Ag
<
0g
7e < 0
1=1
1=1
1/
0 ei) <
1=1
y
0:7 < 0
d
Σ 0
+
Σ
0:7 < 0
(8)
\=m+1
TACT updates z to 2 and g to á via causal trimming, and the resulting prediction is correct if and only if y²- Ağ >> 0, which leads to:
y² Aq > 0
Σ
Σ
Og6;
7/6;
(=M+1
(=m+1
y
ar7(ee e;) > 0
i=m+1
y
0,7 >
(9)
i=m+1
By combining Equation (8) and (9), can derive:
<
Σ
0:7 <0
(10)
i=++)
In addition:
047
Σ
(11)
22

---

A.2 Conditions for trimmed representations to preserve causal features
Proposition 2 (Causal Preservation). For any original representation 2, the trimmed representation
å preserves the correct prediction under the causal decision boundary Ap if any one of the following conditions holds:
Σ
7,am 0
Σ Maan < 0
(6)
d
< Σ 7:077 < E 7:0077
Equation (6) characterizes three cases: (a) the top-m PCs have no contribution to the causal prediction; (b) the top-m PCs has negative influence the causal prediction and thus their removal is beneficial; (c) the top-m PCs has positive contribution, but the representation forms by all PCs contribute even more strongly. When the top-m PCs have no contribution to the causal predictions, they are considered non-causal features. In other words, the removed component z - does not contain causal information. When the top-m. PCs contain causal information, should be selected such that causal information the top-7n PCs contributes less to the prediction compared to the PCs, ensuring that the trimmed representation 2 remains causally informative.
The proof provided here corresponds to this corrected version.
Proof. As the causal decision boundary Ap can classify z correctly, we have:
yz Ap >
0g
ThEe >
1=1
1=1
y/
ma e4) >
1=1
y
7ar%% >
d
7047 +y
Σ
7007% >
(12)
{=
By rearranging Equation (12), we can derive:
Y
7:007 > y
n007%
(13)
(=m-t
Using causal decision boundary to predict 2, the prediction is correct if and only ya Ap > which leads to:
0.
ya. Ap > 0
Σ
y
age
mike; >
1=m++
Σ
maamlelre) e2) >
imm
mar
(14)
Given Equation (13), Equation (14) is satisfied if any one of the following conditions holds:
23

---

y
Miayi > y Σ
7,07% 2 0
(15)
i=++1
i=1
y
Σ
> 0>
7/047/
(16)
1=M+1
1=1
Equation (15) leads to:
7 S 0
(17)
By adding y/ E 77447 to Equation (16), we can derive:
d
may >
(18)
2=1
A.3 Conditions for TACT to preserve correct prediction
Proposition 3. Suppose is correctly classified by the learned decision boundary Dg. The trimmed
representation & obtaiiedd via TACT will still be classified correctly if either of the conditions holds:
1. y(= -g < O, or
2. y(z 2g > 0, and Equation (7) holds, assuming already satisfies the Causal Preserva- tion condition (Proposition 2).
sign
Σ
1RO47
sign Σ
0477i
Equation (7) requires that when classification relies only on the representations formed by the remaining PCs, the learned decision boundary makes the same prediction as the causal decision boundary. Proposition 3 also shows that if correct prediction is made by the learned decision boundary, TACT will preserve this correctness long as the removed part : contributes negatively or does not contribute to the prediction. On the other hand, when the trimmed representation contains sufficient causal information as established in Proposition 2, the learned decision boundary is required to align directionally with the causal decision boundary defined by the remaining PCs.
The proof provided here corresponds to this corrected version.
Proof. As the learned decision boundary Aq classify z correctly, we have: yE Aq
y(> - 2) Aq + ya Aq >
(19)
We rewrite yē Aq as:
ya Ag y
age;
Σ
$=m+1
i=1
m Σ
d Σ
Σ ag4y
Yee +
746
imm+l
imm+l
Σ
y
ag4;
ager
imm+1
d
d Σ
=0 + y
OgE
7
i=m+1
\=m+1
= y.. A
(20)
24

---

By combining Equation (19) and Equation (20), we can derive:
y(= 2) 3q y2 Aq
(21)
The updated prediction by TACT is correct if and only if уē Ag > 0. Equation (21) shows that the value y( - 2). Aq needs to be considered to derive the conditions under which y Aq >
1. When y(z - 2) Aq S O, the removed part does not positively contribute to the prediction using the learned decision boundary, together with Equation (21), we can derive:
y² A > - 2) Aq 2 0
(22)
Equation (22) suggests that yı A > 0 always true when y(z 2) Aq S O.
2. When y(= 2) Aq > the removed part positively contributes to the prediction using the learned decision boundary. We wish to connect with the causal decision boundary understand the conditions. Therefore, we additionally assume & satisfies the Causal Preservation condition (Proposition 2), which suggests ya Ap 0.
The updated prediction is correct, i.e. y² Ag >> 0 if:
sign (y² Ap) = sign (y Aq)
d
d Σ
sign
Cg
sign
Σ
i=m+1
2=1
i=m+1
i=m++1
1
sign
y/
aamirilee . es)
sign
y
Σ
actile - ei)
i=++1
i=m+1
sign
y
amm
sign
y
Σ
am
i=m+1
i=m+1
d
d
sign
Σ
7:00%
= -iign
Σ
am
(23)
1m++
1=m++1
B Data Augmentation for CivilComments
CivilComments considers the following demographics mentioned in a comment: male. female, LGBTQ, Christian, Muslim, other religions, Black, White. We group the demographies into gender (male/female), sexuality (LGBTQ), religion (Christian/Mussim/oteee religions), and race (Black/White). We notice that the comments tend mention only one of the demographics in each group, and some comments mention more than one group. To vary demographics, propose to introduce new demographics to the comments.
We propose to randomly insert a sentence before or after the comment. The sentences being inserted sentences. Each sentence in the set mentions all demographics
are randomly drawn from a in one of the groups. The sentences are not toxic, so they would not affect the toxicity rating of the comment. Toxic comments remain toxic, and non-toxic comments remain non-toxic when the sentence is added. We ask ChatGPT via the web interface (https: / chattptt. com) to generate 20 sentences for each demographic group. We list the sentences below. Sentences from all groups make up the set from which we randomly sample for augmentation.
Gender(male/femaal))
"This is post about females and males."
"The discussion focuses women and men."
"Females and males are the central topic here."
"Women and men both contribute to this conversation."
25

---

"This explores perspectives of females and males."
"The post highlights contributions of women and men."
Bothh females and males are part of the narrative."
Women and men play essential roles this story."
Females and males are equally represented here."
"This covers aspects of both women and men."
"This is post about women and men."
"The discussion centers on ladies and gentlemen."
"Females and males are the key focus here." "
"Girls and boys both play significant roles."
Botth genders are part of this discussion."
"This highlights contributions from men and women."
"Ladies and gentlemen represented here equally."
"The focus is both sexes and their roles."
"Woennfll and menfolk shape this narrative."
"Both males and females are included in this topic."
Sexuality (LGBTQ)
"This is post about LGBTQ+ and heterosexual individuals."
"The discussion focuses sexual minorities and heterosexual communities."
"This highlights experiences of both LGBTQ+ and cisgender people."
"The post compares queer and non-queer perspectives."
"This covers topics relevant to both LGBTQ+ and straight groups."
Srender-diverss and cisgender voices are included this conversation."
"The focus is on LGBTQ+ and heterosexual rights and issues."
"Both sexual minorities and heterosexual people's experiences addressed here."
"This post examines the lives of gender-noneee nondoofoonnnn and cisgender individuals."
"The post explores the intersection of queer and non-queer identities."
"LGBTQ+ and heterosexual people both contribute to this topic."
"This content engages with both gender-divers and cisgender communities."
"The article offers insights into the experiences of L.GBT++ and non-L.GBTQ+ indi- viduals."
"This is a post about LGBTQ+ and heterosexual experiences in society."
"Both sexual minorities and heterosexual groups have a place in this discussion."
"This conversation includes both LGBTQ+ and cisgender perspectives."
"We explore issues affecting both sexual minorities and heterosexual individuals."
"This is about the relationships between LGBTQ+ and heterosexual people."
"The focus is creating unity between LGBTQ+ and cisgender communities."
"This post discusses challenges faced by both gender-divers, and cisgender people."
Religion (ChristianMMuslin/tther religions)
"This is a post about Christians, Muslims, and followers of other faiths."
"The discussion focuses Christians, Muslims, and practitioners of different reli- gions."
"This highlights the experiences of Christians, Muslims, and believers from various traditions."
"The post compares Christian, Muslim, and other spiritual practices."
"This covers topics relevant to Christians, Muslims, and people of other religious backgrounds."
"The voices of Christians, Muslims, and adherents of different faiths included in this conversation."
26

---

"The focus is Christian, Muslim, and interfaith perspectives."
Both Christians, Muslims, and people of other beliefs contribute to this discussion."
"This post examines the lives of Christians, Muslims, and followers of other religions."
"The post explores the intersection Christianity, Islam, and other spiritual practices." "Christians, Muslims, and people from diverse faiths share common values of compas- sion."
"This content engages with Christians, Muslims, and those from various religious traditions."
"The article offers insights into the teachings of Christians, Muslims, and other faith communities."
"This is a post about Christians, Muslims, and adherents of various world religions." "Both Christians, Muslims, and individuals from different belief systems included in this conversation."
"The focus is on how Christians, Muslims, and people of other religions practice faith."
"This conversation includes insights from Christians, Muslims, and followers of other spiritual paths."
"We'll explore issues affecting Christians, Muslims, and people from various religious backgrounds."
"This is about the relationships between Christians, Muslims, and those other beliefs." "
"The post discusses shared values between Christians, Muslims, and adherents of other religions."
Race (Black/White)
"This is post about Black and White communities."
"The discussion focuses African American and Caucasian experiences."
"This highlights the perspectives of Black and White individuals."
"The post compares the lives of Black and White people."
"This covers topics relevant to both Black and White races."
"The voices of African Americans and Caucasians are included in this conversation."
"The focus is Black and White racial dynamics."
"Both Black and White communities contribute this discussion."
"This post examines the experiences of Black and White individuals."
"The post explores the intersection of African American and European American identities."
"Black and White people play vital roles in shaping society."
"This content engages with the experiences of Black and White groups."
"The article offers insights into the lives of Black and White people in different settings."
"This is post about African American and White American experiences." "
"Both Black and White cultures have unique contributions to the world."
"The focus is on both Black and White perspectives in social issues."
"This conversation includes both Black and White voices."
"We'll 'll explore the relationship between Black and White individuals."
"This is about the interactions between African Americans and Caucasians."
"The post discusses challenges faced by both Black and White communities."
C Augmentation Design and Selection
Data augmentation requires careful consideration in order to achieve strong performance. It should heuristically maximize variations along non-causal directions and minimize variations along causal directions, so that the directions corresponding to non-causal features are well identified by Principal Component Analysis.
27

---

In practice, the augmentation can be treated as hyperparameter to search over. The data collection process that raises variation and features that affect the prediction target should be analyzed to propose set of augmentations that are semantically invariant with respect to the prediction target, yet introduce variability in other, non-causal aspects.
For example, for the commonly studied image classification task, we recommend searching over general image augmentations, such as AutoAugment [6] and RandomAugment [7]. These augmenta- tions preserve the critical causal features, particularly the shape information of objects [10]. while simultaneously injecting variability into less essential aspects. Our experiments examine the effect of different augmentation strategies on datasets where images serve as the predictive input. As shown in Table augmentation affects model performance, but AutoAugment and RandomAugment could provide consistent improvements over no adaptation.
The most effective way select the augmentation is to test a small subset of labeled test data.
Table 3: Performance of TACT with different augmentation strategies.
Augmentation
Birdcalls
Camelyon17'
ImageNet-R ImageNet-V2
no TTA
22.74
62.31
41.83
62.97
Stain color jitter/color jitter
1..1441.69
70.17:10.05
41.78.10.01
61.88.10.11
AutoAugment
22.22
72.04.10.12
43,29110.07
63.33.10.10
RandomAugment
32.1911.26
79.71100.07
43.59.10.02
62.99-10.10
D Details of Test-Time Adaptation Experiment
D.1 Model Used for Adaptation
For Birdcalls and Camelyon, to our knowledge, there were no publicly available ViT-B/32 models trained on the datasets. Therefore, we train a model using the standard empirical risk minimiza- tion. The training scripts and models can be found our code repository httpa://githen com/ NancyQuris/TACT The details of the training are:
Birdcalls uses batch size of 16 and is trained for 100 epochs. AdamW is employed as the optimizer, with a learning rate of 5e-5 and weight decay of 0.001. As specified [9], the training starts from weight pretrained on ImageNet, and the best model is selected by maero F1 on the in-distribution validation split.
Camelyon17 uses a batch size of 32 and is trained for 30 epochs. SGD is employed as the optimizer, with learning rate of 5e-5 and momentum 0.9, As instructed in [28], the training starts from a randomly initialized weight, and the best model is selected by the average classification accuracy the validation domain.
For CivilComments, we use the model provided by Wilds [28]. The model was trained on the training domain of CivilComments using empirical risk minimization. The model can be found in https: //worksheets. codalab. /ttp//////////////rrr/oooo/eee// contents/blobbbest. mwd_m.odellp.... pth.
For ImageNet-R and Im2 we use the model published by torchvision. The model was trained on ImageNet using empirical risk minimization. The pretrained weight ViT_B_32_____ Weights. IMAGENET1K_ V1 is loaded to the model for test-time adaptation.
D.2 Hyperparameter Search Space
We perform grid search to find the best hyperparameters for the baseline methods we compared with. For backpropagation-fe free methods, here list the details of the hyperparameters searched:
'The performance of AutoAugment and RandomAugment Camelyon17 is under removal principal components beginning with the 2nd. We observe that removing the first principal component only results in performance degradation. We hypothesize that important causal features might be present in the first principal component.
28

---

T3A: Following [19], .M, the number of representations stored to compute the centroid of each
class searched in {1,5,20.50, 100, N/A], where N/A means storing all representations.
LAME: Following [4], the is used in k-nearest neighbours is searched in {1.3,5), and the kernel to compute distance is searched in {KNN, linear, rbf}.
FOA: Following [36]. we use 3 prompts. The population size is set to 4 x log(prompt dim). The À to balance entropy and representation distance is searched in {0.2, 0.4).
For all backpropagation-baln based methods, we search the learning rate in {1e-3, le-4, le-5, le-6). The adaptation is performed in a non-episodic way. For other hyperparameters used in each method, the details are listed below:
SHOT: The method was originally proposed for source-free domain adaptation [30]. Following [19] that adapts as a TTA strategy, 3, the hyperparameter balance information maximization and cross entropy, is set to 0.1. The hyperparameter to filter confident pseudo-labels is set to 0.9. Adam is used the optimizer. The feature extractor is updated during adaptation. The adaptation step is set to 1.
Tent: Following [51 [51], SGD is used as the optimizer with momentum 0.9. The affine parameters of normalization layers are updated during adaptation. The adaptation step is set to 1.
SAR: Following [38], the margin Eg is set 0.4x In C. where C' is the number of classes. To recover the model, the moving average factor is set to 0.9, and the reset constant set to 0.2. SGD is used the base optimizer with sharpness aware minimization (SAM). The momentum for SGD is set to 0.9. P in SAM is set to 0.05. The affine parameters of shallow normalization layers are updated. Normalization layers in the 9*h-11 dh block in the feature extractor frozen during adaptation. The adaptation step is set to 1.
DeYO; Following [29], we search over the three augmentations {patch shuffling, pixel shuffling, occlusion] to destory causal features. The patch size in patch shufiling is set to 4. For occlusion, the occlusion size is set to [ (H/ x (W/2), where H and W stand for the height and width of the image. The occulsion starts from (H/4) th row and (W/4)th column. The Ye margin is set to 0.5 x In C, and the margin Fig is set to 0.4 C, where
the number of classes. The PLPD threshold is searched in (0.2, 0.3, 0.5}. SGD is used as the optimizer with momentum 0.9. The affine parameters of shallow normalization layers are updated. Normalization layers in the 1hh th block in the feature extractor are frozen during adaptation. The adaptation step set to
TAST: Following [21], we search the number of nearby support examples N, in {1. 4, 8). M, the number of support examples per class is searched in (1,5,20,50,10) 100, N/A). where N/A means storing all representations. The number of adaptation modules N. set to 20. Adam used as the optimizer. The trainable module added on top of the feature extractor is adapted. The adaptation step is searched in 31.
TSD: Following [53], the hyperparameter for feature filter M searched in {1, 20, 50, 100, N/A], where N/A denotes no entropy filter. The tradeoff parameter À to balance TSD loss and MSLC loss set to 0.1. Adam is used as the optimizer. Adapting {affine parameters, classifier, feature extractor, all parameters) } is searched. The adaptation step is set to 1.
PASLE: Following [17], we search the the threshold in {0.2. 0.4, 0.6, 0.8}. The threshold gap set to 1. The Tdes is searched in fle-3, le-4). The buffer size is
16, 1/4 of the batch size we used. Adam is used as the optimizer. Adapting (affine parameters, classifier, feature extractor, all parameters) is searched. The adaptation step is set 1.
D.3 Hardware and Software Used
We perform experiments on the NVIDIA V100 GPU with 32GB memory. When the batch size is set to 64, the memory of GPU is sufficient to perform test-time adaptation using TACT as well as all the baseline methods.
We implement TACT using PyTorch 2.1.2. Singular vector decomposition implemented by torch. www.llgggg... svd is used to compute the principal components, as it is computationally more stable than spectral decomposition. Since the covariance matrix is symmetric positive semi-definite matrix, the singular vectors are the same the eigenvectors.
29

---

E Additional Performance Study
E.1 TTA Performance on Larger Models
We examine TACT's effectiveness on larger models, specifically ViT-B/16 for images and BERT for texts. The experiment setup is consistent with that described in Section 6. Table 4 presents the performance of TACT and other state-of-the-ard architectures. Across all datasets except ImageNet-R, TACT achieves the best performance, ranking second on ImageNet-R. These results demonstrate the scalability of TACT to larger models.
backpropagation-fre
methods on the larger
The models for Birdcalls and Camelyon are trained under the same setting as that for ViT-B/32 stated in Appendix D.1. We follow the guidance of CivilComments' publisher to train BERT. The models we trained are included in our code repository. ViT-B/16 backbone for ImageNet-R and ImageNet-V. published by torchvision.
Table 4: Test-time adaptation performance of backpropagaation-fiee methods larger models. The best performance of each dataset bold.
Method
Birdcalls
Camelyon17
CivilComments
ImageNet-R
ImageNet-V2
No TTA
27.10
65.37
67.62
44.06
69.57
T3A
28.32=1.60
7277
67.46=0.00
43.99=0.08
69.67:10.04
LAME
27.48-1.44
6.15
67.65
44.04:10.04
69.59110.01
FOA
27.89-0.54 0.54
6.675
2.75
69.68
TACT
33.65 2.11
72.85 0.02
6..76:10.4
45.59:10.01
69.71
E,2 Synergy with Training-time Augmentation
The "no TTA" baselines of BirdCalls, Camelyon and CivilComments are trained without the augmentations used by TACT to identify and reduce non-causal features. To assess TACT's synergy with training-tine time augmentation, we trained models using the same augmentations as those applied by TACT and then performed test-time adaptation. For ImageNet-R and ImageNet-V2, the "no TTA" baseline provided by torchvision was trained with AutoAugment using the ImageNet policy.
Table 5 shows the test-time adaptation performance of TACT on models trained with the same augmentation strategy. The results show that, even when models are trained with these augmentations, TACT further improves test-time performance. This highlights TACT's ability to synergize with training-time augmentation and provides strong evidence of its effectiveness and generalizability.
Table 5: Test-time adaptation performance of TACT with training-time augmentation models.
Birdcalls
Camelyon17
CivilComments
ImageNct-R
ImageNet-V V2
no TTA (train time aug)
29.86 30.57=20.96
74.09 77.27=0.03
64.60 68.84+0020
41.83
62.97
+ TACT
43.29-:0.07
63.33:t0.10
E.3 TTA Performance under Different Batch Size
We study the test-time adaptation performance of TACT on ImageNet-R when the test batch size varies. Table 6 shows the result when the test batch size is set to 1, 4, 16, 64 and 128, respectively. The performance remains stable across different batch sizes. Even with a batch size of 1, the performance only decreases by 0.06% compared to a batch size of 64. Moreover, TACT still improves performance by 1.7% over the no-adaptation baseline when only one sample is available per batch during adaptation. The result suggests that TACT is robust to variations in batch size, maintaining high performance even when batch sizes are small. This makes it well-suited for situations where the number of test samples per batch is constrained.
30

---

Table Test-time adaptation performance (%)
TACT on ImageNet-R under different batch sizes.
no TTA
batch size
batch size =
batch size 16
batch size = 64
batch size 128
41.83
43.53:10.02
43.51
43.55=00.06
43.59:0.02
43.56=0.03
E.4
Computational Cust
We compared the computational requirements of TACT with those other backpropagation-frn methods the Birdcalls dataset using ViT-b/32 backbone. As shown in Table 7, TÀCT incurs higher time and GPU memory consumption relative to alternative approaches. Nevertheless, this additional computational cost results in substantial performance gains (Table 1), which justifies the trade-off. Future work may explore optimization strategies, such as more efficient eigendecomposition techniques for PCA, reduce the overhead.
Table 7: Time and GPU memory required by backpropagaation-fee free methods on Bridcalls.
time (second)
GPU memory (MB)
T3A
7.67
667.42
LAME
7.34
667.42
FOA
16.83
667.42
TACT (num aug=128)
112.22
1750.21
TACT (num aug=256)
170.00
2966.21
TACT (num aug=512)
323.62
5398.21
F,5 Additional Visualization of Predictions after Causal Trimming
We provide more GradCAM visualization of the original predictions and the predictions made by TACT on samples from ImageNet-R. Figure 5 shows the visualizations.
Compared to original predictions, predictions made by TACT focus less non-causal information. For example, TACT pays less attention to the background of the warplane example, and the blowfish example. The focus on the information that is semantically correlated with the class retained in predictions made by TACT in the above examples. When the causal information is not important to the original prediction, prediction made by TACT leverages the causal information and thus turn the wrong prediction correct, as shown in the example of jellyfish and bloodhound.
Input
GradCAM
TACT-GradCAM
Input
GradCAM
TACT-GradCAM
ground Iruth: warplane
ground truth: jellylish; fish; prediction: ant
ground truth: blowfish
ground tuuh: Weimaraner; prediction: hlondhaund (b) wrong predictions corrected by TACT
(a) correct predictions
Figure 5: Additional GradCAM visualizations of the original predictions and TACT's predictions.
31

---

Alternative Design of TACT
E1 ICA to Find Non-Causal Directions
We experiment using an alternative direction finding method, Independent Component Analysis (ICA) with TACT. We rank the independent components by the variance of the scalars of features on the components, We remove the top independent components that have maximum variance. Table shows the result on the Birdcalls dataset. ICA performs inferior to Principal Component Analysis (PCA). but better than no adaptation. Although ICA overcomes the orthogonality constraints of PCA, only looks for statistically independent components and assumes each component follows non-Gaussian distribution. Causal and non-causal features might not follow the non-Gaussian distribution assumption under augmentations that vary non-causal features.
Table 8: Performance of TACT with ICA to find non-causal directions.
no TTA
TACT w/ PCA
TACT w/ ICA
22.74
31.14+1.69
25.53+1.06
F.2 Causal Trimming Based on a Threshold
We consider using the variance that the top principal components (PC) account for as threshold to decide whether causal trimming is conducted or not. When the augmentation only changes non-causal features and causal features remain unchanged, datapoints that are invariant augmentations should have smaller variance of the top PCs. Thus, if the variance is smaller than a threshold, causal trimmings will not be conducted on the data. As the range variance is not known and it could change significantly, setting numerical threshold might not be feasible. We consider normalized variance, where we divide the variance of top PCs by the sum of variances of all PCs. Table shows the result the Birdcalls dataset. Removing components based on a threshold does not outperform using no threshold.
Table 9: Performance of TACT when causal trimming is performed based on a threshold т.
no TTA
TACT
TACT (7=0.1)
TACT (r=0.2)
TACT (r=0.3)
22.74
31.14+1.69
30.99-2.18
31.03+22.19
28.03++3.12
32


Reviews:
[{"review_id": "bXX012xt0g", "reviewer": "Reviewer_ptsx", "summary": "This paper actively reduces non-causal features in TTA. To identify non-causal features, the authors analyzed how these representations change when applying targeted perturbations to the input data.", "questions": "How to handle the limitations mentioned in the above weakness part?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strenghts:\n\n1. The paper proposes a novel causal trimming approach to remove non-causal features from representations, improving model robustness under distribution shifts.\n\n2. This idea is novel in TTA research field. The theoretical analysis is solid.\n\n* Weakness:\n\n1. Designing data augmentations that preserve causal features while perturbing non-causal ones requires domain knowledge. Consequently, this limits the method’s generalizability and practical deployment in domains where such knowledge is unavailable or hard to encode.\n\n2. The identification of non-causal components is based on PCA, which assumes linearity and orthogonality of the principal directions. This may fail in practice.", "was_revised": true, "final_justification": "I appreciate the authors' detailed response. My concern has been addressed.  I will raise the score.", "rebuttal": "Thank you for the thoughtful feedback and for recognizing our novelty and theoretical analysis. \nThe limitations raised in the review were acknowledged in section 7 of the main paper. We discuss the two limitations in more detail below. \n\n**1. Domain knowledge to design augmentation**\n\nWe agree that requiring prior knowledge of the data is a limitation of the proposed method. \nUnder the setting of TTA, where only a trained model and test data are given, in the absence of any task-specific or domain knowledge, it is inherently difficult—if not impossible—to distinguish causal from non-causal features [1, 2]. \n\nData augmentation is one approach to identifying non-causal features. In practice, when insufficient domain knowledge is given and the optimal augmentation strategies are unknown, the augmentations can be treated as a hyperparameter to search over. \nSpecifically, our basic assumption is that, given a dataset and a task,  we at least have the knowledge about the prediction target. Based on this, we can design augmentations that are semantically invariant with respect to the prediction target.\nFor example, for the commonly studied image classification task, we recommend searching over general image augmentations, such as AutoAugment and RandomAugment. These augmentations preserve the key causal features, particularly the shape information of objects [3].  Our experiments in the main paper of using them for ImageNet-R and ImageNet-V2 show state-of-the-art performance. \nBirdcall's audio clip is converted to Mel spectrogram, which is treated as image as well.\nOur experiment of using AutoAugment and RandomAugment with Birdcalls and Camelyon show that they can also offer performance gain compared to no adaptation. \n| Augmentation                    | Birdcalls      | Camelyon17     |\n|---------------------------------|----------------|----------------|\n| no TTA                          | 22.74          | 62.31          |\n| Stain color jitter/color jitter | 29.99$\\pm$0.80 | 70.17$\\pm$0.05 |\n| AutoAugment                     | 27.45$\\pm$1.28 | 66.42$\\pm$0.01 |\n| RandomAugment                   | 31.76$\\pm$0.19 | 65.60$\\pm$0.08 |\n\nThe paper wants to highlight that reducing non-causal features by adjusting the learned representations is helpful for TTA.  Alternative to data augmentation, if other information, such as a small set of labeled data from multiple distributions, is given, non-causal features can also be identified by comparing the representations of data with the same labels. \n\n**2. Linearity and orthogonality of the principal directions**\n\nWe use a simplified representation space that satisfies the linearity and orthogonality assumptions of causal and non-causal features to approximate the learned representation space, so that a practical method could be developed. \nNonetheless, the theoretical analysis in section 5 does not rely on the assumption.\nWe show that even when causal and non-causal features are not orthogonal, and causal features are partially trimmed, \nTACT performs well as long as top PCs alone cause the prediction error, or sufficient causal information is preserved after trimming, and the learned decision boundary is directionally aligned with the causal decision boundary. \nEmpirically, the models used for TTA in section 6 do not have their representation space explicitly enforced to satisfy the assumption. \n\n[1] Wang et al., Sound and Complete Causal Identification with Latent Variables Given Local Background Knowledge, NeurIPS 2022.\n\n[2] Schölkopf  et al., Toward causal representation learning, Proceedings of the IEEE 2021.\n\n[3] Geirhos et al., ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, ICLR 2019."}, {"review_id": "SeBgQAu0LR", "reviewer": "Reviewer_5gmS", "summary": "This paper introduces TACT, a backpropagation-free test-time adaptation (TTA) method that actively identifies and removes non-causal components from feature representations to improve robustness under distribution shift.\nThe authors provide theoretical analysis (Propositions 1–3) that formalizes when trimming corrects misclassifications and preserves causal information. Empirically, TACT is evaluated on five real-world out-of-distribution benchmarks—Birdcalls, Camelyon17, CivilComments, ImageNet-R, and ImageNet-V2—showing consistent gains over state-of-the-art backprop-free methods, and further improvements when combined with gradient-based adaptation (TACT-adapt)", "questions": "- In Section 4.1 (Non-Causal Identification), do you project out only the first principal component, or do you remove m components—and if so, how is m chosen for each batch？\n\n- In Table 1, some reported standard deviations for TACT-adapt are quite large (e.g., Birdcalls ±5.68). Could you report standard deviations for all datasets and perform statistical significance tests against the strongest baselines?\n\n- For Figure 4’s hyperparameter study, would it be possible to present heatmaps of performance over various  (n,m) combinations to visualize their interaction and identify robust operating regions?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n\n- Unlike entropy-based or self-training TTA, TACT directly suppresses non-causal features via linear trimming, addressing the root cause of distribution sensitivity.\n\n- By avoiding gradient updates during adaptation, TACT incurs only a single PCA and a few vector operations per batch, making it attractive for latency-sensitive deployment.\n\n- Propositions 1–3 rigorously characterize conditions for error correction, causal preservation, and non-degradation of already-correct predictions, offering clear guidance on choosing the number of components m to trim.\n\nWeaknesses\n\n- TACT requires domain knowledge to choose augmentations that vary only non-causal factors. An empirical study of sensitivity to augmentation choice, or an automated strategy for augmentation selection, would enhance practicality.\n\n- PCA assumes that causal directions are orthogonal to non-causal ones. In complex tasks where causal and spurious features co-vary, this may fail. Exploring non-linear direction-finding (e.g., ICA or contrastive subspace methods) could address this limitation.", "was_revised": true, "final_justification": "The authors address scalability within their tested range, but the method’s substantially higher latency compared to other approaches may limit its applicability in low-latency settings. While further PCA optimizations are suggested, they are left as future work. Given these practical constraints, I would recommend borderline acceptance.", "rebuttal": "Thank you for the detailed feedback and for acknowledging that TACT is a method that addresses the root cause of distribution shift backed by theoretical analysis. \nThe weaknesses and questions raised in the review are discussed below.\n\n### Weakness\n\n**W1.1. Augmentation sensitivity analysis**\n\nThe table below shows the result. The choice of augmentation would influence performance, but we see that general augmentations such as AutoAugment and RandomAugment could provide improvements over no adaptation across datasets.  To our surprise, Birdcalls even performs better with RandomAugment than the existing color jitter.\n\n| Augmentation                    | Birdcalls      | Camelyon17     | ImageNet-R     | ImageNet-V2    |\n|---------------------------------|----------------|----------------|----------------|----------------|\n| no TTA                          | 22.74          | 62.31          | 41.83          | 62.97          |\n| Stain color jitter/color jitter | 29.99$\\pm$0.80 | 70.17$\\pm$0.05 | 41.78$\\pm$0.01  | 61.88$\\pm$0.11 |\n| AutoAugment                     | 27.45$\\pm$1.28 | 66.42$\\pm$0.01 | 43.29$\\pm$0.07 | 63.33$\\pm$0.10 |\n| RandomAugment                   | 31.76$\\pm$0.19 | 65.60$\\pm$0.08 | 43.59$\\pm$0.02 | 62.99$\\pm$0.10 |\n\n**W1.2. Automated strategy to select augmentation**\n\nIn practice, the augmentation can be treated as a hyperparameter to search over.\nThe data collection process that raises variation and features that affect the prediction target should be analyzed to propose a set of augmentations that are semantically invariant with respect to the prediction target, yet introduce variability in other, non-causal aspects.\nFor example, for the commonly studied image classification task, we recommend searching over general image augmentations, such as AutoAugment and RandomAugment, that preserve the key causal features, particularly the shape information of objects [1].\nThe most effective way to select the augmentation is to test on a small subset of labeled test data.\n\n**W2.1. Orthogonality and co-varying of causal and spurious features**\n\nTACT does not rely on perfect orthogonality or disentanglement. \nThe theoretical analysis in section 5 does not rely on the orthogonality assumption.\nWe show that even when causal and non-causal features are not orthogonal, and causal features are partially trimmed,\nTACT performs well as long as top PCs alone cause the prediction error, or sufficient causal information is preserved after trimming, and the learned decision boundary is directionally aligned with the causal decision boundary. \nEmpirically, in Section 6, we show that even approximate separation achieved by PCA was sufficient to yield gains in performance and robustness across diverse datasets.\n\nWhile it is true that causal and spurious features can co-vary in the training data, it is important to note that when we apply augmentations at test time, we are trying to minimize the variation of causal features, so that the influence of non-causal features can be isolated. \n\n**W2.2. Alternative direction finding methods**\n\nWe experiment with using ICA with TACT on Birdcalls. \nWe rank the independent components by the variance of the scalars of features on the components. We remove the top independent components that have maximum variance. \nThe experiment with ICA is searched over the same number of augmentations and number of components. \nThe table below shows the result.\nICA performs inferior to PCA, but better than no adaptation.\nWe agree that ICA can overcome the orthogonality assumption. However, ICA only looks for statistically independent components and assumes each component follows a non-Gaussian distribution. Thus, ICA might not be better at identifying causal and non-causal features, which might not follow the non-Gaussian distribution assumption under augmentations that vary non-causal features. \n\n|           | no TTA | TACT w/ PCA             | TACT w/ ICA    |\n|-----------|--------|-------------------------|----------------|\n| Birdcalls | 22.74  | **29.99$\\pm$0.80** | 25.32$\\pm$0.35 |\n\n### Questions \n\n**Q1. Number of PCs to project out**\n\nWe explain the projection in section 4.2 (Causal Trimming to Reduce Non-Causal Feature). The top-m PCs are removed. We maintain the same m for a dataset across batches.  $m$ is a hyperparameter to be selected. \nWe empirically observe that setting $m=1$, i.e., removing the top PC is frequently adequate. For datasets like ImageNet-R with more complex or layered distribution shifts, removing more PCs could further boost the performance.\n\n**Q2. Statistical significance test**\n\nTo clarify, all experiments are repeated three times, and we report the mean and standard deviation in Table 1.\nWe use the t-test to compute the statistical significance between the results of TACT-adapt and the strongest baselines. The p-value shows how likely the results of the two methods come from the same distribution. The table below shows the result. \nCamelyon, CivilComments, and ImageNet-V2 have a p-value smaller than 0.15.\n\n|         | Birdcalls | Camelyon17 | CivilComments | ImageNet-R | ImageNet-V2 |\n|---------|-----------|------------|---------------|------------|-------------|\n| p-value | 0.7487    | 0.1436     | 0.0000        | 0.5286     | 0.0572      |\n\nWe acknowledge that the improvement on Birdcalls and ImageNet-R does not seem statistically significant, which is likely due to the small number of runs.  Therefore, we performed two additional runs to compute the result. The table below shows the updated result. P-values for the two datasets are improved. We plan to conduct more runs and update the results in the revision.\n\n|            | TACT           | Best baseline  | p-value |\n|------------|----------------|----------------|---------|\n| BirdCalls  | 30.61$\\pm$5.40 | 27.94$\\pm$2.19 | 0.3125  |\n| ImageNet-R | 48.84$\\pm$0.14 | 48.83$\\pm$0.18 | 0.4557  |\n\n**Q3. Hyperparameter and performance**\n\nFigure 4 shows the performance over various $(n, m)$ combinations. \nThe colored lines denote the number of augmentations, highlighting the role of $n$, while the x-axis represents $m$, the number of removed PCs. Performance values (y-axis) are determined by the intersection of each line with the respective m value.\nEmpirically, we find that values of $n \\in$ {128, 256, 512} provide sufficient performance. \nFor the number of removed principal components $m$, we observe that removing the top principal component, which typically captures the dominant non-causal variation, is frequently adequate. \n\n[1] Geirhos et al., ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, ICLR 2019."}, {"review_id": "ANmcPIyThN", "reviewer": "Reviewer_QQc2", "summary": "This paper proposes TACT (Test-time Adaptation by Causal Trimming), a novel backpropagation-free method for test-time adaptation (TTA) under distribution shifts. The key insight is that performance degradation often stems from reliance on non-causal features. TACT addresses this by identifying and removing these non-causal components in the representation space using PCA on augmentations that vary non-causal attributes while preserving causal semantics.\n\nSpecifically, TACT:\n\n* Uses targeted data augmentations to induce variability in non-causal dimensions.\n\n* Applies PCA to extract directions of maximum variance (assumed to align with non-causal features).\n\n* Removes projections onto these principal components (PCs) from both the sample representation and class prototypes.\n\n* Uses a moving average over the trimmed prototypes for stable prediction.\n\n* Provides theoretical guarantees for when causal trimming helps or preserves performance.\n\n* Demonstrates strong empirical results across five diverse datasets, showing consistent improvements over state-of-the-art (SOTA) backpropagation-free and backpropagation-based TTA methods.", "questions": "* Your method assumes that causal features remain invariant under augmentations while non-causal ones vary. How robust is TACT when this assumption is only approximately satisfied?\n\n* PCA assumes that variance-aligned directions are linearly separable. But in high-capacity models like ViT or BERT, representation entanglement is common. How sensitive is TACT to this? Have you considered using nonlinear alternatives (e.g., kernel PCA or ICA)?\n\n* The number of removed PCs (m) appears to significantly influence performance (Figure 4). Can you elaborate on how this is tuned in a realistic, non-oracle setting?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Quality\n### Strengths:\n\n* Solid methodological foundation combining causal assumptions with representational geometry.\n\n* Clear theoretical propositions (with assumptions) analyzing when and why trimming helps.\n\n* Sound and comprehensive experimental protocol using five real-world datasets with meaningful shifts.\n\n* Ablation studies and Grad-CAM visualizations strongly support claims.\n\n### Weaknesses:\n\n* Performance gains rely heavily on carefully selected augmentations that preserve causal content, which may not generalize across tasks or be feasible without domain expertise.\n\n* PCA-based trimming assumes non-causal features dominate variance directions, which might not always hold, especially under entangled representation spaces.\n\n## Clarity\n### Strengths:\n\n* The writing is clear, structured, and technically precise.\n\n* Figures (especially Grad-CAM and ablations) help illustrate key points.\n\n* Theoretical section is accessible and intuitive despite formal proofs being deferred to appendix.\n\n### Weaknesses:\n\n* The role of the hyperparameter m (number of trimmed PCs) is critical but tuning guidance is heuristic and empirical.\n\n* The limitations section could more thoroughly discuss failure cases (e.g., when causal features are not orthogonal to spurious ones).\n\n## Significance\n### Strengths:\n\n* Offers a promising new direction in test-time adaptation that is backpropagation-free yet effective.\n\n* Bridges causal reasoning with representation learning and adaptation  a direction of growing interest. \n\n\n## Originality\n### Strengths:\n\n* The idea of causal trimming based on PCA over augmentations is novel in TTA.\n\n* Synthesizes ideas from disentanglement, causality, and test-time learning in a unique way.\n\n* Extends beyond label-free entropy minimization or pseudo-label refinement by directly adjusting the representation space.\n\n### Weaknesses:\n\nSome ideas, such as augmenting data to identify spurious directions, have been explored in domain generalization and DeYO. However, TACT’s trimming strategy is distinct.", "was_revised": true, "final_justification": "The rebuttal satisfactorily addresses key concerns, showing that generic augmentations (e.g., AutoAugment, RandomAugment) work across datasets and that TACT remains effective under approximate disentanglement and with large models. The PCA reliance is acknowledged, with empirical results suggesting robustness despite potential entanglement. Guidance on choosing *m* is provided, though still heuristic. Overall, the method is novel, well-supported theoretically and empirically, and offers a meaningful advance in backpropagation-free TTA.", "rebuttal": "Thank you for the comprehensive feedback and for praising the proposed method as well as our empirical and theoretical analysis. A detailed discussion of the concerns and questions is presented below.  \n### Weakness\n\n**Quality-W1. Reliance on augmentation and generalizability**\n\nWe agree that augmentations need to be carefully selected to achieve good performance.\nIn practice, the augmentation can be treated as a hyperparameter to search over.\nOur basic assumption is that knowledge about the prediction target is known, based on which augmentations that are semantically invariant with respect to the prediction target can be designed. \nFor example, for the commonly studied image classification task, we recommend searching over general image augmentations, such as AutoAugment and RandomAugment. These augmentations preserve the key causal features, particularly the shape information of objects [1].  Our experiments in the main paper of using them with ImageNet-R and ImageNet-V2 show state-of-the-art performance. \nOur experiment of using AutoAugment and RandomAugment with Birdcalls and Camelyon shows that they can also offer performance gain compared to no adaptation, suggesting the potential generalization ability of TACT.  \n\n| Augmentation                    | Birdcalls               | Camelyon17              |\n|---------------------------------|-------------------------|-------------------------|\n| no TTA                          | 22.74                   | 62.31                   |\n| Stain color jitter/color jitter | 29.99$\\pm$0.80          | 70.17$\\pm$0.05 |\n| AutoAugment                     | 27.45$\\pm$1.28          | 66.42$\\pm$0.01          |\n| RandomAugment                   | 31.76$\\pm$0.19 | 65.60$\\pm$0.08          |\n\n**Quality-W2. Non-causal features dominate variance directions and entangled representation space**\n\nWe select data augmentation strategies to heuristically maximize variations along non-causal directions and minimize variations along causal directions, which leads to improved disentanglement after applying PCA. Therefore, it is critical to select the augmentations carefully.\n\nTheoretically, TACT does not rely on perfect disentanglement. \nWe show that when causal features are partially trimmed, \nif sufficient causal information is preserved and the learned decision boundary is directionally aligned with the causal decision boundary, TACT would perform well. \nEmpirically, in Section 6, we show that even approximate separation of causal and non-causal features achieved by PCA was sufficient to yield gains in performance and robustness across diverse datasets.\n\n**Clarity-W1. Tuning guidance of number of trimmed PCs**\n\nWhen the oracle is not accessible, m in general can be set to 1.\n\n**Clarity-W2. Failure cases**\n\nWhen causal features and spurious features are not orthogonal, removing spurious features will affect the remaining causal features, which likely also degrade performance. In the extreme, feature removal will not work if spurious features are perfectly correlated with causal features in the training data. This also corresponds to our theoretical analysis where sufficient causal features need to be preserved to make correct predictions.\n\n### Questions \n\n**Q1. Robustness when the assumption is only approximately satisfied**\n\nTheoretically, Propositions 2 and 3 suggest that when causal features vary, resulting in the trimming of causal features, sufficient causal information needs to be preserved for TACT to perform well. \n\nEmpirically, in our experiment with ImageNet-R and ImageNet-V2, we only tried to maintain the key causal features, which are object structure and shape. Other causal features that could be helpful in inferring objects, such as color when inferring strawberries, are altered. \nThis experiment shows that TACT is still able to improve performance when the assumption is only approximately satisfied. \n\n**Q2.1. Performance on larger models when representation entanglement is common**\n\nThe table below shows the performance with larger backbones, including ViT-B/16 for images and BERT for texts. TACT achieves the best performance compared to other backpropagation-free methods on all datasets except ImageNet-R. TACT achieves the second best on ImageNet-R.\nThe result suggests that TACT can be robust to changes in the representation space.\n\n| Method | Birdcalls               | Camelyon17              | CivilComments           | ImageNet-R              | ImageNet-V2             |\n|--------|-------------------------|-------------------------|-------------------------|-------------------------|-------------------------|\n| No TTA | 27.10                   | 65.37                   | 67.62                   | 44.06                   | 69.57                   |\n| T3A    | 27.50$\\pm$0.27          | 72.72$\\pm$0.73          | 67.46$\\pm$0.00          | 43.99$\\pm$0.08          | 69.67$\\pm$0.04          |\n| LAME   | 28.11$\\pm$1.38          | 68.50$\\pm$0.11          | 67.65$\\pm$0.04          | 44.04$\\pm$0.04          | 69.59$\\pm$0.01          |\n| FOA    | 28.01$\\pm$0.26          | 67.15$\\pm$0.67          | -                       | **47.53$\\pm$2.73** | 69.68$\\pm$0.04          |\n| TACT   | **35.00$\\pm$2.49** | **72.85$\\pm$0.02** | **69.76$\\pm$0.44** | 45.59$\\pm$0.01          | **69.71$\\pm$0.02** |\n\nThe models for Birdcalls and Camelyon are trained under the same setting as that for ViT-B/32 used in the paper. We follow the guidance of CivilComments' publisher to train BERT. ViT-B/16 backbone for ImageNet-R and ImageNet-V2 is published by torchvision.\n\n**Q2.2. Alternative direction finding methods**\n\nAlthough kernelPCA and ICA overcome the limitations of PCA, they incur additional requirements. \n- kernelPCA transforms the representation space via a kernel, which needs to be selected such that causal and non-causal information are retained and better separated. \nDue to this complexity, we encourage further exploration on this in future works.\n- ICA looks for statistically independent components and assumes each component follows a non-Gaussian distribution.\nCausal and non-causal features might not follow the non-Gaussian distribution assumption under augmentations that vary non-causal features. \n\n\nWe experiment with using ICA with TACT on Birdcalls. \nWe rank the independent components by the variance of the scalars of features on the components. We remove the top independent components that have maximum variance. \nThe experiment with ICA is searched over the same number of augmentations and number of components. \nThe table below shows the result.\nICA performs inferior to PCA, but better than no adaptation.\n\n|           | no TTA | TACT w/ PCA             | TACT w/ ICA    |\n|-----------|--------|-------------------------|----------------|\n| Birdcalls | 22.74  | **29.99$\\pm$0.80** | 25.32$\\pm$0.35 |\n\n**Q3. Selection of the number of removed PCs**\n\nPlease refer to Clarity-W1. \n\n[1] Geirhos et al., ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, ICLR 2019."}, {"review_id": "eO1RczetOi", "reviewer": "Reviewer_DW91", "summary": "The paper proposes a method for test-time adaptation by removing the non-causal components from intermediate representations. At test-time augmentations are applied and PCA is used to remove the components with the highest variance. Experiments were performed on a range of datasets, and supportive results were shown compared to the baselines.", "questions": "- Was the “no TTA” baseline trained with the same augmentations used for TACT?\n- What would be the performance of the method if the hyperparameters were tuned on validation data instead?\n- Have the authors tried removing components based on a threshold instead? If the datapoint is invariant to the augmentation, will this remove the “causal” components too?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper was well written and easy to follow.\n- The experiments were done on a diverse datasets and domains (from audio signals, text to images) with supportive results.\n\n**Weaknesses**\n\n- The selected augmentations seems to assume some knowledge of about the distribution shift, i.e., they are selected to be similar to the shift for each dataset. Was the no TTA baseline trained with these augmentations? If not, It would be useful to see if learning invariances via training time augmentation alone is sufficient.\n- Performance of the method in practice. The paper mentions that oracle selection to used to select the hyperparameters (nb of augmentations, components), allowing for fair comparison across methods. While this can give a fair comparison, it makes it hard to understand the performance of the method in practice. From Fig. 4, there can be large differences in performance.\n- Computation cost. Since the method performs PCA, it can be more expensive that the baselines. Including the computational cost of each method make it easy to assess the trade-offs and make a more informed decision for time sensitive applications.", "was_revised": true, "final_justification": "I appreciate the authors rebuttal and the additional results, they were insightful. However, my concern about the choice of augmentation stills stands. While they work better than no TTA, the standard augmentations that works the best for one dataset, results in the worst performance for another dataset. Thus, I will be keeping my score.", "rebuttal": "Thank you for the extensive feedback and for recognizing our empirical analysis. \nWe discuss the weakness and questions raised in the review below.\n\n### Weakness\n\n**W1. \"no TTA'' baseline and training time augmentation**\n\nThe \"no TTA'' baseline of BirdCalls, Camelyon17, and CivilComments were not trained with the augmentations used by TACT to identify and reduce non-causal features. \nFollowing the instructions of the data publisher, no data augmentation was used when training the baseline with empirical risk minimization. \nTo understand the effectiveness of training time augmentation, we trained the models with the data augmentations used by TACT. The result is shown in the table below. \nTraining time augmentation is not always sufficient, as it achieves worse performance than TACT on BirdCalls and Camelyon17. \nWe further show that, when training time augmentation is performed, TACT can still improve the performance. \nThese results suggest TACT's usefulness when tackling distribution shift. \n\n|                         | Birdcalls      | Camelyon17     | CivilComments  |\n|-------------------------|----------------|----------------|----------------|\n| no TTA                  | 22.74          | 62.31          | 55.38          |\n| + TACT                  | 29.99$\\pm$0.80 | 70.17$\\pm$0.05 | 71.80$\\pm$0.35 |\n| no TTA w/train time aug | 29.86          | 74.09          | 64.60          |\n| + TACT                  | 30.30$\\pm$0.27 | 77.27$\\pm$0.03 | 68.84$\\pm$0.20 |\n\nFor ImageNet-R and ImageNet-V2, the \"no TTA'' baseline provided by torchvision was trained with AutoAugment using the ImageNet policy. The table below shows the performance of TACT on ImageNet-R and ImageNet-V2 with the same augmentation. TACT improves the performance.\n\n|             | ImageNet-R     | ImageNet-V2    |\n|-------------|----------------|----------------|\n| no TTA      | 41.83          | 62.97          |\n| AutoAugment | 43.29$\\pm$0.07 | 63.33$\\pm$0.10 |\n\n**W2.  Performance selected by validation data**\n\nFor Birdcalls, Camelyon, and CivilComments, we use the validation domain for model selection. \nFor ImageNet-R and ImageNet-V2, we use the validation set of ImageNet for model selection. \nThe performance is shown in the table below. \nTACT performs the best on all datasets except for Birdcalls. For Birdcalls, TACT performs the second best.\n\n| Method | Birdcalls               | Camelyon17              | CivilComments           | ImageNet-R              | ImageNet-V2             |\n|--------|-------------------------|-------------------------|-------------------------|-------------------------|-------------------------|\n| No TTA | 22.74                   | 62.31                   | 55.38                   | 41.83                   | 62.97                   |\n| T3A    | 23.71$\\pm$1.74          | 68.54$\\pm$0.06          | 56.43$\\pm$0.00          | 41.66$\\pm$0.04          | 62.93$\\pm$0.02          |\n| LAME   | 22.81$\\pm$0.71          | 62.38$\\pm$0.03          | 56.00$\\pm$0.10          | 41.71$\\pm$0.04          | 63.00$\\pm$0.02          |\n| FOA    | **26.72$\\pm$0.70** | 58.36$\\pm$0.77          | -                       | 41.46$\\pm$0.16          | 62.75$\\pm$0.09          |\n| TACT   | 25.22$\\pm$1.19          | **70.17$\\pm$0.05** | **71.33$\\pm$0.36** | **42.10$\\pm$0.04** | **63.02$\\pm$0.12** |\n\n**W3. Computation cost**\n\nWe compared the computation cost of TACT with other backpropagation-free methods on Birdcalls. The table below shows that TACT requires more time and memory than other backpropagation-free methods. \nHowever, this increase in computational cost leads to a significant improvement in performance shown in Table 1 of the main paper, which we believe justifies the trade-off. \nIn many practical applications, the improved performance is critical and outweighs the added cost. \nFurther optimization techniques on efficient eigendecomposition for PCA could be applied to reduce the overhead if needed, which we leave as future work.\n\n| Method | time (second) | GPU memory (MB) |\n|--------------------|---------------|-----------------|\n| T3A                | 7.67          | 667.42          |\n| LAME               | 7.34          | 667.42          |\n| FOA                | 16.83         | 667.42          |\n| TACT (num aug=128) | 112.22        | 1750.21         |\n| TACT (num aug=256) | 170.00        | 2966.21         |\n| TACT (num aug=512) | 323.62        | 5398.21         |\n\n### Questions \n**Q1. ''no TTA\" baseline**\n\nPlease refer to W1. \n\n**Q2. Performance selected by validation data**\n\nPlease refer to W2. \n\n**Q3. Removing components based on a threshold**\n\nWe consider using the variance that the top PC accounts for as a threshold ($\\tau$) to decide whether causal trimming is conducted or not. \nWhen the augmentation only changes non-causal features and causal features remain unchanged, datapoints that are invariant to augmentations should have smaller variance of the top PCs.\nThus, if the variance is smaller than a threshold, causal trimmings will not be conducted on the data. \nAs the range of variance is not known and it could change significantly, setting a numerical threshold might not be feasible. \nWe consider normalized variance, where we divide the variance of top PCs by the sum of variances of all PCs. \n\nThe table below shows  the performance. Current experiments on Birdcall with threhsold does not improve the performance. \nFuture work can study this issue in more details.\n\n|           | no TTA | TACT           | TACT ($\\tau$=0.1) | TACT ($\\tau$=0.2) | TACT ($\\tau$=0.3) |\n|-----------|--------|----------------|-------------------|-------------------|-------------------|\n| Birdcalls | 22.74  | 29.99$\\pm$0.80 | 28.95$\\pm$0.92    | 29.03$\\pm$1.17    | 25.69$\\pm$1.57    |"}]

Author rebuttal:

Meta-review: This paper proposes a backpropagation-free test-time adaptation method that mitigates distribution shift. Theoretical analysis provides conditions for error correction and causal preservation, while experiments across five datasets (image, text, audio) demonstrate consistent gains over state-of-the-art TTA methods.

The work introduces a principled and broadly applicable approach to TTA, bridging causality and representation learning. Despite practical challenges, the method’s novelty, theoretical rigor, and strong empirical validation justify acceptance.

During the rebuttal. authors addressed concerns by showing robustness to approximate assumptions, generalization of augmentations, and complementarity with training-time augmentation. Ablations with ICA and additional large-model experiments strengthen claims. Reviewers found responses satisfactory; two raised scores and one maintained a borderline accept.
Decision: Accept (poster)