Title: Test-Time Adaptation by Causal Trimming
Venue: NeurIPS 2025
Accepted: yes
Keywords: test-time adaptation; distribution shift; causality; representation;
Abstract: Test-time adaptation aims to improve model robustness under distribution shifts by adapting models with access to unlabeled target samples. A primary cause of performance degradation under such shifts is the model’s reliance on features that lack a direct causal relationship with the prediction target. We introduce Test-time Adaptation by Causal Trimming (TACT), a method that identifies and removes non-causal components from representations for test distributions. TACT applies data augmentations that preserve causal features while varying non-causal ones. By analyzing the changes in the representations using Principal Component Analysis, TACT identifies the highest variance directions associated with non-causal features. It trims the representations by removing their projections on the identified directions, and uses the trimmed representations for the predictions. During adaptation, TACT continuously tracks and refines these directions to get a better estimate of non-causal features. We theoretically analyze the effectiveness of this approach and empirically validate TACT on real-world out-of-distribution benchmarks. TACT consistently outperforms state-of-the-art methods by a significant margin.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for test-time adaptation by removing the non-causal components from intermediate representations. At test-time augmentations are applied and PCA is used to remove the components with the highest variance. Experiments were performed on a range of datasets, and supportive results were shown compared to the baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper was well written and easy to follow.\n- The experiments were done on a diverse datasets and domains (from audio signals, text to images) with supportive results.\n\n**Weaknesses**\n\n- The selected augmentations seems to assume some knowledge of about the distribution shift, i.e., they are selected to be similar to the shift for each dataset. Was the no TTA baseline trained with these augmentations? If not, It would be useful to see if learning invariances via training time augmentation alone is sufficient.\n- Performance of the method in practice. The paper mentions that oracle selection to used to select the hyperparameters (nb of augmentations, components), allowing for fair comparison across methods. While this can give a fair comparison, it makes it hard to understand the performance of the method in practice. From Fig. 4, there can be large differences in performance.\n- Computation cost. Since the method performs PCA, it can be more expensive that the baselines. Including the computational cost of each method make it easy to assess the trade-offs and make a more informed decision for time sensitive applications.\n\nQUESTIONS:\n- Was the “no TTA” baseline trained with the same augmentations used for TACT?\n- What would be the performance of the method if the hyperparameters were tuned on validation data instead?\n- Have the authors tried removing components based on a threshold instead? If the datapoint is invariant to the augmentation, will this remove the “causal” components too?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TACT (Test-time Adaptation by Causal Trimming), a novel backpropagation-free method for test-time adaptation (TTA) under distribution shifts. The key insight is that performance degradation often stems from reliance on non-causal features. TACT addresses this by identifying and removing these non-causal components in the representation space using PCA on augmentations that vary non-causal attributes while preserving causal semantics.\n\nSpecifically, TACT:\n\n* Uses targeted data augmentations to induce variability in non-causal dimensions.\n\n* Applies PCA to extract directions of maximum variance (assumed to align with non-causal features).\n\n* Removes projections onto these principal components (PCs) from both the sample representation and class prototypes.\n\n* Uses a moving average over the trimmed prototypes for stable prediction.\n\n* Provides theoretical guarantees for when causal trimming helps or preserves performance.\n\n* Demonstrates strong empirical results across five diverse datasets, showing consistent improvements over state-of-the-art (SOTA) backpropagation-free and backpropagation-based TTA methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Quality\n### Strengths:\n\n* Solid methodological foundation combining causal assumptions with representational geometry.\n\n* Clear theoretical propositions (with assumptions) analyzing when and why trimming helps.\n\n* Sound and comprehensive experimental protocol using five real-world datasets with meaningful shifts.\n\n* Ablation studies and Grad-CAM visualizations strongly support claims.\n\n### Weaknesses:\n\n* Performance gains rely heavily on carefully selected augmentations that preserve causal content, which may not generalize across tasks or be feasible without domain expertise.\n\n* PCA-based trimming assumes non-causal features dominate variance directions, which might not always hold, especially under entangled representation spaces.\n\n## Clarity\n### Strengths:\n\n* The writing is clear, structured, and technically precise.\n\n* Figures (especially Grad-CAM and ablations) help illustrate key points.\n\n* Theoretical section is accessible and intuitive despite formal proofs being deferred to appendix.\n\n### Weaknesses:\n\n* The role of the hyperparameter m (number of trimmed PCs) is critical but tuning guidance is heuristic and empirical.\n\n* The limitations section could more thoroughly discuss failure cases (e.g., when causal features are not orthogonal to spurious ones).\n\n## Significance\n### Strengths:\n\n* Offers a promising new direction in test-time adaptation that is backpropagation-free yet effective.\n\n* Bridges causal reasoning with representation learning and adaptation  a direction of growing interest. \n\n\n## Originality\n### Strengths:\n\n* The idea of causal trimming based on PCA over augmentations is novel in TTA.\n\n* Synthesizes ideas from disentanglement, causality, and test-time learning in a unique way.\n\n* Extends beyond label-free entropy minimization or pseudo-label refinement by directly adjusting the representation space.\n\n### Weaknesses:\n\nSome ideas, such as augmenting data to identify spurious directions, have been explored in domain generalization and DeYO. However, TACT’s trimming strategy is distinct.\n\nQUESTIONS:\n* Your method assumes that causal features remain invariant under augmentations while non-causal ones vary. How robust is TACT when this assumption is only approximately satisfied?\n\n* PCA assumes that variance-aligned directions are linearly separable. But in high-capacity models like ViT or BERT, representation entanglement is common. How sensitive is TACT to this? Have you considered using nonlinear alternatives (e.g., kernel PCA or ICA)?\n\n* The number of removed PCs (m) appears to significantly influence performance (Figure 4). Can you elaborate on how this is tuned in a realistic, non-oracle setting?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TACT, a backpropagation-free test-time adaptation (TTA) method that actively identifies and removes non-causal components from feature representations to improve robustness under distribution shift.\nThe authors provide theoretical analysis (Propositions 1–3) that formalizes when trimming corrects misclassifications and preserves causal information. Empirically, TACT is evaluated on five real-world out-of-distribution benchmarks—Birdcalls, Camelyon17, CivilComments, ImageNet-R, and ImageNet-V2—showing consistent gains over state-of-the-art backprop-free methods, and further improvements when combined with gradient-based adaptation (TACT-adapt)\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n- Unlike entropy-based or self-training TTA, TACT directly suppresses non-causal features via linear trimming, addressing the root cause of distribution sensitivity.\n\n- By avoiding gradient updates during adaptation, TACT incurs only a single PCA and a few vector operations per batch, making it attractive for latency-sensitive deployment.\n\n- Propositions 1–3 rigorously characterize conditions for error correction, causal preservation, and non-degradation of already-correct predictions, offering clear guidance on choosing the number of components m to trim.\n\nWeaknesses\n\n- TACT requires domain knowledge to choose augmentations that vary only non-causal factors. An empirical study of sensitivity to augmentation choice, or an automated strategy for augmentation selection, would enhance practicality.\n\n- PCA assumes that causal directions are orthogonal to non-causal ones. In complex tasks where causal and spurious features co-vary, this may fail. Exploring non-linear direction-finding (e.g., ICA or contrastive subspace methods) could address this limitation.\n\nQUESTIONS:\n- In Section 4.1 (Non-Causal Identification), do you project out only the first principal component, or do you remove m components—and if so, how is m chosen for each batch？\n\n- In Table 1, some reported standard deviations for TACT-adapt are quite large (e.g., Birdcalls ±5.68). Could you report standard deviations for all datasets and perform statistical significance tests against the strongest baselines?\n\n- For Figure 4’s hyperparameter study, would it be possible to present heatmaps of performance over various  (n,m) combinations to visualize their interaction and identify robust operating regions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper actively reduces non-causal features in TTA. To identify non-causal features, the authors analyzed how these representations change when applying targeted perturbations to the input data.

STRENGTHS AND WEAKNESSES:
* Strenghts:

1. The paper proposes a novel causal trimming approach to remove non-causal features from representations, improving model robustness under distribution shifts.

2. This idea is novel in TTA research field. The theoretical analysis is solid.

* Weakness:

1. Designing data augmentations that preserve causal features while perturbing non-causal ones requires domain knowledge. Consequently, this limits the method’s generalizability and practical deployment in domains where such knowledge is unavailable or hard to encode.

2. The identification of non-causal components is based on PCA, which assumes linearity and orthogonality of the principal directions. This may fail in practice.

QUESTIONS:
How to handle the limitations mentioned in the above weakness part?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05