NeurIPS 2025

Test-Time Adaptation by Causal Trimming
Yingnan Liu1,2
Rui Qiao1,3
Mong Li Lee1,2
Wynne Hsu1,2
1School of Computing, National University of Singapore
2Institute of Data Science, National University of Singapore
3Singapore-MIT Alliance for Research and Technology
{liu.yingnan, rui.qiao}@u.nus.edu, {dcsleeml, dcshsuw}@nus.edu.sg
Abstract
Test-time adaptation aims to improve model robustness under distribution shifts
by adapting models with access to unlabeled target samples. A primary cause of
performance degradation under such shifts is the model’s reliance on features that
lack a direct causal relationship with the prediction target. We introduce Test-time
Adaptation by Causal Trimming (TACT), a method that identifies and removes
non-causal components from representations for test distributions. TACT applies
data augmentations that preserve causal features while varying non-causal ones. By
analyzing the changes in the representations using Principal Component Analysis,
TACT identifies the highest variance directions associated with non-causal features.
It trims the representations by removing their projections on the identified direc-
tions, and uses the trimmed representations for the predictions. During adaptation,
TACT continuously tracks and refines these directions to get a better estimate of
non-causal features. We theoretically analyze the effectiveness of this approach and
empirically validate TACT on real-world out-of-distribution benchmarks. TACT
consistently outperforms state-of-the-art methods by a significant margin. Our
code is available at https://github.com/NancyQuris/TACT.
1
Introduction
Machine learning models often exhibit significant performance degradation when evaluated on data
drawn from a distribution that differs from their training data distribution [13]. To address this
challenge, test-time adaptation (TTA) has emerged as a promising approach. TTA methods adapt
a pretrained model to the test distribution dynamically, using the incoming tes...