IT$^3$: Idempotent Test-Time Training
Authors: Nikita Durasov
Assaf Shocher
Doruk Oner
Gal Chechik
Alexei A Efros
Pascal Fua
Venue: ICML 2025 poster (2025)
Keywords: idempotence;generalization;test-time training
Primary area: deep_learning->robustness
TL;DR: Adapting to distribution shifts at test time by training the model to be idempotent.

Abstract: Deep learning models often struggle when deployed in real-world settings due to distribution shifts between training and test data. While existing approaches like domain adaptation and test-time training (TTT) offer partial solutions, they typically require additional data or domain-specific auxiliary tasks. We present Idempotent Test-Time Training (IT3), a novel approach that enables on-the-fly adaptation to distribution shifts using only the current test instance, without any auxiliary task design. Our key insight is that enforcing idempotence---where repeated applications of a function yield the same result---can effectively replace domain-specific auxiliary tasks used in previous TTT methods. We theoretically connect idempotence to prediction confidence and demonstrate that minimizing the distance between successive applications of our model during inference leads to improved out-of-distribution performance. Extensive experiments across diverse domains (including image classification, aerodynamics prediction, and aerial segmentation) and architectures (MLPs, CNNs, GNNs) show that IT3 consistently outperforms existing approaches while being simpler and more widely applicable. Our results suggest that idempotence provides a universal principle for test-time adaptation that generalizes across domains and architectures.

I Idempotent Test-Time Training
Nikita Durasov
Assaf Shocher
Doruk Oner 3
Gal Chechik 2 Alexei A. Efros* Pascal Fua"
Abstract
encountered by systems operating in the real world often differs substantially from what they were trained on due to data distribution shifts over time or other changes in the environment. This inevitably degrades performance, even in state-of-the-art models (Recht et al., 2018; Hendrycks et al., 2021; Yao et al., 2022). Machine learning systems used in production not only need adapt to distribution shifts but also must do on-the-fly using very limited data.
Deep learning models often struggle when de- ployed in real-world settings due to distribution shifts between training and test data. While ex- isting approaches like domain adaptation and test-time training (TTT) offer partial solutions, they typically require additional data or domain- specific auxiliary tasks. We present Idempotent Test-Time Training (IT*), a novel approach that enables on-the-fly adaptation to distribution shifts using only the current test instance, without any auxiliary task design. Our key insight is that enforcing idempotence-wheer
Thus, in this work, we focus adapting to distribution shifts on-the-fly using only the current test instance or batch, without access to any additional labeled or unlabeled data during inference. During training, the model only has access to the base distribution training data, without any knowledge of the test distribution, which may be different.
repeated appli- cations of function yield the same result-can effectively replace domain-specific auxiliary tasks used in previous TTT methods. We theoretically connect idempotence to prediction confidence and demonstrate that minimizing the distance between successive applications of our model during in- ference leads to improved out-of-distribution per- formance. Extensive experiments across diverse domains (including image classification, aerody- namics prediction, and aerial segmentation) and architectures (MLPs, CNNs, GNNs) show that IT consistently outperforms existing approaches while being simpler and more widely applicable. Our results suggest that idempotence provides a universal principle for test-time adaptation that generalizes across domains and architectures.
Adversarial robustness and domain adaptation address re- lated challenges. However, they typically require additional data either during training or inference, and sometimes rely on specific assumptions about the nature of the shift. While effective when such additional knowledge is available, they are not designed for immediate, instance-level adaptation and are not applicable in the scenario we envision. Test- Time Training (TTT) (Sun et al., 2020) offers a promis- ing alternative by adapting the model during inference by performing an auxiliary self-supervised task on each test sample. This enables the model to handle corrupted and Distribiiititti (OOD) data using only the current test instance or batch, without access to anything else. However, TTT requires performing an auxiliary task specific to the data modality, such as orientation prediction or inpainting for image data (Gandelsman al., 2022). And defining an appropriate auxiliary task is not straightforward in general.
poster / code/ video / web
In this paper, we argue that enforcing idempotence can profitably replace the auxiliary tasks in TTT and yields an approach we dub IT3 that is a versatile and powerful while generalizing well across domains and architectures.
1. Introduction
Supervised learning methods, while powerful, typically as- sume that training and test data come from the same distri- bution. Unfortunately, this is rarely true in practice. Data
More specifically, let ff be a deep network that takes as input vector and a second auxiliary variable that can either be the ground truth label corresponding to x or a neutral uninformative signal 0. In (Durasov et al., 2024b), it was shown that if such a network is trained so that f(x, 0) = [(x,)) = y, then at test time the distance []] [xx 0)) f(x, 0)ll correlates strongly with the prediction error. What if, at test time, could actively minimize this distance
This work was supported in part by the Swiss National Sci- ence Foundation. 'Equal contribution 'CVLAB, EPFL 2NVIDIA 2 "Neura Vision Lab, Bilkent University *UC Berkeley. Correspon- dence to: Nikita Durasov Enikita.durasov @ovi.ia.com>>
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

IT:: Idempotent Test-Time Training
Train
Test
y
L
[ trainn
y
Yo
Y/1
O
fe
X
fe
F
x
Figure 1. Idempotent Test-Time Training (IT*) approach. During training (left), the model fo is trained to predict the label y with without y given to as input. At test time (right), when given a corrupted input, the model is sequentially applicd. then briefly trains with the objective of making fx -) to be idempotent using only the current test input.
whenever we encounter an OOD instance? Could we "pull it" into the distribution? IT uses this distance as a loss for TTT
Distribution Shifts: TTT has emerged as solution to the problem of generalization under distribution shifts. Using pre-trained network and at test-time refining on a single instance each time. In the foundational work of Sun et al. (2020), the model is adjusted in real-time by solving an auxiliary self-supervised task, such as predicting image ro- tations, on each test sample. This on-the-fl fly adaptation has proven good at improving robustness on corrupted and Out- Of-Distribution (OOD) data. As the self self-supervised learning methods became more efficient (He et al., 2022), they could be exploited for TTT (Gandelsman et al., 2022). Extensions such as TTT++ (Liu et al., 2021) require access to the entire test set. TENT (Wang et al., 2021) adapts during inference at the batch level, based on batch entropy, but cannot be ap- plied to single instances or very small batches. Moreover, relies on updating the model's normalization layers, making it architecture dependent. Another problem that many exist- ing TTT approaches face is their task dependency, meaning they are designed to work for a specific data types, which is almost always image classification. The recent method ActMAD (Mirza et al., 2023) addresses this limitation by pulling the mean and variance of data embeddings closer to those of the training data, improving predictions on out- of-distribution or corrupted data. While its effectiveness has been mostly demonstrated on image data, the approach has the potential to be applied to other types of data as well and we will use as one of the baselines we compare against. In (Park et al., 2024), the authors propose a test-time adap- tation method for depth completion, fine-tuning a specific adaptor layer using a consistency loss between two predic- tions. However, the approach is tailored to depth completion and not applicable other tasks.
sessions. Unfolding Mu- Y1 obtain: [x, f 0))
[(x, 0)ll- This makes (x,,)) idempotent. Fortunately, it has been shown that, while not trivial, training a model to achieve idempotence is feasible (Shocher et al., 2024). See further discussion in Appendix B.
This yields a generic method that does not rely on any spe- cific domain properties. This is in contrast prior TTT methods that rely on domain specific auxiliary task. By leveraging the universal property of idempotence, IT3 can adapt OOD test inputs on-the-fly fly across various domains, tasks and architectures. This includes image classification with corruptions, aerodynamic predictions for airfoils and cars, tabular data with missing information, age prediction from faces, and large-scale aerial photo segmentation, using MLPs, CNNs, or GNNs.
2. Related Work
In essence, 2 relies on idempotence to generalize Test- Time Training. We briefly review these two fields.
2.1. Test- Test-Time Training (TTT)
The idea of leveraging test data for model adaptation dates back to methods like transductive learning (Gammer- man et al., 1998). Early approaches, such as transductive SVMs (Collobert al., 2006) and local learning (Bottou & Vapnik, 1992), aimed to adapt predictions for specific test samples rather than generalizing across unseen data.
Training neural networks solely on single test instances, without pre- pre-training training, has been demonstrated in the "deep internal learning" line of work, for many image enhance- ment tasks (Shocher et al., 2018; Gandelsman et al., 2019) and single image generative models (Shocher et al., 2019; Shaham et al., 2019).
TTT vs. Test Time Adaptation (TTA): TTT operates per- instance, with no assumption that future test data will be similar. In contrast, TTA (Liang et al., 2024) adapts using a large test set from the same shift (Batch/Domain TTA), or assuming correlation between instances (Online TTA).
2

---

IT: Idempotent Test-Time Training
Most previous work (Sun al., 2020; Gandelsman al., 2022) have thus treated TTA and TTT as distinct paradigms rather than direct competitors. TTA exploits abundant test data but cannot tune the training process, while TTT shapes training but handles every test instance in isolation. Each approach suits different scenarios.
higher confidence, while divergent predictions signal un- certainty or out-of-distribution (OOD) data. Unlike popu- lar sampling-based uncertainty estimation methods (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Wen et al., 2020; Durasov et al., 2021), ZigZag does not require many forward passes or complex sampling, making it more computationally efficient for real-time applications.
2.2. Idempotence in Deep Learning
3. Method
Idempotence, a concept rooted in mathematics and func- tional programming, refers to an operation whose repeated application yields the same result as a single application. Mathematically, for a function f. being idempotent means
Given pre-trained model, IT's aims to dynamically adapt its weights at inference time using Test- Time Training (TTT) to reduce uncertainty and handle Out-of Diitr---iitiiittiinn (OOD) instances. As discussed above, other TTT approaches rely on satisfying domain specific auxiliary tasks to achieve this, Instead, we rely on the model we train being idempotent on the training set and adapt its weights at inference time to approach idempotence on the test set as new samples are being fed to it. This pulls the representations of OOD inputs back into the distribution of the training data and improves the model's performance on corrupted or OOD instances.
[((())) = [(x): Vr.
In other words, applying the function multiple times has no effect beyond the first one. the context of linear op- erators, idempotence corresponds to orthogonal projection. Over R", these are matrices A that satisfy A A, with eigenvalues of either 0 or I; they can be interpreted as geo- metrically preserving certain components while nullifying others. This concept was recently applied in generative modeling. Idempotent Generative Network (IGN) (Shocher et al., 2024) is a generative model that maps data instances to themselves, f(r) = x, and maps latents to targets that also map to themselves, f(f(=)) = fz)). It was shown to 'project' corrupted images onto the data manifold, effec- tively removing the corruptions without prior knowledge of the degradation.
In this section, we describe how we make our models idem- potent for training set samples, how we use the idempotence loss for TTT during inference, and how we adapt the algo- rithm for online scenarios.
3.1. Making the Network Idempotent at Training Time
Let fe be generic network with weights 0 that takes an input x. We wish to deploy in an environment where the statistical distribution of the samples it receives may change over time. To this end, as in ZigZag (Durasov et al., 2024a), we modify slightly its input layer so that it can accept a second argument that be either y, the desired output of the network given input x, neutral uninformative signal O. During the initial training. we minimize the supervised loss
Energy-Basee Models (EBMs; (Ackley et al., 1985)) offer a related perspective by defining a function f that assigns energy scores to inputs, with higher energy indicating less desirable or likely examples, and lower energy indicating those that fit the model well. IGN introduces similar con- cept but frames it differently: Instead of f directly serving as the energy function, the energy is implicitly defined via the difference 6(y)= = ( y)), where D measures the distance between the model's prediction and its input. In this framework, training f to be idempotent minimizes 5 (x)))) pushing the model toward low-energy configuration where its outputs remain stable under repeated applications. Thus, can be interpreted as transition operator that drives high- energy inputs toward low-energy, stable domain, reducing the need for separate optimization procedures to find the energy minimum.
Luain = fl y) - yll + fl, 0) - yyl,,
(2)
as depicted on the left side of Fig. 1. This enforces
yo = fo(x, 0) y
(3)
Y1 = fo(x, yo) = fo(x.y) my = fo fo (x, 0)) fo(x.
(4)
In other words, 0 has been adjusted so that function fo(x,) :) is as idempotent as possible for all x in the training set. Of course, when x is comes from the test set, there is no guarantee of that because there may be distribution shift between the two sets. In (Durasov et al., 2024a), it is shown that the deviation from the equality of Eq. 4 expressed as [x, fx 0)) fx(x, 0) correlates strongly with the accuracy of the prediction and can be used to detect testing samples that are -istrrribtttinn with respect to the train- ing set. Fig. illustrates this in the case of network trained
In concurrent work, the ZigZag method has first been pro- posed and then extended to recursive networks (Durasov means to et al., 2024a;b). It introduces idempotence Zig7ag op- assess uncertainty in neural network predictions. crates by recursively feeding the model's predictions back as inputs, allowing the model to refine its outputs. The consistency between successive predictions acts as an un- certainty metric, where stable, unchanged outputs indicate
3

---

IT: Idempotent Test-Time Training
to predict the lift-over-drag ratio (L/D) of a 2D airfoil. In other words, here, is 2D outline representing an airfoil and the output is expected the corresponding L/D.
model F. By updating only So and keeping F fixed, we ensure that yo is adjusted minimize the discrepancy with YI, without pulling y1 toward an incorrect yo. A similar idea was employed in the IGN approach (Shocher et al., 2024) when meaningful predictions are required. After each TTT optimization iteration, the dynamic model fo is initialized with O ready for the next input.
Idermpotence Histogrems (Airfoils)
0.8
000 Data optimized)
0CD Data (optimized)
0.7
Test Date
0.6
Train Data
Essentially, IT extends the projection principle of Idem- potent Generative Networks (IGN) (Shocher et al., 2024). IGNs map corrupted inputs onto the distribution of valid data by enforcing idempotence. Similarly, IT3S projects OOD (æ; y) pairs onto the distribution of valid ones by iteratively refining the network's internal representations. While only y explicitly changes, every layer's activa- tions-funcciios of both x and y-adjust to better fit the distribution of in-distribution representations, much like IGN corrects corrupted data by pulling it toward the natural image manifold. See detailed discussion in Appendix B.
0.5
0.4
0.3
0.2
0.1
0.0
0
Idemppteece value [P1 - Vil
Figure 2. Idempotence vs. Out-n--iistriuttoonnss:: We plot the distribution of idempotence errors, measured by the distance [y) y2l in Eq.4, for training, test, and OOD data, For OOD samples, we show the errors both before and after minimizing them globally. OOD samples exhibit significantly larger idempotence errors. which decrease after optimization. Figuratively, IT' pushes the OOD representations to be more similar to those of the training distribution. In Sec.4, we show that this reduction yields improved performance.
3.3. Online IT
We introduce a variant of IT for different scenario: Given data streams, where the distribution shifts continuously over time, in a continual learning setup, we modify IT to operate in an online mode by not resetting fo back to F after each TTT episode, as we did in Section 3.2. We essentially assume that the distribution mostly shifts smoothly and, thus, there is good reason to believe that the current state of fe is a better initialization for the next TTT episode than the original F. This makes the model evolve over time. In this scheme, it can happen that the performance of the model on data from its original training decreases significantly, a phenomenon known as catastrophic forgetting (Kirkpatrick et al., 2017). This is acceptable as the goal is to perform well on data at the present moment, rather than on past examples.
3.2. Test-Time Training
In ZigZag (Durasov et al., 2024a), deviations from idem- potence, measured by the distance between the two pre- dictions of Eq. 4, are used to evaluate the accuracy of prediction. In IT3 we propose to go further and to mini- mize these deviations at inference time to compensate for potential domain shifts between training and test data.
A naive way would be to minimize the loss function
LTTT= = (x, (x, 0)) fo(x, 0)ll
for all samples x received as inference time. However, this can produce undesirable side effects. For instance, if yo = fo( 0) is an incorrect prediction, minimizing Ilyo yill may cause yı = fx( to be pulled toward the incorrect yo. thereby magnifying the error. Another potential failure mode is to encourage (x. -) to become the identity function, with is trivially idempotent.
We make another modification in the second sequen- tial application of the model, that is, when computing F(x, Jo(x, (x, 0)). Since the data keeps shifting, there is no reason to retain the frozen F as an anchor indefinitely. Over time, fa may diverge far from F. making it irrelevant. Rely- ing on the old state of the model would prevent the model from evolving efficiently. Replacing it with the current state of fe is out of the question, it could causes collapse as described in Section 3.2. Instead, we need an anchor that is influenced by reasonable amount of data, yet evolves over time. Our solution is to replace F with an Exponential Moving Average (EMA) of the model fe, denoted as JEMA This means fEMA is a smoothed version of fe over time. The test-time loss in the online setting then becomes
To prevent this, we modify the test-time training procedure as shown on the right side of Fig. 1. We keep a copy of the model as it was at the end of the training phase, denoted as F= fe, where are the weights obtained after the initial training of Sec. 3.1, which will not be updated further. We then take the test-time loss to be
fn = || F fo(x, 0)) fa(x, 0)]l
(6)
Conline = fEMMAXX fo(x, 0)) fo(x,
(7)
where fo is the model being updated at test-time. Here, the first prediction yo = fx. 0) is computed as before, but the second one, y1 = F(x, fx(, 0)), is made using the frozen
By updating both fe and fEMA incrementally, with fEMA serving as stable reference that changes more slowly, the
4

---

IT:: Idempotent Test-Time Training
model adapts to gradual shifts without overfitting to noise or temporary anomalies.
model is a simple Multi-Layer Perceptron (MLP) optimized using the Adam optimizer, and observe that IT consis- tently degrades less compared to other baselines across all OOD levels as depicted in Fig. 3.
4. Experiments
We evaluate our approach across a diverse set of data types and tasks, including age prediction, image classification, and road segmentation in the visual domain, as well as aerodynamics prediction using 3D data and tabular data periments. In all these scenarios, we first train the model using the supervised approach of Section 3.1 and then per- form the test-time training of Section 3.2. For each task, we design an OOD test set for evaluation, that is, data drawn from a shifted distribution with respect to that of the training set. The OOD data is divided into several levels, with higher levels representing data that is progressively further from the training distribution. We evaluate our method for each level, presenting the results as bar plots for different batch sizes. After running the algorithm on particular batch, we reset the model to its original, non-updated weights before evaluating the next batch.
1.0
Not Optimized ACtMAD (batch=2)
AcIMAD (batd)=8) (bTattch-1)
IT' (batchi44)
(batchwm))
ACIMAD (batched)
0.8
MAE
0.6
0.4
0.2
0.0
1
3
4
Out-of-distribution Level
Figure 3. UCT Results on OOD inputs: The plots illustrate the performance of IT compared to other haselines across different OOD levels. The box plot for tabular data shows the distribution of MAE at various OOD levels, where 1 with different batch sizes (lbatch=1, batch=4, batch=8]) degrades less compared to the Not optimized baseline and ActMAD. Larger batch sizes preserve performance more effectively.
In our experiments, we compare our method against a non- optimized model to demonstrate the effectiveness of TTT approaches relative to vanilla model, well as other pop- ular baselines. For the image classification task, we include the original TTT method and newer more versatile ap- proach, ActMAD (Mirza et al., 2023), which we descrihed in Section and apply across all other setups. To further assess the effectiveness of our approach, we also evalu- ate all baselines, except the vanilla model, using different batch sizes. Across all scenarios, our method degrades more slowly than the baselines as the domain shift between train- ing and testing data increases. Additionally, in Appendix A, we provide inference time comparisons for the considered approaches.
4.2. CIFAR
We conducted similar experiments using the CIFAR- 10 (Krizhevsky et al., 2014) dataset, selecting CIFAR- C (Hendrycks & Dietterich, 2019) as the out-of-distribution (OOD) data. CIFAR-C contains the same images as CIFAR- 10 but with various common corruptions, such as Gaussian noise, blur, and contrast variations, simulating real-world conditions. These corruptions are applied at different sever- ity levels, allowing us to evaluate how the model's perfor- mance degrades as the data shifts further from the original CIFAR-10 distribution. For this experiment, we used the Deep Layer Aggregation (DLA) (Yu et al., 2018) network, known for its strong performance in image classification and robustness to overfitting. We trained the model according to the guidelines from the original DLA paper to ensure opti- mal results. Fig.3 shows the evaluation error on CIFAR-C at severity level for different types of corruptions, follow- ing (Sun al.. 2020). As shown, IT's outperforms other baselines, with higher batch sizes yielding the best results.
4.1. Tabular Data
Tabular data consists of numerical features and correspond- ing continuous target values for regression tasks from the UCI tabular datasets (Bay et al., 2000). They are widely used in machine learning research to benchmark regression models. In our case, use The Boston Housing dataset describes housing prices in the suburbs of Boston, Mas- sachusetts. It includes various features related to socioe- conomic and geographical factors that influence housing prices. We take a test set and gradually apply random fea- ture zeroing with increasing probabilities of 5%, 10%, 15%, and 20% mentioned levels of OOD). This random fea- ture dropping simulates out-of-distribution (OOD) data by progressively altering the input features, making the data less similar to the original training distribution. As the probability of feature dropping increases, the data becomes more OOD, which lowers the model's accuracy. The trained
4.3. Age Prediction
To experiment with image-based age prediction from face images, we use the UTKFace dataset (Zhang et al., 2017). large-scale collection containing tens of thousands of face images annotated with age information. The model is trained on face images of individuals aged between 20 and 60, while individuals younger or older than this range are considered out-of-distributin (OOD) (Fig.5). The further the age is from the 20-60 interval, the higher the OOD level
5

---

IT:: Idempotent Test-Time Training
Object recognition task only
ACIMAD (batch=128)
TTT (batch=128)
Ibaach 32)
ACUMAD (balch=8)
TTT (batch=8)
(batch B)
[batchh 128)
ACtMAD (batch-322
TTT (batch-32) 32)
Erot 6
50
40
30
20
0
original
gauss
shot
impulse
defocus
glass
motion
zoom
snow
frost
fog
bright
contrast
elastic
pixelate
Joeg
Figure 4. Test error (%) on CIFAR-10-4 with level 5 corruptions. We compare our approaches, 17 with object recognition without self-supervision, TTT, and ActMAD, 17 improves over other basclines and higher hatch size improves even further.
Figure S. Face Samples. The (top) row shows training images of middle-aged individuals, while (middle) and (bottom) display images of older and younger individuals (OOD).
Figure 7. Road Samples. The road datasct (left) cov- ers urban areas of six different countries while the Mas- sachusetts dataset (right) primarily features rural neighbor- hoods along with some urban areas.
Optimized
Bbttch
(oanch116)
30
[bwtch=4)
20
Rot Oytim ALDOD (bathh15)
(batch (bstcie4)
17* (butc)=221))
0.9
10
0.7
20.6
00.
diz Loval
0.4
Figure 6. Age boxplot results on OOD shapes. Not optimized corresponds to a single model without TTT applied. IT3 with [batch=4, batch=8, and batch=16] represents our method dif- ferent batch sizes. As the data shifts further from the training distribution, our method degrades less, with larger batches preserv- ing performance more effectively.
0.3
0.2
Out-ci-cistr bution Level
Figure 8. Roads results on OOD images. Not optimized corre- sponds to a single model without TTT applied. IT with [batch=4, batch=8, and batch=16] represents our method at different batch sizes. As the data shifts further from the training distribution. our method degrades less compared to the Not optimized, TTT, and ActMAD, with larger batches preserving performance more effectively.
we assign to it. We use a ResNet- 152 backbone with five additional linear layers and ReLU activations. This archi- tecture delivers strong accuracy, outperforming the popular ordinal regression model CORAL (Cao et al., 2020) and matching other state-of-the-an methods (Berg et al., 2021). We train our model on the UTKFace training set (limited to individuals aged 20-60) and then run inference on faces at different OOD levels. Again, IT significantly outperforms the non-optimized model, as shown in Fig.
road segmentation in aerial imagery using the RoadTracer dataset (Bastani et al., 2018). We train a DRU-Net (Wang et al., 2019), on the RoadTracer dataset.
We perform OOD experiments using Massachusetts Road dataset (Mnih, 2013) that primarily comprises rural neigh- borhoods, as depicted in Fig. 7. We sample 450 images, each with dimensions of 1500 x 1500 pixels and divide them into four groups based on the Mean Squared Error (MSE) of the segmentation outputs, effectively creating dif-
4.4. Road Segmentation
Our method can be generalized to segmentation tasks as well. To demonstrate this, we consider the problem of
6

---

IT: Idempotent Test-Time Training
Figure? 9. Qualitative effect of IT on Road Segmentation. From left to right: (I) Original aerial image, (2) Output before optimization, (3) IT output at 5th iteration, (4) IT oulput the 15th iteration, and (5) Ground truth label. The segmentation quality improves significantly with IT iterations, observed in the progressively refined outputs at the 5th and 15th iterations.
ferent levels of distributional shift within the sampled set. We then further train the network on these OOD subsets using the ZigZag method (Durasov et al., 2024a).
Drag
We evaluate road segmentation performance by using Cor- rectness, Completeness and Quality (CCQ) metric (Wiede- mann et al., 1998) which is a popular metric to evaluate de- lineation performance. The Correctness, Completeness and Quality are equivalent to precision, recall and intersection- over-union, where the definition of true positive has been relaxed from spatial coincidence of prediction and anno- tation to co-occurrence within a distance of 5 pixels. As shown in Fig.8, IT' significantly improves performance on OOD images (see Fig.16 16 for more qualitative examples).
Figure 10. Airfoil Samples. Training and testing profiles (left) show reasonable acrodynamies, while OOD samples (right) fea- ture rare, high lift-to-drag shapes. Black arrows indicate pressure, and red lines show lift and drag.
70-
Optimteed
ActMAD (batch=11)) (batch-1)
dostchand (batch-12)
ALIMAD (bulchmd))
40-
4.5. Aerodynamics Prediction
20
Wings. Our method is versatile and can handle various types of data. To illustrate this, we generated a dataset of 2,000 wing profiles, as depicted in Fig. 10, by sampling the widely used NACA parameters (Jacobs & Sherman, 1937). We used the XFoil simulator (Drela, 1989) to compute the pressure distribution along each profile and estimate its lift-to-drag coefficient, a crucial indicator of aerodynamic performance. The resulting dataset consists wing profiles x,, represented by a set of 2D nodes, and the corresponding scalar lift-to-drag coefficient y; for 1 S i 2000.
10-
i
Oup Sstrittittttt Level
Figure 1 1. Airfoil results on OOD shapes. Not optimized cor- responds to single model without TTT applied. ITTT with [batch=1, batch=4, and batch=16] represents our method at dif- ferent batch sizes. As the data shifts further from the training distribution, our method degrades less compared to the Not opti- mized and ActMAD. with larger batches preserving performance more effectively.
We selected the top 5% of shapes, based their ratio, out-of-distribution (OOD) samples. The OOD lev- els were determined using the ground truth lift-to-drag ratio, where higher OOD levels correspond to more aerodynami- cally streamlined shapes. The training set includes shapes with lift-to-drag values ranging from 0 to 60, with anything beyond this threshold considered OOD and excluded from training. We then trained Graph Neural Network (GNN) composed of 25 GMM (Monti et al., 2017) layers, featur- ing ELU activations (Clevert et al., 2015) and skip connec- tions (He et al., 2016), to predict the lift-to-drag coefficient y; from the profile x,, following the approach (Remelli et al., 2020; Durasov et al., 2024a). As with previous ex- periments, I significantly improves performance on OOD shapes and provides more accurate predictions compared to
other baselines, as shown in Fig. 11.
Cars. As for wings, we experimented with 3D car models from a subset of the ShapeNet dataset (Chang et al., 2015), which contains car meshes suitable for CFD simulations, as depicted in Fig. 12. The experimental protocol was the same as for the wing profiles, except used OpenFOAM (Jasak et al., 2007) to estimate drag coefficients and employed a more sophisticated network to predict them from the trian- gulated 3D car meshes.
To predict drag associated to a triangulated 3D car, we uti- lize similar model to airfoil experiments but with increased

---

IT: Idempotent Test-Time Training
Table 1. Qualitative result for Online IT. Evaluation metrics for the road segmentation task (left), airfoils lift-to draddrra prediction
(middle), and car drag prediction (right). Online IT enhances performance compared to the original model and significantly outperforms offline IT.
METHOD
Corr
Comp
Quality
METHOD
MAE
METHOD
MAE
NOT OPTIMIZED
55.7
44.3
39.5
NOT OPTIMIZED
38.2
NoT OPTIMIZED
0.501
IT (BATCH==)
55.7
49.1
16.4
IT (BATCH=1)
37.6
IT (BATCH=
0.446
I (BATCH=8)
58.1
52.0
48.5
IT (BATCH=4)
37.5
IT (BATCH=2)
0.424
IT
(BATCH=16)
57.3
52.7
48.7
IT (BATCH=16)
37.4
IT (BATCH=4)
0.412
IT (ONLINE)
77.5
79.8
69.8
IT (ONLINE) 34.1
IT (ONLINE) 0.385
superior results w.r.t. the offline setup, as our model keeps being trained on the new distribution. A more effective way to constant adaptation over time is to use continuously changing distribution. We test IT3S on increasing cor- ruption / OOD level. In all cases, the online IT3 performs significantly better than the basic anchored variation.
Road segmentation. Building upon our previous road seg- mentation experiments, we further evaluate the effectiveness of online IT In the online IT setup, OOD samples are ranked based on their mean squared error (MSE) loss when passed through the vanilla network. We begin by selecting the samples with low MSE loss, as these are closer to the training distribution given the network's strong performance on them. Gradually, we introduce samples with progres- sively higher MSE loss, smoothly shifting between distri- butions and thereby allowing the model adapt effectively to range of OOD samples. As in previous experiment, use DRU-Net trained on the RoadTracer dataset vanilla model and 890 images are sampled from Massachusetts dataset as OOD images.
Figure 12. Car Samples. The car dataset comprises many regular vehicles (left) and a few streamlined ones (right), which we treat as being out-of off-iistributin. Red and blue denote high and low pressures respectively.
0.8
Not Optimized
ACTMAD (batch=4)
17" (batch=2) (bitch=4)
(batch=2) ActMAD (batch-2)
(batch=1))
0.4
0.0
Firstly, the vanilla network is tested on the Massachusetts dataset without any additional fine-tuning. We then ap- ply online IT's during inference adapt the model to the OOD distribution new data is presented. We evaluate the segmentation performance using the Correctness, Complete- ness, and Quality metrics, described previously. Table I (left) summarizes the results. The application of IT'S im- proved the performance over the initial network and the online IT method significantly outperforms the offline IT3,
3 dustrriittritttion Level
Figure 13. Car results on OOD shapes. Not optimized cor- responds to a single model without TTT applicd. ITTT with [batch=1, batch=2, and batch=4] represents our method at dif- ferent batch sizes. As the data shifts further from the training distribution, our method degrades less compared the Not op- timized baseline and ActMAD, with larger batches preserving performance more effectively.
capacity. Instead of twenty five GMM layers, we use thirty five and also apply skip-connections with ELU activations. Final model is being trained for 100 epochs with Adam optimizer and 10-3 learning rate. As with airfoils exper- iments, IT significantly improves performance on OOD shapes and provides more accurate predictions compared to other baselines, as shown in Fig. 13.
Aerodynamics. We conducted online experiments for air- foils and cars lift-to-drag prediction. We set the data stream such that OOD shapes appear with increasing aerodynamic properties, modeling a continuous domain shift in the data. As with the segmentation results, the online version signifi- cantly outperforms both the offline version and the original network, as shown in Tabs. (middle and right).
4.6. Online 3
4.7. ImageNet
We test our proposed online variation on several tasks. As- suming a data stream online scenario rather than the previ- ous setup. Naturally, when the distribution remains constant (although shifted from the training distribution) we expect
ImageNet-C (Hendrycks & Dietterich, 2019) consists of Im- ageNet (Krizhevsky et al., 2012) test images corrupted using the same transformations CIFAR-10/I00C (Sec. 4.2). We
8

---

IT: Idempotent Test-Time Training
50
cccuracy
40
TENT
ETA
MEMO
ACtMAD
m
20
Optimized
TENT
ETA (batchi:2))
MEMO (batch=:1))
ACTMAD (batch 161
IT? ibatch=16F
10
(botch-16)
MEMO (batch=32)
ACtMAD (batch-
Ibatch=221
(botch-32)
(batch-64)
MEMO (batch-64)
(batch-
IT" (batch=64)
TENT (batch-54)
ETA (batch-
MEMO (batch=128)
ActMAD (batch 128)
(batch-129)
TENT (batch-128)
Corruptions Level
Figure 14. Test accuracy (%) on ImageNet-C across 5 corruption levels. This plot compares our method, IT', with popular adaptation approaches including TENT, ETA, MEMO, and ActMAD the ImageNet-C dataset. IT consistently outperforms all baselines across all corruption severity levels and batch sizes, with performance further improving at higher batch sizes.
corruption levels and batch sizes, and significantly surpasses the original baseline model. As previously observed, larger inference batch sizes improve performance for all methods. These results demonstrate that our approach is also effective in large-scale data scenarios.
Pearson
70
60
Accuracy (%6
50
Corruption
Accuracy vs Idempotence. Similar to the results in Fig. 2, Fig. 15 illustrates the correlation between model perfor- mance and idempotence error. This supports the core idea of our method: optimizing idempotence during inference can improve performance. As shown, idempotence error exhibits strong negative correlation with accuracy. This observation further reinforces the conclusions of (Shocher et al., 2024) and (Durasov et al., 2024a), which highlight that idempotence error across multiple predictions is strong indicator of model performance key motivation behind our approach.
30
20
0.75
1.75
2.00
Idempotence Error
Figure Accuracy vs Idempotence on Corrupted ImageNet: This plot demonstrates strong correlation between model idem- polence and performance. Each point corresponds to one inference batch, with accuracy and idempotence еиrо computed per batch. The color indicates the corruption level from ImageNet-C for the respective batch. The Pearson correlation between idempotence error and accuracy is 0.94, indicating a strong negative trend.
5. Conclusions, Limitations, and Future Work
We have proposed approach to test-time-training relying enforcing idempotence as new samples are being consid- ered. This effectively handles domain shifts and method generic. We have demonstrated that is effective in a wide range of domains without requiring domain -specff knowl- edge, which sets it apart from state-of- the-art methods.
used it for large-scale classification experiments, evaluat- ing performance across 155 corruption types and different severity levels. For our setup, we employed a standard R8 (He et al., 2016) and followed the Pytorch train- ing protocol (Paszke et al., 2017). addition our previous experiments, included several widely used baselines this benchmark: TENT (Wang et al., 2020), ETA (Niu et al., 2022), and MEMO (Zhang et al., 2022).
The flip side is that IT lacks domain expertise. In some cases, it is hard to apply IT to single instances without ad- ditional conditions. This is most common in domains where information within a single input limited. Combining IT3 with camain-specific methods may remove these limitations, which we will explore future work.
The results on ImageNet-C (each bar represents the average accuracy across 15 corruption types) are shown in Table 14, Our method outperforms all other approaches across

---

IT: Idempotent Test-Time Training
Impact Statement
Durasov, N., Bagautdinov, T., Baque, P., and Fua, P.
Masksembles for Uncertainty Estimation. In Conference Computer Vision and Pattern Recognition, 2021.
This paper introduces IT' a new approach that enables more adaptable machine-learming models by refining their predictions for out-of-distribution data at test time. Through this on-the-fly adaptation, IT can reduce training costs, data requirements, and model size, making advanced Al methods more broadly accessible. The technique thus offers a step toward more efficient, flexible deployment of deep learning in real-world scenarios, where conditions often shift beyond the original training domain.
Durasov, N., Dorndoff, N., Le, H., and Fua, P. Zigzag: Universal Sampling-Free Uncertainty Estimation through Two-Step Inference. Transactions on Machine Learning Research, 2024a.
Durasov, N., Oner, D., Le, H., Donier, J., and Fua, P. En- abling Uncertainty Estimation in Iterative Neural Net- works. In International Conference on Machine Learning, 2024b.
References
Ackley, H., Hinton, G. E.,, and Sejnowski, J. A Learn- ing Algorithm for Boltzmann Machines. Cognitive Sci- ence, (1):147-169, 1985.
Gal, Y. and Ghahramani, Z. Dropout as a Bayesian Ap- proximation: Representing Model Uncertainty in Deep Learning. International Conference on Machine Learn- ing, pp. 1050-1059, 2016.
Bastani, F., He, S., Alizadeh, M., Balakrishnan, H., Madden, S., Chawla, S., Abbar, S.. and Dewitt, D. Roadtracer: Automatic Extraction of Road Networks from Aerial Im- ages. In Conference on Computer Vision and Pattern Recognition, 2018
Gammerman, A., Vovk, V., and Vapnik, V. Learning by Transduction, Vol Uai"98, 1998.
Gandelsman, Y., Shocher, A., and Irani, M. "Double-Dip": Unsupervised Image Decomposition via Coupled Deep- Image-Priors. In Conference Computer Vision and Patlern Recognition, pp. 11026-1 1035, 2019.
Bay, S. D., Kibler, D., Pazzani, M. J., and Smyth, P. The UCI KDD Archive of Large Data Sets for Data Mining Research and Experimentation. ACM SIGKDD explo- rations newsletter, 2500 2000.
Gandelsman, Y., Sun, Y., Chen, X., and Efros, A. Test- Time Training with Masked Autoencoders. In Advances in Neural Information Processing Systems, pp. 29374- 29385, 2022.
Berg. A., Oskarsson, M., and O' C'Conoor, M. Deep Ordinal Regression with Label Diversity. International Con- ference on Pattern Recognition, pp. 2740-2741 2021.
He, K., Zhang, X., Ren, S., and Sun, Deep Residual Learn- ing for Image Recognition. In Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Bottou, L. and Vapnik, V. Local Learning Algorithms. Neu- ral computation, 4(6):888-900 900, 1992.
Cao, W., Mirjalili, V., and Raschka, S. Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation. Pattern Recognition, 140:325-332 331, 2020.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked Autoencoders Are Scalable Vision Learners. In Conference on Computer Vision and Patern Recognition, pp. 16000-16009, 2022.
Chang, A., Funkhouser, T., G., I.., Hanrahan, P., Huang, Q.. Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., and Yu, F. Shapenet: An Information-Rich 3D Model Repository. In arXiv Preprint, 2015.
Hendrycks, D. and Dietterich, Benchmarking Neural Network Robustness Common Corruptions and Pertur- bations. In International Conference Learning Repre- sentations, 2019.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In arXiv Preprint, 2015.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo. M., et al. The Many Faces of Robustness: A Critical Analysis of Out-Of-Distribution Generalization. In International Conference on Computer Vision, pp. 8340-8349, 2021.
Collobert, R., Sinz, F., Weston, J., Bottou, L.. and Joachims, T. Large Scale Transductive SVMS. Journal of Machine Learning Research, 7(8), 2006.
Jacobs, E. and Sherman, A. Airfoil section characteristics as affected by variations of the reynolds number. Report- National Advisory Committee for Aeronautics, 227:577- 611, 1937.
Drela, M. XFOIL:
An Analysis and Design System for Low Reynolds Number Airfoils. Conference on Low Reynolds Number Aerodynamics, pp. 1-12, 1989.
10

---

IT: Idempotent Test-Time Training
Jasak, H., Jemcov, A., Tukovic, Z., et al. OpenFOAM:
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E.,
A C++ Library for Complex Physics Simulations. In International workshop coupled methods in numerical dynamics, 2007.
Devito, Z., Lin, Z.., Desmaison, A., Antiga, I.,, and Lerer, A. Automatic Differentiation in Pytorch. In Advances in Neural Information Processing Systems, 2017.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins. G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- Baraaasaaaiin,,,, A., et al. Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the Na- tional Academy of Sciences USA, 114((1):3521-3526, 2017.
Recht, B., Roclofs, R., Schmidt, L., and Shankar, V. Do CIFAR-10 Classifiers Generalize to CIFAR-10? In arXiv Preprint, 2018.
Remelli, E., Lukoianov, A.. Richter, S., Guillard, B., Bagaut- dinov, T., Baque, P., and Fua, P. Meshsdf: Differentiable Iso-Surface Extraction. In Advances in Neural Informa- tion Processing Systems, 2020.
Krizhevsky, A., Sutskever, I., and Hinton, G. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, pp. 1106-1114, 2012.
Shaham, T. R., Dekel, T., and Michaeli, T. Singan: Learning a Generative Model from a Single Natural Image. In International Conference on Computer Vision, pp. 4570- 4580, 2019.
Krizhevsky, A., Nair, V., and Hinton, G. The CIFAR-10 Dataset. online: http:///ww.. cs. foronto. edu/kriz/cifar. html, 55(5), 2014.
Shocher, A., Cohen, N., and Irani, M. "zero-Shot" Super- Resolution Using Deep Internal Learning. In Conference Computer Vision and Pattern Recognition, pp. 3118- 3126, 2018
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems, 2017.
Shocher, A., Bagon, S., Isola, P., and Irani, M. Ingan: Cap- turing and Retargeting The" Dna" of a Natural Image. In International Conference on Computer Vision, pp. 4492- 4501, 2019.
Liang, J., He, R., and Tan, T. A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts. Interna- tional Journal of Computer Vision, pp. 1-34, 2024.
Liu, Y., Kothari, P., Van Delft, B., Bellet-Gurlet, B., Mordan, T., and Alahi, A. Tu+++ When Does Self-Supervised Test- Time Training Fail or Thrive? In Advances in Neural Information Processing Systems, pp. 21808-21820, 2021.
Shocher, A., Dravid, A. V., Gandelsman, Y., Mosseri, I., Rubinstein, M., and Efros, A. A. Idempotent Generative Network. In The Twelfth International Conference on Learning Representations, 2024.
Mirza, M., Soneira, P., Lin, W., Kozinski, M.. Possegger, H., and Bischof, H. Actmad: Activation Matching to Align Distributions for Testtime Training. In Conference on Computer Vision and Pullern Recognition, pp. 24152- 24161, 2023.
Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A., and Hardt, M. Test-Time Training with Sele-suuer for Gen- eralization Under Distribution Shifts. In International Conference on Machine Learning, pp. 9229-9229 2020.
Mnih, V. Machine Learning for Aerial Image Labeling. PhD thesis, University of Toronto, 2013.
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion. arXiv preprint 2.xYv:2006.1072200 2020.
Monti, F., Boscaini, D., Masci, J., Rodolà, E., Svoboda, J., and Bronstein, M. M. Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In Conference Computer Vision and Pattern Recognition, pp. 5425-5434, 2017.
Wang. D., Shelhamer, E., Liu, S., Olshausen, B., and Dar- rell, T. Tent: Fully Test-Time Adaptation by Entropy Minimization. In International Conference on Learning Represemations, 2021.
Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation with- out forgetting. In International conference on machine learning, pp. 16888-16905. PMLR, 2022.
Wang. W., Yu, K., Hugonot, J., Fua, P., and Salzmann, M. Recurrent U-Net for Resource-Constrainee Segmentation. In International Conference on Computer Vision, 2019.
Park, H., Gupta, A., and Wong, A. Test-time adaptation for depth completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20519 22519,, 2024.
Wen, Y., Tran, D., and Ba, J. Batchensemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning. International Conference on Learning Representations, 2020.

---

IT: Idempotent Test-Time Training
Wiedemann, C., Heipke, C., Mayer, H., and Jamet, O. Em- pirical Evaluation of Automatically Extracted Road Axes. In Empirical Evaluation Techniques in Computer Vision, pp. 172-187, 1998.
Yao, H., Choi, C., Cao, B., Lee, Y., Koh, P. W., and Finn, C. Wild-Time: A Benchmark of In-The-Wild Distribution Shift over Time. In Advances in Neural Information Processing Systems, 2022.
Yu, F., Wang, D., Shelhamer, E., and Darrell, T. Deep Layer Aggregation. In Conference Computer Vision and Pattern Recognition, pp. 2403-2412, 2018.
Zhang, M., Levine, S., and Finn, C. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-38642,
2022.
Zhang. Z., Song, Y.,, and Qi, H. Age Progression/regressionn by Conditional Adversarial Autoencoder. In Conference on Computer Vision and Pattern Recognition, 2017.
12

---

IT: Idempotent Test-Time Training
A. Inference Time Comparison
Our method typically requires only 1-3 optimization steps, keeping the overall cost comparable to other well-known TTT methods. Below, we provide a comparison of inference times on out-of-distribution data for three approaches: the base model without optimization, the state-of-the-art TTT method ActMAD, and IT As shown, while our method introduces no significant overhead compared to ActMAD, it remains computationally efficient while achieving substantial improvements in performance. A similar observation can be made about memory consumption, as reported in Tab. 5 for the case of batch size 128 on ImageNet-C, showing peak memory reserved (in GB) using torch. cuda max memory reserved ().
Table 2. Inference Time Comparison (OOD Airfoils)
Table 3. Inference Time Comparison (OOD Cars)
Method
Base Model
ActMAD
IT3
Method
Base Model
ActMAD
IT
Inference Time (1)
1x
3x
4x
Inference Time (L)
1x
4X
5x
Table 4. Inference Time Comparison (OOD Roads)
Table 5. Memory Consumption (OOD ImagetNet), GPU Gb
Method
BASE
ActMAD
IT3
Method
BASE TENT MEMO ETA
ActMAD IT3
Inference Time (I)
1x
4.5x
6x
Memory (L)
4.5
4.8
13.5
4.9
7.2
7.4
B. Extended Discussion: Relating IT to IGN
In this section, elaborate on how IT generalizes the projection principle of Idempotent Generative Networks (IGN) to supervised test-time training setting. We show that both approaches idempotence reppotttencc-ppptteeeeeeeeeeeedd applications of the network function should yield the same result-as as way to "project" off-manifold inputs onto learned manifold of valid data. While IGN enforces idempotence directly on all possible inputs, IT enforces it primarily on training data but adapts on-the-ffy fly at test time to handle outtof-distriuution (OOD) samples.
The Projection Principle in IGN.
IGN (Shocher et al., 2024) learns 9a : 2 > X, mapping from a source distribution Pg (e.g. Gaussian noise) to a target distribution Pæ C X' (e.g. natural images). It imposes:
(0((2)) go(2) Vz E Z,
so second application of ga makes no change. This idempotence implies that once an off-manifold is mapped to go(=), it must already lie on the manifold {a go(x) = x). In effect,
(2) E {o ( =x
One can interpret this projection: drift or energy measure 6x(x)= go (x) || vanishes (f) = if and only if already lies on that manifold. Enforcing (g(((2)) g0(2) ensures 5e (g) = O. Hence, after one forward pass, the corrupted or noisy input is "pulled" onto the learned data manifold, and repeated applications do not alter it further.
Idempotence in 3 Pairwise Function.
3 deals with supervised model
fo N xx + V.
where X is input and y e y its desired output. The training set (xx,):) spans in-distribution Pa,y During training, 3 enforces:
1. fo(x, y) = y for training pairs (x, y). Thus, each real pair a fixed point.
13

---

IT: Idempotent Test-Time Training
2. O) y, using a "neutral" label 0 to predict y.
Combining these yields:
1o(x,0) = =fo(x,0) 0), 5o 50(x, xx,
an idempotence condition parallel IGN's go(g((z)) g0(2). One may define drift-like measure
Ax(x)
|| fo(x, ()) fo(x, 0)
When is in-distribution, training makes (x 0. If is OOD, 4x > 0 initially. Test-time adaptation then updates f on-the-fly push (x) closer to zero, thereby restoring idempotence.
Subtlety: Internal Representations Projection
natural question arises: If the OOD variable x itself stays fixed, how can (x, y) become "on distribution". "? The answer that, inside the network layers, and jointly produce hidden representations. Although x does not physically change, the way participates in the representation does change once and the model parameters 0 are updated. Thus, even if x is not from the training distribution, the pair (x, 3) can enter region of representation space that matches valid training pairs. Formally, each layer of fo has activations that depend on both x and y. By adjusting 0 (but freezing the outer function call) so that
fo fa(x, fo(x,,)) = fo(x,
we effectively project (x, 0) into the manifold {(x,y) fo(x, y) y within the network's internal representation. Hence, even though x remains the same, the final pair (x. y) "valid" the sense that repeated applications are stable.
Conclusion: I Also "Projects" Off-Manifold Pairs.
In IGN, once trained, any input z E Z maps to go(2) on the real-image manifold (go (x) - x). In IT a new OOD pair (x, 0) adapted that (x. fo (x, 0)) belongs the set {(x,y) fe(x. y) = yy From an internal representation viewpoint, this pulls the OOD pair onto the manifold of valid (x,y) relations. Thus, IT extends IGN's core idea of "learned idempotent projection" to supervised test-time training paradigm. Despite leaving intact, the final output indeed corresponds to consistent (x, y) -pair the model's manifold, much like IGN pulls corrupted noise into the real-data manifold.
Appendix C: Detailed Elaboration: Relation between Adaptation and Idempotence
In this appendix, we provide detailed explanation of the rationale behind the idempotence loss used in IT3. Our aim is to demonstrate how the discrepancy between recursive model outputs quantifies the out-of-distribution (OOD) uncertainty and why actively minimizing this discrepancy drives the model toward idempotence with respect its auxiliary input.
Training via the ZigZag Approach
Following (Durasov al., 2024a), the network is modified accept an auxiliary input. During training, for each training pair (x. y). the model f is trained to satisfy:
f(r,0) and f(a = v.
This is achieved by minimizing composite loss: Tarai (s, 0) wll + [1, v) y|1-
(9)
This training strategy ensures that incorporating the auxiliary input (either the true label or "don't 't know" signal) does not hurt the primary task performance while establishing a consistency property in the network.
14

---

IT: Idempotent Test-Time Training
2. Test-Time Recursive Inference and Uncertainty Measurement
At test time, the network is applied recursively:
9o = f(r, 0),
(10)
31 = f(z, yo).
(11)
We define the uncertainty loss as:
(x)a(x) = =1ly1 - yoll
(12)
The rationale is follows:
1. If is in-distribution, then yo se y. and since the network is trained so that f(r, se y, we have y/ yo. Therefore, Lyra (x) is small.
2. If is OOD, then yo is unlikely approximate the true label. In this case, the pair (Ir, yo) is not valid input as per training, leading 1/1 to be unpredictable and significantly different from yo, resulting in a large Lyr
Thus, the magnitude of |/y: yoll serves as a proxy for prediction certainty.
3. From Uncertainty to Idempotence
We now elaborate on how the uncertainty loss translates into enforcing idempotence of f(x,)) for given æ. Recall that function is idempotent if
f(f(x) = f(x).
(13)
For our setting, consider the function g(v) = far v). The desired idempotence condition becomes:
.(o(0)) = =g(0,
(14)
or equivalently,
f(x, f(x,0).
(15)
Thus, can rewrite the uncertainty loss as:
Lers(z) = [1((z, f(x, 0)) - f(x, 0)l-
(16)
Minimizing LiTa (x) drives the network toward the condition that repeated application of fzz does not change the output. When (x)= = 0, the function [ar, :) is idempotent given the input xr This self-consistency indicates that the network's output is aligned with the in-distribution manifold.
4. Addressing Optimization Challenges
Minimizing the idempotence loss is not trivial. Naively reducing Lyrs can lead to pitfalls such as reinforcing erroneous prediction Vo. As discussed in (Shocher et al., 2024), directly optimizing the idempotence loss induces two gradient pathways:
1. A desirable pathway that updates f(r, 0) toward the correct in-distribution manifold.
2. An undesirable pathway that may cause the manifold to expand, thereby including an incorrect f(x, 0).
To counteract the latter, our method avoids passing gradients through the second application of f by using a frozen copy of the network (or updating it via an exponential moving average). Specifically, we compute:
F(z,fzz 0)),
(17)
and redefine the loss as:
LIT [(x, (2x,0)l
(18)
This decoupling ensures that only the first prediction yo is adapted during test-time training, thus preventing error reinforce- ment and ensuring that the minimization of the loss indeed pulls fxx toward idempotence.
15

---

IT: Idempotent Test-Time Training
Image
TTT (batch= 16) Qual=30.08
ActMAD (batch=16) Qual=31.22
rT (batch=16)
Label
Qual=39.23
Qual=30.63
Qual=31.66
Qual=55.79
Qual=16.98
Qual=16.11
Qual=40.64
Qual=19.03
Qual=26.47
Qual=59.27
FR
X
Qual=63.32
Qual=69.84
Qual=77.66
Qual=53.72
Qual=50.39
Qual=77.17
Figure 16. Comparison of different Time Training methods segmentation tasks. This plot shows the predictions of various TTT methods on an segmentation task. As can he seen, our approach consistently enhances prediction quality and outperforms other methods.
16

---

IT: Idempotent Test-Time Training
5. Summary
By minimizing the loss
LIra f f(z,
(19)
we enforce the condition
fa, f(x, 0)) ~ (az
(20)
ic., (x, becomes idempotent given the input æ. This idempotence a critical indicator that the input is aligned with the training distribution. When LITS is small, the prediction is self-consistent and reliable; when is large, it signals that the input is likely OOD. Consequently, actively minimizing this loss through test-time training refines the network's 's prediction and enhances its robustness to distribution shifts.
For further details on related projection perspectives, please refer to Appendix B.
17


Reviews:
[{"review_id": "ga89mFaZ2z", "reviewer": "Reviewer_WBsB", "summary": "The authors:\n1. Make the claim that enfocing idempotence is benificial for test-time training tasks. \n2. Design a paradigm that brings an auxiliary signal representing ground truth, and force the network to learn idempotence through minimizing $\\Vert f_{\\theta}(x, y) - y \\Vert + \\Vert f_{\\theta}(x, 0) - y \\Vert$.\n3. Conduct experment on various dataset and shows promising results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. Below we address all the comments, and will revise the camera-ready version accordingly.\n\n&nbsp;\n\n### **\"The paper's presentation lacks clarity. Specifically, one of its key claim is \"if such a network is trained so that f(x, 0) = f(x, y) = y, then at test time the distance ||f(x, f(x, 0))−f(x, 0)|| correlates strongly with the prediction error\". However, the authors only skim through it, which disrupts the logical flow.\"**\n\nThank you for identifying this important issue. We acknowledge that the paper lacks a clear logical flow connecting idempotence to OOD adaptation. We provide here **[A SKETCHED LOGICAL CHAIN](https://imgur.com/XwHPhbc)** explaining exactly this. The introduction will be rewritten accordingly and a detailed extended version will be added to the paper as appendix C. In short:\n\n1. When using ZigZag approach to training, the discrepancy between recursive calls of the model, $|| y_1 - y_0 ||$, is highly correlated to how far out-of-distribution (OOD) an input is.\n2. Thus, we can use $|| y_1 - y_0 ||$ as a loss, and reduce 'OODness'.\n3. By doing this, we make the network more idempotent: $L_{\\mathrm{IT^3}} = || y_1 - y_0 || = || f(x, f(x,0)) - f(x,0) ||.$\n4.  But properly minimizing $L_{\\mathrm{IT^3}}$ is non trivial, so we employ the approach from IGN (Shocher et al.)\n5. This can be understood as a projection onto a distribution. (Appendix B).\n \n&nbsp;\n\n### **\"I'm interested to see ablation studies examining the impact of enforcing idempotence within the TTT framework.\"**\n\nWe provide empirical evidence analyzing the impact of enforcing idempotence in Fig. 2 of the main paper. It shows histograms of the distance from idempotence across four scenarios. For the original training and validation data, the model demonstrates very high idempotence (with the validation set slightly lower than the training set). For Out-of-Distribution (OOD) data, the model initially shows near-random idempotence. However, after applying our TTT optimization, the distribution of OOD data is significantly \"shifted\" from this random state towards that of the training and validation sets. This supports our claim that optimizing for idempotence indeed makes the model behave on OOD data similarly to how it behaves on in-distribution data.\n\nTo further investigate this, we conducted additional experiments on ImageNet-C, measuring idempotence loss across various corruption types and severity levels. **[(SEE ANALYSIS FIGURE HERE)](https://imgur.com/a/idempotence-vs-imagenet-performance-ym0McKq)**. Each point in the plot represents a batch of data; the x-axis shows the computed idempotence loss for that batch, while the y-axis indicates its classification accuracy. The data covers 15 corruption types and 5 severity levels, which are visualized using the colorbar. As expected, higher severity levels of corruption correspond to larger idempotence loss, further validating the effectiveness of enforcing idempotence within the TTT framework and highlighting its strong negative correlation with model performance (Pearson correlation: –0.94).\n\n&nbsp;\n\n### **\"Section 2.1: The argue \"TTT operates per-instance, with no assumption that future test data will be similar. So previous work have treated TTA and TTT as distinct paradigms\" seems weak. As some works, e.g. [1] and [2] do consider TTT under online setting.\"**\n\nWe conducted a new large-scale experiment on ImageNet-C that also includes comparisons to TTA methods TENT and ETA across various batch sizes. **([SEE RESULTS HERE](https://imgur.com/a/imagenet-corruptions-robustness-bGaru0u))** These will be added to the main paper in the camera-ready version.\n\nAs the reviewer points out, there is previous work [1, 2] about the online setting that we present in sections 3.3 and 4.6. However, unlike in [1,2], ours is intended for continual learning. Furthermore, we want to make a distinction between TTA, along with its many variants, and online-TTT: While the latter is used  in a data-streaming context the former is used on a given dataset or batch, or the training data at hand.\n\n&nbsp;\n\n### **\"In line 235 in the Experiments section: It's unclear which auxiliary task 'the original TTT method' refers to or whether other auxiliary tasks were evaluated.\"**\n\nOur apologies for this lack of clarity. By original TTT, we meant Sun et al. 2020. The auxiliary task used in their work is orientation prediction for images, with validation on image recognition benchmarks. We also compare to ActMAD (Mirza et al., 2023), which we adapted for other types of data and tasks.\n\n&nbsp;\n\n### **\"The authors should reorganize the presentation, as detailed in my \"Other Strengths And Weaknesses\" section.\"**\nThank you for helping us improve the presentation of our paper. We will reorganize accordingly.", "claims_and_evidence": "All of the claims are generally adequate.", "methods_and_evaluation_criteria": "The evaluation benchmark is comprehensive and provides convincing evidence to show the effectiveness of proposed method.", "theoretical_claims": "I checked the derivation and discussion of idempotence in main paper and appendix, and they are adequate.", "experimental_designs_or_analyses": "The experiment covers various aspect of tasks, including image recognition, tabular prediction, which is overall convincing promising.\nOne issue, however, is line 235 in Experiments section: \"we include the original TTT method and a newer more versatile approach\". It’s unclear which auxiliary task \"the original TTT\" refers to, and whether other auxiliary tasks were evaluated. I hope the authors can clarify this point.", "relation_to_broader_scientific_literature": "This paper discusses and brings a now paradigm for test-time training. Which might be instructive for future work and paradigm design in this field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strenths:\n\n1. The idea of bring idempotence to test-time training is innovative. \n2. The experiments is comprehensive. \n\nWeaknesses: \n1. The paper's presentation lacks clarity. Specifically, one of its key claim is \"if such a network is trained so that f(x, 0) = f(x, y) = y, then at test time the distance ||f(x, f(x, 0))−f(x, 0)|| correlates strongly with the prediction error\". However, the authors only skim through it, which disrupts the logical flow. Although efforts are made on explaning how to keep idempotence during pretraining and test-time training phase, they should provide further explanation and intuition on why enforcing idempotence is beneficial for enhancing performance. \n2. Similarly to point (1), I'm interested to see ablation studies examining the impact of enforcing idempotence within the TTT framework.", "other_comments_or_suggestions": "Section 2.1: The argue \"TTT operates per-instance, with no assumption that future test data will be similar. So previous work have treated TTA and TTT as distinct paradigms\" seems weak. As some works, e.g. [1] and [2] do consider TTT under online setting.\n\n[1] Gandelsman, Y., Sun, Y., Chen, X., and Efros, A. Test-time training with masked autoencoders.\n\n[2] Renhao Wang, Yu Sun, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang. Test-time training on video streams.", "questions_for_authors": "The authors should reorganize the presentation, as detailed in my \"Other Strengths And Weaknesses\" section. For the remaining points, the strengths and weaknesses are clear and straightforward in my view, so I probably won't change my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s0HmStViAU", "reviewer": "Reviewer_4J9m", "summary": "This paper proposes a novel test-time learning objective based on idempotent learning. The pipeline is easy to use, appears to be task-agnostic and model-agnostic, and is thus more versatile than previous TTA solutions. Experiments across various tasks demonstrate the effectiveness of the proposed method. My detailed comments are as follows.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and encouraging feedback. We address the reviewer's concerns below and will revise the paper accordingly. \n\n&nbsp;\n\n### **\"The authors claim that their method does not require designing additional auxiliary tasks or using extra data, making the adaptation on-the-fly. However, from my perspective, enforcing idempotency during training can also be considered an auxiliary task,...**\n\nIndeed, enforcing idempotence can be seen as an auxiliary task. However, it is not a domain-specific one. This is in contrast to approaches that use auxiliary tasks that are specific for data-type. Take for instance Sun et al., or Gandelsman et al. Their visual auxiliary tasks cannot be used on other types of data. Some others are architecture-dependent e.g., ActMAD (Mirza et al., 2023). Contrary to existing work, the IT$^3$ mechanism can be used out of the box for any task (we demonstrate image classification, aerodynamics prediction, aerial segmentation and tabular data) and any architecture (MLPs, CNNs, GNNs).\n\n&nbsp;\n\n### **...and it also requires access to training data for supervised learning. Could the authors clarify this further?\"**\n\nDuring the pre-training phase, IT$^3$ uses the training data like any other supervised-learning method. At test time, the challenge is defined so that there is absolutely no access to the training data. At any point, the only thing given is the current test input.\n\n&nbsp;\n\n### **\"For classification results, more comparisons with advanced fully test-time adaptation (TTA) methods, such as TENT, ETA, and DEYO, as well as evaluations on larger datasets like ImageNet-C, would strengthen the justification for the proposed method’s superiority.\"**\n\nFollowing the reviewer’s suggestion, we conducted additional large-scale experiments on **ImageNet-C**, comparing our method against popular TTA baselines including **TENT, ETA, MEMO and ActMAD**. The results are **[PRESENTED IN THIS PLOT](https://imgur.com/a/imagenet-corruptions-robustness-bGaru0u)**, which will be included in the final version of the paper.\n\nAs shown in the plot, our method consistently outperforms the baselines across all severity levels and batch sizes, achieving a 3–5% improvement in accuracy. These results further highlight the effectiveness and scalability of our approach.\n\n&nbsp;\n\n### **\"For segmentation results, could the authors include additional evaluations on commonly used benchmarks such as KITTI-C and nuScenes? This would enable a fairer comparison, as the dataset used in this paper has not been widely adopted in previous works.**\"\n\nThank you for pointing out these benchmarks. They are relevant. We missed them because prior TTT works don't use them. We intend to remedy this when revising but, unfortunately, training on them would take more time than we have for this rebuttal. Thus, to provide additional results during the rebuttal period, we prioritized experiments on ImageNet-C, as discussed in our response to the previous comment.\n\n&nbsp;\n\n### **\"The error bars in Figures 3, 4, and 8 are difficult to recognize and distinguish, particularly for color vision-deficient readers.\"**\n\nIn the revised version, we will improve the clarity and accessibility of the error bars, following the approach used in **[PLOT](https://imgur.com/a/imagenet-corruptions-robustness-bGaru0u)**. Specifically, we made the error bars bolder and more visible, used brighter, more distinguishable colors, and added explicit labels to differentiate the bars. These improvements will be applied to all relevant figures in the paper, along with better-written captions to clearly explain the contents of each plot and guide interpretation—ensuring improved readability, including for color vision-deficient readers.\n\n&nbsp;\n\n### **\"Additionally, could the authors provide more qualitative results instead of bar charts? The absolute differences in bar heights are not intuitive to interpret.\"**\n\nWhen revising, we will add an Appendix. D with some more qualitative segmentation results for the aerial photography road segmentation challenge. You can view these qualitative results via the **[FOLLOWING THIS ANONYMOUS LINK](https://imgur.com/a/JCOWYgq)**.\n\nIn the figure, we display the segmentation predictions of TTT, ActMAD, and IT$^3$ on OOD data. Quality scores are added on top of each prediction because they offer an objective measure of segmentation performance, helping to interpret the results more intuitively.\n\n&nbsp;\n\n### **\"Could the authors provide a computational complexity analysis comparing their method with existing baselines?\"**\n\nRuntime analysis can be found in Appendix A. When revising, we will add analysis of memory consumption to all cases as well. \nOne case for batch size 128 ImageNet-C, peak memory reserved with torch.cuda.max_memory_reserved() in GB:\n\nTENT: 4.8   \nETA: 4.9  \nMEMO: 13.5  (3 augmentations)  \nActMAD: 7.2  \nIT$^3$: 7.4  \nVanilla Model: 4.5", "claims_and_evidence": "The claim \"enables on-the-fly adaptation to distribution shifts using only the current test instance, without any auxiliary task design.\" needs to be further clarified. Please refer to my Questions part.", "methods_and_evaluation_criteria": "Yes, The designs make sense.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "2. For classification results, more comparisons with advanced fully test-time adaptation (TTA) methods, such as TENT, EATA, and DEYO, as well as evaluations on larger datasets like ImageNet-C, would strengthen the justification for the proposed method’s superiority.\n\n3. For segmentation results, could the authors include additional evaluations on commonly used benchmarks such as KITTI-C and nuScenes? This would enable a fairer comparison, as the dataset used in this paper has not been widely adopted in previous works.", "relation_to_broader_scientific_literature": "The problem studied is a fundamental challenge in machine learning with great potential for practical applications.", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "++Pros\n\nThe idea of leveraging idempotent learning for test-time adaptation (TTA) is interesting. This learning pipeline is not restricted to specific model architectures (e.g., MLP, CNN, Transformer) or specific tasks (e.g., classification, regression, segmentation).\n\nThe overall method is technically sound, and simple yet effective.\n\nExtensive experiments on classification, tabular data regression, age prediction, and road segmentation demonstrate the promise of the proposed approach.\n\nThe paper is well-written and easy to follow.", "other_comments_or_suggestions": "5. The error bars in Figures 3, 4, and 8 are difficult to recognize and distinguish, particularly for color vision-deficient readers. Additionally, could the authors provide more qualitative results instead of bar charts? The absolute differences in bar heights are not intuitive to interpret.", "questions_for_authors": "1 The authors claim that their method does not require designing additional auxiliary tasks or using extra data, making the adaptation on-the-fly. However, from my perspective, enforcing idempotency during training can also be considered an auxiliary task, and it also requires access to training data for supervised learning. Could the authors clarify this further?\n\n4 Could the authors provide a computational complexity analysis comparing their method with existing baselines?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "COcYtIG4G6", "reviewer": "Reviewer_tgc2", "summary": "This paper proposes a test-time adaptation (TTT) method named Idempotent Test-Time Training (IT3), that enforces idempotence to the model. Here, idempotence indicates that repeating a function to end in a stationary point (or fixed point). Specifically, IT3 predicts the output with a given input and randomly shown label, where at test-time the model is trained to minimize the ‘first prediction with null label as input’ and the successive model’s prediction with the self-predicted label. The method is tested across multiple domains and architectures, showing promising results.\n\n----\n\n**POST REBUTTAL**\n\nFor a short summary, I will raise the score to weak acceptance, but would like to highlight a small concern that I have.\n\nMost of my concerns are addressed, leaving only one concern. \"Why does this proposed method improve OOD robustness\"?.\n\nFrom a personal perspective as a reviewer (or a reader), the rebuttal did not fully clarify my concern and question. I think Idempotent somewhat makes sense to improve OOD robustness, as OOD makes the model struggle more to have a consistent output. However, it is also somewhat unclear why making Idempotent consistent on the in-distribution sample improves OOD robustness. I think this question is raised by other reviewers as well (e.g., WBsB).\n\nBut also from the author's perspective (I have also done some research on this domain), OOD robustness is hard to prove and sometimes an empirical science. So if the OOD robustness claim is toned down a little, it would be much helpful.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their detailed and constructive feedback. We address the reviewer's concerns below and will revise the paper accordingly. \n\n&nbsp;\n\n### **\"Lack of justification for IT3’s generalization ability to out-of-distribution (OOD) data. While the paper presents IT3 as an effective method for handling OOD data scenarios, the justification for why enforcing idempotence leads to better OOD generalization. The intuition behind this connection needs to be more explicitly stated, and additional ablation studies could help clarify why this approach is effective beyond empirical results.\"**\n\nThank you for identifying this important issue. We acknowledge that the paper lacks a clear logical flow connecting idempotence to OOD adaptation. We provide here **[A SKETCHED LOGICAL CHAIN](https://imgur.com/a/imagenet-corruptions-robustness-bGaru0u)** explaining exactly this. This will be combined into the introduction in the paper. A detailed extended version will be added to the paper as appendix C. A very brief summary is below:\n\n1. When using ZigZag approach to training, the discrepancy between recursive calls of the model, $|| y_1 - y_0 ||$, is highly correlated to how far out-of-distribution (OOD) an input is.\n2. Thus, we can use $|| y_1 - y_0 ||$ as a loss, and reduce 'OODness'.\n3. By doing this, we make the network more idempotent: $L_{\\mathrm{IT^3}} = || y_1 - y_0 || = || f(x, f(x,0)) - f(x,0) ||.$\n4.  But properly minimizing $L_{\\mathrm{IT^3}}$ is non trivial, so we employ the approach from IGN (Shocher et al.)\n5. This can be understood as a projection onto a distribution. (Appendix B).\n\n\nTo empirically support this claim, we also analyzed the relationship between corruption severity and idempotence loss on ImageNet-C. The results are shown **[IN THIS LINK](https://imgur.com/a/idempotence-vs-imagenet-performance-ym0McKq)**. Each point represents a batch of data, with the x-axis showing the computed idempotence loss and the y-axis indicating classification accuracy. As expected, higher severity levels correspond to greater idempotence loss. Notably, we observed a strong negative correlation between idempotence loss and performance (Pearson correlation: –0.94).\n\n&nbsp;\n\n### **\"Unclear contribution of exponential moving average (EMA) to the performance gains.\"**\nThe online version, as described in sec. 3.3 differs from the base version because they aim at different scenarios. In the base version,  the assumption is that each input is a separate test and has no correlation or information about other inputs. Thus the network weights are reset back to the state they were at the end of the pre-training phase. In contrast, the online version is intended for a continual learning data stream with correlation and somewhat smooth transitioning between inputs. In this case, instead of resetting after each input, we leave the weights updated from the previous inputs. \n\nSince the data keeps shifting in this scenario, over time, $f_{\\theta}$ may diverge far from $F$, making it irrelevant. Instead, we need an anchor that is influenced by a reasonable amount of data, yet evolves over time. Our solution is to replace $F$ with an Exponential Moving Average (EMA) of the model. This means $f_{\\text{EMA}}$ is a smoothed version of $f_{\\theta}$ over time.\n\nNote that the online-IT$^3$ results, depicted in Table. 1 achieve higher scores than the base model. This is due to the inherent difference between the base and the online scenarios. A correlated data stream allows aggregating  information during test-time and therefore expected to exploit it and perform better.\n\n&nbsp;\n\n### **\"Limited large-scale experiments... it would be beneficial to include results from larger datasets such as ImageNet.\" / \"I think showing more high-dimensional (label) domain in the image domain will be great.\" / \"Missing baselines... lacks comparison with augmentation-based baselines.\"**\n\nFollowing the reviewer’s suggestion, we conducted additional experiments on **ImageNet-C** and compared also to TTA baselines:   **TENT, ETA and MEMO** across various batch sizes. The results are **[PRESENTED IN THIS PLOT](https://imgur.com/a/imagenet-corruptions-eI7JNfr)**, which will be included in the final version of the paper. For each method and severity level, we present bars composed of 15 accuracy values—one for each corruption type. Our method consistently outperformed with a 3–5% improvement.\n\n&nbsp;\n\n### **\"Computational overhead and fairness in comparisons... A fairer comparison should analyze compute overhead and consider normalized performance metrics that account for increased memory usage (or parameter count).\"**\n\nRuntime analysis can be found in Appendix A. We will add analysis of memory consumption to all cases as well. \nOne case for batch size 128 ImageNet-C, peak memory reserved with torch.cuda.max_memory_reserved() in GB:\n\nTENT: 4.8   \nETA: 4.9  \nMEMO: 13.5  (3 augmentations)  \nActMAD: 7.2  \nIT$^3$: 7.4  \nVanilla Model: 4.5", "claims_and_evidence": "While most of the claims are reasonable, I believe the key idea and engineering technique should be clarified, namely the justification of generalization ability of IT3 and the use of EMA. \n\n**Lack of justification for IT3’s generalization ability to out-of-distribution (OOD) data.** \\\nWhile the paper presents IT3 as an effective method for handling OOD data scenarios, the justification for why enforcing idempotence leads to better OOD generalization. The intuition behind this connection needs to be more explicitly stated, and additional ablation studies could help clarify why this approach is effective beyond empirical results.\n\n**Unclear contribution of exponential moving average (EMA) to the performance gains.** \\\nThe role of EMA in improving performance is not well justified. While it is used to stabilize adaptation in the online setting, the underlying reasons for its significant impact on performance are not well explored. I think this result is quite interesting and justifying well will improve the paper's claim.", "methods_and_evaluation_criteria": "The evaluation criteria are correct for the proposed method. But I do believe the paper needs more baseline and experiments to fully support the paper's claim.\n\n**Limited large-scale experiments.** \\\nThe experiments primarily focus on CIFAR-10 (for image) and other relatively small-scale datasets. To convincingly demonstrate scalability, it would be beneficial to include results from larger datasets such as ImageNet. If it is hard to run ImageNet during the rebuttal it would be great if the author can run ImageNet subset. Without this, it remains unclear how well IT3 scales to more complex real-world settings.\n\n**Missing baselines.** \\\nThe author compares IT3 with some TTT methods but lacks comparison with augmentation-based baselines. While it is true that augmentation-based baselines require domain-specific knowledge, I believe most of the real-world applications actually know which augmentation to use at train/test time. In this regard, I believe it is great to compare with such baselines like \"MEMO: Test Time Robustness via Adaptation and Augmentation.\" [1] Additionally, batch normalization-based approaches such as \"Tent: Fully Test-time Adaptation by Entropy Minimization\" [2] should be included, especially since batch normalization layers can be updated without requiring domain knowledge. BN-based method can be used without architecture modification, but this method requires architecture modification. Also,  comparison with \"single-point BN\" used in MEMO would also be relevant. \n\n**Computational overhead and fairness in comparisons.** \\\nIT3 requires maintaining two networks (original and EMA), which increases memory and computational requirements. This could lead to unfair comparisons with single-network baselines that do not require additional storage for EMA parameters. A fairer comparison should analyze compute overhead and consider normalized performance metrics that account for increased memory usage (or parameter count).\n\nReference\\\n[1] MEMO: Test Time Robustness via Adaptation and Augmentation, NeurIPS 2022\\ \n[2] Tent: Fully Test-time Adaptation by Entropy Minimization, ICLR 2021", "theoretical_claims": "There is no theoretical claim in the paper.", "experimental_designs_or_analyses": "I have checked the soundness and validity of experimental design and analysis, which seems correct. I am not claiming that the experiment is sufficient rather claiming that the presented experiment seems correct (see other parts to find the weakness).", "relation_to_broader_scientific_literature": "I think the main contribution of this paper is introducing a new idea (i.e., Idempotence) to test-time training which is quite interesting. If the author could well connect the reason why Idempotence helps generalization to OOD dataset, it would be very helpful.", "essential_references_not_discussed": "The paper presented a good related work section.", "other_strengths_and_weaknesses": "For strength, I think the overall paper is well written and clearly discusses the difference between test-time adaptation and test-time training. Also, while I still do think the domain agnostic needs comparison with domain specific methods (e.g., augmentation-based), the paper did a great job to consider multiple domains (e.g., Tabular) that lack domain specific knowledge.\n\nOther weakness are presented in other sections. I think the main weakness is the justification of Idempotent for OOD generalization. I kindly ask the authors to address this issue during the rebuttal.", "other_comments_or_suggestions": "**Impact of label dimension (|y|) on performance should be studied.** \\ \nSince the IT3 framework explicitly requires designing a network where y is part of the input, it is important to analyze how performance varies as the number of classes increases. The current experiments mainly focus on small-class problems like CIFAR-10 (in image), but it is unclear whether the method remains effective when dealing with datasets with higher label dimensionality. I think showing more high-dimensional (label) domain in the image domain will be great, as most readers and reviewers are more familiar with image baselines.", "questions_for_authors": "All questions are asked in other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U42aaP2gqn", "reviewer": "Reviewer_Viky", "summary": "This paper introduces IT3, a test-time training method that leverages idempotence to adapt model weights on-the-fly without requiring domain-specific auxiliary tasks. By enforcing that repeated applications of the model yield the same output, IT3 effectively projects out-of-distribution inputs onto the training data manifold, achieving improved robustness across diverse tasks and domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful comments. We address the reviewer's concerns below and will revise the paper accordingly. \n\n&nbsp;\n\n### **\"The experiments primarily involve smaller datasets and omit standard OOD benchmarks like WILDS, leaving broader applicability and real-world relevance underexamined.\" / \"limited set of baselines\" / \"The tasks used in experiments are relatively small-scale\"How does IT³ scale in terms of memory and runtime when model sizes grow, and have you tested it on larger networks (e.g., ImageNet-scale) to confirm its practicality?\"**\n\nFollowing the reviewer’s suggestion, we conducted additional experiments on **ImageNet-C** and compared also to TTA baselines:   **TENT, ETA and MEMO** across various batch sizes. The results are **[PRESENTED IN THIS PLOT](https://imgur.com/a/imagenet-corruptions-robustness-bGaru0u)**, which will be included in the final version of the paper. For each method and severity level, we present bars composed of 15 accuracy values—one for each corruption type. Our method consistently outperformed with a 3–5% improvement. While WILDS  is clearly relevant, prior TTT works did not evaluate on it and we missed it. In the limited rebuttal time we had, we prioritized ImageNet-C and additional baselines experiments. We will test on standard OOD benchmarks when revising the paper.\n\nRuntime analysis can be found in Appendix A. When revising, we will add analysis of memory consumption to all cases as well. \nOne case for batch size 128 ImageNet-C in GB:\n\nTENT: 4.8   \nETA: 4.9  \nMEMO: 13.5  (3 augmentations)  \nActMAD: 7.2  \nIT$^3$: 7.4  \nVanilla Model: 4.5\n\n&nbsp;\n\n### **\"... does not include rigorous derivations confirming that test-time adaptation preserves idempotence on OOD samples.\"**\n\nThank you for identifying this important issue. We acknowledge that the paper lacks a clear logical flow connecting idempotence to OOD adaptation. We provide here **[A SKETCHED LOGICAL CHAIN](https://imgur.com/XwHPhbc)** explaining exactly this. While not a formal proof,  it is a rigorous description of this relation. This will be combined into the introduction in the paper.  A detailed extended version will be added to the paper as appendix C. A very brief summary is below:\n\n1. When using ZigZag approach to training, the discrepancy between recursive calls of the model, $|| y_1 - y_0 ||$, is highly correlated to how far out-of-distribution (OOD) an input is.\n2. Thus, we can use $|| y_1 - y_0 ||$ as a loss, and reduce 'OODness'.\n3. By doing this, we make the network more idempotent: $L_{\\mathrm{IT^3}} = || y_1 - y_0 || = || f(x, f(x,0)) - f(x,0) ||.$\n4.  But properly minimizing $L_{\\mathrm{IT^3}}$ is non trivial, so we employ the approach from IGN (Shocher et al.)\n5. This can be understood as a projection onto a distribution. (Appendix B).\n \nTo empirically support this claim, we also analyzed the relationship between corruption severity and idempotence loss on ImageNet-C. The results are shown **[IN THIS LINK](https://imgur.com/a/idempotence-vs-imagenet-performance-ym0McKq)**. Each point represents a batch of data, with the x-axis showing the computed idempotence loss and the y-axis indicating classification accuracy. As expected, higher severity levels correspond to greater idempotence loss. Notably, we observed a strong negative correlation between idempotence loss and performance (Pearson correlation: –0.94).\n\n&nbsp;\n\n### **\"Briefly discuss tuning hyperparameters (e.g., optimization steps, learning rate) for test-time adaptation to guide practical deployment.**\"\n\nThank you for this important comment. We will expand appendix A to include implementation details. Code will be released upon acceptance.\n\n&nbsp;\n\n### **\"Could you clarify how you prevent confusion between the neutral zero and legitimate zero labels?\"**\n\nWe thank the reviewer for their important comment. The special 0 notation used does not represent an actual zero. Instead, it is a unique signal with the same dimensions as the labels, specifically chosen to differentiate it from actual label values. Our approach builds upon the ZigZag method from [1], where the choice of this signal is extensively discussed and justified. We understand that this notation may appear confusing, so we will revise the paper to include a clearer explanation to avoid any ambiguity.\n\n&nbsp;\n\n### \"**If the model’s initial prediction is incorrect, even with the frozen reference model, the iterative optimization can magnify the error signal.\" / \"When the initial prediction for an OOD sample is substantially off, what mechanisms prevent... reinforcing incorrect outputs?\"**\n\nThis is a valid point. Even with the frozen network trick, it is not impossible for such event to occur. Empirically, such cases are rare in comparison to cases where the TTT session improved the prediction. The number of TTT iterations is very limited. The network is reset after each instance so no long term damage can really accumulate.", "claims_and_evidence": "Most claims are supported by extensive experimental evidence, but the theoretical justification of idempotence as a projection mechanism of the test-time training process are less substantiated.", "methods_and_evaluation_criteria": "While the proposed methods align conceptually with test-time training for handling distribution shifts, the experiments primarily involve smaller datasets and omit standard OOD benchmarks like WILDS, leaving broader applicability and real-world relevance underexamined.\n\nWilds: A benchmark of in-the-wild distribution shifts. ICML, 2021.", "theoretical_claims": "The paper’s theoretical justification relies largely on intuitive arguments relating idempotence to test-time adaptation, rather than providing formal proofs. While the extended discussion references prior work (e.g., IGN), it does not include rigorous derivations confirming that test-time adaptation preserves idempotence on OOD samples, leaving this point insufficiently substantiated.", "experimental_designs_or_analyses": "I examined the experimental design, and a key concern is the limited set of baselines: while the paper occasionally compares IT³ to methods like ActMAD, it does not consistently benchmark against TTT++ across all tasks with matching batch sizes and architectures.\n\n\nTTT++: When Does Self-Supervised Test-Time Training Fail or Thrive?, NeurIPS 2021", "relation_to_broader_scientific_literature": "The paper extends the emerging line of work on test-time adaptation (e.g., TTT, ActMAD) by introducing idempotence as a unifying principle that obviates the need for domain-specific auxiliary tasks.", "essential_references_not_discussed": "No essential references appear to be missing beyond those already mentioned.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The approach leverages a general idempotence principle rather than domain-specific tasks, making it broadly applicable.\n2. It consistently demonstrates improved performance on diverse tasks (e.g., tabular data, image classification, aerodynamics).\n3. The method is relatively straightforward to plug into existing architectures and does not require extensive auxiliary supervision.\n\nWeaknesses:\n\n1. If the model’s initial prediction is incorrect, even with the frozen reference model, the iterative optimization can magnify the error signal.\n2. The tasks used in experiments are relatively small-scale or synthetic, leaving questions about real-world robustness.", "other_comments_or_suggestions": "1. Provide more detail on using zero as a neutral signal in regression tasks, where zero may also be a valid label.\n2. Briefly discuss tuning hyperparameters (e.g., optimization steps, learning rate) for test-time adaptation to guide practical deployment.", "questions_for_authors": "1. Could you clarify how you prevent confusion between the neutral zero and legitimate zero labels in regression tasks, and whether an alternative placeholder value might be more appropriate?\n\n2. How does IT³ scale in terms of memory and runtime when model sizes grow, and have you tested it on larger networks (e.g., ImageNet-scale) to confirm its practicality?\n\n3. When the initial prediction for an OOD sample is substantially off, what mechanisms (if any) prevent the model’s iterative updates from further reinforcing incorrect outputs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a method for test-time training. The main claim is in the assumption of idempotence as the primary mechanism for updating the model parameters at test time and builds on existing work [A, B] and extends it towards test-time training. 

While the reviewers came to a positive consensus, we note that the paper bares similarity to [C], where both utilizes the outcome from an uninformative input as an auxiliary supervision signal and enforces consistency as the primary mechanism for adaptation. However, [C] was not mentioned. 

We also note that reviewer tgc2 had doubts on whether it made sense that the approach should improve OOD robustness and noted that the results are largely empirical, tgc2 ultimately improved their score despite the concern.

[A] Shocher et al.  Idempotent generative network. ICLR 2024.

[B] Durasov et a. Zigzag: universal sampling-free uncertainty estimation through two-step inference. Transactions on Machine Learning Research, 2024. 

[C] Park et al. Test-time adaptation for depth completion. CVPR 2024.
Decision: Accept (poster)