Single Image Test-Time Adaptation for Segmentation
Authors: Klara Janouskova
Tamir Shor
Chaim Baskin
Jiri Matas
Venue: Accepted by TMLR ()

Abstract: Test-Time Adaptation methods improve domain shift robustness of deep neural networks. 
We explore the adaptation of segmentation models to a single unlabelled image with no other data available at test time. This allows individual sample performance analysis while excluding orthogonal factors such as weight restart strategies. We propose two new segmentation \ac{tta} methods and compare them to established baselines and recent state-of-the-art. The methods are first validated on synthetic domain shifts and then tested on real-world datasets. The analysis highlights that simple modifications such as the choice of the loss function can greatly improve the performance of standard baselines and that different methods and hyper-parameters are optimal for different kinds of domain shift, hindering the development of fully general methods applicable in situations where no prior knowledge about the domain shift is assumed.

Chaim Baskin Published in Transactions on Machine Learning Research (05/2024)
Single Image Test-Time Adaptation for Segmentation
Klara Janouskova
kara.jjmoonskoruu@ddd.cc.....
Visual Recognition Group, Faculty of Electricol Engineering Czech Technicol University in Progue
Tamir Shor
temihsshor@@aampp..eeriion...
Technion Israel Institute of Technology, Hoifa, Israel
chaimboskinnttechniinn....
Technion Israel Institute of Technology, Hoifa, Israel
Jiri Matas
matas@fel.ccut..cc
Visual Recognition Group, Faculty of Electricol Engineering
Caech Technical Universiby in Prague
Reviewed on OpenReview: https: openreview. net/ forun? id= 68L WMLsWD
Abstract
Test-Time Adaptation (TTA) methods improve domain shift robustness of deep neural net- works. We explore the adaptation of segmentation models to a single unlabelled image with other data available at test time. This allows individual sample performance analysis while excluding orthogonal factors such weight restart strategies. We propose two segmentation TTA methods and compare them to established baselines and recent state- of-the-art. The methods are first validated on synthetic domain shifts and then tested on real-world datascts. The analysis highlights that simple modifications such as the choice of the loss function can greatly improve the performance standard baselines. and that differ- ent methods and hyper-paramest the development of fully general methods applicable in situations where prior knowledge about the domain shift is assumed.
are optimal for different kinds of domain shift, hindering
Code and data:
www.:ooowwwo.iiiiuu....o.... sithub.oo/sitta-see
1 Introduction
A common challenge in machine learning stems from the disparity between source (training) and target (deployment) data domains. Models optimized to minimize error dataset from specific domain are often expected to perform reliably in different domains. The discrepancy between training and deployment data, known as the domain shift, is very common; in fact, few things do not change in time, and training happens (well) before deployment. A domain shift may substantially degrade model performance at de- ployment time despite proper validation on training data, yet is often not explicitly addressed and most machine learning effort has focused on the generalization problem.
In many practical scenarios, the characteristics of the larget domain are not known beforehand, making the preparation of the model with traditional domain adaptation Rodriguez & Mikolajczyk (2019); Tzeng et al. (2017) techniques nontrivial. Recent advances Wang et al. (2020b): Sun et al. (2020); Gandelsman et al. (2022) suggest that under certain weak assumptions about the domain shift. such as a stable label distribution across domains - it is possible to mitigate the performance degradation with methods based on the information carried by input data received at inference time.
Test- Time Adaptation (TTA) is suitable for priori unknown or difficult-o-ppredict domain shifts. Charac- terized unsupervised and source-free technique, TTA adapts the model directly during inference. The

---

Published in Transactions on Machine Learning Research (05/2024)
segmentation model training
test-time adaplation training
test/deplymment
source dataset S model selection
synthetic dataset S-C derived from S
real-workd domain shift datasets
SITTA hyperparameter tuning
SITTA
model SITTA
model
hyperparams
Figure 1: The proposed experimental framework for Single Image Test-Time Adaptation (SITTA). Hyper- parameters are found on & synthetic dataset derived from the training set by applying a diverse set of corruptions. SITTA methods are then tested on real-world datasets with domain shift.
source-free nature, i.e, without access the original training data, ensures compliance with data governance standards and enables adaptation in memory-constrained environments.
Single Image Test-Time Adaptation (SITTA) tailors a model at test time to each individual image. Since it operates a single image, it does not introduce assumptions about the stability of the data distribution over time. Each time starting from the weights fixed at training time, SITTA is safe to use when any form of memorization of the deployment data is prohibited. A disadvantage is an increased computational time compared to batch methods and the lack of retaining the acquired knowledge. On the other hand, SITTA could be leveraged by only processing the most informative samples. Despite the advantages, only Khurana et al. (2021) primarily addresses SITTA, the mainstream of TTA research deals with continual test-time adaptation in changing environment Niu et al. (2023); Volpi et al. (2022); Wang et al. (2022). These methods typically gradually update model parameters or accumulate image statistics for subsequent adaptation to individual images. Continual TTA strategies are practical in many applications, such as autonomous driving, but they are challenging from the point of view of accurate assessment of relative strengths and weaknesses. The difficulty ariscs from the evaluations conducted on many images with varying levels and kinds of domain shift. Moreover, the sequence in which images are presented can significantly influence performance metrics, adding layer of complexity to assessing the true efficacy of these methods. SITTA streamlines the evaluation process and contributes to understanding the broader class of TTA strategies. Continual TTA analysis has focused on complementary issues such as catastrophic forgetting.
reduce the computational cost of adapting to a sequence of similar images
In this paper, we explore and improve the state of the art in Single Image Test-Time Adaptation for image semantic segmentation. While TTA has been applied to many tasks; the overwhelming majority of methods have been developed for image classification. Image segmentation TTA methods are each evaluated under very different conditions and compared to limited set of baselines, making understanding their performance difficult. We focus semantic segmentation SITTA with self-supervised loss functions. These methods are broadly applicable across various segmentation models and domains without being constrained by specific architectural or segmentation training requirements. Consequently, the many works that rely on batch normalization layers are not included in this work as they are incompatible' with the modern transformer architectures currently dominating the field. Likewise, methods such as image-reconstruction based TTA Wang et al.; Gandelsman al. (2022) are not included due their significant demands for training process modification and model architecture adjustments. Some methods require access to training data before deployment since they train auxiliary network. However, they do not alter the training of the segmentation model.
Six TTA methods are evaluated in the SITTA setting. Two are established baselines: entropy minimization Hu et al. (2019) and pscudolabelling Lee et al. (2013) (self-training) They are the only ones evaluated in the SITTA setup before. The third method Prabhu et al. (2021) is a recently proposed segmentation TTA method that identifies confidently pscudolabelled pixels as consistent across augmentations. The fourth method Nguyen et al. (b) was proposed for image classification; we are the first to use it for image segmentation.
At inference time, batch normalization per-fcature and variance computed over the training datasct. TTA methods also incorporate the data feature statistics. Transformer models typically nver-noomalization layers, which normalize features for each instance independently and thus always the test sample statistics.
2

---

Published in Transactions on Machine Learning Research (05/2024)
The main idea is that if network is robust to small adversarial domain shifts (it is optimized to not change the prediction under an adversarial attack at test time), it will also be robust to real-world domain shifts.
The remaining two approaches are novel. We combine and extend two techniques optimizing self-supervised losses with 1. learnt segmentation mask domain discriminators Tzeng et al. (2017), so far used for unsuper- vised domain adaptation and never applied to TTA, and 2. Segmentation mask refinement 2 modules Karani et al. (2021); Valvano et al. (2021), never considered outside the medical domain TTA.
The idea of training-time unsupervised adaptation with mask discriminators is that a classifier trained to distinguish masks from the source and target domains can be used to supervise adaptation. The source- domain segmentation model is finetuned to produce masks on the target domain that resemble those of the source domain, as predicted by the discriminator. This method is not directly applicable to TTA since the target images to train the discriminator are unavailable. However, the target domain masks can be synthesized. Copying random patches into different image locations was proposed in the modical domain. Such augmentation makes sense when the textures are more important than the shapes of objects, such in the medical domain, but are not general enough for other images. We propose to use the output of the segmenter adversarially attacked images. The adversarial optimization learning rate and the number of iterations control the severity of the domain shift. Further, Karani et al. (2021) proposes to replace the mask discriminator with a denoising autoencoder. Since a discriminator can use shortcuts and only focus on the most discriminative parts of the masks, a model that produces a refined mask is trained to consider all the masks. We explore both options.
We address the problem that arises from the practice in the current landscape of segmentation TTA that the performance assessment is carried out with inconsistent adaptation settings. For instance, koeping batch normalization statistics constant or updating them during entropy minimization can yield substantially different outcomes. If only one of the two options is tested Khurana et al. (2021); Volpi et al. (2022), contradictory results are reported. Often, the evaluation compares only with the established baselines such as entropy minimization Wang et al. (2020a) and batch normalization Ioffe Szegedy (2015a) statistics adaptation Schneider et al. (2020); Nado et al. (2020), ignoring recent progress. Many methods that improve these baselines have been proposed in recent years for both segmentation and classification Nguyen et al. (a); Gao et al. (2023); Niu et al. (2023); Chen et al. (2022), and their relative merit is unknown, since a comprehensive comparison with well-defined methodology is lacking.
Our experimental framework, depicted in Figure 1, consists of three stages: 1. Segmentation model training, which is standard. 2. Test-time adaptation training which tunes TTA ayper-parameters. We utilize an augmented version of the training datasets. This extension incorporates synthetic corruptions inspired by Hendrycks & Dietterich (2019). The corruption types include different kinds of noise and blur, weather conditions such as fog or frost, jpeg compression, and basic image intensity transformations. It can be derived from an arbitrary segmentation training dataset, as opposed to existing synthetic datasets used by previous work such as Ros et al. (2016). It provides precise control over the conditions and facilitates detailed analysis. 3. Evaluation on real-world test datascts with domain shift.
We experiment with two existing pretrained models for semantic segmentation. The models have different architectures and are trained on various datasets. Almost all segmentation TTA methods are evaluated on driving scenes benchmarks- we follow this practice and add a benchmark on common objects.
The main contributions of this paper are:
We conduct a comparative study of six TTA techniques run in SITTA mode for image segmenta- tion: Two establishod baselines, two adapted sratc-of-thh-art methods from image classification and continual TTA, and two proposed mechods.
2Called "denoising autoencoders" in Karani et al. (2021).
"In the domain adaptation literature, the term "unsupervised" refers the situation when images from the target domain are available the method, but they are not labelled. such a setting, training sourco-target domain discriminator is possible, which contradicts the standard implication of the term "unsupervised". The setting where samples from the target domain are available during training referred to as "test-time adaptation" "unsupervised source-free adaption?.
3

---

Published in Transactions on Machine Learning Research (05/2024)
2. Novel methods adapting ideas from unsupervised domain adaptation and medical imaging TTA to non-medical image segmentation are introduced, filling a gap in exploring diverse self-supervised loss functions. The method outperforms the other methods on multiple test datasets and is shown to be powerful on the images with the worst segmentation performance, as measured by Intersection over Union (IoU).
3. Improvements of baselines in the single-image setup by replacing Choss-Entropy (CE) with the loU loss. The performance of pscudo-labelling is improved by 3.51 % and 3.28 % on GTA5-C and COCO- validation sels while with CE loss, the improvements are by 1.7 % and 2.16 % only, respectively.
4. The first work shows the potential of SITTA for segmentation, an underexplored setup essential in applications with strict data governance standards or high variability among individual images.
2 Background
Common approaches to domain adaptation change the style of labelled source images to resemble the training images Tzeng et al. (2017); Zhang et al. (2018) or train domain classifiers guide the adaptation process. In practice, this is not always feasible since source data may not be available for example for privacy memory limitation reasons, or may only have small number of target domain images available when data arrive individually/in /in small batches, rather than all at once. In continually evolving environments, the distribution may change by the time adaplation on & large larget dataset is completed. Various modifications of the traditional domain adaptation scenario tackling the aforementioned limitations have recently emerged, for example by considering no access to source data or a continual domain shift Liu et al. (2021a); Volpi et al. (2022); Wang et al. (2022); Bartler et al. (2022).
In particular, test-time adaptation methods assume no source data is available and aim to exploit the infor- mation from little as single target domain image. Like other domain adaptation methods, TTA meth are often inspired by semi-superviis on minimization of the predictions entropy, a method inspired by Saito et al. (2019). Other methods rely on adapting the batch normalization statistics, inspired by methods like adaptive batch normalization Li et al. (2018), or aggregating statistics to create so-called prototypes Tanwisuth et al. (2021) that can be used to build a classifier.
learning methods. For instance, the most common TTA baseline relies
Some works also distinguish between TTA and Test-Time Training (TTT). The difference between TTA and TTT is that TTA methods such as Nguyen et al. (b); Karani el al. (2021) can be applied to arbitrary pre-trained models without any additional constraints while TTT methods like Gandelsman et al. (2022); Bartlor et al. (2022); Liu et al. (2021b) require modifications to the training process. However, not all works make this distinction and the boundary is not always clear, as some methods like Karani et al. (2021) need to train an auxiliary deep net on the source data but do not modify the model pretrained weights. In this work, both will be jointly referred to as TTA for simplicity.
In Appendix A, other related domain adaptation scenarios and their relation to TTA are described.
Generally, TTA methods can be split into three groups: Adaptation in the input space, feature space and output space. Input space adaptation aims to translate the images from the source domain to the input domain. In practice, Gao et al. (2023) achieve this by feeding target images with added noise to a diffusion model trained on the source data, coupled with reconstruction guidance to preserve semantics. The model doesn't 't retain any knowledge from the adaptation the advantage is it is not susceptible to catastrophic forgetting but the disadvantage is it may limit the adaptation capabilities. Adaptation in the feature space is the most common approach and typically relies on optimizing the network parameters via sclf-supervised loss function. This can be done directly, i.c. through prediction entropy minimization, or by training an auxiliary lask such as image reconstruction. Another set of feature-adaplation approaches are parameter-free and rely on accumulating the image statistics, such as the mean and variance of image features, or by aggregating confident prodiction features into so-called prototypes, which are then used for classification. Output space adaptation techniques aim improve the network output without neither altering the network parameters and statistics the input image. This is done for instance in Karani et al.
4

---

Published in Transactions on Machine Learning Research (05/2024)
(2021) where auxiliary network is trained to predict refined mask. To the best of our knowledge, output space adaptation methods are typically only used to provide pseudo-masks, turning them into feature-space adaptation methods. This helps to iteratively improve the pseudo-masks and adapt to larger domain shifts. All the methods evaluated in this work can be considered as feature space adaptation methods, possibly via output space adaptation.
3
Related Work
Test-Time Adaptation methods for classification. Many recent methods propose improved strategies to update the batch normalization statistics Schneider et al. (2020); Nado et al. (2020). A limitation of these methods is the reliance on presence of batch uromalization, which is often not part of recent transformer-basee architectures. In Wang et al. (2020b), the learnable parameters of the normalization layers are also updated via entropy minimization. While this method is often reported as unstable since single-image statistics may not be sufficient, the method can also only update the normalization layers learnable parameters, without the statistics update, making it generalizable to all currently used architectures.
On classification tasks, many methods outperforming the aforementioned baselines have been proposed. A combination of self-supervised contrastive learning to refine the features and online label refinement with a memory bank is proposed in Chen et al. (2022). Recently, a method based on updating the parameters of the normalization layers of the network by optimizing it for robustness against ad vdresaiiaal perturbation as a representative of domain shift was proposed in Nguyen et al. (b), outperforming similar test-time adaptation approaches. Rotation prodiction is proposed in Sun et al. (2020) as self-supervised task to be learnt alongside the main one and then optimized al inference time. Lately, it was shown that reconstruction with masked auto-encoders is a very strong supervisd task for test-time adaptation of classifiers by Gandelsman et al. (2022).
Test-Time Adaptation methods for segmentation. To the best of our knowledge, the only work also focused on adaptation to a single isolated image Khurana et al. (2021) is based on computing the statistics from augmented version of the input image, assuming batch normalization layers are present in the network. Both Prabhu et al. (2021) and Wang et al. (2022) exploit augmented views of the input images to identify reliable predictions. The method of Prabhu et al. (2021) is based on the consistency of predictions between augmented views, which replaces prediction confidence for selecting reliable pixcls. Cross entropy loss is then minimized on such reliable predictions, together with regularization based on information entropy Li et al. (2020) to prevent trivial solutions. The method achieves impressive results, however, in contrast to our experiments, knowledge of the target domain shift is used for hyper-paramcter tuning. The evaluation assumed a full test set available at once, focusing on source-free domain adaptation, rather than TTA, but the method is applicable to the TTA setup as well. In Volpi et al. (2022), the performance of entropy minimization in a continual setup is explored, proposing parameter restart to tackle weight drift, significantly improving performance. The focus is on driving datasets only. Similarly, Wang et al. (2022) also focus continual adaptation. Again, augmentations of the images are generated to obtain more reliable predictions. Further, the network parameters are stochastically reset to their initial values to prevent forgetting of the source domain knowledge.
Test-Time Adaptation methods for medical imaging. In Karani et al. (2021), an autoencoder is proposed that translates predicted masks into refined mask. At test time, the segmenter optimized produce masks closer to the enhanced ones. However, this work assumes the whole test dataset is available at once, in contrast to our single- image setup. The work of Valvano et al. (2021) is similar to Karani et al. (2021) but instead of a masked-autoencoderr, a GAN-like discriminator trained end-to-end together with the segmenter is used, as well as an auxiliary reconstruction loss.
These works assume domain shifts specific to the medical imaging domain such as the use of a different scanner and thus make the assumption that only low-level features are affected. Under this assumption, these works typically optimize small adapter only, ie. the first few convolutional layers of the segmenter. Nonetheless, these methods are generalizable to image segmentation.
5

---

Published in Transactions on Machine Learning Research (05/2024)
Enhancing existing TTA benchmarks. There multiple concurrent works that identify similar issues and reporting results consistent with our experiments, mostly for image classification. The work of Yu et al. (2023) also highlights the issue of evaluating each method under very different conditions and provides a benchmark for image classification TTA encompassing different adaptation scenarions, as well as diverse backbones and domain shift datasets. Similarly to ours, a significant disparity bet ween synthelic corruptions performance and natural shifts observed. However, the hyper-parameters were selected based on a single kind of domain shift, which may bias the results. Another work adressing the issue of fair comparison of TTA methods is that of Mounsaveng al. (2024) which provides analysis of existing orthogonal classification TTA methods. Class rebalancing is one of the tricks proposed to improve the methods' performance. Also, sample filtration to remove noisy high-entropy images is employed. In contrast, we analyze performance of different methods based on prediction entropy, showcasing some methods can actually be highly effective on those noisy, high-entropy samples. Similarly to ours, the work shows that baselines can be greatly improved by very simple tricks. In Niu al. (2023), label imbalance at test-time again identified as an important factor harming the TTA performance. Again, the works focus is on image classification and epxlores different normalization layer kinds and stabilization techniques of entropy minimization while we focus comparions of cross-entropy and laas-immalaace aware segmentation loss function, the loU. Finally, Yi et al. (2023) study TTA for image segmentations and how well classification methods transfer to semantic segmentation TTA. They conclude that many of the classification TTA improvements do not transfer to segmentation and again highlight the class imbalance, which typically greater for segmentation datasets.
4 Methods
In total, six different methods are implemented and evaluated, including traditional TTA basclines, methods from other tasks and novel methods. All the methods consist of optimizing a self-supervised loss, the specifics of the loss being what differentiates the methods. It can be formalized as follows:
0,+1 = arg min C(f
0,
where x is the input image, 0, are the parameters of the segmentation network f f&at at the i-th iteration and C is the self-supervised loss function.
The methods considered are:
Sitrropy-Minmmizationn (Ent), a method proposed by Wang et al. (2020b) inspired by semi- supervised learning where the self-supervised objective is the prediction entropy. It has been used as a baseline by most of the TTA work. Only normalization layer parameters are typically updated to reduce the computational cost. Whether batch normalization statistics are updated or not varies.
Pseudo-Labelling (PL), also commonly referred to self-training. The model finetuned with pseudo-labels obtained from the pretrained segmentation model. There are many improvements and modifications. The standard approach is to threshold the predicted probabilities and only train the model on the most confident predictions.
Angmuennation-CCosistennng (AugCo), proposed by Prabhu et al. (2021), is a method based on self-training enhanced by also optimizing for consistency between the original prediction and the prediction on augmented views, adapted to the single isolated image scenario.
Adversarial-Attace (Adv) is the method proposed by Nguyen et al. (b) for image classification TTA, adapted to the single, isolated image segmentation.
Mask Refinement (Ref) is one of the proposed methods and can be considered an enhanced pscudolabelling method. The psoudo-labels are obtained by a learnt refinement module that takes logit masks as inputs and outputs a refined segmentation mask. A method based on this idea has been implemented in medical imaging Karani et al. (2021) but never tested non-medical tasks.

---

Published in Transactions on Machine Learning Research (05/2024)
Degp-Intrsacctoo-ovee--hoi (dIoU), the second proposed method is similar to Ref. However, single-scalar quality estimate is predicted by a learnt module and minimized al test time. It is similar to using a GAN-like discriminator, as done in unsupervised domain adaptation literature Tzong ct al. (2017).
Only the necessary modifications to make the methods applicable to the single, isolated image segmentation setup with no assumptions of specific network architecture were applied to the existing methods.
The proposed methods based on learnt mask-refinement and mast-diseriminator modules will be described in the rest of this section. A description of other methods and the details of their modifications in this work are in Appendix B.
The method proposed in this work is presented first and the differences from previous work are detailed aftwerwards.
TTA with mask refinement based on the idea that since the output space changes much less than the input space, mask translation module can be learnt refine mask predictions on images from target distribution to resemble the masks obtained from source images. At lest time, the refinement network can be viewed
enhanced pseudo-label generation method. These pseudo-labels can then be used both supervision for the segmenter or a direct replacement of the segmentation output without any parameter optimization. However, the second option is unlikely to tackle highly distorted masks since the refined mask cannot improve gradually.
To train the refinement network f with Icarnable parameters Φ, images from the source distribution and the pretrained segmentation network fs are required. Given an image x and x' generated from x by syuthesizing a covariate domain shift (not changing the label), let us denote as 8 = [1) and s' (x') the corresponding segmentation masks. Then, fR is trained to predict given s' as input:
arg min (C
(1)
Predicted masks can also be replaced with ground truth g at training time:
arg min CCR (CR(J((s)))))
(2)
where LCE is the cross-entropy loss.
At test-time, adapting to an image x, the model parameters are updated minimize the IoU loss between mask prediction and refined mask estimated by fn:
0 = arg min Liou (fu (z)), (x))
(3)
0,
where 0, are the learnable parameters of fg at optimization iteration i and Jgg denotes no gradient flow throughout the computations of fs.
An overview of the training pipeline and the TTA with mask-refinement is Figure 2.
Refinement module training requires generating masks resembling those that the model would output under domain shift. Since TTA assumes the domain shift is not known in advance, the goal is to synthetically generate diverse realistic segmentation masks representing masks predicted on images with domain shift. The advantage of the refinement module is that only the output space corrupted masks are needed. It doesn't 't malter how these were oblained since the refinement module is independent of the input images. This work simulates the mask corruptions by using mask predictions on the images from the first few iterations of Projected Gradient Descont (PGD) Madry et al. (2017); Kurakin et al. (2018) adversarial attack, using the inverted mask as target. The more iterations of the attack, the higher the mask corruption, but the less realistic it becomes. Examples of generated corrupted masks are shown in Appendix C. The intuition

---

Published in Transactions on Machine Learning Research (05/2024)
train image
train mask
segmenter
Lykes
LSBG
corrupted image
corrupted mask
test image
*
segmenter
refiner
segmenter
refiner
TRAINING
INFERENCE
forward
backward
* frozen parameters
Figure Mask refiner training (left) and Mask Refinement (Ref) TTA inference (right). During training, the segmenter outpuls masks from training image and corrupted version of the training image simulating domain shift. The mask refiner then trained to predict the clean image mask given the corrupted image mask input only gradients flow back to the segmenter. At inference time, the segmenter output is fed into the refiner model. The refined output is then used as a pseudo-label to finetune the segmenter. A single gradient update is performed in each TTA iteration, then the masks are updated. The segmenter output may change with the updated weights, which in turn results in a new, possibly better, pseudo-label from the refiner. Visualized on single class prediction.
behind this adversarial approach is that in the first iterations, the most challenging pixels for the network are converted. Similarly, those image areas could be easily impacted by domain shift. The idea of using adversarial attack as a representative of domain shifts was also used in Nguyen et al. (b) and it was previously shown that adversarial robustness improves robustness to some domain shift kinds Croce et al. (2020).
TTA with Deep-Intersection-vve--- Union almos equivalent to Mask Refinement (Ref). The only difference that rather than learning refined mask, the network is optimized to predict the value of the IoU loss. The IoU loss can be computed from ground-truth at training time but the network receives the predicted mask input only.
Difference from unsupervised domain adaptation literature: There are many domain adaptation methods based on lcarning a discriminator between source and target domain. The key difference is that in this work, a module estimating how corrupled the mask or a mask refinement module learnt, instead of a binary classifier. Hence, the output is either a new segmentation mask, real number, rather than a class (domain) probability. This has the advantage that the mask needs to be considered than allowing the network to focus the most discriminative parts only. Also, the target domain is not known in advance and cannot be used to train the refinement. network masks resembling the target domain masks need to be synthesized.
whole, rather
Difference from medical image segmentation TTA with denoising autoencoders The idea of learning to predict a refined mask the output space is the same as in Karani al. (2021). The main differences stem from the domain difference, since their approach is tailored to the medical domain. In Karani et al. (2021), the corrupted masks obtained through swapping input image patches, heuristic method with many lyper-parameters Such augmentation makes sense when texture is more important than shapes, which is often not the case outside the medical domain. This work proposes exploiting adversarially- allacked images lo synthesize corrupted masks. Further, this work excludes methods that make changes to the segmentation model architecture while Karani et al. (2021) rely on an additional normalization module. Their module architecture is designed with domain shifts such as intensity transformations, typical for the medical domain, in mind. Updates of parameters in the segmentation network itself are not allowed during TTA.

---

Published in Transactions on Machine Learning Research (05/2024)
5 Experiments
The structure of this scction is as follows:
Given the focus of this study on SITTA and per-image performance analysis, underscore the need for an image-level evaluation metric. The widely used mean Intersection over Union (mloU) metric is typically applied at the dataset level and its adaptation for image-level assossment is not standardized.
Evaluation metrics:
Experimental Setup:
Experiment settings shared across experiments such as network architec- tures or hyper-parameters. Creation of the synthotic SITTA training set derived from the segmen- lation training dataset is also explained.
3. Experimental Results and Analysis: Experiment results and analysis. The TTA methods are evaluated two semantic segmentation models pretrained the GTA5 Richter el al. (2016) and COCO Lin et al. (2014) datasets.
5.1 Evaluation metrics
The standard semantic segmentation ovaluation metric is the mIoU, where the IoU score of cach class is computed from predictions aggregated over the whole dataset and the per-class scores are averaged
mloU C
IoU 1
(4)
where m& are the predictions and ground truth values for class k for all pixels across all images. Con- catenating all the masks into a single one and then computing the metric would not change the results, each pixel has the same weight independent. of the image size or difficulty. This metric does not consider the size of objects or the difficulty of individual images. Per-image results cannot be compared, since not all classes are typically present in an image it is not clear what value the score for that class should be.
Two additional metrics are introduced to account for the limitations of the standard mToU and make the evaluation more fine-grained. The first metric is designed to consider class imbalance and difficulty of individual images, locusing on per-class performance. will be referred to as mloU, and is defined as
mloU.
Σ EELe
ToU Jik)
(5)
C
where Is is the set of images in which either the prediction or the ground truth mask contains class and [ the total number of images in Ig.
The second metric is focused more on per-image performance and be computed for single image. It will be referred to mloU, and is defined as
1 [I]
mIoU,
(muk- Ink)
(6)
Cn
nET
REC.
where Cn is the set of classes in the predicted masks the ground truth of image I denotes the set of all images. This is the metric reported in our experiments unless stated otherwise. It allows for per-image performance comparison with the disadvantage of not accounting for class imbalance less frequent classes (on the image level) get smaller weight.
Similar metrics were recently considered by other works Volpi et al. (2022), typically only aggregating over images where the given class appears in the ground truth (as opposed either the ground truth the prediction). This has the advantage that mistakes are only accounted for once, making the metric more
9

---

Published in Transactions on Machine Learning Research (05/2024)
optimistic than ours. On the other hand, information about the errors is lost since the error is only computed for the ground truth class independently of what the incorrectly predicted class is.
On test datasets, the mDice (mean over the per-class dice scores Milletari et al. (2016)) and Accuracy (overall percentage of correctly classified pixels, regardless of class) metrics are also reported.
5.2 Experiment setting
SITTA training set. The SITTA training set for each model is derived from set of 40 images from the segmentation model's training dataset extended with a set of 9 synthetic corruptions at three severity levels from Hendrycks & Dicttorich (2019) such as blur, noise or fog, simulating different domain shifts. The original images are also included since the TTA methods should not harm the model on source domain images. Details about the corruption can be found in Appendix D. These synthetic datasets based on the GTA5 and COCO datasets are referred to as GTA5-C and COCO-C, respectively. Since the original images without corruption are also included, each SITTA training dataset consists of 1200 images (40 images, + corruption, three corruption levels).
For each TTA method, optimizing all the network parameters or normalization parameters is only considered, resulting in at least two different setups for each method. Further, when applicable (the methods compute a segmentation loss based on masks, as opposed to another self-supervised loss such as the prediction entropy), the CE and IoU losses are compared. While training segmentation models with loss that takes class imbalance into account, such as the CE and the Dice loss Milletari et al. (2016), is standard, TTA work image segmentation has relied cross-entropy, which is also suboptimal from the point of view of the evaluation metric. It is desirable to align the optimization metric with the evaluation metric as much as possible. This results in four setups for the Ref, PL, and AugCo methods. The learning rate and number of TTA iterations are considered from learning hyper- ppper-parameter.. The maximum possible number of iterations 10 to limit the computational requirements. Reasonable learning rate values are found via a grid search and then extended with other promising values based on the initial results.
SITTA
Shared implementation details. The refinement network architecture is U-Net Ronneberger et al. (2015) with an EfficentNNetBBO Tan & Le (2019) backbone pre-trained on ImageNet from the Timm library Wightman (2019). It is trained with the AdamW Loshchilov & Hutter (2017) optimizer with learning rate of 1e-3 and the Gross-Entropy (CE) loss. The SGD optimizer is used for the TTA since early experiments with AdamW showed a high divergence rate.
5.3 Experiment results
GTA5 - Cityscapes, ACDC. This experiment explores the performance of the TTA methods on a model trained on a syuthetic driving dataset, GTA5, evaluating on real-world driving datasets under dif- ferent weather conditions. The GTAS-pretrained model is the best-performing model of Volpi et al. (2022) (DeepLabbV)).
Ent norm
PL
Ref
AugCo
Adv
dloU
params loss
full
full
full
norm
norm
full
full
norm
norm
full
full
norm
norm
full
norm
full
norm
ent
ent
iou
cc
iou
iou
iou
iou
iou
kl
kl
NA
38.18
38.18
18
35.18
35,
35.18 18
35. 18
35.18
35.18
35. 18
35. 18
35.18
35.18
35.18
35.20
35, 20
35.18
35.18
TTA.-
38.18
35.58
35.54
37.21
35.60
37.09
35.18
38.69
36.88
36.50
35.27
35.66
35.35
35.39
35.20
35.20
35.18
35.18
AABS
-E 0.39
0.36
2.03
0.42
1.90
-6 3.51
1.70
1.32
0.09
0.48
0.17
0.21
Table 1: mIoU, results aggregated across corruptions and levels the GTA5-C dataset, compared to non- adapted (NA) performance. The TTA Soper-parameters a* were selected for overall best performance of each method. The overall and per-method best results are highlighted. No positive hyper-parameters are denoted by- -e (the performance converges to 0 from below).
10

---

Published in Transactions on Machine Learning Research (05/2024)
Ref
PL
AugCo
Iu
Adv
Corruption level:
Coruption lyss:
2-
fog
kp- molse
shot neise
spatter defoc.
geuns. bright. blur
contr.
JPSg
none
fog
gouss. noise
shot naks
spatcer defoc
ga000. blur
bright
contr.
JP9g
nome
RIK
blur
blur
Cornuption levell
Cornaction level: 1,
12.0
10.0
8
25
2-
-.5
shot spatter defoc. blur
bright.
fog.
shot
deros. blur
bright.
jpeg
blur
Figure 3: GTA5-C mloU, error reduction (%) depending corruption levels. TTA with overall optimal hyper-parameters for GTA5-C.
improver!
deteriorated
hqg
change
improved
deterioratedi
kgg
change
Ref
PL
Ref
PL
Ent
AugCo
Ent
AugCo
mIoU, NA
(b) entropy NA
Figure 4: The relationship between per-image scores (a) entropy (b) before and the score after adaptation the GTA5-C dataset. The difference between non-adapted (NA) mloU, or entropy and the mloU, after TTA is shown (mloU,A). A least-squares line fitted to the points is shown in yellow.
Since current methods do not consider different paper-parameters for individual images, a single set with overall best performance across all corruptions and corruption levels selected. The aggregated results with these overall optimal hyper-parameters on the SITTA training set can be found Table can be observed that the biggest improvements are achieved either by PL with IoU loss, optimizing normalization parameters only, or by Ref with IoU loss, optimizing all the parameters. The best-performing method is Ref, improving by 3.51 % over the non-adapted baseline (NA). Other methods only marginally improve over NA or show no improvement at all. Optimizing CE generally yields worse results than optimizing the IoU. While updating normalization parameters only may stabilize Ent, optimizing all the parameters is essential for optimal performance of Ref. For other methods, the difference is smaller optimizing normalization parameters only faster and thus recommended.
In Figure 3, the total error reduction results with the same of overall optimal aaper-parameters for each method shown but for each corruption level and kind separately. Tt can be seen that it is not possible
11

---

Published in Transactions on Machine Learning Research (05/2024)
Ref
PL
AugCo
Iu
Adv
Corruption level:
Corruption level: 3
3.5
2.0
3.0
15-
2.5
1.0
1.5
0.5
0.5
0.0
fog
gauss. nole
shot noise
spatter
Jefoz.
gaus3. blur
tright.
contr.
jP%
nome
fog
gaus.. nolse
shot nolse
spaster
cafoc blar
Kouss.
bright.
contr.
JP48
none
blur
blur
Corraption level:
Corruption level. 3.
3.0
2,00
2.5
1.70
1.50
2.0
1.20
1.5
1.00
3.0
0.77
0.50
0.5
0.0
fog
shot
spatter
defos.
bright.
fog.
shot
cefoc
bright.
blar
blar
blur
Figure 5: GTA5-C error reduction difference (%) between overall optimal hyperparameters and hyper- parameters selected for each corruption kind separately. The hyper-parameters were selected on GTA3-C
find a single set of ayper-parameet that would perform well across all the corruption levels with these methods. While all methods except for AugCo improve performance level 1 corruptions, from level 3, negative results be observed more often and many methods deteriorate/only slightly improve on level 5. Ref outperforms the other methods the majority of corruption kinds and corruption levels. The aggregated results across all corruptions showed that the negative results are outweighed by the gains on level 1, resultin g in overall positive results for most the methods.
In Figure 5, shown that one could select optimal hyper-parametert for each corruption kind and level, results would improve substantially for many of the corruptions. This analysis suggests that unless the domain shift is known in advance, strategics with method and rrper-parameter selection for each image separately should be explored.
Only the methods with overall positive TTA results are considered for further analysis, namely Ref, PL, AugCo, and Ent. The relationship wetween the non-adapted (NA) performance and the performance im- provement on individual images for different methods visualized in Subfigure 4a. The analysis shows Ref outperforms other methods, espocially on images that had low initial mIoU,, while the performance PL is more consistent across all initial scores but not as powerful for initial low scores. While Ent makes perfor- mance worse for low initial scores and improves more the initial score increases, AugCo shows consistent improvements across all initial scores similarly to PL but to smaller extent.
Ref
PL
Ent
AugCo
12
Ref
PL
Ent
AugCo
10
AmtoU, (%)
L LL
AmloU, (%)
3-
cityscapes
acdo-fog
acdc-night
acdc-rain
acdc-snow
vOC
(a) GTA5 model
(b) COCO model
Figure 6: TTA mToU, reduction (%) test datasets with hyper-parameters select the TTA training datasets.
12

---

Published in Transactions on Machine Learning Research (05/2024)
If the mloU, for each image were known (the ground truth is necessary for its computation), it could be used to either select a method performing best on those values to select hyper-parameters. In Subfigure 4b, an analogous analysis is performed, replacing the mloU, with segmentation prediction entropy, which does not require any supervision. Similar results as with the mloU, can be observed.
After selecting the best hyper-parameters for each method on the SITTA training set, the methods are evaluated on 5 test datasets: ACDC-Rain, ACDC-Fog, ACDC-Night, ACDC-Snow, and Cityscapes. The Cityscapes represent domain shift from synthetic to real images, while ACDC datasets add adverse weather conditions, making the domain shift even greater. The first four datasets are created by splitting the ACDC dataset by different conditions. Each of the test sets consists of 500 images. The test results are reported in Figure 6a and Table 2. Similarly to SITTA training datasets, either Ref or PL methods perform best, depending on the dataset. While not outperforming Ref on all the datasets, the performance consistently better than the other methods while Ref outperformed or matched by the other methods on Cityscapes and ACDC-night. Table 2 shows that in standard metrics that do not consider per-image difficulty, AugCo outperforms the other methods in many cases.
PL is
method
dataset
metric
NA
Ref
PL
Ent
AugCo
mloU,
34.40
34.72
37.14
35.12
35.52
mloU, 28.71
28.65
30.70
29.09
29.53
cityscapes
mloU
42.90
40.37
43.28
43.41
14.05
m Dice
55.78
52.53
56.04
56.17
56.89
Accuracy
86,54
83.92
87.15
87.30
87.50
mloU; 32.03
35.99
35.68
33.92
31.82
mloU, 24.87
27.29
27.52
26.00
24.69
acde-fog
mloU 37.65
38.36
39.39
39.15
37.61
mDice
51.42
51.12
53.29
52.68
51.51
Accuracy
73.23
84.11
75.51
75.91
66.63
mloU, 13.60
14.12
15.09
14.13
14.15
mloU. 10.77
10.96
11.53
10.68
11.01
acdo-night
mloU 15.79
13.77
15.11
15.53
16.25
m Dice
24.38
21.29
23.49
23.74
24.98
Accuracy
52.09
52.47
51.86
52.82
53.09
mloU, 33.52
35.60
37.16
35.05
34.50
mloU, 26.15
27.40
28.47
26.89
26.73
acdo-rain
mloU
36.93
36.21
37.61
37.53
37.98
miice
48.92
47.52
49.44
49.23
50.25
Accuracy
84.56
84.22
85.65
86.13
84.74
m 31.54
35.60
34.15
31.87
31.81
mloU.
25.28
28.09
27.16
25.38
25.45
acde-snow
mloU
35.30
37.89
36.64
35.34
35.40
mDice
48.46
50.35
50.00
48.51
48.53
Accuracy
73.17
81.52
74.48
73.66
73.45
Table 2: ACDC and Cityscapes test datasets results. Hyper-paramelers were selected for overall best performance GTA5-C, Best results for each dataset and metric are highlighted.
The inconsistencies of results between SITTA training and test suggest that unless the domain shift conditions are known in advance, it is difficult to select hyper-parameters based on a general SITTA training set.
COCO - VOC. In this experiment, the performance of TTA methods studied on a model trained on the COCO dataset and evaluated on the VOC dataset. The segmentation model an official Torchvision DeepLabbV3 model with Resnet50 backbone trained the COCO dataset. with subset of 20 VOC classes.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Ref
PL
AugCo
Iou
Adv
Corruption level: 1
Cornu piion level: 3
15
15
10
13
-3
log
gauss. nolse
shot noks
spatter defoc. gouss. bright. bler
conzr.
JPM
none
fog
gauss. noie
shot noise
spatter defoc. blar Corruption level: 1. 1.3.5
gaass. bright. blur
contr.
1pn
nome
+BK
blar
Corraption level: 5
15
13
-5
-10
fog
spatter defoc. blur
bright.
jpeg
fog
shot
spatter
defoc. blar
bright
contr.
jong
blar
blur
Figure 7: COCO-C mloU, error reduction (%) depending on corruption levels. hyper-parameters for COCO-C.
TTA with overall optimal
In contrast to previous experiments, it is real-to-real dataset domain shift.. The results of different methods with parameters selected for the overall best performance across all corruptions and levels can be found in Table 3. Biggest improvements are obtained by the PL and Ref methods. PL purperfomss Ref, in contrast to the GTA5-C experiments. The best improvement is by 3.28 %, reducing the total segmentation error by 7.3 %. Consistently with previous experiments, best results are achieved with the IoU loss, outperforming CE cases. In contrast to GTA5-C, Ent achieves better results when optimizing all the network parameters, as opposed optimizing only the normalization layer parameters. The same holds for PL. For Ref, optimizing all the parameters again important. Other methods' improvements over the non-adapted baseline are marginal.
Ent
AugCo
Adv
dloU
params loes
full
full
full norm
full
full
norm norm
full
full
full
full
lou
iou
kl
NA
55.01
55.01
55.01
55.01
55.01
55.01
55.01
55.01
55.01
55.01
55.01
55.01
55.01
55.01
55.16
55.16
55.01
55.01
TTAp
56.97
56.75
67.17
57.99
57.10
58.30
56.24
57.31
56.56
57.16
55.40
55.59
55.30
56.30
55.16 55.16
55.61
56.74
ABBS
1.96 1.74
2.16
2.98
2.09
3.28
1.23
2.30
1.55
2.15
0.39
0.58
0.29
1.29
0.60 0.73
Table 3: mloU; results aggregated across corruptions and levels in the COCO-C dataset, compared to non- a" were selected for overall best performance of each method. The overall and per-method best results are highlighted. No positive hyper-parameters are denoted by (the performance converges from below).
adapted (NA) performance. The TTA
The total error reduction results with single set of optimal hyper- pareemerrreeeerss for each method are reported for each corruption level and kind in Figure 7. The results slightly differ from those for the GTA5-C, as in this case, PL is consistently the best method, only rarely outperformed by Ref or AugCo. Negative results on some corruption kinds are already reported for level corruptions.
In Figure 9, the results with optimal typer-parameters for cach method, corruption kind and level are shown. The results again improve substantially but this time, the differences between methods are smaller. Interestingly, the dToU method performs much stronger than in the GTA5-C experiments.
14

---

Published in Transactions on Machine Learning Research (05/2024)
improved
deteriorated
kq change
deteriorated
change
Ref
PL
Ref
PL
Ent
AugCo
Ent
AugCo
(a) mloU, NA
(b) entropy NA
Figure 8: The relationship between per-image scores (a) entropy (b) before and the score after adaptation the COCO-C dataset. The difference between non-adapted (NA) mloU, entropy and the mloU, after TTA is shown (mloU,A). A least-squares line fitted to the points is shown in yellow.
Ref
PL
AugCo
bu
Adv
Corruption level:
Corruption level: 3
fog
shot noise
spaster
cefoc. blur
gauss. blur
bright.
contr:
none
fog:
gAUsS noise
shot noise
spacter
defoc. blur
geass. blur
bright.
contr.
iPM5
none
noise
Corruption level:
Corruption level: 1. 3.
fog
shot
cefor. blur
bright.
shot
de'oc.
bright.
iPa5
blur
Figure 9: COCO-C error reduction difference (%) between overall optimal hyperparameters and hyper- parameters selected for each corruption kind separately. The hyper-parameters were selected on the COCO- C.
Figure 10b compares the overall method performance on the SITTA training set. An oracle option is introduced where the method with the best results is picked for each image. There is a significant, gap between the oracle and other methods, which further highlights that different methods are good in various cases and understanding the strengths of cach methods can load to significantly improved performance.
Again, only Ref, PL, AugCo, and Ent are used for further analysis. The relationship between the non- adapted (NA) performance and the performance improvement on individual images for different methods is visualized Subfigure 8a. The distribution of initial non-adapted mIoU, is different. The initial model is stronger than the GTA5 model. All methods show similar behavior more improvement is achieved on images with lower initial score.
15

---

Published in Transactions on Machine Learning Research (05/2024)
30
20
mlou
10
ref
pl
augco adv
oracle
ref tent augco iou adv oracle
(a) Comparison of the overall performance of different
(b) Comparison of the overall performance of different methods the COCO-C validation set.
methods the GTA5-C validation
Figure 10: The difference between non-adapted (NA) mloU,
The relationship of segmentation prediction entropy and mloU; improvement by adaptation is shown in Subfigure 8b, supporting the notion that the entropy of prediction before adaptation a good proxy for mloU,-
method
metric
NA
Ref
PL
Ent
AugCo
mIoU,
77.00
77.62
79.67
78.85
77.75
mloU,
63.38
63.38
66.90
65.38
63.10
mIoU
77.20
77.78
77.55
77.01
76.01
mDice
86.16
86.66
86.47
86.05
85.54
Accuracy
94.69
95.18
94.68
94.69
94.76
Table 4: VOC test dataset results. Hyper-parameters were selected for overall best performance on COCO- C. Best results for each dataset and metric are highlighted.
The results on the VOC test set are shown in Figure 6b and Table 4. PL outperforms other methods in the proposed metrics, all the mothods improved over the non-adapted baseline. When considering standard metrics which do not consider per-image difficulty, Ref is best.
Additional results can be found in Appendix E.
6 Conclusions and limitations
This work investigated the performance of six Single Image Test-Time Adaptation (SITTA) methods semantic segmentation problems. Two established baselines, two adapted state-of-the-art methods from image classification and continual TTA, and two novel methods were considered. We designed framework that allows for hyper-parameter tuning and analysis of performance under diverse conditions on synthetic data inspired by Hendrycks Dietterich (2019) that can be derived from an arbitrary training dataset. We evaluated the methods on real-world datasets.
Experiments on driving-scene datasets dominate the segmentation TTA literature. We followed this practice and experimented with models pretrained the synthetic GTA5 dataset, evaluating real-world driving scenes, including datasets with adverse weather conditions such rain and fog. The common practice is to tune TTA hyperparamcters on the Synthia Ros et al. (2016) datascts, however, this dataset contains the same weather and day-night conditions as the test datasets. Also, it limited to driving datasets with the same set of classes. For this reason, we tuned papameersssserss on a synthetic dataset derived from the training dataset by applying different corruptions, which gives us control over the domain shift conditions, facilitates analysis and applicable any existing dataset. We also added novel benchmark where we evaluated COCO-pretrained model the VOC dataset, focusing common objects.
16

---

Published in Transactions on Machine Learning Research (05/2024)
We proposed two new methods inspired by ideas from unsupervised domain adaptation and medical imaging TTA. The stronger of the methods based on a learnt refinement. module performed best on multiple of the test datasets and we showed that it is powerful on the images with the worst segmentation performance, as measured by IoU. We also showed that if we replace the IoU performance with the entropy of the predicted segmentation mask, which does not require the ground truth to be known, the same behaviour can be observed. This could be used to choose an appropriate saahhodhhyper-paranee
in future work.
We explored the effect of previously neglected design choices. Training with loss function that accounts for class imbalance, a known issue of image segmentation datasets, such the loU or dice loss, is standard when training image segmentation models. When considering small batch sizes or even a single image, the class imbalance further increases, but SITTA methods implement baselines such as pseudo-labelling with the CE loss or entropy minimization with equal weight for all pixels. Our results revealed that while SITTA in the standard setting with CE loss did not improve performance much, substituting the CE with ToU improves performance substantially. The performance of pscudo-labelling was improved by 3.51 % and 3.28 % GTA5-C and COCO-C validation sets while with CE loss, the improvements were by 1.7 % 2.16 % only, respectively. The experiments whether to update all or normalization parameters only show this design choice significantly impacts results but the right option depends on the scttings (method, datasct)- it is an important as unstable for small batch sizes, performs well when the batch-normmllzztioon mean and variance are not updated at test time and only the affine parameters the normalization layers are optimized.
to consider. Further, we find that entropy minimization, often reported
In the GTA5-C synthetic datasets experiments, the refinement SITTA dominates, followed by the pseudo- labelling baseline. While the refinement is significantly better some of the real-world test datasets, other ones, pseudo-labelling performs best. In the COCO-C experiments, the top performers swap places; Pseudo-labelling is followed by refinement. On the test dataset, psendo-labelling remains the best. While the other methods do not perform very well when TTA hyperparameters are optimized across many different corruption kinds, their performance improves when tuning them to specific kinds of domain shift. While there is not a single method performing best over all the test datasets, our results highlight the potential ial of SITTA for semantic segmentation.
Limitations. To limit the scope of the study, we only focused adaptation with self-supervised loss functions and no reliance on batch normalization layers. While these methods tend to perform the best, their iterative optimization comes at an increased computational time. Methods alleviating this burden should be explored, such as only adapting to informative samples or methods inspired by efficient model finetuning.
The synthetic validation sot created by applying artificial corruptions to the training set may not cover the complexity of real-world domain shifts. Label shift, common in real-world datasets, is not considered the solutions are typically complementary.
Finally, only two models were considered and the effect of different model architectures on the individual methods is not known. While our work improves the understanding of TTA for semantic segmentation methods, a benchmark for fair and thorough evaluation of the methods still missing.
Acknowledgements
We greatly appreciate the help of Tomas Jenicek and Ondrej Tybl with proofreading of the manuscript.
Klara Janouskova was supported by the 19/23//77//HK3, portod by the Technology Agency of the Czech Republic, project No. SS73020004 and project FACIS No. VJ02010041. Tamir Shor and Chaim Baskin were supported by NOFAR MAGNET number 78732 by the Israel Innovation Authority.
research grant, and Jiri Matas was sup-
References
N Romero Aquino, Matheus Gutoski, Leandro T Hattori, and Heitor S Lopes. The effect of data augmentation the performance of convolutional neural networks. Braz. Soc. Comput. Intell, 2017.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. arXiv:1077006.53 2016.
Layer normalization.
arXiv preprint
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. Mt3: Meta test-time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Stafistics, pp. 3080-3090. PMLR, 2022.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 295-305, 2022.
Francosco Croce, Maksym Andriushchenko, Vikash Schwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint Xtv:20001 09670, 2020.
Alain Fournier, Don Fussell, and Loren Carpenter. Computer rendering of stochastic models. Communica- tions of the ACM, 25(6):371-33)) 1982.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A Efros. Test-time training with masked autoencoders. arXiv preprint Xrx:22200 07522,, 2022.
Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the souree: Diflusion-driven adaptation to lest-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11786-11796, 2023.
Tan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. 2014.
arXiv preprint
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Represenlations, 2019.
Xiaowei Hu, Chi- Wing Fu, Lei Zhu, and Pheng-Ami Heng. Depth-attentional features for single-image rain removal. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 8022-8031, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456, 2015a.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning, pp. 448 456. pmlr, 2015b.
Neerav Karani, Ertune Erdil, Krishna Chaitanya, and Ender Konukoglu. Test-time adaptable neural networks for robust medical image sogmentation. Medical Image Analysis, 68:101907, 2021.
Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. SITA: Single Image Test-time Adaptation. 12 2021. URL httpp:://axxivv orgooogg//aoo.oooooooooooooooe absp//bs/21111. 02355v3.
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pp. 99 112. Chapman and Hall/CRC, 2018.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, 1CML, volume 3, pp. 896, 2013.
Bo Li, Yezhen Wang, Tong Cho, Shanghang Zhang, Sicheng Zhao, Pengfei Xu, Wei Zhou, Yoshua Ben- gio, and Kurt Keutzer. Rethinking distributional matching based domain adaptation. arXiv preprint arXiv:2006. 13352, 2020.
18

---

Published in Transactions on Machine Learning Research (05/2024)
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80:109-117, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 18th European Conference, Zurich, Swilzerland, Seplember 6-12, 2014, Proceedings, Parl V 13, pp. 740 755. Springer, 2014.
Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 1215 1224, 2021a.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Ttt++: When does self-supervised test-time training fail thrive? Advances in Neural Information Processing Systems, 34:21808-21888 2021b.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Xiv preprint 17X1::711.0555 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial atlacks. ar preprinl or arXie: 176: 1706.06083, 2017.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565 571. Ieee, 2016.
Saypraseuth Mounsaveng, Florent Chiaroni, Malik Boudiaf, Marco Pedersoli, and Ismail Ben Ayed. Bag of tricks for fully test-time adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1936 1945, 2024.
Zachary Nado, Shreyas Padhy, Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint (:Xiv:2006.1066 2020.
A Tuan Nguyen, Thanh "guyen-Tang, Ser-Nam Lim, and Philip H S Torr. TIPI: Test Time Adaptation with Transformation Invariance.
A Tuan Nguyen, Thanh Nguyen-Tang, Ser-Nam Lim, and Philip HS Torr. Tipi: Test time adaptation with transformation invariance. b.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint ar Xin:2330. 12400, 2023.
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Augco: augmentation consistency-guided self-training for sourco-free domain adaptive semantic segmentation. arXiv preprint arXi::2107.10140, 2021.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Computer -ision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 102 118. Springer, 2016.
Adrian Lopez Rodriguez and Krystian Mikolajezyk. Domain adaptation for object detection via style con- sistency. Xixiv preprint X9v 10033,, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-asse assisted interven- tion, pp. 234 241. Springer, 2015.
19

---

Published in Transactions on Machine Learning Research (05/2024)
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 3234 3243, 2016.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8050 8058, 2019.
Steffen Schncider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems,
2020.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer, How to train your vit? data, augmentation, and regularization in vision transformers. orXiv preprinl arXiv:2106. 10. 10270, 2021.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for gencralization under distribution shifts. In International conference on machine learning, pp. 9229-92222222 9248. PMLR, 2020.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-6616166 6114. PMLR, 2019.
Korawat Tanwisuth, Xinjic Fan, Huangjic Zheng, Shujian Zhang, Hao Zhang, Bo Chen, and Mingyuan Zhou. A prololype-oriented framework for unsupervised domain adaplation. Advances in Newral Information Processing Systems, 84:17194-17208, 2021.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 7167-7176, 2017.
Gabriele Valvano, Andrea Leo, and Sotirios A Tsaftaris. Re-using adversarial mask discriminators for test- time training under distribution shifts. arXiv preprint arXiv:2108. 1 11926, 2021.
Riccardo Volpi, Pau De Jorge, Diane Larlus, and Gabriela Csurka. On the road to online adaptation for semantic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19184-19195, 2022.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arxiv preprint 2Xiu:2006.10726, 2020a.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXin:2006. 2020b.
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Contimual test-time domain adaptation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201-7211, 2022.
Renhao Wang, Yu Sun, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang. Test-Time Training on Video Streams. URL https://rido-ttt..iibhie
Ross Wightman. Pytorch image models.
http://pproorrr-mmmmmmdeelllllle 2019.
Chang'an Yi, Haotian Chen, and Yifan Zhang. A critical look at classic test-time adaptation methods semantic segmentation. arXiv preprint arXiv:2310.05331 2023.
Yongcan Yu, Lijun Sheng, Ran He, and Jian Liang. Benchmarking test-time adaptation against distribution shifts in image classification. arXi preprint NNN::3377.0319 2023.
Yue Zhang, Shun Miao, Tommaso Mansi, and Rui Liao. Task driven generative modeling for unsupervised domain adaptation: Application to x-ray image segmentation. In Medical Image Computing and Computer Assisted CnterrentionMMCCAA. 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II, pp- 599 607. Springer, 2018.
20


Reviews:
[{"review_id": "y0TW5NUXGm", "reviewer": "Reviewer_cWBQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The proposed task is interesting and novel.\n2. Complete framework including evaluation metrics, baseline, etc, are developed for this new task.\n3. A novel and effective method is proposed to tackle the proposed setting.\n4. Thorough empirical study are provided to validate the effectiveness of the method.\n\nWeakness:\nNo prominent weakness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on a novel setting, namely single image test-time adaptation. Evaluation metric and framework are developed, and a novel method is proposed to tackle the proposed task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No requested changes.", "broader_impact_concerns": "No broader impact discussion are needed in the reviewer's opinion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h9sdcRF7uU", "reviewer": "Reviewer_dkDE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1: This paper focuses on a valuable problem that needs to tailor a model for each image in the test-time.\n\n2: The authors conduct extensive experiments to verify the effectiveness of the proposed method.\n\n3: The proposed method seems to be simple yet effective.\n\n**Weaknesses**:\n\nThe overall presentation is not ready for publication, and the needed modifications are beyond the scope of the reviewers' responsibility.\n\n1: Grammar and Basic Writing Organization. Here, I take the Introduction as an example, but I believe the problem doesn't only exist in the Introduction.\n\na) \"between source\" --> between data sources?\n\nb) \"difficult to predict\" --> difficult-to-predict?\n\nc) \"acsitta not only streamlines\": What is \"acsitta\"? The logical relation between this and the previous sentences is not clear.\n\nd) \"the many works relying on the existence ...\" --> many works that rely on? Meanwhile, is it possible to convert the batch-normalization to layer-normalization so that such methods can be compatible with transformer architectures?\n\ne) \" an additional method from image classification which robustifies the network to small domain shifts (represented by adversarially-attacked images), expecting increased robustness to other domain shifts as well.\": The meaning of this sentence is cumbersome to read, and it can be refined.\n\nf) \"While medical domain and its domain shifts of interest are different\": What does this sentence mean? Why are the domain and domain shifts comparable? I guess the authors want to emphasize the difference between the medical domain and the general non-medical domain.\n\ng) \"been on prediction entropy and reconstruction loss\" --> \"been on predicting entropy and reconstruction loss\" or \"been on entropy and reconstruction loss prediction\"? \n\nh) The summarized contributions are not clear. I can't find what the novel part is, given the simple description, \"A novel adversarial refinement module.\" I want to know the actual improvements rather than a simple sentence, \"improvements of baselines.\"\n\n2: Format. \n\na) Tables (both Table 1 and Table 2) are beyond the border of the main paper.\n\nb) Legends and main content overlap are not encouraged (Figure 6). Why don't authors align subfigures together?\n\nc) In Figure 8, the left and right subfigures are not aligned well. The text and number on the x-axis overlap.\n\n3: Emphasizing the Proposed Technique Contribution.\n\nFrom the method section and Figure 2, I can't see what the proposed technique contribution is. Figure 2 presents the pipeline of Mask Refinement, but the idea has been proposed in medical imaging, and the adversarial-attack is also proposed before. The only contribution seems to test their effectiveness in the non-medical domain. Meanwhile, the shown inference and training images are the same, which doesn't suggest the challenge of test-time adaptation i.e., the domain gap between training and test images.\n\nAll in all, I appreciate the authors' efforts in conducting extensive experiments on the single image test-time adaptation tasks. However, it's hard for me to evaluate the contribution of this paper, given the current presentation. I feel this paper was written in a hurry and kindly encourage authors to re-organize and polish the paper, especially in emphasizing the paper's contribution. I look forward to the revised version and will re-evaluate this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles Single Image Test-Time Adaptation for the segmentation task, in which only one image is available for adapting the model. The problem is practically valuable. This paper proposes to use augmentation to alleviate the limited data (only one image) problem and tune the needed hyperparameters in SITTA methods. The authors conducted extensive experiments to analyze and verify the performance of many techniques for tackling this task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My current concern mainly focuses on writing. I will re-evaluate after the re-organization and polishment.", "broader_impact_concerns": "The authors discuss the broader impact in the main text.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cYviZ8yC4G", "reviewer": "Reviewer_dCLn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper studies an underinvestigated yet important problem -- single image test time adaptation for segmentation. \n- The experiments cover a comprehensive set of methods that are considered effective for the problem before.\n- The proposed refinement framework, even though not consistently outperform, is simple.\n\nWeakness:\n- The proposed method is not performing best in all cases.\n- From the experiment results, the proposed SITTA is effective when CE is replaced by IoU, however, no justification for why this is the case. \n- regarding the domain shift, the synthetic dataset is generated based on covariate shift, it would be nice to see method performance on datasets a. with label shift b. realistic data. \n- only mIOU is used as evaluation metrics are used, but since the IoU is used as the loss function,  it is hard to disentangle the improvements from replacing the training loss as IoU is more directly related to the evaluation metric.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on test time adaptation of the segmentation task on a single image. The contributions of the paper can be listed as follows:\n1. One of the few works studying single image test time adaptation for segmentation. \n2. A diverse set of methods are validated on synthetic and real datasets. \n3. Replacing Cross-Entropy by IoU loss improves the baseline method performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- More evaluation metrics need to be adopted, such as jaccard index, dice (F1 score), etc. \n- Justification of the improvements from replacing the loss, can you provide the intuition on why this is the case?  and more experiments to support this. \n- Experiments on more realistic domain shift (with more realistic data)\n- Typos: please carefully check the grammar and typos of the submission. e.g. the 4th point of contribution:  \"... an underexplroed setup important in applications ...\", underexplroed should be underexplored.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers lean to acceptance after the rebuttal. However, the authors still need to address the minor remaining issues. Specifically, the reviewers suggested releasing codes or relative materials, which makes this work reproducible. Moreover, the authors should further proofread the manuscript and fix all the typos.
Decision: Accept (poster)