Domain-Conditioned Transformer for Fully Test-time Adaptation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 06:05:38

Abstract: Fully test-time adaptation aims to adapt a network model online based on sequential analysis of input samples during the inference stage. We observe that, when applying a transformer network model into a new domain, the self-attention profiles of image samples in the target domain deviate significantly from those in the source domain, which results in large performance degradation during domain changes. To address this important issue, we propose a new structure for the self-attention modules in the transformer. Specifically, we incorporate three domain-conditioning vectors, called domain conditioners, into the query, key, and value components of the self-attention module.  We learn a network to generate these three domain conditioners from the class token at each transformer network layer. We find that, during fully online test-time adaptation, these domain conditioners at each transform network layer are able to gradually remove the impact of domain shift and largely recover the original self-attention profile. Our extensive experimental results demonstrate that the proposed domain-conditioned transformer significantly improves the online fully test-time domain adaptation performance and outperforms existing state-of-the-art methods by large margins.

Main paper: 
 related work: 
 This work is related to test-time adaptation, source-free unsupervised domain adaptation, parameter efficient transfer learning, and prompt learning.Test-time adaptation (TTA) aims to adapt a pre-trained source model to unlabeled data with domain shift during inference. There are two major approaches: test-time training [10, 35, 46] and fully test-time adaptation [37, 38, 51] . [46] proposes the first test-time training (TTT) method where feature extractor network parameters are updated using a self-supervised loss on a proxy learning task. The TTT++ method [35] improves this approach with a feature alignment strategy based on online moment matching. Extending this line of research, the TTT-MAE method [10] incorporates a transformer backbone and replaces self-supervision with masked auto-encoders [15] . Note that all these TTT methods require specialized training in the source domain.In contrast, fully test-time adaptation methods fine-tune pretrained models during inference without access to the source data. The TENT method [51] proposes fully test-time adaptation by fine-tuning Batch Normalization (BN) layers. The NHL method [49] learns early-layer representations in an unsupervised manner, drawing inspiration from neurobiology-inspired Hebbian learning. Methods have also been developed to update the model inputs instead of the network parameters. For example, the DDA method [11] projects the input data from the target domain into the source domain based on a diffusion model during testing. The method proposed by [9] modifies the target inputs by learning image-level visual prompts, keeping source model parameters frozen during testing. It has been noted that existing online model updating methods suffer from performance degradations due to sample imbalances and distribution shifts. To address this issue, SAR [38] proposes to eliminate noisy samples with high gradients and perform flattening of model weights towards a minimum, thereby enhancing the robustness of the model. DELTA [59] uses moving averaged statistics to perform the online adaptation of the normalized features.Source-free unsupervised domain adaptation (source-free UDA) aims to adapt the model trained on the source domain to the unlabeled target domains without leveraging the source data [19, 24, 26, 27, 30, 45, 55] . Pseudo-labeling [24] methods assign a class label for each unlabeled target sample and uses the label for the supervised learning objective. The SHOT method [30] computed pseudo labels through the nearest centroid classifier and optimized the model with information maximization criteria. The KUDA method [45] utilized the prior knowledge about label distribution to refine modelgenerated pseudo labels. The SFDA-DE method [5] aligned domains by estimating source class-conditioned feature distribution. The HCL method [19] proposed a solution for addressing the lack of source data by introducing both instance-level and category-level historical contrastive learning. The DIPE method [52] focuses on exploring the domain-invariant parameters of the model, rather than trying to learn domain-invariant representations. These source-free methods are offline, requiring access to the complete test dataset. It also costs a number of epochs for model adaptation. In contrast, our fully online test time adaptation adapts the given source model on the fly during testing which only accesses the test samples once.As the model size grows rapidly with the development of foundation models, there has been a growing interest among researchers focusing on Parameter Efficient Transfer Learning [18, 25] . This area of study focuses on adapting large-scale pre-trained models to different downstream tasks with minimal modification of parameters. These methods strategically select a subset of pre-trained parameters and, if needed, introduce a limited number of additional parameters into a pre-trained network. These selected parameters are updated specifically for new tasks, while the majority of the original model parameters are frozen to ensure efficiency and effectiveness. For instance, the method proposed by [32] introduces learnable vectors to rescale keys, values in attention mechanisms, and inner activations in position-wise feed-forward networks through element-wise multiplication. Diff pruning [14] learns an adaptively pruned task-specific "diff" vector extending the original pre-trained parameters. BitFit [57] employs sparse fine-tuning, where only the bias terms (or a subset of them) are modified. AdaptFormer [4] adapts pre-trained Vision Transformers (ViTs) for various vision tasks by replacing the original MLP block with a trainable down-up bottleneck module. In contrast, LST [47] introduces a separate ladder side network, a smaller network that uses intermediate activations as inputs based on shortcut connections, rather than inserting additional parameters inside the backbone networks. It should be noted that all of these parameter efficient transfer learning methods adapt pre-trained models to downstream tasks through supervised learning.The notion of prompts originated in Natural Language Processing (NLP), where linguistic instructions, known as prompts, are added to input text to guide pre-trained language models in specific downstream tasks [33] . Recent methods [25, 28, 34] represent prompts as task-specific vector inputs and optimize them directly through error back-propagation. These prompt-tuning methods learn prompts from downstream data within the input embedding space, requiring fewer parameters to be updated during the adaptation process. Prompt learning has been successfully applied to vision-language models [13, 43, 60, 61] . CoOp [61] fine-tunes the prompt of the text encoder in CLIP [41] . CoCoOp [60] conditions the learned prompt on the model's input data to address out-of-distribution issues. TPT [43] optimizes the prompt of CLIP's text encoder during test time, enhancing generalization performance by minimizing entropy with confidence selection. DAPL [13] constructs a prompt comprising domain-agnostic context, domain-specific context, and class label for the text encoder using a contrastive objective to disentangle semantic and domain representations. It's crucial to note that these prompt methods are based on text encoders in Vision-Language Models, whereas our approach exclusively focuses on the Vision Transformer (ViT) encoder.In addition to text prompt learning, techniques have emerged for learning visual prompts in computer vision tasks. The method proposed by [9] focuses on continuous TTA tasks, where domainspecific and domain-agnostic prompts are learned and attached to target input images on a per-pixel basis. Meanwhile, BlackVIP [39] learns individual prompts for each image through a neural network, without requiring prior knowledge about pre-trained model architectures and parameters. Visual Prompt Tuning (VPT) [12, 21] introduces task-specific learnable parameters into the input sequence of each ViT encoder layer while keeping the pre-trained transformer encoder backbone frozen during downstream training. This approach has found applications in transfer learning for image synthesis [44] . Note that the VPT method introduces a large number of new tokens. It substantially increases the computational complexity of the self-attention mechanism, as the computational complexity of self-attention is quadratic to input token size.In this work, we propose to explore fully online test-time adaptation of transformer models by designing adaptable self-attention modules. Compared to existing work, the major contributions of this work can be summarized as follows: (1) We introduce a new design of the self-attention module in the transformer networks, which is able to capture the domain-specific characteristics of test samples in the target domain and is able to clean up the domain shift perturbations in the test samples. (2) We learn a lightweight neural network called domain-conditioner generator to generate the domain conditioners from the class token at each layer, enabling the transformer model to better align its self-attention profiles with the source domain. (3) Our experimental results demonstrate that our  
method: 
 In this section, we present our method of Domain-Conditioned Transformer (DCT) for fully test-time adaptation. When adapting transformer-based models to new domains, we observe that their self-attention distance profiles, defined as the spatial distribution of self-attention between tokens, for image samples in the target domain deviate significantly from those in the source domain. Let {Ω 𝑙 𝑖 |1 ≤ 𝑖 ≤ 𝑁 } be the set of 𝑁 embedding tokens at the network layer 𝑙, and C = [𝑐 𝑙 𝑖 𝑗 ] 𝑁 ×𝑁 be their selfattention weight matrix. Let D = [𝑑 𝑙 𝑖 𝑗 ] 𝑁 ×𝑁 be the distance matrix for these tokens where 𝑑 𝑙 𝑖 𝑗 represents the pixel distance in the original input image between tokens 𝑖 and 𝑗. The attention distance [6, 42] for network layer 𝑙 is then defined as The attention distance 𝑑 𝑙 for all network layers is referred to as the self-attention profile, denoted as {𝑑 (𝑙)}. In Figure 2 , we plot the attention profile for the clean image, images with different levels of domain corruptions (Defocus Blur), and the attention profile recovered by our DCT method. Conceptually, this attention distance is similar to the receptive field size in Convolutional Neural Networks (CNNs). It indicates that lower layers of the ViT model tend to focus on local regions more, as evidenced by a lower mean attention distance. In contrast, higher layers primarily integrate global information, leading to a higher attention distance. When there is a data distribution shift during testing in the target domain, the attention distance distribution is shifted. As illustrated in Figure 2 , as the level of image corruption increases, the corresponding attention distance becomes large. Once this self-attention profile has been perturbed by the domain changes or corruptions, the performance of the transformer model will degrade significantly. It can be seen that, using the DCT method, the attention distance profiles of the corrupted images can be largely recovered to their minimum level (Level 1), approaching the attention distance profile of the clean image. Meanwhile, the Attention Rollout for the target samples exhibits improved focus compared to the source model, as illustrated in Figure 6 .EQUATIONIn this work, we propose to introduce a new self-attention structure that is able to capture the domain perturbations and gradually remove them from the image features. As shown in Figure 3 , at layer 𝑙 of the proposed domain-conditioned transformer (DCT), we append three domain-conditioning vectors, [𝐶 𝑙 𝑞 , 𝐶 𝑙 𝑘 , 𝐶 𝑙 𝑣 ], into the query, key, and value components [𝑄 𝑙 , 𝐾 𝑙 , 𝑉 𝑙 ] of its self-attention module. At each transformer network layer, we learn a lightweight neural network (domain-conditioner generator Φ 𝑙 ) to generate these three domain conditioners [𝐶 𝑙 𝑞 , 𝐶 𝑙 𝑘 , 𝐶 𝑙 𝑣 ] from the class token 𝐶 𝑙 . During fully test-time adaptation, Φ 𝑙 is updated during the inference process. These domain conditioners at each transform network layer are able to gradually remove the impact of domain shift and significantly recover the original self-attention profile. In the following section, we explain the proposed DCT method in more detail.Self-attention mechanisms have demonstrated remarkable performance in various computer vision tasks by capturing correlation between image patches. The output matrix of self-attention is defined as:EQUATIONwhere 𝑑 represents the dimensions of the query, key, and value.For convenience, we consider layer 𝑙 of the network and omit the superscript 𝑙 here in this section. The self-attention weights are computed from the correlation between patch embeddings. Certainly, the self-attention distance profile defined in the previous section changes when the input image is perturbed by domain shifts. Write the query 𝑄 ∈ R 𝑛×𝑑 , key 𝐾 ∈ R 𝑛×𝑑 , and value 𝑉 ∈ R 𝑛×𝑑 matrices of the self-attention mechanism for the 𝑛 embedding tokens (n = N+1, including class token) as𝑄 = [q 1 , q 2 , • • • , q n ] ⊤ , 𝐾 = [k 1 , k 2 , • • • , k n ] ⊤ , 𝑉 = [v 1 , v 2 , • • • , v n ] ⊤ .The original correlation matrix between 𝑄 and 𝐾 is denoted by 𝑄𝐾 ⊤ = [𝛼 𝑖,𝑗 ] 𝑛×𝑛 . In our proposed DCT method, we introduce three domain conditioning vectors 𝐶 𝑞 ∈ R 1×𝑑 , 𝐶 𝑘 ∈ R 1×𝑑 , and 𝐶 𝑣 ∈ R 1×𝑑 and append them to the the query, key, and value matrices, respectively, and obtain the following augmented query Q ∈ R (𝑛+1) ×𝑑 , key K ∈ R (𝑛+1) ×𝑑 , and value V ∈ R (𝑛+1) ×𝑑 :EQUATIONThe correlation matrix between Q and K with domain conditioners is:EQUATIONNow, the new self-attention weight matrix 𝑊 = [𝑤 𝑖,𝑗 ] (𝑛+1) × (𝑛+1) is given by:𝑤 𝑖,𝑗 = softmax Q K⊤ =                        exp(𝛼 𝑖,𝑗 ) 𝑛 𝑗 =1 exp(𝛼 𝑖,𝑗 ) +exp(q i 𝐶 ⊤ 𝑘 ) , 𝑖 ≠ 𝑛 + 1, 𝑗 ≠ 𝑛 + 1; 𝑒 q i 𝐶 ⊤ 𝑘 𝑛 𝑗 =1 exp(𝛼 𝑖,𝑗 ) +exp(q i 𝐶 ⊤ 𝑘 ) , 𝑖 ≠ 𝑛 + 1, 𝑗 = 𝑛 + 1; exp(𝐶𝑞 k j ⊤ ) 𝑛 𝑗 =1 exp(𝐶𝑞 k j ⊤ ) +exp(𝐶𝑞𝐶 ⊤ 𝑘 ) , 𝑖 = 𝑛 + 1, 𝑗 ≠ 𝑛 + 1; exp(𝐶𝑞𝐶 ⊤ 𝑘 ) 𝑛 𝑗 =1 exp(𝐶𝑞 k j ⊤ ) +exp(𝐶𝑞𝐶 ⊤ 𝑘 ) , 𝑖 = 𝑛 + 1, 𝑗 = 𝑛 + 1.(EQUATIONThe conditioned self-attention output is:EQUATIONFrom ( 5) and ( 6), we can see that the original self-attention weights have been modified by the domain conditioner. The introduction of domain conditioners adds an additional context to the attention mechanism. The softmax score of 𝛼 𝑖,𝑗 is re-calibrated to consider both input data and the contextual information of the target domain.In our proposed DCT method for fully test time adaptation, these domain conditioners [𝐶 𝑞 , 𝐶 𝑘 , 𝐶 𝑣 ] are generated by a network that is learned online during the test-time adaptation process, which will be explained in the following section.At each transformer network layer, we introduce a dedicated lightweight network Φ 𝑙 , called domain conditioner generator to generate the three domain conditioners [𝐶 𝑙 𝑞 , 𝐶 𝑙 𝑘 , 𝐶 𝑙 𝑣 ] from the class token 𝐶 𝑙 :EQUATIONThis domain conditioner generator network is learned during the test-time adaptation process. Specifically, in the current mini-batch B 𝑡 , when training the network Φ 𝑙 , we aim to minimize the loss function L (𝜃 𝑡 ; 𝑥) with respect to the learnable weights 𝜃 𝑡 of network Φ 𝑙 . The loss function L (𝜃 𝑡 ; 𝑥) in test-time adaptation is commonly defined by the entropy of the given batch. In practice, we find such minimization will cause model collapse. To address this issue, we use the reliable entropy minimization along with the sharpnessaware minimization [7, 38] . The reliable entropy minimization filters out testing samples with relatively large entropy to reduce the impact of noisy samples on the model's fine-tuning and makes it more robust to incomplete or noisy data. The sharpness-aware entropy minimization encourages the model weights to converge to a flat minimum, indicating that the model is robust to small perturbations in the weights. The overall optimization loss is defined as:EQUATIONwhere I[E(𝜃 𝑡 ; 𝑥) < 𝐸 0 ] is the mask to filter out test samples when entropy is larger than the threshold 𝐸 0 , and E is the entropy function.Figure 4 We observe that the domain conditioners play a crucial role in capturing and gradually removing the domain perturbation from the image features throughout the transformer network layers. Figure 5 shows the class tokens 𝐶 𝑙 at different layers of our domainconditioned transformer. Specifically, the first 5 plots show the class token at layers 1, 3, 6, 11, and 12 of the domain-conditioned transformer. Each plot shows the samples from 5 different target domains (Gaussian Noise, Frost, Defocus Blur, Contrast, and Fog) with each domain being plotted with a different color. We can see that, with the proposed domain conditioning learning and adaptation, the domain information is being gradually removed from the class tokens. In the 5-th plot for layer 12, we can hardly see any domain difference among these samples. For comparison, in the 6-th plot, we also show the class token of layer 12 from the source model without using the proposed DCT method. We can see that the domain information is clearly seen in the final layer of the transformer model. This will significantly degrade the performance of the network in the target domain. 
experiments: 
 In this section, we conduct experiments on multiple online test-time adaptation settings and multiple dataset benchmarks to evaluate the performance of our proposed DCT method.In our experiments, we select the widely used ImageNet-C benchmark dataset [17] , consisting of 50, 000 instances distributed across 1, 000 classes. Additionally, we test in ImageNet-R [16] , a dataset containing 30,000 images presenting diverse artistic renditions of 200 classes from the ImageNet dataset. The results also include VisDA-2021 [2] , a dataset designed to assess models' ability to adapt to novel test distributions and effectively handle distributional shifts. Additionally, we utilize the Office-Home [50] dataset, which has a total of 15, 500 images spanning 65 object categories across four distinct domains. We compare our proposed DCT method against the following fully online test-time adaptation methods: no adaptation which is the source model, T3A, CoTTA, DDA, MEMO, TENT, AdaContrast, CFA, DePT-G, and SAR.Following the official implementations of SAR, we use the ViT-B/16 backbone for all experiments unless explicitly stated otherwise.The pre-trained model weights are obtained from the timm repository [54] . Specifically, for the Office-Home dataset, we fine-tune the ViT-B/16 model by replacing the original classifier head with a new classifier head. To ensure fair performance comparisons, all methods within each experimental condition share identical architecture and pre-trained model parameters. We employ the SGD optimizer with Sharpness Aware Minimization [8] . The batch size is set to 64 for all experiments, except for the condition Batch size = 1. Our reported experimental results are the mean and standard deviation values obtained from three runs, each with random seeds chosen from the set {2021, 2022, 2023}. It should be noted that we use the matched normalization setting for the pre-trained timm model (mean = [0.5, 0.5, 0.5], std = [0.5, 0.5, 0.5]), which is different from the code of the original SAR paper [38] . All models are tested on a single NVIDIA RTX3090 GPU. The source code will be released.We evaluate the performance of our DCT method under three different test conditions on the ImageNet-C dataset. We report the reproduced top-1 accuracy using the official codes for all methods under comparison. Specifically, (1) we first evaluate our approach under the Normal i.i.d assumption and compared it with other TTA methods. The results of this experiment are shown in Table 1 .Our method outperforms other baseline methods for almost all 15 corruptions. On average, our method outperforms the second-best method by 2.7%. ( 2) Then, we evaluate our approach under the Imbalanced label shifts test condition with the same imbalanced sample sequence. The results of this experiment are shown in Table 2 . We can see that our method improves the average classification accuracy of all 15 corruption types by 1.1%. (3) We evaluate our approach under the challenging Batch size = 1 test condition, which is known to be particularly difficult for TTA methods. As shown in Table 3 , our method improves the average classification accuracy of all 15 corruption types by 1.6%, demonstrating its superior robustness and adaptability with small batch sizes. We also conduct experiments on the ImageNet-R and VisDA-2021 datasets to verify the effectiveness of our method. For ImageNet-R, we use the same pre-trained ViT-B/16 backbone and set the output size to 200 following the procedure in [16] . From Table 5 and To explore the explainability of the domain-conditioned transformer, we visualize the attention map by Attention Rollout [1] following ViT [6] . As shown in Figure 6 , given the corruption image, we can see that the adapted transformer attention focuses more on the object than the source model. This demonstrates that our Domain-Conditioned Transformer (DCT) method significantly enhances the attention in the target domain.Additionally, we performed an ablation study on the domainconditioner generator shown in Table 7 . When solely integrating learnable domain-conditioners into query, key, and value without the domain-conditioner generator's conditional generation based on the class token, the average accuracy improved by 0.3%. In contrast, when adapting the domain-conditioner generator to generate domain conditioners conditioned by the class token, we observed a substantial improvement of 2.7%. It demonstrates the significant contribution of the class token conditioned domain-conditioner generator in enhancing the model's performance.We conduct parameter sensitivity analysis on the learning rates for the domain-conditioner generator with the Normal setting in ImageNet-C with ViT-B/16. As shown in Figure 7 , we can see that the performance is best when the learning rate of the domainconditioner generator is set to 0.01.  
conclusion: 
 Fully test-time adaptation is a challenging problem in computer vision, particularly in the presence of complex corruptions and shifts in the test data distribution. In this work, we have tackled the critical challenge of adapting transformer-based models to new domains, focusing on the significant deviation in self-attention profiles encountered in the target domain compared to the source domain. Specifically, we have introduced three domain-conditioning vectors, called domain conditioners into the self-attention module. By integrating these domain conditioners into the query, key, and value components of the self-attention module, we have effectively mitigated the impact of domain shift observed during inference. The dynamic generation of these domain conditioners at each transformer network layer, derived from the class token, allowed for a gradual removal of domain shift effects, thereby enabling the recovery of the original self-attention profile in the target domain. Our experimental results demonstrated that our proposed DCT method is able to significantly improve fully test-time adaptation performance. 
: 
 In layer 1, the features exhibit domain-separability and class-inseparability due to the presence of domain shift, with a considerable distance between domains and a small distance between classes. Our DCT method effectively mitigates the influence of domain shift over successive layers. Consequently, the domain distance decreases while the class distance increases, leading to the features domain-inseparable yet class-separable across the layers of the Domain-Conditioned Transformer.of TTA methods: (1) test-time training (TTT) [10, 35, 46] and (2) fully test-time adaptation [37, 38, 49, 51] , depending on whether source domain data is accessed or not. In this study, we focus on fully test-time adaptation.For fully test-time adaptation, the TENT method [51] updates the batch normalization module by minimizing entropy loss. The MEMO method [58] optimizes the entropy of averaged predictions over multiple random augmentations of input samples. Meanwhile, the VMP method [22] introduces perturbations into model parameters based on variational Bayesian inference. Note that these approaches assume a sufficiently large number of samples in a minibatch and a well-balanced label distribution in each mini-batch of the target domain. However, this assumption does not always hold in practice. To address this challenge, the SAR method [38] proposes a sharpness-aware and reliable optimization scheme, which eliminates samples with significant gradients and encourages model weights to converge to a flat minimum. The TTN method [31] optimizes interpolation weights during the post-training phase, requiring access to labeled source data. The RoTTA method [56] introduces robust batch normalization through category-balanced sampling.Recently, transformer-based methods have achieved remarkable success in various machine learning tasks due to their powerful self-attention capabilities. In this work, we propose to explore how transformer networks can be successfully adapted to new domains during the testing stage. During our experiments, we find that when transformer models are applied to new domains, their self-attention distance profiles, defined as the spatial distribution of self-attention between tokens, for image samples in the target domain deviate significantly from those in the source domain. Note that the selfattention is one of the core modules in the transformer network design. Once this self-attention profile has been perturbed by the domain changes or corruptions, the performance of the transformer model will degrade significantly. A research question arises: how do we remove these perturbations caused by the domain shifts from the self-attention profile so as to improve the test-time adaptation performance of transformer models?To address this challenge, we propose to explore a new approach, called domain-conditioned transformer, for fully test-time adaptation of transformer models. Specifically, we introduce three domainconditioning vectors, called domain conditioners, into the query, key, and value components of the self-attention modules. These domain conditioners are designed to capture domain-specific perturbation information and remove these perturbations layer by layer. We learn a domain conditioner generation network to generate these domain conditioners from class tokens of the previous network layers, containing both semantic and domain information. We observe that, our proposed approach gradually mitigates the impact of domain shift. As shown in Figure 1 , the domain information is gradually removed and the class information is enhanced across the domain-conditioned transformer layers. This gradual adjustment process facilitates the recovery of the original self-attention profile, allowing the model to maintain its performance across diverse domains. Our extensive experimental results demonstrate that the proposed domain-conditioned transformer significantly improves the online test-time domain adaptation performance and outperforms existing state-of-the-art methods by large margins. 
 
 Appendix:  


Decision: The paper initially got three borderline rejections. The authors have provided a rebuttal. After checking the rebuttal, two of the reviewers are satisfied with the rebuttal and changed their ratings to borderline accepts. However, the other reviewer kept the original borderline rejection, as he/she thinks the description of the domain conditioner generator network is not clear enough. The AC has carefully checked the rebuttal about the domain conditioner generator network and found that the authors have already clearly described it. Hence, the AC thinks all the concerns of the reviewers have been solved during rebuttal and this is a borderline accept paper. In the camera ready, the AC strongly encourages the authors to include all discussions in the rebuttal, provide more details of important parts, and release the source code.

Reviews: review: paper_summary: paper_summary: This paper focuses on fully test-time adaptation (TTA). The proposed domain-conditioned transformer construct domain-conditioners to capture domain-specific perturbation information and remove these perturbations layer by layer. In addition, domain-conditioned transformer learns a domain conditioner generation network to generates the abovementioned domain conditioners, which contains semantic and domain information. Experimental evaluation shows the effectiveness of the proposed method.
strength_weakness: strength_weakness: *	The idea seems interesting for dealing with TTA.
*	The paper did thorough experiments across multiple benchmarks with performance gains.
limitations: limitations: *	The motivation behind the adopted method, particularly regarding domain-conditioned self-attention, is not sufficiently clarified. It is essential to provide a more comprehensive explanation of the rationale behind this choice to strengthen the foundation of the proposed approach.
*	The implementation details of the domain-conditioner generator network are missing. The authors are suggested to provide more technical details about how to implement the domain-conditioner generator network.
*	In Tab. 7, what dataset was used to illustrate the ablation study results? Additionally, the backbone utilized in the ablation study was not provided.
*	The experiments on SAR [1] are all based on the ImageNet-C dataset. If the experimental results of SAR are reproduced by the public code, please given the relevant elucidations.

[1] Niu et al. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023.
paper_summary: paper_summary: In this paper, the authers explore fully online test-time adaptation of vision transformer models.  
They introduce a new design of the self-attention module in the transformer, which can capture the domain-specific characteristics of test samples in the target domain and is able to clean up the domain perturbations in the test samples.  
To be specific, they learn a lightweight neural network called domain-conditioner generator to generate the domain conditioners from the class token at each layer, and incorporate the three domain conditioners into the query, key, and value components of the self-attention module, respectively.  
Experimental results demonstrate effictiveness.
strength_weakness: strength_weakness: 1) The method  improves the online fully test-time domain adaptation performance and outperforms existing state-of-the-art methods.  
2) The running time cost of the proposed approach is promising.  
3) The paper is good writing and has elegant figures and tables.
limitations: limitations: 1) what is the complexity and structure of the domain conditioner generator? It is crucial to know model size of DC-generator because the component plays a major role in performance improvement according to Table 7.  
2) There are some typos, e.g. excess "the" in the line 448.  
3) The description of figures is not clear. For instance, the authers need to state that "Gaussian Noise", "Defocus Blur" and "Snow" indicate domains in Figure 1.
paper_summary: paper_summary: This paper proposes a Domain-Conditioned Transformer for fully test-time adaptation, introducing domain-conditioning vectors into the self-attention mechanism. During test time, a network is learned to generate these domain conditioners. Experimental results and analysis demonstrate the effectiveness of the proposed approach.
strength_weakness: strength_weakness: 1. The authors conduct extensive experiments on various online test-time adaptation settings across multiple dataset benchmarks, demonstrating that the proposed model outperforms previous SOTA method SAR.
2. The visualizations of output class tokens in Figure 1 and Figure 5 exhibit that the proposed method effectively reduces the inter-domain distance while increasing the inter-class distance,  thereby mitigating the influence of domain gap. Figure 6 also shows improvements in attention within the target domain.
limitations: limitations: 1. The proposed method need further explanation. The domain conditioners, which are actually prompts generated from the class tokens, require a clearer illustration regarding how these vectors leverage information from class tokens to mitigate the impact of domain shift. It would be also beneficial for the authors to further discuss the relationship with prompt learning.
2. In Section 3.1 and Figure 2, the authors investigate the self-attention profile. There is confusion regarding why domain shift results in larger attention distances and how increased attention distance affects performance.
3. Figure 4 shows the t-SNE visualization of the domain conditioners, but I am wondering that whether different domains can be separated so distinctly. Is there any post-processing applied? I recommend the authors provide a detailed illustration of the visualization procedure.