Controllable Continual Test-Time Adaptation

## Abstract

Abstract Continual Test-Time Adaptation (CTTA) is an emerging and challenging task where a model trained in a source domain must adapt to continuously changing conditions during testing, without access to the original source data. CTTA is prone to error accumulation due to uncontrollable domain shifts, leading to blurred decision boundaries between categories. Existing CTTA methods primarily focus on suppressing domain shifts, which proves inadequate during the unsupervised test phase.
In contrast, we introduce a novel approach that guides rather than suppresses these shifts.
Specifically, we propose C ontrollable Co ntinual T est- T ime A daptation (C-CoTTA), which explicitly prevents any single category from encroaching on others, thereby mitigating the mutual influence between categories caused by uncontrollable shifts.
Moreover, our method reduces the sensitivity of model to domain transformations, thereby minimizing the magnitude of category shifts.
Extensive quantitative experiments demonstrate the effectiveness of our method, while qualitative analyses, such as t-SNE plots, confirm the theoretical validity of our approach. Our code is available at https://github.com/RenshengJi/C-CoTTA.

## 1 Introduction

Continual Test-Time Adaptation (CTTA) [] is becoming an emerging field, which explores the adaptability of any machine learning model during test time in dynamic environments.
The primary objective of CTTA is to enable a pretrained model to adapt to continuously changing scenarios, where the distribution of data shifts over time.
CTTA is practical in many long-term intelligent applications, such as autopilot [, , ], monitoring [], medical image analysis [, ], where models need to remain robust and accurate against possible changes over extended periods.

The main challenge of CTTA is the accumulation of errors caused by the lack of real labels and the continuous domain shifts.
Existing methods mainly avoid error accumulation by using strategies such as Mean Teacher [], augmentation-averaged predictions [, , ], and selecting reliable samples [, , , ] to minimize the impact of domain shift.
However, these methods focus on suppress the shifts, few methods explicitly attempt to guide or control the shifts.
This is becuase the lack of labels in the target domain and the inability to obtain source domain data in CTTA.
We evaluate this on CIFAR10C, as shown in Fig. 1(a), we find that in the domain adaptation process, multiple categories to lean towards a confusion region.
In fact, CTTA meets continuous unknown domain shifts, making this shift almost uncontrollable.
This phenomenon will worsen with the changing domain over time, resulting in error accumulation.

Therefore, we hypothesize that instead of attempting to suppress the shift, guiding or controlling the shift to remain class-separable may also achieve effective adaptation.
On top of this, the main focus of this paper is to study how to achieve controllable domain shift in CTTA.
Generally, controlling domain shifts means to control the direction of the shift in feature space.
First of all, we first need to accurately represent the direction of domain shift.
Inspired by the interpretable machine learning [], we represent shift directions using the tool of Concept Ativation Vectors (CAV) [], which represents the transformation path from one concepts to another.
In CTTA, the CAV can be represented by the vector from one prototype to another.
First, for a specific category of domain shift direction, we obtain it by subtracting the category prototype of the target domain in the feature space from the category prototype of the source domain. Then, for the overall domain shift direction, we obtain it by subtracting the overall prototype of the target domain in the feature space from the overall prototype of the source domain.

To further control domain shift, we construct the Control Domain Shift (CDS) loss and the Control Class Shift (CCS) loss.
CDS refers to controlling the shift of the overall domain by constraining the model’s sensitivity in that direction, thus reducing the impact of domain shift on model performance.
CCS controls the shift of specific categories by constraining the shift direction of each category to avoid biasing other categories.
As shown in Fig. 1(b), our method achieve controllable domain shift, and the direction of the shift will not blur the classification boundaries.
Extensive experiments are conducted on three large-scale benchmark datasets to validate the effectiveness of the proposed C-CoTTA framework in various challenging and realistic scenarios.

Figure: Figure 1: t-SNE visualization of controllable domain shift in CTTA. (a) For CoTTA [], due to the lack of control over domain shift, categories being biased towards others, resulting in fuzzy classification boundaries. (b) In contrast, our method achieves controllable domain shift, so even if categories are shift, it will not lead to confusion among categories.
Refer to caption: x1.png

Our contributions are three-fold:

- (1)
We evaluate and find that only suppress domain shifts is insufficient, which may lead to blurred classification boundaries. In contrast, we propose to guide and control shifts to keep the class-separability.
- (2)
We propose a simple and effective direction representation based on Concept Ativation Vectors (CAV) in interpretable machine learning, which utilizes the difference between two prototypes in the feature space.
- (3)
We propose to explicitly control the direction of specific category bias by preventing any category from leaning towards other categories, in order to prevent the blurring of classification boundaries; at the same time, by reducing the sensitivity of the model to domain shift, we control the overall domain shift to alleviate the impact of domain shift on domain adaptation.

## 2 Related work

### 2.1 Continual Test-Time Adaptation

Continual Test-Time Adaptation (CTTA) [, , ] is an emerging paradigm within the machine learning community designed to address the dynamic nature of real-world data distributions. Unlike traditional Test-Time Adaptation (TTA) [, , ], which typically assumes a fixed target domain in a source-free and online manner, CTTA operates under the assumption that the target domain may change over time. The main challenge in CTTA is the potential for catastrophic forgetting and error accumulation. During the test time, as the model adapts to new distributions, it risks losing previously learned knowledge, which can result in a degradation in performance known as catastrophic forgetting []. Moreover, the utilization of pseudo-labels derived from the model’s own predictions can introduce errors, which may accumulate over time [, ], especially when there are frequent domain shifts.

To address the challenges of error accumulation in CTTA, researchers have developed various strategies.
A number of works [, , , ] employ augmentation-averaged predictions for the teacher model to boost the teacher’s confidence, while others[, ] add perturbations to the student to enhance the model’s robustness.
Various methods[, , , , ] focuses on selecting reliable samples to eliminate the impact of misclassified samples on domain adaptation.
As to the challenge of catastrophic forgetting, Wang et al. [] and Brahma et al.[]
believe that the source model is more reliable, thus they are designed to restore the source parameters.
While these studies address the CTTA issue at the model level, other research efforts [, , ] leverage visual domain prompts or a limited subset of parameters to extract ongoing target domain knowledge.
However, these approaches primarily focus on suppressing domain shift and there are few methods that explicitly attempt to guide or control domain shift.

### 2.2 Concept Activation Vectors

Concept Activation Vectors (CAVs) is an interpretability tool for explaining decision-making processes in deep learning models. Originally, the authors of [] define CAV as the normal to a hyperplane that separates examples without a concept from examples with a concept in the model’s latent activations. This hyperplane is commonly computed by solving a classification problem, for example, using Support Vector Machines (SVMs)[], ridge[], lasso[] or logistic regression [].
Given its ability to effectively orient concepts, CAVs have been employed for a plethora of tasks in recent years, such as concept sensitivity testing [], model correction for shortcut removal  [, , ],
knowledge discovery by investigation of internal model states [],
and training of post-hoc concept bottleneck models [].
However, common regression-based methods tend to deviate from the true conceptual direction due to factors such as noise in the data []. To that end, signal-pattern-based CAVs (referred to “signalCAVs”) have been proposed [], which are more robust against noise [, , , ].
However, during test-time, we may not have access to all samples of a prototype, and due to the lack of true labels, misclassified samples may contaminate the prototype. Therefore, the construction of a prototype is different from interpretable machine learning.

## 3 Methodology

### 3.1 Problem Definition

Given a classification model pre-trained on a source domain, CTTA methods adapt the source model to the unlabeled target data, where the domain continuously changes.
The unsupervised dataset of target domains are denoted as $\mathcal{D}^{k}=\{x^{k}_{m}\}^{N^{k}}_{m=1}$, where $k$ is the target domain index.
As shown in Figure 1, in the process of CTTA, if the domain shifts are not controlled, some categories may generate bias towards other categories, resulting in blurred classification boundaries.
In this paper, we propose to explicit control over the shift direction in CTTA.
specific categories and the overall domain.
In the following, we first study how to represent domain shifts in Sec. 3.2.
Then, we propose to control the shift within the process of CTTA in Sec. 3.3.

### 3.2 Representing Shift via CAV

Domain shift refers to the distribution shift of each class that occurs in the feature space, generally caused by differences between the target domain distribution in the testing phase and the source domain distribution in the training phase.
Therefore, how to represent the domain shift direction in the feature space in CTTA is a challenge.

In the field of interpretable machine learning [, ], CAV [] refer to the normal to a hyperplane that separates examples without a concept from examples with a concept in the model’s latent activations feature space.
CAV is widely used in areas such as model correction for shortcut removal [, , ].
The concept in CAV generally refers to high-level semantic information, such as whether there are a large number of striped structures in an image.
For example, in the concept of stripe, the label for features extracted from images with stripes is 1, and without stripes is 0.

CAV can be calculated in different methods, we use the signal-pattern-based CAVs (SCAV) [] which provide a simple but effective way to represent the CAV $\mathbf{v}$ as follows:

$$ $\mathbf{v}=\frac{\textrm{cov}[f(x),t]}{\textrm{cov}[t,t]}=\frac{\sum(f(x_{i})- \overline{f}(x))(t_{i}-\overline{t})}{\sum(t_{i}-\overline{t})^{2}},\quad t_{i }=\begin{cases}1&\text{if }x_{i}\in\mathcal{X}_{c},\\ 0&\text{if }x_{i}\in\mathcal{X}_{n}.\end{cases},$ (1) $$

where $f(\cdot)$ represents the feature extractor and $t$ represents the concept label of the features, $\mathcal{X}_{c}$ and $\mathcal{X}_{n}$ denote sets of samples that either possess or lack the specified concept, respectively.
Mean feature $\overline{f}(x)=\frac{1}{N}\sum f(x_{i})$ and mean label $\overline{t}=\frac{1}{N}\sum t_{i}$.

Figure: Figure 2: The pipeline of C-CoTTA. (a) Based on the mean teacher framework, perturb the student to enhance model robustness, while optimizing using symmetric cross-entropy. (b) Control the overall domain shift by constraining the model’s sensitivity to domain shift directions. (c) Control the shift of a specific category by directly controlling the shift direction of any category to prevent bias towards other categories.
Refer to caption: x2.png

Nevertheless, in the traditional CAV application scenario, the concept label $t$ is manually annotated offline. In contrast, in unsupervised CTTA scenario, to represent the domain shift direction in an online manner is intractable.
Therefore, we use the pseudo-label $y$ obtained by the existing model to represent concepts automatically.
The shift representation for the direction in CTTA can be computed as follows:

$$ $\mathbf{v}=\frac{\textrm{cov}[f(x),y]}{\textrm{cov}[y,y]}=\frac{\sum(f(x_{i})- \overline{f}(x))(y_{i}-\overline{y})}{\sum(y_{i}-\overline{y})^{2}},\quad y_{i }=\begin{cases}1&\text{if }x_{i}\in\mathcal{X}_{t},\\ 0&\text{if }x_{i}\in\mathcal{X}_{s}.\end{cases},$ (2) $$

At the same time, through deduction (refer to Appendix A for details), it is determined that the direction can be further represented as the difference between two prototypes in the feature space.

$$ $\mathbf{v}=\frac{1}{|\mathcal{X}_{t}|}{\sum\limits_{x_{i}\in\mathcal{X}_{t}}f( x_{i})}-\frac{1}{|\mathcal{X}_{s}|}{\sum\limits_{x_{i}\in\mathcal{X}_{s}}f(x_{ i})}=\mathbf{p}_{t}-\mathbf{p}_{s}.$ (3) $$

Then, we can use this to calculate various directional vectors, in order to further control domain shift.

Domain-level source-to-target shift.
We represent the shift of the overall domain as below:

$$ $\mathbf{v}^{s\to t}=\mathbf{p}^{t}-\mathbf{p}^{s},$ (4) $$

This calculates by subtracting the target domain prototype $\mathbf{p}^{t}$ composed of all samples in each batch at test-time from the source domain prototype $\mathbf{p}^{s}$ composed of prototypes of all categories in the source domain as shown in Figure 2(b).

Class-level source-to-target shift.
We also represent the shift of a specific category as below:

$$ $\mathbf{v}_{i}^{s\to t}=\mathbf{p}^{t}_{i}-\mathbf{p}^{s}_{i},$ (5) $$

where we construct it by subtracting the prototype $\mathbf{p}^{t}_{i}$ composed of trustworthy samples belonging to that category in each batch at test-time from the source domain prototype $\mathbf{p}^{s}_{i}$ of that category as shown in Figure 2(c). Moreover, during test time, some samples may be misclassified due to the lack of real labels, leading to distortion in the extracted prototypes. Therefore, we estimate the entropy $H(x^{t})$ for each sample $x^{t}$ from the target domain using the model. We then set aside samples with an entropy exceeding a predefined threshold $E_{0}$ following [].

Class-to-class source shift.
We construct class relative shift from category $i$ to $j$ as below:

$$ $\mathbf{v}_{i\to j}^{s}=\mathbf{p}^{s}_{j}-\mathbf{p}^{s}_{i}.$ (6) $$

This calculates by using the source domain category prototypes $\mathbf{p}^{s}_{i}$ and $\mathbf{p}^{s}_{j}$ as shown in Figure 2(c).

### 3.3 Control of shift

We control the shift at domain and class levels.
On one hand, the shift of different categories are influenced by the overall domain shift, showing a certain degree of convergence, meaning that after domain shift, the distances between different categories in the feature space become closer. This finding is similar to the conclusions drawn in studies such as  [, , ], where researchers found that the feature norms in the target domain are relatively small, so we control the overall domain shift.
On the other hand, the shift of different categories is related to their characteristics.

#### Domain-level shift controlling.

First, we propose to control the shift of the overall domain at the domain level. Specifically, we model the shift sensitivity by considering the gradient of the model output $h(g(x_{i}))$ with respect to the feature map $g(x_{i})$, and then combine it with the overall domain shift direction ${\mathbf{v}^{s\to t}}$. We can then reduce the sensitivity of the model to domain shift by constraining the direction gradient as follows:

$$ $\displaystyle\mathcal{L}_{\mathrm{CDS}}=\sum\limits_{x_{i}\in{\mathcal{X}_{b}} }\left|\frac{\partial h(g(x_{i}))}{\partial g(x_{i})}\cdot{\mathbf{v}^{s\to t} }\right|,$ (7) $$

where $h(\cdot)$ represents the remaining part of the model. Intuitively, the Control Domain Shift (CDS) loss $\mathcal{L}_{\mathrm{CDS}}$ enforces the model output to not change when slightly adding or removing activations along the bias direction as follows:

$$ $\displaystyle\lim_{\epsilon\rightarrow 0}\frac{h(g(\mathbf{x})+\epsilon{ \mathbf{v}^{s\to t}})-h(g(\mathbf{x}))}{\epsilon}=0.$ (8) $$

Thus, by minimizing $\mathcal{L}_{\mathrm{CDS}}$, the model becomes insensitive towards the domain shift direction, thereby reducing the impact of domain shift on the domain adaptation process.

Figure: Algorithm 1 Controllable Continual Test-Time Adaptation

#### Class-level shift controlling.

Then, in order to prevent uncontrollable shifts of each category,
we propose to control class-level shift to avoid any category leaning towards other categories.
This requires that the direction of shift $\mathbf{v}_{i}^{s\to t}$ for any category is preferably the opposite direction of the direction $\mathbf{v}_{i\to j}^{s}$ of other categories relative to that category as shown in Fig. 2(c).
This means that the dot product of $\mathbf{v}_{i}^{s\to t}$ and $\mathbf{v}_{i\to j}^{s}$ should be as small as possible. The Control Class Shift (CCS) loss is calculated as follows:

$$ $\displaystyle\mathcal{L}_{\mathrm{CCS}}=\sum_{i=1}^{c}\sum_{j\neq i}^{c}\frac{ \mathbf{v}_{i}^{s\to t}}{\|{\mathbf{v}_{i}^{s\to t}}\|_{2}}\cdot\frac{{\mathbf {v}_{i\to j}^{s}}}{\|{\mathbf{v}_{i\to j}^{s}}\|_{2}},$ (9) $$

where we normalize $\mathbf{v}_{i}^{s\to t}$ and $\mathbf{v}_{i\to j}^{s}$.
The loss $\mathcal{L}_{\mathrm{CCS}}$ is to prevent any category from shifting towards other categories, achieving controllable domain shift, effectively preventing the decrease in classification performance caused by blurred category boundaries in continuous domain adaptation.

### 3.4 Overall Objective

In addition, our work also utilizes $\mathcal{L}_{\mathrm{SCE}}$ derived from [], which following the mean teacher framework, which involves simply averaging the weights of a student model over time, the resulting teacher model provides a more accurate prediction function than the final function of the student, meanwhile perturbs the student to enhance the robustness of the model [, ] and employs the symmetric cross-entropy (SCE) [] which has superior gradient properties compared to the commonly used cross-entropy loss. For enhancing the output of students $q$ and teachers $p$, the symmetric cross-entropy is defined as follows:

$$ $\displaystyle\mathcal{L}_{\mathrm{SCE}}=-\sum\nolimits_{c=1}^{C}\mathbf{q}_{c} \,\mathrm{log}\,\mathbf{p}_{c}-\sum\nolimits_{c=1}^{C}\mathbf{p}_{c}\,\mathrm{ log}\,\mathbf{q}_{c}.$ (10) $$

The overall objective of our proposed continual test-time adaptation method is as follows:

$$ $\displaystyle\mathcal{L}=\mathcal{L}_{\mathrm{SCE}}+\lambda_{1}\mathcal{L}_{ \mathrm{CDS}}+\lambda_{2}\mathcal{L}_{\mathrm{CCS}},$ (11) $$

where $\lambda_{1}$, and $\lambda_{2}$ are the hyperparameters. $\mathcal{L}_{\mathrm{CDS}}$ refers to controlling the shift of the overall domain, which constrains the model’s sensitivity in that direction. $\mathcal{L}_{\mathrm{CCS}}$ controls the shift of specific categories, which constrains the shift direction of each category to avoid biasing other categories. The overall frame diagram is shown in Figure 2.

We illustrate the whole algorithm in Algorithm 1. First, before domain adaptation begins, we use the source domain category prototypes to calculate the inter-class relative direction vector $\mathbf{v}_{i\to j}^{s}$. During domain adaptation, on one hand, we calculate the shift direction $\mathbf{v}_{i}^{s\to t}$ for specific categories and constrain it through the loss $\mathcal{L}_{\mathrm{CCS}}$; on the other hand, we calculate the shift direction $\mathbf{v}^{s\to t}$ for the entire domain and constrain it through the loss $\mathcal{L}_{\mathrm{CDS}}$. Additionally, we compute the symmetric cross-entropy loss for the prediction logits of the student and teacher, optimize it via Eq. 11, and update the student and teacher models.

## 4 Experiments

### 4.1 Experimental Setting

Datasets.
We evaluate our proposed method on three continual test-time adaptation benchmark: CIFAR10-C, CIFAR100-C, and ImageNet-C. Each dataset contains 15 types of corruptions with 5 levels of severity, ranging from 1 to 5. For simplicity in tables, we use Gauss., Impul., Defoc., Brit., Contr., Elas. and Pix. to represent Gaussian, Impulse, Defocus, Brightness, Contrast, Elastic,and Pixelate, respectively.

Pretrained Model.
Following previous studies [, ], we adopt the pretrained WideResNet-28 [], ResNeXt-29 [] and ResNet-50 [] for CIFAR10-C, CIFAR100-C and Imagenet-C, respectively. Similar to CoTTA, we update all the trainable parameters in all experiments.

Methods to be Compared.
We compare our C-CoTTA with the original model (Source) and multiple state-of-the-art (SOTA) methods such as BN [, ], TENT [], CoTTA [], RoTTA [], SATA [], SWA [], PETAL [], RMT [], DSS []. All compared methods utilize the same backbone and pretrained model.
All experiments are conducted on a single RTX 4090.

### 4.2 Results for Continual Test-Time Adaptation

Experiments on CIFAR10-C.
We first evaluate the effectiveness of the proposed model on the CIFAR10-C dataset. We compare our method to the source-only baseline and nine SOTA methods.
As shown in Table 1, directly using pre-trained model without adaptation yields a high average error rate of 43.5%. BN method improve the performance by 23.1% compared to the source-only baseline.
Among all comparison methods, SWA achieve the lowest error rate of 11.2%, 11.8% and 11.5% on Defoc., Motion and Fog, respectively. Both PETAL and SATA achieve the lowest error rate of 7.4% on Brit.. In other conditions, our proposed method outperforms or is comparable to all the above methods. In conclusion, our method achieve the lowest average error rate, which is reduced to 14.7%.

**Table 1: Classification error rate (%) for standard CIFAR10-C continual test-time adaptation task.**
| Method | Gauss. | Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Brit. | Contr. | Elas. | Pix. | Jpeg | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Source | 72.3 | 65.7 | 72.9 | 46.9 | 54.3 | 34.8 | 42.0 | 25.1 | 41.3 | 26.0 | 09.3 | 46.7 | 26.6 | 58.5 | 30.3 | 43.5 |
| BN [, ] | 28.1 | 26.1 | 36.3 | 12.8 | 35.3 | 14.2 | 12.1 | 17.3 | 17.4 | 15.3 | 08.4 | 12.6 | 23.8 | 19.7 | 27.3 | 20.4 |
| TENT [] | 24.8 | 20.6 | 28.5 | 15.1 | 31.7 | 17.0 | 15.6 | 18.3 | 18.3 | 18.1 | 11.0 | 16.8 | 23.9 | 18.6 | 23.9 | 20.1 |
| CoTTA [] | 24.5 | 21.5 | 25.9 | 12.0 | 27.7 | 12.2 | 10.7 | 15.0 | 14.1 | 12.7 | 07.6 | 11.0 | 18.5 | 13.6 | 17.7 | 16.3 |
| RoTTA  [] | 30.3 | 25.4 | 34.6 | 18.3 | 34.0 | 14.7 | 11.0 | 16.4 | 14.6 | 14.0 | 08.0 | 12.4 | 20.3 | 16.8 | 19.4 | 19.3 |
| RMT  [] | 24.1 | 20.2 | 25.7 | 13.2 | 25.5 | 14.7 | 12.8 | 16.2 | 15.4 | 14.6 | 10.8 | 14.0 | 18.0 | 14.1 | 16.6 | 17.0 |
| PETAL [] | 23.7 | 21.4 | 26.3 | 11.8 | 28.8 | 12.4 | 10.4 | 14.8 | 13.9 | 12.6 | 07.4 | 10.6 | 18.3 | 13.1 | 17.1 | 16.2 |
| SATA [] | 23.9 | 20.1 | 28.0 | 11.6 | 27.4 | 12.6 | 10.2 | 14.1 | 13.2 | 12.2 | 07.4 | 10.3 | 19.1 | 13.3 | 18.5 | 16.1 |
| DSS [] | 24.1 | 21.3 | 25.4 | 11.7 | 26.9 | 12.2 | 10.5 | 14.5 | 14.1 | 12.5 | 07.8 | 10.8 | 18.0 | 13.1 | 17.3 | 16.0 |
| SWA [] | 23.9 | 20.5 | 24.5 | 11.2 | 26.3 | 11.8 | 10.1 | 14.0 | 12.7 | 11.5 | 07.6 | 09.5 | 17.6 | 12.0 | 15.8 | 15.3 |
| Ours | 22.7 | 17.9 | 23.8 | 11.6 | 24.3 | 12.8 | 09.5 | 13.1 | 12.4 | 11.6 | 08.0 | 09.5 | 16.4 | 11.4 | 15.6 | 14.7 |

Experiments on CIFAR100-C.
To further demonstrate the effectiveness of the proposed method, we evaluate it on the more difficult CIFAR100-C task with the source-only baseline and eight SOTA methods. The experimental results are shown in Table 2. Generally speaking, our method not only achieved the lowest error rates on the Snow, Frost, Fog, and Pix. tasks but also had the lowest average error rate. We improve the performance by 16.5% and 0.4% compared to the source-only baseline and SATA, respectively.

**Table 2: Classification error rate (%) for standard CIFAR100-C continual test-time adaptation task.**
| Method | Gauss. | Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Brit. | Contr. | Elas. | Pix. | Jpeg | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Source | 73.0 | 68.0 | 39.4 | 29.3 | 54.1 | 30.8 | 28.8 | 39.5 | 45.8 | 50.3 | 29.5 | 55.1 | 37.2 | 74.7 | 41.2 | 46.4 |
| BN [, ] | 42.1 | 40.7 | 42.7 | 27.6 | 41.9 | 29.7 | 27.9 | 34.9 | 35.0 | 41.5 | 26.5 | 30.3 | 35.7 | 32.9 | 41.2 | 35.4 |
| TENT [] | 37.2 | 35.8 | 41.7 | 37.9 | 51.2 | 48.3 | 48.5 | 58.4 | 63.7 | 71.1 | 70.4 | 82.3 | 88.0 | 88.5 | 90.4 | 60.9 |
| CoTTA [] | 40.1 | 37.7 | 39.7 | 26.9 | 38.0 | 27.9 | 26.4 | 32.8 | 31.8 | 40.3 | 24.7 | 26.9 | 32.5 | 28.3 | 33.5 | 32.5 |
| RoTTA  [] | 49.1 | 44.9 | 45.5 | 30.2 | 42.7 | 29.5 | 26.1 | 32.2 | 30.7 | 37.5 | 24.7 | 29.1 | 32.6 | 30.4 | 36.7 | 34.8 |
| RMT  [] | 40.2 | 36.2 | 36.0 | 27.9 | 33.9 | 28.4 | 26.4 | 28.7 | 28.8 | 31.1 | 25.5 | 27.1 | 28.0 | 26.6 | 29.0 | 30.2 |
| PETAL [] | 38.3 | 36.4 | 38.6 | 25.9 | 36.8 | 27.3 | 25.4 | 32.0 | 30.8 | 38.7 | 24.4 | 26.4 | 31.5 | 26.9 | 32.5 | 31.5 |
| DSS [] | 39.7 | 36.0 | 37.2 | 26.3 | 35.6 | 27.5 | 25.1 | 31.4 | 30.0 | 37.8 | 24.2 | 26.0 | 30.0 | 26.3 | 31.1 | 30.9 |
| SATA [] | 36.5 | 33.1 | 35.1 | 25.9 | 34.9 | 27.7 | 25.4 | 29.5 | 29.9 | 33.1 | 23.6 | 26.7 | 31.9 | 27.5 | 35.2 | 30.3 |
| Ours | 38.1 | 34.8 | 36.4 | 27.1 | 34.3 | 27.7 | 26.1 | 28.7 | 28.5 | 30.9 | 24.1 | 26.2 | 28.2 | 26.2 | 31.2 | 29.9 |

Experiments on ImageNet-C.
The last experiment is conducted on ImageNet-C to further demonstrate the effectiveness of the proposed method. The experimental results can be seen in Table 3. Compared with the SOTA methods, the proposed method achieve the lowest average error rate. Noteworthily, the proposed method outperforms SATA by a large margin for the Shot(71.6% vs. 72.9%), Impul.(68.7% vs. 71.6%), Defoc.(74.0% vs. 75.7%) and Glass.(71.6% vs. 74.1%) corruptions.

**Table 3: Classification error rate (%) for standard ImageNet-C continual test-time adaptation task.**
| Method | Gauss. | Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Brit. | Contr. | Elas. | Pix. | Jpeg | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Source | 95.3 | 95.0 | 95.3 | 86.1 | 91.9 | 87.4 | 77.9 | 85.1 | 79.9 | 79.0 | 45.4 | 96.2 | 86.6 | 77.5 | 66.1 | 83.0 |
| BN [, ] | 87.7 | 87.4 | 87.8 | 88.0 | 87.7 | 78.3 | 63.9 | 67.4 | 70.3 | 54.7 | 36.4 | 88.7 | 58.0 | 56.6 | 67.0 | 72.0 |
| TENT [] | 81.6 | 74.6 | 72.7 | 77.6 | 73.8 | 65.5 | 55.3 | 61.6 | 63.0 | 51.7 | 38.2 | 72.1 | 50.8 | 47.4 | 53.3 | 62.6 |
| CoTTA [] | 84.7 | 82.1 | 80.6 | 81.3 | 79.0 | 68.6 | 57.5 | 60.3 | 60.5 | 48.3 | 36.6 | 66.1 | 47.2 | 41.2 | 46.0 | 62.7 |
| RoTTA  [] | 88.3 | 82.8 | 82.1 | 91.3 | 83.7 | 72.9 | 59.4 | 66.2 | 64.3 | 53.3 | 35.6 | 74.5 | 54.3 | 48.2 | 52.6 | 67.3 |
| RMT  [] | 79.9 | 76.3 | 73.1 | 75.7 | 72.9 | 64.7 | 56.8 | 56.4 | 58.3 | 49.0 | 40.6 | 58.2 | 47.8 | 43.7 | 44.8 | 59.9 |
| PETAL [] | 87.4 | 85.8 | 84.4 | 85.0 | 83.9 | 74.4 | 63.1 | 63.5 | 64.0 | 52.4 | 40.0 | 74.0 | 51.7 | 45.2 | 51.0 | 67.1 |
| DSS [] | 84.6 | 80.4 | 78.7 | 83.9 | 79.8 | 74.9 | 62.9 | 62.8 | 62.9 | 49.7 | 37.4 | 71.0 | 49.5 | 42.9 | 48.2 | 64.6 |
| ViDA [] | 79.3 | 74.7 | 73.1 | 76.9 | 74.5 | 65.0 | 56.4 | 59.8 | 62.6 | 49.6 | 38.2 | 66.8 | 49.6 | 43.1 | 46.2 | 61.2 |
| SATA [] | 74.1 | 72.9 | 71.6 | 75.7 | 74.1 | 64.2 | 55.5 | 55.6 | 62.9 | 46.6 | 36.1 | 69.9 | 50.6 | 44.3 | 48.5 | 60.1 |
| Ours | 75.1 | 71.6 | 68.7 | 74.0 | 71.6 | 65.1 | 56.4 | 55.7 | 61.0 | 49.3 | 41.3 | 61.9 | 49.1 | 44.8 | 46.0 | 59.4 |

**Table 4: Ablation study on class and domain under severity 5.**
| No. | CCS | CDS | CIFAR10-C | CIFAR100-C | ImageNet-C |
| --- | --- | --- | --- | --- | --- |
| 1 |  |  | 15.66 | 31.19 | 60.56 |
| 2 | ✓ |  | 14.94 | 30.42 | 59.67 |
| 3 |  | ✓ | 15.16 | 30.68 | 59.75 |
| 4 | ✓ | ✓ | 14.71 | 29.92 | 59.43 |

### 4.3 Ablation Studies

We perform ablation study experiments to evaluate the effectiveness of major components of C-CoTTA on three benchmarks. For simplicity, we denote the Control Class Shift as CCS and Control Domain Shift as CDS. As shown in Table 4, C-CoTTA decrease error rates in all benchmarks after adding CCS or CDS to the network, which indicate mitigating inter-category interference or reducing the model’s sensitivity to overall domain shifts can improve the classification accuracy. Furthermore, the combination of CCS and CDS can further improve the classification accuracy of the C-CoTTA framework.

### 4.4 t-SNE Visualization for Class Shift

We use t-SNE [] for dimensionality reduction to visualize domain shift situations of different methods during test-time. As shown in Figure 3, it can be observed that directly using pre-trained model(Source), CoTTA, and SATA methods do not explicitly control domain shift, resulting in unpredictable domain shift directions, leading to blurry classification boundaries and mutual interference between categories. In contrast, our method implements controllable domain shift, so it can be seen that although categories also experience shift, the direction of the shift is benign and does not cause mutual interference between categories.

Figure: Figure 3: Visualization of the t-SNE dimensionality reduction of three classes from CIFAR10-C dataset (three easily misclassified animals: bird, deer, frog) transferred from the source domain to the target domain (zoom)
Refer to caption: x3.png

### 4.5 Inter-Class Distance and Inter-Domain Distance

Inter-Class Distance. The inter-class distance can be formulate as $d_{ic}=\sum_{i=1}^{c}\sum_{j\neq i}^{c}\left\|\mathbf{p}_{i}^{t}-\mathbf{p}_{j
}^{t}\right\|_{2}^{2}$, where $\mathbf{p}_{i}^{t}$ and $\mathbf{p}_{j}^{t}$ denote target domain category prototype.
The inter-class distance comparison between our method, CoTTA and SATA can be seen in Figure 4(a). Compared to CoTTA and SATA, our method has a larger inter-class distance, indicating better separability between classes and reflecting the effectiveness of controlling class shift direction.

Inter-Domain Distance. The inter-domain distance can be computed as $d_{id}=\left\|\mathbf{p}^{s}-\mathbf{p}^{t}\right\|_{2}^{2}$, where $\mathbf{p}^{s}$ denotes overall prototype of the source domain while $\mathbf{p}^{t}$ denotes overall prototype of the target domain. As shown in the Figure 4(b), compared to SATA, our method has smaller inter-domain distances, indicating that the model is less sensitive to domain transformations and reflecting the effectiveness of controlling domain shift. At the same time, although the CoTTA method has relatively small inter-domain distances in the early stage, as the target domain changes at test time, its inter-domain distances gradually increase. This indicates that the model is sensitive to domain shift, further emphasizing the importance of reducing the sensitivity of the model to domain shift in controlling overall domain shift.

Figure: Figure 4: (a) Inter-class distance: can indicate the separability between classes. (b) Inter-domain distance: can indicate the sensitivity of the model to domain transformations.
Refer to caption: x4.png

### 4.6 Results for Gradual Test-Time Adaptation

In the standard setup described above, corruption types change abruptly at the highest severity level. We will now present the results of the gradual setup. We design the sequence by gradually changing the severity for the 15 types of corruption. When the type of corruption changes, the severity level is at its lowest. The distribution shift within each type is also gradual. Table 4.6 shows that the proposed method outperforms competing methods.

Figure: Table 5: Results for Gradual Adaptation

### 4.7 Results for Corruption Loops Test-Time Adaptation

In real-world scenarios, test domains may occur in cycles. To assess the long-term adaptation performance of the method, we evaluated the test conditions for 10 consecutive cycles. This means that at level 5 severity, the test data will be reanalyzed and readjusted nine times. The complete result can be found in Table 4.6. The results also show that our method outperforms the other methods in this long-term adaptation scenario. It illustrates the effectiveness of domain offset controllability as well as category offset controllability.

### 4.8 Results for Random Order Test-Time Adaptation

For a more comprehensive evaluation of the proposed method, CIFAR10-C, CIFAR100-C, and ImageNet-C experiments are conducted on over ten sequences of various damage types with a severity level of 5. As shown in the Table 7, C-CoTTA is consistently outperforming CoTTA and other competing methods.

**Table 7: Average error of standard ImageNet-C experiments over 10 diverse corruption sequences.**
| Avg. Error (%) | Source | TENT | CoTTA | SATA | Ours |
| --- | --- | --- | --- | --- | --- |
| CIFAR10-C | 43.5 | 20.1 | 16.3 | 16.3 | 14.7 |
| CIFAR100-C | 46.4 | 61.3 | 32.6 | 32.8 | 29.5 |
| ImageNet-C | 83.0 | 61.8 | 57.9 | 64.5 | 55.5 |

## 5 Conclusion and Limitation

In this work, we introduce C-CoTTA, a novel framework designed to prevent any category from leaning towards other categories by explicitly controlling the offset direction to avoid fuzzy classification boundaries, and reduce the sensitivity of the model in the domain shift direction to reduce the impact of domain shift on domain adaptation. This fills the gap left by traditional methods, which can only mitigate the impact of domain drift. C-CoTTA can explicitly control domain shift, opening up a new solution pathway for CTTA. Through extensive quantitative experiments and qualitative analysis, such as t-SNE plots, we demonstrate the effectiveness and theoretical validity of C-CoTTA.

However, our method also has certain limitations. During test-time, we may not have access to all samples of a specific domain prototype, and due to the lack of true labels, misclassified samples may contaminate the prototype. As a result, the representation of the prototype may be poor, further affecting the accuracy of the constructed direction, leading to ineffective or even erroneous domain shift control. Therefore, in the future, addressing how to obtain high-quality prototypes and directional representations is a task that needs attention.

## Appendix A Equivalent Representation of SCAV

Further derivation of scav is conducted to obtain a simpler equivalent representation, as follows:

$$ $\displaystyle\text{cov}[f(x),y]$ $\displaystyle=\sum_{x_{i}\in\mathcal{X}_{c}\cup\mathcal{X}_{n}}(f(x_{i})- \overline{f}(x))(y_{i}-\overline{y})$ $\displaystyle=\frac{|\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|} \sum_{x_{i}\in\mathcal{X}_{c}}(f(x_{i})-\overline{f}(x))+\frac{|\mathcal{X}_{c }|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\sum_{x_{i}\in\mathcal{X}_{n}}(f(x_{i} )-\overline{f}(x))$ $\displaystyle=\frac{|\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|} \left(\sum_{x_{i}\in\mathcal{X}_{c}}f(x_{i})-|\mathcal{X}_{c}|\cdot\overline{f }(x_{i})\right)-\frac{|\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|} \left(\sum_{x_{i}\in\mathcal{X}_{n}}f(x_{i})-|\mathcal{X}_{n}|\overline{f}(x_{ i})\right)$ $\displaystyle=\frac{|\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|} \sum_{x_{i}\in\mathcal{X}_{c}}f(x_{i})-\frac{|\mathcal{X}_{n}||\mathcal{X}_{c} |}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\overline{f}(x_{i})-\frac{|\mathcal{X}_ {c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\sum_{x_{i}\in\mathcal{X}_{n}}f(x_{i })+\frac{|\mathcal{X}_{c}||\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n }|}\overline{f}(x_{i})$ $\displaystyle=\frac{|\mathcal{X}_{n}||\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+| \mathcal{X}_{n}|}\left(\frac{1}{|\mathcal{X}_{c}|}{\sum_{x_{i}\in\mathcal{X}_{ c}}f(x_{i})}-\frac{1}{|\mathcal{X}_{n}|}{\sum_{x_{i}\in\mathcal{X}_{n}}f(x_{i} )}\right)$ $\displaystyle\text{cov}[y,y]$ $\displaystyle=\sum_{x_{i}\in\mathcal{X}_{c}\cup\mathcal{X}_{n}}(y_{i}- \overline{y})^{2}$ $\displaystyle=\sum_{x_{i}\in\mathcal{X}_{c}}(1-\frac{|\mathcal{X}_{c}|}{| \mathcal{X}_{c}|+|\mathcal{X}_{n}|})^{2}+\sum_{x_{i}\in\mathcal{X}_{n}}(0- \frac{|\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|})^{2}$ $\displaystyle=\sum_{x_{i}\in\mathcal{X}_{c}}\frac{|\mathcal{X}_{n}|^{2}}{(| \mathcal{X}_{c}|+|\mathcal{X}_{n}|)^{2}}+\sum_{x_{i}\in\mathcal{X}_{n}}\frac{| \mathcal{X}_{c}|^{2}}{(|\mathcal{X}_{c}|+|\mathcal{X}_{n}|)^{2}}$ $\displaystyle=\frac{|\mathcal{X}_{c}||\mathcal{X}_{n}|^{2}+|\mathcal{X}_{n}|| \mathcal{X}_{c}|^{2}}{(|\mathcal{X}_{c}+|\mathcal{X}_{n}|)^{2}}$ $\displaystyle=\frac{|\mathcal{X}_{c}||\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|X_{ n}|}$ $\displaystyle\frac{\text{cov}[f(x),y]}{\text{cov}[y,y]}$ $\displaystyle=\frac{1}{|\mathcal{X}_{c}|}{\sum_{x_{i}\in\mathcal{X}_{c}}f(x_{i })}-\frac{1}{|\mathcal{X}_{n}|}{\sum_{x_{i}\in\mathcal{X}_{n}}f(x_{i})}$ $$

## Appendix B Corruption loops Test-Time Adaptation

We present the results of 10 cycles on CIFAR10-C. As depicted in the Fig. 5, it is evident that over time, the error rate of CoTTA and SATA methods has gradually increased, whereas our method continues to decrease. Consequently, the performance gap is widening. This phenomenon is most likely related to the fact that the first two methods lack reasonable control over the category and the offset direction of the domain.

Figure: Figure 5: Under the CIFAR10-C dataset, it can be observed that the performance of each method under 10 corruption cycles varies. The error rates of CoTTA and SATA methods start to gradually increase in the later stages, whereas our method continues to decrease or maintain stability.
Refer to caption: x5.png

## Appendix C Hyperparameter analysis

In this section, we delve into the critical examination of hyperparameters $\lambda_{1}$ and $\lambda_{2}$ in Eq. 11 within our C-CoTTA framework in ImageNet-C, which substantially influence the model’s performance. Through meticulous experimentation, we fine-tuned these hyperparameters to identify their optimal values, ensuring the harmonious interplay between Control Domain Shift (CDS) and Control Class Shift (CCS) components.

Our investigation revealed that the selection of $\lambda_{1}$ and $\lambda_{2}$ is pivotal in balancing the contributions of CDS and CCS to the overall objective function. We experimented with a spectrum of values for these hyperparameters, meticulously recording the impact on classification accuracy and domain adaptation efficacy. The empirical results, illustrated in Fig. C, present a compelling case for the optimal balance that our chosen hyperparameters provide, underscoring the model’s robustness against various disturbances.

Figure: Figure 6: Analysis Hyperparameter of $\lambda_{1}$ for CDS and $\lambda_{2}$ for CCS on ImageNet-C.
Refer to caption: x6.png

Figure: Figure 7: Analysis Hyperparameter of $E_{0}$ on CIFAR10-C.
Refer to caption: x7.png

## Appendix D Reliable Sample Selection Analysis

In our control category offset method, in order to reduce the contamination of category prototypes by misclassified samples and thus affect the control of category shift direction, we remove samples with entropy values exceeding the predefined threshold $E_{0}$, which is set as $0.4\times\ln C$ based on []. We have verified the rationality of this operation through experiments. As shown in Fig. C, when the threshold is large, the effectiveness of the CCS method deteriorates. This may be because the conditions are too loose, leading to a large number of misclassified samples when calculating the prototype. On the other hand, when the threshold is small, the CCS method also deteriorates. This may be because the conditions are too strict, resulting in too few samples used to calculate the prototype, making the generated prototype not representative.

## Appendix E Class Confusion Matrix

We observed the confusion matrix of category in the domain adaptation process. As shown in Fig. 8, compared to the CoTTA and SATA methods, our method significantly reduced the degree of category confusion, demonstrating the effectiveness of controllable domain shift.

Figure: Figure 8: confusion matrix of category in the domain adaptation process. The vertical axis represents the true labels, and the horizontal axis represents the predicted labels.
Refer to caption: x8.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: * How are the hyperpaprameters lambda_1, lambda_2 tuned? Is any validation corruption used?
* Can you report numbers for the two settings when i. lambda_1 = 0, and ii. lambda_2 = 0 to analyze the contribution of different losses.
* Is the paper not just an application of CAV for continual TTA? Is it non-trivial to apply CAV for settings such as TTA?
* Are the DSCL and CSCL losses not applicable to domain adaptation problems in general? If not, why does it make sense for TTA?

**Minor Comments**
* Line 82-83: Typo "Concept Ativation Vectors" --> "Concept Activation Vectors"
* Algorithm 1 does not define p^t_i 
* Some other typos, please proofread

**References**
1. Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11920–11929, 2023.
2. Lee, Daeun, Jaehong Yoon, and Sung Ju Hwang. "BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation." International Conference on Machine Learning, 2024.
soundness: 3
strengths: * Utilizing CAV for the problem of continual TTA
* Proposing domain and class shift controlling losses
* Improving performance on the state-of-the-art benchmarks
summary: This paper addresses the problem of continual test-time adaptation (TTA) by guiding and controlling shifts by utilizing Concept Activation Vectors (CAV), an interpretability tool for deep learning models.
They control the domain shift by constructing the domain and class shift to control losses.
Experiments show better performance in widely used continual TTA benchmarks.
weaknesses: * Using pseudo-labels for prototype computation can lead to more error accumulation in real-world
* Motivation for why CAV makes sense for continual TTA and not any deep learning problem, in general, is lacking
* Comparisons with recent approaches such as EcoTTA [1], and BeCoTTA [2] are missing

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: The conceptual contribution of this work remains unclear. As described, it appears to be a rephrasing of the inter-class distance maximisation and inter-domain distance minimisation approach commonly used in CTTA. How does this approach differ meaningfully from existing methods in CTTA?

While the results slightly outperform previous approaches on average, I have concerns about whether hyperparameter tuning was conducted in a way that avoids overfitting to the evaluation benchmark. Please clarify this in the rebuttal.

My current assessment leans toward rejection. I encourage the authors to address these points in their response.
soundness: 3
strengths: The method demonstrates empirically plausible results, though with reservations that I describe in the weaknesses.

Since this area is outside my expertise, I would defer to my fellow reviewers on the following:

- The relevance of the benchmark tasks and datasets used in this work
- The significance of the reported results
- Any potential biases or issues in the experimental setup
summary: Continual Test-Time Adaptation (CTTA) - While most CTTA methods focus on suppressing domain shifts, this paper introduces an approach that aims to guide rather than suppress these shifts. They propose Controllable Continual Test-Time Adaptation (C-CoTTA), designed to maximise inter-class distances while minimising inter-domain distances.
weaknesses: The terms "guide" and "control" are vague. More precise language is needed to clearly convey the conceptual mechanism of C-CoTTA. Specifically, could the authors clarify whether the method is fundamentally maximising inter-class distances and minimising inter-domain distances in the representation space? This objective appears consistent with the aims of many test-time adaptation methods. Detailed technical explanation on how C-CoTTA differs from prior approaches would be beneficial, especially avoiding abstract terms like "suppress," "guide," or "control."

It may not be fair to summarise all baseline methods (Tables 1, 2, and 3) as merely "suppressing the shift." Is it accurate to say that these methods do not involve any "guiding" or "controlling" aspects? Further technical support for this claim would strengthen the paper. Perhaps the authors could include an analysis in §4.5 and Figure 4 for all baseline methods listed in Tables 1, 2, and 3.

The terms "guiding" and "controlling" remain ambiguous, especially given that Equation 7 still seems focused on "suppressing the shift" as well. Clear definitions of these terms within the context of C-CoTTA are recommended.

Since many methods are compared in the benchmark with rather small performance gaps, I wonder about the possibility of test-set overfitting. This is especially important given the low-resource nature of the setup (where information about the test distribution is supposed to be only partially available to the learner). It would be helpful if the authors could describe their efforts to minimise this risk (e.g., by confirming if the hyperparameters and design choices in §C were tuned without reference to the test/evaluation split). This would impact the fairness of the empirical comparisons, given the small performance gaps (15.3 vs 14.7 in Table 1, 30.2 vs 29.9 in Table 2, 59.9 vs 59.4 in Table 3).

nit: please run spell/grammar checker
- "Ativation" --> "Activation"
- "only suppress domain shift is insufficient" --> "only suppressing domain shift is insufficient" 
- "explicit control" --> "explicitly control"

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Could you clarify how "random order" is defined and implemented in the experiments mentioned in section 4.8?
2. Does the availability of source data impact the performance of your method? Testing with varying amounts of source data could provide insights into its practical utility when labeled source data is limited.
3. The computation symbol "cov[]" used in formula (1) is unclear. Could you provide a definition or explanation?
4. Can you elaborate on the relationship between formulas (7) and (8)? What is the underlying intuition linking these formulas?
5. Considering practical constraints where not all classes may appear in each batch (e.g., batch sizes less than 1000), how might this affect the calculation of domain shift direction and the applicability of formula (9)?
6. Typo in lines 71 and 73. DSCL $\rightarrow$ CSCL?
soundness: 3
strengths: 1. The paper addresses a significant challenge in CTTA by proposing a method to control domain shifts, which is crucial for applications in dynamic environments.
2. The approach uses Concept Activation Vectors (CAVs) to represent and control shift directions, which is a well-founded technique in interpretable AI.
summary: The paper presents a novel framework for continual test-time adaptation (CTTA), aiming to control domain shifts dynamically during model inference without access to source data or labels. The approach, named C-CoTTA, introduces mechanisms to guide domain shifts to maintain clear class boundaries and minimize error accumulation.
weaknesses: 1. Marginal Improvements: The reported improvements in classification accuracy are relatively marginal (0.6%, 0.4%, 0.5% on CIFAR-10C, CIFAR100-C, and ImageNet-C, respectively). This raises concerns about the practical significance and robustness of the proposed method.
2. Inconsistency in Compared Methods: There is a lack of consistency in the methods compared across different datasets. For instance, ViDA is only included in the ImageNet-C experiments but not in CIFAR experiments. Additionally, the paper does not include comparisons with more recent and potentially more effective methods in segmentation tasks.
3. Metric Validity and Relevance Concerns: The paper uses inter-class and inter-domain distances as metrics to assess class separability and sensitivity to domain shifts. However, the relevance and validity of these metrics are questionable. For instance, the paper does not account for the potential scaling of feature values; if feature vectors $p_i^t$ and $p_j^j$ in line 414 are simply scaled up by a factor (e.g., doubled), the computed distances would also increase, suggesting greater separability without actual improvement in class distinction. This is illustrated by the example where different domains with similar inter-class and inter-domain distances exhibit significant accuracy disparities, such as between the 'Brightness' and 'Contrast' conditions of ImageNet-C, which have a roughly 20% gap in accuracy despite similar distance metrics. 
4. The paper does not discuss several relevant works such as [1-5]. This comparison is crucial for situating the novelty of the proposed method within the existing literature. 
5. Given the use of labeled source data in the adaptation process, the experiments may not offer a fair comparison to results from methods that do not use labeled source data, rely only on statistics of source data, or do not use source data at all. Clarifying the conditions under which each method is evaluated is essential for understanding and interpreting the experimental results accurately.

Addressing these concerns could significantly strengthen the manuscript, enhancing its contribution to the field and its potential for a higher rating.

[1] Lee et al., "Becotta: Input-dependent online blending of experts for continual test-time adaptation", ICML 2024.

[2] Song et al., "Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization", CVPR 2023.

[3] Gong et al., "SoTTA: Robust Test-Time Adaptation on Noisy Data Streams", NeurIPS 2023.

[4] Liu et al., "Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation", CVPR 2024.

[5] Yang et al., "A Versatile Framework for Continual Test-Time Domain Adaptation: Balancing Discriminability and Generalizability". CVPR 2024.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 1
questions: 1. Why the symmetric cross-entropy loss is used rather than KL or reverse KL in eq. 10?

2. What happens if a batch contains samples from multiple domains? I wonder how much performance degradation would result from that scenario.

3. Figure 3 is quite intuitive. Can you show similar figures for different target domains over time?

4. The performance of RMT recorded in this paper is lower than the results reported in the original RMT paper, with a noticeable difference. What is the reason for this?
soundness: 3
strengths: The strength of this paper lies in introducing a new loss function to actively control domain shifts within the CTTA framework. This approach not only tackles error accumulation but also provides a way to maintain clear boundaries between classes during continual adaptation.
summary: This paper proposes Controllable Continual Test-Time Adaptation (C-CoTTA), a method that enables models to adapt to continuously changing domains during test time without access to the original source data. 

Unlike traditional CTTA approaches, which primarily focus on suppressing domain shifts to mitigate error accumulation, this approach controls the direction of domain shifts to prevent category confusion. By leveraging Concept Activation Vectors (CAV), the method controls the feature shift direction for each class, while domain- and class-level shift control loss functions enhance classification performance.
weaknesses: 1. There are a lot of grammatical errors that hinder readers from concentrating on the paper. 
For example 

- Line 109: focuses ⇒ focus

- Line 112: they are designed ⇒ they designed

- Line 124: Andres et al. (2022). no parenthesis in the reference.

- Line 157: refer to ⇒ refers to

- Line 225: calculates ⇒ is calculated

- Line 231: weird sentence

- etc.

Also, there are quite a lot of notational errors

- eq 2: $\mathcal{X}_t$ and $\mathcal{X}_s$ are undefined

- eq 3 and 4: inconsistent notation (both subscripts and superscripts are used to indicate the same objects)

- Line 216: $i$ is undefined

Poorly structured English and inconsistent symbols can indeed make the core concepts harder to follow, even if the underlying ideas are strong. Improved clarity in language and notation would make it easier for readers to fully appreciate the novel approach to CTTA and better understand the implementation of the new loss functions for domain control.

I recommend that the authors thoroughly revise the grammar and notation before resubmitting the paper to a new venue.

2.The idea of using a domain prototype was originally proposed in the UDA task, so it’s not a new concept. However, it has not been actively utilized in TTA because it’s challenging to obtain an accurate domain prototype given the nature of TTA, which receives online mini-batches as input. What is the mini-batch size used in this experiment? Mini-batch size is considered a significant issue in TTA.

3. It still inherits the weaknesses of previous works that use pseudo labels. Depending on the accuracy of the pseudo labels, there is a possibility that the class shift may be inaccurately estimated.