Beware of Model Collapse! Fast and Stable Test-time Adaptation for Robust Question Answering
Authors: Yi Su
Yixin Ji
Juntao Li
Hai Ye
Min Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Question Answering
Roboustness
Test-time adaptation

Abstract: Although pre-trained language models (PLM) have achieved great success in question answering (QA), their robustness is still insufficient to support their practical applications, especially in the face of distribution shifts.
Recently, test-time adaptation (TTA) has shown great potential for solving this problem, which adapts the model to fit the test samples at test time. 
However, TTA sometimes causes model collapse, making almost all the model outputs incorrect, which has raised concerns about its stability and reliability.
In this paper, we delve into why TTA causes model collapse and find that the imbalanced label distribution inherent in QA is the reason for it.
To address this problem, we propose Anti-Collapse Fast test-time adaptation (Anti-CF), which utilizes the source model‘s output to regularize the update of the adapted model during test time.
We further design an efficient side block to reduce its inference time.
Extensive experiments on various distribution shift scenarios and pre-trained language models (e.g., XLM-RoBERTa, BLOOM) demonstrate that our method can achieve comparable or better results than previous TTA methods at a speed close to vanilla forward propagation, which is 1.8× to 4.4× speedup compared to previous TTA methods.

Beware of Model Collapse! Fast and Stable Test-time Adaptation for Robust Question Answering
Yi Su'; Yixin Ji Juntao Li Hai Ye', Min Zhang' Institute of Computer Science and Technology, Soochow University, China
Department of Computer Science, National University of Singapore yi sunnp@@uttoook com; jiyixin1 1lyyixin699@mail com; yehaiêcomp.. nus edu. sg; {ljt, minzhang)@sudaa edu. cn
Abstract
2020), and so on. This situation is known as distri- bution shift. Unfortunately, even the most advanced models currently available, such as ChatGPT, do not perform well under the distribution shift (Ye et al., 2023; Wang et al., 2023).
Although pre-trained language models (PLM) have achieved great success in question answer- ing (QA), their robustness is still insufficient to support their practical applications, especially in the face of distribution shifts. Recently, test- time adaptation (TTA) has shown great poten- tial for solving this problem, which adapts the model to fit the test samples at test time. How- ever, TTA sometimes causes model collapse, making almost all the model outputs incorrect, which has raised concerns about its stability and reliability. this paper, we delve into why TTA causes model collapse and find that the imbalanced label distribution inherent in QA is the reason for it. To address this problem, we propose Anti- Anti-Collapse Fast test-time adaptation (Anti-CF), which utilizes the source model's output regularize the update of the adapted model during test time. We further design an efficient side block to reduce its inference time. Extensive experiments various distribution shift scenarios and pre-trained language mod- els (e.g.. RoB.Ra, BLOOM) demon- strate that our method can achieve comparable or better results than previous TTA methods at a speed close to vanilla forward propagation, which is 1.8x to 4.4x speedup compared to pre- vious TTA methods. Our code is available at https://gi thtp::://iitub..... wwwm//isnl/l/nnt......
To address this problem, researchers have pro- posed many approaches such as adversarial training (Zhu et al., 2020; Wang et al., 2021a), data augmen- tation (Zhou al., 2021; Chen et al., 2021a). These methods improve the robustness of the model by changing the training strategy, but according to the No Free Lunch Theorem (Wolpert and Macready, 1997), a fixed model still cannot perform perfectly in all catributionnshifteee scenarios. Therefore, some works (Wang et al., 2021b; Sun et al., 2020; Niu et al., 2022; Ye et al., 2022) explore how to up- date the model during the testing phase to adapt it to the distribution shifts of the test samples, called Test Time Adaptation (TTA). A typical approach (Wang al., 2021b) uses the Shannon entropy of the probability given by the model as the loss to update itself. However, due to the unreliable output of the model, TTA may accumulate erroneous in- formation learned in test samples, leading to model collapse and sharp decline in model performance, which makes TTA extremely unstable and unreli- able in practical applications.
To solve this problem, take QA task an ample and investigate why TTA causes the model collapse. Our experiments indicate that the main reason for the model collapse is the imbalanced label distribution of the test data. In contrast to the direct inference, TTA exacerbates this imbal- anced distribution, making all outputs of the model to be specific class. Therefore, we propose Anti- Anti-Collapse Fast test-time adaptation (Anti-CF), which utilizes the output of the source model as a soft label regularize the update of the adapted model during test time to ensure that the adapted model will not deviate too far from the source model, thus avoiding model collapse.
1 Introduction
Pre-trained language models (PLMs) have achieved great success on many NLP tasks (Devlin al., 2019; Liu et al., 2019; Lewis et al., 2020a; Raf- fel al., 2020; Brown et al., 2020; OpenAI, 2022, 2023; Touvron et al., 2023). However, their success is based on the assumption that the test distribution is consistent with the training distribution. In many scenarios, this assumption is not true, such as ad- versarial attack (Wang et al., 2022), cross-lingual (Li et al., 2021), cross-domain (Ramponi and Plank,
"Equal Contribution. Corresponding author.

---

However, obtain the output of the source model and the adapted model, we need keep the parameters of two models and conduct forward propagation twice, which will bring a lot of addi- tional costs in practical applications. Therefore, we freeze the source model and add an efficient side block the adapted model to reduce the cost of ad- ditional forward propagation and back propagation. Extensive experiments on various distribution shift scenarios and PLMs demonstrate that our method can achieve comparable or better results than pre- vious TTA methods at a speed close to vanilla for- ward propagation, which is 1.8x to 4.4x speedup compared to previous TTA methods.
[CLS] token, indicating the absence of an answer in that particular span.
Assume that we have training data pairs (o vill= E Da with the distribution Pa, where oy E X2 refers to the span-question pair and v e V refers to the corresponding position of the an- swer. After successful training on the training data, we obtain model fe X Vg.. In the test phase, have test data samples {a:]) ==1 A with un- derlying corresponding labels {ut {9iT==1 E Va. The distribution of the test data is Pe 7 P.- If we use the trained model fo to predict test samples, the model's performance will degrade due to the distribution shift. Test-time adaptation provides a promising paradigm to mitigate performance degra- dation, which updates the model during the testing phase to accommodate the distribution shifts of test data. Tent (Wang et al., 2021b) and EATA (Niu et al., 2022) use the model's prediction probability of the test sample as soft label to optimize the model by minimizing the entropy:
Overall, our contributions in this work include:
We investigate why TTA causes model collapse in QA and find that the imbalanced label distri- bution inherent in QA is the reason for it.
We propose Anti-Collapse Fast test-time adap- tation (Anti-CF) to solve the problem that TTA sometimes leads to model collapse.
((xr) = - (velare log (velwe)
Experimental results show that Anti-CF can ef- fectively prevent the model from collapsing with fast inference speed. It improves the stability and reliability of TTA in practical upplications.
(1)
where (Nelaz) is the probability of the c-th cate- gory of 2t.
2.2 The Risk of TTA Leading to Model Collapse
2 Preliminary
In this section, we begin by introducing extractive question answering and the application of TTA to enhance its robustness. Subsequently, Tent (Wang et al., 2021b) as a case discuss the potential risks of TTA, specifically the occurrence of model collapse. Additionally, we conduct an analysis of generative QA in Section 5.5.
Wc use XLM- LLM-RoBBRTa-base (Conneau et al., 2020) the backbone to train a source model on SQuAD (Rajpurkar al., 2016) and evaluate it on NaturalQA (Kwiatkowski et al., 2019), which a cross-domen setting. We compare the results direct inference and Tent. We experimented with various optimizers and learning rates for Tent, as illustrated in Figure 9, We find that no matter what kind of optimizer and what learning rate we set, the performance of the model will decrease. Obvi- ously, the smaller the learning rate, the smaller the impact of TTA on the model. However, if we set the learning rate to be very small, this will make TTA almost ineffective, and if we set the learning rate to a more reasonable range, the model may col- lapse. TTA does not always improve the robustness of the model and even has a risk of model collapse, which would seriously hinder the application of TTA in real-world scenarios.
focus
2.1 Test-time Adaptation for Question Answering
In extractive QA. the input of the model is com- bination of context and a question. The goal is to determine the start and end positions the an- swer within the context, where the text between them represents the answer. However, in practice, the context- queesxtt-tuetionnnneoeo pairs are often too long to be directly processed by the model. To address this, we divide the context into smaller spans. For each span, the model predicts the start and end positions of the answer within that specific span. In cases where the model determines that the answer does not exist within a given span, will output the start and end positions of a special token, such as the
To explore why TTA causes model collapse. we study the entropy of test data. As Figure 2 shows, the entropy of Tent sharply decreases until approaching 0, which means the collapsed model

---

300
300
300
200
200
200
1810
1100
2000
2550
530
1000
20000
2550
502
1030
1500
2000
3530
Spmn
Span
Spon
(a) Forward
(b) Tent
(c) Anti-CF
Figure 1: The start positions NaturalQA given by the model using Forward, Tent, and Anti-CF. Forward: direct inference.
3.2) aims to reduce the inference time by building a small network next to the backbone.
Forward Text
Ant-CP
3.1 Entropy Minimization with Source Constraints
Entropy
1.0
To solve the problem we mentioned Section 2.2, we want to use the output of the source model as a constraint to the adapted model during test time so that the adapted model does not deviate too far from the source model, thus avoiding model collapse. also
0.5
100
Batch
200
300
Like many previous TTA methods, choose entropy minimization as one of our opti- mization goals, which can formulated as:
Figure 2: The entropy of NaturalQA predicted by the model using Forward, Tent, and Anti-CF. Forward: di- rect inference.
T
C.
-
Σ Pa (uelax)) ) log Pa (yelx;) (2)
makes wrong predictions with high confidence. We further explored the start positions given by the model (Figure 1, the ground truth and end posi- tions can be found in the Appendix C). We find that 0 (indicates that the answer is not in this span) accounts for the majority. Furthermore, after using Tent, the model's prediction tends be more and more inclined towards O, which directly leads to almost all of the model's predictions being incor- rect. Therefore, the imbalanced distribution of test labels has led to TTA leaning too much towards the majority class during the update process, resulting in all outputs being biased towards some specific classes, which is why TTA causes model collapse.
where {-}] is batch of test samples and Pa (Mel2z)) is the prediction probability of x; given by the adapted model. We use forward Kullback-Leibler (KL) divergence to constrain the update of the adapted model, which will make the output of the adapted model close to that of the source model:
1 Le -
- Σ (o((o) log 1=0
Ps (s(eee (T))
(3)
Pa (ye (x4)
where Ps (Vel (2)) the probability of the c-th cat- egory given by the source model. We introduced a to balance the two losses, so the loss function of Anti-CF is:
3 Method
L= (1 a)Le aLe
(4)
In this section, we propose Anti-collapse Fast Test- time adaptation (Anti-CF). Anti-CF consists of two strategies. (1) Entropy minimization with source constraints (Section 3.1) seeks to ensure that the adapted model does not deviate too far from the source model, thus avoiding the occurrence of model collapse. (2) Efficient side block (Section
We can briefly analyze why Anti-CF can avoid model collapse. Suppose model collapse has curred, with extremely low entropy like that in Section 2.2. At this point, the first part of the loss L. close to O, and the loss approximately has only the second part L. Therefore, the main objec- tive of the loss is to pull the adapted model closer

---

Luth.
ranges from to the number of layers in the ef- ficient side block. Both hg and so are initialized as embedding outputs. For example, the XLM- RoBERTa-large has 24 Transformer layers, we take 12 adapter modules as the side block.
LayerNorm
Adapted output
Source output
Classifier
Classifier
Nontinear
When a sample is given, since the backhone and side block are parallel, only one forward propagation is needed to obtain the output of the source model and the adapted model. During back propagation, the backbone frozen and only the parameters of the efficient side block are up- dated, which prevents gradient propagation in the backbone, thus significantly accelerating the back- propagation speed.
Adapter
W
Layer
Adapter
Layer
Adapter
Layer
Updated
Layer
H
Inidilleee
Forward
Backward
Embedding
SB
Inputs
Add
Since the efficient side block is additionally plugged into the backbone in the TTA phase, it is not trained in the training phase. Thus, its param- eters are randomly initialized. We believe that the efficient side block without learning task aaee-specific information may cause performance degradation of TTA, so we train the efficient side block before per- forming TTA, which we call the warmup process. Since the warm-up phase only learns task-related information, the warmup data can be either the training data of the original model or other avail- able data of the same task.
Figure 3: The structure of efficient side block. There adapter added between every two layers. Gradients only propagate through adapters, and the backbone frozen. The adapted output will be used the final output for prediction, while the source output is only used to constrain the update of the efficient side block.
to the source model, which effectively avoids the occurrence of model collapse.
3.2 Efficient Side Block
To minimize Eq.4, we need to obtain the predicted probability of the source and adapted model. How- ever, this requires at least two forward propaga- tion and one back propagation for each sample, which undoubtedly dramatically increases the cost practical application. To break this dilemma, we propose an efficient side block, which plugged into the backbone as the adapted model so that we only need one forward propagation to obtain the two outputs simultaneously. In addition, the gradi- ent only back propagates through the efficient side block, reducing the cost of back propagation.
Experiments
4.1 Datasets and Evaluation Metrics
To verify the effectiveness of our proposed Anti-CF, we conduct experiments in three distribution shift scenarios: adversarial attack, cross-lingual, and cross-domain. Datasets use the following: NoiseQA (Ravichander al., 2021) adds three common types of noise to the SQuAD datasct (Ra- jpurkar et al., 2018): speech recognition, keyboard inputs, and translation systems. Then, NoiseQA adds natural and synthetic noise, generating two subsets named NoiseQA-syn and NoiseQA-na. XuuAD (Artetxe et al., 2020) is cross-lingual QA dataset. XQuAD translates a subset of the development set from SQuAD into ten languages. MLQA (Lewis et al., 2020b) is also a cross-lingual dataset. Compared to XQuAD, annotations the new dataset from English Wikipedia and translates it into six languages.
As shown in Figure 3, the efficient side block consists of series of adapter modules (Houlsby et al., 2019). We plug an adapter between every two Transformer layers (Vaswani et al., 2017). The input of each adapter is the combination of the hidden state of its corresponding two layers and the hidden state of the previous adapter:
h& Layer (hx- 21, 2i
(5)
MRQA (Fisch et al., 2019) is cross-domain dataset which includes HotpotQA (Yang al., 2018), NaturalQA (Kwiatkowski et al., 2019), NewsQA (Trischler et al., 2017), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017).
9; = Adapter, + f2g + h2++1)
(6)
where h2 is the hidden state of the k-th Trans- former layer, s is the hidden state of the i-th adapter,

---

NoiseQA-xx -syn
NoiseQA-na
XQuAD
MLQA
MRQA
Avg.
Models
Time
EM
F1
EM
F1
EM
EM
F1
EM
F1
EM
ximr-base
66.61
78.64
66.08
77.92
55.59
71.43
48.27
65.83
40.12
52.79
55.33
69.32
4.25
+ Tent
68. 14
79.47
67.44
78.78
56.40
71.74
48.11
65.61
20.77
32.35
52.17
65.59
9.43
+EATA
68.25
79.55
67.57
78.88
56.42
71.64
48.11
65.53
19.16
30.62
51.90
65.24
11.01
+OIL
68.36
79.55
67.74
79.01
56.69
71.92
48.28
65.62
37.65
49.60
55.74
69.14
23.07
+SAR
67.63
79.30
67.36
78.76
55.94
71.77
48.01
63.41
31.71
42.68
54.13
67.58
18.30
+Ours
68.08
79.45
67,50
78.82
56.75
71.72
48.32
65.66
40.16
52.86
56.16
69.70
5.23
xTune
70.92
81.49
69.72
80.65
58.81
73.77
50.66
67.86
43.36
56.04
58.70
71,96
4.25
+ +Tent
71.52
81.89
70.78
81.24
59.48
74.08
51.10
68.07
26.96
38.24
55.97
68.70
9.43
+EATA
71.49
82.01
70.67
81.14
59.36
73.91
51.30
68.24
24.51
35.70
55.47
68.20
11.01
+OIL
71.42
81.91
70.80
81.15
59.66
74.16
51.36
68.28
39.54
51.63
58.57
71.42
23.07
+SAR
70.90
81.46
70.34
80.85
59.23
73.93
51.19
68.11
36.73
48.50
57.68
70.57
18.30
+Ours
71.32
81.75
70.19
80.91
59.01
73.83
51.18
68.28
43.44
56.05
59.02
72.16
5.23
xlmr-large
65,55
79.91
64.17
78.37
63.15
78.77
54.40
72.50
46.17
59.45
58.69
73,80
12.37
+ +Tent
70.07
82.74
67.21
80.34
63.66
79.02
54.50
72.51
27.65
36.83
56.62
70.29
26.27
+EATA
70.59
83.03
68.37
81.07
63.65
78.97
54.60
72.52
24.87
33.43
56.42
69.81
29.94
+OIL
69.83
82.92
68.22
81.05
63.86
79.18
54.77
72.72
40.98
52.29
59.53
73.63
66.37
+SAR
69.29
82.38
67.70
80.56
63.52
78.98
54.43
72.37
46.41
59.25
60.27
74.71
52.47
+Ours
71.53
83.27
69.28
81.08
63.78
79.00
54.93
72.81
46.24
59.47
61.15
75.13
15.36
Table 1: Main results (%). We select three source models: XLM-RoBERTS. (xlmr-base), <Tune-XLM:
RoBERTa-base (xTune), and XLM--RBERTa--arge (ximr-large). Bold: the best results.
We report the EM (Exact Match) and F1 score for each dataset. We also provide the Time of each TTA method, which means the time (ms) the model requires to process sample.
model on SQuAD using the default training setup from XTREME (Hu et al., 2020).
For all baselines, to speed up TTA as much as possible, we follow the setup of Su et al. (2023) and only tune all LayerNorm parameters. When reproducing EATA, discard the step of filter- ing redundant samples following (Niu al., 2022) because this method unsuitable for NLP data. For Anti-CF, we set the adapter's 's hidden size the same as the source model's hidden size. Unlike the setting in OII,, we believe that TTA should not select a set of hyper-parametery for each test set individually because in complex and variable real- world scenario, we cannot make careful hyper- parameter selection for each distribution shift.
4.2 Baselines
We use the following strong baselines as compar- ison to verify the effectiveness of Anti-CF. Tent (Wang et al., 2021b) updates the model by minimizing the entropy of test samples.
EATA (Niu et al., 2022) only updates samples with low entropy to improve the reliability of pseudo- labels. In addition, restricts the updating of the model from changing too much to avoid forgetting. OIL (Ye et al., 2022) uses a teacher-student paradigm to increase the reliability of the pseudo- labels. It utilizes the output of the teacher as the pseudo-labels to guide the update of the student. SAR (Niu et al., 2023) utilizes the sharpness-awwe optimizer to enhance the reliability of the model updating process and enable adaptation to highly challenging environments.
We rum all experiments with different random seeds three times and take the averaged result as the final experimental results. We tune the model with the learning rate in [5e-5, le-4, 5e-4) and set the batch size 8. We use the validation set of SQuAD to warmup the efficient side block for epoch with the learning rate of 5e-4. All experiments are completed on NVIDIA RTX 3090 GPU. Details of all Sayper-parameters are given in Appendix B.
4.3 Implementation Details
In our main experiments, utilize the XLM- RoBERRaa-basellrgee the backbone model. In ad- dition, we use xTune (Zheng et al., 2021), 1), a strong robustness tuning method to train a source model on LLMM-OBBRRT--aae We train the source
4.4 Main Results
Table shows the main results of each TTA method. We have the following observations: Anti-CE CF can effectively avoid model collapse. On

---

MRQA, Tent and EATA cause the model collapse. resulting in a significant decrease in performance compared to the source model, with the highest EM decreasing by 21.3% (46.17% + 24.87%). Although OIL can alleviate model collapse with imitation learning from the mean teacher, there will still be significant performance degradation, at most from 46. 16.17% 40.98% EM. Even the latest baseline SAR cannot completely avoid model collapse. However, Anti-CF has no performance degradation on any dataset, avoiding the model col- lapse that other TTA methods may encounter. We also plot Anti-CF's start position distribution (Fig- ure I())) and entropy of NatrualQA (Figure 2). We note that the entropy on Anti-CF decreases slowly compared to Tent, and the predictions avoid com- pletely lean towards the majority of the labels. This corroborates that Anti-CF can effectively prevent the model collapse caused by TTA.
Exact Match 60%
Tent
BATA
AntS-CF
55%
50%
45%
40%
2e-5
Se-5
1e-4
2e-4
Se-4
1e-3
2e-3
Se-3
Learning Rate
Figure 4: Effect learning rate.
tion (Ye et al., 2022), resulting in a much slower inference speed than vanilla forward, increasing its cost in real-world applications. The inference speed of Anti-CF is close to vanilla forward, which can meet the speed needs of practical applications.
Anti-CF has superior inference speed among all TTA methods. Anti-CF is about 1.8 times faster than Tent, two times faster than EATA, 4.4 times faster than OIL, 3.4 times faster than SAR, and only about 20% slower than vanilla forward. This speed is faster than all existing TTA methods and has a vast advantage in real-world applications.
5 Further Analysis
5.1 Effects of the Learning Rate
The learning rate is a very important hyper- parameter of TTA. Choi et al. (2022) shows that a large learning rate may cause the model to collapse, while small learning rate can make TTA almost ineffective. However, careful hyper-parameter lection for TTA during test time is not feasible in practice. Therefore, an advanced TTA approach should be less sensitive to the learning rate. We use the FLM-RoBERTa--aae as the source model to test the sensitivity of each TTA method to the learning rate on the XQuAD dataset. The result is shown in Figure 4. We can observe that Tent, EATA and SAR are very sensitive to the learning rate. With the increase in the learning rate, the per- formance of them drops rapidly after reaching the maximum, which indicates that they are prone to model collapse under large learning rate. OIL per- forms better than Tent, EATA and SAR, but it still rapidly deteriorates after maintaining performance for a while. In contrast, Anti-CF is less sensitive to the learning rate. As the learning rate increases, the performance of Anti-CF will slowly decline until it approaches the performance of the source model.
Anti-CF can achieve comparable or better re- sults than other TTA methods. On the NoiseQA, XQuAD, and MLQA datasets, each TTA method performs well and can achieve performance im- provements based on the source model. Anti-CF can achieve comparable better results than other TTA methods without model collapse. Among them, when using llmre as the source model, the EM of Anti-CF is 5.98% higher than that of vanilla forward and 0.94% higher than the best per- formance among other TTA methods on NoiseQA- syn. On average, Anti-CF has stable improvement effect on all source models.
Anti-CF has great potential in real-world appli- cations. Previously, TTA was challenging to apply in real-world applications due to its instability. Al- though achieves performance improvements, will also sometimes causes the model to collapse, resulting in almost all output being false, which is unacceptable in real- world applications. Anti- CF avoid it. In addition, many existing TTA methods are becoming increasingly complex, incor- porating technologies such as contrastive learning (Chen et al., 2022), data augmentation (Liu et al., 2021; Zhang et al., 2022), and knowledge distilla-
5.2 Effects of
an important hyper-parameter of Anti-CE, sig- nificantly influencing the results. To thoroughly in- vestigate its impact, we conduct experiments on the NaturalQA dataset. Figure shows the effects a.

---

59.93
25000
Momory usage (MB) for each TTA method
60
60.86
60.76
61.05
61.18
61.01
20100
42.69
40
43.75
44.09
44.2
43.79
140033-4027
12972
30
10200
$167
5008
2011
10
2.57
1.42
Jlne buoe
alase lango
Burward
Text
EATA
OTL
3AR
Auci
0.0 0.2
0.4
0.6
0.8
1.0
Figure 7: Memory usage of various TTA methods.
Figure 5: EM and F1 of Anti-CF on NaturalQA under different We xlmr-base our source model and 5e-4 as the learning rate.
and when the amount of warmup data reaches the thousand level, the performance of Anti-CF reaches plateau. It indicates that warmup is essential harness the power of Anti-CF and is data-efficient, requiring only hundreds or thousands of samples to achieve exciting performance gains when data resources are limited. We speculate that because the model capacity of the efficient side block is small, there is less demand for training data.
REM FL
5.4 Memory Usage for TTA Methods.
5000
In practical applications, TTA requires additional memory, which poses challenges when deploying lightweight devices with limited memory sources. We discover that the efficient side block of Anti-CF can potentially solve this problem by reducing the memory required for back propaga- tion. To demonstrate this, we record the mem- ory required by each TTA method in Figure 7. It is evident that Anti-CF incurs minimal additional memory compared to vanilla forward. This is be- cause Anti-CF does not require passing through the backbone during back propagation and does not need record gradients within it. contrast, Tent and EATA require approximately twice the amount of memory. Furthermore, OIL maintains the state of teacher and student models, resulting in significant memory requirement. This renders OIL almost impractical for real-world applications. Thus, the memory efficiency of Anti-CF makes it promising choice for deployment, especially in resource-constrained scenarios.
Figure 6: EM and F1 of Anti-CF under different quanti- ties warmup data.
When a is set to O, indicating that Anti-CF does not impose any constraints on the adapted model, the model quickly collapses. However, even a slight in- crease in such as 0. provides enough constraint to prevent the model from collapsing, resulting in remarkable improvement in the EM score from 1.42% to 42.69%. This change demonstrates the effectiveness Anti-CE.
5.3 Effects of Warmup Data
We explore the impact of the amount of warmup data on Anti-CF, We use ximr-large as the source model on the NoiseQA-syn dataset and conducted experiments with different amounts of warmup data. We randomly sample the warmup data from the validation set of SQuAD. As shown in Figure 6, even if we do not perform warmup, Anti-CF still achieves performance over direct inference, which may be because source constraints make the efficient side block learn the knowledge from the source model the process of TTA. As the amount of warmup data grows, Anti-CF performs better,
5.5 Applications in Generative QA
With the recent amazing progress in generative large language models (LLMs),, generative QA is becoming increasingly valuable for research and application. In light of this, we also investigate the

---

0.8
distribution shift. TTA uses self-supervised signals to update the model the inference stage. has achieved surprising performance in various tasks (Liang et al., 2023). Depending on whether or not modifying the objectives of the training phase and accessing the training data, TTA can be divided into test-time training and fully test-time adapta- tion. Test-time training (Sun et al., 2020; Liu et al., 2021; Bartler et al., 2022; Gandelsman et al., 2022; Chen al., 2022) dedicated to designing a distri- bution shift-aware auxiliary task during the training phase and using this task to update the model dur- ing the testing phase. On the other line, the fully test time adaptation (Wang et al., 2021b) updates the model with test data using a self-training strat- egy without modifying the training process. How- ever, fully test-time adaptation faces performance degradation due the low quality of pseudo or soft labels, so current works (Niu al., 2022: Zhang et al., 2022; Jang and Chung, 2022; Gong et al., 2022; Niu et al., 2023; Song et al., 2023) focus on improving the quality of pseudo-labels, designing more robust and efficient tuning methods. In NLP, TTA has also attracted researchers' attention. Ye et al. (2022) first evaluate the effectiveness of fully test-time adaptation in the QA task and introduce an imitation learning paradigm ensure the quality of pseudo-labels.
Forward
Anc-CF
0.6
0.5
0.4
0.3
200
400
600
800
1000
Figure 8: Probability distribution of tokens generated BLOOM. The ordinate represents sum of probabilities of generating the top-k tokens.
Model
NoiseQA-syn
NoiseQA-na
Bloom
48.77
48.12
Tent
49.08
48.85
Ours
50.70
49.83
Table
EM on NoiseQA.
potential of TTA, especially Anti-CF in We train BLO0M-1.77 (Scao et al., 2022) on the SQuAD dataset using generative QA setup. Firstly, we study the distribution of generated to- kens in generative QA. We employ three methods during inference: direct inference, Tent, and Anti- CF. For each method, record the probability of each token appearing throughout the inference process. We then sum the probabilities of the top-k tokens. The results this analysis are presented in Figure 8. The analysis reveals the presence of an imbalanced label distribution in generative QA. High-frequence words are easier to be generated by LLMs. Moreover, Tent will still exacerbate the imbalance. However, by utilizing Anti-CF, this problem can be alleviated to some extent. Anti-CF helps mitigate the imbalanced label distribution's effects and promotes a more balanced generation of words during the inference process. Table shows the EM score of each method on NoiseQA. From the results, we can see that the effect of Anti-CF is better than that of Tent. TTA still play role in generative LLMs, and applying TTA in LLMs is feasible path with great potential.
Robustness NLP Training a sufficiently robust model is prerequisite for the practical application of a trustworthy NLP system. Wang et al. (2022) divide robustness in NLP into two categories: ad- versarial robustness under artificial attacks and nat- urally occurring distribution shifts. For adversarial robustness, researchers (Ebrahimi et al., 2018; Li et al., 2018: Alzantot et al., 2018; Jía et al., 2019; Garg and Ramakrishnan, 2020; Li et al., 2020; Lin et al., 2021; Ravichander et al., 2021) have pro- posed many methods for constructing adversarial attack samples to evaluate the model's adversarial robustness. NLP, naturally occurring distribu- tion shifts also have rich real-world scenarios, such as cross-lingual (Hu et al., 2020; Jafari et al., 2021; Zheng et al., 2021; Yang et al., 2022; Mao et al., 2022), cross-domant (Hendrycks et al., 2020; Ram- poni and Plank, 2020; Malinin et al., 2021), and different styles of text. Many robustness tuning methods are proposed improve the robustness of NLP models, such as data augmentation (Kaushik et al., 2020; Khashabi et al., 2020; Chen et al., 2020, 202 lb; Zhou et al., 2021) and adversarial training
6 Related Work
Test-Time Adaptation Test-Time Adaptation (TTA) is a promising paradigm to deal with the

---

(Miyato al., 2017: Madry al., 2018; Zhu et al., 2020; Wang et al., 2021a).
Acknowledgements
This work is supported by the National Science Foundation of China (NSFC No. 62206194) 94), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488).
7 Conclusion
In this paper, we attempt to improve the robustness of QA models by testing time adaptation (TTA) but find that TTA causes the models collapse. We thoroughly investigate why previous TTA methods cause the model collapse and find that the imbal- anced label distribution is the main reason. We ad- dress this problem by adding constraints between the source and adapted model during the TTA pro- cess. We also design an efficient side block to speed up the inference time. Sufficient experimental re- sults show that our proposed method effective and efficient, making TTA big step closer to being upplied in real-world scenarios.
References
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho. Mani Srivastava, and Kai- Wei Chang. 2018. Generating natural language adversarial ex- amples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 2890-2896, Brussels, Belgium. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pages 4623-4637.
Limitations
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. 2022. M13: Meta test-time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics, pages 3080-3090. PMLR.
Although our proposed Anti-CF has made signif- icant progress in terms of stability and inference efficiency compared with existing TTA methods, there are still some limitations:
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Anti-CF constrains the adapted model's predic- tion with the source model's prediction, which prevents TTA from model collapse and effec- tively improves the lower bound of the model performance. However, the source model tends to perform poorly under distribution shift, and strong constraint similar to KL divergence can limit the upper bound the model performance. The experimental results in Table also imply this limitation. Therefore, we will explore more flexible constraints in the future.
Dian Chen, Dequan Wang. Trevor Darrell, and Sayna Ebrahimi. 2022. Contrastive test-time adaptation. Proceedings of the IEEF/CVE Conference on Com- puter Vision and Pattern Recognition, pages 295- 305.
Jiaao Chen, Dinghan Shen, Weizhu Chen, and Diyi Yang. 2021a. HiddenCut: Simple data augmenta- tion for natural language understanding with better generalizability. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4380-4390 Online, Association for Computational Linguistics.
In this paper, we empirically design the efficient side block without exploring the impact of its structure and capacity on the performance of TTA. The side block still has great potential to be explored, and we hope that future work will focus on this aspect.
Jiaao Chen. Dinghan Shen, Weizhu Chen, and Diyi Yang. 2021b. Hiddeneut: Simple data augmenta- tion for natural language understanding with better generalizability. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11 International Joins Conference on Natural Language Processing (Volume 1: Long Papers), pages 4390-4390.
In this paper, we only validate the effectiveness of Anti-CF in extractive and generative QA. How- ever, Anti-CF is task-independent and model- independent,
believe that Anti-CF is also applicable to other NLP tasks and pre-trained lan- guage models. But we do not verify it because of time and computing power constraints. We will further validate the generalizability of Anti-CF in the future.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix- Text: Linguistically-inee informmed interpolation of hid- den space for semi-supervised text classification. In

---

Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2147- 2157, Online. Association for Computational Lin- guistics.
Robust continual test-time adaptation against tempo- ral correlation. In Advances in Neural Information Processing Systems.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve di-ttii-itiiitoion robustness. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 2744-2751 Online. Association for Computa- tional Linguistics.
Sungha Choi, Seunghan Yang, Scokeon Choi, and Sun- grack Yun. 2022. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, Octo- ber 23-27, 2022, Proceedings, Part XXXIII, pages 440-458. Springer.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, 2019. Parameter-efficient transfer learning for nlp. In Inter- national Conference on Machine Learning.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-ling lingull representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig. Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross- moos-llnggull generalisa- tion. In International Conference on Muchine Learn- ing, pages 4411-4422 PMLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
Amir Reza Jafari, Behnam Heidary, Reza Farahbakhsh, Mostafa Salehi, and Mahdi Jalili. 2021. Transfer learning for multi-lingual tasks a survey. CoRR, abs/2110.02052.
Minguk Jang and Sae- Young Chung. 2022. Test-time adaptation via self-training with nearest neighbor in- formation. arXiv preprint 22%:::2007.10922
Matthew Dumn. Levent Sagun, Mike Higgins. V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from search engine. arXiv preprint 1597077
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. In Proceeulings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language P Processing (EMNIP-IJCNIP), pages 4 4129-4142, Hong Kong, China, Association for Computational Linguistics.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 31-36.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Funsol Choi, and Danqi Chen. 2019, MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China. Association for Computa- tional Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume I: Long Papers), pages 1601-161 I, Vancouver, Canada. Association for Computational Linguistics.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. 2022. Test-time training with masked autoen- coders. In Advances in Neural Information Process- ing Systems.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a differ- ence with caatterfactually-ugeeeee data. In Inter- national Conference on Learning Representations.
Siddhant Garg and Goutham Ramakrishnan. 2020. BAE: BERT-based adversarial examples for text clas- sification. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 6174-6181 Online. Association for Computational Linguistics.
Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. 2020. More bang for your buck: Natural perturba- tion for robust question answering. In Proceedings of the 2020 Conference on Empirical Methods Natu- ral Language Processing (EMNLP), pages 163-170, Online. Association for Computational Linguistics.
Taesik Gong, Jongheon Jeong, Taewon Kim. Yewon Kim, Jinwoo Shin, and Sung-Ju Ju Lee. 2022. NOTE:
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti,

---

Danielle Epstein, Iilia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wa Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark JF Gales, Alexey Noskov, An- drey Ploskonosov. Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. 2021. Shifts: A dataset of real distributional shift across multiple large-scale tasks, arXiv preprint arXiv:2107.07455.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Chazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. Bart: Denoising sequence-to peuvence-tosequeeee toseeunecee pre pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880.
Yuren Mao, Yaobo Liang, Nan Duan, Haobo Wang. Kai Wang, Lu Chen, and Yunjun Gao. 2022. Less- forgetting multi- lingual fine tuning. In Advances in Neural Information Processing Systems.
Patrick Lewis. Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020b. MIqa: Evaluat- ing cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7315- 7330.
Takeru Miyato, Andrew M Dai, and Ian Goodfel- low, 2017. Adversarial training methods for semi- supervised text classification.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen. Shijian Zheng, Peilin Zhao, and Mingkui Tan. 2022. Efficient test-time model adaptation without forgetting. arXiv preprint arXiv:2204.02610.
Jinfeng Li. Shouling Ji. Tianyu Du, Bo Li, and Ting Wang. 2018 Texthugger: Generating adversarial text against real-world applications. arXiv preprint arXiv.1812.02271.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. 2023. Towards stable test-time adaptation in dy- namic wild world. In The Fleventh International Conference on Learning Representations.
Juntao Li, Ruidan He, Hai Ye, Hwee Tou Ng. Lidong Bing. and Rui Yan. 2021. Unsupervised domain adaptation of a pretrained cross-lingual language model. In Proceedings of the Twenty-Ninth Interna- tional Conference International Joint Conferences on Artificial Intelligence, pages 3672-3678.
OpenAL. 2022. for dialogue. Open Al, blog.
Chatgpt: Optimizing language models
OpenAL. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Linyang Li, Ruotian Ma. Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial at- tack against bert using bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1193-6202..
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the lim- of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Jian Liang. Ran He, and Tien-Ping Tan. 2023. A com- prehensive survey on test-time adaptation under dis- tribution shifts. ArXiv, abs/2303. 15361.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018 Know what you don't know: Unanswerable ques- tions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789.
Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. Rockner: A simple method to create adversarial examples for evaluating the robust- ness of named entity recognition models. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language P Processing, pages 1723-773.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Alan Ramponi and Barbara Plank. 2020. Neural un- supervised domain adaptation in nlp- a survey, In Proceedings of the 28th International Conference on Computational Linguistics, pages 6838-6855.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. 2021. Ttt++ When does self-supervised test-time training fail or thrive? Advances in Neural Informa- tion Processing Systems, 34:21 208-21820.
Abhilasha Ravichander, Siddharth Dalmia, Maria Ryskina. Florian Metze, Eduard Hovy, and Alan Black. 2021. Noiseqa: Challenge set evaluation for

---

user-centric question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Vol- ume, pages 2976-2992.
Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. Measure and improve robustness in NLP models: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-aceess multilingual language model. arXiv preprint 1rX50:221..01000
Seattle, United States. Association for Computational Linguisties.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, et al. 2013. Ontonotes release 5.0 1dc2013119. Linguistic Data Consortium, Philadelphia, PA, 23.
Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. 2023, Ecotta: Memory- -efirreefffiiennt continual test- time adaptation via self-distille distilled regularization, In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 11920- 11929.
David H Wolpert and William G Macready. 1997. No free lunch theorems for optimization. IEEE transac- tions on evolutionary computation, ((66722-
Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. 2022. Enhancing cross-lingual transfer by manifold mixup. In International Conference on Learning Representations.
Yi Su, Yixin Ji, Juntao Li, Hai Ye, and Min Zhang. 2023, Test-time adaptation with perturbation consistency learning. arXiv preprint arXiv:2304.1 120..4244.
Yu Sun, Xiaolong Wang. Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. 2020. Test-time training with sulf-suue for generalization un- der distribution shifts. In International conference on machine learning, pages 9229-9248. PMLR.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. 2023. Ilama: Open and effi- cient foundation language models. arXiv preprint a9X7
Hai Ye, Yuyang Ding, Juntao Li, and Hwee Tou Ng. 2022. Robust question answering against distribution shifts with test-time adaptation: An empirical study.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Har- ris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehen- sion dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui. Qi Zhang, and Xuanjing Huang. 2023. A comprehen- sive capability analysis of gpt-3 and gpt-3.5 series models. ArXiv, abs/2303. 10420.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Ilia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Marvin Mengxin Zhang, Sergey Levine, and Chelsea Finn. 2022. MEMO: Test time robustness via adap- tation and augmentation. In Advances in Neural Information Processing Systems.
Bo Zheng, Li Dong, Shaohan Huang. Wenhui Wang. Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Consistency reg- ularization for cross- wooss-liggull fine-tuning. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3403-3417
Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo and Jingjing Liu. 2021a. Info bert]: Improving robustness of language models from an information theoretic perspective. In International Conference on Learning Representations.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2021b. Tent: Fully test-time adaptation by entropy minimization. In In- ternational Conference on Learning Representations.
Kun Zhou, Wayne Xin Zhao, Sirui Wang. Fuzheng Zhang, Wei Wu, and Ji-Rong Wen. 2021. Virtual data augmentation: A robust and general framework for mue-tuning pre-trained models. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 3875-3887, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jindong Wang. Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Weirong Ye, Xiubo Geng, Binxing Jiao, Yue Zhang, and Xingxu Xie. 2023. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. ArXiv, abs/2302. 20055.

---

Exact Match
- AdamW Adam SGD Forward
50%
40%
lncle
30%
20%
500
1000
1500
2000
2500
Span
10%
Figure 10: Ground truth of start positions on NaturalQA.
1e-5
2e-5
Se-S
1e-4
2e-4
Se-4
1e-3
Learning Rate
Figure 9: EM (%) on NaturalQA of Forward and Tent under different optimizers and learning rates.
2903
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Gold- stein, and Jingjing Liu. 2020. Freelb: Enhanced ad- versarial trining for natural language understanding. International Conference on Learning Representa- tions.
(a) Ground Truth
(b) Forward
50
100
A The Impact of Different Optimizers and Learning Rates on Tent
Tent
(d) Anti-CF
B
Hyper parameters
Figure 1I: The end positions of NaturalQA given by the model.
Table 3 shows the detail of hyper parameters.
D Further analysis on Named Entity Recognition (NER)
Methods
xhmr-base
xTune
xlmr-large
Tent
[R: 5e-5
IR: 5e-5
IR: 5e-5
LR: 5c-5
LR:
LR: 5c-5
In cases where the label distribution in the test dataset is extremely imbalanced, most TTA meth- ods can lead to model collapse, and Anti-CF can effectively avoid this situation. Based on this idea, we design the following experiment.
EATA
1/2000
1/2000
1/2000
Eg: 0.41m(512)
Eg: 0.41m(512)
Eg: 0.41m(512)
LR: le-4
LR: lc-4
LR: 5c-4
K:
K:
K:
OIL
2:
y: 0.5
7:
as 0.99
We have extended Anti-CF to NER. We RockNER (Lin al.. 2021) conduct our experi- ment, which is generated from entity-level adver- sarial attacks on the OntoNotes (Weischedel et al., 2013) dataset. We want to demonstrate that Anti- CF can effectively improve model performance on RockNER and will not collapse in extreme situa-
f:: 1
LR: lc-4
LR: lc-4
LR: 5c-4
Fm, 0.9
Fig 0.9
Ey, 0.9
SAR
0.05
p: 0.05
0.05
Ep: 0.41n(512)
En: 0.41m(5112
Ea: 0.41n(512)
LR: 5c-5
LR: 5c-5
LR: 5c-5
a: 0.2
0.2
0.01
Table 3: Details of hyper parameters.
Methods
RockNER
Corrupted
Time
BERT-base
$6.9
56.9
0.51
C Other start and end positions
+Tent
60.6
10.7
1.71
+EATA
60.5
15.7
1.80
+OIL.
58.3
56.8
4.25
Figure 10 shows the start positions of ground truth on NaturalQA and Figure shows the end posi- tions.
+Ours
60.2
58.0
0.78
Table 4; Results on NER.

---

tions. To prove the second point, artificially corrupted RockNER by placing samples with more 'O' in front of it. The model will first infer and update samples labelled with more 'O', making the model easier to collapse due to this extreme label distribution. The results are shown in Table 4.


Reviews:
[{"review_id": "dWunoXmA9S", "reviewer": "Reviewer_kMrG", "summary": "", "questions": "PCL NB", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your valuable comments. We hope the following responses can address your concern:\n\nQ1: \n\nAnti-CF uses KL divergence to constrain the adapted model's output to not deviate too much from the source model, which limits the upper bound of performance to some extent. More flexible constraint methods could be explored.\n\nA1: \n\nThis is correct, and we have also indicated this issue in our Limitation. In fact, it is possible to have better constraints that enable the model to avoid collapse while not limiting the upper bound of performance. But we want to say that KL divergence is already a very good constraint, so considering our limited time and computing resources, we did not explore other constraints.\n\n\n\nQ2: \n\nCould the author compare with stronger TTA methods? For example, SAR , AdaNPC, LAME.\n\nA2: \n\nThank you for your suggestion. In fact, most of the existing TTA methods are only validated in CV tasks, and we have included existing baselines in the NLP field. Nevertheless, to address your concerns, we will still migrate some baselines in the CV field to NLP. We have added two new baselines, SAR and LAME (we don't select AdaNPC because it requires retraining the classification head with KNN, which may result in an unfair comparison). All experimental settings are the same as the other baselines in the paper. We repeat our experiments three times under different random seeds and report their mean values. We will add these two baselines to the final version of our paper. Due to space limitations, we only report the EM values here.\n\n\n\n| Models     | NoiseQA-syn | NoiseQA-na | XQuAD | MLQA  | MRQA  | Avg.  | Time  |\n| ---------- | ----------- | ---------- | ----- | ----- | ----- | ----- | ----- |\n| xlmr-base  | 66.61       | 66.08      | 55.59 | 48.27 | 40.12 | 55.33 | 4.25  |\n| +SAR       | 67.63       | 67.36      | 55.94 | 48.01 | 31.71 | 54.13 | 18.30 |\n| +LAME      | 66.43       | 65.88      | 55.47 | 48.10 | 39.20 | 55.02 | 4.62  |\n| +Ours      | 68.08       | 67.50      | 56.75 | 48.32 | 40.16 | 56.16 | 5.23  |\n| xTune      | 70.92       | 69.72      | 58.81 | 50.66 | 43.46 | 58.70 | 4.25  |\n| +SAR       | 70.90       | 70.34      | 59.23 | 51.19 | 36.73 | 57.68 | 18.30 |\n| +LAME      | 70.64       | 69.40      | 58.52 | 50.32 | 42.46 | 58.27 | 4.62  |\n| +Ours      | 71.32       | 70.19      | 59.01 | 51.18 | 43.44 | 59.02 | 5.23  |\n| xlmr-large | 65.55       | 64.17      | 63.15 | 54.40 | 46.17 | 58.69 | 12.37 |\n| +SAR       | 69.29       | 67.70      | 63.52 | 54.43 | 46.41 | 60.27 | 52.47 |\n| +LAME      | 65.35       | 63.94      | 62.97 | 54.12 | 45.35 | 58.34 | 12.63 |\n| +Ours      | 71.53       | 69.28      | 63.78 | 54.93 | 46.24 | 61.15 | 15.36 |\n\n\n\nFrom the table, we can see that Anti-CF still outperforms these two stronger baselines.\n\nThe performance of SAR is stronger than the other baselines selected in our paper. It searches for the optimal solution within the neighborhood, making the performance of TTA more stable. Therefore, it can alleviate the problem of model collapse to some extent. But due to its need for two forward propagation and two backward propagation, its inference time is approximately twice that of Tent. \n\nLAME does not update the parameters of the model, only adjust the output probability. Although this can avoid model collapse, it also greatly limits the performance of TTA. In addition, due to the differences between CV tasks and NLP tasks, LAME has almost no effect on our datasets.\n\n\n\n\n\nQ3: \n\nAnti-CF is currently validated only in the question answering task, and it can be extended to other NLP tasks in the future.\n\nA3: \n\nAnti-CF can achieve comparable or better performance in most scenarios where TTA performs well. In addition, in cases where the label distribution in the test dataset is extremely imbalanced, most TTA methods can lead to model collapse, and Anti-CF can effectively avoid this situation. Based on this idea, we design the following experiment.\n\nWe have extended Anti-CF to NER. We use RockNER (Lin et al., 2021) to conduct our experiment , which is generated from entity-level adversarial attacks on the OntoNotes (Weischedel et al., 2013) dataset. We want to demonstrate that Anti-CF can effectively improve model performance on RockNER and will not collapse in extreme situations. To prove the second point, we artificially corrupted RockNER by placing samples with more 'O' in front of it. The model will first infer and update samples labelled with more 'O', making the model easier to collapse due to this extreme label distribution. The results are shown in the table below. More experimental details will be presented in the final version of our paper.\n\n| methods   | RockNER | Corrupted | Time |\n| --------- | ------- | --------- | ---- |\n| BERT-base | 56.9    | 56.9      | 0.51 |\n| +Tent     | 60.6    | 10.7      | 1.71 |\n| +EATA     | 60.5    | 15.7      | 1.80 |\n| +OIL      | 58.3    | 56.8      | 4.25 |\n| +Ours     | 60.2    | 58.0      | 0.78 |\n\nWe can see that Anti-CF can still achieve comparable performance compared to other TTA methods and can prevent the model from collapsing with the fastest inference speed.\n\n\n\nQ4:\n\nThe selection of hyperparameters in the experiments could be more systematic. It needs further investigation whether different datasets and models require different combinations of hyperparameters. \n\nA4:\n\nWe have briefly explained the principles for selecting hyperparameters under our settings in lines 336-340. We will provide a more detailed explanation here. \n\nThe goal of TTA is to take effect in real-world application scenarios, and TTA does not make any assumptions about the distribution of test samples. Therefore, it is unreasonable to choose different hyperparameters for different datasets, as we do not know how TTA performs in the test set. In addition, if we can choose different hyperparameters for different datasets, then TTA will never cause the model collapse, because if we have found that TTA causes the model collapse, we can directly set the learning rate to 0, which is a type of cheating.  LAME (Malik et al., 2022) also proposed the same viewpoint.\n\nIt is reasonable to choose different hyperparameters for different models, because we can use experience to determine which models require a higher learning rate and which require a lower learning rate, which is feasible in real-world applications. So, our setting is: once a model and a TTA method are determined, their hyperparameters cannot be changed.\n\nIn our experiment, we have explored a learning rate range of {5e-5, 1e-4, 5e-4}, which can make all TTA methods in our experiment effective. However, in the table, Tent and EATA tend to choose 5e-5 because model collapse can be alleviated at low learning rates. Regarding hyperparameters beyond the learning rate, we have respectively referred to the hyperparameter settings of OIL and EATA.\n\nIt should be noted that Tent and EATA still perform well on other datasets under 5e-5, so there is no suspicion that we intentionally lowered the baseline. We report the EM values of Tent under different learning rates and datasets under xlmr-base.\n\n| LR   | NoiseQA-syn | NoiseQA-na | XQuAD | MLQA  | MRQA  | Avg.  |\n| ---- | ----------- | ---------- | ----- | ----- | ----- | ----- |\n| 5e-5 | 68.14       | 67.44      | 56.40 | 48.11 | 20.77 | 52.17 |\n| 1e-4 | 68.40       | 67.67      | 56.78 | 48.16 | 16.52 | 51.51 |\n| 5e-4 | 67.12       | 66.79      | 56.50 | 47.70 | 6.34  | 48.90 |\n\nIn addition, more precise selection of hyperparameters can also improve the performance of Anti-CF. We report the EM values of Anti-CF under xlmr-large and NoiseQA-syn with different hyperparameters.\n\n| LR/$\\alpha$ | 0.01  | 0.2   | 0.8   |\n| ----------- | ----- | ----- | ----- |\n| 5e-5        | 68.21 | 68.08 | 66.92 |\n| 1e-4        | 67.51 | 67.82 | 66.86 |\n| 5e-4        | 65.61 | 67.00 | 66.65 |\n\n\n\nQ5:\n\nThe computational and memory complexity of Anti-CF can be analyzed in more detail.\n\nA5:\n\nWe have reported the inference speed of Anti-CF in Section 4.4 and the memory requirements in Section 5.4. We believe that the results in our paper fully demonstrate the superiority of Anti-CF over other TTA methods. Due to time constraints during rebuttal period, we will add further analysis in the final version of our paper.\n\n\n\nReferences:\n\n- Lin et al., 2021, A simple method to create adversarial examples for evaluating the robustness of named entity recognition models, EMNLP. \n- Weischedel et al., 2013, Ontonotes release 5.0 ldc2013t19, Linguistic Data Consortium.\n- Malik et al., 2022, Parameter-free Online Test-time Adaptation, CVPR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. By analyzing, it was discovered that the imbalance in label distribution in the test set of the question answering task is a major cause of model collapse in TTA methods. This finding is of great significance and provides important guidance.\n2. The Anti-CF method is proposed, which uses the output of the source model as soft label constraints for the adapted model in the TTA process, effectively preventing model collapse.\n3. Anti-CF is demonstrated to achieve stable performance improvements in various distribution shift scenarios, making TTA more reliable in practical applications.", "reasons_to_reject": "1. Anti-CF uses KL divergence to constrain the adapted model's output to not deviate too much from the source model, which limits the upper bound of performance to some extent. More flexible constraint methods could be explored.\n2. Could the author compare with stronger TTA methods? For example, SAR (https://openreview.net/pdf?id=g2YraF75Tj), AdaNPC (https://arxiv.org/abs/2304.12566), LAME (https://arxiv.org/abs/2201.05718).\n3. Anti-CF is currently validated only in the question answering task, and it can be extended to other NLP tasks in the future.\n4. The selection of hyperparameters in the experiments could be more systematic. It needs further investigation whether different datasets and models require different combinations of hyperparameters. The computational and memory complexity of Anti-CF can be analyzed in more detail.", "questions_to_authors": "PCL NB", "paper_topic_and_main_contributions": "This paper proposes a new TTA method called Anti-CF to address the problem of performance degradation of pre-trained language models in distribution shift scenarios. It discusses the causes of potential model collapse issues that could be caused by TTA, and improves the stability and efficiency of TTA in practical applications.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "54sDvG90fx", "reviewer": "Reviewer_CXeg", "summary": "", "questions": "Anti-CF with xlmr-base and xTune is underperforming compared to the other baseline models for NoiseQA-Syn, NoiseQA-na and XSQuAD tasks, whereas it is performing well for MLQA and MRQA. Is there any specific explanation for this?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and acknowledgment. We hope the following responses can address your concern:\n\n\n\nQ1:\n\nAlthough this study shows significant improvements in TTA for extractive QA task, it did not show the performance measure of any other task.  Thus, it is not clear about the generalizability of the proposed method for other tasks. Offcourse, it is hard to include several tasks within the given space, some discussions along the line would have been great.\n\nA1:\n\nAnti-CF can achieve comparable or better performance in most scenarios where TTA performs well. In addition, in cases where the label distribution in the test dataset is extremely imbalanced, most TTA methods can lead to model collapse, and Anti-CF can effectively avoid this situation. Based on this idea, we design the following experiment.\n\nWe have extended Anti-CF to NER. We use RockNER (Lin et al., 2021) to conduct our experiment , which is generated from entity-level adversarial attacks on the OntoNotes (Weischedel et al., 2013) dataset. We want to demonstrate that Anti-CF can effectively improve model performance on RockNER and will not collapse in extreme situations. To prove the second point, we artificially corrupted RockNER by placing samples with more 'O' in front of it. The model will first infer and update samples labelled with more 'O', making the model easier to collapse due to this extreme label distribution. The results are shown in the table below. More experimental details will be presented in the final version of our paper.\n\n| methods   | RockNER | Corrupted | Time |\n| --------- | ------- | --------- | ---- |\n| BERT-base | 56.9    | 56.9      | 0.51 |\n| +Tent     | 60.6    | 10.7      | 1.71 |\n| +EATA     | 60.5    | 15.7      | 1.80 |\n| +OIL      | 58.3    | 56.8      | 4.25 |\n| +Ours     | 60.2    | 58.0      | 0.78 |\n\nWe can see that Anti-CF can still achieve comparable performance compared to other TTA methods and can prevent the model from collapsing with the fastest inference speed.\n\n\n\n\n\nQ2: \n\nAnti-CF with xlmr-base and xTune is underperforming compared to the other baseline models for NoiseQA-Syn, NoiseQA-na and XSQuAD tasks, whereas it is performing well for MLQA and MRQA. Is there any specific explanation for this?\n\nA2:\n\nEven though Anti-CF can effectively prevent model collapse, it still limits the upper bound of performance to some extent, as we  have discussed in the Limitation section of our paper. So on datasets such as NoiseQA-syn, NoiseQA-na, and XQuAD that already perform well with TTA, the performance of Anti-CF may be slightly lower. On datasets with poor TTA performance such as MLQA and MRQA, Anti CF performs better than other methods. \n\nFrom another perspective, MLQA and MRQA have longer sentence lengths, resulting in more segmentation during preprocessing, resulting in more labels falling at position 0 (indicating the answer is not in the current span). Therefore, other TTA methods perform mediocrely on datasets such as MLQA and MRQA, and this is where Anti-CF can come into play.\n\n\n\n\n\nReferences:\n\n- Lin et al., 2021, A simple method to create adversarial examples for evaluating the robustness of named entity recognition models, EMNLP. \n- Weischedel et al., 2013, Ontonotes release 5.0 ldc2013t19, Linguistic Data Consortium.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work proposes an approach to prevent model collapse during test time adaptation. \n2. By introducing a side-block along with the main model, it has significantly reduced the computational cost and presents an efficient approach\n3. Experiments are carefully designed and presented nicely", "reasons_to_reject": "Although this study shows significant improvements in TTA for extractive QA task, it did not show the performance measure of any other task. Thus, it is not clear about the generalizability of the proposed method for other tasks. Offcourse, it is hard to include several tasks within the given space, some discussions along the line would have been great.", "questions_to_authors": "Anti-CF with xlmr-base and xTune is underperforming compared to the other baseline models for NoiseQA-Syn, NoiseQA-na and XSQuAD tasks, whereas it is performing well for MLQA and MRQA. Is there any specific explanation for this?", "paper_topic_and_main_contributions": "This paper aims to improve the application of pre-trained language models (PLMs) on tasks/scenarios where a distribution shift is expected. It has been observed that the performance of PLMs deteriorates when the test distribution does not match the training distribution. In literature, some attempts have been made to update the model during the testing phase to make the model adapted for distribution shift. However, due to the unreliable output, the test time adaptation (TTA) strategy also fails. This study contributes in the following ways -\n1. Consider a particular task of Question-Answering and investigates why TTA causes model collapse\n2. Proposes Anti-Collapse Fast test-time adaptation (Anti-CF) to mitigate model collapse issue. Anti-CF treats outputs of the source model as a soft label to regularize model updates during test time.\n3. Extensive empirical studies to show the proposed Anti-CF can effectively take care of model collapse.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zPI49HCI5Y", "reviewer": "Reviewer_76Fj", "summary": "", "questions": "- Lines 182-184 state that \"Therefore, the imbalanced distribution of test labels has led to TTA leaning too much towards the majority class during the update process\". Could you clarify the meaning of \"labels\" and \"the majority class\" here?\n\n\n- When computing the predicted probability for the adapted model, do the efficient side block's parameters require forward propagation? If so, lines 237-239 \"... as the adapted model so that we only need one forward propagation to obtain the two outputs simultaneously\" are incorrect.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and constructive suggestions.  We hope the information attached below can address your concern.\n\n\n\nQ1:\n\nWhether Tent was tuned properly in Section 2.2 is still questionable.\n\nA1:\n\nIn fact, we have made a reasonable parameter selection for Tent. \n\nIn Section 2.2, We used the AdamW optimizer and selected a learning rate of 1e-4. This is because such settings perform well in most datasets, such as NoiseQA-syn. We have list the performance of Tent in NoiseQA-syn below for reference.\n\n| LR         | 1e-5  | 2e-5  | 5e-5  | 1e-4  | 2e-4  | 5e-4  | 1e-3  |\n| ---------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| xlmr-base  | 66.61 | 66.61 | 66.61 | 66.61 | 66.61 | 66.61 | 66.61 |\n| Tent+AdamW | 67.11 | 67.51 | 68.14 | 68.52 | 68.54 | 67.87 | 66.11 |\n| Tent+Adam  | 67.20 | 67.54 | 68.07 | 68.54 | 68.49 | 67.90 | 65.85 |\n| Tent+SGD   | 66.69 | 66.67 | 66.67 | 66.64 | 66.95 | 67.11 | 67.42 |\n\n\n\nWe also repeat the experiments in the preliminary under different hyperparameters. For batch size, as SAR (Niu et al., 2023) discussed it in detail, BN layers make Tent sensitive to batch size. However, in Transformer based models, there is no BN layer, only LN layers. Therefore, Tent is not sensitive to batch size in our experiment, and our previous practical process has proven this.\n\nWe have made a more detailed selection of learning rates and optimizers. All other experimental settings are the same as in Section 2.2. We report the EM values due to space limitations.\n\n| LR         | 1e-5  | 2e-5  | 5e-5  | 1e-4  | 2e-4  | 5e-4  | 1e-3  |\n| ---------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| xlmr-base  | 43.44 | 43.44 | 43.44 | 43.44 | 43.44 | 43.44 | 43.44 |\n| Tent+AdamW | 38.15 | 28.53 | 18.59 | 11.65 | 7.78  | 3.32  | 1.70  |\n| Tent+Adam  | 38.80 | 28.36 | 16.76 | 13.39 | 9.17  | 2.71  | 1.52  |\n| Tent+SGD   | 42.72 | 42.61 | 42.28 | 23.78 | 14.86 | 6.28  | 1.89  |\n\nObviously, regardless of learning rates and optimizers, Tent cannot perform well on the NaturalQA dataset. And the trend is that the higher the learning rate, the lower its accuracy, which is also in line with expectations. When the learning rate is set to 0, it is obvious that the model will never collapse, and as the learning rate increases, the model collapses faster.\n\nIt should be noted that the goal of TTA is to take effect in real-world application scenarios, and TTA does not make any assumptions about the distribution of test samples. Therefore, it is unreasonable to choose different hyperparameters for different datasets, as we do not know how TTA performs in the test set. In addition, if we can choose different hyperparameters for different datasets, then TTA will never cause the model collapse, because if we have found that TTA causes the model collapse, we can directly set the learning rate to 0, which is a type of cheating.  LAME (Malik et al., 2022) also proposed the same viewpoint.\n\n\n\n\n\nQ2：\n\nThe authors are encouraged to discuss the proposed side block compared to existing PEFT methods (e.g., LoRA) to justify its novelty.\n\nA2：\n\nSide block can be considered as a type of PEFT, and its advantage is that it can simultaneously obtain source output and adapted output. However, other PEFT methods such as LoRA cannot achieve this, as they require at least two forward propagation to get two different outputs. In addition, the gradient of Anti-CF only propagates in the side block and no propagation will occur in the backbone, which greatly reduces the time and memory required for backpropagation.\n\nIn fact, the idea that we only adjust LayerNorm comes from PEFT, which can also be seen as a simple PEFT, similar to BitFit.\n\nWe conduct the entire experiment with Tent again using LoRA. We repeat the experiment three times under different random seeds and report its mean value. Due to space limitations, we only report EM values. The rank we use in LoRA is 4.\n\n| Models     | NoiseQA-syn | NoiseQA-na | XQuAD | MLQA  | MRQA  | Avg.  | Time  |\n| ---------- | ----------- | ---------- | ----- | ----- | ----- | ----- | ----- |\n| xlmr-base  | 68.63       | 67.54      | 57.09 | 48.15 | 16.28 | 51.54 | 10.97 |\n| xTune      | 71.54       | 70.25      | 59.55 | 50.06 | 23.21 | 54.92 | 10.97 |\n| xlmr-large | 70.51       | 68.70      | 63.61 | 52.51 | 7.62  | 52.59 | 33.66 |\n\nWe can see that the results of Tent obtained using LoRA are similar to those in our main experiment in terms of accuracy and inference speed.\n\nIn addition, we also try to use both LoRA and KL constraints simultaneously. We make two copies of the source model, one frozen and the other updated using LoRA. These two models are called the source model and the adapted model respectively. We obtain the source output through a forward propagation in the source model the adapted output through a forward propagation in the adapted model, then update the adapted model using Eq. (4) in out paper. Obviously, this method requires two forward propagation and one backpropagation in the backbone. The results of the experiment are shown in the table below. We only report EM values due to space limitations.\n\n| Models             | NoiseQA-syn | NoiseQA-na | XQuAD | MLQA  | MRQA  | Avg.  | Time  |\n| ------------------ | ----------- | ---------- | ----- | ----- | ----- | ----- | ----- |\n| xlmr-base+LoRA+KL  | 68.26       | 67.55      | 56.75 | 48.42 | 40.27 | 56.27 | 15.12 |\n| xlmr-base+Anti-CF  | 68.08       | 67.50      | 56.75 | 48.32 | 40.16 | 56.16 | 5.23  |\n| xTune+LoRA+KL      | 71.85       | 71.20      | 59.76 | 51.17 | 43.16 | 59.43 | 15.12 |\n| xTune+Anti-CF      | 71.32       | 70.19      | 59.01 | 51.18 | 43.44 | 59.02 | 5.23  |\n| xlmr-large+LoRA+KL | 70.81       | 68.57      | 63.88 | 54.26 | 46.91 | 60.28 | 45.84 |\n| xlmr-large+Anti-CF | 71.53       | 69.28      | 63.78 | 54.93 | 46.24 | 61.15 | 15.36 |\n\nThe results in the table indicate that this approach may perform slightly better than Anti-CF, but it will be about three times slower than Anti-CF. We believe that it is not worth trading such inference time costs for an extremely limited increase in performance. In fact, we have tried this method before, but for faster inference speed, we designed the side block.\n\n\n\n\n\nQ3:\n\nLines 182-184 state that \"Therefore, the imbalanced distribution of test labels has led to TTA leaning too much towards the majority class during the update process\". Could you clarify the meaning of \"labels\" and \"the majority class\" here?\n\nA3:\n\nLabels refer to the answer position of the current span. When each sample is cut into many spans, the answer is likely not in the current span, so the position defaults to 0. So, many span labels are 0, which is what I call the major class. \n\n\n\nQ4:\n\nWhen computing the predicted probability for the adapted model, do the efficient side block's parameters require forward propagation? If so, lines 237-239 \"... as the adapted model so that we only need one forward propagation to obtain the two outputs simultaneously\" are incorrect.\n\nA4:\n\nThere will be forward propagation in the side block, but the side block is relatively small compared to the backbone and can generally be ignored. Our side block is similar to a PEFT method, just like LoRA with an additional classifier. We generally cannot say that it obtains output with forward propagation twice. It is incorrect to consider its forward propagation in the backbone and in the low rank matrix as two forward propagation.\n\n\n\nQ5:\n\nIn Eq. (2), incorporating KL divergence between two different predicted probabilities is similar to that of X-MIXUP (the last term of Eq. (7) in Yang et al. (2021)). Although this paper computes the loss at test time, it is worth mentioning/citing it.\n\nA5:\n\nAdding KL divergence constraints to normal training is a common idea, so we may have overlooked this paper. We will add this citation to the final version of our paper.\n\n\n\nQ6:\n\nIn Table 1, reporting inference time is helpful. It would be nicer if the paper also shows the total number of parameters for each model.\n\nA6:\n\nWe have reported the usage of memory in Section 5.4, which can reflect the relative size of parameter quantities. The final version of our paper will show the total number of parameters of each model in Table 1.\n\n\n\n\n\nQ7:\n\nIn Table 3, the learning rate (LR) choices are strange. Typically, the same LR does not work well for different pre-trained LMs (e.g., XLM-RoBERTa-base vs. XLM-RoBERTa-large). This would lead to suboptimal performance.\n\nA7:\n\nWe have briefly explained the principles for selecting hyperparameters under our settings in lines 336-340. We will provide a more detailed explanation here. \n\nThe goal of TTA is to take effect in real-world application scenarios, and TTA does not make any assumptions about the distribution of test samples. Therefore, it is unreasonable to choose different hyperparameters for different datasets, as we do not know how TTA performs in the test set. In addition, if we can choose different hyperparameters for different datasets, then TTA will never cause the model collapse, because if we have found that TTA causes the model collapse, we can directly set the learning rate to 0, which is a type of cheating.  LAME (Malik et al., 2022) also proposed the same viewpoint.\n\nIt is reasonable to choose different hyperparameters for different models, because we can use experience to determine which models require a higher learning rate and which require a lower learning rate, which is feasible in real-world applications. So, our setting is: once a model and a TTA method are determined, their hyperparameters cannot be changed.\n\nIn our experiment, we have explored a learning rate range of {5e-5, 1e-4, 5e-4}, which can make all TTA methods in our experiment effective. However, in the table, Tent and EATA tend to choose 5e-5 because model collapse can be alleviated at low learning rates. Regarding hyperparameters beyond the learning rate, we have respectively referred to the hyperparameter settings of OIL and EATA.\n\nIt should be noted that Tent and EATA still perform well on other datasets under 5e-5, so there is no suspicion that we intentionally lowered the baseline. We report the EM values of Tent under different learning rates and datasets under xlmr-base.\n\n| LR   | NoiseQA-syn | NoiseQA-na | XQuAD | MLQA  | MRQA  | Avg.  |\n| ---- | ----------- | ---------- | ----- | ----- | ----- | ----- |\n| 5e-5 | 68.14       | 67.44      | 56.40 | 48.11 | 20.77 | 52.17 |\n| 1e-4 | 68.40       | 67.67      | 56.78 | 48.16 | 16.52 | 51.51 |\n| 5e-4 | 67.12       | 66.79      | 56.50 | 47.70 | 6.34  | 48.90 |\n\nIn addition, more precise selection of hyperparameters can also improve the performance of Anti-CF. We report the EM values of Anti-CF under xlmr-large and NoiseQA-syn with different hyperparameters.\n\n| LR/$\\alpha$ | 0.01  | 0.2   | 0.8   |\n| ----------- | ----- | ----- | ----- |\n| 5e-5        | 68.21 | 68.08 | 66.92 |\n| 1e-4        | 67.51 | 67.82 | 66.86 |\n| 5e-4        | 65.61 | 67.00 | 66.65 |\n\n\n\n\n\nReferences:\n\n- Niu et al., 2023, Towards stable test-time adaptation in dynamic wild world, ICLR\n- Malik et al., 2022, Parameter-free Online Test-time Adaptation, CVPR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed Anti-CF model sounds reasonable and would be applicable to other tasks.", "reasons_to_reject": "This paper pointed out the problem of Tent (Wang et al., 2021), which can lead to model collapse. However, whether Tent was tuned properly in Section 2.2 is still questionable because Tent can be sensitive to hyperparameters (e.g., learning rate, batch size, etc.) and the optimizer. The results in Table 1 also show that Tent performs reasonably well on other datasets. The authors are encouraged to discuss the proposed side block compared to existing PEFT methods (e.g., LoRA) to justify its novelty. ", "questions_to_authors": "- Lines 182-184 state that \"Therefore, the imbalanced distribution of test labels has led to TTA leaning too much towards the majority class during the update process\". Could you clarify the meaning of \"labels\" and \"the majority class\" here?\n\n\n- When computing the predicted probability for the adapted model, do the efficient side block's parameters require forward propagation? If so, lines 237-239 \"... as the adapted model so that we only need one forward propagation to obtain the two outputs simultaneously\" are incorrect.", "paper_topic_and_main_contributions": "This paper proposes Anti-Collapse Fast (Anti-CF) test-time adaptation (TTA) for extractive question answering. TTA methods (in particular TENT (Wang et al., 2021)) are not stable and may cause model collapse (i.e., producing almost all incorrect predictions at test time). The paper presents a regularization method that computes KL divergence between the predicted probability produced by the adapted model and the one made by the source model. In addition, only small parameters on the so-called side block are tuned to reduce the parameter size and inference time. The results on various QA datasets indicate that Anti-CF performs on par or better than other models regarding EM/F1 scores but is faster than all adaptive models.\n\nComments\n- In Section 2.2, the paper discusses why the performance of Tent (Wang et al., 2021) significantly drops when adapted to the NaturalQA datasets. However, this issue can also be caused by various factors. In Tent's paper, they used different optimizers, batch sizes, and learning rates for ImageNet and other datasets. I am unsure whether this paper had experimented with a sufficient range of hyperparameters before concluding that Tent leads to model collapse.\n\n- In Eq. (2), incorporating KL divergence between two different predicted probabilities is similar to that of X-MIXUP (the last term of Eq. (7) in Yang et al. (2021)). Although this paper computes the loss at test time, it is worth mentioning/citing it.\n\n- Eq. (4) can be thought of as a special case (i.e., test-time case) of X-MIXUP's loss (Eq. (8)), where L_task is replaced with L_s and the MSE term is omitted.  \n\n- The idea of the efficient side block is closely related to Parameter-Efficient Fine-Tuning (PEFT) (e.g., LoRA (Hu et al., 2021)). Discussing the pros/cons between the efficient side block and those existing techniques would make the paper more solid.\n\n- In Table 1, reporting inference time is helpful. It would be nicer if the paper also shows the total number of parameters for each model.\n\n- In Table 3, the learning rate (LR) choices are strange. Typically, the same LR does not work well for different pre-trained LMs (e.g., XLM-RoBERTa-base vs. XLM-RoBERTa-large). This would lead to suboptimal performance. \n\nReferences:\n- Wang et al., 2021, Tent: Fully Test-Time Adaptation by Entropy Minimization, ICLR. https://openreview.net/forum?id=uXl3bZLkr3c\n- Yang et al., 2022, Enhancing Cross-lingual Transfer by Manifold Mixup, ICLR. https://openreview.net/forum?id=OjPmfr9GkVv\n- Hu et al., 2022, LoRA: Low-Rank Adaptation of Large Language Models. ICLR. https://openreview.net/forum?id=nZeVKeeFYf9\n- Zheng et al., 2021, Consistency Regularization for Cross-Lingual Fine-Tuning, ACL. https://aclanthology.org/2021.acl-long.264/\n\n\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Hu et al., 2022, LoRA: Low-Rank Adaptation of Large Language Models. ICLR. https://openreview.net/forum?id=nZeVKeeFYf9"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new TTA method called Anti-CF to address the problem of performance degradation of pre-trained language models in distribution shift scenarios. The authors applied their methods to extractive question answering.

Reviewers agree that the approach is reasonable and that could even be applied to other NLP tasks. Some reviewers highlight the importance of the analysis leading to discover that the imbalance in label distribution in the test set of the question answering task is a major cause of model collapse in TTA method. Some reviewers also mention that the approach if efficient, and that the authors have designed a careful set of experiments to demonstrate that their proposed approach achieves stable performance improvements in various distribution shift scenarios.

The main concern raised by one reviewer is related to novelty, mainly because the authors could explain better how their proposed method differs and/or builds on existing approaches (e.g. LAME and PEFT techniques). Some of these clarifications were provided during the rebuttal period, showcasing that the authors are aware of these techniques and their relationship with their work. As such, they are strongly encouraged to include these comparisons and, in turn, adapt their novelty claims. Other concerns were related to comparisons with stronger models, which were presented during rebuttal, as well as details on hyperparameter settings (also added during rebuttal). Finally, there was some concern regarding using KL divergence and the limits it brings to the approach, which the authos have acknowledged is a limitation (it is stated in the corresponding section in the paper) and plan on tackling it in future work.
Decision: Accept-Main