Test-Time Self-Adaptive Small Language Models for Question Answering
Authors: Soyeong Jeong
Jinheon Baek
Sukmin Cho
Sung Ju Hwang
Jong C. Park
Venue: EMNLP 2023 Findings (2023)
Keywords: Language Models
Question Answering
Test-Time Adaption

Abstract: Recent instruction-finetuned large language models (LMs) have achieved notable performances in various tasks, such as question-answering (QA). However, despite their ability to memorize a vast amount of general knowledge across diverse tasks, they might be suboptimal on specific tasks due to their limited capacity to transfer and adapt knowledge to target tasks. Moreover, further finetuning LMs with labeled datasets is often infeasible due to their absence, but it is also questionable if we can transfer smaller LMs having limited knowledge only with unlabeled test data. In this work, we show and investigate the capabilities of smaller self-adaptive LMs, only with unlabeled test data. In particular, we first stochastically generate multiple answers, and then ensemble them while filtering out low-quality samples to mitigate noise from inaccurate labels. Our proposed self-adaption strategy demonstrates significant performance improvements on benchmark QA datasets with higher robustness across diverse prompts, enabling LMs to stay stable. Code is available at: https://github.com/starsuzi/T-SAS.

Test-Time Self-Adaptive Small Language Models for Question Answering
Soyeong Jeong'
Jinheon Back² Sukmin Cho Sung Ju Hwang Jong C. Park 1*
School of Computing' Graduate School of A1
Korea Advanced Institute of Science and Technology1.2
(starsuzi, baek, nelllpic, jona kr.
Abstract
(A] Document Augmmett-- Question
[B] Self- Ensemble
Huuge Paretti
Question
Who write the tong Can't Hep Fallin
Huga Pareti
Recent instruction-finetuned large language models (LMs) have achieved notable perfor- mances in various tasks, such question- answering (QA). However, despite their ability to memorize a vast amount of general knowl- edge across diverse tasks, they might be sub- optimal on specific tasks due to their limited capacity to transfer and adapt knowledge to tar- get tasks. Moreover, further finetuning LMs with labeled datasets is often infeasible due their absence, but it is also questionable if we can transfer smaller LMs having lim- ited knowledge only with unlabeled test data. In this work, we show and investigate the ca- pabilities of smaller self aelf-adaptive LMs, only with unlabeled test data. In particular, we first stochastically generate multiple answers, and then ensemble them while filtering out low- quality samples to mitigate noise from inaceu- rate labels. Our proposed self-adaption strat- egy demonstrates significant performance im- provements on benchmark QA datasets with higher robustness across diverse prompts, en- abling LMs to stay stable, Code is available at: https://gi http:/ggttthu..... cwm/s httm//sttsuui/-sta.
Love?
External Document
Huga Portii
Presey American arrger (19611 withen
Cant Faling Lowe song
LMs
Probs.
Preske
Filtering
Figure 1: Illustration of our proposed T-SAS that includes self-ensemble and filtering strategies for self-adapting LMs.
their knowledge to target tasks. One possible so- lution is to additionally fine-tune LMs, but this is often impractical in realistic scenarios where labeled datasets scarce. Furthermore, while large LMs with hundreds of billions of parameters may solve specific target tasks without fine-tuning, they are rarely accessible. Motivated by these chal- lenges, we focus on investigating the self-adaptive capabilities of smaller LMs during the test-time.
While several studies have shed light on the po- tential for self- iaf-iimprovemntt of LMs, they mainly focus augmenting the supervised labeled data with additional self-generated labels (He al., 2020; Chen al., 2023; Wang al., 2023b), which significantly differs from ours with a more chal- lenging setup that does not rely on labeled data. Note that there exists a recent work (Huang et al., 2022) that has shown the self-adaptive ability of large LMs with 540B parameters by further train- ing with their generated answers, focusing on the reasoning tasks. However, the utilization of large LMs requires substantial costs and restricted ac- cessibility. Therefore, our attention shifts towards smaller LMs, whose potential to be adapted to tar- get tasks with their limited capability of storing knowledge remains largely unexplored.
1 Introduction
Language models (LMs) have gained the ability to learn generalizable representations that are applica- ble to diverse tasks by being trained on massive text corpora with increased parameters (Brown al., 2020; Kojima al., 2022). Moreover, enhance the transferability unseen tasks, LMs are further fine-tuned on instructions that are verbalized from a vast amount of the supervised datasets, showing a remarkable zero-shot ability across a wide range of tasks (Wei et al., 2022a; Sanh et al., 2022). However, despite their ability to store a vast amount of general knowledge across diverse tasks, LMs show suboptimal performances specific downstream tasks when transferring and adapting
Note that the adaptation of smaller LMs to down- stream tasks brings forth new and critical chal- lenges. First, it seems evident that large LMs with extensive knowledge are adept at facilitating self- adaption. However, the question arises regarding the self-adaptive capability of smaller LMs, specif- ically when the supervised labeled datasets are ab- sent. Second, it may be suboptimal to use all the
Corresponding author

---

self-generated labels for training, as some of them could contain possibly incorrect information, lead- ing to significant performance degradation (Zhou et al., 2023). Third, unlike the large LMs, adapting smaller LMs to specific tasks would require the incorporation of external knowledge, due to their limited capabilities of storing specific knowledge.
ios, self-adaptive models have gained substantial attention (Wang et al., 2021a; Shu al., 2022; Veksler, 2023). In particular, some work has sug- gested to augment supervised labeled training data with self-generated labels (He et al., 2020; Chen et al., 2023; Wang et al., 2023b). In contrast, we assume more realistic test-time setup without labeled data. Note that there are recent studies self-consistent LMs. Specifically, Huang et al. (2022) demonstrated that large LMs can be further trained with the most consistent labels among mul- tiple self-generated labels. On the other hand, our focus lies on more practical and accessible smaller LMs with novel strategies of answer sampling and filtering. While some research focuses on self- consistent prompts by regularizing LMs to generate consistent outputs across different prompts (Zhou et al., 2022: Zeng and Gao, 2023; Wan et al., 2023), the focus different and orthogonal ours which proposes to self-adapt smaller LMs for specific tar- get tasks associated with external knowledge.
In this work, we aim at addressing the challenges of self-adaptive abilities in smaller LMs without ad- ditional labeled datasets, focusing on the QA tasks augmented with external knowledge. To this end, we first stochastically generate multiple answers according to a given unlabeled question with its related document. Then the most plausible answer is selected through majority voting, serving pseudo-label for training during test-time. Note, however, that we should be aware of the possi- bility of unreliable results from the self-ensemble. To mitigate this, we further propose to filter out potentially incorrect samples based on low agree- labels, as illustrated in Figure 1. We refer to our proposed method as Test-time ddaf-ve Small LMs (T-SAS). We validate our T-SAS on QA datasets with smaller LMs, on which T-SAS significantly improves the self-adaptive capabilities of smaller LMs.
ment among self-generaal
Self-adaption for Extractive QAs Several previ- ous studies explored the self-adaptive capabilities of the traditional pre-trained language models, but they mainly addressed classification problems un- der the extractive setting (Li et al., 2020; Shakeri et al., 2020; Banerjee et al., 2021; Wang et al., 2021b; Ye et al., 2022). However, we focus on the generative setting, which makes a large difference due to fundamentally different objectives. To be specific, in an extractive setting, self-adaptation is based on probabilities, while in generative set- ting, self-adaptation is done using generated text. In situations where filtering is further applied, fil- tering is based on probabilities in the extractive setting, whereas, it is done using the generated text in the generative setting. Furthermore, Li et al. (2020) and Shakeri et al. (2020) assume an unsu- pervised QA setting, where the context- questi--- answer triplets are not available, thus requiring an additional seery-generation module. Such a pair generation approach different and orthogonal ours, since we aim to enhance answer generation directly from the provided context and question.
Our contributions and findings are threefold:
We first explore the self-adaptive capability of the smaller LMs in realistic setting only with the unlabeled data during the test-time.
We ensure that the high quality of the self- generated labels is maintained by proposing a novel self-ensemble scheme with filtering.
We show that our T-SAS method achieves out- standing performance on the QA tasks.
2 Related Work
Language Models Pre-trained language mod- els have shown decent advancements in diverse tasks (Brown et al., 2020) and further improved by increasing the number of parameters to bil- lions (Touvron et al., 2023; Anil et al., 2023) and leveraging instruction-finetuninn
techniques with a vast amount of supervised labeled datasets (Wei al., 2022a; Sanh et al., 2022). Despite their recent successes (Wei et al., 2022b; Wang al., 2023a), however, we see that LMs still encounter difficul- ties in effectively addressing downstream tasks due to their limited task-specific adaptation.
Method
3.1 Preliminaries
Question Answering Let Dtrain be a labeled QA training set, where each instance consists of a ques- tion qi, a gold answer af, and its associated doc-
Self-adaptive LMS Due to the frequent occur-
rence of domain or data shift in weal-world scenar-

---

Table 1: Exact Match (EM) and Fl three QA benchmark datasets with varying sizes of FLAN.
Base (250M)
Large (780M)
XI. (3B)
Datasets
Methods
EM
Fl
EM
F1
EM
F1
Finetuned Training Set
67.47
75.28
73.41
80.79
74.36
81.72
Naive LM w/o Ext.
3.50
7.20
7.19
12.24
12.09
18.24
Naive LM
37.21
45.00
53.76
65.34
54.53
66.39
NQ
Self- Adff-ivee w/ Greody
32.75
40.69
56.78
67.69
59.34
71.04
Self Belf-Adaptive w/ Soft
39.53
49.24
54,29
65.98
58.17
71.03
Sclf-Adaptive LMSI
39.81
49.36
36.79
68.08
74.17
T-SAS (Ours)
41.70
50.22
63.90
74.20
63.96
75.29
Finetuned w/ Training Set
71.78
77.28
80.84
85.01
81,93
86.40
Naive LM Ext.
5.96
11.54
13.29
19.57
25.30
31.14
Naive LMM
$1.83
60.79
69.77
76.69
75.27
81.17
TQA
Self-Adaptive Greedy
52.95
60.98
69.05
75.45
76.94
82.72
Self-Adaptive w/ Soft
49.57
58.31
69.02
76.19
75.15
81.64
Adelt-Ad w/ LMSI
56.55
65.45
70.47
77.42
77.34
83,37
T-SAS (Ours)
60.67
67.93
74.46
80.60
78.38
84.01
Finetuned Training Set
69.94
81.72
74.26
75.58
86.55
Naive LM w/o Ext.
2,00
6.37
3,55
9.01
5.86
12.17
Naive LM
59.93
71.25
83,00
SQD
Sell-Adaptive w/ Grevdy
$7.20
68.93
64.62
76.77
73.03
84.03
Self-Adaptin w/ Soft
51.79
65.48
65.78
79.02
71.13
83.41
Self-Adaptive LMSI
60.42
72.57
69.19
80.87
73.42
84.39
T-SAS (Ours)
63.02
74.68
71.34
82.57
73.84
84.72
ument dj that contains af, as follows: Dtrain {qi, di, } with of E dz. Similarly, an unlabeled test set is defined as follows: Drest {(q, d.)). Assume that LM is language model parameterized with 0, which has been instruction-finetune on massive datasets. Then, given a pair of the question and its relevant document (4;, d;), LM generates an answer, as follows: a, = LM(d, : 0). Note that, in order to generate correct answer, i.e., af = a. it is beneficial to train LM with labeled training set by minimizing the loss (e.g., cross-entropy) be- tween correct answer at and model prediction ā; follows: L(a,*, āz)- Then, after training phase, LM more likely to correctly predict of on the unlabeled test set Diest of the target task.
Stochastic Self-Ensemble Note that relying on single o;" generated by LM, which is trained on general domains, may result in inaccurate predic- tions when adapted to the target task, there ists possibility of incorrectly self-generated a; To mitigate this, we propose to make LM generate multiple answers {amي}ー= with diverse points view. Note that, while existing work (Huang et al., 2022; Wang et al., 2023a) proposed to use Top-k or nucleus sampling (Fan et al., 2018; Holtzman et al., 2020) when generating (05,) ) for reason- ing tasks, their diversities might be limited due to answer sampling based single representa- tion. Instead, we propose leverage multiple rep- resentations generated through Monte-Carlo (MC) dropout (Gal and Ghahramani, 2016) by randomly masking weights on LM during test-time, as follows:
Self-Adaptive LM While recent LM is capable of answering questions, directly using to the target task may yield suboptimal results, thus requiring transfer learning or adaption. In order to do so, our idea is to maximally leverage the unlabeled test set Dzest that we have in hand for the target task. In particular, we have Diest and given that, possi- ble solution is to train LM with its self-gend eeff-generated pseudo label, ,." In other words, LM can be further trained on Dreat_sel] self (qs, di, a, where ai is generated from the unlabeled test sample (g:, de) with LM, for self-adaption to target tasks.
{a75);=1 = [LMM-M qe M) (1)
where M is a distribution of mask weights and M a sampled mask weight. Once we have generated multiple answers (aT., )= our next objective is to assign one pseudo label a from the set using a majority voting strategy, which selects ā;" with the highest number of occurrences among {ai,; J After acquiring the self-generated label a," for the unlabeled Diest, we now aim training LM the
D Diest self with following loss term: L(a a.).
3.2 Test-time Self-Adaptive Small LMs
Filtering However, in contrast Huang al. (2022) and Wang et al. (2023a) who use all sam- ples and their associated pscudo labels ai for train- ing, relying entirely on them can largely prob-
We describe our test-time self-adaptive LMs (T- SAS) with proposed strategies for effectively gen- erating and utilizing pseudo labels during test-time.

---

Datasets
Methods
EM
Methods
Large (780M)
XL (3B)
Naïve
71.80
80.31
SciQ
T-SAS (Ours)
73.40
81.30
T-SAS (Ours)
74.20
75.29
Naïve LM
51.69
72.93
w/o Stochastic
67.67
74.10
cpgQA
T-SAS (Ours)
53.42
74.30
w/o Filtering
68.25
73.24
Naïve LM T-SAS (Ours)
64.55
76.97
TyDiQA
67.95
79.31
Naive LM
65.34
66.39
Table 2: Results additional QA datasets with FLAN-T5- XI. SeiQ, epgQA, and TyDiQA (only using the English data).
Table Ablation studies the NQ dataset with FLAN-TS- Large and FLAN-TS-XL. based models.
lematic since LM lacks specific training make valuable predictions particular tasks. There- fore, motivated by the fact that the substantial per- formance improvements of LMs are primarily at- tributed to their training on high-quality data (Zhou et al., 2023), we further propose an automatic fil- tering strategy to identify and exclude samples, (q:; d4, az:") a}") C Diest_self f. that are likely to have incorrect a;". We determine this by removing sam- ples labeled with a;" that have vote count propor- tionally lower than certain threshold.
4.2 Results
Here, we show the overall performance T-SAS. Please see Appendix B for more results.
Main Results As Table 1 shows, T-SAS signif- icantly outperforms all baselines of varying sizes, particularly with FLAN-Large model. Note that Naïve LMs show substantially lower performance than the supervision finetuned LMs. This corrob- orates our hypothesis that LMs are not fully opti- mized for target tasks. Also, by integrating external knowledge, the performance of all models, espe- cially smaller ones, is largely improved. Furthermore, ensembling multiple self- generated predictions significantly enhances performance, compared to baselines with Greedy decoding. This indicates that T-SAS, considering diverse points of view, reduces the likelihood of encountering performance-degrading scenarios caused by relying single prediction.
4 Experiments
4.1
Experimental Setups
this subsection, we describe experimental setups. Further details are shown in Appendix A.
Datasets We use three QA datasets, augmented with external documents from Wikipedia, pre- processed by Karpukhin et al. (2020): 1) Natu- ral Questions (NQ) (Kwiatkowski al., 2019), TriviaQA (TQA) (Joshi et al., 2017), and 3) SQuAD v1.1 (SQD) (Rajpurkar et al., 2016).
However, when compared to the baselines with multiple self-generated predictions (i.e., Self- Adaptive w/ Soft and w/ LMSI), the results indicate that a filtering strategy is required. Solely relying on the final result of the self-ensemble should be avoided with smaller LMs, contrasting the observa- tion by Huang et al. (2022) with large LMs (540B) on the reasoning task. Interestingly, the model trained soft labels, which must leverage all the predictions, shows even lower performance than the Naïve LMs, emphasizing the negative impact of training on inaccurate self-generated labels.
Baselines and Our Model We compare our T- SAS against other baselines using unlabeled test data. We use the FLAN (Chung et al.. 2022) and the same prompt aeross all models. 1) Finetuned w/ Training Set is an indicator model, which is finetuned on the labeled training set. 2) Naïve LM w/o Ext. a naive baseline without external knowl- edge and self-adaptive training. 3) Naïve LM is a baseline without self-adaptive training, but in- corporates external knowledge. 4) Self-Adaptive w/ Greedy is trained on the gelf-generated pseudo- labels via Greedy decoding. 5) Self-Adaptive w/ Soft is trained the result of soft voting. 6) Self- Adaptive w/ LMSI trained on the result of ma- jority voting, using Top-k sampling (Huang et al., 2022). 7) T-SAS (Ours) is ours, which incorpo- rates both self-ensembling and filtering strategies.
Comparing the performance with large LMs is outside our scope, since our work targets at inves- tigating the effectiveness of smaller LMs on self- adaption. Note that large LMs and smaller LMs are not directly comparable to each other, due to their discrepancy in capacity. However, report the performance of the large zero-shot LM as indicator in Figure Surprisingly, our T-SAS significantly outperforms a much larger zero-shot LM, which further signifies the effectiveness of the proposed adaptive strategies for smaller LMs.
additionally

---

85
85
ga2
75
80
80
g=7
75
Average
75
14
71
19
Ny Greedy
LM
69
7-AS w/ MC.
LM GPT3
70
1-545 w Top-k
MSI
60
ESAS (Durs)
SAS 10urs)
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
NQ
TQA
SQD
Self-Adaptive LMs
Filter Threshold
Figure 2: Results on three QA datasets with FLAN-T5-XL based models and GPT-3 (tex--)avinc-00033
Robustness to the diverse prompts, the TQA using the FLAN- T5-XL models with 11 different prompts.
Figure 3:
Figure 4: Comparison of MC drop and Top-k sampling with varying thresholds NQ with FLAN-TS-Large.
Robustness Diverse Prompts Recent LMs have been observed be prompt-sensitive (Cho al., 2023; Ishibashi al., 2023), which undesir- able as consistent outputs are expected across sim- ilar prompts, particularly real-world scenarios with diverse users. As shown in Figure 3, T-SAS shows substantial robustness to diverse prompts, which are attributed to the proposed stochastic gen- eration and filtering strategies. In other words, T- SAS effectively mitigates the impact of inaccurate predictions by filtering them from multiple perspec- tives, even for specific prompts.
Stochastic Generation Strategy with Filtering We compare MC dropout and Top-k sampling with varying filtering thresholds. As shown in Figure 4, both strategies benefit from a filtering strategy by mitigating noises introduced during stochastic gen- eration processes. Moreover, MC dropout consis- tently outperforms Top-k sampling, which can be attributed to its higher lexical diversity (0.24 vs. 0.14). These findings suggest that diversity, com- bined with a filtering strategy, allows LMs to effec- tively identify and remove low-quality outputs with higher variances, consequently resulting overall performance improvement.
Effectiveness on Specific Domains In addition to evaluating T-SAS in general domains, we further conduct experiments specific domains, includ- ing the science domain, SciQ (Welbl et al., 2017), and the clinical domain, cpgQA (Mahbub al., 2023), to assess its domain adaptability. Table shows consistent improvements from T-SAS, high- lighting its ability to capture and enhance transfer- ability across domain shifts. These findings indi- cate the applicability of T-SAS in domains where high-quality labeled data is scarce.
Ablation Studies In order to how each of the stochastic generation and filtering strategies contributes the overall performance, we provide the ablation studies with two variants of T-SAS with Large and XL. sizes. To be specific, in T-SAS w/o Stochastic, low-quality samples are filtered out based on the generation probability of single prediction, and in T-SAS w/o Filtering, the major- ity voting results are directly used as pseudo-labels without applying our proposed filtering strategy. As shown in Table 3, both of the proposed strategies positively contribute the overall performance, by substantially improving the performance of Naive LMs both model sizes. Furthermore, these strate- gies indicate that they are in a complementary re- lationship, suggesting that they work together to enhance overall performance.
Effectiveness on Unseen Datasets Recent instruction-finetune language models have been extensively trained on various QA datasets, making it challenging to evaluate them on new datasets. Therefore, in Table 2, further show clear im- provements even on the unseen datasets, cpgQA and TyDiQA (Clark al., 2020). Also, it is worth noting that the performance improvement achieved by applying self-adaptation to the already trained data not our weakness, but rather a strength. To be more specific, even though an LM has been trained comprehensively on diverse datasets in-
5 Conclusion
In this work, we investigated and improved the self-adaptive capabilities of the recent smaller LMs only using unlabeled test data, on the QA task. Specifically, our proposed method involves self- ensembling the stochastically generated labels and filtering strategy remove possibly incorrect la- bels, thereby enabling training with high-quality self-generated labels. The experimental results and analyses indicate that our method significantly im- proves QA performances during test-time.
cluding the target dataset, the LM remains as the general-purpose model and is not tailored to the specific target dataset. However, by using our pro- posed T-SAS on the target dataset, the model can further achieve improved performance thanks to its self-adaptation capabilities over the target dataset.

---

Limitations
Yuanzhong Xu, Yujing Zhang, Gustavo Herándezz Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury. Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng. Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz. Nan Du Ethan Dyer, Vladimir Feinberg, Fangxi- aoyu Feng. Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. 2023. Palm 2 technical report. arXiv preprint 0rxiv23305.0040
While we show clear advantages of using our SAS to address the realistic challenges of the recent LMs regarding their self-adaptive capabilities, is important to acknowledge that our validation was conducted under the assumption of having gold external documents containing the answers. How- ever, in real-world scenarios, obtaining such gold documents may not be feasible, necessitating the incorporation of additional retrieval modules to retrieve query-relevant information. Although the integration of retrieval modules is beyond the scope of our current work, our exploration of the potential benefits of an external noowlede-auggmntee set- ting for self-adaptive smaller LMs opens up fruitful avenues for future research. We also believe that in- vestigating the integration of retrieval modules with T-SAS to further enhance the practical applicabil- ity of self-adaptive LMs in real-world applications holds a significant value.
Pratyay Banerjee, Tejas Gokhale, and Chitta Baral. 2021. Sulf--upe test-time learning for reading com- prehension. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIIT 2021, Online, June 6-11, 2021, pages 1200-1211, Association for Computa- tional Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Ethics Statement
The experimental results confirm the effectiveness of T-SAS in adapting unlabeled test-time data. However, is important to consider and address the potential bias of the underlying LM when utilizing its extensive knowledge for training self-adaptive models. We suggest that implementing specific fil- tering strategies targeted at mitigating these biases can be potential solution to ensure the safety and reliability of self-adaptive models.
Wei-Lin Chen, An-Zi Yen, Hen-Hsen Huang, Cheng- Kuang Wu, and Hsin-Hsi Chen. 2023. ZARA: im- proving few-shot self-rationalizatiinn for small lan- guage models. arXiv preprint arXiv.2305.03355.
Acknowledgements
This work was supported by Institute for Informa- tion and communications Technology Promotion (IIPP) grant funded by the Korea government (No. 2018-0-00582, Prediction and augmentation of the credibility distribution via linguistic analysis and automated evidence document collection) and Ba- sie Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2023-00275747),
Sukmin Cho. Soyeong Jeong. Jeongyeon Seo, and Jong C. Park. 2023. Discrete prompt optimization via constrained generation for zero-shot re-ranker. In Findings of the Association for Computational Lin- guistics: ACL 2023. Association for Computational Linguisties.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun. Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finetueed arXiv Preprint 20.
References
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstens Gau- rav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao,
finetuned language models.
Jonathan H. Clark, Jennimaria Palomaki, Vitaly Niko- laev, Eunsol Choi, Dan Garrette, Michael Collins,

---

and Tom Kwiatkowski. 2020. Tydi QA: bench- mark for iiformationnseeking question answering in typologically diverse languages. Trans. Assoc. Com- put. Linguistics, 8:454-470.
and Wen-tau Yih. 2020. Dense passage retrieval for open- open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781 - Associa- tion for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. pages -89-898, 898. Association for Computational Lin- guistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In NeurlPS.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452- 466.
Yarin Gal and Zoubin Ghahramani, 2016. Dropout bayesian approximation: Representing model uncertainty in deep learning. In Pruceedings of the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050~1059, JMLR.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc' Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Zhongli Li. Wenhui Wang, Li Dong, Furu Wei, and Ke Xu. 2020. Harvesting and refining question- answer pairs for unsupervised QA. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6719-6728, Association for Com- putational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenKeview.net.
Maria Mahbub, Edmon Begoli, Susana B. Martins, Alina Peluso, Suzanne Tamang, and Gregory D. Pe- terson. 2023. cpgqa: A benchmark dataset for ma- chine reading comprehension tasks on clinical prac- tice guidelines and a case study using transfer learn- ing, IEEÉ Access, 11:3691-3705.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguistics.
Jiaxin Huang, Shixiang Shane Gu. Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.1 16700.
Yoichi Ishibashi, Danushka Bollegala, Katsuhito Su- doh, and Satoshi Nakamura. 2023. Evaluating the robustness of discrete prompts. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Creatia, May 2-6, 2023, pages 2365-2376. Association for Computational Linguis- tics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey. M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Mattco Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao. Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer: 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pages 1601-1611. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen,
Siamak Shakeri, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh

---

Nallapati, and Bing Xiang. 2020. End-to-end syn- thetic data generation for domain adaptation of ques- tion answering systems. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2020, Online, November 16-20, 2020, pages 5445-5460. Association for Com- putational Linguistics.
International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25- 25-29, 2022. OpenReview.net.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022b. Chain-of-thought prompt- ing elicits reasoning in large language models. In NewrIPS.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. 2022. Test-time prompt tuning for zero-shot general- ization in vision-language models. In NeurIPS.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsoureing multiple choice science questions. In Proceedings of the 3rd Workshop generated Text, NUT@ EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 94-106. Associ- ation for Computational Linguistics.
Noisy User-
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 23X2:23322.3971 13971.
Hai Ye, Yuyang Ding, Juntao Li, and Hwee Tou Ng. 2022. Robust question answering against distribution shifts with test-time adaption: An empirical study, In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6179 6192. Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Olga Veksler. 2023. Test time adaptation with regular- ized loss for weakly supervised salient object detec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7360-7369
Fengzhu Zeng and Wei Gao. 2023. Prompt to be sistent is better than self-consistent? few-shot and zero-shot fact verification with pre-trained language models. In Findings of the Association for Com- putational Linguistics: ACL 2023. Association for Computational Linguistics.
Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Ö. Arik, and Tomas Pfister. 2023. Better zero-shot reason- ing with self-adaptive prompting. arXiv preprint arXiv.2305 14106.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. 2021a. Tent: Fully test-time adaptation by entropy minimization. 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OeenReview.net.
Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Prompt con- sistency for zero-shot task generalization. ings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11. 2022, pages 2613-2626. Association for Computational Linguistics.
Find-
Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Gra- ham Neubig. 2021b. Efficient test time adapter en- sembling for low-resource language varieties. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November. 2021, pages 730-737. Association for Computational Linguistics.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: less is more for alignment. arXiv preprint a00. 7 17006, abs/2305.1 11206.1206..
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2023a. Self- consistency improves chain of thought reasoning in language models. In 11th International Conference on Learning Representations, ICLR 2023. OpenRe- view.net.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generater generated instructions. In Asso- ciation for Computational Linguisties: ACL 2023. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M. Dai, and Quoe V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth

---

76
Sizes
Methods
EM
F1
Naive LM
23.98
31.92
Small (80M)
74
T-SAS (Ours)
31.28
40.05
73-
Naïve LM
60.73
70.93
XXL (IIB)
T-SAS (Ours)
66.52
76.58
72
T TSAS via Stocasatio
71
Table 4: Results of FLAN-Sminn and FLAN-XXL, on NQ.
Number of Dropout masks
Figure 5: F1 scores with varying numbers of dropout masks.
A Experimental Setups
in decreasing the possibility of selecting inaccurate self- self-generated answers.
Datasets We use three QA datasets, augmented with external documents from Wikipedia, prepro- cessed by Karpukhin et al. (2020). 1) Natural Questions (NQ) (Kwiatkowski et al., 2019) is com- posed of questions from the Google Search engine. 2) TriviaQA (TQA) (Joshi et al., 2017) is collected from the Web, which consists of trivia questions. 3) SQuAD v1.1 (SQD) (Rajpurkar et al., 2016) constructed by the annotators by writing questions after reading passages.
Effectiveness on Diverse Model Sizes In addi- tion to the performance on three sizes shown in Table I, we additionally conduct experiments for the FLAN-Small (80M) and FLAN-XXL (11B) models. The results in Table 4 demonstrate that our T-SAS consistently enhances performances the much smaller or larger model sizes.
Effectiveness on TO-3B In addition to the FLAN series, conduct experiments another LM, TO (Sanh et al., 2022) with 3B size. As shown in Table 5, our T-SAS consistently improves the performance on three QA datasets, which indicates the applicability of T-SAS on diverse LMs.
Metrics We evaluate models with Exact Match (EM) and Fi-score (FI), following the standard protocol for the QA task (Rajpurkar et al., 2016; Karpukhin et al., 2020).
Implementation Details For a fair comparison, we compare the models using the instruction- finetuned FLAN-T5 (Chung et al., 2022) model with three different sizes, Base (250M), Large (780M), and XL (3B) with the same prompt: 'Read this and answer the questioon/contex///qeeeeee For the models larger than 3B, we trained them adopting a low-rank adaptation (LoRA) method (Hu et al., 2022), For hyperparameters, we set the training epoch as 5 for all the self-adaptive models and 1 for the indicator model. Also, we set the number of stochastically generated predictions 15 and set the filtering threshold as 0.7.
Effectiveness of Augmented External Document We have observed significant performance improve- ment with the augmented external documents all models as shown in Table Here, we fur- ther analyze the importance of augmenting external documents, especially for the models that require training. As shown in Table 6, the overall per- formance without external knowledge is largely devastated for all models. Interestingly, the perfor- mance degradation for an indicator model trained on the supervised training dataset is remarkable. These findings corroborate our proposed challenge that external knowledge is necessarily required for training self self-adaptive and especially smaller LMs, whose capabilities of storing specific knowledge are highly limited.
B Additional Experimental Results
Dropout Mask Variations To investigate the im- pact of the number of masks in MC dropout on performance, we conduct experiments with vary- ing mask numbers. Figure 5 illustrates that in- creasing the number of masks leads to improved performance, but stabilized after reaching tain number of masks. Note that the stochastically perturbed models, with multiple dropout masks, offer more diverse perspectives compared to the ablated model without stochastic generation or the model with only one dropout mask, thus resulting
Case Study We conduct a case study, mainly comparing our T-SAS against adaptive base- line model with LMSI, in Table 7. The first exam- ple shows the robustness of our T-SAS approach in addressing prompt-sensitive situations. where LM exhibits significantly degraded performance for a specific prompt, 'Read the following article and answer the question. Article: |) Question: (). While both models stochastically generate in-

---

Datasets
Methods
EM
F1
Naïve LM
42.08
54.78
NQ
T-SAS (Ours)
50.17
61.61
Naïve LM
62.43
70.23
TQA
T-SAS (Ours)
68.70
75.49
Naïve LM
50.15
62.73
SQuAD
T-SAS (Ours)
56.86
69.10
Table Results with TO-3B on QA datasets.
Methods
EM
F1
Naïve LM
54.53
66.39
Naïve LM w/o Ext.
12.09
18.24
Finetuned. w/o Ext.
7.16
12.79
T-SAS (Ours) w/o Ext.
12.29
18.43
Table 6: Results without using external documents, on NQ.
accurate predictions that are totally unrelated to the question or document, and subsequently vote based these erroneous labels, our T-SAS pre- vents further training with these unreliable predic- tions by employing filtering strategy. This demon- strates the adaptability of our T-SAS real-world scenarios with diverse user prompts. Morcover, the second example highlights the effectiveness of a filtering strategy in addressing diverse answers generated by our stochastic eeff-eeneration strat- egy. Note that MC dropout produces answers with higher lexical diversity than Top-k sampling, which enables the removal of low-quality outputs with higher variance. both cases, the removal of low- quality labels is crucial for the self-adaptive LMs, and our proposed strategies have demonstrated the effectiveness achieving this goal.

---

Table Examples from two self-adaptive LMS without using labeled data, during test-time.
Case # 1: Our T-SAS exhibits robustness in handling diverse prompts.
Document: ... This was necessary because EMI owned another record label called Columbia, which operated every market except North America, Spain and Japan. CBS sold the record company 1988 to (Answer) Sony, 1991, the CBS label olticially renamed Columbia Records and the company was renamed Sony Music Entertuinment.
Question: Which Japanese compuny bought CBS records in 1988?
Prompt: Read the following article and answer the question. Article: (context) Question: {question]
Self-Adaptive w/ LMS1
T-SAS (Ours)
Self-Generated Answers: 'iv!, (C), 'a)., '(4)., '(a). 'iv.' 'iil'. Id],', 'liv'", 'sony". "(vv), '(i)). 'B)., 'iv.. "sony"
Self-Generated Answers: [[v]', *b)., '(4),, '6v)., '[i]', 'd.' 'lil'. 'fil', 'sony'. 'sony', 'fil', 'fiv]". 'Bil". "fil', 'iv)'
3
Self-Ensemble Result: 'iv.'
Self-Ensemble Result: Tip < Threshold)
15
Final Prediction: "[iv]'
Final Prediction: 'sony"
Case 2: Our T-SAS shows effectively combines diverse answer generation and filtering strategies.
Howard the 2007 film Spider-Man and by (Answer) Emma Stone the 2012 rebuot film The Amazing Spider-Man and sequel The Amazing Spider-Man Created writer Stan Lee and artist Steve Ditko, Gwen Stacy first appeared in The Amazing Spider-Man 31 (December 1965). In her initial appearances, Peter Parker met Gwen while both were studying undergraduates Empire State University, but with Aunt May the hospital, Peter was troubled and ignored her advunces. ...
Ducument:
Question: Real name of Gwen Stacy in Amazing Spiderman
Prompt: Read this and answer the question [context] [question]
Self-Adaptive w/ LMS1
T-SAS (Ours)
Self-Gemerated Answers: 'peter purker's intellect', 'pe- ter parker's scientific rigor and', 'peter parker's 's interest in science'. 'peter parker'. 'peter'. "peter parker's scientific approach solving mysteries', 'peter parker's intelligence', "peter parker', 'peter parker', pptee parker', 'peter's 's intel- lect and', 'peter parker', 'peter parker's 's scientific prowess". "peter parker's 's intellect and sass', 'peter parker's under- standing of science"
Self-Generated Answers:
'howard'. "gwen stacy'. 'howard', 'howard', 'emma stone', 'howard', 'gwen stacy', 'emma stone' 'peter parker's wit and intelligence'. 'howard'; ; 'howard', 'gwen stacy', 'emma stone', 'emma stone', 'emma stone"
Self-Ennemblee Result: 'peter parker'
Self-Ensemble Result: -heward
15. <
Threshold)
Final Prediction: 'peter parker'
Final Prediction: 'emma stone'


Reviews:
[{"review_id": "vpv3PSCSZG", "reviewer": "Reviewer_ruV4", "summary": "", "questions": "For the experiments, have you run multiple times for each baseline? ", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Weakness 1-1)** The novelty seems to be limited.\n\nWe strongly believe that our work is novel and that its contributions are substantial, which we summarize as follows:\n\n* First of all, we introduce **a novel and realistic problem** by exploring the self-adaptive capabilities of the **recently proposed smaller LMs** (See Lines 37-47), specifically in a realistic test-time setting where the labeled data is unavailable. \n* Then, we address this problem with two novel and effective strategies (Section 3.2): self-ensemble strategy with stochastic generation via majority voting (Lines 180-207) and filtering strategy which removes samples with low-quality according to their agreement (Lines 208-222).\n* We experimentally showed that our proposed T-SAS significantly improves the performance and that each of the proposed self-ensemble and filtering strategies is effective (Table 3). Furthermore, our T-SAS is substantially robust to diverse prompts (Lines 281-292 and Lines 676-689), which effectively addresses an undesirable but realistic prompt-sensitive problem in recent LMs.\n\n\n>**Weakness 1-2)** Some related work exists [A1-A4].\n\nWe thank you for your suggestion on the related work. However, the suggested work and ours are **largely different**, since the previous studies focus on solving classification problems under the **extractive setting** mainly using BERT series models. In contrast, ours focuses on the **generative setting**. Please note that as the objectives of extractive and generative settings are different, the test-time adaption strategies are fundamentally different. To be specific, in an extractive setting, self-adaptation is based on probabilities, while in a generative setting, self-adaptation is done using generated text. In situations where filtering is further applied, in the extractive setting, filtering is based on probabilities, whereas in the generative setting, it is done using the generated text (We discovered that it is less effective to filter out samples based on a single generation probability in Table 3). This makes it difficult to directly adopt the previous test-time adaptive strategies used for the extractive models to the generative setting. Therefore, while several previous work has investigated test-time adaption approaches to solving classification tasks under the extractive setting, it has been questionable and largely underexplored how we can self-adopt the small LMs over the generative setting.\n\nAlso, please note that the difference in extractive and generative settings makes it difficult to directly compare our generative T-SAS against the extractive models through experiments.\n\nWe concretely further discuss the differences between each of the suggested papers and ours as follows: \n\n[A1] Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study\n* This work targets an **Extractive** task (QA), which is largely different from our generative setting, as aforementioned above.\n* Also, since this work aims to correctly classify the tokens in an extractive setting, the filtering approach in this work is based on a single token probability. Please note that such a filtering approach applied to our generative setting is already presented as a baseline in Table 3 (w/o Stochastic), which indicates the effectiveness of our stochastic filtering strategy.\n\nAlthough conducting a direct apples-to-apples comparison with this work is infeasible due to the distinct extractive/generative natures of two different settings, we have further compared our T-SAS against the suggested model, OIL, along with other baseline models PL and Tent in this work. Specifically, we tried our best to make a fair assessment by 1) comparing models with similar parameter sizes (FLAN-T5-base (250M) vs XLM-RoBERTa-base (270M)) and 2) finetuning all the models with the SQuAD dataset before the test-time adaption. As shown in Table A.1, our T-SAS significantly outperforms the extractive baseline models.\n\nTable A.1: Comparison between the extractive models from [A1] and our T-SAS on the NQ.\n|Methods|EM|F1|\n|---|---|---|\n|Naïve LM|31.72|40.84|\n|Tent|43.14|56.11|\n|Tent+xTune|44.98|59.67|\n|PL|43.53|56.32|\n|PL+xTune|44.89|59.53|\n|OIL|43.00|56.44|\n|OIL+xTune|45.30|60.16|\n|T-SAS (Ours)|**58.27**|**69.29**|\n\n[A2] Efficient test time adapter ensembling for low-resource language varieties. \n* This work targets at an **Extractive** task (cross-lingual NER and POS tagging), which is largely different from our generative QA setting, as aforementioned above.\n* This work aims at ensembling multiple models for diverse languages to robustly classify tokens in an unseen language. In contrast, instead of ensembling multiple different models, our self-ensemble strategy considers self-generated outputs from a single model.\n\n[A3] PADA: Example-based prompt learning for on-the-fly adaptation to unseen domains.\n* This work does not target at a **Test-Time Adaption** task, but rather aims at generating an appropriate prompt for a test sample, without further training the model during the test-time.\n* This work targets at an **Extractive** task (text classification and sequence tagging), which is largely different from our generative setting, as aforementioned above.\n\n[A4] Self-supervised test-time learning for reading comprehension.\n* This work targets an **Extractive** task (QA), which is largely different from our generative setting, as aforementioned above. \n* The direction of the test-time adaption is different and rather orthogonal to ours. Specifically, our test-time adaption focuses on effectively addressing the given test-time input questions by generating synthetic answers. However, the previous work aims at generating synthetic question-answer pairs based solely on the provided test-time context, without taking into account the actual test-time input questions. As a result, direct adaptation to the test-time input questions may not be feasible in this setting, given that the adapted model does not have visibility into the actual test-time input questions.\n* In addition, this work does not consider the quality of the synthetically generated data. In contrast, we propose self-ensemble and filtering strategies to ensure training models with high-quality data.\n\n>**Question 1)** For the experiments, have you run multiple times for each baseline?\n\nYes, we have run each baseline multiple times, by using different prompts, as shown in Figure 2. Specifically, we run each model with 11 different prompts and Figure 2 illustrates the average F1 scores with corresponding standard deviation. Figure 2 indicates that our model not only consistently outperforms other baselines, but also is highly robust to different prompts (Lines 286-292). \nAdditionally, we further report the average and standard deviation on the NQ over three different runs in Table A.2, where we use the same prompt with different seeds. Here, we observe that our T-SAS shows clear performance improvements over the baselines and is statistically significant at p < 0.01.\n\nTable A.2: Three runs on the NQ using the FLAN-T5-XL models.\n|Methods|EM|F1|\n|---|---|---|\n|Self-Adaptive w/ Greedy | 59.31 &pm; 0.41 | 70.86 &pm; 0.35 |\n|Self-Adaptive w/ Soft | 57.85 &pm; 0.35 | 70.56 &pm; 0.44|\n|Self-Adaptive w/ LMSI | 62.11 &pm; 0.68 | 73.74 &pm; 0.40|\n|T-SAS (Ours) | **63.91 &pm; 0.09**|**75.20 &pm; 0.12**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The setting is practical and reasonable. \nTransductive learning should be very useful in practice since it can explore the information of the test data, so it can find out more correct labels on the test data, though it is expensive to refine the model. \n\n2. The method is sound with enough experimental results, given it is a short paper. \n\n", "reasons_to_reject": "Though the setting is practical and reasonable, the novelty seems to be limited. \nTest-time fine-tuning or adaptation has been studied for NLP tasks in the past few years, so the setting studied in this work is not new.\n\nRelated work: \n[1] Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study\n[2] Efficient test time adapter ensembling for low-resource language varieties. \n[3] PADA: Example-based prompt learning for on-thefly adaptation to unseen domains. \n[4] Self-supervised test-time learning for reading comprehension. \n\n", "questions_to_authors": "For the experiments, have you run multiple times for each baseline? ", "paper_topic_and_main_contributions": "This paper studies test-time fine-tuning with unlabeled test data. The initial model is the one tuned to follow instructions. In this paper, it uses FLAN-series models. \n\nThe initial model will be fined tuned with unlabeled test data, which is a transductive-learning setup. In the method, the initial model has generate pseudo labels on the test data, then the pseudo labels will be used to fine-tune the model. \n\nThe paper proposes a method which samples several pseudo labels then filter the noisy labels, to make sure the training more robust.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study\n[2] Efficient test time adapter ensembling for low-resource language varieties. \n[3] PADA: Example-based prompt learning for on-thefly adaptation to unseen domains. \n[4] Self-supervised test-time learning for reading comprehension. "}}, {"review_id": "GBQgxVh9No", "reviewer": "Reviewer_FMJN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Weakness 1)** Directly selecting the pseudo label via majority voting strategy, i.e., the highest number of occurrences, is unreliable, since the generated results of smaller LMs are less reliable compared to large LMs. The part requires more discussion.\n\n We do not directly select the pseudo-label via majority voting. Instead, as you mentioned, we also pointed out that the generated data from smaller LMs can be unreliable (Lines 89-94), and further proposed to filter out these potentially incorrect samples having a low agreement (Lines 208-222). Also, we already provided analysis and ablation studies in Figure 3 and Table 3, showing that our proposed filtering strategy effectively abstains from training with low-quality pseudo labels, thus contributing to significantly improved performance.\n\nTo be more clear, we additionally provide the effectiveness of our proposed filtering strategy for the smaller LMs with different sizes. As shown in Table B.1, all models show significant performance enhancements, and the performance gains for the smaller models are more remarkable. This highlights the importance of filtering out unreliable self-generated samples, especially for the smaller LMs. We will include them in the revision.\n\nTable B.1: Effectiveness of our filtering strategy with diverse FLAN-T5-series models on the NQ.\n|Sizes|Methods|F1|Increasement|\n|---|---|---|---|\n| Base (250M) | T-SAS w/o Filtering  | 45.00 | |\n| Base (250M) | T-SAS   | 50.22 | **+11.60%** |\n| Large (780M) | T-SAS w/o Filtering | 68.25 |  |\n| Large (780M) | T-SAS |  74.20 | **+8.72%** |\n| XL (3B) | T-SAS w/o Filtering | 73.24 |  |\n| XL (3B) | T-SAS |   75.29 | **+2.80%** |\n\n>**Weakness 2)** To show the effectiveness of the proposed method, it also needs to compare the performance with those large LMs (zero-shot).\n\nComparing the performance with large LMs is outside of our scope, since our work targets at investigating the effectiveness of smaller LMs on self-adaption (Lines 42-47). Please note that large LMs and smaller LMs are not directly comparable, considering their discrepancy in capacity.\nHowever, as requested, we additionally report the performance of the large zero-shot LM (text-davinci-003) as an indicator in Table B.2. Surprisingly, our T-SAS significantly outperforms a much larger zero-shot LM, which further signifies the effectiveness of the proposed self-adaptive strategies for smaller LMs.\n\nTable B.2: Results on three QA datasets with FLAN-T5-XL based models and GPT-3 (text-davinci-003).\n|Datasets|Methods|EM|F1|\n|---|---|---|---|\n| NQ | Naïve LM | 54.53 | 66.39 |\n| NQ | GPT-3 | 55.53 | 70.57 |\n| NQ | T-SAS (Ours) |  **63.96** | **75.29** |\n| TQA | Naïve LM |  75.27 |  81.17 |\n| TQA | GPT-3 | 71.75 | 81.57 |\n| TQA | T-SAS (Ours) | **78.38** |  **84.01** |\n| SQuAD | Naïve LM |  71.66 |  83.00 |\n| SQuAD | GPT-3 | 57.16 | 75.16 |\n| SQuAD | T-SAS (Ours) | **73.84** |  **84.72** |\n\n>**Weakness 3)** The authors only evaluate the proposed strategy on a single backbone model.\n\nWe already provided the performance of another one of the most widely used smaller LMs, T0-3B, in Table 5. As described in Lines 652-657, our T-SAS with T0 consistently enhances the performance across three QA datasets, which supports the applicability of T-SAS on various recent smaller LMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Instead of using conventional strategies, such as Top-k or nucleus sampling, to obtain multiple self-generated answers based on a single representation, this work proposes to leverage multiple representations generated through Monte Carlo dropout, which could increase the diversity of answer sampling. This idea is simple yet effective.\n2. The experiments are comprehensive and the performance gains demonstrate the effectiveness of the proposed methods.\n3. The paper is clearly written and easy to follow.", "reasons_to_reject": "1. Despite the performance gains, directly selecting the pseudo label via majority voting strategy, i.e., the highest number of occurrences, is unreliable. since the generated results of smaller LMs are less reliable compared to large LMs. There is a higher probability that all the generated answers are incorrect, and using these pseudo labels for training may hinder the model's performance. The part requires more discussion.\n2. To show the effectiveness of the proposed method, it also needs to compare the performance with those large LMs (zero-shot). Even if it still cannot beat the large LMs, it would still be very encouraging if it shows a narrowing of the performance gap between smaller LMs and large LMs.\n3. The author only evaluates the proposed strategy on a single backbone model. It is hard to see whether this method can be generalized to various language models or not.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Although large language models show a strong ability to store a vast amount of general knowledge across diverse tasks, they struggle with suboptimal performances on specific downstream tasks when transferring and adapting their knowledge to target tasks. Fine-tuning LMs is a possible solution, but it is impractical when labeled datasets are scarce. \n\nThis work investigates the capabilities of smaller self-adaptive LMs, only with unlabeled test data. The author devises a self-adaption strategy that stochastically generates multiple answers, and then ensemble them while filtering out low-quality samples to mitigate noise from inaccurate labels. Instead of utilizing the large LMs, which require substantial costs and restricted accessibility. This work focuses on smaller LMs and aims to mine their capacity to solve downstream QA tasks without fine-tuning by using external knowledge. Experiments show significant performance improvements on various QA tasks with higher robustness across diverse prompts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hg2XQxcjlt", "reviewer": "Reviewer_kW16", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Weakness 1-1)** The first reason for rejection I see is that at least ⅔ datasets used in this study are part of the training set used to instruction-tuned Flan-T5 models. \n\nPlease note that FLAN-T5 series models have gained recognition for their extensive training across nearly all available benchmark QA datasets, thus making it difficult to evaluate them with previously unseen benchmark QA datasets. However, it is important to mention that these models were not previously trained on certain QA datasets (e.g., cpgQA in Table 2), and we experimentally show that our T-SAS exhibits clear improvements even on them.\n\nAdditionally, we have extended our analysis to include experiments on another unseen QA dataset, namely TyDiQA [C3], only using the English data in Table C.1. As shown in Table C.1, the proposed T-SAS model further demonstrates substantial enhancements in performance.\n\nAlso, it is worth noting that the performance improvement achieved by applying self-adaptation to the already trained data is not our weakness, but rather a strength. To be more specific, even though an LM has been trained comprehensively on diverse datasets including the target dataset, the LM remains as the general-purpose model and is not tailored to the specific target dataset. However, by using our proposed T-SAS on the target dataset, the model can further achieve improved performance thanks to its self-adaptation capabilities to the target dataset.\n\nTable C.1: Results on the unseen dataset, TyDiQA with English, with FLAN-T5-XL based models.\n|Methods|EM|F1|\n|---|---|---|\n| Naïve LM | 64.55 | 76.97 |\n| T-SAS | **67.95** | **79.31** |\n\n[C1] TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages\n\n>**Weakness 1-2)** The authors do not discuss the implications of this fact (i.e., at least ⅔ datasets used in this study are part of the training set used for instruction fine-tuning Flan-T5) in the paper, which could be grounds for rejection.\n\nPlease refer to **Weakness 1-1** for a detailed answer. We will discuss these implications in the revision. Specifically, in Section 4.2, we will include the experimental results of the additional unseen data (Table C.1). Moreover, in the Limitation section, we will further discuss the challenge associated with evaluating FLAN-T5 series models on unseen benchmark QA datasets due to their limited availability. We hope that these updates, requiring minor revisions, will address your concerns. \n\n\n>**Weakness 2)** More discussion on the other related work [C2-C3].\n\nWe thank you for suggesting related work. However, please note that the suggested papers and ours are not directly comparable for the following reasons:\n\n* **Extractive QA vs Generative QA**:  The suggested papers target at the extractive QA setting, while our T-SAS is designed under a generative QA setting. Since the extractive QA models aim at extracting exact answers from the given context, they address the problem as the token classification task. Please note that as the objectives of extractive and generative settings are different, the test-time adaption strategies are fundamentally different. To be specific, in an extractive setting, adaptation is based on probabilities, while in a generative setting, adaptation is done using generated text. In situations where filtering is further applied, in the extractive setting, filtering is based on probabilities, whereas in the generative setting, it is done using the generated text (We discovered that it is less effective to filter out samples based on a single generation probability in Table 3). This makes it difficult to directly compare our generative T-SAS against the extractive models. Therefore, while several previous work has investigated self-training approaches to solving classification tasks under the extractive setting, it has been questionable and largely underexplored how we can self-adapt the small LMs over the generative setting.\n\n* **Question-Answer pair generation vs Answer generation**: As the suggested papers assume an unsupervised QA setting, the context-question-answer triplets are not available. Therefore, they are required to first generate question-answer pairs with an additional query-generation module. Also, such a pair generation approach is different and rather orthogonal to ours, since we aim to enhance answer generation directly from the provided context and question.\n\n\nWe concretely further discuss the differences between each of the suggested papers and ours as follows:\n\n[C2] Harvesting and Refining Question-Answer Pairs for Unsupervised QA\n* This work is an **Extractive** task (QA), which is largely different from our generative setting, as aforementioned above.\n* This work focuses on the **Question-Answer pair generation** strategy, requiring an additional query generation module.\n* In addition, the filtering approach in this work is based on a single token probability. Please note that such a filtering approach applied to our generative setting is already presented as a baseline in Table 3 (w/o Stochastic), which indicates the effectiveness of our proposed stochastic filtering strategy. \n\n[C3] End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems\n* This work is an **Extractive** task (QA), which is largely different from our generative setting, as aforementioned above.\n* This work focuses on the **Question-Answer pair generation** strategy, requiring an additional query generation step.\n* Furthermore, we already compared the filtering approach used in this work as a baseline in Table 3 (w/o Stochastic), which is based on the likelihood of the generated question-answer pair, showing that our proposed filtering strategy outperforms this baseline.\n\n\n>**Weakness 3)** The paper requires more explicit contextualization and more clarity in its limitations.\n\nPlease refer to **Weakness 2** for more explicit contextualization and **Weakness 1** for more clarity on limitations. To summarize the answers on them, regarding contextualization, we will further discuss how our proposed T-SAS is different and orthogonal to the suggested unsupervised learning scheme and question-answer pair generation work in Section 2. Also, we will further clarify that the setting of test-time adaption under the generative QA setting is largely underexplored. Furthermore, we will discuss the problems of the limited availability of the unseen QA benchmark datasets in the Limitation section, as mentioned in **Weakness 2-1**. We believe that these revisions are minor to address and hope that you are satisfied with them.\n\n\n>**Weakness 4)** Minor issues of terminology and presentation.\n\nThank you for your suggestion and we apologize for the confusion. We will update them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Exploring the ability of open-source models to perform on QA tasks without supervision is an important and impactful research direction.\n- Proposed sampling methodology is interesting and novel.\n- Empirical results are sound, baselines are strong and thorough and all the necessary ablations are present.", "reasons_to_reject": "- The first reason for rejection I see is that at least ⅔ datasets used in this study are part of the training set used to instruction-tune Flan-T5 models. The authors do not discuss the implications of this fact in the paper, which essentially could change the self-adaptive or self-training setting into more of a semi-supervised or even data-augmentation setting. This could be grounds for rejection.\n- The second important reason for rejection would be that the current work is not contextualized thoroughly enough. There is some mention of recent self-training works such as Wang et al. 2023 or Chen et al. 2023 but it is unclear how this work provides a “more realistic test-time setup without labeled data”. This paper is very related to QA self-training papers like the following but no discussion about this is added.\n   - https://aclanthology.org/2020.acl-main.600.pdf\n   - https://aclanthology.org/2020.emnlp-main.439.pdf \n- To be clear, I believe this paper does expand on the current literature but it still requires more explicit contextualization and more clarity in its limitations.\n- Minor issues:\n   - FLAN and FLAN-T5 are different models, please clearly state that you are using FLAN-T5 throughout the paper.\n   - The tables and figures on page 4 are very crammed, you should have just one caption with (left, center, right) divisions or split the figures up.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a “self-adaptive” technique which enables smaller LM fine-tuning without labeled data. The authors introduce a novel sampling technique which uses Monte-Carlo dropout to create many potential answers for an unlabelled question which allows for improved filtering of noisy answers. They provide thorough empirical evidence showing that their proposed sampling and filtration method is more effective than previous methods.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://aclanthology.org/2020.acl-main.600.pdf\n\nhttps://aclanthology.org/2020.emnlp-main.439.pdf"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses a self-learning strategy for fine-tuning relatively small LM for Question Answering tasks using only unlabelled data. 
The proposed solution is relatively simple but sound and intuitive and the results observed on multiple datasets are remarkable.

There is enough substance for a short paper.
Related works should be better discussed.
Decision: Accept-Findings