Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers
Authors: Kangda Wei
Sayan Ghosh
Rakesh R Menon
Shashank Srivastava
Venue: EMNLP 2023 Findings (2023)
Keywords: test-time adaptation
learning from language
data programming

Abstract: Recent approaches have explored language- guided classifiers capable of classifying examples from novel tasks when provided with task-specific natural language explanations, instructions or prompts (Sanh et al., 2022; R. Menon et al., 2022). While these classifiers can generalize in zero-shot settings, their task performance often varies substantially between different language explanations in unpredictable ways (Lu et al., 2022; Gonen et al., 2022). Also, current approaches fail to leverage unlabeled examples that may be available in many scenarios. Here, we introduce TALC, a framework that uses data programming to adapt a language-guided classifier for a new task during inference when provided with explanations from multiple teachers and unlabeled test examples. Our results show that TALC consistently outperforms a competitive baseline from prior work by an impressive 9.3% (relative improvement). Further, we demonstrate the robustness of TALC to variations in the quality and quantity of provided explanations, highlighting its potential in scenarios where learning from multiple teachers or a crowd is involved. Our code is available at: https://github.com/WeiKangda/TALC.git.

Leveraging Multiple Teachers for Test-Time Adaptation of Segguage-Guided Classifiers
Kangda Wei!
Sayan Ghosh Rakesh R. Menon Shashank Srivastava²
Department of Computer Science and Engineering, Texas A&M University Department of Computer Science, UNC Chapel Hill (sayghosh, rrmenon, ssrivastava @cc. unc.
kangdal amu edu
Abstract
Task: Will customer apply for travel-insurance?
Recent approaches have explored language- guided classifiers capable of classifying exam- ples from novel tasks when provided with task- specific natural language explanations, instruc- tions or prompts (Sanh et al., 2022; R. Menon al., 2022). While these classifiers can gen- eralize in zero-shot settings, their task perfor- mance often varies substantially between dif- ferent language explanations in unpredictable ways (Lu et al., 2022; Gonen et al., 2022). Also, current approaches beled examples that may be available in many scenarios. Here, we introduce TALC, a frame- work that uses data programming to adapt sanuuage-guiee classifier for new task dur- ing inference when provided with explanations from multiple teachers and unlabeled test exam- ples. Our results show that TALC consistently outperforms competitive baseline from prior work by impressive 9.3% (relative improve- ment). Further, we demonstrate the robust- ness of TALC to variations in the quality and quantity of provided explanations, highlight- ing its potential scenarios where learning from multiple teachers or a crowd involved. Our code is available at: https: /// //g/tttt...//////////////////tht com/ We i Kangkannnnaaaaa /TALC. git.
people travel abroad have taken travel insurance.
Travelers
are 29 end older take traveler insurance.
greduate have taken travel insurance.
Input X
Traveled abroad
Language-guided Classifier
Age
Education
Yes
32
High-school
leverage unla-
Y
Graphical Aggregation
Figure TALC leverages data programming to perform test-time adaptation of language-guidde guidddd classifier. Nat- ural language explanations (E' = {c1, 62, e3)) provided by multiple teachers and unlabeled examples (X1.)) for new task are fed to language-guides classifier pair-wise resulting in multiple pseudo-labels for the un- labeled examples. TALC uses graphical aggregation to weigh the pscudo-labels from different explanations to decide the final predicted label ( TALC is highly flexible in aggregating labels as it conceptually sider a broad variety of factors, such as the complexity of explanations. consistency between explanation pre- dictions, identity of the explanation provider, etc.
1 Introduction
Inductive learning from examples has underpinned many successful machine learning applications. However, classifiers trained solely from labeled examples often struggle to generalize in scenarios with limited labeled data. In contrast, humans can learn new concepts through natural language con- versations (Chopra et al., 2019; Tomasello, 1999). Inspired by this phenomenon, recent approaches natural language explanations, instructions, and prompts train (anguage-guidee classifiers (Sri- vastava et al., 2017: Andreas et al., 2018: Murty et al., 2020; Wang et al., 2020; Ye et al., 2020). While these classifiers can perform zero-shot classi- fication, they have several limitations. Firstly, they
lack principled strategy for weighing language pervision from multiple sources (or teachers). Sec- ondly, they fail utilize the available unlabeled data for new task during inference. Additionally, the impact of the quality of sources, and the inclu- sion of low-quality explanations, remains largely unexplored.
To address these limitations, we present TALC (Test-time Adaptation of Language-guide Classifiers), a framework for adapting language-

---

guided classifier on novel tasks during inference. also known as test-time adaptation. TALC assumes a priori access to the entire test set (unlabeled sam- ples) of the novel task and the task-specific expla- nations, which aligns with real-world situations, such as developing a product-category classifier for an e-commerce platform. In the context of TALC, the multiple explanations available for each task are considered distinct supervisory signals. Leveraging the power of data programming (Rat- ner al., 201 8b), TALC aggregates and combines the supervision provided by these multiple expla- nations effectively.
2 Related Work
Learning From Language Using natural lan- guage explanations to inform or train classifiers has garnered significant interest in recent years (Gold- wasser and Roth, 2014; Srivastava al., 2017; Han- cock et al., 2018; Murty et al.. 2020). While Murty et al. (2020) enhance supervised BERT models (De- vlin et al., 2019) for relation extraction tasks, other approaches employ language explanations for few- shot learning. For instance, Hancock et al. (2018) convert explanations to labeling functions via se- mantic parsing, leveraging unlabeled data for weak labels. More recently, R. Menon et al. (2022) uti- lize natural language explanations in entailment- based model for classification decisions.
Figure 1 illustrates the TALC framework. TALC uses subset the test data, called the adaptation set, for adapting the anguage-guided classifier. For each pair of explanation and test example in the adaptation set, pseudo-label is generated using the base language-guided classifier. TALC learns label aggregator the pseudo-labels generated for the adaptation using EM ($3). The label aggregator is trained to consider the contribution of each explanation during adaptation, thus, in prin- ciple, allowing it to weigh different sources of lan- guage supervision. Finally, TALC uses the learned aggregator over the entire test set to obtain final predictions for test set examples.
Test-time Adaptation Test-time adaptation has been extensively studied in computer vision by em- ploying batch-normalization statístics (Nado al., 2020; Khurana et al., 2021; Schneider et al., 2020), test-time entropy minimization (Wang et al., 2020; Sivaprasad and Flcuret, 2021), prediction consis- tency maximization over augmentations (Zhang et al., 2021), and classifier adjustment (Iwasawa and Matsuo, 2021). In the realm of natural lan- guage processing, Banerjee et al. (2021) explore test-time adaptation for question- antteernn using self-supervision. In contrast, we introduce a new test-time adaptation approach that leverages data programming to adapt a base eagguage-guided clas- sifier during inference by utilizing natural language explanations.
We evaluate TALC on six classification tasks from the CLUES-Real dataset (R. Menon et al., 2022), where each task is paired with natural lan- guage explanations ($4). TALC outperforms strong baselines by 3.3% on average (absolute). Through qualitative and quantitative analysis, we investi- gate TALC's robustness with respect to the size of the adaptation set, number of explanations, and ex- planation quality. In the subsequent sections, we describe TALC in detail ($3), present experimen- tal results and analysis ($4), and conclude by dis- cussing contributions, limitations, and ethical considerations. Our contributions are:
Data Programming Data programming (Ratner et al., 2017) employs a combination of multiple labeling functions and generative models to create probabilistic training labels for unlabeled datasets. Prior work (Ratner et al., 201 8b; Hancock et al., 2018) has demonstrated successful applications of this paradigm to create systems that allow users to label large datasets programmatically. Here, we repurpose data programming in the test-time adap- tation setting to improve classifiers on unseen tasks.
We introduce TALC, test-time adaptation frame- work, that uses label aggregation improve language-guided classifiers.
3 TALC
In this section, we present the details of our frame- work, TALC. TALC leverages data programming to adapt base natural language explanation-guidee classifier on a novel task during inference.
We demonstrate the effectiveness of TALC on multiple real-world classification tasks from CLUES Reea (R. Menon et al., 2022).
We present comprehensive analyses to evaluate the robustness of TALC w.r.t the quantity and quality of explanations.
Problem Setup. We assume a language-guin -gudded classifier, MyC, which can take an explanation e

---

from teacher and example X predict a label Mic(X,e) A language-guided classifier refers to a classifier that utilizes one or more natural lan- guage explanations to make predictions. Our objec- tive is to make predictions for a batch of test sam- ples, represented as {etest Ytest I:m, where Yest represents the unobserved ground-truth labels cor- responding to the samples in Xteat, and denotes the number of samples. During test-time adapta- tion, our aim is to effectively adapt the classifier to the specific task hand and infer the true labels for Xtost. Existing methods for test-time adaptation typically assume an online setting, where exam- ples are processed one at a time (Sun et al., 2020; Banerjee et al., 2021). In contrast, we assume priori access to the entire test set of the task. This assumption allows to leverage the empirical dis- tribution of the unlabeled data for semi-supervised learning.
Algorithm TALC
Inputs:
Language-guided guided classifier MIC. test
set Xtest, task explanations E, adaptation ratio
Nadapt + [Xtestt]
2: Xeest + Xtesel: Nadapt)
3: X hheld-out <- Xtost Nadapt :]
test
4: Train the label aggregator C on Xadapt
① + argmax Peo (x MIM (using EM)
5: Infer YTALO for Xtest using the learned w. YTALC argmax Pa (Y F,Mcc)
return PTALC
set and the held-out test. The adaptation set is utilized by TALC to adapt MLC. The proportion of the test set that forms the adaptation set is defined by an adaptation ratio, a E [0, 1], defined as a = Indaptation sell test set
We also partition the labeling matrix M into Madapt and Mheld-out by choosing the rows corresponding to samples in the adaptation set and held-out set, respectively.
Our setting aligns with weal-world scenarios, such as developing a product-category classifier for e-commerce platform, where the complete database of products (including the test set) is known in advance. For situations where test sam- ples are observed one at a time, it is still possible to utilize TALC for adapting a base classifier. This in- volves a "warm-up" phase, where the base classifier is used off-the-sheeff for a few samples, followed by adaptation using TALC. While this usage scenario is not the primary focus of our work, we provide description of how TALC can be employed in such cases in Appendix $A for brevity.
To model the dependence between the (latent) inferred labels and Madapp, we use data program- ming techniques (Ratner et al., 2019) to train a label aggregator, Lase with task-specific param- eters w. We use the learned parameters (which correspond to weights Iearned for each explana- tion) to aggregate predictions in M (both M and Miheld and infer the labels, YTALC-
Label Aggregator. The label aggregator is a graphical model that defines the joint probability of explanations E, examples X and latent (true) labels Y for a given language-guides classifier as:
Overview. As depicted in Figure 1, for new task Tnew, we are provided with natural lan- guage explanations E {e1, .2.,...... cm). and a set of examples {X; E ). To generate the classifier outputs, we iterate through each ex- planation Ej for every example X,, and compute Mj == (x,). This yields a labeling matrix M with a shape of n x m. Next, we in- troduce a test-time adaptation procedure: TALC, to compute the final labels Ý utilizing the M. This procedure essentially implements function f M Ra + Y e R", which we describe in the rest of this section.
E, & exp w OX,EEYY (1)
Here, e is a feature-vector of features that can be computed using X, E, Y and M and w is a weight vector corresponding to each of those fea- tures. In general, this can subsume very broad range of features, including features that can indi- cate the complexity of an explanation, or its prove- nance 1 We also note that in particular, since the labeling matrix M is computed from X, E and MLc. á can include features that depend on M and Y. For simplicity, our instantiation incorpo- rates the labeling rates of each explanation (how
Test-time Adaptation. The objective of TALC is adapt the language-guider classifier, MIc. on a novel task, T Tnow during inference. We illustrate the adaptation procedure in Algorithm 1. First, we split the test set into two disjoint sets the adaptation
'So, for example, the aggregator can automatically leam to Ican more/less on complex explanations, or trust explanations from specific sources more than from others

---

frequently an explanation assigns label?) and the correlations between the pseudo-labels from differ- ent explanations to estimate the accuracies of each individual explanation in an unsupervised manner. Specifically, the label aggregator is defined in terms of two types of features; accuracy AoAce E (uxm") and propensity (oProp E R=xm)) Each value in Acc and Prop is defined as:
et al., 2022) benchmark. Out of the sixteen real- world tasks in the test split of CLUES, we focus on six tasks for evaluation due to the limited num- ber of test samples (< 10) in the remaining tasks, which restricts their suitability for test-time adap- tation. Figure I presents an illustrative example showcasing the nature of these tasks and provides examples of the corresponding natural language explanations. Appendix $B provides further details regarding the six tasks selected for evaluation. We utilize the ExEnt model (R. Menon et al., 2022) as the base language-guided classifier (Mic) in alignment with our choice of the CLUES dataset'. The ExEnt model leverages textual en- tailment to establish the correspondence between explanations and tabular inputs, enabling label pre- dictions. To aggregate predictions from multiple explanations, ExEnt adopts a mean-pooling strat- egy. aggregating the predictions obtained from each explanation-input pair to derive the final label. It's important to note that ExEnt is not trained for ab- stention, meaning it always assigns a label regard- less of the quality of the explanations. In $4.4, we further explore the scenario of abstention, which consider a more realistic use case for language- guided classifiers.
Oij (AM0 =1 [[M vi)
(2)
Prop (M,Y) = (M,s # yabstain] Oij
(3)
where yi is the label for 2thh sample and Vabstoin is special label that denotes MLC has abstained from predicting label based on the j explanation. The accuracy factor fires if the inferred label (Y;) for an unlabeled example X; matches the predicted label from an explanation j. The propensity factor fires whenever the classifier doesn't t abstain from predicting a label from an explanation.
Since here we only define two types of features for each explanation, vector corresponding to weights for the accuracy factor and propensity factor for each explanation. The weights are learned by maximizing the log- likelihood log P(X, F) = log Ey ( X, E, Y) (EM) algo-
1R is learnable
using the oppectation-maxxiiizat rithm (since we don't t have ground-trune truuh labels for Y at test-time). We compute the MAP estimate YTALC argmax Pa (Y|X, F) using Gibbs sam- Y
4.2 Baseline and Evaluation Metries
We compare TALC against the following baselines: 1. ExEnt: This refers to the base ExEnt model (R. Menon et al., 2022) that has been trained on real-world training tasks from the CLUES dataset.
pling to predict the final labels. Note that while we learn the weights á on the adaptation set (line 4 in Algorithm 1), the learned weights are used to aggre- gate predictions in both the adaptation and the held- out examples, to predict the labels, ALC (line in Algorithm 1). We implement the label aggre- gator using Snorkel-Metal (Ratner et al., 201 2018a). Appendix $C provides task-specific details of the label aggregator training.
2. ExEnt-MV: For each example X,, we generate set of pseudo-labels corresponding to each of the m task explanations. The final predicted label is determined by selecting the label that appears most frequently among the m pseudo- labels (majority vote). Unlike ExEnt, which uses mean-pooling for aggregation, ExEnt-MV applies a mode-pooling operation.
4 Experiment and Analysis
In this section, we evaluate the zero- zero-shot adaptation performance of TALC on classification tasks, fol- lowed by a detailed analysis of TALC's robustness.
3. ExEnt-FT: Similar to our approach of fine- tuning TALC with the predicted labels from the label aggregator, L we also include self- training baseline approach by fine-tuning ExEnt. This involves utilizing ExEnt's own predictions as labels on the adaptation set.
4.1 Data
We assess the performance of TALC on real-world classification tasks from the CLUES (R. Menon
We use classification accuracy as the evaluation metric to compare the utility of different methods.
"The pseudo-label corresponding to an explanation can either a class label, or a special label, Yalstein (e.g. an explanation does not upply for an example) Phttps:/ / //wwww snorkel. orrg
At the time of writing, this is the best model on CLUES with publicly available code.

---

4.3 Results
ExEnt is based on NLI (R. Menon et al., 2022), where neutral label can be naturally mapped an abstention. We train ExEnt-A on the same tasks as ExEnt with the modification of having 'abstain' as an additional class label for each task.
Table 1 shows the zero-shot classification accuracy of TALC and the baselines on the six evaluation tasks. The findings reveal several key insights. Firstly, we observe that majority voting (ExEnt- MV) performs better than vanilla ExEnt on av- erage across the six tasks. Secondly, fine-tuning ExEnt on its own predictions (ExEnt-FT) results in better zero-shot accuracies than the base ExEnt model, demonstrating the value of self self-training on unlabeled data. Furthermore, the performance of ExEnt-FT increases with an increase in the amount of test data used for adaptation (35.7 - 36.8 as we increase the adaptation ratio from 0.3 + 1.0).
Table 2 shows the results of TALC-A, and the baselines when abstention is allowed ("-A' denotes abstention). We find that TALC-A achieves the best overall accuracy. More importantly, comparing Table 1 and Table 2, we observe that TALC has a smaller drop in performance in comparison to ExEnt and ExEnt-FT suggesting that TALC is bet- ter at adapting to multiple teachers even when cer- tain teachers choose to abstain from prediction.
We note that TALC obtains better performance average across all evaluation tasks compared to the three baselines. Specifically, TALC improves the accuracy by around 3.3% on average (abso- lute) over the state-of-the-aar ExEnt model. In fact, both TALC variants, at adaptation ratio 0.5 and 1.0, perform better than ExEnt on all tasks except for indian-liver- liver--atientt,t The utilization of the label ag- gregator in TALC results the biggest improvement (~~ 25% relative) the tic-tac-toe- toeee--a-ed task. We attribute this improvement to the label aggre- gator's ability to give higher weightage to high- quality explanations, resulting in more accurate predictions. For the tasks where the performance of TALC is close for a = 0.5 and a = 1.0 in Table 1, we observed that the aggregation weights for each explanation learned by the data programming framework are roughly similar for the two settings. As result, the aggregation over the pseudo labels in the labeling matrix produces similar final predic- tions and hence similar accuracies.
40
Accuracy
36
34
0.2
0.8
1.0
Adaptation Ratio EXEnt-FT
TALC
Figure 2: Accuracy (averaged over 6 tasks) of ExEnt and TALC when training label aggregator with different adaption set sizes. Overall, increasing the adaptation ratio does not impact performance of ExEnt-FT, but improves performance of TALC.
Effect of adaptation set size. We analyze the performance of ExEnt-FT and TALC by varying size of the adaptation set. Specifically, we vary the adaptation ratio, from 0.2 to 1.0 (in increments of 0.1) for all six evaluation tasks.
Intuitively, we expect that the accuracy of ExEnt- FT and TALC to improve with increase in adaptation ratio. However, we empirically observe that the performance of ExEnt-FT fluctuates with change in a and does not show a consistent trend of im- provement as shown in Figure 2. Meanwhile, shown in Figure 2, we observe that larger adap- tation set enhances the performance of TALC from 37.6% + 38.8% as increases from 0.2 1.0.
4.4 Analysis
Abstention. Our previous experimental results treat labels from individual explanations the same, irrespective of the confidence of the model in its predictions on those examples. This is because the base-language classifier used in experiments, ExEnt, always chooses a label during inference rather than performing selective predictions. How- ever, the TALC framework allows for differential modeling of abstentions, where a model can choose refrain from assigning class label if the expla- nation does not apply to the example. To explore this, we design a variant of ExEnt, referred to as ExEnt-A, A, that can refrain from assigning a class la- bel during inference. This is straightforward since
Robustness to number of explanations. Next, analyze the robustness of ExEnt-FT and TALC to changes in the number of explanations provided for adaptation on the new task. We will refer to the fraction of explanations used for adaptation as the
"The results of TALC and ExEat-FT on each individual task with different adaptation ratios can be found in Appendix $G.

---

Non-Adaptation Baselines
Adaptation Ratio 0,5
Adaptation Ratio 1.0
ExEnt
Tasks
ExËnt
ExEnt-FT
TALC
ExEnt-IT
TALC
bankkote-auuheeniiation
46.9
48.4
(0.1
49.5
45.2(0.2)
49.7(0.3)
tie lad-ooe-n
32.8
32.3
32.3
41.1(0.0)
32.3
(1.1(0.0)
cur-evaluation
10.7
17.6
4.6
14.1(a.4)
3.8(0.4)
16.5(u.0)
contraceptive-shenn
42.7
43.7
43.4(0.0)
44.0(0.7)
43.4(p.0)
43.8(0.3)
indian-liver-patien
48.7
40.0
(7.5
44.3ç2.5)
62.0(4.0)
47.8(u.0)
travel-insurance
31.9
33.9
(4.2
34.2(o.o)
(4.2(0.0)
34.2(0.0)
Average
35.6
36.0
35.7
37.9
36.8
38.9
Table I: Comparison zero-shot accuracies (higher is better) between non-adaptation-basel ExEnt baselines, We report the and
ExEnt-FT, and proposed method, TALC, the 6 different tasks from standard deviation for the accuracy across three runs for adaptation-bust -basedd methods. The numbers in bold indicate the best accuracies across methods.
Non-Adaptation Baselines
Adaptation Methods
Tasks
ExEnt-A
FEEnt-FT-A
TALC-A
MV-A
banknote-authenticne
authentication
8.0
36.7
27. (2.5;
53.9 (0.1)
tic-tac-toe toe-endgame
2.6
30.2
(2.3
(2.4(
car-evaluation
2.3
2.6
14.8
13.3
contraceptive-choic
22.7
32.8
19.3
31.6(1.)
indian-liver-patiee
36.5
31.7
30.1
(0.2
travel-insurance
15.3
24.6
15.2(2.0)
23.4(u.))
Average
14.6
26.9
23.1
32.5
Table 2: Comparison of zero-shot accuracies between TALC and the baselines when allowing the ExEnt model to abstain from making a prediction (the modified model is denoted as ExEnt-A), 'A' stands for 'Abstention' for all the models in the table. For the adaptation methods (ExEnt-FT, TALC), we report mean and standard deviation across 9 adaptation ratios (0.2 to 1.0). Numbers in bold denote the best accuracies across methods.
ratio (a) fixed at 1.0 for this analysis.
40
Figure shows the variation in performance of ExEnt-FT and TALC with changes in the explana- tion ratio averaged over the six evaluation tasks. The accuracy of TALC drops when increasing the explanation ratio as 0.3 + 0.5, buts shows a consis- tent increasing trend (from 33.5% - 38.8%) when increasing the explanation ratio from 0.5 + 1.0. In contrast, the performance of ExEnt-FT fluctuates as the number of available explanations changes. This shows that TALC is comparatively more sensi- tive the number of explanations used for adapta- tion.
38
Accuracy
36
34
32
0.2
0.4
0.6
0.8
1.0 TALC
Ratio of Explanations Used
ExEnt-FT
Figure 3: Results for ExEnt-FT and TALC when varying the number of explanations used for training the label aggregator. Results are averaged over the six evaluation tasks. With increase in number of explanations, the accuracies using TALC improve while performance of ExEnt-FT is not affected.
Robustness quality explanations. Here we analyze the role of explanation quality on the per- formance of TALC. However, quantifying the qual- ity of explanations in the absence of annotations is a challenging and open research problem. To circumvent this issue, we explore two approaches to quantify explanation quality:
explanation ratio # available expls # of all expls Specifically, we vary the explanation ratio from 0.2 to 1.0, by randomly choosing explanations without replace- ment, when training the ExEnt-FT model and the label aggregator in TALC. We keep the adaptation
Individual explanation accuracy: Here, we as- sume there exists an oracle which has access to all the explanations, the base language guided-

---

classifier, and the labeled examples. This oracle evaluates the accuracy of each individual expla- nation of the task by evaluating it on the labeled clas-
TALC: Using Top XX Explanetions (Ranked by Accurecyl
80-
Accuracy (K)
80
examples with the base language-guii sifier. We term this accuracy as the individual explanation accuracy and use it a proxy for quan-
40
tifying the quality of an explanation. For each of the six evaluation tasks, we provide the individ- ual explanation accuracies in Appendix $F. Perplexity of an explanation: Assuming access to all labeled examples (needed for the above approach) may be unrealistic for many scenarios. Hence, we also explore a surface-level metric, the perplexity of the explanation, to quantify the quality of an explanation. We obtain perplexity scores for each explanation by using the GPT2- Large pre-trained model (Radford et al., 2019). We provide perplexity scores of each explanation for the six evaluation tasks in Appendix $F.
He- spe aetten
bantente
endgging
malution
contracapdie
Jedia patlent
Insutanca
Acerage
Chnice
20%
B0%
TALC: Using Top X% Explanations (Ranked Perplexity)
80
Accuracy (30)
80-
40
20
benenooee
toe endggame
cer mnkiuaioon
contran duike
Buer
inouctess
Avarago
These aforementioned approaches to quantify the quality of an explanation) can filter out poor quality explanations or selectively choose good quality explanations for adapting the base language- guided classifier. We explore the following scenar- ios (with adaptation ratio, a 1) to understand the impact of explanation quality:
20%
40%
60%
BO%
100%
Figure 4: TALC's performance only using the top X% explanations, where X = 20, 40, 60, 80, 100. On aver- age, TALC has the best performance when only using the explanations with the highest quality. The performance of TALC decreases and then increases as we add explana- tions with lower quality. We see this trend because only the explanations with high quality are used at first and adding explanations with lower quality distract the label aggregator at first, but the label aggregator is able to distinguish high-quality explanations when the number of explanations keeps increasing.
Using the top X percentage of explanations: We rank the explanations by accuracy or perplexity for each task and only use the top X percent of the ranked explanations for TALC, where X 20, 40, 60, 80, 100. The results are shown in Figure 4. On average, we observe that TALC performs the best when using only the top 20% of explanations ranked by both accuracy and perplexity. As X increases from 20 > 40- , 60, the average performance of TALC decreases, and then keeps increasing. We
TALC: Removing the explanation with highest accuracy
80
Accuracy (%)
60
40
20
attribute this trend to the fact that the training of the label aggregator may be sub-optimal with a smaller number of explanations, and improve with more explanations. These results also clearly show that the label aggregator is able to distinguish explanation quality. We note a roughly similar trend when the explanations are ranked by lowest perplexity instead of highest accuracy. This is encouraging result, and indicates that perplexity of explanations can actually be reasonable basis for filtering from a large pool of explanations.
banknote au auteettcaaio caslen
ensgeme
evatuation
cestraceslive method
pallent
Bror
travel Iesurancc
Average
w/ high highest-accurac explanation
w/o highest-accuracy explanation
Figure 5: When ranking the explanations by their indi- vidual accuracy, removing the best explanation leads to 1.3% drop in performance on average.
planation from the set of explanations for each the task and adapt TALC. Figure 5 shows that re- moving the best explanation hurts performance consistently across tasks, as expected. We ob- serve a 1.3% in accuracy drop when ranking the
Removing the best explanation: We remove the best (highest accuracy or lowest perplexity) ex-

---

explanations by accuracy and a 1.0% drop when ranking by perplexity on average across the six tasks (shown in Appendix $1).
TALC: Replacing high-accuracy explanations with malicious explanations
80
Accuracy
80
TALC: Adding the explanation with lowest accuracy
40
Accuracy (%)
20
60
40
banknote authanticatio fert
(ic-tc endiaima
iraliattio
contraceptin sivea
indian
Infurance
Average
patient
choice
w/o malicious explanation
w/ 3 malicious explanation
Sumtentcction
tic tac-to
endgamn
evaluation
cuntracaptiv tiracppiiv method choice
indian liver patient
travel Insirance
Average
Figure 7: Comparison of TALC's performance before and after replacing good-quality explanations with ma- licious explanations. drops by 7.3% good explanations replaced by malicious explanations.
w/o aaccurcyy explanation
w/ lowest- lowest-accuracy explanation
average, TALC's performance
Figure 6: Comparison of TALC's performance before and after adding a low-quality explanation to set of high-quality explanations. On average, the performance decreases by 1.5% when ranking by accuracy.
malicious explanations. When explanations are ranked by perplexity, the results are similar (de- tails in Appendix ŞI). Surprisingly, for the 'car- evaluation task, the performance increased from 16.5% to 21.4% on modifying the best expla- nations to malicious explanations when ranking by accuracy. From the average drop in perfor- conclude that TALC is susceptible text-based attacks that may occur through the explanations provided during adaptation. Future work can address the challenge of learning distinguish between beneficial and adversarial explanations.
Adding low-quality explanation to set of high-quality explanations: Next, we study the TALC.
impact of low-quality explanations For this, we consider two setups. In the first setup TALC utilizes just the top-3 explanations as per their individual accuracies. The indi- vidual explanations accuracies can be found in Appendix SF. In the second setup, TALC utilizes the top-3 and the worst explanation (as per individual explanation accuracy) for adaptation. Figure 6 shows the performance of TALC for these settings. When ranking by accuracy, the average decrease in performance due to the addition of low-quality explanation is 1.5%, demonstrating the robustness of TALC to low-quality explanations. We observe a similar trend in results when the explanations are ranking by their perplexity (details in Appendix $1).
mance,
Agnostic nature of TALC w.r.t language-guided classifier. The flexibility of choosing different models as the underlying language- -uieeeg classi- fiers is an advantage of the TALC framework. The modular design of TALC, i.c, decoupling of (1) how we obtain predictions w.r.t each explanation using language-guided classifier and (2) how we com- bine these individual predictions, makes TALC highly generalizable and flexible framework. To empirically validate this flexibility, we experiment with different LLMs as the underlying language- guided classifier. Table 3 compares the accuracies of TALC and two baselines by using TO-3B (Sanh et al., 2022), OPT-2.7B (Zhang et al., 2022), and Flan-T5-XXX (Chung et al., 2022) as the underly- ing language-guided classifiers. As basclines, we consider two settings, (1) MV majority vote of the predictions made by the LLM corresponding to individual explanations and (2) Concat predict- ing by considering all explanations (concatenated) together in the context of the LLM. For both set-
Replacing best explanations with malicious explanations: Next, we create malicious expla- nations by flipping the labels mentioned by the original explanations. For example, taking the ex- planation from Figure for the travel-insurance task, we convert 'most college graduates have taken travel insurance' to 'most college gradu- ates have not taken travel insurance'. We repeat this process for the top-3 explanations ranked by accuracy perplexity for each of the six evalua- tion tasks. The results in Figure show drop in performance of TALC (from 38.8% to 31.5%), as expected, when the top-3 explanations (ranked by their individual accuracies) are modified into

---

TO(3B)
OPT(2.7B)
FLAN-TS-XXL
Tasks
Concat
MV
TALC
Concat
MV
TALC
Concat
MV
TALC
sunknote-authennicatin
44.7
44.7
44.7.0.0)
44.7
45.1
(4.7(o.0)
54.2
31.3
43.6ro.0)
67.7
67.7
67. (7.7.
32.3
19.0
(1.
32.3
10.6
56.8(u.o)
car-evaluation
3.8
71.7
(1.7(0.0)
2.3
3.8
3.8(o.o)
3.8
26.6
26.6(o.o)
contraceptive- choice
23.4
23.9
23.4(q.u)
27.5
33.2
33. 33.2
38.3
45.4
45.4(0.0)
indian- indian-liver-pate liver-patient
67.8
33.0
(8.7
32.2
32.2
32.2
66.1
48.7
56.5(0.4)
travel-ine insurance
38.9
65.8
65.8
42.5
65.8
65.8(o.m)
71.9
71.9
72.6 (2.4(
Average
41.1
51.1
53.7
30.2
38.2
40.8
44.4
44.1
50.3
Table 3: Comparison of accuracies between TALC and baselines when using different LLMs as the language-guided classifier on the 6 different tasks from CLUES-Reat We report the mean and standard deviation for the accuracy across three runs for adaptation-base methods. The numbers in bold indicate the best accuracy across methods.
ting, the prediction is done by prompting the LLM. We provide the prompt templates in Appendix $D. Table shows that TALC outperforms both base-
metrics to judge the quality of explanation mains an open and pressing research challenge.
lines for all of the three LLMs demonstrating the robustness of TALC to the choice of the underlying hanguage-guided classifier.
To analyse robustness of TALC w.r.t malicious explanations, created malicious explanations by flipping the labels mentioned in the best explana- tion for a task. However, there could other ways of creating malicious or adversarial explanations. which are more subtle than just flipping a label. For example, one subtle way of altering an existing explanation to malicious one could be by estab- lishing unwanted correlations between a protected attribute (e.g. gender) and the label for a down- stream task (e.g. whether the loan should be ap- proved). Analyzing and improving the robustness of TALC to more nuanced adversarial/maliciouss ex- planations remains to be explored.
5 Discussion & Conclusion
In this paper, we introduce TALC, a framework for test-time adaptation of language-guided classifiers that leverages multiple sources of supervision. One conceptual advantage of TALC is its agnosticism towards the choice of language-guu gugided classifier, leaving room for future exploration with different models. TALC is flexible in terms of what aspects of explanations, teachers and unlabeled examples are used to train the label aggregator. While our approach here trains label-aggregator for every new task (since our features for the label aggrega- tor include identities of individual explanations), in principle it should be possible to train unified label aggregator across tasks based featurized representations of tasks and explanations. Scaling up TALC to new datasets with a larger number of tasks would provide valuable insights into its gen- eralizability. Our experiments reveal TALC's sus- ceptibility to malicious attacks and bad-faith actors, which future works can improve on. Despite these challenges, TALC suggests exciting opportunities for harnessing the collective wisdom of teachers in real-world applications.
The adapted model obtained by using TALC is task dependent, as uses explanations and unla- beled data specific to the downstream task for adap- tation (specifically, for training the label aggregator component). Hence, for every novel task for which we want adapt base language-guider classifier, need access explanations and unlabeled sam- ples. This requirement (especially obtaining good explanations for adaptation) can be challenging issue for some real-world scenarios. Improving TALC to reduce its dependence on the amount of explanations and/or unlabeled data while still re- taining downstream accuracy adattaddatttiion) is an interesting direction for future work. The base language-guided classifier used in our experiments, ExEnt, designed to work with maximum of 512 tokens in its context. Usage of longer context models or even large-scale pre-trained models re- mains to be explored. The effectiveness of TALC under multilingual setting is also unexplored.
Limitations
To analyse the impact of quality of explanation during test-time adaptation, we use individual ex- planation accuracy as a surrogate measure for its quality in lieu of a standardized metric of expla- nation quality. However, developing standardized

---

Ethics and Broader Impact
the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 [Long Pa- pers), pages 2166-2179, New Orleans, Louisiana. Association for Computational Linguistics.
The experiments described in this work are per- formed over tasks from a publicly available bench- mark, CLUES (R. Menon et al., 2022). The data for these tasks do not contain any personally identifi- able information. We do not collect or annotate any additional data. For all the experiments in the paper we evaluate using automatic metrics and do not perform any human evaluation.
Pratyay Banerjee, Tejas Gokhale, and Chitta Baral. 2021. Self-supervised test-time learning for reading com- prehension. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1200-1211, Online. Association for Computational Linguistics.
TALC is agnostic to the base ainuaag--guided classifier. We do not foresee major risks with our framework if the inputs provided are appropriate. Like any other natural language guided method there potential concerns of misguiding model deliberately by providing erroncous inputs. Mea- sures to detect such bad actors and rectifying er- roneous inputs is beyond the scope of this work. However, there is a risk of classifiers perpetuat- ing biases present in the input natural language explanations (for example, some explanations may describe the label in terms of sensitive inappro- priate features). Biased or discriminatory explana- tions can result in biased predictions and contribute to unjust outcomes.
Sahil Chopra, Michael Henry Tessler, and Noah D Good- man. 2019. The first crank of the cultural ratchet: Learning and transmitting concepts through language. In Proceedings of the st Annual Meeting of the Cognitive Science Society, pages 226-232.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus. Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros. Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra. Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scaling crrruction-iinttnee language models.
The broader impact of this work can lead to de- velopment of frameworks that enable efficient adap- tation of Al systems. Developing Lunguage-guided adaptable systems can improve the impact and us- ability of Al systems in daily life, especially on the long tail of tasks with limited labeled data. How- ever, the responsible development and deployment of these models would require domain-specific ex- pertise, involving collaboration with experts and stakeholders to understand the implications and en- sure ethical considerations are met. Close attention should be paid to the specific contexts in which the classifiers are applied to minimize negative conse- quences and maximize positive impacts.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dan Goldwasser and Dan Roth. 2014. Learning from natural instructions. Machine Learning, 94(2):205- 232.
Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation.
A Acknowledgments
Braden Hancock, Paroma Varma, Stephanie Wang. Mar- tin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language ex- planations. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1884-1895, Mel- bourne, Australia. Association for Computational Linguistics.
The authors would like to thank the anonymous reviewers for their suggestions and feedback on the work. This work was supported in part by NSF grant DRL21 The views contained in this article are those of the authors and not of the funding agency.
Yusuke Iwasawa and Yutaka Matsuo. 2021. Temt-time classifier adjustment module for model-agnostic do- main generalization. In Advances in Neural Infor- mation Processing Systems, volume 34, pages 2427- 2440. Curran Associates, Inc.
References
Jacob Andreas, Dan Klein, and Sergey Levine. 2018 Learning with latent language. In Proceedings of

---

Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas. and Gaurav Aggarwal. 2021. Sita: Single image test-time adaptation.
Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Seao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompled training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity.
Shikhar Murty, Pang Wei Koh, and Percy Liang. 2020. ExpBERT Representation engineering with natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2106-2113, Online. Association for Computational I Linguistics.
Steffen Schneider. Evgenia Rusak, Luisa Bck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. 2020. Improving robustness against common corrup- tions by covariate shift adaptation.
Prabhu Teja Sivaprasad and François Fleuret. 2021. Test time adaptation through perturbation robustness.
Zachary Nado, Shreyas Padhy, D. Sculley, Alexan- der D' Amour, Balaji Lakshminarayanam, and Jasper Snock. 2020. Evaluating prediction-ting batch nor- malization for robustness under covariate shift.
Shashank Srivastava, Igor Labutov, and Tom Mitchell, 2017. Joint concept learning and semantic parsing from natural language explanations. In Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Language Processing, pages 1527-1536, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Rakesh R. Menon. Sayan Ghosh, and Shashank Sri- vastava. 2022. CLUES: A benchmark for learning classifiers using natural language explanations. In Pruceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 3523-6546, Dublin, Ireland. As- sociation for Computational Linguistics.
Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. 2020. Test-time training with self supervispe for generalization un- der distribution shifts, In ICML.
Alec Radford. Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):9
Michael Tomasello. 1999. The Cultural Origins of Hu- man Cognition. Harvard University Press.
Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré. 2018a. Snorkel metal: Weak supervision for multi-task learning. In Proceed- ings of the Second Workshop on Data Management for End-To- To-End Machine Learning, pages 1-4.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2020. Tent: Fully test-time adaptation by entropy minimization.
Ziqi Wang*. Yujia Qin", Wenxuan Zhou, Jun Yan. Qinyuan Ye, Leonardo Neves, Zhiyuan Liu, and Xi- ang Ren. 2020. Learning from explanations with neural execution tree. In International Conference on Learning Representations.
Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel. Proceedings of the VLDB Endowment, 11(3):269-25 282.
Qinyuan Ye, Xiao Huang, and Xiang Ren. 2020. Teach- ing machine comprehension with compositional ex- planations. CoRR, abs/2005.00806.
Alexander Rainer, Braden Hancock, Jared Dunn- mon, Roger Goldman, and Christopher Ré. 2018b. Snorkel metal: Weak supervision for multi-task learn- ing. In DEEM' 18: International Workshop on Data Management for End-to- to-End Machine Learn- ing, 11(3)44
Marvin Zhang, Sergey Levine, and Chelsea Finn. 2021. Memo: Test time robustness via adaptation and aug- mentation.
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. 2019. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 4763-4771.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab. Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey,
M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han

---

Appendix
Models
Prompt
Explanations: <explanation 1>. <explana- tion_2. ... <explanation Note Details:
A Usage of TALC
TO
1. An example of real-world cases where the entire set of test samples can be realistically accessed:
<feat_ 1> equal to <feat_ Ifeat___valuu>.. value>, <feat_ 2>
FLAN-
equal to <feat_2 value> ... From the expla- nations, is the note fake or original? Answer:
T5
Let's consider the of product cate- gory classifier for products the Amazon database. In this case, developers will first define classifiers using some training data and deploy the classifier on the entire database to label examples.
The following is classification task that the following explanations. Based explanation classify the subsequent sam- ple:
Explanations:
2. Method to use TALC when test samples are observed one-by-one:
<explanation 1>
<explanation 2>
Even if we not have access to the entire test set and the classifier observes unlabeled samples one by one, TALC can be deployed in practice as:
OPT
<explanation_n>
Note
Details:
<feat_ 1>
equal to
<feat 1 values.
<feat 2> equal
(a) For predetermined number of sam- plcs, the language-guided classifier is de- ployed off-the-shelf (Note: In this work. this would be the same as using ExEnt for those samples).
<feat 2_ values......
From explanations, note fake or original?
Answer:
(b) The aforementioned samples can now be pooled together as an adaptation set, and we can adapt the language-guided classi- fier using TALC.
Table 4: Prompt templates used for large language model experiments 4.4.
In other words, we incur a "warm-up" phase, where the un-adapted classifier is used, follow- ing which we adapt the classifier using TALC by considering the set of samples observed during warm-up adaptation set.
saiitii-ta--toc-ndgaaaa uul//contaccpiive-eettooo--dt uci/indian-liver-
uci/car-evaluation.
patient, and kaggle/tavel-insurance respectively.
C Hyperparameter and Compute Details
We train the ExEnt model following the hyperpa- rameters in R. Menon al. (2022), e.g. learning rate of le-5 for 5 epochs, batch size of 2, and eval- uation batch size of 16. For the label aggregator training, we did of the tasks and report the best Hyper-parameten in Table 5.
B Details of evaluation tasks
We use 6 real world classification tasks from R. Menon et al. (2022) tasks considered are cec//annknoe--authnnieeee uei/tic-tac-toe endgame, uci/car-evaluation uci/indian- liver-patient, and kaggle/navel-insuranee Examples of these tasks can be found at the CLUES website with the following link: https: http://cluee-beccaaaaaaa wiitmaarr....t........... io. Among the above tasks, uci/car-evaluation and are multi-class classification tasks while the rest tasks are binary classification tasks. The numbers of examples test set of each task are 275, 195, 346, 295, 115, 398 for htt.baanknote-uuhhnticaaa
evaluation tasks. The
search for cach
For fine-tuning the ExEnt model, compute time ranged from hr for the shortest jobs with smaller data sizes to 2 hours on I RTX 2080Ti GPU. For fine-tuning the label aggregator, compute time is within minute.
D Prompt Templates for LLM Experiments
For the experiments using large language models, we used the prompts elaborated Table 4.

---

TASK Ranknote-authentiatin
LEARNING RATE ÉrOCH
this paper in Table 8 to 13. In Table 8 to 13, We also report the accuracy when using only one ex- planation at a time with ExEnt and the perplexity
4.38e 08
500
li-lac-toe-endgame
7.31e U8
40
car-evaluation
2.48c 04
500
petraapptive-mettdd-hhhe
5.53e 03
100
of each explanations. In Figure 10, we analyze the correlation between accuracy and perplexity of all explanations. There is positive correlation
indian liver piveent
7.91c
travel-insurance
7.31e 08
40
between accuracy and perplexity of all the explana- tions.
Table 5: The best hyper- parameters for training label aggregator.
G Results for Models Without Abstention
E Learned Label Aggregator Explanation Weight
Here, we show ExEnt-FT and TALC experiment re- sults without abstention different adaptation size for each 6 tasks from ILUES-Real in Figure 11.
We analyze the learned values of the weights in the label aggregator, f interpret the contribution each explanation towards the final prediction of TALC at an adaptation ratio of 1.0. First, we cal- culate the accuracy of each individual explanation of task by using it with ExEnt for classification on the entire test set. These individual explanation accuracies serve a proxy for their relative quality. The visualization for the 6 datasets' learned expla- nation weights of label aggregators and the learned explanation weight trends are shown Figure
H Few Shot Learning
Here, run few-shot supervised version of ExEnt. We fine-tune the ExEnt model using k samples with gold labels from the evaluation tasks, where k 4, 8, 16, 12. We report the results in Table 7. We observed that the test accuracy of this few-shot trained ExEnt is better than TALC. The performance with a few-shot model is better as the gold labels are quite different from noisy ag- gregated labels used by TALC for adaptation. We observed huge label imbalance in the intermedi- ate ExEnt model that results in lower accuracy for both TALC and ExEnt-FT, both of which leverage ExEnt 's predictions as noisy pseudo labels.
Here, we also show the average learned explana- tion weight for explanations with/without quanti- fiers and the average learned explanation weight for explanations with/without conjunctions in Table 6. Quantifiers are words like 'always' and "ssually'. Conjunctions are words like 'and' and 'or'. We use the same quantifier and conjunction words follow- ing R. Menon et al. (2022)
Dataset
A:
k
k
banknote
47.3(2.))
51.2(
34.9
53.6(a.)
authentication
Dataset
Quanttifeer No Quantifier
Conjunction
No Conjonction
tic-tac-toe
55.6
59.1
63.7
64.2
endgame
banknote
autheniication
61.7
52.6
68. 68.1
69.422.)
tic-tac-tee
0.20
0.12
0.16
evaluation
endganee
contraceptive 34.4(4.)
evaliation
car
0.01
0.18
0.15
0.18
chuice
32.9(44)
36.8(o.))
40.0(2..)
contncepptive
choice
0.15
0.21
0.14
0.22
indian-liver 66.9(2.))
paticnt travel
67.4
63.8(a.4)
67.5
indian- liver potient
0.51
0:33
0.57
53.9
57.0
53.9
62.5
trwel
insurance
0.18
0.18
Average
53.3
53.4
56.9
59.5
Average
0.14
0.22
0.21
0.24
Table 7: Few-shot fine-tuning with ExEnt. We report the mean and standard deviation for the accuracy across three runs using different seeds.
Table 6: The average learned explanation weight for explanations w/wo quantifiers and explanations w/wo conjunctions of label aggregators for each task. Empty values in the table indicate that the linguistic element was absent in the explanations for the corresponding dataset.
We would like to emphasize that few-shot su- pervised model is not an ideal bascline for our framework. This is because TALC is designed work as an unsupervised approach for test-time adaptation. Hence, few-shot fine-tuning would represent an upper bound for TALC. Alternatively, few-shot finetuning can be treated as a comple-
F Individual Explanation for Each Task
We show all the available natural language expla- nations for the six CLUES-Read dataset we use in

---

banénot eveu ihentiottinnn Label Aggragator Lewrned Explanat Weight
too endgame Aggregator Explan ation Weight
weg
0.20
benknott-auuhheniiation
tic-tac-toeeendgame
carluvalatioon Label Aggregator Learned Explanation Weight
contraceptive- Apgimgator Explanation
implenaiin
explanation wight
0.1
0.2
0.3
0.6
0.5
0.6
0:30
2.20
0.30
(c) car-evaluation
(d) contraccpiiv--mechnndchoiiii
indlan-liver.aatienn Apgrogator Explanation Weight
Ira Travellissurncee Label Aggregator Learned Explanation Weight
wignt
weight
(c) linn----eee---ieenn
(f) travel-insurance
Figure 8: Learned label aggregator accuracy factor for the CLUES-Reaa evaluation datasets used in our work.
mentary approach TALC and can be paired with TALC for real-world time adaptation scenarios. Combining these two paradigms for improved test time adaptation is an interesting direction for future work and is beyond the scope of this paper.
Ranking by Perplexity
We show the results of the ablations studies de- scribed in Section 4.4 ranking by perplexity here in Figure 12 to 14.
For the setting where add one low-quality ex- planation a set of high-quality explanation, when ranking by perplexity, the average performance in- creases by 0.6%, caused by the performance in- crease of the the other tasks' performance either decreases or stays the same. According to Table 8, the added explanation for highest perplexity and the second highest accuracy, suggesting the accuracy metric may have stronger impact the performance than perplexity does.
task while
has the

---

banknote-authenticatiin
tic game
Accuracy
Accuracy
0.05
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
Leamed Explanation Weight
Learned Explanation Weight
(a) Pearson correlation coefficient -0.12.
(b) Pearson correlation coefficient 0.12.
car-evaluation
contracepive--ethoodcciiie
100
Accuracy
Accuracy
40
0.1
0.2
0.3
0.4
0.5
0.6
0.05
0.10
0.15
0,20
0.25
0,30
Leamed Explanation Weight
Learned Explanation Weight
(c) correlation coefficient 0.23.
(d) Pearson correlation coefficient -0.21.
indian-liver-patient
travel-insurance
100
100
80
80
Accuracy
Accuracy
0.3
0.4
0.5
0.6
0.7
0.05 0.10 0.15
0.20
0.25 0.30 0.35 0.4a
Learned Explanation Weight
Accuracy Factor Weight
Pearson correlation coefficient -0.68.
Pearson correlation coefficient 0.16.
Figure 9: Relationship between learned accuracy factor weight and the corresponding explanations' accuracy for (a) Lanknote-autbenticaton.. (b) tie-tactoo-enndgaee, (c) car-evaluation, (d) cnnrracetive-mettoo--hooii indian-liver-patien,,, and (f) travel-insurance.

---

Normalized perplexity accuracy for all explanations
Unnormalized perplexity accuracy explanations
80
80
Accuracy
0.6
Accuracy
20
0.0
0.2
0.8
1000
2000
3000
4000
5000
Perplexity
Perplexity
(a) Pearson correlation coefficient 0.18.
(b) Pearson correlation coefficient 0.15.
Figure 10: Relationship between perplexity and accuracy of explanations: Normalized, (b) Un-normalized
banknote-authetiction
lie-tac-toe-nndamme
car-evaluation
Apcuracy
Accuracy
0.2
Adaptation Ratio
Adaptation Ratio
Adaptation Ratio
conflaeepive--mettoo-chhoeee
indian-liver-patien!
travel-insurance
Accuracy
Accurscy
Adaptation Ratio
Adaptation Ratio
Adaptation Ratio
ExEnt-FT
TALC
Figure 11: Performance ExEnt-FT and TALC with different adaptation ratios for each of the 6 evaluation tasks.

---

No. EXPLANATION
ACCURACY
PERPLEXITY
If the variance of the note is a negative number, it's more likely to be an original note.
56.7%
25.05
If the kurtosis of the bill is a negative number, it's more likely to be fake note.
41.1%
31.82
A fake banknote has a variance level over 4.0.
46.2%
177.85
Notes that have a negative Variance level will also have a negative Entropy level.
45.1%
39.02
Notes that have a positive Skewness level will have negative Kurtosis level.
45.8%
61.06
Variance above 4 leads the results of FAKE.
43.3%
160.22
6
Below 3,80 skewness leads to the original.
56.0%
548.93
Variance above I 1.00 leads to the FAKE.
53.1 %
105.15
Below the 3.75 Sketwness leads to the ORIGINAL.
54.2%
410.40
Entropy is low value so it is fake.
49.1%
160.25
10
Kurtosis is high value it is original.
54.9%
346.58
Table 8: All explanations for banknote-authenticatinn task used in this paper.
No.
EXPLANATION
ACCURACY PERPLEXITY
The game usually not over yet when there are at least two blank squares.
34.4%
56.82
The game usually over when the players have taken all the middle squares.
42.7%
51.84
The top middle square results in winning.
33.3%
800.38
The bottom middle square results in winning.
32.3%
1132.84
4
When the middle-right- square is left blank, x is less likely to win.
32.3%
94.01
Whoever marks the top-left-square is unlikely to win.
38.5%
78.83
The player who moves second much more likely to lose if they place in the middle-left-square.
37.0%
45.58
The middle-right-sqquare the square that most likely go unused during game.
35.9%
36.20
Top left square X indicates the Negative group.
61.5%
969.94
Without b categories in middle middle square comes under the
32.3%
2158.20
Positive group.
10
An O in both the top-left and bottom-right is likely to be posi- tive.
39.6%
24.43
11
A blank in the middle-right will likely lead to negative.
65.1 %
139.76
Table All explanations for ticttac-toe-endgame task used this paper.

---

No.
EXPLANATION
ACCURACY
PERPLEXITY
0
If safety is high, then the car will not be unacceptable.
22.3%
56.95
maintemance cost medium, then the car will not be umac-
42.8%
121.90
ceptable.
A capicity for 4 or more persons makes the vehicle acceptable for resale.
9.0%
115.36
High safety ratings generally make vehicles acceptable for resalc.
9.8%
158.10
A low buying cost a good indicator of vehicle being accept- able for resale.
8.4%
55.96
Most people having the passenger capacity of 4 more have good acceptability for car resale.
18.2%
117.22
6 Cars with low buying and maintenance cost are highly accept- able for resale
5.5%
436.58
Cars with higher safety and capacity are highly acceptable for resale.
12.4%
156.38
8 Cars with higher safety and medium luggage boot size are highly acceptable for resale.
4.0%
323.92
Table 10: All explanations for car-evaluation task used in this paper.

---

No.
EXPLANATION
ACCURACY
PERPLEXITY
A husband's s education has high chance of long-term 44.7% traceptive being used.
85.88
A wife's education level usually determines if long term contraceptive used.
46.8%
166.37
2
Women with low education are more likely to use short-term
39.3%
12.18
contraception.
Women with high education age 40 and under are more likely to use long-term contraception.
43.4%
22.01
4 Women with two or more children used short and long term methods.
42.7%
47.27
The least educated women either used short-term method or didn't use any contraceptive method.
24.1%
72.05
the wife's education is high, then the contraceptive method used is long-term.
41.0%
44.04
If the wife's education not high, then the contraceptive method used is no-use or short-term.
27.5%
41,12
If the wife's education is not high, then the contraceptive method used is not long-term.
23.4%
31.23
If the wife's education is high, then the contraceptive method is short-term.
38.6%
37.03
Table 11: All explanations for wwwpecceppiie-metth----ehhee task used in this paper.
No.
EXPLANATION
ACCURACY
PERPLEXITY
The SGPT High percentage so the liver patient was yes.
49.6%
4184.78
The SGPT Low percentage so the liver patient was no.
52.2%
4891.02
Patients over the age of forty are liver patients.
42.6%
54.92
Patients who has SGOT range greater than forty are liver pa-
42.6%
373.03
tients.
Decreased SGPT Value ensures liver patient.
52.2%
4404.52
5 Age group above 40 ensures liver patient.
37.4%
4994.60
Some people have more age and have the SGOT and they are liver patient.
54.8%
272.55
Some people have minimum age and they are liver patient. They are somewhat accurate.
50.4%
226.11 11
Table 12: All explanations for indian didiaan-iver-patienn task used in this paper.

---

No.
EXPLANATION
ACCURACY
PERPLEXITY
Frequent flyers with an annual income over million usually take travel insur- ance.
33.2%
104.36
People with an unnual income over million and under 30 years old usually take travel insurance.
30.4%
53,40
Travelers who and older take traveler insurance.
34.4%
105.36
3 Travelers who have not traveled abroad before are more likely to take traveler
34.2%
22.74
insurance.
4
Frequent flyer travelers with an annual income above million frequently take truvel insurance.
31.2%
121.70
5
Travelers older than 25 years old and with an income below million do not usually take travel insurance.
42.5%
27.45
Most passengers who are not frequent fliers do not use travel insurance.
65.8% 33.28
Most passengers who have traveled ahroad do travel insurance.
77.1%
42.87
8
Most passenger with income higher than 100,000 use travel insurance.
27.9%
131.69
9 People who have never traveled abroad before are more likely to have taken
34.2%
15.95
travel insurance.
People with an annual income below 1,000,000 are less likely have traveled
23.15 1
8.36
ahroad than those with annual incomes ahove 1,000,000.
People with an annual income above 1,000,000 are more likely to have taken travel insurance.
39.9%
26.63
More frequent flyers
travel insurance.
34.9%
334.32
13 More people that travel abroad have taken travel insurance.
34.2%
114.91
People who are non-frequent flyers and are college graduates are less likely to get travel insurance.
27.4%
22.28
People who make a million or more and are frequent fliers are more likely to get travel insurance.
32.4%
22.73
Most people didn't travel abroad before had taken Insurance.
34.2%
84.50
17 Most college graduate have taken Travel Insurance.
64.8%
780.98
18 Those who are college graduates and in their 20s are somewhat likely to pur-
22.9%
28.62
chase travel insurance.
Those who have never travelled abroad and are not frequent flyers often do not purchase travel insurance.
67.1%%
23.85
Travelled Abroad Before "No" categories indicates the "No" Travel Insurance Taken.
24.4%
490.45
21
College Graduate "Yes" categories to the "Yes" Travel Insurance Taken.
30.9%
678.44
22
Frequent Flyer "No" indicates the "No" Travel Insurance Taken.
53.8%
194.31
23
Annual income categories above 1049999 indicates the "Yes" Travel Insurance Taken.
43.5%
2291.89
24
Most college graduates that make more than 1000000 annually have taken travel
37.2%
231.38
insurance.
Most college graduates from 26 to 34 have taken travel insurance.
43.2%
185.20
26
About have of college graduates that have travelled abroad before have taken travel insurance.
34.2%
186.52
27
Frequently flyer "No" categories indicates the "No" travel insurance taken
65.6%
586.77
group.
28
Annual income above 1049999 indicate the "Yes" travel insurance taken group. 37.4%
769.28
29
Frequent flyer "No" categories indicates the "No" travel insurance taken group. 29.6%
375.25
30
Annual income above 1049999 indicates the "Yes" travel insurance taken group. 42.5%
695.34
Table 13: All explanations for traveliinsurance task used in this paper.

---

TALC: Removing the explanation lowest perplexity
Accuracy (96)
60
20
hanknote aud nuthnticatiio
Di Uctaa-106 andgamei
entracsptive evaluation
Inttan
travel Insurance
Averape
patiant
cheice
w/ lowest-penn perrlet-erplexitt explanation
w/o pereet-peppexxtty explanation
When ranking the explanations by their indi- vidual perplexity, removing the best explanation leads 1.0% drop in performance average.
Figure 12:
TALC: Adding explanation with highest perplexity
Accuracy (%)
60
40
banimote authetkkatoon iitti
endgame
tac--tol tol
evelsaten
contra contracaptivn scppppiv
Indian method choice
patlent
Insurence
ttavel
Average
w/o highest- perplexity perpletttyyy explanation
w/ highest- perplexity explanation
Figure 13: Comparison of TALC's performance before and after adding a lua-quality explanation to a set of high-quality explanations. On average, the performance increases by 0.6% when ranking by perplexity.
TAL C: Replacing low-perplexity explanations with malicious explanations
Accuracy
40
20
banknote Axthentication
th Bh-tac-tte 106
endgamm
car Mn
contracaptive method cholice
Indkan
trawel mnsunaicc
Average
potient
w/o ma milicuuus explanation
w/ malicious explanation
Figure 14: Comparison of TALC's performance before and after replacing qoul-quality explanations with ma- licious explanations. On average, TALC's performance drops by 5.6% as good explanations are replaced by malicious explanations.


Reviews:
[{"review_id": "SeUIoGTbGH", "reviewer": "Reviewer_U5MY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our proposed method novel and applicable to real world problems. We additionally thank the reviewer for considering our paper well-written and our analysis comprehensive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is novel and applicable in the real world by tackling the problem of adaptability of language-guided classifiers on unseen tasks.\n- The authors compare their work against prior work, showcase strong improvement on a given dataset and analyse the pros and cons of their method.", "reasons_to_reject": "In general the paper is well written and includes a comprehensive analysis. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work introduces a framework to adapt language guided classifiers on new tasks. Given explanations from multiple teachers and unlabeled test examples and leveraging data programming the proposed method, TALC, showcases competitive performance when compared to prior work on real world classification tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZxCCDg5LoZ", "reviewer": "Reviewer_7rv9", "summary": "", "questions": "I will just copy here the question from above:\n\n1. Why is it sensible to model the joint distribution P(X, E, Y) with Eq. 1?\n\n2. Why to use EM as opposite to variational inference? \n\n3. What are the disadvantages of using MAP estimates in this problem?\n\n4. Why didn't you consider a different baseline (i.e. one not based on the ExEnt model)?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review! We are encouraged by the fact that the reviewer recognizes the relevance of the issue we address and how it relates to real-world situations. Additionally, we appreciate the acknowledgment of our sensitivity analysis. Below we address some of the concerns raised by the reviewer:\n\n* **Why is it sensible to model P(X, E, Y) ?** The observations in this case are the unlabeled instances X and the explanations E. Thus, we would like to infer the latent labels Y that would maximize the probability of these observations, i.e. P(X, E). For this, the natural choice would be to model the joint probability P(X, E, Y) and marginalize out Y. \n\n* **Why use EM over VI?** We need to not only infer the latent labels Y for the unlabeled data, but also learn the model parameters for the graphical aggregator (which is why we need both the inference in the E step and the learning in the M step). If the reviewer’s question is why we don’t use a variational inference approach in the E-step (rather than a sampling-based approach), this was simply because of ease of implementation (the Gibbs sampling updates for this specific case are really simple). It is possible that a VI based approach might work better. \n\n* **Disadvantages of MAP estimates**: MAP normally works better when datasets are small. In our case, the test set we used from CLUES normally has around 200 to 300 examples, which makes MAP suitable for the tasks. \nAs for situations where MAP can be disadvantageous for our particular application, MAP is sensitive to the outliers. Hence, for lower adaptation ratios, where we do not work with the entire test set, if the adaptation set and the held-out set come from disjoint distributions (through random seeding), the MAP estimates can be substantially affected.\n\n* **Other baselines for test-time adaptation**: \nWe would like to point out that our self-training baseline (namely, ExEnt-FT) already subsumes the entropy minimization objective proposed for test-time adaptation in Wang et al. (2021). We will make this clearer in our final version.\\\n\\\nApart from this, another approach for test-time adaptation is pretraining the model on the target domain/task data using a task-independent objective (e.g., masked language modeling). However, this approach is not suitable for our setting as the target tasks are also structured (tabular in nature) and masked language modeling is not a natural task in this format. We are not aware of any other baseline strategies to compare against. \\\n\\\nWhy do we compare against baselines built with ExEnt ? – Our evaluation focuses on showing the improvement of using TALC by forming a direct comparison to previous work. For this, ExEnt is the most natural choice since it the best current model pre-trained on the structured (tabular) training data of the CLUES dataset.\n\n\n\n* **Reproducibility Concern**: We would like to emphasize that all data as well as base models used in the paper are public. All hyperparameters for our main experiments have been provided in the Appendix as well. We will release our code on first publication, to aid reproducibility and further research in this direction.\n\n* **Typos**: Thanks for pointing out the typos. We will correct them in the final version.\n* **More details about EM algorithm**: As suggested we will provide more information about the EM algorithm (including the update equations) in the Appendix of the final version.\n\n**Reference**:\nWang et al. 2021: Tent: Fully Test-time Adaptation by Entropy Minimization, ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles the problem of test-time adaptation for unlabeled classification which, as pointed out by the authors, aligns well with real-world scenarios.\n\n- According to the authors, their methodology represents a first application of the data programming paradigm to test-time adaptation for unlabeled classification.\n\n- The authors studied the sensibility of their methodology to plausible perturbations.", "reasons_to_reject": "- The main components of the methodology proposed by the authors could definitely be explained better. It feels the authors assume every reader is well acquainted with both, the data programming paradigm of Ratner et al. (2016) and the ExEnt model of Menon et al. (2022). The label aggregator, for example, represents the heart of the method. Yet it's not very easy to follow the description provided by the authors. \n\nWhy is it sensible to model the joint distribution P(X, E, Y) with Eq. 1?; Why to use EM as opposite to variational inference?; What are the disadvantages of using MAP estimates in this problem?\n\nNone of these points are discussed. \n\nCould you include, be in the Appendix or in Algorithm 1, the update EM equations?\n\n- The baselines are all modifications of the language-guided classifier of Menon et al. (2022). The proposed methodology also leverages this language-guided classifier. It's therefore not difficult to believe that the proposed methodology outperforms the baselines, since the former builds on the latter. Why didn't you consider a different baseline?\n\n[Note: I am not knowledgeable about the problem of test-time adaptation for unlabeled classification.  I understand that it might be that no other method has been developed for this problem in the past, and hence that comparing with another baseline is difficult. It'd be worth elaborating on this, if this were to be the case]\n\n- No code/repo was provided (or at least I can't find it). This, together with the first point above, makes it very difficult to both understand the method and reproduce any of the reported results.", "questions_to_authors": "I will just copy here the question from above:\n\n1. Why is it sensible to model the joint distribution P(X, E, Y) with Eq. 1?\n\n2. Why to use EM as opposite to variational inference? \n\n3. What are the disadvantages of using MAP estimates in this problem?\n\n4. Why didn't you consider a different baseline (i.e. one not based on the ExEnt model)?", "paper_topic_and_main_contributions": "This paper proposes a framework for classification of unlabeled data, tailored to an scenario in which both (i) a set of natural language explanations from multiple sources and (ii) a set of unlabeled data examples are available. Specifically, it makes use of the data programming paradigm of Ratner et al. (2016) to train a function which aggregates the explanation-dependent predictions of a base language-guided classifier. The latter is chosen to be the ExEnt model of Menon et al. (2022).\n\nThe authors empirically showed that their method generally outperforms the baselines. The authors also included an interesting analysis of the sensibility of their method to the number and quality of the available explanations. \n\n\n*Links to references*\n\n- Ratner et al (2016): https://proceedings.neurips.cc/paper/2016/file/6709e8d64a5f47269ed5cea9f625f7ab-Paper.pdf\n\n- Menon et al. (2022): https://aclanthology.org/2022.acl-long.451/", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Typo. Lines 342: aggregator **in** results\n\n- Typo. Lines 549: in **principal**\n \n- Could you include, be in the Appendix or in Algorithm 1, the update EM equations?"}, "extra_text": {}}, {"review_id": "NObIy9plj6", "reviewer": "Reviewer_382n", "summary": "", "questions": "* One hypothesis of TALC is that it assigns \"higher weightage to high-quality explanations\" (Line 344). However in Figure 9 it seems that the learned weight is not having strong positive correlation with the explanation's quality (reflected by accuracy). Could you please elaborate on this observation?\n* Line 375: Is TALC-A another method or is it the same as the TALC method in Table 1? From my understanding TALC already takes abstention into account.\n* Table 2: What's the reasoning behind reporting \"mean and standard deviation across 9 different adaptation ratios\"?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review! We are encouraged by the fact that the reviewer acknowledges the importance of the research problem addressed in our work and the superior performance of our framework against the baselines. Below we address some of the concerns raised by the reviewer:\n\n* **Technical contribution**: Traditional data programming techniques require access to labeling functions (essentially semantic parsers that are either learned or provided as inputs). On the other hand, TALC does not require any such labeling function/semantic parsers. TALC focuses on adapting *any* language guided classifier during test-time. The underlying language guided classifier plays the role of labeling function by assigning pseudo labels while being more generalizable and robust than traditional labeling functions (since the language-guided classifier can use natural language explanations directly rather than parsing them into a specific format).\\\nOur framework, TALC, provides the first proof-of-concept in leveraging data programming to adapt any language guided classifier during test-time. TALC is agnostic to the choice of language-guided classifier or the label aggregation algorithm. Through specific choices of using ExEnt and Metal as the classifier and label aggregation algorithm we empirically show the efficacy of TALC. Our experiments also highlight the robustness of TALC with the number and quality of explanations and with test-time adaptation set size.\nExploring other alternatives of classifiers and label aggregation techniques is beyond the scope of this work and can be a separate research avenue in itself, which we leave for future work. \n\n\n* **Baselines with Large Language Models**: Larger language models can certainly be applied as the language-guided classifier of the framework (i.e, one could use any LLM in place of ExEnt under the same TALC framework). This flexibility of choosing different models as the underlying language-guided classifiers is an advantage of the TALC framework that we advocate. \n\n* **Takeaways from Analysis**: Thanks for the suggestion! We summarize the takeaways from Section 4.4 more concisely below and will ensure to add these takeaways to the final version of the draft:\n    - If we have enough compute, α=1 is the best adaptation ratio to utilize for test-time adaptation, i.e. utilize the entire set of available test data to adapt. \n    - Given the sensitivity of TALC to the number of explanations, it is best to always utilize all available explanations for a task during adaptation (Figure 3). If you have a metric (such as, perplexity of the explanation) to compute the quality of explanations, however, utilizing the top 20% of the available explanations as per the metric is also a viable option to reduce computation (Figure 4).\n    - From Figure 6, if the user is privy to the fact that an explanation is malicious, then they should remove such explanations as the performance of the adapted classifier can be drastically impacted by such explanations.\n* **TALC assigns higher weightage to higher-quality explanations, but there is no strong positive correlation**:  We would like to clarify that we do not mention the association between explanation accuracy and weightage to be *strong* positive correlation in the draft. As rightly pointed out by the reviewer, these weights indicate a mild positive correlation. We conjecture that this lack of *strong* correlation could be attributed to factors other than accuracy, such as perplexity of explanations, that also affects the weightage assigned by the label aggregator of TALC. As noted in R. Menon et al. (2022), ExEnt (which is the underlying language-guided classifier of TALC) does not handle explanations with conjunctions, negations, or quantifiers well and hence that potentially affects the explanation rankings as well.\n\n* **TALC-A v.s TALC**: While in principle TALC can handle abstention, the usage of ExEnt as the language-guided classifier was a limitation as ExEnt was not trained to abstain from prediction. Hence, to analyze efficacy of TALC under abstention, we consider a modified version of ExEnt, namely ExEnt-A which can now abstain from prediction. We do this by re-directing the neutral logit from NLI step to a separate “abstain” label. More details about this can be found in L#356-383. The version of TALC which uses this modified language-guided classifier, ExEnt-A, is referred as TALC-A in our paper. \n\n* **Why report mean, std over different adaptation ratios?** The aim of Table 2 is to analyze the performance of different approaches when models can abstain from classification. In table 2, we provide a concise statistic that captures performance of TALC and baselines across a range of adaptation set sizes by computing the mean and standard deviation over the nine adaptation ratios (thus reflecting small to large sizes of adaptation set).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Training language-guided classifiers is an important research topic. It is a unique learning paradigm as opposed to supervised learning and is very interesting given the rise of large language models.\n* TALC has strong empirical results, outperforming ExEnt (previous state-of-the-art method) by 3.3% on average.\n* Extensive analysis on the proposed TALC framework under different settings.", "reasons_to_reject": "* Need more justification regarding technical contributions and novelty: As mentioned in the related work section (data programming) and method introduction (Line 205,262), the work is closely related to existing data programming technique such as Snorkel-MeTaL. MeTaL proposes to \"learn a re-weighted model of labeling functions\". For now I cannot clearly distinguish the difference between TALC and MeTaL and would request more justification on this from the authors.\n* Need baselines with larger language models: As stated in the abstract, large models such as GPT-3 are good at learning from explanations and instructions. It would be extremely helpful to provide the baselines on these larger models for the language-guided classifier task. In addition, since TALC is an flexible framework (Line 543), would it be possible to apply TALC when these larger models are used to produce the $M\\in\\mathbb{R}^{n\\times m}$ matrix?\n* Need direct/clear takeaways from the analysis. Sec 4.4 provides comprehensive analysis on TALC's performance under different scenarios. I appreciate these extensive efforts. However, it is not very clear how these analysis can be used to guide the application of TALC. For example, should one always use adaptation ratio $\\alpha=1$? Given Figure 3, how many explanations is sufficient for TALC to work well?", "questions_to_authors": "* One hypothesis of TALC is that it assigns \"higher weightage to high-quality explanations\" (Line 344). However in Figure 9 it seems that the learned weight is not having strong positive correlation with the explanation's quality (reflected by accuracy). Could you please elaborate on this observation?\n* Line 375: Is TALC-A another method or is it the same as the TALC method in Table 1? From my understanding TALC already takes abstention into account.\n* Table 2: What's the reasoning behind reporting \"mean and standard deviation across 9 different adaptation ratios\"?", "paper_topic_and_main_contributions": "Language-guided classifiers refers to the problem of classifying $n$ test examples when given $m$ natural language explanations describing the classification task. \nThis paper proposes TALC: Test-time Adaptation of Language-guided classifiers, which extends from prior work ExEnt (R. Menon et al., 2022) and includes a new label aggregation component for test time adaptation.\nIn ExEnt, the predictions are made by mean-pooling results of running $m$ NLI predictions (whether each of the $m$ explanation entails the classification result).\nIn TALC, the label aggregator has learnable parameters $w$ which weight the predictions from the $m$ explanations. $w$ is learned on unlabeled test examples using EM algorithm.\nOn 6 tasks in the CLUES benchmark, TALC achieves strong empirical performance (outperforms ExEnt by 3.3%). Additional analysis is done regarding abstention, effect of adaptation-set size, number of explanations and quality of explanations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 252: It may be helpful to provide more description on the EM algorithm used to learn $w$."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an empirically sound work which proposes an approach for test-time adaptation of language-guided classifiers. In contrast to prior works, this paper uses EM to learn some parameters on unlabeled test examples to enable better aggregation. During the discussion period there were some questions as to (1) novelty compared to existing data-programming methods and (2) whether this method would extend to LLM-based models. The authors ran extensive experiments during the rebuttal period to answer (2), but questions regarding the methodological contribution above existing work remain.
Decision: Accept-Findings