kNN-CM: A Non-parametric Inference-Phase Adaptation of Parametric Text Classifiers
Authors: Rishabh Bhardwaj
Yingting Li
Navonil Majumder
Bo Cheng
Soujanya Poria
Venue: EMNLP 2023 Findings (2023)
Keywords: nearest neighbors
text classification
semi-parametric models
non-parametric models
kNN-CM
kNN-LM

Abstract: Semi-parametric models exhibit the properties of both parametric and non-parametric modeling and have been shown to be effective in the next-word prediction language modeling task. However, there is a lack of studies on the text-discriminating properties of such models. We propose an inference-phase approach---\textit{k}-Nearest Neighbor Classification Model (\textit{k}NN-CM)---that enhances the capacity of a pre-trained parametric text classifier by incorporating a simple neighborhood search through the representation space of (memorized) training samples. The final class prediction of \textit{k}NN-CM is based on the convex combination of probabilities obtained from \textit{k}NN search and prediction of the classifier. Our experiments show consistent performance improvements on eight SuperGLUE tasks, three adversarial natural language inference (ANLI) datasets, 11 question-answering (QA) datasets, and two sentiment classification datasets.

ANN-CM: A Non-parametric Inference-Phase Adaptation of Parametric Text Classifiers
Rishabh Bhardwaj Yingting Li Li Navonil Majumder
Bo Cheng Soujanya Poria
Singapore University of Technology and Design, Singapore Beijing University of Posts and Telecommunications
rishabh_ haiaabbhbbaarddwaeemmmaaaalll sutd. edu. sg, cindyytingebupt edu. cn navonil_man navomil_majjumderssutt edu. sg, chengboebupt. edu. cn sporia@sute edu.sg
Abstract
of parameters in function that defines suppos- edly linear relationship between (independent) in- put variables and (dependent) output variables. As a complex composition of many linear regressions, many neural architectures, such as Transformer- based models (Vaswani et al., 2017), can thus be classified as purely parametric.
aani-parametri models exhibit the properties both parametric and non-parametri mod- eling and have been shown to be effective in the next-word prediction language modeling task. However, there lack of studies on the text-discriminating properties of such models. We propose an inference- phase approach- k- Nearest Neighbor Classification Model (KNN- Cha enhances the capacity of pre- trained parametric text classifier by incorporat- ing simple neighborhood search through the representation space of (memorized) training samples. The final class prediction ANN- CM is based the convex combination of probabilities obtained from ANN search and prediction of the classifier. Our experiments show consistent performance improvements on eight SuperGLUU tasks, three adversarial nat- ural language inference (ANLI) datasets, 11 question answering (QA) datasets, and two sen- timent classification datasets. The source code of the proposed approach is available at https: //gi thub. com/ Bhardwaj dwaadddwashhaeeee Rrihabh//KNN-Ot
However, there has been little research on the utility of non non-parametric models for NLP. In con- trast to parametric models which need predefined function, such linear model, non-parametric models seek training data help define the func- tion form itself. Thus, they provide flexibility fit a wide range of possible shapes of ground truth function. A widely known non-parametric model is the k-nearest neighbor (kNN) where inference on test samples are drawn from the information provided by the neighborhood formed by train set samples (Fix and Hodges. 1989).
A ANN model provides memorization capabili- ties and captures rare patterns from the training set that otherwise are ignored by parametric model (as studied by Khandelwal et al. (2019)). Language models (LMs) with non- non-parametric properties have shown impressive gains in next-word prediction tasks (Yogatama et al., 2021; Bhardwaj et al., 2022; He al., 2021; Khandelwal et al., 2019). Addition- ally, these models do not need explicit parameter learning via optimization, thus cutting the model training time completely-the lack of such charac- teristics in purely parametric model motivates the proposed approach.
1 Introduction
The recent advancements in Natural Language Pro- cessing (NLP) have largely been attributed to the learning of contextual representations of text from large language models acting backbone. Most of these language models, such as versions of BERT (Devlin et al., 2018), GPT (Radford al., 2018), T5 (Raffel et al., 2020), are parametric, i.e., they encode information required to solve task purely in its parameters.
This work explores the importance of querying neighbors to solve classification tasks in the text domain. We hypothesize that underlying language model representations have a high-dimensional spatial proximity relation between input instances which can be leveraged enhance prediction performance-
A parametric model, irrespective of the size of the dataset, assumes that the output variables are dependent on the input variables through pre- defined class of functions. The exact function is ascertained by learning its fixed set of parameters. For instance, a linear regression model fits a set
the capacity of the classifier. Hence, we propose a paramerracttiic model RNN- CM (k Nearest Neighbor Classification Model)
Equal Contribution.

---

which constitutes parametric classifier and non- parametric memory (i.e. datastore) for neighbor- hood retrieval. In reinforcement learning (RL), classifiers are often employed as tools to aid policy learning or state representation. They can help es- timate the quality of different actions or policies, and classify text into different categories, thus pos- sessing high utility for the recent paradigm shifts in generative models (von Werra al., 2020).
I-SE pe Kna
FSNE SHat
Figure 1: Motivation behind nearest neighborhood ap- proach. The left and right figures show the pre-final layer and final layer mapping of ANLI training samples. The red crosses represent test samples where CM predic- tion is incorrect, which are corrected after incorporating predictions from kNN.
Contributions, We propose an inference phase semi-parametric modeling approach ANN-CM that enhances the capacity of a given parametric clas- sifier model by incorporating an external memory datastore. In the inference phase, kNN-CM per- forms k-neighborhood search through the datas- tore and merges the neighborhood predictions with the prediction obtained from the parametric classi- fier. Since the expansion of the CM to ANN-CM happens in the inference phase, it allows one to enhance the capacity of most of the existing pre- trained neural classifiers. By performing an exten- sive set of experiments, we demonstrate the impor- tance of neighborhood search through the mem- orized samples on eight SuperGLUU tasks, three NLI datasets, 11 QA tasks, and two aspect-based sentiment classification tasks. We also show how the semi semi-parametric method can still outperform CM in out-of-domain scenarios. Furthermore, we test ANN-CM by tasking it under various cases of domain adaptation. Since kNN-CM introduces pre- diction latency compared with CM, we demonstrate how one employ an entropy- based divergence measure filter out the samples that use KNN re- trieval facility. Additionally, we illustrate the im- portance of memorization the low-resource sce- nario. In the end, we point out potential extensions of the proposed approach in conversation modeling and continual learning.
formed image classification without class supervi- sion but considering every instance as class.
Recommendation systems. For session-based recommendation (based on current user interac- tion with the system), Kamehkhosh et al. (2017); Jannach and Ludewig (2017) showed that a neighborhood basddd-bbddd--- model output performs GRU- neural model the studied tasks. ANN has been widely popular in collaborative filter- ing (memory-based) where the recommendation is done by the user or item neighborhood search (Su and Khoshgoftaar, 2009; Sarwar et al., 2001).
Language models. The advantage of querying nearest neighbors from set of pre-trained LM's representations of the training set (datastore) first observed by Khandelwal al. (2019) by proposing kNN-LM. This is followed by several works such as improving retrieval speed (He et al., 2021), ANN- LM adaptation (Bhardwaj et al., 2022), adaptive interpolation (Drozdov et al., 2022; Yo- gatama et al.. 2021), and masked language model- ing (Min et al., 2022).
To the best of our knowledge, this work is the first attempt extensively study the importance of neighborhood search for text classification enhance the capacity of parametric classifiers in the inference phase. Figure 1 gives an illustrative to motivate the idca.
2 Related Work
Computer vision. For the image captioning task, Karpathy and Fei-Fei (2015); Devlin et al. (2015) proposed nearest neighbor baselines where they as- sign an unseen sample the (consensus of) captions of the training set images closest to it. Wang et al. (2019b) studied the utility of (transformed) neigh- borhood while performing few-shot object classi- fication tasks. KNN has also been used to analyze learned image representations (Wallace and Hari- haran, 2020) as well as to classify images (Zhang et al., 2023). For instance, Wu et al. (2018) per-
Methodology
For a given domain, we obtain training data S== S==((11,1)) put text in instance space X' and yı is the label of its class defined in label space y A learning sys- tem, a classifier, comes up with a prediction rule (hypothesis) that maps the input from X' to a probability distribution over class labels in Y
(IN, yN)} where x, denotes in-

---

Classifier (CM). Without the loss of general- ity, we consider CM constitutes backbone large language model and trainable classification head. During the pre-trainin treiiinin phase, the language model assumed to have learned semantic high- dimensional text representations that can be tuned to solve given downstream task. The input to CM is sequence of tokens {wi,.... Wn and output is probability distribution over the class la- bels. CM is tasked approximate the ground truth input-output mapping by learning parameters of a predefined function, i.c. the neural connections within CM. We denote the task-specific trained parametric predictor by l.M-
nearest neighbor classifier highly depends on the effectiveness of its search space (datas- tore), which is defined by the vector represen- tations of text provided by CM.
Explicit control over a model capacity: Inte- grating ANN with CM provides explicit con- trol over the model's capacity. For instance, a change in the & value changes the model's 's bias and variance shown in the non-parametric estimator's study by Geman et al. (1992). Changing model's bias-variance characteris- ties directly affects the model's capacity to fit a wider class of functions".
KNN. We use well-known algorithm k- k-nearest neighbors for non-parametric modeling (Fix and Hodges, 1989). Using training samples of task, we construct a datastore D=(u(a;), yi M where v(z) denotes the high-dimensions vector embeddings of text x obtained from the classifier model. For a given classification task, an unseen test sample & is classified based on the nearest neighborhood training samples <xx). Let arg min; denote the index of k training samples that return least distance values from 2
We hypothesize that neighborhood search is important when the classifier is confused be- tween classes and prone to do mistakes around the decision boundary. We quantify this aspect and call model more confused if the classi- fier's output probabilities resemble uniform distribution. Thus, one can choose between CM and ANN-CM depending on the unseen sample under testing. We study this aspect in detail in Section
Next, we define the error made by hy over m test samples
K arg ming d(v(()) SE[N]
Hi E [m] arg max (xx + vill
p(y V) Eiex [y: y)
jEy
k
(3)
m
where d(.) denotes the distance function be- tween v(t) and v(z;) yı denotes the label of zyj Similar to Khandelwal al. (2019); Bhardwaj et al. (2022), use euclidean distance for d(-). Hence, we obtain a non-pannnn con-parametric hypothesis haNN. We define the eemi-parametric classifier model KNN- CM as a linear combination of the two probability distributions with coefficient > C [o,
where hi is probability assigned by the hypothesis to class j. is the cardinality of the set and [m] = (1....,m) Note that denotes the accuracy of the semi-parametric classifier.
Time and Space Complexity. Similarity search can be computationally expensive and introduce high memory requirements for each task, thus we use Faiss (Johnson et al., 2019) an efficient sim- ilarity search algorithm and clustering (indexing) algorithm of high dimensional data. The cluster- ing of similar vectors in high dimensions obviates the need to search through the whole training set (datastore). For small-scale datasets, we use In- dexFlatL2 which queries through all the vectors in the datastore. The complexity is thus (n) where n the number of elements in the datastore. For large-scale datastores, we use IndexFlatIVF to first
b À ĀENN (1 A) hcM-
There are several aspects of this formulation:
While performing ANN search, parametric classifier parameters are kept frozen.
Strong dependence of hANN on hcM: Unlike commonly used ensemble methods where the underlying classifiers undergo independent training and inference, the errors made by
Brooddyy, model with & = N will learn fewer patterns from data while model with A 1 can Ieam as many patterns as N. where N is the number of training samples.
Morcover, one can weight each prediction with exp( xxp(-d(v()), v(x))).

---

still fails to capture the nuance of the neighbor- hood precisely.
60
40
4.2 NLP datasets
20
We base our main experiments on the SuperGLUE benchmark and large variety of existing NLP datasets to solve NLI, Question Answering (QA), and Sentiment Classification.
-70
-60
a bench- mark dataset to evaluate a model on its language un- derstanding capabilities. BoolQ (Clark et al., 2019) is QA task where a yes/no question is asked on short passage. CB (De Marneffe et al., 2019) is textual entailment task where given a premise, the model is asked to predict how committed the author toward the truth of the (clause) hypothesis. COPA (Roemmele et al., 2011) is a causal reasoning task to identify the cause or effect of a premise from the set of given choices. Mul tiRC (Khashabi et al., 2018) is a multi-choice QA task where a question is asked about context paragraph and the answer choices are provided. ReCoRD (Zhang et al., 2018) is multi-choice QA task where, from passage, masked-out entity to be predicted from a set of entities. RTE (Haim al., 2006) is another textual entailment dataset with two classes, entailment and not entailment. Wic (Pilehvar and Collcho-Collados, 2018) is task of word sense disambiguation. Given two texts and a word appear- ing in both sentences, the task is to determine if the word is used in the same sense in both sentences. WSC (Levesque et al., 2012) is conference resolu- tion task where example consists of a pronoun and list of noun phrases, the task is to identify the correct pronoun referent.
SuperGLUE (Wang et al., 2019a).
-00
20
Figure 2: Error analysis of a neural classifier clus- tered synthesised data.
cluster the vectors in datastore and then perform an NN search in each cluster. The time complexity of this method is (n.m) where n. and denote the number of clusters and the average number of elements in clusters, respectively. The space com- plexity of IndexFlatL2 is (nds) and IndexFladVV is Ond, + md.), where d, denotes the dimension- ality of the vector the datastore. Contrary to non con-parametric, the time and space complexity of a parametric model, such as CM, is predefined and does not vary with the number of train samples.
4 Experiments
4.1 Toy dataset
We hypothesize that CM prone miss neigh- borhood cues from the training data. To test this, we set up a toy experiment on a neural network comprised of one hidden layer of 100 nodes acti- vated with ReLU. To test the capacity of this net- work, we synthesize dataset by randomly initial- izing 20 cluster centers {c: N~,N( 1.5), E 1. 1...20}. each of which constitutes 20 points {cij Cty = + Py; Py 1 1), E 1... 20), where cluster center and Pj are independently sampled for each of their dimensions. All the data points lie in the space of 1000 We randomly split the clusters into four classes. Figure? shows the 2- dimensional t-SNE plot of the generated data with samples shown in the same color belonging to the same class and vice versa. Circles represent sam- ples used to learn the network parameters, black dots denote the correctly classified test cases and the red squares denote the test samples incorrectly classified by the network. Red squares provide ev- idence for the hypothesis, i.c., while the model is able to identify correct clusters of several test cases,
BoolQ, COPA, COPA, Wic, WSC, and RTE are bi- nary classification tasks, CB three-class classifica- tion, Mul MultiRC and ReCoRD cast binary class classification where the correct choice (or entity) is labeled and incorrect is labeled 0.
ANLI (Nie et al., 2020). Adversarial Natural Language Inference is a large-scale benchmark NLI dataset constructed by an adversarial human- The dataset is subdivided into three datasets A1, A2, and A3 with increasing task dif- ficulty. ANLI aims solve textual entailment task where given a premise, the model is asked predict if a hypothesis entails, contradicts, or is neutral to the premise. We use ANLI to represent combination of A1, A2, and A3.

---

Question Answering. For QA tasks, we experi- ment on ten datasets: QASC (Question Answering via Sentence Composition) (Khot et al., 2020) is a fact retrieval from a large corpus to answer a question given eight choices, only one of which is correct. PIQA (Physical IQA) (Bisk et al., 2020) tests physical knowledge of language models by asking them to select the correct choice from the given two. SIQA (Social IQA) (Sap al., 2019) is common sense reasoning dataset for the context of social situations. Given social situation and three choices to select from, the task is to select the cor- rect choice. CQA (Commonsense(AA) (Talmor et al., 2019) is a commonsense QA based on ConceptNet knowledge (Speer et al., 2017). For a question, the task is to choose one of five given choices. CQA-2 (CommonsenseeAA 2.0) (Talmor al., 2021) is an- other recent commonsense QA dataset constructed with model-in-looy approach. It consists of com- monsense questions from various categories of rea- sons, with the answer being yes or no. SWAG and (H-SWAG) (Zellers et al., 2018) are datasets for grounded inference. Given an incomplete event description, the task is to find the correct ending from a set of four choices. CosmosQA (Huang et al., 2019) is dataset for commonsense-basede reading comprehension. The task is identify correct choice from the given four for the question asked about a paragraph. CICERO v1, v2 (Ghosal et al., 2022b; Shen et al., 2022) are dialogue QA dedi- cated datasets. Given a question about given ut- terance taken from a dialogue, the task is to choose the correct answer from the choices.
CM Setup. For all the tasks in the SuperGLUE benchmark, we utilize RoBERTa-base ((Liu et al., 2019)) as the backbone language model. Fol- lowing the success of parameter-efficient adapters (Houlsby et al., 2019), and their competitive per- formance with full-mode fine-tuning (Liu et al., 2022; Hou et al., 2022: Bhardwaj et al., 2022), we obtain a task-specific classifier (CM) by training adapter modules inserted in between LM layers and attaching a classification module (head) on top of the LM 4, All the tasks are formulated as a classification problem 5. We follow a similar setup for language inference tasks (ANLI) and sen- timent analysis tasks. For QA datasets, we use the DeBERTa-laree (He et al., 2020) based classifier. Following TEAM (Ghosal et al., 2022a) which has shown better than baseline performance numer- ous QA tasks, we formulate all the multi-choice QA tasks as binary classification where the cor- rect choices are labeled as I and incorrect choices are labeled as O. Therefore, in the training phase, the classifier model aims to minimize the binary- cross entropy objective function. During the in- ference phase, we select the choice with the maxi- mum class 1 probability score. Since our approach improves the model performance in the inference phase. liberate ourselves from classifier training by downloading the model checkpoints generously provided by Ghosal et al. (2022a). The classifica- tion head uses 
 from RoBERTa and [CLS] from DeBERTa (generally used as classification tokens).
kNN Setup. For each task under study, use the task-specific trained CM obtained via the method described above and construct a datastore using the train set samples. We obtain hidden representations of each sample by performing one forward pass through CM. For fast neighbor search and mak- ing the datastore memory-efficient, the obtained vectors are indexed using Faiss.
Aspect-Based Sentiment Classification. We also compare the proposed approach on two aspect- based sentiment classification datasets-Lapp and Res taurant. The datasets are a set of restau- rant and laptop reviews obtained from Pontiki et al. (2015, 2016)3. We convert the given review in the form W4 W2 ... <aspect term> ... Wn. where <.> encloses the aspect term for which the sentiment (posiivvehnggativeeeetaal) to be predicted.
Hyperparameters. Since the approach is applicable to the inference phase, the primary set of hyperparameters comes from KNN search. For each task, we find the best interpolation parameter À (Equation (2)) and the optimal number of neighbors to search & using the validation set, À E {0.001, 0.01, 0.1, 0.2, ... 0.8, 0.9, 0.99, 0.999) and k < (1,2, 4, 8, 16, 32, 64, 128, 256, 512).
4.3 Experimental Setup
The RNN-CM is an inference phase approach that does not require task-specific CM parameter tuning.
We either train a classifier or utilize existing pre- trained task task-specific classifiers to obtain a baseline CM on a given task.
"Trainable parameters are =x1.2% classifier parameters. Datasets downloaded from https: htt.//hgggggaaccccccc face. Tace.coeoo co/ gatasets/super_gluss
We utilize the data collected by Zhou et al. (2021)

---

Task
ANN-CM
Task
Acc.
CM
Ace.
ENN-CM
Acc.
F1
Acc.
F1
A1
46.90
46.70
47. 47.330 (10.8555)
47. (7.04
CB
91.07
91.04
(2.86))
92.37 (11:408)
A2
43.20
42.33
44. 14.10((2.0e) (12.08%)
(3.558
COPA
56,00
55.84
64.00
63.87 (14.99))
A3
42.33
40.16
45.0 (10.0099)
44.72
WSC
63.46
44.29
(314000)0005)
47 41 (17.01%%
ANLI
41.84
40.32
44.72
43.7 (3871(184199)
RTE
77.98
77.6
79.42 (11.86))
79. 18 (12.049)
Table 2: Results the test set of NLI tasks. ANLI is combined dataset AI, A2, A3.
WiC
69.12
69.08
69. 28 (0..33%)
69. 28 (10.2998)
BonlQ
80.49
78.62
81.16 (1.116ro..cm (10.88))
79. 79.86 (11.66)))7
MultiRC
66.56
60.73
70.40 (15.77%)
69. (0.26
ReCoRD
61.17
61.82
(2.055(1.4555)
62. (11.4888)
CM
RNN-CM
Task
Table I: Performance comparison of &NN-CM vs. CM
Bin.
Inst.
Bin.
Inst.
the SuperGLUE development set. The results are reported on the validation set.
QASC
91.27
75.05
91.10
75.92 (11.88))))%))
QASC-IR
95.39
88.01
95.40
(8.44(10
PIQA
73.59
86.45
73.91
86. 86.94 (10.82%)
SIQA
81.01
80.71
80.98
81.27 (10.69%)
5 Results and Discussion
CQA
88.06
82.80
88.12
83.29 (10.59%)
CQA-2
54.37
57.97
61.20
60.8 60.84(11.88) (11955)
Table 1 shows results on SuperGLUE datasets. We observe kNN to correct predictions of CM in tex- tual entailment tasks such as CB and RTE. The as- sistance of neighbors from the hidden space rep-
SWAG
91.46
93.14
91.47
93. 24 (10.119))
CosmosQA
87.63
86.30
87.86
86.77 (10.519)
H-SWAG
94.44
96.15
94,74
96. 26.244. (10, (10.09%)
CICERO-VI
88.53
83.55
88.50
83.70 (10. 18%))
CICERO-v2
87.62
90.34
87.67
90.56 (1.5566110200) (10.219)
resentations also shows a huge improvement (by *7%) to resolve the ambiguity in pronoun in WSC. However, the improvement in WiC is comparably less (~0.3%). After investigation, we found that CM and KNN share the same set of samples where they make erroncous predictions. The conclu- sions are made on relatively low improvements on BoolQ when compared with MultiRC and ReCoRD, Amongst all, we observe a huge improvement for over 14% in COPA. We notice that ANN alone can surpass the baseline COPA accuracy by over three points. While a combination of both gives boost of over eight points in both performance matrices. Table shows the improvement due to kNN in- volvement during predictions. We find the neigh- borhood search to help more as the task complexity increases, thus the observed improvement for A3 is more than the other adversarial datasets. More improvement in Fl score indicates that neighbor- hood search is less impacted by the class imbalance when compared with the CM-only setting. The best identified for ANLI tasks is between 1-4 with high& À (=0.99), thus, reflecting the informa- tion provided by the closest few neighborhoods is important and sufficient.
Table 3: ANN-CM vs. CM on QA datsets. Bin. and Inst. denote binary and instance classification accuracy.
the instance accuracy of KNN 80.45% while the CM performance is 80.98% with errors made on a similar set of samples. Table 4 shows the results on sentiment analysis tasks. Similar to WiC, the reasons for poor perfor- mance on Restaurant were found to be the same set of erroneous predictions, and thus no explicit probability corrections were made by the neighbor samples. In contrast to this, we observed good per- formance on Laptop because the nearest neighbors help boost recall and precision.
Out-Of-Domain Performance. We retrieve Su- perGLUE diagnostic datasets AX and AXg (test only) and perform ANLI out-of-domain evalua- tions 6. Table 5 shows that the neighbor search on ANLI datastore not only improves on in-domain datasets but also shows over 12% Fl improve-
Since we evaluating OOD performance, use the same metric used in ANLI evaluation.
In QA tasks (Table 3), the observed improve- ment is relatively lower when compared with Super- GLUE and ANLI After investigation, we observed the prediction that the predictions made by CM and KNN are similar. This indicates an effective clus- tering by the (DcBERTa-large) CM to perform the binary classification task. For instance, on SIQA,
CM
ANN-CM
Task
Acc.
F1
Acc.
Fl
Laptop
88.42
77.96
88. (10.1%))
79.94 (r.
Restaurant
88.01
79.13
88. (900.20%
79.36 (0000299))) 29%)
Table 4: ANN-CM vs. CM performance on test split of sentiment classification datasets.

---

Task
ANN-CM
Accuracy with change in threshold
Acc.
F1
Acc.
F1
AX
59.78
49.30
61 (12
55.58 55.58
2.9k
AXg
50.84
37.80
50.28 1.10%)
39.37 (1.18)
2.3k
2.8k
44.0
Table 5: ANLI out of domain evaluation on AXp, AXg.
Accuracy
43.0
1.5k
kNN, CM,
ANNa CMa
ANN: CMa
kNN CMa
Metric
CMa
42.5
70.3k
Acc.
41.07
50.00
39.29
75.00
60.71
F1
32.19
34.73
32.47
53.82
46.58
0.8
Table ANLI-CB domain adaptation without CM fine- tuning, adding domain-speciff datastore. kNN,- CM2 is a &NN-only classifier that constructs a datastore from RoBERTa-base LMM. CM2 denotes an ANLI classi- fier. KNN subscripts 'a' and 'e' indicate the datastore is constructed using ANLI or/and CB training set.
Figure 3: Impact of 7 ANLI accuracy. Red annota- tions are the number samples query KNN.
hood search is important for CM to make important decisions around the decision boundaries which leads to model confusion. We assume the model to be more in need of external aid when the CM predictions are close to the uniform distribution over labels. Thus, we define a neighbor require- ment score r for a given text x as the normalized KL-divergence of CM prediction with respect to a discrete uniform distribution over classes:
ment on AX and around 4% improvements on AXg OOD datasets. There are improvements in Acc. for AXb and an observed poorer performance (by 11%) of KNN-CM on AXg- To investigate it fur- ther, we found kNN to improve the precision and recall score of the poor performing class by slightly trading-off with the precision and recall of higher performing class, the overall impact improves F1 significantly, however, accuracy degrades.
r(x) = KL(hcM Uy) log((>")
For given input x. redefine the predictor:
Domain Adaptation without Classifier Tuning. We carry out KNN-CM domain adaptation of ANLI to CB without explicitly fine-tuning the classifier but including the domain-specific datastore. In Table we observe the datastore from CB domain is im- portant to help boost the performance significantly. Even on an untrained classifier, merely including domain-specific datastore constructed on purely pre-trained LM (CM,) and classifying using kNN- only, gives 50% accuracy. The best-perforning model is kNN,-CM, (a CB datastore constructed on ANLI classifier) with an accuracy of around 75% and Fl score of over 53%. Merging available ANLI datastore with CB datastore, however, tends to reduce the performance. We posit the reason is very small fraction of neighbors belonging to CB compared to ANLI (>0..5%%) Rescoring meth- ods can help adapt the existing datastore other domains (Bhardwaj et al., 2022).
X h2Nx(2) + (1-A) hcM(s), ifr( S hcc(()).
hy(x):=
otherwise
where À [0, 1], |J]| (cardinality of the label set) denote the number labels, My denotes the uniform distribution over the labels the set y. h is classifier's probability over label i, t defines a threshold on divergence value below which KNN will be involved in model predictions. In Figure 3, observe ANLI accuracy to converge at 7 0.7. filter
Thus, using entropy-based measures, samples for kNN reduce inference time.
Layer Importance for Retrieval. Following Khandelwal et al. (2019); Bhardwaj et al. (2022), we create datastore on representations obtained from different layers of ANLI-based classifier and perform hyperparameter search (k and X) on ANLI development set. Figure 4 shows the layer-wise test set performance increases when we go deeper in the network. For ANLI, we find the best-performing neighborhood representations to be layer normal- ization in the fre-final layer. our initial experi- ments on the SuperGLUE benchmark, on average, we found the final layer to be the best performing.
Filtering Samples for k-NN Search. As dis- cussed in section 3, we hypothesize the neighbor-
We use hyperparameters & and X, obtained from ANLI dev set, across the settings in Table

---

ANLI: Layer-wise retrieval performance
ANN-CM Time Overhead. Being semi- parametric model, the KNN search space tends linearly increase with the increase the number of training samples to memorize. Thus, we study the time-overhead, i.e., added inference-time latency due to the neighborhood search. Without the loss of generality, we base our analysis on the ANLI dataset. On the CPU the per-sample CM infer- speech is z/2 ms, and for the KNN retrieval 9, is #29 ms. On the GPU 10 the per-sample in the CM stage is =9 ms, and for the kNN stage is x2 ms. Thus, a flat KNN search increases inference time by around 40% on CPU and 20% on GPU.
IRMN-CMI (CN) F1 (CM)
44
Accuracy
2
10
Layer
Figure 4: Impact of different layers ANLI The hori- zontal axis denotes layer representations considered for datastore construction and the vertical axis denotes the test accuracy/F1.
Utterance Classification. We also trained clas- sifiers on datasets for emotion recognition in con- versation. Given an utterance from conversation with an appended set of eight utterances preced- ing it, we aim to classify it in one of the emo- tion classes. Our experiments on MELD (Poria et al., 2018), 8), DailyDialogue (Li et al., 2017), and IEMOCAP (Busso al., 2008) shows very insignif- icant improvements in Accuracy and F1 scores when the model equipped with kNN search. We leave the precise semi- aani-parametric modeling for ut- terance classification as future work.
CM
ANN-CM
Task
Acc.
F1
Acc.
F1
20%
33.37
16.68
35.38
35.09
40%
33.37
16.68
33.37
31.18
60%
33.37
16.68
36.11
36.21
80%
46.47
44.90
48.09
47.43
100%
41.84
40.32
44.72
43.71
Table Low-resource performance on ANLI.
Neighbors Weighting. We compare the frequency-based -baeed probability computations Equation 1 with weighting neighbors with their distances from the query, thus
Low-Resource. In Table 7, observe that when data availability is reduced below 60%, the clas- sification model performs worse equivalent random classification with uniform probabilities. When KNN is introduced, the performance of the KNN-CM model tends to be better than assigning the random labels to the test instances. It shows that, even in low resource cases, there are clustered vectors that &NN can exploit to boost the classifica- tion performance, while large parametric classifiers fail to capture proximity relations. This to some extent provides evidence to the hypothesis of Khan- delwal et al. (2019), i.e., learning the similarity be- tween texts is easier than predicting the next word, given our problem is reformulated as a label (text) generation. On the other hand, when the amount of training data is greater than 80% (of full-set), our baseline performed well and ANN adds further improved by nearly 4%-8% Thus, irrespective of the baseline performance, ANN tends maintain better performance than random classification and it is evident that it supports the CM relentlessly even in the low- rew-resuurce regime.
1 yy==yy exp ( ((((())))()))
p(y EJ)=
In our initial experiments on QASC dataset with 8 E (2, 10, 20). we found the validation set per- formance to be 75.81%, 75.92%, 75.92% all of which are higher than baseline CM but lower than frequency-basee computations. We posit there is no generic weighting scheme that works for all the tasks, hence we leave methods to find a task- adaptive neighborhood weighting for classification for future work.
Conclusion
In this work, we presented kNN-CM, semi- parametric paradigm that augments the inference
AMD Ryzen Threadripper 3960X 24-Core. "We implement IndexFlatL22 search.
UVIDDI RTX A6000.

---

phase with neighborhood search through the train- ing set. We studied the impact of adding non- parametric characteristies to a parametric classi- fier on 24 language understanding tasks. We fur- ther demonstrated the generalizability of kNN- CM by studying their out-of-domain and domain adaptation performance. We also showed its ef- ficacy in low-resource scenarios where CM per- formance reduces dramatically and neighborhood search emerges as savior. Toward the end, we leave a few important remarks on utterance clas- sification and neighborhood weighting that carry the potential to motivate future research directions, which we elaborate on in the limitations section.
learning applications involving text. Another inter- esting area will be propose methods that allow task-specific datastore representation tuning, more interestingly through backpropagation. Since the datastore size increases linearly with the number of training samples, scaling semi-parametrn systems can be challenging task. Thus, deploying such systems edge devices with constrained compu- tational capacity and memory is another interesting future research direction.
Acknowledgement
We thank the anonymous reviewers for their con- structive feedback. This project is supported by the AcRF MoE Tier-2 grant (Project no. T2MOE2008, and Grantor reference no. MOE-T2EP20220- 0017) titled: "CSK-NLP: Leveraging Common- sense Knowledge for NLP", and the SRG grant id: TISRIS19149 titled "An Affective Multimodal Dialogue System". This work also generously supported by BUPT Excellent Ph.D. Students Foun- dation CX2021229.
7 Limitations
We discuss the potential limitations of semi- parametric modeling and considerable future work:
Non-parametric characteristics introduce chal- lenges in the interpretability of the predic- tions.
Since the function form highly depends on the size of the training set, the memory footprint grows linearly.
References
Rishabh Bhardwaj. George Polovets, and Monica Sunkara. 2022. Adaptation approaches for near- est neighbor language models. arXiv preprint arXiv:2211.00 11.0728..
Learning a good representation of the dataset is still a bottleneck task and predominantly relies on the parametric models. Thus, the performance and function form of non- parametric models depends on the effective- ness of the data representations.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical com- monsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432 7439.
Since nearest neighbor computation requires pairwise similarity between test and samples in the train set, the inference time increases with the increase in the dimensionality of space and size of the train set. Several tools such as Faiss (Johnson al., 2019) assist in significant reduction of computational over- head with the trade-off in the performance of the model.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Temocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-359,
Christopher Clark, Kenton Lee, Ming-Wa Beng-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 190s 10044.
One can compute &NN probabilities by using exponential of negative of distance (Khandel- wal et al., 2019). However, simple averaging shows considerable improvements, and find- ing better probability computations is left for future work.
Marie-Catherine De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmenbank:: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
In the future, we see huge potential of ANN's in tackling the catastrophic forgetting in continual

---

Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, and C Lawrence Zitnick, 2015. Exploring nearest neighbor approaches for image captioning. arXiv preprint TXXv::5005.04::
Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP), pages 2391 2391-2401.
Dietmar Jannach and Malte Ludewig. 2017. When recurrent neural networks meet the neighborhood for session-based recommendation. In Proceedings of the eleventh ACM conference on recommender systems, pages 306-310.
Andrew Drozdov, Shufan Wang, Razich Rahimi, An- drew McCallum, Hamed Zamani, and Mohit lyyer. 2022. You can't pick your neighhors, or can you? when and how to rely on retrieval in the & nn-lm. arXiv preprint arXiv:2210.58852
Jeff' Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEFE Transactions on Big Data, 7(3):535-544 557.
Evelyn Fix and Joseph Lawson Hodges. 1989. Dis- criminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Re- view/Revee Internationale de Statistique, 57)3)238 247.
Iman Kamehkhosh, Dietmar Jannach, and Malte Ludewig. 2017. A comparison of frequent pattern techniques and a deep learning method for session- based recommendation. In RecTemp @ RecSys, pages 50-56.
Stuart Geman, Elie Bienenstock, and René Dour- sat. 1992. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 3128- 3137.
Deepanway Ghosal, Navonil Majumder, Rada Mihal- cea, and Soujanya Poria. 2022a. Two is better than many? binary classification as an effective approach to multi-choice question answering. arXiv preprint arXiv:2210. 16495.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019, Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv: 191 1.00722.
Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2022b. Cicero: A dataset for contextualized commonsense inference in dialogues. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5010-5028.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth. Shyam Upadhyay, and Dan Roth. 2018 Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Pa- pers), pages 252-262.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the Second PAS- CAL Challenges Workshop on Recognising Textual Entailment, volume 7.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence compo- sition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082-8090.
Junxian He, Graham Neubig, and Taylor Berg- Kirkpatrick. 2021. Efficient nearest neighbor lan- guage models. arXiv preprint 1xXiv:2109.02211 04212.
Pengeheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced hert with disentangled attention. arXiv preprint vrXiv:2006.03644
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
Zejiang Hou, Julian Salazar, and George Polovets. 2022. Meta-learning the difference: preparing large language models for efficient adaptation. ArXiv, abs/2207.03509.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao. and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.0399
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. ArXiv. abs/2205.0566 05638.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 1907.1
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qu: Machine reading com- prehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical

---

Sewon Min. Weijia Shi, Mike Lewis. Xilun Chen, Wen- tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Nonparametric masked language modeling. arXiv preprint arXiv:2212.01.. 349.
Siqi Shen, Deepanway Ghosal, Navonil Majumder, Henry Lim, Rada Mihalcea, and Soujanya Poria. 2022. Multiview contextual commonsense infer- ence: A new dataset and task. arXiv preprint arXiv:2210.02890.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language under- standing. In P Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. As- sociation for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multlingual graph of gen- eral knowledge. In Thirty-first AAAI conference on artificial intelligence.
Mohammad Taher Pilehvar and Jose Gamacho-Colladon 2018. Wie: the word-in- in-contextt dataset for evaluat- ing context-sensitive meaning representations. arXiv preprint arXiv:1808.09 10121..
Xiaoyuan Su and Taghi M Khoshgoftaar. 2009. A sur- vey of collaborative filtering techniques. Advances in artificial intelligence, 2009.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsensequ: A question answering challenge targeting commonsense knowl- edge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4159-4158.
Maria Pontiki, Dimitrios Galanis. Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Moham- mad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. 2016. Semeval- 2016 task 5: Aspect based sentiment analysis. In In- ternational workshop on semantic evaluation, pages 19-30.
Maria Pontiki, Dimitrios Galanis, Harris Papageorgiou. Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment analy- sis. Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 486- 495.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bha- gavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021. Commonsenseqa 2.0: Exposing the limits of ai through gamification. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round) 1).
Soujanya Poria, Devamanyu Hazarika, Navonil Ma- jumder, Gautam Naik, Erik Cambria, and Rada Mi- halcea. 2018. Meld: multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint 1870.025000
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 Improving language under- standing by generative pre-training.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lam- bert. 2020. Trl: Transformer reinforcement learning. https: //gi wwwpttt.....w.ert.eeeeteee com/lvwerra/trl.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. Peter Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67,
Bram Wallace and Bharath Hariharan. 2020. Extend- ing and analyzing supersuue learning across do- mains. In European Conference on Computer Vision. pages 717-734. Springer.
Melissa Roemmele. Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Com- monsense reasoning about social interactions. In
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463-4473.
Y Wang, WL Chao, KQ Weinberger, and L Simpleshot van der Maaten. 2019b. Revisiting nearestneighhor classification for few-shot learning. Preprint.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pages 285-295.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pat- tern recognition, pages 3733-3742.

---

Dani Yogatama, Cyprien de Masson A'ttmmme and Lingpeng Kong. 2021. Adaptive semiparametric lan- guage models. Transactions of the Association for Computational Linguistics, 9:362-373.
Rowan Zellers, Yonatan Bisk. Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceed- ings of the 2018 Conference Empirical Methods Natural Language Processing, pages 93-104, Brus- sels, Belgium. Association for Computational Lin- guistics.
Renrui Zhang, Liuhui Wang, Ziyu Guo, and Jianbo Shi. 2023. Nearest neighbors meet deep neural net- works for point cloud analysis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1246-1255.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. arXiv preprint
12885.
Yan Zhou, Fuqing Zhu, Pu Song, Jizhong Han, Tao Guo, and Songlin Hu. 2021. An adaptive hybrid framework for cross domsin aspect-based sentiment analysis. In Proceedings the AAAI Conference on Artificial Intelligence, volume 35, pages 14630- 14637.


Reviews:
[{"review_id": "OUFyT94aes", "reviewer": "Reviewer_rD6m", "summary": "", "questions": "1. Broadly, the semi-parametric approach attempts to smoothen the feature space distribution as a consequence of introduction of kNN search. In the literature, temperature parameter during softmax stage  attempts to smoothen the label space. IMO, the baselines should be strengthened by comparing few of the experiments by playing with the temperature parameter.\n\n2. The paper considers only one baseline, which is the classification model itself. I feel its more of an ablation experiments than an attempt to compare with baselines. I would suggest authors to consider more pragmatic baselines from the semi-parametric literature.\n\n3. Discussion of \\lambda - \\lambda hyperparameter plays a cruical role in identifying the utility of semi-parametric approach. The results table should include an additional \\lambda parameter value adopted for each of the experiments.\n\n4. Extension to sequence modeling tasks - IMO, couple of experiments for sequence-to-sequence tasks should also be considered to validate the generalization ability of such an approach.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Clarification 1]: Label smoothening vs. $k$NN LM**\n\nThanks for asking this question. While label smoothening is implemented during the training phase, $k$NN-CM is an inference time approach that acts on an off-the-shelf CM model. Thus these two can be highly complementary approaches rather than competing ones. One can leverage the benefits of both by first training the model with temperature parameters (training phase) followed by including $k$NN during the inference phase. To provide evidence, we have conducted new experiments on ANLI (A1+A2+A3):\n\n|         |    Temp: | 1 |    Temp: | 2 |    Temp: | 4 |     Temp: | 16 |\n|---------|--------|----------|--------|----------|--------|----------|--------|----------|\n|         |    Acc(1) | F1(1) |    Acc(2) | F1(2) |    Acc(4) | F1(4) |    Acc(16) | F1(16) |\n| CM | 41.84 |   40.32 | 47.50 |   47.06 | 47.94 |   47.60 | 50.44 |   49.87 |\n| $k$NN-CM  | **46.25** |   **45.88** | **49.81** |   **49.57** | **49.25** |   **49.03** | **51.00** |   **50.56** |\n\n_Table 1: Performance of CM vs. $k$NN-CM when CM is trained with temperature parameter. Acc(2) denotes accuracy with temperature parameter 2. F1 denotes macro-F1._\n\n[Observation] $k$NN provides complementary advantages to CM trained with label smoothing. As shown in Table 1, when we train the model with different temperature parameters: {1,2,4,16} we observe adding $k$NN to CM during inference time increases the performance between 0.5% to 5% across temperature scales. Thus, $k$NN does not compete with label smoothing but rather provides complementary improvements as both act in different phases of the CM model. Thanks for asking this question, if the reviewer feels that these experiments should be included, we will update them in the draft.\n\n===\n\n**[Clarification 2]: Regarding choice of baselines**\n\nSince $k$NN-CM is presented to fill the gaps left behind by $k$NN-LM literature, to be consistent, we followed the baselines as proposed by $k$NN-LM, $k$NN-MT, and Adaptation-$k$NNLM [1][2][3]. We also try to cover all the experiments performed by existing literature as depicted for more clarity of the reviewer below:\n| Experiment                      | $k$NN-LM     | $k$NN-MT            | Adaptation-$k$NN-LM | $k$NN-CM (this work)      |\n|----------------------------|------------|-------------------|--------------|-------------------------|\n|                 | generation  | generation | generation | classification |\n| In-domain                  | Yes        | Yes               | Yes          | Yes                     |\n| Out-of-domain (zero shot)  | No         | No                | Yes          | Yes                     |\n| Adaptation                 | Yes        | Yes               | Yes          | Yes                     |\n| Lambda Analysis            | Yes        | Yes               | No           | Yes                     |\n| Layer-importance Analysis  | Yes        | No                | Yes          | Yes                     |\n| Low-resource analysis      | No         | No                | Yes          | Yes                     |\n| # tasks                    | 2 datasets | 17 language pairs | 7 datasets   | 24 classification tasks |\n\n\n$k$NN-CM is a contribution in the direction of showing the advantages of non-parametric memory when attached to off-the-shelf models, which has been missing in the current literature focusing on text-to-text tasks ([1][2][3]). We try to convince readers that $k$NN is a complementary approach and thus can enhance the performance at the inference time of a given off-the-shelf model _irrespective of the algorithm used to train them_.\n\n===\n\n**[Clarification 3]: Lambda and other important hyperparameters**\n\nThanks for pointing it out, we have mentioned the important hyperparameters adopted for each dataset selected on the respective validation split. As per the reviewer's suggestion, we will include them in the corresponding tables of the paper. $k$, $\\lambda$, $\\tau$ denotes the number of top neighbors used, the coefficient of $k$NN in equation 2, and the neighborhood threshold, respectively.\n\n| Dataset      | cb   | copa | wsc  | rte  | wic  | boolq | multirc | record | qasc | qascir | piqa | siqa | cqa   | cqa2 | swag | cosmos | hswag | cicerov1 | cicerov2 | A1 | A2 | A3 | ANLI | laptop | rest |\n|--------------|------|------|------|------|------|-------|---------|--------|------|--------|------|------|-------|------|------|---------|-------|-----------|-----------|-------|-------|-------|------|--------|------------|\n| topk         | 1    | 4    | 16   | 16   | 512  | 32    | 32      | 128    | 16   | 128    | 64   | 8    | 32    | 16   | 8    | 1       | 256   | 2         | 1         | 1     | 2     | 1     | 1    | 128    | 1          |\n| lambda       | 0.5  | 0.999| 0.1  | 0.6  | 0.7  | 0.8   | 0.999   | 0.84   | 0.999| 0.5    | 0.6  | 0.6  | 0.4   | 0.99  | 0.4  | 0.2     | 0.8   | 0.1       | 0.9       | 0.5   | 0.7   | 0.5   | 0.5  | 0.999  | 0.2        |\n| kl_threshold | 1.1  | 0.1  | 0.9  | 1.1  | 1.2  | 1     | 0.5     | 0.6    | 0.6  | 0.7    | 0.7  | 0.3  | 0.2   | 1.7   | 0.5  | 0.5     | 0.3   | 0.6       | 0.7       | 0.9   | 0.7   | 1     | 1.1  | 0.9    | 0.7        |\n\n===\n\n**[Clarification 4]: Sequence to sequence tasks**\n\nThanks for asking this question, there is a strong reason we did not include sequence-to-sequence results because such results have been published before. $k$NN-MT, $k$NN-LM, and Adaptation-$k$NNLM [1][2][3] have shown the advantages of $k$NN for text-to-text tasks. The primary motivation to push this paper into the community is to fill the gap in this area by exploring the utility of $k$NN for classifiers. If the reviewer feels this point should be emphasized in the draft, we would love to do it. We draw the table below to show where existing literature stands and where we contribute:\n\n| Experiment                      | $k$NN-LM                              | $k$NN-MT                     | Adaptation-$k$NN-LM                       | $k$NN-CM (this work)                                    |\n|----------------------------|-------------------------------------|----------------------------|------------------------------------|-------------------------------------------------------|\n|                 | generation  | generation | generation | classification |\n| In-domain                  | Yes, 2.9% reduction in perplexity  | Yes, 1. 5 increase in BLEU | Yes, 15.8% reduction in perplexity | Yes, 3.12% improvement in F1 score  (across 25 tasks) |\n| Out-of-domain (zero-shot)  | No                                  | No                         | Yes, 4.6% reduction in perplexity  | Yes, 8.45% improvement in F1 score on AXg and AXb     |\n| Adaptation                 | Yes, 12.28% reduction in perplexity | Yes, 9.2 increase in BLEU  | Yes, 33.7% reduction in perplexity | Yes, 21.63% improvement in adapting ANLI to CB.       |\n| Lambda Analysis            | Yes                                 | Yes                        | No                                 | Yes                                                   |\n| Layer-importance Analysis  | Yes                                 | No                         | Yes                                | Yes                                                   |\n| Low-resource analysis      | No                                  | No                         | Yes                                | Yes                                                   |\n| # tasks                    | 2 datasets                          | 17 language pairs          | 7 datasets                         | 24 classification tasks                               |\n\nReferences:\n- [1] Khandelwal, Urvashi, et al. \"Generalization through memorization: Nearest neighbor language models.\" arXiv preprint arXiv:1911.00172 (2019).\n- [2] Zheng, Xin, et al. \"Adaptive nearest neighbor machine translation.\" arXiv preprint arXiv:2105.13022 (2021).\n- [3] Bhardwaj, Rishabh, George Polovets, and Monica Sunkara. \"Adaptation approaches for nearest neighbor language models.\" arXiv preprint arXiv:2211.07828 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The presentation of this paper is quite clear.\n\n2.The experiments in the paper are extensive, reasonable and systematic. The experiment results are persuasive.\n\n3. Even though novelty seems to be limited, I find experimental section intriguing and in favor of semi-parametric methods over only parametric methods.\n", "reasons_to_reject": "See Questions for the Authors section.", "questions_to_authors": "1. Broadly, the semi-parametric approach attempts to smoothen the feature space distribution as a consequence of introduction of kNN search. In the literature, temperature parameter during softmax stage  attempts to smoothen the label space. IMO, the baselines should be strengthened by comparing few of the experiments by playing with the temperature parameter.\n\n2. The paper considers only one baseline, which is the classification model itself. I feel its more of an ablation experiments than an attempt to compare with baselines. I would suggest authors to consider more pragmatic baselines from the semi-parametric literature.\n\n3. Discussion of \\lambda - \\lambda hyperparameter plays a cruical role in identifying the utility of semi-parametric approach. The results table should include an additional \\lambda parameter value adopted for each of the experiments.\n\n4. Extension to sequence modeling tasks - IMO, couple of experiments for sequence-to-sequence tasks should also be considered to validate the generalization ability of such an approach.", "paper_topic_and_main_contributions": "The paper presents a semi-parametric approach for classification tasks. Extensive experimental study across diverse range of classification tasks such as QA, NLI, ABSA is performed. The adopted approach is a rehash of previous non-parametric approaches for classification and language modeling tasks. During inference time, additional kNN search is computed over already computed representation by the classification model over the feature space. The expectation is that such additional kNN search helps in correctly classifying support vectors. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 154: 'this works is a first ... ' -> 'this work is a first ...'\nLine 158: 'gives an illustrative to ' -> '... an illustration to..'"}, "extra_text": {}}, {"review_id": "RBDLezm8NG", "reviewer": "Reviewer_ifUU", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Clarification1]: The technical novelty is limited to some extent. The KNN-based methods are widely studied in LM and MT, etc. This work provides few insights into such a non-parametric method for text classification.**\n\nThanks for giving us a chance to elaborate on our contributions. We want to emphasize that this work is a contributing successor of kNN-LM and $k$NN-MT literature, thus most of the experiments are focused on classification environment rather than text generation. We believe there is a potential misunderstanding that $k$NN-CM completely ignores the contribution of $k$NN-LM and $k$NN-MT, rather we try to supplement the existing literature showing the effectiveness of $k$NN in downstream classification tasks in addition to text generation as studied by existing works. To be consistent, we followed the baselines proposed by $k$NN-LM, $k$NN-MT, and Adaptation-$k$NN-LM [1][2][3], covering all the experiments by existing literature. Let us give a better comparison below to convey how it fills the gap in the current literature:\n\n| Experiment                      | $k$NN-LM                              | $k$NN-MT                     | Adaptation-$k$NN-LM                       | $k$NN-CM (this work)                                    |\n|----------------------------|-------------------------------------|----------------------------|------------------------------------|-------------------------------------------------------|\n|                 | generation  | generation | generation | classification |\n| In-domain                  | Yes, 2.9% reduction in perplexity  | Yes, 1. 5 increase in BLEU | Yes, 15.8% reduction in perplexity | Yes, 3.12% improvement in F1 score  (across 25 tasks) |\n| Out-of-domain (zero-shot)  | No                                  | No                         | Yes, 4.6% reduction in perplexity  | Yes, 8.45% improvement in F1 score on AXg and AXb     |\n| Adaptation                 | Yes, 12.28% reduction in perplexity | Yes, 9.2 increase in BLEU  | Yes, 33.7% reduction in perplexity | Yes, 21.63% improvement in adapting ANLI to CB.       |\n| Lambda Analysis            | Yes                                 | Yes                        | No                                 | Yes                                                   |\n| Layer-importance Analysis  | Yes                                 | No                         | Yes                                | Yes                                                   |\n| Low-resource analysis      | No                                  | No                         | Yes                                | Yes                                                   |\n| # tasks                    | 2 datasets                          | 17 language pairs          | 7 datasets                         | 24 classification tasks                               |\n\nIf the author feels that the above table and clarification make the contributions stand out, we would be happy to incorporate them into the paper.\n\n===\n\n**[Clarification2]** Related work in OOD and DA.\n\nThanks for asking this question, to the best of our knowledge, there is no inference-time baseline in the semi-parametric literature. The main aim of this work is to fill the gaps left by $k$NN-LM literature [1][2][3] which primarily compares with LM-only models. Thus, we compare OOD and DA with respect to the CM-only model. Existing DA works mainly propose changes to the network architecture or learning algorithms that work during the training phase, in contrast to this, $k$NN-CM is an inference phase approach similar to $k$NN-LM, $k$NN-MT, and Adaptation-$k$NN-LM [1][2][3]. Hence, the most appropriate baseline we could find is the CM-only models. Here, CM can be replaced by any model with a better DA or OOD score, we claim that $k$NN will still provide complementary benefits to it. To make it more evident, we conduct further experiments. As shown in Table-1 of reviewer rD6m and Table-1 of fnKr, the inference phase $k$NN can be complementary to approaches applicable to the training phase. If the reviewer strongly feels such a discussion can strengthen the discussion, we will include it in the paper.\n\n===\n\nReferences\n\n- [1] Khandelwal, Urvashi, et al. \"Generalization through memorization: Nearest neighbor language models.\" arXiv preprint arXiv:1911.00172 (2019).\n- [2] Zheng, Xin, et al. \"Adaptive nearest neighbor machine translation.\" arXiv preprint arXiv:2105.13022 (2021).\n- [3] Bhardwaj, Rishabh, George Polovets, and Monica Sunkara. \"Adaptation approaches for nearest neighbor language models.\" arXiv preprint arXiv:2211.07828 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper incorporates a nonparametric KNN method into a classification model to improve the test-phrase adaptation performance.\n\n2. Experimental results are extensive on a range of datasets and show strong results.", "reasons_to_reject": "1. The technical novelty is limited to some extent. The KNN-based methods are widely studied in LM and MT, etc. This work provides few insights into such a non-parametric method for text classification.\n\n2. The OOD and DA performance lacks comparisons to related works. This can weaken the contributions of this work. \n\n**Post Rebuttal:**\nThe authors claim in the response that ``most of the experiments are focused on classification environment rather than text generation\". However, KNN is a classical method for non-parametric classification. I still think the contribution/novelty of this paper is marginal, it seems like an incremental work on using KNN for supervised learning. Thus, I would like to learn about other reviews' comments on my score changing.\n\n3. I notice that there exists work on KNN-based text-time adaptation [1][2]. The authors' rebuttal cannot address my concerns about the technical novelty of this paper.\n\nRef.  \n- [1] Minguk Jang et al. \"Test-Time Adaptation via Self-Training with Nearest Neighbor Information\". Published in ICLR 2023.\n- [2] Zhang et al. AdaNPC: \"Exploring Non-Parametric Classifier for Test-Time Adaptation\". Published in ICML 2023.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The researchers propose a method called k-Nearest Neighbor Classification Model (kNN-CM) that enhances the capabilities of a pre-trained parametric text classifier by incorporating a neighborhood search through the representation space of training samples. This approach improves the text-discriminating properties of semi-parametric models and shows consistent performance improvements on various language tasks, including SuperGLUE tasks, adversarial natural language inference datasets, question-answering datasets, and sentiment classification datasets.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jJ8SXzxOQH", "reviewer": "Reviewer_fnKr", "summary": "", "questions": "A. How does the proposed method perform using other language model backbones except for RoBERTa-base?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Clarification A]: How does the proposed method perform using other language model backbones except for RoBERTa-base?**\n\nThanks for pointing it out! we believe including it as a separate paragraph in the paper will enhance the quality. Below we show the experiment on three ANLI classifier models RoBERTa-base, RoBERTa-large, and DeBERTa-base. \n\n|Model| CM | $k$NN-CM|% Improvement|\n|-------|------|-------------|-------------------|\nRoBERTa-base|41.84|**46.25**|4.41$\\uparrow$|\nRoBERTa-large|48.59|**49.47**|0.88$\\uparrow$|\nDeBERTa-base|58.56|**59.53**|0.97$\\uparrow$|\n_Table 1:Benefits of including $k$NN search during inference phase of three different classifier models on ANLI (A1+A2+A3)._\n\nWe observe that during the inference phase, $k$NN significantly improves the performance of the CM-only model between ~1\\%-4.4% across model architectures when tested on ANLI. \n\n_Note: While in the given time we could perform experiments on one setup, if the reviewer feels the experiments should be carried out on multiple tasks and more architectures, we would do so for the main draft. Another point to note is that in the paper, we use two architectures DeBERTa-large (for QA tasks) and RoBERTa-base (for other tasks). It also shows that our idea is applicable to off-the-shelf models and not only the RoBERTa backbone._\n\n===\n\n**[Clarification 1]: It is important to specify the memory overhead of semi-parametric models $k$NN**\n\nAbsolutely, $k$NN datastores for classification incur significantly less ($<$ 1%) memory and parameter overhead w.r.t RoBERTa which has 125M parameters and occupies 500 MBs of disk space. We provide space utilization below:\n\n| Dataset          | CB    | COPA  | WSC   | RTE   | WiC   | BoolQ | MultiRC | ReCoRD | QASC  | PIQA  | SIQA  | CQA   | SWAG  |\n|-------------------|-------|-------|-------|-------|-------|-------|---------|--------|-------|-------|-------|-------|-------|\n| Memory overhead (in MBs)      | 0.75  | 1.2   | 1.66  | 0.842 | 0.889 | 0.953 | 1.2     | 2.4    | 0.932 | 1.1   | 1.3   | 0.958 | 1.4   |\n| # Params (in K) | 197.0 | 315.0 | 435.0 | 221.0 | 233.0 | 250.0 | 315.0   | 629.0  | 244.0 | 288.0 | 341.0 | 251.0 | 367.0 |\n| % param/memory-overhead    | 0.157% | 0.25% | 0.350% | 0.177% | 0.186% | 0.2%   | 0.252%  | 0.503% | 0.195% | 0.230% | 0.273% | 0.201% | 0.294% |\n\n| Dataset          | H-Swag | CICERO | A1    | A2    | A3    | ANLI  | Laptop | Restaurant  |\n|-------------------|--------|--------|-------|-------|-------|-------|--------|-------|\n| Memory overhead (in MBs)      | 1.2    | 1.4    | 1.3   | 1.1   | 1.5   | 2.4   | 3.4   | 0.848 |\n| # Params (in K) | 367.0  | 341.0  | 288.0 | 393.0 | 629.0 | 891.0 | 222.0  | 218.0 |\n| % param/memory-overhead    | 0.294% | 0.273% | 0.230% | 0.314% | 0.503% | 0.712% | 0.178% | 0.174% |\n\nAs per the reviewer's suggestion, we will include it in the updated draft. Thanks for pointing it out.\n\n===\n\n**[Clarification 2]: If a larger parametric model with equal memory cost is adopted, is it possible that the larger parametric model win?**\n\n_We believe it would not be a fair comparison as $k$NN does not compete with CM_: $k$NN-CM does not require learning of added parameters (acts in inference phase) while adding parameters to CM would require model tuning (acts in training phase). Due to this, the two approaches are complementary rather than competing and provide distinct benefits. Comparing the two models across different axes will make the analysis less interesting. To convince the reviewer more, we provide experimental evidence below:\n\n| Metric  | classifier   | classifier+$k$NN | Adapter+classifier  | Adapter+classifier+$k$NN |\n|---------|-------|----------|-------|----------|\n| ACC     | 41.07 | **75.00**    | 91.07 | **92.86**    |\n| F1      | 32.19 | **53.82**    | 91.04 | **92.37**    |\n| %extra params      | 0 | 0.25%    | 11.2% | 11.45%    |\n| training?  |No| No| Yes| No |\n\n_Table2: Performance on CB task using RoBERTa (pre-tuned on ANLI)_ \n\nIt can be seen that $k$NN is complementary to CM as well as Adapter-CM with only 0.25% parameter overhead. Thus it adds to the performance of both the un-finetuned and fine-tuned parametric models with little trade-off in the number of parameters. This provides evidence that both parametric models and non-parametric models have their own advantages and work on different axes of learning from data.\n\n===\n\n**[Clarification 3]: Regarding baselines**\n\nSince $k$NN-CM is presented to fill the gaps left by $k$NN-LM literature, to be consistent, we followed the baselines as proposed by $k$NN-LM, $k$NN-MT, and Adaptation-$k$NNLM [1][2][3]. We also try to cover all the experiments performed by existing literature as depicted for more clarity of the reviewer below:\n\n| Experiment                      | $k$NN-LM     | $k$NN-MT            | Adaptation-$k$NN-LM | $k$NN-CM (this work)      |\n|----------------------------|------------|-------------------|--------------|-------------------------|\n|                 | generation  | generation | generation | classification |\n| In-domain                  | Yes        | Yes               | Yes          | Yes                     |\n| Out-of-domain (zero shot)  | No         | No                | Yes          | Yes                     |\n| Adaptation                 | Yes        | Yes               | Yes          | Yes                     |\n| Lambda Analysis            | Yes        | Yes               | No           | Yes                     |\n| Layer-importance Analysis  | Yes        | No                | Yes          | Yes                     |\n| Low-resource analysis      | No         | No                | Yes          | Yes                     |\n| # tasks                    | 2 datasets | 17 language pairs | 7 datasets   | 24 classification tasks |\n\nWe want to emphasize that the experiments conducted for zero-shot and low-resource settings were to claim the effectiveness of $k$NN in these regimes as well. An existing zero-shot baseline would see a similar improvement when $k$NN is employed in its inference phase. Table 1 in clarifications for reviewer rD6m and Table 1 above provide evidence for such a hypothesis. While the emphasis of the paper is not to promote zero-shot and low-resource with RoBERTa, rather it is to illustrate the importance of $k$NN, if the reviewer strongly feels new zero-shot baselines such as [4] and zero-shot/few-shot capabilities of recent language models should be included as a part of our main experiments, we will include them in the draft.\n\n===\n\nReferences\n\n- [1] Khandelwal, Urvashi, et al. \"Generalization through memorization: Nearest neighbor language models.\" arXiv preprint arXiv:1911.00172 (2019).\n- [2] Zheng, Xin, et al. \"Adaptive nearest neighbor machine translation.\" arXiv preprint arXiv:2105.13022 (2021).\n- [3] Bhardwaj, Rishabh, George Polovets, and Monica Sunkara. \"Adaptation approaches for nearest neighbor language models.\" arXiv preprint arXiv:2211.07828 (2022).\n- [4] Wei, Jason, et al. \"Finetuned language models are zero-shot learners.\" arXiv preprint arXiv:2109.01652 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed knn strategy in this paper can be used as a plug-in for RoBERTa's inference phase to improve the performance of text classification with additional computational cost.\n2. The experiments in this paper are comprehensive. Experiments show that the method can be applied to out-of-domain classification, domain adaptation classification and low-resource classification.\n", "reasons_to_reject": "1. It is important to specify the memory overhead of semi-parametric models? If a larger parametric model with equal memory cost is adopt, is it possible that the larger parametric model win? The experiment lacks a comparison. \n2. In scenarios involving out-of-domain or low-resource situations, it is crucial to consider state-of-the-art (SOTA) baselines that are specifically designed for such scenarios. However, the experiment lacks a direct comparison with these baselines. ", "questions_to_authors": "A. How does the proposed method perform using other language model backbones except for RoBERTa-base?\n", "paper_topic_and_main_contributions": "For Text classification, this paper proposed a KNN-CN method to improve the classification performance by integrating KNN and language model in the inference phase. \nExperiments show that the performance of the proposed method has been improved compared with using only language models on 8 SuperGLUE tasks, 3 adversarial natural language inference datasets, 11 question-answering datasets, and 2 sentiment classification datasets.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a semi-parametric approach, called kNN-CM for text classification tasks by integrating a non-parametric memory for neighborhood retrieval to a pre-trained parametric text classifier to explore the importance of querying neighbors to solve classification tasks. Reviewers agree that the paper is well-written and appreciate the comprehensive experimental evaluation conducted in the paper. The experiments cover a diverse set of classification tasks and provide persuasive results.

However, reviewers express concerns about the contribution of the approach, as kNN-based methods have been studied extensively in the context of language models and machine translation. I see the authors already addressed these concerns during the rebuttal phase. I believe integrating these explanations and providing more insights into the unique aspects of their kNN-CM approach to clarify the contribution will address the potential concerns in this direction. Another point that lacks clarity and has not been raised by the reviewers is whether the applicability of kNN-CM is limited to masked language models or if it can also be employed with autoregressive models. I think clarifying this point is also important.

One additional aspect worth noting is the authors' mention of potential extensions for the proposed approach in conversation modeling and continual learning. However, the paper does not delve into the specifics of how kNN-LM could be applied in these contexts. I would recommend either providing more detailed insights on these potential extensions or considering the removal of this reference if further elaboration is not possible. This would help to maintain clarity and avoid raising expectations without sufficient follow-through.
Decision: Accept-Findings