Retrieval-Augmented Few-shot Text Classification
Authors: Guoxin Yu
Lemao Liu
Haiyun Jiang
Shuming Shi
Xiang Ao
Venue: EMNLP 2023 Findings (2023)
Keywords: Retrieval-augmented methods
Few-shot text classification
TL;DR: We propose two new training objectives to adapt retrieval-augmented methods in few-shot scenarios.

Abstract: Retrieval-augmented methods are successful in the standard scenario where the retrieval space is sufficient; whereas in the few-shot scenario with limited retrieval space, this paper shows it is non-trivial to put them into practice. First, it is impossible to retrieve semantically similar examples by using an off-the-shelf metric and it is crucial to learn a task-specific retrieval metric; Second, our preliminary experiments demonstrate that it is difficult to optimize a plausible metric by minimizing the standard cross-entropy loss.
The in-depth analyses quantitatively show minimizing cross-entropy loss suffers from the weak supervision signals and the severe gradient vanishing issue during the optimization. To address these issues, we introduce two novel training objectives, namely EM-L and R-L, which provide more task-specific guidance to the retrieval metric by the EM algorithm and a ranking-based loss, respectively. Extensive experiments on $10$ datasets prove the superiority of the proposed retrieval augmented methods on the performance.

Retrieval-Angmentte Few-shot Text Classification
Guoxin Yu
Lemao Liu
Haiyun Jiang Shuming Shi Xiang Ao
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS),
Institute of Computing Technology, CAS, Beijing 100190, China. Peng Cheng Laboratory.
University of Chinese Academy of Sciences, Beijing 100049, China.
Tencent Al Lab, China. Institute of Intelligent Computing Technology, Suzhou, CAS.
(yuguoxin20g, aoxiang)@ict. ac. cn
(redmondliu, haiyunjiang, Shumingshijjteencetttom
Abstract
On the other hand, retrieval- methods have witnessed success on various natural language pro- cessing tasks, thanks to their capability of incorpo- rating retrieved memory alongside parameters for better generalization. These methods retrieve rele- vant examples memories from large-scale cor- pus through either static retrieval metric (Lewis et al., 2020; Wang et al., 2022) or a joint learning- based metric (Cai et al., 2021; Siriwardhana et al., 2023) and then the retrieved examples are used to make a prediction. In this way, their general- ization ability is achieved by not only the model parameters but also the retrieved memory.
Retrieval-augneen augmented methods are successful in the standard scenario where the retrieval space is sufficient; whereas in the few-shot scenario with limited retrieval space, this paper shows it non-trivial to put them into practice. First, it is impossible retrieve semantically simi- lar examples by using an aff-the-shelf metric and it crucial learn task-specific retrieval metrie; Second, our preliminary experiments demonstrate that it is difficult to optimize a plausible metric by minimizing the standard cross-entropy loss. The in-depth analyses quan- titatively show minimizing cross- entropy loss suffers from the weak supervision signals and the severe gradient vanishing issue during the optimization. address these issues, we intro- duce two novel training objectives, namely EM- I. and R-I., which provide more task-specific guidance the retrieval metric by the EM al- gorithm and ranking-hasee hased loss, respectively. Extensive experiments on 10 datasets prove the superiority of the proposed retrieval augmented methods on the performance.
Despite the theoretical potential of promoting generalization by using retrieved memory, previous retrieval-augment methods empirically struggle to showcase compelling ability in few-shot learning scenarios, where the retrieval space (i.e., the few- shot training data) is limited. Specifically, static trieval may lack neighbors with high metries in the case of limited retrieval space. Even though such neighbors exist, static retrieval cannot be reliable for retrieving really helpful samples for target pre- diction, because its metric is not task-specific. particular, for joint learning-based retrieval which minimizes the standard cross-entropy based loss, although the retrieval metric is updated towards the downstream task, it suffers from the gradient vanishing problem during the optimization process as quantitatively measured in Fig. 2 (see $5.2 later). As result, in few-shot scenario, the retrieval met- ric might be not optimized well due to insufficient training data.
1 Introduction
Few-shot text classification, which entails learn ing new task based on limited training data, has been advanced by pre-trained language mod- els (PLMs) (Brown et al., 2020; Liu al., 2023) and prompt engincering (Gao et al., 2021; Chen et al., 2022a). However, since training numerous parameters of PLMS on scarce data is prone to pro- duce over-fitting (Liu et al., 2021) and unstable generalization, only using the trained parameters for inference usually leads to unsatisfactory perfor- mance on unseen test data.
To overcome the aforementioned challenges, we propose two novel training objectives, namely Expectation Maximization-basat Loss (EM-L) and manking-based Loss (R-L), for learning to retrieve
"Wor done while this author was an intern at Tencent.
*Corresponding authors.

---

examples from limited space more effectively. Both objectives are committed to obviating the gra- dient vanishing problem and prioritizing more ben- eficial examples for specific downstream tasks. In the EM-L approach, the retrieved examples are treated as latent variables, and an iterative process of Expectation-step and Maximization-ster is em- ployed until convergence (Dempster et al., 1977). The posterior distribution of the latent variable es- timated to measure the importance of candidate ex- amples in the E-step, while the M-step maximizes the expectation log-likelihood. By approximating the retrieval metric according to the posterior prob- ability, more productive examples could be recalled for downstream tasks with limited training data.
2 Revisiting Retrieval-augmented Methods in Few-shot Learning
2.1 Retrieval-augmmnted Methods
In this paper, we revisit the retrieval -uggmeeeedd methods in few-shot text classification and formu- late the task in general framework. Our primary objective is to retrieve examples from limited train- ing data to improve the few-shot text classification.
Model Formulation All retrieval methods could comprise an example retriever and text classi- fier. We provide the formal formulation inspired by Singh et al. (2021) and Izacard et al. (2022);
Po,olu|x) = Σ 2,lu 5=1
Following similar idea, R-L optimizes an addi- tional ranking loss function to provide more direct supervision to the examples retriever, which draws inspiration from pair-wise ranking algorithm (Fre- und and Schapire, 1997; Burges et al., 2005; Rudin and Schapire, 2009). Such a tailored loss measures the consistency between the retrieval metric and the auxiliary function associated with each exam- ple for classification purposes. Minimizing the loss could effectively strengthen the supervision signals for the example retriever.
2z))) softmax(far (x 2j)), Pa(zz(x) = fretr(X, zx),),
where x and 2, denote the representations of orig- inal input and a retrieved example from the train- ing set, and y corresponds to the class associated with input æ. felr and fretr serve as the text clas- sifier and the example retriever, which selects ex- amples according to retrieval metric. O and ø denote the trainable parameters of the text classi- fier and examples retriever. m is a hyperparameter that denotes the number of fetched examples. The operation @ signifies concatenation, and the term softmax refers to the normalized exponential func- tion. Specifically, z corresponds to a set of retrieval examples, which can either be ((x", ,v")) pairs or {x*). The latter form is adopted in this paper for simple experiments.
Our experimental evaluation on ten text classifi- cation datasets demonstrates the superiority of EM- L and R-L over cxisting retrieval methods within limited retrieval space. The comparative analyses further confirm that EM-L and R-L alleviate the weak supervisory signals and gradient vanishing issue suffered by joint learning-basee based retrieval. Our contributions could be summarized as follows:
The standard cross entropy is employed to opti- mize the classifier and example retriever as follows:
We discuss the weak supervision signals and gra- dient vanishing problem encountered by existing retrieval methods minimizing the standard cross- entropy loss, quantitatively measured in $5.2.
n
C
Σ log (o.o(ux(x))
(2)
where is the total number of training instances and y is the gold label of the i-th instance. During inference, for all retrieval methods, we select top m examples according to (z)))xx and get the final classification results using the first line of Eq. (1).
We introduce two novel training objectives, namely EM-L and R-L, which optimize the re- triever more effectively, thus recalling more pro- ductive examples from limited space.
Extensive experiments and analyses demonstrate that the proposed methods achieve better perfor- mance on few-shot text classification and alle- viate the supervision insufficiency and gradient vanishing issues.
Static Retrieval Given an input sentence x and retrieval corpus, static retrieval aims to search for set of relevant examples Z according to fixed retrieval metric (Borgcaud et al., 2022; Wang et al., 2022; Li et al., 2022). Following the Eq. (1), its

---

retrieval metric is defined as follows:
static retrieval even underperforms methods with- out retrieval in some few-shot tasks (see Table 1). Such failure can also be attributed to data limitation in few-shot scenarios, where examples with high static similarities are scarce or non-existent.
Pa(zz|x) = frour(x, 2j) = sim(x, Zz). (3)
Here, sim(x, zj) represents a fixed metric with- out any trainable parameters, such as TF- IDF (Sparck Jones, 1972), BM25 (Robertson et al., 2009), and semantic similarity encoded by PLMs. Such fixed metrics cannot adapt to the downstream task and prioritize the most helpful examples. Par- ticularly, this limitation will be amplified in few- shot learning with scarce training data.
In addition, joint learning-based retrieval meth- ods (Ren et al., 2021; Cai et al., 2021; Siriward- hana et al., 2023) are good solutions enhance the adaptability of the retrieval to downstream tasks, However, our study demonstrates that learnable metrics struggle to be trained as anticipated and are inferior to static metrics in several few-shot tasks (see Table I). The main underlying factors are the scarcity of data and the weak supervision signals provided to the learnable retrieval metric. In more detail, the retrieval metrics in joint learning- based methods are adjusted solely based on distant supervision from the downstream tasks, which is significantly further weakened by the limited data. This fact is further supported by quantifying the gradient of retrieval parameters: the gradient norm of the parameters in retrieval metric is more than le 6 for only about 10% updates in some datasets as shown in Figure 2 (see $5.2 later).
Joint Learning based Retrieval Static retrieval assumes that higher similarity between 2, and x im- plies a greater auxiliary effect of 2, on x. However, the assumption failed to hold in tasks where inputs with high similarity have distinct labels, such as sentiment classification. To address this limitation, joint learning- based retrieval (Cai et al., 2021; Gao al., 2022; Siriwardhana et al., 2023) unifies the retriever and the downstream model to jointly train them for specific tasks. Following Eq. (1),
exp(x 2
Pa(zz[x)= = (x,))( Σラ= exp(x 2,
In this paper, our objective is to meet the chal- lenges of weak supervision signals for the retriever and insufficient data, aiming to retrieve the most helpful examples to promote model generalization.
fretr(X, 2y) is trainable dot product attention. No- tably, the absence of ground truth for (o(z (x) makes challenging to determine which zj is the most beneficial one, and it relics implicitly on dis-
3
Methodology
3.1 Overview
tant supervision from text classification.
Both static retrieval and joint learning-based re- trieval are proposed to retrieve examples from a large-scale corpus. In this paper, we mainly focus on seo-shot text classification and retrieve the most helpful examples from the limited training set.
Given the limitations posed by limited data and weak supervision signals, existing retrieval meth- ods are inadequate for addressing these challenges. To address these limitations, we propose two novel training objectives, which are achieved by two loss functions: Expectation Maximization-based Loss (EM-L) and Ranking-based Loss (R-L). Both methods aim to enhance the retrieval quality by giv- ing the retriever more supervisory signals and pri- oritizing examples that are more beneficial for the specific task with limited training data. In essence, we seek to maximize the consistency between the metric distribution P(zz (x) and the classification distribution P(yl x, z,) [vs] as much as possible. In this way, more suitable examples are retrieved and the performance of text classification could be im- proved even in the few-shot scenario. Additionally, we integrated EM-L,, R-L., and two existing retrieval methods with two popular text classification back- bones to compare their respective performance.
2.2 Challenges in Few-shot Learning
While the above retrieval- auggeented methods have shown advancements in various natural language processing tasks, their performance in few-shot learning remains unconvincing. In other words, re- trieving examples from a narrow space to improve few-shot learning is still challenging due to lim- ited training data. Previous studies (Li et al., 2022; Siriwardhana et al., 2023) have revealed that static retrieval may not fetch the most helpful examples in tasks where similar inputs correspond to different labels, primarily due to their unreasonable assump- tion that higher similarity implics better suitability for the downstream task. Moreover, we also find

---

3.2 Backbone
where Po(y)x, 2z) and Po(zz|x) are obtained from classifier feir and examples retriever freur in Eq. (1) respectively. m denotes the number of retrieved examples.
Fine-tune Pre-trained Language Models For each sentence, we use PLMs to tokenize the in- put sentence into {[CLS], 21, ... 21, [SEP]] with (I + 2) tokens and extract the representation x of [CLS] as the sentence embedding. In the same way, the j-th retrieved example is represented as
In the M-step, the parameters updated by maximizing the expected log-likelihood, which is taken with respect the estimated posterior Po.o (2,|x,
2/. These tensors are subsequently fed into the example retriever and classifier, producing the final probability estimated for label y-
(x, y) in the E-step:
Po.cuux)= = Σ Poolzz]xx.1) log Po(yix, 2,)).
Prompt Learning Another backbone is to trans- form the text classification into a cloze question problem (Schick and Schütze, 2021). Let M be masked language model with vocabulary V. and Y denote the label set of specific downstream task A. Prompt learning employs function to convert an input sentence into phrase containing a prompt with a [MASK] token. Then an injective function : + V is utilized to map each label to a word from MM' vocabulary V. We first obtain the representation of [MASK] and determine the most suitable word from V for filling the [MASK]. For instance, the application of prompt learning sen- timent classification can be outlined as follows:
(7)
Since sample m examples from the training set by (z(z)) x) and estimate Po.o |xx y) based on m examples in the E-step, more supervision will be provided to the retriever during the optimization in the M-step. Please refer to Appendix A for proof of rationality of (q.(6) and why EM-L can minimize the likelihood-basse loss defined in Eq. (2).
3.4 Ranking-based Loss (R-L)
Following the main idea claimed in $ 3.1, Ranking- based Loss (R-L) considers the process of retriev- ing Zj as ranking task. Unlike EM-L, R-L em- ploys a ranking loss to enhance the consistency between Po(y)x, 2y)[u]] and Po(z,Ix) and provide more direct signals to the retriever. The optimiza- tion objective of R-L aims to ensure that zj with higher (o(y) x. 2;) [BA( has higher Po by min- imizing the following Lx:
P(x) = [CCLS], #1, Zt. was [MASK], [SEP]]
Ply|x) = g(P((MMSK] (x))
v(y) E (great, terrible),
(5) where x the representation of [MASK], g con- verts the probability of label words to classes, and is sentence length. The representation Zj of retrieved example is yielded from [MASK] token in the same way.
Lu Σ
max (Px (x)
(8)
3.3 Expectation Maximization-basee Loss (EM-L)
Po(lz)) 5,0).
Here, Po(y|x, 2,) and Po(z)1x) are obtained from fer and freir in Eq. (1), m and n denote the number of retrieved examples and training instances. ō is a margin parameter imposing the distance between two distributions to be larger than 6.
Considering the absence of the ground truth for Pa(zzix) in Eq. (1), regard z as latent variable and propose an EMMbased retrieval objective to estimate Po [z,] [x). This method alternates between an Expectation-step and a Maximization--tey until convergence. In the E-step, the current parameters are used to estimate the posterior distribution of the latent variable given the observed data. Specifically, we retrieve examples from the training set and compute the conditional probabilities of the latent variable using:
The ranking loss LR is added to the overall loss C in Eq. (2) with a weight À every step:
Lsum L A CR:
1, step mod t= 0;
(9)
A
0, otherwise;
Po(vxx, Pa
(z,0(z)(Xx)) (2))))
Σイ Poly
where > 0 is a hyperparameter to trade off both loss terms, and step denotes the training steps.
(6)

---

Single Sentence
Sentence Pair
ABSA
Avg.
Model
SST-2
MR
CR
TREC
QQP
QNLI
MNLI
SNLT
RES
LAP
Prompr Learning wich RoBerto-Lagge
Vanilla
84. 34.544a.50)
(7.88
28.3
27.20
64,25 (7.00
6000
64.56)4.08)
71.81
73.87
Static
88, n0
83.67 (0.00)
90.956 (7.06
68.31 (2.00)
(6.2T(ras)
60.388
68.17 68.17 (3.00)
70.96p5-0)
73.02
75.74
Joint
90. ml..
86.83
86.83
30.57:amm)
67 26
63. 15jras)
61. 06
67 (4
71 71.07
73. 73.32
15.83
EM-L
22.32
87.58 L.an)
o.000
1.413/1-45)
74.41 (o.74)
(.77
04.85/221
69.52 69.52 (s.00)
23. 23.24/
76.02 1.000
78.72
R-1.
9
87.47 (o.
89. (1.70)
92.86(1.1) m1
(2.09)2.00) (2.28)
(6.79
(3.88
(a.20)
76.79 (2.00
75.59
79.46
Fine RoBerto Large
Vanilla
81. 81.59
73.59 (000
$1.63
85, 55.95
(1.42((.19)
57.20kpa9)
69.90
59.19
69.21;414)
71.06
70.07
Static
81. ay
p3.)))
87 27.02(225)
60. (0.23(g.0))
57. (7.11
61.69(4.7
62 (2.65)g.0)
70.48/s74)
71.37
70.10
Joint
(3.49
24.89
(0.63
(1.33:3177
63.500
57. 37.00
60.90 (0.99
(1.01
70.23
70.62:447)
70.94
EM-L
85.38 a.g
75.80 puaa))
83.87 (5:00)
29.3
(0.93 (.m
62.24 (5:14)
(a.20)
(1.64
(2.69
73.27
R-L
84.69
y5.35
(3.17
(..92
(0.53
(1.37
62.18 1..18 :2)
(a.3)
(3.28
(2.69
73.85
Table 1: Comparison results on 16-shot text classification. "Vanilla" denotes methods without retrieval, which only consists of a sentence encoder and classifier. "Static" and "Joint" are static retrieval and joint learning-based retrieval, which are introduced in $2. "EM-L" and "R-L" are methods implemented with our proposed new objectives. All the reported results average Accuracy and the standard deviation in the subscript.
4 Experimental Results
for classification. Kendall's 's T is defined as follows:
4.1 Experimental Settings
T
2
m
sign(u, uj - ux) sign(w; -Kk,,
We compared the proposed EM-L and R-L approaches with existing retrieval methods by conducting experiments on 10 widely used text classification datasets, including single-sentence classification, sentence pair classification, and aspect-based sentiment classification. We created few-shot datasets following Gao et al. (2021). For more details, please refer to Appendix B.
Datasets
m m 1) jck
u ~ Pa(z|/x), u Polulx; 2) [-1, 1],
(10)
where sign(-) E {-1,0,13 is a sign function. A ranking pair (j. k) is concordant if their ranks have the same order in Po(z(x:) and Po(y)x;, 2) [ui]. Consequently, positive indicates positive cor- relation between two distributions, and vice versa. For n instances x; in the training set, we calculate the proportion of x; with TÚ > 0 as follows:
To prove the effectiveness of retrieving examples from the training set, we develop base- line method without retrieval for comparison. It comprises input encoder described in $ 3.2 and feed-forward neural network for classification. For comparing different retrieval methods, we evalu- ated our EM-L and R-L against static retrieval and joint learning- based retrieval. We combine them with two widely used backbones for text classifica- tion: pre-trained language models fine-tuning and prompt learning. Please refer to Appendix C for more implementations, such as hyper-parametent and templates in prompt learning.
Baselines
7' Σ step(Ti) TT)
(11)
0, T
step(Ti)
1, 7>0
The reported Kendall's in the following experi- ment is actually 7', which represents the proportion of instances with r; > 0.
4.2 Main Results
The experimental results for 16-shot setting on 10 datasets are reported in Table 1, where different retrieval-based methods are combined with two backbones. Several insightful observations could be drawn from the results.
Evaluation. We evaluate all the retrieval meth- ods using two metries: Accuracy and Kendall's т. Accuracy represents the proportion of correctly classified instances out of the total number of in- stances. Kendall's T is employed to measure the consistency and correlations between the retrieval metric (z(x;) and its auxiliary Po (v(2) (2, z) [v4]
Retrieving examples from the training set effec- tive in few-shot scenarios. Firstly, in most datasets, retrieval-augmented models outperform the vanilla

---

Kendall's
SST-2
CR
QQP
QNLI
RES
Accuracy
SST-2
MR
TREC
QQP
Stutic
0.5344
0.5837
0.4307
0.5312
0,47857
Vanilla
80.22
60.71
86.05
64.27
Joint
0.5413
0.6129
0,4776
0.5937
0,4732
EM-L
0.6853
0.6457
0.6265
0.7500
0.6598
Static
76.58
67.51
86.94
60.30
R-L
0.7442
0.6562
0.6057
0.7185
0.6125
Joint
85.41
71.01
86.57
61.92
EM-L
87.30
78.75
87.52
67.90
Table 2: Kendall's of (z(z)(x))) and Po(y//x, Lu).
R-L
89.79
77.38
88.78
66.77
Table 3: Comparison results on 8-shot text classification. Standard deviations are omitted to save space.
model with two backbones, indicating that retriev- ing examples from the training set could enhance the generalization, Secondly, the joint learning-basee based retrieval, EM- I, and R-L perform better than the static retrieval, which is even less effective than the vanilla model. We hold that this is because static retrieval fetches some examples with high semantic similarities but is detrimental to the downstream tasks. In contrast, the learnable retrieval methods, i.c. joint learning- based retrieval, EM-L, and R-L, are more likely to align with the goals of specific tasks.
with narrow search scope.
MR
TREC
RES
LAP
Accuracy
Vanilla
90.80
96.80
86.53
80.87
Static
91.40
97.60
87.50
81.19
Joint
90.90
97.80
87.58
82.13
EM-I
91.70
98.00
88.04
82.76
R-I.
91.45
98.00
88.48
83.22
Kendail's
Static
0.4340
0.5280
0.5705
0.4310
EM-L and R-L approaches train the retriever more effectively than static retrieval and joint Searning-based retrieval. At first, our proposed EM-L and R-L achieve significantly higher accu- racy across different backbones, proving their effec- tiveness in fetching helpful examples and adapting specific downstream tasks. Furthermore, av- erage. R-L outperforms EM-L, potentially due to its utilization of more direct ranking loss that provides more significant signals and flexible guid- worth
Joint
0,5075
0,6580
0.7187
0.7492
EM-L
0.9195
0.7880
0.8700
0.8564
R-I.
0.9090
0.7160
0.8889
0.8903
Table 4: Comparison results with full supervision of the original datasets. Standard deviations are omitted save space.
and R-L indicates that they could prioritize more helpful examples according to their correspond- ing metrics and improve the performance by train- ing more effective retrievers. However, retriev- ing examples according to static metrics and joint learning beaseing-based metrics may result in the inclusion of harmful examples in the final performance.
ance to the example retriever. Finally, noting that EM-L and R-L show smaller standard deviations on most datasets than other methods, we conjecture that the proposed training objectives enhance the stability of generalization by incorpo- rating retrieval memory alongside parameters.
The advantages of EM-L and R-L are more pro- nounced on challenging tasks, such as sentence pair classification, and aspect-based sentiment anal- ysis. In this regard, EM-L and R-L achieve im- provements of more than 0.3 on most datasets for sentence pair classification and ABSA, whereas the improvement on the single-sentence classifi- cation ranges from 0.1 to 0.2, which gain further highlights the effectiveness EM-L and R-L.
4.4 Auxiliary Experiment
We further conduct additional experiments in both 8-shot and full supervision settings to investigate the advantages of EM-L and R-L on different data scales. The results are presented in Table 3 and Table 4, respectively. It is obvious that EM-L and R-L. consistently exhibit excellence in both settings. Particularly, we note more significant improve- ment of our methods in the 8-shot setting, which manifests that the proposed training methods train the retriever more effectively, especially when the training data is scarce.
4.3 Consistency Experiments
The Kendall's r' defined in Eq. (11) on selected datasets are reported in Table 2, which measures the consistency between retrieval metrics of fetched examples and their auxiliaries to downstream tasks. Combing the results Table 1, higher 7' of EM-L
Moreover, another interesting phenomenon emerged: although E.MLL and R-L achieve higher Kendall's r' in the full supervision setting, their im- provements in text classification are comparatively

---

smaller compared that in few-shot scenarios. We believe this can be attributed the fact that the classifier in the full supervision setting already well-trained so the potential improvement from a better retrieval memory is relatively limited.
based retrieval provide weaker supervision signals to the retrievers and suffer from severe vanishing issues in few-shot text classification while EM-L and R-L alleviate such limitations.
5.3 Case Study
5 Analysis
Finally, we present illustrative example from the LAP dataset along with the retrieved examples using different methods in Fig. In the input sen- tence, the aspect term "startup times" negative Although static retrieval fetches a semantic similar example, it includes information that could poten- tially mislead the sentiment prediction, such as the term "spectacular". The joint learning based trieval retrieves an example that seems unrelated to the input sentence, possibly indicating that weak supervision signals for the retriever are prone worse retrieval results. In contrast, our EM-L and R-L methods are capable of retrieving examples that may not possess high semantic similarity but are more beneficial for sentiment prediction.
5.1 Effects of the Number of Retrieved Examples
To examine the effects of the number on var- ious retriever training methods, present line charts in Fig. 1 that depict the relationship between Accuracy and m. First, the charts demonstrate retrieving examples could enhance the performance fho-shot text classification, except for a slightly lower accuracy of static retrieval and joint learning- based retrieval when m takes specific values. This could be attributed to the instability of their train- ing process. Second, most methods achieve their peak performance at m
or m 10. As m continues increase, the performance may start to deteriorate. We guess the reason that retrieving too many examples increases the training difficulty. Third, we observe EM-L and R-L maintain sus- taining advantages and stability as m varies, which verifies their stronger supervision signals. Another observation is that the joint learning based method falls behind the static method on LAP. This find- ing suggests that in certain tasks, a poorly trained learnable metric even exhibits inferior performance compared to a static metric.
Related Work
6.1 Retrieval-augmenttr Methods
Retrieval-augmented methods enhance the ability of the Pre-trained Language Models in processing various natural language tasks by fetching relevant examples from the training set or external knowl- edge base and prepending them with the original input. These methods have improved the perfor- mance of lot of tasks, such as neural machine translation (Zhang et al., 2018; Cai et al., 2021; Li et al., 2022; Wang et al., 2022), question an- swering (Li et al., 2020; Karpukhin et al., 2020; Singh et al., 2021; Wang et al., 2022; Siriward- hana et al., 2023; Li et al., 2023; Hofstätter et al., 2023), dialog generation (Fan et al., 2021; Thulke et al., 2021; King and Flanigan, 2023), text classi- fication (Izacard et al., 2022; Lewis al., 2020), keyphrase generation (Gao al., 2022), etc. Ac- cording to retrieval metrics, these methods could be categorized as static retrieval methods and joint lcarning-base methods, which use fixed retrieval metric and jointly learnable metric respectively. Different from the above methods, which fetch relevant examples from the large-scale corpus, we propose two novel training objectives retrieve examples in restricted retrieval space and analyze their advantages. Following Singh Izacard et al. (2022), we formulate the retrieval- augmented methods into a retriever and a classifier
5.2 Gradient Updates
In order to assess the supervision signals exerted on the retrievers by different methods, we quantify the average gradients of all retrievers' parameters. This evaluate the guidance provided by each method to the retriever during the training process. Fig. 2 illustrates the percentage of training steps where the average gradients of all re- trievers' parameters exceed the threshold of le 6.
measurement allows
For clarity, we exclude static retrieval from this fig- ure since its retriever has no trainable parameters Our analysis revealed that certain datasets, the gradient norm of the joint learning- based retriever exceeds the threshold of le 6 for only about 40% of the steps, whereas EM-L and R-L surpass this threshold in over 60% of the steps. This obser- vation suggests that both static and joint learning-
(2021);
This corresponds to a constant proportion of zero for steps with a gradient norm excceding Ic-6.

---

State
0.8
2011
0.75
0.95
Ace
0.65
0.90
0.7
[ML
0
0
15
10
25
The number in scarchod examples
The number scarched examples
The number mn scarched examples
(a) CR
(b) QQP
LAP
Figure 1: Effects of the number m of retrieved examples. The results are average Accuracy on the validation set.
6.3 Few-shot Text Classification
1.0
The proportion of steps %
Joint Learinn
Few-shot Text Classification trains classifier with limited data for each class, which can also predict unseen classes. Existing studies for few-shot text classification encompass various approaches such as prototypical networks (Jake et al., 201 7), XLNet- based methods (Zhilin et al., 2019). (ER)))) based methods (Chen et al., 2020, 2022a), Pattern- exploiting training (Schick and Schütze, 2021), prompt tuning (Lester al., 2021; Gao al., 2021), etc. And common tab-tasks in text classification consist of intention classification, topic classifica- tion, sentiment classification, etc. We evaluate our methods on different text classification tasks, with focus on adapting the idea of retrieval-augmentt -ugmmetted methods to the few-shot scenarios through the de- sign of new training objectives.
0.8
ONLI
Datasets
Figure 2: proportion steps which average
gradient of retriever's all parameters is more than le
in Eq. (1) for a fair comparison.
6.2 Prompt Engineering
Fueled by the birth of large-scale language mod- els (Brown et al., 2020), prompt-based learn- ing (Liu et al., 2023) for the Pre-trained Lan- guage Models has been developed to convert dif- ferent downstream tasks into cloze-style mask lan- guage model objectives, achieving impressive per- formance in text classification (Wang et al., 2021;
Conclusion
This paper studies the retrieval-augmeettd methods for few-shot text classification and demonstrates the challenges which hinder their success: it is im- possible retrieve semantically similar examples by using an off-the-shelf metric and it is difficult to optimize plausible metric by minimizing the standard cross-entropy loss. Accordingly, it pro- poses two novel training objectives, EM-L and R-L, which provide stronger supervision signals to train the retrieval metric effectively in few-shot scenar- ios. It is worth mentioning that the idea of scarch- ing within limited examples bears similarity the concept of demonstration selection in recent large language models (LL.Ms). Exploring the applica- tion our methods LLMs holds promise for future research.
Gao al., 2021; Hambardzumyan et al., 2021;
Lester et al., 2021; Schick et al., 2020; Schick and Schütze, 2021), sentiment classification (Seoh et al., 2021; Yan et al., 2021: Chen and Qian, 2020; Zhang et al., 2021), named entity recogni- tion (Cui et al., 2021), relation extraction (Chen et al., 2022b,b), question answering (Lewis et al., 2019; Khashabi al., 2020), commonsense reason- ing (Shwartz et al., 2020), etc. Orthogonal these studies of prompt learning, our paper focuses on the comparison of different retrieval methods, where prompt learning is just employed as a backbone.

---

Input: Startup times incredibly long : two minules. The sentiment polarity of startup times <mask>
Labels for Retrieved
Methods
Predictions
Retrieved Examples
Examples
The internet speed is spectacular. The sentiment polarity of internet speed was <mask>
Static
positive
positive
That included the extra Sony Sonie Stage software the speakers
and the subwoofer got -LRB- that WAS worth the money- RRB- the bluetooth mouse for my supposedly bluetooth enabled computer, , the extended life battery and the docking port. The sentiment polarity of docking port was <mask>
Joint
positive
neutral
Its not just slow the internet, its slow in general. The sentiment polarity of internet was <mask>
EM-L
negative
negative
Another thing that after only month the keyboard broke and costed $175 to send it in to fix it The sentiment polarity of keyboard was <mask>
R-L.
negative
negative
Figure 3: Case Study. "Input" denotes an input sentence from LAP, "Predictions" represents the predicted sentiment polarities of different methods, and "Retrieved Examples" is the fetched examples with the highest metric in the training set. "Labels for Retrieved Example" denotes sentiment labels of the fetched examples.
Limitations
motion Association CAS, Beijing Nova Program 220110000682066.
There are three primary limitations of our methods. Firstly, EM-L and R-L require additional training time compared to existing retrieval methods. It is due to the alternation between the E-step and M-step in EM-L and the optimization of addi- tional loss of R-L. Specifically, the training time for EM-L per epoch approximately 1.5 times that of static retrieval and 1.2 times that of joint learning- based retricval. Similarly, the training time for R-L per epoch is about 1.8 times that of static retrieval and 1.5 times that of joint sorming-bbsed retrieval, Although our proposed methods require more time, they still fall within the acceptable range. Secondly, we didn't 't focus designing more sophisticated templates for prompt engineering, as our main em- phasis was on exploring different retrieval methods, Thirdly, we evaluate our methods in few-shot set- tings constructed from widely used datasets, rather than real-world scenes. This could limit the gener- alizability of our findings to practical applications.
References
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, -aan-ttptstte Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206 2240. PMLR.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508. met..05088000
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Chris Burges. Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference Machine learning, pages 89-96.
A Acknowledgements
The research work is supported by the National Key R&D Plan No. 2022YFC3303303, the Na- tional Natural Science Foundation of China un- der Grant (No.61976204). This study is also sup- ported by grants from the Major Key Project of PCL (Grant Number: PCL,022D01) and the CAAT Huawei MindSpore Open Fund. Xiang Ao is also supported by the Project of Youth Innovation Pro-
Deng Cai, Yan Wang. Huayang Li, Wai Lam, and Lemao Liu. 2021. Neural machine translation with monolingual translation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 7307-7318, Online. Association for Computational Linguistics.

---

Xiang Chen, Lei Li, Ningyu Zhang, Xiaozhuan Liang. Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022a. Decoupling knowledge from memorization: Retrieval-augmenter prompt learning. arXiv preprini arXiv:2205. 14704.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-leve Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 4921-4933, Online, Association for Computational Linguistics.
Xiang Chen, Ningyu Zhang. Xin Xie, Shumin Deng. Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022b. Knowprompt: Knowledge- with synergistic optimization
Sebastian Hofstätter, Jiccao Chen, Karthik Raman, and Hamed Zamani. 2023. Fid-light: Efficient and effec- tive
aware prompt for relation extraction. In Proceedings of the ACM Web Conference 2022, pages 2778-2788.
text generation. In Proceed- ings of the 46th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, pages 1437-1447
Xilun Chen, Asish Ghoshal, Yashar Mchdad, Luke Zettlemoyer, and Sonal Gupta. 2020. Low-resource domain adaptation for compositional fask-oriented semantic parsing. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP). pages 5090-5100, Online. As- sociation for Computational Linguistics.
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Atlas: Few-shot learning with retrieval augmented language models.
Zhuang Chen and Tieyun Qian. 2020. Enhancing aspect term extraction with soft prototypes. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2107-2117, Online. Association for Computational Linguistics.
Snell Jake, Swersky Kevin, and Zemel Richard. 2017. Prototypical networks for few-shot learning. Ad- vances in Neural Information Processing Systems, 30.
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang, 2021. Template-based named entity recognition us- ing bart, arXiv preprint (XXi:2106600160)
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu. Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical series (methodological), 39(1):1-23
society:
Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2021. Augmenting transformers with knn- based composite memory for dialog. Transactions of the Association for Computational Linguistics, 9:82- 99.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. UNIFIEDQA: Crossing for- mat boundaries with a single QA system. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online, Association for Computational Linguistics.
Yoav Freund and Robert E Schapire. 1997. A decision- theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139
Brendan King and Jeffrey Flanigan. 2023. Diverse retrieval-augmented in-context learning for dialogue state tracking. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5570- 5585.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making tre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 38 1836-3,,,, Online. Association for Computa- tional Linguistics.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficieet prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 0045-3059) Online and Punta Cana, Domini- can Republic, Association for Computational Lin- guistics. Yifan Gao, Qingyu Yin. Zheng Li, Rui Meng Tong Zhao, Bing Yin. Irwin King, and Michael Lyu. 2022. Retrieval-augmeente
multilingual keyphrase gener- ation with retriever-generator iterative training. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1233-1246.
Patrick Lewis, Ludovic Denoyer, and Sehastian Riedel. 2019. Unsupervised question answering by cloze translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics,

---

pages 4896-4910, Florence, Italy, Association for Computational Linguistics.
Stephen Robertson, Hugo Zaragoza. et al. 2009. The probabilisticc relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 334)3333-3
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih. Tim Rock- täschel, et al. 2020. Retrieval- auugeetel generation for wetwwledgeinntesivve nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474.
Cynthia Rudin and Robert E Schapire. 2009, Margin- based ranking and an equivalence between adaboost and rankboost.
Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as labels for few-shot text classification. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 5569-5578, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Huayang Li. Yixuan Su, Deng Cai. Yan Wang, and Lemao Liu. 2022, A survey on text generation. arXiv preprint arXiv:2202.011
Shiyang Li, Yifan Gao, Haoming Jiang, Qingyu Yin, Zheng Li, Xifeng Yan, Chao Zhang, and Bing Yin. 2023. Graph reasoning for question answering with triplet retrieval. arXiv preprint 23055 18742.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
Xiaoya Li, Yuxian Meng, Mingxin Zhou, Qinghong Han, Fei Wu, and Jiwei Li. 2020. Sac: Accelerating and structuring self-attention via sparse adaptive con- nection. Advances in Neural Information Processing Systems, 33: 17008-
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 1-(9):1-35.
Ronald Seoh, Ian Birle, Mrinal Tak, Haw-Shiuan Chang, Brian Pinette, and Alfred Hough. 2021. Open aspect target sentiment classification with natural language prompts. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing. pages 63 l 1-6322, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qiam, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too, arXiv preprint 103.
Vered Shwartz, Peter West, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 4615-4629, Online. Association for Computa- tional Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 16692.
Suresh Manandhar. 2014. Semeval-200 task 4: As- pect based sentiment analysis. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014 4), pages 27-35.
Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. 2021. End-to-end train- ing of multi multi-docummntt reader and retriever for open- domain question answering. Advances in Neural Information Processing Systems, 04-25968-2588)
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cs/0409058.
Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Trans- actions of the Association for Computational Linguis- tics, 11:1-17.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.000 05250.
Ruiyang Ren, Yinggi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji- Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2825-2835, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empiri- cal methods in natural language processing, pages 1631-1642.

---

Karen Sparek Jones. 1972. statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11-21.
Yang Zhilin, Dai Zihang, Yang Yiming, Carbonell Jaime, Salakhutdinov Ruslan, and Quoe V. Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. The 33rd Conference on Neu- ral Information Processing Systems (NeurlPS 2019).
David Thulke, Nico Daheim, Christian Dugast, and Hermann Ney. 2021. Efficient retrieval augmented generation from unstructured knowledge for task- oriented dialog. arXiv preprint WrXiv:2102.0464.
A Proof of the EM-L Method
Proposition. Optimizing the following two like- lihood functions is equivalent in EM-L:
Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 200-207.
II (o.o(v)x)) =
max 0,0
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.
n
m
max 0.0
ΣΣ
Po.o(zz|xs; v) log Polulx; 24),
Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint 1OX.::8844.746..
where (0.0(zz(2x:)
Poly) X, Zi) Po xz)
Chengyu Wang, Jianing Wang, Minghui Qiu, Jun Huang, and Ming Gao. 2021. TransPrompt: Towards an automatic transferable prompting framework for few-shot text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2792-2802, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mj m Po x 23 Zz)
(12)
where x, is the representation of the i-th sentence. For each x,, the retriever fetches m examples from the corpus to assist x; in text classification, where each example is represented as Zj.
Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data is more valuable than you think: A simple and effective method by retrieving from training data. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 3170-3179, Dublin, Ireland. Association for Computational Lin- guistics.
Proof. We first use variational inference to derive the lower bound of the original likelihood:
max II (vlx())
- max log II (y)x())
Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint 1rXhv:1704.05422
max
log Pe. (e.((y)x;)
Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 2021. A unified generative framework for aspect-based sentiment analysis. of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2416-2429 Online. Association for Computational Linguistics.
max
log
2,)
Proceedings
n
max
log
Pa.l(.22(2x))
(13)
Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Gra- ham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. arXiv preprint arxiv: 180400255999
Let Q(z;) be a random distribution of 2y:
12
172
max
log
P Po.o (x:)
Wenxuan Zhang, Yang Deng, Xin Li, Yifei Yuan, Li- dong Bing, and Wai Lam. 2021. Aspect sentiment quad prediction as paraphrase generation. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 9209- 9219. Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
(2, Q(z))
max
log
((z))
(14)
m
m
Poely, 2z(xi)
max Σ
>>8 Q(z,) log
Q(z;)

---

The last step is according to Jansen inequality and equals and only if Qzz,) is proportional to (y. 2j) (x) and è is a constant. Such a propor- tional relationship can be expressed as:
Since Pe, 2z|xx) Po(u) Xi, Pols)())) we can further simplity Eq.(18) as follows:
m Σ ((z,) Po.
max Σ
Q(2)) log ((z)))
((2))
m
(0, 2x(xx) C, is constant
= max
((z,) log Po(u|x 2j) Po
(15)
(Po = Q(zz): Vi,j
Q(z)) log Q(z;))
m
Since ((z)) 1, we can sum z on both sides of the equation:
Σ Q(z,) log Po(y|xx, 23)
max
+
Q(z)) log Po(z,(xi)
m
Po.
Σ
Q(z,) log Q(2)))*
(16)
1
c
max
((z)) log Po 2)))
m Po Pa(zzio v) log Po(v/x;,,,)
max
Now we can derive a lower bound of II: (v)x)) by substituting e into (15( and then substituting ((z;) (14):
(19)
Specifically, in the step denoted with
*
Σ. m Q(z,) log Pa(zz|xx) and
Σ ΣS Q(2,) log Q(z,)) can be canceled
out, because Q(z,) Po(z)|x(,)) s Pa(z)(x;) in Eq. (17).
Po, 2, xx)
Q(z;)
>) th Po 221xx)
Further proof for convergence and equality of
Polyixs, 2j) (o(
(17) the original two optimizations is ordinary to derive as the proof of the EM algorithm, which is omitted here.
En Po(y) xs, 2j) (o(z))
Pa.o (2)(x..1)
B Dataset Detail
B.1 Original Datasets
All the retrieval methods are evaluated on three types of datasets: single-sentence classification, sentence pair classification, and aspect-based senti- ment analysis (ABSA). The single-sentence classi- fication consists of SST-2 (Socher et al., 2013), MR (Pang and Lee, 2004), CR (Hu and Liu, 2004), and TREC (Voorhees and Tice, 2000). The sentence pair classification includes QQP? 2
max II (v)x()
max
Q(z;) log
(18)
x m
Σ Σ (zzj) log Q(z,))
>htps://quoruaataqqurr..om
3

---

Dataset
Input
Output
Train
Test
Type
positive negative
SST-2
sentence a:
6,920
872
sentiment classification
positive negative
MR
sentence a:
8,662
2,000
sentiment classification
positive negative
CR
sentence
1,775
2,000
sentiment classification
Personality
Advisor
2: Conclusion Human
TREC
sentence x
5,452
500
question classification
4 Assignment
5: Minute
entailment
QQP
sentence 3r, #2
363,846
40,431
paraphrase
O: not entailment
1: entailment
QNLI
sentence 21, a2
104,743
5,463
Natural Language Inference
O: not entailment
2: entailment
MNLI
sentence 31,
I: neutral
392,702
9,815
Natural Language Inference
contradiction
entailment neutral
SNLI
sentence Ti, 222
549,367
9,842
Natural Language Inference
O: contradiction
positive
RES
sentence aspect
I: neutral
3,044
800
aspect-based sentiment analysis
negative
2: positive neutral
LAP
sentence x; aspect a
3,048
800
aspect-based based sentiment analysis
negative
Table 5: Dataset details. The column labeled "Train" represents the number of instances in the original training set, while "Test" denotes number instances the test set. The "Type" column describes the task type associated with each dataset.
QNLI (Rajpurkar et al., 2016), SNLI (Bowman et al., 2015), and MNLI (Williams et al., 2017). The aspect-based sentiment analysis datasets are RES (Manandhar, 2014) and LAP (Manandhar, 2014). Particularly, for SST-2, MNLI, and QNLI from GLUE (Wang et al., 2018) and SNLI, utilize their original validation sets for testing pur- poses.
erage evaluation scores on the original test set are reported.
C Experimental Settings
C.1 Hyper-parameter Selection
We adopt grid search to choose the hyper- parameters of different methods. Specifically, the learning rates are taken from {1e-5, 2e- 5e-5). the batch sizes are from (4, 8, 16), and the numbers of retrieved examples are taken from (5, 10, 15).
B.2 Few-shot Datasets
Following the few-shot setting of Gao et al. (2021), we randomly select 16 or 8 examples from the train- ing set to create 16-shot or 8-shot experiments. Specifically, generate five distinct few-shot datasets using different seeds and train models on each of them. noted that we use consistent five seeds on different datasets and retrieval methods to conduct a fair comparison. The best model is chosen based on the validation results, and the av-
The parameter that determines the update fre- quency of loss CR is searched from (5, 10, 15). The loss coefficient À in ranking-based loss is set to (0.5, 1, For each dataset, we set the max training steps as 800 steps and use early stopping to avoid over-fitting, In each trial, we validate the model in each epoch and save the best checkpoint.
We adopt the AdamW optimizer and accumu- late gradients for each batch. The code imple-

---

Dataset
Template
Label
SST-2
1: positive good
MR
Input sentence æ; it was <mask>.
0: negative + terrible
CR
0: -essonaltyy=PPeronnaliyy
1: Advisor->Advisor
2: Conclusion-+CConclusion
TREC
Input sentence 2; it was <mask>.
3: Human
:Assigument
5: Minute
QQP
1: entailment Yes
Input sentence 21, <mask>, 2-2
QNLI
0: entailment No
MNLI
2: positive- positive
Input sentence 21, <mask>, Z2-
SNLI
1: neutral + neutral
0: negative + nagative
RES
2: positive-> positive
Input sentence v, the a was <mask>,
LAP
I: neutral neutral
0: negative - nagative
Table 6: Templates and label words for different datasets that used for prompt-based fine-tuning
mented with PyTorch 1.9.0 and transformers 4.1.1 and launched on an Ubuntu server with single NVIDIA Tesla V100 (32G) or NVIDIA 4090. In addition, we will test our model with Mindspore, which is new deep-learning framework
C.2 Templates of Prompt-based Fine-tuning We use RoBgRTa-Large (Liu et al., 2019)* with 1024 dimensions to encode the input sentences with the related template. The templates for var- ious datasets are shown in Table 6. Since our main aim to investigate the difference among retrieval methods, we adopt the widely used and effective templates for these tasks in comptt-based fine-tuning refer to Gao al. (2021). The specific templates shown in Table 6.
hhnps://www.minddppr..c////// fhanms//gggaaa.ooommnnggnnifeeem


Reviews:
[{"review_id": "NWjaT1VZFa", "reviewer": "Reviewer_uG9t", "summary": "", "questions": "(1) Some of the symbols in Equation 1 are not explained, please add them.\n(2) Regarding the author's mentioned solution to weak supervision signals for the retriever and insufficient data, can it be simply interpreted as using only quality data and discarding the rest?\n(3) This paper lacks an overall framework figure.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable suggestions, which undoubtedly contribute to enhancing the clarity and coherence of our paper. We structure our responses based on distinct categories of questions, including article organization, comparison with similar studies, missing references, etc. \n\n**1. Article organization and unclear details**\n\n**W1: We are willing to adjust the length of Section 2 and Section 3 and try to mitigate redundancy between Section 2 and Section 6.** Given our endeavor to claim the limitation of existing retrieval methods and the challenges they faced in few-shot scenarios in Section 2 and the advantages of EM-L and R-L in Section 3, the length has extended slightly. An overlap between Section 2 and Section 6 has inadvertently arisen because both of them delineate some representative retrieval-augmented methods but Section 6 is more comprehensive. \n\n**W2: We will improve the details of datasets and baseline methods in Section 4.1.**\n\nWe have put details of datasets and the construction methodologies applied to create few-shot datasets in Appendix B due to space constraints. Specifically, in our main experiment, we evaluate different models on $10$ datasets, categorized as follows: single-sentence classification (**SST-2**, **MR**, **CR**, and **TREC**), sentence pair classification (**QQP**, **QNLI**, **MNLI**, and **SNLI**), and aspect-based sentiment classification (**Res14**, and **Lap14**).\n\nWe primarily compare three baselines, namely **Vanilla**, **Static**, and **Joint**, to prove the effectiveness of EM-L and R-L. All the methods acquire sentence representations either via Pre-trained Language Models (PLMs) or prompt learning using manual templates. **Vanilla** is a simple model without retrieval, which only adopts a feed-forward neural network as a text classifier to identify the category of each sentence. **Static** consists of a text classifier and an examples retriever, the operational mechanics of which are explained in Eq. (1) and Eq. (3).  Similarly, **Joint** also consists of a text classifier and an examples retriever, which are explained in Eq. (1) and Eq. (4). All the three baselines are optimized by minimizing a cross-entropy loss of text classification, as described in Eq. (2). \n\n**W3: To alleviate the simple experimental form, we shall add the supplementary results and more 8-shot experiments** in the final version, if space permits. We will consider including these supplementary materials in the Appendix If space constraints arise.\n\n**Q1: We will add explanations for several symbols in Eq. (1), as outlined below**:  $\\theta$ and $\\phi$ denote the trainable parameters of the text classifier and examples retriever. $z_j$ is the $j$-th retrieved example. $f_{clf}$ and $f_{retr}$ serve as the text classifier and the example retriever, both comprising feed-forward neural networks. $y$ corresponds to the class associated with input $x$. The operation $\\oplus$ signifies concatenation, and the term \"softmax\" refers to the normalized exponential function.\n\n**Q3: We are willing to supplement a figure about the model overview if necessary.**  Since we have concentrated on different retrieval processes under the same prevailing retrieval framework, we paid more attention to the theory statement. \n\nYour suggestions for article organization are helpful to us, and we will improve it and fix all the typos. \n\n**2. Comparison with other similar studies**\n\n**W2: Following your valuable recommendations in W2, we add two additional baselines.**\n\n(1) **Vanilla+KNN** combines a KNN mechanism with the Vanilla Baseline. It retrieves the top $m$ relevant examples and merges their labels with the logits derived from Vanilla according to the relevance scores. \n\n(2) **Gao et al. 2022 (NAACL) [1]** proposed a Retriever-Generator Iterative Training method for the keyphrase generation task, which mines parallel passage pairs with high qualities according to retrieval scores. This method could be directly helpful for training the keyphrase generation module and is executed based on Parallel data $D_{PAR}$ and Non-parallel data $D_{NP}$. However, the context delineated in Gao et al. 2022 (NAACL) diverges from our present investigation, resulting in the nonexistence of the aforesaid $D_{PAR}$ and $D_{NP}$ within our study. \n\nIn response to the reviewers' mention of Gao et al. 2022 (NAACL) in the missing references, we construct Parallel data $D_{PAR}$ and Non-parallel data $D_{NP}$ for our task to create a new baseline. Then the iterative algorithm elucidated by Gao et al. 2022 (NAACL) is incorporated to facilitate the training of our retriever and text classifier.\nSpecifically, for each training example $x$, we first use the pre-trained language model to compute the semantic similarities among $x$ and other examples in the training set. We then select examples whose similarities with $x$ exceed 0.5 to form $D_{PAR}$ while the remaining examples form $D_{NP}$. Finally, we adopt the iterative algorithm in Gao et al. 2022 (NAACL)  to train our model. We set the iterations as $5$ across different datasets. \n\n**We make an additional analysis compared with Vanilla+KNN and Gao et al. 2022 (NAACL).** According to the results reported in Table A, **Vanilla+KNN** performs better than Vanilla, which proves the advantages of retrieval. However, Vanilla+KNN is inferior to Vanilla on CR. We think that the KNN fetches examples with high semantic similarities but harms the performance of downstream tasks, which is similar to the Static baseline analyzed in lines 389 to 392. \n**Gao et al. 2022 (NAACL) [1]** performs better than Vanilla and shows comparable performance on most datasets. However, on most datasets, Gao et al. 2022 (NAACL) [1] is less effective than EM-L and R-L. The reason might be that there is not enough initial Parallel data $D_{PAR}$ in our few-shot setting, which limits the exertion of the iterative algorithm. \n\n**Table A**. The comparison results with the two new baselines on selected datasets.  The parameters of the two baselines are maintained consistent with other baselines. We adopt the backbone of prompt learning.\n| Model                       | TREC         | CR           | SST-2        | Res14        | Lap14        |\n| --------------------------- | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Vanilla                 | 87.20        | 88.36        | 84.84        | 72.05        | 71.81        |\n| Static                  | 90.95        | 87.06        | 88.60        | 70.95        | 73.01        |\n| Joint                   | 90.57        | 86.76        | 90.71        | 71.07        | 73.32        |\n| EM-L                    | 92.13 | 90.00 | 91.31 | 73.74 | 76.02 |\n| R-L                     | **92.86**    | **89.93**    | **91.58**    | **76.79**    | **75.59**    |\n| **Vanilla+KNN**             | 90.38        | 85.79        | 91.24        | 73.35        | 72.95        |\n| **Gao et al. 2022 (NAACL)** | 91.42        | 84.79        | 87.09        | 72.28        | 75.11        |\n\n[1] Yifan Gao, Qingyu Yin, Zheng Li, Rui Meng, Tong Zhao, Bing Yin, Irwin King, and Michael Lyu. 2022. Retrieval-augmented multilingual keyphrase generation with retriever-generator iterative training. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1233–1246.\n\n**3. Solution to weak supervision signals and insufficient data.**\n\n**Q2**: Yes, the proposed two retrieval methods could retrieve examples with high qualities from a limited retrieval space, which can improve the performance of the downstream tasks more effectively. The rest of the examples after retrieval will remain unutilized for downstream tasks.\n\n**4. Missing references**\n\nOur study focuses on different retrieval-augmented methods for few-shot text classification, so some studies that only concentrate on either retrieval-augmented methods or few-shot text classification are not included in the references. Following the reviewer's suggestions, we will cite the following similar published papers and check our references carefully. \n\n[1] Yifan Gao, Qingyu Yin, Zheng Li, Rui Meng, Tong Zhao, Bing Yin, Irwin King, and Michael Lyu. 2022. Retrieval-augmented multilingual keyphrase generation with retriever-generator iterative training. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1233–1246.\n\n[2] Li S, Gao Y, Jiang H, et al. Graph Reasoning for Question Answering with Triplet Retrieval[C]// Findings of the Association for Computational Linguistics: ACL 2023, pages 3366--3375.\n\n[3] Hofstätter S, Chen J, Raman K, et al. Fid-light: Efficient and effective retrieval-augmented text generation[C]//Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023: 1437-1447.\n\n[4] King B, Flanigan J. Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking[C]//Findings of the Association for Computational Linguistics: ACL 2023. 2023: 5570-5585.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The method proposed in this paper have some practical significance.\n(2) The ideas and methods of the paper are clear.\n(3) The experiments in this paper are sufficient and the experimental angles are varied.", "reasons_to_reject": "(1) The overall organization of the article needs to be adjusted, e.g., “Revisiting Retrieval-augmented Methods in Few-shot Learning” and methods both take up one page and there is duplication between “Revisiting Retrieval-augmented Methods in Few-shot Learning” and related work.\n(2) The dataset section does not explicitly give which dataset was used.The baseline section does not give a clear method of comparison and does not compare to similar studies.\n(3) The results of the experimental part show a simple form.", "questions_to_authors": "(1) Some of the symbols in Equation 1 are not explained, please add them.\n(2) Regarding the author's mentioned solution to weak supervision signals for the retriever and insufficient data, can it be simply interpreted as using only quality data and discarding the rest?\n(3) This paper lacks an overall framework figure.", "paper_topic_and_main_contributions": "This paper addresses the problem of weak supervision signals for the retriever and insufficient data. They propose two novel training objectives, namely Expectation Maximization-based Loss (EM-L) and Ranking-based Loss (R-L), for learning to retrieve examples from a limited space more effectively.The main contributions of this paper are as follows:\nThey discuss the weak supervision signals and gradient vanishing problem encountered by existing retrieval methods minimizing the standard cross-entropy loss,\nThey introduce two novel training objectives, namely EM-L and R-L, which optimize the retriever more effectively, thus recalling more productive examples from a limited space.\nExtensive experiments and analyses demonstrate that the proposed methods achieve better performance on few-shot text classification and alleviate the supervision insufficiency and gradient vanishing issues.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "(1) A large portion of the references come from arXiv.\n(2) Some references are not cited in the text. For example Yifan Gao, Qingyu Yin, Zheng Li, Rui Meng, Tong Zhao, Bing Yin, Irwin King, and Michael Lyu. 2022. Retrieval-augmented multilingual keyphrase generation with retriever-generator iterative training. In Findings ofthe Association for Computational Linguistics: NAACL 2022, pages 1233–1246.", "Typos_Grammar_Style_and_Presentation_Improvements": "(1) In model formulation of section 2.1, the sentence “where x and zj denote the representations of original input and a retrieved example from the training set, fclf and fretr serve as…” lacks a conjunction.\n(2) In last paragraph of section 3.4, the sentence “where λ > 0 is a hyperparameter to trade off both loss terms, step denotes the training steps” lacks a conjunction."}}, {"review_id": "PvEgvCmw6v", "reviewer": "Reviewer_U8Tr", "summary": "", "questions": "Other than using the Jansen inequality to show lower bound, would the maximization objective be obtained from normal EM derivation (using ELBO)? What are the differences between your M objective with standard EM (I cannot tell by looking at the final form)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.  We will solve all the formulation problems and clarify the details in the final version. We answer your questions in order as follows. \n\n**W1：Formulation notations**\n\nIn line 140, $s$ in $x^s$ and $y^s$ denote an example sentence, while $x^s$ and $y^s$ respectively stand for the input and the label of the example s. We consider $y$ as a distribution and $y_i$ is a specific label. By your guidance, we will remove the unnecessary spaces in Appendix formulas and follow your suggestions to use math symbols. The employment of the \"*\" symbol in Eq. (9) serves to indicate a forthcoming description of this step, which is shown in lines 945-946. We will revise these problems in the final version.\n\n**W2: Retrieved examples**\n\nOur study found that some helpful retrieved examples do not necessarily maintain a very high semantic similarity with the input sentence. On the contrary, examples with very high semantic similarities may be harmful to downstream performance (refer to lines 389-394). EM-L and R-L tend to exchange a little semantic similarity for greater help to downstream tasks. Furthermore, in Figure 3, the similarity between the retrieved example by R-L and the input sentence lies in that both of them describe a certain high attribute (the \"setup time\" or \"costs\") thus expressing negative sentiment polarities.\n\n**W3: Training time**\n\n(1)In alignment with your insightful recommendations, we further report the training times on selected datasets in Table B. The comparison results meet the results acknowledged in the limitations section. \n\n(2)We also report the training times of different models when the number m of retrieved examples ranges from $1$ to $9$, which are shown in Table C. \n\n**Table B**. Training times under the backbone of prompt learning on selected datasets. The number m of retrieved examples is $5$.\n| Training Time (ms/epoch) | SST-2    | MR       | CR       | TREC      | QNLI     | Res14    | Lap14    | Avg.     |\n| ------------------------ | -------- | -------- | -------- | --------- | -------- | -------- | -------- | -------- |\n| Static                   | 317.7609 | 407.1764 | 419.9924 | 1155.6439 | 388.6265 | 498.2818 | 529.7172 | 531.0284 |\n| Joint                    | 416.3982 | 474.4482 | 385.0958 | 1127.0368 | 396.7046 | 584.0690 | 527.6005 | 558.7647 |\n| EM-L                     | 401.8793 | 626.1048 | 553.6871 | 1329.2260 | 471.6467 | 664.9293 | 639.2867 | 669.0948 |\n| R-L                      | 568.6276 | 739.5091 | 671.3948 | 1500.2369 | 559.8521 | 797.4810 | 731.7583 | 795.5514 |\n\n\n**Table C**. Training times under the backbone of prompt learning when m ranges from $1$ to $9$ on Lap14.\n| Training Time (ms/epoch) | m=1       | m=3      | m=5      | m=7       | m=9      |\n| ------------------------ | --------- | -------- | -------- | --------- | -------- |\n| Static                   | 517.0159  | 525.0875 | 529.7172 | 584.45692 | 572.6985 |\n| Joint                    | 517.5697  | 523.3082 | 527.6005 | 529.60372 | 572.5355 |\n| EM-L                     | 602.61846 | 636.1904 | 639.2867 | 679.8043  | 695.4593 |\n| R-L                      | 706.64191 | 730.2668 | 731.7583 | 739.98213 | 772.3598 |\n\n**W4: Details of $f_{retr}$ and $f_{cls}$**\n\nIn our experiment, we configured $f_{retr}$ and $f_{cls}$ functions as separate RoBERTa. And we will consider your suggestion to release our code. \n\n**Q1: About proof**\n\nBoth EM and ELBO are motivated by the idea of Variational Bayesian Inference. In our derivation process, we provide a detailed presentation of the variational inference procedure by employing Jansen’s inequation to emphasize the practical significance of Eq.(7). Thus obtaining the maximization objective using ELBO is fully valid and justifiable.\n\nUnlike the standard EM algorithm, our results explicitly incorporate Y to emphasize the forward inference process of X -> Z -> Y in our algorithm. This is also done for the convenience of readers to understand and reproduce our algorithmic procedure. After all, deriving the loss function based on the original EM algorithm is not as intuitive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work is theoretically sound and could be significant for few-shot retrieval-augmented classification, potentially also generalizable to retrieval-augmented LLM tasks as well (as the author also indicated in the conclusion). The results are promising in data scarcity settings (especially good in 8-shot experiments) for a limited number of retrieval sample (m as small as 10 is well enough, according to Figure 1). ", "reasons_to_reject": "* The formula notations are not clean. E.g., what is \"s\" in x^s at line 140? Can we write y in Y=y or y_i? And y should be explained when it first occurs in Eq. 1. Also, many unnecessary spaces in Appendix formulas, using math symbols instead of \\text for in-formula texts, and not using \\left( and \\right) makes the derivation unpleasant to read. Plus, there is a \"*\" (star) at the final steps of Eq. 19 for no reason. Overall, these issues are minor and non-critical, but they indicate the potential for polishes.\n\n* The retrieved example in R-L case of Figure 3 looks dubious, as it is irrelevant to the input.\n\n* The training time is important aspects in the evaluation of proposed methods. I suggest the authors give full results on training times instead of mentioning it in the Limitations section (including the effects of m and training samples n).\n\n* Lacking implementation details in terms of f_retr and f_cls functions, are they separate RoBERTa? Or one is stacked on top of the other? Also, I would recommend the authors to state the willingness of publishing their code, as they are critical for understanding the details of this work.", "questions_to_authors": "Other than using the Jansen inequality to show lower bound, would the maximization objective be obtained from normal EM derivation (using ELBO)? What are the differences between your M objective with standard EM (I cannot tell by looking at the final form)?", "paper_topic_and_main_contributions": "This work utilized the idea of EM algorithm to model and to jointly train the retrieval-augmented methods for a better classifier using a retriever.\nThe latent distribution in Eq. 6 can be modeled and estimated by the classifier and retriever in the E-step, subsequently, both the classifier and retriever are jointly optimized in the M-step from the variational distribution (evidence lowerbond). In addition, the paper also proposed a Ranking-based Loss where a high probability prediction given the augmented retrieval results is supposed to have a higher probability to generate these retrieval results.\n\nThe main contribution of this paper is introducing the idea of EM algorithm to model the retrieval-augmented methods and being able to show this is a data-efficient way to optimize both classifier and retriever than previous methods such as Joint Learning based Retrieval. Besides, this work shows the modeling can alleviate the vanishing gradient issue in training deep models, presumably by offering more training signals (with the number of loss terms proportionally to the sampled retrieved results m, I assume). ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3OvgxpY8Xh", "reviewer": "Reviewer_VkCo", "summary": "", "questions": "1. It is not clear why EM-L method does not suffer from the data scarcity problem. \n2. Is it possible to combine EM-L and R-L? If yes, why the paper does not combine them? If not, is there any reason for that? \n3. Will the EM-L suffer from worse training efficiency? Since it alternatively performs the E-step and M-step. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will improve this paper according to your suggestions. We answer your questions in order as follows. \n\n**W1: The combination of EM-L and R-L**\n\nFirst, we didn't combine EM-L and R-L because of their different calculation methods of relevance scores and different optimization objectives.  Although the idea of EM-L and R-L are consistent, the relevance scores of EM-L are computed by estimating the posterior distribution of the latent variable (in Eq. (6)), while that of R-L are calculated by the examples retriever directly. \n\nSecond, we didn't combine EM-L and R-L because they consider the retrieval augmented classification tasks from different perspectives. In the EM-L approach, the retrieved examples are treated as latent variables (lines 282 to 285). In the R-L approach, the retrieved examples are treated as a ranking process (lines 312 to 313). \n\nOf course, according to the reviewer's suggestion, we are happy to conduct this additional experiment: we combine EM-L and R-L to obtain the \"EM-L+R-L\" model and report the comparison results on selected datasets in Table D. As the results show, \"EM-L+R-L\" has no obvious improvement compared with EM-L and E-L. \n\nWe will add the above reasons in the final version for a more comprehensive experimental analysis. \n\n**Table D**. Comparison results. \"EM-L + R-L\" denotes the combination of EM-L and R-L.\n| Model      | TREC      | CR        | SST-2     | Res14     | Lap14     |\n| ---------- | --------- | --------- | --------- | --------- | --------- |\n| EM-L       | 92.13     | **90.00** | 91.31     | 73.74     | 76.02     |\n| R-L        | **92.86** | 89.93     | **91.58** | **76.79** | 75.59     |\n| EM-L+R-L | 92.54     | 89.88     | 91.24     | 73.95     | **76.45** |\n\n**Q1: Data scarcity problem of EM-L**\n\nAs shown in Eq. (6),  on the one hand, EM-L could compute the conditional probabilities of the latent variable based on m retrieved examples, which are more comprehensive. On the other hand, EM-L takes into consideration the auxiliary function $P_θ(y|x, z_j)$ of retrieved example $z_j$ to the downstream task. The above two characteristics provide more evidence for training than existing retrieval methods and alleviate the data scarcity problem, so as to better train the retriever and select more helpful examples. \n\n**Q2: Why not combine EM-L and R-L**\n\nPlease refer to the answer to W1.\n\n**Q3: Training efficiency**\n\nDue to the limited number of retrieved examples and few-shot training data, both the posterior distribution estimation in E-step and the maximization in M-step will not increase too much time. The extra training time caused by alternatives still falls within the acceptable range. Please refer to the limitation section and answer for **W3 to Reviewer U8Tr** for specific efficiency comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper studies a very interesting problem, namely retrieval-based few-shot text classification, which is of great significance to the NLP researchers and practitioners. \n2. The paper is technically sound and the proposed algorithm effectively mitigate the shortcomings of the existing algorithms. \n3. The paper is well written and easy to follow. \n4. The experiments are very solid and comprehensive. The results look great. ", "reasons_to_reject": "1. There are still some unclear issues to be better elaborated (e.g., why the paper does not combine EM-L and R-L?). ", "questions_to_authors": "1. It is not clear why EM-L method does not suffer from the data scarcity problem. \n2. Is it possible to combine EM-L and R-L? If yes, why the paper does not combine them? If not, is there any reason for that? \n3. Will the EM-L suffer from worse training efficiency? Since it alternatively performs the E-step and M-step. ", "paper_topic_and_main_contributions": "The paper studies the retrieval-augmented few-shot classification. Motivated by the fact that the existing retrieval methods all suffer from data scarcity and vanishing gradient problem, the paper proposed an EM algorithm which alternatively retrieves the best m training data and update the parameters by maximizing the log-likelihood. Besides, it also proposed a R-L algorithm which minimizes the discrepancy between the retriever and the classifier. The comprehensive empirical studies verifies the effectiveness of the proposed two techniques. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers saw both strengths and weaknesses in the submitted version of the paper. The authors have provided a rebuttal that appeared to alleviate some concerns. After some discussion, the overall verdict was leaning toward an acceptance of this submission. However, there are still some issues to be addressed as raised by Reviewer uG9t. So, I am leaning toward a Borderline Sound on this submission.
Decision: Accept-Findings