DRAFT: Dense Retrieval Augmented Few-shot Topic classifier Framework
Authors: Keonwoo Kim
Younggun Lee
Venue: EMNLP 2023 Findings (2023)
Keywords: few-shot topic classification
real-world application
TL;DR: We propose a novel approach, DRAFT, for addressing few-shot classification across diverse topics.

Abstract: With the growing volume of diverse information, the demand for classifying arbitrary topics has become increasingly critical. To address this challenge, we introduce DRAFT, a simple framework designed to train a classifier for few-shot topic classification. DRAFT uses a few examples of a specific topic as queries to construct Customized dataset with a dense retriever model. Multi-query retrieval (MQR) algorithm, which effectively handles multiple queries related to a specific topic, is applied to construct the Customized dataset. Subsequently, we fine-tune a classifier using the Customized dataset to identify the topic. To demonstrate the efficacy of our proposed approach, we conduct evaluations on both widely used classification benchmark datasets and manually constructed datasets with 291 diverse topics, which simulate diverse contents encountered in real-world applications. DRAFT shows competitive or superior performance compared to baselines that use in-context learning, such as GPT-3 175B and InstructGPT 175B, on few-shot topic classification tasks despite having 177 times fewer parameters, demonstrating its effectiveness.

DRAFT: Dense Retrieval Augmented Few-shot Topic Classifier Framework
Keonwoo Kim*
Younggun Lee Neosapience yg@neosapience.com
Seoul National University
VRCREW Inc.
keonwookim@bdai.. snu.
snu. ac. kr
Abstract
demand to detect specific topics within the text gen- erated by LLMs. The use of L.Mss often contains ethical concerns with the issues of hallucination, as they possess the capability to generate morally inappropriate or unintended content (Ganguli et al., 2022; Perez et al., 2022). However, as the amount of content shared on social media and generated by LLMs increase exponentially, verifying each piece of content becomes challenging for individuals.
With the growing volume of diverse informa- tion, the demand for classifying arbitrary topics has become increasingly critical. address this challenge, we introduce DRAFT, simple framework designed to train a classifier for few- shot topic classification. DRAFT uses few examples of specific topic as queries to con- struct Customized dataset with a dense retriever model. Multi-query retrieval (MQR) algorithm, which effectively handles multiple queries re- lated to specific topic, applied to construct the Customized dataset. Subsequently, we fine- tune a classifier using the Customized dataset identify the topic. To demonstrate the ef- ficacy of our proposed approach, we conduct evaluations on both widely used classification benchmark datasets and manually constructed datasets with 291 diverse topics, which simu- late diverse contents encountered in real-world applications. DRAFT shows competitive perior performance compared to baselines that use in-context learning, such as GPT-3 175B and InstructGPT 175B, on few-shot topic clas- sification tasks despite having 177 times fewer parameters, demonstrating its effectiveness.
In natural language processing (NLP), research related to the challenges mentioned earlier can be considered topic classification task since the de- mand for automatically classifying texts on spe- cific topic exists. Recent pre-trained language mod- els (Devlin et al., 2018; Liu et al., 2019; Clark et al., 2020; He et al., 2020) have gained consid- erable recognition for their ability to achieve high performance in topic classification tasks.
To train a topic classification model, com- mon practice to rely on supervised learning using a training dataset with labeled data. Most existing methods on topic classification have primarily relied on benchmark datasets (Zhang al., 2015; Auer et al., 2007). To the best of our knowledge, they have predominantly focused on improving model performance on benchmark datasets that involve a limited number of topics rather than ad- dressing long-tailed arbitrary topics in weal-world scenarios. Regarding real-world applications, the availability of labeled datasets that cover diverse topics is often limited due to the substantial cost of building such datasets. This constraint poses challenge for directly deploying topic classification models in practical scenarios, thereby calling for a solution that enables their flexible application.
1 Introduction
With the prevalence of the Internet and social me- dia, there is a significant demand for classifying or detecting texts related to specific topics within the vast amount of information pouring in from the Internet. For instance, on social media plat- forms where overwhelming volume of content is generated, there may exist need monitor and filter content associated with particular issues (e.g.. drugs). Additionally, with the remarkable progress large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022; Chowdhery et al., 2022; Rac et al., 2021; Scao et al., 2022; Thoppilan et al., 2022) in recent times, there is also an increasing
Few-shot classification, which performs classi- fication using few text examples, can be applied even without training dataset for predefined topic, enabling its extension to tasks classifying a diverse range of arbitrary topics. To the best of our knowledge, existing few-shot classification
"Work done while at Neosapience.

---

methods operate exclusively on tasks with two or more defined classes and cannot conduct in one- class classification tasks (Zhang et al., 2023; Chen et al., 2022a; Mukherjee and Awadallah, 2020; Sun
attempt to classify a topic in texts through a dense retriever model.
3. We introduce the MQR algorithm, which is the first accommodate multiple queries si- multaneously as inputs for the retriever.
et al., 2019). However, by leveraging LLMs with in-context learning (Brown et al., 2020; Holtzman et al., 2021; Min et al., 2021), we can conduct one-class few-shot topic classification tasks across various topies, showcasing superior performance. This approach garnered significant attention due to its applicability in scenarios with limited labeled data (Zhao et al., 2021; Holtzman et al., 2021; Min et al., 2021). However, successful implementation of in-context learning relies on LLMs with billions of parameters (Kaplan et al., 2020a). Such models suffer from high computational costs, extensive re- source requirements, and slow inference speed to apply in real-world applications.
4. The results of extensive experiments show that DRAFT achieves competitive performance compared large language models.
Related Works
Retrieval-augmented methods Information re- trieval aims to find semantically relevant docu- ments based on a query. Traditional methods employ lexical approaches to retrieve support documents using sparse vectors (Hiemstra, 2000; Robertson et al., 2009). With the recent develop- ment of deep learning (Vaswani et al., 2017), neural network- based methods have demonstrated high performance. The bi-encoder structure (Karpukhin et al., 2020; Izacard al., 2022a) offers the advan- tage of pre-encoding document candidates in an offline setting, which allows for faster computation. Nonetheless, the lack of token-level interaction be- tween query and document tokens in the bi-encoder models can lead to lower performance compared to the cross-encoder models (Devlin et al., 201 2018). Nevertheless, we use bi-encoder in DRAFT for efficient retrieval, which facilitates the immediate creation of classifier for any given topic.
To address the few-shot topic classification more efficiently in real-world applications, we pro- pose a simple framework called Dense Retrieval Augmented Few-shot Topic classifier framework (DRAFT), which can classify arbitrary topics given limited labeled data. DRAFT uses a dense retriever model (Karpukhin et al., 2020) to construct Cus- tomized dataset, using examples of target topic provided queries. Subsequently, a pre-trained language model finetuned with the Customized dataset for the topic classification task. We eval- uate the performance of DRAFT by conducting experiments on general benchmarks and manually constructed datasets with 291 topies. The former datasets are widely regarded as benchmark datasets in recent classification research. In contrast, the latter datasets manually constructed to replicate real-world scenarios, allowing for a thorough as- sessment of the diverse topic classification capabil- ity. They are used for one-class classification tasks where only a single topic is defined. Our experi- ment results demonstrate that DRAFT consistently achieves competitive or superior performance com- pared to LLMs, such as GPT-3 175B and Instruct- GPT 175B, which have 177 times more parameters on topic classification tasks. These findings provide strong empirical support for the efficacy of DRAFT in tackling few-shot topic classification tasks. The contributions are summarized as follows:
There exist various retrieval-augmenned method- ologies (Cai et al., 2022; Izacard et al., 2022b) in recent NLP research. They encompass solu- tions for tasks such fact retrieval (Thorne et al., 2018), open-domain question answering (Chen et al., 2017; Izacard and Grave, 2020; Lewis et al., 2020; Guu et al., 2020), and others. They also include techniques applied during inference to re- duce perplexity in language modeling (Khandelwal et al., 2019) and strategies operate akin to memory for specific knowledge or dialogues (Chen et al., 2022b; Fan et al., 2021). All existing retrieval- augmented methods universally handle only sin- gle query as the input to the retriever and subse- quently execute downstream tasks. However, un- like existing approaches. DRAFT processes multi- ple queries simultaneously as input for retrievers.
1. We propose DRAFT as a simple but effective framework that classifies texts related to arbi- trary topics using few labeled data.
In-context learning with LLMs Recent LLMs (Thoppilan et al., 2022; Zhang et al., 2022; Scao et al., 2022; Rae et al., 2021; Shoeybi et al.,
2. To the best of our knowledge, we are the first

---

(a)
(b)
Building Customized dataset
Data Collection
Semantic Search
Posttive Samples
Negative Samples
number
Querry esample
Query Encoder
Query esample
Query example
Queries (positive samples)
Training a classifier
Passages retrieved as positive samples through Semantic Search
Customized dataset
Language
MLP Layer
clessification
Remaining passagea Data Collection (negutive samples)
Figure I: (a) Overall pipeline of DRAFT. DRAFT receives n queries as input, and a trained classifier is only used in the test phase. (b) Illustration of MQR in two-dimensional space. A cirele represents the normalized embedding space of texts in Data Collection. For each query, passages only within angle size 0, calculated as a threshold from n query vectors, are retrieved as positive samples, while others are classified as negative samples.
2019; Chowdhery et al., 2022) represent critical development in NLP and have been considered an attempt to develop intelligent language systems with fluency approaching that of humans. Accord- ing to (Kaplan et al., 2020b), as the scale of the language model increases, its performance is also improved on many tasks that typically require models with hundreds of billions of parameters. LLMs primarily perform tasks through in-context learning (ICL) (Brown al., 2020; Holtzman et al., 2021; Min et al., 2021), which feeds concatenated prompt and input-target examples (referred as "k-shot') into the model without weight updates. It exhibits superior performance over zero-shot inference across an extensive array of tasks (Zhao et al., 2021; Liu et al., 2021a). For classification tasks, models predict label from predefined set of labels with the highest probability. ICL offers the key benefit of allowing single model to instantly handle multiple tasks with only limited number of labeled examples, known as few-shot learning. In the context of our research on few-shot topic classification across diverse topics, creating classifier for each topic without having training dataset can be regarded
3 Method
In this section, we begin by describing the appli- cation of DRAFT to a simple binary classification and elaborate on the extension of DRAFT. DRAFT comprises two stages for few-shot topic classifi- cation: (1) constructing Customized dataset from multiple queries using dense retriever model and (2) training elassifier. Figure (a) illustrates the overall process of DRAFT.
3.1 Building Customized dataset
In the first stage, we use a pre-trained dense re- triever, bi-encoder consisting of query encoder and passage encoder, to construct Customized dataset. Customized dataset is created by employ- ing few texts related to target topic as queries. To efficiently retrieve relevant passages from Data Collection, which serves the external knowledge base (e.g.. Wikipedia), we employ Maximum In- ner Product Search (MIPS) algorithm (Shrivastava and Li, 2014) that finds the vector with the highest inner product value with a given query vector. The retriever is defined by employing query encoder Equery and a passage encoder Epassage:
distinct task. Thus, we ICL with LLMs, which perform diverse tasks with only a few labeled examples, baseline models in the experiments.
p(2 x)) x exp (Equer((a) Expassage (z))-
Equery embeds query I, and Epaseage embeds pas- sages & E Z,, where Z indicates Data Collection.

---

We select the top-k passages with the highest prior probability p(= 4), which is proportional to the inner product of the query and passage vectors.
token representing sentence-level embedding ob- tained from the hidden vector after passing through the encoder layers through an MLP layer. The classifier is trained using binary cross-entropy loss.
Algorithm 1: Multi- Multi-Query Retrieval
Result: An array C' of target samples
2(x++ma
Input: An array S with queries
L = 2(n + m)
Σ 1=1
وu=ةlnqu mi), jE(0,1]
Σ
(0 <- 1
..C2 > Ejff sim (((),
for g = 1,2., n2 do
with an indicator function 1, a text Hi a label for the corresponding text yi E {0, 1}. and a probabil- ity on model's prediction 9.
retrieve k passages using MIPS for passage .........k do
a- < cos((S(() (=sssa())
denotes cosine similarity function if cos then
3.3 Expanding the Capabilities of DRAFT
Negative query We can improve DRAFT's per- formance by introducing an additional stage after training a classifier. In this stage, we manually incorporate negative queries that belong to se- mantically similar category but are different from the target topic. Constructing negative samples follows the process outlined in Section 3.1 for con- structing positive samples, which uses MQR. After training the classifier with the updated Customized dataset, it becomes more robust in handling hard negatives that are difficult to classify. This expan- sion of DRAFT enables it to handle complex cases and enhance its overall performance.
C < C + passagee]
end
end
end
We define Customized datasel as the construc- tion of a collection of positive and negative samples, including multiple queries, specifically designed to train a topic classifier. To build positive samples within Customized dataset, we propose the Multi- Query Retrieval (MQR) algorithm, depicted in Fig- ure 1 (b). As outlined in Algorithm 1, MQR begins by gathering sentences or keywords related to the specific topic to form queries. dense retriever model for each query is employed retrieve top-k passages, where & denotes the subspace size. From the retrieved n x & passages, only passages vectors that exceed a threshold cos(f) determined by the average pairwise cosine similarity score among the n query vectors are retained. Unlike general dense retriever models that take single query as input, it accept multiple queries input. By combining the queries and the m passages retained from Data Collection, we collect total of m posi- tive samples. Subsequently, we create an equivalent number of negative samples by randomly selecting passages from Data Collection. thereby forming Customized dataset, which comprises 2(n + m) samples.
Multi-class classification expansion of the class set, possible develop a multi-class classifier. For each class, MQR em- ployed to construct class-specific positive samples. The training dataset for a multi-class classifier is created by merging the aggregated class-specific constructed positive samples for each class, elimi- nating the need for separate process of construct- ing negative samples within each class due the presence of independent other defined classes. Sub- sequently, the classifier trained with the merged dataset using cross cross-entropy loss.
Moreover, through the
Experiments
To evaluate the performance of DRAFT, we con- duct three experiments with limited labeled data in classification tasks.
3.2 Training topic classifier in DRAFT
We assess the performance of DRAFT in few-shot topic classification by manually con- structing datasets comprising 291 unique top- ics that simulate real-world scenarios.
In the following stage, we proceed with fine-tuning a pre-trained language model the classification task, employing Customized dataset constructed in the preceding stage. The fine-tuning process is similar to (Devlin et al., 2018) for downstream tasks. It passes [CLS] token, a special classification
We explore the expanding capabilities of DRAFT using negative queries on two addi- tional manually constructed datasets.

---

3. We evaluate the multi-class classification ca- pabilities of DRAFT using three commonly used benchmark datasets.
50 samples that belong the positive class. For each dataset, we construct negative samples, com- posed of easy negatives and hard negatives, to the same size as the positive samples. Easy negatives are randomly sampled from subtopics that belong to a different subcategory than the positive ones; hard negatives are randomly mined from subtopics within the same subcategory the positive class.
4.1 DRAFT Setup
We use a Wikipedia dump from 2018 for Data Col- lection. All encoders, a bi-encoder, and classifier used in DRAFT are initialized with SimCSE (Gao et al., 2021) trained through supervised learning. While each encoder shares the same weights and contains approximately 330 million parameters, these weights are not shared during the training of DRAFT. We first train the bi-encoder in DRAFT on Natural Question (Kwiatkowski et al., 2019) using contrastive InfoNCE loss (Oord et al., 2018). Once trained, the weights of the bi-encoder remain fixed, only requiring fine-tuning of a classifier when the type of topic is modified. Since the number of sam- ples extracted from MQR varies for each dataset, the size of Customized dataset used by a classifier in DRAFT for training also fluctuates accordingly.
We set LLMs as baseline models due to their capability to serve as classifiers for vari- ous topics when provided with a few labeled sam- ples for ICL. The format of ICL is described in Appendix B.3. We use two types of LLMs, GPT- (Brown al., 2020) and InstructGPT (Ouyang et al., 2022), both with two versions of model sizes: 2.7B and 175B. InstructGPT is an improved ver- sion of GPT-3, fine-tuned through reinforcement learning with human feedback to enhance its un- derstanding of prompts.
Baselines
Detail settings Given the absence of training datasets, we adopt Quick Facts, a collection of five texts associated with each subtopic in FactsNet, as queries for DRAFT and examples of ICL,, serv- ing as sample instances of the positive class. We employ various ICL examples from Quick Facts for LLMs, specifically selecting one, three, or five examples. For DRAFT, we configure a set of five examples as queries, with a subspace size of 10,000 allocated for each query. The examples of Quick Facts can be seen in Appendix B.4.
When fine-tuning classifier, we use the AdamW (Loshchilov and Hutter, 2017) optimizer with learning rate of le-5 using linear learning rate scheduler. We divide Customized dataset into an 80% to 20% split to form the training and valida- tion datasets, respectively. The classifier is trained by employing early stopping against the validation loss with the patience of two epochs. We perform all experiments using the Py PoTcch framework and three 32GB Tesla V100 GPUs
To assess the capability of classifying diverse topics, derive rankings based the Fl score for each major category across all methodologies and subsequently compute an average rank based on the five major categories. The robustness of each methodology in topic classification can be assessed through the average rank.
4.2 Few-shot topic classification task
Dataset To evaluate the effectiveness of DRAFT in few-shot topic classification, specifically in di- verse topic scenarios, we construct 291 test datasets. Given the absence of existing benchmark datasets specifically designed for its task, we perform web crawling on the FactsNet website', which covers a broad spectrum of subjects comprising 291 distinct topics. The content on the website has a three-level hierarchical structure consisting of major cate- gory, a subcategory, and a subtopic. The major categories are comprised of five types: Lifestyle, History, Nature, World, and Science. Additional information about datasets is in Appendix A.
Results and analysis Table I presents the eval- uation results of the few-shot classification tasks on diverse topics. We evaluate the F1 score for all 291 subtopics and aggregate the results based on the five major categories. Among the baselines with billions of parameters, except for InstructGPT 175B 1-shot, DRAFT, only with millions of pa- rameters, demonstrates superior performance com- pared to the others in all categories. When consid- ering the average rankings across five major cate- gories, DRAFT achieves the highest rank of 1.4, followed by InstructGPT 175B 1-shot with an aver- age rank of 1.6, implying DRAFT's optimality.
In our dataset construction, we leverage contents in subtopics. We consider classifying each subtopic as distinct topic classification task, resulting in 291 test datasets. We select subtopics with at least
'The website is as follows: https://facte.

---

GPT-3
GPT33 175B
InstructGPT 2.7B
InstructGPT 175B
Major eategory
DRAFT
I-shor
3-shrc
S-chot
I-shor
3-shot
5-shot
1-shot
3-shot
S-shot
1-shot
3-shor
S-ahot
Lifestyle
80.5(1)
55.164)
52.7(5
48.5(9)
23.0(13)
50.3(8)
56.4(3)
45.8(11)
52.206)
47.5(10)
79.1(2)
52.666)
35.7(12)
History
84.7(1)
53,405)
53.4K)
48.818)
23.7(13)
51.6(7)
58.2(3)
40.5(11)
48.3(9)
47.8(10)
82.8(2)
55.04)
37.1(12)
Nature
79.6(2)
54,8(7)
59.70)
56.714)
16.7(13) 45.7(9)
55.016)
40,8(11)
46.9(8)
39.3(12)
82.3()
55.8(5)
41,0(10)
World
82.1(2)
54.906)
56.0.5)
52.6(8)
17.3(13)
54,7(7)
61.2()
41,6(11)
46.9(9)
45,4(10)
86.9(1)
59.1(4)
39.0(12)
Science
80.2(1)
54.364)
54.005)
47.2(9)
20:4(13)
49.6(7)
54.6(3)
44.7(10)
49.308)
43.3(11)
79.9(2)
50.506
35,2(12)
Avg. Rank
5.2
4.6
13
3.6
10.8
8
1.6
11.6
Table I: Fl scores on few-shot topic classification tasks. We use k-shot setting, where & denotes the number of examples with labels for ICL on LLMs. We present the average score for each subtopic within its corresponding major category. The values within parentheses indicate the ranking of the 13 methods based on the highest scores for each row. 'Ag. Rank' represents the average ranking for each method.
Method
in Table excluding InstructGPT 175B 1-shot. Also, the most straightforward approach, Random method, surpasses some LLMs. Thus, we empha-
Major category
DRAFT
Random
Noun.
Dense.
Lifestyle
80.5(1)
50.2(4)
69.7(2) 58.0(3)
History
84.7(1)
49.4(3)
69.3(2)
42.5(4)
size that LLMs are ineffective for few-shot topic classification tasks across diverse topics.
Nature
79.6(1)
50.8(4)
62.9(3)
71.6(2)
World
82.1(1)
50.2(3)
56.7(2)
45.3(4)
Science
80.2(1)
48.5(4)
54.5(3)
58.7(2)
4.3
Including negative queries on DRAFT
Avg. Rank
1.0
3.6
2.4
3
Dataset We manually construct two additional test datasets with Religion and South Korea topics using a similar format to Section 4.2. In contrast to the automatically constructed datasets in Sec- tion 4.2 by crawling the website, five annotators manually construct datasets that determine whether each sample belongs to the positive, easy negative, or hard negative class. The easy negative and hard negative in the test dataset share the same nega- tive class label. However, the diffcrentiation is made to demonstrate the construction process of the negative dataset. We define easy negatives as samples unrelated to the positive class in terms of their semantic content. In contrast, hard negatives samples that fall into semantically similar cate- gories to positive ones but have distinct content. In Religion dataset, we define 'Jewish' and 'Islam' positive classes. contrast, the hard negatives con- sist of content that falls within the religion category but pertains to different specific religions, such as 'Buddha' and 'Hinduism'. The easy negatives are composed of content unrelated to religion alto- gether. The examples of classes within the dataset Religion dataset, additional details about South Korea dataset, and instructions for annotators can be found in Appendix A.
Table 2: Results show F1 scores on few-shot topic classi- fication tasks. DRAFT, Noun-bar boun-based (Noun.), and Dense- based (Dense.) employ the same 5-shot setting.
Ablation study We conduct an ablation study to highlight the difficulty of few-shot topic classifi- cation across diverse topics. We establish three simple baselines in this experiment using the same datasets in Table 1. (I) Random method randomly assigns classes
sample with uniform distri- bution of positive and negative classes, where the positive class corresponds to a target topic while the negative class corresponds to all other topics. (2) Nosnd method classifies sample as a target topic if any nouns from the queries exist in the sample. We use NLTK package² to extract all nouns. (3) Dense-based method classifies a sample using embeddings obtained from [CLS] token. If the cosine similarity score between a query vector and a sample vector surpasses threshold deter- mined by MQR for any of the queries, the sample is classified as a target topic. The queries used in (2) and (3) are identical to those used in DRAFT.
We observe a clear trend of DRAFT outperform- ing three baselines across all main categories in Ta- ble 2. implies the difficulty of the task and high- lights the limitations of simple approaches, such as Noun-based or Dense-based methods. Neverthe- less, the boun- method outperforms LLMs
Baselines
We investigate the impact of different methods on building negative datasets in DRAFT using three distinct methods: M1, M2, and M3. MI is defined by its use of random sampling, while M2 exclusively employs negative queries as dictated by MOR. M3 is a combination of two methods, with
2NLTK package is from https: https://www.nnt nl www.

---

50% of the dataset constructed from M1 and the remaining 50% from M2.
topic classification task. TREC (Voorhees and Tice, 2000) a dataset for 6-way question classifica- tion task, which differs from topic-based content. Additional details can be found in Appendix A.
Detail settings We use three positive and two negative queries in Religion dataset, whereas four positive and three negative queries in South Korea dataset. The subspace size associated with each query is set at 10,000. Detailed instances of two types of queries can be seen in Appendix B.4.
Baselines Among various LLMs, we consider GPT-2 XL with 1 1.3B parameters (Radford et al., 2019) and GPT-3 (Brown et al., 2020) with differ- ent model sizes, such as 175B and 2.7B parameters.
Detail settings We conduct ICL with LLMs evaluate their performance given labeled ran- domly sampled examples from the training dataset for each class, with k being 1, 4, and 8. Similarly, we randomly select eight samples corresponding to each class from the training dataset to serve queries for DRAFT. Empirically, experiments ducted in Section 4.2 and Section 4.3, which solely rely on one-class samples few-shot samples, akin to one-class classification, are more challenging compared to multi-class classification, which pro- vides few-shot samples for all classes. Considering the difficulty of the task, we set the subspace size to 50 in this experiment.
Dataset
Method
Fl (5)
Acc (%)
(S6)
EN (%)
HN (5)
M1
79.0
79.5
96.8
100.0
40.0
Religion
M2
69.0
76.9
64.5
86.4
84.0
96.8
79.5
73.2
76.5
S.K.
M2
72.6
69.6
80.4
18.2
85.3
M3
85.4
86.6
78.6
100.0
91.2
Table 3: Evaluation results on Religion and South Korea (S.K) datasets. Positive (P), easy negative (EN), and hard negative (HN) are class labels classified during the dataset construction process.
Results and analysis We evaluate the Fl score and accuracy for each methodology on the Reli- gion and South Korea datasets. To further examine the impact of negative queries, we measure the ac- curacy based on the three distinct classes defined during the dataset construction process, which in- clude the positive, easy negative, and hard negative classes. In Table 3, M1 demonstrates proficient classification of casy negatives, albeit struggles in effectively identifying hard negatives. Conversely, M2 exhibits reasonable performance in classifying hard negatives but the expense of lower accu- racy for casy negatives. Remarkably, M3, which builds a negative dataset constructed through a com- bination of random sampling and employment of negative queries, consistently shows superior per- formance. Our experimental findings emphasize the significance of M3 in effectively mitigating bias associated with negative queries while ensuring ac- curate classification of semantically independent content from positive samples. It verifies the poten- tial of employing negative queries to enhance the performance of DRAFT.
k-shot AGNews
DBpedia
TREC
45.4 8.4
33.6 18.9
21.5 6.2
GPT-2 XL (L.BB)
44.6 12
53.0 14.8
23.1 5.9
57.1
66.0 3.6
32.7 7.
33.0 5.1
25.9 4.4
24.3 6.4
GPT-3 (2.7B)
43.3 8.5
61. 12.8
25.8 11.5
50.8 7.5
72.6 4.6
29.3 8.0
62.1
79.3
57.7
GPT:3 (175B)
61.0 10.9
84.6 a.8
60.2 7.6
79.1
82.3 r.8
45.6 a.0
DRAFT
82.8 1.3
94.7 1.
36. 6.2
Table 4: Results show accuracy on benchmark datasets. 'ksshot' represents the number of examples used in ICL, All values are presented in the meon,ed format.
Results and analysis Experiments are run five times, each iteration using different random seed for sampling queries and examples. Table 4 presents the accuracy results in classification benchmark datasets. DRAFT outperforms all base- lines on the topic classification tasks in both AG- News and DBpedia. Comparing DRAFT with the best performance cases of GPT-3 175B, the results show a difference of 3.7%p accuracy in AGNews and 12.4%p in DBpedia. While DRAFT achieves a lower accuracy than GPT-3 175B in TREC, it still outperforms GPT-2 XL and GPT-3 2.7B.
4.4 Multi-class classification task
Dataset We employ three general benchmark datasets evaluate the performance of DRAFT with a limited number of samples. AGNews (Zhang et al., 2015) is a collection of news articles used for a w-w topic classification task. DBpedia (Auer et al., 2007) is an ontology dataset for a 14-way
Rosults on baselines are reported from (Zhao et al., 2021)

---

Figure 2. Increasing the number of queries ex- hibits a positive correlation with accuracy when the subspace size fixed. The highest accuracy of 88% is achieved with a subspace size of 10 and 50 queries. Although the increase in the number of queries is limited to 50 due to computational resource constraints, the consistent trend implies that more queries can improve performance.
We find that DRAFT has the potential for im- provement with an increased number of queries. Unlike DRAFT, LLMs can suffer from degraded performance due to the majority label bias (Zhao et al., 2021) when the number of examples from the same class increases in few-shot samples. In Section 4.2, InstructGPT 175B demonstrates no- ticeable decline in performance as the value of k increases in k-shot settings. Considering majority label bias and experimental results, DRAFT shows a more robust performance than LLMs.
the number of queries
Figure 2: Heatmap shows the impact of the number of queries and subspace size DRAFT using AGNews.
We find that the performance of DRAFT varies across different benchmark datasets, suggesting that the attribute of Customized dataset plays crucial role. As DRAFT uses Data Collection to construct Customized dataset, the choice of Data Collection strongly influences performance. In our experiments, by leveraging Wikipedia as Data Collection, which primarily consists of topic-based content, DRAFT consistently outperforms all base- lines with lower variance AGNews and DBpedia, which also consist of topic-based content. How- ever, TREC, which involves attributes different from based content, DRAFT exhibits lower performance compared GPT-3 175B. These re- sults indicate that DRAFT works outstanding for classifying topics but may not exhibit robust per- formance in other classification tasks.
Data Collection We underscore the importance of Data Collection Section 4.4. DRAFT shows the highest accuracy on DBpedia due to the simi- lar distribution between the test dataset and Data Collection. However, classifying topics that re- flect recent content becomes challenging issuc for DRAFT, which uses a Wikipedia dump from 2018 for Data Collection, since there is a discrep- ancy between the distribution of the training dataset and that of topics that reflect the recent content. ARAFT can simply solve the problem by building Data Collection with recent knowledge. From the perspective of injecting external knowledge into the model or framework, DRAFT offers a more straightforward approach than LLMs (Meng al., 2022; Mitchell et al., 2021).
5 Discussion
Potential points in DRAFT In DRAFT, the num- ber of queries and the subspace size considered the most crucial factors among various hyperparam- eters. Queries provide information related to the target topic, while subspace size determines how many passages related to each query are retrieved from Data Collection. They influence the perfor- mance of DRAFT, as they significantly impact the construction of Customized dataset, which directly impacts the process of training classifier.
Real-world applications DRAFT can effectively classify content related specific topics from the Internet or SNS. It can swiftly generate a classi- fier for the given topic upon user request with few queries. However, a challenge emerges as the num- ber of tasks for classifying topics escalates sig- nificantly with the increasing number of users on the Internet or the growing quantity of topics re- quested. Storing the weights of all individually trained classifiers for each task is inefficient. Thus. effectively implement DRAFT in real-world sce- narios, one must consider an approach centered on parameter-efficient learning (Houlsby et al., 2019;
We investigate the relationship between the num- ber of queries and the subspace size by varying both variables using AGNews. The experiments are repeated five times with different configurations, and the average accuracy results are presented in
Liu et al., 202 lb; Hu et al., 2021) to allow the effi- cient management of weights for each task.

---

6 Conclusion
8 Ethics Statement
We explicitly mention the copyright of the FactsNet dataset, crawled from the website. We firmly state that we use the Factset dataset simply for evalu- ation due to the absence of a benchmark dataset containing diverse topics. Also, we employ five annotators while building two additional manually constructed datasets in Section 4.3, Our annotators are affiliated with our company and receive com- pensation through wages for their labeling work, Detailed instruction for the annotation task is in Ap- pendix A. Also, we strongly discourage any misuse of DRAFT for illegal activities.
In this study, we introduce DRAFT, a simple but ef- fective approach that first applies dense retriever model for few-shot classification across diverse topics. Despite possessing 177 times fewer param- eters than LLMs, DRAFT demonstrates superior performance in few-shot topic classification. These results imply the effectiveness of DRAFT in clas- sifying a diverse array of real-world topics. We anticipate that DRAFT holds the potential to be implemented in practical contexts and actively con- tribute to addressing diverse societal challenges, including those encompassing specific topics.
7 Limitations
References
In Section 4.3, the negative query is defined as a text with content similar to the positive class but with different topic. Although DRAFT demon- strates the ability improve performance upon re- ceiving negative query, the automatic generation or construction of negative queries is necessary for worldworld applications since people cannot manu- ally provide a negative query for every topic.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann. Richard Cygamiak, and Zachary Ives. 2007, Dbpedia: A nucleus for web of open data. In The semantic web, pages 722-735. Springer.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Also, some ambiguous topics need to be cali- brated in automatically constructed 291 datasets us- ing the FactsNet website. For example, the queries in 'Sports' topic primarily revolve around baseball and golf. After training DRAFT, it can effectively classify content related to baseball and golf. How- ever, after manually examining the test dataset for 'Sports' topie, it becomes apparent that examples of other sports, such as basketball and tennis, belonged to the positive class. Although basket- ball and tennis undoubtedly fall under the sports category, the queries are composed solely of con- tent related to specific examples of sports, such as baseball and golf. The ambiguity of the test dataset poses a challenge for DRAFT, proving to be a diffi- cult task even for humans to classify these contents with the same few-shot samples accurately. There- fore, conducting human manual reviews for each of the 291 subtopics within FactsNet filter out ambiguous topics could enhance the reliability of experimental results.
Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. 2022. Recent advances in retrieval- augmmeted text generation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 3417-3419.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. arXiv preprint IrXiv:1704.0000 700.00051.
Junfan Chen, Richong Zhang, Yongyi Mao, and Jie Xu. 2022a. Contrastnet: A contrastive learning frame- work for few-shot text classification. In Proceedings of the AAAI Conference on Artificial Intelligence. pages 10492-10500.
Xiang Chen, Lei Li, Ningyu Zhang, Xiaozhuan Liang, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022b. Decoupling knowledge from memorization: Retrieval-augmentee prompt learning. Advances in Neural Information Processing Systems, 0223908-23922
Ke-Li Chiu, Annie Collins, and Rohan Alexander. 2021, Detecting hate speech with gpt-3, arXiv preprint arXiv:2103.124000
Morcover, limitation of DRAFT lies in the need for mathematically rigorous proof for the va- lidity of its MQR. As a result, in future research, we plan to undertake endeavors to address the quality issue of FactsNet and establish rigorous mathemat- ical proof for evaluating the effectiveness of MQR.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.

---

Kevin Clark, Minh-Thany Luong. Quoc V Le, and Christopher Manning. 2020. Electra: treitraining text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555
Gautier Izacard and Edouard Grave. 2020. Leverag- ing passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint arXiv: 1010:088000400
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022b. Few-shot learning with re- trieval augmented language models, arXiv preprint 2:Xi::2008.03299
Angela Fan. Claire Gardent, Chloé Braud, and Antoine Bordes. 2021. Augmenting transformers with knn- based composite memory for dialog. Trunsactions of the Association for Computational Linguistics, 9:82- 99.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020a. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to re- duce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0777 077058.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020b. Scaling laws for neural language models. arXiv preprint arXiv:2001.08:
Tianyu Gao. Xingeheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint wxXiv:210400883
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Tdduno, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint N0Xx::000404900
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasu- pat. and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International Con- ference Machine Learning, pages 3929-3938. PMLR.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.0017.
Pengcheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decondng--nhanned bert with disentangled attention. arXiv preprint arXiv:2006.0366 03654.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Djoerd Hiemstra. 2000. probabilistic justification for using tfx idf term weighting in information retrieval. International Journal on Digital Libraries, 3(2):131- 139.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competi- tion: Why the highest probability answer isn't 't always right. arXiv preprint 2xxiv:204.008/5..
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval laurieeela generation for knowledge-intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019, Parameter-efficient transfer learning for nlp. In In- lernational Conference on Machine Learning, pages 2790 2799. PMLR.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021a. What makes good in-context examples for gpt-3? arXiv preprini 20X.v:2201.066:
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Zhu, Yuanzhi Li. Shean Wang. Lu Wang. and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.00 09685.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. 202 P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2 10.0762..
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022a. Unsupervised dense in- formation retrieval with contrastive learning, arXiy preprint arXiv:2
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1907. 16622.

---

Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05101
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. 2019. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint 10XXv:190900800
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual asso- ciations in GPT. Advances in Neural Information Processing Systems, 35.
Anshumali Shrivastava and Ping Li. 2014. Asymmetric 1sh (alsh) for sublinear time maximum inner prod- uct search (mips). Advances in neural information processing systems, 27.
Sewon Min, Mike Lewis. Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108.04105
Shengli Sun, Qingfeng Sun, Kevin Zhou, and Tengchao Lv. 2019. Hierarchical attention prototypical net- works for few-shot text classification. In Proceed- ings of the 2019 conference on empirical methods in natural language processing and the 9th interna- rional joint conference on natural language process- ing (EMNLP-IJCNLP), pages 476-485.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv:2110.1144
Subhabrata Mukherjee and Ahmed Awadallah. 2020. Uncertainty-ave self-training for few-shot text clas- sification. Advances in Neural Information Process- ing Systems, 33:21199-2122 21212.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam
Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.0222
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807,03748.
James Thorne, Andreas Vlachos. Christos Christodlollopoulo,, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint a:Xiv:2203.02155.
1803.05353
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Ethan Perez, Saffron Huang, Francis Song. Trevor Cai, Roman Ring. John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red team- ing language models with language models. arXiv preprint 2202
Ellen M Voorhees and Dawn Tice. 2000. Building a question answering test collection. Proceedings of the 23rd annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 200-207.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):9.
Ruohong Zhang, Yau-shian Wang, Yiming Yang, Dong- han Yu, Tom Vu, and Likun Lei. 2023. Long-tailed extreme multi-label text classification by the retrieval of generated pseudo label descriptions. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1062-1076.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1 11446.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen. Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li. Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint or Xxxx:2205.00006
Stephen Robertson, Hugo Zaragoza, al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 3(4)333--89.
Xiang Zhang, Junbo Zhao, and Yann LeCun 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Teven Le Scao, Angela Fan. Christopher Akiki, EI- lie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter acen-access multilingual language model. arXiv preprini 201.0:221..05100.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR

---

A Details of datasets
lowing sentence is not related to religion, choose If the following sentence is related to Buddha or Hinduism, choose 2. Answer: 0, 1, 2'. The final label for all classes is determined through a majority vote. Religion dataset consists of 78 sam- ples from https://www. history. com/toopics/reLigion The composition of the label is: (O:Positive, 1:Neg- ative), where the negative class contains both easy negative and hard negative. Examples of the dataset can be seen in Table
Data information
Major category
Topic-num
Sample-num
Avg. token longth
Lifestyle
46
220
39.4
History
56
166
45.3
Nature
54
169
43.5
World
113
203
46.0
Science
22
214
44.0
Total
291
193
44.1
Table 5: The detailed information includes the 'Topic- num', 'Samppe-num', and 'Avg. token length', which represent the number of subtopics, the average number of samples, and the average token count, respectively.
South Korea For South Korea, the test dataset consists of 112 samples from http://www.... facts. net, https://pitchhork.ccm con, ht https://ww.brrtammtt
www.:wriiie www.briiinnn.....mmmm com,,
and https://en yna. yna.co.kr. This dataset is con- structed using similar method to Religion dataset. Positive samples are related to 'South Korea'. Hard negatives are composed of content related to other Northeast Asian countries, such 'North Korea', 'China', and 'Japan', which fall under the same country category but differ from South Korea. The instruction for annotators regarding samples is as follows: 'Please choose the correct label the following sentence. If the following sentence is not related to the country, choose 1. If the following sentence is related to South Korea, choose O. If the following sentence is related to North Korea, China, or Japan, choose 2. Answer: 0, 1, 2'. The com- position of the label is: (O:Positive, I:Negative). Examples of the dataset can be seen in Table 9.
In this study, we use total of six datasets. Fact- sNet is used in Section 4.2. Religion and South Korea are used in Section 4.3. AGNews, DBpedia, and TREC, are used in Section 4.4.
FactsNet We manually construct 291 test datasets for few-shot topic classification tasks across diverse topies by crawling http://www. facts. net. It has a three-level hierarchical structure con- sisting of a major category, a subcategory, and subtopic. The major category comprises six topics ("Lifestyle,' 'History,' 'Nature,' 'World,' 'Science.' and 'General"). However, we exclude 'General' when constructing the dataset since it consists of ambiguous content that does not align well with topic classification. Detailed information for five major categories can be found Table 5. Table provides types of subcategories under each major category and all subtopics under each subcategory. In constructing the dataset, we set the positive class 0, the easy negative class as 1, and the hard neg- ative class 2. The composition of the label is: (O:Positive, I:Negative), where the positive class belongs content related the subtopic, and the negative class contains easy negative and hard neg- ative. Table 7 shows example samples of datasets.
AGNews A collection of news articles from Heea is used for a 4-way topic clas- sification problem. We use a test dataset consist- ing of 7,600 samples from https://huggingfaae co/ datase taget_newee// news. The composition of the label is: (0: World, 1: Sports, 2:Buisness, 3:Sci/Tec).
DBpedia DBpedia ontology classifica- tion dataset is used for a 14-way topic classification problem. We use a test dataset consisting of 70,000 samples from https://huggingface http://cc...cccattttesppppaaaaaa composition of the label is: (0:Company, I:EducationalInssiturio,, 2:Artist, 3:Athlete, 4:0f- ficeHolder, 5: 6:Building 7: 7:NaturalPlace, 8: Villlage, 9:Animal, 10:Plant, 11:Album, 12:Film, 33WrritteWoork)
Religion Unlike FactsNet dataset, we determine whether samples are appropriate for positive and negative classes by examining each sample indi- vidually. Five annotators label each sample to de- termine if it contains content related to the topics defined in Religion. Samples for the hard negative class are constructed with the same method. The instruction for annotators regarding all samples is as follows: 'Please choose the correct label for the following sentence. If the following sentence is related to Jewish or Islam, choose O. If the fol-
The
TREC
Text Retrieval Conference Question Clas- sification is used for 6-way question classification tasks, which differ from topic-based content clas- sification. The sample consists of a question, and

---

the labels are divided into six types of question answers. A test datasct consisting of 500 samples is used from https:// https://huggingaacce httpeeee..//aatttttteeeeeeeee The composition of the label is: (0:Abbreviation, 1:Eniity, 2:Descripiion and abstract concept, 3:Hu- man being, 4:Location, 5:Numeric value).
B.4 Query examples
In Section 4.2, the queries obtained from DRAFT, sourced from Quick Facts, are additionally utilized as ICL examples for the LLMs. On the other hand, in Section 4.3, five annotators formulate both posi- tive and negative queries relevant defined classes in Religion and South Korea. Queries used in Sec- tion 4.2 and Section 4.3 can be seen in Table 10, For Section 4.4, we select queries randomly from the training dataset, using the random seed referred to in Appendix B.2.
B Details of experiment
B.1 Resources
When training a dense retriever model in DRAFT, bi-encoder, we employ a Distributed Data Parallel (DDP) setting with three GPUs. However, we only use a single GPU to train the classifier for the topic classification task in DRAFT.
During the experiments with LLMs, we use Ope- nAI's API and conduct ICL. The cost for GPT- 3 and InstructGPT models, with a model size of 175B, is $0.02 per thousand tokens, while the cost for model size of 2.7B is $0.0004 per thousand tokens. Due to API costs, we run the LLMs ex- periments directly only in Section 4.2, and in Sec- tion 4.4, take the results from (Zhao et al., 2021). Therefore, the candidates for k in the k-shot setting differ between them.
B.2 Hyperparameters
Hyperparameter settings for training bi-encoder are identical to those of (Karpukhin et al., 2020). all experiments, all classifiers in DRAFT use a fixed batch size of 256 and maximum token length of 128. As mentioned in Section 4.4, the subspace and number of queries tuned using grid search for hyperparameter search, using ac- curacy the criterion, while the remaining hy- perparameters are not tuned. The values used for the subspace size and the number of queries are [1, 5, 10, 50, 100, 500, 1000] and [5, 10, 15, 20, 25, 50J, respectively. The samples extracted from the training datasets of benchmark datasets in Sec- tion 4.4 are randomly selected using five random seeds: [1234, 5678, 1004, 7777, 9999].
B.3 Prompts format
We refer to (Chiu et al., 2021) for the prompt format LLMs for ICL in Section 4.2. Table 11 lists examples of prompt formats. The prompt format for LLMs in Section 4.4 can be found in (Zhao et al., 2021).

---

Mujor
Subcalegary
Subtopic
Entertiinees
Books, BTS, Chrismas Sougs, Dinney, Emeetianent, Friends, Harry Poner, Lego, Lard Of The Rings Marvel, Minceran, My Cousin Vinny, Netflis, Pukumun, Rock Peper Scieors Spotify, Sor Wirs, Tital, Video Came
Lifestyle
Foxl
Asucads Rear Berskfad Avand The Wirld. Cleodaae ConaColn, Culliee, Comflikes, Egg. Food, kee Cream, McDonald, Nutella, Fincopple, Pizza Pizoa Hie Starwacks Strawhery, Wattrmekon
Health
Hodlh
Sports
Baschall, Baikeball,
Rusning, Soccer, Sports
Cullure
Elypian Godlesses, Coliree, Langugg,, St. Patrick's Day, Thanksgiving, Mayans, Valentine's Day
History
Uistorical Events
Of July. Boston Party, Cinco de Historical Lveeni, Korean War, Poul Hurbrr Wodl War World War
People
Alexander the Great, Andie the Gant, Aphrodze. Auschwizz Barack Obama. Beethoven, Bill Gates, Bruce Lee. Cleopatra, Confucius, Donald Trump. Nightingaic,
Washingtos, Hades, Hero, Maya Angelon, Modher Toresa, Mozar. Araistinn. Jacksod, Facsbar. People, Prosidon, Pytluporas, Johs, Opeah Winfrey
Cap-
Maric Marilyn Manene, Martin
The Beatles, US Presideats, William Shakespeau, Womes Leaders
Religinn
Buddhisn, Easter
Animals
Bobeat, Camel. Cut. Chicken. Dolglin, Elephastt Frng. Golden Retrieves, Hippo, Horse, Killer Whale, Litle Red Hyyng Fox, Lobster, Meguiodon, Monsrch Buttefly, Morkey, Ones, Owl, Parasile, Penguin, Pigean, Rabhin, Red Tailed Hawk Shark, Sibesian Hasky, Snake, Tiger, Stufish, Whale, Welf
Animal,
Bec.
Chibadbus,
Human Body
Baby, Human Body, Modhcod, Sex
Sunflower
Univere
Earth, Juptier, Moen, Northem Lights, Plato, Satum, Univene
Cities
Rarcelum, Berlin, City. Calapagos Ishand, London, York, Paris, Pom- poùi, Tokyo, Vanke
World
Argentnaa, Relgium, Bolivis, Brasel, Cameka, China, Colombis, Casa Rizi, Country, Denmark, Dominican Republic Ecundor, ligspt, Greece, Germany. Green- land, Guntmmla,, Hais, Hondurss. Hong Kong. lusland, Iraiis, Inm. benee, Irall. Japan, Mexico. Mangolia, New Zealand, Nigetia, North Korea, Norway Pakistun, Philipjines Poland, Portspal, Puznoo Rico, Rassia, Scotland, Sings- Korea, Spain. Sweden. Switzeland, Haalaand Turkey, Venezuela,
Aighonistan,
ligpl,
Landmarks
Arco 51, Big Ben. Buckingham Palice, Bui Khalifa, Colasseun, E.ffel Tower, Golden Cose Briige. Grand Canyon Natsonal Pork, Michu Picchn, Mount Rushnore, Niggra Note Cathedal, Pasaita Eibetty Sunneengs, Sydnuy Open Pirk.
Fulls, Winch
rster House, Yelowstone
State
Illoois, Indiaso, Kasses, Kemncky, gas, Leuivirr, Maine, Maryland Micligun Mimssati,, Mivseui, Nebraska,
Alaska, Counecteata,
lampphhire, Mexzoo Canelina, Olio,, Ongan, Penaahhniis Rhude South Curolins, Trxas, US States, Virgima, Virginia Plan VS New Jersey Plan, Wiscorsin
Biology
Bictert, Biology
Science
Cheminyy
Gold, Lithium
Geography
Deforestation, Desert, Goography, Great Pacific Garbaag Patck, Rainforest Taiga Biomes, Weather
Physie
Tedhnology
Amason, Apollo 11, Cac, Internet, Linkedin, Plypal, Tochnology. WhitsApp. Yahoo
Table 6: The major categories, subcategories, and subtopics in the FactsNet dataset, which follows threc-level hierarchical structure, are as follows. There are 291 topics in the subtopic column, and each subtopic has its unique test dataset. The subtopics are listed in alphabetical order.

---

Topic
Text
Class
Many sculptures portray Poseidon with curly and beard. Since he is the god of the sea, Poseidon was also depicted with wet look distinguished by his hair.
Poscidon
Aside from entanglements with each other, the gods have also been known to pursue affairs with humans and animals alike.
Contrary to popular belief, sharks do have cars, although they aren't 't visible N(EN) like most species.
These interesting
include a shark's similarities cats.
N(EN)
The Danish politician took office in 201 But when Helle was still in high school, she was already taking part in antiapartheid efforts and peace movements.
N(HN)
Other organizations she served as chair to are the Congressional Black Caucus, N(HN)
Urban Affairs Committees, and Veteran's 's Affairs and Banking.
the Han dynasty entered period political turmoil, epidemics and viruses began to weaken the Han Dynasty.
China
From 1279 1368, the Yuan dynasty's 's vast size resulted more widespread foreign trade.
Since then, the website's 's services had grown from air beds and shared spaces to higher-end properties including the houses, upartments, private rooms, and other properties.
N(EN)
Unlike Accoleo's acquisition where Airbub bought the company before putting up an office Germany, Airbub established 2nd international office in London before acquired Crashpadder.
N(EN)
Wheat made up the higgest part of Ancient Egypt's harvests, used to make bread of various kinds.
N(HN)
Jewelry, in particular, provided a colorful contrast to the usual plain appear- ance of Ancient Egyptian clothing.
N(HN)
Table 7: 'Poseidon' and 'China' are examples of the 291 topies used in the few-shot topic classification task.
Class
Text
P
Their God communicates believers through prophets and rewards good deeds while also punishing evil.
P
Muslims are monotheistic and worship one, all-knowing God, who in Arabic is known as Allah.
EN
Enter your destination & your Tesla will automatically include Supercharging stops in your route.
EN
Comments section of Yahoo controlled by alt-reality hiased moderators supporting lies harmful to the Nation.
HN
The religion's 's founder, Buddha, is considered an extraordinary being, but god.
HN
Hinduism is unique in that it's not single religion but compilation of many traditions and philoso- phies.
Table 8: Examples from the test dataset under Religion dataset. During evaluation, easy negative (EN) and hard negative (HN) in Class column are used as the same negative class. We only use positive (P) and negative (N) labels for evaluation.

---

Class
Text
P
Bong Joon Ho's Parasite made history for bagging awards at the 2020 Oscars, which was most any film nominated.
P
Hangul classifies one of the Altaic languages, is affiliated to Japanese, and contains some Chinese loanwords.
EN
Recently after more than 20 years as Google account holder my YouTube channel was suspended without warning and without any reason given.
EN
Jordi Cruyff has signed his contract as FC Barcelona's new sporting director foothall. He has already been serving the role since July 1.
HN
The greatest health threat North Korea hunger.
HN
The Huizhou Ancient Town is famous historical and cultural city in southern Anhui Province with over 2000 years of history.
Table 9: Examples of the manually constructed dataset related South Korea.
Dataset
Query
Class
Poseidon is the Greek god of the sea.
Positive
He is also Greek god of storms, horses, and earthquakes. Poseidon's most famous brothers are Hades Zeus. His parents are known as Cronus and Rhea.
Positive
Poseidon
Positive
Positive
Poseidon is one the twelve Olympians included in ancient Greek religion and mythology.
Positive
China's 's population around 1.4 billion.
Positive
China hosted the 2008 Summer Olympic Games in Beijing- China's 's population 4 times larger than the United States. 60.7% of the population China lives in urban areas.
Positive
China
Positive
Positive
The Chinese civilization dates back 7,000 BC.
Positive
Judaism is the world's oldest monotheistic religion, dating back nearly 4,000 years.
Positive
Religion
Christianity is the most widely practiced religion in the world, with more than 2 billion followers.
Positive
Islam is the second largest religion in the world after Christianity, with about 1.8 billion Muslims worldwide. Buddhism is a faith that founded by Siddhartha Gautama ("the Buddha"") than 2,500 years ago in India.
Positive
Negative
Hinduism is the world's oldest religion, according many scholars, with roots and customs dating back more than 4,000 years.
Negative
South Korea
'South Korea', 'K 'K-pop, 'Seoul', 'Kimchi' 'North Korea'. 'Japan'. 'China'
Positive
Negative
Table 10: Examples of queries for manually constructed datasets. 'Poseidon' and 'China' example subtopics taken from the 291 subtopics in FactsNet. They only use positive class since tasks in Section 4.2 are one-class classification tasks. On the other hand, Section 4.3, Religion South Korea accept negative queries as well, resulting in two types of labels. The queries in South Korea comprise individual words rather than complete sentences. It underscores DRAFT's independence from the text format type of query, showcasing its capability to accept input regardless of whether it is in sentence or word format.

---

Input
Class
following text related to Poseidon? Answer yes or no.
He is also Greek god of storms, horses, earthquakes. yes Poscidon's most famous brothers are Hades Zeus. yes
ICI. Format
Positive
His parents are known as Cronus and Rhea." yes
Many sculptures portray Poseidon with curly hair and a beard. Since he is the god of the sea, Poseidon was also depicted with a wet look distinguished by his hair.:
Table 11: Formats used in LLMs in Section 4.2. The example shows a 3-shot setting for ICL, which have similar format in (Chiu et al., 2021). We first concatenate the problem description and examples, followed by the test sample. The 'Class" column represents the actual label of the test sample.


Reviews:
[{"review_id": "SxGNnu85vs", "reviewer": "Reviewer_1bbv", "summary": "", "questions": "Refer to Reasons To Reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your dedicated review of our work. Your critical feedback have helped us to extend and refine the paper. We provide a detailed response to your comments.\n\n$\\textbf{Q1}$  The article only chooses LLM as the baseline for comparison. Despite having a large number of parameters, LLM may not necessarily achieve the best performance for tasks in specific domains. You are suggested to compare more baselines based on PTMs.\n\n$\\textbf{Reply}$  In our study, we emphasized the significance of accurately classifying long-tailed arbitrary topics in real-world scenarios. The primary experiment of our research is the task of classifying 291 topics as detailed in Section 4.2 (Few-shot topic classification task). This task differs from general topic classifications, which define two or more classes. Task in Section 4.2 is a one-class classification task where only a single topic is defined and tested. To the best of our knowledge, existing few-shot classification methods operate exclusively on tasks with two or more defined classes and cannot conduct in one-class classification tasks. In contrast, DRAFT, through our proposed MQR algorithm, can automatically construct a Customized dataset for training. This allows DRAFT to perform the classification task even when samples for only one topic are provided, showcasing its novelty over other few-shot classification methods.\n\nFor quantitative assessment of our proposed framework, DRAFT, the best choice for a baseline model is the Large Language Model (LLM), and here's why: While LLM is a general-purpose model and might not be the best for specific domain tasks, it is currently the only model capable of performing classification tasks across a variety of topics where only one class is defined (one-class few-shot topic classification). We will include a detailed explanation in our manuscript as to why the LLM is the most appropriate baseline for our main experiment: the one-class few-shot topic classification task.\n\nIn Section 4.2, the baseline models used for the one-class few-shot topic classification task are detailed in Table 1, which includes LLM (GPT-3, InstructGPT), and in Table 2, which lists three weak baseline methods. As an extended experiment for this task, we further evaluated the SimCSE (same encoder type used in a classifier of DRAFT) , which was finetuned using few-shot samples that define only a single class. As anticipated, the results in the attached below Table show a lower performance due to the training being conducted on a dataset defined by a limited number of one-class samples. However, this additional experiment results with an extra baseline model has allowed us to underscore the excellence demonstrated by DRAFT as well as emphasize the difficulty of the one-class few-shot topic classification task. As shown in the Table below, we will integrate these results into Table 2 in our manuscript alongside the existing weak baselines.\n\n  | Major category | DRAFT | SimCSE | Random | Noun. | Dense. |\n  |----------|----------|----------|----------|----------|----------|\n  | Lifestyle  | 80.5(1)  | 67.1(3)  | 50.2(5) | 69.7(2)  | 58.0(4)  |\n  | History  | 84.7(1)  | 66.5(3)  | 49.4(4)  | 69.3(2)  | 42.5(5)  |\n  | Nature | 79.6(1)  | 67.0(3)  | 50.8(5)  | 62.9(4)  | 71.6(2)  |\n  | World | 82.1(1)  | 66.7(2)  | 50.2(4)  | 56.7(3)  | 45.3(5)  |\n  | Science  | 80.2(1)  | 66.7(2)  | 48.5(5)  | 54.5(4)  | 58.7(3)  |\n  | $\\textbf{Avg. Rank}$ | $\\textbf{1.0}$ | 2.6  | 4.6  | 3.0  | 3.8  |\n\n\n$\\textbf{Q2}$  Despite effectiveness, the proposed model is a little simple, which has been popular in retrieval-augmented methods in NLP.\n\n$\\textbf{Reply}$ There exist various retrieval-augmented methodologies in recent NLP research. These methods include solutions for tasks such as fact retrieval, open-domain question answering, and others which provide answers to questions (e.g., FiD, RAG, REALM), techniques that apply during inference to reduce perplexity in Language Modeling (e.g., kNN-LMs), and strategies that operate akin to memory for specific knowledge or dialogues (e.g., RETROPROMPT, KIF). To the best of our knowledge, all existing retrieval-augmented methods universally handle only a single query as the input to the retriever and subsequently execute various downstream tasks. (We will incorporate not only the papers you recommended but also cover various retrieval-augmented methods being studied in NLP tasks within the related work section. We are grateful for your suggestions on the missing references.)\n\nHowever, in our research, we introduce the MQR algorithm, which is the first to accommodate multiple-query inputs for the retriever. To address the functionality of the MQR algorithm: it's not merely a naive extension that takes a single query input multiple times. Instead, it operates by considering the semantic similarities among the queries. We have validated the efficacy of the MQR algorithm through experiments, specifically by examining the performance of a classifier trained on a Customized dataset constructed using MQR. All experimental results presented in our paper (Table 1 in Section 4.2, Table 3 in Section 4.3, and Table 4 in Section 4.4) consistently demonstrate the significant performance of DRAFT, indirectly affirming the validity of MQR. Also, by referencing the additional experiment results in Table presented in the Q1 Reply, we can observe that the SimCSE, which was only finetuned on few-shot samples without undergoing the MQR process in DRAFT, exhibits notably low performance which highlights the efficacy of MQR.\n\nUpon examining Figure 2 in our manuscript, which presents an ablation study based on the number of queries in DRAFT, a positive correlation between DRAFT's performance and the number of queries is evident. This suggests that using multiple queries as inputs proves more beneficial than relying solely on a single query. If DRAFT were to employ a naive retriever that can only handle a single query, as opposed to our MQR, the number of queries used to construct the Customized dataset would be limited to one, consequently adversely affecting its classification performance. Although the MQR algorithm might appear straightforward, its importance is accentuated when processing multiple queries concurrently to ensure optimal performance. We consider this feature a distinctive novelty in our research. Furthermore, the MQR algorithm can be universally applied across all the retrieval-augmented studies previously mentioned. We consider the introduction of this simple and effective idea – being able to process multiple queries simultaneously as input for retrieval – as one of the primary contributions of our research. We will place greater emphasis on the stated contribution in the manuscript.\n\n$\\textbf{Q3}$  A case study or visualization study might give clearer explanations about why a dense retriever could improve the effectiveness.\n\n$\\textbf{Reply}$ Table 8 and Table 9 in the Appendix provide examples of the Religion dataset and South Korea dataset, respectively, which were meticulously curated and reviewed by our research team on a sample-by-sample basis. In the Religion dataset, texts related to Jewish and Islam are designated as positive samples for our target topic, while texts associated with other religions, such as Buddha or Hinduism, are considered as hard negative samples. For the South Korea dataset, texts related to South Korea serve as the target topic's positive samples, with texts pertaining to North Korea, China, and Japan classified as hard negative samples. Given the similar semantics shared between the positive and hard negative samples in each dataset, it is a challenging task to perform fine-grained classifications. \n\nIn Table 3 in Section 4.3, the M3 method, which employs both the MQR algorithm and a random sampling technique to construct negative samples for the target topic within the Customized dataset, demonstrates the highest performance. The M3 method, in comparison to the M1 method which constructs negative samples only with random texts from Data Collection, achieves a higher F1 score. The M3 method even exhibits greater accuracy when handling hard negative samples. This underscores the enhanced effectiveness of employing a dense retriever when constructing the Customized dataset. For instance, in constructing the negative dataset for the South Korea topic, the M3 method utilizing a dense retriever correctly classifies texts related to 'Kim Jong-il, the leader of North Korea,' as the Negative class, indicating they don't pertain to South Korea. On the other hand, the M1 method, which doesn't use a dense retriever, mistakenly categorizes them under the Positive class associated with South Korea, likely due to the significant interrelation between the two Koreas. Thus, the dense retriever plays a pivotal role in facilitating the assembly of a training dataset apt for fine-grained topic classifications, proving its efficacy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Meaningful tasks and innovative work. This work first applies a simple dense retriever model for few-shot classification across diverse topics. This work holds significant innovative value and achieves notably superior performance compared to LLM. The topic classification task in this paper has certain application values. \n2. Detailed content. The introduction to the background, model design, experimental design, and other aspects are all written in a detailed and clear manner.\n3. Well-structured experiments, with thorough consideration of experimental conditions. Datasets are diverse and comprehensive, and the experimental results and analysis are convincing.\n", "reasons_to_reject": "1. The article only chooses LLM as the baseline for comparison. Despite having a large number of parameters, LLM may not necessarily achieve the best performance for tasks in specific domains. You are suggested to compare more baselines based on PTMs. \n2. Despite effectiveness, the proposed model is a little simple, which has been popular in retrieval-augmented methods in NLP.\n3. A case study or visualization study might give clearer explanations about why a dense retriever could improve the effectiveness. ", "questions_to_authors": "Refer to Reasons To Reject.", "paper_topic_and_main_contributions": "This paper proposes a framework designed to train a classifier for few-shot topic classification called DRAFT, which first applies a dense retriever model for few-shot classification across diverse topics. This approach demonstrates superior performance in few-shot topic classification than LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Why is the first letter of classifier framework in the title not capitalized?"}, "extra_text": {"Missing_References": "Similar methods applied for other NLP tasks:\n[1] Generalization through memorization: Nearest neighbor language models.\n[2] Decoupling knowledge from memorization: Retrieval-augmented prompt learning.\n[3] Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. \n[4] Recent advances in retrieval-augmented text generation. \n[5] Augmenting transformers with knn-based composite memory for dialogue. \n[6] Dense passage retrieval for open-domain question answering.\n...\n\nThe references are suggested to be added in Related Work. They are some examples of retrieval-augmented methods in other NLP tasks. "}}, {"review_id": "AUpTzqjWoD", "reviewer": "Reviewer_dsxP", "summary": "", "questions": "Respectively for reasons to reject,\n\n1.1. What is the novel part of this method? It seems that MQR (multi-query retrieval) is proposed by the authors, but I am wondering why this additional step is necessary if we can construct a customized data set for each topic directly by the MIPS algorithm. \n\n1.2. Is a single classifier built for each topic? If then, can we still see it as more efficient than the LLMs with in-context learning since we need repeated dataset construction and model tuning by the number of topics. \n\n\n2.1. Text classification has been long studied in Natural Language Processing and Data Mining community. It is a shame that there are not any discussion or comparison with these approaches. How is the proposed model different from the classical text classification methods?\n\n2.2. Similarly, few-shot classification has been actively studied recently. While the authors compare ICL LLMs with the proposed models, they are more like general-purposed models not necessarily optimized for topic classification tasks. How is the existing few-show classification method different from this method?\n\n2.3. Is it a fair comparison if only the proposed method exploits the external knowledge base (for data collection)? How much does it affect the performance improvement?\n\n\n3.1. Besides, is there a particular reason why the authors constructed a manual data set for evaluation?\n\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're grateful for your detailed evaluation and insight on our research paper. We will make sure to incorporate the parts that you suggested for clarity and reflect their feedback on paper. Also, in addition to the papers you mentioned under missing references, we will incorporate other studies related to few-shot text classification in the related work section. We appreciate your pointing out the relevant research that we may have overlooked.\n\n$\\textbf{Q1.1}$ What is the novel part of this method? It seems that MQR (multi-query retrieval) is proposed by the authors, but I am wondering why this additional step is necessary if we can construct a customized data set for each topic directly by the MIPS algorithm.\n\n$\\textbf{Reply}$ As you pointed out, our proposed MQR algorithm, capable of processing multiple queries, operates differently from existing retrievers that solely utilize the MIPS algorithm for retrieving. To address the functionality of the MQR algorithm: it's not merely a naive extension that takes a single query input multiple times. Instead, it operates by considering the semantic similarities among the queries. Unlike scenarios such as open-domain question answering or fact retrieval where there's a definitive answer, when constructing a training dataset for topic classification, we posit that retrieving using multiple queries through MQR — rather than approximately retrieving relevant passages for a single query via the MIPS algorithm — leverages more information. This, in turn, enhances the quality of the retrieved passage for a specific topic. Experimentally, as illustrated in Figure 2 of Section 4.4, we've demonstrated a positive correlation between the number of queries and task performance, thereby validating the superiority of employing multiple queries over a single one, which directly uses MIPS algorithm, for the topic classification task using the MQR algorithm. If DRAFT were to employ a naive retriever that can only handle a single query, as opposed to our MQR, the number of queries used to construct the Customized dataset would be limited to one, consequently adversely affecting its classification performance. \n\nAlthough the MQR algorithm might appear straightforward, its importance is accentuated when processing multiple queries concurrently to ensure optimal performance. We consider this feature a distinctive novelty in our research. Furthermore, the MQR algorithm can be universally applied across all the retrieval-augmented studies previously mentioned. We consider the introduction of this simple and effective idea – being able to process multiple queries simultaneously as input for retrieval – as one of the primary contributions of our research. We will place greater emphasis on the stated contribution in the manuscript.\n\n$\\textbf{Q1.2}$ Is a single classifier built for each topic? If then, can we still see it as more efficient than the LLMs with in-context learning since we need repeated dataset construction and model tuning by the number of topics.\n\n$\\textbf{Reply}$   As you pointed out, DRAFT creates an independent classifier for each unique topic. When considering time efficiency, DRAFT takes longer to construct its classifiers since it needs finetuning stage compared to an API-utilizing LLM. However, in environments where API usage is restricted and only the model is compared, handling models of 175B size or more demands significant resources and costs, often burdensome for individual researchers or even for most companies. In contrast, DRAFT only requires three V100 GPUs in a local environment for building Customized dataset and training, while only one V100 GPU is sufficient for inference. Even in situations where ample GPU resources are available to run models like InstructGPT, additional processing and validation are required due to inherent biases in in-context learning, such as the majority label bias. Another reason we emphasize DRAFT's efficiency is its cost-effective approach. Unlike typical classifiers that need manually constructed training data in a supervised setting, DRAFT autonomously creates its training dataset (Customized dataset), presenting a more cost-efficient alternative.\n\n$\\textbf{Q2.1}$ Text classification has been long studied in Natural Language Processing and Data Mining community. It is a shame that there are not any discussion or comparison with these approaches. How is the proposed model different from the classical text classification methods?\n\n$\\textbf{Reply}$ We agree on the importance of addressing text classification methodologies in our paper. We will make sure to highlight the distinctions between DRAFT and other text classification methods in the related work section. General text classification methods predominantly utilize supervised learning, training classifiers to discern patterns in texts corresponding to each class. Similarly, DRAFT employs supervised learning from training data to construct its classifier. However, a primary distinction between traditional approaches and DRAFT lies in the construction process of training data. While most traditional studies rely on manually curated training data, incurring high human annotation costs, DRAFT efficiently and autonomously constructs its own training data using only a few samples corresponding to each class. In summary, while the objective functions during training are similar, DRAFT offers a more efficient and practical methodology for constructing the prerequisite training data, making it more applicable in real-world scenarios.\n\n\n$\\textbf{Q2.2}$ Similarly, few-shot classification has been actively studied recently. While the authors compare ICL LLMs with the proposed models, they are more like general-purposed models not necessarily optimized for topic classification tasks. How is the existing few-show classification method different from this method?\n\n$\\textbf{Reply}$ In our study, we emphasized the significance of accurately classifying long-tailed arbitrary topics in real-world scenarios. The primary experiment of our research is the task of classifying 291 topics as detailed in Section 4.2 (Few-shot topic classification task). This task differs from general topic classifications, which define two or more classes. Task in Section 4.2 is a one-class classification task where only a single topic is defined and tested. To the best of our knowledge, existing few-shot classification methods operate exclusively on tasks with two or more defined classes and cannot conduct in one-class classification tasks. In contrast, DRAFT, through our proposed MQR algorithm, can automatically construct a Customized dataset for training. This allows DRAFT to perform the classification task even when samples for only one topic are provided, showcasing its novelty over other few-shot classification methods.\n\nFor quantitative assessment of our proposed framework, DRAFT, the best choice for a baseline model is the Large Language Model (LLM), and here's why: As you pointed out, while LLM is a general-purpose model and might not be the best for specific domain tasks, it is currently the only model capable of performing classification tasks across a variety of topics where only one class is defined (one-class few-shot topic classification). We will include a detailed explanation in our manuscript as to why the LLM is the most appropriate baseline for our main experiment: the one-class few-shot topic classification task.\n\n$\\textbf{Q2.3}$ Is it a fair comparison if only the proposed method exploits the external knowledge base (for data collection)? How much does it affect the performance improvement?\n\n$\\textbf{Reply}$ Using Data Collection is inherently a part of the framework we propose. Therefore, we believe it's challenging to discuss fairness when referencing external knowledge that doesn't directly contain the answers. To be precise, even the LLM model referred to more data during its pretraining stage and possesses a larger parameter size, giving it advantages compared to DRAFT. Given the absence of an entirely fair baseline model for the one-class few-shot topic classification task, we opted to use LLM in our experiments, as it represents the most suitable baseline for this particular task.\n\nAs mentioned in the response to Question 2.2, there isn't an appropriate baseline to compare with DRAFT in a completely fair setting. Hence, our decision to use LLM, which demonstrates the strongest performance in NLP, as the baseline was the most optimal choice. To confirm the efficacy of the external knowledge base, we conducted additional experiments under the same conditions as Section 4.2, evaluating 291 datasets. In these experiments, we compared the performance of DRAFT when utilizing the external knowledge base to when it was trained solely on queries, excluding the external knowledge base. The Table below presents the results of an additional experiment where we finetuned SimCSE (same encoder type used in a classifier of DRAFT) using only a few-shot samples, without utilizing an external knowledge base (Data Collection). Setting aside the limited size of the training data, this task provided information only for the positive class corresponding to the target topic (a one-class sample). As a result, the classifier couldn't distinguish the negative class, leading to an extremely high recall and markedly low precision score. These results underscore the critical role of the external knowledge base, which enables the functioning of the MQR algorithm.\n\n  | Major category | DRAFT | SimCSE | Random | Noun. | Dense. |\n  |----------|----------|----------|----------|----------|----------|\n  | Lifestyle  | 80.5(1)  | 67.1(3)  | 50.2(5) | 69.7(2)  | 58.0(4)  |\n  | History  | 84.7(1)  | 66.5(3)  | 49.4(4)  | 69.3(2)  | 42.5(5)  |\n  | Nature | 79.6(1)  | 67.0(3)  | 50.8(5)  | 62.9(4)  | 71.6(2)  |\n  | World | 82.1(1)  | 66.7(2)  | 50.2(4)  | 56.7(3)  | 45.3(5)  |\n  | Science  | 80.2(1)  | 66.7(2)  | 48.5(5)  | 54.5(4)  | 58.7(3)  |\n  | $\\textbf{Avg. Rank}$ | $\\textbf{1.0}$ | 2.6  | 4.6  | 3.0  | 3.8  |\n\n$\\textbf{Q3.1}$ Besides, is there a particular reason why the authors constructed a manual data set for evaluation?\n\n$\\textbf{Reply}$  Recent studies in topic classification have primarily focused on improving algorithms or methodologies on benchmark datasets with a limited number of topics. However, when considering real-world applications, there's a demand for classifying topics that span hundreds of categories. While most benchmark datasets for topic classification are designed for tasks involving the classification among topics with two or more defined classes, the real-world scenario often deviates from this norm. For instance, in platforms overflowing with text information like social media, there's a significant demand to classify texts related to a specific topic (one-class topic classification). We believe that existing benchmarks do not adequately capture real-world applications. Therefore, in response to the aforementioned considerations, we took the initiative to construct an evaluation dataset comprising 291 topics, tailored for one-class topic classification, to more accurately reflect these real-world demands.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Easy-to-understand presentation and writing\n2. Extensive evaluation and discussion\n3. Higher performances than in-context large language models.\n", "reasons_to_reject": "1. A straightforward implementation with a combination of existing works\n2. Lack of comparison with existing (few-shot) text classification methods ", "questions_to_authors": "Respectively for reasons to reject,\n\n1.1. What is the novel part of this method? It seems that MQR (multi-query retrieval) is proposed by the authors, but I am wondering why this additional step is necessary if we can construct a customized data set for each topic directly by the MIPS algorithm. \n\n1.2. Is a single classifier built for each topic? If then, can we still see it as more efficient than the LLMs with in-context learning since we need repeated dataset construction and model tuning by the number of topics. \n\n\n2.1. Text classification has been long studied in Natural Language Processing and Data Mining community. It is a shame that there are not any discussion or comparison with these approaches. How is the proposed model different from the classical text classification methods?\n\n2.2. Similarly, few-shot classification has been actively studied recently. While the authors compare ICL LLMs with the proposed models, they are more like general-purposed models not necessarily optimized for topic classification tasks. How is the existing few-show classification method different from this method?\n\n2.3. Is it a fair comparison if only the proposed method exploits the external knowledge base (for data collection)? How much does it affect the performance improvement?\n\n\n3.1. Besides, is there a particular reason why the authors constructed a manual data set for evaluation?\n\n", "paper_topic_and_main_contributions": "This paper addresses a topic classification task with arbitrary topics and limited labeled data. They use a dense retriever model to construct a customized dataset with a multi-query retrieval algorithm and fine-tune a pre-trained model for the classification task. Manually collected data sets and benchmark data sets are used for evaluation. The result shows that the proposed model is comparable with or outperforms the large language models with hundreds of billions of parameters. They argue that this is the first work to exploit a dense retriever model for a topic classification task. ", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There are many existing few-shot text classification methods (even more for the classical text classification task).\n\nLong-tailed Extreme Multi-label Text Classification by the Retrieval of Generated Pseudo Label Descriptions, Findings of ACL 2023\n\nContrastnet: A contrastive learning framework for few-shot text classification, AAAI 2022\n\nUncertainty-aware Self-training for Few-shot Text Classification, NeurIPS 2020\n\nHierarchical Attention Prototypical Networks for Few-Shot Text Classification, EMNLP 2019\n\n\nAlso, there is a retrieval augmented language model which may be directly comparable with the proposed model especially in an few-show learning with retrieval augmentation.\n\nAtlas: Few-shot Learning with Retrieval Augmented Language Models, Arxiv 2022"}}, {"review_id": "LVosmuEYgw", "reviewer": "Reviewer_r9Me", "summary": "", "questions": "Please refer to the reasons to reject.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your careful reading of our paper, and your feedback. Your insightful feedback has guided us in expanding and improving our paper. We prepared a detailed response to your questions.\n\n$\\textbf{Q1}$ Although I can understand the meaning of each paragraph in the introduction chapter, the cohesion between paragraphs can be strengthened, thus improving the fluency of the paper. Some expressions in this paper are confusing, such as positive samples, negative samples, and negative quires.\n\n$\\textbf{Reply}$ Thank you for your feedback regarding the cohesion between the paragraphs in the introduction. We recognize the importance of fluidity and will take steps to enhance the transitions and connectivity between paragraphs. Also, to mitigate any confusion regarding terminology within the manuscript, we provide clearer definitions of the terms.\n\n- Positive samples: These are datasets related to the target topic for classification, corresponding to the positive class in the Customized dataset.\n\n- Negative samples: These are datasets that do not relate to the target topic, representing the negative class in the Customized dataset. \n\n- Positive queries: These are the multiple queries used to extract positive samples related to the target topic. Through the MQR algorithm, these multiple queries are processed simultaneously.\n\n- Negative queries: In the default setting of DRAFT, negative samples in the Customized dataset are randomly sampled from Data Collection. However, as mentioned in Section 3.3 (Expanding the Capabilities of DRAFT), by providing queries for the hard negative class similar to positive queries as input to DRAFT, performance can be enhanced. These queries are referred to as negative queries.\n\nFurthermore, we will expand upon and further clarify any terms in the paper that may be potentially confusing to readers.\n\n$\\textbf{Q2}$ The novelty of this paper is limited. The retrieval augmented methods have been widely used in many NLP tasks, such as machine reading comprehension, dialog generation, questioning answering, text classification, and sentiment analysis. However, there are no related works about retrieval-augmented methods in this paper.\n\n$\\textbf{Reply}$ There exist various retrieval-augmented methodologies in recent NLP research. These methods include solutions for tasks such as fact retrieval and open-domain question answering which provide answers to questions (e.g., FiD, RAG, REALM), techniques that apply during inference to reduce perplexity in Language Modeling (e.g., kNN-LMs), and strategies that operate akin to memory for specific knowledge or dialogues (e.g., RETROPROMPT, KIF). To the best of our knowledge, all existing retrieval-augmented methods universally handle only a single query as the input to the retriever and subsequently execute various downstream tasks. (As you pointed out, we will incorporate various retrieval-augmented methods being studied in NLP tasks within the related work section. We are grateful for your notification on the missing references.)\n\nHowever, in our research, we introduce the MQR algorithm, which is the first to accommodate multiple-query inputs for the retriever. To address the functionality of the MQR algorithm: it's not merely a naive extension that takes a single query input multiple times. Instead, it operates by considering the semantic similarities among the queries. We have validated the efficacy of the MQR algorithm through experiments, specifically by examining the performance of a classifier trained on a Customized dataset constructed using MQR. All experimental results presented in our paper (Table 1 in Section 4.2, Table 3 in Section 4.3, and Table 4 in Section 4.4) consistently demonstrate the significant performance of DRAFT, indirectly affirming the validity of MQR. Also, by referencing the additional experiment results in Table below, we can observe that the SimCSE, which was only finetuned on few-shot samples without undergoing the MQR process in DRAFT, exhibits notably low performance which highlights the efficacy of MQR.\n\n  | Major category | DRAFT | SimCSE | Random | Noun. | Dense. |\n  |----------|----------|----------|----------|----------|----------|\n  | Lifestyle  | 80.5(1)  | 67.1(3)  | 50.2(5) | 69.7(2)  | 58.0(4)  |\n  | History  | 84.7(1)  | 66.5(3)  | 49.4(4)  | 69.3(2)  | 42.5(5)  |\n  | Nature | 79.6(1)  | 67.0(3)  | 50.8(5)  | 62.9(4)  | 71.6(2)  |\n  | World | 82.1(1)  | 66.7(2)  | 50.2(4)  | 56.7(3)  | 45.3(5)  |\n  | Science  | 80.2(1)  | 66.7(2)  | 48.5(5)  | 54.5(4)  | 58.7(3)  |\n  | $\\textbf{Avg. Rank}$ | $\\textbf{1.0}$ | 2.6  | 4.6  | 3.0  | 3.8  |\n\nUpon examining Figure 2 in our manuscript, which presents an ablation study based on the number of queries in DRAFT, a clear positive correlation between DRAFT's performance and the number of queries is evident. This suggests that utilizing multiple queries as inputs proves more beneficial than relying solely on a single query. If DRAFT were to employ a naive retriever that can only handle a single query, as opposed to our MQR, the number of queries used to construct the Customized dataset would be limited to one, consequently adversely affecting its performance potential. While the MQR algorithm may seem straightforward at first glance, it becomes indispensable when dealing with multiple queries concurrently, which we regard as a distinctive novelty in our work. Furthermore, the MQR algorithm can be universally applied across all the retrieval-augmented studies previously mentioned. We consider the introduction of this straightforward idea – being able to process multiple queries simultaneously as input for retrieval – as one of the primary contributions of our research.\n\n$\\textbf{Q3}$ Although the task is meaningful in practice, the proposed method is common in other NLP tasks. Even some studies proposed new retrieval methods, such as joint learning-based retrieval and retriever pre-training for specific downstream tasks. What do you think is the biggest difference between your retrieval method and theirs? Or what is the improvement of the retrieval model you made to solve the topic classification?\n\n\n$\\textbf{Reply}$  Existing research on retrievers has predominantly focused on enhancing the performance of the retriever itself, often through approaches like joint learning or pretraining. However, all of these existing retrieval studies have remained confined to accepting only a single query as input and retrieving semantically relevant passages. In our study, we introduce the MQR algorithm, which is designed to take multiple queries simultaneously as input and leverages the similarities between these queries to extract higher quality passages. By expanding the number of queries that retrieval can process at once through MQR, we have enabled the performance of a new task: the one-class few-shot topic classification task, which was previously unattainable using traditional retrievers. Our ablation study results presented in Figure 2 of the manuscript demonstrate experimentally how classification performance improves as the number of queries increases.\n\nAdditionally, the main experiment in Section 4.2 (Few-shot topic classification task) addresses a one-class few-shot topic classification task, wherein only one class related to the desired topic is defined. In the manuscript, we note that not only are there no existing retrieval-augmented few-shot classification methodologies suitable for this task, but there are also no standard few-shot classification models that can be applied to a one-class few-shot topic classification task. To the best of our knowledge, aside from the general-purpose model LLM, DRAFT is the pioneering approach capable of addressing this task. DRAFT introduces a straightforward methodology that employs a retrieval model for one-class few-shot topic classification tasks. Although it is a simple approach, there have been no prior attempts to employ retrieval for topic classification tasks. Our introduction of the MQR algorithm and its application represents a significant contribution of our research.\n\n\n$\\textbf{Q4}$ The case of some phrases in the article is confusing. Why should the first letter “C” of Customized dataset be capitalized? Why is the first letters “c” and “f” of the Topic classifier framework not capitalized in the title? Similar problems need to be reconfirmed. In Table 1, the improvement of DRAFT is limited, did the authors do a significance test? For example, DRAFT achieves 80.2 and InstructGPT 175B achieves 79.9.\n\n\n$\\textbf{Reply}$ Thank you for pointing out the crucial aspects of our paper. Our responses to each comment are provided below.\n\n- The term \"Customized dataset\" refers to a training dataset that is tailored for classifying specific topics, constructed by DRAFT accepting n queries as input. As it's a novel concept we introduced, we treated it akin to a proper noun, capitalizing the initial 'C'.\n\n- The reason for not capitalizing the 'c' and 'f' in the title was to represent the acronym DRAFT. However, in line with your feedback, we recognize the potential for confusion this might cause and will update them as capital letters.\n\n- Conducting a significance test on the 291 test data used in Table 1 was not feasible due to the high API costs associated with LLM. Nevertheless, in Section 4.2, both DRAFT and LLM utilize the same fixed query set in their experimental settings which assures no significant sources of randomness in the results. Moreover, the main point we aim to convey from the results in Table 1 is not that DRAFT significantly outperforms InstructGPT 175B, but rather that DRAFT achieves comparable performance with a smaller model. Therefore,we believe that there is no inconsistency in the overall logical flow of our argument in our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposed a simple method to solve the topic classification task, which is very significant and suffers from the problem of limited data in real-world applications.\n2. This paper conducted extensive experiments on benchmarks and manually constructed datasets to demonstrate the effectiveness of DRAFT.\n3. The dataset's details and experiment settings are declared clearly.", "reasons_to_reject": "1. Although I can understand the meaning of each paragraph in the introduction chapter, the cohesion between paragraphs can be strengthened, thus improving the fluency of the paper. Some expressions in this paper are confusing, such as positive samples, negative samples, and negative quires. \n2. The novelty of this paper is limited. The retrieval augmented methods have been widely used in many NLP tasks, such as machine reading comprehension, dialog generation, questioning answering, text classification, and sentiment analysis. However, there are no related works about retrieval-augmented methods in this paper. \n3. Although the task is meaningful in practice, the proposed method is common in other NLP tasks. Even some studies proposed new retrieval methods, such as joint learning-based retrieval and retriever pre-training for specific downstream tasks. What do you think is the biggest difference between your retrieval method and theirs? Or what is the improvement of the retrieval model you made to solve the topic classification?\n4. The case of some phrases in the article is confusing. Why should the first letter “C” of Customized dataset be capitalized? Why is the first letters “c” and “f” of the Topic classifier framework not capitalized in the title? Similar problems need to be reconfirmed.\nIn Table 1, the improvement of DRAFT is limited, did the authors do a significance test? For example, DRAFT achieves 80.2 and InstructGPT 175B achieves 79.9. ", "questions_to_authors": "Please refer to the reasons to reject.", "paper_topic_and_main_contributions": "This paper aims to solve the task of arbitrary topic classification. The authors designed a simple framework, named DRAFT, to train a classifier for few-shot topic classification. DRAFT first uses a few examples for specific topics to construct a customized dataset by a MQR algorithm. Finally, they fine-tune a topics classifier based on the customized dataset. Experiments on widely used datasets and manually constructed datasets demonstrate the effectiveness of DRAFT. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "Please refer to the reasons to reject."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework called DRAFT for few-shot topic classification using a dense retriever model. The model is shown to outperform large language models (LLMs) in experiments, but the novelty of the approach is questioned by the reviewers as it is similar to existing retrieval-augmented methods. The paper is well-written and provides detailed content and thorough experimental results, but the reviewers feel that more comparison with other methods is suggested. The dataset construction method is also questioned as it uses a multi-query retrieval algorithm, and more details are needed to make it more clear. The reviewers also suggest more discussion and comparison with existing methods is needed.
Decision: Accept-Findings