Retrieval Augmented Zero-Shot Text Classification

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-25 16:18:04

Abstract: Zero-shot text learning enables classifiers to handle unseen classes efficiently, alleviating the need for task-specific training data. A
simple approach often relies on comparing embeddings of queries to those of potential classes. However, the embeddings of a simple
query may lack rich contextual information, which hinders performance. Traditionally, this has been addressed by improving the embedding models’ quality with expensive training. We introduce QZero, a novel training-free approach that reformulates queries by retrieving supporting categories from Wikipedia to improve zero-shot classification performance. Our experiments across six diverse datasets demonstrate that QZero enhances performance for state-of-the-art static and contextual embedding models without the need for retraining. Notably, in News and medical topic classification tasks, QZero improves the performance of even the largest OpenAI embedding model by at least 5% and 3%, respectively. Acting as a knowledge amplifier, QZero enables small word embedding models to achieve performance levels comparable to those of larger contextual models, leading to significant computational savings. Additionally, QZero offers meaningful insights that illuminate query context and verify topic relevance, aiding in understanding model predictions. Overall, QZero improves embedding-based zero-shot classifiers while maintaining their simplicity. This makes it particularly valuable for resource-constrained environments and domains with constantly evolving information.

Main paper: 
 introduction: 
 Zero-shot learning enables classifiers to handle unseen classes efficiently, alleviating the need for task-specific training data. Unfortunately, supervised classification models encounter significant challenges in scenarios that are characterized by an unconstrained label space [1, 7] . For example, consider the task of classifying recipes into categories based on their ingredients or regional cuisines. The diversity of culinary traditions and the continuous emergence of new dishes create an expansive and evolving label space. This complexity makes it difficult to define and collect labeled data for all possible label types, thus limiting the model's ability to accurately classify novel or unique recipes. The conventional practice of retraining a model for each new label set becomes impractical due to the substantial increase in expenses associated with annotation and computation. This has necessitated the widespread adoption of zero-shot text classification.Generative Large Language Models (LLMs) have revolutionized the field of natural language processing [3, 23] , especially for zeroshot learning. Their remarkable zero-shot abilities enable them to tackle diverse tasks with impressive efficiency and adaptability. However, applying generative LLMs directly for zero-shot text classification tasks presents challenges due to their massive size and computational demands. These models require significant computational resources for inference, which can render them impractical in resource-constrained settings. In addition, these models tend to make predictions that are independent of user-specified classes [35] . For instance, when classifying cuisine as Chinese or Mexican, a generative LLM model might incorrectly predict Italian, even though it wasn't an option. This lack of user control over the classification process poses a significant limitation to achieving targeted results.A straightforward and efficient alternative for zero-shot text classification involves assigning a class (or label) to a query (or text) by comparing the embeddings of the query and potential classes using a distance metric such as cosine similarity [15, 27] . This relatively cheap approach removes the need for retraining or additional data labeling and allows for control over the classification process. However, when applied to queries that do not explicitly reflect the context of the class or align well with the model's training data, the accuracy of this technique decreases. For example, consider Table 1 , which illustrates the contrast between explicit and implicit queries for a text input whose ground truth class is Technology. In the explicit example, keywords like "Artificial Intelligence" directly suggest Technology as the correct class. In contrast, the implicit example does clearly indicate the ground truth class. Here, a model's ability to infer Technology heavily relies on prior knowledge stored within its representations. For example, if the model lacks prior knowledge that "Sam Altman" and "Anthropic" are related to technology, it will fail to generate appropriate embeddings. Consequently, queries lacking sufficient contextual cues will result in reduced accuracy and recall. [20] . We introduce QZero, a simple retrieval augmentation approach to enhance the quality of embeddings used in zero-shot classification. Retrieval systems have emerged as powerful, cost-effective tools for various knowledge augmentation tasks, offering access to relevant information to keep models updated [12, 33, 24, 1] . They also benefit tasks like evidence-based modeling and text generation by expanding vocabulary and facilitating domain adaptation. Despite their potential, retrieval models remain under-explored in improving the quality of embeddings in zero-shot text classification tasks. The QZero paradigm explores reformulating queries by retrieving supporting information from Wikipedia. This approach enhances the quality of input text and achieves better zero-shot classification performance without model retraining. QZero streamlines the classification process by ensuring the embedding is enriched with a diverse vocabulary while remaining current with minimal resources.Specifically, QZero, adopts a two-step approach for zero-shot classification tasks, as illustrated in Figure 1 . First, it utilizes a retrieval model to identify relevant categories within supporting documents for the input text. These retrieved categories are then reformulated as the new input, depending on the type of embedding model used. For contextual embedding models, the concatenation of the categories serves as the reformulated input. Static word embedding models, on the other hand, utilize keywords and frequencies extracted from the retrieved categories. Consequently, the model utilizes this reformulated input to generate embeddings for downstream text classification tasks.We evaluate QZero on six diverse datasets, using embedding models of different sizes, ranging from a simple Word2Vec to Open AI's text embeddings. Our results demonstrate that QZero benefits models of all scales. Notably, the additional context provided by QZero acts as a knowledge boost for smaller models, allowing them to achieve performance levels comparable to larger models. This translates to significant computational cost savings, as smaller models require less processing power. In addition, QZero offers meaningful insights that illuminate query context and verify topic (class) relevance, aiding in understanding model predictions. 
related work 2.1 zero-shot text classification with generative llms: 
 Zero-shot classification has seen significant advancements through various approaches. One prominent avenue utilizes large-scale generative models, like GPT (Generative Pre-trained Transformer) [3] , for inference tasks. Trained on massive text datasets, these models excel at understanding and generating natural language, making them attractive for zero-shot classification tasks. However, their immense size and computational demands limit their accessibility and efficiency for resource-constrained environments. In addition, these models tend to make predictions outside the scope of userdefined classes, hindering the model's applicability in scenarios where class-based predictions are required.Beyond large generative models, an alternative approach leverages embedding techniques to compare semantic similarity between the text and potential classes during classification. This approach involves encoding textual information, like words or sentences, into vector spaces using innovative methods like [21, 2, 29, 25] . Pioneering work by [5] and [8] laid the groundwork for this strategy. However, early embedding techniques had limitations in accuracy because of their reliance on simple modeling approaches, hindering their ability to effectively capture the nuanced relationships and semantic context within textual data. Recent research by [7] and [15] addressed this by fine-tuning pre-trained models with external knowledge sources like Wikipedia to improve the embedding quality for zero-shot classification. Although cheaper, these methods don't entirely eliminate the need for retraining, which can be challenging for rapidly evolving domains.To alleviate the need for retraining, recent studies like [28, 31] have explored retrieval-augmented approaches. These methods leverage external information by retrieving documents relevant to the query at inference time. These retrieved documents are then prepended to the query itself, serving as a form of query expansion. This approach improves the performance of language models on various tasks without further training the entire model. In addition, retrieval systems excel at handling new information efficiently, allowing for quick updates when new information arrives. Inspired by these studies, we investigate whether a similar retrieval strategy can improve the performance of text embedding models for zero-shot text classification tasks. Notably, while retrieval has been explored for text embeddings before (e.g., ERATE [24] ), our objectives differ. ERATE focuses on reducing the high computational cost incurred from generating embeddings using large models. In contrast, QZero prioritizes enriching query context for improved classification.Query enrichment, the process of reformulating or augmenting a query with supplementary information, has been widely explored in Natural Language Processing (NLP) tasks as a means to enhance semantic understanding and overall performance. While earlier research predominantly concentrated on its utility in information retrieval tasks [37, 6, 32, 14, 4] , recent investigations have extended its applicability to few-shot [9] and short-text classification tasks [34] . Techniques leveraging linguistic resources like synonyms, knowledge graphs, and pseudo-relevance feedback have shown promising outcomes. Our work builds upon these prior studies by investigating the application of query enrichment specifically for zero-shot classification tasks. 
methodology 3.1 qzero: the retrieval augmented query reformulation pipeline: 
 We present QZero, a simple retrieval augmented query reformulation pipeline designed to enhance the zero-shot classification performance of embedding-based models. This system enhances the representation of essential information by transforming input text into refined queries, resulting in richer embeddings. Figure 1 provides an illustration of the QZero pipeline. Consider the input example in the figure: "Sam Altman to discuss possible collaboration with Anthropic." The input text becomes a query to the retrieval system, which returns a list of relevant document categories. The first set of categories pertains to information about Sam Altman, while the second set describes Anthropic. Afterward, the original input is disregarded, and the reformulated query becomes the concatenation of the retrieved categories. Subsequently, the embedding of the reformulated query is compared to the embeddings of the potential classes using cosine similarity. The class with the highest cosine similarity becomes the model's prediction. In this section, we describe the components of the QZero pipeline in detail.We used Wikipedia2 as our knowledge corpus across all experiments. Wikipedia articles offer concise information on a wide range of subjects, organized into categories located at the bottom of the page. These categories serve as topics and keywords associated with an article. We indexed English Wikipedia articles, focusing on article content and categories. Articles with fewer than 20 words and those with no assigned categories were excluded, resulting in 5.85 million documents with at least one category.The QZero scheme can be used with arbitrary retrieval systems.To build our retrieval system, we constructed a one-time index of the selected Wikipedia articles, leveraging it for subsequent retrieval of relevant article categories. Each article, represented as a unique document, contains attributes such as content and categories. For classification tasks, the retrieval process focuses solely on returning the categories of the best-matched articles, ensuring a concise presentation of information.Wikipedia category names are usually lengthy and more detailed than conventional label names, which poses a challenge for static word embedding models. For such models, we used different keyword extraction strategies to harness the extensive information encapsulated in the Wikipedia category as described below:• SpaCy's POS tagging 3 : it identifies keywords within a sentence by breaking down the sentence into tokens. then it employs a trained statistical model to analyze each token and assign it a Part-of-Speech (POS) tag. We selected only Noun tokens as keywords for our experiments.• Capitalization: SpaCy's POS tagging struggles with proper nouns representing nationalities (e.g., Indians, Filipinos). To improve accuracy with such datasets, we switched to a simpler method based on word capitalization using simple Regex functions.• MedCAT [16] : When dealing with a specialized domain like medicine, we used the Medical Concept Annotation Tool.(MedCAT) for more precise extraction of medical terms and sentences.MedCAT is an open-source tool that was pretrained to identify medical terms in sentences and link them to standardized medical vocabularies.QZero takes a query 𝑥 as an input to the retrieval system (see Figure 1 ). The system ranks Wikipedia articles based on their relevance to the query and returns the categories associated with the top-ranked articles. We select the categories of the top 50 articles, ensuring comprehensive coverage of potentially relevant information. Each article has a varying number of categories, and similar categories might appear across multiple articles. Our approach retains all categories from each retrieved article and we also keep repeated categories to emphasize their potential importance. We explore the use case of the reformulated queries for contextual models that are optimized for generating meaningful embeddings from sentences and static word embedding models optimized for word inputs (details in Section 3.6). The reformulated query for the contextual embedding models comprises a concatenation of all the retrieved categories. This concatenation process is executed following the order of their respective articles' ranks, as depicted in Figure 1 and outlined in the equation below:𝑥 𝑠 𝑅 = (𝐶 1 , 𝐶 2 , .. . , 𝐶 𝑛 ) In the above equation, 𝐶 1 represents the retrieved categories for an article. Post-retrieval, the categories for each article are denoted as 𝐶 𝑖 , where i signifies the rank of the article (1 for the top-ranked article). Therefore, the reformulated query 𝑥 𝑠 𝑅 is the concatenation of the retrieved categories in the sequence of their corresponding article's rank. 3 https://spacy.io/usage/linguistic-features While contextual embedding models offer advantages, they have limitations regarding the number of input tokens. Concatenating too many categories can introduce noise into the reformulated query, and processing longer queries increases computational cost. To address these limitations, we restrict all contextual embedding models to use only the first 512 tokens of the concatenated categories as their refined queries.Static word embedding models are limited to single words as inputs; thus, we extracted keywords 𝐾 from the reformulated query using the appropriate strategy from Section 3.4 based on the dataset. Upon obtaining our keywords, we assign weights 𝑤 to them based on their frequency across the reformulated query, facilitating effective measurement of each keyword's importance (since keywords will be repeated across the reformulated query). This results in a refined query where each keyword is paired with its weight, forming a structured representation as a list of tuples expressed as:𝑥 𝑤 𝑅 = ((𝐾 1 , 𝑤 1 ), (𝐾 2 , 𝑤 2 ), . . . , (𝐾 𝑛 , 𝑤 𝑛 ))In the equation, 𝑥 𝑤 𝑅 represents the reformulated word representation that will be used as input into the static word embedding model, and (K, w) is the keyword and its corresponding weight.We explore zero-shot text classification using both contextual embedding and static word embedding models (Table 3 ) to leverage the distinct advantages of each embedding type. Contextual embeddings capture the overall meaning of the reformulated query, while static word embeddings allow for finer-grained analysis of the individual keywords in the reformulated query. Both approaches are explained as follows:• Contextual Embedding Models: To perform zero-shot classification, we use the contextual embedding models to obtain embeddings for the reformulated query 𝑥 𝑠 𝑅 and each potential class label. Next, we compute the cosine similarity (a measure of closeness between embeddings) between 𝑥 𝑠 𝑅 and each class. The class with the highest cosine similarity is assigned to 𝑥 𝑠 𝑅 (See Figure 1 ).• Static Word Embedding Models: First, we compute the embedding of each class label. For class labels with single words, we obtain their representation directly from the embedding model, while we average the representations of the constituent words within the label if it is a phrase. We also obtain the representation for each keyword 𝐾 in the 𝑥 𝑤 𝑅 . To determine the best matching topic, we compute the cosine similarity between each keyword 𝐾 in 𝑥 𝑤 𝑅 and each class label 𝑦, subsequently multiplying this similarity by the corresponding weight 𝑤 of 𝐾. These weighted similarities across all keywords in 𝑥 𝑤 𝑅 are then aggregated to yield cumulative scores for each class label. Finally, the class label with the highest accumulated score is assigned to 𝑥 𝑤 𝑅 . See Algorithm 1 for details.  
experimental setup 4.1 datasets: 
 For a comprehensive understanding of our study and to ensure a thorough examination of textual nuances across diverse domains, we evaluated QZero on six distinct publicly available text classification datasets. These include the AG News articles [36] , and the DBPedia factual knowledge base [17] , which contains a summary of Wikipedia extracts. We also utilize the Yahoo! Answers community-driven knowledge exchange [36] , and the Yummly Table 3 : Embedding Models EvaluatedEmbedding Type Word2Vec [21] static word GloVe [22] static word FastText [2] static word All-mpnet-base-v2 [29] contextual text-embedding-3-small (GPT-3-small) contextual text-embedding-3-large (GPT-3-large) contextual dataset [13] of recipes from various regional cuisines, obtained from Kaggle's What's cooking challenge 4 . Additionally, we employed the TagMyNews dataset [13] 5 , containing news from RSS feeds as adopted by [18] , and the Ohsumed corpus [10] 6 , a collection of medical abstracts about various diseases. We used labels similar to [15, 7, 13, 18] , which we show in Table 2 . We report the classification accuracy averaged over three runs on the test sets. We summarize all dataset statistics in Table 2 .We evaluated the impact of the QZero pipeline on embedding models by comparing their performance utilizing the original input versus reformulated queries. Six different static word and contextual embedding models (See Table 3 ) were tested.• Zero-shot classification via Contextual Embeddings: The approach is similar to the contextual embedding classification method described in Section 3.6. The only difference is that instead of the reformulated query, we compare the cosine similarity between the embeddings of the original text input and class labels to achieve zero-shot classification. • Zero-shot classification via Static Word Embeddings: The key difference between the baseline static word embedding approach in Section 3.6 and this baseline approach is the absence of weights in the input to be classified. This means we compared each class label to the average vector representation of words in the original input text.We accessed the text-embedding-3 small and large models through the OpenAI API7 in March 2024. while the All-mpnet-base-v2 model was accessed via hugging face 8 . We describe the All-mpnet base-v2 model, text-embedding-3 small and large models as All-mpnet, GPT-3-small, and large, respectively, throughout the manuscript. Unlike the All-mpnet base-v2 model, with a 512-token limit, Ope-nAI models can handle longer inputs. To ensure consistency and address the 512-token limit across all models, we employed the GPT-2 tokenizer9 when reformulating queries.To retrieve supporting categories, we explored two methods: a dense and a sparse retriever. The sparse retriever utilized the BM25 [26] algorithm implemented in Pyterrier [19] with default settings. For the dense retriever, we employed the Contriever [11] model, trained specifically on Wikipedia passages. The BEIR [30] framework enabled indexing and retrieval for the Contriever model. 
results and discussion 5.1 effect of retrieval augmentation on zero-shot performance: 
 Table 4 showcases the benefits of the retrieval augmentation pipeline (QZero), demonstrating performance improvements across all model sizes, especially in News topic classification datasets. In the TagMyNews dataset, the smallest model (Word2vec) experienced a significant 13.00% boost in accuracy, while even the largest model (GPT-3-large) saw a 6.61% increase. Similarly, in the AG News dataset, all models achieved a minimum accuracy gain of 4.17%, except for GPT-3-large, which exhibited a slight 1.57% drop. This drop in GPT-3-large's performance suggests potential noise introduced by uninformative categories in the reformulated query. QZero enhances smaller models to achieve performance comparable to larger models without QZero. In TagMyNews, a QZeroenhanced Word2vec outperformed GPT-3-large and GPT-3-small (with original input) by substantial margins of 3.56% and 9.27%, respectively. Similarly, in the AG News dataset, Word2vec outperformed GPT-3-small by 3.4% while achieving similar accuracy with GPT-3-large. This is particularly valuable for scenarios with limited computational resources or tight financial constraints, where utilizing OpenAI's expensive embeddings might not be feasible.Furthermore, QZero enriches the original input with useful context information that may be outside of the model's training data.For example, in the Ohsumed disease topic dataset, GPT-3-large and Word2vec (a model with limited medical knowledge) achieved a minimum of 5.00% increase in accuracy. On the Yummyly recipe datasets, the static word embedding models achieved a boost as high as approximately 38.00%. In addition, even the All-mpnet-base-v2 model, also lacking training data in the culinary domain, improved by 17.54% on the Yummyly recipes. This is impressive considering the limited medical and culinary domain information present in the general Wikipedia corpus, which was the only Knowledge corpus for QZero. These results highlight the promising potential of QZero as a cost-effective solution for domain adaptation challenges.Our results also demonstrate that retrieval augmented query reformulation is effective for classifying topics outside of the model's training data. By transforming the input query into a knowledge space the model is more familiar with, reformulation bridges the gap between the model's knowledge and unfamiliar topics. This enhances the model's adaptability, allowing it to handle a wider range of tasks and domains without extensive retraining. Additionally, this reformulation provides support for a more effective representation, leading to models that are better equipped to tackle unseen data or generalize across domains.Interestingly, we observed that across datasets, QZero rarely hurts the performance of the static word embedding models, and in cases where it does, the performance decline is typically minimal, under 1.00%. However, the impact on larger models is more diverse, especially for common evaluation sets like DBpedia and Yahoo Answers. Here, QZero's effect can vary in magnitude, with the largest decrease observed in GPT-3-large. This disparity might be linked to the training data used for these larger models. The Yahoo Answers dataset, for instance, was part of the training data for All-mpnet-base-v2 10 . Similarly, GPT-3 models are trained on massive-scale datasets that might already contain the information encoded in the reformulated queries. As a result, applying QZero for such datasets might introduce redundancy or contradict existing knowledge within these larger models, leading to performance drops.The findings presented in Table 4 highlight QZero's robustness, showcasing its compatibility with dense and sparse retrievers. The BM25 (sparse) retriever performs better on the Ohsumed dataset, which contains lengthy documents that most dense neural retrievers might struggle with. Interestingly, the BM25 retriever also outperforms the Contriever (dense) retriever on the Yummly dataset. This could be because Contriever's training data was not wellsuited for the specific domain of recipes in Yummly. In contrast, the Contriever (dense) retriever excels in tasks related to News topics, general QA, and Wikipedia topics, domains likely aligning better with its training data. These findings demonstrate how QZero adapts to the complementary strengths of each retrieval method, ultimately broadening its applicability across various use cases. Table 5 shows that QZero's capabilities go beyond simply classifying queries. By leveraging Wikipedia's knowledge base, it generates insightful details for each query, including relevant categories and keywords. These insights illuminate the context of the query and verify its connection to specific topics, enriching our understanding of the model's predictions. For instance, for the query "Today's executives address Lauer and Vieiera's exit buzz, " QZero identifies categories like "American television shows, " confirming and validating the entertainment ground truth. Furthermore, QZero's outputs can help in understanding the query's focus. Even without prior sports knowledge, QZero's outputs reveal the sports-related nature of the query "Martinez leaves bitter like Roger Clemens did almost exactly eight years earlier, Pedro Martinez has left the Red Sox apparently bitter about the way he was treated by management," aiding in interpreting the model's predictions. Our analysis of incorrect predictions reveals a few key sources of error. In some cases, the predicted topic and the annotated ground truth might differ, but both could be valid interpretations of the query's meaning. Alternatively, the ground truth itself could be inaccurate. For example, in Table 5 , the query "Why do I not walk correctly when I have sinusitis? Your equilibrium could be 'off' due to your sinusitis, which could cause problems with your inner ear" clearly has no connection to the ground truth, "Business & Finance. " Other errors stem from retrieving irrelevant categories or limitations of the embedding model itself. By understanding these nuances, we can leverage the model's capabilities to interpret its predictions, refine our evaluation methods, and ultimately enhance model accuracy.Figure 2 shows how QZero's performance changes with respect to the number of documents retrieved for reformulating queries. This applies to both GloVe and All-mpnet embedding models. We see a trend across all datasets: as more documents are retrieved initially, QZero's performance improves. However, once the number of retrieved documents exceeds 50, the accuracy plateaus in the case of the All-mpnet model due to the fixed number of maximum tokens 7) , (offering, 5), (scandals, 5), (finance, 4), (securities, 4), (litigation, 4), (stock, 3),Business the model can take as input. On the other hand, the accuracy of the GloVe embedding model starts to decline. This suggests that there's a point of diminishing return where retrieving more documents starts to include irrelevant ones. Retrieving too many documents dilutes the pool of relevant categories that appropriately describe the input query, ultimately reducing QZero's effectiveness. 
conclusions and future work: 
 Embedding models present an effective solution for zero-shot text classification. However, the conventional method of retraining or fine-tuning the entire model to improve embedding quality proves costly, especially in rapidly evolving domains. QZero addresses this challenge by offering a training-free solution that improves the embedding quality. It achieves this through the adoption of a retrieval augmented query reformulation, which is considerably more cost-effective to update. Our experiments and results demonstrate that QZero significantly boosts classification accuracy across a wide range of embedding models, ranging from smaller options like Word2Vec to larger models such as OpenAI's text embeddings.Furthermore, QZero enhances the performance of smaller models to match their larger counterparts. Beyond improving classification accuracy, QZero incorporates relevant information from Wikipedia articles, providing valuable insights into the context of queries and validating their pertinence to specific topics. This dual functionality strengthens the rationale behind model predictions, leading to more reliable and trustworthy classification outcomes. In summary, QZero improves the performance of embedding-based zero-shot classifiers while maintaining their simplicity, paving a promising path for more efficient and interpretable zero-shot classification techniques.While QZero shows promise, there are limitations that require future exploration. First, the current query reformulation process for contextual embedding models could benefit from further refinement. Secondly, there might be cases where the model's knowledge is simply insufficient, and query reformulation alone would not be enough to improve performance. Lastly, our experiments focused on embedding models. Investigating whether query reformulation or other forms of retrieval augmented learning can benefit other models, such as those for natural language inference or generation, would be an interesting avenue to explore. 
 
 Appendix: We downloaded wiki version: enwiki-20230820-pages-articles-multistream.xml.bz2 Conference'17, July 2017, Washington, DC, USA Anon.https://www.kaggle.com/competitions/whats-cooking/datahttps://github.com/AIRobotZhang/STCKAhttps://disi.unitn.it/moschitti/corpora.htmhttps://platform.openai.com/docs/guides/embeddingshttps://huggingface.co/sentence-transformers/all-mpnet-base-v2https://huggingface.co/docs/transformers/en/model_doc/gpt2training datasets can be found: via:https://huggingface.co/sentence-transformers/allmpnet-base-v2 


Decision: This paper propose a retrieval augmentation to enhance zero-shot classification.  The setup is to use frozen text-embeddings from models like Word2Vec / OpenAI text-embeddings API for classification. Under this setup, the authors argue that when the input text is short, the embedding models often lack knowledge, and cannot embed the text close to its class labels. The authors therefore propose to retrieve similar wikipedia passages (BM25/Contriever), add the wikipedia passages' categories to the input text, and re-embed the text for classifcation. Experiments show that this method significantly improve models like Word2Vec and FastText, but the gains on newer/strong embedding models such as OpenAI text-embeddings-3 is limited.

The paper is clearly written. The motivation is very intuitive. The method is simple yet can boost the quality of weak embedding models. 

However, the current version of the paper is not sufficiently convincing due to the following reasons:
1) On many datasets, the proposed method cannot outperform vanilla OpenAI text-embeddings-3-large. The complexity added from this method is not justified. 
2) Related to 1, the proposed seems to only benefit weak models like Word2Vec, and it needs to be explained why people would chose to use those older models over newer embedding models.
3)  Using frozen text-embeddings is one approach for zero-shot classification. E.g., one could prompt LLMs in a generative manner to generate the class labels. The authors should add these common zero-shot / few-shot classifcation baselines.

Therefore, we do not think the paper is ready for ICTIR. We encourage the authors to revise the paper, in particular adding stronger baselines to justify the claims.

Reviews: review: main_review: main_review: The work is quite understandable thanks to a linear writing and free of great confusion. The ease of reading has certainly favored my appreciation of the relevance of the problem also due to the self-interest cultivated for a long time. This is also why I did not find the proposed methods particularly original since it seemed to me to recognize many other similar methods described in the literature dedicated to Query Expansion. For limitations of personal knowledge I cannot say with certainty that the methods are original in the context of the WE and automated classification by neural networks and the like. However, I think others in the ICTIR community can find food for discussion and reflection.
main_review: main_review: The presentation is generally good and I think the ICTIR community will be interested.
main_review: main_review: * The paper employs some baselines, such as Word2Vec and other static word embedding models, which are outdated. It also fails to incorporate some of the latest open-source large models, such as LLaMa, for experimentation.

* From Table 4, it is evident that large models like GPT-3 do not perform as well in many classification tasks after using RAaG (Retrieval-Augmented Generation) as they do in their base form. This suggests that the method proposed by the authors has limitations, particularly when applied to large models.

* The paper primarily focuses on text classification based on text embedding methods. However, for large models, it is more common to use generative approaches for classification, which involves directly inputting the candidate categories and the text to be classified into the large model and allowing the model itself to generate the category. Additionally, RAaG is more suitable for generative methods.
main_review: main_review: (1) Did you find the paper easy to read and understand? 

Yes, the paper is easy to follow.

(2) Do you find the problem relevant? 

Yes.

(3) Do you find the methods original? 

While this paper may offer original contributions to the field of zero-shot classification, the proposed framework lacks originality in broader domains, such as retrieval-augmented generation.

(4) Do you find the results interesting? 

I gained little insight from the results. Most of them align with expectations.

(5) Do you think that others in the ICTIR community will be interested in this work?

Have some interests.