Introduction
Text classification is the most important tool for NLP practitioners, and there has been substantial progress in advancing the state-of-the-art, especially with the advent of large, pretrained language models (PLM) (Devlin et al., 2019 ) . Modern research focuses on in-context learning (Brown et al., 2020 ) , pattern exploiting training (Schick and
Schütze, 2021a , b ; Schick and Schütze, 2022 ) , adapter-based fine-tuning with learned label embeddings (Karimi Mahabadi et al., 2022 ) , and parameter efficient fine-tuning (Liu et al., 2022a ) . These methods have achieved impressive results on the SuperGLUE (Wang et al., 2019 ) and RAFT (Alex et al., 2021 ) few-shot benchmarks, but most are difficult to use because of their reliance on billion-parameter PLMs, pay-to-use APIs, and/or prompting. Constructing prompts is not trivial and may require domain expertise.

One exception to these cumbersome systems is SetFit. SetFit does not rely on prompting or billion-parameter PLMs, and instead fine-tunes a pretrained Sentence Transformer (ST) (Reimers and Gurevych, 2019 ) under a contrastive learning paradigm. SetFit has comparable performance to more unwieldy systems while being one to two orders of magnitude faster to train and run inference.

An important application of text classification is aiding or automating content moderation, which is the task of determining the appropriateness of user-generated content on the Internet (Roberts, 2017 ) . From fake news to toxic comments to hate speech, it is difficult to browse social media without being exposed to potentially dangerous posts that may have an effect on our ability to reason (Ecker et al., 2022 ) . Misinformation spreads at alarming rates (Vosoughi et al., 2018 ) , and an ML system should be able to quickly aid human moderators. While there is work in NLP with this goal (Markov et al., 2022 ; Shido et al., 2022 ; Ye et al., 2023 ) , a general, practical, and open-sourced method that is effective across multiple domains remains an open challenge. Novel fake news topics or racial slurs emerge and change constantly. Retraining of ML-based systems is required to adapt this concept drift, but this is expensive, not only in terms of computation, but also in terms of the human effort needed to collect and label data.

SetFit’s performance, speed, and low cost would make it ideal for effective content moderation, however, this type of text classification proves difficult for even state-of-the-art approaches. For example, detecting hate speech on Twitter (Basile et al., 2019 ) , a subtask on the RAFT few-shot benchmark, appears to be the most difficult dataset; at time of writing, it is the only task where the human baseline has not been surpassed, yet SetFit is among the top ten most performant systems. 2 2 2 https://huggingface.co/spaces/ought/raft-leaderboard (see ”Tweet Eval Hate”).

Here, we propose a modification to SetFit, called Like a Good Nearest Neighbor ( LaGoNN ). LaGoNN introduces no learnable parameters and instead modifies input text by retrieving information from its nearest neighbors (NN) seen during optimization. Specifically, we append the label, distance, and text of the NNs in the training data to a new instance and encode this modified version with an ST (see Figures 5 and 1 and Table 1 ). By making input data appear more similar to instances seen during training, we inexpensively exploit the ST’s pretrained or fine-tuned knowledge when considering a novel example. Our method can also be applied to the linear probing of an ST, requiring no expensive fine-tuning of the large embedding model. Finally, we propose a simple alteration to the SetFit training procedure, where we fine-tune the ST on a subset of the training data. This results in a more efficient and performant text classifier that can be used with LaGoNN . We summarize our contributions as follows:

We propose LaGoNN , an inexpensive modification to Sentence Transformer- or SetFit-based text classification.

We suggest an alternative training procedure to the standard fine-tuning of SetFit, that can be used with or without LaGoNN , and results in a cheaper system with similar or improved performance to the more expensive SetFit.

We perform an extensive study of LaGoNN , SetFit, and standard transformer fine-tuning in the context of content moderation under different label distributions, and in general and multilingual text classification settings.