RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
A.4 LaGoNN ’s computational expense
In this Appendix section we discuss and provide results for LaGoNN ’s computation time. LaGoNN is more computationally expensive than Sentence Transformer- or SetFit-based text classification. LaGoNN introduces additional inference with the encoder, NN-lookup, and string modification. As the computational complexity of transformers increases with sequence length (Vaswani et al., 2017 ) , additional expense is created when LaGoNN appends textual information before inference with the ST. In Table 6 , we provide a speed comparison of comparable methods computed on the same hardware. 21 21 21 We used a 40 GB NVIDIA A100 Tensor Core GPU. On average, LaGoNN introduced 24.2 additional seconds of computation compared to its relative counterpart.
A.7 Additional results: general text classification
In this Appendix section, we provide additional results from our general text classification experiments in the main text, Section 6 . Here we show results comparing LaGoNN lite against SetFit lite and LaGoNN exp against SetFit exp , but we include results for one to five neighbors with LaGoNN LabDist , Figures 8 and 9 , respectively. The measure is average precision for IMDB, macro-F1 elsewhere.

In general, the number of neighbors we consider does not appear to have a large impact on LaGoNN ’s predictive power and our method continues to be a more stable classifier than SetFit and can generally be expected to improve SetFit’s performance. We also see that continued fine-tuning with the embedding model is only helpful for cases when the dataset has a relatively large number of labels. One exception to this is the case of Student Question Categories, where there are four labels. While it is clear that SetFit lite is a stronger model than LaGoNN lite, if we consider the more expensive alternatives, the story changes; if we continue to fine-tune, the prediction curves are essentially the same, and LaGoNN exp seems to have a slight edge on SetFit exp as we add training data.

LIAR, both the collapsed version we considered in our content moderation experiments and the original version (Orig Liar) we examine in our general text classification experiments here, seems to be a very difficult dataset. Adding examples or increased fine-tuning does not appear to consistently increase model performance. We observed this across all experimental settings and balanced regimes and is a sensible finding, as it should be very difficult to determine the truth of a specific statement without additional context.
A.8 Additional results: multilingual text classification
In this Appendix section, we provide multilingual text classification results from experiments where we compare SetFit exp and SetFit lite against LaGoNN exp and LaGoNN lite respectively. For these experiments, we used the Multilingual Amazon Reviews Corpus Keung et al. ( 2020 ) , which has five labels, where each label is a star rating in Chinese, English, French, German, Japanese, or Spanish. 22 22 22 https://huggingface.co/datasets/amazon_reviews_multi To create the mapping from label to text, we used code from the ADAPET Tam et al. ( 2021 ) port in the official SetFit repository. 23 23 23 https://github.com/huggingface/setfit/blob/main/scripts/adapet/ADAPET/utilcode.py In these experiments, we used the same multilingual pretrained Sentence Transformer for all models under the balanced sampling regime. 24 24 24 https://huggingface.co/sentence-transformers/paraphrase-multilingual-mpnet-base-v2 In the case of LaGoNN exp and LaGoNN lite , we use LabDist and search over one to five neighbors, reporting all results.

Figure 10 shows our results for expensive and inexpensive models. We note in all cases all models perform similarly. This supports our assertion in Section 6 that when the training data is balanced and we have only a handful of labels or less, it is sufficient to fine-tune the Sentence Transformer on only a subset of available training data. A classifier can then be fit on all available data, encoded with the fine-tuned ST. We observed this for SST-5 and observe it again here, especially clearly on the Chinese subset of this dataset. SetFit exp plateaus on the fifth step and stops learning, with different versions of LaGoNN exp outperforming it on later steps. However, if we move down on row, we see that all cheaper models continue to learn on all steps.
A.9 Ablations
In this Appendix section, we perform ablation studies with LaGoNN to support our findings in the main text.
A.9.1 Ablation: LaGoNN configurations
Here, we provide an in-depth comparison between all LaGoNN configurations, LABEL , DISTANCE , LabDist , TEXT , and ALL (see Table 1 ) for all content moderation datasets, balances, and levels of expense. The evaluation measure is the mean average precision ( \times 100 ) over five seeds in all cases except for Hate Speech Offensive where the measure is the macro-F1.

Below, Figures 11 through 15 are the results for the LaGoNN cheap training strategy, Figures 16 through 20 are the results for LaGoNN , Figures 21 through 25 are the results for LaGoNN lite , and Figures 26 through 30 are the results for LaGoNN exp . We place the figures on a new page for ease of viewing.

In the case of LaGoNN cheap , if we do not fine-tune the embedding model we see little variation in the standard deviation bands, with the exception of the LIAR dataset, which seems to be a very difficult dataset. When we do fine-tune, we see a great deal of variation, especially in cases of label imbalance, which is expected as the representations are altered more.
The performance of TEXT and ALL is very unstable, often being the worst performers, while sometimes being the best. Interestingly, we note that DISTANCE , LABEL , and LabDist often show very similar performance. In our opinion. LabDist seems to be the most consistent and stable performer, especially in cases when the embedding model is fine-tuned, LaGoNN , LaGoNN lite , and LaGoNN exp .

Overall, we believe that LabDist is the most performant/stable configuration of LaGoNN , and it is about this version that we present results in the main text. We note that we could have presented the best performer for each evaluation scenario, however, this is not in the spirit of our work as it adds yet another hyperparameter to configure, standing in the way of practical usage and convoluting our analysis. However, in our codebase, we hope that we have made it easy for one to change these configurations for their own usage, be it scientific or otherwise.
A.9.2 Ablation: LaGoNN k nearest neighbors
Here, at the suggestion of an anonymous reviewer, we present ablation results and analysis of searching over one to five nearest neighbors when modifying input via LaGoNN . We present results over all LaGoNN configurations under the LaGoNN lite fine-tuning strategy and with all balance regimes for the content moderation datasets. For the general text classification setting, we present results for both LaGoNN lite and LaGoNN exp fine-tuning under the balanced regime for all datasets with the LabDist and TEXT configurations.

If we consider all LaGoNN configurations and balance regimes in the case content moderation, Figures 31 through 55 , the number of neighbors does not appear to be an important hyperparameter; the learning curves for a given dataset and balance regime are very similar. While there is variation, the trend appears to be that the first NN results in the stablest, most performant, and most consistent model.

However, if we only focus on LabDist (Figures 31 through 35 ), the default LaGoNN configuration, we see that it can be a very important hyperparameter to consider in cases of extreme imbalance or when we have balanced data but few data points. For example, performance is boosted by up to five points for Hate Speech Offensive by the tenth step (1000 examples) with five neighbors under the extreme balance regime, yet for the balanced regime, the performance curves are roughly the same. For Toxic Conversations, in the balanced regime, we see that we can increase performance by up to seven points on the second step (200 examples) by considering more neighbors.

Turning our attention now to the general classification experiments, we see that the number of neighbors for both the LabDist and TEXT configurations continues to consistently not really make much of a difference, with all models showing very similar performance curves for all datasets. We note however that LabDist appears to be the most performant configuration of our method. While continued fine-tuning on datasets with a large number of labels does increase performance, we observe essentially the same boost for all neighbors. We also observe similar instability and performance degradation when we fine-tune on a large number of examples in cases when we have few labels.
A.9.3 Ablation: the effect of encoding distance
Here, at the suggestion of an anonymous reviewer, we present ablation results and analysis of how encoding distance affects LaGoNN , because PLMs often struggle to understand numbers. Note that during our development stage, we ensured that our tokenizer was capable of encoding floats with trailing digits. To examine the effect of trailing digits on LaGoNN , we consider the DISTANCE configuration (see Table 1 ), where we append only the Euclidean distance to the input text. In this ablation, however, we round to different levels of precision. For example, if the distance were a float of 0.123456789, we round it to the nearest whole number, 0.0, single digit float, 0.1, three digit float, 0.123, six digit float, 0.123457, and finally keep it unrounded, that is, the original DISTANCE configuration, 0.123456789. The below results are only for the LaGoNN lite training strategy. We chose LaGoNN lite for this ablation because it provides insight into both how distance affects full-model fine-tuning and only refitting the classification head. The results can be seen below in Figures 60 through 64 . We place the figures on a new page for ease of viewing.

Interestingly, we tend to observe very similar performance curves for all rounding precisions. The exceptions to this would perhaps be Amazon Counterfactual and Hate Speech Offensive in the balanced regime where DISTANCE and rounding to the third trailing digit respectively exhibit large instability.

Although not always the case, it appears that providing the model with the distance rounded to the nearest whole number tends to result in the strongest and stablest performer, however, we emphasize that in general there does not seem to a dramatic difference between the rounding precisions we considered. Longer digits slightly worsen model performance and the model might learn the most from simpler or abbreviated representations of distance. This finding motivated us to consider the ablation in Appendix A.9.4 .
A.9.4 Ablation: support for LabDist
The results from the ablation in Appendix A.9.3 suggest that rounding the distance to the nearest whole number results in a stronger classifier than appending the unrounded distance. Thus far, we have asserted that LabDist , where we append both the gold label of the NN and unrounded distance is the most performant version of LaGoNN (see Table 1 ). To demonstrate that this is reasonable, in this ablation study, we compare the original LabDist configuration against three models, namely the LABEL configuration, distance rounded to near whole number (Whole), and finally a new configuration similar to LabDist , but where we append the gold label and distance rounded to a whole number, which we refer to as LabRound . As in Appendix A.9.3 , in this ablation we consider only the LaGoNN lite fine-tuning strategy. We chose LaGoNN lite for this ablation because it provides insight into both how the different configurations affect full-model fine-tuning and only re-fitting the classification head. The results can be seen below in Figures 65 through 69 . We place the figures on a new page for ease of viewing.

In general, we note very similar performance curves for these four models. In the case of Insincere Questions, appending the distance after rounding it to the nearest whole number (Whole, the red curve), is a strong model, except in the balanced regime where we note large instability. The results for Amazon Counterfactual tell a different story, where rounding the Euclidean distance to the nearest whole number causes large instability and even degrades performance on the fifth step.

For the other evaluation scenarios, it is unclear what is the strongest method as sometimes LabDist is the best performer and sometimes it is Whole (the red curve). However, we believe that in general LabDist is the most stable model while also often being the most performant. We therefore choose it as our default LaGoNN configuration as a compromise between strength and stability. It is about this configuration which we report results in the main text. Our interpretation of this is that passing the model both a discrete prediction (the gold label of the NN) and a truly continuous measure of similarity (the unrounded Euclidean distance) gives it the most consistent and dependable reasoning ability.

We note, as we did in Appendix A.9.1 , that we could have presented the best performer for each evaluation scenario, however, it is not the goal of our work to create even more hyperparameters that must be iterated over. However, we hope that our codebase has made it easy for one to change these configurations for their own purposes.
Figure 1: LaGoNN LabDist uses an ST to encode training data, performs NN lookup, appends the NN’s gold label and distance, and optionally SetFit to fine-tune the embedding model. We then embed this new instance and train a classifier. During inference, we use the embedding model to modify the test data with its NN’s gold label and distance from the training data, compute the final representation, and call the classifier. Input text is in quotation marks, the NN’s gold label and distance are in brackets, and the integer label is in parenthesis.; LaGoNN LabDist uses an ST to encode training data, performs NN lookup, appends the NN’s gold label and distance, and optionally SetFit to fine-tune the embedding model. We then embed this new instance and train a classifier. During inference, we use the embedding model to modify the test data with its NN’s gold label and distance from the training data, compute the final representation, and call the classifier. Input text is in quotation marks, the NN’s gold label and distance are in brackets, and the integer label is in parenthesis.
Figure 2: First row: performance for all LaGoNN configurations and balance regimes for the Hate Speech Offensive dataset. Second row: LaGoNN performance for one to five neighbors for all balance regimes on a collapsed version of the LIAR dataset. We use the LaGoNN lite fine-tuning strategy (see Section 5.1 ).; First row: performance for all LaGoNN configurations and balance regimes for the Hate Speech Offensive dataset. Second row: LaGoNN performance for one to five neighbors for all balance regimes on a collapsed version of the LIAR dataset. We use the LaGoNN lite fine-tuning strategy (see Section 5.1 ).
Figure 3: Average performance for all sampling regimes on Toxic Conversations. More expensive models, such as LaGoNN exp , SetFit exp , and RoBERTa full perform best when the label distribution is imbalanced. As the distribution becomes more balanced, inexpensive models, such as LaGoNN lite , show similar or improved performance. The measure is average precision and we only consider one neighbor for the LaGoNN -based methods (see Appendix A.6 for additional results).; Average performance for all sampling regimes on Toxic Conversations. More expensive models, such as LaGoNN exp , SetFit exp , and RoBERTa full perform best when the label distribution is imbalanced. As the distribution becomes more balanced, inexpensive models, such as LaGoNN lite , show similar or improved performance. The measure is average precision and we only consider one neighbor for the LaGoNN -based methods (see Appendix A.6 for additional results).
Figure 4: Average performance on four datasets in the balanced sampling regime; the measure is average precision for IMDB, macro-f1 elsewhere. First row: SetFit lite compared to LaGoNN exp LabDist with modifications for one to five neighbors. Second row: SetFit exp compared to LaGoNN exp . See Appendix A.7 for additional results.; Average performance on four datasets in the balanced sampling regime; the measure is average precision for IMDB, macro-f1 elsewhere. First row: SetFit lite compared to LaGoNN exp LabDist with modifications for one to five neighbors. Second row: SetFit exp compared to LaGoNN exp . See Appendix A.7 for additional results.
Figure 5: We embed training data, retrieve the text, gold label, and distance for each instance from its nearest neighbor and modify the original text with this information. Then we embed the modified training data and train a classifier. During inference, the NN from the training data is selected, the original text is modified with the text, gold label, and distance from this NN, and the classifier is called.; We embed training data, retrieve the text, gold label, and distance for each instance from its nearest neighbor and modify the original text with this information. Then we embed the modified training data and train a classifier. During inference, the NN from the training data is selected, the original text is modified with the text, gold label, and distance from this NN, and the classifier is called.
Figure 6: Average performance in the imbalanced and balanced regimes relative to comparable methods. We include RoBERTa full results for reference. The measure is macro-F1 for Hate Speech Offensive, average precision elsewhere.; Average performance in the imbalanced and balanced regimes relative to comparable methods. We include RoBERTa full results for reference. The measure is macro-F1 for Hate Speech Offensive, average precision elsewhere.
Figure 7: Average performance for all the moderate and balanced sampling regimes on Amazon Counterfactual and Hate Speech Offensive. More expensive models, such as LaGoNN exp , SetFit exp , and RoBERTa full perform best when the label distribution is imbalanced. As the distribution becomes more balanced, inexpensive models, such as LaGoNN lite , show similar or improved performance. The measure is average precision for Amazon Counterfactual and the macro F1 for Hate Speech Offensive. We only consider one neighbor for the LaGoNN -based methods.; Average performance for all the moderate and balanced sampling regimes on Amazon Counterfactual and Hate Speech Offensive. More expensive models, such as LaGoNN exp , SetFit exp , and RoBERTa full perform best when the label distribution is imbalanced. As the distribution becomes more balanced, inexpensive models, such as LaGoNN lite , show similar or improved performance. The measure is average precision for Amazon Counterfactual and the macro F1 for Hate Speech Offensive. We only consider one neighbor for the LaGoNN -based methods.
Figure 8: SetFit lite performance compared against one to five neighbors for LaGoNN lite LabDist . The measure is average precision for IMDB, macro-F1 elsewhere.; SetFit lite performance compared against one to five neighbors for LaGoNN lite LabDist . The measure is average precision for IMDB, macro-F1 elsewhere.
Figure 9: SetFit exp performance compared against one to five neighbors for LaGoNN exp LabDist . The measure is average precision for IMDB, macro-F1 elsewhere.; SetFit exp performance compared against one to five neighbors for LaGoNN exp LabDist . The measure is average precision for IMDB, macro-F1 elsewhere.
Figure 10: Multilingual classification experiments. In the first row, we display results from expensive models on German, English, Spanish data, with their cheaper counterparts in the following row. In the third and fourth row, we do the same but for French, Japanese, and Chinese. The measure is macro-F1 in all cases.; Multilingual classification experiments. In the first row, we display results from expensive models on German, English, Spanish data, with their cheaper counterparts in the following row. In the third and fourth row, we do the same but for French, Japanese, and Chinese. The measure is macro-F1 in all cases.
Figure 11: LaGoNN cheap performance for all configurations and balance regimes on the Insincere Questions dataset. The relevant balance is in the title of each panel.; LaGoNN cheap performance for all configurations and balance regimes on the Insincere Questions dataset. The relevant balance is in the title of each panel.
Figure 12: LaGoNN cheap performance for all configurations and balance regimes on the Amazon Counterfactual dataset. The relevant balance is in the title of each panel.; LaGoNN cheap performance for all configurations and balance regimes on the Amazon Counterfactual dataset. The relevant balance is in the title of each panel.
Figure 13: LaGoNN cheap performance for all configurations and balance regimes on the Toxic Conversations dataset. The relevant balance is in the title of each panel.; LaGoNN cheap performance for all configurations and balance regimes on the Toxic Conversations dataset. The relevant balance is in the title of each panel.
Figure 14: LaGoNN cheap performance for all configurations and balance regimes on the Hate Speech Offensive dataset. The relevant balance is in the title of each panel.; LaGoNN cheap performance for all configurations and balance regimes on the Hate Speech Offensive dataset. The relevant balance is in the title of each panel.
Figure 15: LaGoNN cheap performance for all configurations and balance regimes on the LIAR dataset. The relevant balance is in the title of each panel.; LaGoNN cheap performance for all configurations and balance regimes on the LIAR dataset. The relevant balance is in the title of each panel.
Figure 16: LaGoNN performance for all configurations and balance regimes on the Insincere Questions dataset. The relevant balance is in the title of each panel.; LaGoNN performance for all configurations and balance regimes on the Insincere Questions dataset. The relevant balance is in the title of each panel.
Figure 17: LaGoNN performance for all configurations and balance regimes on the Amazon Counterfactual dataset. The relevant balance is in the title of each panel.; LaGoNN performance for all configurations and balance regimes on the Amazon Counterfactual dataset. The relevant balance is in the title of each panel.
Figure 18: LaGoNN performance for all configurations and balance regimes on the Toxic Conversations dataset. The relevant balance is in the title of each panel.; LaGoNN performance for all configurations and balance regimes on the Toxic Conversations dataset. The relevant balance is in the title of each panel.
Figure 19: LaGoNN performance for all configurations and balance regimes on the Hate Speech Offensive dataset. The relevant balance is in the title of each panel.; LaGoNN performance for all configurations and balance regimes on the Hate Speech Offensive dataset. The relevant balance is in the title of each panel.
Figure 20: LaGoNN performance for all configurations and balance regimes on the LIAR dataset. The relevant balance is in the title of each panel.; LaGoNN performance for all configurations and balance regimes on the LIAR dataset. The relevant balance is in the title of each panel.
Figure 21: LaGoNN lite performance for all configurations and balance regimes on the Insincere Questions dataset. The relevant balance is in the title of each panel.; LaGoNN lite performance for all configurations and balance regimes on the Insincere Questions dataset. The relevant balance is in the title of each panel.
Figure 22: LaGoNN lite performance for all configurations and balance regimes on the Amazon Counterfactual dataset. The relevant balance is in the title of each panel.; LaGoNN lite performance for all configurations and balance regimes on the Amazon Counterfactual dataset. The relevant balance is in the title of each panel.