A.5 Additional results: initial experiments
Here we provide additional results from our initial experimental setup that, due to space limitations, could not be included in the main text. We note that a version of LaGoNN outperforms or has the same performance of all methods, including our upper bound RoBERTa full , on 54% of all displayed results, and is the best performer relative to Sentence Transformer-based methods on 72%. This excludes LaGoNN cheap . This method showed strong performance on the Insincere Questions dataset, but hurts performance in other cases.
In cases when SetFit-based methods do outperform our system, the performances are comparable, usually within a point, yet they can be quite dramatic when LaGoNN -based methods are the strongest. Below, we report the mean average precision \times 100 for all methods over five seeds with the standard deviation, except in the case of Hate Speech Offensive, where the evaluation measure is the macro-F1. Each table shows the results for a given dataset and a given label-balance distribution on the first, fifth, and tenth step followed by the average for all ten steps. In the table caption we provide a summary/interpretation of the results for a given setting.
The LIAR dataset seems to be the most difficult for all methods. This is expected because it likely does not include enough context to determine the truth of a statement.

Method Insincere-Questions Imbalanced 1^{st} 5^{th} 10^{th} Average RoBERTa full 39.8_{5.5} \textbf{53.1}_{4.6} \textbf{55.7}_{1.2} 50.6_{4.4} SetFit exp 43.7_{2.7} 52.2_{1.9} 53.8_{0.9} 51.4_{2.9} LaGoNN exp \textbf{44.5}_{4.5} 52.7_{2.4} 55.4_{2.0} \textbf{51.8}_{3.0} RoBERTa freeze 39.8_{5.5} 44.1_{3.6} 46.3_{2.4} 44.0_{2.0} k NN 23.9_{2.2} 30.3_{3.0} 31.6_{2.4} 30.0_{2.1} SetFit 43.7_{2.7} 47.6_{1.6} 50.1_{2.1} 47.6_{1.8} LaGoNN \textbf{44.5}_{4.5} 48.1_{2.2} 50.3_{1.7} 48.1_{1.9} Probe 40.4_{4.2} 49.4_{2.3} 52.3_{1.7} 49.0_{3.3} LaGoNN cheap 40.8_{4.3} 51.1_{2.4} 54.5_{1.4} 50.4_{4.0}

Method Insincere Questions Moderate 1^{st} 5^{th} 10^{th} Average RoBERTa full 48.1_{2.3} \textbf{54.7}_{1.9} \textbf{57.5}_{1.5} 53.9_{2.9} SetFit exp 48.9_{1.7} 53.9_{0.7} 54.2_{1.5} 52.3_{1.6} LaGoNN exp \textbf{49.8}_{1.6} 52.2_{1.9} 53.2_{3.3} 52.0_{1.4} RoBERTa freeze 48.1_{2.3} 50.2_{2.2} 52.0_{1.4} 50.2_{1.4} k NN 28.0_{2.4} 33.9_{2.8} 33.6_{2.0} 33.5_{1.9} SetFit 48.9_{1.7} 53.6_{1.9} 55.8_{1.7} 53.3_{2.2} LaGoNN \textbf{49.8}_{1.6} 54.4_{1.3} 56.9_{0.5} \textbf{54.2}_{2.2} Probe 45.7_{2.1} 52.3_{1.8} 54.4_{1.1} 51.4_{2.5} LaGoNN cheap 45.7_{2.2} 54.4_{1.6} 56.4_{0.6} 53.2_{3.2}

Method Amazon Counterfactual Imbalanced 1^{st} 5^{th} 10^{th} Average RoBERTa full 68.2_{4.5} \textbf{81.0}_{1.7} \textbf{82.2}_{1.0} 79.2_{3.9} SetFit exp 72.0_{2.1} 78.4_{2.8} 78.8_{1.2} 78.0_{2.1} LaGoNN exp \textbf{74.3}_{3.8} 80.1_{1.4} 79.0_{1.6} \textbf{79.5}_{1.9} RoBERTa freeze 68.2_{4.5} 75.0_{2.2} 77.0_{2.4} 74.2_{2.6} k NN 51.0_{4.1} 60.0_{3.1} 61.3_{2.1} 59.7_{3.0} SetFit 72.0_{2.1} 74.4_{2.3} 76.7_{1.8} 74.8_{1.4} LaGoNN \textbf{74.3}_{3.8} 76.1_{3.6} 77.3_{3.2} 76.1_{1.0} Probe 46.6_{2.8} 60.3_{1.4} 64.2_{1.2} 59.2_{5.2} LaGoNN cheap 38.2_{3.2} 55.3_{1.8} 61.0_{1.2} 54.4_{6.7}

Method Amazon Counterfactual Moderate 1^{st} 5^{th} 10^{th} Average RoBERTa full 73.9_{2.5} 80.0_{1.0} 80.1_{2.3} 79.1_{2.1} SetFit exp 76.5_{1.6} 77.0_{2.4} 74.7_{0.5} 76.5_{1.0} LaGoNN exp \textbf{78.6}_{2.2} 78.0_{2.1} 76.3_{4.9} 78.2_{1.0} RoBERTa freeze 73.9_{2.5} 76.6_{1.4} 78.5_{0.7} 76.4_{1.7} k NN 54.5_{3.1} 64.2_{1.9} 66.6_{1.3} 64.7_{3.5} SetFit 76.5_{1.6} 80.6_{0.5} 81.2_{0.3} 80.0_{1.4} LaGoNN \textbf{78.6}_{2.2} \textbf{81.2}_{1.4} \textbf{81.6}_{1.1} \textbf{80.8}_{0.9} Probe 52.3_{2.0} 64.1_{1.8} 67.2_{1.4} 63.1_{4.3} LaGoNN cheap 47.3_{3.4} 60.7_{1.5} 65.2_{1.4} 59.5_{5.2}

Method Toxic Conversations Extreme 1^{st} 5^{th} 10^{th} Average RoBERTa full 7.9_{0.5} 21.2_{3.7} \textbf{33.8}_{5.5} 21.9_{9.3} SetFit exp 8.8_{1.2} 18.1_{3.4} 24.7_{4.1} 17.6_{5.5} LaGoNN exp 8.9_{1.7} 17.4_{6.6} 26.4_{5.2} 17.9_{6.0} RoBERTa freeze 7.9_{0.5} 12.8_{2.4} 19.1_{3.2} 13.5_{3.5} k NN 7.9_{0.0} 8.7_{0.4} 8.7_{0.2} 8.5_{0.3} SetFit 8.8_{1.2} 13.1_{2.5} 16.3_{3.0} 13.0_{2.6} LaGoNN 8.9_{1.7} 13.8_{3.9} 17.1_{4.8} 13.4_{2.6} Probe \textbf{13.1}_{2.8} \textbf{24.6}_{2.6} 30.1_{2.1} \textbf{23.9}_{5.6} LaGoNN cheap 11.3_{2.2} 21.7_{2.7} 27.4_{2.3} 21.3_{5.3}

Method Toxic Conversations Imbalanced 1^{st} 5^{th} 10^{th} Average RoBERTa full \textbf{24.1}_{5.6} 43.1_{3.4} 52.1_{2.5} 42.4_{8.2} SetFit exp 21.8_{6.6} 44.5_{4.1} 51.4_{1.9} 42.1_{9.3} LaGoNN exp 22.7_{9.8} \textbf{49.1}_{5.6} \textbf{53.4}_{2.3} \textbf{45.6}_{9.8} RoBERTa freeze \textbf{24.1}_{5.6} 31.2_{4.4} 34.0_{4.0} 30.5_{3.1} k NN 11.5_{2.5} 14.7_{4.0} 15.3_{3.2} 14.6_{1.1} SetFit 21.8_{6.6} 26.7_{5.3} 30.2_{4.0} 26.6_{2.7} LaGoNN 22.7_{9.8} 27.6_{8.9} 30.3_{8.7} 27.4_{2.4} Probe 23.3_{2.7} 33.0_{2.8} 37.1_{1.8} 32.5_{4.2} LaGoNN cheap 20.5_{3.2} 31.1_{3.2} 35.6_{1.8} 30.5_{4.6}

Method Toxic Conversations Moderate 1^{st} 5^{th} 10^{th} Average RoBERTa full 34.2_{3.4} 45.5_{1.9} 52.4_{3.3} 45.7_{5.6} SetFit exp 33.6_{2.9} 47.2_{2.2} 46.6_{3.3} 44.3_{4.3} LaGoNN exp \textbf{36.6}_{4.2} \textbf{48.2}_{2.7} \textbf{49.9}_{3.7} \textbf{48.0}_{4.4} RoBERTa freeze 34.2_{3.4} 38.4_{2.1} 39.5_{1.8} 38.0_{1.5} k NN 19.4_{1.9} 21.5_{3.4} 22.4_{2.9} 21.6_{0.8} SetFit 33.6_{2.9} 39.2_{2.9} 41.6_{2.7} 38.6_{2.4} LaGoNN \textbf{36.6}_{4.2} 42.7_{3.7} 45.0_{3.5} 42.0_{2.5} Probe 29.0_{2.7} 36.1_{1.2} 39.1_{1.5} 35.5_{3.3} LaGoNN cheap 26.1_{2.7} 34.3_{1.3} 37.5_{1.8} 33.6_{3.6}

Method Toxic Conversations Balanced 1^{st} 5^{th} 10^{th} Average RoBERTa full 32.3_{1.1} 42.7_{1.8} \textbf{54.1}_{3.4} 43.8_{6.3} SetFit exp 35.7_{3.4} 32.6_{6.2} 37.4_{2.7} 36.5_{1.9} LaGoNN exp \textbf{40.4}_{4.4} 40.2_{6.6} 39.8_{7.5} 40.0_{1.2} RoBERTa freeze 32.3_{1.1} 39.2_{1.5} 41.0_{0.6} 38.5_{2.4} k NN 17.4_{0.8} 23.7_{2.6} 24.3_{2.7} 23.1_{2.0} SetFit 35.7_{3.4} 44.5_{2.9} 46.1_{2.8} 43.6_{2.9} LaGoNN \textbf{40.4}_{4.4} \textbf{46.6}_{2.7} 48.1_{2.2} \textbf{46.1}_{2.2} Probe 29.5_{2.4} 35.9_{0.9} 40.2_{0.9} 36.1_{3.5} LaGoNN cheap 26.8_{2.7} 34.5_{1.3} 38.5_{0.8} 34.4_{3.7}

Method Hate Speech Offensive Extreme 1^{st} 5^{th} 10^{th} Average RoBERTa full 30.2_{1.4} 43.5_{2.5} \textbf{51.2}_{2.2} \textbf{44.3}_{7.4} SetFit exp 30.3_{0.8} \textbf{44.0}_{1.3} 51.1_{2.0} 43.8_{6.5} LaGoNN exp 30.3_{0.7} 40.7_{2.9} 49.1_{4.4} 42.2_{6.2} RoBERTa freeze 30.2_{1.4} 33.5_{3.1} 34.4_{3.4} 33.1_{1.4} k NN \textbf{31.5}_{1.2} 35.9_{2.7} 37.4_{2.0} 35.8_{1.7} SetFit 30.3_{0.8} 38.4_{2.5} 41.1_{1.5} 37.8_{3.3} LaGoNN 30.3_{0.7} 35.7_{2.6} 39.1_{2.4} 35.6_{2.7} Probe 29.0_{0.2} 34.7_{1.5} 40.1_{2.1} 35.1_{3.8} LaGoNN cheap 29.0_{0.1} 36.9_{1.8} 40.5_{2.1} 36.2_{3.7}

Method Hate Speech Offensive Imbalanced 1^{st} 5^{th} 10^{th} Average RoBERTa full 50.6_{3.0} 65.2_{3.9} \textbf{70.3}_{1.2} 64.2_{5.3} SetFit exp 54.4_{4.3} 66.3_{1.8} 68.9_{2.0} 64.3_{4.5} LaGoNN exp \textbf{57.0}_{5.2} \textbf{67.0}_{4.4} 69.8_{2.1} \textbf{64.9}_{4.6} RoBERTa freeze 50.6_{3.0} 54.1_{1.6} 55.3_{2.3} 54.1_{1.3} k NN 55.6_{4.8} 57.3_{2.3} 58.8_{3.6} 57.4_{1.1} SetFit 54.4_{4.3} 57.0_{3.9} 58.2_{3.8} 57.2_{1.1} LaGoNN \textbf{57.0}_{5.2} 58.2_{4.1} 58.3_{3.4} 58.3_{0.6} Probe 46.5_{2.2} 57.8_{1.7} 60.3_{1.2} 56.5_{4.5} LaGoNN cheap 47.1_{1.3} 56.5_{2.2} 59.5_{2.5} 55.6_{3.8}

Method Hate Speech Offensive Moderate 1^{st} 5^{th} 10^{th} Average RoBERTa full 61.9_{3.4} 70.8_{1.0} \textbf{72.5}_{1.4} 69.9_{3.2} SetFit exp \textbf{64.3}_{4.2} 70.6_{2.4} 72.4_{0.5} 69.8_{2.8} LaGoNN exp 63.8_{4.9} \textbf{71.0}_{2.1} 72.3_{1.0} \textbf{70.0}_{3.0} RoBERTa freeze 61.9_{3.4} 63.2_{4.1} 64.1_{4.5} 63.2_{0.6} k NN \textbf{64.3}_{4.0} 63.3_{2.9} 63.9_{2.5} 63.7_{0.4} SetFit \textbf{64.3}_{4.2} 67.3_{3.2} 67.6_{2.3} 66.9_{1.1} LaGoNN 63.8_{4.9} 65.0_{5.3} 66.7_{5.9} 65.3_{0.9} Probe 55.6_{1.7} 63.8_{0.8} 66.1_{0.3} 63.2_{3.0} LaGoNN cheap 56.0_{3.6} 62.2_{1.4} 66.0_{0.9} 62.3_{2.9}

Method Hate Speech Offensive Balanced 1^{st} 5^{th} 10^{th} Average RoBERTa full 59.7_{3.5} \textbf{66.9}_{1.2} \textbf{69.2}_{1.8} \textbf{66.4}_{2.7} SetFit exp 60.7_{1.3} 66.3_{1.6} 67.5_{0.9} 65.9_{2.2} LaGoNN exp \textbf{61.5}_{1.7} 66.4_{1.4} 67.7_{0.9} 66.1_{1.8} RoBERTa freeze 59.7_{3.5} 60.4_{2.7} 63.1_{2.3} 61.0_{1.3} k NN 60.7_{1.3} 59.6_{2.8} 59.5_{2.5} 59.5_{0.5} SetFit 60.7_{1.3} 62.5_{0.7} 63.4_{1.0} 62.3_{1.0} LaGoNN \textbf{61.5}_{1.7} 62.8_{1.5} 64.2_{1.0} 63.0_{0.9} Probe 54.9_{1.4} 58.5_{0.9} 60.9_{0.4} 58.7_{1.7} LaGoNN cheap 54.2_{2.3} 58.6_{0.6} 60.6_{0.5} 58.5_{1.8}

Method LIAR Extreme 1^{st} 5^{th} 10^{th} Average RoBERTa full \textbf{32.0}_{2.7} \textbf{34.7}_{2.9} 35.1_{4.3} 33.7_{1.0} SetFit exp 31.2_{3.8} 30.4_{3.1} 31.8_{2.9} 31.5_{0.7} LaGoNN exp 30.6_{4.7} 30.3_{2.0} 31.3_{2.0} 31.1_{0.6} RoBERTa freeze \textbf{32.0}_{2.7} 32.8_{4.5} 34.2_{5.0} 33.2_{0.7} k NN 27.0_{0.5} 27.3_{0.8} 27.9_{0.8} 27.4_{0.3} SetFit 31.2_{3.8} 33.7_{5.1} \textbf{35.7}_{5.1} \textbf{34.3}_{1.6} LaGoNN 30.6_{4.7} 32.0_{4.6} 33.7_{5.4} 32.6_{0.9} Probe 30.7_{2.0} 30.6_{3.9} 31.7_{2.9} 31.1_{0.4} LaGoNN cheap 30.7_{2.0} 30.5_{3.8} 31.4_{2.6} 31.0_{0.4}

Method LIAR Imbalanced 1^{st} 5^{th} 10^{th} Average RoBERTa full 31.4_{3.2} 35.8_{2.6} \textbf{40.0}_{4.3} 36.2_{2.4} SetFit exp \textbf{32.3}_{4.5} 35.9_{3.1} 36.4_{2.2} 35.2_{1.1} LaGoNN exp \textbf{32.3}_{4.6} 35.7_{3.4} 36.5_{2.3} 35.7_{1.4} RoBERTa freeze 31.4_{3.2} 34.1_{2.6} 35.6_{3.2} 34.0_{1.4} k NN 27.0_{0.2} 28.5_{1.0} 29.0_{1.0} 28.7_{0.7} SetFit \textbf{32.3}_{4.5} \textbf{36.5}_{3.1} 38.5_{3.4} \textbf{36.3}_{2.0} LaGoNN \textbf{32.3}_{4.6} 34.9_{2.2} 36.9_{2.5} 35.3_{1.4} Probe 30.7_{3.0} 32.8_{1.8} 35.0_{1.6} 33.5_{1.5} LaGoNN cheap 30.4_{3.0} 32.9_{1.8} 35.4_{1.7} 33.5_{1.7}

Method LIAR Moderate 1^{st} 5^{th} 10^{th} Average RoBERTa full 33.9_{3.1} 38.4_{2.7} \textbf{43.9}_{2.2} \textbf{39.5}_{3.0} SetFit exp 33.0_{2.6} 37.2_{1.8} 38.7_{1.5} 37.4_{1.6} LaGoNN exp \textbf{34.1}_{3.4} \textbf{38.7}_{2.3} 39.0_{1.8} 37.8_{1.5} RoBERTa freeze 33.9_{3.1} 35.3_{2.6} 36.8_{2.2} 35.4_{1.0} k NN 29.2_{0.8} 29.7_{1.5} 30.0_{0.6} 29.8_{0.3} SetFit 33.0_{2.6} 37.2_{3.9} 39.4_{3.5} 37.0_{1.8} LaGoNN \textbf{34.1}_{3.4} 37.0_{3.1} 38.6_{3.0} 36.8_{1.3} Probe 31.6_{1.1} 34.7_{2.5} 37.0_{2.5} 34.9_{1.7} LaGoNN cheap 31.4_{0.9} 35.3_{2.3} 37.6_{2.0} 35.3_{1.9}

Method LIAR Balanced 1^{st} 5^{th} 10^{th} Average RoBERTa full 33.8_{2.1} \textbf{39.4}_{2.4} \textbf{43.5}_{1.7} \textbf{40.2}_{3.2} SetFit exp \textbf{34.4}_{2.3} 36.7_{1.7} 37.0_{1.3} 36.5_{1.1} LaGoNN exp 33.8_{1.8} 34.2_{2.7} 37.2_{1.9} 36.2_{1.4} RoBERTa freeze 33.8_{2.1} 36.6_{1.6} 38.6_{1.5} 36.7_{1.5} k NN 30.1_{0.4} 31.3_{2.1} 30.6_{1.1} 30.9_{0.4} SetFit \textbf{34.4}_{2.3} 38.3_{2.5} 40.0_{2.0} 37.9_{1.6} LaGoNN 33.8_{1.8} 38.3_{1.3} 40.6_{0.6} 38.1_{2.0} Probe 32.1_{1.9} 35.2_{1.4} 37.2_{2.5} 35.2_{1.7} LaGoNN cheap 31.9_{1.9} 36.0_{1.0} 37.5_{2.5} 35.7_{1.8}