Achilles-Bench: A Challenging Benchmark for Low-Resource Evaluation (2024)

Abstract: With promising yet saturated results in highresource settings, low-resource datasets have gradually become crucial benchmarks (e.g., BigBench Hard, superGLUE) for evaluating the learning ability of advanced neural networks. In this work, we find that there exists a set of “hard examples” in low-resource settings that challenge neural networks but are not well evaluated, which causes over-estimated performance. We first give a theoretical analysis on which factors bring the difficulty of low-resource learning. It then motivates us to propose a challenging benchmark AchillesBench to better evaluate the learning ability, which covers 11 datasets, including 8 natural language process (NLP) datasets and 3 computer vision (CV) datasets. Experiments on a wide range of models show that neural networks, even pre-trained language models, have sharp performance drops on our benchmark, demonstrating the effectiveness of evaluating the weaknesses of neural networks. On NLP tasks, we surprisingly find that despite better results on traditional low-resource benchmarks, pre-trained networks, does not show performance improvements on our benchmarks. there is still a large robustness gap between existing models and human-level performance, highlighting the need for robust low-resource learning models. 1

heading: 1 Introduction
text: Large-scale models have shown strong capabilities in learning from a handful of examples (Scao et al., 2022; Touvron et al., 2023a; OpenAI, 2023), resulting in an increased demand for low-resource benchmarks. Numerous research studies have highlighted the rapid adaptability of such models to new tasks, utilizing techniques like in-context learning (Dong et al., 2022). Consequently, the evalua-
*Equal Contribution 1Code and data are available on https://github.com/
Qian2333/Achilles-Bench.
tion of large-scale pre-trained models has shifted towards assessing their ability to quickly learn new downstream tasks with limited available samples, including superGLUE (Wang et al., 2019) and BIGBench Hard (Suzgun et al., 2022b).
However, many low-resource datasets usually use random or manual selection methods to sample data from the cleaned and balanced training data. They struggle to capture the data biases and increased difficulty commonly encountered in realworld scenarios. Consequently, these benchmarks fall short in evaluating the true learning gap between existing models and human-level models. While some models can surpass human performance on these benchmarks (e.g., SST-2) (Yang et al., 2019; Nangia and Bowman, 2019; He et al., 2021), many studies have revealed that these robust models still face challenges such as spurious correlation (Sagawa et al., 2020; Hu et al., 2023a) or bias (Bolukbasi et al., 2016), which are relatively uncommon in human learning. As depicted in Figure 1, models on a randomly sampled lowresource set demonstrate performance comparable
1 2057
to human-level in sentiment analysis. However, their performance significantly deviates from human level when confronted with challenging examples. It motivates us to propose a challenging low-resource benchmark.
In this work, we aim to find challenging examples given any tasks. This approach differs significantly from existing challenging benchmarks, which are either focused on complex tasks such as Big-Bench-Hard (Suzgun et al., 2022a) or specific to extremely few-shot settings like fewGLUE. In contrast to these previous studies (Hu et al., 2024), our proposed benchmark aims to generate difficult examples for any given task2. In addition, realworld low-resource data samples often exhibit biases towards specific domains, such as blank backgrounds in image detection or short sentences in handwritten hate speech. Therefore, our evaluation also includes a bias assessment. Specifically, we consider two dimensions: misleading examples with smaller classification margins for performance evaluation, and biased examples for robust evaluation. We begin by conducting a comprehensive analysis of how these two dimensions impact lowresource learning. Based on the insights derived from our analysis, we present an empirical solution to construct a challenging low-resource benchmark. The final benchmark encompasses 3 computer vision datasets and 8 natural language processing datasets.
To prove the effectiveness of the constructed benchmark, we evaluate 13 models, including 8 pre-trained models, such as T5 (Raffel et al., 2020), Llama (Touvron et al., 2023a), etc. All these models struggle to handle our benchmarks, with a large performance gap compared with randomlysampled low-resource benchmarks. On NLP tasks, we surprisingly find that despite better results on traditional low-resource benchmarks, pre-trained networks, do not show performance improvements on our benchmarks. The contribution of this paper is summarized as: 1) We propose AchillesBench, a challenging benchmark designed to expose Achilles’ heel (weaknesses) of neural networks. This benchmark provides a reflective view of the current progress in the field of low-resource learning. 2) We conduct a comprehensive analysis to identify the factors that particularly exacerbate the difficulty of low-resource learning. 3)
2To ensure the exclusion of mislabelled examples, we have implemented a human-check process in our work.
Experimental results demonstrate that our proposed benchmark effectively challenges existing models, including robust pre-trained networks and large language models.

heading: 2 Related Work
text: Low-resource Evaluation Learning on lowresource datasets has recently come into the spotlight with the introduction of more powerful models (Radford et al., 2019; Brown et al., 2020). Recent low-resource benchmarks use a transfer learning setting (Dumoulin et al., 2021; Zheng et al., 2021) as well as in-context learning (Schick and Schütze, 2020; Bragg et al., 2021), and they have also added up on dataset difficulty (Wang et al., 2018). Among these, there are two major types of low-resource benchmark: natural low-resource datasets, and sampled low-resource datasets. The former requires additional dataset curation (Wang et al., 2018; Koh et al., 2021; Srivastava et al., 2022) and currently, most lowresource benchmarks are uniformly sampled from larger datasets (Kolesnikov et al., 2020; Schick and Schütze, 2020; Brown et al., 2020; Logan IV et al., 2021; Alayrac et al., 2022).
Challenging Benchmark Previous approaches in constructing challenging benchmark mainly curate from natural data (Schick and Schütze, 2020; Zheng et al., 2021; Xu et al., 2021; Koh et al., 2021). These methods require heavy annotation and faces misalignment between human-perceived difficulty and samples hard for models. Our methods, however, create an annotation-free framework for building challenging training sets, which has the potential to quickly apply to any available task. Other work involved benchmarking a more comprehensive and challenging list of tasks (Ye et al., 2021; Mukherjee et al., 2021; Hu et al., 2023b), which deviates from our focus in finding model weakness on common tasks.
Data Pruning Our approach is similar to data pruning literature in that we both hope to find a difficult subset in a large dataset. Previously, data pruning methods (Toneva et al., 2018; Hacohen and Weinshall, 2019; Paul et al., 2021; Sorscher et al., 2022; Zhang et al., 2024) use data difficulty metrics including GradNorm and Loss Score to rank and prune datasets. However, we approach dataset sampling from a drastically different goal as we hope to challenge low-resource learning models.
2

heading: 3 Understanding the Difficulty of Low-Resource Learning
text: To better understand the challenges of low resource learning, we first look at the teacher-student setting in learning perceptrons. Consider a large curated dataset of N examples D = {xi, yi}i∈[N ] where xi ∈ Rd are i.i.d. random Gaussian inputs xi ∼ N (0, Id), with labels generated by a teacher perceptron T ∈ Rd as yi = sign(Txi). The number of samples N → ∞ but sample per parameter α = Nd = O(1) to remain trainable. Now we consider the low resource scenario where the number of training samples available P is much less than N , where αlow = Pd → 0. For convenience, we sample the data for low resource learning from dataset D such that Dlow = {xµ, yµ}µ∈[P ] ⊂ D. Learning on Dlow, we obtain a new student perceptron J that has generalization error ϵg.
Intuitively, three dimensions amount to the difficulty of learning perceptron J : (1) the number of training samples P (here we base the study of data scarity on the sample per parameter variable αlow); (2) the classification difficulty of the data samples, denoted by the margin m = minµ J(xµyµ); (3) the bias of the training dataset: here we look at a specific type of bias, spurious correlation, which draws correlation based on peripheral attributes of data items with a target variable, denoted as a student perceptron Jbias. We explore the difficulty of low-resource learning by altering our selection procedure for Dlow and explore how ϵg changes. Specifically, we look at three settings and use simulation experiments for analysis. 1) Low-resource learning, where Dlow is uniformly sampled from D. 2) Hard low-resource learning, where the margin of each sample is calculated mµ = T (xµyµ) and the samples with the smallest margins are se-
lected from D, as shown in Figure 2. 3) Biased low-resource learning, where a biased probe Jbias with θ angle to T is chosen as the spurious classifier. Then data that satisfies both yi = sign(Jbiasxi) and yi = sign(Txi) is uniformly sampled from D, as shown in Figure 2.
We elaborate on simulation settings in the Appendix.
Difficult data especially challenges low resource learning. We first compare the setting that increases data difficulty to the random-sampled version of Low-resource Learning. We vary our dataset size from 1% to 500% trainable parameters. As shown in Figure 3, the dark blue line corresponds to the setting where data is uniformly selected, and lighter lines range in data difficulty from margin 0.1 to 1. The functions of ϵg to α yield a crossover between the function for randomsampled training data and the one for increased difficulty training data, showing that increased data difficulty affects low resource settings more than sufficient data settings. Also, the increase in generalization error is more distinct for slightly larger training sets. As when the low-resource training set only has a few samples, it requires model to have strong generalization ability to beat the rule of generalization ϵ ∝ α−1 and the task is challenging enough.
Low resource learning is more sensitive to spurious correlations. In the biased learning scenario as shown in Figure 4, we compare students trained on biased datasets (red lines) to students trained on random-sampled datasets (blue lines). When the bias probe is more distinct from the teacher (larger θ), the drop in performance is more distinct. This is in line with the phenomenon that when a model overfits on spurious features that contain information distant from semantics, the model tends to suffer on generalization. Also, for smaller bias, low resource learning sees a larger drop in generalization while models with abundant data barely suffer. This show that low-resource learning is sensitive to even small biases.
Theoretical perspective Here we use theoretical analysis in addition to simulations to study the scenario that results in failed generalization in low resource learning. Again, we focus on the scenario where we have a large dataset D that represents the natural task distribution P . We sample a low resource dataset Dlow from D that form the distribution Plow. We theoretically show that the generalization error for the model trained on the low-
3
resource dataset is bounded by a function of data difficulty and the distribution bias of low-resource dataset.
Theorem 3.1. (Low-resource Generalization Measured by Distribution Shift and Data difficulty) Let H be the hypothesis space X → Rd. flow is the empirical risk ϵPlow(f) minimizer, and f is the hypothesis that minimizes expected risk ϵQ(f), m is the smallest margin of D to decision boundary of f . MMD(Plow, P ) describes the Maximum Mean Discrepancy (Gretton et al., 2012) between the sampled distribution and the original distribution. Then with probability over 1-δ,
ϵQ(flow) ≤ ϵQ(f) + c
√ |H| lnm+ ln ( 2 δ )
m
+ MMD(Plow, P ) + ϵα + ϵH (1)
where ϵα, and ϵH are small constants describing the error that occurred in training and the hypothesis space complexity, while c is the constant describing the scale of the effect of margin on generalization. Details are shown in Appendix.
The value of the Equation 1 right-hand side increases when m decreases and the term MMD(Plow, P ) increases, corresponding to the increase in data difficulty and the presence of data bias. This theorem applies not only to our simu-
lated scenario of perceptron learning but also to deeper models. In our biased learning setting, the distribution gap between low resource data distribution is larger for biased training set than randomsampled training set, i.e., MMD(P θlow, P ) > MMD(P randomlow , P ), since data samples forming P randomlow are sampled uniformly from P .
Based on our simulation experiments and theoretical results in the previous section, we find that low-resource learning is more likely to suffer from performance drop due to data difficulty and dataset bias. However, these scenarios are not covered in previous low-resource benchmarks. This motivates us to propose a challenging benchmark AchillesBench for better evaluation.

heading: 4 Achilles-Bench Challenge
text: We propose a new challenging benchmark that elevates low-resource learning difficulty on some well-known datasets. Unlike previous low-resource datasets that are randomly sampled from a training set, we curate the benchmark by selecting one of the most challenging low-resource training sets from GLUE, CIFAR10, CIFAR100, and ImageNet.
Following our theoretical analysis, we introduce the simple yet effective approach to build hardBench: First, we train a predictor for only one epoch on a large benchmark, obtaining a biased predictor; then, we score each sample on data difficulty for this stage of training. For each label, we pick the top k samples as our selected low-resource training set. We elaborate on the data difficulty metrics and the biased predictor respectively in section 4.1.

heading: 4.1 Metrics Measuring Data Difficulty
text: Previous literature in curriculum learning (Hacohen and Weinshall, 2019), data pruning (Paul et al., 2021), and continual learning (Toneva et al., 2018) propose metrics for data sample difficulty based on loss or gradient norms. Here we restate three metrics: Loss score, GradNorm score and explain how they can be applied in our problem scenario.
Loss Score Paul et al. (2021) and Sorscher et al. (2022) state this metric in the EL2N method, which intuitively measure data samples difficulty by looking at whether they can be learned correctly. Data samples with a higher loss score after training are more likely to be near the decision boundary. Therefore, we can select the hardest samples by ranking the loss score on the dataset. We call
4
datasets constructed via loss scores as AchillesBench (Loss). Examples with higher losses are selected as hard examples.
Gradient Norm Score Paul et al. (2021) discussed using gradient norm as an indicator of data importance. Samples with larger gradient norms shape the training geometry. However, there is little discussion on the connection between gradient norm and data difficulty. Here we give a brief and casual explanation. Based on previous analysis, we can find hard samples by checking their margin to the decision boundary of our model f , f(x0) = 0. Therefore, we can define the Lp norm margin as,
m(x) = min x0
||x− x0||p, s.t.f(x0) = 0 (2)
We use Taylor’s approximation for an approximate solution, following Elsayed et al. (2018).
m(x) ≈ |f(x)|∥∇xf(x)∥q , (3)
When the numerator is constrained (For a classification problem, we can constraint logits f(x) within 1 using sigmoid function), we can maximize the gradient norm to minimize margin. We call datasets constructed via gradient norm scores as AchillesBench (GradNorm). Examples with higher gradient norm scores are selected as hard examples.

heading: 4.2 Introducing Bias with Early Stopping
text: As shown in the above sections, we need to train a student predictor to estimate the decision boundary and thereby calculate the data difficulty score. However, we find that we can easily introduce bias into our selected benchmark dataset if we early stop training on the student predictor. We will give an explanation based on the Loss Score.
The Loss Score effectively estimates the difficulty of data examples to be classified correctly when the student predictor is exactly the same as the teacher model, i.e. θ = 0. However,
when the student model is undertrained, there would exist a gap γ between student g(x) = sgn(Jx) and teacher f(x) = sgn(Tx). For any x, the loss function would be L(x) = g(x) − f(x) = (J−T )x. Therefore, the resulting selected dataset Dlow = {(xi, yi)|xi = maxi=1,2,...Px (J − T )x, yi = sgn(Txi)} is isotropic in the nullspace of J − T , inducing a bias of γ2 .
This intuitively explains that we can use an early stopped predictor as well as data difficulty metrics to select a biased and difficult low-resource dataset that mimics the real-world setting. In the following sections, we use this approach to curate our Achilles-Bench.

heading: 5 Experiments

heading: 5.1 Benchmark Metric
text: Traditional low-resource benchmarks usually randomly choose a subset from the full-size training data as the training set. In this paper, we also follow this setting and extract hard examples from the fullsize data as the training data in our benchmark. To be specific, we implement three benchmarks in this work, which are described as follows. RandomBench. For each label, we randomly select k examples as the training set. We randomly select 3 subsets and report the average results. Achilles-Bench (Loss). For each label, we choose top-k hard examples based on losses scores. Achilles-Bench (GradNorm). For each label, we choose top-k hard examples based on gradient norm scores.

heading: 5.2 Benchmark Settings
text: Our framework is not limited to specific tasks, allowing for flexibility across various tasks. We benchmark on from-scratch models, pre-trained models, as well as large language models. In our implementation, we have chosen 11 tasks to generate a comprehensive and challenging benchmark. NLP Tasks We choose 8 datasets from GLUE (Wang et al., 2018), a collection of understanding datasets. We select a subset of the full-size training set as a training set. Following previous studies, we use the validation set as the test set considering the hidden test set. For the convenience of the demonstration, we show all the results with accuracy scores. For all NLP datasets, we implement BERT trained with one epoch as a biased predictor to select hard examples. For all NLP datasets, we extract 500 examples for each label (except for WNLI with 100 examples) as
5
the training set for our main results. Regarding large language models, we adopted the in-context learning paradigm, details can be find in Appendix E. We also build more variants with less training data. More results can be found at Appendix F. CV Tasks We also explore 3 widely-used image classification datasets, CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), and ILSVRC-2012 ImageNet (Deng et al., 2009) to demonstrate the generality of our approach. For each dataset, we select a subset as the training set in our benchmark, with 500 examples in CIFAR10, 50 examples in CIFAR-100, 100 examples in ImageNet-1K. results can be found at Appendix F.

heading: 5.3 Results Achilles-Bench challenges neural networks As
text: Table 1, Table 2 and Table 3 illustrate, AchillesBench can mislead neural networks with worse generalization errors. We re-implement strong understanding models, which have shown promising results in various low-resource tasks. For example, in Random-Bench, RoBERTa shows the near-human performance on SST-2 with 91% accuracy, which drops sharply on Achilles-Bench with only 51.01% accuracy on Achilles-Bench (GradNorm) and 50.55% accuracy on Achilles-Bench (Loss), nearly random-guessing results. Similar results are observed on CV datasets. For example, DenseNet-121 trained on a random sampling set achieves high test results with 71.33% accuracy on CIFAR-10. The accuracy drops to 59.87% on Achilles-Bench (GradNorm) and to 44.81% on Achilles-Bench (Loss). For LLMs, LLaMA-7B and LLaMA2-7B consistently demonstrate the low-
est performance on Achilles-Bench. Regarding BLOOM-1.1B’s performance on QQP, it is noteworthy that the model’s results are subpar compared to the label distribution, where "not duplicate" constitutes 63.2% of the dataset. The large performance drop also indicates that there is still a large gap between existing models and humanlevel models. All these drops demonstrate that our benchmark poses a great challenge.
Pre-trained networks show strong generation results on CV benchmarks, but still suffer from handling NLP tasks Compared with randomlyinitialized models, pre-trained networks show better generalization results in CV datasets, as shown in Table 3. For example, ViT-B/16 does not yield obvious performance drops on Achilles-Bench. As a comparison, pre-trained networks have much worse results on NLP tasks. On Random-Bench , pre-trained networks bring large performance improvements over random-initialized baseline (Transformer). However, on our benchmark, all pre-trained networks yield surprising performance drops. These results demonstrate that the results of pre-trained models on NLP tasks are more easily over-estimated.
Achilles-Bench (Loss) is more challenging than Achilles-Bench (GradNorm) We implement two metrics to select hard examples, including loss and gradient norm. Despite similar motivation, Achilles-Bench (loss) is more challenging than Achilles-Bench (GradNorm) according to our experimental results. On NLP tasks, Achilles-Bench (loss) also witnesses the worst results. Loss is the
6
most direct signal to see how neural networks understand an example. These difficult examples confuse neural networks, which barely learn core features. This learning weakness is not covered by existing low-resource benchmarks. Achilles-Bench provides a new perspective for understanding the learning abilities of different models.
Data augmentation slightly improves results Table 4 shows the results on CIFAR-10 with data augmentation techniques, cutmix (Yun et al., 2019). We can see that data augmentation brings slight performance improvements, but also faces the challenges of generalization on our benchmarks.
Achilles-Bench (Loss) demonstrate greater challenges compared to FewGLUE (Schick and Schütze, 2020) Table 5 presents a performance comparison between RoBERTa and GPT-2 on the 32-shot RTE task. The performance of GPT-2 under both the Achilles-Bench (Loss) and FewGLUE approaches tends to resemble random selection. Regarding RoBERTa, FewGLUE does not seem significantly more challenging than Random-Bench , whereas Achilles-Bench (Loss) demonstrates a higher level of difficulty.
Results on different metrics Table 6 presents the outcomes obtained on the 500-shot datasets from CIFAR-10 using the forget statistic technique (Toneva et al., 2018). Achilles-Bench (Loss) surpasses the forget statistic approach in all models,
7
including pre-trained models. The forget statistic technique does not appear to be more challenging than Random-Bench for pre-trained models.

heading: 5.4 Ablation Studies
text: Massive sampling fails to find a challenging benchmark In Random-Bench, we report the average results over 3 random samplings. In this part, we conduct 100 samplings and report the worst result in Figure 6 to figure out whether our methods can be replaced with massive sampling. As we can see, there is still a large gap between the worst results on Random-Bench and Achilles-Bench, indicating that the proposed method is an effective method to build challenging benchmarks.
Results on the selected set as the test set Figure 7 shows results on the selected set as the test set. As we can see, these “hard examples” capture the weakness of neural networks. If neural networks has not seen these examples, they fail on them.
Ablation studies on different models as predictors In our framework, we introduce a weak classifier as a biased predictor. For simplification, we choose FFN for CV datasets and BERT for NLP datasets. We conduct experiments on more networks to see whether the choice of predictors affects our conclusions. Table 7 and Table 8 show the attack results on SST-2 and CIFAR-10. For SST2, we test two more models: randomly-initialized Transformer and GPT2, as predictors. For CV models, we test two more models: ResNet-18 and ViTB/16, as predictors. All models show consistent
performance drops, indicating that our method is a universal model to generate challenging datasets to attack various models.

heading: 5.5 Explaining the Effectiveness of Achilles-Bench with Visualization
text: In this section, we compare samples selected by our Achilles-Bench with samples from RandomBench to demonstrate our approach reaches the goal of building difficult low-resource training set with shifted distributions. To make our observation more straightforward, we show visualizations in the Appendix G. We make the following observations based on these visualization results: Achilles-Bench induces bias in the low-resource training set From visualizations, we can see that both GradNorm and Loss variations of AchillesBench construct training sets that are drastically different from the data distribution. For SST2 task, specifically, Random-Bench exhibits ordinary statements containing words with clear emotional expressions. In contrast, both GradNorm and the Loss variations of Achilles-Bench opt for shorter sentences, incorporating statements with implicit emotional nuances. Similar biases are evident in other classes, showing that our approach successfully induces bias in the low-resource training set.
Achilles-Bench find challenging samples The method selects tough examples from datasets using difficulty metrics, notably in GradNorm and Loss. In the SST2 task, it favors terse, uninformative samples or input sentences that use sophisticated
8
vocabularies.

heading: 6 Conclusion
text: This paper proposes a challenging benchmark for low-resource learning. We first analyze which factors affect the difficulty of low-resource learning. We prove that low-resource generalization results in worse performance with more difficult and biased datasets. Hence we choose two metrics for measuring data difficulty, which result in two variants, Achilles-Bench (Loss) and Achilles-Bench (GradNorm) . Experiments show that both can better tell the learning gap between existing models than randomly-sampled low-resource datasets.

heading: 7 Acknowledge
text: This research is supported by the National Key Research and Development Program of China 2020AAA0106700, the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N_HKU714/21.

heading: A Limitation and Future Work
text: The method proposed in this paper can be extended to a wider range of tasks and datasets. In future studies, we aim to expand the tasks to more challenging datasets, such as superGLUE.
Furthermore, we did not test the results on the latest models, such as GPT-4. We intend to extend the tasks to include the latest large-scale models in the future.

heading: B Perceptron Model of Low Resource Learning
text: In this section, notations are defined as follows. We look at the teacher-student setting in learning perceptrons. Consider a large curated dataset of N examples D = {xi, yi}i∈[N ] where xi ∈ Rd are i.i.d. random Gaussian inputs xi ∼ N (0, Id), with labels generated by a teacher perceptron T ∈ Rd as yi = sign(Txi). The number of samples N → ∞ but sample per parameter α = Nd = O(1) to remain trainable. Now we consider the low resource scenario where the number of training samples available P is much less than N , where αlow = Pd → 0. For convenience, we sample the data for low resource learning from dataset D such that Dlow = {xµ, yµ}i∈[P ] ⊂ D. Learning on Dlow, we obtain a new student perceptron J that has generalization error ϵg.
In the basic low-resource learning scenario, we use a uniform sampling strategy to obtain Dlow from D. We model ϵg as a function of αlow. The results are as follows.
Lemma B.1. (Low-resource Learning, Seung et al. (1992)) For student perceptron J learned on high dimension dataset Dlow, the generalization error satisfies,
ϵg ∝ α−1low (4)
For other settings, the generalization error is only related to the angle between teacher model T and learned student model J . ϵg = arccosR/π, R = JT|J ||T | . Based on different low-resource dataset sampling strategies, we calculate the teacher-student overlap R with the geometry of each dataset distribution. Sorscher et al. (2022) has proved similar results in data pruning with our hard and biased learning setting. Here we only cite their results and don’t elaborate on proofs. Note that despite the proofs being similar, we use a different setting in perceptron learning. Their main objective is to understand how data-pruning can improve data efficiency, while we take an inverse stand, trying to understand challenging settings of low-resource learning.
Lemma B.2. (Hard Low-resource Learning, Sorscher et al. (2022)) Dlow is sampled from D such that ∀xµ ∈ Dlow,∀xγ ∈ D/Dlow, their margins satisfy |Txµ| ≥ |Txµ|. Let J be the student perceptron learned on high dimension dataset Dlow, and κ be the minimum margin minµ J(xµyµ). If the perceptron is trained to maximum margin, the generalization error of J satisfies,
ϵg = arccosR/π (5)
where R satisfies the saddle point equation,
R = 2α
f √ 2π √ 1−R2
∫ κ
−∞ Dt exp
( − R 2t2
2(1−R2)
)
· [ 1− exp ( − γ(γ − 2Rt)
2(1−R2)
)] (κ− t)
(6)
in which γ = H−1(N−P2N ), p(z) = e−
z2
2 N√ 2πP Θ(γ − |z|) The proof for this lemma can be found in Sorscher et al. (2022) A.5.1 and is omitted here for brevity.

heading: C Low-Resource Generalization
text: C.1 Proof for Theorem 3.1 Lemma C.1. Define ϵS(h, f) := Ex∼S |δ(h(x))− δ(f(x))|. For any hypothesis h, h′ ∈ H, there exists ϵH > 0 which satisfies,
12
|ϵPlow(h, h′)− ϵP (h, h′)| ≤ MMD(H, Plow, P ) + ϵH 2
(7)
ϵH is a constant for the complexity of hypothesis space.
Lemma C.2. Let flow be the trained classifier on the low resource distribution Plow, and f be the trained classifier on distribution P . Since Plow is formed by a subset of the training examples, when training error ϵP (f) → 0 and ϵPlow(flow) → 0, ϵPlow(flow, f) ≤ ϵα, where ϵα is a constant approaching zero.
Proof. ∣∣∣ϵPIk ( h, h′ ) − ϵP ( h, h′ )∣∣∣ ≤ sup h,h′∈H ∣∣∣ϵPIk ( h, h′ ) − ϵP ( h, h′ )∣∣∣
= sup h,h′∈H
∣∣∣Px∼PIk [ δ(h(x)) ̸= δ(h′(x)) ] −Px∼P [ δ(h(x)) ̸= δ(h′(x)) ]∣∣∣
= sup h,h′∈H
∣∣∣Px∼PIk [ h(x) ̸= h′(x) ] −Px∼P [ h(x) ̸= h′(x) ]∣∣∣
= sup h,h′∈H
∣∣∣∣ ∫
X 1h(x)̸=h′(x)dµPIk −
∫
X 1h(x)̸=h′(x)dµP ∣∣∣∣ (8)
Lemma C.3. Let f be the trained classifier on dataset D that is drawn i.i.d. from distribution P . f is tested on a test dataset S that is also drawn i.i.d. from the distribution P . Let m be the maximum margin of classifier f . Then with probability at least 1− δ,
ϵS(f) ≤ ϵD(f) + c
√ |H| lnm+ ln ( 1 δ )
m (9)
where ϵD(f) is the error on training set, and ϵS(f) be the error on test set.
Following Ben-David et al. (2010), we use Lemma C.1 and C.2 to prove Theorem 3.1. Proof
ϵQ(flow) ≤ ϵQ(f) + ϵQ(flow, f) = ϵQ(f) + ϵPlow(flow, f) + (ϵQ(flow, f)− ϵP (flow, f)) + (ϵP (flow, f)− ϵPlow(flow, f)) ≤ ϵQ(f) + ϵPlow(flow, f) + |ϵP (flow, f)− ϵQ(flow, f)|+ |ϵPlow(flow, f)− ϵP (flow, f)| ≤ ϵQ(f) + ϵα + |ϵP (flow, f)− ϵQ(flow, f)|+ MMD(Plow, P ) + ϵH
(10) In which,
|ϵP (flow, f)− ϵQ(flow, f)| = ∣∣∣∣ ∫
X 1flow(x)̸=f(x)dµP −
∫
X 1flow(x)̸=f(x)dµQ
∣∣∣∣
= | n∑
i=1
1flow(xi) ̸=f(xi) − EQ1flow(x)̸=f(x)| (11)
Here suppose test set Q matches the distribution of data for this classification task, and P is constructed by sampling n i.i.d. samples from the distribution Q. Using Lemma C.3 we have,
P (|ϵP (flow, f)− ϵQ(flow, f)| > c
√ |H| lnm+ ln ( 2 δ )
m ) ≤ δ (12)
Therefore, with a probability over 1− δ, we have
ϵQ(flow) ≤ ϵQ(f) + MMD(Plow, P ) + ϵα + ϵH + c
√ |H| lnm+ ln ( 2 δ )
m (13)
13

heading: D Models and Hyperparameters
text: We implement the following models for experiments in this paper. 1) FFN, a feed-forward neural network with two convolution and pooling layers and three feedforward layers. 2) VGG (Simonyan and Zisserman, 2014), a classical convolutional neural network. We use the VGG-16 with 13 convolution layers and three fully connected layers as implementation. 3) ResNet (He et al., 2016), a residual neural network. We use the ResNet-18 with 16 residual blocks, one convolution layer, and one fully connected layer as implementation. 4) DenseNet (Huang et al., 2017). We use DenseNet-121 with 121 layers, one convolution layer, and one fully connected layer as reimplementation.3 Besides, to verify the attack ability Gradon the pre-trained models, we also re-implement two pre-trained models: 1) Transformer-based ViT (Dosovitskiy et al., 2021)4 and 2) Convolutional-based EfficientNetV2 (Tan and Le, 2021)5. For FFN, VGG, ResNet, ResNeXt, and DenseNet on ImageNet, we resize all the images into 256× 256 and then center-crop them into 224× 224. For ViT on CIFAR, we resize all the images into 224× 224, while 384× 384 for EfficientNetV2.
We list hyper-paramters in Table 9. All the SGD optimizers are with a momentum of 0.9. For Adam/AdamW, we set β = (0.9, 0.999). We employed the torch.optim.lr_scheduler.MultiStepLR module to dynamically adjust the learning rate during training. Specifically, we set the milestones at epochs 20 and 40 (for ImageNet-1K, epochs 10 and 20 respectively) to adaptively update the learning rate based on the progress of training. The corresponding learning rate values used during three periods in our experiments are provided in Table 9. We conduct all the experiments on a single A100 GPU. We use Adam as the optimizer for all the NLP tasks with a learning rate of 2e− 5 and a linear scheduler.
3For VGG, ResNet, ResNeXt, and DenseNet on CIFAR and MNIST, we use the implementation from https://github. com/kuangliu/pytorch-cifar. As for ImageNet, we use the implementation from torch.models.
4We use the implementation from https://huggingface.co/google/vit-base-patch16-224 5We use the implementation from torch.models.
14

heading: E Details of ICL and LLMs
text: Considering the sentence length of different tasks and limitations of the GPU, we tested SST2 and COLA with 16-shots, MNLI, QNLI, MRPC, RTE, WNLI with 8-shots each, and QQP with 4-shots.

heading: F Results with Different Shots
text: To better explore the effectiveness of Achilles-Bench (GradNorm) and Achilles-Bench (Loss) , we demonstrate the results with different shots. The results are shown in the following tables.
Based on the results presented in Table 11 and Table 12, as compared to the findings discussed in Section 5.4, it is evident that all the model demonstrates a notable decline in performance when trained on a more limited dataset (20, 50-shot) in Achilles-Bench (Loss) and Achilles-Bench (GradNorm) , as compared to Random-Bench . This observation suggests that Achilles-Bench (Loss) and Achilles-Bench (GradNorm) pose greater challenges with fewer shots. While the pretrained model exhibits some level of robustness in Section 5.4, its performance still suffers when faced with more limited data. This highlights the significance of employing challenging benchmarks that incorporate scenarios with limited training data.
The results in NLP, as shown in Table 14, Table 15, Table 16, are generally consistent with the previous findings presented in Section 5.4. Nonetheless, a few specific models, characterized by inadequate few-shot learning capabilities, exhibited poor performance across all three benchmarks.
G Visualization
G.1 Selected Sentences by Achilles-Bench for NLP tasks
G.2 Visualization of the “Average Examples (Random)”
G.3 Visualization of the Searched “Hard Examples (GradNorm)”
G.4 Visualization of the Searched “Hard Examples (Loss)”

heading: Future Work
text: In future studies, we aim to expand the tasks to more challenging datasets, such as superGLUE. Furthermore, we did not test the results on the latest models, such as GPT-4. We intend to extend the tasks to include the latest large-scale models in the future