NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Tasks
Year: 2023

Abstract:
Finetuning large language models inflates the costs of NLU applications and remains the bottleneck of development cycles. Recent works in computer vision use data pruning to reduce training time. Pruned data selection with static methods is based on a score calculated for each training example prior to finetuning, which involves important computational overhead. Moreover, the score may not necessarily be representative of sample importance throughout the entire training duration. We propose to address these issues with a refined version of dynamic data pruning, a curriculum which periodically scores and discards unimportant examples during finetuning. Our method leverages an EL2N metric that we extend to the joint intent and slot classification task, and an initial finetuning phase on the full train set. Our results on the GLUE benchmark and four joint NLU datasets show a better timeaccuracy trade-off compared to static methods. Our method preserves full accuracy while training on 50% of the data points and reduces computational times by up to 41%. If we tolerate instead a minor drop of accuracy of 1%, we can prune 80% of the training examples for a reduction in finetuning time reaching 66%.

text: Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 129–146 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: State-of-the-art natural language understanding (NLU) systems rely on finetuning large language models (LLMs) with considerable amounts of human-labelled examples. Such overparametrized models succeed at learning to classify nearly all training samples correctly. However, not all training examples are created equal: proper classification of out-of-distribution outliers requires more iterations for the model to converge. Moreover, many examples are correctly learned after only a few epochs of training and could be ignored further on without impacting test accuracy.
∗ Equal contributions.
This raises a question about data efficiency: can we remove a proportion of training examples to reduce the number of calculations during finetuning while preserving accuracy at inference? Since finetuning LLMs is computationally intensive and represents a bottleneck in time and costs of development cycles of NLU applications, such an approach appears promising.
Previous works in computer vision suggest that an important fraction of training examples, up to 50% for some datasets, can be removed without significant loss in accuracy (Toneva et al., 2018; Paul
129
et al., 2021; Raju et al., 2021). Assuming total train steps increase proportionally with the number of samples, this represents a considerable reduction of training costs while being completely compatible with other existing strategies to make finetuning more efficient (Gupta et al., 2015; Micikevicius et al., 2018; Zhang and He, 2020; Zaheer et al., 2020; Xiong et al., 2021). The core idea is to score each training example (xi, yi) with an importance metric, and to remove a proportion of the samples based on ranked scores: the most important examples are kept, whereas others are discarded.
In this work, we expand the analysis of such a method to text classification across various datasets. In agreement with Raju et al. (2021), we observe that many examples have low importance when evaluated at early training but with high variability further on. Thus, a single evaluation often fails at predicting their impact on the loss at a later stage. To address this problem, we also adopt a curriculum in which pruning is performed periodically at a fixed number of epochs. The model is first trained with all examples for a few epochs before ranking is performed with respect to metric χ̂. Then, a proportion ρ of the samples with the lowest scores is set aside, and the model is trained with the retained subset for a cycle of T epochs. All examples are re-scored at every cycle’s end to obtain an updated subset. New cycles are computed until training is complete. Since χ̂ is cheap to compute, the overhead of periodic calculations for a new pruned subset remains low. Figure 1 provides an overview of our proposed method.
Our contributions are as follows:
• We investigate the effectiveness of data pruning methods applied to text classification in reducing finetuning time while maintaining accuracy, including experimental results on six datasets from the GLUE benchmark and four joint NLU datasets.
• We improve the dynamic pruning method with a key initial training phase on the full train set and novel application of the EL2N score in a dynamic setting.
• We extend the EL2N importance score to multioutput text classifications supported by mathematical derivations, specifically for joint intent classification and slot filling.
• We release an open source version of our imple-
mentation1 built around the transformers library (Wolf et al., 2020).

heading: 2 Related Work
text: Previous works have investigated data pruning methods based on metrics calculated from training characteristics. Toneva et al. (2018) propose to score examples based on their number of transitions from correct to incorrect classification over the course of training. They observed that training accuracy is not considerably affected by removing a significant proportion of the rarely forgotten training examples. Nevertheless, this method is not directly applicable to efficiency improvements since evaluating forgetting scores requires a full round of finetuning. Coleman et al. (2020) address this issue by conducting data selection using a small proxy model. However, this involves supporting an additional model, which makes the process more complex. Paul et al. (2021) derives GraNd and EL2N scores as approximations to sample importance (Alain et al., 2015; Katharopoulos and Fleuret, 2018). They observe that expectation over 10 initializations improves metric quality when evaluated early in training. Although this represents a considerable computational cost, they show that they can prune large proportions of lowimportance samples without test accuracy reductions. Others have applied GraNd/EL2N scores to text datasets, notably in natural language inference (Fayyaz et al., 2022) and to reduce gender bias (Zayed et al., 2022). Sorscher et al. (2022) criticize previous work for experimenting only on two computer vision datasets: CIFAR-10 and CIFAR-100. Other metrics have been proposed (Sorscher et al., 2022; Mindermann et al., 2022; Yang et al., 2022), but they all involve further computations.
Raju et al. (2021) and Swayamdipta et al. (2020) observe that training dynamically separates examples in three categories: easy-to-learn, hard-tolearn and ambiguous. Ambiguous samples are characterized by higher variability across training, and an early evaluation of their importance metric can incorrectly predict late training behaviour. To address this, a curriculum is proposed to train on dynamically selected subsets, leading to better performance than EL2N-based static pruning and random subset selection at high pruning rates.
Other related topics also include coreset selection (Har-Peled and Kushal, 2005; Munteanu et al.,
1github.com/jpcorbeil-nuance/nlu_data_diets
2018; Campbell and Broderick, 2018), active learning (Ash et al., 2019; Kim, 2020; Park et al.), dataset size reduction (Mishra and Sachdeva, 2020) and curriculum learning (Bengio et al., 2009; Kumar et al., 2010; Platanios et al., 2019; Wan et al., 2020; Christopoulou et al., 2022).

heading: 3 Methods

heading: 3.1 Estimating Training Example Importance
text: Importance refers to the degree to which a particular training example affects the model’s performance at learning to correctly classify other examples, as formally defined in Paul et al. (2021). A low-importance sample has little impact on training and can be removed without much effect.
We consider supervised classification, with training set S composed of training examples (xi, yi) with i ∈ [1, N ] and N ∈ N, drawn i.i.d. from an unknown data distribution. Here, xi ∈ Rd are the input vectors of dimension d ∈ N and yi ∈ {0, 1}K are one-hot labels with K classes. Model f , composed of weights w ∈ W ⊆ RD with D ∈ N, produces probability vector p = f(xi) ∈ [0, 1]K , and is trained to estimate yi using the cross-entropy loss:
Li = −yTi · log(f(xi)). (1) Paul et al. (2021) showed that the norm of the gradient of Li is an upper bound to the importance of sample (xi, yi). They also introduced a computationally more efficient approximation known as the EL2N score:
χ(xi, yi) = E||f(xi)− yi||2, (2) where the expectation is taken over the randomly
initialized weights w of model f .

heading: 3.1.1 Extending EL2N to NLU
text: For the joint NLU task, each training example has sequence-level intent labels yintenti and M ∈ N word-level slot labels ysloti,m with m ∈ {1,M}. The loss for sample i is expressed as:
Lnlui = Lintenti + M∑
m=1
Lsloti,m , (3)
where intent and slot losses are calculated from eq. 1 with yintenti and y slot i,m respectively.
In order to estimate joint NLU sample importance, we extend the EL2N formulation to a such case in Appendix A. We provide a derivation for the
EL2N score in three contexts: EL2N for sequencelevel classification:
χ̂intent(xi, yi) = ||f(xi)− yintenti ||2, (4)
EL2N for multiple-output classifications, which introduces a second ℓ2 norm over the sequence length:
χ̂slot(xi, yi) =
√√√√ M∑
m=1
||f(xi)− ysloti,m ||22, (5)
and for joint tasks, e.g. joint intent detection and slot filling: χ̂nlu(xi, yi) = √ χ̂intent(xi, yi)2 + χ̂slot(xi, yi)2.
(6)

heading: 3.1.2 Averaging χ̂ Over Time
text: Averaging EL2N over multiple initializations improves importance approximation, but increases compute costs. Instead, following Raju et al. (2021), we consider an exponential moving average (EMA) over time to evaluate χ̂:
χ̂ema(x, y)←α · χ̂nlu(x, y) +(1− α) · χ̂ema(x, y)
(7)
This is possible since dynamic pruning is performed periodically over multiple cycles. Because importance estimation for some examples varies significantly across train steps, EMA provides an adequate trade-off between emphasizing the current estimate and leveraging previous evaluations.

heading: 3.2 Dynamic Pruning
text: With dynamic pruning, the model is trained for τ epochs, then, we train for cycles of T epochs until training is complete. At the beginning of each cycle, we score all training examples with eq. 7, and the proportion with the highest scores is selected. In algorithm 1, we present pseudo-code for the proposed method, where S′ is the retained training subset and E is the number of training epochs.

heading: 4 Experimental Details

heading: 4.1 Experimental Setup
text: In our experiments, we compare five methods:
• Full Train Set: baseline approach with standard training using the full train set.
Algorithm 1: Dynamic Pruning Inputs: S, f , ρ, T , τ , E S′ ← {}; χ̂ema ← {}; cycle← 1; for epoch in τ do
train f with S; end while cycle < ⌊(E−τ)/T ⌋ do
χ̂ema ← αχ̂nlu + (1− α)χ̂ema; sort S with respect to χ̂ema, ↓ order; S′ ← top (1− ρ) elements from S; cycle← cycle+ 1; for epoch in T do
train f with S′; end
end
• Static Pruning: method from Paul et al. (2021). The training subset is obtained from EL2N scores averaged over 10 randomly initialized models trained for 10 epochs, and finetuning is then performed on the fixed subset.
• Single Pruning: training is performed on the full trainset for τ epochs, followed by a single subset selection using EL2N. Finetuning is then performed on the fixed subset.
• Dynamic Pruning: proposed method which trains on the full trainset for τ epochs, followed by periodic data pruning using χ̂ema as described by algorithm 1.
• Dynamic Random Pruning: same as dynamic pruning, but the subset selection is random.
For the full train set method, we finetune the model for E epochs. We assume that E and τ correspond to constant numbers of training steps defined from the full training set size. For the other four methods, all experiments are performed over E epochs, to which we remove a number of train steps equivalent to ρ · (E − τ). We do not report results for the full train set method with reduced training steps as they are comparable to those of the dynamic random pruning method. The number of epochs E is set to 10 for GLUE following Liu et al. (2019) and to 40 for the joint NLU tasks following Chen et al. (2019).

heading: 4.2 Datasets
text: We present two series of experiments to address text classification problems with either a single output or multiple outputs. For the single output case, we consider six datasets from the GLUE benchmark (Wang et al., 2019): COLA, MNLI, MRPC, QQP, RTE, and SST2. We exclude STS-B since it is a regression task, WNLI because it has a very small train set, and QNLI as it is an artificially built dataset.
For the multi-output case, we use four opensource joint NLU datasets: ATIS (Price, 1990; Tur et al., 2010), SNIPS (Coucke et al., 2018), SLURP (Bastianelli et al., 2020) and MTOP (Li et al., 2021) — English version only. All examples from these datasets are labelled with an intent (sequence-level class) and entity tags (token-level class). ATIS focuses on the airline travel domain, and the other three datasets represent voice assistants across various domains, such as music, weather, and calendar, among others. We use the original versions of these datasets without any modification.
Tables 2 and 3 in Appendix D provide the general characteristics of these datasets. Overall, they cover a large diversity of train set sizes and tasks. For the joint NLU tasks, we have a variety of domains, intents and entities.

heading: 4.3 Model Architecture
text: We use the pre-trained transformer model RoBERTa base (Liu et al., 2019). For the GLUE datasets, we use single sequences or concatenations of two sequences as model inputs for paraphrase detection and natural language inference (NLI) tasks. For the joint NLU tasks, we use the JointBERT approach (Chen et al., 2019). To handle entities, we apply a mask on special tokens as well as subsequent subwords if a word is decomposed into multiple subword tokens. Additionally, we reinitialize the last layer of the encoder (Kovaleva et al., 2019; Tamkin et al., 2020), which can improve the final performance.

heading: 4.4 Accuracy Evaluation
text: We evaluate the performance of our models on the GLUE datasets following the original instructions. Since the test sets are not publicly available, we report the averaged results on the development sets following previous approaches from (Devlin et al., 2019; Liu et al., 2019). Specifically, we report the accuracy metric for RTE and SST2, accuracy on the
matched and mismatched test set splits for MNLI, and both accuracy and F-1 metrics for MRPC and QQP. For COLA, we compute the Matthews correlation coefficient. For the joint NLU tasks, we focus on full-sequence accuracy, which requires matching both the intent and all the entities. We also provide the intent accuracy and entity micro F1 in Appendix F.

heading: 4.5 Runtime Evaluation
text: We report GPU finetuning time in minutes or seconds. To avoid the impact of CPU overhead, such as pre-processing and batching, we measure the time for each training step and forward pass for scoring EL2N separately, and then sum the results at the end of the fine-tuning process. We provide the mathematical expressions for the total GPU runtime and the minimum cycle Tmin to improve efficiency in Appendix B. Finally, we used 10 GPUs to parallelize the computation of EL2N scores with static pruning, as this method requires training 10 independent models.

heading: 5 Results

heading: 5.1 GLUE Benchmark
text: In Table 1, we present the results obtained on the GLUE benchmark with a 50% pruning rate. Overall, dynamic pruning achieves accuracy nearly on par with full training while reducing the finetuning time by approximately 30%. The largest accuracy decrease is observed on the COLA and MRPC datasets, at only 1% absolute. We also note that the smallest time reduction occurs on RTE and is due
to the different pruning settings required by this dataset of smaller size. We observe a sharper drop in performance with single pruning and dynamic random pruning, despite being faster. Static pruning underperforms in terms of both accuracy and finetuning time. This is especially the case on the RTE dataset and is likely due to the combination of a high pruning rate and a smaller number of training examples. This suggests that dynamic pruning may be affected at lower data amounts, as reported for static pruning by Sorscher et al. (2022).

heading: 5.2 Joint Intent Classification and Slot Filling
text: In Figures 2 and 3, we present the full-sequence accuracy and GPU finetuning time obtained with the studied methods on the joint NLU tasks. In Appendix F, we also include the related intent accuracy and slot micro F1 score.

heading: 5.2.1 Accuracy Analysis
text: Overall, our dynamic pruning method achieves accuracy comparable to full training, even when using aggressive pruning rates. It exhibits a flatter and smoother decrease compared to other techniques, which results in a better time-accuracy trade-off. For instance, at a pruning rate of 40%, our approach leads to similar scores as the full-training baseline while being about 25% faster. Other methods underperform by 0.5 to 2.5% in the same setting, excluding the ATIS dataset for which all methods perform above baseline. Single pruning outperforms static pruning in almost every setting, despite static pruning leveraging EL2N scores calculated from multiple models. This emphasizes the importance
of the initial training phase on the full dataset. Surprisingly, dynamic random pruning performs better than static and single pruning methods. This is further discussed in section 6.
At an aggressive pruning rate of 80%, dynamic pruning only incurs a marginal drop of 1-2%, with
a total finetuning time of nearly a third of the full training. In comparison, dynamic random pruning achieves a few absolute percentage points less than dynamic pruning, and static pruning decreases even faster. Single pruning shows a similar decrease, except on the ATIS dataset. At an extreme prune rate of 90%, dynamic random pruning performs better than dynamic pruning for two datasets: SLURP and MTOP. This is because these datasets have many mislabeled data points, usually associated with high-importance scores, as discussed in section 6. We also notice a significant decrease in accuracy from the static and single pruning methods for the same reason. Regarding domains, number of intents, and number of entities, SLURP and MTOP are the most complex.
Results on the ATIS dataset show a different trend, especially at a low prune rate. We observe an improvement over the full training set method for single, static and dynamic pruning, which perform equally well. ATIS is a smaller dataset with simple, repetitive samples on a single domain and consequently contains a larger proportion of easyto-learn examples (see Section 6). Even at moderate prune rates, the composition of the pruned subset remains stable over time, explaining the adequate performance of the static method. At higher prune rates, the accuracy of static pruning drops abruptly.
Figures 7 and 8 of Appendix F show that our
dynamic pruning method preserves the full-trainset intent accuracy and slot micro F1 score with prune rates up to 80%.

heading: 5.2.2 Finetuning Time Analysis
text: Regarding finetuning time, we empirically observe the linearly decreasing relationship with ρ as described by equation 31 in Appendix B. Dynamic random pruning is faster than dynamic pruning since it does not require the periodic forward passes needed for calculating χ̂ema. However, we can observe from this difference that the overhead is relatively small compared to the whole training process. Single pruning, which only requires a single computation of the importance score, is faster than dynamic pruning. In contrast, static pruning is significantly slower due to the overhead from the initial 10 runs of 10 epochs, even if computations are parallelized across ten GPUs.

heading: 5.3 Balancing Accuracy and Efficiency
text: In this section, we present an analysis of the timeaccuracy trade-off for our dynamic data pruning method in relation to T and τ on the MTOP dataset, with a fixed prune rate of 70%. Similar trends are observed with the other datasets, as shown in Appendix F. The results are illustrated in Figures 4 and 5.
We found that training on larger τ and considering shorter T results in higher accuracy. We note that τ is key to our approach since it impacts accuracy the most. For instance, a τ of 2 is less effective than 16, while the difference between 4 and 8 is marginal. Despite pruning 70% of the data points, we achieved similar results to full training with T = 2 and τ = 16. Considering that we are at a high prune rate, we can achieve a better time-accuracy trade-off at T = 4 and τ = 4. This results in a marginal sacrifice of 1.2% for twice
the reduction in finetuning time. Specifically, the finetuning time is reduced from 826 s (−36%) to 533 s (−59%) compared to full training at 1295 s.
Furthermore, we observe the inverse relationship between the finetuning time and T in Figure 5 as described by equation 31 in Appendix B. For small T , this inverse relationship significantly increases the computational time. For instance, the difference between T = 1 and T = 2 is nearly 2 minutes, while being negligible between 8 and 16.
Due to periodical re-evaluation of the pruned subset, we note that performance obtained with τ , ρ, and T is less sensible to dataset characteristics, and the time-accuracy trade-off it provides remains robust across various settings. We provide additional recommendations about selecting these parameters in Appendix E.

heading: 6 Data Selection Analysis
text: We analyze pruning dynamics for joint intent classification and slot filling. The analysis is presented for SLURP, but observations are similar to other datasets.
First, we present a data map in Figure 6a, that shows variance and mean of χ̂nlu over 10 cycles of T = 4 epochs for all training examples. The pruning proportion is set to 50%. We discern three regions delimited by dashed curves: the bottomleft corner (low variance, low mean) corresponds to easy-to-learn examples consistently pruned from the train set, the top (high mean) corresponds to hard-to-learn examples consistently retained, and examples in-between (mid/high variance, low/mid mean) are ambiguous.
Figure 6b shows the distribution for the number of times a given example is selected in the retained subset over ten pruning cycles. For SLURP, we observe that about a third of the examples are con-
sistently either always selected or never selected, corresponding to hard-to-learn and easy-to-learn cases, respectively. Other examples are approximately evenly distributed across selection counts. In comparison, we also present the distribution for the ATIS dataset, for which most training examples are either never selected or always selected.
Figure 6c presents scoring metrics averaged over the full train set for intent and slot classifications with respect to the training epoch. On average, sequences have one intent classification term per 6.9 slot classification terms. Overall, contribution to χ̂nlu is comparable between intent and slots, meaning that individual score is typically smaller for slots. However, scores associated with the prevalent no_entity class are relatively low compared to other slot labels, which have a predominant contribution to χ̂slots.
In Table 5 of Appendix F, we present 13 examples of easy-to-learn, hard-to-learn and ambiguous samples, with their score averaged over time and the number of selections with four different initializations. Examples with the highest score typically have mislabeled intent (1) or slot (2), or have an
infrequent class label (3, 0.03% of all samples), or are unique (4). Examples with lowest χ̂nlu generally have short utterances with no entity label and with highly represented intent labels having many redundant tokens across class examples (see 5-7). Note that Spearman correlation between χ̂nlu and the number of words (r = 0.096), number of entities (r = 0.205) and intent class prevalence (r = −0.109) remains low when considering full train set.
Ambiguous examples, however, are more difficult to characterize qualitatively. We present several examples with an increasing scoring metric which are all under the same intent label for comparison purposes (see examples 8-13). There is no clear distinction between samples. In fact, when considering different initializations, the same ambiguous sample obtains different selection counts in the retained subset. This shows the necessity of using dynamic pruning: a static pruning algorithm will target easy-to-learn samples but will suffer from the variability of ambiguous samples. This high variability also explains the unexpectedly high performance of random pruning. Dynamic pruning can adjust the selection of ambiguous samples during training based on the model’s performance, which leads to more efficient training.
When the proportion of ambiguous examples is low, as with the ATIS dataset, single and static pruning outperform random dynamic pruning, as shown in Figure 2, since the pruning subsets remain consistent across training.

heading: 7 Conclusion
text: In summary, this work presents a dynamic approach to data pruning to improve finetuning efficiency. It introduces an extension of the EL2N metric for multi-output classifications. We use this metric to periodically select a subset of the data during training in contrast to static pruning. We empirically show that our method provides a better trade-off between accuracy and efficiency, especially at high prune rates. We also show that this trade-off benefits from initial training with the full train set for a few epochs.
We apply our method to NLP classification datasets, particularly six tasks from the GLUE benchmark and four datasets for joint intent classification and slot filling. By pruning 50% of training examples, we preserve full accuracy on the test set while reducing the finetuning time by up to 41%. If
we tolerate an accuracy reduction of 1% absolute, we can prune 80% of the training data, corresponding to time reductions of up to 66%. Finally, we provide insights into how performance is affected by the characteristics of the training set.
As future work, we envision further investigations including more natural language tasks, more challenging datasets (e.g. SuperGLUE) and improved importance approximations. Overall, dynamical evaluation of sample importance remains largely unexplored beyond improving efficiency, notably in data augmentation, active learning, mislabel detection or pre-training data selection.
Limitations
This study is limited to classification tasks with an encoder architecture, short texts (e.g. utterances), datasets with at least several thousand examples, and a relatively low amount of mislabeled data. In theory, we could apply our method to longer texts, but our takeaways might not directly apply. For similar reasons, our conclusions could be different on very small datasets. Our approach is also sensible to mislabeled data points, but this weakness is mitigated by the fact that our method can contribute to improving the identification of such mislabels. We are also aware of further efficiency optimizations, such as calculating scores directly from mini-batches during training for the retained subset, which we leave to future work.
Ethics Statement
We bring up two main ethical considerations. First, this empirical study uses a large language model requiring a considerable amount of computations (at most 1,500 GPU hours), which is not without environmental consequences. However, since this study aims at making training more efficient, it will help reduce energy consumption in the future. Moreover, this study focuses on accuracy as a measure of performance, which can hide pervasive effects on under-represented marginalized groups. However, since our method is about evaluating importance of training examples over train steps, it can lead to improving techniques to decrease bias in the training process, particularly when marginalized groups are under-represented in the data and therefore challenging to identify accurately.

heading: Acknowledgements
text: We would like to express our gratitude to François Beaulieu, Paul Vozila, Rupert Brook, and Alessandro Sordoni for their helpful comments. We also appreciate the valuable feedback provided by the anonymous reviewers, which contributed to the improvement of our paper.

heading: A Derivations of Pruning Metrics
text: A.1 Unique Single-Label Classification on CLS token
We begin by defining the GraNd score, χt(x, y) ∈ R≥0, at time t > 0 and for any sample data (x, y) as a measure of sample importance (Paul et al., 2021):
χt(x, y) = Ew [||gt(x, y)||2] , (8) where gt is the gradient of the sample (x, y) obtained with model f at time t. For simplicity, we define χ′t(x, y) as the same expression as in Equation (8) but without the expectation term. We compensate for the improvement of this expectation with dynamic pruning by leveraging the exponential moving average:
χ′t(x, y) = ||gt(x, y)||2. (9) Because processing per-sample gradients can be computationally burdensome, we look for an upper
bound. First, we use the result from Katharopoulos and Fleuret (2018), which implies that the full gradient is upper bounded by a constant β ∈ R>0 multiplied by the gradient of the last classification layer. Let Gt(x, y) ∈ RK×d denote the gradient of the last classification layer, where K, d ∈ N (number of classes and hidden dimensions, respectively). We can write this result as
χ′t(x, y) ≤ β||Gt(x, y)||2. (10) Since the last layer is a classification with the cross-entropy loss on a softmax activation, it naturally takes the form of a K × d matrix defined by the outer product of the error vector p⃗(x) − y⃗ and the transposed contextual word embedding h⃗CLS(x) ∈ Rd. For simplification purposes, we do not carry the notation marking the dependence on x for p⃗ and h⃗CLS :
Gt(x, y) = (p⃗− y⃗) h⃗TCLS . (11) By replacing Equation (11) in Equation (10), we
obtain
χ′t(x, y) ≤ β|| (p⃗− y⃗) h⃗TCLS ||2. (12) Using the property of outer products, we can separate the matrix norm into the product of the vector norms:
χ′t(x, y) ≤ β||⃗hCLS ||2||p⃗− y⃗||2. (13) Given that we want to rank samples {(x, y)}1..N and that norm of the contextual word embeddings of h⃗CLS can roughly be assumed constant across samples due to layer normalization, we obtain:
χ′t(x, y) ∝ ||p⃗− y⃗||2 (14) This metric corresponds to the EL2N score proposed by Paul et al. (2021), and provides scoring function for intent classification:
χ̂intent(x, y) = ||p⃗(x)− y⃗||2 (15)
A.2 Sequence of Single-Label Classifications The derivation for multi-output classification is similar to that of single-label classification shown above, with some modifications. In this case, we consider a summation over the sequence length m ∈ [1,M ], where M ∈ N, to update the entity classifier with the sum of the gradients from all tokens. We assume that all sequences are padded up to length M .
To begin, we introduce the sequence-wise error vectors δ⃗m = p⃗m − y⃗m, and rewrite the gradient Gt(x, y) of equation (11) as follows:
Gt(x, y) = M∑
m=1
δ⃗mh⃗ T m, (16)
where h⃗m is the contextual embedding for the m-th token. By substitution in equation (10), we obtain the following upper bound:
χ′t(x, y) ≤ β|| M∑
m=1
δ⃗mh⃗ T m||2. (17)
We then use the triangle inequality to derive a new bound as follows:
χ′t(x, y) ≤ β M∑
m=1
||δ⃗mh⃗Tm||2. (18)
Since the sum over m ∈ [1,M ] can be interpreted as a ℓ1 norm over the sequence, we replace it with a ℓ2 norm leveraging the equivalence of topology of p-norms. This norm is less sensitive to the sequence length and puts more emphasis on the highest norms. This results in a new upper bound:
χ′t(x, y) ≤ β
√√√√M M∑
m=1
||δ⃗mh⃗Tm||22. (19)
This substitution is better suited for our metric for two reasons. First, the square applied to each individual norm of element m reduces the impact of small norms compared to larger ones, resulting in a metric that scales less with the length of the sequence M . Second, it matches the ℓ2 norm used in the previous section, which combines more naturally in the mathematical development of the next section for joint tasks.
We can apply the inner ℓ2 norm separately on the outer product of vectors. Since the amplitude of all contextual word embeddings h⃗m are similar, we can assume ||⃗hm|| ≈ ||⃗hm′ || ≈ ||⃗h|| with m′ ∈ [1,M ] across tokens. Thus, we obtain the following simplified bound:
χ′t(x, y) ≤ β √ M ||⃗h||2
√√√√ M∑
m=1
||δ⃗m||22. (20)
Since we are interested in ranking samples {(x, y)}1..N by scores as before, we can simplify
this bound for multi-output classification tasks such as slot filling. We also replace δ⃗m by its definition.
χ̂slot(x, y) =
√√√√ M∑
m=1
||p⃗m(x)− y⃗m||22 (21)
A.3 Joint Single-Label Classification and Sequence of Single-Label Classification
We aim to combine the gradient matrices of two independent classification heads into one gradient matrix using an ℓ2 norm. We can use the direct sum to express this combination for a joint task, such as joint intent classification (indicated by the int superscript) and slot filling (indicated by the slot superscript). The direct sum of matrices generates a block diagonal matrix that combines two vector spaces. Under a norm, this matrix representation is equivalent to the flattened one.
Let us denote the gradient matrices as G(int,t)(x, yint) and G(slot,t)(x, yslot), where t denotes the training step, x is the input data, and yint and yslot is the corresponding labels. For simplification, we are leaving the dependence on x, yint and yslot on the right side of our equations. The combined gradient matrix is obtained by taking the direct sum of these matrices as follows:
Gt(x, y) = G (int,t) ⊕G(slot,t). (22)
Substituting this equation in (10), we get:
χ′t(x, y) ≤ β||G(int,t) ⊕G(slot,t)||2. (23)
We consider the squared Frobenius norm and apply the sum of square terms block-wise, where the off-diagonal zero terms are ignored. This gives us the sum of the square norms of each gradient matrix as follows:
χ′t(x, y) 2 ≤ β2 ( ||G(int,t)||22 + ||G(slot,t)||22 ) . (24)
Using the definitions of the gradient norms obtained from the previous derivations, we get the following:
χ′t(x, y) 2 ≤ β2||⃗hCLS ||22||p⃗int − y⃗int||22+
β2||⃗h||22 M∑
m=1
||p⃗m − y⃗m||22. (25)
As ||⃗hCLS || ≈ ||⃗h|| and by identification with the two previous derivations, we obtain the final form for the joint task, such as joint intent classification and slot filling, as follows:
χ̂nlu(x, y) = √ χ̂2intent + χ̂ 2 slot. (26)

heading: B Derivations of GPU Runtime Equations
text: B.1 GPU Runtime Dependency to Pruning Factor and Epoch Cycle
The total GPU time T contains three terms as follow
T = Tinit + Tprune + Tscore, (27)
where Tinit is the initial amount of epochs finetuned with the full training set, Tprune is the remaining amount of epochs to finetune with the pruned dataset, and Tscore is the time to compute the importance score.
We define τ ≤ E ∈ N, the epoch at which we start the data pruning and the total amount of epochs, respectively. We have B ∈ N, the number of steps in one epoch (i.e. the ceiling of training set length divided by the batch size). We define an epoch cycle to prune the train data set as well T ∈ N, with T ≤ E − τ . The proportion of pruned samples is defined by ρ ∈ (0, 1). Finally, we have ∆tstep ∈ R≥0, the average time to process one mini-batch, and ∆tforward ∈ R≥0, the time to compute a complete forward pass on the training dataset.
Then, we can explicitly formulate
Tinit = τB∆tstep (28)
Tprune = (E − τ) (1− ρ)B∆tstep (29)
Tscore = ⌊ E − τ T ⌋ ∆tforward, (30)
where ⌊·⌋ is the floor operator. Therefore, the complete GPU runtime equation based on eq. 27 takes the form
T = EB∆tstep − (E − τ)B∆tstepρ
+ ⌊ E − τ T ⌋ ∆tforward
(31)
B.2 Lower-Bound Epoch Cycle We can find Tmin, which is a lower bound to T , to achieve an effective pruning in terms of computational time. We need to satisfy:
T < Tbaseline (32) Given the previous definitions, we can find
Tbaseline = EB∆tstep (33) Considering eq. 33 and 31 for T , we deduce
Tmin > ∆tforward ∆tstepBρ
(34)

heading: C Training Setup
text: We use the following HuggingFace libraries distributed under the Apache License 2.0: transformers v4.21.3 (Wolf et al., 2020) and datasets v2.5.1 (Lhoest et al., 2021). We set our back-end to Pytorch v1.10.1 (Paszke et al., 2017, 2019). We used the Adam optimizer (β1 = 0.9, β2 = 0.999 and ϵ = 1 × 10−8) with no warm up and no scheduler. We run fine-tunings and evaluations on an on-premise cluster of NVIDIA V100 32 Gb GPUs. All the results are the average of 5 runs with different random seeds unless it is stated otherwise.
We run our fine-tuning on the GLUE datasets with the following hyper-parameters: number of epochs of 10, a learning rate of 2× 10−5, a batch size of 32 and a maximum sequence length of 128 (except for the RTE dataset at 256). For the joint intent and slot filling datasets, we set: number of epochs of 40, a learning rate of 2× 10−5, a batch size of 32, a maximum sequence length of 50 and λ = 0.5. For χ̂ema calculations, we use α = 0.8.

heading: D Dataset Statistics
text: In Tables 2 and 3, we provide the statistics about the GLUE datasets and the joint NLU datasets. We verify that all samples in the datasets do not contain offensive language and personal information.

heading: E Parameter Selection
text: Values for τ , ρ, and T must be selected to achieve the desired time-accuracy trade-off. A valid first step is to set τ at about 10% of E for datasets with more than a few thousand samples or 20-40% for smaller datasets to achieve higher accuracy. Larger τ significantly hinders efficiency at the expense of accuracy. Then, we calculate Tmin as a lower bound to obtain an efficiency gain with a given prune rate ρ using equation 34. We recommend ρ ∈ [0.4, 0.7] depending if accuracy or efficiency is more important, respectively. Finally, we want to set T such that it is above Tmin to be efficient but small enough to provide many pruning updates for higher accuracy — about ten cycles calculated with⌊ E−τ T ⌋ being a valid target. In Table 4 of Appendix F, we present values of Tmin for ρ ∈ {0.1, 0.5} for all our datasets, along with the forward pass time ∆tf , train step time ∆tstep and the number of steps per epoch B.

heading: F Further Results
text: In Table 4, we show Tmin (i.e. the minimum cycle to achieve an efficiency improvement over the baseline) for ρ ∈ {0.1, 0.5} for all datasets along their ∆tforward and ∆tstep.

heading: U ATIS 3.7 0.065 156 3.6 0.7
text: racy and the slot micro F1 score for the joint intent classification and slot filling tasks as complementary results to sub-section 5.2.
In Figures 9 and 10, we present the extensive results on all joint NLU datasets for the experiments on the prune epoch cycle T and prune epoch τ .
In table 5, we present representative examples of hard-to-learn, easy-to-learn and ambiguous samples from fine-tunings on the SLURP dataset, as discussed in section 6.

heading: Future Work
text: As future work, we envision further investigations including more natural language tasks, more challenging datasets (e. g. SuperGLUE) and improved importance approximations. Overall, dynamical evaluation of sample importance remains largely unexplored beyond improving efficiency, notably in data augmentation, active learning, mislabel detection or pre-training data selection. Limitations
This study is limited to classification tasks with an encoder architecture, short texts (e. g. utterances), datasets with at least several thousand examples, and a relatively low amount of mislabeled data. In theory, we could apply our method to longer texts, but our takeaways might not directly apply