Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning (2024)

Abstract: Neural Machine Translation models are extremely data and compute-hungry. However, not all data points contribute equally to model training and generalization. Data pruning to remove the low-value data points has the benefit of drastically reducing the compute budget without a significant drop in model performance. In this paper, we propose a new data pruning technique: Checkpoints Across Time (CAT), that leverages early model training dynamics to identify the most relevant data points for model performance. We benchmark CAT against several data pruning techniques including COMET-QE, LASER and LaBSE. We find that CAT outperforms the benchmarks on Indo-European languages on multiple test sets. When applied to English-German, EnglishFrench and English-Swahili translation tasks, CAT achieves comparable performance to using the full dataset, while pruning up to 50% of training data. We inspect the data points that CAT selects and find that it tends to favor longer sentences and sentences with unique or rare words.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 9407–9426 August 11-16, 2024 ©2024 Association for Computational Linguistics
Neural Machine Translation models are extremely data and compute-hungry. However, not all data points contribute equally to model training and generalization. Data pruning to remove the low-value data points has the benefit of drastically reducing the compute budget without a significant drop in model performance. In this paper, we propose a new data pruning technique: Checkpoints Across Time (CAT), that leverages early model training dynamics to identify the most relevant data points for model performance. We benchmark CAT against several data pruning techniques including COMET-QE, LASER and LaBSE. We find that CAT outperforms the benchmarks on Indo-European languages on multiple test sets. When applied to English-German, EnglishFrench and English-Swahili translation tasks, CAT achieves comparable performance to using the full dataset, while pruning up to 50% of training data. We inspect the data points that CAT selects and find that it tends to favor longer sentences and sentences with unique or rare words.

heading: 1 Introduction
text: In recent years, there has been significant improvement in the quality of Neural Machine Translation (NMT) models (Johnson et al., 2017; Arivazhagan et al., 2019; Liu et al., 2020; Fan et al., 2021; Reid et al., 2021; Ramesh et al., 2022; Costa-jussà et al., 2022; Bapna et al., 2022; Gala et al., 2023). The success of these models is primarily due to many factors including pretraining data size, compute abundance and the ever-increasing model size. Despite the gains due to growing datasets, large dataset sizes have also posed significant hurdles for maintaining data quality. Significant portions of web-scraped data used for language model pretraining have been shown to be of low quality, machinegenerated spam, pornographic content (Kreutzer
et al., 2022). Given the ever-growing size of parallel corpora, it often becomes laborious for humans to assess the quality at scale (Freitag et al., 2021; Agarwal et al., 2022; Longpre et al., 2023). Quality of pretraining data for large NMT models often faces amplified quality issues because of extensive bitext mining approaches to automatically extracting translation pairs from either monolingual corpora (Guo et al., 2018; Schwenk et al., 2021a,b; Costa-jussà et al., 2022; Ramesh et al., 2022; Vegi et al., 2022; Gala et al., 2023) or document-aligned corpora (Bañón et al., 2020; Steingrímsson et al., 2021; Ramesh et al., 2022; Gala et al., 2023). Bitext mining approaches, although far more costeffective than human curation, have been shown to introduce significant noise into datasets by relying on sub-optimal sentence embedding models (Thompson and Koehn, 2019; Feng et al., 2022; Heffernan et al., 2022) to match sub-optimal translation pairs (Kreutzer et al., 2022).
Recent work has shown that quality matters. Large neural models trained on high-quality data outperform models trained on noisy data (Khayrallah and Koehn, 2018; Arora et al., 2021; Abdulmumin et al., 2022; Gala et al., 2023) or match performance using far fewer data points (Sorscher et al., 2022; Siddiqui et al., 2023; Koneru et al., 2022; Li et al., 2018; Xu et al., 2023). In this work, we ask if we can arrive at a rigorous estimator of data quality through data pruning. Our goal is to focus on metrics which are scalable across large datasets, and which identify a subset that preserves performance.
We leverage distinctive periods of training to indicate examples most critical to model generalization. Specifically, we propose Checkpoints Across Time (CAT): a metric that leverages the variability in perplexity across early checkpoints to identify such critical data points when training MT models. We use this to rank data points, selecting the instances with the highest variability and continuing
9407
to train the model solely on this subset. This scoring is used to remove the data points estimated to be least important, thus creating gains in efficiency for training.
CAT is motivated by fundamental work in machine learning which has shown that there are distinctive periods to learning in deep neural network training. Prior work shows that more common, easier features are learned earlier in training, with the most challenging features learned in the last stages of training (Jiang et al., 2020; Achille et al., 2017; Siddiqui et al., 2023). Prior work has also shown that variance in gradients with respect to inputs (Agarwal et al., 2022) or gradient patterns themselves vary depending on the period of learning (Faghri et al., 2020). However, gradients remain expensive to leverage as a learning signal to identify relevant instances. Furthermore, almost the entirety of this work has focused on a computer vision setting, with the exception of Swayamdipta et al. (2020) which uses differences in the model’s confidence in the true class, and the variability of this confidence across epoch to cluster data. Here – we explore applying the insights that there are distinctive periods of training to an NLP setting and leveraging for the purpose of pruning. We propose a more efficient method to compute signal relative to prior methods which have required computing gradients instead. Our hypothesis is that those with the largest variability indicate the easy examples which show the quickest learning and gains in certainty early in training.
Compared to existing data pruning strategies in NMT, e.g. the popular (dual) cross-entropy method (Axelrod et al., 2011; Junczys-Dowmunt, 2018) and contrastive data selection (Wang et al., 2018), our approach doesn’t require an initial clean dataset to train on as this is one limitation that often prevents the adoption of such methods to low-resourced languages. Additionally, it does not require significant time to run as we rely only on two early checkpoints rather than a training run over multiple epochs. This makes CAT methods suitable for low-compute and data-scarce environments which are usually co-occurring (Ahia et al., 2021). Additionally, it eliminates dependency on other models that may not capture information from various languages and domains.
We study the effectiveness of CAT across diverse linguistic contexts. Specifically, we utilize English-German and English-French pairs from the Indo-European family using WMT datasets,
and the English-Swahili pair from the Bantu family with automatically aligned datasets. Through a series of extensive experiments, we show that CAT results in significant improvements in translation quality over random pruning (randomly selecting X% subset of the training data) and other embedding-based quality estimation methods such as LaBSE (Feng et al., 2022). We summarize our key contributions below:
1. Focusing on Machine Translation, we perform an empirical evaluation of existing quality estimation techniques and embedding models as a means of data pruning. These techniques were not designed for data pruning however we observe that quality estimation techniques are less effective for pruning high-quality MT datasets compared to embedding models used for automatic alignment.
2. We propose a novel method, Checkpoints Across Time (CAT), that leverages perplexity variation to prune MT datasets with significant improvements over existing methods for Indo-European languages. We show that CAT results in significant improvements in translation quality over random pruning (randomly selecting X% subset of the training data) and other embedding-based quality estimation methods such as LaBSE (Feng et al., 2022). Using our best CAT-based method on EnglishGerman, English-French and English-Swahili translation tasks, we are able to achieve comparable performance while pruning up to 50% of the training data.
3. We complement our findings with extensive analysis, studying the relationships between performance metrics and sentence characteristics such as sentence length in MT.

heading: 2 Background

heading: 2.1 Data Pruning in ML
text: Data pruning has been extensively studied in the work of (Marion et al., 2023; Li et al., 2018; Raju et al., 2021; Swayamdipta et al., 2020; Boubdir et al., 2023). Li et al. (2018); Sorscher et al. (2022) and Raju et al. (2021) focus on data pruning for computer vision whereas Swayamdipta et al. (2020) focuses on data pruning for language tasks. Various techniques have been proposed for data pruning, such as utilization of loss profile (Siddiqui
et al., 2023), gradients confidence score (Agarwal et al., 2022), and data classification using a model (Swayamdipta et al., 2020). Increasingly techniques have been extended from primarily computer vision settings (Sorscher et al., 2022) to more recently expanding the treatment of data pruning language settings (Marion et al., 2023). These methods quantify the importance of data points to model generalization. Our work follows along these lines of identifying critical data points by leveraging the difference in perplexity between early checkpoints.

heading: 2.2 Model-based Data Pruning for NMT
text: Specifically for the task of NMT, model-based data pruning approaches have been most popular and originate from the cross-entropy scoring method, proposed by Moore and Lewis (2010). Data points are ranked either based on their perplexity under an in-domain language model (LM), or based on the difference of this in-domain cross-entropy and the cross-entropy of a general-domain LM. Axelrod et al. (2011) first apply this technique for MT data selection with a bilingual extension with LM cross-entropy scores for source and target sides. Duh et al. (2013) use early neural LMs instead of ngram LMs, and Junczys-Dowmunt (2018) replace LMs with NMT models trained in both directions. Zhang et al. (2020) propose to use scores from pretrained large language models like GPT for domain filtering instead of custom-trained models. van der Wees et al. (2017) make the data selection dynamic, by gradually fine-tuning an NMT model on newly selected data points every epoch. While the focus of model-based data selection was originally on in-domain data selection, it was later extended to generally filtering noisy data sources. Wang et al. (2018) propose an online data denoising approach, measuring noise through differences in log probabilities between noisy and denoised NMT models. Most recently, MT quality estimation (QE) models have also been leveraged for corpus mining and filtering (Kocyigit et al., 2022; Batheja and Bhattacharyya, 2023). Chimoto and Bassett (2022) show that COMET-QE (Rei et al., 2020) improve over random data selection in active learning MT settings.
All these approaches rely on the existence of a small trusted data set or trusted models to extrinsically define what “good data” looks like. Our approach, while also using model perplexity as a metric, however, does not require such pre-selection,
and focuses on model training dynamics instead. Thereby, it can be applied more broadly. Our experiments aim at this less constrained setting without any given high-quality data, so we compare against model-based selection with an MT-QE model and a multilingual LLM.

heading: 2.3 Embedding-based Data Pruning for NMT
text: Embedding-based similarity metrics have recently become popular for NMT data selection, as they rely on unsupervised learning and intrinsic notions of quality, assuming that high-quality parallel sentences have high similarity in a cross-lingual representation space (Schwenk, 2018). Few examples of language-agnostic sentence embedders include MUSE (Lample et al., 2018), XLM-R (Conneau et al., 2020), LaBSE (Feng et al., 2022) and LASER (Heffernan et al., 2022). Schwenk et al. (2021a,b); Costa-jussà et al. (2022) utilize the LASER sentence embedder for bitext mining, whereas Ramesh et al. (2022); Gala et al. (2023) use the LaBSE sentence embedder for the same purpose. In our experiments, we compare against data pruning with LASER and LaBSE embeddings.

heading: 2.4 Comparison Studies
text: With this plethora of filtering methods and application scenarios, there have been few works to systematically compare them. The WMT shared tasks for corpus filtering (Koehn et al., 2018, 2019, 2020) have shed light on how different languages and resourcefulness conditions pose different challenges for above-described filtering methods in practice, and how they affect downstream NMT performance. Bane and Zaretskaya (2021) compare data filtering techniques with a more qualitative angle: They find that NMT scores and MUSE embeddings have the highest correlation with human quality judgments, NMT scores work best for in-domain selection, and MUSE/XLMR for out-of-domain generalization. Herold et al. (2022) compare multiple data noise detection techniques according to their effectiveness for filtering out specific types of data noise. They find that LASER does not detect incorrect languages, and both LASER and cross-entropy perform weakly on detecting misaligned sentences and over/under translation. Similarly, Bane et al. (2022) find crossentropy filtering empirically superior to other methods (XLM-R, MUSE, LASER, COMET) for filtering out various synthetic noise types (Khayrallah and Koehn, 2018). In their setup, COMET fails at
filtering out misaligned sentences but shows particular sensitivity for target-side omissions, NMT scoring for source-side omissions, while LASER and COMET do not filter mismatching numbers well. Dakwale et al. (2022) compare LASER and LaBSE on low-resource corpus filtering tasks and find them competitive if languages are included in the pretraining of the underlying embedding models.
In this work, we provide both a systematic empirical comparison of different types of data pruning techniques on downstream NMT performance for three target languages, as well as qualitative analyses of what kind of data is getting selected.
3 CAT: Checkpoints Across Time
Our goal is to extract the most valuable data points that would contribute to model generalization and train the model solely on this. Our approach, while simple, leverages the difference in learning stages in deep neural network optimization. Firstly, we train an NMT model on the full dataset for only a few epochs, ideally a small fraction of the total time needed for the model to converge. Subsequently, we use the checkpoints from this initial stage to compute the perplexity of each data point. These perplexity scores serve as the basis for selecting candidate data points for subsequent pruning using different perplexity profiles. This approach draws upon the findings of Swayamdipta et al. (2020) that categorize training data relevance based on the learning dynamics during training, and Agarwal et al. (2022) which leverage the difference in gradients across training to identify easy versus challenging examples. Unlike Agarwal et al. (2022) which uses computationally expensive variance of gradients, we use perplexity. For CAT we can utilize the perplexity scores in two ways:

heading: 3.1 CAT-DIFF
text: Let X = {x1, x2, x3, ..., xn} be all instances in the training data. To select s ⊆ X instances to be pruned, we compute the perplexity of each instance at epoch j of all data points in X using the model trained on X with weight parameters θj :
PPLj(xi) = exp ( − 1 T T∑
t=1
logPθj (xi,t)
) (1)
where T is the number of words in each example xi.
We then rank all data points based on differences in perplexity between the two epochs that form our checkpoints:
∆PPL(xi) = PPL1(xi)− PPLk(xi) (2)
From the ranked data points, we select those with the highest differences between early epochs,1 according to a threshold at a chosen percentile. For example, for pruning of 50% of data points, we keep all data points that rank higher than the 50th percentile.

heading: 3.2 CAT-VAR
text: An alternative is to compute the perplexity at N > 1 epoch checkpoints, but to rank data points according to the variance of the perplexity across the checkpoints:
Var(PPL(xi)) =
1
N
N∑
j=1
( PPLj(xi)− 1
N
N∑
l=1
PPLl(xl)
)2 (3)
In our experiments, we use N = 3 corresponding to epochs 1, 3 and 5.
In contrast to CAT-DIFF, we then select data points around the median with a range according to the pruning percentage that we want to achieve, e.g. data points between the 25th and 75th percentile for a pruning percentage of 50%. This aligns with several works which have found variance in model signal to be predictive of difficulty (Swayamdipta et al., 2020; Agarwal et al., 2022). Swayamdipta et al. (2020) show that data points that fall on the tails of the variance tend to be either too easy or too difficult and thus the model may not benefit most from being trained on such examples.

heading: 4 Experimental Setup
text: To fully test the CAT methods for pruning we explore the following variations: (1) different target languages, (2) different pruning levels, (3) different test domains, and (4) configurations such as model sizes and choice of checkpoints. We compare the effectiveness of the pruning against embeddingbased, LLM-based, and random selection.

heading: 4.1 Training Data
text: We conduct our experiments on datasets that translate from English into three languages: French,
1Either 5th and 1st or 2nd and 1st.
German, and Swahili. For German, we use the English-German WMT19 dataset (Barrault et al., 2019), comprising 38M parallel sentences sourced from various origins, and for Swahili, the EnglishSwahili WMT22 dataset (Costa-jussà et al., 2022; Adelani et al., 2022b), consisting of 22M parallel sentences sourced from the web (Bañón et al., 2020) then automatically identified using LASER. We use the English-French WMT15 (Bojar et al., 2015) dataset for French. Due to compute constraints, the majority of our experiments and ablations were carried out on a randomly selected subset of 3.8M parallel sentences from all datasets, which constitutes 10% of the original German and French datasets and 17% of the Swahili dataset. Subsequently, we implement the best variant of our method on the entire dataset to evaluate the scalability of our approach.

heading: 4.2 Evaluation Data
text: Our evaluation sets consist of WMT18 (Bojar et al., 2018), WMT15 (Bojar et al., 2015), FLORES (Goyal et al., 2022; Costa-jussà et al., 2022) and MAFAND-MT (Adelani et al., 2022a) test sets. These test sets were curated using human annotators thereby mitigating common issues associated with automatically aligned datasets, such as erroneous alignments. The WMT18 and WMT15 test sets were released as part of the WMT machine translation task consisting of 2998 and 1500 parallel sentences respectively from various sources of the news domain. WMT18 was used to evaluate German models while WMT15 test set was used to evaluate French. The FLORES test set was used to evaluate all languages: German, French and Swahili. The FLORES test set consists of 1012 sentences from 3 sources: WikiNews, WikiJunior and WikiVoyage. We use the MAFAND-MT test set for Swahili model evaluation. MAFAND-MT test set consists of 1835 sentences from the news domain.

heading: 4.3 Training Details
text: We train all our NMT models using the fairseq library2 (Ott et al., 2019). Our models follow the same architecture as the vanilla transformer (Vaswani et al., 2017) with GeLU activation (Hendrycks and Gimpel, 2016) instead of ReLU activation (Nair and Hinton, 2010). We report all the hyperparameters used for training the models
2https://github.com/facebookresearch/fairseq
in Table 1. We conduct all our experiments on 4 A100 40GB GPUs with the duration for runs at 90%, 70%, and 50% prune level, and full dataset utilization approximately taking 1, 2, 3, and 4.5 hours, respectively. Furthermore, we train separate SentencePiece tokenizers (Kudo and Richardson, 2018) for source and target languages with a vocab size of 16K using 3.8M translation pairs for both English-German and English-Swahili experiments. We use SacreBLEU library3 (Post, 2018) to compute BLEU score (Papineni et al., 2002) and used it as our evaluation metric. We also calculate chrF++ and COMET scores and report them in the Appendix. All the metrics exhibit the same trend in our experiments. Our baseline models are trained on the entire 3.8M translation pairs and denoted as “full” for all language pairs; English-German, English-French and English-Swahili. We investigate the impact of data pruning with sparsity levels of 50%, 70% and 90% using various techniques briefly described below.

heading: 4.4 Baselines
text: As outlined in Section 2, we compare with the most common NMT data pruning techniques:
1. Random Selection: This involves choosing data instances without specific criteria purely by chance. Although this approach may seem intuitively suboptimal, Azeemi et al. (2023) shows that this also performs competitively with different pruning strategies for a few cases. Therefore, we include this approach as one of our baselines.
2. COMET-QE: Rei et al. (2020) proposed model-based MT evaluation metric that utilizes embeddings from XLM-RoBERTa (Conneau et al., 2020) to access translation quality when provided source, labels and target translations or just source and target translations.
3. LaBSE & LASER: Both are multilingual sentence embedding models trained on large aligned corpora covering a number of languages. LASER (Heffernan et al., 2022) covers 200 languages and was trained with an encoder-decoder LSTM architecture. LaBSE (Feng et al., 2022) on the other hand covers 93 languages and was trained on top of BERT (Devlin et al., 2019).
3With parameters: nrefs:1|case:mixed|eff:no| tok:13a|smooth:exp|version:2.3.2
4. BLOOM LLM: Using the multilingual pretrained BLOOM LM (Scao et al., 2022), we explore the selection of relevant data points by computing the perplexity of the reference sentence.4 This is the closest to the CAT method, as it also leverages insights from a trained Transformer model, where we expect easy and hard-to-learn examples to fall on the tails and thus will not enable the model to learn effectively. In contrast to CAT, its perplexity scores might be more expressive, because it was trained on more data and with more parameters. On the other hand, we do not get any insights on training dynamics. We use BLOOM variants with 560M, 1.1B and 1.7B parameters for our experiments and assess the impact of model size on perplexity computation.
In the following, we will summarize embeddingbased selection and QE-based selection as Translation Quality Estimators.

heading: 5 Results and Discussion
text: We conducted experiments on English-German and English-Swahili datasets, employing various pruning techniques. We pruned 90%, 70% and 50% of the training data and compared this to training on the full set. To evaluate these models, we used WMT18 and FLORES test sets for German, and FLORES and MAFAND-MT as the test sets for Swahili. Furthermore, we applied the most effective pruning techniques to an English-French
4Past work has found that NMT is more susceptible to target-side noise (Khayrallah and Koehn, 2018), so we focus on target-side perplexity. In principle, this method could be extended to a combination of source and target perplexities as in the cross-entropy method (Moore and Lewis, 2010), but it would require twice the computation.
dataset to verify the consistency of performance across different language pairs.

heading: 5.1 CAT techniques outperforms random pruning in German and Swahili
text: Figure 1a demonstrates that CAT-DIFF(1,5) and CAT-VAR outperform random selection in the German test sets. On the other hand, in Swahili, only CAT-DIFF(1,5) consistently outperforms random selection, while selecting data randomly yields better results than CAT-VAR (refer to Figure 1b). CAT techniques achieve efficient pruning while maintaining an upwards of 75% performance compared to training on the full dataset, even after pruning 90% of the data. For German, both CAT-DIFF(1,5) and CAT-VAR maintain over 80% of the performance achieved by training on 100% of the data. In the case of Swahili, CAT-DIFF(1,5) retains above 80% of the performance, whereas CAT-VAR tends to perform worse, retaining only around 30% of the performance. A noteworthy efficiency of CAT techniques lies in their ability to sidestep the inferencing with fully converged models, thus one doesn’t need a lot of compute to utilize these techniques.
Table 2 presents the ablation study conducted for CAT-DIFF to compare the performance of CAT pruning techniques based on different checkpoints. We evaluate the difference between checkpoints 1 and 5, referred to as CAT-DIFF(1,5) and checkpoints 1 and 2, denoted as CAT-DIFF(1,2). We see that CAT-DIFF(1,5) yields better performance than CAT-DIFF(1,2). However, CAT-DIFF(1,2) offers the advantage of using fewer resources albeit at the cost of a slight dip in performance.

heading: 5.2 Translation quality estimators are less effective for German
text: We find translation quality estimators to be less effective in pruning German than Swahili. For instance, Figure 2a indicates that random data pruning outperforms translation quality estimators at 90% and 70% pruning levels for both test sets. However, with Swahili, we observe a different behavior, where LABSE outperforms random pruning on all pruning levels, as depicted in Figure 2b. Interestingly, pruning Swahili data with LaBSE and LASER yields superior performance compared to training on the full data set, particularly for the 50% pruning level for the FLORES test set. Further, translation quality estimators consistently outperform random pruning in Swahili and do not sacri-
fice performance even at 90% pruning level with the exception of LASER on the MAFAND test set. We believe that these differences in behavior can be attributed to varying data quality between the two languages. The German dataset contains sentences with high variance in sentence length and included sentences with foreign language. Conversely, the Swahili dataset consists of processed automatically aligned sentences, where there is a low sentence length variance and the sentences do not contain any foreign language. See Figure 7 and Figure 8 in the Appendix. Therefore, selecting the best sentences ensures the inclusion of high-quality translation pairs.

heading: 5.3 LLM perplexity techniques are competitive for Swahili but sub-par for German
text: Figure 3 illustrates that utilizing LLMs for pruning provides competitive results for Swahili but shows sub-par performance for German. We further observe that the LLM perplexity technique across different BLOOM model sizes performs superior compared to random pruning across pruning levels for Swahili (see Figure 3b). Among the various BLOOM model sizes, the smallest model, BLOOM 560M consistently outperforms the others, followed by the BLOOM 1B model. It is important to note that LLM techniques perform worse than random for German. This suggests that in cases where limitations in data and compute double bind, utilizing pretrained LLMs can save on the limited resources available.

heading: 5.4 What pruning techniques work best?
text: Figure 4 shows the comparison of random, LaBSE and CAT-DIFF pruning techniques for German,
French and Swahili. It is evident that CAT-DIFF, outperforms other methods for German, whereas LaBSE demonstrates superior performance for Swahili. Specifically, CAT-DIFF(1,5) surpasses the other techniques for German at each pruning level and retains 99% of the full data’s performance even at 50% prune level (refer to Figure 4a). We observe similar trends for French in Figure 4b with CAT-DIFF(1,5) also generally outperforms LaBSE and random selection, except for FLORES at the 50% prune level. Conversely, for Swahili, LaBSE is consistently superior to other pruning techniques, followed by CAT-DIFF(1,5) (see Figure 4c). We also find that pruning using LaBSE for Swahili yields competitive performance as training on the full dataset.
We also report the statistical significance results with paired bootstrap resampling(Koehn, 2004) for SacreBLEU and COMET in the Appendix, see Tables 4 to 6.

heading: 5.5 Scaling best pruning techniques
text: To verify the generalizability of CAT-DIFF along with baselines such as LaBSE and Random Selection, we trained NMT models on the entire dataset,
∼23M parallel sentences for Swahili and ∼38M parallel sentences for German and French. We compare CAT-DIFF(1,5), LABSE and random data pruning at 90% prune level. We find the results to be similar to 3.8M scale experiments. Table 3 show that CAT-DIFF(1,5) outperforms random in German and French while pruning using LaBSE performs better in Swahili.

heading: 6 Analysis of Sentence Selection for German, French, and Swahili
text: To better understand the performance gap between pruning techniques for German, French(IndoEuropean languages) and Swahili(a Bantu language), we analyze the sentences selected randomly as well as those selected using CATDIFF(1,5) and LaBSE. Figure 5 illustrates that longer sentences correlate with higher BLEU scores for German and French when considering target sentence length. This observation of longer sequence lengths giving better scores is also seen in Junczys-Dowmunt (2019). Most of the points in the
top right quadrant are chosen by CAT-DIFF(1,5), indicating a preference for longer sentences and performs better. However, we observe that picking the longest sentences for Swahili doesn’t lead to better BLEU scores. We, therefore also look at the lexical diversity to distinguish the sentences selected.
Figure 6 shows that CAT-DIFF(1,5) selects sentences that have more unique and rare words as opposed to German which is less lexically diverse and thus results in lower mean frequency. Simi-
larly, CAT-DIFF(1,5) also selects more unique and rare words for Swahili. Although LaBSE performs relatively better in selecting more unique and rare words for French, it still falls short compared selection of words using CAT-DIFF(1,5) for German. This implies that CAT-DIFF(1,5) selection is far more complex than selecting the longest sentences. We leave a deeper investigation of sentences selected using various techniques for future studies.

heading: 7 Conclusion
text: Identifying optimal subsets of a given dataset for training a machine translation model is a crucial problem that enables data efficiency and saves on compute. In this paper, we present Checkpoints Across Time (CAT) methods as efficient techniques to prune datasets. CAT uses the change in perplexity scores of individual examples over the first few training epochs to find good subsets. Using 3.8M sentences for English-German, English-French and English-Swahili, we show that CAT techniques offer highly computationally efficient pruning that
achieves above 75% of full performance even when 90% pruning is applied to all target languages. We also show that LaBSE, when used to select training examples from an automatically aligned dataset provides strong signals on examples crucial for model generalization as the examples selected perform on par or surpass training on the full dataset. Interestingly, we find that utilizing LLMs for pruning does not offer benefits in selecting data for German but does for Swahili.

heading: 8 Limitations
text: In this work, we focus on data pruning in a bilingual setting for 3 languages and limit the data size to 3.8M sentences. This is due to computation constraints. This means that the results may not apply to other languages. Also, it would be interesting to investigate performance on larger scales as we only conducted experiments on German and English for CAT-DIFF and random selection. Moreover, fur-
ther research is needed to explore the multilingual setup where data pruning would be conducted simultaneously on several languages.

heading: 9 Acknowledgements
text: EAC acknowledges a grant from the Carnegie Corporation of New York (provided through the African Institute for Mathematical Sciences). The statements made and views expressed are solely the responsibility of the authors.

heading: A Dataset Analysis
text: Figure 7 and Figure 8 provide analysis of the 3.8M dataset for German, Swahili and French. Both German and French exhibit outliers in the sentence length with some sentences being 500 words or longer while Swahili seems to be filtered by sentence length as no sentence has more than 250 words. See Figure 7. We also see that foreign languages were removed from the Swahili dataset as both source and target sentences do not contain any other language whereas both German and French contain foreign language.

heading: B Other evaluation metrics and significance tests
text: We ran additional experiments to calculate chrF++ and COMET scores and report the statistical significance results with paired bootstrap resampling (Koehn, 2004) using sacreBLEU and COMET libraries with default configurations for the respective metrics below for all our experiments with on the 3.8M data scale. We find that CAT-DIFF demonstrates competitive or superior performance compared to other methods on en-de and en-fr pairs across most test sets. Bolded numbers in Tables 4 to 6 show that with 1000 resamples, CAT-DIFF outperforms the other technique 95% of the time thus CAT-DIFF is statistically significance at 95% confidence level. These results highlight the effectiveness of our proposed method.

heading: Future Work
text: We leave a deeper investigation of sentences selected using various techniques for future studies