Frequency Balanced Datasets Lead to Better Language Models
Authors: Rodolfo Joel Zevallos
Mireia Farrús
Núria Bel
Venue: EMNLP 2023 Findings (2023)
Keywords: Language Models
word frequency
pre-training corpus
low-resource languages

Abstract: This paper reports on the experiments aimed to improve our understanding of the role of the amount of data required for training attention-based transformer language models. Specifically, we investigate the impact of reducing the immense amounts of required pre-training data through sampling strategies that identify and reduce high-frequency tokens as different studies have indicated that the existence of very high-frequency tokens in pre-training data might bias learning, causing undesired effects. In this light, we describe our sampling algorithm that iteratively assesses token frequencies and removes sentences that contain still high-frequency tokens, eventually delivering a balanced, linguistically correct dataset. We evaluate the results in terms of model perplexity and fine-tuning linguistic probing tasks, NLP downstream tasks as well as more semantic SuperGlue tasks. The results show that pre-training with the resulting balanced dataset allows reducing up to three times the pre-training data.

Frequency Balanced Datasets Lead to Better Language Models
Rodolfo Zevallos
Núria Bel
Mireia Farrús
Universitat Pompeu Fabra
Universitat Pompeu Fabra Barcelona, Spain nuria. bel@upf.eel.ff..mm edu
Universitat de Barcelona Barcelona, Spain mfarrus@ub.. edu
Barcelona, Spain
rodolfojoel zevallosuuf.l edu
Abstract
proved be behind some phenomena related to the geometry the representations causing problems for so called semantic tasks that rely on similar- ity assessments (Ethayarajh, 2019; Fuster- Baggee-t and Fresno, 2022). In this paper, we report on our experiments to better understand the impact of to- ken frequency in the pre-training data language model (LM) quality, in terms of perplexity, and the quality of the representations that these mod- els provide for fine-tuning. Our ultimate goal is to study to what extent the quantity of pre-training data could lessened, since this could of great importance for training LM for low-resource lan- guages (LRLs).
This paper reports the experiments aimed to improve our understanding of the role of the amount data required for training attention- based transformer language models. Specif- ically, we investigate the impact of reducing the immense amounts of required pre-training data through sampling strategies that identify and reduce high-freee seh-freeuencyy tokens as differ- ent studies have indicated that the existence of very high-free frequerrycncy tokens in pre-training data might bias learning, causing undesired ef- fects. In this light, describe our sampling algorithm that iteratively assesses token fre- quencies and removes sentences that contain still high-freyuency tokens, eventually deliver- ing a balanced, linguistically correct dataset. We evaluate the results in terms of model per- plexity and fine-tuning linguistic probing tasks, NLP downstream tasks as well as more seman- tic SuperGlue tasks. The results show that pre- training with the resulting balanced dataset al- lows reducing up three times the pre-training data.
Texts are known to follow Zipf"s law with some tokens occurring very frequently, some occurring with medium frequency, and many tokens that rarely occur forming long tail. Different dis- ciplines have addressed the problems of using long-tailed training data where some elements ap- pear on most of the data but most of the elements are under-represented. Under-sampling and over- sampling techniques to directly adjust the number of examples by removing and adding data respec- tively have been proposed (Cui et al., 2019; Rau- nak et al., 2020). In this context, we propose under-sampling algorithm whose objective bal- ance token frequency by removing the sentences in which high-frequency tokens (and bigrams) occur, thus improving the estimation model probabil- ities for low-frequency tokens. By ensuring that tokens have a balanced exposure during training, the model should become more capable of learn- ing more diverse linguistic patterns, resulting in improved model perplexity and in better represen- tations for NLP tasks. Our experiments show that sampling the data avoid highly skewed token frequency distribution delivers results equivalent to pre-training the model with average three times more but unbalanced, raw data.
1 Introduction
Historically, using more training data has been considered a key to improving the performance of natural language processing (NLP) tools. The seems to be true for transformer- based large language models (LLMs) such as BERT (Devlin al., 2019) and RoBERTa (Liu al., 2019) that exhibit their best performance when being trained with really massive quantities of texts. However, the question arises whether more training data is the only factor for improving results or whether specific characteristics of the provided texts have particular impact the learning efficiency of these powerful models. Recent research (Wei et al., 2021; Razeghi al., 2022) has found evidence about the impact of the absolute frequency of pre- training data tokens in the prediction capacities the model. Besides, high-frequency terms have
In this paper, we report on how our algorithm

---

processes the dataset, how it iteratively assesses token frequencies and removes sentences that con- tain still high-frequency tokens, eventually deliver- ing a balanced, linguistically correct dataset. We evaluate the results in terms of model perplexity for English general and domain texts and for four other languages of different morphological com- plexities. Furthermore, we evaluate the approach in terms of the quality of the representations for fine-tuning to a range of tasks: linguistic probing tasks, NLP tasks and more semantic SuperGlue tasks. Our hypothesis was that if Pruttr-BBagettu and Fresno (2022) analysis is correct, reducing fre- quency biases would also improve the results in semantic tasks.
training data, although in a GPT model, and found strong correlation of token frequencies with the resulting model performance.
Puccetti et al. (2022) provided evidence directly linking the presence of outlier dimensions (Koval- eva et al., 2021) with the frequency of tokens in the pre-training data. Outlier dimensions are those dimensions which parameters with an unusually high magnitude- -in terms of standard deviation- - are found consistently across the model layers. Puccetti et al. (2022) results suggest that outliers are due to the highly skewed token frequency dis- tribution in the textual pre-training data. Moreover, Fuster- Fuster-Baggetto Baggetto and Fresno (2022) demonstrated that the con-finettuned BERT models contain to- ken frequency biases that distort the embedding representation space. The distortion leads to the already observed poor performance on semantic tasks because of what Gao et al. (2019) already diagnosed as anisotropy in transformer LMs or the representation degeneration problem: the embed- dings concentrate in hypercone instead of occu- pying the whole space. Consistently, Ethayarajh (2019) found that randomly sampled words tend to be highly similar when measured by cosine similar- ity. Kovaleva al. (2021) have also found evidence that different transformer-base LMs have similar behavior regarding poor semantic isometry, even when they differ in number of parameters, architec- ture or learning objective.
In what follows, section reviews the related work; section describes the algorithm imple- mented, as well as its impact on the initial and the resulting datasets. Section describes the ex- perimental setup for training ten RoBERTa mod- els with different data and in different languages and domains. Also in section 4, we describe the fine-tuning experiments for eleven different tasks that were meant to get thorough information about the impact of training the model with a balanced dataset, specially in semantic tasks. Section 5 de- scribes the results of the experiments in terms of perplexity of the models and of F1-score of the different tasks. Results are analysed and discussed in section 6. Finally, section 7 is devoted to sum up the conclusions and contributions.
As for bias-removal techniques, our approach is different from Fuster-Baggetto and Fresno (2022), that preferred not to remove frequent tokens from sentences because it would be affecting the ability of the LM. Instead, they removed the embeddings after processing, before pooling like Jiang et al. (2022) and Yan et al. (2021). In our experiments. remove the sentences with high frequency token to guarantee that only grammatical sentences are processed. Our approach is, thus, in line with Samuel et al. (2023) findings that got a noticeable difference in downstream tasks performance when using the British National Corpus of 100M tokens, as a sample of a curated data set that proved to be enough to learn the required basic linguistic skills.
2 Related Work
A number of works have studied the impact of word frequency on different aspects of LLM and, in particular, on the quality of the delivered rep- resentations. Kassner et al. (2020) have studied BERT models and possible memorization based on token frequency, demonstrating that if token appears fewer than 15 times, the model will dis- regard it, while a token that appears 100 times or more will be predicted more accurately. Zhou et al. (2022) demonstrated that high frequency words and low frequency words are represented differently by transformer LLM,, in particular by BERT. Wei et al. (2021) found evidence that BERT models struggle correctly predict a verb when it belongs a set of word pairs (S-V), or bigrams, that appears less than 10 times or more than 100 times the train- ing corpus. Razeghi et al. (2022) also examined the strong impact of the frequency of terms in pre-
3 Frequency
In this section, we report on our algorithm that processes a raw text corpus and returns a balanced dataset. As explained in this section, which we set to a size 10M tokens aas we will explain later on

---

in more detail. The processing includes, the one hand, calculating the number of occurrences of content tokens in the whole dataset and assessing to what extent they usually co-occur with other less frequent tokens, thus forming to be kept bigrams. On the other hand, frequency counts are updated after each sentence removal.
Counting Token Occurrences (Freq): We per- formed a count of how many times each token appeared in the dataset.
4. Bigram Counting (Bi): In addition to single tokens, we also considered bigrams, which are pairs of tokens appearing together. By count- ing the occurrences of bigrams, wanted take into account that very frequent tokens can consistently co-occur with other less fre- quent token that should be kept in the curated dataset.
Algorithm Removing sentences from dataset
Require: S, Tmars Buins Freq, Bi
Ensure: Stalanced
1: Sbalanced S-S
2: deleted feleaeed_praass++ + True
the steps described above, we ob- tain Freq and Bi. Freq is list { fo). (Wi, (0)..,(wn, fn)]. wi is a content token and fi is the fre- quency of wy. Similarly, Bi is a list {((wo, ). fbo), ((,,,, ((w,, ), fba)). where (w;;) ) represents bigrams and fb; is the frequency of (w:.).
3: while deleted deletedd_praaess do
4: stop_process +- False
where
5: for each 8; in balanced do
W S-ssspplssteeeeeessss sentence(ss))
7:
all wy in W, Freq(us]
Tmae and Bi((w;s)) )] > Bmin then Remove 8; from Sualanced
8:
9:
Update wi from Freq
3.2 A frequency-algorithe for getting a balanced dataset
10:
Update (wi,) ) from Bi
11:
stop_procest <- True end if end for
12:
One of the most important steps in balancing a dataset is to setup an upper threshold for the oc- currence of any token (Tmar) or token and usual co-occurrences (Bmin).
if sttpssppopeeee poop__rocesssss_ False then geleteddpprraser +- False 16: end if
15:
Taking into account the previous research, we decided to consider both token frequencies and co-occurrence frequencies in our algorithm. Fur- thermore, in order adapt the algorithm to each dataset, we decided to set T maz value as rela- tive value, calculated based on the average frequen- cies of the type tokens (TT), excluding outlier fre- quencies (OF). To remove the outlier frequencies. we used the " contlier_uttlls library from Python. Our algorithm utilizes Bmin of 10 and Tmaz is obtained using the following equations:
17:
end while
3.1 Calculating token frequencies
To calculate the token frequencies of a dataset, we implemented the following steps:
1. Word split: We divided the dataset into in- dividual tokens by using the space character (" ") delimiter. We opted for this easy approach it enough for the purpose of our experiments.
Σ Freq
Freqavg
--=
(1)
2. Stopword Removal: We eliminated stopwords from the dataset because they are mainly grammatical words that among the most frequent tokens in any language. For ex- periments, used the Stopword function from NLTK! library, which contains stop- words list of German, French and Turkish. We got Quechua stopword from a Github reposi- tory.
Z
where:
Z Number of TT without OF.
Freq, Frequency of token,
Freqavg Average frequency
Frequv
if Freq 100
T Tmax
(2)
100
if Freqaye 100
'https://www. http:/wwwwwt.....mmm.. www.n.lkorrgggg... org
'nttps://github cwt.uu...oosttttee com/sstopwods-isso stapwords-iso
'https:/ http:///yppp.ooggpooooo http://pp....rrppooeeecr/otttttt #description

---

Algorithm 1 (removing sentences from dataset) is used create list of the sentences (Shalanced) that not contain high frequency tokens. It takes as parameters all the sentences from the dataset (5), the list of token frequencies (Freq), the list of bigrams (Bi), Tmar and Bmin-
more data achieve the target. However, for most datasets, an average increase of four times was sufficient to reach 10M tokens, except for Turkish, which required approximately seven times more data.
To analyze each sentence (s;) in Sialanced where {50, (., Siy 8n). split 8 using (" ") as delimiter, obtaining W [wo, Wi: ...; Wn), being each wi a token. Then, for each element of W we obtain the values Freq[we] and Bi((ws,) )]. Freq[wi] and Bi((w;;) ) values are compared with Tmaz and Buin respec- tively. If in all Wi Tmaz and Bmn are lower than Freq [rs] and Bi((ur,) }}, we remove 8; from Sbalanced; and we update the frequency counts in the lists Freq and Bi. The process is repeated each time this condition fulfilled, until the condition is not fulfilled any 8,. The output of this process is Saalanced
4 Experimental Setup
V 8 Sallancdd
In this section, we describe the setup of the differ- ent experiments. We trained ten RoBERTa mod- els with different processed and non-processed datasets for English, French, German, Turkish data and measured their perplexity to assess the impact of balancing token frequency in pre-training data, We also applied the algorithm for assessing how it could be used in a LRLs scenario, such as the one of Quechua, with only 6M tokens corpus that got reduced into only 1M. After that, we describe the fine-tuning of the English models for four different typical NLP tasks, for linguistic probing classifier and for four different SuperGLUE tasks.
3.2.1 Dataset size details
4.1 Language models
In order to conduct the experiments, we followed Warstadt et al. (2020b), in which models created with 10M and 100M datasets where compared, and we used the algorithm to create balanced datasets of 10M tokens, except for the LRL Quechua, which we set at IM tokens due to the lack of large cor- pus. Table shows the amount of data consumed for each dataset to reach 10M tokens using our algorithm.
For training all the LMs, we followed the configura- tion outlined by Warstadt et al. (2020b) and specifi- cally, employed the hyperparameters from their Med-Small model, which consisted of attention heads, a hidden size of 512, ferd-fo network dimension of 2048, and total of 45 million param- eters. In line with common practice in transformer- based LM development, we employed the BPE tokenizer (Sennrich et al., 2015), utilizing a vocab- ulary size of 52k tokens. Additionally, we adopted identical parameter values for dropout, attention dropout, and learning rate decay. For comprehen- sive overview of these parameters, plcase refer Table in the appendix.
Size needed
Dataset
Language
Times
Wikipedia
English
36M
3.6x
News
English
29M
2.9x
Reviews
English
24M
2.4x
BioMedic
English
30M
3.0x
4.1.1
English Wikipedia corpus
CS
English
19M
1.9x
English training data source is the Wikipedia cor- pus used by Devlin et al. (2019). It relevant note that this corpus has a total 2.5 billion tokens but for our experiments we have only downloaded 50M tokens. We compared our roBERTa models trained with a balanced corpus of 10M tokens, ob- tained by using the frequency-algorinn already available roberta-base models trained on 10M' and 100M datasets.
German
41M
4.1x
OSCAR
French
39M
3.9x
Turkish
72M
7.2x
Llamacha
Quechua
6M
6x
Table 1: Details of raw datasets to result in 10M
with the
balanced one in number of tokens and reduction times.
The amount of data required reach 10M tokens varies significantly depending on the language and domain. In the case of Quechua, the size of the training data was 1M, and it required six times
"nttps://tugiingaacc.cco//
roberta-base-1ee 10M-2
htttps: http://hugggnggcccec coteeee/////iimaeeeeee roberta- roberta-base-100 maase-100M-1

---

4.1.2 English domain corpora
holds in the real-world between two entities, as a sample of a more semantic NLP task. Different re- lations are taken from an inventory of symbolic relation types Fause-Effect, Content-Container, Instrument-Ageney, etc. The data were taken from Tenney et al. (2019).
In addition to the experiments explained above, we trained RoBERTa models with data from dif- ferent domains using our frequency-algoriihm for sampling the data. The importance of using our rrequency-algorithn on corpora other than Wikipedia lies in the need to observe how our algo- rithm behaves with domain-specifie words and how they are distributed in specialized corpus. We se- lected four specific domains for these experiments: news, product reviews, biomedicine, and computer science.
All fine-tuning models used same hyperparam- eters. For comprehensive overview of these pa- rameters, please see the appendix.
4.3 BLiMP tasks
For the biomedicine and computer science do- mains, we utilized data obtained from Lo et al. (2019), with technical knowledge and specialized terminology in those fields. Regarding the news domain, we used data extracted from RealNews (Zellers et al., 2019). Finally, for the Amazon prod- uct reviews domain, obtained training data from HuggingFace".
We utilized the BLiMP testset (Warstadt al., 2020a) evaluate the quality of representations regarding individual grammatical phenomena in English. BLiMP is a collection of 67 tasks, each of which contains 1000 minimal pairs of sentences that highlight specific morphological, syntactic, or semantic phenomena. These BLiMP minimal pairs consist of two sentences that differ in single edit but contrast in their grammatical acceptability. The purpose of BLiMP is to conduct unsupervised eval- uation of LMs through a forced-choice acceptabil- ity judgment task. A LM is considered to classify a minimal pair correctly if it assigns higher prob- ability to the acceptable sentence. Thus, BLiMP provides a valuable tool for measuring and com- paring the performance of models. For evaluation, we followed the MLM scoring method proposed by Salazar al. (2020).
4.1.3 Other languages
order to validate the robustness of the approach with other languages that are morphologically dis- tinct from English -hat is, with different dis- tribution of tokens and types-, we conducted ex- periments building models for German, French, Turkish, and Quechua. As mentioned in the intro- duction, we applied the algorithm Quechua for assessing how it could be used LRL scenario with only a 6M tokens corpus.
The resources used for German, French, and Turkish were obtained from the OSCAR corpus (Ortiz Suárez et al., 2019). Each corpus has dif- ferent size. German has 21 billion tokens (Scheible et al., 2020), French has 32.7 billion tokens (Mar- tin et al., 2020) and Turkish has 11.5 million doc- uments (Toraman et al., 2022). For Quechua, we used the 6M corpus from Zevallos et al. (2022).
4.4 SuperGLUE tasks
SuperGLUE is benchmark set consisting of eight classification-basel language understanding tasks Wang et al. (2019). We evaluated English LMs fine-tuning five SuperGLUE tasks in order to the impact of training with a balanced dataset in tasks that are considered to involve understand- ing such as Boolean Questions (BoolQ) (Clark et al., 2019), Commitment Bank (CB) (De Marn- effe et al., 2019), Choice of Plausible Alternatives (CPA) (Gordon et al., 2012), Recognizing Textual Entailment (RTE) (Wang et al., 2019) and Word in Context (WiC) (Pilchvar and Callaco-Collados. 2019). Our interest was whether reducing fre- quency biases would also improve the results in semantic tasks. Description of the tasks and the hyperparameter search range used for each task are described in the appendices.
4.2 NLP downstream tasks
For English, we additionally fine-tuned a number of classifiers for traditional NLP tasks: Part-of- Speech tagging (POS), Dependency Labeling (DL.) and Named Entity Labeling (NER). POS and NER task data were obtained from Weischedel et al.
(2013)7 and DL from Silveira al. (2014).
We also addressed Relation Classification (REL) task, which the task of predicting the relation that
"https: http://hhugggaaaae http:///ggggngaaae..ceaeatatttte wttp///aaat/t/aaaaaeaaaa
reviews /ttttps:/cctalogg 1dc. ude..u.en.cc.mmm www..uppee.enece22222 watm..ddLLDD211111

---

English Wikipedia
English Domains
Model
Perplexity
Domain
roberta-base
roberta-freq
roberta-base- 100M
4.61
News
19.42
13.59
roberta-base-100 10M
10.78
Reviews
25.17
18.26
roberta-freq-100 10M
4.93
BioMedic 17.31
10.15
Computer 15.48
8.63
Table Model perplexity of language models trained with raw data and balanced word token frequency. The word "freq" in the model name refers to the use siquuency-balance data after applying our algorithm.
Science
Table 3: Model perplexity of language models trained with different domains and applying the frequency- algorithm to get 10M tokens training data. These models were trained from scratch for this experiment.
5 Results
We now describe the results of the different experi- ments. In all of them, the use of balanced dataset for training the LMs equals improves its quality in terms of perplexity of the LM in terms of accuracy in fine-tuned tasks..
from different languages to see the impact of us- ing our frequency algorithm. As shown Table 4, German achieves a reduction perplexity of more than 7 points using the balanced corpus (roberta- freq). Similarly, French, Turkish and Quechua re- duce perplexity by 8.26, 8.65 and 62.75 points spectively, confirming that morphologically complex languages, with more word types due inflection, also greatly benefit of balanced pre- training dataset.
5.1 Language models and pre-training data
5.1.1 English Wikipedia
To assess the results, we evaluated the impact of balancing the pre-training datasets by comparing the perplexity of the model obtained with the per- plexity of models trained with the same amount but raw data.
Other Languages
Language German
roberta-base
roberta-freq
Table 2 shows the perplexity of "roberta-base- 10M" and 'roberta-base 100M" as obtained with the models from Zhang et al. (2020), as well as the model trained with the frequency balanced corpus. The perplexity is reduced in almost points, from 10.78 4.93, getting very close 4.61 perplexity of the model when trained with 100M words corpus raw data.
17.05
9.76
French
13.81
10.88
Turkish
35.18
26.53
Low-Resource Language
Language Quechua
roberta-base
roberta-freq
358.47
295.72
Table 4: Model perplexity of language models trained with different languages when applying the frequency- algorithm. models are trained with 10M tokens except for Quechua, which is trained with only IM tokens. These models were trained from scratch for this experiment.
5.1.2 Domains
We also evaluated different domain specific train- ing sets collected from domains such as news, re- views, biomedical, and computer science texts. As shown in Table the reduction of model perplexity is clear in all the domains, with 6 points of reduc- tion in average, which supports the robustness and consistency of the impact of balancing word token frequency.
5.2 NLP downstream tasks
As can be seen in Table 5, the use of our frequency reduction algorithm to process the training dataset leads to improvements in the four tasks: POS, NER, REL and Dependencies. The "roberta-freq- 110M model increases performance compared to "roberta- base- 10M", with better results for tasks that were more semantic such as REL. Moreover, note that the results in terms of FI -core are the same for
5.1.3 Other languages
Like with the roBERTa model of English, we eval- uated the perplexity of models trained with corpora
*Improvements are in all cases statistically significant 0.05 as assessed with sign test

---

NLP Task
Model
Part-of-Speech
NER
REL
Dependencies
roberta-bai seartta-ase--iie 100M
0.98
0.97
0.76
0.92
Soberta-base-10M
0.96
0.95
0.67
0.88
roberta-freq-10M
0.97
0.97
0.76
0.92
Table 5: NLP results in Fl-score the different language models trained according to: training size and
frequency reduction technique.
BLiMP
Task
roherta-base-100
roberta-freq-10N.
voberta-base-10mt
ANA. AGR.
91.1
96.7
97.2
ARG. STR
70.1
78.9
79.1
Binding
71.6
75.2
75.4
CTRL RAIS
70.7
79.7
79.6
D-N ARG.
91.6
93.1
94.5
Ellipsis
86.0
90.8
91.6
Filler GAP
67.3
79.1
78.8
Irregular
84.3
92.5
92.7
Island
53.6
61.3
63.0
NPI
75.6
77.3
77.2
Quantifiers
58.6
62.7
64.7
S-V ARG.
77.0
86.8
87.5
Table BLiMP results for 10M algorithm.
100M roberta-base models and the roberta model using our 10M frequency-
almost all cases that the scores obtained with the "roberta-base-100MM model.
5.4 SuperGLUE tasks
The experiments conducted on the SuperGLUE dataset reveal important findings regarding the per- formance of the evaluated models. The "roberta- freq-10M model achieves comparable or slightly better performance than the baseline models in most of the tasks evaluated in SuperGLUE.
5.3
BLiMP Tasks
The experiments conducted with the BLiMP dataset reveal improvements regarding the refer- ence models used: "roberta-base- 10M" (see Table 6) in most of the tasks. In the ANA.AGR task, score of 96.7 is achieved with the "roberta-free" freq- 10M" model, surpassing "roberta-base- 10M" by 5.6 and coming close to the "roberta-base- 1 model with 97.2. In the CTRL.RAIS task, the "roberta-freq-10M model achieves a score of 79.7, surpassing "roberta-base- 10M" by 9.0 and "roberta- base-100M" by 0.1. Similarly, in the NPI task, the "roberta-freq- 10M" model demonstrates superior- ity over the "roberta-bare basetttabbbb-beeee 10M" and "roberta-bare basc-- 100M" models by 1.7 and 0.1 respectively. As for the Filter.GAt GAP task, the "roberta-freq" model out- performes both base models, achieving a score of 79.1. Lastly, the fubertaa-ree--10M model sur- passes the "roberta-base- 1OMM--OMe--- model in all tasks.
As can be seen Table 7, the most highlighting task where the "roberta-freq-10M equals "roberta- Pase-100Ms is WiC, achieving a score of 0.70. Moreover, the "roberta-freq- 10MM overcomes the "roberta-base sintta-base-100 model in all tasks, with an erage of 0.05 points.
Discussion
Since word frequency shows a highly skewed distri- bution in texts, when using balanced dataset (especially when pre-training with a small 10M corpus), the potential biases due highly-frequent tokens should be moderated, and the system should be able to perform a better computation of the prob-

---

SuperGL.UE
Task
roberta- basertt-bas-lie 10M
roberta-freq-100
roberta-base 100M
BoolQ
0.68
0.73
0.74
CB
0.81
0.86
0.89
Copa
0.57
0.60
0.61
RTE
0.57
0.62
0.64
WiC
0.65
0.70
0.70
Table 7: SuperGlue results for the 10M and 100M roberta-base models and the roberta model using our 10M requency-algoritin
abilities of medium and rare words.
the semantic representation and makes the system perform significantly better especially in semantic tasks.
The results of our experiments consistently show that perplexity in all our experiments improves when working with a balanced corpus. In particu- lar, we have seen that using balanced datasets for modeling morphological complex languages with more number of types due to inflection like Ger- man, French, Turkish and Quechua also reduces model perplexity, confirming the robustness of the approach with different probability distributions.
7 Conclusions
In the current paper, reported algorithm that processes corpus assessing token frequencies to remove sentences that contain high-frequency tokens, eventually delivering balanced, linguisti- cally correct data set. We have shown that training with a balanced corpus improves the quality of the model, as shown in a significant reduction in per- plexity for English general and domain texts and for five other languages of different morphological complexities. Furthermore, the results also show the quality of the representations for fine-tuning tasks: linguistic prohing tasks, NLP tasks and Su- perGlue tasks also improve the results obtained with a non-balanced, raw dataset, in many cases delivering results that equals the results obtained with models trained with 100M tokens of raw text. In particular, such better results in semantic tasks confirm previous researches that signalled high- frequency tokens distorting the semantic space cre- ated by the models.
The improvement of rare tokens modeling after climinating fighffrequency tokens is clearly shown in the BLIMP results, which show better perfor- mance for model trained with balanced 10M corpus than model trained with 100M words corpus for tasks related to medium or rare words like Negative Polarity Items (NPT) like 'ever' or Control/Raising verbs like 'oblige' or 'promise'. Note that negative sentences are minority in En- glish written texts ranging from less than 10% to 32% of the sentences in different reference cor- pora (Jiménez-Zafra et al., 2020). Also note that Filler-ap phenomena mostly occur question sentences that are less frequent in any corpus of English too (see, for instance, Liu et al. (2022)).
Moreover, since figh-freeuency words seem to responsible for representations with outlier com- ponents distorting the geometry of the created hy- perspace, semantic fine-tuned tasks should improve more than others that do not rely on semantic simi- larity. For instance, we see an important improve- ment in the Relation Classification task (REL), which predicts the relation that holds between two entities, and also improvement that equals the results of pre-training with 100M in the task of Word in Context (WiC)- word sense disambigua- tion task. Therefore, our results support Fuster- Baggetto and Fresno (2022) because we found that the lack of frequency bias improves the quality of
Summing up, the contributions of our research are the algorithm for balancing textual data, the different balanced corpora that are available with the code at ANONYMIZED, and further evidence about the impact of high-frequency tokens for train- ing transformed-based LMs. Finally, we think that our findings will be of great interest for LRLs, since have defined the characteristics of the data be gathered or created that, using less data, the created models can achieve the best results.
Limitations
We have limited ourselves to studying only four domains other than Wikipedia because the majority

---

of other types of domains are not publicly accessi- ble or do not have the minimum required data for our experiments.
gating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, pages 107-124.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
On the other hand, we have chosen to use Quechua as the language of low-resource settings for being one of the few languages that meets the necessary requirements for our experiments. Quechua is a morphologically complex language (agglutinative), and it has dataset of 6M tokens for training LMs.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCN) Hong Kong, China. Association for Computational Linguistics.
Ethics Statement
The datasets used in this paper for the training and evaluations of the pre-trained models, and fine- tuned models have been extracted from various pre- vious articles and open-access repositories. There- fore, we abide by the ethical rules by citing the orig- inal authors of each dataset. In addition, we encour- age authors who use the resources in this article to cite the original sources. Finally, it is important to mention that one of the authors is an active mem- ber of various initiatives and non-governmental organizations (NGO) aimed at the preservation and revitalization of minority languages. As a result, this provides a deeper understanding of the issues and concerns surrounding minority languages.
pages 55 55-65,
Alejandro Tuster-Baggetee and Victor Fresno. 2022. Is anisotropy really the cause of BERT embeddings not being semantic? In Findings of the Asvociation for Computational Linguistics: EMNLP 2022, pages 4271-4281, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Jun Gao, Di Hc, Xu Tan, Tao Qin, Liwei Wang, and Tic- Yan Liu, 2019. Representation degeneration problem in training natural language generation models.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roem- mele. 2012. SemEval-2201 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexicul and Computational Seman- tics Volume I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Eval- uation (SemEval 2012), pages 394-398, Montréal, Canada. Association for Computational Linguistics.
Acknowledgements
This research was partially funded by the project LUTEST, Project 1199019-104512BBB terio de Ciencia, Innovación y Universidades and Agencia Estatal de Investigación (Spain). The first author has been supported by a FI grant of the Cata- lan Funding Agency for Research and Universities (AGAUR).
Minis-
Ting Jiang, Jian Jiao, Shaohan Huang. Zihan Zhang. Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang, 2022. Prompt- BERT: Improving BERT sentence embeddings with prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8826-8837, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
References
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2924-2936, Minncapolis, Min- nesota. Association for Computational Linguistics.
Salud María Jiménez-Zafra, Roser Morante. María Teresa Martín- Valdivia, and L. Alfonso Ureña-López. 2020. Corpora annotated with negation: An overview. Computational Linguistics. 46(1):1-52.
Yin Cui, Menglin Jia, Tsung-Yo Lin. Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples.
Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. Are pretrained language models symbolic reasoners over knowledge? arXiv preprint arXiv:2006.1 10413.
Marie- Cathe-Ceee De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The comnitmenthank: Investi-
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. BERT busters: Outlier

---

dimensions that disrupt transformers. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 3392-3405, Online. Association for Computational Linguistics.
Findings of the Association for Computational Lin- guistics: EMNLP 2022. pages -40-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Yingtong Liu, Rachel Ryskin, Richard Futrell, and Ed- ward Gibson. 2022. A frame frequency account of constraints long-distance dependencies in en- glish. Cognition, 222:104902.
Julian Salazar, Davis Liang, Toan Q Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguisties.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1900.11699
David Samuel, Andrey Kutuzov, Lilja Øvrelid, and Erik Velldal. 2023. Trained on 100 million words and still in shape: BERT meets British National Corpus. Findings of the Association for Computational Lin- guistics: EACL 2023, pages 1954-1974, Dubrovnik, Croatia. Association for Computational Linguisties.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Dan Weld. 2019. S2orc: The seman- tic scholar open research corpus. arXiv preprint arXiv:191..02782.
Raphael Scheible, Fabian Thomezyk, Patric Tippmann, Victor Jaravine, and Martin Boeker. 2020. Gotthert: pure German language model. arXiv preprint 1rXiv:2012.02110.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Ville- monte De La Clergerie, Djamé Seddah, and Benoit Sagot. 2020. Camembert: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203-7219.
Rico Sennrich, Barry Haddow, and Alexandra Birch, 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07900 07909.
Natalia Silveira, Timothy Dozat, Marie-Catherine De Marneffe, Samuel R Bowman, Miriam Connor, John Bauer, and Christopher D Manning. 2014. A gold standard dependency corpus for english. In LREC, pages 2897-2901 2904. Citeseer.
Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for process- ing huge corpora on medium to low-resource infras- Chal-
tructures. In Proceedings of the Workshop lenges in the Management Large Corpora (CMLC- 7) 2019. Cardiff. 22nd July 2019, pages 9 - 16. Mannheim. Leibniz-Institut für Deutsche Sprache.
Ian Tenney, Dipanjan Das, and Ellic Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Mohammad Taher Pilehvar and Jose Camacho- Coolados. 2019. WIC: the word-in-context dataset for evalu- ating sonsexe- meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinuç. and Oguzhan Ozcelik. 2022. Impact of tokenization on language models: An analysis for Turkish. arXiv preprint 220.::2202408832
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose purpose language under- standing systems. Advances in neural information processing systems, 32.
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell' Orletta. 2022. Outlier dimensions that disrupt transformers are driven by frequency. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 1286-1304, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey. Wei Peng. Sheng-Fu Wang, and Samuel R Bowman. 2020a. Blimp: The benchmark of linguis- tic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377- 392.
Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. 2020, On long-tailed phenomena in neural machine translation. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pages 3088-3095, Online. Association for Computa- tional Linguistics.
Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020b. Learning which features matter: RoBERTa acquires preference for linguistic generalizations (eventually). In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In

---

217-235, Online, Association for Computational Lin- guistics.
A Appendices
A.1 Language Models details
Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 232-948. Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
To train all LMs, we followed the choices by Warstadt et al. (2020b) for their RoBERTa Med- Small model with 45M parameters. Table 8 show the hyperparameters used in our experiments.
Description
Value
Number of Layers
6
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ni- anwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, and et 2013. Ontonotes release 1dc2013t19. Linguistic Data Consortium, Philadel- phia, PA.
Hidden size
512
Feed-forward network dimension
2048
Attention heads
8
Dropout
0.1
Attention dropout
0.1
Yuanmeng Yan, Rumei Li. Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A con- trastive framework for self-supervised sentence repre- sentation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065-5075, Online. Association for Computational Linguistics.
Max Steps
10K
Learning rate decrease
5E-4
Batch Size
512
Number of parameters
45M
Table 8: Configuration used for all RoBERTa model trainings
We ran all training on 5 server equipped with an Intel Xeon E5-2650 v4 CPU (12 cores, 2.2GHz 30MB Cache 2400MHZ 105W) and Gigabyte Geforce GTX 1080 Ti TURBO 11.72GB GPU. We trained each model for 10k steps. The training time was over 4 days for models of 10M and 15 hours for models of 1M. The entire LMs creation experiment took approximately 8 days.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Advances in neural information processing systems, 32.
Rodolfo Zevallos and Núria Bel. 2023. Quantified repro- ducibility assessment of NLP results. In Proceedings of the 61th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), Toronto, Canada. Association for Computational Lin- guistics.
A.2 NLP downstream tasks details
From the pretrained RoBERTa models, and still following Zevallos and Bel (2023), we generated representations of the token span and trained clas- sifiers that predict whether a given label correctly describes the input span for NER, POS, Depen- dency Labeling (DL) and Relation Classification (REL).
Rodolfo Zevallos, John Ortega, William Chen, Richard Castro, Núria Bel, Cesar Toshio, Renzo Venturas, Hilario Aradiel, and Nelsi Melgarejo. 2022. Intro- ducing QuBERT: A large monolingual corpus and BERT model for Southern Quechua. In Proceedings of the Third Workshop on Deep Learning for Low- Resource Natural Language Processing, pages 1-13, Hybrid. Association for Computational Linguistics.
In order to obtain the best and validated results in all tasks, we performed a 10-fold macro-Fl score cross-validation. We set a batch size of 16 and a learning rate 2E-5 in all our fine-tuning models. Furthermore, for evaluating the four tasks, we used macro-Fl score.
Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R Bowman. 2020. When do you need bil- lions of words of pretraining data? arXiv preprint arXiv:2011.04946
Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, and Dan Jurafsky. 2022. Problems with cosine measure of embedding similarity for high frequency words. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Short Pupers), pages 401-423 Dublin, Ireland. Association for Computational Linguistics.
On the other hand, have used I server equipped with Intel Xeon E5-2650 v4 CPU (12 cores, 2.2GHz, 30MB Cache, 2400MHZ, 105W) and Gigabyte GeForce GTX 1080 Ti TURBO 11.72GB GPU to train all these models. The train- ing time was approximately 5 hours per task and

---

model. In total, we trained 12 models over period of approximately days.
A.3 BLiMP and SuperGlue details
The tasks of BliMP and SuperGlue are described in Table and Table 10 respectively.

---

Task
Fullname
Description
the requirement that reflexive pronouns like himself' (a.k.a. anaphora) agree with their antecedents in person, number, gender, and animacy.
Anaphor agreement
ANA. AGR.
The ability of different verbs to appear with different types of arguments. For instance, different verbs can appear with direct object, participate in the causative alternation, or take an inanimate argument.
Argument structure
ARG. STR
The structural relationship between a pronoun and its antecedent.
Binding
Binding
Syntactic and semantic differences between various types of predicates that embed an infinitival VP. This includes control, raising, and tough- predicates.
CTRL RAIS
Control/raising
Determiner-noun
Number agreement between demonstrative determiners (e.g., this/these) and the associated noun.
D-N ARG.
agreement
The possibiliity of omitting expressions from sentence. Because this difficult illustrate with sentences of equal length, our paradigms cover only special cases of noun phrase ellipsis that meet this constraint. Dependencies arising from phrasal movement in, for example, wh-questions.
Ellipsis
Ellipsis
Filter GAP
Filler-gap
Irregular morphology on English past participles (e.g., broken). We are unable to evaluate models on nonexistent forms like #breaked because such forms out of the vocabulary for some LMs.
Irregular
Irregular forms
Restrictions on syntactic environments where the gap in filler-gap dependency may oceur.
Island
Island effects
Restrictions on the distribution of negative polarity items like any and ever limited to, for example, the scope of negation and only.
NPI
NPT licensing
Restrictions on the distribution of quantifiers. We cover two such restrictions: superlative quantifiers (e.g., at least) cannot embed under negation, and definite quantifiers and determiners cannot be subjects in existential-there constructions.
Quantifiers
Quantifiers
Subject-verb agreement
S-V ARG.
Subjects and present tense verbs must agree in number.
Table 9: Description of each BLiMP task used in our experiments.

---

Task
Fullname
Description
BQ, is a QA task where cach example consists of short passage and yes/no question about the passage. The questions were provided by users of Google SE, and afterwards paired with a paragraph from a Wikipedia article containing the answer. CB is a corpus of short texts in which at least one sentence contains an embedded clause. The embedded clause is annotated with the degree of commitment expressed by the author of the sentence, and the task is to predict it. COPA a causal reasoning task that chooses the cause or effect of a premise sentence from two possible choices.
Boolean Questions (Clark al., 2019)
BoolQ
CommitmentBank (De Marneffe et ul., 2019)
CB
Choice of Plausible Alternatives (Gordon et al., 2012)
CОРA
RTE task is about text entailment derived from the Natural Language Inference (NLI) dataset in which given a premise sentence and hypothesis sentence, the task is to predict whether the premise entails the hypothesis not.
Recognizing Textual Entailment (Wang et al., 2019)
RTE
WiC is kind of disambiguation task. Given two text snippets and polysemous word that appears in both sentence, the task is to decide whether the polysemous word is used with the same sense in both sentences.
Word in Context
WiC
(Pilehvar and Camacho-Collados, 2019)
Table 10: Description of each SuperGLUE task used in our experiments.


Reviews:
[{"review_id": "3EW5iboILN", "reviewer": "Reviewer_gnCA", "summary": "", "questions": "Have you tried varying the size of the filtered datasets to get a sense of the range at which is the proposed filtering approach the most effective?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear R1,\n\nWe appreciate very much your positive opinion about our line of research. Indeed, it is crucial to find out: 1. that LM can also deliver good enough results for languages that do not count on very large pre-training datasets, 2. to understand what are the characteristics of optimal pre-training data sets maybe to make design guidelines to support low-resource languages or domains, and 3., last but not least, to be the first ones to set up the basis to empirically confirm that high frequency tokens distort the semantic space and harms similarity-based tasks.\n\nWe have investigated other filtering methods and mostly learned from Fuster-Bagetto and Fresno (EMNLP 2022) results, as reported in section 2.  Note that since we are addressing most frequent words, only deleting them would produce rare, non-correct sentences that might not be suitable for the model that is assessing the probabilities of sequences of words. \n\nAs for your question:\n\n1. Have you tried varying the size of the filtered datasets to get a sense of the range at which is the proposed filtering approach the most effective?\n\nWe have not done it in a systematic way, but only for Quechua, which gets a filtered dataset of 1M. Note that our ultimate goal is not the filtering method, but to understand the importance of having a word-frequency-balanced dataset when working with small datasets, and we take the 10M size as a minimal, critical amount for learning as demonstrated in Zhang et al. 2020. We leave as future work a more systematic comparison of the size of the filtered datasets for the other languages to better know the optimal range.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The authors propose a simple method to filter a large corpora down by on average around 4x by removing sentences containing frequent tokens.\n2) Reducing the required training data, and thus computation, of language model pretraining is an important area of research.\n3) Authors demonstrated the efficacy of their approach on a comprehensive list of tasks and probes.\n4) The authors intend to publish the datasets and code.\n5) Idea and writing was easy to follow.", "reasons_to_reject": "1) The explored design space of filtering algorithms was limited.", "questions_to_authors": "Have you tried varying the size of the filtered datasets to get a sense of the range at which is the proposed filtering approach the most effective?", "paper_topic_and_main_contributions": "This paper proposes to reduce the amount of data needed to achieve good language modeling perplexity and downstream task performance, especially that on semantic tasks, by removing sentences consisting entirely of high frequency tokens.\n\nList of contributions: new data resources, NLP engineering experiment, publicly available software (upon publication).", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Tables in general: I might consider using underline or bolding rather than red color to indicate emphasis for readability."}, "extra_text": {}}, {"review_id": "LLW4TXil9w", "reviewer": "Reviewer_oSA3", "summary": "", "questions": "- How is perplexity calculated? Is it on a held-out dev set?\n- What's the vocabulary? It says 52k, but is it the same vocabulary for all models?\n- What happens if you apply your method to 100M? Would it improve over the 100M baseline?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear R2, \n\nThanks a lot for your positive comments. Indeed we are interested in how to prevent a masked language model (Roberta) from focusing too much on high frequency tokens and ignoring low-frequency ones. As you suggest, it seems to us that brute force could solve many problems by raising the number of samples for all tokens that would no longer be ignored. However, this approach seems unsustainable, as there are no massive amounts of data for all languages, or  even all domains.\n\nAs for your questions:\n\n1. How is perplexity calculated? Is it on a held-out dev set?\n\nWe used the code provided in https://github.com/facebookresearch/fairseq to compute perplexity, in the same way as Liu et al (2019). The dev set is a held out 10% of the dataset.\n\n2. What's the vocabulary? It says 52k, but is it the same vocabulary for all models?\n\nAll the experiments used BPE as tokenizer, and the vocabulary size was set at 52K in all cases. It is true that the perplexity could be affected if the different models have different vocabularies. After your comment, we have compared the words in the vocabularies of the 3 different English models used for the experiments. Percentage of coincidences is in this table. As you can see, the percentage of coincidences is almost 100% for the two models working with a corpus of 10M. \n\n_Table 1: Comparison of vocabulary tokens that share between datasets._\n\n| Datasets     | **10M** | **10M-Freq** | **100M** |\n|--------------|:-------:|:------------:|:--------:|\n| **10M**      |    X    |     99.6%    |   97.1%  |\n| **10M-Freq** |  99.6%  |       X      |   97.8%  |\n| **100M**     |  97.1%  |     97.8%    |     X    |\n\n3. What happens if you apply your method to 100M? Would it improve over the 100M baseline?\n\nFor computational reasons, and with an approach to work with low-resourced languages, we focussed on our experiments up to 10M words. In the future, however, we plan to extend the parameters of the experiments, and to analyze to what extent the number of samples of medium to low-frequency words, that would be more in a larger corpus, could affect the balancing effects of the algorithm, and therefore, the performance of the models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting idea that attempts to deal with power-law issues that may imply we as a field are brute-forcing in a non-intelligent way. The results seem promising - though I have some questions for the authors.", "reasons_to_reject": "- I'm a bit unclear on the experimental setup for the perplexity experiments. Are the vocabularies the same? This can impact perplexity calculations? Was a held out dev set used? 5.1.1 doesn't appear to be, but maybe 5.1.2 (different domains are)? \n- Generalization beyond this model architecture and data size (though the experiments are rather thorough)", "questions_to_authors": "- How is perplexity calculated? Is it on a held-out dev set?\n- What's the vocabulary? It says 52k, but is it the same vocabulary for all models?\n- What happens if you apply your method to 100M? Would it improve over the 100M baseline?", "paper_topic_and_main_contributions": "I've updated my score. However, I really think that the authors need to include the information about the overlap in vocab terms in the paper. It makes a big difference to a reader on comparing perplexities. I would discount the paper without it.\n\n```````````````````````\n\nThis paper looks at what happens when data is preprocessed to remove sentences that have high frequency terms. The overall idea is to prevent a masked language model (Roberta) from focusing too much on high frequency tokens and instead to focus on low-frequency. Experiments were run on a large set of tasks – though mostly classification based and perplexity. Experiments were run on a large set of tasks – though mostly classification based and perplexity. Overall, the authors look at 5 languages including English and showed that performance improved over the baseline 10M, while approaching the 100M.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "3.1 first time language is mentioned ... it feels like the intro was edited to be more general and then in section 3 the reader is supposed to know all the languages\n\nIn the conclusion, the phrase \"linguistically correct\" is a verrrrry strong claim that is likely not justified ...."}}, {"review_id": "kSofVh7ilv", "reviewer": "Reviewer_VqbC", "summary": "", "questions": "A. How does align the results for roberta-base-100M and roberta-base-10M, that was taken with previous works (i.e. BLiMP, English Wikipedia)? It seems the different sampling strategies could lead to different results.\nB. What are the results of training on the full dataset used for obtaining the 10M-freq dataset? I.e. on 36M-wiki and others. For competitive comparison, I suggest including it.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear R3, \n\nWe really appreciate your detailed comments and contributions to improve our paper.  We would like you to note that, trying to clarify your following comment “It suggests a quite simple and effective method to reduce the dataset size. With the suggested approach it seems reasonable to achieve better downstream results with a smaller dataset.”. Our aim was to understand  the impact of high-frequent words, not of the dataset size, although it is a side-effect. Indeed, there are other algorithms that just try to reduce the size, but not paying attention to what in a corpus is less efficient, which after our results are high-frequent words. Our results demonstrate that for equal size corpora, the one with balanced frequency delivers better results. \n\nAlthough it is not one of your questions, we would also like to clarify what is the setup that could benefit from the findings of our research as you expressed doubts about it. We think that for low-resource languages (or for domains) that need to plan how to develop larger pre-training datasets it could be of paramount importance not to try to wildly collect as much data as possible, but to design a balanced dataset.\n\nAs for your questions:\n\n1. How does align the results for roberta-base-100M and roberta-base-10M, that was taken with previous works (i.e. BLiMP, English Wikipedia)? It seems the different sampling strategies could lead to different results.\n\nWe took results for English, and when possible, from previous publications to save resources. However, we would like to note that we are working with subsets, like Wardstat et al. 2020. We would also like to draw your attention to the fact that we are interested in getting evidence about the impact of having a balanced pre-training dataset vs. raw dataset of the same amount of data. This is why we keep comparing with other 10M corpora, and then with a 100M to see the actual impact of the improvement.\n\n2. What are the results of training on the full dataset used for obtaining the 10M-freq dataset? I.e. on 36M-wiki and others. For competitive comparison, I suggest including it.\n\nWe have not compared the balance dataset with the raw-source dataset because the main focus of the paper was to demonstrate that high frequency words harm the modeling, and that balancing word frequency benefits the final results for two equal size datasets. But we agree that the comparison you suggest would also be interesting. We will include it if the paper is accepted, or anycase in the future versions as we plan to extend our experiments and send the results to a journal paper where all these details could be added", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The suggested approach is quite easy to implement and can lead to better results with smaller datasets.", "reasons_to_reject": "1)  Need to evaluate the obtained results with full datasets used for obtaining 10M-freq. Not clear whether the baseline comparison is correct (whether were sampled same data)\n2) The suggested approach is quite straight-forward and there is no specification in what setups it could be beneficial", "questions_to_authors": "A. How does align the results for roberta-base-100M and roberta-base-10M, that was taken with previous works (i.e. BLiMP, English Wikipedia)? It seems the different sampling strategies could lead to different results.\nB. What are the results of training on the full dataset used for obtaining the 10M-freq dataset? I.e. on 36M-wiki and others. For competitive comparison, I suggest including it.", "paper_topic_and_main_contributions": "The paper presents the analysis of how the balanced datasets could help to achieve similar results on downstream tasks with smaller dataset sizes. In line with previous research [1], the authors show that smaller datasets are enough because they take into consideration most syntactic and semantic features. The paper introduces the method for cutting the sentences with the most frequent words (including bigrams) and compares the obtained corpus of 10M words on different downstream tasks: English Wiki, PoS, NER, BLiMP, several tasks from SuperGLUE, and others. Also, several multilingual comparisons are done for German, French, Turkish, and low-resource Quechua languages. Almost for all benchmarks the obtained 10M-freq dataset overperforms 10M dataset.  All experiments are conducted with RoBERTa backbone.\n\nThe paper is well-written and effectively conveys the main idea. It suggests a quite simple and effective method to reduce the dataset size. With the suggested approach it seems reasonable to achieve better downstream results with a smaller dataset.\n\nHowever, I have a few unclear points, that I want to discuss. First, it seems that the results for roberta-base-100M and roberta-base-10M are taken from previous works (i.e. BLiMP, English Wikipedia), and different sampling strategies could lead to different results. Second. I am not sure that a direct compare 10M and 10m-freq is the correct setup. As far as 10M-freq usually is obtained from 3x-4x larger number of data I suggest providing the comparison of 10M-freq with full data, which was used to obtain smaller corpus.\n\nStrengths:\n\n1) The suggested approach is quite easy to implement and can lead to better results with smaller datasets.\n\nWeaknesses:\n\n1)  Need to evaluate the obtained results with full datasets used for obtaining 10M-freq. Not clear whether the baseline comparison is correct (whether were sampled same data)\n2) The suggested approach is quite straight-forward and there is no specification in what setups it could be beneficial \n\n[1] [When Do You Need Billions of Words of Pretraining Data?](https://aclanthology.org/2021.acl-long.90) (Zhang et al., ACL-IJCNLP 2021)", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Suggest to rename the misleading tokens to words to align with previous works."}, "extra_text": {}}, {"review_id": "G2uHyG2Byv", "reviewer": "Reviewer_zg87", "summary": "", "questions": "Questions:\n- Other than stop words and split by whitespace are punctuations removed?\n- Is the perplexity scores calculated based on 100M corpus even though it’s trained on 10M or is the model trained on 10M measured on the 10M? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear R4, \n\nThanks for your encouraging comments. Before answering your questions we would like, for clarification, comment on the weaknesses you mention in the review. \n\nWe are also very curious about how to explain that perplexity improvements do not improve the results in downstream tasks for English, and to get evidence of these tasks for other languages. This is one of our future lines of research, and, if accepted, we will highlight it in the conclusions and limitations.\n\nWe have mentioned other filtering methods that address, as we do, high-frequent words. Fuster-Baggetto & Fresno (2022) are the only ones that evaluate an approach to get rid of frequent words, but not balancing the data-set: just removing embeddings, mostly of very frequent stop words. To the best of our knowledge, there are no other experiments comparable to ours. Therefore, it is not possible to provide a comparison with other filtering algorithms in this case. Also note that our results support other’s results related to balanced datasets: Samuel et al (2023) found that a noticeable difference in downstream task performance was achieve used the 100M BNC corpus, that it is not balanced with respect high frequency words, but it is a curated corpus. \n\n\nAs for your questions:\n\n1. Other than stop words and split by whitespace are punctuations removed?\n\nAll the models in our experiments have been trained with their respective punctuation marks. However, in the frequency count phase, we removed the punctuation marks, since they are not relevant both in frequency term and co-occurrence.\n\n2. Is the perplexity scores calculated based on 100M corpus even though it’s trained on 10M or is the model trained on 10M measured on the 10M?\n\nThe perplexity of each model is based on the size of the training set. For instance, if we train a model of 10M in English, the dev set for the perplexity is 1M. Both datasets (train and dev) are taken from the 100M of the English corpus.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Simple approach, clear writing. A marginal accept, 3.5 out of 5 score for accepting the paper. More details, see strength above.", "reasons_to_reject": "Fundamentally, the paper is sound but a little extra work (in suggestions above) would make the paper a lot better. Would prefer to read the paper if the suggestions are incorporated in revision / future re-work of the paper. See weakness above for details.", "questions_to_authors": "Questions:\n- Other than stop words and split by whitespace are punctuations removed?\n- Is the perplexity scores calculated based on 100M corpus even though it’s trained on 10M or is the model trained on 10M measured on the 10M? ", "paper_topic_and_main_contributions": "\nSummary: This paper presents a method to select/sample data for language modelling (LM) by removing sentences with sentences with high frequency tokens, hypothesizing that resulting corpus will be a more balance corpus. \n\nHighlights:\n\n- [Competitive Results with less data] On the intrinsic LM metric is the achievement of comparable LM perplexity with 10x less data on the English wikipedia \n- [SOTA Results] Across various domains, for English, models trained with proposed data filtering method achieved lower perplexity\n- [SOTA Results] Comparatively, for non-English other languages chosen in the experiments, models trained with filtered data using the proposed algorithm achieved lower perplexity\n- On downstream superglue tasks, their proposed data filtering method improved model’s performance with comparable corpus size but the model trained on 10x corpus size outperforms the model trained on filtered data\n- On downstream blimp tasks, their proposed data filtering method achieved competitive performance as compared to models trained on 10x corpus size\n\n\nStrength:\n\n- Simple proposal to reduce corpus size to achieve LM perplexity compared to models trained on 10x data size\n- Achieved competitive results in downstream tasks that test the grammatical integrity of the language model (blimp)\n\nWeakness:\n\n- While the proposed method is compared to model trained with no filtering, it would be good to at least see the comparison of other filtering methods, e.g. simple perplexity filtering (filter low/high perplexity items based on n-gram / other pre-tained language model).\n- Downstream tasks were all English centric, it’s hard to determine the efficacy of the proposed method in non-English language esp. when the perplexity improvements don’t significantly improve downstream tasks for English\n\n\nSuggestions:\n- Use bold in tables, avoid using red in tables (or generally colours on paper to accommodate colour-blind readers)\n- To improve soundness/technical strength score, a post-hoc (after tokenised training) analysis should be done to see coverage of the tokens for low-frequency work, e.g. section 2 from https://aclanthology.org/2023.americasnlp-1.18\n- To improve your “excitement” score, capitalise on the chosen language families Quechua and Turkish (both agglutinative), if you have a specific downstream task that test generation/labelling/annotation for agglutination that’ll make the paper very strong accept (even if the model don’t perform well, you’ve explored the limitation of the model)\n- To make it a paper even more \"exciting\", add machine translation to the mix of downstream task, even though the model is trained monolingual, fine-tuning the model with a small corpus in the languages available in the pre-trained LM would show how well a user can capitalise on the “balancedness” of the corpus the paper suggested to create better/worse cross lingual information (shown by proxy with MT scores)", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Maybe refer to this https://arxiv.org/pdf/2102.09690.pdf and compare some works on the \"common token bias\" in the Related Works section of this paper. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to reduce the amount of data needed to achieve good language modeling perplexity and downstream task performance, especially that on semantic tasks, by removing sentences consisting entirely of high frequency tokens.

Reasons To Accept:
- The authors propose a simple method to filter a large corpora down by on average around 4x by removing sentences containing frequent tokens.
- Reducing the required training data, and thus computation, of language model pretraining is an important area of research.
- Authors demonstrated the efficacy of their approach on a comprehensive list of tasks and probes.
- The authors intend to publish the datasets and code.
- Idea and writing was easy to follow.
- This is an interesting idea that attempts to deal with power-law issues that may imply we as a field are brute-forcing in a non-intelligent way.

Reasons To Reject:
- The explored design space of filtering algorithms was limited.
- Need to evaluate the obtained results with full datasets used for obtaining 10M-freq. Not clear whether the baseline comparison is correct (whether were sampled same data)
- The suggested approach is quite straight-forward and there is no specification in what setups it could be beneficial
Decision: Accept-Findings