How far does the sequence of compositions impact Multilingual Pre-Training?
Authors: Leonardo Ranaldi, Giulia Pucci, Fabio Massimo Zanzotto (University of Edinburgh, University of Aberdeen, Università di Roma Tor Vergata, 2024)
Topic: Language Models

Abstract
An Efficient strategy for conducting pre-training of language models is the concatenation of contiguous sequences of text of fixed length through causal masking that estimates the probability of each token given its context. Yet earlier work suggests that this technique affects the performance of the model as it might include misleading information from previous text sequences during pre-training. To fill this gap, intra-context and rank-based causal masking techniques have been proposed, in which the probability of each token is conditional only on the previous ones in the same document or ranked sequences, avoiding misleading information from different contexts. However, the sequences provided by the use of these techniques have been little explored, overlooking the opportunity to optimise the composition by manipulating the volume and heterogeneity in the sequences and improving unbalance pre-training settings. In this paper, we demonstrate that organising text chunks based on a policy that aligns with text similarity effectively improve pre-training, enhances the learning and cross-lingual generalisation capabilities of language models, maintains efficiency, and allows for fewer instances. %Our code is publicly released at the following \href{https://github.com/lranaldii/pretraining-order}{link

Introduction
Large language models (LLMs) are pre-trained on huge amounts of documents by optimizing a language modelling objective and show an intriguing ability to solve various downstream NLP tasks. \citet{ranaldi-etal-2023-modeling} in multilingual settings and later \citet{zhao2024analysingimpactsequencecomposition} highlighted the importance of pre-training data quality, diversity and composition methodologies. Our research takes a step further by exploring the influence of the pre-training sequences heterogeneity for cross-lingual generalisation. This potentially leads to significant advancements in understanding LLMs' learning properties. [t] \includegraphics[width=0.96\linewidth]{figures/pipelines.pdf randomly samples documents from all corpora to construct pre-training sequences, which can pack documents from different sources; Sequence-based randomly samples documents from a single source to construct a sequence; Retrieve-based operate via ranking-based construction process. The down block represents a document Collector that caches a set of documents randomly sampled between the corpora. In decoder-only architectures pre-training, the constructions of the instances are based on packing that combines randomly sampled texts (i.e., documents) into a chunk that matches the size of the context window without using any selection policy. Then, the causal masking predicts the next token conditioned on the previous, including those from different documents (portions of non-contiguous texts) in the chunk. The ways to mitigate this arbitrary procedure are: (i) \intraMask AUTHOR, where the likelihood of each token is conditioned on the previous from the same document AUTHOR and retrieval-based masking AUTHOR where similar documents retrieved by retrieval systems condition likelihood. To study the role of heterogeneity and volume of samples in sequence composition strategies (i.e., packing and masking pipelines), we pre-train language models using different masking approaches (described in \S sec:masking) and compare them with models pre-trained via the traditional causal masking with different packing approaches by varying amount of the sequence composition of the documents in the pre-training chunks. Whilst for studying the impact on cross-lingual generalisation we use cross-lingual settings (i.e., Italian English). Complementing the foundation approaches proposed in AUTHOR,we operate via bilingual corpora. Hence, we analyse the results produced by a commonly used baseline method that randomly samples and packs documents (\MixChunk), a process that samples and packs documents from the same source based on their composition and origin (\UniChunk), and then operate via efficient retrieval-based packing method, which retrieves and packs related documents (\S sec:bm25chunk). The experimental results indicate that operating via causal masking (\MixChunk) with arbitrary sequence patterns of documents leads to the inclusion of misleading information that stems from different context during pre-training (\S sec:pretraining), impacting in a negatively the performance of the models in downstream tasks (\S sec:downstream_tasks). Instead, \intraMask, which avoids the misleading phenomena during pre-training, significantly improves the models' performance and does not impact the runtime. Although \intraMask performs well, it limits the operability of sequence composition mixing documents from different corpora (in our case in different languages as well). As revealed by \citet{zhao2024analysingimpactsequencecomposition} as well, this is partly solved by \UniChunk's avoidance of packing documents from different distributions, which improves the performance of causal masking models in downstream tasks but still does not allow individual sequences to be selected. Hence, we use a retrieval-based packing method, which allows operating directly on sequences by improving cross-lingual models' language modeling, in-context learning and generative capabilities by using causal masking and thus paying a small fee for document sorting but achieving tangible results. Our main findings can be summarised as follows: [leftmargin=*,noitemsep,nolistsep] \item By analyzing different pre-trained strategies in cross-lingual settings we reveal that operating through causal masking and considering the order and patterns sequence represented in documents, leads to significant improvements. In addition, retrieval-based techniques provide resilience and allow for the selection of pre-training sequences by guaranteeing heterogeneity and reducing data (\S sec:pretraining). \item We show important benefits on the in-context learning capabilities of downstream models. We observe that in low-resource settings, it is possible to achieve the same performance and in some cases cross-lingual generalisation (in our case, English-Italian) (\S sec:downstream_tasks). \item In conclusion, we show that the retrieval-based packing method allowing for a flexible sequence composition process benefits unbalanced cross-lingual learning tangible benefits by using less pre-training data.

Conclusion
The role of pre-training sampling is a strategic component. We analyse the impact of sequencing by pre-training several language models on multilingual corpora. We showed that causal masking involves misleading documents that confound the pre-training of language models and impact the performance in downstream tasks. Hence, we find that improving sequence correlation in pre-training chunks reduces potential distractions while improving the performance of language models without reducing pre-training efficiency. In the future, we will study whether these findings archive benefits in fine-tuning pipelines AUTHOR as well. \clearpage \appendix