Are All Languages Equal? Curriculum Learning over Different Languages
Authors: Giulia Pucci, Leonardo Ranaldi, Fabio Massimo Zanzotto (Università di Roma Tor Vergata, Idiap Research Institute, 2023)
Topic: Language Models

Abstract
Curriculum Learning (CL) is emerging as a relevant technique to reduce the cost of pre-training Large Language Models. The idea, tested for the English language, is to train LLMs by organizing training examples from the simplest to the most complex. Complexity measures may depend on the specific language. Hence, this paper aims to investigate whether CL and the complexity measure can be easily exported to other languages. For this reason, we present a set of linguistically motivated measures to determine the complexity of examples, which has been used in English: these measures are based on text length, rarity, and comprehensibility. We then test the approach on two Romance languages: Italian and French. Our results show that the technique can be easily exported to languages other than English without adaptation.

Introduction
Transformers-based models have disrupted natural language understanding methods, outperforming previous methods and sometimes even humans in many tasks. Unsupervised learning on huge corpora, regardless of the domain, seems to be the key to increasing performance; however, besides the high costs, there are difficulties with the data. This leads to a significant carbon footprint, which contradicts global sustainability goals. Various approaches are being explored to address the AI carbon footprint problem, ranging from using more carbon-efficient energy sources to applying more efficient AI models and training algorithms. Despite their success, transformers are essentially large memory-based models, necessitating better ways to train them. Bengio et al. proposed Curriculum Learning (CL) as a specific class of efficient training strategies for deep learning models. The naïve approach for training Large Language Models (LLMs), which involves feeding randomly sampled textual batches from the training corpus, is revisited in CL, where the model is refined with a sequence of progressively more challenging examples. This method emulates human learning, starting with simpler concepts and gradually building up more complex ones. Soviany et al. show that CL improves model performance and accelerates convergence. In this paper, we conduct a deep analysis of learning divergencies by training from scratch with BERT and GPT2 on the same corpus in multiple languages. Additionally, we apply our CL-LRC metrics based on length, rarity, and comprehensibility to reduce computational costs and mitigate divergences. Using the same small corpus in three different languages—English (original), Italian, and French (translated)—experimental results show that loss values during training vary across languages. Furthermore, this difference appears to be less pronounced in terms of perplexity scores when pre-training block sizes increase incrementally.

Conclusion
In this paper, we explored the effectiveness of Curriculum Learning (CL) in reducing the cost of pre-training and improving results. We trained Large Language Models (LLMs) by organizing examples from the simplest to the most complex, leveraging complexity measures. Specifically, we pre-trained BERT and GPT2 from scratch using standard baselines and CL approaches. After a thorough analysis, we demonstrated that learning divergence can be mitigated using CL approaches, which were reinforced by measures designed to determine the complexity of examples. These measures, applied during pre-training to sort the corpus based on complexity, yielded outstanding results. While the original approach was tested and validated for the English language, this research aimed to determine whether CL and its associated complexity measure could be applied to other languages without significant adaptation. Experiments conducted in a low-resource environment showed that the proposed method led to better performance, with lower loss during learning and improved perplexity on test data.