# eSCorpius-m: A Massive Multilingual Crawling Corpus with a Focus on Spanish
## 2 State-of-the-Art  
Foundational models refer to large-scale language models that serve as the basis or foundation for various downstream applications and tasks [2; 3; 4]. They have become a fundamental building block for a wide range of AI applications covering natural language understanding (text classification including sentiment analysis, spam detection and topic categorization, named entity recognition, language translation, etc.) [5; 6; 7; 8], text generation (content creation, code generation for programming languages, etc.) [9; 10], question answering, conversational AI [11; 12], language summarization [13; 14], content recommendation and moderation [15; 16], search engines, web pages and documents ranking [17], and data extraction and knowledge graph creation [18; 19]. Application domains include key areas such as education (e.g., assistants, facilitators, virtual tutors and assessment aids [20; 21]), healthcare (medical education, medical histories summarization, patient data analysis, conversational agents for patients, preliminary diagnosis, or clinical skills assessment [22]), legal and compliance (legal research, document generation, legal information providing, legal analysis [23]), e-government (assisted citizen service, interaction with public, FAQs, gathering feedback, accessibility to information, personalized support to groups, data analysis, summaries, automated decision-making, improving cybersecurity [24]), marketing and financial analysis (risk analysis, generating reports, interactive data analysis, generating summaries and financial news briefs, automated customer interaction [25; 26; 27]), and accessibility (text-based communication, multilingual support, 24/7 access, customization and personalization, integration with assistive technologies [28]), among others. These models are initially pre-trained on vast and diverse text corpora to learn large language patterns, grammar, syntax, and world knowledge. This pre-training phase helps the model acquire a broad understanding of language. After pre-training, these models can be fine-tuned on specific datasets or tasks to make them more specialized. Fine-tuning adapts the model's knowledge to perform specific tasks such as text generation, translation, sentiment analysis, and more. Foundational models are designed to generalize well across a wide range of natural language understanding and generation tasks. These models are often very large, with billions or even trillions of parameters, which allows them to capture intricate language nuances and patterns. However, their size also requires significant computational resources for training and inference. Transfer learning from these models has become a standard practice in many NLP applications. Therefore, clean and well-curated text data is crucial for LLMs' training, and their performance hinges on the quality and volume of this data. Modern models are trained on hundreds of billions of tokens, and each developer adopts distinct training methods including their own recipes for creating the datasets [29; 30; 31]. As commercial interests in LLMs grow, there is a trend towards proprietary datasets [32], underscoring the importance of accessible, clean data. Despite the positive results reported in the literature for English pre-trained models, many voices highlight the disparity in the availability and quality of models and data for other languages. As noted in ref. [33], most NLP research is conducted in English, followed by Mandarin Chinese, and at a great distance to other languages, including Spanish despite its large number of speakers around the world. This situation has negative repercussions on the development of fair NLP technology for all [34], e.g., disparate access to clinical NLP for speakers of different languages [35]. To address this issue, language models are being trained on monolingual corpora in different languages.