# eSCorpius-m: A Massive Multilingual Crawling Corpus with a Focus on Spanish
## 4 Data Download and Cleaning Process
### Common Crawl Repository  
Common Crawl is a web archive that contains petabytes (see [[https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlsize](https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlsize)]([https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlsize](https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlsize)) (accessed on 15 September 2023) to explore current crawling sizes) of data collected since 2008. The repository contains raw web page data in the WARC file format, request/response metadata files in WAT format, and text data contents extracted from WARC and stored in WET files. Some corpora such as mC4 use WET files to generate the corpus content. However, this has several disadvantages. The main problem with WET is that the process followed \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline  & **OSCAR 22.01 [43]** & **mC4 [36]** & **CC-100 [44]** & **ROOTS [39]** & **ParaCrawl v9 [45]** & **esCorpius-m (Ours)** \\ \hline Size & 5 2 TB & 3.5 TB & 2.0 TB & 1.2 TB & 24 GB & 2.5 TB \\ \hline Dos & 560M & - & - & 521M & & 1427M \\ \hline Words & 381B & 3567B & 284B & 189B & 8B & 242B \\ \hline WARC/WET & WET & WET & WET & WET & WARC & WARC \\ \hline Lang. identification & fastText & CLD3 & fastText & fastText & CLD2 & CLD2 + fastText \\ \hline Content identification & WET heuristic & WET heuristic & WET heuristic & Sentence Alignment & JustText (modified) \\ \hline Elements & Document & Document & Document & Document & Sentence & Document and paragraph \\ \hline Parsing quality & Medium & Low & Medium & High & High & High \\ \hline Cleaning quality & Low & No cleaning & Low & High & High & High \\ \hline Deduplication & No & No & No & SimHash + SupfikArray & Bicleaner & dLIFF \\ \hline If parallel & - & - & - & - & ✓ & - \\ \hline Traceability & URL & URL & URL & URL & URL & URL + WARC \\ \hline Licence & CC BY 4.0 & OCD-BY 1.0 & Common Crawl & CC-BY-SA-4.0 & CC0 & CC-BY-NC-ND 4.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of the main state-of-the-art multilingual corpora (excluding English excerpts). to remove HTML tags and extract the text is error-prone. After HTML tags are removed, frequently the text is either divided into unconnected pieces or merged with other unrelated textual information. For example, the sentence \"We offer <b>fast</b> transportation\" could be divided into [\"We offer\", \"fast\", and \"transportation\"], thus loosing the integrity of the original text. Additionally, the text extractions from WET files are HTML-content agnostic in the sense it is not checked whether the content belongs to less relevant parts of the web (e.g., a menu or a footer). Hence, for our work, we decided to use the original WARC files as some transformations made for creating the WETs are irreversible or, at least, very costly to repair. A quick look at the same example in WARC and WET format shows this difference (same example in WET: [[https://gist.githubusercontent.com/Smerity/e750f0](https://gist.githubusercontent.com/Smerity/e750f0) ef0 ab9 aa366558/raw/313b675a01 ee1d1f05829439165a9 eb991571547/bbc.wet]([https://gist.githubusercontent.com/Smerity/e750f0](https://gist.githubusercontent.com/Smerity/e750f0) ef0 ab9 aa366558/raw/313b675a01 ee1d1f05829439165a9 eb991571547/bbc.wet) (accessed on 15 September 2023) and WARC: [[https://gist.githubusercontent.com/Smerity/e750f0](https://gist.githubusercontent.com/Smerity/e750f0) ef0 ab9 aa366558/raw/313b675]([https://gist.githubusercontent.com/Smerity/e750f0](https://gist.githubusercontent.com/Smerity/e750f0) ef0 ab9 aa366558/raw/313b675) a01 ee1d1f05829439165a9 eb991571547/bbc.warc (accessed on 15 September 2023) formats). The reason why the related work utilizes WET [29; 46] is that in WET format there is a close relation between the file size and the amount of textual data it contains: nearly 100% of the contents of WET files is pure text, thus optimizing the resources invested in processing the files.