# eSCorpius-m: A Massive Multilingual Crawling Corpus with a Focus on Spanish
## 3 esCorpius-m at a Glance
\begin{table}
\begin{tabular}{l c c c c c c c} \hline
**Lang** & **000 Words** & **\% Words** & **Par.** & **\% Par.** & **URLs** & **\% URLs** & **Size, GB** \\ \hline Afrikaans & 114,386 & 0.05\% & 248,588 & 0.04\% & 245,361 & 0.04\% & 0.66 \\ Arabic & 4,542,747 & 1.88\% & 11,502,118 & 1.78\% & 11,403,653 & 1.79\% & 49.38 \\ Bengali & 594,746 & 0.25\% & 1,772,168 & 0.27\% & 1,759,258 & 0.28\% & 10.44 \\ Catalan & 1,680,554 & 0.69\% & 4,284,590 & 0.66\% & 4,227,541 & 0.66\% & 10.95 \\ Czech & 5,764,676 & 2.38\% & 16,978,203 & 2.63\% & 16,667,666 & 2.62\% & 44.12 \\ Danish & 3,155,961 & 1.30\% & 7,551,657 & 1.17\% & 7,410,853 & 1.17\% & 21.15 \\ German & 38,899,553 & 1.06\% & 109,705,164 & 16.99\% & 107,831,398 & 16.88\% & 297.11 \\ Greek & 5,151,481 & 2.13\% & 12,042,049 & 1.86\% & 11,923,822 & 1.87\% & 61.87 \\ Spanish & 28,618,504 & 11.81\% & 57,808,392 & 8.95\% & 57,327,087 & 9.01\% & 195.7 \\ Basque & 116,918 & 0.05\% & 40,207,725 & 0.07\% & 414,677 & 0.07\% & 0.99 \\ Persian & 6,712,191 & 2.55\% & 12,631,282 & 1.96\% & 12,475,728 & 1.96\% & 155.13 \\ Finnish & 2,793,619 & 1.15\% & 7,945,694 & 1.23\% & 7,805,202 & 1.23\% & 25.68 \\ French & 40,739,063 & 16.82\% & 94,895,680 & 14.69\% & 92,999,095 & 14.62\% & 298.24 \\ Galician & 223,711 & 0.09\% & 603,020 & 0.09\% & 595,481 & 0.09\% & 1.5 \\ Hindi & 1,326,133 & 0.55\% & 2,735,525 & 0.42\% & 2,714,560 & 0.43\% & 16.93 \\ Croatian & 1,222,123 & 0.50\% & 4,067,720 & 0.63\% & 0.406,201 & 0.63\% & 8.93 \\ Italian & 21,879,938 & 9.03\% & 50,844,721 & 7.87\% & 50,094,971 & 7.88\% & 149.39 \\ Japanese & 913,584 & 0.38\% & 23,856,552 & 3.69\% & 23,593,379 & 3.71\% & 111.23 \\ Korean & 1,711,686 & 0.71\% & 3,810,567 & 0.59\% & 378,029 & 0.59\% & 18.45 \\ Maltsev & 24,340 & 0.01\% & 51,934 & 0.01\% & 514,507 & 0.01\% & 0.2 \\ Dutch & 12,015,439 & 4.96\% & 32,497,299 & 5.03\% & 31,779,475 & 5.00\% & 80.77 \\ Norwegian & 2,361,749 & 0.97\% & 5,725,188 & 0.89\% & 564,580 & 0.89\% & 15.59 \\ Cocitan & 7,371 & 0.00\% & 20,631 & 0.00\% & 20,323,005 & 0.00\% & 0.05 \\ Punjabi & 54,946 & 0.02\% & 104,737 & 0.02\% & 103,879 & 0.02\% & 0.7 \\ Polish & 11,731,521 & 4.84\% & 33,017,904 & 5.11\% & 32,436,496 & 5.10\% & 92.34 \\ Portuguese & 22,577,860 & 9.32\% & 50,055,128 & 7.75\% & 49,535,350 & 7.79\% & 149.34 \\ Romanian & 5,351,974 & 2.21\% & 11,995,288 & 1.86\% & 11,851,961 & 1.86\% & 36.29 \\ Slovenian & 902,213 & 0.37\% & 2,296,724 & 0.36\% & 2,262,941 & 0.36\% & 6.36 \\ Serbian & 445,014 & 0.18\% & 907,411 & 0.14\% & 894,739 & 0.14\% & 5.2 \\ Swedish & 6,135,501 & 2.53\% & 14,777,204 & 2.29\% & 14,528,724 & 2.28\% & 48.93 \\ Turkish & 6,393,599 & 2.64\% & 21,232,882 & 3.29\% & 21,023,030 & 3.31\% & 56.65 \\ Ukrainian & 3,483,220 & 1.44\% & 9,066,207 & 1.40\% & 8,958,138 & 1.41\% & 46.51 \\ Urdu & 335,09 & 0.14\% & 568,139 & 0.09\% & 563,963 & 0.09\% & 2.81 \\ Chinese & 4,806,600 & 1.98\% & 39,750,371 & 6.16\% & 39,533,316 & 6.22\% & 504.62 \\ \hline
**Total** & **242,248,582** & **100.00\%** & **645,772,362** & **100.00\%** & **636,047,388** & **100.00\%** & **2524.24** \\ \hline \end{tabular}
\end{table} 
Table 1: Data per language. Figure 1: Distribution of the languages in esCorpius-m by the amount of words. The enhancements incorporated in esCorpius-m compared to other similar, large, crawled corpora can be summarized as follows:  
1. Contrary to most other corpora--with the exception of ParaCrawl [45]--our content is extracted directly from WARC files. This method is notably more reliable than sourcing from the frequently error-prone WET files. 2. Language identification is a two-step process: first, using the less computationally-intensive cld2, followed by fastText for paragraph candidates. The latter is recognized for its superior quality (As can be demonstrated in [[https://modelpredict.com/language-identification-survey](https://modelpredict.com/language-identification-survey)]([https://modelpredict.com/language-identification-survey](https://modelpredict.com/language-identification-survey)), accessed on 15 September 2023).