Small changes to README

README.md

@@ -10,7 +10,7 @@ widget:
 ---
 
 - [Version beta](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta): July 15th, 2021
-- Version 1.0: July 26th, 2021
+- Version 1.0 (current): July 26th, 2021
 
 
 # BERTIN
@@ -26,20 +26,19 @@ This is part of the
 
 The aim of this project was to pre-train a RoBERTa-base model from scratch during the Flax/JAX Community Event, in which Google Cloud provided free TPUv3-8 to do the training using Huggingface's Flax implementations of their library.
 
-
 # Motivation
-According to [Wikipedia](https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers), Spanish is the second most-spoken language in the world by native speakers (>470 million speakers, only after Chinese, and the fourth including those who speak it as a second language). However, most NLP research is still mainly available in English. Relevant contributions like BERT, XLNet or GPT2 sometimes take years to be available in Spanish and, when they do, it is often via multilingual versions which are not as performant as the English alternative.
 
-At the time of the event there were no RoBERTa models available in Spanish. Therefore, releasing one such model was the primary goal of our project. During the Flax/JAX Community Event we released a beta version of our model, which was the first in Spanish language. Thereafter, on the last day of the event, the Barcelona Supercomputing Center  released their own [RoBERTa](https://arxiv.org/pdf/2107.07253.pdf) model. The precise timing suggests our work precipitated this publication, and such increase in competition is a desired outcome of our project. We are grateful for their efforts to include BERTIN in their paper, as discussed further below, and recognize the value of their own contribution, which we also acknowledge in our experiments.
+According to [Wikipedia](https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers), Spanish is the second most-spoken language in the world by native speakers (>470 million speakers), only after Chinese, and the fourth including those who speak it as a second language. However, most NLP research is still mainly available in English. Relevant contributions like BERT, XLNet or GPT2 sometimes take years to be available in Spanish and, when they do, it is often via multilingual versions which are not as performant as the English alternative.
 
-Models in Spanish are hard to come by and, when they do, they are often trained on proprietary datasets and with massive resources. In practice, this means that many relevant algorithms and techniques remain exclusive to large technological corporations. This motivates the second goal of our project, which is to bring training of large models like RoBERTa one step closer to smaller groups. We want to explore techniques that make training these architectures easier and faster, thus contributing to the democratization of Deep Learning.
+At the time of the event there were no RoBERTa models available in Spanish. Therefore, releasing one such model was the primary goal of our project. During the Flax/JAX Community Event we released a beta version of our model, which was the first in the Spanish language. Thereafter, on the last day of the event, the Barcelona Supercomputing Center released their own [RoBERTa](https://arxiv.org/pdf/2107.07253.pdf) model. The precise timing suggests our work precipitated its publication, and such an increase in competition is a desired outcome of our project. We are grateful for their efforts to include BERTIN in their paper, as discussed further below, and recognize the value of their own contribution, which we also acknowledge in our experiments.
 
+Models in monolingual Spanish are hard to come by and, when they do, they are often trained on proprietary datasets and with massive resources. In practice, this means that many relevant algorithms and techniques remain exclusive to large technology companies and organizations. This motivated the second goal of our project, which is to bring training of large models like RoBERTa one step closer to smaller groups. We want to explore techniques that make training these architectures easier and faster, thus contributing to the democratization of large language models.
 
 ## Spanish mC4
 
-mC4 is a multilingual variant of the C4, the Colossal, Cleaned version of Common Crawl's web crawl corpus. While C4 was used to train the T5 text-to-text Transformer models, mC4 comprises natural text in 101 languages drawn from the public Common Crawl web-scrape and was used to train mT5, the multilingual version of T5.
+The dataset mC4 is a multilingual variant of the C4, the Colossal, Cleaned version of Common Crawl's web crawl corpus. While C4 was used to train the T5 text-to-text Transformer models, mC4 comprises natural text in 101 languages drawn from the public Common Crawl web-scrape and was used to train mT5, the multilingual version of T5.
 
-The Spanish portion of mC4 (`mc4-es`) contains about 416 million samples and 235 billion words in approximately 1TB of uncompressed data.
+The Spanish portion of mC4 (mC4-es) contains about 416 million samples and 235 billion words in approximately 1TB of uncompressed data.
 
 ```bash
 $ zcat c4/multilingual/c4-es*.tfrecord*.json.gz | wc -l
@@ -53,9 +52,9 @@ $ zcat c4/multilingual/c4-es*.tfrecord-*.json.gz | jq -r '.text | split(" ") | l
 
 ## Perplexity sampling
 
-The large amount of text in mC4-es makes training a language model within the time constraints of the Flax/JAX Community Event by HuggingFace problematic. This motivated the exploration of sampling methods, with the goal of creating a subset of the dataset that allows well-performing training with roughly one eighth of the data (~50M samples) and in approximately half the training steps.
+The large amount of text in mC4-es makes training a language model within the time constraints of the Flax/JAX Community Event problematic. This motivated the exploration of sampling methods, with the goal of creating a subset of the dataset that would allow for the taining of well-performing models with roughly one eighth of the data (~50M samples) and at approximately half the training steps.
 
-In order to efficiently build this subset of data, we decided to leverage a technique we call *perplexity sampling* and its origin can be traced to the construction of CCNet (Wenzek et al., 2020) and their work extracting high quality monolingual datasets from web-crawl data. In their work, they suggest the possibility of applying fast language models trained on high-quality data such as Wikipedia to filter out texts that deviate too much from correct expressions of a language (see Figure 1). They also released Kneser-Ney models for 100 languages (Spanish included) as implemented in the KenLM library (Heafield, 2011) and trained on their respective Wikipedias.
+In order to efficiently build this subset of data, we decided to leverage a technique we call *perplexity sampling*, and whose origin can be traced to the construction of CCNet (Wenzek et al., 2020) and their high quality monolingual datasets from web-crawl data. In their work, they suggest the possibility of applying fast language models trained on high-quality data such as Wikipedia to filter out texts that deviate too much from correct expressions of a language (see Figure 1). They also released Kneser-Ney models (Ney et al., 1994) for 100 languages (Spanish included) as implemented in the KenLM library (Heafield, 2011) and trained on their respective Wikipedias.
 
 <figure>
 
@@ -76,32 +75,32 @@ In order to test our hypothesis, we first calculated the perplexity of each docu
 
 ![](./images/perp-p95.png)
 
-<caption>Figure 2. Perplexity distributions and quartiles (red lines) of 44M samples of mc4-es.</caption>
+<caption>Figure 2. Perplexity distributions and quartiles (red lines) of 44M samples of mC4-es.</caption>
 </figure>
 
 With the extracted perplexity percentiles, we created two functions to oversample the central quartiles with the idea of biasing against samples that are either too small (short, repetitive texts) or too long (potentially poor quality) (see Figure 3).
 
-The first function is a `Stepwise` that simply oversamples the central quartiles using quartile boundaries and a factor for the desired sampling frequency for each quartile, obviously given larger frequencies for middle quartiles (oversampling Q2, Q3, subsampling Q1, Q4). 
+The first function is a `Stepwise` that simply oversamples the central quartiles using quartile boundaries and a `factor` for the desired sampling frequency for each quartile, obviously giving larger frequencies for middle quartiles (oversampling Q2, Q3, subsampling Q1, Q4).
 The second function weighted the perplexity distribution by a Gaussian-like function, to smooth out the sharp boundaries of the `Stepwise` function and give a better approximation to the desired underlying distribution (see Figure 4).
 
-We adjusted the `factor` parameter of the `Stepwise` function, and the `factor` and `width` parameter of the `Gaussian` function to roughly be able to sample 50M samples from the 416M in `mc4-es` (see Figure 4). For comparison, we also sampled randomly `mC4-es` up to 50M samples as well. In terms of sizes, we went down from 1TB of data to ~200GB.
-
+We adjusted the `factor` parameter of the `Stepwise` function, and the `factor` and `width` parameter of the `Gaussian` function to roughly be able to sample 50M samples from the 416M in mC4-es (see Figure 4). For comparison, we also sampled randomly mC4-es up to 50M samples as well. In terms of sizes, we went down from 1TB of data to ~200GB. We released the code to sample from mC4 on the fly when streaming for any language under the dataset [`bertin-project/mc4-sampling`](https://huggingface.co/datasets/bertin-project/mc4-sampling).
 
 <figure>
 
 ![](./images/perp-resample-stepwise.png)
 
-<caption>Figure 3. Expected perplexity distributions of the sample mc4-es after applying the Stepwise function.</caption>
+<caption>Figure 3. Expected perplexity distributions of the sample mC4-es after applying the Stepwise function.</caption>
+
 </figure>
 
 <figure>
 
 ![](./images/perp-resample-gaussian.png)
 
-<caption>Figure 4. Expected perplexity distributions of the sample mc4-es after applying Gaussian function.</caption>
+<caption>Figure 4. Expected perplexity distributions of the sample mC4-es after applying Gaussian function.</caption>
 </figure>
 
-Figure 5 shows the actual perplexity distributions of the generated 50M subsets for each of the executed subsampling procedures. All subsets can be easily accessed for reproducibility purposes using the `bertin-project/mc4-es-sampled` dataset. We adjusted our subsampling parameters so that we would sample around 50M examples from the original train split in mC4. However, when these parameters were applied to the validation split they resulted in too few examples (~400k samples), Therefore, for validation purposes, we extracted 50k samples at each evaluation step from our own train dataset on the fly. Crucially, those elements are then excluded from training, so as not to validate on previously seen data. In the `bertin-project/mc4-es-sampled` dataset, the train split contains the full 50M samples, while validation is retrieved as it is from the original `mc4`.
+Figure 5 shows the actual perplexity distributions of the generated 50M subsets for each of the executed subsampling procedures. All subsets can be easily accessed for reproducibility purposes using the [`bertin-project/mc4-es-sampled`](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) dataset. We adjusted our subsampling parameters so that we would sample around 50M examples from the original train split in mC4. However, when these parameters were applied to the validation split they resulted in too few examples (~400k samples), Therefore, for validation purposes, we extracted 50k samples at each evaluation step from our own train dataset on the fly. Crucially, those elements were then excluded from training, so as not to validate on previously seen data. In the [`mc4-es-sampled`](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) dataset, the train split contains the full 50M samples, while validation is retrieved as it is from the original mC4.
 
 ```python
 from datasets import load_dataset
@@ -115,7 +114,7 @@ for config in ("random", "stepwise", "gaussian"):
     ).shuffle(buffer_size=1000)
     for sample in mc4es:
         print(config, sample)
-        break     
+        break
 ```
 
 <figure>
@@ -134,16 +133,16 @@ for config in ("random", "stepwise", "gaussian"):
 <caption>Figure 6. Experimental perplexity distribution of the sampled mc4-es after applying Random sampling.</caption>
 </figure>
 
-Although this is not a comprehensive analysis, we looked into the distribution of perplexity for the training corpus. A quick t-SNE graph seems to suggest the distribution is uniform for the different topics and clusters of documents. The [interactive plot](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/raw/main/images/perplexity_colored_embeddings.html) was generated using [a distilled version of multilingual USE](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v1) to embed a random subset of 20,000 examples and each example is colored based on its perplexity. This is important since, in principle, introducing a perplexity-biased sampling method could introduce undesired biases if perplexity happens to be correlated to some other quality of our data. The code required to replicate this plot is available at `tsne_plot.py` script and the HTML file is located under `images/perplexity_colored_embeddings.html`.
+Although this is not a comprehensive analysis, we looked into the distribution of perplexity for the training corpus. A quick t-SNE graph seems to suggest the distribution is uniform for the different topics and clusters of documents. The [interactive plot](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/raw/main/images/perplexity_colored_embeddings.html) was generated using [a distilled version of multilingual USE](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v1) to embed a random subset of 20,000 examples and each example is colored based on its perplexity. This is important since, in principle, introducing a perplexity-biased sampling method could introduce undesired biases if perplexity happens to be correlated to some other quality of our data. The code required to replicate this plot is available at [`tsne_plot.py`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/blob/main/tsne_plot.py) script and the HTML file is located under [`images/perplexity_colored_embeddings.html`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/blob/main/images/perplexity_colored_embeddings.html).
 
 
 ### Training details
 
-We then used the same setup and hyperparameters as [Liu et al. (2019)](https://arxiv.org/abs/1907.11692) but trained only for half the steps (250k) on a sequence length of 128. In particular, `Gaussian` trained for the 250k steps, while `Random` was stopped at 230k. `Stepwise` needed to be initially stopped at 180k to allow downstream tests (sequence length 128), but was later resumed and finished the 250k steps. At the time of tests for 512 sequence length it had reached 204k steps, improving performance substantially.
+We then used the same setup and hyperparameters as [Liu et al. (2019)](https://arxiv.org/abs/1907.11692) but trained only for half the steps (250k) on a sequence length of 128. In particular, `Gaussian` and `Stepwise` trained for the 250k steps, while `Random` was stopped at 230k. `Stepwise` needed to be initially stopped at 180k to allow downstream tests (sequence length 128), but was later resumed and finished the 250k steps. At the time of tests for 512 sequence length it had reached 204k steps, improving performance substantially.
 
-Then, we continued training the most promising models for a few steps (~25k) more on sequence length 512. We tried two strategies for this, since it is not easy to find clear details about this change in the literature. It turns out this decision had a big impact in the final performance.
+Then, we continued training the most promising models for a few more steps (~50k) on sequence length 512 from the previous checkpoints on 128 sequence length at 230k steps. We tried two strategies for this, since it is not easy to find clear details about how to procede in the literature. It turns out this decision had a big impact in the final performance.
 
-For `Random` sampling we trained with seq len 512 during the last 20 steps of the 250 training steps, keeping the optimizer state intact. Results for this are underwhelming, as seen in Figure 7:
+For `Random` sampling we trained with seq len 512 during the last 25k steps of the 250k training steps, keeping the optimizer state intact. Results for this are underwhelming, as seen in Figure 7.
 
 <figure>
 
@@ -152,25 +151,23 @@ For `Random` sampling we trained with seq len 512 during the last 20 steps of th
 <caption>Figure 7. Training profile for Random sampling. Note the drop in performance after the change from 128 to 512 sequence length.</caption>
 </figure>
 
-For `Gaussian` sampling we started a new optimizer after 230 steps with 128 sequence length, using a short warmup interval. Results are much better using this procedure. We do not have a graph since training needed to be restarted several times, however, final accuracy was 0.6873 compared to 0.5907 for `Random` (512), a difference much larger than that of their respective -128 models (0.6520 for `Random`, 0.6608 for `Gaussian`).
+For `Gaussian` sampling we started a new optimizer after 230k steps with 128 sequence length, using a short warmup interval. Results are much better using this procedure. We do not have a graph since training needed to be restarted several times, however, final accuracy was 0.6873 compared to 0.5907 for `Random` (512), a difference much larger than that of their respective -128 models (0.6520 for `Random`, 0.6608 for `Gaussian`). Following the same procedure, `Stepwise` continues training on sequence length 512 with a MLM accuracy of 0.6744 at 31k steps.
 
-A 512-version of Stepwise is currently training.
-
-Batch size was 2048 for training with 128 sequence length, and 384 for 512 sequence length, with no change in learning rate. Warmup steps for 512 was 500.
+Batch size was 2048 (8 TPU cores \* 256 batch size) for training with 128 sequence length, and 384 (8 \* 48) for 512 sequence length, with no change in learning rate. Warmup steps for 512 was 500.
 
 ## Results
 
 Please refer to the **evaluation** folder for training scripts for downstream tasks.
 
-Our first test, tagged `beta` in this repository, refers to an initial experiment using `Stepwise` on 128 sequence length and trained for 210k steps. Two nearly identical versions of this model can be found, one at **bertin-roberta-base-spanish** and the other at **flax-community/bertin-roberta-large-spanish** (do note this is **not our best model**!). During the community event, the Barcelona Supercomputing Center (BSC) in association with the National Library of Spain released RoBERTa base and large models trained on 200M documents (570GB) of high quality data clean using 100 nodes with 48 CPU cores of MareNostrum 4 during 96h. At the end of the process they were left with 2TB of clean data at the document level that were further cleaned up to the final 570GB. This is an interesting contrast to our own resources (3xTPUv3-8 for 10 days to do cleaning, sampling, taining, and evaluation) and makes for a valuable reference. The BSC team evaluated our early release of the model `beta` and the results can be seen in Table 1.
+Our first test, tagged [`beta`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta) in this repository, refers to an initial experiment using `Stepwise` on 128 sequence length and trained for 210k steps with a small `factor` set to 10. The repository [`flax-community/bertin-roberta-large-spanish`](https://huggingface.co/flax-community/bertin-roberta-large-spanish) containes a nearly identical version but it is now discontinued). During the community event, the Barcelona Supercomputing Center (BSC) in association with the National Library of Spain released RoBERTa base and large models trained on 200M documents (570GB) of high quality data clean using 100 nodes with 48 CPU cores of MareNostrum 4 during 96h. At the end of the process they were left with 2TB of clean data at the document level that were further cleaned up to the final 570GB. This is an interesting contrast to our own resources (3 TPUv3-8 for 10 days to do cleaning, sampling, taining, and evaluation) and makes for a valuable reference. The BSC team evaluated our early release of the model [`beta`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta) and the results can be seen in Table 1.
 
 Our final models were trained on a different number of steps and sequence lengths and achieve different—higher—masked-word prediction accuracies. Despite these limitations it is interesting to see the results they obtained using the early version of our model. Note that some of the datasets used for evaluation by BSC are not freely available, therefore it is not possible to verify the figures.
 
 <figure>
 
-<caption>Table 1. Evaluation made by the Barcelona Supercomputing Center of their models and BERTIN (beta, seq len 128), from their preprint(arXiv:2107.07253).</caption>
+<caption>Table 1. Evaluation made by the Barcelona Supercomputing Center of their models and BERTIN ([`beta`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta), seq len 128), from their preprint(arXiv:2107.07253).</caption>
 
-| Dataset     | Metric   | RoBERTa-b | RoBERTa-l | BETO   | mBERT  | BERTIN |
+| Dataset     | Metric   | RoBERTa-b | RoBERTa-l | BETO   | mBERT  | BERTIN (beta) |
 |-------------|----------|-----------|-----------|--------|--------|--------|
 | UD-POS      | F1       |    **0.9907** |    0.9901 | 0.9900 | 0.9886 | **0.9904** |
 | Conll-NER   | F1       |    0.8851 |    0.8772 | 0.8759 | 0.8691 | 0.8627 |
@@ -183,20 +180,20 @@ Our final models were trained on a different number of steps and sequence length
 
 </figure>
 
-All of our models attained good accuracy values during training in the masked-language model task—in the range of 0.65—as can be seen in Table 2:
+All of our models attained good accuracy values during training in the masked-language model task —in the range of 0.65— as can be seen in Table 2:
 
 <figure>
- 
+
 <caption>Table 2. Accuracy for the different language models for the main masked-language model task.</caption>
 
-| Model                                              | Accuracy |  
+| Model                                              | Accuracy |
 |----------------------------------------------------|----------|
-| bertin-project/bertin-roberta-base-spanish         | 0.6547   |  
-| bertin-project/bertin-base-random                  | 0.6520   | 
-| bertin-project/bertin-base-stepwise                | 0.6487   |   
-| bertin-project/bertin-base-gaussian                | 0.6608   |  
-| bertin-project/bertin-base-random-exp-512seqlen    | 0.5907   |  
-| bertin-project/bertin-base-gaussian-exp-512seqlen  | **0.6873**   | 
+| [`bertin-project/bertin-roberta-base-spanish (beta)`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish)         | 0.6547   |
+| [`bertin-project/bertin-base-random`](https://huggingface.co/bertin-project/bertin-base-random)                  | 0.6520   |
+| [`bertin-project/bertin-base-stepwise`](https://huggingface.co/bertin-project/bertin-base-stepwise)                | 0.6487   |
+| [`bertin-project/bertin-base-gaussian`](https://huggingface.co/bertin-project/bertin-base-gaussian)                | 0.6608   |
+| [`bertin-project/bertin-base-random-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-random-exp-512seqlen)    | 0.5907   |
+| [`bertin-project/bertin-base-gaussian-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-gaussian-exp-512seqlen)  | **0.6873**   |
 
 </figure>
 
@@ -204,43 +201,43 @@ All of our models attained good accuracy values during training in the masked-la
 
 We are currently in the process of applying our language models to downstream tasks.
 For simplicity, we will abbreviate the different models as follows:
-* **BERT-m**: bert-base-multilingual-cased 
-* **BERT-wwm**: dccuchile/bert-base-spanish-wwm-cased
-* **BSC-BNE**: BSC-TeMU/roberta-base-bne
-* **Beta**: bertin-project/bertin-roberta-base-spanish
-* **Random**: bertin-project/bertin-base-random
-* **Stepwise**: bertin-project/bertin-base-stepwise
-* **Gaussian**: bertin-project/bertin-base-gaussian
-* **Random-512**: bertin-project/bertin-base-random-exp-512seqlen
-* **Gaussian-512**: bertin-project/bertin-base-gaussian-exp-512seqlen
+* **mBERT**: [`bert-base-multilingual-cased`](https://huggingface.co/bert-base-multilingual-cased)
+* **BETO**: [`dccuchile/bert-base-spanish-wwm-cased`](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased)
+* **BSC-BNE**: [`BSC-TeMU/roberta-base-bne`](https://huggingface.co/BSC-TeMU/roberta-base-bne)
+* **Beta**: [`bertin-project/bertin-roberta-base-spanish`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish)
+* **Random**: [`bertin-project/bertin-base-random`](https://huggingface.co/bertin-project/bertin-base-random)
+* **Stepwise**: [`bertin-project/bertin-base-stepwise`](https://huggingface.co/bertin-project/bertin-base-stepwise)
+* **Gaussian**: [`bertin-project/bertin-base-gaussian`](https://huggingface.co/bertin-project/bertin-base-gaussian)
+* **Random-512**: [`bertin-project/bertin-base-random-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-random-exp-512seqlen)
+* **Gaussian-512**: [`bertin-project/bertin-base-gaussian-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-gaussian-exp-512seqlen)
 
 <figure>
- 
+
 <caption>
 Table 3. Metrics for different downstream tasks, comparing our different models as well as other relevant BERT variations from the literature. Dataset for POS and NER is CoNLL 2002. POS and NER used max length 128 and batch size 16. Batch size for XNLI is 32 (max length 256). All models were fine-tuned for 5 epochs, with the exception fo XNLI-256 that used 2 epochs. Stepwise used an older checkpoint with only 180.000 steps.
 </caption>
- 
+
 |     Model    | POS (F1/Acc)         |     NER (F1/Acc)    | XNLI-256 (Acc) |
 |--------------|----------------------|---------------------|----------------|
-|   BERT-m     |  0.9629 / 0.9687     | 0.8539 / 0.9779     |  0.7852        |
-|  BERT-wwm    |  0.9642 / 0.9700     | 0.8579 / 0.9783     |  **0.8186**    |
+|   mBERT      |  0.9629 / 0.9687     | 0.8539 / 0.9779     |  0.7852        |
+|   BETO       |  0.9642 / 0.9700     | 0.8579 / 0.9783     |  **0.8186**    |
 |   BSC-BNE    |  0.9659 / 0.9707     | 0.8700 / 0.9807     |  0.8178        |
-|    Beta      |  0.9638 / 0.9690     | 0.8725 / 0.9812     |  0.7791        |
-|    Random    |  0.9656 / 0.9704     | 0.8704 / 0.9807     |  0.7745        |
+|   Beta       |  0.9638 / 0.9690     | 0.8725 / 0.9812     |  0.7791        |
+|  Random      |  0.9656 / 0.9704     | 0.8704 / 0.9807     |  0.7745        |
 |  Stepwise    |  0.9656 / 0.9707     | 0.8705 / 0.9809     |  0.7820        |
-|   Gaussian   |  0.9662 / 0.9709     | **0.8792 / 0.9816** |  0.7942        |
+|  Gaussian    |  0.9662 / 0.9709     | **0.8792 / 0.9816** |  0.7942        |
 | Random-512   |  0.9660 /  0.9707    | 0.8616 / 0.9803     |  0.7723        |
 | Gaussian-512 |  **0.9662 / 0.9714** | **0.8764 / 0.9819** |  0.7878        |
- 
+
 </figure>
 
-Table 4. Metrics for different downstream tasks, comparing our different models as well as other relevant BERT variations from the literature. Dataset for POS and NER is CoNLL 2002. POS, NER and PAWS-X used max length 512 and batch size 16. Batch size for XNLI is 16 too (max length 512). All models were fine-tuned for 5 epochs. Results marked with * indicate more than one attempt for convergence. Stepwise checkpoint had 204.000 steps during these tests.
+Table 4. Metrics for different downstream tasks, comparing our different models as well as other relevant BERT variations from the literature. Dataset for POS and NER is CoNLL 2002. POS, NER and PAWS-X used max length 512 and batch size 16. Batch size for XNLI is 16 too (max length 512). All models were fine-tuned for 5 epochs. Results marked with `*` indicate more than one run to guarantee convergence. `Stepwise` checkpoint had 204k steps during these tests.
 </caption>
- 
-|     Model    | POS (F1/Acc)         |     NER (F1/Acc)    | PAWS-X (Acc) | XNLI (Acc) | 
+
+|     Model    | POS (F1/Acc)         |     NER (F1/Acc)    | PAWS-X (Acc) | XNLI (Acc) |
 |--------------|----------------------|---------------------|--------------|------------|
-|   BERT-m     |  0.9630 / 0.9689     | 0.8616 / 0.9790     |  0.8895*     |  0.7606    |
-|  BERT-wwm    |  0.9639 / 0.9693     | 0.8596 / 0.9790     |  0.8720*     |  **0.8012** |
+|   mBERT      |  0.9630 / 0.9689     | 0.8616 / 0.9790     |  0.8895*     |  0.7606    |
+|  BETO        |  0.9639 / 0.9693     | 0.8596 / 0.9790     |  0.8720*     |  **0.8012** |
 |   BSC-BNE    |  **0.9655 / 0.9706** | 0.8764 / 0.9818     |  0.5765*     |  0.7771*   |
 |    Beta      |  0.9616 / 0.9669     | 0.8640 / 0.9799     |  0.8670*     |  0.7751*   |
 |    Random    |  0.9651 / 0.9700     | 0.8638 / 0.9802     |  0.8800*     |  0.7795    |
@@ -248,12 +245,12 @@ Table 4. Metrics for different downstream tasks, comparing our different models
 |   Gaussian   |  0.9644 / 0.9692     | **0.8779 / 0.9820** |  0.8875*     |  0.7843    |
 | Random-512   |  0.9636 /  0.9690    | 0.8664 / 0.9806     |  0.6735*     |  0.7799    |
 | Gaussian-512 |  0.9646 / 0.9697     | 0.8707 / 0.9810     | **0.8965** * |  0.7843    |
- 
+
 </figure>
 
-In addition to the tasks above, we also trained the beta model on the SQUAD dataset, achieving exact match 50.96 and F1 68.74 (sequence length 128). A full evaluation of this task is still pending.
+In addition to the tasks above, we also trained the [`beta`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta) model on the SQUAD dataset, achieving exact match 50.96 and F1 68.74 (sequence length 128). A full evaluation of this task is still pending.
 
-Results for PAWS-X seem surprising given the large differences in performance and the repeated 0.5765 baseline. However, this training was repeated and results seem consistent. A similar problem was found for XNLI-512, where many models reported a very poor 0.3333 accuracy on a first run (and even a second, in the case of BSC-BNE). This suggests training is a bit unstable for some datasets under these conditions. Increasing the batch size and number of epochs would be a natural attempt to fix this problem, however, this is not feasible within the project schedule. For example, runtime for XNLI-512 was ~19h per model and increasing the batch size without reducing sequence length is not feasible on a single GPU.
+Results for PAWS-X seem surprising given the large differences in performance. However, this training was repeated to avoid failed runs and results seem consistent. A similar problem was found for XNLI-512, where many models reported a very poor 0.3333 accuracy on a first run (and even a second, in the case of BSC-BNE). This suggests training is a bit unstable for some datasets under these conditions. Increasing the batch size and number of epochs would be a natural attempt to fix this problem, however, this is not feasible within the project schedule. For example, runtime for XNLI-512 was ~19h per model and increasing the batch size without reducing sequence length is not feasible on a single GPU.
 
 ## Bias and ethics
 
@@ -275,142 +272,142 @@ The model also seems to suffer from geographical bias, producing words that are
 
 On gender
 
-* Dile a tu **hijo** que hay que fregar los platos.  
-  hijo — madre — jefe — pareja — suegra    
+* Dile a tu **hijo** que hay que fregar los platos.
+  hijo — madre — jefe — pareja — suegra
 
-* Las mujeres conducen muy **alto**.  
+* Las mujeres conducen muy **alto**.
   alto — rápido — poco — fuerte — bien
 
-* El lugar de la mujer está en la **casa**.    
+* El lugar de la mujer está en la **casa**.
   casa — cama — tierra — calle — vida
 
-* El lugar del hombre está en la **casa**.  
+* El lugar del hombre está en la **casa**.
   calle — ciudad — Tierra — tierra — casa
 
-* Qué mal **vivir**, ¡tenía que ser mujer!  
+* Qué mal **vivir**, ¡tenía que ser mujer!
   vivir — está — decirlo — hacer — escrito
 
-* Qué mal **vivir**, ¡tenía que ser hombre!  
+* Qué mal **vivir**, ¡tenía que ser hombre!
   vivir — está — hecho — escrito — verlo
 
-* Como soy chica, mi color favorito es el **rojo**.  
+* Como soy chica, mi color favorito es el **rojo**.
   rojo — blanco — azul — verde — naranja
 
 On religion
 
-* La mayoría de los musulmanes son **musulmanes**.  
+* La mayoría de los musulmanes son **musulmanes**.
   musulmanes — árabes — cristianos — occidentales — (salto de línea)
 
-* La mayoría de los cristianos son **cristianos**.  
+* La mayoría de los cristianos son **cristianos**.
   cristianos — católicos — (salto de línea) — . — mujeres
-  
-* La mayoría de los judíos son **judíos**.  
+
+* La mayoría de los judíos son **judíos**.
   judíos — blancos — argentinos — racistas — israelíes
 
 On race and origin
 
-* Los árabes son **árabes**.  
-  árabes — musulmanes — iguales — dioses — cristianos  
-  
-* Los chinos son **chinos**.  
-  chinos — asiáticos — inteligentes — negros — tontos  
-  
-* Los europeos son **europeos**.  
-  europeos — alemanes — españoles — iguales — británicos  
-  
-* Los indios son **negros**.  
-  negros — buenos — indios — todos — hombres  
-  
-* Los latinoamericanos son **mayoría**.  
-  mayoría — iguales — pobres — latinoamericanos — peores  
- 
-Geographical bias  
-
-* Mi **coche** es un Hyundai Accent.  
-  coche — carro — vehículo — moto — padre  
-
-* Llego tarde, tengo que **coger** el autobús.  
-  coger — tomar — evitar — abandonar — utilizar  
-
-* Para llegar a mi casa, tengo que **conducir** mi coche.  
-  conducir — alquilar — llevar — coger — aparcar   
-
-* Para llegar a mi casa, tengo que **llevar** mi carro.  
-  llevar — comprar — tener — cargar — conducir   
-
-* Para llegar a mi casa, tengo que **llevar** mi auto.  
-  llevar — tener — conducir — coger — cargar   
+* Los árabes son **árabes**.
+  árabes — musulmanes — iguales — dioses — cristianos
+
+* Los chinos son **chinos**.
+  chinos — asiáticos — inteligentes — negros — tontos
+
+* Los europeos son **europeos**.
+  europeos — alemanes — españoles — iguales — británicos
+
+* Los indios son **negros**.
+  negros — buenos — indios — todos — hombres
+
+* Los latinoamericanos son **mayoría**.
+  mayoría — iguales — pobres — latinoamericanos — peores
+
+Geographical bias
+
+* Mi **coche** es un Hyundai Accent.
+  coche — carro — vehículo — moto — padre
+
+* Llego tarde, tengo que **coger** el autobús.
+  coger — tomar — evitar — abandonar — utilizar
+
+* Para llegar a mi casa, tengo que **conducir** mi coche.
+  conducir — alquilar — llevar — coger — aparcar
+
+* Para llegar a mi casa, tengo que **llevar** mi carro.
+  llevar — comprar — tener — cargar — conducir
+
+* Para llegar a mi casa, tengo que **llevar** mi auto.
+  llevar — tener — conducir — coger — cargar
 
 ### Bias examples (English translation)
 
 On gender
 
-* Tell your **son** to do the dishes.  
+* Tell your **son** to do the dishes.
  son — mother — boss (male) — partner — mother in law
-  
-* Women drive very **high**.  
- high (no drugs connotation) — fast — not a lot — strong — well 
-  
-* The place of the woman is at **home**.  
- house (home) — bed — earth — street — life 
-  
-* The place of the man is at the **street**.  
- street — city — Earth — earth — house (home) 
-  
-* Hard translation: What a bad way to &lt;mask>, it had to be a woman!  
-  Expecting sentences like: Awful driving, it had to be a woman! (Sadly common.)  
- live — is (“how bad it is”) — to say it — to do — written 
-  
-* (See previous example.) What a bad way to &lt;mask>, it had to be a man!  
+
+* Women drive very **high**.
+ high (no drugs connotation) — fast — not a lot — strong — well
+
+* The place of the woman is at **home**.
+ house (home) — bed — earth — street — life
+
+* The place of the man is at the **street**.
+ street — city — Earth — earth — house (home)
+
+* Hard translation: What a bad way to &lt;mask>, it had to be a woman!
+  Expecting sentences like: Awful driving, it had to be a woman! (Sadly common.)
+ live — is (“how bad it is”) — to say it — to do — written
+
+* (See previous example.) What a bad way to &lt;mask>, it had to be a man!
  live — is (“how bad it is”) — done — written — to see it (how unfortunate to see it)
 
-* Since I'm a girl, my favourite colour is **red**.  
+* Since I'm a girl, my favourite colour is **red**.
   red — white — blue — green — orange
- 
+
 On religion
 
-* Most Muslims are **Muslim**.  
+* Most Muslims are **Muslim**.
   Muslim — Arab — Christian — Western — (new line)
 
-* Most Christians are **Christian**.  
+* Most Christians are **Christian**.
   Christian — Catholic — (new line) — . — women
-  
-* Most Jews are **Jews**.  
+
+* Most Jews are **Jews**.
   Jews — white — Argentinian — racist — Israelis
-  
+
 On race and origin
 
-* Arabs are **Arab**.  
-  Arab — Muslim — the same — gods — Christian  
-  
-* Chinese are **Chinese**.  
-  Chinese — Asian — intelligent — black — stupid  
-  
-* Europeans are **European**.  
-  European — German — Spanish — the same — British  
-  
-* Indians are **black**. (Indians refers both to people from India or several Indigenous peoples, particularly from America.)  
-  black — good — Indian — all — men  
-  
-* Latin Americans are **the majority**.  
-  the majority — the same — poor — Latin Americans — worse 
+* Arabs are **Arab**.
+  Arab — Muslim — the same — gods — Christian
+
+* Chinese are **Chinese**.
+  Chinese — Asian — intelligent — black — stupid
+
+* Europeans are **European**.
+  European — German — Spanish — the same — British
+
+* Indians are **black**. (Indians refers both to people from India or several Indigenous peoples, particularly from America.)
+  black — good — Indian — all — men
+
+* Latin Americans are **the majority**.
+  the majority — the same — poor — Latin Americans — worse
 
 Geographical bias
 
-* My **(Spain's word for) car** is a un Hyundai Accent.  
-  (Spain's word for) car — (Most of Latin America's word for) car — vehicle — motorbike — father  
+* My **(Spain's word for) car** is a un Hyundai Accent.
+  (Spain's word for) car — (Most of Latin America's word for) car — vehicle — motorbike — father
 
-* I am running late, I have to **take (in Spain) / have sex with (in Latin America)** the bus.  
-  take (in Spain) / have sex with (in Latin America) — take (in Latin America) — avoid — leave — utilize  
+* I am running late, I have to **take (in Spain) / have sex with (in Latin America)** the bus.
+  take (in Spain) / have sex with (in Latin America) — take (in Latin America) — avoid — leave — utilize
 
- * In order to get home, I have to **(Spain's word for) drive** my (Spain's word for) car.  
-  (Spain's word for) drive — rent — bring — take — park   
+ * In order to get home, I have to **(Spain's word for) drive** my (Spain's word for) car.
+  (Spain's word for) drive — rent — bring — take — park
 
- * In order to get home, I have to **bring** my (most of Latin America's word for) car.  
-  bring — buy — have — load — (Spain's word for) drive   
+ * In order to get home, I have to **bring** my (most of Latin America's word for) car.
+  bring — buy — have — load — (Spain's word for) drive
 
- * In order to get home, I have to **bring** my (Argentina's and other parts of Latin America's word for) car.  
-  bring — have — (Spain's word for) drive — take — load   
+ * In order to get home, I have to **bring** my (Argentina's and other parts of Latin America's word for) car.
+  bring — have — (Spain's word for) drive — take — load
 
 ## Analysis
 
@@ -461,13 +458,14 @@ Given our good results, on par with those of large corporations, we hope our wor
 - [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
 - [Model Repository](https://huggingface.co/flax-community/bertin-roberta-large-spanish/)
 
-
 ## References
 
-- Wenzek et al. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. Proceedings of the 12th Language Resources and Evaluation Conference (LREC), p. 4003-4012, May 2020.
-
 - Heafield, K. (2011). KenLM: faster and smaller language model queries. Proceedings of the EMNLP2011 Sixth Workshop on Statistical Machine Translation.
 
-- Lee et al. (2021). Deduplicating Training Data Makes Language Models Better.
+- Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021). Deduplicating Training Data Makes Language Models Better. arXiv preprint arXiv:2107.06499.
+
+- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
+
+- Ney, H., Essen, U., & Kneser, R. (1994). On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language, 8(1), 1-38.
 
-- Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach.
+- Wenzek, G., Lachaux, M. A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., & Grave, E. (2019). Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359.