Update README.md

README.md

@@ -155,7 +155,7 @@ Our final models were trained on a different number of steps and sequence length
 
 <figure>
 
-<caption>Table 1. Evaluation made by the Barcelona Supercomputing Center of their models and BERTIN (beta, seq len 128).</caption>
+<caption>Table 1. Evaluation made by the Barcelona Supercomputing Center of their models and BERTIN (beta, seq len 128), from their [preprint](https://arxiv.org/pdf/2107.07253.pdf).</caption>
 
 | Dataset     | Metric   | RoBERTa-b | RoBERTa-l | BETO   | mBERT  | BERTIN |
 |-------------|----------|-----------|-----------|--------|--------|--------|
@@ -170,11 +170,11 @@ Our final models were trained on a different number of steps and sequence length
 
 </figure>
 
-All of our models attained good accuracy values, in the range of 0.65, as can be seen in Table 2:
+All of our models attained good accuracy values during training in the masked-language model task—in the range of 0.65—as can be seen in Table 2:
 
 <figure>
  
-<caption>Table 2. Accuracy for the different language models.</caption>
+<caption>Table 2. Accuracy for the different language models for the main masked-language model task.</caption>
 
 | Model                                              | Accuracy |  
 |----------------------------------------------------|----------|
@@ -187,6 +187,8 @@ All of our models attained good accuracy values, in the range of 0.65, as can be
 
 </figure>
 
+###Downstream Tasks
+
 We are currently in the process of applying our language models to downstream tasks.
 For simplicity, we will abbreviate the different models as follows:
 * **BERT-m**: bert-base-multilingual-cased 
@@ -202,55 +204,26 @@ For simplicity, we will abbreviate the different models as follows:
 <figure>
  
 <caption>
-Table 3. Metrics for different downstream tasks, comparing our different models as well as other relevant BERT variations from the literature. Dataset for POS nad NER is CoNLL 2002. 
+Table 3. Metrics for different downstream tasks, comparing our different models as well as other relevant BERT variations from the literature. Dataset for POS nad NER is CoNLL 2002. POS, NER adn PAWS-X used max length 512 and batch size 8. 
 </caption>
  
 |     Model    | POS (F1/Acc)            | NER (F1/Acc)         |  PAWS-X (Acc) | XNLI-256 (Acc) |  XNLI-512 (Acc) | 
 |--------------|-------------------------|----------------------|--------------|--------------|--------------|
-|   BERT-m     |    0.9629 / 0.9687      | 0.8539 / 0.9779      |    |    |    |
-|  BERT-wwm    |    0.9642 / 0.9700      | 0.8579 / 0.9783      |    |    |    |
-|   BSC-BNE    |     0.9659 / 0.9707     |  0.8700 / 0.9807     |    |    |    |
-|    Beta      |    0.9638 / 0.9690      |  0.8725 / 0.9812     |    |    |    |
-|    Random    |   0.9656 / 0.9704       | 0.8704 / 0.9807      |    |    |    |
-|  Stepwise    |   0.9656 / 0.9707       |  0.8705 / 0.9809     |    |    |    |
-|   Gaussian   |   0.9662 / 0.9709       |  **0.8792 / 0.9816** |    |    |    |
-| Random-512   |   0.9660 /  0.9707      |   0.8616 / 0.9803    |    |    |    |
-| Gaussian-512 |   **0.9662 / 0.9714**   |  **0.8764 / 0.9819** |    |    |    |
+|   BERT-m     |    0.9629 / 0.9687      | 0.8539 / 0.9779      |  0.5765      |    |    |
+|  BERT-wwm    |    0.9642 / 0.9700      | 0.8579 / 0.9783      |  0.8720      |    |    |
+|   BSC-BNE    |     0.9659 / 0.9707     |  0.8700 / 0.9807     |  0.5765      |    |    |
+|    Beta      |    0.9638 / 0.9690      |  0.8725 / 0.9812     |  0.5765      |    |    |
+|    Random    |   0.9656 / 0.9704       | 0.8704 / 0.9807      |  0.8800      |    |    |
+|  Stepwise    |   0.9656 / 0.9707       |  0.8705 / 0.9809     |  0.8825      |    |    |
+|   Gaussian   |   0.9662 / 0.9709       |  **0.8792 / 0.9816** |  0.8875      |    |    |
+| Random-512   |   0.9660 /  0.9707      |   0.8616 / 0.9803    |  0.6735      |    |    |
+| Gaussian-512 |   **0.9662 / 0.9714**   |  **0.8764 / 0.9819** |  **0.8965**  |    |    |
  
 </figure>
 
+In addition to the tasks above, we also trained the beta model on the SQUAD dataset, achieving exact match 50.96 and F1 68.74 (sequence length 128). A full evaluation of this task is still pending.
 
-### SQUAD-es
-Using sequence length 128 we have achieved exact match 50.96 and F1 68.74.
-
-
-POS
-All models trained with max length 512 and batch size 8, using the CoNLL 2002 dataset. 
-
-NER 
-All models trained with max length 512 and batch size 8, using the CoNLL 2002 dataset. 
-
-## PAWS-X 
-All models trained with max length 512 and batch size 8. These numbers are surprising both for the repeated instances of 0.5765 accuracy and for the large differences in performance. However, experiments have been repeated several times and the results are consistent. 
-
-<figure>
- 
-<caption>Table 5. Results for PAWS-X.</caption>
-
-| Model                                              | Accuracy |  
-|----------------------------------------------------|----------|
-| bert-base-multilingual-cased                       | 0.5765   |
-| dccuchile/bert-base-spanish-wwm-cased              | 0.8720   | 
-| BSC-TeMU/roberta-base-bne                          | 0.5765   |  
-| bertin-project/bertin-roberta-base-spanish         | 0.5765   |  
-| bertin-project/bertin-base-random                  | 0.8800   | 
-| bertin-project/bertin-base-stepwise                | 0.8825   |   
-| bertin-project/bertin-base-gaussian                | 0.8875   |  
-| bertin-project/bertin-base-random-exp-512seqlen    | 0.6735   |  
-| bertin-project/bertin-base-gaussian-exp-512seqlen  |  **0.8965**   |
-
-</figure>
-
+To note: not intense tuning, epochs, etc. Still, good?? PAWS-X: weird (large differences and repeated base value). Repeated and same, with minor differences.
 
 ### XNLI