run_pl_glue.py (almost equivalent performance with non-english bert models)
Author: timpal0l
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: `4.0.0.dev0`
- Platform: `Ubuntu 20.04.1 LTS`
- Python version: `3.8.5`
- PyTorch version (GPU?): `1.7.0` (GPU - yes)
- Tensorflow version (GPU?):
- Using GPU in script?: Yes (GeForce GTX Titan X)
- Using distributed or parallel set-up in script?: distributed

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 --> perhaps @sgugger

## Information
I tested running the glue benchmark with a few non-english models such as; Arabic, Swedish and Chinese.
Models I am using: `asafaya/bert-base-arabic`, `KB/bert-base-swedish-cased`, `bert-base-chinese`.

I recieve almost identical results as in [Run PyTorch version](https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-pytorch-version), it differs with a few percentages for each task, where some are even slightly better than using the default `bert-base-cased`

Am not sure this is a bug, but it seems a bit strange that with using different embeddings that are really far away from English such as Arabic and Chinese I get very similair results.

The problem arises when using:
* [X] the official example scripts: [run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue.py)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: GLUE, (sts-b in this example)
* [ ] my own task or dataset: (give details below)

## To reproduce

I get almost identical results when running a non-english bert on the glue benchmark. In this case on `stsb` using the `bert-base-chinese`, `asafaya/bert-base-arabic` and `KB/bert-base-swedish-cased`.

```
export TASK_NAME=stsb

python run_glue.py \
  --model_name_or_path bert-base-chinese \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/$TASK_NAME/
```
Chinese:
```
11/18/2020 17:10:42 - INFO - __main__ -   ***** Eval results stsb *****
11/18/2020 17:10:42 - INFO - __main__ -     eval_loss = 0.8410218954086304
11/18/2020 17:10:42 - INFO - __main__ -     eval_pearson = 0.7922208042884891
11/18/2020 17:10:42 - INFO - __main__ -     eval_spearmanr = 0.7956508384154777
11/18/2020 17:10:42 - INFO - __main__ -     eval_combined_score = 0.7939358213519834
11/18/2020 17:10:42 - INFO - __main__ -     epoch = 3.0
```
Arabic:
```
11/18/2020 17:14:04 - INFO - __main__ -   ***** Eval results stsb *****
11/18/2020 17:14:04 - INFO - __main__ -     eval_loss = 0.8082903027534485
11/18/2020 17:14:04 - INFO - __main__ -     eval_pearson = 0.8357733212850804
11/18/2020 17:14:04 - INFO - __main__ -     eval_spearmanr = 0.8386964712863125
11/18/2020 17:14:04 - INFO - __main__ -     eval_combined_score = 0.8372348962856965
11/18/2020 17:14:04 - INFO - __main__ -     epoch = 3.0
```
Swedish:
```
11/18/2020 17:32:26 - INFO - __main__ -   ***** Eval results stsb *****
11/18/2020 17:32:26 - INFO - __main__ -     eval_loss = 0.7071832418441772
11/18/2020 17:32:26 - INFO - __main__ -     eval_pearson = 0.8379047445076137
11/18/2020 17:32:26 - INFO - __main__ -     eval_spearmanr = 0.8350383734219187
11/18/2020 17:32:26 - INFO - __main__ -     eval_combined_score = 0.8364715589647662
11/18/2020 17:32:26 - INFO - __main__ -     epoch = 3.0
```
Is expected behaviour? Meaning that the readaption of the embedding matrices can work with non english vocabs such as Chinese and Arabic since they perhaps contain some latin characters. 

With English model `bert-base-cased` we get pearson: `83.95` and Arabic model `asafaya/bert-base-arabic` pearson: `83.57`. 
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
Thanks!
## Expected behavior
Not sure..
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
I can't say with certainty, but I actually think it's entirely feasible that this is legitimate result. Here's a [recent ACL paper](https://www.aclweb.org/anthology/2020.acl-main.421/) showing that a monolingual model can be fine-tuned on another language with competitive performance. The authors do learn a new embedding layer for the new target language in an intermediate pre-training step, so it's not entirely the same, but I wouldn't find this result too surprising. It's also likely that these non-English models had exposure to some English that wasn't scrubbed from their pre-training corpora, in which case the model might already have decent embeddings for tokens sourced from English text just from pre-training.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.