noamaanMulla-03/transformers-issues huggingface dataset:
mBART: incorrect <mask> token id
Author: OlegPlatonov
Pull request: no

# 🐛 Bug

## Information

Model I am using: mBART

## To reproduce

```
from transformers import MBartTokenizer
tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-cc25')
print(tokenizer.convert_tokens_to_ids(['<mask>', 'ar_AR']))
```
The output for the above code is `[250001, 250001]`  - two different special tokens are mapped to the same id.

## Expected behavior

As far as I can tell, `<mask>` token should be mapped to id 250026.

I've checked [fairseq implementation](https://github.com/pytorch/fairseq/blob/master/fairseq/tasks/multilingual_denoising.py) and it seems that `<mask>` token is added after all the language codes, so it should be the last token in the vocab.

Currently, when I try to use mBART to denoise text with `<mask>` tokens, it mostly just ignores them, but if I replace mask ids with 250026, the model actually generates new text in place of `<mask>` tokens:

```
from transformers import MBartTokenizer, BartForConditionalGeneration

tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-cc25')
model = BartForConditionalGeneration.from_pretrained('facebook/mbart-large-cc25')

text = 'I highly recommend <mask> - it is one of the best <mask> ever read!'
inputs = tokenizer.prepare_translation_batch([text], src_lang='en_XX')

outputs = model.generate(inputs['input_ids'], decoder_start_token_id=tokenizer.lang_code_to_id['en_XX'], 
                         num_beams=5)
print(tokenizer.batch_decode(outputs)[0])
```

The output is:
```
en_XX
 highly recommend - it is one of the best ever read!
```
Replacing mask ids:

```
where = (inputs['input_ids'] == 250001)
inputs['input_ids'][where] = 250026

outputs = model.generate(inputs['input_ids'], decoder_start_token_id=tokenizer.lang_code_to_id['en_XX'], 
                         num_beams=5)
print(tokenizer.batch_decode(outputs)[0])
```

The output is:
```
en_XX
 highly recommend this book - it is one of the best books I have ever read!
```

(In both cases, the model also skips the first input token when generating output, as discussed in #5755.)

I've also noticed that fairseq is using [language code tokens](https://github.com/pytorch/fairseq/blob/108bb2560b1ec01524ba723bc7c69186875afa0a/fairseq/tasks/multilingual_denoising.py#L62) of the form `[en_XX]` rather than just `en_XX`, which can lead to different tokenization if words like  `en_XX` appear in the text, but that's a rather contrived case.


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2


@sshleifer


Comments:
great catch and incredibly detailed description, I'll fix this one! Thanks!
[fix] add bart to LM_MAPPING
Author: sshleifer
Pull request: yes

This fixes 2/5 failing slow tests in #6094 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6099?src=pr&el=h1) Report
> Merging [#6099](https://codecov.io/gh/huggingface/transformers/pull/6099?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/06834bc33255f5fb8fabb72c9ff114764b3c7ce5&el=desc) will **increase** coverage by `0.16%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6099/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6099?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6099      +/-   ##
==========================================
+ Coverage   77.77%   77.93%   +0.16%     
==========================================
  Files         146      146              
  Lines       26325    26325              
==========================================
+ Hits        20474    20517      +43     
+ Misses       5851     5808      -43     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6099?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6099/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `78.48% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6099/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `65.42% <0.00%> (-29.91%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6099/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.71% <0.00%> (-0.51%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6099/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `98.79% <0.00%> (+33.89%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6099?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6099?src=pr&el=footer). Last update [06834bc...1c7f573](https://codecov.io/gh/huggingface/transformers/pull/6099?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[Fix] position_ids tests again
Author: sshleifer
Pull request: yes

Yesterday, in my haste, I added authorized_missing_keys to `BertModel`, rather than `BertPretrainedModel`. Since position_ids are allowed to be missing for all Bert variants, we want the latter, not the former.

I also improved the traceback for the failing test.

Comments:
Failure is test_tokenization_auto.py, and spurious .
Deebert Examples test failure
Author: sshleifer
Pull request: no

`DeeBertTests.test_glue_deebert`
Excerpt from [CI](https://pipelines.actions.githubusercontent.com/SFFqAjp6ciVZiZmfZfjie9y9Q96dfpUE8sJvWAtTDWoFlixGkf/_apis/pipelines/1/runs/4783/signedlogcontent/3?urlExpires=2020-07-23T20%3A02%3A20.7871420Z&urlSigningMethod=HMACV1&urlSignature=XL%2Fj%2F2NSBD3Q174ifxC4jTOQUloQMx3UFZ%2ByaM%2Beomk%3D)

```python
2020-07-23T00:30:23.1537872Z         with patch.object(sys, "argv", train_args):
2020-07-23T00:30:23.1538109Z             result = run_glue_deebert.main()
2020-07-23T00:30:23.1538324Z             for value in result.values():
2020-07-23T00:30:23.1538573Z >               self.assertGreaterEqual(value, 0.75)
2020-07-23T00:30:23.1538856Z E               AssertionError: 0.6666666666666666 not greater than or equal to 0.75
2020-07-23T00:30:23.1539030Z 
```



Comments:
@Ji-Xin would you mind taking a stab at this?
Command:
```bash
RUN_SLOW=1 pytest examples/deebert/ 
```
Fix deebert tests
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6102?src=pr&el=h1) Report
> Merging [#6102](https://codecov.io/gh/huggingface/transformers/pull/6102?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/06834bc33255f5fb8fabb72c9ff114764b3c7ce5&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6102/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6102?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #6102   +/-   ##
=======================================
  Coverage   77.77%   77.77%           
=======================================
  Files         146      146           
  Lines       26325    26325           
=======================================
+ Hits        20474    20475    +1     
+ Misses       5851     5850    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6102?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6102/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+0.25%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6102?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6102?src=pr&el=footer). Last update [06834bc...c00d3b1](https://codecov.io/gh/huggingface/transformers/pull/6102?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
StopIteration error in RobertaForMultipleChoice
Author: h56cho
Pull request: no

Hello,
I am trying to execute the line below for my `RobertaForMultipleChoice` model:
```python
# retrieve the resulting mc_loss
mc_loss = model(input_ids = input_ids, attention_mask = attention_mask, labels =  mc_labels)[0]
```
but this generates the following error:

```python
Traceback (most recent call last):
  File "STAT946_final_project_code_v4.py", line 623, in <module>
    success_rate_list_diag_normal = main_function_diag_normal('/home/ec2-user/test.txt', 'test_ans_num.txt', num_iter, log_interval)
  File "STAT946_final_project_code_v4.py", line 414, in main_function_diag_normal
    best_model_RobertaForMultipleChoice_diag_normal = train_loop(model_RobertaForMultipleChoice, tokenizer, optimizer_1, scheduler_1, log_interval, svi_diag_normal, guide_diag_normal, best_model_RobertaForMultipleChoice_diag_normal)
  File "STAT946_final_project_code_v4.py", line 341, in train_loop
    optimizer, scheduler, log_interval, svi, guide, epoch)
  File "STAT946_final_project_code_v4.py", line 236, in train_mc_head
    mc_loss = model(input_ids = input_ids, attention_mask = attention_mask, labels =  mc_labels)[0]
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyro/nn/module.py", line 413, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 441, in forward
    output_hidden_states=output_hidden_states,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyro/nn/module.py", line 413, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 732, in forward
    extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 228, in get_extended_attention_mask
    extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 159, in dtype
    first_tuple = next(gen)
StopIteration
```

How can I get around this type of error? Thank you,

Comments:
5 Slow test failures
Author: sshleifer
Pull request: no

traceback: [here](https://github.com/huggingface/transformers/runs/916838508?check_suite_focus=true)

```bash
=========================== short test summary info ============================
FAILED tests/test_modeling_auto.py::AutoModelTest::test_model_for_pretraining_from_pretrained
FAILED tests/test_modeling_auto.py::AutoModelTest::test_model_from_pretrained
FAILED tests/test_modeling_bart.py::BartModelIntegrationTests::test_bart_base_mask_filling
FAILED tests/test_modeling_bart.py::BartModelIntegrationTests::test_bart_large_mask_filling
FAILED tests/test_modeling_common.py::ModelUtilsTest::test_model_from_pretrained
==== 5 failed, 1423 passed, 489 skipped, 384 warnings in 1609.33s (0:26:49) ====
```

I'm investigating

Comments:
ALL FIXED YEEHAW
No button for creating new post at the forum.
Author: guotong1988
Pull request: no

![image](https://user-images.githubusercontent.com/4702353/88753699-76b01080-d18f-11ea-80c3-905a006ad31a.png)
discuss.huggingface.co

Comments:
![image](https://user-images.githubusercontent.com/4702353/88774445-0b2d6980-d1b6-11ea-88ba-e993190b434c.png)


I approved your post there @guotong1988
Incorporate HuggingFace 'nlp' library in examples
Author: nateraw
Pull request: no
Labels: name: wontfix

name: External

# 🚀 Feature request

I propose we replace the custom data downloading/preprocessing logic found within the examples directory with the new [**HuggingFace `nlp` Library**](https://github.com/huggingface/nlp) where applicable.

## Motivation

The examples directory is filled with custom shell scripts that download and process common research datasets. These scripts work great, but are at times tricky to follow. I'm sure this can be discouraging for new users looking to try out `transformers` for the first time.

I'm hoping `nlp` will make the examples generally more accessible for both new and experienced users. And yeah...I guess it's probably not too bad for the brand either. 😉 

## Your contribution

I'll get a WIP PR pushed up this weekend. I'll focus on the `pytorch_lightning` examples for now.

Comments:
Love this idea. @thomwolf @julien-c what do you guys think about adding `nlp` as a dependency in `examples/requirements.txt`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
❓ [TPU] [Trainer] Moving model to device before setting optimizer slow the training 
Author: astariul
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

On master, after applying the fix of #4450, the training on 8 TPU cores is much slower.

* Before the fix : **20 min / epoch (8 iterations / s)**
* After the fix : **1h30 / epoch (2~3 iterations / s)**

Of course the training before the fix was not working (loss was not decreasing).

But this slow is not expected : for the TF2 equivalent with the same dataset, it takes **20 min / epoch**.

---

Anyone meeting the same problem ?


Comments:
I have the same problem, but I'm not sure if it's really a problem. I thought the extreme speed before the fix was because it wasn't training properly, and that the slower speed now is supposed to be this way, but that's just my guess.

Hi, you're right @LeonieWeissweiler. Previous to that fix, the optimizer wasn't actually adjusting weights, resulting in a major speed-up (but the script in itself wasn't working).

@Colanim, do you mind specifying what exactly you're training on? When training on TPU there's a lot you should take into account: batch size, sequence length, number of cores being the most important. Do you mind giving a bit of context as to what you're trying to run?

 We've also merged this PR https://github.com/huggingface/transformers/pull/4467 which solves quite a few issues with the TPU training. Please make sure to install from source to benefit from that commit.

From my tests, on TPU with 8 cores (v3-8), on MNLI I reach 22 minutes/epoch with a batch size of 8, but **6 minutes/epoch** (with a 2 minute tracing, that isn't necessary for the following epochs) with a batch size of 128 (which does train with a final accuracy of 81% using `bert-base-cased`, single epoch). 

I'm training ELECTRA for Extractive Text Summarization.

I will try to increase the batch size and see the results, thanks for the pointer.

What bother me is that I used TFElectra and I could train the model at ~8 iterations per sec.

Same model and same hyper-parameters on Pytorch and it's slower.

But I realized that in my model, since it's extractive summarization, I'm extracting the [CLS] representation of each sentence. These CLS position varies from sample to sample. Maybe that's why it's slower on pytorch-xla ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
batch_encode_plus returns same lengths when enable pad_to_max_length
Author: binh-vu
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the following script

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
sents = [
    "I can eat glass without harm",
    "I cannot eat glass"
]
resp = tokenizer.batch_encode_plus(sents, pad_to_max_length=True, return_lengths=True)
print(resp['length'])
# >>> get [8, 8], should be [8, 6]
```
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The function batch_encode_plus should return correct lengths of sentences before padded to max length. Which should be [8, 6] in the above example. Otherwise, we can just get the length from the last dimension of the mask.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Ubuntu
- Python version: 3.7
- PyTorch version (GPU?): CPU 1.5
- Tensorflow version (GPU?): No
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
This is not a bug but expected behaviour. The length of the tokenized input is only calculated after padding.

https://github.com/huggingface/transformers/blob/a08652772791fdaeed6f263b1a99926ca64be5dc/src/transformers/tokenization_utils.py#L1981-L1982

Perhaps you are right, though, and it would be more useful to get the size before padding!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
KeyError ' '- run_ner.py - Transformers 2.8.0
Author: calusbr
Pull request: no
Labels: name: wontfix

name: Ex: Named Entity Recognition

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): PT

The problem arises when using:
* [ X ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

I'm having trouble running a dataset in Portuguese. It is in the conll pattern

Traceback (most recent call last):
  File "/home/lucasrodrigues/code/transformers-2.8.0/examples/ner/run_ner.py", line 292, in <module>
    main()
  File "/home/lucasrodrigues/code/transformers-2.8.0/examples/ner/run_ner.py", line 170, in main
    if training_args.do_train
  File "/home/lucasrodrigues/code/transformers-2.8.0/examples/ner/utils_ner.py", line 124, in __init__
    pad_token_label_id=self.pad_token_label_id,
  File "/home/lucasrodrigues/code/transformers-2.8.0/examples/ner/utils_ner.py", line 207, in convert_examples_to_features
    print([label_map[label]] + [pad_token_label_id] * (len(word_tokens) - 1))
KeyError: ''

## Expected behavior

Execution of the NER.

## Environment info
     
- `transformers` version: 2.8.0
- Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?:  <fill in>
- Using distributed or parallel set-up in script?: <fill in>


Comments:
Have you got two blank lines consecutive? Look at the end of the file, maybe you got an extra blank line or an empty line at the end of the file.

Additionally, the dataset format must be in a `Token Label` (delimiter is a space) format.

The error message shows, that an unexpected label ( `"`) was found, so I could imagine, that the dataset format is not consistent.

To check this, just do the following on your training, development and test set:

```bash
cat train.txt dev.txt test.txt | grep -v "^$" | cut -d " " -f 2 | sort | uniq
```

This should give you all labels. If you see other tokens, then there's something wrong in the dataset.



This problem also happened to me when I trained with my dataset.

Possible causes:
1. There is an extra character in labels
1. The word and label is not separated by space but tab space `\t` <- my main culprit

Hi can I look into this issue. I am very interested to start contributing in this project

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using DistillBert to train Bert (run_languge_modeling.py) for some languge from scratch
Author: mahdirezaey
Pull request: no
Labels: name: wontfix

name: Ex: LM (Pretraining)

name: Distillation

# ❓ Questions & Help

Is it a right thing to use 'distilbert-base-cased'  for training that with more data for some language ( which exists in the list of multilingual languages , and doesn't have a separate model still)

Will we reach good results as good as doing that with 6 layer BERT ?
Is there any differences between  6 layer BERT and distillBert (to use as the model for run_languge_modeling.py)

Comments:
I guess you can, since the architecture of distilbert is more or less the same as BERT (just half the layers), but I would not expect great performance. The power of distilling lies (next to the training objective) in having a teacher model (e.g.. a full BERT model) and initializing the distilled student network with weights from the teacher. If you don't do that it might not be easy to have good initialisation. In addition, the triple training loss would not make sense then since you have no teacher predictions to compare your distilled model with.

I can see that the language_modeling script allows to use distilbert, but I would assume that is intended for fine-tuning rather than pre-training. cc @VictorSanh 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add BERTweet and PhoBERT
Author: datquocnguyen
Pull request: yes
Labels: name: model card

I'd like to add BERTweet and PhoBERT to the huggingface's transformers library. 
Users can now use these models directly from transformers. E.g: 

    tokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-base")
    bertweet = BertweetModel.from_pretrained("vinai/bertweet-base")

Comments: