noamaanMulla-03/transformers-issues huggingface dataset:
XLM-R Tokenizer now passes common tests + Integration tests
Author: LysandreJik
Pull request: yes

XLM-R Tokenizer had a lot of issues that were not identified as no testing was done on it.

closes #2993 
closes #2795
closes #2741
closes #2727
closes #2508

This fixes all the above issues, and works for all official checkpoints as well as other SPM files.

However, there are a few things I dislike about the way things stands, which I'm detailing in the comments below.

Comments:
This solved a CUDA runtime error for me. Strange! Thanks for this PR!
Init card for model
Author: DukeEnglish
Pull request: yes
Labels: name: model card

We create card for this model. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3321?src=pr&el=h1) Report
> Merging [#3321](https://codecov.io/gh/huggingface/transformers/pull/3321?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/38a555a83c8aceae77895d325174af5bd576cec7&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3321/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3321?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3321      +/-   ##
==========================================
- Coverage   77.14%   77.10%   -0.04%     
==========================================
  Files         100      100              
  Lines       16972    16972              
==========================================
- Hits        13093    13087       -6     
- Misses       3879     3885       +6     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3321?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3321/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.40% <0.00%> (-1.08%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3321?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3321?src=pr&el=footer). Last update [38a555a...1630a1f](https://codecov.io/gh/huggingface/transformers/pull/3321?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks! [`model page`](https://huggingface.co/clue/roberta_chinese_3L312_clue_tiny)

By the way, I sent an email to the address listed [in your paper](https://arxiv.org/pdf/2003.01355.pdf).
Let me know if you got it.

Thank you.
Yes, please. And my friend has sent an e-mail to you via that email.

best,
Junyi

On Wed, 18 Mar 2020 at 20:27, Julien Chaumond <notifications@github.com>
wrote:

> Thanks! model page
> <https://huggingface.co/clue/roberta_chinese_3L312_clue_tiny>
>
> By the way, I sent an email to the address listed in your paper
> <https://arxiv.org/pdf/2003.01355.pdf>.
> Let me know if you got it.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/3321#issuecomment-600593364>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEMFSVEG4UOVDKYZRLCCSN3RIC43XANCNFSM4LOEAGWQ>
> .
>


-- 
Junyi Li
+ 86 136 0354 2466
dukeenglish.github.io
Dockerhub images huggingface/transformers_cpu for version 2.5.1 has version 2.5.0 installed
Author: edwardcqian
Pull request: no

# 🐛 Bug

## Information

Model I am using any model introduced in 2.5.1

The problem arises when using:
pulling `huggingface/transformers_cpu:2.5.1` from dockerhub

## To reproduce

Steps to reproduce the behavior:

1. pull docker image from dockerhub
2. run docker container
3. run `pip freeze` to see `transformers==2.5.0`

## Expected behavior
`pip freeze` in the docker container should show: `transformers==2.5.1`

Comments:
Should be fixed, I'll push updated images for all the others Dockerfile in the next hours. 

Thanks for reporting @edwardcqian 

I'm closing, feel free to reopen if I missed something 👍
Trouble fine tuning distilbertmodel
Author: drisspg
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Trying to run lm_finetuning on the distilbert multilanguage

Getting the following error when I run:
python lm_finetuning.py \
    --model_type='distilbert' \
    --model_name_or_path=distilbert-base-multilingual-cased\
    --train_data_file=small.txt \
    --output_dir=output \
    --mlm \
    --do_train\
    --save_total_limit=2 \
    --save_steps=1000 \
    --no_cuda


similar error when trying to run on GPU.

```
Traceback (most recent call last):
  File "lm_finetuning.py", line 712, in <module>
    main()
  File "lm_finetuning.py", line 662, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "lm_finetuning.py", line 299, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_distilbert.py", line 550, in forward
    masked_lm_labels.view(-1))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2009, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97
Epoch:   0%|                                                                                                                                                                 | 0/1 [00:02<?, ?it/s]
Iteration:   0%|  
 ```


Comments:
According to https://github.com/huggingface/transformers/issues/2418#issuecomment-571721526, until a fix is released you should change the `-100` in your script to `-1`. It worked for me with Albert.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
changing the attention head size in MultiBert
Author: mhajiaghayi
Pull request: no

## 🐛 Bug

<!-- Important information -->
I'm trying to use a MultiBERT model with not all 12 attention heads but just 8 attention heads. so, in config file, I changed the following keys
    config.num_attention_heads = 8
    config.hidden_size = 512
    config.pooler_fc_size = 512

I assumed similar to layer size that we have flexibility in changing it, we may have a similar freedom in changing the head size. however, the run_xnli.py code throws the following error. 



>         size mismatch for bert.encoder.layer.11.output.LayerNorm.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in curr
> ent model is torch.Size([512]).
>         size mismatch for bert.pooler.dense.weight: copying a param with shape torch.Size([768, 768]) from checkpoint, the shape in current model is t
> orch.Size([512, 512]).
>         size mismatch for bert.pooler.dense.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Si
> ze([512]).



Model I am using MultiBERT. 

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [X ] an official GLUE/SQUaD task: run_xnli.py
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. get the run_xnli.py script and added the following config change codes 
    config.num_attention_heads = 8
    config.hidden_size = 512
    config.pooler_fc_size = 512
    config.pooler_num_attention_heads = 8



## Environment

* OS: windows 
* Python version: 3.6
* PyTorch version: 1.1 
* PyTorch Transformers version (or branch): 2.2.1 
* Using GPU ? yes
* Distributed or parallel setup ? parallel 
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
I got it working
improve doctstring for tf and pt generate() method
Author: patrickvonplaten
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3327?src=pr&el=h1) Report
> Merging [#3327](https://codecov.io/gh/huggingface/transformers/pull/3327?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e8f44af5bf44a79f102678f5d7bb737cd6da3b52&el=desc) will **decrease** coverage by `0.10%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3327/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3327?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3327      +/-   ##
==========================================
- Coverage   77.10%   77.00%   -0.11%     
==========================================
  Files         100      100              
  Lines       16953    16953              
==========================================
- Hits        13071    13054      -17     
- Misses       3882     3899      +17     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3327?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3327/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `84.43% <ø> (-3.23%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3327/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.09% <ø> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3327?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3327?src=pr&el=footer). Last update [e8f44af...183952e](https://codecov.io/gh/huggingface/transformers/pull/3327?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create README.md
Author: DukeEnglish
Pull request: yes

albert_chinese_tiny card

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3337?src=pr&el=h1) Report
> Merging [#3337](https://codecov.io/gh/huggingface/transformers/pull/3337?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/20139b7c8d88f380f1e4e0ae2baf0b0ac9351039&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3337/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3337?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3337   +/-   ##
=======================================
  Coverage   77.63%   77.63%           
=======================================
  Files         100      100           
  Lines       16943    16943           
=======================================
  Hits        13154    13154           
  Misses       3789     3789           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3337?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3337/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.19% <0.00%> (-0.18%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3337/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.36% <0.00%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3337?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3337?src=pr&el=footer). Last update [20139b7...7fbca7b](https://codecov.io/gh/huggingface/transformers/pull/3337?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create README.md
Author: DukeEnglish
Pull request: yes

roberta_chinese_base card

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3338?src=pr&el=h1) Report
> Merging [#3338](https://codecov.io/gh/huggingface/transformers/pull/3338?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/20139b7c8d88f380f1e4e0ae2baf0b0ac9351039?src=pr&el=desc) will **decrease** coverage by `0.18%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3338/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3338?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3338      +/-   ##
==========================================
- Coverage   77.63%   77.44%   -0.19%     
==========================================
  Files         100      100              
  Lines       16943    16943              
==========================================
- Hits        13154    13122      -32     
- Misses       3789     3821      +32
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3338?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3338/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `82.46% <0%> (-5.91%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3338/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.36% <0%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3338?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3338?src=pr&el=footer). Last update [20139b7...a7ff5ff](https://codecov.io/gh/huggingface/transformers/pull/3338?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create README.md
Author: DukeEnglish
Pull request: yes

albert_chinese_small card

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3336?src=pr&el=h1) Report
> Merging [#3336](https://codecov.io/gh/huggingface/transformers/pull/3336?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/20139b7c8d88f380f1e4e0ae2baf0b0ac9351039&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3336/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3336?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3336      +/-   ##
==========================================
- Coverage   77.63%   77.60%   -0.04%     
==========================================
  Files         100      100              
  Lines       16943    16943              
==========================================
- Hits        13154    13148       -6     
- Misses       3789     3795       +6     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3336?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3336/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `91.15% <0.00%> (-0.85%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3336/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.19% <0.00%> (-0.18%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3336/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.36% <0.00%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3336?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3336?src=pr&el=footer). Last update [20139b7...38b38af](https://codecov.io/gh/huggingface/transformers/pull/3336?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Unexpected output from feature extraction pipeline
Author: Stuffooh
Pull request: no

Hi everyone,

I'm not sure if it is a bug or if I simply overlooking something so I did not want to submit a bug report yet. I have the following example code:

```
from transformers import pipeline, AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased',
add_special_tokens=False)
#initialize pipeline
nlp = pipeline('feature-extraction', model='bert-base-uncased', config='bert-base-uncased', tokenizer=tokenizer, device=1)
features = nlp("Why is Howard asking questions about the food after Leonard gives him a carton ?")
features = np.squeeze(features)
print(features.shape)
```

I expect the output: (15,768)
But I receive the output: (18,768)

I think there are only 15 tokens but somehow the shape is 18. What am I missing here?

Is this output expected and am I simply missing something or is there more to it?

Comments:
The tokenizer adds special tokens (here, specific to BERT) at the beginning and end of the sentence. 

You can check that with:
```python
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
len(tokenizer.encode(TEXT)) == 18
```

@julien-c is there any way to avoid the special tokens when extracting the features? I expected "add_special_tokens=False" would prevent this from happening?

I would suggest not using the Pipeline and just doing `tokenizer.encode()` then `outputs = model(input_ids)`
Install error , Win10,anaconda3,python3.5,pytorch
Author: caucsunjiahui
Pull request: no



 When I pip install the transformers,its not successfull,my environment is Win10,anaconda3,python3.5,the error is as follows,what is wrong with it,thank you!.


## Details
 Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\sjh\AppData\Local\Temp\pip-install-fu05dcfq\sentencepiece\setup.py", line 29, in <module>
        with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
      File "C:\Users\sjh\Anaconda3\Lib\codecs.py", line 895, in open
        file = builtins.open(filename, mode, buffering)
    FileNotFoundError: [Errno 2] No such file or directory: '..\\VERSION'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\sjh\AppData\Local\Temp\pip-install-fu05dcfq\sentencepiece\ 


Comments:
seems to be a sentencepiece issue, please open an issue at https://github.com/google/sentencepiece
GPT-2 attention_mask reshaping uses input_ids first dimension
Author: lazarevskiVsg
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

## To reproduce
Use attention_mask in **GPT2LMHead**, while feeding **inputs_embeds** instead of **input_ids**
The code fails because line 427 of modeling_gpt2.py uses input_ids first dimension to reshape the mask
```
        if attention_mask is not None:
            batch_size = input_ids.shape[0]
            attention_mask = attention_mask.view(batch_size, -1)
```
I fix it by changing **input_ids.shape[0]** to **attention_mask.shape[0]**, but I think it would be more correct to obtain single batch_size from one of the available input formats.
**Update**
   I think `batch_size = input_shape[0]` is the best way
- `transformers` version: **master**
- Python version: 3.7
- PyTorch version (GPU?):1.4



Comments:
Hey @lazarevskiVsg, thanks a lot for pointing this out!
It should be fixed now :-)
pipelines.ipynb mask should be [MASK]
Author: shibing624
Pull request: no

transformers/notebooks/03-pipelines.ipynb fill-mask task, use newest version, <mask> is error, should be [MASK].

Comments:
Hi @shibing624, 

Thanks for opening this issue.

`fill-mask` pipeline uses Roberta under the hood and the mask token is actually `<mask>` which is the one used in the notebook.

However, it's not the safest way to use the pipeline I agree. I'll update the notebook with 

```python
nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in ' + nlp_fill.tokenizer.mask_token)
```

This way it will be compatible with any model.

Morgan

Dont hesitate to reopen if I missed something :)