noamaanMulla-03/transformers-issues huggingface dataset:
Attention mask always returns array of ones for CamembertTokenizer.batch_encode_plus
Author: Wissben
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

The model i'm using is CamebertTokenizer for the French language

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

- First init the models with the following code:
```python
camembert_tokenizer = CamembertTokenizer.from_pretrained('../models/', cache_dir='./models')
camembert_tf_model = TFCamembertModel.from_pretrained('../models/', output_hidden_states=True,
                                                          cache_dir='./models'
                                                     )
camembert_tf_model.trainable = False
```
- Prepare the input data :
```python
text = ' '.join(['je' for i in range(25)])
texts = [ text, "je suis cool"]
input_ids = camembert_tokenizer.batch_encode_plus(texts,
                                             add_special_tokens=True,
                                             max_length=8,
                                            return_tensors='tf')
print(input_ids)
```
## Expected behavior

What should happen is that the padded tokens should have a mask value of 0 if I've correctly understood the doc. so the output of the snippet should be : 
```
{'input_ids': <tf.Tensor: shape=(2, 8), dtype=int32, numpy=
 array([[   5,   50,   50,   50,   50,   50,   50,    6],
        [   5,   50,  146, 4261,    6,    1,    1,    1]], dtype=int32)>,
 'token_type_ids': <tf.Tensor: shape=(2, 8), dtype=int32, numpy=
 array([[0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 1, 1, 1]], dtype=int32)>,
 'attention_mask': <tf.Tensor: shape=(2, 8), dtype=int32, numpy=
 array([[1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 0, 0, 0]], dtype=int32)>}
```
Instead, i'm always getting an attention_mask full of ones like this : 
```
{'input_ids': <tf.Tensor: shape=(2, 8), dtype=int32, numpy=
 array([[   5,   50,   50,   50,   50,   50,   50,    6],
        [   5,   50,  146, 4261,    6,    1,    1,    1]], dtype=int32)>,
 'token_type_ids': <tf.Tensor: shape=(2, 8), dtype=int32, numpy=
 array([[0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 1, 1, 1]], dtype=int32)>,
 'attention_mask': <tf.Tensor: shape=(2, 8), dtype=int32, numpy=
 array([[1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1]], dtype=int32)>}
```

## Environment info
- `transformers` version: 2.5.1
- Platform: Linux-5.3.0-40-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: (False)
- Using distributed or parallel set-up in script?: (False)




Comments:
**EDIT** : 
I've been accidentally testing this code on the 2.4.1 version in a different environment. Since I've updated to 2.5.1 the behavior is as expected.
Create README.md
Author: dreasysnail
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3218?src=pr&el=h1) Report
> Merging [#3218](https://codecov.io/gh/huggingface/transformers/pull/3218?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d6de6423baf02a971d38ee69824104a1f0f85ad2&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3218/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3218?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3218      +/-   ##
==========================================
- Coverage   78.14%   78.14%   -0.01%     
==========================================
  Files          98       98              
  Lines       16668    16668              
==========================================
- Hits        13026    13025       -1     
- Misses       3642     3643       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3218?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3218/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.54% <0.00%> (-0.20%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3218?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3218?src=pr&el=footer). Last update [d6de642...a3ccef9](https://codecov.io/gh/huggingface/transformers/pull/3218?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
could you add an option to transfer variables from float32 to float16 in GPT2 model to reduce model size and accelerate the inference speed 
Author: carter54
Pull request: no
Labels: name: wontfix

## 🚀 Feature
could you add an option to transfer variables from float32 to float16 in GPT2 model
## Motivation
reduce model size and accelerate the inference speed
## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

Comments:
You should give a look at NVIDIA's apex library and PyTorch `model.half()` method.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@thomwolf Do you know if the gpt2 model needs to be pre-trained with apex support in order to use NVIDIA's apex library (e.g. O1 mode) at inference time? Was mixed precision used during the training of the gpt2 model? Is there a way I can verify that?

Do we have models trained with mixed precision enabled for gpt2? I can't find them in huggingface's repo.
few changes due to the torch version inconsistency in summarization example
Author: junxu-ai
Pull request: yes
Labels: name: wontfix

This small change intends to fix the issue #2297.
it's generally a version inconsistent issue.
in ver 1.1.0, torch.gt outputs:

_torch.gt(torch.tensor([[1, 2], [3, 4]]), torch.tensor([[1, 1], [4, 4]]))
tensor([[ 0, 1],
[ 0, 0]], dtype=torch.uint8)_

while in ver 1.2.0, it outputs:

_torch.ge(torch.tensor([[1, 2], [3, 4]]), torch.tensor([[1, 1], [4, 4]]))
tensor([[True, True], [False, True]])_

Thus, I added a version checking function, and revise the tensor type in tensor.gt()

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
revise T5 code to support one step decoding during generation
Author: eelxpeng
Pull request: yes
Labels: name: wontfix

@thomwolf Hi, I'm new to contribute to this project. I revised the T5 code to support one step decoding during generation based on your implementation. Besides adding `decode_step` function, I also revised some others to pass in `cache` variable. Meanwhile, I added `bos_token` in tokenizer_t5 so that `bos_token` can be used as the first token during decoding. The example usage is as follows:
```
Examples:
            encoder_hidden_states = model.encode(input_ids)
            cache = model.init_state_from_encoder(encoder_hidden_states)
            next_token = input_ids.new_full((batch_size, 1), tokenizer.bos_token_id)
            generated = [next_token]
            for i in range(100):
                output, cache = model.decode_step(cache, input_ids=next_token)
                next_token = torch.argmax(logits, dim=-1).unsqueeze(-1)
                generated += [next_token]
            generated = torch.cat(generated, dim=1).tolist()
```
Let me know whether it is useful and can be merged.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Improve TextDataset building/tokenization (6x faster; Enable large dataset file usage)
Author: mttcnnff
Pull request: yes
Labels: name: wontfix

## This PR:
- Chunks the reading of the dataset file used to create a `TextDataset` for training, if you used a file of any larger size (for my case I had 3.5GB txt file I was able to get finish in ~45 min) the program would just hang at `f.read()` 😢 
- Speeds up `lowercase_text` in the `BasicTokenizer` with a simpler regex scheme
- Add `@functools.lru_cache()` to several functions responsible for acting on individual chars
- Use multiprocessing to drastically speed up tokenization inside `TextDataset` constructor

## Checkouts & Performance Profiling
Benchmark script I used to clock speeds 👉 [gist](https://gist.github.com/mttcnnff/f192d8933f2a8d2b58e14b53841c4080)

Comparison script I used to compare results 👉 [gist](https://gist.github.com/mttcnnff/503f9facd3c317e7efc61a43295a335f)

### Performance on master with no changes on a ~16.5MB txt file (~1 min):
<img width="703" alt="image" src="https://user-images.githubusercontent.com/17532157/70959441-4aa1ae00-204a-11ea-95ad-b1d89baf2e09.png">

### Performance after all changes applied on same ~16.5MB txt file (~10 seconds):
<img width="788" alt="image" src="https://user-images.githubusercontent.com/17532157/70959476-6c9b3080-204a-11ea-9228-6bd55332fff0.png">

Comments:
@thomwolf or @julien-c any idea what's going on with the seg fault in the `build_py3_torch_and_tf`  [run](https://circleci.com/gh/huggingface/transformers/9811?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link)? was able to run and pass locally...
![image](https://user-images.githubusercontent.com/17532157/71120378-47253880-21aa-11ea-9f62-d0a0fd6e4ff3.png)
  

> @thomwolf or @julien-c any idea what's going on with the seg fault in the `build_py3_torch_and_tf` [run](https://circleci.com/gh/huggingface/transformers/9811?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link)? was able to run and pass locally...
> ![image](https://user-images.githubusercontent.com/17532157/71120378-47253880-21aa-11ea-9f62-d0a0fd6e4ff3.png)

I've encountered this problem too while adding my PR some days ago. It was broken something inside Transformers library, but @LysandreJik solved this bug two days ago in #2207. Therefore, you can re-launch your tests and you'll see that now it works as expected! :)

UPDATE: you've to install transformers from source code `master` branch through `pip install git+https://github.com/huggingface/transformers.git`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add "Train on Valohai" buttons to README
Author: ruksi
Pull request: yes
Labels: name: wontfix

This pull request adds two "Train on Valohai" buttons to README:

![Screenshot from 2019-12-19 16-59-54](https://user-images.githubusercontent.com/2681608/71184210-1284b000-2282-11ea-8bb8-72bf93014672.png)

and...

![Screenshot from 2019-12-19 17-00-07](https://user-images.githubusercontent.com/2681608/71184213-13b5dd00-2282-11ea-84c7-393afbbae7ea.png)

When clicked, it will automatically create a project on Valohai and let you run the project examples without any further setup. Effectively the same as "Deploy to Heroku" button if you are familiar with that.

## The flow looks like this (after login):

![Screenshot from 2019-12-19 17-00-19](https://user-images.githubusercontent.com/2681608/71184512-88891700-2282-11ea-855b-5a187ef2e700.png)
![Screenshot from 2019-12-19 17-00-34](https://user-images.githubusercontent.com/2681608/71184528-8d4dcb00-2282-11ea-8228-6322236360a9.png)
![Screenshot from 2019-12-19 17-05-31](https://user-images.githubusercontent.com/2681608/71184531-8e7ef800-2282-11ea-970a-aefa5d127069.png)
![Screenshot from 2019-12-19 17-06-18](https://user-images.githubusercontent.com/2681608/71184536-9048bb80-2282-11ea-98b0-7fd371a78677.png)


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/huggingface/transformers/pull/2227?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/62c1fc3c1ecdfab787ee3c34d1ec1eba65c18877?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2227/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2227?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2227   +/-   ##
=======================================
  Coverage   81.47%   81.47%           
=======================================
  Files         122      122           
  Lines       18344    18344           
=======================================
  Hits        14946    14946           
  Misses       3398     3398
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2227?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2227?src=pr&el=footer). Last update [62c1fc3...6bfc181](https://codecov.io/gh/huggingface/transformers/pull/2227?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Big fan of the images explaining the usage :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
tokenization progress made more sensible via tqdm
Author: iedmrc
Pull request: yes
Labels: name: wontfix

Because tokenization takes a relatively long time, having progress being visualized via tqdm would be nice.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1944?src=pr&el=h1) Report
> Merging [#1944](https://codecov.io/gh/huggingface/transformers/pull/1944?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5d3b8daad2cc6287d30f03f8a96d0a1f7bc8d0dc?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1944/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1944?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1944      +/-   ##
==========================================
+ Coverage      84%   84.01%   +<.01%     
==========================================
  Files          97       97              
  Lines       14340    14341       +1     
==========================================
+ Hits        12047    12048       +1     
  Misses       2293     2293
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1944?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1944/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `92.16% <100%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1944?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1944?src=pr&el=footer). Last update [5d3b8da...36b211d](https://codecov.io/gh/huggingface/transformers/pull/1944?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok for me.
Ok for you @LysandreJik?

I feel this might output too much text when tokenizing a lot of small sequences (which is the case for practically every example). It would be useful when tokenizing large datasets though. Maybe test if the length is superior to, say, 10000 before? What do you think?

You might be right, I have never thought about that. But it's a stubborn fact that when the time comes to tokenize larger datasets. How could we test about length? It is not about the `tokenized_text` 's length.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Some bug in using eval_all_checkpoints
Author: nike00811
Pull request: yes
Labels: name: wontfix

when using --eval_all_checkpoints
checkpoints will find a pytorch_model.bin just under output_dir
when calling evaluate(args, model, tokenizer, prefix=global_step) will get a FileNotFoundError

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2105?src=pr&el=h1) Report
> Merging [#2105](https://codecov.io/gh/huggingface/transformers/pull/2105?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0cb163865a4c761c226b151283309eedb2b1ca4d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2105/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2105?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2105   +/-   ##
=======================================
  Coverage   82.67%   82.67%           
=======================================
  Files         111      111           
  Lines       16162    16162           
=======================================
  Hits        13362    13362           
  Misses       2800     2800
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2105?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2105?src=pr&el=footer). Last update [0cb1638...4757840](https://codecov.io/gh/huggingface/transformers/pull/2105?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Sample a constant number of tokens for masking in LM finetuning
Author: rakeshchada
Pull request: yes
Labels: name: wontfix

@thomwolf @LysandreJik @julien-c Re-creating this PR as the original PR https://github.com/huggingface/transformers/pull/1555 that I created about a month ago has become quite stale. Please review.


For Masked LM fine-tuning, I think both the original BERT and RoBERTa implementations uniformly sample x number of tokens in *each* sequence for masking (where x = mlm_probability * 100 * sequence_length)
However, The current logic in run_lm_finetuning.py does an indepdendent sampling (from bernoulli distribution) for each token in the sequence. This leads to variance in the number of masked tokens (with the average number still close to x%).

The below example illustrates an extreme case, of the current logic, where no token in the input sequence is masked.

```
In [1]: import numpy as np
   ...: import torch
   ...: from transformers import BertTokenizer
   ...: 
   ...: mlm_probability = 0.15
   ...: tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
   ...: 
   ...: tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode('please mask me, o lord!', add_special_tokens=True))
   ...: 
   ...: input_ids = tokenizer.convert_tokens_to_ids(tokens)
   ...: 
   ...: inputs = torch.Tensor([input_ids])
   ...: 
   ...: labels = inputs.clone()
   ...: 
   ...: probability_matrix = torch.full(labels.shape, mlm_probability)
   ...: 
   ...: special_tokens_mask = [tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()]
   ...: probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
   ...: masked_indices = torch.bernoulli(probability_matrix).bool()
   ...: 
   ...: 

In [2]: masked_indices
Out[2]: tensor([[False, False, False, False, False, False, False, False, False]])

```

This PR modifies the logic so the percentage of masked tokens is constant (at x).

Separately, the existing and the new masking logic both rely on boolean tensors of pytorch.
So, this also updates README to include the minimum pytorch version needed. (1.2.0)

Comments:
Hi @rakeshchada, we've thought about this. It is interesting and we might merge it, but will probably will be behind a flag :)

Thanks for contributing!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix GPT2LMHeadModel.from_pretrained(from_tf=True)
Author: leogao2
Pull request: yes
Labels: name: wontfix

GPT2LMHeadModel.from_pretrained(from_tf=True) doesn't work because pointer points to the GPT2LMHeadModel instance, not the GPT2Model instance. 

This bug causes errors like:

'GPT2LMHeadModel' object has no attribute 'h'

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1796?src=pr&el=h1) Report
> Merging [#1796](https://codecov.io/gh/huggingface/transformers/pull/1796?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b5d330d11820f4ac2cc8c909b1a6a77e0cd961e0?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1796/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1796?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1796      +/-   ##
==========================================
- Coverage   84.03%   84.02%   -0.02%     
==========================================
  Files          94       94              
  Lines       14032    14034       +2     
==========================================
  Hits        11792    11792              
- Misses       2240     2242       +2
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1796?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/1796/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2dwdDIucHk=) | `83.91% <0%> (-0.54%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1796?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1796?src=pr&el=footer). Last update [b5d330d...60ebaa5](https://codecov.io/gh/huggingface/transformers/pull/1796?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks good to me!

Are you loading from a TF 1.0 or a TF 2.0 model?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.