noamaanMulla-03/transformers-issues huggingface dataset:
Fast tokenizers padding and prefix space
Author: updbqn
Pull request: no
Labels: name: Core: Tokenization

name: Should Fix

# 🐛 Bug
`pad_to_max_length=True` Doesn't seem to do anything when using fast tokenizers.

```python
tokenizer_roberta = RobertaTokenizer.from_pretrained('roberta-base')
tokenizer_roberta_fast = RobertaTokenizerFast.from_pretrained('roberta-base')

tokenizer_bert = BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer_bert_fast = BertTokenizerFast.from_pretrained('bert-base-uncased')

def test_encode_decode(name, text, tokenizer):
    encoded = tokenizer.encode(text, max_length=10, pad_to_max_length=True)
    decoded = tokenizer.decode(encoded)
    print(name)
    print(encoded)
    print(decoded)
    print()
    
text = 'hello huggingface'

test_encode_decode('bert', text, tokenizer_bert)
test_encode_decode('bert_fast', text, tokenizer_bert_fast)
test_encode_decode('roberta', text, tokenizer_roberta)
test_encode_decode('roberta_fast', text, tokenizer_roberta_fast)
```

**Output:**
```
bert
[101, 7592, 17662, 12172, 102, 0, 0, 0, 0, 0]
[CLS] hello huggingface [SEP] [PAD] [PAD] [PAD] [PAD] [PAD]

bert_fast
[101, 7592, 17662, 12172, 102]
[CLS] hello huggingface [SEP]

roberta
[0, 20760, 31164, 9021, 2, 1, 1, 1, 1, 1]

 hello huggingface
<pad><pad><pad><pad><pad>

roberta_fast
[0, 42891, 31164, 9021, 2]

hello huggingface

```
Additionally I can't seem to make `add_prefix_space=True` to work with `RobertaTokenizerFast`. I can only get the same output as `RobertaTokenizer` if i manually prepend a space. I saw the warning and link to #2778 but I'm not sure if I follow completely.

Thanks for taking the time!

Edit: I'm up to date with master, latest commit 53ce3854a16ad2a715bc6ac8af3e30c18b5a1d11

Comments:
This should be fixed in master on commit cc6775cdf5b20ad382613d3bdbf0dd8364d23219.

If you want to give it a try, otherwise the first maintenance release will soon be live.

That fixed it for me, thank you!

Also figured out what i was doing wrong with `add_prefix_space`. Instead of passing it to encode as for the regular tokenizers it should be provided at init e.g.

```python
from transformers import RobertaTokenizerFast
t = RobertaTokenizerFast.from_pretrained('roberta-base', add_prefix_space=True)
t.decode(t.encode('hello huggingface'))
# '
 hello huggingface
'
```
fix _update_memory fn call in transformer-xl
Author: andompesta
Pull request: yes

Fix bug related to #2970 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2971?src=pr&el=h1) Report
> Merging [#2971](https://codecov.io/gh/huggingface/transformers/pull/2971?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/92487a1dc03c919afa8a961ed7d8ba78fafa21bd?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2971/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2971?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2971      +/-   ##
==========================================
+ Coverage   77.14%   77.15%   +<.01%     
==========================================
  Files          98       98              
  Lines       16003    16003              
==========================================
+ Hits        12346    12347       +1     
+ Misses       3657     3656       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2971?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/2971/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsLnB5) | `75.63% <100%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2971/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.53% <0%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2971?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2971?src=pr&el=footer). Last update [92487a1...7209cf3](https://codecov.io/gh/huggingface/transformers/pull/2971?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Attempt to increase timeout for circleci slow tests
Author: sshleifer
Pull request: yes

@LysandreJik can you help me test this?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2844?src=pr&el=h1) Report
> Merging [#2844](https://codecov.io/gh/huggingface/transformers/pull/2844?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f54a5bd37f99e3933a396836cb0be0b5a497c077?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2844/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2844?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2844   +/-   ##
=======================================
  Coverage   75.02%   75.02%           
=======================================
  Files          93       93           
  Lines       15275    15275           
=======================================
  Hits        11460    11460           
  Misses       3815     3815
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2844?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2844?src=pr&el=footer). Last update [f54a5bd...68880a1](https://codecov.io/gh/huggingface/transformers/pull/2844?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Julien verbal approval :)

@sshleifer:
```
Configuration errors: 1 error occurred:

* In step 4 definition: step type "no_output_timeout" is not a valid type
```

in https://app.circleci.com/jobs/github/huggingface/transformers/18406
I am getting repetitive output when running "python run_generation.py"
Author: sunshinelala1991
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Here is the command I used to run the code:
python  run_generation.py --model_type=gpt2 --model_name_or_path=gpt2 --length 100
Here is the input and output I got:
Model prompt >>> nice to meet you
nice to meet you.

"I'm sorry, but I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet!

I have tried different inputs but the output is always repeated.


Comments:
I guess you can tune the model for better results like selecting medium large gpt model changin temp and top - p to get different predictions. 
If your new try using [write with transformer](https://transformer.huggingface.co/doc/gpt2-large) to  get an idea about it.

You could add a `repetition_penalty`. Running 
python run_generation.py --model_type=gpt2 --model_name_or_path=gpt2 --length=100 --repetition_penalty=1.2

would give:

```
Model prompt >>> nice to meet you
nice to meet you.
I'm sorry, but I don't know what's going on here." She said with a smile that made me feel like she
was trying hard to be nice and not mean or anything… "You're just saying it because we've been 
together for so long…" Her voice sounded very serious as if someone had asked her about the past 
couple of days before they'd met up in person at all!
```

You would need a very up-to-date version of transformers to make sure that the PR #2303 is included in your code to be sure that the `repetition_penalty` is working correctly.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Updates to the templates
Author: BramVanroy
Pull request: yes

This PR updates the existing GitHub templates. Main changes are:

- motivate users to post general question on Stack Overflow, tagged [huggingface-transformers](https://stackoverflow.com/questions/tagged/huggingface-transformers)
- removing the 'additional context' section as it might not add much and just bloats the template
- changed references to pytorch-transformers to transformers

closes https://github.com/huggingface/transformers/issues/2529

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2664?src=pr&el=h1) Report
> Merging [#2664](https://codecov.io/gh/huggingface/transformers/pull/2664?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ea2600bd5f1d36f2fb61958be21db5b901e33884?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2664/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2664?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2664   +/-   ##
=======================================
  Coverage   74.51%   74.51%           
=======================================
  Files          87       87           
  Lines       14920    14920           
=======================================
  Hits        11117    11117           
  Misses       3803     3803
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2664?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2664?src=pr&el=footer). Last update [ea2600b...1cfc4af](https://codecov.io/gh/huggingface/transformers/pull/2664?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Bug in transfo_xl function call
Author: andompesta
Pull request: no
Labels: name: Core: Modeling

# 🐛 Bug
Miss-order in the function call.
In line 785  of modeling_transfo_xl.py you call the ``new_mems = self._update_mems(hids, mems, mlen, qlen)``. Yet, the function declaration is ``def _update_mems(self, hids, mems, qlen, mlen)``

Apparently you invert the order of ``qlen`` and ``mlen``.

Yet the training performances are not affected if ``ext_len = 0`` which is the default setting.

## Information
Related to https://github.com/kimiyoung/transformer-xl/issues/96

Model I am using (Bert, XLNet ...): Transformer-XL (modeling_transfo_xl.py)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
There is no issue during training, but it might affect the performance in evaluation

## Environment info
     
- `transformers` version: 2.5.0
- Platform: MACOS
- Python version: 3.6.0
- PyTorch version (GPU?): 1.2.0 
- Tensorflow version (GPU?):
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: yes


Comments:
Removing redundant model weights
Author: bacicnikola
Pull request: no
Labels: name: wontfix

## 🐛 This is not a bug, more like implemetation detail

I am using BertForTokenClassification model for my binary token classification problem. If my understanding is right, BertForTokenClassification has one layer on top with num_classes output neurons (one for each class) with softmax activation function + CrossEntropyLoss().

Now, if your problem has >2 classes this is completely fine, but if num_classes=2 you are both modeling P(input = 0) and P(input = 1), and it's easy to see why this is redunant.

num_classes=2 is special case and it should be implemented with only one output neuron with sigmoid activation function + binary cross-entopy.


Please correct me if I am wrong :)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Albert Hyperparameters for Fine-tuning SQuAD 2.0
Author: ahotrod
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I want to fine-tune `albert-xxlarge-v1` on SQuAD 2.0 and am in need of optimal hyperparameters.  I did not find any discussion in the Albert original paper regarding suggested fine-tuning hyperparameters, as is provided in the XLNet original paper.  I did find the following hard-coded parameters in the Google-research Albert `run_squad_sp.py` code:
```
'do_lower_case' = True
'train_batch_size' = 32
'predict_batch_size' = 8
'learning_rate' = 5e-5
'num_train_epochs' = 3.0
'warmup_proportion' = 0.1
```
With fine-tuning on my 2x GPUs taking ~69 hours, I'd like to shrink the number of fine-tuning iterations necessary to attain optimal model performance.  Anyone have a bead on the optimal hyperparameters?

Also, Google-research comments in `run_squad_sp.py` state that `warmup_proportion` is "Proportion of training to perform linear learning rate warmup for."  "E.g.,  0.1 = 10% of training".  Since 3 epochs, batch size = 32 while fine-tuning SQuAD 2.0 results in approximately 12.5K total optimization steps, would I set `--warmup_steps = 1250` when calling Transformers' run_squad.py?

Thanks in advance for any input.

Comments:
Wondering this as well but for GLUE tasks. There don't seem to be a good consensus on hyperparameters such as weight decay and such

Results using hyperparameters from my first post above, varying only batch size:
```
albert_xxlargev1_squad2_512_bs32:
{
  "exact": 83.67725090541565,
  "f1": 87.51235434089064,
  "total": 11873,
  "HasAns_exact": 81.86572199730094,
  "HasAns_f1": 89.54692697189559,
  "HasAns_total": 5928,
  "NoAns_exact": 85.48359966358284,
  "NoAns_f1": 85.48359966358284,
  "NoAns_total": 5945
}

albert_xxlargev1_squad2_512_bs48:
{
  "exact": 83.65198349195654,
  "f1": 87.4736247587816,
  "total": 11873,
  "HasAns_exact": 81.73076923076923,
  "HasAns_f1": 89.38501126197984,
  "HasAns_total": 5928,
  "NoAns_exact": 85.5677039529016,
  "NoAns_f1": 85.5677039529016,
  "NoAns_total": 5945
}
```
![lr](https://user-images.githubusercontent.com/44321615/70385030-2bb26600-193e-11ea-9af3-0ed6460c3b7a.jpg)
![loss](https://user-images.githubusercontent.com/44321615/70385034-3240dd80-193e-11ea-8133-2720567426e6.jpg)


@ahotrod There is a table in the appendix section of the ALBERT paper, which shows hyperparameters for ALBERT in downstream tasks:
![image](https://user-images.githubusercontent.com/26627380/71256582-1db40a80-236c-11ea-8141-e74278e5f5c5.png)
 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
missing name entity recognition link
Author: autoih
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2967?src=pr&el=h1) Report
> Merging [#2967](https://codecov.io/gh/huggingface/transformers/pull/2967?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/94ff2d6ee8280c5595b92c1128c0f18e44925e56?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2967/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2967?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2967      +/-   ##
==========================================
+ Coverage   77.11%   77.12%   +<.01%     
==========================================
  Files          98       98              
  Lines       15977    15977              
==========================================
+ Hits        12321    12322       +1     
+ Misses       3656     3655       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2967?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2967/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.53% <0%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2967?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2967?src=pr&el=footer). Last update [94ff2d6...bede5f4](https://codecov.io/gh/huggingface/transformers/pull/2967?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Change masking to direct labeling for TPU support.
Author: srush
Pull request: yes

Torch XLA conversion is very sensitive to certain core pytorch operations. These result in TPU being slower than CPU operation. The most obvious are binary masking and calls to item(). 

https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md

This PR replaces some of these calls that are in the example/NER pathway. 


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2982?src=pr&el=h1) Report
> Merging [#2982](https://codecov.io/gh/huggingface/transformers/pull/2982?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d1ab1fab1be7199e082129dfbe46eb52bca92799?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `62.5%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2982/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2982?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2982      +/-   ##
==========================================
+ Coverage   74.09%   74.09%   +<.01%     
==========================================
  Files          93       93              
  Lines       15249    15253       +4     
==========================================
+ Hits        11298    11301       +3     
- Misses       3951     3952       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2982/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `64.95% <0%> (-0.31%)` | :arrow_down: |
| [src/transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/2982/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19kaXN0aWxiZXJ0LnB5) | `95.87% <100%> (+0.01%)` | :arrow_up: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2982/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `87.92% <100%> (+0.02%)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2982/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.25% <50%> (+0.04%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2982?src=pr&el=footer). Last update [d1ab1fa...757e2c3](https://codecov.io/gh/huggingface/transformers/pull/2982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
batch_encode_plus with pad_to_max_length is not padding the output
Author: dipanjannag
Pull request: no
Labels: name: Core: Tokenization

name: Version mismatch

# 🐛 Bug

## Information

Model I am using (BertTokenizer,):

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts.

The tasks I am working on is:
* [ ] Simple batch tokenization task

## To reproduce
```
from transformers import BertTokenizer, BertForQuestionAnswering
import torch

import numpy as np

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

qt1 = ('What is the name of the repository?', 'pipeline have been included in the huggingface/transformers repository')
qt2 = ('What is the name of the repository?', 'What can be done with this???')

inp_raw = [qt1, qt2]
tck_temp = tokenizer.batch_encode_plus(inp_raw, max_length=20, pad_to_max_length=True)

print(tck_temp)
inp_ids = tck_temp['input_ids']
tck_type_ids = tck_temp['token_type_ids']


print(len(inp_ids[0]) == len(inp_ids[1]))
## This is coming false
```

## Expected behavior

The code snippet should print True
But it prints False

## Environment info
     
- `transformers` version: 2.3
- Platform: Linux
- Python version: 3.6
- PyTorch version (1.4CPU)


[EDIT] : correct transformers version

Comments:
The code snippet prints `True` for me.
I have the versions:
transformers version: 2.5.1
Platform: Linux
Python version: 3.6.9
PyTorch version: 1.4.0+cpu
Can you try `pip install --upgrade transformers` and see whether the error is still there?

(Posted on behalf of @patrickvonplaten who's having issues with Github)

Thanks @TevenLeScao it seems the issue was only for transformers ~2.3