noamaanMulla-03/transformers-issues huggingface dataset:
Zero-shot PPLs vary wildly across gpt2 model sizes
Author: g-karthik
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

## Details

We've been experimenting with multiple datasets for fine-tuning both OpenAI GPT and GPT-2.

What we've noticed is that the zero-shot PPL of GPT-2 small and medium are in the order of `3e+40`. However, the zero-shot PPL of GPT-2 large and XL on the same datasets are much more reasonable, like ~30. The zero-shot PPL of OpenAI GPT on the same datasets are also reasonable, on the order of 200-300.

We're trying to understand why the pre-trained GPT-2 small and medium models display such vastly different zero-shot PPLs in comparison to large, XL and OpenAI GPT.

Has anyone else also noticed the same in their experiments? What might the cause for this be?



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Examples link to the master branch which seems to be ahead of the pip install version?
Author: shenkev
Pull request: no
Labels: name: wontfix

Documentation for training a language model points to https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py

The import 
`from transformers import (
    CONFIG_MAPPING,
    MODEL_WITH_LM_HEAD_MAPPING,
    AutoConfig,
    AutoModelWithLMHead,
    AutoTokenizer,
    DataCollatorForLanguageModeling,
    HfArgumentParser,
    LineByLineTextDataset,
    PreTrainedTokenizer,
    TextDataset,
    Trainer,
    TrainingArguments,
    set_seed,
)`

leads to "No name ___ in module transformers" errors.

I installed transformers using pip.

`pip install -U transformers`

The import problems seem to be gone when I install from source.

Comments:
I would use the version installed from source but training a tokenizer is substantially slower than the pip install version. When I run the code snippet below, the pip install version finishesin ~5 minutes but the install-from-source version takes 20+ (I didn't wait for it to finish). For some reason there's a new "Reading files" stage which takes forever to finish.

```import os
from tokenizers import ByteLevelBPETokenizer, BertWordPieceTokenizer


tokenizer = ByteLevelBPETokenizer(lowercase=False)

paths = ["./data.txt"]

tokenizer.train(files=paths, vocab_size=52000, min_frequency=2, special_tokens=[
    "
",
    "<pad>",
    "
",
    "<unk>",
    "<mask>",
])

out_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "token")
os.makedirs(out_path, exist_ok=True)

tokenizer.save(out_path)```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Tf to pytorch
Author: Sagar1094
Pull request: no

Hi, I have XLNet checkpoint file and want to convert it to pytorch. I have converted the file to pytorch and when I am using the same file with simpletramsformers for text classification. It says the number of labels is not matching. Where should I specify the number of classes while converting from tf to pytorch?

Comments:
#Code used to convert from tf to Pytorch

import torch

from transformers import (
    CONFIG_NAME,
    WEIGHTS_NAME,
    XLNetConfig,
    XLNetForQuestionAnswering,
    XLNetForSequenceClassification,
    XLNetLMHeadModel,
    load_tf_weights_in_xlnet,
    XLNetModel
)


tf_checkpoint_path="./model/"
xlnet_config_file = "./model/config.json"
pytorch_dump_path="./xlnetpytorch/"

config = XLNetConfig.from_json_file(xlnet_config_file)
print("Building PyTorch model from configuration: {}".format(str(config)))
model = XLNetForSequenceClassification(config)

#Load weights from tf checkpoint
model =   load_tf_weights_in_xlnet(model, config, tf_checkpoint_path)

#Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
model.save_pretrained(pytorch_dump_path)

Use config.num_labels = 536 (number of labels) in my case  to define the number of labels in custom data set before initialising model i.e., before line model = XLNetForSequenceClassification(config)
T5 Masking:
Author: zbush548
Pull request: no

While this works, the outputs are not according to what is specified in "model.generate."

```
# tokenizer = T5Tokenizer.from_pretrained(model_name)
# model = T5ForConditionalGeneration.from_pretrained(model_name)
# # Input text

# text = 'This <extra_id_0> sentence. 
'

# encoded = tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')
# input_ids = encoded['input_ids']

# # Generating 20 sequences with maximum length set to 5
# outputs = model.generate(input_ids=input_ids, 
#                           num_beams=200, num_return_sequences=20,
#                           min_length = 5, max_length=10)

# _0_index = text.index('<extra_id_0>')
# _result_prefix = text[:_0_index]
# _result_suffix = text[_0_index+12:]  # 12 is the length of <extra_id_0>

# def _filter(output, end_token='<extra_id_1>'):
#     # The first token is <unk> (inidex at 0) and the second token is <extra_id_0> (indexed at 32099)
#     _txt = tokenizer.decode(output[2:], skip_special_tokens=False, clean_up_tokenization_spaces=False)
#     if end_token in _txt:
#         _end_token_index = _txt.index(end_token)
#         return _result_prefix + _txt[:_end_token_index] + _result_suffix
#     else:
#         return _result_prefix + _txt + _result_suffix

# results = list(map(_filter, outputs))
```

Example Output:

`This perception holds no validity.`

As can be seen, "holds no" is only two words despite that I put the "min_length" as five.

Comments:
`min_length` corresponds to the minimum number of tokens, not words.

Feel free to reopen if this does not answer your question.
Fix typo in training
Author: ELanning
Pull request: yes

Fixes a small typo in the code example.

Ref code: https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1211

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5510?src=pr&el=h1) Report
> Merging [#5510](https://codecov.io/gh/huggingface/transformers/pull/5510?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/58cca47c16149e43d1b516623d59e3c5d97f695e&el=desc) will **decrease** coverage by `0.41%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5510/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5510?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5510      +/-   ##
==========================================
- Coverage   77.83%   77.42%   -0.41%     
==========================================
  Files         141      141              
  Lines       24634    24634              
==========================================
- Hits        19175    19074     -101     
- Misses       5459     5560     +101     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5510?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `26.92% <0.00%> (-68.47%)` | :arrow_down: |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/5510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `28.84% <0.00%> (-58.66%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `79.59% <0.00%> (+0.33%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+1.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `98.76% <0.00%> (+32.51%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5510?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5510?src=pr&el=footer). Last update [58cca47...121648f](https://codecov.io/gh/huggingface/transformers/pull/5510?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks a lot!
Typo fix in `training` doc
Author: arnavsharma93
Pull request: yes

`provides` -> `provided`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5495?src=pr&el=h1) Report
> Merging [#5495](https://codecov.io/gh/huggingface/transformers/pull/5495?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8438bab38e1ea60efca181c92ebc7e4602f91848&el=desc) will **increase** coverage by `0.76%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5495/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5495?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5495      +/-   ##
==========================================
+ Coverage   76.97%   77.74%   +0.76%     
==========================================
  Files         141      141              
  Lines       24617    24617              
==========================================
+ Hits        18950    19138     +188     
+ Misses       5667     5479     -188     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5495?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/5495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `71.83% <0.00%> (-23.95%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.41% <0.00%> (-2.27%)` | :arrow_down: |
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/5495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `94.21% <0.00%> (+1.32%)` | :arrow_up: |
| [src/transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/5495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdHJhbnNmb194bC5weQ==) | `42.48% <0.00%> (+8.92%)` | :arrow_up: |
| [src/transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/5495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtLnB5) | `82.93% <0.00%> (+66.66%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5495?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5495?src=pr&el=footer). Last update [8438bab...74a8457](https://codecov.io/gh/huggingface/transformers/pull/5495?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for the fix!
Cannot loading SpanBERT pre-trained model 
Author: changyeli
Pull request: no
Labels: name: wontfix

I have some questions regarding SpanBert loading using the transformers package.

I downloaded the pre-trained file from [SpanBert](https://github.com/facebookresearch/SpanBERT) GitHub Repo and ```vocab.txt``` from Bert. Here is the code I used for loading:

```python
model = BertModel.from_pretrained(config_file=config_file,
                                  pretrained_model_name_or_path=model_file,
                                  vocab_file=vocab_file)
model.to("cuda")
```

where 

- ```config_file``` -> ```config.json```
- ```model_file``` -> ```pytorch_model.bin```
- ```vocab_file``` -> ```vocab.txt```

But I got the ```UnicodeDecoderError``` with the above code saying that ```'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte```

I also tried loading SpanBert with the method mentioned [here](https://huggingface.co/SpanBERT/spanbert-large-cased). But it returned ```OSError: file SpanBERT/spanbert-base-cased not found```.

Do you have any suggestions on loading the pre-trained model correctly? Any suggestions are much appreciated. Thanks!

Comments:
@palooney 
Can you check again ?. The following call worked for me
```
model = AutoModel.from_pretrained("SpanBERT/spanbert-base-cased")
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Closing, feel free to reopen if it isn't fixed.
[fix] pin sacrebleu to fix CI ImportError
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5529?src=pr&el=h1) Report
> Merging [#5529](https://codecov.io/gh/huggingface/transformers/pull/5529?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/58cca47c16149e43d1b516623d59e3c5d97f695e&el=desc) will **decrease** coverage by `0.34%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5529/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5529?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5529      +/-   ##
==========================================
- Coverage   77.83%   77.49%   -0.35%     
==========================================
  Files         141      141              
  Lines       24634    24634              
==========================================
- Hits        19175    19090      -85     
- Misses       5459     5544      +85     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5529?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `26.92% <0.00%> (-68.47%)` | :arrow_down: |
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/5529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `33.33% <0.00%> (-24.33%)` | :arrow_down: |
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/5529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `71.83% <0.00%> (-23.95%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `79.59% <0.00%> (+0.33%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.21% <0.00%> (+1.00%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `98.76% <0.00%> (+32.51%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5529?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5529?src=pr&el=footer). Last update [58cca47...e21636a](https://codecov.io/gh/huggingface/transformers/pull/5529?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
incorrect typehint for PreTrainedTokenizer.convert_ids_to_tokens() return value
Author: andifunke
Pull request: no

# 🐛 Bug

## Information

The return value of `tokenization_utils.PreTrainedTokenizer.convert_ids_to_tokens()` is declared as `Union[int, List[int]]` while the method clearly returns a `Union[str, List[str]]` object type.

Would you like me to open a PR? ;)

Thanks for the great work!
Andreas

PS: Excuse me for skipping my system specs since this is a rather small issue and I'm just a bit annoyed by PyCharm's inspection warning on this ;)

Comments:
We want to make a clean patched release soon, so I opened #5551 to make sure it's fixed in it.
In the future, don't hesitate to directly open a PR for an issue with a clear fix like this :-)

Closed via #5551
getting different model result from tokenizer vs tokenizer.encode function
Author: monk1337
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

I am using gpt2 model to encode sentences. I am confused between tokenizer.encode vs tokenizer : 

If I am using tokenizer.encode: 

```

sentence = 'checking single sentences'

from transformers import GPT2Tokenizer, GPT2Model
import torch

    
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
model     = GPT2Model.from_pretrained('gpt2-medium')
input_ids = torch.tensor(tokenizer.encode(sentence)).unsqueeze(0)  # Batch size 1
outputs   = model(input_ids)
outputs[0]
```

output is  : 

```
tensor([[[ 0.6804,  0.4182,  0.3483,  ..., -0.3102,  0.0341,  0.4901],
         [-0.4060,  0.7790,  0.2695,  ..., -0.4763,  0.1817,  0.0600],
         [ 0.7916,  0.6078,  0.4642,  ..., -0.5557, -0.1571, -0.1220]]],
       grad_fn=<ViewBackward>)
```

While using tokenizer alone : 

```
from transformers import GPT2Tokenizer, GPT2Model
sentence = 'checking single sentences'


tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
model     = GPT2Model.from_pretrained('gpt2-medium')
inputs    = tokenizer(sentence, return_tensors="pt")
outputs   = model(**inputs)
outputs[0]
```

output is : 

```
# tensor([[[ 0.5511,  0.4063,  0.2453,  ..., -0.4217,  0.1519,  0.0898],
#          [-0.1138, -0.0232,  0.1736,  ..., -0.5408,  0.0145,  0.2985],
#          [ 0.1856,  0.3127,  0.2430,  ..., -0.7532, -0.2332,  0.1506]]],
#        grad_fn=<ViewBackward>)
```

I tried to examine the output of both tokenizer : 

```
from transformers import GPT2Tokenizer, GPT2Model
import torch

sentence = 'checking single sentences'


tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
input_ids = tokenizer.encode(sentence)
inputs    = tokenizer(sentence, return_tensors="pt")

print(input_ids)

print(inputs)
```

output : 

```
[41004, 2060, 13439]
{'input_ids': tensor([[41004,  2060, 13439]]), 'token_type_ids': tensor([[0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1]])}
```

Which is recommended tokenizer function? 


Comments:
GPT2 tokenizer should not output token type IDs
Author: LysandreJik
Pull request: yes

The GPT-2 and OpenAI GPT tokenizers should not output `token_type_ids` by default.

closes https://github.com/huggingface/transformers/issues/5517

code quality will pass when merged in master.

Edit (@thomwolf):
This will also fix #4922

Comments:
WARNING:transformers.tokenization_utils:Keyword arguments {'add_space_before_punct_symbol': True} not recognized.
Author: songproducer
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
ctrl

Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: (give details below)
```
python run_generation.py \
    --model_type ctrl \
    --model_name ctrl --length=100 --temperature 0.2 --num_return_sequences=5 --p=0.8 --seed=17 --repetition_penalty=1.2 
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

```
- `transformers` version: 3.0.1
- Platform: Darwin-19.5.0-x86_64-i386-64bit
- Python version: 3.7.4
- PyTorch version (GPU?): 1.2.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```

Comments:
AdamW step device error
Author: ZhaofengWu
Pull request: no

I have a model:
```
class M(nn.Module):
  def __init__(self, ...):
    self.A = nn.Parameter(...)
    self.B = nn.Parameter(...)
    self.C = torch.einsum(..., self.A, self.B)

  def forward(self, D):
    return func(self.C.to(D.device), D)
```

However, I'm getting the following error when training this model on GPU:
```
File "file.py", line 71, in optimizer_step
optimizer.step()
File ".../lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
return wrapped(*args, **kwargs)
File ".../lib/python3.7/site-packages/transformers/optimization.py", line 155, in step
exp_avg.mul_(beta1).add_(grad, alpha=1.0 - beta1)
RuntimeError: expected device cuda:0 but got device cpu
```

This is happening in 2.11.0. Am I doing something incorrectly here?

Comments: