noamaanMulla-03/transformers-issues huggingface dataset:
Why does padding affect the embedding results for XLNet? Pre-padding returns different embeddings than post-padding. Which one should be used?
Author: osmanbaskaya
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello, I am confused with different results of XLNet depending on padding. For Bert, padding doesn't affect the outputs, but for XLNet with **pre** padding (which I saw in https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L281), returns very different results for the same sentence, with and without padding. The difference between no padding and post padding for XLNet returns similar results. Why did "pre" padding is used for run_glue.py? Does XLNet expect post padding or pre padding? Is there any document I am missing to clarify those important distinctions?

Here is a demonstration of the differences for pre, post padding for BERT and XLNet: https://colab.research.google.com/drive/1PCiU3icdfUB-nrLbrKAhgpePoIcFRlqX

Thanks,
Osman

Comments:
I might be wrong, but intuitively I would say that that makes things easier. XLNet expects single sequences that look like this `tok1 tok2 ... SEP CLS`. So in contrast with BERT, the classification token is at the end of a sentence rather than beginning. This is before padding. So if you use post-padding, the position of the CLS element can differ for each element in your batch, but if you use pre-padding, then you can access the CLS element by its `-1` index.

That's not to say that it's not possible to retrieve the CLS element in XLNet when you've used post-padding. Something like this should work. Find the position (indices) where the input IDs are the classification token, then use those indices to slice the output.

```python
output = output[torch.where(input_ids == tokenizer.cls_token_id)]
```

If you've used pre-padding, this can be simplified to

```python
output = output[:, -1]
```

Hey @BramVanroy, thanks for the answer. You may be right that pre padding for XLNet makes things easier (i.e., getting `[cls]` token from the last index) but the question I want to be answered is not why we would like "pre" padding but why pre padding and post padding gives different answers. Maybe I should change the title further. If you see the notebook I shared, depending on padding you're getting different results.

Sorry, I fear I can't help with that. I am also wondering when padding is necessary and when it isn't. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Maybe too late for this thread, but any answers to this issue?

Any answer on this issue?

Any answer on this issue?
run_ner.py crashes with RoBERTa because of incorrect sequence length
Author: oadams
Pull request: no

# 🐛 Bug

## Information

I'm running `examples/token-classification/run_ner.py` with RoBERTa. An assert statement fails:

```
assert len(input_ids) == max_seq_length
AssertionError
```

Looks like the cause is a mismatch between the value of roberta's `tokenizer.num_special_tokens_to_add()` and the number of special tokens that is actually added to the sequence in `utils_ner.py::convert_examples_to_features()`.

Specifically, `tokenizer.num_special_tokens_to_add()` is 2 (presumably for `
` and `
`). However, `convert_examples_to_features()` adds an extra `
` token at line 331, in addition to the `
` token and first `
` token. So the result is that there are three special tokens, and the sequence ends with `
 
`.

`convert_examples_to_features()` relies on `num_special_tokens_to_add()` to determine how many content tokens from the sequence to use, but because of the mismatch above, you can end up with a sequence length of 129 even when the sequence length was set to a max of 128.

To reproduce this, just follow the instructions at `examples/token-classification/README.md` except use the flag `--model_name_or_path roberta-base`.

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)


## Environment info
- `transformers` version: 2.11.0
- Platform: Linux-4.15.0-101-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: fails in both cases
- Using distributed or parallel set-up in script?: no



Comments:
getting same error. any solution yet

The quick temp fix would be to change the num_special_tokens_to_add() to be 3 instead OR not write the extra sep tag since it's single sequence tagging.

I'm guessing the second option is more appropriate because the code that adds the extra sep tag is in utils_ner.py, and as far as I'm aware NER should never involve more than one sep tag.

I will prepare a fix for that soon!
Attribute error while using run_language_modeling.py with Transformer-XL
Author: TakLaszlo
Pull request: no

python3 /home/username/.local/lib/python3.7/site-packages/transformers/transformers/examples/run_language_modeling.py \
    --output_dir=_outpath_
    --model_type=Transformer-XL\
    --model_name_or_path=transfo-xl-wt103 \
    --do_train \
    --train_data_file=_pathtodataset_
    --per_gpu_train_batch_size=1\
    --gradient_accumulation_steps=30\
    --train_epochs=50

Every time I run the above script I get the following error:

Traceback (most recent call last):
  File "/content/Essay/run_language_modeling.py", line 284, in <module>
    main()
  File "/content/Essay/run_language_modeling.py", line 208, in main
    model.resize_token_embeddings(len(tokenizer))
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py", line 336, in resize_token_embeddings
    model_embeds = base_model._resize_token_embeddings(new_num_tokens)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py", line 351, in _resize_token_embeddings
    new_embeddings = self._get_resized_embeddings(old_embeddings, new_num_tokens)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py", line 372, in _get_resized_embeddings
    old_num_tokens, old_embedding_dim = old_embeddings.weight.size()
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 576, in __getattr__
    type(self).__name__, name))
**AttributeError: 'AdaptiveEmbedding' object has no attribute 'weight'**
| | |
|----|---|
|OS |Fedora 29/Google Colab|
|Python|3.6|
|Pytorch|1.4.0/1.5.0|

The problem arises in each 4 cases with the XL model but not with other models.

Comments:
Hey, sorry, it's been a while, but this was fixed in #4759 since.
Create README.md
Author: ipuneetrathore
Pull request: yes
Labels: name: model card

Adding readme file for the model. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4975?src=pr&el=h1) Report
> Merging [#4975](https://codecov.io/gh/huggingface/transformers/pull/4975?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ca5e1cdf8e314288bd0242a531815a6c75d8178e&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4975/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4975?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4975      +/-   ##
==========================================
- Coverage   77.26%   77.26%   -0.01%     
==========================================
  Files         128      128              
  Lines       21851    21851              
==========================================
- Hits        16884    16883       -1     
- Misses       4967     4968       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4975?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4975/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.38% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4975/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.26% <0.00%> (-0.16%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4975/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.61% <0.00%> (+0.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4975?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4975?src=pr&el=footer). Last update [ca5e1cd...03bddda](https://codecov.io/gh/huggingface/transformers/pull/4975?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Changes to fix working for transformer-xl
Author: bajajahsaas
Pull request: yes

Update arguments (`lm_labels` to `labels`)
Use labels as data; labels are shifted inside the model (https://github.com/bajajahsaas/transformers/blob/master/src/transformers/modeling_transfo_xl.py#L855)

Comments:
Sorry, it's been a while, but this has since been fixed in #3716.
Hans data
Author: sgugger
Pull request: yes

This is the first step toward solving #4742: to be able to use the Trainer API, we first need to remove the TensorDataset and have datasets with dict items. This PR addresses that and updates the training and evaluation script accordingly.

It takes the multiple choice as a reference implementation, using the same file structure (hence the removal of "hans_processor.py") and implements in "utils_hans.py":
- a `HansDataset` and a `TFHansDataset` that implement the logic of the old method `load_and_cache_examples`
- a `HansProcessor` (copied from before)
- a `hans_convert_examples_to_features` with the same logic as before but using the tokenizer method for padding instead of re-implementing it.

Side question: it doesn't look like the `TFMultipleChoiceDataset` I use as a reference for this implementation uses caching, maybe it should be added in the future?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4854?src=pr&el=h1) Report
> Merging [#4854](https://codecov.io/gh/huggingface/transformers/pull/4854?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ca5e1cdf8e314288bd0242a531815a6c75d8178e&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4854/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4854?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4854   +/-   ##
=======================================
  Coverage   77.26%   77.26%           
=======================================
  Files         128      128           
  Lines       21851    21851           
=======================================
  Hits        16884    16884           
  Misses       4967     4967           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4854?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4854/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.38% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4854/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.61% <0.00%> (+0.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4854?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4854?src=pr&el=footer). Last update [ca5e1cd...a58291b](https://codecov.io/gh/huggingface/transformers/pull/4854?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
run_pplm.py bug fix
Author: songyouwei
Pull request: yes

`is_leaf` may become `False` after `.to(device=device)` function call.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4867?src=pr&el=h1) Report
> Merging [#4867](https://codecov.io/gh/huggingface/transformers/pull/4867?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9f5d5a531d769d07403f59661884e254f8420afe&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4867/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4867?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4867   +/-   ##
=======================================
  Coverage   76.55%   76.56%           
=======================================
  Files         128      128           
  Lines       21502    21502           
=======================================
+ Hits        16461    16462    +1     
+ Misses       5041     5040    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4867?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4867/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.12% <0.00%> (-0.48%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4867/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.62% <0.00%> (+0.94%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4867?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4867?src=pr&el=footer). Last update [9f5d5a5...26de790](https://codecov.io/gh/huggingface/transformers/pull/4867?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@songyouwei  run_pplm.py  Can you follow the Readme steps to success run it?   And  I report  this error?  Could you teach me? 

![image](https://user-images.githubusercontent.com/49581245/84686252-334a6b80-af6e-11ea-86c5-b973424ac8e7.png)
Increase pipeline support for ONNX export.
Author: mfuntowicz
Pull request: yes

 Supports all kind of models for all the pipelines we've except `summarization` because of `triu`not supported by ONNX.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5005?src=pr&el=h1) Report
> Merging [#5005](https://codecov.io/gh/huggingface/transformers/pull/5005?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9931f817b75ecb2c8bb08b6e9d4cbec4b0933935&el=desc) will **increase** coverage by `0.37%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5005/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5005?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5005      +/-   ##
==========================================
+ Coverage   76.89%   77.26%   +0.37%     
==========================================
  Files         128      128              
  Lines       21854    21854              
==========================================
+ Hits        16804    16886      +82     
+ Misses       5050     4968      -82     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5005?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5005/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.79% <0.00%> (+0.40%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/5005/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG0ucHk=) | `75.85% <0.00%> (+19.75%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5005?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5005?src=pr&el=footer). Last update [9931f81...9f4da89](https://codecov.io/gh/huggingface/transformers/pull/5005?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[🚀 Feature request] upload bart-base checkpoint
Author: patil-suraj
Pull request: no

# 🚀 Feature request

`bart-base`  checkpoint is now available in fairseq repo 
https://github.com/pytorch/fairseq/tree/master/examples/bart#pre-trained-models
so now it can be made available here  `facebook/bart-base` 

@sshleifer 

Comments:
Thanks for letting me know @patil-suraj !

This is almost done, blocked on 1 minor [issue](https://github.com/pytorch/fairseq/issues/2242)

If I don't get an answer there by Wednesday I will assume that the tokenizer is the same as bart.large and move forward.

Thanks @sshleifer  for immediate response and PR :)
Add bart-base
Author: sshleifer
Pull request: yes

Also adds two `@slow` integration test for fill mask tasks.
I made some guesses on the correct conversion, which will hopefully be confirmed by the authors in this [issue](https://github.com/pytorch/fairseq/issues/2242)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5014?src=pr&el=h1) Report
> Merging [#5014](https://codecov.io/gh/huggingface/transformers/pull/5014?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/66bcfbb130a3a60d873597fdca05a8d539ef3401&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5014/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5014?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5014      +/-   ##
==========================================
+ Coverage   77.20%   77.26%   +0.05%     
==========================================
  Files         128      128              
  Lines       21854    21854              
==========================================
+ Hits        16873    16885      +12     
+ Misses       4981     4969      -12     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5014?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/5014/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmFydC5weQ==) | `95.12% <ø> (ø)` | |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5014/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.25% <0.00%> (-0.24%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5014/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.26% <0.00%> (-0.16%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5014/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `78.65% <0.00%> (+2.29%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5014?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5014?src=pr&el=footer). Last update [66bcfbb...0dd162e](https://codecov.io/gh/huggingface/transformers/pull/5014?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


add a model card too if needed/useful

Will and model cards for this and bart-large at some point.
Latest merge [Benchmark] Memory benchmark utils #4198 fails at Windows
Author: songsuoyuan
Pull request: no

# 🐛 Bug

## Information

Model I am using: Bert

## To reproduce

Steps to reproduce the behavior:

1. Windows
2. Install transformers from source

## Environment info
python = 3.7.6
pytorch = 1.5
cuda = 10.2

I think the problem happens because in code: 
https://github.com/huggingface/transformers/blob/master/src/transformers/__init__.py

`from .benchmark import PyTorchBenchmark, PyTorchBenchmarkArguments`

and in code:
https://github.com/huggingface/transformers/blob/master/src/transformers/benchmark/benchmark_utils.py

there is `from signal import SIGKILL`

> signal.SIGKILL
> 
> Kill signal.
> It cannot be caught, blocked, or ignored.
> Availability: Unix.

I think it will fail for Windows if you try `from signal import SIGKILL`.

@patrickvonplaten 
https://github.com/huggingface/transformers/pull/4198

Comments:
Thanks for noting that - you're 100% correct :-)