noamaanMulla-03/transformers-issues huggingface dataset:
add_ctags_to_git_ignore
Author: patrickvonplaten
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2984?src=pr&el=h1) Report
> Merging [#2984](https://codecov.io/gh/huggingface/transformers/pull/2984?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/129f0604acb9e8b9cebd2897437324198fa37a0a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2984/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2984?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2984   +/-   ##
=======================================
  Coverage   77.17%   77.17%           
=======================================
  Files          98       98           
  Lines       15997    15997           
=======================================
  Hits        12345    12345           
  Misses       3652     3652
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2984?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2984?src=pr&el=footer). Last update [129f060...5636868](https://codecov.io/gh/huggingface/transformers/pull/2984?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
evaluating on race dataset with checkpoints fine tuned on roberta with fairseq
Author: qshi95
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I fine tuned a model on race dataset with reberta, following the fairseq instruction, got the result:
| epoch 004 | valid on 'valid' subset:  | loss 0.913 | nll_loss 0.003 | ppl 1.00 | num_updates 21849 | best_accuracy 0.846563 | accuracy 0.836129
| epoch 004 | valid on 'valid' subset:  | loss 0.913 | nll_loss 0.003 | ppl 1.00 | num_updates 21849 | best_accuracy 0.846563 | accuracy 0.836129
| epoch 004 | valid on 'valid' subset:  | loss 0.913 | nll_loss 0.003 | ppl 1.00 | num_updates 21849 | best_accuracy 0.846563 | accuracy 0.836129
| epoch 004 | valid on 'valid' subset:  | loss 0.913 | nll_loss 0.003 | ppl 1.00 | num_updates 21849 | best_accuracy 0.846563 | accuracy 0.836129
| saved checkpoint checkpoints/checkpoint4.pt (epoch 4 @ 21849 updates) (writing took 145.8246190547943 seconds)
| done training in 76377.9 seconds

But I load the weight to the transformers with convert_roberta_original_pytorch_checkpoint_to_pytorch script:

python convert_roberta_original_pytorch_checkpoint_to_pytorch.py --roberta_checkpoint_path ../pytorch-transformers-master/data/roberta-best-checkpoint/ --pytorch_dump_folder_path ../pytorch-transformers-master/data/roberta-best-checkpoint/

then evaluate on RACE dataset, I got terrible results on dev set:
model           =data/models_roberta_race/
total batch size=8
train num epochs=5
fp16            =False
max seq length  =512
eval_acc = 0.4808676079394311
eval_loss = 1.352066347319484

and on test set:
model           =data/models_roberta_race/
total batch size=8
train num epochs=5
fp16            =False
max seq length  =512
eval_acc = 0.6015403323875153
eval_loss = 1.3087183478393092

I don't know why. Could anyone can help? Thank you!

Comments:
Do you get any improvement? 
The ACC of eval and test has a huge gap.


 --classification-head when converting the models 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am unable to reproduce the results on the RACE dataset. If anyone has been able to reproduce it, could you kindly share the weights of the fine-tuned model ?
```glue_convert_examples_to_features``` for sequence labeling tasks
Author: antgr
Pull request: no
Labels: name: wontfix

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
I would like a function like ```glue_convert_examples_to_features``` for sequence labelling tasks.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->
The motivation is that I need much better flexibility for  sequence labelling tasks. Its not enough to have a final, and decided for me model, for this task. I want just the features (a sequence of features/embeddings I guess).

## Additional context
This can be generalized to any dataset with a specific format.

<!-- Add any other context or screenshots about the feature request here. -->


Comments:
Do you mean the one already into Transformers in the [glue.py](https://github.com/huggingface/transformers/blob/d46147294852694d1dc701c72b9053ff2e726265/transformers/data/processors/glue.py) at line 30 or a different function? 

> glue_convert_examples_to_features

A different one.  Does this proposal makes sense?

> A different one. Does this proposal makes sense?

Different in which way? Describe to us please the goal and an high-level implementation.

Thanks for the reply! First of all I just want to clarify that I am not sure that my suggestion makes indeed sense. I will try to clarify: 1) This implementation ``` def glue_convert_examples_to_features(examples, tokenizer,``` is for glue datasets, so it does not cover what I suggest 2) in line 112 we see that it can only support "classification" and "regression". The classification is in sentence level. I want classification in tokens level.

So to my understanding, this function will give back one ```features``` tensor that is 1-1 correspondence with the one label for this sentence. In my case we would like n  ```features``` tensors, that will have 1-1 correspondence with the labels for this sentence, where n the number of tokens of the sentence.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Need pretrained XLNET on Squad which can be loaded from_pre_trained
Author: karthik19967829
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
It will be great ,if someone could share their username/model for XLNET pretrained on squad  
<!-- A clear and concise description of the question. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BertTokenizerFast ignores `pad_to_max_length`
Author: ranamihir
Pull request: no
Labels: name: Core: Tokenization

name: Should Fix

# 🐛 Bug

Hi,

I noticed some strange behavior with the fast tokenizers in `v2.5.0`, which I think is a bug:
It seems `BertTokenizerFast` is ignoring the `pad_to_max_length` argument, as shown below:

```python
>>> from transformers import AutoTokenizer, BertTokenizer
>>> tok_auto = AutoTokenizer.from_pretrained('bert-base-uncased')
>>> tok_bert = BertTokenizer.from_pretrained('bert-base-uncased')
>>> a, b = 'Sentence 1', 'Sentence 2'
>>> 
>>> tok_bert.encode(a, b, max_length=10, pad_to_max_length=True)
[101, 6251, 1015, 102, 6251, 1016, 102, 0, 0, 0]  # <-- Expected behavior
>>> tok_auto.encode(a, b, max_length=10, pad_to_max_length=True)
[101, 6251, 1015, 102, 6251, 1016, 102] # <-- Actual behavior
```

Also, can some please explain the reason for the warning below that's raised when I set `pad_to_max_length=False` (which is only there in the fast tokenizer)?
```python
>>> tok_auto.encode(a, b, max_length=10, pad_to_max_length=False)
Disabled padding because no padding token set (pad_token: [PAD], pad_token_id: 0).
To remove this error, you can add a new pad token and then resize model embedding:
	tokenizer.pad_token = '<PAD>'
	model.resize_token_embeddings(len(tokenizer))
[101, 6251, 1015, 102, 6251, 1016, 102]
>>> tok_bert.encode(a, b, max_length=10, pad_to_max_length=False)
[101, 6251, 1015, 102, 6251, 1016, 102] # <-- No warning
```

Thanks!

Comments:
Duplicate of #2947

Thanks @ranamihir @fte10kso, 

I'll have a look today. 

Thanks @mfuntowicz!
Python Tokenizer batch_encode_plus doesn't pad input if asked to do so.
Author: mfuntowicz
Pull request: no
Labels: name: Core: Tokenization

name: Should Fix

```python
input_p = tokenizer_p.batch_encode_plus(
    ["This is a simple input 1", "This is a simple input 2"], 
    max_length=15, 
    pad_to_max_length=True
)
```

Output is not padded 

Comments:
Add keyword arguments to batch_encode_plus() to match encode_plus()
Author: sergicastellasape
Pull request: no

## 🚀Consistent Keyword arguments for batch_encode_plus() to match encode_plus()
 Currently, features such as `return_special_tokens_mask` that are available for the `encode_plus()` method are not available for `batch_encode_plus()`. It would be nice if all keyword arguments worked in a similar fashion.

## Motivation
`batch_encode_plus()` is extremely useful to seamlessly tokenize batches, however, the lack of features forces to fall back to encode_plus() and an _uglier_ implementation.

Comments:
This should require adding a simple `**kwargs` at the end of 

https://github.com/huggingface/transformers/blob/f1e8a51f08eeecacf0cde33d40702d70c737003b/src/transformers/tokenization_utils.py#L977
Testing that batch_encode_plus is the same as encode_plus
Author: LysandreJik
Pull request: yes

Spoiler alert: it wasn't.

closes #2960 
closes #2658 
closes #2654 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2973?src=pr&el=h1) Report
> Merging [#2973](https://codecov.io/gh/huggingface/transformers/pull/2973?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c36416e53c29da8b6193f4a36d7b024c5f513495?src=pr&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `92.85%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2973/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2973?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2973      +/-   ##
==========================================
+ Coverage   77.14%   77.17%   +0.03%     
==========================================
  Files          98       98              
  Lines       16006    16020      +14     
==========================================
+ Hits        12348    12364      +16     
+ Misses       3658     3656       -2
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2973?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/2973/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `95.83% <100%> (+0.11%)` | :arrow_up: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2973/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `91.06% <92.3%> (+0.45%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2973?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2973?src=pr&el=footer). Last update [c36416e...9e3275a](https://codecov.io/gh/huggingface/transformers/pull/2973?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
`AutoModel.from_pretrained` sends config kwargs to model
Author: bryant1410
Pull request: no
Labels: name: Core: Modeling

name: Should Fix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert (may apply to more)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python
from transformers import AutoModel
AutoModel.from_pretrained('bert-base-uncased', output_attention=True)
```

([example from the docs](https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoModel.from_pretrained))

It crashes:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[...]/transformers/src/transformers/modeling_auto.py", line 384, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "[...]/transformers/src/transformers/modeling_utils.py", line 463, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'output_attention'
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

That the code returns a correct model, without crashing

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.0 (master)
- Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: no

Comments:
Seems related to #2694

Indeed! I fixed the misleading documentation with #2998.
Create README.md for the new model fine tuned for Spanish POS tagging
Author: mrm8488
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2999?src=pr&el=h1) Report
> Merging [#2999](https://codecov.io/gh/huggingface/transformers/pull/2999?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8194df8e0cff8e5866ec2bcbda34e3892f10eb39?src=pr&el=desc) will **decrease** coverage by `1.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2999/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2999?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2999      +/-   ##
==========================================
- Coverage   77.16%   76.14%   -1.02%     
==========================================
  Files          98       98              
  Lines       15999    15999              
==========================================
- Hits        12345    12182     -163     
- Misses       3654     3817     +163
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2999?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.71% <0%> (-10%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.48% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `96.03% <0%> (-2.21%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.2% <0%> (-1.35%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.53% <0%> (+0.32%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2999?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2999?src=pr&el=footer). Last update [8194df8...b00f716](https://codecov.io/gh/huggingface/transformers/pull/2999?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Wrong tokenization in Transformer-XL documentation
Author: DavidNemeskey
Pull request: no

## 🐛 Bug

This is a documentation-related bug. In the [TransfoXL documentation](https://huggingface.co/transformers/model_doc/transformerxl.html), the tokenization example is wrong. The snippet goes:
```
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
...
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
```
This code outputs the tokens `[24, 617, 3225, 23, 16072]`, of which `24` is `<unk>`.

The problem comes from the fact that Transformer-XL does **not** use a wordpiece vocabulary, but a regular (whole-word) one. Also, in WT-103, punctuation marks are split from the words. Consequently, the example should read instead (note that space in from of `,`):
```
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
...
input_ids = torch.tensor(tokenizer.encode("Hello , my dog is cute")).unsqueeze(0)  # Batch size 1
```

It would also be nice to warn the user about this fact in the documentation, perhaps in `TransfoXLTokenizer`'s docstring?

Comments:
Indeed, do you want to fix this in a PR?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
what is the most efficient way to store all hidden layers' weights?
Author: vr25
Pull request: no
Labels: name: wontfix

Hi,

I am following this [post](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/) for getting all 12 hidden layers' weights for every token in a sentence.

Consider I have a short text with 2 sentences: `He stole money today. He is fishing on the Mississippi riverbank.`

I want to store 5 + 8 = 13 tokens - all 12 hidden layers weights where each tensor's size=768. So, I will have 13 x 12 = 156 tensors.

I want to save all the weights in a file and I am wondering if I should use `pickle` or `hd5` format (I am working with long text documents.) I am planning to separate two sentences by a blank line, please suggest if any better ways to do it.

Thanks!

Comments:
PyTorch has its [own saving utility](https://pytorch.org/tutorials/beginner/saving_loading_models.html): `torch.save`, which sounds good for your use case as you can easily save/load the tensors. It's based on pickle.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
add explaining example to XLNet LM modeling
Author: patrickvonplaten
Pull request: yes

1) adds example explaining how XLNet could be used for standard auto-regressive modelling.
2) puts add_special_tokens=True for simple example to make sure no <sep> and <cls> are added to input. The are used for special training (similar to BERT) and might be confusing to user

Comments:
> Cool! Indeed, if I remember the paper correctly there is no need to shift the labels. Maybe we could update the [associated docstring](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_xlnet.py#L980-L985) to make sure no one gets confused again?

Done :-) 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2997?src=pr&el=h1) Report
> Merging [#2997](https://codecov.io/gh/huggingface/transformers/pull/2997?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/21d8b6a33ebf96680b6a0aabd27fa7eaa068da93?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2997/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2997?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2997   +/-   ##
=======================================
  Coverage   77.19%   77.19%           
=======================================
  Files          98       98           
  Lines       16013    16013           
=======================================
  Hits        12361    12361           
  Misses       3652     3652
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2997?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2997/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `75.77% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2997?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2997?src=pr&el=footer). Last update [21d8b6a...1868b91](https://codecov.io/gh/huggingface/transformers/pull/2997?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).