noamaanMulla-03/transformers-issues huggingface dataset:
Better serialization for Tokenizer and Config classes (BERT, GPT, GPT-2 and Transformer-XL)
Author: thomwolf
Pull request: yes

This PR add standardized serialization to all the tokenizers (BERT, GPT, GPT-2, Transformer-XL) through a `tokenizer.save_vocabulary(path)` method.

Also add a serialization method to all the Configuration classes: `Config.to_json_file(file_path)`

Added clean examples for serialization best practices in README and examples.

Also fixes Transformer-XL "split on punctation" bug mentioned in #466.

Comments:
Implement the QuickStart but got an error when  using BertForMaskedLM to predict a masked token
Author: JJJJane
Pull request: no

## ❓ Questions & Help

I was running the BERT example following the instruction of pytorch-transformers' docs, but when Iusing BertForMaskedLM to predict a masked token, an error occured:
"INFO:pytorch_transformers.modeling_utils:Weights from pretrained model not used in BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']"

Any idea how to fix this? Thanks!

Comments:
Hi, this is not an error, but a warning. This warning tells you that some of the weights that were in your pretrained model were not used by the model with which you loaded them. In this case, it concerns the classification layer weight/bias.

I see, thanks!
[Help] How to do mean/max pooling to get sentence embedding?
Author: brytjy
Pull request: no

Hi, I read a few questions raised before regarding sentence embedding and came across mean/max pooling suggestions. 

Not too sure how to go about doing mean/max pooling.

Is my implementation correct for mean pooling? I simply took the sum of all the token vectors and divide them by total sequence length.

![image](https://user-images.githubusercontent.com/46053996/63739517-04201c00-c8c0-11e9-97ca-259efc8c60d9.png)

Thanks :)



Comments:
Hi! Yes this would work, but it would certainly be slower than using [the torch `mean` function](https://pytorch.org/docs/stable/torch.html#torch.mean)!

Understood thanks! :)
Does RoBERTa needs input_type_ids as Bert ?
Author: Lawiss
Pull request: no

## ❓ Questions & Help

Hello,

I'm trying to fine-tune RoBERTa for a sentence-pair classification task. With Bert, I used the token_type_ids to identify sentence A and B. But it seems that the Roberta "token_type" Embedding is configured with a dictionnairy of size 1 from what I understand of the model summary : (token_type_embeddings): Embedding(1, 768).

So, does RoBERTa needs token_type_ids ? If not, why there is an Embedding layer for token_type_ids ?
The documentation of the RobertaModel class omit to talk about the token_type_ids present among the parameter : [modeling_roberta.py](https://github.com/huggingface/pytorch-transformers/blob/e08c01aa1ad63efff83548ea69d5ba3ce4a75acc/pytorch_transformers/modeling_roberta.py#L97).
Thank you in advance.


Comments:
RoBERTa does not use `token_type_ids`. We made a choice to still have an embedding layer (which is all zeros, so they don't contribute anything additively) so that we use the exact same implementation as BERT.

Understood, thanks for the quick answer ! :)
sample_text.txt is broken (404 ERROR)
Author: zbloss
Pull request: no

## ❓ Questions & Help

When I try to access the sample_text,txt file at this link, I find an nginx 404 server error.

https://huggingface.co/pytorch-transformers/samples/sample_text.txt

Comments:
Hi, where did you retrieve this link from?

Hey this is listed on the huggingface.co documentation page here ([https://huggingface.co/pytorch-transformers/examples.html?highlight=sample_text](https://huggingface.co/pytorch-transformers/examples.html?highlight=sample_text))

This one is now in the tests at: https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/tests/fixtures/sample_text.txt

We'll fix the doc, thanks!
Delete nonexistent parameter from documentation fix #1115
Author: CrafterKolyan
Pull request: yes

Changed documentation of GPT2Model, GPT2LMHeadModel and GPT2DoubleHeadsModel

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116?src=pr&el=h1) Report
> Merging [#1116](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/e08c01aa1ad63efff83548ea69d5ba3ce4a75acc?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1116   +/-   ##
=======================================
  Coverage   79.61%   79.61%           
=======================================
  Files          42       42           
  Lines        6898     6898           
=======================================
  Hits         5492     5492           
  Misses       1406     1406
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfZ3B0Mi5weQ==) | `75.84% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116?src=pr&el=footer). Last update [e08c01a...c8933bb](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1116?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @CrafterKolyan!
Wrong parameter name in documentation
Author: CrafterKolyan
Pull request: no

Documentation web page: https://huggingface.co/pytorch-transformers/model_doc/gpt2.html#pytorch_transformers.GPT2DoubleHeadsModel

See `Inputs -> multiple_choice_labels`. There is actually no such parameter in `GPT2DoubleHeadsModel.forward` method. It was renamed to `mc_labels`. Also it is presented two times in documentation which seems to be just a copy past error.

Please change `multiple_choice_labels` to `mc_labels` and delete second occurrence of this parameter in documentation.

Created #1118 to fix documentation. Also you may squash it with fix of #1115

Comments:
Yes!
Documentation fix #1117
Author: CrafterKolyan
Pull request: yes

Rename parameter in documentation + Delete its second occurrence.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118?src=pr&el=h1) Report
> Merging [#1118](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/e08c01aa1ad63efff83548ea69d5ba3ce4a75acc?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1118   +/-   ##
=======================================
  Coverage   79.61%   79.61%           
=======================================
  Files          42       42           
  Lines        6898     6898           
=======================================
  Hits         5492     5492           
  Misses       1406     1406
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfZ3B0Mi5weQ==) | `75.84% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118?src=pr&el=footer). Last update [e08c01a...26bda77](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1118?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed!
Tons of warnings on use of TransfoXLModel. masked_fill_ input dtype torch.uint8 should be changed to torch.bool
Author: CrafterKolyan
Pull request: no

## 🕑 Usage of deprecated behaviour

Using example from documentation web page: https://huggingface.co/pytorch-transformers/model_doc/transformerxl.html#pytorch_transformers.TransfoXLModel
```
import torch
from pytorch_transformers import *

tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLModel.from_pretrained('transfo-xl-wt103')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)
outputs = model(input_ids)
last_hidden_states, mems = outputs[:2]
```

Get tons of same warnings:
> /pytorch/aten/src/ATen/native/LegacyDefinitions.cpp:14: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead.
![image](https://user-images.githubusercontent.com/9883873/63766644-4b180c80-c8d4-11e9-890c-9da640b5c0e8.png)

Created #1120 to fix it.

Comments:
Temporary fix for RoBERTa's mismatch of vocab size and embedding size - issue #1091
Author: amirsaffari
Pull request: yes

I added an optional input argument so you can pass the starting index when adding new tokens.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096?src=pr&el=h1) Report
> Merging [#1096](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/df9d6effae43e92761eb92540bc45fac846789ee?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `85.71%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1096      +/-   ##
==========================================
+ Coverage   79.61%   79.62%   +<.01%     
==========================================
  Files          42       42              
  Lines        6898     6900       +2     
==========================================
+ Hits         5492     5494       +2     
  Misses       1406     1406
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `86.31% <85.71%> (+0.08%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096?src=pr&el=footer). Last update [df9d6ef...9a950be](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1096?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @amirsaffari. We actually don't need this fix anymore.

@LysandreJik has updated the vocabulary on the AWS S3 bucket to include the missing (unused) tokens (called `makeupword0000`, `makeupword0001` and `makeupword0002`). So that the vocabular now has the same length as the last token index. We're adding a test as well.

If you have the latest release you can delete your cached vocabulary to download the updated version. If you have installed from master, you can just force the download and overwriting of the new vocabulary with `tokenizer = RobertaTokenizer.from_pretrained('your-model', force_download=True)` 

👍
TypeError: expand_as() takes 1 positional argument but 5 were given
Author: anxingle
Pull request: no
Labels: name: wontfix

[Model.py](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py) line [870](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L870):
`head_mask = head_mask.expand_as(self.config.num_hidden_layers, -1, -1, -1, -1)`   
got  the mistake,  I tried head_mask=torch.tensor([1, 2, 3]) or just like that.

Comments:
Oh yes, this will be fixed in the coming PR #711.
Head mask is an option to explore the model internals, it's not for production.
See the `bertology.py` example script.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Cannot import DistilBert classes
Author: delip
Pull request: no

Tried installing from the master, and couldn't do it. 
![image](https://user-images.githubusercontent.com/347398/63883240-91c73e80-c988-11e9-9a22-241ac48479c3.png)


Comments:
My bad. Forgot to ignore pip cache
```
pip install git+https://github.com/huggingface/pytorch-transformers --no-cache-dir
```
swap order of optimizer.step() and scheduler.step()
Author: adai183
Pull request: yes

The current code results in the following warning:

```
UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
```

Comments:
indeed, thanks @adai183 

🤗
Include a reference on in-domain LM pre-training for BERT
Author: lopuhin
Pull request: no
Labels: name: wontfix

From https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/lm_finetuning#introduction

> As such, it's hard to predict what effect this step will have on final model performance, but it's reasonable to conjecture that this approach can improve the final classification performance, especially when a large unlabelled corpus from the target domain is available, labelled data is limited, or the target domain is very unusual and different from 'normal' English text.
> If you are aware of any literature on this subject, please feel free to add it in here, or open an issue and tag me (@Rocketknight1) and I'll include it.

Hi @Rocketknight1 this paper https://arxiv.org/pdf/1905.05583.pdf studies within-task and within-domain pre-training for BERT in section 5.4 and they achieve a good boost from it.

Comments:
Ah, thank you very much for this! I'll read over the paper and include it as a reference soon.

Finally read it once I had some free time at the weekend and added PR #715. Thank you!

Thank you @Rocketknight1 !

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Low SQuADv2 F1 & EM Score
Author: pachiko
Pull request: no
Labels: name: wontfix

Hi,

I used the default settings and run_squad.py script (with the exception of a batch size of 4 since my GPU has low memory) to train for 3 epochs. Turns out I got an EM & F1 score of 43% and 48% respectively. AvNA looks decent at 65%. Is this due to a small number of epochs or the small batch size? Note that training for 3 epochs on my single 1070-TI took 12 hours.

Thanks.

Comments:
@thomwolf 
Hi, it seems there is something wrong with the training code in this repo.
I used Google's official BERT training code and I could get decent results even with a small batch size: 
`{'EM': 73.80717341230668, 'F1': 77.11048422305339, 'AvNA': 80.78315235274762}`

Here's the settings I used for Google's and this repo's:
```
--vocab_file=uncased_L-12_H-768_A-12/vocab.txt
--bert_config_file=uncased_L-12_H-768_A-12/bert_config.json
--init_checkpoint=uncased_L-12_H-768_A-12/bert_model.ckpt
--do_train=True
--train_file=../squad/data/train-v2.0.json
--do_predict=True
--predict_file=../squad/data/dev-v2.0.json
--train_batch_size=5
--learning_rate=3e-5
--num_train_epochs=2.0
--max_seq_length=384
--doc_stride=128
--output_dir=gugel_bert
--version_2_with_negative=True
--do_lower_case=True
```
```
--bert_model=bert-base-uncased
--output_dir=try_bert_2
--train_file=data/train-v2.0.json
--predict_file=data/dev-v2.0.json
--do_train
--do_predict
--do_lower_case
--train_batch_size=5
--predict_batch_size=5
--num_train_epochs=2.0
--learning_rate=3e-5
--version_2_with_negative
```
EDIT:
Forgot to say that I also can get the expected results using SQuAD v1.1 with this repo's code.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
loss explosion
Author: g-jing
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am using Bert-CRF model to do Named Entity Recognition task. I am using the average of the last four layers as the input of CRF model.  But the loss will increase and become Nan in a few batches. Anyone meet that problem before? Any suggestions will be appreciated!

Comments:
just because of a large learning rate
BPE vocab
Author: sashank06
Pull request: no
Labels: name: wontfix

Do you guys have the functionality to support BPE with the models? 

Comments:
Not speaking for the core developers, but `pytorch-pretrained-BERT` supports it, because:

* GPT-1 use BPE, see code [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization_openai.py#L73)
* GPT-2 use BPE on byte level, see code [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization_gpt2.py#L88)

BERT uses a variant of BPE (word pieces) and the pretrained language model for Transformer-XL was trained on WikiText-103 (so it is a word-based model).



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to get pooler state's (corresponds to CLS token) attention vector?
Author: Akella17
Pull request: no

The following model definition returns the attention vector for tokens corresponding to the input sequence length, i.e. ```x.size(1)```. How do I procure the attention vector of the pooler state (output embedding corresponding to the CLS token)?

```Python
model = BertModel.from_pretrained('bert-base-uncased', output_attentions=True)
outputs = self.model(x, attention_mask = x_mask)
last_layer_attentions = outputs[2][-1] # [batch_size, num_heads, x.size(1), x.size(1)]
# I want the attention vector for pooler state, i.e. [batch_size, num_heads,  1, x.size(1)]
```


Comments:
Hi, the pooler takes as input the last layer hidden-state of the first token of the sentence (the `[CLS]` token). So the attention used to compute the pooler input is just the attention for this token.

@thomwolf If my understanding is right, the last layer's attention vector should be of size ```[batch_size, num_heads, (x.size(1)+1), (x.size(1)+1)]```, corresponding to the **[CLS]** embedding and x.size(1) token embeddings. However, ```output[2][-1]``` only returns ```[batch_size, num_heads, x.size(1), x.size(1)]``` dimensional attention map, which, I am guessing, corresponding to the input sequence (**x.size(1)**) and not the **[CLS]** token.

How do I get the attention vector corresponding to the **[CLS]** token? Also, can you mention which of the two **x.size(1)** axes corresponds to the input layer and the output layer?
Closing bracket is missing in token_counts.py for DistilBERT
Author: tomohideshibata
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): DistilBERT

The problem arise when using:
* [x] the official example scripts: (give details)

I'm trying DistilBERT, and get an "invalid syntax error" when I run `examples/distillation/scripts/token_counts.py`.

A closing bracket seems missing at line 27.

```
    parser.add_argument("--data_file", type=str, default="data/dump.bert-base-uncased.pickle",
                        help="The binarized dataset."
```
 

## Environment

* OS: Linux
* Python version: 3.6.9
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 1.1.0
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information:


Comments:
Hi! Indeed, the closing bracket was missing, fixed it with caf1d11! Thanks for the bug report :).
GPT2-large fails to load the tokenizer
Author: pywirrarika
Pull request: no

## 🐛 Bug

Using: GPT2-Large

## To Reproduce
When I load the gpt2-large model, in the same way as gpt2 and gpt2-medium I get an NoneType when loading the tokenizer. 

```
self.tokenizer = GPT2Tokenizer.from_pretrained("gpt2-large", bos_token="_start_", unk_token='_unk_', eos_token="_eos_", sep_token="_delimiter_", cls_token="_classify_", pad_token='_pad_' )
self.model = GPT2LMHeadModel.from_pretrained("gpt2-large")
```
At this point in the debugger self.tokenizer == None is True
The issue is clear when I try to use the tokenizer.
```
File "gpt2_train.py", line 70, in load
    num_added_toks = self.tokenizer.add_special_tokens(special_tokens_dict)
AttributeError: 'NoneType' object has no attribute 'add_special_tokens'
```

## Expected behavior

Get a tokenizer object

## Environment

* OS: Linux
* Python version: 3.7.3
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.1.0 (last github pull)
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information:



Comments:
Hello! Could you try to install from source (1.2.0)? Is there any warning in your terminal such as 
```
Model name 'gpt2-large' was not found in model name list (gpt2, gpt2-medium). We assumed 'gpt2-large' was a path or url but couldn't...
```
or something along those lines?

Thank you! Solved with the update. And yes, checking again I got your mentioned error message.