noamaanMulla-03/transformers-issues huggingface dataset:
the execution order of `scheduler.step()` and `optimizer.step()`
Author: boy2000-007man
Pull request: no

## ❓ Questions & Help

About current readme, related to the execution order of `scheduler.step()` and `optimizer.step()`
https://github.com/huggingface/pytorch-transformers#optimizers-bertadam--openaiadam-are-now-adamw-schedules-are-standard-pytorch-schedules
```python
### In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_total_steps)  # PyTorch scheduler
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
    **scheduler.step()**
    **optimizer.step()**
    optimizer.zero_grad()
```
While following the example code, i meet the warning which indicate the order is not expected according to the pytorch official document.
```bash
/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:82: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: 
`optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
```
I wonder if the readme need to be update to fit this announcement.

Thx

Comments:
I believe the order doesn't actually matter as long as it reflects what you're trying to do with your learning rate.

Readme fixed, thanks!
can this project select the specific version of BERT?
Author: bytekongfrombupt
Pull request: no

## ❓ Questions & Help
I dont know if this project can select the version of BERT which I need. For example, i want use BERT-wwm not BERT-basic, what should i do？ Can you help me, plz. 
<!-- A clear and concise description of the question. -->

Comments:
Hi. You can check the documentation about the different checkpoints available for each model [here](https://huggingface.co/pytorch-transformers/pretrained_models.html).

If you're looking for BERT whole word masking, there are the following pretrained models that might be of interest: `bert-large-uncased-whole-word-masking`, `bert-large-cased-whole-word-masking`, `bert-large-uncased-whole-word-masking-finetuned-squad` and `bert-large-cased-whole-word-masking-finetuned-squad`.

@LysandreJik Thanks for your advice. But, in my situation, I have to use my corpus to train a new BERT with whole word mask, so I cant use the pre-trained BERT model, what I should do in this situation?

Training an entire BERT model from scratch takes a lot of resources, and we don't have any scripts/examples that show how to do it with our library.

You could look at [Microsoft's repository](https://github.com/microsoft/AzureML-BERT) that uses our implementation to pre-train/fine-tune BERT.

@LysandreJik notice that the bert had been updated by adding whole word mask, do you updating your pytorch-transformer with this trick when you convert BERT from tf to pytorch
Index misplacement of Vocab.txt BUG BUG BUG
Author: hackerxiaobai
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (BertTokenizer):

Language I am using the model on (Chinese):

The problem arise when using:

**pytorch tokenizer**
```
t = tokenizer.tokenize('[CLS]哦我[SEP]')
i = tokenizer.convert_tokens_to_ids(t)
print(i)
[101, 1522, 2770, 102]
```

**tensorflow tokenizer**
```
t = tokenizer.tokenize('[CLS]哦我[SEP]')
i = tokenizer.convert_tokens_to_ids(t)
print(i)
[101, 1521, 2769, 102]
```

> Due to index misalignment, When the last word （##😎）in the  Vocab.txt  appears in the training set ，out of range error.

Comments:
Gradual unfreezing and discriminative fine-tuning for BERT
Author: dchang56
Pull request: no
Labels: name: wontfix

Three of the tips for fine-tuning proposed in ULMFIT are slanted triangular learning rates, gradual unfreezing, and discriminative fine-tuning.

I understand that BERT's default learning rate scheduler does something similar to STLR, but I was wondering if gradual unfreezing and discriminative fine-tuning are considered in BERT's fine-tuning implementations. Has anyone had experience implementing these two features in BERT fine-tuning? I'd like to hear your thoughts on it. Thanks!

Comments:
I've tried a bit to play with these training schemes on a deep transformer for our [tutorial on Transfer Learning in Natural Language Processing](https://naacl2019.org/program/tutorials/#t4-transfer-learning-in-natural-language-processing) held at NAACL last week but I couldn't get gradual unfreezing and discriminative fine-tuning to out-perform a standard fine-tuning procedure (multi-tasking did help, however).

You can have a look at the results by reading the "Hands-on" parts of the tutorial here: https://tinyurl.com/NAACLTransfer.

You can give it a try your-self with the associated Colab notebook which is here: https://tinyurl.com/NAACLTransferColab (and a full stand-alone codebase is here: https://tinyurl.com/NAACLTransferCode).

It's possible that I just didn't spend enough time scanning the hyper-parameters or maybe these two training variants are better suited to LSTM than Transformers.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> I've tried a bit to play with these training schemes on a deep transformer for our [tutorial on Transfer Learning in Natural Language Processing](https://naacl2019.org/program/tutorials/#t4-transfer-learning-in-natural-language-processing) held at NAACL last week but I couldn't get gradual unfreezing and discriminative fine-tuning to out-perform a standard fine-tuning procedure (multi-tasking did help, however).
> 
> You can have a look at the results by reading the "Hands-on" parts of the tutorial here: https://tinyurl.com/NAACLTransfer.
> 
> You can give it a try your-self with the associated Colab notebook which is here: https://tinyurl.com/NAACLTransferColab (and a full stand-alone codebase is here: https://tinyurl.com/NAACLTransferCode).
> 
> It's possible that I just didn't spend enough time scanning the hyper-parameters or maybe these two training variants are better suited to LSTM than Transformers.

I find the standard fine-tuning procedure having a unstable issue, that different shuffle order affects a lot. I wander if unfreezing help the bert finetune get a relative stable result?
This is also mentioned in bert paper, they just use different random seeds to get a best result.
GPT2 774M weights released!
Author: moinnadeem
Pull request: no

## 🚀 Feature

Hi! OpenAI released the 774M weights in GPT2, is it possible to integrate this into pytorch-transformers?

https://twitter.com/OpenAI/status/1163843803884601344

Also, sorry for the obnoxiously quick ask! Thanks for all the great work you do for the community.

Thanks!

Comments:
I did the following:

1. Run `download_model.py 774` from [here](https://github.com/openai/gpt-2)
2. Create a file named `config.json` with the following contents (Might be correct but I am not super sure):
```json
{
    "vocab_size": 50257,
    "n_ctx": 1024,
    "n_embd": 1280,
    "n_head": 20,
    "n_layer": 36,
    "n_positions": 1024,
    "embd_pdrop":0.1,
    "attn_pdrop": 0.1,
    "resid_pdrop": 0.1,
    "layer_norm_epsilon": 1e-5,
    "initializer_range": 0.02
}
```

3. Clone this repo

4. Run ```python .\pytorch-transformers\pytorch_transformers\convert_gpt2_checkpoint_to_pytorch.py --gpt2_checkpoint_path models/774M --pytorch_dump_folder_path ./ --gpt2_config_file config.json```

5. Use it with 
```
config = GPT2Config.from_pretrained("config.json")
model = GPT2LMHeadModel.from_pretrained("pytorch_model.bin", config=config)
```

6. Realize there's no way you can fine-tune this your PC's GPU you need to rent something with more memory.

We've added it on master.
You can install from source and use the shortcut name `gpt2-large` to use it (but beware, it's big!)

Question: Will the gpt2-large be added to Write With Transformer? I've been eagerly looking forward to that since the moment the 774M was released!

@zacharymacleod Glad you asked! We're definitely planning on adding it in the near future :)

Seems to me as if this has been addressed via #1064 . Closing the feature request now!
Missing tf variables in convert_pytorch_checkpoint_to_tf.py
Author: 4everlove
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): bert-base-uncased

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Fine tune BERT model using examples/run_glue.py
2. Use convert_pytorch_checkpoint_to_tf.py
3. Use run_classifier.py provided by BERT GitHub repo to do the prediction task.

Tensorflow will fail in loading the converted checkpoint, due to the missing variables 'global_step' and 'output_bias' (and maybe other variables)

Comments:
Indeed.
But I don't think we will aim for two-sided compatibility with the original Bert repo anyway.
In your case, you will need to adjust the original Bert repo code to be able to load the converted pytorch model (remove the unused variables or, more simple, tweak the checkpoint loading method).

Great. Thanks for your help, @thomwolf . Closing the ticket.
Fix "No such file or directory" for SQuAD v1.1
Author: cooelf
Pull request: yes

This solves the exception for SQuAD v1.1 evaluation without predicted null_odds file.

Traceback (most recent call last):
  File "./examples/run_squad.py", line 521, in <module>
  File "./examples/run_squad.py", line 510, in main
    for checkpoint in checkpoints:
  File "./examples/run_squad.py", line 257, in evaluate
    na_prob_file=output_null_log_odds_file)
  File "/home/zhangzs/pytorch-transformers-master/examples/utils_squad_evaluate.py", line 291, in main
    with open(OPTS.na_prob_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'squad/squad-debug/null_odds_.json'

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079?src=pr&el=h1) Report
> Merging [#1079](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/e00b4ff1de0591d5093407b16e665e5c86028f04?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1079      +/-   ##
==========================================
- Coverage   79.61%   79.58%   -0.03%     
==========================================
  Files          42       42              
  Lines        6898     6898              
==========================================
- Hits         5492     5490       -2     
- Misses       1406     1408       +2
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2JlcnQucHk=) | `93.89% <0%> (-0.94%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079?src=pr&el=footer). Last update [e00b4ff...61f14c5](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1079?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I think I fixed the issue about month ago?
https://github.com/huggingface/pytorch-transformers/blob/e00b4ff1de0591d5093407b16e665e5c86028f04/examples/run_squad.py#L248-L251

Thx! It looks fine now. My version is out of date. I'll close the comment.
Training Dataset of GPT-2
Author: ngarneau
Pull request: no
Labels: name: wontfix

What is the training dataset for the pre-trained GPT-2 Model?

Comments:
The pretrained model is trained on the WebText dataset. It's a collection of documents from outgoing Reddit links that have above 3 karma (ensures quality).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Performing MRPC task after Fine Tuning
Author: whitewolfos
Pull request: no

## ❓ Questions & Help

Sorry if this is really basic; I'm new to BERT and machine learning in general. I want to perform the MRPC task. I went ahead and did the fine-tuning and got the files/model okay. But now that I have this fine-tuned model, I'm confused how to do the actual MRPC task (i.e. given two sentences, produce a 1 if they are paraphrases or a 0 if they are not).

I think that I generally have the setup correct (see the code below), but my main problem is what to do with the tuple that is produced from the model. How do you turn that tuple output into the desired 0 or 1?

Thank you in advance for the help!

Code:

```
import torch
from pytorch_transformers import (BertForSequenceClassification, BertTokenizer)

#Creating two sentences to compare
sen1 = "I made a discovery."
sen2 = "I discovered something."

#Creating the tokenizer and model
fine_tuned_model_loc = '../pytorch-transformers/tmp/MRPC'
tokenizer = BertTokenizer.from_pretrained(fine_tuned_model_loc)
model = BertForSequenceClassification.from_pretrained(fine_tuned_model_loc)

#Prepare tokenized input
sen1_tokens = ["[CLS]"] + tokenizer.tokenize(sen1) + ["[SEP]"]
sen2_tokens = tokenizer.tokenize(sen2) + ["[SEP]"]

indexed_tokens = tokenizer.convert_tokens_to_ids(sen1_tokens + sen2_tokens)
token_type_ids = [0]*len(sen1_tokens) + [1]*len(sen2_tokens)
attention_mask = [1]*len(sen1_tokens + sen2_tokens)

#Turning things into a tensor
tokens_tensor = torch.tensor([indexed_tokens])
ids_tensor = torch.tensor([token_type_ids])
attention_tensor = torch.tensor([attention_mask])

#Run the model on the given info
model.eval()
with torch.no_grad():
    output = model(input_ids=tokens_tensor, token_type_ids=ids_tensor, \
                         attention_mask=attention_tensor)
```

Comments:
You can refer the code to run inference which I had written for sentiment classfication. HTH.
https://github.com/nikhilno1/nlp_projects/blob/master/pytorch-transformers-extensions/examples/run_inference.py

Ah, that was exactly what I needed; thank you!

One final thing, though: I'm still a bit confused on exactly what the "labels" are they you put into the model. Looking at your code, it seems like they can either have a value of "0" or "1", but I'm confused when it should be one over the other. Or does that not really matter when you are doing inference?

It does not matter. It is just a dummy input.

Alright! Thanks again for all of your help!
Wrong documentation example for RoBERTa
Author: CrafterKolyan
Pull request: no

Documentation web page: https://huggingface.co/pytorch-transformers/model_doc/roberta.html#pytorch_transformers.RobertaModel
See `Inputs -> input_ids`:
`tokens: [CLS] is this jack ##son ##ville ? [SEP][SEP] no it is not . [SEP]`  
and
`tokens: [CLS] the dog is hairy . [SEP]`  
are wrong examples. Because `RobertaTokenizer` says that its `cls_token` is `<cls>` not `[CLS]`.
Same for `sep_token`: it is `<sep>`, not `[SEP]`.

Using tokens `[CLS]` and `[SEP]` doesn't create any special token ids which causes errors when you try to use your encoded input in model. Adding the `add_special_tokens=True` to `encode` of course helps but you've added 2 extra tokens `[CLS]` and `[SEP]` to the input which are not known by model and possibly can decrease its quality.
Please change `[CLS]` and `[SEP]` to `<cls>` and `<sep>`.

Comments:
Hi, thank you for the bug report. It has been [changed](https://huggingface.co/pytorch-transformers/model_doc/roberta.html).
keeping encoder fixed from pretrained model but changing classifier
Author: ghost
Pull request: no

Hi
I need to pretrain the bert on one dataset and finetune it then on other datasets, so basically
removing classifier from first part and substitute it with a new one with the specific number of 
labels, currently with current codes, it will be error to do it when loading pretrained model,
could you please assist me how I can do this? I have a deadline soon and really appreciate your help urgently.
thanks
Best
Julia 

    model = model_class.from_pretrained(args.model_name_or_path, from_tf=bool('.ckpt' in args.model_name_or_path), config=config)
  File "julia/libs/anaconda3/envs/transformers/lib/python3.5/site-packages/pytorch_transformers/modeling_utils.py", line 461, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for BertRUBIForSequenceClassification:
	size mismatch for classifier.weight: copying a param with shape torch.Size([174, 768]) from checkpoint, the shape in current model is torch.Size([3, 768]).
	size mismatch for classifier.bias: copying a param with shape torch.Size([174]) from checkpoint, the shape in current model is torch.Size([3]).


Comments:
Closing this as it is a duplicate of the issue #1108 you opened 4 hours ago.
Working GLUE Example for XLNet (STS-B) 
Author: thomwolf
Pull request: yes

Same as #776 but let's merge it on XLNet for the moment.

`run_glue.py` is now a single script able to train BERT, XLNet and XLM on all GLUE tasks.

Example for XLNet:
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 python ./examples/run_glue.py   --do_train  --task_name=sts-b     --data_dir=${GLUE_DIR}/STS-B   --output_dir=./proc_data/sts-b-110   --max_seq_length=128   --per_gpu_eval_batch_size=8   --per_gpu_train_batch_size=8   --max_steps=1200  --model_name=xlnet-large-cased   --overwrite_output_dir   --overwrite_cache --warmup_steps=120
```
These hyper-parameters (same as the original one) give a pearsonr > 0.918.


Comments:
Working GLUE Example for XLNet (STS-B)
Author: thomwolf
Pull request: yes

`run_glue.py` is now a single script able to train BERT, XLNet and XLM on all GLUE tasks.

Example for XLNet:
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 python ./examples/run_glue.py   --do_train  --task_name=sts-b     --data_dir=${GLUE_DIR}/STS-B   --output_dir=./proc_data/sts-b-110   --max_seq_length=128   --per_gpu_eval_batch_size=8   --per_gpu_train_batch_size=8   --max_steps=1200  --model_name=xlnet-large-cased   --overwrite_output_dir   --overwrite_cache --warmup_steps=120
```
These hyper-parameters (same as the original one) give a pearsonr > 0.918.


Comments:
Improvements to GPT-2 (special_tokens, fine-tuning, medium model) + repo code coverage metric
Author: thomwolf
Pull request: yes

- adding method to add special tokens to GPT-2 (like it's done for GPT).
- adding code coverage tracking for tests.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@b832d5b`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `70.37%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master     #560   +/-   ##
=========================================
  Coverage          ?   66.04%           
=========================================
  Files             ?       18           
  Lines             ?     3673           
  Branches          ?        0           
=========================================
  Hits              ?     2426           
  Misses            ?     1247           
  Partials          ?        0
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/modeling\_openai.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvbW9kZWxpbmdfb3BlbmFpLnB5) | `79.68% <100%> (ø)` | |
| [pytorch\_pretrained\_bert/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `84.86% <100%> (ø)` | |
| [pytorch\_pretrained\_bert/modeling\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvbW9kZWxpbmdfZ3B0Mi5weQ==) | `80.16% <61.9%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560?src=pr&el=footer). Last update [b832d5b...db98a4a](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/560?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Updating network handling
Author: thomwolf
Pull request: yes

This PR adds:
- a bunch of tests for the models and tokenizers stored on S3 with `--runslow` (download and load one model/tokenizer for each type of model BERT, GPT, GPT-2, Transformer-XL)
- relax network connection checking (fallback on the last downloaded model in the cache when we can't get the last eTag from s3)

Comments:
Update to fifth release
Author: thomwolf
Pull request: yes

Mostly a bug fix update for loading the `TransfoXLModel` from s3:
- this fixes a bug in the loading of the pretrained `TransfoXLModel` from the s3 dump (which is a converted `TransfoXLLMHeadModel`) and the weights were not loaded.
- I also added a fallback of `OpenAIGPTTokenizer` on BERT's `BasicTokenizer` when SpaCy and ftfy are not installed. Using BERT's `BasicTokenizer` instead of SpaCy should be fine in most cases as long as you have a relatively clean input (SpaCy+ftfy were included to exactly reproduce the paper's pre-processing steps on the Toronto Book Corpus) and this also let us use the `never_split` option to avoid splitting special tokens like `[CLS], [SEP]...` which is easier than adding the tokens after tokenization.
- I also updated the README on the tokenizers options and methods which was lagging behind a bit.

Comments: