noamaanMulla-03/transformers-issues huggingface dataset:
warmup_linear for BertAdam and OpenAIAdam
Author: lukovnikov
Pull request: no

1. OpenAIAdam version of `warmup_linear` does not linearly increase lr, instead it looks like this:
![figure_1](https://user-images.githubusercontent.com/1732910/53427158-b26a0800-39e8-11e9-9f4e-5df1b9f64566.png)
This is different from BertAdam version of `warmup_linear`. Should they not be the same (Bert version)?

2. if `t_total` is specified incorrectly (too small), learning rate becomes negative after t_total for both versions. Probably it would be better to set lr to 0 to avoid situations like [Issue#297](https://github.com/huggingface/pytorch-pretrained-BERT/issues/297). Also, with a too small `t_total`, there is a drop in lr right after `warmup` is reached:
![figure_1](https://user-images.githubusercontent.com/1732910/53427736-c5c9a300-39e9-11e9-8f16-8160ac325490.png)

Let me know if I should PR a fix for both.



Comments:
I just ran into this problem while running BERT on large samples from `run_squad.py`. I think a fix would be welcome because this is a really disturbing and hard to catch issue. 

It would probably be enough to move the optimizer creation + computing of `num_train_optimization_steps` inside the train loop.

Happy to welcome a PR on this indeed.
I'm not super fan of silently hide a wrong `t_total` by setting `lr` to zero so maybe sending a warning `logger. warning` at the same time would be nice too.

made a PR: https://github.com/huggingface/pytorch-pretrained-BERT/pull/327

Fixed in master now, thanks @lukovnikov!
Issue in running run_glue.py in Roberta, XLNet, XLM in latest release
Author: leslyarun
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Roberta, XLM, XLNet

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: run_glue.py

The tasks I am working on is:
* [x] my own task or dataset: binary classification on my own dataset

## To Reproduce

Steps to reproduce the behavior:

1. Upon running glue.py with the following : python ~/pytorch-transformers/examples/run_glue.py --task_name cola --do_train  --do_eval  --do_lower_case --data_dir ~/bert-data/ --model_type roberta --model_name_or_path  roberta-base  --max_seq_length 512  --learning_rate 2e-5 --num_train_epochs 1.0   --output_dir ~/data/roberta-1/

Getting the following error:
```
08/16/2019 14:18:21 - WARNING - pytorch_transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (513 > 512). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
  File "/home/pytorch-transformers/examples/run_glue.py", line 494, in <module>
    main()
  File "/home/pytorch-transformers/examples/run_glue.py", line 447, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "/home/pytorch-transformers/examples/run_glue.py", line 283, in load_and_cache_examples
    pad_token_segment_id=4 if args.model_type in ['xlnet'] else 0,
  File "/home/pytorch-transformers/examples/utils_glue.py", line 485, in convert_examples_to_features
    assert len(input_ids) == max_seq_length
AssertionError
```
2. Trying the same as above with XLNet and XLM, gives the following error:

```
08/16/2019 14:26:59 - INFO - __main__ -   Creating features from dataset file at /home/new-bert-data/keyword_data/
Traceback (most recent call last):
  File "/home/pytorch-transformers/examples/run_glue.py", line 494, in <module>
    main()
  File "/home/pytorch-transformers/examples/run_glue.py", line 447, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "/home/pytorch-transformers/examples/run_glue.py", line 282, in load_and_cache_examples
    pad_token=tokenizer.encoder[tokenizer.pad_token] if args.model_type in ['roberta'] else tokenizer.vocab[tokenizer.pad_token],
AttributeError: 'XLMTokenizer' object has no attribute 'vocab'
```

## Environment

* OS: Debian
* Python version: 3.6.9
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 1.1.0
* Using GPU: Yes
* Distributed of parallel setup: Multi-GPU 


Comments:
Thanks for the report! I'm looking into it.

Could you try the changes in [this commit ](https://github.com/huggingface/pytorch-transformers/commit/a93966e608cac8e80b4ff355d7c61f712b6da7f4)on your own dataset and tell me if you still have errors?

@LysandreJik Ya this code works fine. Thanks for the quick fix

Great, glad I could help!
Add an argument --model_size to convert_gpt2_checkpoint_to_pytorch.py
Author: chrisyue1998
Pull request: yes
Labels: name: wontfix

Add an argument --model_size to convert_gpt2_checkpoint_to_pytorch.py that lets the user specify whether they want to convert a checkpoint from the 117M model or from the 345M model.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Thanks I don't think we'll add this since converted models are already provided.
Adding additional model loading functionality
Author: zphang
Pull request: yes

(Porting over some functionality from my old fork)

This PR adds additional methods to `PreTrainedModel` for loading models for `state_dict`s. Currently, `from_pretrained()` does a lot of the heavy lifting, but is primarily designed to load from file/folders. This adds additional options for users with different model-loading workflows.

Comments:
Hi Jason,

Can you give me a little more information on the model-loading workflow you are using so I can understand the whys and wherefores of these proposed modifications?

Hey Thomas,

Sorry for the delay. My thinking is this: the `from_pretrained` method current does two things: resolve the path/archive for loading a pretrained model, and the specialized model loading logic (e.g. handing the fact that the current model may have different heads from those in the loaded weights). My proposed change is to separate the two and allow the user to just do the second. 
This would be useful in cases where the user already has access to the `state_dict` in memory (e.g. if they have a different model saving workflow/format).

Hi Jason,
There is a `state_dict` option in `from_pretrained` that, I think, let you do just that!
See here for instance: https://huggingface.co/pytorch-transformers/main_classes/model.html#pytorch_transformers.PreTrainedModel.from_pretrained

Closing this for now. Feel free to re-open if the provided solution doesn't solve your problem, Jason.
Fix the gpt2 quickstart example
Author: oliverguhr
Pull request: yes

You need to add the SEP (seperator) token to the tokenizer, otherwise the tokenizer.decode will fail with this error:
`ERROR:pytorch_transformers.tokenization_utils:Using sep_token, but it is not set yet.`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1070?src=pr&el=h1) Report
> Merging [#1070](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1070?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/6f877d9daf36788bad4fd228930939fed6ab12bd?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1070/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1070?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1070   +/-   ##
=======================================
  Coverage   79.61%   79.61%           
=======================================
  Files          42       42           
  Lines        6898     6898           
=======================================
  Hits         5492     5492           
  Misses       1406     1406
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1070?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1070?src=pr&el=footer). Last update [6f877d9...3248388](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1070?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue was fixed upstream with #1087
Thanks @oliverguhr
Can't Using Binarization Script for DistilBERT
Author: SreeramV181
Pull request: no

## 🐛 Bug

<!-- Important information -->

I'm currently using DistilBERT and running into issues when I run scripts/binarized_data.py. I get the following error: 

Traceback (most recent call last):
  File "scripts/binarized_data.py", line 25, in <module>
    from ..utils import logger 
ValueError: attempted relative import beyond top-level package

I haven't modified anything within the package.

Comments:
Hello @SreeramV181,
it should be fixed in commit 803c1cc4eacd38f1b854578d7d717b5e4a1ada47.
Thanks for pointing that out!
Victor
Update apex fp16 implementation
Author: anhnt170489
Pull request: yes

As the issue I raised here: https://github.com/huggingface/pytorch-transformers/issues/1143
I updated the implementation of apex fp16, following latest Apex version's document. 

Comments:
Thanks, these examples will probably be deprecated and replaced by the new more general `run_lm_finetuning` example which can train several models with normal and masked language modeling.
Why still using old implementation of apex fp16
Author: anhnt170489
Pull request: no

## 🚀 Feature

According to Nvidia apex fp16 documentary: https://nvidia.github.io/apex/amp.html

The implementation of new apex version doesn't require using FP16_Optimizer wrapping over FusedAdam. So I wonder why the team still kept the old implementation. Is there any special reason ? If not, I will make a pull request of implementation due to new apex version

Comments:
AFAIK no particular reason, feel free to open a PR
fix adding special tokens
Author: epwalsh
Pull request: yes

Currently there is a bug when adding `additional_special_tokens` in the form of a tuple, instead of a list. To reproduce:

```python
from pytorch_transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
tokenizer.add_special_tokens({"additional_special_tokens": ("@a@", "@b@")})
tokenizer.all_special_tokens
```

Results in:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-81549ce398a5> in <module>()
----> 1 tokenizer.all_special_tokens

~/GitHub/pytorch-transformers/pytorch_transformers/tokenization_utils.py in
all_special_tokens(self)
    677         set_attr = self.special_tokens_map
    678         for attr_value in set_attr.values():
--> 679             all_toks = all_toks + (attr_value if isinstance(attr_val
ue, (list, tuple)) else [attr_value])
    680         all_toks = list(set(all_toks))
    681         return all_toks

TypeError: can only concatenate list (not "tuple") to list
```

Comments:
Look good to me (the failing test on `head_masking` is not related to this PR).
Thanks @epwalsh!
low accuracy when fine tuning for the MRPC task with large model
Author: syu0000
Pull request: no
Labels: name: wontfix

I noticed that on the website you said:"Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking model to reach a F1 > 92 on MRPC." However, when I fine tuned the model with max_sequence_length=128 and batch_size=12 on a single 11G GPU, it gives the accuracy of 0.68.
acc = 0.6838235294117647
acc_and_f1 = 0.7480253018237863
eval_loss = 0.6240295206799227
f1 = 0.8122270742358079
global_step = 918
loss = None
I wonder what have let to that.
the command I used "python  run_classifier.py   --bert_model bert-large-uncased-whole-word-masking    --task_name MRPC --do_train   --do_eval   --do_lower_case   --data_dir E:\Users\..... --max_seq_length 64   --train_batch_size 12   --learning_rate 2e-5   --num_train_epochs 3.0  --output_dir E:\Users\...."

Comments:
The batch size will be 8 times smaller with only one GPU, increase it by a factor of 8 using gradient accumulation, e.g. `--train_batch_size 96 --gradient_accumulation_steps 8`

Thank you for your help. However, when I use this command: 
python  run_classifier.py   --bert_model bert-large-uncased-whole-word-masking  --task_name MRPC –-do_train --do_eval --do_lower_case --data_dir E:\Users\...\MRPC --max_seq_length 128 --train_batch_size 48 --gradient_accumulation_steps 8 --learning_rate 2e-5 --num_train_epochs 3.0  --output_dir E:\Users\...\MRPC_result
it still gives me a too low acc result.
btw I am not using the latest version of package, could that be the cause?



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_classifier.py:TypeError: forward() missing 1 required positional argument: 'input_ids'
Author: SlinZhang
Pull request: no
Labels: name: wontfix

when I use my dataset， forward() missing 1 required positional argument: 'input_ids' ,but I can get "input_ids, input_mask, segment_ids, label_ids" in batch.

Comments:
I ran into the same error and fixed it by using this instead:

[Pull Request](https://github.com/huggingface/pytorch-pretrained-BERT/pull/604)

Haven't tried my full dataset yet but on a slice, it worked well!

Edit: On the full dataset I still get the error.

Edit2: 
Change the train data loader to drop the last "incomplete" batch as a workaround:

> train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=args.train_batch_size, drop_last=True)

Same error. I solved this problem by add "CUDA_VISIBLE_DEVICES=0" in command line.
I guess this problem caused by "DataParallel" which splits and sends batch to GPUs. In some case, batch_size is less than the number of GPU. In consequence, model on extra gpu just get an empty input.
BTW，you should use  @AlanHassen 's solution in training stage.

same error when use my dataset.   File "run_classifier.py",   TypeError: forward() missing 1 required positional argument: 'input_ids'.

Add:
i  change the default eval_batch_size from 8 to 12, it can avoid this error.
you can change the eval_batch_size to some other number.

of course, this bug should finally be solved by code. 



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Grover generator support
Author: asafamr
Pull request: no
Labels: name: wontfix

Grover released their trained model:
https://github.com/rowanz/grover
I think it should be similar to GPT-2 large. Any plans to support it? 

Comments:
Maybe if they open source a model larger than the current GPT-2 large.

I'm also happy to welcome PRs to port additional models (as long as they are provided with test/doc/example)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GPT & GPT2: binary classification fails
Author: epsdg
Pull request: no
Labels: name: wontfix

Using `OpenAIGPTDoubleHeadsModel` for binary classification fails.

`CrossEntropyLoss`, requires that the logits dim matches num_classes.

If `input_ids.size()` is (batch x 1 x seq_len) (only one copy of the input sequence) but mc_labels are {0, 1} (two classes), the loss fn returns a shape mismatch.  The only way it seems to work is using two copies of the input sequence, one for class 0 and a second for class 1.

I tweaked the double heads model to use `BCEWithLogitsLoss` for binary:

https://github.com/epsdg/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling_openai.py#L955-L960

https://github.com/epsdg/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling_gpt2.py#L943-L948

...and it works fine, but did I miss an intended use pattern?

Thanks for this fantastic port of these models - much more user-friendly than the original code.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
distillation: fix ModuleNotFoundError error in token counts script
Author: stefan-it
Pull request: yes

Hi,

I'm currently trying out "distillation" 😅

This PR a `ModuleNotFoundError` message in the `token_counts.py` script (same error was recently fixed in 803c1cc4eacd38f1b854578d7d717b5e4a1ada47 🤗

Comments:
Great, thanks @stefan-it!
Bert initialization
Author: Albert-Ma
Pull request: no

When I train bert model from scratch, it can not convergence and the loss does not decrease. Still don't work even try different learning rate many times.

But It works when I tried the tf version. I checked code and there have no much difference except the initialization.

So does anybody have some ideas about this?

Comments:
Hi, I think you'll find this [particular issue interesting](https://github.com/huggingface/pytorch-transformers/issues/202).

[Thomas Wolf's comment](https://github.com/huggingface/pytorch-transformers/issues/202#issuecomment-522613642) in particular may be of help.

> Hi, I think you'll find this [particular issue interesting](https://github.com/huggingface/pytorch-transformers/issues/202).
> 
> [Thomas Wolf's comment](https://github.com/huggingface/pytorch-transformers/issues/202#issuecomment-522613642) in particular may be of help.

Thanks. But I am not saying to train a language model from scratch, I am saying to train a glue task from scratch. So I think there has much difference between this.
FP16_Optimizer is not an Optimizer when fp_16
Author: anhnt170489
Pull request: no

## 🐛 Bug

When using fp_16, the scheduler should be: 

scheduler = WarmupLinearSchedule(optimizer.optimizer, warmup_steps=args.warmup_steps,
                                         t_total=num_train_optimization_steps)

if not, that will be "TypeError: FP16_Optimizer is not an Optimizer"

Comments:
I think this issue should be fixed on master (even though I'm not exactly sure which script you are referring to).

I already fixed this in my pull request. This error is caused by FP16_Optimizer, by using new Apex implementation, we deprecated FP16_Optimizer, so this bug is no longer issued

Ok closing the issue then, thanks.
May I get the details of Bert pre-train procedure?
Author: zhuhaozhe
Pull request: no

## ❓ Questions & Help
I want to measure the time consume in BERT pre-train procedure. May I ask some questions here:
1. What is the pre-train data? Is that same in paper : BooksCorpus & English Wiki.
2. Is there some code for pre-train can I utilize?
3. May I know how much epochs should I train when pre-train 'base' version model or 'large' version model?

Comments:
Hi, you should refer to the original Bert repository and papers for details on the pretraining: https://github.com/google-research/bert

Thank you! Get it.
Erroneous Code
Author: LeoLai930603
Pull request: no
Labels: name: wontfix

I guess there is a minor mistake in this line.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/80684f6f86c13a89fc1e4feac248ef96b013765c/pytorch_pretrained_bert/modeling_transfo_xl.py#L1385

In `TransfoXLLMHeadModel`, the forward computation requires the target (if available) has the shape [batch_size, sequence_length], that is, it is rank-2 here. However, in the `ProjectedAdaptiveLogSoftmax` (case when `config.sample_softmax < 0`, and the default value 
of `config.sample_softmax` was set to -1), the forward compuation (indicated in the above line) requires the target (if available) has the shape [batch_size * sequence_length], which means it is rank-1 here. This could not passed the assertion check in the forward computation and raise the error.

Luckily I have checked the computation logic for the Transformer-XL part, most of them are correct. Therefore I suggest just a minor change from `softmax_output = self.crit(pred_hid.view(-1, pred_hid.size(-1)), target) ` to `softmax_output = self.crit(pred_hid.view(-1, pred_hid.size(-1)), target.reshape(-1)) `

If you prefer PR, I could do this little favor;)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Write with Transformer doesn't show 774M model?
Author: zacharymacleod
Pull request: no

According to [this twitter post](https://twitter.com/huggingface/status/1166368535221870592), Write With Transformer has been updated to include the Large model.

However, the Model & Decoder Settings only shows Small and Medium options for model size.

Comments:
(nevermind: refreshing the page fixed it... it's weird though because I've been checking every day!)

Hi! Your browser cache was probably playing tricks on you :)

That's exactly what I figured! Forgot the term for it. Kinda sucks that it did so for several days! :(

Anyways, I'm having fun with it now so it's all good!

Glad you like it!!
XLNet resize embedding size ERROR
Author: Saner3
Pull request: no

## ❓ Questions & Help

I add new tokens to XLNetLMHeadModel and use resize function
```
tokenizer.add_tokens(["<token1>", "<token2>"])
model.resize_token_embeddings(len(tokenizer))
```
But when running, the following error occurs:
```
Traceback (most recent call last):
 ...
  File "/nas/home/jsun/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "../pytorch_transformers/modeling_xlnet.py", line 1059, in forward
    logits = self.lm_loss(transformer_outputs[0])
  File "/nas/home/jsun/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/nas/home/jsun/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "/nas/home/jsun/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1410, in linear
    output += bias
RuntimeError: The size of tensor a (32003) must match the size of tensor b (32000) at non-singleton dimension 2
```

It is because in `resize_token_embeddings`, it changes the embedding size and calls `tie_weight` function to resize the LM head weight. But it forgot to change the size of `bias`, since XLNet has 
```
self.lm_loss = nn.Linear(config.d_model, config.n_token, bias=True)
```
while other models have bias=False

Comments:
Should be fixed on master by @LysandreJik's PR!