noamaanMulla-03/transformers-issues huggingface dataset:
No gradient clipping in AdamW
Author: OlegPlatonov
Pull request: no
Labels: name: wontfix

Hi!

After moving from pretrained-bert to transformers I've noticed that the new AdamW optimizer does not perform gradient clipping, even though both BertAdam and OpenAIAdam used to do it. 

Also, in finetune_on_pregenerated example bias correction is turned off only for FusedAdam, but not for AdamW.

Comments:
Yes, the LM fine-tuning example will be refactored.

Adding the removal of gradient clipping to the list of breaking changes, thanks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can you please share the pre-processed text dump of the bookcorpus and wikipediacorpus?
Author: kamalravi
Pull request: no

## ❓ Questions & Help
I am trying to train distilbert with different architecture. If you can share the text dump for the pre-training, it would be great. Thanks! 
<!-- A clear and concise description of the question. -->


Comments:
Hello @kamalravi 

For the English Wikipedia data, I followed the scripts in XLM [here](https://github.com/facebookresearch/XLM#train-your-own-monolingual-bert-model). It downloads the latest dump and does the necessary pre-processing.
For BookCorpus, as you probably know, TBC is not distributed anymore and it's not clear to me whether I can distribute it here (I prefer not to). However, there is open-source options to collect a similar dataset (like [this one](https://github.com/soskek/bookcorpus)).
If you are ever interested in Reddit-based dataset, I used [OpenWebTextCorpus](https://skylion007.github.io/OpenWebTextCorpus/) following RoBERTa to distill DistilGPT2.

Having the raw text dumps, I simply use `scripts/binarized_data.py` to pre-process the data.

Victor
Using Fp16 half precision makes Bert prediction slower.
Author: Oxi84
Pull request: no
Labels: name: wontfix

When I use:

     model = BertForMaskedLM.from_pretrained('bert-large-cased') 
     model = model.half()
     model.eval()
     model.to('cuda')

by adding Fp16:

     model = model.half()

It runs around 50% slower. Why is that?
 I run it on ubuntu 18.04, cuda 9, pytorch 1.1 and python 3.6.8




Comments:
gtx 1080. Is there any other way to make the predictions faster?

Hi, you need at least a Volta GPU to get benefits from fp16 unfortunately.

@thomwolf  Does P100 applicable?

I don't think so

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Updating run_swag script for new pytorch_transformers setup
Author: yakazimir
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/pytorch-transformers/blob/f2a3eb987e1fc2c85320fc3849c67811f5736b50/examples/single_model_scripts/run_swag.py#L35

It appears that WEIGHTS_NAME and CONFIG_NAME have moved out of {pytorch_transformers/pytorch_pretrained_bert}.file_utils (and instead can be imported directly from pytorch_transformers), as shown below : 

``` 
>>> import pytorch_transformers as p
>>> p.__version__
'1.0.0'
>>> from pytorch_transformers.file_utils import WEIGHTS_NAME, CONFIG_NAME
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'WEIGHTS_NAME'
>>> from pytorch_transformers.file_utils import CONFIG_NAME
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'CONFIG_NAME'
>>> from pytorch_transformers import WEIGHTS_NAME
>>> 
```

This seems incorrect in the new run_swag script (as shown above) (if true,  are there any other important imports that were overlooked here?)

Comments:
I found a few other issues: 1) the script uses the old BertAdam (instead of AdamW)

https://github.com/huggingface/pytorch-transformers/blob/44dd941efb602433b7edc29612cbdd0a03bf14dc/examples/single_model_scripts/run_swag.py#L431

2) the train and test loops still use the old version of forward, i.e., 

https://github.com/huggingface/pytorch-transformers/blob/44dd941efb602433b7edc29612cbdd0a03bf14dc/examples/single_model_scripts/run_swag.py#L450
https://github.com/huggingface/pytorch-transformers/blob/44dd941efb602433b7edc29612cbdd0a03bf14dc/examples/single_model_scripts/run_swag.py#L525

In the former case, I added the following: 

```
                #loss = model(input_ids, segment_ids, input_mask, label_ids) # line 450 in train
                outputs = model(input_ids, segment_ids, input_mask, label_ids)
                loss = outputs[0]
```
 
And the latter: 

```
                ## tmp_eval_loss = model(input_ids, segment_ids, input_mask, label_ids) ## line 525
                output = model(input_ids, segment_ids, input_mask, label_ids)
                tmp_eval_loss,logits = output[:2]
```

I'd be curious if the last two versions are correct. I'm actually not able to reproduce the Swag results I was getting before the refactoring (reported here: https://github.com/huggingface/pytorch-transformers/blob/v0.6.2/README.md). Rather than getting around 80%, I'm stuck at around 78%. 

Yes the run_swag script hasn't been updated to the new API yet.

Do you want to give it a look and submit a PR? I don't have plan to work on it in the short-term.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLNet large low accuracy
Author: hzeng-otterai
Pull request: no
Labels: name: wontfix

I was running run_glue.py on one of my classification problems. Using XLNet-base-cased, everything seems to be fine, the classification accuracy converge to around 92%. But using XLNet-large, the accuracy is 89% at the first checkpoint and then drop into 24.85% at the second checkpoint. The data should be ok because I have been running many different algorithms on it. 

Some of the logs as below. What could be the possible cause?

```
07/31/2019 05:35:28 - INFO - __main__ -   Saving features into cached file /data/_working/sentiment/large/cached_dev_xlnet-large-cased_128_sentimentall
07/31/2019 05:35:29 - INFO - __main__ -   ***** Running evaluation  *****
07/31/2019 05:35:29 - INFO - __main__ -     Num examples = 10000
07/31/2019 05:35:29 - INFO - __main__ -     Batch size = 8
07/31/2019 05:38:49 - INFO - __main__ -   ***** Eval results  *****
07/31/2019 05:38:49 - INFO - __main__ -     acc = 0.8902
07/31/2019 05:38:55 - INFO - __main__ -   Saving model checkpoint to /data/_working/sentiment/large/output/checkpoint-500000/1250 [03:19<00:00,  6.16it/s]
07/31/2019 13:56:16 - INFO - __main__ -   Loading features from cached file /data/_working/sentiment/large/cached_dev_xlnet-large-cased_128_sentimentalls]
07/31/2019 13:56:16 - INFO - __main__ -   ***** Running evaluation  *****
07/31/2019 13:56:16 - INFO - __main__ -     Num examples = 10000
07/31/2019 13:56:16 - INFO - __main__ -     Batch size = 8
07/31/2019 13:59:39 - INFO - __main__ -   ***** Eval results  *****
07/31/2019 13:59:39 - INFO - __main__ -     acc = 0.2485
07/31/2019 13:59:44 - INFO - __main__ -   Saving model checkpoint to /data/_working/sentiment/large/output/checkpoint-100000/1250 [03:22<00:00,  6.16it/s]
 Iteration:  72%|██████████████████████████████████████████████████████████████████▉                          | 142278/197500 [23:46:00<9:18:52,  1.65it/s]07/31/2019 22:22:27 - INFO - __main__ -   Loading features from cached file /data/_working/sentiment/large/cached_dev_xlnet-large-cased_128_sentimentalls]
07/31/2019 22:22:27 - INFO - __main__ -   ***** Running evaluation  *****
07/31/2019 22:22:27 - INFO - __main__ -     Num examples = 10000
07/31/2019 22:22:27 - INFO - __main__ -     Batch size = 8
07/31/2019 22:25:49 - INFO - __main__ -   ***** Eval results  *****
07/31/2019 22:25:49 - INFO - __main__ -     acc = 0.2485
07/31/2019 22:25:55 - INFO - __main__ -   Saving model checkpoint to /data/_working/sentiment/large/output/checkpoint-150000/1250 [03:21<00:00,  6.00it/s]

```

Comments:
The first thought could be that the learning rate is too high and you overfit.
You probably should try changing the batch size too.
You can have a look at #795 where we discussed similar questions for SST-2. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Chinese BERT broken
Author: Liangtaiwan
Pull request: no
Labels: name: wontfix

There are still some bug after #860 
The same issue is also mention in #903

I'm running on Chinese-Style SQuAD dataset (DRCD).
I can train Chinese-Bert successfully about half year ago.
However, I could not train the model successfully but I can train Multi-Bert successfully.

I'm not able to find out the reasons.
@thomwolf I think there should be more test in this repo as the project is fast growing.

Comments:
Yes you need to install from master for now. We have not yet done a new release with the fix of #860.

@thomwolf Not related to this specific issue here, but do you think it makes sense to add the following policy to the newly introduced issue templates: all bug reports should be filed against latest `master` version of PyTorch-Transformers (incl. pip install with git master url) 🤔

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Wrong refactoring of mandatory parameters for run_squad.py
Author: mathiasburger
Pull request: no
Labels: name: wontfix

When only running evaluation on a squad dev set, it should *not* be mandatory to add a --train_file because only the --predict_file is necessary.

Current script invocation:
```
python run_squad \
    --model_type bert \
    --model_name_or_path xxx \
    --output_dir xxx \
    --train_file UNNECESSARY_BUT_MANDATORY \
    --predict_file xxx \
    --version_2_with_negative \
    --do_eval \
    --per_gpu_eval_batch_size 2
```

Desired script invocation without --train_file param:
```
python run_squad \
    --model_type bert \
    --model_name_or_path xxx \
    --output_dir xxx \
    --predict_file xxx \
    --version_2_with_negative \
    --do_eval \
    --per_gpu_eval_batch_size 2
```

Before refactoring in 50b7e52 the behavior was correct.

Comments:
indeed, we could remove this

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
_convert_id_to_tokens for XLNet not working
Author: chris-boson
Pull request: no
Labels: name: wontfix

```
text = self.tokenizer.convert_ids_to_tokens(token_list)
  File "/home/lambda/repos/pytorch-transformers/pytorch_transformers/tokenization_utils.py", line 444, in convert_ids_to_tokens
    tokens.append(self._convert_id_to_token(index))
  File "/home/lambda/repos/pytorch-transformers/pytorch_transformers/tokenization_xlnet.py", line 170, in _convert_id_to_token
    token = self.sp_model.IdToPiece(index)
  File "/home/lambda/python-envs/research/lib/python3.6/site-packages/sentencepiece.py", line 187, in IdToPiece
    return _sentencepiece.SentencePieceProcessor_IdToPiece(self, id)
TypeError: in method 'SentencePieceProcessor_IdToPiece', argument 2 of type 'int'
```

I find that if I explicitly convert ids to integers it works fine. In `tokenization_xlnet.py`
```
def _convert_id_to_token(self, index, return_unicode=True):
    """Converts an index (integer) in a token (string/unicode) using the vocab."""
    token = self.sp_model.IdToPiece(int(index))
    if six.PY2 and return_unicode and isinstance(token, str):
        token = token.decode('utf-8')
    return token
```

Comments:
Which command can we use to reproduce the behavior?

Upon further testing, looks like this tokenizer doesn't like numpy arrays, the other ones seem to be fine
```
import numpy as np
from pytorch_transformers import XLNetTokenizer, TransfoXLTokenizer, BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print(tokenizer.convert_ids_to_tokens(np.array([3, 4, 6, 2356])))
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
print(tokenizer.convert_ids_to_tokens(np.array([3, 4, 6, 2356])))
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
print(tokenizer.convert_ids_to_tokens(np.array([3, 4, 6, 2356]).tolist()))
print(tokenizer.convert_ids_to_tokens(np.array([3, 4, 6, 2356]))) # Above error
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
convert int to str before adding to a str
Author: luranhe
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1487?src=pr&el=h1) Report
> Merging [#1487](https://codecov.io/gh/huggingface/transformers/pull/1487?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6596e3d56626c921b3920e313866b7412633b91a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1487/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1487?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1487   +/-   ##
=======================================
  Coverage   85.59%   85.59%           
=======================================
  Files          91       91           
  Lines       13526    13526           
=======================================
  Hits        11578    11578           
  Misses       1948     1948
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1487?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1487?src=pr&el=footer). Last update [6596e3d...dd904e2](https://codecov.io/gh/huggingface/transformers/pull/1487?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks!
model.to(args.device) in run_glue.py taking around 10 minutes. Is this normal?
Author: pydn
Pull request: no

## ❓ Questions & Help

Currently line 484 of run_glue.py `model.to(args.device)` is taking close to 10 minutes to complete when loading the bert-base pretrained model. This seems like a long time compared to what I was seeing in pytorch-transformers.

My configuration:
    Tesla V100 - Driver 418.87.00
    Cuda toolkit 10.1
    PyTorch 1.3.0

The code I am running is: 
`python example/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $(MY TASK) \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $(MY_DIR) \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=64   \
    --per_gpu_train_batch_size=64   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir $(MY_OUTDIR) \
    --overwrite_output_dir \
    --fp16`

Is this behavior expected or am I doing something wrong? Thanks!




Comments:
This seems weird, I'm looking into this.

By running the run_glue.py script as it is right now with your exact parameters, I timed to model.to and it took 6.4 seconds

Ok, thanks for looking into that! I'm using my own dataset so I made adjustments to the processor, but I don't think that should be causing the issue when transferring the model to the GPU. I'll run a few more tests and see if I can pinpoint what is going on. It's super helpful to know that you are seeing it take only 6.4 seconds. Thank you!

I just tested again using the SST-2 data keeping the run_glue.py code as is and I'm still having the same issue. My guess is that there is something with my VM set up that's causing the hanging issue. I'm having a hard time identifying what might be the exact cause of the issue.

Hmm do you think you can reproduce it on another VM? Are you running into the same issue if you simply put the model on the device in a standalone script?

Ok, it's definitely an issue with my setup. I have the same issue when running the following: 
`from torchvision import models

model = models.densenet121(pretrained=True)
model.to('cuda')`

I'll close the issue and keep troublehsooting on my end. Thanks!

Reopening because I found the issue and hopefully it can help someone else. I was comparing model loading times to what I was seeing on the hosted runtimes in Google Colab notebooks. 

Even through they have cuda toolkit 10.1 installed as you can see when running the command !nvidia-smi, when you run torch.version.cuda they have 10.0.130 installed instead of the 10.1 version. They are also running pytorch 1.2.0. 

I downgraded my environment to match and the model from models.densenet121(pretrained=True) loaded in 4.9 seconds.

Thanks for the help!
Replace TensorboardX with Pytorch's built in SummaryWriter
Author: bilal2vec
Pull request: no

## 🚀 Feature

Import `SummaryWriter` from `from torch.utils.tensorboard` instead of from `tensorboardX`

If you're interested, I can make a pull request to merge the changes that I made in my [fork](https://github.com/bkkaggle/transformers) into the main repository.

## Motivation

TensorboardX isn't needed anymore now that Pytorch 1.2 has been released. The relevant Pytorch docs are available [here](https://pytorch.org/docs/stable/tensorboard.html).

Comments:
You cannot assume that suddenly _everyone_ is on 1.2. You'll need a fallback for people who are 1.x. Something like

```python
try:
    from torch.utils.tensorboard import SummaryWriter
except AttributeError
    from tensorboardX import SummaryWriter
```

That's a good way to 'ease into' a definite change.

Fixed.

 I undid the commits removing tensorboardX from the requirements and added a try-except block around the imports to check if the user's pytorch version comes with tensorboard (It's 'experimental' in 1.1.0 and is now stable in 1.2.0).

I updated my fork to be in line with the recommendations for contributing in #1448

I created the pull request:

closing now that #1454 has been merged in
Updating simple_lm_finetuning.py for FP16 training
Author: crowegian
Pull request: no
Labels: name: wontfix

in simple_lm_finetuning the recent updated code doesn't work with the old optimizer specifications. 

When not running with --fp16 
`
optimizer = BertAdam(optimizer_grouped_parameters,
                     lr=args.learning_rate,
                     warmup=args.warmup_proportion,
                     t_total=num_train_optimization_steps)
// In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=args.learning_rate, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = WarmupLi
`

fixes the problem as suggested.

But when running --fp16 
`scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_train_optimization_steps)  # PyTorch scheduler`
will error out with FP16_Optimizer saying that 

> TypeError: FP16_Optimizer is not an Optimizer

Does the FuseAdam object need to be passed into WarmupLinearSchedule instead?

Comments:
Hi, has this been fixed? I've tried updating my language modeling script to match but still getting errors.

Having the same problem at the moment ...

I guess the preferred way is to use  `apex.amp` like in this example?
https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using memory states with XLNet / TransfoXL
Author: chris-boson
Pull request: no
Labels: name: wontfix

I would like to fine-tune XLNet / TransfoXL for a classification task where I classify each sentence in the context of a large document. Is there an example for how to use the memory states in XLNet and TransfoXL? 
This example only uses memory states for inference but there is no example for training:
https://github.com/huggingface/pytorch-transformers/blob/xlnet/examples/single_model_scripts/run_transfo_xl.py

This example doesn't use the memory states:
https://github.com/huggingface/pytorch-transformers/blob/24ed0b9346079da741b952c21966fdc2063292e4/examples/run_xlnet_classifier.py

Naively feeding in the memory states leads to some dimension mismatch at the end of the training epoch:
```
  File "/home/lambda/repos/research/trainer/models/xlnet.py", line 86, in forward
    mems=new_mems
  File "/home/lambda/python-envs/research/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lambda/repos/pytorch-transformers/pytorch_transformers/modeling_xlnet.py", line 959, in forward
    new_mems = new_mems + (self.cache_mem(output_h, mems[i]),)
  File "/home/lambda/repos/pytorch-transformers/pytorch_transformers/modeling_xlnet.py", line 792, in cache_mem
    new_mem = torch.cat([prev_mem, curr_out], dim=0)[-self.mem_len:]
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 14 and 32 in dimension 1 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71
```

Comments:
Which command did you use to "naively" feed in the memory states?
You can just feed the mems that you get from the previous forward pass, but the inputs need to be the continuation of the previous input. So the batch_size, in particular, should stay the same.

I found this post that talks about how to organize the inputs: https://mlexplained.com/2019/07/04/building-the-transformer-xl-from-scratch/

Is there an example for how to adapt this for classification? Right now the data is organized as `(batch_size x max_seq_length)` and labels are `batch_size`. Each example in the batch represents  a sentence, where multiple sentences may come from the same document. For simplicity we can assume they are all from the same document.

I was looking at doing the same with TransformerXL, but ran into this same issue regarding how to adapt the label vector to work with the data matrix when trying to do classification. I'd appreciate any help from people that have successfully implemented this.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.