noamaanMulla-03/transformers-issues huggingface dataset:
error when do python3 run_squad.py
Author: directv00
Pull request: no
Labels: name: Need more information

Hello,
I am newbie of pytorch-pretrained-Bert.
After successfully converted from init-checkpoint of tensorflow to pytorch bin,
I found an error when I do run_squad.
Guessing I should've included some configuration ahead, could anyone can help?
See below.
```bash
File "run_squad.py", line 37, in <module>
    from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE, WEIGHTS_NAME, CONFIG_NAME
ImportError: No module named pytorch_pretrained_bert.file_utils
```


Comments:
Did you install pytorch-pretrained-bert as indicated in the README?
`pip install pytorch_pretrained_bert`

You don't have to convert the checkpoints yourself, there are already converted.

Try reading the installation and usage sections of the README.

Of cause I installed,
More precisely, error code is slightly changed.
```bash
Traceback (most recent call last):
  File "run_squad.py", line 37, in <module>
    from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE, WEIGHTS_NAME, CONFIG_NAME
ImportError: cannot import name WEIGHTS_NAME
```

Hmm you are right, the examples are compatible with `master` only now that we have a new token serialization. I guess we'll have to do a new release (0.6.2) today so everybody is on the same page.
Let me do that.

Actually, we'll wait for the merge of #506.

In the meantime you can install from source and it should work.

Oh it's done immediately when I installed from source. Thanks.

Ok great.

Just a side note on writing messages in github: you should add triple-quotes like this: \``` before and after the command line, errors and code you are pasting. This way it's easier to read.

Ex:
\```
pip install -e .
\```

will display like:
```
pip install -e .
```

Good point(triple quotes).
I didn't know what to do, but now I have it all.
Thanks.

> Actually, we'll wait for the merge of #506.
> 
> In the meantime you can install from source and it should work.

how to "install from source"?

@YanZhangADS 

You can install from source with this command below
```
git clone https://github.com/huggingface/pytorch-pretrained-BERT.git
cd pytorch-pretrained-BERT
python setup.py install
```

Same problem with "ImportError: cannot import name WEIGHTS_NAME". However, after building **0.6.1** from source, I get: 
```
from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear
    ImportError: cannot import name 'warmup_linear'
```
I don't need the warmup, so I removed the import, but letting you guys know that this is an import error as well. Thanks!

Thanks for that @dumitrescustefan, we're working on it in #518.
I'm closing this issue for now as we start to deviate from the original discussion.

I just built from source. I'm still getting the same error as in original issue.

The version 0.4.0 doesn't give this issue.
pip install pytorch_pretrained_bert==0.4.0
Store Model cards in the repo
Author: julien-c
Pull request: yes


Comments:
Thanks for importing the readme files :heart: 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2701?src=pr&el=h1) Report
> Merging [#2701](https://codecov.io/gh/huggingface/transformers/pull/2701?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d426b58b9e32a2ffc8c8a1196143270e22054a46?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2701/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2701?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2701   +/-   ##
=======================================
  Coverage   74.25%   74.25%           
=======================================
  Files          92       92           
  Lines       15216    15216           
=======================================
  Hits        11298    11298           
  Misses       3918     3918
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2701?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2701?src=pr&el=footer). Last update [d426b58...d126da9](https://codecov.io/gh/huggingface/transformers/pull/2701?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Using model output by transformers (v2.0) in older versions (0.4.0 or 1.0.0) 
Author: Genius1237
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I have a bert model finetuned on in-domain data on the latest version of the package (`2.0`). I would now like to use this in some code that is written with the older version of the package (say `0.4.0` or `1.0.0`). Would this be possible?

I tried pointing the code which imports version `0.4.0` to the model output and it gave an error saying `bert_config.json` not found, but there was a `config.json` in the model folder. I renamed `config.json` file and made the code run again and it seems to run.

Am I on the right track? Is this all I have to do to get it run?


Comments:
Models' architectures should have stayed relatively the same between versions. If you did not get warnings telling you that some layers had not been loaded, then you should be good to go!

You could try and compare inferences between two environments which have different versions of the library installed to make sure that they're the same.

Well, I don't get any warnings, so that is good.

Could someone comment on `bert_config.json` vs `config.json`. Has there been any change in the naming scheme with the newer versions?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
TypeError: convert_examples_to_features() got an unexpected keyword argument 'sequence_a_is_doc'
Author: snijesh
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert):

Language I am using the model on (English):

The problem arise when using:
```
!python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-large-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file train-v1.1.json \
  --predict_file dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Fine-tuning on SQuAD


## Environment

* OS:
* Python version: 3.6
* PyTorch version: 1.3.1'
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? Yes



Comments:
Please, try to install Transformers library **from source code**, and **not from PyPi**. The former one is the up-to-date version. In fact, if you see in [utils_squad.py](https://github.com/huggingface/transformers/blob/master/examples/utils_squad.py) at row 197, there is a parameter called `sequence_a_is_doc` in the definition of the `convert_examples_to_features()` method. Try it out and keep us updated on this problem!

> ##  Bug
> Model I am using (Bert):
> 
> Language I am using the model on (English):
> 
> The problem arise when using:
> 
> ```
> !python run_squad.py \
>   --model_type bert \
>   --model_name_or_path bert-large-uncased \
>   --do_train \
>   --do_eval \
>   --do_lower_case \
>   --train_file train-v1.1.json \
>   --predict_file dev-v1.1.json \
>   --per_gpu_train_batch_size 12 \
>   --learning_rate 3e-5 \
>   --num_train_epochs 2.0 \
>   --max_seq_length 384 \
>   --doc_stride 128 \
>   --output_dir /tmp/debug_squad/
> ```
> 
> The tasks I am working on is:
> 
> * [ ]  an official GLUE/SQUaD task: Fine-tuning on SQuAD
> 
> ## Environment
> * OS:
> * Python version: 3.6
> * PyTorch version: 1.3.1'
> * PyTorch Transformers version (or branch): 2.1.1
> * Using GPU ? Yes

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Abruptly model training was stopped
Author: ellurunaresh
Pull request: no
Labels: name: wontfix

I'm fine tuning NER custom dataset using BERT cased model. I'm using script of examples/run_ner.py to fine tune.  **In 2nd epoch training was stopped abruptly without displaying any error.**

Epoch:  67%|██████▋   | 2/3 [2:48:41<1:24:21, 5061.32s/it]
Iteration:  98%|█████████▊| 3734/3816 [1:22:30<01:48,  1.32s/it]
Iteration:  98%|█████████▊| 3735/3816 [1:22:31<01:47,  1.33s/it]

**The training was stopped at 98%.**

Training details are given here:
no. of training sentences: 122095, batch size=32, num_epochs=3, save_steps=750, GPU server: Tesla K40m

Could anybody help me out how to solve this issue and please let me know if you need any further information.

Thanks in advance


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix typo in examples/utils_ner.py
Author: falcaopetri
Pull request: yes

`"%s-%d".format()` -> `"{}-{}".format()`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2707?src=pr&el=h1) Report
> Merging [#2707](https://codecov.io/gh/huggingface/transformers/pull/2707?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ddb6f9476b58ed9bf4433622ca9aa49932929bc0?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2707/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2707?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2707   +/-   ##
=======================================
  Coverage   74.25%   74.25%           
=======================================
  Files          92       92           
  Lines       15216    15216           
=======================================
  Hits        11298    11298           
  Misses       3918     3918
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2707?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2707?src=pr&el=footer). Last update [ddb6f94...dd19c80](https://codecov.io/gh/huggingface/transformers/pull/2707?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Good catch, thanks
Benchmark not replicable
Author: Pointy-Hat
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Hello. I wanted to test if everything is allright with my downloads and so I ran the code snippet you provided in the section **Fine-tuning Bert model on the MRPC classification task** in the main README file (the only difference being the number of gpus-I use 4). However, my evaluation results are well below the ones you mention. I get 
```
acc = 0.8725490196078431
acc_and_f1 = 0.888829254329469
f1 = 0.9051094890510949
```
in my terminal.

There is also no output file in the specified folder.

Do you know what could cause this?

Thanks for answer
MS

Comments:
Hello @Pointy-Hat 

Please tell us more, as explained [here](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md).

Technical data of the cluster used for computation:
System: CentOS Linux release 7.5.1804 (Core)
Python: 3.7.4
Pytorch: 1.3.1

Code run:
```
python3 -m torch.distributed.launch --nproc_per_node 4 ./examples/run_glue.py   \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --task_name MRPC \
    --do_train   \
    --do_eval   \
    --do_lower_case   \
    --data_dir $GLUE_DIR/MRPC/   \
    --max_seq_length 128   \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5   \
    --num_train_epochs 3.0  \
    --output_dir /tmp/mrpc_single/ \
    --overwrite_output_dir   \
    --overwrite_cache 

```

Expected results (as per README.txt):

```
acc = 0.8823529411764706
acc_and_f1 = 0.901702786377709
f1 = 0.9210526315789473
```

Obtained results:
```
acc = 0.8725490196078431
acc_and_f1 = 0.888829254329469
f1 = 0.9051094890510949
```

GLUE data obtained via their `download_glue_data.py` script, as recommended in README.

did you try multiple random seeds?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_summarization_finetuning.py
Author: jmamou
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Hi
Does your code run_summarization_finetuning.py implement the abstractive summarization approach described in "Text summarization with pretrained encoders." by Liu, Yang, and Mirella Lapata?
Thanks
<!-- A clear and concise description of the question. -->


Comments:
I think so. The script called `run_summarization_finetuning.py` has the goal of extracting text summary given as input a text document. This code shows how to fine-tune a text summarizer model (called **BertSum**) with two different datasets: CNN and Daily Mail.

> ##  Questions & Help
> Hi
> Does your code run_summarization_finetuning.py implement the abstractive summarization approach described in "Text summarization with pretrained encoders." by Liu, Yang, and Mirella Lapata?
> Thanks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Changing the docs as per Pytorch v1.1+
Author: AdityaSoni19031997
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

[Docs Link](https://huggingface.co/transformers/migration.html#optimizers-bertadam-openaiadam-are-now-adamw-schedules-are-standard-pytorch-schedules)

```
# From the Docs
### In Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps)  # PyTorch scheduler
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
    scheduler.step()
    optimizer.step()
```

As per the Pytorch 1.1+, 

>Prior to PyTorch 1.1.0, the learning rate scheduler was expected to be called before the optimizer’s update; 1.1.0 changed this behavior in a BC-breaking way. If you use the learning rate scheduler (calling scheduler.step()) before the optimizer’s update (calling optimizer.step()), this will skip the first value of the learning rate schedule. If you are unable to reproduce results after upgrading to PyTorch 1.1.0, please check if you are calling scheduler.step() at the wrong time.

[Pytorch Reference Link](https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate)

Thanks.

PS Not sure if the issue category selected is apt.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
DistributedDataParallel for multi-gpu single-node runs in run_lm_finetuning.py
Author: Genius1237
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
Modify `run_lm_finetuning.py` with DDP for multi-gpu single-node jobs.


## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->
In it's current state, `run_lm_finetuning.py` does not run in DDP for multi-gpu single-node training jobs. This results in all but the first GPU having very low utilization (as low as 50%, when the first one is in the high 80%) due to the way simple DP works. Once implemented, the load would be more evenly balanced across all the GPUs.
## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
I can help implementing this feature, but would need guidance on what should/shouldn't be modified to get this working properly.

Comments:
As far as I can see, the script fully supports DDP:

https://github.com/huggingface/transformers/blob/2ba147ecffa28e5a4f96eebd09dcd642117dedae/examples/run_lm_finetuning.py#L282-L286

I haven't run the script myself, but looking at the source this should work with the [torch launch](https://pytorch.org/docs/stable/distributed.html#launch-utility) utility. Your command would then look like this when using a single node with four GPUs.

```bash
python -m torch.distributed.launch --nproc_per_node 4 run_lm_finetuning.py [arguments]
```


Ah ok, I wasn't aware that it had to be launched this way. I was looking at the code and thought it DDP would happen only when the process was launched across multiple nodes.

Thanks for the help @BramVanroy
BertForMultipleChoice
Author: apratim-mishra
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Shape Error when using BrtForMultipleChoice

![image](https://user-images.githubusercontent.com/26097066/69660490-0ff9b680-1046-11ea-89a3-fa60b769be97.png)


Below is the model i used:

![image](https://user-images.githubusercontent.com/26097066/69660530-26a00d80-1046-11ea-9cbd-b71d56a8bd22.png)


Comments:
Hi, could you try and test this on a more recent version of the library and let me know if it works?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Changes to S3 Roberta / RobertaForSequenceClassification
Author: frankfka
Pull request: no
Labels: name: wontfix

Hello,
I'm wondering if there have been any changes in either the pretrained Roberta model or the configuration of RobertaForSequenceClassification within the past month or so. I am initializing it as `RobertaForSequenceClassification.from_pretrained(...)` and running it as demonstrated in `run_glue.py`.

For a custom dataset, I am noticing that the fine-tune accuracy has decreased by several percentage points on newly trained models as opposed to ~ 1 month ago. This happens consistently (i.e I've tried retraining multiple times) and using the same hyperparams.

In addition, I've noticed that the new training times are cut by ~1/2, so the model seems to train faster, but is less performant.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rectified Adam + LARS
Author: i404788
Pull request: no
Labels: name: wontfix

## 🚀 Feature
There has been a lot of buzz around the new Radam and Ralamb (Radam + LARS) optimizers, and I was wondering if it could also be implemented in pytorch-transformers.

## Motivation

It seems to have consistent performance improvements. It also seems to handle different learning rates a lot better. And with LARS it also allows for really large batch sizes without regressing.

## Additional context

https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20
https://github.com/mgrankin/over9000
https://twitter.com/jeremyphoward/status/1162118545095852032
https://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b

Comments:
From what I can tell, Radam makes automatic warmup and LARS is good but requires more calculations per batch. Before implementing it here it's worth do testing to tell if it's a good idea.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


any update?

For anyone interested in testing, I've created a fork that uses Radam+LARS+LookAhead, https://github.com/i404788/transformers

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Added README.md to Swedish BERT models from National Library of Sweden
Author: marma
Pull request: yes

Following the lead of others these are not actual model cards but rather the README.md-files from https://github.com/Kungbib/swedish-bert-models

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2716?src=pr&el=h1) Report
> Merging [#2716](https://codecov.io/gh/huggingface/transformers/pull/2716?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2ba147ecffa28e5a4f96eebd09dcd642117dedae?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2716/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2716?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2716   +/-   ##
=======================================
  Coverage   74.09%   74.09%           
=======================================
  Files          93       93           
  Lines       15248    15248           
=======================================
  Hits        11298    11298           
  Misses       3950     3950
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2716?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2716?src=pr&el=footer). Last update [2ba147e...e46c8bf](https://codecov.io/gh/huggingface/transformers/pull/2716?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @marma! 

As also mentioned on https://github.com/huggingface/transformers/pull/2720#issuecomment-581430234, we'll find a way for users to get merge rights on their model cards (via a GitHub bot maybe)