noamaanMulla-03/transformers-issues huggingface dataset:
How can I assign a specific gpu when using examples/run_language_modeling.py?
Author: ridiculouz
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hello, I'm wondering if I can assign a specific gpu when using examples/run_language_modeling.py to train a language model?
Lots of thanks!

Comments:
This might answer your question: https://stackoverflow.com/questions/39649102/how-do-i-select-which-gpu-to-run-a-job-on

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Conversion script to export transformers models to ONNX IR.
Author: mfuntowicz
Pull request: yes

This PR adds conversion script to export our models to ONNX IR. 

Plan is to support both PyTorch and TensorFlow: 

- [x] PyTorch
- [x] TensorFlow

TensorFlow currently blocked because of an issue in the conversion script provided by ONNX (seems fixed on master : https://github.com/onnx/tensorflow-onnx/issues/876)

_PS: PR recreated because the first one got corrupted (#4176)._

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4253?src=pr&el=h1) Report
> Merging [#4253](https://codecov.io/gh/huggingface/transformers/pull/4253?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c558761ad11beba5e70dd505e1c97bdc4eb6ee9c&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4253/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4253?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4253   +/-   ##
=======================================
  Coverage   78.16%   78.16%           
=======================================
  Files         120      120           
  Lines       20005    20005           
=======================================
  Hits        15637    15637           
  Misses       4368     4368           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4253?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4253?src=pr&el=footer). Last update [c558761...c558761](https://codecov.io/gh/huggingface/transformers/pull/4253?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


For the notebook:
```
  "environ[\"OMP_NUM_THREADS\"] = str(cpu_count(logical=True))\n",
  "environ[\"OMP_WAIT_POLICY\"] = 'ACTIVE'
```
  It is better to place the environ setting before importing onnxruntime. I used to find that placing these settings after importing onnxruntime does not work.


```
   options.execution_mode = ExecutionMode.ORT_SEQUENTIAL 
```
ORT_SEQUENTIAL is the default value so the line is optional.

@tianleiwu I agree with your regarding optional argument, especially GPT2 and the past context which might provide a significant boost in performance.

For the scope of this release, I decided to not provide support for this, as it would have required much more thinking / changes in the library. 

We might have a look at possible solutions for this in short/mid term 👍
[Pipeline, Generation]  tf generation pipeline bug
Author: patrickvonplaten
Pull request: yes

Generation can currently not be used for TF due to a small bug. This PR fixes it and adds tests.

I will update the usage docs and pipeline notebook tomorrow.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4217?src=pr&el=h1) Report
> Merging [#4217](https://codecov.io/gh/huggingface/transformers/pull/4217?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8bf7312654d40cea1a399e86f9fe8e39e1ea3a1e&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4217/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4217?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4217   +/-   ##
=======================================
  Coverage   78.38%   78.39%           
=======================================
  Files         120      120           
  Lines       19916    19918    +2     
=======================================
+ Hits        15611    15614    +3     
+ Misses       4305     4304    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4217?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/4217/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `76.36% <100.00%> (+0.09%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4217/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.96% <0.00%> (-0.13%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4217/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.93% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4217/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `74.38% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4217?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4217?src=pr&el=footer). Last update [8bf7312...0e6bf4e](https://codecov.io/gh/huggingface/transformers/pull/4217?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


^^ I borked your PR unfortunately, @patrickvonplaten  (me = 💩). Sorry about that. Care to open a new one?

(or commit your change directly as others have approved)

No worries, I used my bullet proof manual rebase of copy-paste files :D. 

This looks good to me. Can you check quickly and merge? @julien-c
NER: some issues in PyTorch Lightning example
Author: stefan-it
Pull request: no
Labels: name: wontfix

Hi,

I wanted to try out the new NER example script (`./ner/run_pl_ner.py`) that uses PyTorch Lightning.

Here are some bugs that I've found:

Dataset preparation method is not called. Usually, InputBatch batches or input features are written and stored in a file. However, the `prepare_data()` [1] method is not called and no input features are written. I fixed that adding this method to the `train_dataloader()` [2] function, but I'm not sure if it's the right place. 

Model training will work then.

Evaluation is currently not working correctly. The checkpoint output file name is:

```bash
# ls
'checkpointepoch=0.ckpt'  'checkpointepoch=1.ckpt'  'checkpointepoch=2.ckpt'
```

so the pattern `checkpointepoch=<number_epoch>.ckpt` is used, whereas the main script expects an output checkpoint pattern of `checkpoint_<number_epoch>.ckpt` [3]

[1] https://github.com/huggingface/transformers/blob/master/examples/ner/run_pl_ner.py#L56-L80
[2] https://github.com/huggingface/transformers/blob/master/examples/ner/transformer_base.py#L126-L139
[3] https://github.com/huggingface/transformers/blob/master/examples/ner/run_pl_ner.py#L220

Comments:
After fixing the these issues the output predictions seems very weird, maybe this is in an label 2 id bug?

```bash
Nachdem I-OTH
er I-PERpart
1907 B-ORGderiv
nicht I-PER
zum I-PER
Direktor I-PER
der I-PERderiv
Bibliothek B-ORGpart
berufen I-PER
wurde I-PER
, I-PER
verließ B-PERderiv
Gardthausen I-PERderiv
den I-ORG
Bibliotheksdienst B-ORGpart
. I-PERderiv
```

😂

cc @srush (via #3053)

Thanks I will take a look. Can you verify you are on the latest pytorch-lightning? `prepare_data` was just added. 

Also can you post your log. What was the Val accuracy? Are you on single GPU?

@srush Please see the PR #3180 I have updated the bash script and README to run effortlessly.  

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Xlnet, Alberta, Roberta are not finetuned for CoLA task
Author: SarikGhazarian
Pull request: no

## 🐛 Bug

I am currently trying to finetune pretrained models on CoLA task by using run_glue.py. Some of the models such as Bert and DistilBert are finetuned correctly as it is expected (the training loss goes down and the evaluation result is as what has been reported). Even though, for other models such as Roberta, Albert and Xlnet the training loss remains the same. I finetune the models for CoLA accordingly:

python run_glue.py --data_dir=./glue_data/CoLA/  --model_type=roberta 
--model_name_or_path=roberta-base --task_name=CoLA 
--output_dir=./model_roberta/  --max_seq_len=128 --do_train  --do_eval --num_train_epochs=3.0 --save_steps=50  --learning_rate=5e-5

## Observed behavior
(for Roberta-base)
"loss": 0.5889998215436936, "step": 50
"loss": 0.649243945479393, "step": 100
"loss": 0.6612952649593353, "step": 150
"loss": 0.6241107112169266, "step": 200
...
"loss":  0.6236384356021881, "step": 50
...
"loss": 0.6253059101104737, "step": 800
...
(I trained for more epochs (different lr )and still, the loss is near 0.5-0.6)
By debugging the code it seems that the model's output (softmax of logits) during training no matter what is the input is always label 1.

Another hint: I tried to finetune Roberta for other tasks such as STS-B and it finetuned well (loss was going down and the output of the model was not identical for all different inputs).

I was wondering if someone also has faced the same issue. How should I solve this?

OS type and version: Linux-3.10.0
Python: 3.7
Pytorch: 1.3.1


Comments:
NM, I was able to solve it by changing some of the hyperparameter values.

Hi, glad you could make it work! Do you mind sharing what hyperparameter values you tuned in order to make it work?

> Hi, glad you could make it work! Do you mind sharing what hyperparameter values you tuned in order to make it work?

Hi, have you solved the problem? I used bert-base-cased, roberta-base, and xlnet-base-cased to predict CoLA test.tsv, and I got 51.8, 55.6 and 24.7 respectively, I don't know why xlnet got such low Matthew's Corr. Can you help me? thx
run_squad with early stopping on a validation set
Author: yonatanbitton
Pull request: no

Hello. 
Is there a way to use run_squad with early stopping as a validation set? 

I have 3 files: train-v1.1.json, dev-v1.1.json, and test-v1.1.json.
I want to train on the train file, stop the training when the loss on the dev file starts to increase, and then do the final prediction and answers output on the test set. 

At Keras it's pretty straight forward: 
`history = model.fit(X_train, y_train, validation_data=(X_dev, y_dev), epochs=4000, verbose=0, callbacks=[EarlyStopping(monitor='dev_loss', mode='min'])`

How can I have a similar functionallity in huggingface's run_squad? I'm asking whether is there a built in way before implementing it from scratch. 

I thought of changing the code in the point of: 
`--evaluate_during_training` - it activates the evaluation during training - I can change the fie to the dev-v1.1.json file in this point, and stop the training if the validation loss increases. 

I would like to know if there is a better way. 

Thanks

Comments:
Hi, this would be something you would need to build yourself.

Alternatively you could use our Layers as Keras Layers?
Issue with lr during training from scratch
Author: bokertof
Pull request: no

I'm trying to resume my pretraining of RoBERTa model and I noticed that everything is OK except learning rate. I thought that it should continue to decrease from last checkpoint (lr = 7e-7) but lr is much bigger (lr = 5e-5) than it has to be.

this is my parameters to run script:

- 	--model_name_or_path ./ROBERTA-small-v1/checkpoint-30000
-   --train_data_file TEXT.txt
-   --output_dir ./ROBERTA-small-v2              
- 	--mlm
- 	--config_name ./ROBERTA_SMILES-v1/checkpoint-30000
- 	--tokenizer_name ./ROBERTA
- 	--do_train
- 	--line_by_line
- 	--num_train_epochs 10
- 	--save_total_limit 1
- 	--save_steps 2000
- 	--per_gpu_train_batch_size 16
- 	--seed 42
- 	--logging_steps 250


Comments:
Are you restarting from your checkpoint with the exact same set of script parameters?

@julien-c  Actually, no. And I guess I undestood what the reason of that behavior is. Originally I used to train with number of epochs = 5. After that I selected 10 epochs and hoped that it'll continue pretraining from 5'th epoch up to 10'th. However as far as I understand lr-value is calculated based on the number of epochs. The current value of lr is the same as I got at 15k iteration (30k/2). So I have should selected 10 (or whatever number) epochs originally

Yes. You can always implement this behaviour yourself if you need it.
max_qa_length is needed for funetune on multiple-choice problems
Author: MagicFrogSJTU
Pull request: no

# 🚀 Feature request
In examples/multiple-choice/run_multiple_choice.py, the max_qa_length is not implemented as that in google/Albert, which would cause trouble when funetuning RACE.
This should be easy becauce there are similar codes in codes for run_squad.py.
### Your contribution
I have done the script and get similar RACE performance compared to google/Albert.

# Current Bug
In examples/multiple-choice/utils_multiple_choice.py
```python
# From 537 lines: 
            # It should be => 
            # inputs = tokenizer.encode_plus(
            #    text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True, return_overflowing_tokens=True
            # )
            # Or the log info below would never be activated.
            inputs = tokenizer.encode_plus(
                text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True,
            )
            if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                logger.info(
                    "Attention! you are cropping tokens (swag task is ok). "
                    "If you are training ARC and RACE and you are poping question + options,"
                    "you need to try to use a bigger max seq length!"
                )
```


Comments:
Indeed, you're totally correct. Thank you for reporting the issue, I'm fixing it in #4385.
Allow for None gradients in GradientAccumulator.
Author: jarednielsen
Pull request: yes

The [TFTrainer commit](https://github.com/huggingface/transformers/commit/aad50151f35b934039c455242c433a24f011cc93#diff-2a765415189cf1edda4414e1758b0e79) seems to have reverted the ability for gradients to be None in GradientAccumulator. This behavior is necessary when pretraining a model, such as AlbertForPreTraining, and using only one loss (only SOP or only MLM).

The [OpenNMT repository](https://github.com/OpenNMT/OpenNMT-tf/blob/master/opennmt/optimizers/utils.py) the code was taken from lacks this ability, but it's important. This commit restores the ability for gradients to be None.

Comments:
[Bart/Marian] ignore output_attentions when invoked through AutoModel
Author: sshleifer
Pull request: no
Labels: name: marian

[ ] should mimic behavior of other models
[ ] backwards compat?

Comments:
this works for me:
```python
modelname='Helsinki-NLP/opus-mt-en-de'
model = MarianMTModel.from_pretrained(modelname,output_attentions=True, output_hidden_states=True)
output_tuple = model(**model.dummy_inputs)
assert len(output_tuple) == 6
```

#4388 is the real issue.
DistilGPT2 Finetuning with GPU
Author: orientino
Pull request: no

Hi, i'm a beginner in this field and would like to finetune the model DistilGPT2 using my laptop. I have an Nvidia 1660Ti and followed the guide example 

```
export TRAIN_FILE=/path/to/dataset/wiki.train.raw
export TEST_FILE=/path/to/dataset/wiki.test.raw

python run_language_modeling.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE
```

But during the finetuning my GPU stays at 0% and all the work is done by the CPU, is there a way to force the GPU to finetune?

Comments:
It should be automatic.

What does torch.cuda.is_available say?

torch.cuda.is_available return False.
This is what my setup is by running nvidia-smi
```
Thu May 14 15:04:38 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 431.87       Driver Version: 431.87       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 166... WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   57C    P8     3W /  N/A |    153MiB /  6144MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```

Yes just checked. torch is not able to find cuda

You should upgrade your nvidia driver.

You probably installed pytorch 1.5.0 bumped up the required version of nvidia-driver.

yes.
`pip install torch==1.4.0`
will solve the issue.
Knowing the specific data set used for DistilBertForQuestionAnswering
Author: valldabo2
Pull request: no
Labels: name: wontfix

name: Ex: Question Answering

name: Distillation

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi,
I am using the pipeline for question answering and am wondering what dataset was used to train the underlying model. The loaded model is DistilBertForQuestionAnswering. But I need to know if it used Squad 1.1 or Squad 2.0 which includes the possibility of no answer?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Performance Issue about pretrained bert migration from tensorflow to pytorch
Author: eagle705
Pull request: no
Labels: name: wontfix

name: Migration

# ❓ Questions & Help

Hi, 
I have a question about migration from tensorflow to pytorch.
if I change pretrained BERT with google tensorflow 1.x version to pytorch transformers BERT through transformers-cli, performance drops a lot. I wonder why.

Is there any similar case?

Comments:
Are you using the latest version? `transformers` and not `pytorch-transformers`. It works with both TF2 and PT. You can test both and compare.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Unknown task fill-mask
Author: DesiKeki
Pull request: no

# 🐛 Bug

## Information

I am trying to create a "Pipeline" for fill-mask task

Language I am using the model on is English

The problem arises when using:
My own modified scripts: (give details below)
```
import transformers
transformers.__version__
```
>>'2.3.0'
```
from transformers import pipeline
nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in <mask>')
```
>>---------------------------------------------------------------------------
>>KeyError                                  Traceback (most recent call last)
>><ipython-input-22-d415d13f7528> in <module>
>>----> 1 nlp_fill = pipeline('fill-mask')
>>     2 nlp_fill('Hugging Face is a French company based in <mask>')
>>
>>~\Anaconda3\lib\site-packages\transformers\pipelines.py in pipeline(task, model, config, tokenizer, modelcard, **kwargs)
>>    848     # Retrieve the task
>>    849     if task not in SUPPORTED_TASKS:
>>--> 850         raise KeyError("Unknown task {}, available tasks are {}".format(task, list(SUPPORTED_TASKS.keys())))
>>    851 
>>    852     framework = get_framework(model)
>> **KeyError**: "Unknown task fill-mask, available tasks are ['feature-extraction', 'sentiment-analysis', 'ner', 'question-answering']"

The tasks I am working on is:
My own task or dataset. Trying using Pipeline.

## To reproduce

Steps to reproduce the behavior:

from transformers import pipeline
nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in <mask>')



## Expected behavior
Should have got a filled output with a score.

## Environment info
'transformers-cli' is not recognized as an internal or external command, operable program or batch file.
     
- `transformers` version: 2.3.0
- Platform: Windows Jupyter notebook
- Python version: 3.7.4



Comments:
Can you upgrade your version of transformers?

Same issue I am not able to run the command `transformer-cli env` even after installing from source @julien-c .

`transformers-cli is not recognized as an internal or external command operable program or batch file` 

Just wondering if this is due to the fact of using `conda` as in latest versions it is not preferring to modify the PYTHONPATH. hence the non visibility of the command?



Upgrading the transformers version to 2.9.1 works!