noamaanMulla-03/transformers-issues huggingface dataset:
Gradient overflow issue when i try to train gpt2 with run_language_modeling in fp16 with 02. Any idea why that maybe happen?
Author: Nkonstan
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
![Screenshot from 2020-05-31 13-49-05](https://user-images.githubusercontent.com/35643708/83350909-172cb480-a348-11ea-97a5-d8745626e774.png)
I get this message every 2000 steps. 
"Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 262144.0"

The train run on rtx 2070 and seems to not have other problem but that message keeps going on every 2000 steps.

pytorch 1.5.0
python 3.7
cuda 10.1


Do you have the same issue with opt level O1? Using O2 is discouraged. The issue may also be related to PyTorch 1.5, so if switching to O1 does not help, try a previous PyTorch version. Note that it is highly likely that this is an AMP problem, not a transformers issue. Have a look here https://github.com/NVIDIA/apex/issues/318

@BramVanroy  Yes i guess you 're right. It seems problem of the AMP. I asked and in APEX the same question and their answer was : 

"The loss scaler tries to increase the loss scaling factor after a threshold of successful steps was reached. In your case it seems that the scaling factor is being downgraded to the same value, so it should be fine."

So, according to that answer, is not a problem. But still i am not sure. 



I guess that skipping one step every 2000 steps is not a problem. You can monitor the loss, and as long as it seems to decrease normally, then you should be fine. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is it possible to fine-tune models on TPUs using TensorFlow?
Author: SebiSebi
Pull request: no
Labels: name: wontfix

I have looked at the release notes and found out that:

"Training on TPU using free TPUs provided in the TensorFlow Research Cloud (TFRC) program is possible but requires to implement a custom training loop (not possible with keras.fit at the moment).
We will add an example of such a custom training loop soon." (Note from September 26).

Is this observation still true? Can we train the transformer models on TPUs in TF?

Comments:
following, would love to know if this is possible

We have some code in the `tpu-experiment` branch, for instance here: https://github.com/huggingface/transformers/tree/tpu-experiments/examples/TPU/tensorflow

And planning to make it clean in the mid-term (not sure that will be before the end of the year though).

cc @LysandreJik 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any updates on this one?

Same here

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Same here 

bump! Would love this! If you guys have your hands full, let me know if I can help in anyway :)

Hi! We recently have updated all of our scripts with `Trainer` classes, for both TensorFlow and PyTorch. Both trainers now have TPU support!

The [examples README](https://github.com/huggingface/transformers/tree/master/examples#running-on-tpus) has been updated accordingly.

That's great! Will try it out and report! 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
SpanBert always predicts the same token
Author: Oxi84
Pull request: no
Labels: name: wontfix

I tried to use this implementation - https://huggingface.co/SpanBERT/spanbert-base-cased, but as a prediction I always get the same exect outoput no matter where I put [MASK] in a sentence. Here is the code.  Am I doing something wrong?

    import torch
    from transformers import AutoTokenizer,AutoModel
    tokenizer = AutoTokenizer.from_pretrained("SpanBERT/spanbert-base-cased") 
    model = AutoModel.from_pretrained("SpanBERT/spanbert-base-cased")
    model.eval()
    model.to('cuda')

    text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
    tokenized_text = tokenizer.tokenize(text)
    print("tokenized_text",tokenized_text)

    themask = 2
    tokenized_text[themask] = '[MASK]'

    indexes = tokenizer.convert_tokens_to_ids(tokenized_text)
    indexes_tensor = torch.tensor([indexes])
    indexes_tensor = indexes_tensor.to('cuda')


    with torch.no_grad():
        outputs = model(indexes_tensor)
        predictions0 = outputs[0]


    the_index = torch.argmax(predictions0[0, themask]).item()
    theresult = tokenizer.convert_ids_to_tokens([the_index])[0]
    print("theresult",theresult)
    print("the_index",the_index)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is the separation token absolutely necessary if I use GPT2DoubleHeadsModel with token_type_ids?
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

I am trying to use the GPT2DoubleHeadsModel to process the multiple choice questions.
For the pre-processing of the multiple choice questions, I didn't add any special separating token between the multiple choice question and the multiple choice option. Instead, I generated the token_type_ids, which denote 0 for the question portion of the text, and 1 for the multiple choice option. Then I tried to make the GPT2DoubleHeadsModel to predict the correct answer by doing:

```python
gpt2DoubleHeadsModel(input_ids=input_ids, token_type_ids = token_type_ids)
```

Is this practice acceptable? or do I absolutely need to insert a special separation token between the question text and the multiple choice option text?

Thank you,  


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Converting model to pytorch
Author: rlpatrao
Pull request: no
Labels: name: wontfix

# 🐛 Bug
Folks, I am trying to convert the Biobert model to Pytorch. Here are the things that I did so far:

**1. For the vocab:**  I am trying to convert the vocab using solution from #69 : 
```tokenizer = BartTokenizer.from_pretrained('/content/biobert_v1.1_pubmed/vocab.txt')```

I get :
`OSError: Model name '/content/biobert_v1.1_pubmed' was not found in tokenizers model name list (bart-large, bart-large-mnli, bart-large-cnn, bart-large-xsum). We assumed '/content/biobert_v1.1_pubmed' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.`
	
I don’t have the vocab.json, so I how do I convert the vocab for the tokenizer ? 
	
**2. For the model:** As the out of the box `pytorch_pretrained_bert.convert_tf_checkpoint_to_pytorch` did not work I customized it per #2 by adding: 

```
excluded = ['BERTAdam','_power','global_step']
init_vars = list(filter(lambda x:all([True if e not in x[0] else False for e in excluded]),init_vars))

```
With this the model 'seems' to be converting fine.  But When I load this using:

`model = BartForConditionalGeneration.from_pretrained('path/to/model/biobert_v1.1_pubmed_pytorch.model') `

I still get 

`UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
`

Can you pl. help me to understand what is going on here ? 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[s2s] tiny QOL improvement: run_eval prints scores
Author: sshleifer
Pull request: yes

its annoying to have to cat a file to see the scores after calling run_eval.py

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6341?src=pr&el=h1) Report
> Merging [#6341](https://codecov.io/gh/huggingface/transformers/pull/6341?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/322dffc6c9a44fd504b24b0efcbcaa419b577a93&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6341/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6341?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6341      +/-   ##
==========================================
+ Coverage   78.37%   78.51%   +0.13%     
==========================================
  Files         148      148              
  Lines       27196    27196              
==========================================
+ Hits        21316    21354      +38     
+ Misses       5880     5842      -38     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6341?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6341/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.53% <0.00%> (-63.20%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6341/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.71% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6341/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `97.41% <0.00%> (+32.94%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6341?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6341?src=pr&el=footer). Last update [322dffc...37aad56](https://codecov.io/gh/huggingface/transformers/pull/6341?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Model parallelism support?
Author: orenmelamud
Pull request: no
Labels: name: wontfix

Hi. I'm running into memory issues when trying to fine tune the CTRL language model on a 16 GB GPU. Is there any built-in support for splitting models across more than one GPU? I suppose that mapping the input embedding layer to a different GPU or even the CPU would do the trick.


Comments:
How do you training CTRL ? Did you check this: https://github.com/huggingface/transformers/blob/0477b307c7501ea76e01b03cb387a2312db752b3/examples/run_lm_finetuning.py#L198

You can also decrease batch size up to 1.

And also gradients can be accumulated:
https://github.com/huggingface/transformers/blob/0477b307c7501ea76e01b03cb387a2312db752b3/examples/run_lm_finetuning.py#L389

@iedmrc Thanks for the suggestions. I'm using a slightly modified version of run_lm_finetuning.py but for CTRL even with a batch size of 1 the fine tuning consumes more than 16 GB of memory. I tried also to fine tune with half precision (fp16), but for some reason that didn't help either (I have seen other people complain about similar issues here).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


See https://github.com/huggingface/transformers/pull/3578.
Is it necessary to provide attention_mask, or model will calculate itself?
Author: saahiluppal
Pull request: no

Is it necessary to provide attention_mask, or model will calculate itself?

Comments:
If you do not pass an `attention_mask` then the `attention_mask` will automatically set to all ones (`[1, 1, 1, ...,]`). Meanning that every token is attended to (no token is masked). Also check out the docs here: https://huggingface.co/transformers/glossary.html#attention-mask.

If your `input_ids` contain <PAD> tokens, then the `attention_mask` will not automatically be calculated. You can leverage the tokenizers though to automatically retrieve the correct `attention_mask`.
Documentation bug in GPT2Config
Author: miguelvictor
Pull request: no

In [this page](https://huggingface.co/transformers/model_doc/gpt2.html), GPT2Config's initializer_range default value is 16. But the source code [says](https://github.com/huggingface/transformers/blob/master/src/transformers/configuration_gpt2.py#L130) it is 0.02.

Comments:
Good catch!

#6352
Save & load sparse models from the models database
Author: borisdayma
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Handle saving and loading pipeline for sparse models such as PruneBERT.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

New sparse models will be extremely useful if we can use them by downloading the compressed. version with the `.from_pretrained` functions

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

An example is already part of the current repo so I can try to create a PR later: https://github.com/huggingface/transformers/blob/master/examples/movement-pruning/Saving_PruneBERT.ipynb


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
tensorflow2_gpt2 Slow speed
Author: only-yao
Pull request: no
Labels: name: wontfix

# ❓ The speed of pytorch_gpt2 based on transformers is 4-5 times faster than that of tensorflow_gpt2, which is unreasonable. Where is my problem

**pytorch_gpt2**

```
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

model.to(dev)
raw_text_2 = "And if so be ye can descrive what ye bear,"

inputs = tokenizer.encode(raw_text_2, add_special_tokens=False, return_tensors="pt")
inputs = inputs.to(dev)
generated = model.generate(inputs, top_k=0, max_length=512)

```

**tensorflow2_gpt2**
```

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = TFGPT2LMHeadModel.from_pretrained('gpt2')

raw_text = "And if so be ye can descrive what ye bear,"

tokens = tokenizer.encode(raw_text,  return_tensors="tf")
output_ids = model.generate(tokens, top_k = 0, max_length=512)

```


Comments:
With tf. Varibable model for packaging
```
    @tf.function(experimental_relax_shapes=True)
    def model_static(self, model_inputs):
        outputs = self(**model_inputs)
        return outputs

```

Same problem here
@only-yao Did you find a solution yet?

> @ only-yao这是同样的问题，您找到解决方案了吗？

Wrap the model with @tf.function，the model as a static diagram

Very interesting! Thanks for your code snippet @only-yao - I will take a closer look in a week or so :-) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
resize_token_embeddings not implemented for TFGPT2LMHeadModel
Author: virginianegri
Pull request: no
Labels: name: wontfix

I am using a TFGPT2LMHeadModel pretrained model and new special tokens to the gpt tokenizer.
However, the method resize_token_embeddings is not implemented in all gpt2 tf models.
Will it be added? Or are there any workarounds?

Thank you!

Comments:
Hi @virginianegri, 

Could you specific for which model the method `resize_token_embeddings` does not work? 
Can you add a code snippet that reproduces the error?

The model is the TFGPT2LMHeadModel. This is my code:
```
from transformers import GPT2Tokenizer, TFGPT2LMHeadModel 

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
special_tokens_dict = {'cls_token': '<CLS>'}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)

gpt2 = TFGPT2LMHeadModel.from_pretrained('gpt2')
gpt2.resize_token_embeddings(len(tokenizer)) 
```

When running the resize_token_embeddings method it launches a NotImplementedError

Yes we should implement this soon!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is there any need to fine-tune the already pre-trained GPT-2 models?
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

If I am using the pre-trained GPT-2 for my research, should I still fine-tune the already pre-trained models with my dataset? I am a bit confused by the term "pre-trained".

Thank you,

Comments:
"Finetuning" a GPT-2 model is generally used for generation, to make it output text more similar to a given dataset.

If you are using it for other tasks (e.g. binary predictions), you may not have to finetune it.

Hello,

Thank you for your reply.
So say if I am trying to solve multiple choice questions with GPT2DoubleHeadsModel, should I just use the pre-trained model without fine-tuning?

Thanks, 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can I use TorchText Iterator output as the input_ids for Hugging Face Transformer?
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

For a given text, `TorchText` iterator, such as the `BPTTIterator`, returns the string of the text after each token is converted to their respective integer ID.

So for example, if the integer ids are assigned as the following manner:
"I" = 53, "like"=753, "dogs" = 2

Then for the string "I like dogs", a `TorchText` iterator would return `[53, 753, 2]`.

Is it okay to use this type of TorchText Iterator output directly as an `input_ids` for Hugging Face Transformers, providing that the Transformer models I use is not the Hugging Face pre-trained ones?

Thank you,


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
CUDA Out of Memory
Author: mc2259
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     https://stackoverflow.com/questions/63335442/how-do-i-deal-with-cuda-out-of-memory-while-finetuning-bart
     -->

## Details
<!-- Description of your issue -->
I was trying to finetune BART on google collab using the xsum dataset and the finetuning script and I got this:
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.73 GiB total capacity; 13.67 GiB already allocated; 15.88 MiB free; 13.72 GiB reserved in total by PyTorch)
Does this mean I have to use a smaller model?

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:
https://stackoverflow.com/questions/63335442/how-do-i-deal-with-cuda-out-of-memory-while-finetuning-bart

Comments:
Yes it seems like the GPU that was allocated to you does not provide enough GPU memory for the model
hi
Author: xinghao302001
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 tensorflow: @jplu 
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Colab button
Author: sgugger
Pull request: yes

This PR adds a "open in colab" button on the tutorials of our documentation. For each of those tutorials, three notebooks are available: mixed version (with cells PyTorch and TensorFlow), PyTorch-only and TensorFlow only, so hovering on the button makes a dropdown appear with the three different links.

Those notebooks are generated automatically from the docs rst files and the script in the [notebooks repo](https://github.com/huggingface/notebooks/blob/master/utils/convert_doc_to_notebooks.py).

Comments: