noamaanMulla-03/transformers-issues huggingface dataset:
Massive text generation slowdown when using repetition_penalty param on GPU
Author: minimaxir
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Text generation when using the `repetition_penalty` takes about 2x-10x longer on a GPU, which is disproportionate and implies the GPU may not be used in that instance.

Reported from https://github.com/minimaxir/aitextgen/issues/34

The `enforce_repetition_penalty_` function at 

https://github.com/huggingface/transformers/blob/08c9607c3d025f9f1a0c40e6d124d5d5d446208e/src/transformers/modeling_utils.py#L817

may be using CPU ops instead of Tensor ops.

## To reproduce

Demo Colab notebook w/ time benchmarks: https://colab.research.google.com/drive/1SzYUEC0xikHN8OEp2WOTWHlT6JxT9rRf?usp=sharing




Comments:
Thanks for the notebook - I can reproduce the results both on a local GPU and on colab. You are right, this slowdown is disproportionate! 
I'm suspecting the double for loop to be the reason for the heavy slow down. It might be possible to replace the two for loops by some smart matrix operations, but not sure.

Also pinging our PyTorch GPU master @mfuntowicz - do you maybe have some insight here?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to visualize the output of the encoder using T-sne plots? 
Author: jlim13
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
I was thinking of just mean or max pooling (along the sequence length dimension) the output of the encoder and visualizing that, but I was wondering if there were better ways of doing so. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
❓ Why multiplying the output of T5 by some scalar before LM head ?
Author: astariul
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I'm wondering why multiplying the outputs of T5 by some scalar before inputting in the LM head :

https://github.com/huggingface/transformers/blob/1d2332861f5225aef17bd7e75abc670a72239081/src/transformers/modeling_tf_t5.py#L1147



Comments:
I think this would be a great question for our new forum, with a title like "T5 architecture": https://discuss.huggingface.co/ - would you mind posting it there? This seems like an interesting question people would probably like to see there.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fail to run trainer.train() with huggingface transformer
Author: wjzhan
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- I am trying to set up a TensorFlow fine-tune framework for a question-answering project. Using hugging-face/transformer as the prototype, but cannot run through the trainer.

The experiment is conducted at Databricks, the pre-trained model loaded is base-bert, train and dev sets are downloaded from hugging-face examples SQUAD 2.0 https://github.com/huggingface/transformers/tree/master/examples/question-answering

The error log complains about the unexpected keyword argument 'is_impossible', which is a SQUAD 2 data format feature. -->

<!-- Here is the link to my question on stackoverflow -->
**https://stackoverflow.com/questions/62879960/fail-to-run-trainer-train-with-huggingface-transformer**:

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to use (and preferably finetune) BART for text infilling? 
Author: tomaszgarbus
Pull request: no
Labels: name: Help wanted

name: wontfix

name: Examples

[Here](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration) is shown how to use BART for simple mask filling (one <mask> token = one generated token), but how to use it for text infilling? The BART paper states that the model was pretrained on such task so it should be possible.

Is the only solution to simply take the `facebook/bart-large` model for summarization and finetune it on a dataset with <mask> tokens or is there a better way?


Comments:
@julien-c , @sshleifer ?

Sorry for the slow response.
Unfortunately, text infilling is not yet supported. It would be a welcome contribution! I think the equivalent fairseq task is called `DenoisingTask` 
https://github.com/pytorch/fairseq/blob/aa79bb9c37b27e3f84e7a4e182175d3b50a79041/fairseq/tasks/denoising.py#L27

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BART/T5 eli5 in model hub
Author: clmnt
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Would love to have https://huggingface.co/qa in the model hub mostly for the inference widget/api (the demo is down quite regularly)

## Motivation

A lot of companies might want to test it/use it!

## Your contribution

🔥 emojis on slack

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BART MNLI + yahoo answer in the model hub for inference API
Author: clmnt
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

would love to be able to use https://huggingface.co/zero-shot/ with the inference API

## Motivation

it would help many companies run zero-shot classification of long documents

## Your contribution

I can provide 🔥 emojis on slack


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Longformer output_hidden_states=True outputs sequence length=512 for all inputs of different lengths
Author: xuewyang
Pull request: no

Can someone explain why cc[2][0].shape != dd[2][0].shape?

>>> from transformers import LongformerModel, RobertaModel
>>> import torch
>>> from transformers import RobertaTokenizer
>>> tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
>>> aa = RobertaModel.from_pretrained('roberta-base', return_dict=True)
>>> bb = LongformerModel.from_pretrained('allenai/longformer-large-4096', return_dict=True)
Some weights of LongformerModel were not initialized from the model checkpoint at allenai/longformer-large-4096 and are newly initialized: ['longformer.embeddings.position_ids']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> cc = aa(**inputs, output_hidden_states=True)
>>> cc[2][0].shape
torch.Size([1, 8, 768])
>>> dd = bb(**inputs, output_hidden_states=True)
>>> dd[2][0].shape
torch.Size([1, 512, 1024])
>>> dd[0].shape
torch.Size([1, 8, 1024])


Comments:
When I use longformer-base-4096, it works well.

The reason is that `LongformerModel` pads its input to the `window_size` configuration param, which is 512 for `allenai/longformer-large-4096`. Therefore all `hidden_states` are of length 512. We could think about cutting the hidden states to the actual inputt length...

Gotcha. But I also tested with longformer-base. It works fine with hidden states cut. Thank you.
Fine-tune BERT for regression problem
Author: rxlian
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
I was wondering in order to fine-tune BERT for regression problem, do I just need to put num_labels = 1 inside the BertForSequenceClassification function? Does there anything else need to modify? I'm a newbie to transformers and still confused with basic things. Thanks in advance.
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

Comments:
Hey @rxlian, 

We are trying to move more general and "researchy" questions to our discussion forum here: https://discuss.huggingface.co/ and use github issues mainly for bugs. 

Would you mind posting your questions at the forum - other people might be interested as well :-)  

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
google/reformer-enwik8 tokenizer was not found in tokenizers model name list
Author: pzelasko
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Reformer

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Enter https://huggingface.co/google/reformer-enwik8
2. Look at "Hosted inference API"

The model's tokenizer cannot be found; I'm getting the same error in scripts as the one displayed on your webpage:

```⚠️ This model could not be loaded by the inference API. ⚠️
Error loading tokenizer Model name 'google/reformer-enwik8' was not found in tokenizers model name list (google/reformer-crime-and-punishment). We assumed 'google/reformer-enwik8' was a path, a model identifier, or url to a directory containing vocabulary files named ['spiece.model'] but couldn't find such vocabulary files at this path or url. OSError("Model name 'google/reformer-enwik8' was not found in tokenizers model name list (google/reformer-crime-and-punishment). We assumed 'google/reformer-enwik8' was a path, a model identifier, or url to a directory containing vocabulary files named ['spiece.model'] but couldn't find such vocabulary files at this path or url.")
```

## Expected behavior

Tokenizer loaded without issues

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: latest
- Platform: your own
- Python version: ?
- PyTorch version (GPU?): ?
- Tensorflow version (GPU?): ?
- Using GPU in script?: ?
- Using distributed or parallel set-up in script?: ?


Comments:
That's because only the crime and punishment modell has an uploaded tokenizer.


`google/reformer-enwik8` is the only model that is a char language model and does not need a tokenizer. If you take a look here: https://huggingface.co/google/reformer-enwik8#reformer-language-model-on-character-level-and-trained-on-enwik8 , you can see that the model does not need a tokenier but a simple python encode and decode function. 

@julien-c @mfuntowicz  - how do you think we can include char lms to `pipelines`? Should we maybe introduce a `is_char_lm` config variable? Or just wrap a dummy tokenizer around the python encode and decode functions?

Add a `tokenizer_class` optional attribute to config.json which overrides the type of Tokenizer that's instantiated when calling `.from_pretrained()`?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
InvalidArgumentError: Cannot convert a Tensor of dtype resource to a NumPy array.
Author: Douboo
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert):


The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python
def build_model(item_dim, num_layers, num_heads, max_len):
    config = BertConfig(hidden_size=item_dim, num_hidden_layers=num_layers,
                        num_attention_heads=num_heads, intermediate_size=item_dim*4,
                        max_position_embeddings=max_len)
    bert = TFBertMainLayer(config=config)
    inputs = Input(shape=(max_len, item_dim), dtype=tf.float32, name='inputs')
    # pre-training vectors to bert
    seq_emb = bert(inputs=None, inputs_embeds=inputs)[0]
    print(seq_emb)
    print(seq_emb[:, -1, :])
    last_token_emb = seq_emb[:, -1, :]
    outputs = Dense(1, activation='sigmoid')(last_token_emb)
    model = Model(inputs=inputs, outputs=outputs)
    return model
```

Errors with
`InvalidArgumentError: Cannot convert a Tensor of dtype resource to a NumPy array.`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
* Run the code successfully when feed pre-training vectors to bert.
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
https://colab.research.google.com/gist/Douboo/adf6e136e45f8406b1070d88f4041a49/untitled2.ipynb
     
- `transformers` version: 3.0.2
- Platform: google colab
- Python version: 3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.2.0
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add "Fine-tune ALBERT for sentence-pair classification" notebook to the community notebooks
Author: NadirEM
Pull request: yes

Hello,

I'm adding to the community notebooks a tutorial on fine-tuning ALBERT and other BERT-based models for sentence-pair classification.
The main features of this tutorial are :
[1] End-to-end ML implementation (training, validation, prediction, evaluation)
[2] Easy adaptability to your own datasets
[3] Facilitation of quick experiments with other BERT-based models (BERT, ALBERT, ...)
[4] Quick training with limited computational resources (mixed-precision, gradient accumulation, ...)
[5] Multi-GPU execution
[6] Threshold choice for the classification decision (not necessarily 0.5)
[7] Freeze BERT layers and only update the classification layer weights or update all the weights
[8] Reproducible results with seed settings


Comments:
Some weights of AlbertModel were not initialized ['albert.embeddings.position_ids'] 
Author: vgaraujov
Pull request: no

Hello!

There seems to be a problem with the current code to load a pre-trained Albert model. This warning appears in any configuration of the Albert model:
`Some weights of AlbertModel were not initialized from the model checkpoint at albert-base-v2 and are newly initialized: ['albert.embeddings.position_ids']`
`You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.`

I found this happens only when I install it from the source. Models load correctly (without warning) when installing the library with pip.

Comments:
Hello! Which checkpoint are you trying to load? Is it one of your checkpoints or is it one of the checkpoints hosted on the modelhub?

This is probably because of this PR: https://github.com/huggingface/transformers/pull/5773 , but should not pose a real problem. I guess we just have to add the position ids as "allowed" non-initialized weights.

@LysandreJik I used model hub checkpoints. I runned the following lines with pytorch:

`from transformers import AlbertForPreTraining`
`model = AlbertForPreTraining.from_pretrained('albert-base-v2')`


Hey @vgaraujov, I can reproduce - this PR: #6700 should fix it. Thanks for reporting it :-) 

@patrickvonplaten Thank you for your help! 💯 

Position_ids seems unnecessary to be saved? Why not use register_buffer with persistent=False

> Position_ids seems unnecessary to be saved? Why not use register_buffer with persistent=False

It's a fantastic suggestion, @ruotianluo!

But, alas, It can't be used at the moment, since:
1. this functionality  [was added just a few months ago](https://github.com/pytorch/pytorch/pull/37191) (can't require recent `torch`)
2. [it doesn't yet work with torchscript](https://github.com/pytorch/pytorch/issues/45012)

See the following solution: https://github.com/huggingface/transformers/pull/7224
[example/glue] run_glue compute metrics fail for bart like models
Author: patil-suraj
Pull request: no

This PR #7126 introduced multiple predictions for trainer. This breaks the `compute_metrics_fn` of `run_glue.py`  for `bart` like models which return multiple predictions.

For `BartForSequenceClassfication` `p.predictions`  is a `tuple`, so following code fails
https://github.com/huggingface/transformers/blob/1d90d0f386af2af52017d51c421e71a51ec94de0/examples/text-classification/run_glue.py#L154

@sgugger 


Comments:
I didn't expect that side effect, thanks for fixing it so quickly!
weights partially missing for CamembertForMaskedLM 
Author: raphael0202
Pull request: no

## Environment info

- `transformers` version: 3.1.0
- Platform: Darwin-18.7.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
 
@louismartin 

## Information

When loading "camembert-base" with `CamembertForMaskedLM` with:

    from transformers import CamembertForMaskedLM
    model = CamembertForMaskedLM.from_pretrained("camembert-base")

the bias of the LM head decoder is not loaded:

    Some weights of CamembertForMaskedLM were not initialized from the model checkpoint at camembert-base and are newly initialized: ['lm_head.decoder.bias']
    You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

As I understand `lm_head.decoder.bias` is therefore initialized randomly.
I checked the original `camembert-base` model as published by the author, and the lm_head decoder bias is missing too, which is not discussed in the camembert or roberta publication.

Comments:
Hi @raphael0202 , we converted the fairseq checkpoints using [this script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py), maybe the language model head was not transfered during the conversion. 
You can either use the model in fairseq as per [our website](https://camembert-model.fr/) or try to understand if and why the LM head hasn't transfered from fairseq to transformers.

Check this tutorial as well for huggingface: https://huggingface.co/camembert-base

After a bit of search I found out the warning comes from the way the decoder bias is defined in the code, the same warning is issued when loading the roberta model (see issue 6193). I checked that the LM head decoder bias is the same as the one in the original pytorch checkpoint.
Thank you for your help!

Ok perfect, thanks for fixing this.
Loading fine tuned BertForMaskedLM
Author: tgriseau
Pull request: no

Hi, 

I tried to fine tune BertForMaskedLM and it works. But i'm facing issues when I try to load the fine tuned model. 
Here is the code I used to load the model : 

```
model_state_dict = torch.load("./finetunedmodel/pytorch_model.bin",  map_location='cpu')
model_fine = BertForMaskedLM.from_pretrained(pretrained_model_name='bert-base-multilingual-cased', state_dict=model_state_dict, cache_dir='./data')
```

The error I'm facing is : __init__() got an unexpected keyword argument 'state_dict' 

Does someone already faced this issue ? 

Thanks

Edit : I trained my model on gpu and try to use it on cpu. When I use it on gpu it works !

Comments:
Maybe update to a recent version of `pytorch-pretrained-bert`?

I'm already using the last release.

I don't have any issues running it on gpu. The problem append when using map_location


yeah yes. if you trained model in GPU, can't be loaded. we will change map_location="CPU" in modeling.py line 511. 
https://github.com/huggingface/pytorch-pretrained-BERT/blob/0a9d7c7edb20a3e82cfbb4b72515575543784823/pytorch_pretrained_bert/modeling.py#L511

`state_dict = torch.load(weights_path,  map_location="CPU" )`

now it will load our finetuned model. this is only for inference running on CPU. 

All good ! It works, thanks. 

Maybe adding a device parameter to the function from_pretrained could be usefull.

Thanks for your help.

Great, thanks @MuruganR96 

@tgriseau   - I want to fine-tune bert on MaskedLM using domain-specific text. could you please provide an example of how you fine-tuned or provide some details about what kind of inputs need to be passed? will I be using the true sentence as the output for fine-tuning?
Why does examples/seq2seq/finetune.py only use sortish sampler for train? 
Author: sshleifer
Pull request: no
Labels: name: seq2seq

name: Examples

Would it make eval faster?
Would it make validation sanity check obnoxiously slow?
This should be a ~1 line PR with a longer PR description.



Comments:
@sshleifer I can help out here. What exactly are you looking for?

I'll take care of it actually, sorry! There is a harder one at #6785 if you're interested.

Fixed, using sortish sampler for val. Much faster!