noamaanMulla-03/transformers-issues huggingface dataset:
Why I can't generate phrases in batches if I include an attention mask? (GPT2)
Author: Barbara931120
Pull request: no

Assuming these are my input phrases and model:

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<PAD>')

prompt_text = [
    "are there any good coaching institutes for civil services preparations in bangalore? ->"]
```

If I try to generate phrases in batches with the corresponding attention mask it doesn't work. It outputs the input phrase without any new words on it:

```
# encode plus batch handles multiple batches and automatically creates attention_masks
seq_len = 100
encodings_dict = tokenizer.batch_encode_plus(prompt_text, max_length=seq_len, pad_to_max_length=True)

input_ids = torch.tensor(encodings_dict['input_ids'])
attn_mask = torch.tensor(encodings_dict['attention_mask'])

encoded_result = model.generate(input_ids, attention_mask=attn_mask, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.eos_token_id, num_return_sequences=10, top_k=50, top_p=0.95, do_sample=True, max_length=100)

for er in encoded_result:
    print(tokenizer.decode(er, skip_special_tokens=True))
```

However, if I generate phrases one by one (without batches) then it works:

```
encoded_text = tokenizer.encode(prompt_text[0], return_tensors='pt')
encoded_result = model.generate(encoded_text,eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.eos_token_id, num_return_sequences=10, top_k=50, top_p=0.95, do_sample=True, max_length=100)
print(tokenizer.decode(encoded_result[0], skip_special_tokens=True))
```


## Details
Any ideas what could be causing this problem?

Thanks!!

Comments:
Probably of interest to @patrickvonplaten 

Hi @Barbara931120,

Batch generation is sadly currently not implemented in the `.generate()` method. Also, see https://github.com/huggingface/transformers/issues/3021 for reasons why. It's on our roadmap to implement this functionality soon :-)
How to trace the BertForQuestionAnswering
Author: stu1130
Pull request: no
Labels: name: wontfix

I followed the example [here](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering) and want to convert BertForQuestionAnswering to TorchScript. 
Here is my code
```
from transformers import BertTokenizer, BertForQuestionAnswering
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad', torchscript=True)

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
input_ids = tokenizer.encode(question, text)
token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]

input_tensor = torch.tensor([input_ids])
token_type_ids_tensor = torch.tensor([token_type_ids])

# The way I traced the model could be wrong here
traced_model = torch.jit.trace(model, (input_tensor, token_type_ids_tensor))
traced_model.eval()
start_scores, end_scores = traced_model(input_tensor, token_type_ids_tensor)

all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
answer = ' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1])

print(answer)
```
The answer I got is `jim henson was a nice puppet [SEP]`
Do you have any idea how to make it correct?
Thanks

Comments:
I found a workaround solution, just to pass the complete inputs including `input_ids, token_type_ids and attention_mask` to the trace method and invoke forward along with those 3 inputs. 
But it would be great to know how I can just pass in input_ids and token_type_ids

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Mean reduce over last hidden state
Author: celsofranssa
Pull request: no
Labels: name: wontfix

One of the outputs of [TFBert](https://huggingface.co/transformers/model_doc/bert.html#transformers.TFBertModel.call) is the `last_hidden_state` which is a tensor of shape `(batch_size, sequence_length, hidden_size)`.

How someone could proceed to compute the `mean pooling` of the valid embeddings? I mean, as the `attention_mask` avoids performing attention on padding token indices, it can be used as weight to average only over the real input embeddings ignoring the pad embeddings.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Metrics are coupled to the run_glue.py tasks.
Author: rosenjcb
Pull request: no
Labels: name: wontfix

The metrics used for evaluating a run_glue task is coupled to the task itself (i.e. regression or classification). We wrote a `TrinarySentimentProcessor` which grabs a positive, neutral, or negative sentiment from a text, but we found that the simple accuracy measure was not the right one. We wanted to use log loss (cross entropy) and so we added an log loss function to the metrics (`__init__.py`) package and added another elif to the chain.

Why is the metric so coupled to the task itself? What if you wanted to use a different metric for any particular task? I understand that `run_glue.py` is an "example" but we've been building upon the architecture to reduce the workload of training for tasks outside of GLUE and SQUAD.  Maybe you could add a flag to the `run_glue.py` file like `--metric=cross-entropy`. Any thoughts? Are we just misusing the library by extending `glue.py`?

Comments:
As you say yourself, the core of the library is to provide you with models (and recently also pipelines and in the future even more exciting things) to build your own projects. The examples show you some implementations that are in themselves usable but that are by no means meant to be exhaustive for all problems. As you indicate yourself, you are invited to adapt these examples to your own use case.

If you feel that your changes are useful for the whole community, then feel free to request a PR!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Regarding distilbert-multilingual-uncased model
Author: divyag11
Pull request: no
Labels: name: wontfix

I am using pretrained distilbert-multilingual-uncased model to get the embeddings of a sentence. I  want to ask which layer would be great for taking the semantic embedding of a sentence

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GlossBert adding
Author: Oxi84
Pull request: no

Here is a link https://github.com/HSLCY/GlossBERT - it should be able to do better for word vectors representations and there is already pretrained model than maybe need to be converted to a different format.

Comments:
 Actually it already can be loaded and seem to work in some cases,a ll i tested now.                              
 model_a = BertModel.from_pretrained("/folder/")
 tokenizer_a = BertTokenizer.from_pretrained("/folder/")

It can be downloaded from the link.

As you say, it is already available for download in their repository. If you want them to add their model to the transformers model hub, you can open an issue on their GitHub repo and ask them to add the model [here](https://huggingface.co/models?search=glossbert).
[Benchmark] QUAERO French Medical Corpus for Named Entity Recognition
Author: PieterDujardin
Pull request: no
Labels: name: wontfix

# 🖥 Benchmarking `transformers`

## Benchmark

I am trying the Transformers library out on [QUAERO French Medical Corpus](https://quaerofrenchmed.limsi.fr/) NER-dataset (consisting of Medline titles and EMEA documents). Using Camembert-base 'out of the box' with default hyperparameters, I get an F1 measure of 85% on the EMEA dataset and 64% on Medline, while [one of the few papers](http://ceur-ws.org/Vol-1391/158-CR.pdf) I found that did an experiment on the same dataset reported F1 of 70% and 52%, respectively, using a classic CRF. 

Since extensive grid search for hyperparameter optimization is computationally expensive even with a GPU, and given that I am relatively new to the field, I was wondering how to actually go about further optimizing the current model that I have. Is it even worth doing a lot of hyperparameter optimization or is it common that out of the box transformer models already do a decent job?

In particular, I'm not so sure which of the following things are worthwhile trying out;
- Making the input for the BERT model longer than 1 sentence
e.g. 3 input sentences [CLS] sentence1 sentence2 sentence3 [SEP]
- Extending the vocab of the tokenizer by adding tokens related to the medical domain? Not sure 
if this even makes sense doing..
- Which hyperparameters do I focus on the most? 
I assume epochs and batch size are the most important, but which others are worth trying out as well?
- Other suggestions to improve this model?

If you need more information regarding the setup I will be happy to provide it, thanks!



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
When I used the add_special_tokens function in the BertTokenizer, it assigns 2 different tokens with the same ID. Is this done on purpose?
Author: arnavc1712
Pull request: no
Labels: name: Need more information

name: wontfix


Comments:
We would need a bit more information to understand the issue. A reproducible code example would be even better.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_selec
Author: ankit-ai
Pull request: no
Labels: name: wontfix

I wrapped the code base into a flask container and tried to run it on a GPU and I am running into a GPU device copy issue. Looking for pointers.

I have verified that the model actually gets moved to the GPU:
`        self.model.to(torch.device("cuda" if torch.cuda.is_available() and not "store_true" else "cpu"))
        self.model.eval()  # TO HERE`

```
backend_1        |   File "/app/run_generation.py", line 285, in hook
backend_1        |     num_return_sequences=args.num_return_sequences,
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
backend_1        |     return func(*args, **kwargs)
backend_1        |   File "/app/transformers/modeling_utils.py", line 979, in generate
backend_1        |     attention_mask=attention_mask,
backend_1        |   File "/app/transformers/modeling_utils.py", line 1016, in _generate_no_beam_search
backend_1        |     outputs = self(**model_inputs)
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
backend_1        |     result = self.forward(*input, **kwargs)
backend_1        |   File "/app/transformers/modeling_gpt2.py", line 599, in forward
backend_1        |     inputs_embeds=inputs_embeds,
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
backend_1        |     result = self.forward(*input, **kwargs)
backend_1        |   File "/app/transformers/modeling_gpt2.py", line 465, in forward
backend_1        |     inputs_embeds = self.wte(input_ids)
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
backend_1        |     result = self.forward(*input, **kwargs)
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 114, in forward
backend_1        |     self.norm_type, self.scale_grad_by_freq, self.sparse)
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1484, in embedding
backend_1        |     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
backend_1        | RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_selec
```

Comments:
Are the inputs also being cast to GPU?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
missing import in BartForConditionalGeneration example
Author: cgnorthcutt
Pull request: no
Labels: name: Need more information

name: wontfix

Add 

`from transformers import AutoTokenizer, AutoModelWithLMHead`

to Bart for Conditional Generation example in 

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
Please complete the issue template, I don't understand what the issue is.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
which iterator to use for different hugging face transformer models for solving multiple choice questions?
Author: h56cho
Pull request: no
Labels: name: Help wanted

name: Discussion

name: wontfix

Hello,

I have used Hugging Face GPT-2 models to do Natural Language Processing.
When I used the GPT-2 model, I used the ```BPTTIterator``` from TorchText to pre-process my text data, since GPT-2 essentially performs regular language modelling (next token prediction).

I am wondering, when I use GPT-2, BERT and XLNet for **multiple-choice solving**:

1. What type of iterator should I use for multiple-choice question solving? If there is no specific iterator that can accommodate multiple-choice question solving, how should I pre-process my text (multiple-choice questions) before feeding them into the different transformer models?

2. if hugging face transformer models (BERT, XLNet, GPT-2)  use special tokens to separate questions from multiple-choice options, what are those special tokens? I want to make the use of the pre-trained models rather than training a new model with new special tokens on my own.

3. Is the format of multiple-choice questions that can be processed with, for example, BERT different than the multiple-choice question format that can be processed by GPT-2? or can each of the transformer models process any type of multiple-choice questions? (for example, BERT could only be used to solve fill-in-the-blank multiple-choice questions, whereas multiple-choice question for GPT-2 does not have to have the fill-in-the-blank format?)

Thank you,

Comments:
Do I need to build my own iterator for this?
How should the tokens be arranged for multiple choice question solving?

e.g. (question) (token1) (token2) (MCoption)(answer1)
. ----> is this the right format for all of the GPT-2, BERT and XLNet? 

Thank you

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
EncoderDecoderModel forwards return different values every time.
Author: mmsamiei
Pull request: no

# 🐛 Bug in EncoderDecoderModel

I am using EncoderDecoderModel and I have tested the sample code of it which is written in its page.

```
from transformers import EncoderDecoderModel, BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = EncoderDecoderModel.from_encoder_decoder_pretrained(
    'bert-base-uncased', 'bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)

output = model(input_ids=input_ids, decoder_input_ids=input_ids)[0]
```

but every time I run this code i will get different values for output! I also have used model.eval() but it also couldn't help.




Comments:
Could you please provide the whole code you use? Your structure works ideally for me, this code outputs the same values:
```
from transformers import EncoderDecoderModel, BertTokenizer
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = EncoderDecoderModel.from_encoder_decoder_pretrained(
    'bert-base-uncased', 'bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)

outputs = []
for _ in range(5):
    result = model(input_ids=input_ids, decoder_input_ids=input_ids)[0]
    outputs.append(result)
outputs
```


I see, on each step you initialize your EncoderDecoder model. AFAIU the difference is caused by a randomly initialized layers for decoder in this architecture. You can check it with this code:

```
params1, params2, models = [], [], []
for _ in range(2):
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

    model = EncoderDecoderModel.from_encoder_decoder_pretrained(
        'bert-base-uncased', 'bert-base-uncased')
    
    models.append(model)

pars = models[0].decoder.bert.encoder.parameters()
for _ in range(1000):
    try:
        params1.append(next(pars))
    except:
        break

pars = models[1].decoder.bert.encoder.parameters()
for _ in range(1000):
    try:
        params2.append(next(pars))
    except:
        break

[torch.all(params1[i] == params2[i]).item() for i in range(len(params1))]
```

Thanks for answering @Aktsvigun ! Yes, in the encoder decoder framework, when you instantiate an encoder-decodel using two pretrained BERT models the cross attention layer weights are added and randomly initialized. This is an expected behavior. When you set your log level to INFO you will receive a notification about this as well :-)
run_tf_ner.py output_dir/saved_model empty
Author: jx669
Pull request: no

# ❓ Questions & Help

https://github.com/huggingface/transformers/tree/master/examples/token-classification

I followed the example here and saw the logging message:
` Saving model in model/saved_model`

Then I went to the folder saved_model and found it is empty.

Is this expected?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
I got my answers here: https://github.com/huggingface/transformers/issues/3246
so I am closing this issue.
Why init specific layers rather than whole model  in BART
Author: BramVanroy
Pull request: no
Labels: name: Core: Modeling

In BartForSequenceClassification I can see that rather than calling `self.init_weights()` (which most other models use) only specifically the classification head is initialized.

https://github.com/huggingface/transformers/blob/c58e6c129a153ca1a5021e5d7e642d00bf011e20/src/transformers/modeling_bart.py#L1046-L1047

Is there any advantage of doing this for the head(s) only rather than for the whole model? I can think of a speed improved, but apart from that I'm not sure.


Comments:
The line 
```python
self.model = BartModel(config)
```
already calls `init_weights`, so no need to run it twice.

Ah, you are of course correct. I noticed this because the other models (XXXForXXX) seem to just re-call init_weights, e.g.

https://github.com/huggingface/transformers/blob/e33fdc93b4ecb571dd7a8002a74789ec8bfffc09/src/transformers/modeling_bert.py#L1008-L1015

But I guess that in practice it does not matter, it'll just be a bit slower but the result will be the same.

Thanks!
❓ How to use Gradient Accumulator in TF_Trainer ?
Author: astariul
Pull request: no

# ❓ Questions & Help

According to the documentation of Gradient Accumulator :

> the accumulator should be called in a
  replica context. Gradients will be accumulated locally on each replica and
  without synchronization. Users should then call ``.gradients``, scale the
  gradients if required, and pass the result to ``apply_gradients``.

The default optimizer for TF_Trainer does not provide gradient accumulation.

**Is there any example available showing how to use Gradient Accumulator with TF_Trainer ?**

Comments:
Hello!

The usage of the gradient accumulator is fully integrated since the beginning in the TF Trainer, the default value of accumulation is 1, if you want to change it you have to fill the `--gradient_accumulation_steps` parameter.

That's awesome ! Thank you very much for your answer