noamaanMulla-03/transformers-issues huggingface dataset:
Don't pass in token_type_ids to BART for GLUE
Author: ethanjperez
Pull request: yes

# What does this PR do?

Without this fix, training a `BARTForSequenceClassification` model with `run_pl_glue.py` gives `TypeError: forward() got an unexpected keyword argument 'token_type_ids'`, because BART does not have token_type_ids. I've solved this issue in the same way as it's solved for the "distilbert" model, and I can train BART models on SNLI without errors now.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten



Comments:
Thanks  for the fix @ethanjperez ! Looks good to me!
Relative Attention Bias not initialized for T5ForConditionalGeneration in version 4.0.0
Author: gsarti
Pull request: no

## Environment info
     
- `transformers` version: 4.0.0
- Platform: Linux 18.04
- Python version: 3.6.9
- PyTorch version (GPU?): 1.70 (True)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

@patrickvonplaten

## Information

Model I am using: T5ForConditionalGeneration, specifically the "allenai/unifiedqa-t5-large" checkpoint from the model hub

The problem arises when I try to load the checkpoint following standard loading procedures under `transformers==4.0.0`. The same doesn't happen in version `3.5.0`.

## To reproduce

```python
from transformers import AutoTokenizer, T5ForConditionalGeneration
model_name_or_path = "allenai/unifiedqa-t5-large"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
# Warnings in version 4.0, not in 3.5.0 an preceding ones
model = T5ForConditionalGeneration.from_pretrained(model_name_or_path)
```
```
Some weights of the model checkpoint at allenai/unifiedqa-t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.blo
ck.0.layer.1.EncDecAttention.relative_attention_bias.weight']                                                                               
- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another
 architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).                                    
- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly ident
ical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

## Expected behavior

A consistent behavior across versions, either always or never raising the warning at loading time.


Comments:
Hey @gsarti, 

yeah there was a tiny bug in T5 previously. T5 actually never had relative positional encodings for the EncoderDecoderLayer, so the unnecessary weight was deleted after 3.5.0. This should not really affect the performance however and is no problem now, see: https://github.com/huggingface/transformers/pull/8518

Thank you for the clarification!
sorry I mistakenly submitted a issue twice. Plz ignore (help delete) this one. 
Author: XiangLi1999
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
- `transformers` version: 3.2.0
- Platform: Linux-4.4.0-1111-aws-x86_64-with-debian-stretch-sid
- Python version: 3.6.8
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
@stas00 


 

## Information

Model I am using (Bert, XLNet ...): BART 

The problem arises when using:
*--pt16 --fp16_opt_level O1 + conda-forge apex w/ --fp16 --amp_backend='apex' to run finetune.py 

Got this error: 

  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 429, in fit
    self.accelerator_backend.setup(model)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 53, in setup
    model = self.trainer.precision_connector.connect(model)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/precision_connector.py", line 78, in connect
    model, optimizers = self.backend.connect(model, self.trainer.optimizers)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/plugins/apex.py", line 38, in connect
    self.trainer.reinit_scheduler_properties(optimizers, self.trainer.lr_schedulers)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/trainer/optimizers.py", line 143, in reinit_scheduler_properties
    scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 74, in __init__
    self.optimizer.step = with_counter(self.optimizer.step)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 56, in with_counter
    instance_ref = weakref.ref(method.__self__)
AttributeError: 'function' object has no attribute '__self__'



This is the same error reported in one of the reply in https://github.com/huggingface/transformers/issues/8403... 

The tasks I am working on is:
* [ ] to finetune BART on XSUM. 



Comments:
Fix typo for `modeling_bert` import resulting in ImportError
Author: machelreid
Pull request: yes

# What does this PR do?
Self-explanatory ;) - Fixes a typo resulting in an `ImportError` in the convert RoBERTa from fairseq to HF - Hope it helps!


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Add model card
Author: sarnikowski
Pull request: yes
Labels: name: model card

# What does this PR do?

Adds a model card.

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Comments:
Gradients of BERT layer outputs to inputs
Author: NitinTitus
Pull request: no

I am trying to find the gradient of the output of a layer of BERT to its inputs, token wise. But I keep getting the error saying: 'RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.' Below is the code snippet:

   for count, data in enumerate(iter(data_loader)):

    input_ids=torch.squeeze(data['input_ids'],dim=0)

    attention_mask=torch.squeeze(data['attention_mask'],dim=0)

    last_hidden_state, pooled_output, hidden_states = bert_model(input_ids=input_ids,attention_mask=attention_mask)

    bert_layer_i_output=hidden_states[i][0]

    print(bert_layer_i_output.shape)

    bert_layer_j_output=hidden_states[j][0]

    #print(torch.autograd.grad(bert_layer_j_output,bert_layer_i_output,retain_graph=True, create_graph=True))

    for k in range(bert_layer_i_output.shape[0]):
      gradient=torch.autograd.grad(bert_layer_j_output[k],bert_layer_i_output[k],grad_outputs=torch.ones_like(bert_layer_j_output[k]))
      print(gradient.shape)
      print(torch.norm(gradient))
      break
    break


Below is the stack trace of the error:

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
    202     return Variable._execution_engine.run_backward(
    203         outputs, grad_outputs_, retain_graph, create_graph,
--> 204         inputs, allow_unused)
    205 
    206 

RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.

Am i doing something wrong? Ideally both the tensors should be part of the same computational graph right?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Why BertSelfAttention reshape Q,K,V from 3-D tensor to 4-D tensor
Author: daydayfun
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description
https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py
def transpose_for_scores(self, x):
        new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
        x = x.view(*new_x_shape)
        return x.permute(0, 2, 1, 3)
query_layer = self.transpose_for_scores(mixed_query_layer)
 key_layer = self.transpose_for_scores(mixed_key_layer)
 value_layer = self.transpose_for_scores(mixed_value_layer)

## Open source status
Question:  
1. Why we must transpose Q,K,V from 3-D tensor to 4-D tensor ?
2. What if we just use 3-D Q,K,V to do torch.matmul ?


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Fix QA pipeline on Windows
Author: sgugger
Pull request: yes

# What does this PR do?

As reported on the [forum](https://discuss.huggingface.co/t/pipeline-example-in-the-doc-throws-an-error-question-answering/2632), there is a problem in the current pipeline on Windows. The problem's root is that numpy int arrays have a different default on Linux and Windows, the current snippet:
```
import numpy as np
x = np.array([1, 2, 3])
x.dtype
```
will print `dtype('int64')` on Linux/MacOS but `dtype('int32')` on Windows. So this means that just doing `torch.tensor(some_numpy_array)` may result in a tensor of dtype `int32` which PyTorch does not like. For future reference, the error:
```
Expected tensor for argument #1 'xxx' to have scalar type Long; but got torch.IntTensor instead
```
is usually a clear indicator of this behavior happening.

The PR fixes the QA pipeline by casting the tensors to long if they have the int type.

Comments:
Implement a TF2 version of `GPT2ForSequenceClassification`
Author: LysandreJik
Pull request: no
Labels: name: Good First Issue

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

The architecture `GPT2ForSequenceClassification` was added in #7501 in PyTorch. It would be great to have it in TensorFlow as well.

Below is a list of items to follow to make sure the integration is complete:

- Implement `TFGPT2ForSequenceClassification` in `modeling_tf_gpt2.py`
- Test that architecture in `tests/test_modeling_tf_gpt2.py`
- Add that architecture to `__init__.py` and ` docs/source/model_doc/gpt2.rst`.

Taking a look at the code changes in #7501 would be a good start, as this PR would essentially be a TF2 copy of it.

A very good first issue to get acquainted with the library and its architectures!

Comments:
May I take a crack at this? :)

Yes, you may! Thanks @y2s82!

Hi @LysandreJik , I have completed development for this FR. can you please assign it to me, so i'll raise PR for it.

Feel free to open a PR
TFBertModel NOT learning at all! 
Author: ivankrstev7
Pull request: no

Hi, i am trying to implement a simple Keras model where the first inputs are the input_ids and the attention_mask and then i have a `TFBertModel.from_pretrained('bert-base-uncased')` layer to extract the word embeddings and everything compiles okay, but when I train the model I get a constant accuracy of 0.5 (it is a binary classification problem).
Here is how I've defined my model:
![image](https://user-images.githubusercontent.com/48191509/101362514-80b1f080-38a0-11eb-9dd3-05ab66efade6.png)
And I am using `BertTokenizer.from_pretrained('bert-base-uncased')` to prepare the dataset. I might also have a problem with how i feed the data to the model, I am not sure, so here is a scr of that too:
![image](https://user-images.githubusercontent.com/48191509/101362915-06ce3700-38a1-11eb-965a-9151228d8294.png)


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Fine-tuning on Language Model using two tasks
Author: sajastu
Pull request: no

Hi,

I'm reading the language modeling example on the documentation: https://huggingface.co/transformers/v2.0.0/examples.html#language-model-fine-tuning

It seems that the fine-tuning is done based on Masked Language Modelling (MLM) loss. While in the BERT paper, it appears that the LM fine-tuning is done by optimizing two tasks: 1) Masked Language Modeling, and 2) Next sentence prediction. I'm looking for the second part in Huggingface's implementation, but it seems that this part is either not implemented or I'm missing something? 

Comments:
Hi, you're not missing anything, this part is not implemented in the examples, as fine-tuning a model using only MLM yields similar downstream results than fine-tuning a model with both tasks. 

However, we have the `BertForPreTraining` architecture which is implemented, and which can train a model using the two objectives. You would have to tweak the example scripts to manage this case, however.

Also, we try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!

> Also, we try to keep the github issues for bugs/feature requests.
> Could you ask your question on the [forum](https://discusss.huggingface.co) instead?
> 
> Thanks!

Sure, thank you for clarifying!
Use word_ids to get labels in run_ner
Author: sgugger
Pull request: yes

# What does this PR do?

As #8958 pointed out, the current way labels are computed in the `run_ner` script using offset mappings does not work for sentencepiece-based tokenizers. This PR fixes that using the `.word_ids` method which is more elegant and more reliable.

In passing it adds an early check that the tokenzier is fast (otherwise the script just doesn't work).

Fixes #8958


Comments:
[T5] Add open / closed book answering models
Author: patrickvonplaten
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description
 Check here: https://github.com/google-research/google-research/tree/master/t5_closed_book_qa
<!-- Important information -->

## Open source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
Done: https://huggingface.co/models?search=ssm
Create README.md
Author: Quangtruong1999
Pull request: yes
Labels: name: model card

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
This doesn't seem correct