noamaanMulla-03/transformers-issues huggingface dataset:
what's the difference between TFBertOutput and TFBertSelfOutput?
Author: xiongma
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->

`TFBertOutput` with `TFBertSelfOutput` seem same in their codes. why did you write two same layers? is there for some reasons?

Comments:
Hey @policeme, fair point they seem to be exactly the same. There is a logical difference though `TFBertOutput` is the output of a `TFBertLayer` while `TFBertSelfOutput` is the output a `TFBertAtteniton` (Self-attention -> thus "SelfOutput"). 
But yeah this might seem a bit confusing at first.
Longformer slow than Bert
Author: Maybewuss
Pull request: no

When i set max_length = 2048, i found Longformer's speed is slower than commen bert, why? 

Comments:
well BERT can only generate up to 512 tokens and both models are Autoencoding models and usually not used for causal language generation (Autoregressive models are used for this). You can check out the difference here: https://huggingface.co/transformers/model_summary.html.
How to fine-tune GPT2 on Arithmetic Problem
Author: KYRIEZX
Pull request: no

# #  I want to use the GPT2 to solve simple Arithmetic Problem, how should I make the dataset?



Comments:
Hey @KYRIEZX, 

we are trying to move "non-bug" questions to our forum - would you mind posting the question there again: https://discuss.huggingface.co/ ? Thanks :-)
Failing bart-base slow test
Author: sshleifer
Pull request: no

```
=================================== FAILURES ===================================
____________ BartModelIntegrationTests.test_bart_base_mask_filling _____________
[gw0] linux -- Python 3.7.6 /home/hf/actions-runner_transformers/_work/transformers/transformers/.env/bin/python

self = <tests.test_modeling_bart.BartModelIntegrationTests testMethod=test_bart_base_mask_filling>

    @slow
    def test_bart_base_mask_filling(self):
        pbase = pipeline(task="fill-mask", model="facebook/bart-base")
        src_text = [" I went to the <mask>."]
        results = [x["token_str"] for x in pbase(src_text)]
        expected_results = ["Ġbathroom", "Ġrestroom", "Ġhospital", "Ġkitchen", "Ġcar"]
>       self.assertListEqual(results, expected_results)
E       AssertionError: Lists differ: ['Ġlibrary', 'Ġhospital', 'Ġbathroom', 'Ġmovies', 'Ġpolice'] != ['Ġbathroom', 'Ġrestroom', 'Ġhospital', 'Ġkitchen', 'Ġcar']
E       
E       First differing element 0:
```

https://github.com/huggingface/transformers/runs/996031882?check_suite_focus=true

Comments:
all_hidden_states indentation bug in modeling_bert.py
Author: YuanEric88
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@lavanyashukla 
## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L490
## Expected behavior
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L490 There is a small bug here, I suppose Line491 and Line491 should be inside loop chunk. 
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
My bad, I ignore some code in the file.
Longformer pretrained weights are not really pretrained?
Author: dvirginz
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:3.0.2
- Platform:Ubuntu 18.04
- Python version:3.7
- PyTorch version (GPU?):1.6 Yes
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

 @patrickvonplaten


Model I am using (Bert, XLNet ...): Longformer

The problem arises when using:
* [V] my own modified scripts

The tasks I am working on is:
* [V] my own task or dataset: (give details below)

## To reproduce

As a preprocessing step in my pipeline I train a pre-trained model on a subset of Wikipedia dataset.
When using roBERTa the results of my fine-tuning steps are as follows (LM task):
![image](https://user-images.githubusercontent.com/31047807/90328740-71ccc880-dfa7-11ea-9df5-0163ae48533b.png)

Unfortunately, when simply plugging in longformer (and pointing the pretrained path to `allenai/longformer-base-4096`
![image](https://user-images.githubusercontent.com/31047807/90328734-5e216200-dfa7-11ea-888b-53e759fd2e29.png)
Which seems like the weights are simply random-initialized.
I both cases (roberta and longformer) I get the message stating the weights have been initialized.
What went wrong?


Comments:
Hey @dvirginz,

Can you post a code snippet here so that we can reproduce your results or at least see what might have been falsely configured?

@patrickvonplaten Hi!
I'm sorry to have troubled you, the problem was with how I initialized and loaded the weights for the model.
It is now working, Thanks!
Create README.md
Author: rohanrajpal
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6346?src=pr&el=h1) Report
> Merging [#6346](https://codecov.io/gh/huggingface/transformers/pull/6346?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9f57e39f7165fa8bd6ac911852221a76d4b79ebe&el=desc) will **decrease** coverage by `0.18%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6346/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6346?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6346      +/-   ##
==========================================
- Coverage   79.79%   79.61%   -0.19%     
==========================================
  Files         148      148              
  Lines       27196    27196              
==========================================
- Hits        21701    21652      -49     
- Misses       5495     5544      +49     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6346?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/6346/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `66.95% <0.00%> (-25.22%)` | :arrow_down: |
| [src/transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/6346/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxuZXQucHk=) | `66.66% <0.00%> (-23.43%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6346/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.71% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/tokenization\_dpr.py](https://codecov.io/gh/huggingface/transformers/pull/6346/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZHByLnB5) | `57.65% <0.00%> (+4.50%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6346?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6346?src=pr&el=footer). Last update [9f57e39...9591bf9](https://codecov.io/gh/huggingface/transformers/pull/6346?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Typo fix in 04-onnx-export
Author: SidJain1412
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6595?src=pr&el=h1) Report
> Merging [#6595](https://codecov.io/gh/huggingface/transformers/pull/6595?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ab42d74850233cff9df87701d257d9b975435f66&el=desc) will **increase** coverage by `1.10%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6595/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6595?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6595      +/-   ##
==========================================
+ Coverage   79.42%   80.52%   +1.10%     
==========================================
  Files         156      156              
  Lines       28127    28127              
==========================================
+ Hits        22339    22650     +311     
+ Misses       5788     5477     -311     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6595?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6595/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `28.84% <0.00%> (-58.66%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6595/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `71.84% <0.00%> (-23.17%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/6595/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `79.69% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6595/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.29% <0.00%> (+0.97%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6595/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+1.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/6595/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.91% <0.00%> (+72.35%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6595?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6595?src=pr&el=footer). Last update [ab42d74...a49a870](https://codecov.io/gh/huggingface/transformers/pull/6595?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
An Implementation of ERNIE For Language Understanding (including Pre-training models and Fine-tuning tools) 
Author: manhlab
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition
ERNIE 2.0 is a continual pre-training framework for language understanding in which pre-training tasks can be incrementally built and learned through multi-task learning. ERNIE 2.0 builds a strong basic for nearly every NLP tasks: Text Classification, Ranking, NER, machine reading comprehension, text genration and so on.
## Model description

<!-- Important information -->
https://github.com/PaddlePaddle/ERNIE

## Open source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
Can we update to add this into transformers

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[docs] Copy code button misses '...' prefixed code
Author: romainr
Pull request: yes

Tested in a local build of the docs.

e.g. Just above https://huggingface.co/transformers/task_summary.html#causal-language-modeling

Copy will not copy the full code, e.g.

`for token in top_5_tokens:`

Instead we should get it all:

```
for token in top_5_tokens:
     print(sequence.replace(tokenizer.mask_token, tokenizer.decode([token])))
```

>>> for token in top_5_tokens:
...     print(sequence.replace(tokenizer.mask_token, tokenizer.decode([token])))
Distilled models are smaller than the models they mimic. Using them instead of the large versions would help reduce our carbon footprint.
Distilled models are smaller than the models they mimic. Using them instead of the large versions would help increase our carbon footprint.
Distilled models are smaller than the models they mimic. Using them instead of the large versions would help decrease our carbon footprint.
Distilled models are smaller than the models they mimic. Using them instead of the large versions would help offset our carbon footprint.
Distilled models are smaller than the models they mimic. Using them instead of the large versions would help improve our carbon footprint.

Docs for the option fix:
https://sphinx-copybutton.readthedocs.io/en/latest/

Comments:
Feels more like a false positive failure?
Add model card for facebook/mbart-large-en-ro
Author: sshleifer
Pull request: no

Suggested content:

___
tags:
- translation
language: en
___

link to fairseq readme
link to docs.

Comments:
```
---
language:
- en
- ro
---
```

no?

I didn't know you can have two! Will inference api still populate the text box with "My name is wolfgang"?

Yes it will use the first one to populate the widget if there are several

Done: new config/moon-landing bug

![image](https://user-images.githubusercontent.com/6045025/90800838-2c7c0400-e2e3-11ea-8d4d-40e0aeac0850.png)
[DistilBert] Flaky tests
Author: patrickvonplaten
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-61-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyTorch version (GPU?): 1.6.0+cpu (False)
- Tensorflow version (GPU?): 2.1.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Reproduce Error:

```python 
#!/usr/bin/env python3
import torch
from transformers import DistilBertModel, DistilBertConfig

input_ids = torch.tensor([[55, 40, 88, 37, 12,  6, 20],
        [33, 87, 56,  6, 34, 92,  2],
        [ 4, 25, 95, 19,  9, 14, 80],
        [96, 45, 71, 10, 78, 33, 68],
        [72, 40, 59, 90,  5, 78, 44],
        [36, 15, 11, 18, 74, 40, 30],
        [84, 25,  5, 61, 18, 77, 35],
        [70, 87,  9, 42, 24, 65, 11],
        [28,  0, 28, 45, 92, 83, 96],
        [75, 41, 69, 61, 83, 31, 81],
        [94, 93, 79, 48, 24, 17,  9],
        [97,  5, 38, 94, 75,  8, 59],
        [31, 71, 87, 39, 97, 10, 22]])

attention_mask = torch.tensor([[1, 1, 1, 1, 0, 1, 0],
        [0, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1],
        [0, 1, 1, 1, 0, 0, 1],
        [1, 1, 1, 0, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1],
        [0, 0, 1, 0, 0, 1, 1],
        [1, 0, 1, 0, 0, 0, 1],
        [0, 1, 0, 1, 1, 1, 0],
        [0, 1, 1, 0, 0, 0, 0],
        [0, 1, 0, 1, 1, 0, 1],
        [0, 1, 1, 1, 1, 1, 1],
        [0, 1, 0, 1, 0, 0, 0]])


distil_bert_config = {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 32,
  "dropout": 0.1,
  "hidden_act": "gelu",
  "hidden_dim": 37,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 4,
  "n_layers": 5,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "return_dict": True,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": False,
  "vocab_size": 99
}

config = DistilBertConfig(**distil_bert_config)
torch.manual_seed(0)
model = DistilBertModel(config).eval()

last_hidden_state = model(input_ids, attention_mask=attention_mask)[0]

if torch.isnan(last_hidden_state).any().item():
    print("Error with DistilBert")
```

This code example allows yields nan values.

Comments:
Will investigate now @sgugger @VictorSanh . My first guess is that it because of "inf" values because of masking because this error does not happen when the attention mask is not passed to forward.
[Some weights or buffers of the PyTorch model TFGPT2LMHeadModel were not initialized] convert GPT2 pytorch to tensorflow model 
Author: gyin94
Pull request: no

# ❓ Questions & Help

How can we convert gpt2 fine tuned model from run_language_model.py from pytorch to tensorflow model? I came into the following warning when importing pytorch model GPT2LMHeadModel into TFGPT2LMHeadModel.

```
WARNING:transformers.modeling_tf_pytorch_utils:Some weights or buffers of the PyTorch model TFGPT2LMHeadModel were not initialized from the TF 2.0 model and are newly initialized: ['transformer.h.2.attn.bias', 'transformer.h.4.attn.masked_bias', 'transformer.h.4.attn.bias', 'transformer.h.3.attn.bias', 'transformer.h.3.attn.masked_bias', 'transformer.h.0.attn.masked_bias', 'transformer.h.5.attn.masked_bias', 'transformer.h.1.attn.bias', 'transformer.h.0.attn.bias', 'transformer.h.5.attn.bias', 'transformer.h.1.attn.masked_bias', 'lm_head.weight', 'transformer.h.2.attn.masked_bias']
```

Or do we have any tutorial to train a GPT2 language model by TFTrainer? 

## Details
<!-- Description of your issue -->

Reproduce

```
from transformers import *
model = GPT2LMHeadModel.from_pretrained("distilgpt2")
model.save_pretrained("./gpt2_pt")
model = TFGPT2LMHeadModel.from_pretrained("./gpt2_pt", from_pt=True)
```

However we won't see the above warning if we use 
```
model = TFGPT2LMHeadModel.from_pretrained("distilgpt2")
```

If we already use `run_language_model.py` to train a GPT2LMHeadModel model and import it into TFGPT2LMHeadModel, can we safely use the converted model even if we see the warning?

Comments:
Hi! The way you converted your model is the recommended way :ok_hand: 

The warning is irrelevant to this conversion, we should try to make that clearer.

The warning is very confusing and can be fixed by pull request #6604.

The warning is very confusing and can be fixed by new pull request #6623
504 Gateway Time-out when trying to access Uploaded Model page 
Author: paulowoicho
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I recently uploaded a model available at `https://huggingface.co/paulowoicho/t5-podcast-summarisation`. I made a few changes to the model card and reuploaded. Now, the model's page does not load at all. Instead, it shows a 504 Gateway Time-out error.

![image](https://user-images.githubusercontent.com/28223751/90789131-71616400-e2fe-11ea-928c-070da9ffb449.png)

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Comments:
cc @julien-c I'm also experiencing this.

Yes, the reason is that your model card's metadata is not valid YAML.

I had fixed it on S3 but you overrode it with the previous version – this is a known limitation of our model card system right now and why:
- we recommend submitting the model cards to this git repo for now
- we are working on a way better system to be released in the coming weeks/months

(cc @sgugger @JetRunner and others for context ^^)

I've fixed our website to handle those model pages by displaying an error notice

https://huggingface.co/paulowoicho/t5-podcast-summarisation