noamaanMulla-03/transformers-issues huggingface dataset:
wrong default value of argument "ignore_index" in CrossEntropyLoss for loss calculation in models forward method
Author: shyrma
Pull request: no

Hi guys!

You calculate loss in models forward() methods as follows:
`loss_fct = CrossEntropyLoss()`
for example:
[BertForMaskedLM::forward](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L1328)
[BartForConditionalGeneration:forward](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_bart.py#L1305)

that means you don't take into account labels ids = -100. However this is something one usually doesn't expect, because usually     pad_id != -100. In my opinion DEFAULT behavior should be:
## Expected default behavior
`loss_fct = CrossEntropyLoss(ignore_index=self.config.pad_token_id)`
that is ignore deposits from paddings in loss. At least this is what tensorflow [tutorial](https://www.tensorflow.org/tutorials/text/transformer#loss_and_metrics) is talking about.

I'm aware about [note](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_bart.py#L1271-L1272) in  documentation and possible work around may be
`labels[labels == model.config.pad_token_id] = -100`
but it could be additional not obvious step for ordinary user I'm afraid. 

### Who can help
- @LysandreJik
- @patrickvonplaten

Thanks!


Comments:
@LysandreJik @sgugger - we could add a warning that fires if one hasn't set `-100` in the labels, but has padding tokens instead - what do you think? 

Would it not hurt performance to do the check at each forward?

My opinion on this is that the documentation explicitly states how the `labels` should be created and which value is ignored in the loss, so I'm fine with leaving it like this.

Also `-100` is the default ignored index in the `CrossEntropyLoss`  PyTorch object so it doesn't seem too arcane of a choice.

We could throw a warning the first time it's detected (and not on subsequent calls) if it's really important.

Thank you for consideration 
I suspect a lot of people using your library out of the box (usual situation in production) will miss this step (replacing padding ids by -100), at least because it is absent in examples (as far as I know) and will always get wrong (a slightly bigger) loss :-) 
But this is my opinion and I could be mistaken of course.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Convert blenderbot checkpoint to tensorflow (TF)
Author: sooftware
Pull request: no

Hi! Thank you for a great project.  
I wonder if I can convert blenderbot checkpoint to tensorflow.  
If I can, how I convert checkpoint? Give me some comment.

Comments:
@patrickvonplaten Can you help me?

I convert Parl-AI's checkpoint to huggingface using `convert_blenderbot_original_pytorch_checkpoint_to_pytorch.py`. 
And I convert pytorch checkpoint to tf checkpoint using `convert_pytorch_checkpoint_to_tf2.py`.  
  
If there is something wrong, please comment.

Hey @sooftware,

Could you add a code snippet you are trying to execute here? E.g. which checkpoint do you want to convert exactly?

Hi @patrickvonplaten !!  
  
I want to convert Parl-AI's blanerbot (3B, 9B) models.  
I tried to convert Parl-AI to huggingface checkpoint by `convert_blenderbot_original_pytorch_checkpoint_to_pytorch.py`.  
  
I change some keys.  
  
```python
def rename_layernorm_keys(sd):    
    keys = [        
        "model.encoder.layernorm_embedding.weight",        
        "model.encoder.layernorm_embedding.bias",        
        "model.decoder.layernorm_embedding.weight",        
        "model.decoder.layernorm_embedding.bias",    
    ]
```
  
Ex) `model.encoder.layernorm_embedding.weight` => `encoder.norm_embeddings.weight`. 
  
And I got a config file by `wget https://huggingface.co/facebook/blenderbot-3B/resolve/main/config.json`.  
   
Next, I tried to convert huggingface pytorch checkpoint to tensorflow checkpoint by `huggingface convert_pytorch_to_tf.py`.  
I set `model_type` by `bart`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Blender 9B model
Author: hyunwoongko
Pull request: no

Are there  any plans to release the blender 9B model in the transformers library?

Comments:
I think @patil-suraj talked about it at some point?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

![스크린샷 2021-06-17 오전 12 44 24](https://user-images.githubusercontent.com/38183241/122251094-2c4d9480-cf05-11eb-8355-d5741c30a06c.png)

I ported myself :)
rfc: new benchmark tool
Author: stas00
Pull request: no
Labels: name: Benchmarks

name: WIP

This issue is to collect notes and ideas on creating a new benchmarking tool.

This is not about the other speed/memory regression project we have been discussing elsewhere. 

This is about integration and various comparisons that we need to run in order to give users the best advice on how to deploy transformers in the most efficient way.

Please share the comments ideas/suggestions/concerns/needs, and I will compile them here. 

- important: not part of examples - the goal is performance and integration tooling and not user-facing - totally different needs and priorities
- the cmd line has to continue working the same months later - so that old benchmarks could be re-run - ok to change interface with back-compat option so that the old benchmarks can be still re-validated and compared to
- ideally work with any transformers model - a single tool to rule them all
- minimal amount of arguments - just the important ones
- ability to generate markdown table entries directly and json files that contain not just the outcome but also the key variables that are being tested - 
- the report to include critical hardware/software params as well in a compact form and allow these to be merged from multiple recordings - i.e. if the hw/sw are the same - they can be merged into a single report. will need to figure out how to record hardware nuances 
    *  e.g. the same DDP test with 2 gpus connected w/ NVLink gives dramatically different results than the same 2 gpus w/o NVLink.
    * not sure how to record CPU-capacity/ free RAM, etc., since all these impact the outcome
- crucial to be able to truncate the dataset

Comments:
I was thinking about one feature if possible,

How about when we run an example script a benchmarking script is automatically run and store the results in one file if the user passes an optional argument.

When the user uploads the model on the model hub we can directly sort the model based on benchmarking results file.

All data files on the model hub for the same model arch will give the same speed performance results, since they are just data points. 

Therefore it's the model code that needs to be benchmarked (and the trainer if there is more than one).

And given that currently we have only one model implementation of each there is nothing to compare it to.

The main idea of this issue is to do regression testing, to ensure that we don't accidentally make models slower while changing the code. For an example of this happening, please see: https://github.com/huggingface/transformers/pull/11218
[Docs] Broken Link in the Benchmarks.rst
Author: bhadreshpsavani
Pull request: no

## Issue info

In the documentation at [Benchmarks page](https://huggingface.co/transformers/benchmarks.html) the last link is broken due to the reordering of examples folders

It is 
```
With the new `benchmark` tools, it is easier than ever to share your benchmark results with the community
:prefix_link:`here <examples/benchmarking/README.md>`.
```
should be changed to
```
With the new `benchmark` tools, it is easier than ever to share your benchmark results with the community
- :prefix_link:`Pytorch Benchmarking Results<examples/pytorch/benchmarking/README.md>`.
- :prefix_link:`Tensorflow Benchmarking Results<examples/tensorflow/benchmarking/README.md>`.
```
or a separate documentation page can be created for benchmarking the results of the model. Please Let me know if I can help or if it is being covered by another ongoing effort.

## Who can help?
@patrickvonplaten
@sgugger

Comments:
Don't hesitate to submit a PR with a fix!
Use yaml to create metadata
Author: sgugger
Pull request: yes

# What does this PR do?

This PR leverages `pyaml` to avoid writing yaml manually, as suggested by @julien-c .

Comments:
note that this can ultimately be in `huggingface_hub` rather than here (but it's great to be able to experiment with this here)

Yes we were talking about it with @LysandreJik, probably for after the upcoming release!
Remove extra self from _save_checkpoint call
Author: orf
Pull request: yes

Currently this code is completely broken with non-distributed training. I'm not clear on how it has ever worked:

```
  File "run.py", line 152, in <module>
    trainer.train() #resume_from_checkpoint=get_last_checkpoint("/opt/ml/checkpoints"))
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1105, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1202, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/sagemaker/trainer_sm.py", line 245, in _save_checkpoint
    super()._save_checkpoint(self, model, trial, metrics=metrics)
TypeError: _save_checkpoint() got multiple values for argument 'metrics'
```

This is because the `self` argument shouldn't be passed, so `trial` ends up as `metrics` via it's position.

Comments:
This is a PR against an older version of Transformers, which we do not accept. This code has been completely removed since then and is now fully integrated into Trainer.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
blenderbot checkpoint convert script has bug.
Author: hyunwoongko
Pull request: no

- error

![image](https://user-images.githubusercontent.com/38183241/122261506-70459700-cf0f-11eb-9c51-d30fc6362352.png)

- original parlai checkpoint

![image](https://user-images.githubusercontent.com/38183241/122261541-79ceff00-cf0f-11eb-938f-0be13cadd471.png)

- so, we should fix codes like below.
```python

def rename_layernorm_keys(sd):
    keys = [
        "encoder.norm_embeddings.weight",
        "encoder.norm_embeddings.bias",
        "decoder.norm_embeddings.weight",
        "decoder.norm_embeddings.bias",
    ]
    for k in keys:
        v = sd.pop(k)
        new_k = "model." + k.replace("norm_embeddings", "layer_norm")
        assert new_k not in sd
        sd[new_k] = v


IGNORE_KEYS = ["START"]
```

Comments:
Now I have almost fixed the bug. I will PR soon.
XLM-RoBERTa MLM Trainer not saving 'sentencepiece.bpe.model' file
Author: gabrieltardochi
Pull request: no

## Environment info (Colab)

- `transformers` version: 4.7.0.dev0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.5.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Information

Model I am using xlm-roberta-base:

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

Example: [RoBERTa/BERT/DistilBERT and masked language modeling, using HuggingFace Trainer with your own train file](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

Dataset: csv file with only one column named "text", containing one sentence per row.

## To reproduce

Steps to reproduce the behavior:

1. Follow the instructions displayed in this [pytorch language-modeling examples](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) page (RoBERTa/BERT/DistilBERT and masked language modeling, using HuggingFace Trainer with your own train file).

Used command: 
`!python train.py --model_name_or_path xlm-roberta-base --train_file custom_train_dset.csv --save_steps 300000 --line_by_line --do_train --output_dir xlm-roberta-base-mlm-tuned-example`

## Expected behavior

I did the very same thing, but with less data(same custom dataset, less rows) two days ago(2021/06/14) and I got the desired output:
![image](https://user-images.githubusercontent.com/60230715/122219560-f0bfb580-ce85-11eb-99a7-982c474d8050.png)


Now, this is the output that I am getting(**wrong**):
![image](https://user-images.githubusercontent.com/60230715/122218776-287a2d80-ce85-11eb-8878-622f58f8fb39.png)

@sgugger 

Comments:
Without seeing your training script, it's impossible to diagnose what went wrong. I just tried a `tokenizer.save_pretrained(...)` with this model and I get all the files.

Hey @sgugger, thanks for the quick reply! I was making a very stupid mistake(typo) and haven't noticed it until now.
I was using 'roberta-base' instead of 'xlm-roberta-base', that is why there was no 'sentencepiece.bpe.model' file when saving it.
Sorry for taking your time!
[Docs] fixed broken link
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?

Fixes #12200 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
Documentation: @sgugger

Comments:
Thanks again!
What is the correct way to pass labels to DetrForSegmentation?
Author: nateraw
Pull request: no

The [current documentation](https://huggingface.co/transformers/master/model_doc/detr.html#transformers.DetrForSegmentation.forward) for `DetrModelForSegmentation.forward` says the following about `labels` kwarg:

> The class labels themselves should be a torch.LongTensor of len (number of bounding boxes in the image,), the boxes a torch.FloatTensor of shape (number of bounding boxes in the image, 4) and the **masks a torch.FloatTensor of shape (number of bounding boxes in the image, 4).**

But when I looked at the tests, it seems the shape of `masks` is `torch.rand(self.n_targets, self.min_size, self.max_size)` . 

https://github.com/huggingface/transformers/blob/d2753dcbec7123500c1a84a7c2143a79e74df48f/tests/test_modeling_detr.py#L87-L103

---

I'm guessing this is a documentation mixup!

Anyways, it would be super helpful to include a snippet in the DETR docs that shows how to correctly pass masks/other labels + get the loss/loss dict. 😄 

CC: @NielsRogge 

Comments:
Thanks for noticing, that's a mistake. The masks needs to be torch.FloatTensor of shape (number of bounding boxes in the image, height, width) - with height and width equal to those of the `pixel_values`. 

Note that predicting boxes is required for the training to be possible, since the Hungarian matching is computed using distances between boxes.

However, I've got no less than 5 notebooks coming up that illustrate how to use DETR ;) 

I will fix this docs issue, together with some other small improvements, in a PR. 

No worries! I got it working after this. Training is a bit finicky though 😅. Looking forward to those notebooks!!
Update pretrained_models.rst
Author: mnskim
Pull request: yes

# What does this PR do?

Updates the description of facebook/bart-base and facebook/bart-large in Pretrained models to specify the number of encoder and decoder layers according to #11574 

Comments:
Hi @patil-suraj , 
I added the (N encoder and decoder layers) to the existing descriptions for facebook/bart-base and facebook/bart-large. Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Reformer inference widget is broken
Author: hamelsmu
Pull request: no

From: https://huggingface.co/google/reformer-enwik8?text=My+name+is+Julien+and+I+like+to

![image](https://user-images.githubusercontent.com/1483922/117581699-17cede80-b0b3-11eb-8cb7-1bd790f2e8c2.png)

@patrickvonplaten

Comments:
Yeah Reformer has no tokenizer so this doesn't work...sadly I also think since Reformer doesn't work super well, it's low priority to fix this (cc @Narsil )

Seems like the tokenizer fix is not that hard to make given the README (simply byte shift+2).

However the code seems to refer to a spiece.model : https://github.com/huggingface/transformers/blob/master/src/transformers/models/reformer/tokenization_reformer.py#L37

How bad is reformer performing ? (I am also curious to know how bad are character level models)

Edit: OK, this one is different because it is character level, but it's different from regular transformers that uses spiece, I see.

The character level Reformer  model does quite well at text generation

Here is a demo (from a while back) for generating Wikipedia like entries: 
https://colab.research.google.com/drive/1Oao8vBtDkz6v1E1efUhTlug5uuxC_BnE?usp=sharing

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Punctuation in Wav2Vec2
Author: ghost
Pull request: no

Playing around with Wav2Vec2, and some of the sentences contain punctuation (like apostrophes, etc.) but i don't see any periods, question marks, commas etc. in my output. 

I'm not sure if this a new feature, or just a way of handling the processing, but would it be possible to get punctuation in the Wav2Vec2 outputs ? 

Comments:
Hey @krrishdholakia,

Wav2Vec2 was fine-tuned mostly on Librispeech which does not contain any punctuation so it'll be very difficult for the model to predict punctuation. You could take a pre-trained Wav2Vec2 model and fine-tune it on a downstream task that contains punctuation. This should then just work fine

I came across [this](https://huggingface.co/flexudy/t5-small-wav2vec2-grammar-fixer) t5 model trained to correct grammar outputs of wav2vec2. Not sure if this is what you were looking for @krrishdholakia, but you can pass the text to [this](https://huggingface.co/flexudy/t5-small-wav2vec2-grammar-fixer) model and get back punctuated sentences. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Deberta v2 Fast Tokenizer
Author: ShubhamSanghvi
Pull request: no

Fast tokenizers for deberta models were requested in #10498. For the deberta (v1) model, they were implemented in #11387. 

Deberta v2 fast tokenizers are yet to be implemented.

Comments:
Any kind soul please add fast tokenizer for Deberta V2. Would be really helpful and thanks in advance!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Tokenizer encoding skips � character
Author: seahrh
Pull request: no

## Environment info
- `transformers` version: 4.5.1
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

- tokenizers: @LysandreJik

## Information

Model I am using (Bert, XLNet ...): Electra

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
c = "foo � bar"
print(f"c[4:5]={c[4:5]}")
e = tokenizer(c, return_offsets_mapping=True)
print(repr(e))
"""
{'input_ids': [101, 29379, 3347, 102], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1], 'offset_mapping': [(0, 0), (0, 3), (6, 9), (0, 0)]}
"""
i = e.char_to_token(4)
print(f"i={repr(i)}")  # i=None
```

## Expected behavior

Problem: � character was not encoded by the tokenizer.

� character should be encoded as some token <UNK> or otherwise.

Said character appears in the SquadV2 dataset with ID `5acd29f507355d001abf3774`:
```
Question
What is the glyph that Apple's Last Resort font displays?

Context
Rendering software which cannot process a Unicode character appropriately often displays it as an open rectangle, or the Unicode "replacement character" (U+FFFD, �), to indicate the position of the unrecognized character. Some systems have made attempts to provide more information about such characters. The Apple's Last Resort font will display a substitute glyph indicating the Unicode range of the character, and the SIL International's Unicode Fallback font will display a box showing the hexadecimal scalar value of the character.

Answer
�
```


Comments:
Maybe @n1t0 has an idea!

This is totally expected behavior. This tokenizer uses the same cleanup steps that were used in BERT, and this character is specifically removed.

Cf here on line 492:
https://github.com/huggingface/transformers/blob/32dbb2d/src/transformers/models/bert/tokenization_bert.py#L487-L498