noamaanMulla-03/transformers-issues huggingface dataset:
Adding performer fine-tuning research exampke
Author: TevenLeScao
Pull request: yes

# What does this PR do?

This PR adds a performer fine-tuning research example based on `run_mlm_flax.py`. The user can fine-tune a Performer/FAVOR+ Bert starting from the Bert checkpoint or blank model of their choice, and compare it to a vanilla Bert model, also from a checkpoint or blank.

@patrickvonplaten

Comments:
@sgugger - we want to do some fine-tuning experiments with the new performer model: https://arxiv.org/abs/2009.14794 before adding it to the `src/transformers/`. I think this is a good first place for it where we don't have to be super careful about the API choices yet. Is that fine for you?

Yes, this completely works for me!
Update the README of the text classification example
Author: sgugger
Pull request: yes

# What does this PR do?

This PR updates the main README of the examples folder and the one in the text classification example to take into account the recent changes in the scripts. In particular, I re-ran the command shown for all tasks with/without FP16 to make a clean table of results.

I moved all stuff about distributed training/TPUs in the general README of the examples as all example scripts now use Trainer, so have this working out of the box.


Comments:
Help: How to deploy a fine tuned t5 model in production
Author: as-stevens
Pull request: no

Hi All,

I am trying to deploy a fine-tuned t5 model in production. This is something new to me, to deploy a PyTorch model in production. I went through the presentation from Hugging Face on youtube, about how they deploy the model. And some of the other blog posts.

It is mentioned by HF that they deploy the model on Cython environment as it gives a ~100 times boost to the inference. So, is it always advisable to run a model in production on Cython?
Converting a model in Pytorch to TF does it help and is advisable or not?
What is the preferred container approach to adopt to run multiple models on a set of GPUs?

I know some of these questions would be basic, I apologize for it, but I want to make sure that I follow the correct guidelines to deploy a model in production.

Thank you
Amit

Comments:
Hey @as-stevens,

Could you maybe post this question on the forum: https://discuss.huggingface.co/? We try to move more user-specific questions to the forum and limit Github mostly to bug reports. Thank you!

@patrickvonplaten thank you much! I will close this issue.
Does provide any scripts about how to convert mpnet pretrain model to transformers pretrain one?
Author: RyanHuangNLP
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

I want to convert my pretrain chinese mpnet with the fairseq format to transformers format

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
Maybe the author of MPNet can help here, @StillKeepTry 

@patrickvonplaten problem is fix
Model embedding size and tokenizer size mismatch; resizing embedding will cause CUDA assert error
Author: huu4ontocord
Pull request: no

## Environment info
Google colab

### Who can help

 T5: @patrickvonplaten
 

## Information

I'm noticing something strange with T5. The model embedding size and the tokenizer size does not match.
When I try to resize the model to have a smaller embedding this crashes CUDA. This is probably two bugs - one for the size mismatch, and one for shortening the embedding causing a crash.

```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("t5-base")
print (len(tokenizer))
model = AutoModel.from_pretrained("t5-base")
print (model.shared)
model.resize_token_embeddings(len(tokenizer))
model.to('cuda')
```
## Expected behavior

Expected behaviour is regular loading of the model into cuda.

What I got instead was:
32100
Some weights of T5Model were not initialized from the model checkpoint at t5-base and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Embedding(32128, 768)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-145ba8d0b52c> in <module>()
      5 print (model.shared)
      6 model.resize_token_embeddings(len(tokenizer))
----> 7 model.to('cuda')

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in convert(t)
    608             if convert_to_format is not None and t.dim() == 4:
    609                 return t.to(device, dtype if t.is_floating_point() else None, non_blocking, memory_format=convert_to_format)
--> 610             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
    611 
    612         return self._apply(convert)

RuntimeError: CUDA error: device-side assert triggered

Comments:
Hey @ontocord, I cannot reproduce your error on master...

```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("t5-base")
print (len(tokenizer))
model = AutoModel.from_pretrained("t5-base")
print (model.shared)
model.resize_token_embeddings(len(tokenizer))
model.to('cuda')
```
works fine for me.

I am able to correctly shorten the embedding matrix

@patrickvonplaten Thank you. It's also working now in my code too with latest version of transformer. Thanks for looking into this!
T5 tokenizer.vocab_size and config.vocab_size mismatch?
Author: ArvinZhuang
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.1.1
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1
- tokenizers: 0.9.4
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


## Information
Hi @patrickvonplaten 
I am trying to train a "t5-base" model and I directly use from_pretrained tokenizer, config and model. However, I found the vocabulary size given by the tokenizer and config is different (see to reproduce). Does this is expected? 

If I use the model `T5ForConditionalGeneration.from_pretrained('t5-base', config=config)` to do predictions, this will result in the last dimension of lm_logits is different from `tokenizer.vocab_size`.


## To reproduce

Steps to reproduce the behavior:

```
>>> from transformers import T5Tokenizer, T5Config
>>> tokenizer = T5Tokenizer.from_pretrained("t5-base")
>>> config = T5Config.from_pretrained("t5-base")
>>> print(tokenizer.vocab_size)
32100
>>> print(config.vocab_size)
32128
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
```
>>> print(tokenizer.vocab_size)
32128
>>> print(config.vocab_size)
32128
```

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/4875.

I see, I simply ignored this mismatch and seems nothing wrong with prediction.
Thank you!
MarianMT = How to return 5 best candidates for a translation.  
Author: Oxi84
Pull request: no
Labels: name: High-Level feature

name: marian

This is a code for normal translation. This is the most probable translation. How can I return lets says 5 best candidates for translation for every single word (beam size would be 1)?

This model only return just the best word, which gives us a better translation. But I want to use it as a language model.

Is this even possible. I was looking at classes and code but I am not sure how would I do it.

    

    from transformers import MarianMTModel, MarianTokenizer
    src_text = [
        '>>fr<< this is a sentence in english that we want to translate to french',
    ]    
    model_name = 'Helsinki-NLP/opus-mt-en-ROMANCE'
    tokenizer = MarianTokenizer.from_pretrained(model_name)
    print(tokenizer.supported_language_codes)
    model = MarianMTModel.from_pretrained(model_name)
    translated = model.generate(**tokenizer.prepare_translation_batch(src_text))
    tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]

Comments:
That is available on this page - https://marian-nmt.github.io/faq:

Can I generate n-best lists?
 Yes. Just use --n-best and the set --beam-size 6 for an n-best list size of 6.

I do not know how to apply it here.

That feature is not supported in our implementation, unfortunately.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


You can try passing `num_return_sequences=5` to generate, but that behavior is untested, and I have never used it.

> You can try passing `num_return_sequences=5` to generate, but that behavior is untested, and I have never used it.

However, do not forget to set the beams accordingly. The beams must be defined at least as large as the desired number of alternative results.
torch.hub colab doesn't work
Author: StevenJokess
Pull request: no

https://colab.research.google.com/github/pytorch/pytorch.github.io/blob/master/assets/hub/huggingface_pytorch-transformers.ipynb#scrollTo=T_3y0655Bqbj
```
%%bash
pip install tqdm boto3 requests regex sentencepiece sacremoses
```

Then all cells left don't work!

Comments:
Fix link to old language modeling script
Author: mrm8488
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Fix link to bertabs/README.md
Author: mrm8488
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Add speed metrics to all example scripts + template
Author: sgugger
Pull request: yes

# What does this PR do?

This does the same as #9198 but on all examples scripts and the example template.

Comments:
The eval metrics are already reported with the other metrics, so need to add anything for them. Not sure about the refactor since this shouldn't really be a function in transformers (nothing to do with transformers models) so we would have to define it in every one of those scripts, which kind of takes the same length.

re: eval/train - except n_objs is missing from metrics - remember we had to add it separately in `finetune_trainer` when you refactored it?

re: refactor: I haven't suggested anything for the core - we have utils.py for that.

Yes, but this is something I just did in `finetune_trainer` to have it output the same things as before, for other scripts I don't want that reported several times (it's already logged at the beginning of training/evaluation).
Model Templates for Seq2Seq
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds the possibility to generate Encoder-Decoder models via the cookie-cutter tool.
Model is correctly generated for PT and TF with all tests passing. Two test files are added.
These templates should very much facilitate the addition of Pegasus, Blenderbot, Marian as separate model files as well as adding BigBird, etc...

Please note that for now the safety checks: `# Copied from transformers.models.bart.modeling_bart...` are only added for very few layers because:

- Bart has some hacks that we should not copy for new models, but that we need to keep for backwards compatibility. E.g. positional embeddings have an offset hack leading to slightly too large positional embeddings, which we should not repeat (same as in RoBERTa), automatic creation of `decoder_input_ids` is a special feature and not the default case, Sinusoidal position embeddings are IMO also not general enough to be in the templates
- `modeling_bart.py` still has the `add_layer_norm` hacks which are not copied to the model templates. When Bart is separated into Pegasus, etc... those if-else hacks can be deleted from `modeling_bart.py` at which point some more `# Copied from transformers.models.bart.modeling_bart...` should be added to the Seq2Seq model templates


Comments:
Improvements to TFBart: https://github.com/huggingface/transformers/pull/9252 are now included in this PR as well.
add tests for the new sharded ddp fairscale integration
Author: stas00
Pull request: yes

This PR adds tests for the just added sharded ddp fairscale integration https://github.com/huggingface/transformers/pull/9139

Obviously these won't run on CIs w/o having fairscale installed... hope we will sort this out down the road. the problem is building fairscale - no binary wheel - I will ask them if they could make a jit version.

@sgugger 

Comments:
I made a request for a quicker/simpler binary: https://github.com/facebookresearch/fairscale/issues/264

And added an issue to track this so we won't forget: https://github.com/huggingface/transformers/issues/9178
Fix gpt2 document
Author: xu-song
Pull request: yes

# What does this PR do?


Fixes gpt2 document error.
```
AttributeError: 'GPT2DoubleHeadsModelOutput' object has no attribute 'lm_logits'
```


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?



Comments:
Fix param error
Author: xu-song
Pull request: yes



# What does this PR do?


Fixes error
```
TypeError: forward() got an unexpected keyword argument 'token_type_ids'
```


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments: