noamaanMulla-03/transformers-issues huggingface dataset:
Improve T5 docs 
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR aims to clarify and explain some of the magic that's happening when using T5 for training/inference. It includes:
- explaining that the model automatically creates the `decoder_input_ids` based on the `labels` (a lot of people were still confused by this, see e.g. #11977, #13213 
- added code examples, to show a basic forward pass that includes the fact that padding token ids of the labels should be replaced by -100 (at least, for PyTorch, I see that for FLAX one uses the `decoder_attention_mask` to skip padding tokens), and code examples for inference (both batched/not batched)
- adding info about T5's variants, including T5v1.1, mT5 and byT5, with links to their docs.
- additional tips & tricks, based on what I found on the forum (learning rate, training on TPU, etc).

In addition, I've added T5v1.1 to the main README as well making it have its own documentation page, and some more info to the mT5 docs.

Comments:
Add template for adding flax models
Author: changjonathanc
Pull request: yes

# What does this PR do?

Fixes #12440

## From Patrick
@LysandreJik @sgugger Previously both the PT and TF encoder-decoder templates tests weren't run because the test name didn't match the "*template*" regex - I've fixed that here as pointed out in the comment below. 

@cccntu added both FlaxBERT and FlaxBART templates and we've added two tests to make sure those work as expected. 

The PR should be good for review now :-)

Comments:
It's mostly done. I can successfully create a new model with flax. See #12454
TODO:
* Finish modeling_flax for encoder-decoder architecture
* Tests are mapped from tf to np, need fixes.
* Refactor coockie-cutter workflow, make flax blend in with pytorch, tensorflow.
* ??

I think only the test for template is failing now, can someone take a quick look? Thanks!
@patil-suraj @patrickvonplaten @LysandreJik 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Maybe I should rebase and open another PR?

Sorry for being so late on this one! I'll take a look tomorrow!

yaaaay - finally all the tests are green! @cccntu - amazing work. I've done a lot of small fixes that are quite time-consuming and tedious to make everything pass, but the main work was all done by you - thanks a mille!

Thank you very much @patrickvonplaten! 🤗
This is my biggest open source contribution so far, so I really appreciate you help fixing so many stuff and writing the test! (I merely copied the tf version and did some simple substitution.)

I am curious how do you run the template tests? It was confusing to run them locally because the it would overwrite the tracked files and leave unused files, making the work tree messy. I guess `commit -> test -> clean, reset --hard` would work, but I always fear that I would accidentally delete the wrong thing.
[MPNet] example of fine-tuning MPNet language model on domain specific corpus
Author: vdabravolski
Pull request: no

# 🚀 Feature request

I'd like to understand if it's possible to fine-tune MPnet model on domain specific corpus. I tried to run following script for MPNet and it seemed to be working (or at least on throwing any errors).
`python run_mlm.py --model_name_or_path microsoft/mpnet-base --dataset_name wikitext --do_train --output_dir tmp/mpnet-output --dataset_config_name wikitext-2-raw-v1`

However, since MPNet combines both MLM and PLM objectives, I'm not clear whether MPNet will actually train properly. 

## Motivation
MPNet establishes SOTA benchmarks on number of tasks. It could be useful to have some examples on how to fine-tune MPNet model on specific corpuses and downstream tasks.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Speech2Text] Slow tests are failing on master
Author: patrickvonplaten
Pull request: no

Currently the following tests are failing on master:

```
FAILED tests/test_modeling_speech_to_text.py::Speech2TextModelIntegrationTests::test_generation_librispeech
FAILED tests/test_modeling_speech_to_text.py::Speech2TextModelIntegrationTests::test_generation_librispeech_batched
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Wav2Vec2] Slow pretraining tests are failing on CPU
Author: patrickvonplaten
Pull request: no

The following tests are failing on CPU currently: 

```
tests/test_modeling_wav2vec2.py::Wav2Vec2ModelIntegrationTest::test_inference_integration
tests/test_modeling_wav2vec2.py::Wav2Vec2ModelIntegrationTest::test_loss_pretraining
```

-> check if they also fail on GPU. If not add a skip CPU decorator

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Which files are essential when customize and modify a specific pre-trained model？
Author: Doragd
Pull request: no

For some reason, I want to customize and modify a specific pre-trained model to adapt my experiments, e.g. BERT.  

However, there are so many unrelated files in the  `src/transformers`, and it really confuses me a lot. 

I only want to extract the essential source files to support my experiments without installing `transformers`.

**So I would like to know Which files are essential when customize and modify a specific pre-trained model.**

For example, for the BERT model, I think the files related to BERT are in the `src/transformers/models/bert`. 
Besides that, are there any source files essential?

Could you help me at your convenience? @LysandreJik  @patrickvonplaten @qqaatw 

Comments:
For example, I find https://github.com/microsoft/unilm/tree/master/unilm-v1/src/pytorch_pretrained_bert  use the `pytorch_pretrained_bert` without installing `transformers`. The file tree is very clean and clear. I also want to do this too, without unnecessary irrelevant source files.
![image](https://user-images.githubusercontent.com/26213546/131616444-d4fe68bf-5406-40ce-9914-03b525621678.png)

Hi,

Basically, it's not readily to extract `BertModel` or `BertTokenizer` as an independent module because they're inheriting some base classes resided at other folders, intending to provide some easy-to-use utilities such as `from_pretrained` method.

IMO, I think you can have `transformers` as a git submodule placed in your repository and modify the features you want accordingly.  This way you can still pull the latest features from upstream `transformers` and have your custom BERT (or other models).

> Hi,
> 
> Basically, it's not readily to extract `BertModel` or `BertTokenizer` as an independent module because they're inheriting some base classes resided at other folders, intending to provide some easy-to-use utilities such as `from_pretrained` method.
> 
> IMO, I think you can have `transformers` as a git submodule placed in your repository and modify the features you want accordingly. This way you can still pull the latest features from upstream `transformers` and have your custom BERT (or other models).

Thanks for your reply.
I plan to inherit the origin class and modify it. Maybe this is a compromise solution for me.

Do you think so? @qqaatw 



Yeah, I think this way is comparatively easy to achieve your goal.

Thanks.
[Flax] Fix BigBird
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

PyTorch BigBird was changed in a recent PR: https://github.com/huggingface/transformers/commit/ba1b3db70907b975b5ca52b9957c5ed7a186a0fa but the Flax version wasn't changed accordingly.

Thanks for spotting it @sgugger 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
I see -> yeah we can do this!
[Flax/run_hybrid_clip] Fix duplicating images when captions_per_image exceeds the number of captions, enable truncation
Author: edugp
Pull request: yes

# What does this PR do?

Do not duplicate images when the number of captions in an example is smaller than `captions_per_image`.
Truncate tokens when number of tokens exceeds `max_length`.

Currently, if an example contains a number of captions smaller than `captions_per_image`, the dataset will get scrambled, because the image paths are going to be duplicated but the number of captions is not.
For example, if an example contains a single caption and `captions_per_image` is equal to 2, one caption will be appended to the captions list, but two image path will be appended to the `image_paths list`. This results in mismatching file_path-caption pairs.
This pull request fixes this issue.

Currently, if a tokenized sentence contains a number of tokens > `max_length`, the dimensions of inputs within a batch won't match.
This pull request fixes this issue by truncating the tokens to max_length.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patil-suraj 



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
wav2vec2-large-xlsr-53 Tokenizer unable to load
Author: felixlut
Pull request: no

## Environment info

- `transformers` version: 4.10.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten 

## Information
Model I am using (Bert, XLNet ...):

I'm using the Wav2Vec2-XLSR-53 model, and it is specifically the loading part that doesn't work properly. The script below is provided on the model's huggingface page, and the tokenizer (2:nd line) does not load, while the actual model does work (3:d line). 

```python
from transformers import AutoTokenizer, AutoModelForPreTraining
  
tokenizer = AutoTokenizer.from_pretrained("facebook/wav2vec2-large-xlsr-53")

model = AutoModelForPreTraining.from_pretrained("facebook/wav2vec2-large-xlsr-53")
```
## To reproduce

Steps to reproduce the behavior:

1. Run the script above

## Expected behavior

I would expect the tokenizer to load.


Comments:
https://github.com/huggingface/transformers/blob/ecd5397106e243021ef28e65ca566881bb825bcb/examples/research_projects/wav2vec2/run_common_voice.py#L343-L366

The pretrained model has no tokenizer, you need to built your own one.

If you want to pretrain there is an example script here: https://github.com/huggingface/transformers/blob/master/examples/research_projects/wav2vec2/run_pretrain.py

Ok, thanks for the quick response!
src_lang/tgt_lang missing in mbart example
Author: zijwang
Pull request: no
Labels: name: WIP

## Environment info

- `transformers` version: 4.6
- Platform: linux
- Python version: 3.8
- PyTorch version (GPU?): 1.8

### Who can help

Models:

- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj

Library:
- tokenizers: @LysandreJik

## Information

Model I am using (Bert, XLNet ...): mbart

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)


## To reproduce

I am running the official example in the doc [here](https://huggingface.co/transformers/model_doc/mbart.html) under `Supervised training`. However, there is a warning of 

```
Keyword arguments {'src_lang': 'en_XX', 'tgt_lang': 'ro_RO'} not recognized.
```

when running

```
inputs = tokenizer(example_english_phrase, return_tensors="pt", src_lang="en_XX", tgt_lang="ro_RO")
```

Is this normal?

Comments:
Tagging @patrickvonplaten @patil-suraj @LysandreJik again in case you know what was going on here. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @zijwang , thanks a lot for spotting this.

The tokenizer API is changed a bit,  instead of passing `src_lang` and `tgt_lang` to tokenizes `__call__` method, we can now pass these when initializing the tokenizer, or we could set those properties as well. Here's a minimal example 

```python
from transformers import MBartForConditionalGeneration, MBartTokenizer

tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO")

# to change the src_lang
tokenizer.src_lang = "fr_XX"
```
Add TFSpeech2Text
Author: stancld
Pull request: no
Labels: name: WIP

# 🚀 Feature request
Add TensorFlow implementation of Speech2Text model.

## Your contribution

I'll try to do this.

**Reviewers:** @patil-suraj 

Comments:
Is this still in progress? I didn't see a WIP PR, but I'd like to use the TensorFlow version if possible.

Hello @will-rice, I have this in progress. I'll try to finish some missing components tomorrow and then open the PR for a review :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[docs] Update perplexity.rst to use negative log likelihood
Author: madaan
Pull request: yes

# What does this PR do?

`forward` returns the negative log likelihood. The document correctly defines and calculates perplexity, but the description and variable names are inconsistent, which might cause confusion. This patch fixes the description to reflect the correct behavior (i.e., use negative log-likelihood).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sashavor @sgugger 


Comments:
Finetuning GPT-2 on small datasets
Author: Elysium1436
Pull request: no

I have a relatively small dataset that i've scraped on my discord server. I wanted to make a gpt-2 chatbot with it, but the data is relatively small (3782031 characters counting the eos token). Training for a small number of epochs did nothing for any checkpoint related to gpt-2 (I tried distilbert, gpt-2, dialoGPT-small, and other), and training for a large number of epochs absolutely destroyed the whole model, it was barely able to generate coherent at all, it was either special characters, jumble, or nothing at all. I've tested the same script with a much larger dataset and it worked just fine, so I can only assume it's because of the dataset size. 
I was trying to find ways to freeze the gpt-2 base model and leave just the LMHead, but since the LMHead is somehow tied to the embedding layer, that wouldn't be possible... If there isn't a way to freeze the head of the model, what else should I do then? I've been trying to complete this personal project for quite a while now, and i'm out of options at this point. I'm using a custom TF script from the example folder on TPU, since the pytorch version makes the memory usage blow up on colab.

Comments:
I've finally found [this article](https://arxiv.org/pdf/2103.05247.pdf), and it seems promising. Going to try it out i'll say how it went.

For training-related questions, please refer to the [forum](https://discuss.huggingface.co/). We like to keep Github issues for bugs/feature requests.

For example, you can find all fine-tuning GPT-2-related questions [here](https://discuss.huggingface.co/search?q=fine-tuning%20gpt2).

Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.