noamaanMulla-03/transformers-issues huggingface dataset:
Improve performance docs
Author: lvwerra
Pull request: yes

# What does this PR do?

As discussed with @stas00 this PR does the following things:
- adds files for missing sections so contributors can better find the place to add content.
- adds a disclaimer at the beginning stating that a lot of general training information is in the single GPU training section
- fixes the link of CPU inference

Looking at the ToC I was wondering whether we should add subsections like it is done for the tasks to make the main ToC a bit slimmer. Otherwise we have the main performance docs (the entry point) there plus all the other sections (~8-10). What do you think?

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Sounds good @sgugger, I added the "Coming soon" and restructured the ToC.

Communications are difficult. 

what I proposed is to add normal documents - not "Coming Soon"

inside those documents they should all explicitly point the user to read the first document that is already filled out and that we will expand this other currently mostly empty doc with the missing material.

In other words e.g,, perf_infer_gpu_many.mdx should have:

1. read perf_train_gpu_one.mdx first
2. this document will be completed soon.

If the docs remain "Coming Soon" nobody will read them and will miss out on the already rich performance docs we have.

And the impetus for this change was that we went from a complete solution - all performance notes in one doc, to very incomplete solution, making it look like we only have advise for those with 1 gpu and doing training.

The original discussion back in winter was that all performance docs will be filled out, but it was dropped after the first document and no signs of new docs coming any time soon. So this proposal was my attempt to rescue the situation.

Ok I added references to each document and a small text explaining what will come there. Is that what you had in mind @stas00?
BLOOM minor changes on tokenizer
Author: younesbelkada
Pull request: yes

# What does this PR do?

- Attempts to fix minor issues with the BloomTokenizer
- Remove unused args
- Added a new attribute on `TokenizerTesterMixin` for models that are not agnostic to sequence length (typically models that use ALiBi positional embeddings)

Still need to discuss if it is worth it to force the padding side to the left

cc @SaulLu 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

We may merge this at the same time as https://github.com/huggingface/transformers/pull/17837 to make it a patch release of minor fixes

Let's not add a new attribute and change the common tests for one model only. You can override the test in the subclass of the main model tester, leaving the common tests as they are.
Translation troubleshooting 17459 
Author: F02934
Pull request: yes

# What does this PR do?

<!--

-->

<!-- Remove if not applicable -->

Fixes # (issue) Translation in Italian of Troubleshooting 17459


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@mfumanelli 



Comments:
@mfumanelli Hi, I don't know why it contains commits from previous pull request.

I will remake it because I made a mistake with the branch @mfumanelli
Add pipeline for cross-modal / uni-modal ranking
Author: ggoggam
Pull request: no

### Feature request

Given queries and keys, the proposed pipeline returns a ranked list of keys that are most similar to each respective query.
This pipeline should support uni-modal and cross-modal retrieval, i.e.

- Text-to-Text
- Text-to-Image
- Image-to-Text
- Image-to-Image

Prominent use cases would be:
- Using BERT family of models to perform text-to-text retrieval
- Using multi-modal models such as CLIP to perform any of the retrieval methods above. There can be multiple ranking methods for different multi-modal models, for instance
      - For VILT,  we can use [CLS] pooled image-text matching score for ranking
      - For CLIP, we can use logits_per_modality for cross-modal similarity score for ranking
      - For ALBEF (https://github.com/huggingface/transformers/issues/17224), we have a two-stage (coarse-to-fine) ranking (image-text similarity -> [CLS] pooled image-text matching score)


### Motivation

I was looking for a use-case for CLIP for cross-modal retrieval, but the current pipeline for CLIP does not seem to support cross-modal retrieval. I believe there is a demand for this pipeline.

### Your contribution

- I can help with the implementation once we polish the parameter definitions and outputs!

Comments:
I think there are some image-text retrieval capabilities already, such as [ViltForImageAndTextRetrieval](https://huggingface.co/docs/transformers/model_doc/vilt#transformers.ViltForImageAndTextRetrieval). But these can only work for a toy example set of queries and keys due to its interaction-based nature. A true retrieval (cross-model or not) would probably need `datasets` and `faiss`. I think that may be too complicated for for a `pipeline`?

- I believe there is no unified pipeline for cross-modal search that can be applied to different models.
- Thank you for clarifying. What I had in mind was more of a ranking than a retrieval since I do not expect to have index search baked into the pipeline.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update modeling_cvt.py type hints
Author: F02934
Pull request: yes

As shown in the colab notebook I added the missing type hints for " CvtForImageClassification
CvtModel
"

# What does this PR do?

Add missing type hints for CTV pytorch. #16059  following [this Colab notebook](https://colab.research.google.com/drive/1EvZTslb50yfRqIcXjCZFrbod4HrPdA0G?usp=sharing)

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@Rocketknight1 






Comments:
_The documentation is not available anymore as the PR was closed or merged._
Change no trainer image_classification test
Author: muellerzr
Pull request: yes

# What does this PR do?

I noticed that this test has been failing for the last few months, and it's due to the fact that on a single GPU or CPU the tests pass (we hit 50% accuracy), but on multi GPU it's a hit or miss whether or not it does (each epoch/batch is either 50% or 0%, and we happen to miss that coin flip each time).

This PR modifies the no_trainer test to mimic the equivalent pytorch example tests, and I can confirm it passes repeatedly on a single GPU, CPU, and multi GPU: https://github.com/huggingface/transformers/blob/main/examples/pytorch/test_pytorch_examples.py#L390-L417


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @NielsRogge 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger I've addressed the speed concern in the test, but you won't actually see it show up due to a bug in the scheduler step recalculation. I'll open up another PR for that fix and then merge this one after that's done
Nezha Pytorch implementation
Author: sijunhe
Pull request: yes

# What does this PR do?

This PR adds a pytorch implementation of the NEZHA model to transformers. [NEZHA](https://arxiv.org/abs/1909.00204) was introduced by Huawei Noah's Ark Lab in late 2019 and it is widely used in the Chinese NLP community. This implementation is based on the official pytorch implementation of NEZHA and the current BERT pytorch implementation . The model checkpoints are also from the [official implementation](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA-TensorFlow). 

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Since the model is quite similar to bert, maybe @LysandreJik?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

ready for review! I'll upload the rest of the pre-trained models later today

addressed all the comments from @sgugger and uploaded the two remaining models. Ready for a final round of review.

Thanks again for your contribution!
Index RNG states by global rank in saves
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out in #17829, in a multi-node training the `Trainer` saves the RNG states with the same filenames in the various nodes. This causes problems when the nodes share the same file system, so it's easier to just save each file indexed by global rank instead.

Fixes #17829 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix broken test for models with batchnorm
Author: Rocketknight1
Pull request: yes

One of the Keras tests assumed that fitting a model for one iteration with a learning rate of zero would not change any weights. This is not true for `BatchNorm`, which updates its running means and variances regardless! As a result, the model after the iteration had slightly different outputs, which caused the test to be very flaky. We now reinitialize the model after the single training epoch to make sure this doesn't happen.

Comments:
Related to #17427, cc @amyeroberts .

_The documentation is not available anymore as the PR was closed or merged._

🧠 

Give @amyeroberts the brain emoji for that one - she identified the whole problem, I just fixed the test!
Properly calculate the total train iterations and recalculate num epochs in no_trainer scripts
Author: muellerzr
Pull request: yes
Labels: name: Examples

# What does this PR do?

This PR fixes a situation where your `max_train_steps` was less than that of a single epoch, but yet the script would still continue through the entire first epoch due to how the number of batches recalculation was performed. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger new timings when compared to old. Main decreases repeatedly were seen between image_classification, swag, and squad (though squad is an iffy one. the previous two I can guarantee):

<html><body>
<!--StartFragment--><google-sheets-html-origin>

Example | Before | After
-- | -- | --
image_classification | 99.7 | 40.55
swag | 65.21 | 55.31
squad | 59.45 | 41.67
clm | 37.45 | 35.69
ner | 28.34 | 25.51
glue | 21.88 | 19.35
mlm | 18.52 | 15.47

<!--EndFragment-->
</body>
</html>
fp16+xlnet did not gain any speed increase 
Author: fyubang
Pull request: no
Labels: name: wontfix

Hi, 
I tried fp16 + xlnet, it did not work. 
when I set opt_level='O2', the memory was half, but it was much slower than fp32.
when I set opt_level='O1', the memory was original, and it has similar speed with fp32.

Environment: v100, cuda, 10.1, torch 1.1
The environment is ok, because I tried bert + fp16 and it was much faster than fp32.
I thought it is the problem of torch.einsum, but I am not that sure.

I used the code here to test: https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_squad.py

Comments:
XLNet makes heavy use of `torch.einsum()` but I'm not sure this method is fp16 compatible.
It's also quite slow currently so maybe in the mid/long-term it would be good to change these einsum to standard matmul. I won't have time to do that very soon though.

As as a suggestion, you can add ```apex.amp.register_half_function(torch, 'einsum')``` somewhere near the top of your driver script (examples/run_squad.py for instance).

This forces `amp` to cast the inputs to einsum to `torch.half` before executing, allowing you to get the perf benefits of fp16 + TensorCores when appropriate.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, @slayton58, could `torch.einsum` be automatically processed by `apex.amp` now?
Your response will be appreciated!
Bad readme for HF model codeparrot/codeparrot-small
Author: shi-kejian
Pull request: no
Labels: name: bug

### System Info

```shell
Not system-dependent 
https://huggingface.co/codeparrot/codeparrot-small
The README is not updated
```


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import AutoTokenizer, AutoModelWithLMHead
  
tokenizer = AutoTokenizer.from_pretrained("lvwerra/codeparrot-small")
model = AutoModelWithLMHead.from_pretrained("lvwerra/codeparrot-small")

### Expected behavior

```shell
The model loading should be successful
The model card is "codeparrot/codeparrot-small" but in example usage it's "lvwerra/codeparrot-small". 
A simple update to the model card README will do
```


Comments:
Hi!

Thanks for reporting. We now have a [discussion and PR feature](https://huggingface.co/codeparrot/codeparrot-small/discussions) on the hub, meaning that you can directly open an issue on the hub for a particular repository. So feel free to ping them there!

Thanks for reporting - it is fixed now!

Closing the issue in that case!
SimMIM output num_channels should not be hardcoded
Author: ccaapton
Pull request: no

### Feature request

In all 3 simmim models,  reconstructed images channel is hardcoded as 3. This should be configurable as num_channels
```
deit/modeling_deit.py
swin/modeling_swin.py
vit/modeling_vit.py

nn.Conv2d(in_channels=config.hidden_size, out_channels=config.encoder_stride**2 * 3, kernel_size=1)
```
@NielsRogge 

### Motivation

I'm training a grayscale model, but the reconstructed image has different dimension as the input image.

### Your contribution

None

Comments:
Improve vision models
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR improves the vision models by:
- removing `to_2tuple`
- sanity checking whether the channel dimension of pixel values provided to the model match with `config.num_channels`
- replacing hardcoded 3 with `config.num_channels` for `xxxForMaskedImageModeling` models (fixes #17727) 
- replacing hardcoded 3 by `config.num_channels` in Flax models (ViT, BEiT)

To do:

- [x] ViT
- [x] BEiT
- [x] DeiT
- [x] Swin
- [x] PoolFormer
- [x] DPT
- [x] YOLOS
- [x] ViLT
- [x] GLPN
- [x] DPT
- [x] Data2VecVision
- [x] MaskFormer
- [x] ViTMAE
- [x] TF and Flax implementations
- [x] Corresponding test files
- [x] add more Copied from statements (e.g. DropPath)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Improve encoder decoder model docs
Author: Threepointone4
Pull request: yes

# What does this PR do?
This PR improves the documentation of encoder decoder model.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
       Issues [link](https://github.com/huggingface/transformers/issues/16135)


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@patrickvonplaten 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @



 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Agree with the reviews of @ydshieh and @NielsRogge ! @Threepointone4 do you want to apply them ? Think we can merge after :-)

Great job @Threepointone4 ! Merging :-)
Fix Splinter test
Author: ydshieh
Pull request: yes

# What does this PR do?

`test_multi_gpu_data_parallel_forward` is not mean to run for `SplinterForQuestionAnswering`, due to the number of question tokens might be different on different replicas. This PR skips this test for `SplinterForQuestionAnswering`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
BLOOM enhance alibi creation
Author: younesbelkada
Pull request: yes

# What does this PR do?

Thanks to @justheuristic 's contribution alibi tensor is better created/communicated during the forward pass. The tests seem to pass but still stays as an experimental feature.

cc @justheuristic

This probably will break with accelerate offloading because when initialising alibi tensor we do it only once at the beginning of the forward pass with the device of the first hidden states. In the previous version we used to dynamically change alibi's `device` which was fine for accelerate offloading


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Great thanks ! I think that you are right :) will merge it as soon as the lights are all green 🟢 

It looks like a bad rebase happened, moved the PR at: #17866
[CodeGen] support device_map="auto" for sharded checkpoints
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR adds the `_no_split_modules` attribute in `CodeGenPreTrainedModel` to be able to load the sharded checkpoint with `device_map="auto"`

cc @rooa 

Comments:
_The documentation is not available anymore as the PR was closed or merged._