noamaanMulla-03/transformers-issues huggingface dataset:
fix position bias related logic after prune heads in T5 model
Author: J-shang
Pull request: yes

# What does this PR do?

A follow up pr after #17968

If the attention layer `self.has_relative_attention_bias == False`, then the position bias shape will be wrong, the head number should be the original model (before prune heads) head number `self.n_heads + len(self.pruned_heads)`

## Who can review?

- t5: @patrickvonplaten @patil-suraj

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19097). All of your documentation changes will be reflected on that endpoint.

Maybe of interest to @ArthurZucker :)

Hey, before diving a bit deeper, sorry for the long delay, and thanks for the PR. 
Would you mind adding a test? 🤗 I can take care of it otherwise! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing in favor of #20106. Thanks for your contribution
Generate: add Bloom fixes for contrastive search
Author: gante
Pull request: yes

# What does this PR do?

Bloom has a different cache format, where the batch size and the number of heads are packed in a single dimension. Contrastive search needs to manipulate the cache at the batch dimension, so naturally it fails.

This PR adds functionality to convert Bloom's cache back and forth between its own format and the standard cache format. Then, propagates the use of these new functions to places where the conversion logic was already being used, and finally fixes Bloom's contrastive search.

All slow tests are passing.
____________________________________

This fix was also requested [here](https://huggingface.co/spaces/joaogante/contrastive_search_generation/discussions/1#636e23d1c441b42489215026)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20213). All of your documentation changes will be reflected on that endpoint.

(rebasing to include #20200 in CI, the related test was failing)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20213). All of your documentation changes will be reflected on that endpoint.

Stumbled on the same issue and found this fix. Thanks lots!
BLOOM past_key_values bug
Author: rpryzant
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): 2.9.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yers
- Using distributed or parallel set-up in script?: no


### Who can help?

@LysandreJik 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, AutoModel, MT5ForConditionalGeneration
from transformers import AutoModelForCausalLM, AutoTokenizer


model_name = 'bigscience/bloomz-7b1'
model = AutoModelForCausalLM.from_pretrained( model_name, device_map='auto')
print(model.config)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = 'this is a'
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(input_ids=encoded_input['input_ids'].cuda())

print([x.size() for x in output.past_key_values[0]])


### Expected behavior

As per the documentation and generation interface (https://huggingface.co/docs/transformers/main_classes/output) past_key_values should have tuples of 4-dimensional tensors with shape (batch_size, num_heads, sequence_length, embed_size_per_head). 

The BLOOM models returns 3-dimensional tensors, for example the first tuple in the example code has tensors of shape ( torch.Size([32, 128, 3]), torch.Size([32, 3, 128]) ).

This means that the code crashes when you try to use some decoding algorithms, for example the new contrastive decoding algorithm. 

Comments:
Closing (fixed by https://github.com/huggingface/transformers/pull/20213)
Add Mask2Former
Author: shivalikasingh95
Pull request: no
Labels: name: New model

### Model description

Mask2Former is a single architecture for panoptic, instance and semantic segmentation.

**Mask2Former Paper Abstract**:
Image segmentation is about grouping pixels with different semantics, e.g., category or instance membership, where each choice of semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).


### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/abs/2112.01527

Github repo (and weights): https://github.com/facebookresearch/Mask2Former

Comments:
@NielsRogge I'd like to work on adding this model if no one is working on it yet?

cc'ing @alaradirik, yes we're planning to add this model. If you're interested in it, feel free to get started with a draft PR. Note that we already have [MaskFormer](https://github.com/huggingface/transformers/tree/main/src/transformers/models/maskformer) implemented, and I've heard Mask2Former only adds minor modifications.

Could you give me your email address, such that we can add you on Slack for easier communication?

Thanks @NielsRogge that would be great! You can use this email (shivalikasingh95@gmail.com) to add me on Slack!

I'll get started on a draft PR. But, I may need some guidance as this is my first time contributing to transformers.
I'll get started by understanding the [MaskFormer](https://github.com/huggingface/transformers/tree/main/src/transformers/models/maskformer) implementation.

Hi @NielsRogge just a gentle reminder to add me on slack :)

Hi, I've pinged someone to add you.

Hello.
Any updates about the Mask2Former integration? Thanks

Hi @ArthurOuaknine I'm working on it. Currently there is an open PR on my transformers fork. Will try to close this in next couple of days. 

Thanks for your work, it will definitely help :)
Incorrect Document Content in BlenderBot Tokenizer
Author: chujiezheng
Pull request: no

`The BlenderBot tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not.` However, the examples in BlenderBot Tokenizer (`BlenderbotTokenizer`) are the same:

https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/models/blenderbot/tokenization_blenderbot.py#L105

The same issue also occurs in `BlenderbotTokenizerFast`:

https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/models/blenderbot/tokenization_blenderbot_fast.py#L64

Comments:
Might be of interest to @ArthurZucker 

Okay, this is simply because the model used, `BlenderbotTokenizer.from_pretrained("facebook/blenderbot-3B")` has the attribute `add_prefix_space` set to `True` by default. If you set it to `False` we have the expected different output. 
Let me open a PR to fix this.
remove two tokens that should not be suppressed
Author: ArthurZucker
Pull request: yes

# What does this PR do?
As mentionned in #20123, two tokens `'` and `-` were supressed. This was probably from the [late commit](https://github.com/openai/whisper/commit/8cf36f3508c9acd341a45eb2364239a3d81458b9) that appeared after I started working on it. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[ASR Examples] Update README for Whisper
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Summarises changes from https://github.com/huggingface/transformers/pull/19519 and the blog post at https://huggingface.co/blog/fine-tune-whisper, providing examples for fine-tuning Whisper using the example script `run_speech_recognition_seq2seq.py`.

Required some re-jigging of the Speech-Encoder-Decoder Model examples in the section "[Automatic Speech Recognition with Sequence-to-Sequence](https://github.com/huggingface/transformers/pull/20230#sequence-to-sequence)".

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add padding image transformation
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds padding to the image transforms library. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19838). All of your documentation changes will be reflected on that endpoint.

@NielsRogge I've asked for re-review as the logic has changed a bit from when you first reviewed. 

> Thanks for adding! Do you think it'd be useful to add a pytorch padding vs our implementation equivalence test?

@NielsRogge What do you think should be covered wrt equivalence - is there logic you want to make sure is always aligned between the two? These transformations aren't meant to be a np copy of the torch library so there isn't a 1:1 mapping. 

@NielsRogge I'm going to merge. If we decide to add the equivalence tests I'll add in a follow up PR. 

Ok fine, it's just that I was working on a model (#19784) that leverages torch.nn.functional.pad as seen [here](https://github.com/mv-lab/swin2sr/blob/7eeebfba849bbc934ea254ec4cfa8e9d6fc0672c/models/network_swin2sr.py#L891). So it'd be nice to check equivalence between PyTorch and our NumPy implementation.
Pin TensorFlow as new release breaks `pip install transformers["all"]`
Author: sgugger
Pull request: yes

# What does this PR do?

This PR pins TensorFlow for now, as it seems there is no version of `tensorflow-text` compatible with it.

Comments:
Merging as it unlocks the styling checks, will further fix the rest of the failing tests in a separate PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20313). All of your documentation changes will be reflected on that endpoint.
Implement pytorch model with BetterTransformer in pytorch1.12?
Author: Lucas-Luoling
Pull request: no

### Feature request

BetterTransformer is a fastpath for the PyTorch Transformer API. The fastpath is a native, specialized implementation of key Transformer functions for CPU and GPU that applies to common Transformer use cases.

### Motivation

Faster.

### Your contribution

Pytorch official blog: https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/

Comments:
Hey @luoling1993, we're currently in talks with PyTorch to see how to best approach this. cc @erichan1

Hey there @luoling1993! I'm from the PyTorch team - we are working on this. My working PR https://github.com/erichan1/transformers/pull/2

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @luoling1993 !
This has been implemented now. Please check: https://medium.com/pytorch/bettertransformer-out-of-the-box-performance-for-huggingface-transformers-3fbe27d50ab2
`test_run_squad_no_trainer` is flaky
Author: amyeroberts
Pull request: no

### System Info

Environment: Circle CI image running `examples_torch`





### Who can help?

@muellerzr 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Due to the nature of the issue, unfortunately it cannot be reliably replicated. Examples of this occurring can be found here:

* https://app.circleci.com/pipelines/github/huggingface/transformers/49621/workflows/14c25312-58a5-4b0b-8b41-6c5bec668043/jobs/593213
* https://app.circleci.com/pipelines/gh/huggingface/transformers/49224/workflows/fbae76ab-9259-4695-bb06-475357172587/jobs/589262

### Expected behavior

Occasionally, `test_run_squad_no_trainer` fails on CI runs, even when the PR is not touching code related to the test e.g. I would expect the [output of the tested run](https://github.com/huggingface/transformers/blob/a23819ed6ab852df6d8f04815306440531418260/examples/pytorch/test_accelerate_examples.py#L199), `result`, to be either deterministic or pass reliably when the are otherwise no other changes to the code. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

ping @muellerzr ;-)
Fix flakey no_trainer test with seed
Author: muellerzr
Pull request: yes
Labels: name: Tests

name: Examples

name: bug

# What does this PR do?

This PR adds a seed param to the squad no trainer test to ensure reproducibility. Ran 5x times on single and multi GPU to ensure that the test will pass. 

Fixes # (issue)

Closes https://github.com/huggingface/transformers/issues/19733


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
TF: future proof our keras imports
Author: gante
Pull request: yes

# What does this PR do?

On the release notes for [TF 2.11](https://github.com/tensorflow/tensorflow/releases/tag/v2.11.0), we can read:

```
tensorflow/python/keras code is a legacy copy of Keras since the TensorFlow v2.7 release, and 
will be deleted in the v2.12 release. Please remove any import of tensorflow.python.keras and 
use the public API with `from tensorflow import keras` or `import tensorflow as tf; tf.keras`.
```

On top of that, `hdf5_format` was moved from `keras.saving` to `keras.saving.legacy` in v2.11 (this one is not in the patch notes). 

This PR prepares us for v2.11 and beyond, while keeping retrocompatibility through gated imports.

Note: I've tested the v2.11 imports locally :) (the CI is running against v2.10)

Comments:
Other than changing `save_attributes_to_hdf5_group` to `hdf5_format.save_attributes_to_hdf5_group`, no changes are needed now -- but they will be in v2.12, so might as well 🤷 

_The documentation is not available anymore as the PR was closed or merged._
RAG performance on WebQuestion dataset lower than expected 
Author: lkfafds
Pull request: no

Hi, I recently fine-tuned the Rag model (based on Rag-token-base) in WebQuestion, but the EM score is only 28. The performance in the paper is 45. Do you have any saved models that can match the performance, or does there any tricks during the fine-tuning?

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only.
🚨 🚨 🚨 Fix ViT parameter initialization
Author: alaradirik
Pull request: yes

# What does this PR do?
This PR aims to rectify the discrepancy between the training performances of HF and Timm ViT implementations.

- Initializes torch and flax ViT dense layer weights with trunc_normal instead of normal (consistent with the TF implementation.
- Initializes cls_token and positional_embeddings with trunc_normal
- Updates DeiT copy

Partially fixes # ([19305](https://github.com/huggingface/transformers/issues/19305))


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
     This issue was brought up over `[here.](https://github.com/huggingface/transformers/issues/19305)`
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?




Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for your PR! For such PRs, please ping both @sgugger and @amyeroberts who are better suited to do a review than I am. Thank  you!

@alaradirik I believe setting eps in layernorm to 1e-6 rather than 1e-12 is also important as mentioned in https://github.com/huggingface/transformers/issues/19305 by @rwightman

Yes, @alaradirik, but I think for newer users, or anyone new to ViT or just setting up ViT to train, it would be much better if the default was set to 1e-6 rather than 1e-12 so that they don't have to relook for bugs and most of the time the eps value will not be the first thing they look for, or worst case last thing.

Also, shouldn't Kaiming initialization be used for the nn.Conv2d rather than .normal_() initialization in the class ViTPreTrainedModel or any class that directly inherits from PretrainedModel? And the biases of the nn.Conv2d in ViT should be initialized the same way as PyTorch? (https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv2d) @alaradirik @LysandreJik @NielsRogge @sgugger
ZeroDivisionError: division by zero
Author: JackKuo666
Pull request: no
Labels: name: bug

### System Info

```shell
transformers v4.9.2
python v3.8
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

when i was training my bert model use  ` CustomIterableDataset(torch.utils.data.IterableDataset)`, rise the follow error:
```
Traceback (most recent call last):
  File "train.py", line 102, in <module>
    main(args)
  File "train.py", line 85, in main
    trainer.train() 
  File "/home/jack/anaconda3/envs/py38trans4.19.2/lib/python3.8/site-packages/transformers/trainer.py", line 1316, in train
    return inner_training_loop(
  File "/home/jack/anaconda3/envs/py38trans4.19.2/lib/python3.8/site-packages/transformers/trainer.py", line 1627, in _inner_training_loop
    self.state.epoch = epoch + (step + 1) / steps_in_epoch
ZeroDivisionError: division by zero
```
i find that this line have a litter bug in [`/transformers/trainer.py`, line: 1516](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1526):
```
            steps_in_epoch = (
                len(epoch_iterator)
                if len_dataloader is not None  
                else args.max_steps * args.gradient_accumulation_steps
            )
```
The logic of  ` if len_dataloader is not None`  should be to determine whether `train_dataset` is  `IterableDataset`，But  whether `train_dataset` is  `IterableDataset` or not, len dataloader is 1，so, it should be change as follows:
```
            steps_in_epoch = (
                len(epoch_iterator)
                if not isinstance(self.train_dataset, torch.utils.data.IterableDataset) 
                else args.max_steps * args.gradient_accumulation_steps
            )
``` 

### Expected behavior

```shell
i will fix this bug
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Did you fix this problem,I'm having the same bug