noamaanMulla-03/transformers-issues huggingface dataset:
Skip pt/flax equivalence tests in pytorch `bigbird` test file
Author: ydshieh
Pull request: yes

# What does this PR do?

#21023 fixed random attention issue in Flax bigbird model, and skipped the pt/flax equivalence tests in flax bigbird test file with
```txt
reason="Current Pytorch implementation has bug with random attention -> it always uses it not matter if we are in eval/train mode"
``` 
We need to skip the pt/flax equivalence tests in **pytorch** bigbird test file too.

Currently on `main`, the tests fail

https://app.circleci.com/pipelines/github/huggingface/transformers/63217/workflows/5d512271-f535-44be-a2ec-b95024f8f165/jobs/780069

Comments:
cc @Bearnardd

_The documentation is not available anymore as the PR was closed or merged._

Good catch @ydshieh! I will fix pytorch's big bird implementation today/tomorrow.

@Bearnardd Thank you 🤗 . Luckily, there is no `TFBigBirdModel` 😆
🚨🚨🚨 [`Blip`] remove labels masking
Author: younesbelkada
Pull request: yes

# What does this PR do?

Addresses https://github.com/huggingface/transformers/pull/23004#issuecomment-1523776082

This PR aims to harmonize the training procedure for most of recent additions in `transformers`. It should be users' responsibility to fill_mask the padding tokens of the labels with the correct value. This PR addresses the issue that was raised by other architectures such as Luke or Pix2Struct

However I realize that even if this patch is applied, we still mask fill the labels with -100 [here](https://github.com/huggingface/transformers/blob/9435cc6670b7b8656b33e8ff28d3bbe9bafbca9d/src/transformers/models/blip/modeling_blip.py#L1133), similarly as in [T5](https://github.com/huggingface/transformers/blob/9435cc6670b7b8656b33e8ff28d3bbe9bafbca9d/src/transformers/models/t5/modeling_t5.py#L868)

I would love your feedback on this, @amyeroberts !


Comments:
_The documentation is not available anymore as the PR was closed or merged._
added type hints for blip_text model
Author: iamarunbrahma
Pull request: yes

# What does this PR do?

Added type hints for ```blip_text``` pytorch model as tasked in #16059

@Rocketknight1 Could you review this?

Comments:
Error logs suggest that to use ```pip install "black[jupyter]"``` but unable to understand what to do after that. @Rocketknight1 could you suggest how to fix test failures?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23020). All of your documentation changes will be reflected on that endpoint.

Hi @iamarunbrahma, `black` is a code formatting tool. After you've installed it, run `make style` or `make fixup` in the `transformers` directory. This should reformat your file for you and get the tests in the CI to pass.

@Rocketknight1 getting this error while running ```make fixup```:
```
/bin/sh: line 3: black: command not found
/bin/sh: line 4: ruff: command not found
make: *** [Makefile:10: modified_only_fixup] Error 127
```
 and while running ```make style```:
```
make: black: No such file or directory
make: *** [Makefile:68: style] Error 127
```
I have installed both ```black``` and ```ruff```

@iamarunbrahma looks like `black` wasn't installed after all! The easiest way to get it is to `cd` to the `transformers` source directory you're working on and `pip install .[quality]`. You can also just type `pip install transformers[quality]` anywhere, but this may get slightly older versions of the code quality tools (it's usually fine though).

Once the tools you need are installed, `make fixup` or `make style` should work.
Fix model parallelism for `BridgeTower`
Author: ydshieh
Pull request: yes

# What does this PR do?

Make `BridgeTower` work with model parallelism. The test `test_model_parallelism` is still skipped as tiny version hits edge cases.
With larger values of `hidden_size`, `num_hidden_layers` etc., it works now while failed before.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: prepare assisted generation for release
Author: gante
Pull request: yes

# What does this PR do?

This PR makes a few final adjustments to assisted generation before its release, including:
1. Merge previously named step 7 [the forward pass after matching with assistant tokens] into step 6 [slicing variables based on the number of matches] -- the variables are already computed in step 3 [selecting the model's next tokens based on the logits], so the code becomes more concise and it helps me explain what's going on more easily. See the (partially bugged) gif below, which is WIP for the blog post.
2. Swaps the order of step 6 [slicing variables] with step 5 [updating the number of candidates for the next iteration] -- makes more sense that the update step for the next iteration is the last one :)
3. Better variable names and improved comments (so the implementation becomes self-documenting)

![assisted_gen](https://user-images.githubusercontent.com/12240844/235239007-d9d0224a-063e-49f3-9c1f-6b90b818345e.gif)


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Using FNet model in Encoder Decoder Models
Author: Parmida-Granfar
Pull request: no

Hello every one


I want to train a model with an encoder with FNet model and decoder with another transformer model like gpt. I searched and found EncoderDecoderModel in hugging face library which makes such changes easier. I put the link below:

[(https://huggingface.co/transformers/v3.5.1/model_doc/encoderdecoder.html#transformers.EncoderDecoderModel](url)
. I want to use the FNet model in Encoder Decoder Models but cannot. because I face this error: 

> TypeError: forward() got an unexpected keyword argument 'attention_mask'

I understand that this is because FNet does not have attention, but I do not know how to resolve it.

I searched the internet and found out that EncoderDecoderModel does not work for all transformer models and I wanted to know why and wanted to suggest adding FNet.

Comments:
Hi @Parmida-Granfar 

I would suggest to make the necessary changes to the `EncoderDecoderModel` and/or `FNetModel` code according your own need.
As you have already observed `FNet does not have attention` (very nice finding 💯 ), you can remove the following line

https://github.com/huggingface/transformers/blob/5fd4e3c87c685fba2dd9615be62131748a8b5ee3/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L592

and see if everything works then. If you still need any further help, I am more than happy to answer (but in a thread on [Hugging Face Forums](https://discuss.huggingface.co/) instead)

With all the modeling architectures coming out at a fast pace nowadays, it's not practical and realistic to make composite modeling like `EncoderDecoder` to handle all pairs of encoder and decoder models. But the good thing is the code is open source, and everyone can make changes to it :-). 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Why there is no data send to data_collator?
Author: Luoyang144
Pull request: no

### System Info

- `transformers` version: 4.27.0.dev0
- Platform: Linux-3.10.0-514.el7.x86_64-x86_64-with-centos-7.3.1611-Core
- Python version: 3.7.13
- Huggingface_hub version: 0.12.0
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

**I'm tring to us code below**


And get error `KeyError: 'seq2seq'`, after output feature input to data_collator, get output like follow:
`[{}, {}, {}, {}]`
Why this happened? I output train dataset and get correct result, but when using `trainer.train()` it can get data I need.
[train.txt](https://github.com/huggingface/transformers/files/11035839/train.txt)


### Expected behavior

How can I send data in training process? Thanks for your help.

Comments:
That's related to the data you are preprocessing, not the Transformers library or its examples. There is simply no `"seq2seq2"` in the features you prepare with your function. I suggest posting on the [forums](https://discuss.huggingface.co/) to get help from the larger community.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Training wav2vac2 requires a lot of compute power  
Author: ngawang88
Pull request: no

I am trying to Fine tune wav2vac2 model for my national language and I have 15k data points but while trainings my system could only train on 1k datapoints and if I increase the datapoints my system either crashes or I get CUDA out of memory. so I am wonder is their any other alternatives ??

secondly can I first train on 1k data and then save the model locally and again load the model train with another 1k new data to improve my model ?? will it actually work ?  

Comments:
Hi @ngawang88, thanks for opening an issue! 

This is a question that is better suited to the [forums](https://discuss.huggingface.co/). We try and keep github issues reserved for bugs and feature requests. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Moved labels to enable parallelism pipeline in Luke model
Author: katiele47
Pull request: yes

# What does this PR do?
Fixes #22561 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Please let me know if there's anything I need to correct! Thanks

@sgugger 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22907). All of your documentation changes will be reflected on that endpoint.

@katiele47  - thanks for the PR! 

Apologies, I reviewed another PR implementing the changes for Luke quickly this morning without properly reading the issue and realising this PR was also open. As the PR #22909 notes, there was just a small change that needed to happen on L2232, updating `logits` -> `reshaped_logits`. Otherwise the PR all looked good and after updating would have been merged :) 

I'm sorry for my mistake - I hope this doesn't discourage you from contributing and we welcome any PRs that you'd like to open in the future. 

@sushmanthreddy Anyone in the community is able to review PRs. If you spot something in the code that needs updating, could you comment directly on the PR instead of opening another one? 

@amyeroberts . I don't see, where I have gone wrong. I am new to open source and actually have worked on this issue before the @katiele47 did, but I haven't just mentioned a reviewer to review.
this is the proof for that [link](https://github.com/huggingface/transformers/pull/22900/files), actually, I have made the same changes needed just due to branch conflicts that haven't kept the proper pr.
anyways sorry for that if my mistake is there, I hope u understand 

Hi @amyeroberts thanks for spotting the small change and no worries! Now that it has been fixed by @sushmanthreddy PR #22909 should I close this PR? 

@katiele47 Yes, this PR can now be closed. Thanks again for opening - we look forward to future contributions!

@sushmanthreddy You haven't made a mistake, don't worry :) It's just a request to make it easier for maintainers to keep track of issues and PRs in the codebase.
added type hints in graphormer
Author: dewasahu2003
Pull request: yes

@Rocketknight1 I added type hints for `Graphormers` for pytorch as described in [issue #16059 ](https://github.com/huggingface/transformers/issues/16059)



Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23067). All of your documentation changes will be reflected on that endpoint.
WhisperTokenizer for two languages at once.
Author: BakingBrains
Pull request: no

In the blog https://huggingface.co/blog/fine-tune-whisper I see the code piece.

```tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-small", language="Hindi", task="transcribe")```

What if I want to include both French and Hindi?

Any suggestions here?

Thanks and Regards.

Comments:
cc @ArthurZucker and @sanchit-gandhi 

Hey @BakingBrains! You'll have to set the prefix tokens each time you switch language. You can do this using the [`.set_prefix_tokens`](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperTokenizer.set_prefix_tokens) method, e.g. for French:
```python
tokenizer.set_prefix_tokens(language="French", task="transcribe")
# encode French target text to label ids 
batch["labels"] = tokenizer(batch["sentence"]).input_ids
```
Then for Hindi:
```python
tokenizer.set_prefix_tokens(language="Hindi", task="transcribe")
# encode Hindi target text to label ids 
batch["labels"] = tokenizer(batch["sentence"]).input_ids
```

@sanchit-gandhi Thanks a lot. Can't we use both language at once. Like for example I have an audio file where a person speaks Hindi and in between switches the language to French. How can I use the tokenizer in this case?

Thanks and Regards

I remember trying with a file that contained both French and English and whisper just transcribe the french part as if it was english. The same happens when you try to transcribe some audio that is in english but force the language code to another language: it will write english phonemes corresponding to the sounds that it hears. 
- Now if you want to have a batch, and in the batch you have different language, you can use the `generate` method and provide a batch of `decoder_input_ids` and set the `forced_tokens` to be None. 
- If you have another language in the middle, I suggest trying with the `retur_timestamp` option, which will split the audio. My best recommendation is to do something similar to what openAI's long audio decoding strategy does: when you reach the end of a timestamp, you re-generate. But this is gonna be slow as you don't know if the next sentence is in english or hindi, you cut the next sentence. 

@ArthurZucker Thanks, I have been thinking the same.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Have you considered using falsh attention to speed up?
Author: macheng6
Pull request: no

### Feature request

using flash attention to speed up

### Motivation

none

### Your contribution

none

Comments:
cc @younesbelkada 

Hi @macheng6 
Thanks a lot for your interest in this! 
Indeed there is a similar integration to this called `BetterTransformer` that uses flash attention in the backend I believe. 
This support most of the encoder and decoder models (if you use the `main` branch of `optimum`, please refer to this documentation page: https://huggingface.co/docs/optimum/bettertransformer/overview

cc @fxmarty @michaelbenayoun 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
torch_compile fail with multi-gpus on samples
Author: frank-dong-ms
Pull request: no

### System Info

platform: ubuntu 20.04
Pytorch version: nightly
transformer version: build from source

### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. ubuntu 20.04, python 3.8, install Pytorch nightly, transformers from source
2. install necessary dependency
3. go to official example:  transformers/examples/pytorch/text-classificationg
4. Run sample with torch_compile: python run_glue.py --model_name_or_path finiteautomata/bertweet-base-sentiment-analysis --task_name mnli --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 1 --learning_rate 2e-5 --num_train_epochs 1 --overwrite_output_dir --output_dir ./outputs/ --per_device_eval_batch_size 1 --seed 1137 --fp16 True --max_train_samples 1000 --**torch_compile**
5. Got exception from pytorch: Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'.

try to use single gpu as: export CUDA_VISIBLE_DEVICES=0 adn rerun the sample command, this time it runs without issue. 
Looks like if multi-gpu found, torch model will be also wrapped with nn.DataParallel: https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1387 this seems cause issue with torch compile.


### Expected behavior

Sample runs without issue with torch.compile with multi-gpu

Comments:
I don't think `torch.compile` support `DataParallel`. You should launch your script in a distributed fashion using `torchrun`.

Make sense, in that case I think transformers trainer should be refined to let torch.compile work on multi-gpu

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
line 714  bsz, seq_len = input_shape may crash in CLIPTextTransformer
Author: qilei123
Pull request: no

### System Info

Hi, there,
When input_ids is more than 2 dimentions , the  input_shape = input_ids.size() will give more than 2 numbers. Then bsz, seq_len = input_shape will crash.
I don't know if I was right. Just face the issue and change the code in my machine and it works.


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

when use CLIPTextTransformer, pass a input_ids with more than 2 dimentions.

### Expected behavior

crash: unpack more than expected 

Comments:
Hi @qilei123, thanks for raising this issue.

Could you share a minimal code sample which reproduces the error and information about the environment the code was run in (run `transformers-cli env` to get this info)?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix grammar error in summarization pipeline
Author: SKaplanOfficial
Pull request: yes

# What does this PR do?

Fixes a minor grammar error I noticed while using the summarization pipeline.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed.

- pipelines: @Narsil

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you very much !