noamaanMulla-03/transformers-issues huggingface dataset:
[torch] remove deprecated uint8 in favor of bool
Author: ArthurZucker
Pull request: yes

# What does this PR do?
This should fix #21013. 
Have not ran the tests yet, so leaving as draft

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Waiting for the CI to work again. 

Ok now the real clean up is starting, have to identify all attention masks, vs the causal masks. I think it is a good thing, will help understanding 


The way we handle attention mask is not normalised throughout the library. 
Diving deeper might not be the best idea as there can be some backward incompatibilities: 
- if we modify all the methods that handle attention masks that are define at the beginning of most of the modeling files, it is gonna break things for potential users. 
- if we modify the output of the tokenizer, it is the same.

In conclusion the simplest fix is to dig where `uint8` are used, otherwise ignore. Whenever the uint8 are converted to `torch.bool` the rest of the code that depends on it should also be updated.

Last check, I need to make sure these new mask don't go through a `1.0 - mask` afterwards and will be good to go.
EDIT: looks good, everything goes to a torch.where

Test failing is unrelated, merging
[`tests`] add `accelerate` marker
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR introduces a nice utils for users and contributors (such as myself) that want to run just `accelerate` specific tests. Thanks to @fxmarty that has introduced to me this util

By using `pytest.mark` you can run `accelerate`-specific tests easily. With this PR, if let's say I want to run `accelerate` tests on OPT I can just do:
```bash
RUN_SLOW=1 pytest -m accelerate_tests tests/models/opt/test_modeling_opt.py 
```

cc @ydshieh @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @younesbelkada ! I understand the advantage of adding this. However, I would recommend to follow what we have like `is_pt_tf_cross_test` as in
https://github.com/huggingface/transformers/blob/619d51e01f326d298687912d1f65f8d460f2a6e2/src/transformers/testing_utils.py#L150
Along this approach, you also have to check

https://github.com/huggingface/transformers/blob/619d51e01f326d298687912d1f65f8d460f2a6e2/src/transformers/testing_utils.py#L142
and
(I am not familiar with this part hoever)
https://github.com/huggingface/transformers/blob/619d51e01f326d298687912d1f65f8d460f2a6e2/conftest.py#L34

This is a suggestion based my intuition - I never did this before in `transformers`. You can wait a bit to hear what @sgugger says.


Thanks a lot for sharing that! 
I agree for consistency it would make sense to do this, however this solution seems IMO much simpler, given the number of affected tests (usually for each model there are only 3 `accelerate` tests), let's see what @sgugger will say!

> Thanks a lot for sharing that! I agree for consistency it would make sense to do this, however this solution seems IMO much simpler, given the number of affected tests (usually for each model there are only 3 `accelerate` tests)

My response here is not to convince @younesbelkada the suggested approach, but just want to point out the above argument doesn't seem very valid: if you look  `tests/test_modeling_common.py`, there is only one test decorated with `is_pt_tf_cross_test`, and yet we still use that approach.

The difference with the PT/TF cross tests is that we want to skip them unless a special env var is set (to not run them in the torch job and tf job for instance). This is not necessary here since as we don't want to prevent those tests from running in the regular jobs, this is just for convenience.

The failing test seems to be unrelated to this PR, merging!
Fix PyTorch Perceiver `PerceiverFourierPositionEncoding` with fp16
Author: fxmarty
Pull request: yes

Passing `torch_dtype=torch.float16` with perceiver is currently broken on main. The error comes from a parameter generated on the fly always with `torch.float32` dtype, hence raising issues later on. Reproduction:

```python
from transformers import AutoImageProcessor, PerceiverForImageClassificationConvProcessing
from PIL import Image
import requests
import torch

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("deepmind/vision-perceiver-conv")
model = PerceiverForImageClassificationConvProcessing.from_pretrained("deepmind/vision-perceiver-conv", torch_dtype=torch.float16).to("cuda")

inputs = image_processor(images=image, return_tensors="pt").pixel_values.to("cuda")
inputs = inputs.to(torch.float16)
outputs = model(inputs=inputs)
logits = outputs.logits
list(logits.shape)

# model predicts one of the 1000 ImageNet classes
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])
```

Raising `RuntimeError: expected scalar type Float but found Half`

This PR fixes the issue. Let me know if you would like me to add tests for this. I'm actually surprised this was not catched in an existing test.

## Who can review?

@amyeroberts @sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger @amyeroberts Feel free to merge, it appears I have no merge rights even after approval on transformers.
Fix nn.init.trunc_normal_ call on torch.float16 data
Author: fxmarty
Pull request: yes

Following https://github.com/huggingface/transformers/pull/20803 that gave the idea, but was still buggy for some cases, for example:

```python
from transformers import ViTForMaskedImageModeling
import torch

model = ViTForMaskedImageModeling.from_pretrained('hf-internal-testing/tiny-random-vit', torch_dtype=torch.float16).to("cuda")
```

still raising `RuntimeError: "erfinv_vml_cpu" not implemented for 'Half'`.

Let me know if you would like me to add tests.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger @younesbelkada Feel free to merge, it appears I have no merge rights even after approval on transformers.
[WIP] Add Seaformer
Author: inderpreetsingh01
Pull request: yes

<!-- Remove if not applicable -->

# What does this PR do?

Fixes #21668 
Seaformer is a two-branch architecture with Squeeze enhanced Axial Transformer.
<br> Initialized as
**tokenizer_type** = Standalone <br>  **is_encoder_decoder_model** = False, since its encoder only.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? #21668 
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@alaradirik thanks for offering help with this PR, please let me know about any changes required.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @inderpreetsingh01, SeaFormer is a mobile-friendly semantic segmentation model but I see that the PR is for a language model?

For reference, the best way to add a new model is to identify a similar model in the library (I'd say SegFormer in this case), create a new branch and initialize the files with `transformers-cli add-new-model-like`.  You can refer to [this page](https://github.com/huggingface/transformers/blob/main/templates/adding_a_new_model/README.md) for more information.

Thanks @alaradirik i have done the changes you mentioned (initialized model as SegFormer) in new branch and raised the new PR #21819.
Fix gradient checkpointing bug in gptneox
Author: KMFODA
Pull request: yes

# What does this PR do?

This PR fixes a bug that a user can encounter while using generate and models that use gradient_checkpointing.

Fixes Issue https://github.com/huggingface/transformers/issues/21737

cc @younesbelkada or @gante

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Apologies, hand't pushed the latest commit. All done now!

Awesome, thank you for the contribution <3
UL2 Mixture-of-Denoiser loss
Author: gaceladri
Pull request: no

### Feature request

The losses applied to the paper **UL2: Unifying Language Learning Paradigms**
The Mixture-of-Denoisers losses are described in the UL2 paper, which can be found at the following link: https://arxiv.org/abs/2205.05131
The code is based on T5x (which is JAX/FLAX): https://github.com/google-research/t5x


### Motivation

I am requesting the addition of new losses applied in the UL2 paper called Mixture-of-Denoisers. These new losses have been shown to improve the performance of unsupervised learning models and I believe they could benefit the HuggingFace community.

### Your contribution

Opening the request

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Slightly better WandbCallback
Author: nbroad1881
Pull request: yes

# What does this PR do?

Allows for more environment variables to be used with the `WandbCallback`. Prioritizes variables set in `TrainingArguments`


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21101). All of your documentation changes will be reflected on that endpoint.

@stevhliu, I seemed to have messed up the code quality steps. I'll fix that soon.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Calling parallelize() on T5ForConditionalGeneration for ByT5 results in device_map error
Author: yunyu
Pull request: no

### System Info

4.25.1

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
model = T5ForConditionalGeneration.from_pretrained("google/byt5-xl")
model.parallelize()
```

Results in:
```
The device_map contains more attention blocks than this model has. Remove these from the device_map: {...}
```

### Expected behavior

The model should parallelize attention blocks properly. This is needed because ByT5 has a 3x deeper encoder than decoder, so the same device_map can't be used for both.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Note that the `parallelize` API is going to be deprecated soon. You should load your model like this to use Accelerate instead:
```python
model = T5ForConditionalGeneration.from_pretrained("google/byt5-xl", device_map="balanced")
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add: task guide for zero shot object detection
Author: MKhalusova
Pull request: yes

This PR adds a task guide for zero-shot object detection. Unlike other task guides, there is no fine-tuning and complex preprocessing of custom data. However, the task illustrates different ways of inferencing with OWL-ViT, such as using the pipeline, manual inference with text queries, manual inference for a batch of examples, and image-guided object detection. 
The task guide is based on [this notebook](https://github.com/huggingface/notebooks/blob/main/examples/zeroshot_object_detection_with_owlvit.ipynb) with some additions (e.g. pipeline example) and some modifications. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix quality with `ruff==0.0.253`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix quality with `ruff==0.0.253`.

Merged to avoid CI failing due to the new version of `ruff==0.0.253`.

 - The change with this new version is valid so I decide to go with it instead of pining older version.
 - The change also works with previous `ruff` versions (`0.0.252` and  `0.0.243`), so contributors don't need to upgrade their ruff.

cc @sgugger for comments if any.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21828). All of your documentation changes will be reflected on that endpoint.
introduce `logger.warning_once` and use it for grad checkpointing code
Author: stas00
Pull request: yes

This PR:

1. introduces a new `warning_once` logger method - to prevent repeating the same warning more than once
2. use it for gradient_checkpointing functionality - where one would get thousands of these warnings at the moment should they have `use_cache==True` - the other solution is to assert

(I did this for m4, so thought to sync here as well)

The rename was done automatically with:
```
perl -0777 -pi -e 's|(logger.warning)(\(\W+\S\Suse_cache=True)|logger.warning_once$2|msg' src/transformers/models/*/mode*py
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Rename `MobileViTModelTest` to `TFMobileViTModelTest`
Author: ydshieh
Pull request: yes

# What does this PR do?

@sayakpaul Let's give TF a bit more love ❤️ 🙏.

(joking aside, having consistent and proper prefixes makes things easier)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

No worry :-)
transformers  == 4.26.0 has a bug
Author: tongzhao315
Pull request: no

### System Info

Traceback (most recent call last):
    from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval
ImportError: cannot import name 'BridgeTowerProcessor' from 'transformers' (lib/python3.7/site-(packages/transformers/__init__.py),could you fix this bug。i use transformers  == 4.26.0(pip )      
 
/cc @xianbaoqian 

### Who can help?

@xianbaoqian 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval
import requests
from PIL import Image

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"]

processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

# forward pass
scores = dict()
for text in texts:
    # prepare inputs
    encoding = processor(image, text, return_tensors="pt")
    outputs = model(**encoding)
    scores[text] = outputs.logits[0, 1].item()

### Expected behavior

fix this bug

Comments:
If you want to use this model, you should build from source. It was merged to main on the 25th of January, the release does not include it. Next release will make it available. 
Closing as I can import on main.
Fix gradient checkpointing bug BioGpt
Author: saswatmeher
Pull request: yes

# What does this PR do?
This PR fixes a bug that a user can encounter while using generate and models that use gradient_checkpointing.

Fixes issue #21737 for BioGpt.

cc @younesbelkada, @gante  

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`T5`] Fix torchquant issue
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes #21839

This PR fixes a bug that was introduced with https://github.com/huggingface/transformers/pull/21281 - before this PR, the snippet below was working: 
```python
import torch
from transformers import T5ForConditionalGeneration, T5Tokenizer
model_name = "google/flan-t5-small"
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)
model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)
input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write a whole Haiku in a single tweet?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids
output = model.generate(input_ids)
```
On the `main` branch the snippet does not work anymore due to the check `self.wo.weight.dtype` since ` torch.quantization.quantize_dynamic` converts all `nn.Linear` layers to a bound function, leading to an error.
Since the users were able to run this snippet on previous versions, I think that we should support this feature.

Added also a cool test for that

cc @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can confirm t5 & int8 slow tests are passing, merging!
[`Blip2`] Add `Blip2Model`
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds a new class `Blip2Model`, so that the model can be mapped in `AutoModel` mapping and also used by users that wants to conveniently extract text, image, & the so-called q-former features from the model. 

This PR also addresses this comment: https://github.com/huggingface/transformers/pull/21708#pullrequestreview-1308704909

I decided to still keep `AutoModelForCausalLM` & `AutoModelForSeq2SeqLM` for `self.language_model` as using `AutoModel` there leads to keys that are not properly loaded from the Hub. Let me know if you think that this is a mistake and should be addressed differently

cc @sgugger @MKhalusova @stevhliu 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks! Once it's merged, I'll create a small PR to update the troubleshooting section.
BLIP computes loss no matter what
Author: NielsRogge
Pull request: no

### System Info

transformers-4.27.0.dev0

### Who can help?

@younesbelkada 

### Reproduction

Currently, `BlipForConditionalGeneration` will always return a loss, even when the user hasn't provided any labels:

```
from PIL import Image
import requests
from transformers import AutoProcessor, BlipForConditionalGeneration

processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")

outputs = model(**inputs)
print(outputs.keys())
```
which prints:
```
odict_keys(['loss', 'decoder_logits', 'image_embeds', 'last_hidden_state'])
```
And what's also weird is that this loss is NaN:
```
print(outputs.loss)
```
returns
```
tensor(nan, grad_fn=<AddBackward0>)
```



### Expected behavior

Returning a loss in case the user doesn't provide `labels` isn't really compliant to the design of all models in the Transformers library, which only return a loss in case the user provides labels.

Hence I'm advocating to remove [the following lines of code](https://github.com/huggingface/transformers/blob/5b67ab9924cf7587b39b59eb0bf0abd3d099e8b9/src/transformers/models/blip/modeling_blip.py#L1008-L1009). Similar to models like `T5ForConditionalGeneration`, `BartForConditionalGeneration`, it is the user's responsibility to create the appropriate labels.

Moreover, I'm not sure why [the following lines of code](https://github.com/huggingface/transformers/blob/5b67ab9924cf7587b39b59eb0bf0abd3d099e8b9/src/transformers/models/blip/modeling_blip.py#L1005-L1006) are included, as they don't seem necessary and also lead to unexpected behaviour. Again here, models like T5 and BART don't set the `input_ids` automatically to the decoder start token ID in case the user doesn't provide `input_ids`. It's the user's responsibility to provide `input_ids`. Instead, those lines should also simply be removed IMO.

Comments:
i would like to work on this can you assign me