noamaanMulla-03/transformers-issues huggingface dataset:
Fix `test_finetune_bert2bert`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix the CI error. See the comment in the change of this PR.
```
tests/trainer/test_trainer_seq2seq.py::Seq2seqTrainerTester::test_finetune_bert2bert
(line 162)  ValueError: Make sure to set the pad_token_id attribute of the model's configuration.
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Falcon: batched generation
Author: gante
Pull request: yes

# What does this PR do?

This PR does three things:
1. Fixes the minimum float number added to the attention mask, in the positions the attention mask is `0`. In some numerical precisions, the numerical attention mask was getting `-inf`, which wrecked downstream computations.
2. Adds the `position_ids` input to Falcon, which is needed for proper batched generation. When it is not passed, the forward pass builds the position ids from the sequence length, which does not account for the left-padding in batched generation -- the model could still generate, but the results should be slightly better after the fix.
3. Add tests for batched generation with left padding



Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker woops, sorry, there are still tests to fix, I will ping you again when they are fixed!

@ArthurZucker ready now

> Let's be careful with padding as we have been getting a lot of issues regarding this!

@ArthurZucker 100% agreed! If you come across a new model, plz make sure there is a test for this 🙏 

@ArthurZucker suggestions applied 💪
Use Version instead of version.parse to compare versions
Author: riteshghorse
Pull request: yes

# What does this PR do?

Use `packaging.version.Version` to compare torch version instead of `packaging.version.parse`.




Fixes #25669


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
More explicit bounding box formats
Author: tonydavis629
Pull request: no
Labels: name: Feature request

### Feature request

I am having a difficult time managing different bounding box formats in this library. For example, the DETR model preprocessor input format is COCO (xywh), but the output of the postprocessor is a pascal_voc (xyxy). Why not keep everything the same format? Furthermore, the documentation does not say which format is which so it's required to dig through the source to identify that information.

### Motivation

Ease of use

### Your contribution

Just a request

Comments:
Hi @tonydavis629, thanks for raising this issue. 

Yes - this is something we've been thinking about a lot recently and it's on our roadmap to address! 

cc @rafaelpadilla
[docs] last hidden state vs hidden_states[-1]
Author: MKhalusova
Pull request: yes

Intuitively one may think that `output.hidden_states[-1]` (returned when `output_hidden_states` is set to `True`) should match the `output.last_hidden_states` exactly. However, this is not always the case. Models like CLIP, ClipSeg, GroupVit, OWLViT, and X-CLIP apply `layernorm` before returning the `last_hidden_states`. Some other models apply `post_layernorm` or `norm`.

This PR adds a small note in the docs to address possible confusion.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Getting equivalence between torchvision and image transforms when normalizing without rescaling
Author: NielsRogge
Pull request: no

### System Info

Torchvision version 0.14.0
Transformers version 4.27.0.dev0

### Reproduction

Trying to have equivalent performance for the following (relevant for #26092):

```
from transformers.image_transforms import normalize

from PIL import Image
import requests
import numpy as np
import torch

from torchvision import transforms


url = "https://dl.fbaipublicfiles.com/dinov2/images/example.jpg"
image = Image.open(requests.get(url, stream=True).raw)

def make_depth_transform() -> transforms.Compose:
    return transforms.Compose([
        transforms.ToTensor(),
        lambda x: 255.0 * x[:3], # Discard alpha component and scale by 255
        transforms.Normalize(
            mean=(123.675, 116.28, 103.53),
            std=(58.395, 57.12, 57.375),
        ),
    ])

# prepare image using torchvision
transform = make_depth_transform()
original_pixel_values = transform(image)

# prepare image using Transformers' image transforms
pixel_values = normalize(np.array(image), mean=(123.675, 116.28, 103.53), std=(58.395, 57.12, 57.375))
pixel_values = np.moveaxis(pixel_values, -1, 0)
pixel_values = torch.from_numpy(pixel_values).float()

assert original_pixel_values.shape == pixel_values.shape
assert torch.allclose(original_pixel_values, pixel_values)
```
The assertion above currently fails.

### Expected behavior

I would like to have equivalent behaviour between torchvision and our image transforms when normalizing without rescaling.

cc @amyeroberts 

Comments:
@NielsRogge This is happening because the dtype of the images when calling the torch and image_transforms `normalize` aren't the same. If you cast the dtype to float32, then you'll get the equivalent behaviour: 

```py
pixel_values = normalize(np.array(image).astype(np.float32), mean=(123.675, 116.28, 103.53), std=(58.395, 57.12, 57.375))
```

This is because `ToTensor()` is doing three things: casting to float32, rescaling, converting to a torch tensor. 


Thanks a lot, this was what I needed!
Add type hints for tf models final batch
Author: nablabits
Pull request: yes

# What does this PR do?

Addresses, hopefully :crossed_fingers: , the last the type hints for tf models in https://github.com/huggingface/transformers/issues/16059:

1. `RegNet`
	1. `TFRegNetForImageClassification`
	2. `TFRegNetModel`
2. `TFSamModel`
3. `TFSegformerDecodeHead`
4. `TransfoXL`
	1. `TFTransfoXLLMHeadModel`
	2. `TFTransfoXLModel`
5. `TFWav2Vec2ForSequenceClassification`
6. `TFXLMModel`

## Who can review?
@Rocketknight1 please

Comments:
There are some tests failing likely because they are running on python 3.8.12 and the `|` notation for type hints was introduced in python 3.10:

![image](https://github.com/huggingface/transformers/assets/33068707/648f4de7-1120-4ba6-ab4f-450e298c15d2)

It seems that it's only the [RegNet model](https://github.com/huggingface/transformers/blob/f353448a4dd4fb412be789a863ff6cabb3f96c70/src/transformers/models/regnet/modeling_tf_regnet.py#L380), so I will replace it.

It's not clear to me, though, why the others are not complaining :thinking:  
Anyways, I assume that eventually the circle ci version will be bumped and that will make them compliant in terms of type hints.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25883). All of your documentation changes will be reflected on that endpoint.

> Woah, I never thought we'd actually be able to finish the type hints list. This is amazing! I added a couple of optional comments, but other than that this is good to merge. Let me know whenever you're ready!

Yeah, it's the magic of perseverance :sparkles: 

I'm happy for you to merge, let me know if you want me to open a follow up issue for the [circle ci thingy](https://github.com/huggingface/transformers/pull/25883#discussion_r1312612246)

Type hinting project is officially complete as of this PR!
Add type hints for tf models batch 1
Author: nablabits
Pull request: yes

# What does this PR do?

Addresses some of the type hints for tf models in https://github.com/huggingface/transformers/issues/16059:

1. `TFBlipTextModel`
2. `DPRModel` family:
	1. `TFDPRContextEncoder`
	2. `TFDPRQuestionEncoder`
	3. `TFDPRReader`
3. `LED` family
	1. `TFLEDForConditionalGeneration`
	2. `TFLEDModel`
4. `TFLxmertForPreTraining`
5. `Marian` family
	1. `TFMarianMTModel`
	2. `TFMarianModel`
6. `Rag` family
	1. `TFRagModel`
	2. `TFRagTokenForGeneration`

## Who can review?
@Rocketknight1 please


Comments:
It seems that some tests failed because there was a connection error to `huggingface.co`:

![image](https://github.com/huggingface/transformers/assets/33068707/8257ce49-0a77-4812-86ef-799d6b1359e1)

And

![image](https://github.com/huggingface/transformers/assets/33068707/5df53424-725f-42b8-b071-8a504a657368)

I don't seem to have permissions to retrigger them so anyone who has can do so. Alternatively we can wait until some comment needs to be addressed (pretty likely)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25853). All of your documentation changes will be reflected on that endpoint.

> This looks great! Good catches with the copy-pasted docstrings from torch as well. Is this ready to merge, or do you want to add anything else?

Now with the consistency tweaks I'm happy for you to merge, thanks :hugs:
Add type hints for several pytorch models (batch-4)
Author: nablabits
Pull request: yes

# What does this PR do?
Addresses some of the models in https://github.com/huggingface/transformers/issues/16059:
1. Mgpstr
	1. `MgpstrForSceneTextRecognition`
	2. `MgpstrModel`
2. PLBart
	1. `PLBartForConditionalGeneration`
	2. `PLBartModel`
3. `Pix2StructTextModel`
4. `RagModel`
5. `SamModel`
6. `Swin2SRModel`

## Who can review?

@Rocketknight1, please

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25749). All of your documentation changes will be reflected on that endpoint.
Add type hints for pytorch models (final batch)
Author: nablabits
Pull request: yes

# What does this PR do?
Addresses the last type hints for pytorch models in https://github.com/huggingface/transformers/issues/16059:

1. TableTransformer: 
	1. `TableTransformerForObjectDetection`
	2. `TableTransformerModel`
2. `TimesformerModel`
3. `TimmBackbone`
4. TVLT: 
	1. `TvltForAudioVisualClassification`
	2. `TvltForPreTraining`
	3. `TvltModel`
5. Vivit
	1. `VivitForVideoClassification`
	2. `VivitModel`


## Who can review?
@Rocketknight1 please


Comments:
Thank you again for the diligent work on the type hints, by the way - your PRs were really clean and required very little work to review, and they really helped push that project!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25750). All of your documentation changes will be reflected on that endpoint.

> Thank you again for the diligent work on the type hints, by the way - your PRs were really clean and required very little work to review, and they really helped push that project!

Thank you very much, really appreciated!! TBH it has been a great learning experience for me, so thanks for the opportunity and the patience with the explanations

> Thank you again for the diligent work on the type hints, by the way - your PRs were really clean and required very little work to review, and they really helped push that project!

My apologies, in advance, for stepping in here. 
I would like to highlight  as well  the positive impact  @nablabits  brings to the HF OSS ecosystem. 
It´s also great how this user helps other contributors to thrive  [#1](https://github.com/SoyGema/contrib_schema/issues/1) and [#2](https://github.com/SoyGema/contrib_schema/issues/2) bringing learning, reciprocity, and fun.





> > Thank you again for the diligent work on the type hints, by the way - your PRs were really clean and required very little work to review, and they really helped push that project!
> 
> My apologies, in advance, for stepping in here. I would like to highlight as well the positive impact @nablabits brings to the HF OSS ecosystem. It´s also great how this user helps other contributors to thrive [#1](https://github.com/SoyGema/contrib_schema/issues/1) and [#2](https://github.com/SoyGema/contrib_schema/issues/2) bringing learning, reciprocity, and fun.

Gema, thanks for the kind words :hugs: 
@SoyGema
Add type hints for several pytorch models (batch-3)
Author: nablabits
Pull request: yes

# What does this PR do?
Addresses some of the models in https://github.com/huggingface/transformers/issues/16059:
1. ErnieM
	1. `ErnieMForInformationExtraction`
	2. `ErnieMForMultipleChoice`
	3. `ErnieMForQuestionAnswering`
	4. `ErnieMForSequenceClassification`
	5. `ErnieMForTokenClassification`
	6. `ErnieMModel`
2. `EsmForProteinFolding`
3. `GraphormerModel`
4. `InstructBlipQFormerModel`
5. `LayoutLMForMaskedLM`
6. `LukeForEntitySpanClassification`

## Who can review?

@Rocketknight1, please

Comments:
cc @Rocketknight1 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25705). All of your documentation changes will be reflected on that endpoint.
Add type hints for several pytorch models (batch-2)
Author: nablabits
Pull request: yes

# What does this PR do?
Addresses some of the models in https://github.com/huggingface/transformers/issues/16059

## Who can review?

@Rocketknight1, please


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25557). All of your documentation changes will be reflected on that endpoint.

This looks good to me now! Ping me whenever you're finished and you want me to merge it.

> This looks good to me now! Ping me whenever you're finished and you want me to merge it.

Matt, this is fine by me so I'm happy for you to merge it

@Rocketknight1
Add support for COCO style datasets for instance segmentation
Author: roboserg
Pull request: no

### Feature request

Create a standard dataset loader capable of taking datasets in the JSON COCO style format and converting them into the Huggingface format. The DatasetDict will be generated with the correct features and configurations, making it suitable for various downstream tasks, such as instance segmentation fine-tuning with the Mask2Former mode from Huggingface hub.

The loader should include a flag that allows users to specify the type of segmentation they want to load, such as "panoptic," "semantic," or "instance." These different segmentation tasks store data differently within the COCO format. It is important to note that COCO segmentation masks can be represented in two ways inside the JSON file - either as a polygon or a bitmask.

### Motivation

COCO style formatted datasets are prevalent in computer vision, especially in segmentation. Adopting them would ease the transition for Deep Learning practitioners as data preparation is often the main hurdle.

### Your contribution

I am versed as to how the COCO format is structured but I am a total newbie to Huggingface. 

Comments:
Hi @roboserg, thanks for opening this feature request! 

`transformers` isn't responsible for datasets preparation, and so we wouldn't add a converter as part of the standard library. 

The best place to put logic like this is in an example script or demo notebook. 
* Scripts: https://github.com/huggingface/transformers/tree/b0f23036f16b961e14cf480522fef8581c4bf19c/examples/pytorch e.g. 
* Notebooks: https://github.com/huggingface/notebooks/tree/main

This would provide a working example of how to transform a coco-style dataset   

Ideally datasets should be converted and then uploaded to their DatasetDict equivalent on the hub. Perhaps a script would be useful to add so users could quickly convert and upload their own datasets? 

cc @rafaelpadilla 

Hi @roboserg :) 

I created a COCO dataset for bounding boxes only. Maybe it could be useful to you:
https://huggingface.co/datasets/rafaelpadilla/coco2017

You can find a [COCODataset](https://huggingface.co/datasets/rafaelpadilla/coco2017/blob/main/cocodataset/dataset.py#L14) class, which takes the `loaded_json` dictionary representing the JSON containing COCO's bounding boxes. If you find it useful, you could adapt it for other cases (panoptic and semantic). This way, you can have a dataset class to represent COCO's samples and annotations used in a dataloader.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`PEFT`] Fix PEFT + gradient checkpointing
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/25841 

When using Trainer + PEFT + gradient checkpoitning, it is important to add a forward hook on the input embedding to force the input `requires_grad` to be set to `True`. 

We do it in PEFT here: https://github.com/huggingface/peft/blob/85013987aa82aa1af3da1236b6902556ce3e483e/src/peft/peft_model.py#L334 

The fix is to call that same method inside `gradient_checkpointing_enable()` only if an adapter (or many) is attached to the model.

Added also a nice test!


cc @amyeroberts @pacman100 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks!
cc @amyeroberts for a core maintainer review! 🙏
Flex xpu bug fix
Author: abhilash1910
Pull request: yes

# What does this PR do?

For some Flex Intel XPUs there is support for fp16 mp ; hence this exception  should not be raise if fp16 is provided as mp dtype .
cc @muellerzr  @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26135). All of your documentation changes will be reflected on that endpoint.

@muellerzr Could you confirm if this fix is OK and if there's anywhere else that needs to be updated?
Maskformer is missing dataclass decorator
Author: cchan-lm
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.9.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.2
- PyTorch version (GPU?): 1.13.0+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hello! We are working on utilities for model investigations.  Our utility attempts to process generic input/output types at the layer level, accounting for `dataclass` objects as well. We ran into an issue with Maskformer where an intermediate output from `MaskFormerPixelDecoderOutput` was not an expected dataclass.

To reproduce, run the below:

```python
from dataclasses import is_dataclass

from transformers.models.maskformer.modeling_maskformer import MaskFormerPixelLevelModuleOutput
from transformers.models.maskformer.modeling_maskformer import MaskFormerPixelDecoderOutput


pixel_level_module = MaskFormerPixelLevelModuleOutput()
assert is_dataclass(pixel_level_module)  # passes

decoder_output = MaskFormerPixelDecoderOutput()
assert is_dataclass(decoder_output)   # will fail
```

Looking at https://github.com/huggingface/transformers/blob/80f29a25a7d2c945c769b41c0fd41e89cf7de31d/src/transformers/models/maskformer/modeling_maskformer.py#L121, the `@dataclass` decorator appears to be missing from `MaskFormerPixelDecoderOutput`. I think currently this is the only `ModelOutput` subclass in `transformers.models` that does not have the decorator.

Thank you in advance for your response!

### Expected behavior

I expect `MaskFormerPixelDecoderOutput` to be a `dataclass` object and `assert is_dataclass(decoder_output)` to pass.

Comments:
@cchan-lm Indeed! Would you like to open a PR to add the decorator? This way you get the github contribution for spotting this :) 

Sure! I'll do so from my personal account, @rachthree.

To avoid this in the future, what are your thoughts on adding a test to check that all direct subclasses of `ModelOutput` are dataclasses, and/or add a check in `ModelOutput.__init__subclass__`?

@cchan-lm Great! Yep - a small test would be great to add alongside. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Resolved by https://github.com/huggingface/transformers/pull/25638.
[BLIP-2] Improve conversion script
Author: NielsRogge
Pull request: yes

# What does this PR do?

When investigating an issue reported [here](https://github.com/salesforce/LAVIS/issues/418), I've reran and improved BLIP-2's conversion script (based on InstructBLIP).

It's important to compare apples-to-apples, so I had to fork the LAVIS repo and make sure the original model is also run in float32.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Opened PRs on the respective repos to update the layer norm eps:

- https://huggingface.co/Salesforce/blip2-flan-t5-xl/discussions/1
- https://huggingface.co/Salesforce/blip2-flan-t5-xxl/discussions/6
- https://huggingface.co/Salesforce/blip2-flan-t5-xl-coco/discussions/2
- https://huggingface.co/Salesforce/blip2-opt-2.7b/discussions/14
- https://huggingface.co/Salesforce/blip2-opt-2.7b-coco/discussions/1
- https://huggingface.co/Salesforce/blip2-opt-6.7b/discussions/4
- https://huggingface.co/Salesforce/blip2-opt-6.7b-coco/discussions/1

PR itself can be merged already.

Feel free to merge :)
IDEFICS: allow interpolation of vision's pos embeddings
Author: leot13
Pull request: yes

# What does this PR do?

Allows vision position embeddings to be interpolated. Thus allowing bigger images to be passed to the model.
Fixes issue #26154

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@ArthurZucker @amyeroberts


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for all the comments and suggestions! They should all be answered now