noamaanMulla-03/transformers-issues huggingface dataset:
Any config for DeBERTa series as decoders for TSDAE? 
Author: bobox2997
Pull request: no
Labels: name: Feature request

### Feature request

Seems that there is no config for DeBERTa v1-2-3 as decoder (while there are configs for BERT/RoBERTa et similia models)... This is needed in order to perform TSDAE unsupervised fine tuning.

*(TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning)*

### Motivation

Here a reference for the related sentence transformer issue: https://github.com/UKPLab/sentence-transformers/issues/2771

TSDAE demonstrated to be a powerful unsupervised approach, and DeBERTa is proven to be a really strong base model for further fine tuning (also, v2 has a an xxlarge 1.5B version and v3 demonstrated strong performance and efficiency with its ELECTRA-style pretraining)

For context, here the TSDAE paper: https://arxiv.org/abs/2104.06979

### Your contribution

I'm not sure if I can contribute to the repository... 

Anyway, i can certainly open source multiple domain adapted models, including models in a size range (1.5B) where there is not much choices while working with encoder-only models

Edited

Comments:
Hi @bobox2997, thanks for opening this PR! 

What I would suggest is adding checkpoints, configs and possibly updated modeling files directly on the hub, and have as much support as we can there. It will be easier to integrate than directly into transformers. Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models) if that sound good to you!

I'm not sure that I understood that correctly... What checkpoint should I add?

On TSDAE implementation the decoder portion is tied to the encoder and is not used at inference, I just need an "is_decoder" argument (and related configs of course) in the config of DeBERTa as there is for BERT, RoBERTa et similia models...

I'm sorry if those are naive or dumb questions, I'm still learning.

Thank you so much for your time! 

@bobox2997 Ah, OK, I thought there were checkpoints available trained with this method.

In terms of changes in the transformers library, we're very unlikely to accept changes to the architecture or configurations files to add new features like this, especially for older, popular models and anything which doesn't have official checkpoints available. 

The great thing about open source is that you are free to build upon and adapt the available code (license permitting) for your own projects. It should be possible to add as a new architecture on the hub, keeping compatibility with the transformers library and allowing you to use the same API.

If you or anyone else in the community would like to implement this, feel free to share you project here!
dependencies: `keras-nlp<0.14` pin
Author: gante
Pull request: yes

# What does this PR do?

On the latest `keras-nlp` [release](https://github.com/keras-team/keras-nlp/releases/tag/v0.14.0) -- "KerasNLP no longer supports Keras 2.". This means we don't support this version, as we only support Keras 2

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31684). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Fine with pinning to keep things working for now, but would be good to have a plan for future keras updates. @Rocketknight1  could we remove keras-nlp? 

also cc @ydshieh for reference 

@amyeroberts @ydshieh I followed [our docs](https://www.notion.so/Circle-Ci-Docker-images-3bedc8bc602e4ca88c34c6c194accd74?pvs=4) for version updates, but the [resulting PR](https://github.com/huggingface/transformers/pull/31687) doesn't look right.

What's the expected procedure to update breaking versions? 🤔 

I've confirmed that:
- having `keras-nlp==0.14.0` causes these test errors
- calling `pip install "keras-nlp<0.14" -U` fixes it

However, since the docker images are not updated in this PR, CI will continue to be red

@gante Tbh, I'm not sure. @ArthurZucker is the best person to ping here, as he wrote the docs 

Let's go on slack @gante ! 

@amyeroberts the `dev-ci` commit ran against the `:dev` docker images, which stem from the latest commit [here](https://github.com/huggingface/transformers/pull/31687). The failed test is unrelated and reported on slack, the TF issues are fixed as it can be seen :)

If you approve these changes, after the hub failures are resolved:
1. I will add a commit `[push-ci-image]` to the other PR, which will push the updated docker images. After the CI on this commit runs, all models will run their CI with that docker image, fixing the issues on CI.
2. merge this PR, which fixes the issue on local installs and ensures future docker image updates include this change.
3. Update the CI docker docs with the learnings in this process, to make them more clear to everyone :)

(btw cc @Wauplin for viz)

Thanks for the detailed explanation @Rocketknight1 ! 

Thanks for the ping, good to keep in mind. And thanks @Rocketknight1 for the full context!

Docker image pushed in #31687 -- merging 🤞
Fix Gemma2 4d attention mask
Author: hiyouga
Pull request: yes

# What does this PR do?

(Might) continuation of #31670 

Fixes #31673 

The attention mask here should be a 4d mask, while the current implementation mistakenly treated it as a 2d mask. Additionally, the mask needs to be inverted to prevent label leakage.

Before this pr:

![image](https://github.com/huggingface/transformers/assets/16256802/05eb6053-f6e0-485a-bf45-622433e24975)

After this pr:

![image2](https://github.com/huggingface/transformers/assets/16256802/b8be2906-0590-468b-a21d-730dcdf1c936)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31674). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The patch is out for this! Thanks again 🤗 https://pypi.org/project/transformers/4.42.2/ 

Thank you all for your hard work. I've been dealing with this issue recently as well.
Pre-training reference URL in Idefics2 codebase
Author: BiliBraker
Pull request: no

The below snippet from the `modeling_idefics2.py` script points to "the original codebase", but the link is incorrect.
https://github.com/huggingface/transformers/blob/a564d10afe1a78c31934f0492422700f61a0ffc0/src/transformers/models/idefics2/modeling_idefics2.py#L1347-L1349

Could you provide the correct link for the pre-training configuration for `Idefics2`?

Comments:
This is due to a `Copied from` statement from llava: https://github.com/haotian-liu/LLaVA/tree/main/llava.

The link in `modeling_idefics2.py` should actually be removed

Edit: actually 7 models have this comment, is only applicable for llava

Could you open a PR to remove these links?

Sure, but I've only found 4 models (except llava) with this comment:
https://github.com/huggingface/transformers/blob/a564d10afe1a78c31934f0492422700f61a0ffc0/src/transformers/models/llava_next/modeling_llava_next.py#L239
https://github.com/huggingface/transformers/blob/a564d10afe1a78c31934f0492422700f61a0ffc0/src/transformers/models/vipllava/modeling_vipllava.py#L142
https://github.com/huggingface/transformers/blob/a564d10afe1a78c31934f0492422700f61a0ffc0/src/transformers/models/video_llava/modeling_video_llava.py#L134
https://github.com/huggingface/transformers/blob/a564d10afe1a78c31934f0492422700f61a0ffc0/src/transformers/models/idefics2/modeling_idefics2.py#L1349


Ok I was searching on "important: this ported version" which gave me 7 results, feel free to only keep it for llava

Opened the PR!
Btw, is there any code related to the pre-training configuration/implementation of `Idefics2`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Mistral compile not working on T4 GPU (torch 2.3 + cu121)
Author: ydshieh
Pull request: no

### System Info

Mistral compile not working on T4 GPU

torch 2.3 + cu121
transformers commit: 4ef85fee718969f1703d7dffa134deb72f4de828

See https://github.com/huggingface/transformers/pull/30991#discussion_r1612042080

> tests/models/mistral/test_modeling_mistral.py::tests/models/mistral/test_modeling_mistral.py::test_compile_static_cache

TODO: enable  `test_compile_static_cache` for mistral model test once fixed.




Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
remove incorrect urls pointing to the llava repository
Author: BiliBraker
Pull request: yes

# What does this PR do?
This PR removes 4 incorrect URLs pointing to the LLaVA repository.


Fixes # (issue)
https://github.com/huggingface/transformers/issues/31101

## Before submitting
- [ x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@NielsRogge 
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@BiliBraker For the quality checks, could you run `make fix-copies` and push the changes?

@BiliBraker Running `make fixup` should fix the failing quality checks
Move some test files (`tets/test_xxx_utils.py`) to `tests/utils`
Author: ydshieh
Pull request: yes

# What does this PR do?

For our daily CI (and other scheduled CI) workflow, we collect the tests by using the folders under `tests`. However, there are some files under `tests` too. For some test files, they contains `TestCase` but their tests are not collected in the CI workflow run.

This PR moves `tets/test_xxx_utils.py` into `tests/utils`.

To be very sure we don't miss any test, it's probably better to come up with a solution that will collect the tests from the files under `tests` but not in any of its sub-directory. But it's more complex and might take a bit more time to deal with.




Comments:
Hi @amyeroberts. The failures are irrelevant I think

- `ERROR tests/utils/test_file_utils.py - RuntimeError: Failed to import transformers.models.gpt2.tokenization_gpt2_tf because of the following error (look up to see its traceback):
cannot import name 'ops' from 'keras' (/usr/local/lib/python3.10/site-packages/keras/__init__.py)'`

- FAILED tests/utils/test_chat_template_utils.py::JsonSchemaGeneratorTest::test_multiple_complex_arguments - AssertionError: {'nam[195 chars]': ['string', 'integer'], 'nullable': True, 'd[47 chars]x']}} != {'nam[195 chars]': ['integer', 'string'], 'nullable': True, 'd[47 chars]x']}}

But maybe it's better to wait them being fixed ? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31730). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yeah it's irrelevant. Let's wait until the CI is fixed !
Docker image: keras-nlp<0.14 pin
Author: gante
Pull request: yes

Temporary PR to update the docker image

Comments:
this doesn't look right either 🤔 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31687). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

If the docker images are build correctly, the dev tag should exist


pushed image, closing the PR

@gante it seems something hasn't worked as there's still failures on the CI due keras

This does not seem to be triggering [the build job](https://github.com/huggingface/transformers/actions/workflows/build-ci-docker-images.yml), despite having the correct commit message (see [here](https://github.com/huggingface/transformers/blob/cf85e86e9ad22ddd02c519d21c4783e3dbf72178/.github/workflows/build-ci-docker-images.yml#L25))

Could it be because the branch is from my fork, as opposed to from the repo? Going to test it

Yup, that was it!
Fix mistral ONNX export
Author: fxmarty
Pull request: yes

The ONNX torchscript export fails after https://github.com/huggingface/transformers/pull/30642 as `exclude_mask |= ...` gets mapped to `aten::__ior_` which is not exportable. `aten::bitwise_or` is.

Using `torch.bitwise_or(exclude_mask, second_mask, out=exclude_mask)` breaks as well the ONNX export so using the inplace version of the operator instead.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31696). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

keras CI looks broken
Suport sdpa for RoBERTa and XLM-RoBERTa models
Author: kiszk
Pull request: no
Labels: name: Feature request

### Feature request

Enable sdpa for RoBERTa and XLM-RoBERTa models

### Motivation

While BERT, which is similar to RoBERTa and XLM-RoBERTa, support sdpa, RoBERTa and XLM-RoBERTa do not support sdpa yet. This enablement is straight-forward.

Our applications need latency reduction by sdpa. The performance advantage in BERT is already shown at #28802.

### Your contribution

I will submit a PR.

Comments:
🚨🚨 TextGenerationPipeline: rely on the tokenizer default kwargs
Author: gante
Pull request: yes

# What does this PR do?

### Issue
`TextGenerationPipeline.preprocess`, where tokenization happens, has defined a few optional kwargs whose default value does not match the tokenizer defaults. This was causing: 
1. A mismatch between the tokens resulting from `tokenizer` calls and the tokens `pipeline` was seeing, using default arguments on both
2. The issue seen in [this thread](https://huggingface.co/google/gemma-2-9b-it/discussions/17#66828e78fcfdfa8c5a514891) -- `add_special_tokens` had to be manually set to `True` in the pipeline, despite it already being the default in the tokenizer.

(2 is a consequence of 1 :) )

### This PR
🚨🚨 This PR changes the code to rely on the tokenizer's defaults when these flags are unset. This means some models using `TextGenerationPipeline` previously did not add a `<bos>` by default, which (negatively) impacted their performance. In practice, this is a breaking change.

Example of a script changed as a result of this PR:
```py
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2-9b-it", torch_dtype=torch.bfloat16, device_map="auto")
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe("Foo bar"))
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31747). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Very nice, I agree it makes sense to make the behaviours consistent!
Docker image: keras-nlp<0.14 pin (round 2)
Author: gante
Pull request: yes

# What does this PR do?

Follow-up to #31687 -- see [this comment](https://github.com/huggingface/transformers/pull/31687/#issuecomment-2202918704).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31746). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Make tool JSON schemas consistent
Author: Rocketknight1
Pull request: yes

Our helper for generating JSON schemas from functions had an inconsistency, because the order of types inside a `Union` is not well-defined (I suspect internally it gets converted to a set somewhere).

As a result, they came out in somewhat arbitrary order, and this caused the CI to break when we changed some random aspect of the test runner that flipped the order to the other one. This PR fixes that by using `sorted()` to ensure a consistent ordering for a union of types.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31756). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merging exceptionally under the following rationale: [cc @amyeroberts @LysandreJik ]
1. Our core maintainers are both off
2. The PR LGTM
3. It fixes breaking CI, so it unblocks other work
4. It's very small

🤗
add `post_process_depth_estimation` to image processors
Author: the-neural-networker
Pull request: yes

# What does this PR do?

This PR adds a `post_process_depth_estimation` method to the `DPTImageProcessor` class. The method takes the raw outputs of `DPTForDepthEstimation` and an optional `target_sizes` parameter, and returns a list of depth estimation maps resized to the specified target sizes (if provided) or in their original size. Since `DepthAnything` model also uses the `DPTImageProcessor`, this feature is applicable to `DepthAnything` as well.

Addresses #30917

## Who can review?

@NielsRogge

Comments:
Any suggestions are more than welcome!

@the-neural-networker Thanks for working on this! 

The main two requests would be: 
* Add this for all depth estimation models
* Add tests to make sure the output is as expected 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Ensured the proper handling of imports
Author: Imran-ml
Pull request: yes

I used the try except blocks to handle optional imports correctly.



Comments:
Hi @Imran-ml, thanks for opening this PR! 

This try/except block shouldn't be necessary. Could you share a code snippet which reproduces an error or a related issue this is addressing ?

Hi @amyeroberts, The try/except blocks are necessary in this code because the code should to be flexible and run in environments where PyTorch and TensorFlow might not be installed otherwise it will through ImportError. I faced this issue on one of my env. 

@Imran-ml Could you share the code which was triggering an error? The `is_xxx_available` if statements should help provide guards for imports when using transformers

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Arm-based (M1, M2, M3) Mac Install Issues for `pip install transformers` (<4.23.0)
Author: ai-nikolai
Pull request: no

### System Info

**This is not critical just FYI:**
Tested the following on: `Mac OS 14`, with `arm-based chip` (M2 Pro).

```bash
pip3 install transformers==4.19
pip3 install transformers==4.20
pip3 install transformers==4.21
pip3 install transformers==4.22
```

All of the above fail. 
`pip3 install transformers==4.23.0` is the first one that works.


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

On: `Mac OS 14` with `arm-based` chip.

```bash
pip3 install transformers==4.19
```

### Expected behavior

Installs successfully OOB.

Comments:
Hi @ai-nikolai, thanks for digging into this and opening this issue! 

Making the previous releases installable probably isn't something we'd be able to address (any changes to code on the repo would just go in the most recent release). 

Is there a reason for trying to install these versions specifically? v4.19 for example is two years old now. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.