noamaanMulla-03/transformers-issues huggingface dataset:
[Deprecated due to mistakes] Add Prompt Depth Anything Model 
Author: haotongl
Pull request: yes

# What does this PR do?

This PR adds the **Prompt Depth Anything Model**. [Prompt Depth Anything](https://promptda.github.io/) builds upon [Depth Anything V2](https://depth-anything-v2.github.io/) and incorporates **metric prompt depth** to enable accurate and high-resolution metric depth estimation.

The implementation leverages [Modular Transformers](https://huggingface.co/docs/transformers/main/en/modular_transformers). The main file can be found [here](https://github.com/haotongl/transformers/blob/modeling_prompt_depth_anything/src/transformers/models/prompt_depth_anything/modular_prompt_depth_anything.py).


## Before submitting
- [ N/A] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). 
- [ ✅] Did you read the [contributor guideline] (https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ N/A] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ✅] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ✅] Did you write any new necessary tests?





Comments:
Add compile test for fast image processor
Author: yonigozlan
Pull request: yes

# What does this PR do?

Add a test in `ImageProcessingTestMixin`, which is only ran on fast image processors. The test compiles a fast image processor, run a processing pass and compares the result with an eager image processor.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Who can review?



<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35184). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
During the training process, what happens if tf32 and bf16 are enabled at the same time?
Author: Bonytu
Pull request: no

### System Info

transformers 4.34.1

### Who can help?

@muellerzr  @pacman100 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. use trainer and set --tf32 True and --bf16 True

### Expected behavior

Hi, when I was training llama2-13b, I set both --tf32 True and --bf16 True at the same time. I'm confused because the trainer worked normally when both of these parameters were enabled. During this process, which parts used tf32 and which parts used bf16? How exactly does it work when both are turned on at the same time?

Also I found many tutorial set these two params at the same time. [(https://www.philschmid.de/instruction-tune-llama-2)](tutorial)

Comments:
Hi，could you please help me to solve this issue? @muellerzr  @pacman100 
Thanks！

By setting bf16, the model weights are saved in bf16, but gradients (computed in half-precision but converted to full-precision for the update) and optimizations (optimizer states) are still done in full 32-bit precision. By additionally setting tf32, you speed up that very calculations at the expense of precision, but since ultimately those weights are saved as bf16 anyway, it _probably_ has no effect on accuracy.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> By setting bf16, the model weights are saved in bf16, but gradients (computed in half-precision but converted to full-precision for the update) and optimizations (optimizer states) are still done in full 32-bit precision. By additionally setting tf32, you speed up that very calculations at the expense of precision, but since ultimately those weights are saved as bf16 anyway, it _probably_ has no effect on accuracy.

@tdolega  Assuming you are using Trainer API, you will get model weights trained in mixed precision but saved in FP32 by default if setting --bf16.
warmup LR schedulers start from LR=0
Author: cfhammill
Pull request: no
Labels: name: bug

### System Info

transformers commit: 52ea4aa589324bae43dfb1b6db70335da7b68654 (main at time of writing)
the rest isn't relevant.

### Who can help?

trainer: @muellerzr @SunMarc

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run any example with a warmup scheduler, observe the the effective LR is 0 for the first step, unnecessarily wasting compute. See similar discussion for this issue on torchtune https://github.com/pytorch/torchtune/issues/2010. See the code at

https://github.com/huggingface/transformers/blob/52ea4aa589324bae43dfb1b6db70335da7b68654/src/transformers/optimization.py#L182

and evaluate for step 0. Observe it returns LR factor = 0, weights will not be updated.

### Expected behavior

Expect every optimizer step to adjust the weights of my model unless there is a good reason not to.

Comments:
The proposed solution in the torchtune discussion is to add a `min_lr` argument with the default set to a small number (e.g. 1e-7), and return `max(min_lr, computed_lr)`

Hey @cfhammill, thanks for starting the discussion. I wanted to know if adding a small number really makes a difference compared to just keeping to 0. Another solution would be to skip the forward if the LR is 0. cc @muellerzr 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Llama model, torch.compile output for custom device does not match with eager/cpu when generation_config.use_cache set to True
Author: vpandya-quic
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.43.2
- Platform: Linux-5.15.0-126-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: 1.0.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0a0+gitee1b680 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @gone

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

For a custom device I am working on adding `torch.compile()` with `CPP`  inductor backend.
I am trying run `"TinyLlama/TinyLlama-1.1B-Chat-v1.0"` and it has output difference when using KV cache in generation.
if I use following config output of compiled mode matches with eager mode.
`compiled_model.generation_config.use_cache = False`

And for large content length generation I see output similar to https://github.com/huggingface/transformers/issues/30347

### Expected behavior

Please help me debugging this further so that my backend generates correct output with compile mode even with KV cache.

Comments:
Hey! could you upgrade to the latest version of `transformers`? 🤗 

Thanks @ArthurZucker with latest version 4.47.1 things works as per expectations!
Disable  `.github/workflows/self-comment-ci.yml` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

Let's rework it for more security later.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35366). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

OK, I will just remove it
dinov2 with REGISTERS
Author: betterze
Pull request: no
Labels: name: New model

name: Vision

### Model description

Dear huggingface team,

The fair team published an improved version of dinov2 [VISION TRANSFORMERS NEED REGISTERS](https://arxiv.org/abs/2309.16588). The models and checkpoints are available in the [dinov2 website](https://github.com/facebookresearch/dinov2#pretrained-backbones-via-pytorch-hub), but not in hugging face. 

Could you add this new model? I really appreciate your work.

Best Wishes,

Zongze

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

[<VISION TRANSFORMERS NEED REGISTERS>](https://arxiv.org/abs/2309.16588)
[dinov2 reg checkpoint](https://github.com/facebookresearch/dinov2#pretrained-backbones-via-pytorch-hub)

Comments:
Hello! Could you kindly assign this task to me? I'm eager to take it on as my first contribution and greatly appreciate any guidance or considerations you can provide. Thank you in advance.

I am not a team member of hugging face, I can not 'assign the task to you'. But I believe you are very welcome to work on it, a lot of people will be benefited from your work. Thx


Hello @amyeroberts @NielsRogge!
Can I please know your opinion about this endeavor? Thank you in advance.

After reading the paper, I fully agree that DINOv2 with registers has better performance. 

What's the current progress of this issue? @mhdirnjbr Did you submit a PR? I think timm has an implementation with registers

@StarCycle @mhdirnjbr If any of you would like to, please feel free to open a PR to add this to the library!

Is Anyone working on this?

No. It's only 2 lines of code different compared to DINOv2 so feel free to take it up

Thanks for replying @NielsRogge, and I would take it up if I knew what I was doing. Maybe you could point me in the right direction I could give it a go.

Edit: I tried looking at both timm and Huggingface implementations but tbh I don't have a clue as to what's happening. If you do manage to take I would be grateful for your help.
Add dtype check in check_quantized_param for bnb4
Author: MekkCyber
Pull request: yes
Labels: name: Quantization

# What does this PR do?
Adding dtype check as suggested by the comment in `check_quantized_param` for bnb4

## Who can review ?
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35183). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support saving models trained with DeepSpeed in Trainer callbacks
Author: dwyatte
Pull request: no
Labels: name: trainer

name: Feature request

name: DeepSpeed

### Feature request

Trainer callbacks [pass a model](https://github.com/huggingface/transformers/blob/a7cab3c283312b8d4de5df3bbe719971e24f4281/src/transformers/trainer_callback.py#L418) to any registered callback, but this model cannot be saved if training with DeepSpeed Stage 3 (needs access to `Trainer.accelerator` and `Trainer.model_wrapped`)

### Motivation

I have a custom callback that logs a model to a tracking server at the end of training, but need access to `Trainer.accelerator` and `Trainer.model_wrapped` in my callback to prevent skipping saving sharded tensors. I believe this may lead to a bug in the official `wandb` callback which uses a ["fake trainer"](https://github.com/huggingface/transformers/blob/b6c9f47fd6f911450024c52e382e544e5d04387a/src/transformers/integrations/integration_utils.py#L850) (I don't use `wandb` so can't confirm, but see an error similar to one reported in another repo if I try the "fake trainer" approach in my custom callback e.g., https://github.com/OpenAccess-AI-Collective/axolotl/issues/1092)

### Your contribution

I can contribute this feature, but wanted to get guidance on the design. Roughly

* `trainer_callback.CallbackHandler` receives an additional arg, `accelerator`, which will be passed `Trainer.accelerator` on instantiation
* `trainer_callback.CallbackHandler` has an attribute `model_wrapped` which gets updated with `Trainer.model_wrapped` by `Trainer`
* `trainer_callback.CallbackHandler.call_event` will pass along `self.accelerator` and `self.model_wrapped` when calling  callbacks

This allows my callback to do something like `state_dict = accelerator.get_state_dict(model_wrapped)` and pass that along to `model.save_pretrained` as to not skip saving sharded tensors

If there is a better design or if this is better classified as a bug, please let me know

Comments:
cc @muellerzr @SunMarc 

Support saving models trained with DeepSpeed to wandb

Supoort it, because if we want to monitor per layers grad in the callback or wandb, we also need to call `self.model_wrapped` in callback~

Also interested in this feature request: I want to save checkpoints when using deepspeed stage 3. Currently I want to upload them to huggingface but empty tensors get pushed.
Add DINOv2 with registers
Author: NielsRogge
Pull request: yes
Labels: name: New model

name: Vision

# What does this PR do?

This PR adds DINOv2 with registers, this time using the new modular tool.

Fixes #27379

To do:

- [x] make @bernardzach a co-author

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35348). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for review, revert the drop path update cause it wasn't need. Looks like we can merge :)

One thing left: All model checkpoints are on the hub, would need to be transferred to the facebook org: https://huggingface.co/models?other=dinov2_with_registers (and then replace `nielsr` by `facebook` everywhere)



@NielsRogge Would it be possible to adapt the `interpolate_pos_encoding` function to use the same approach as Dinov2 (but just with options for antialiasing):

https://github.com/huggingface/transformers/blob/6e0515e99c39444caae39472ee1b2fd76ece32f1/src/transformers/models/dinov2/modeling_dinov2.py#L78-L114

This current version has issues with tracing of the graph (see https://github.com/huggingface/transformers/pull/33226).
Drop inplace operation for loss computation with gradient accumulation
Author: qgallouedec
Pull request: yes

# What does this PR do?

Fix 

```
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
```

reported here: https://huggingface.co/answerdotai/ModernBERT-base/discussions/9#6769e0390519344317755b3b

Initially introduced in #35207

The solution does solve the issue, tested locally.
 
## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr @SunMarc @ArthurZucker who reviewed #35207

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35416). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

~Not related, but relate to the same lines: https://github.com/huggingface/transformers/pull/35207#issuecomment-2562016482 @yzhangcs~ Ignore: point raised wrong
`GPT2Attention()` class with `_attn()` method when `is_cross_attention=True`.
Author: CHLEE-Leo
Pull request: no
Labels: name: New model

### Model description

It seems like `GPT2Attention()` class only allows`_attn()` method with `causal_mask` when `is_cross_attention=False`, but not with when `is_cross_attention=True`. 

It would be more productive if `GPT2Attention()` supports `_attn()` method with `causal_mask` even with `is_cross_attention=True`.

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
CLIP Tokenizer Returns Different DTypes for Numpy versus Pytorch
Author: ivanthewebber
Pull request: no
Labels: name: bug

### System Info


- `transformers` version: 4.46.2
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.12.7
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.5.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
from transformers import AutoProcessor

texts = ["a photo of a cat", "A photo of a dog."]

for model_id in ("openai/clip-vit-base-patch16", "openai/clip-vit-large-patch14-336"):
    tokenizer = AutoProcessor.from_pretrained(model_id)
    print(model_id)
    # `CLIPTokenizerFast.from_pretrained` and no kwargs gives string dtype
    # However, no kwargs with AutoProcessor fails with warning
    for i, kwargs in enumerate((dict(padding='longest'), dict(padding='longest', truncation=True))):
        pytorch_inputs = tokenizer(text=texts, return_tensors='pt', **kwargs)
        for k, v in pytorch_inputs.items():
            # int64 (correct)
            print(f"pytorch {i} {k}: {v.dtype} {v.shape}")

        numpy_inputs = tokenizer(text=texts, return_tensors='np', **kwargs)
        for k, v in numpy_inputs.items():
            # int32 (incorrect)
            print(f"numpy {i} {k}: {v.dtype} {v.shape}")
    print()
```

### Expected behavior

Creating numpy and/or pytorch tensors should have equivalent data types.

Comments:
I am assuming the pytorch dtypes are correct because that's the model's framework.

@ivanthewebber I'm not sure one of these is more 'correct', since all of the numbers should be inside the `int32` range regardless. Is there an issue caused by the dtype difference?

Okay, I wasn't sure if that was the case so I thought truncation was possible. I mainly just thought it was inconsistent, but with that knowledge it's easy to cast them. I was just trying to export the model to onnx and verify it. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.