noamaanMulla-03/transformers-issues huggingface dataset:
Add support for Apple's Depth-Pro
Author: geetu040
Pull request: no
Labels: name: New model

name: Vision

### Model description

**Depth Pro: Sharp Monocular Metric Depth in Less Than a Second.**

Depth Pro synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

- Research Paper: [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://arxiv.org/pdf/2410.02073)
- Authors: [Aleksei Bochkovskii](https://arxiv.org/search/cs?searchtype=author&query=Bochkovskii,+A), [Amaël Delaunoy](https://arxiv.org/search/cs?searchtype=author&query=Delaunoy,+A), and others
- Implementation: [apple/ml-depth-pro](https://github.com/apple/ml-depth-pro)
- Models Weights: [apple/DepthPro](https://huggingface.co/apple/DepthPro)

Comments:
If someone from the core-team is not already working on this, I would really love to contribute this model to huggingface with some help. Thanks.

cc: @pcuenca

Very cool! 🙌

I'll let someone from the transformers Vision team answer about how best to proceed.

Hi @geetu040 thanks for your interest, that might be a great addition to current depth estimation models in transformers! We would greatly appreciate your contribution!

Here is ["how to add a model guideline"](https://huggingface.co/docs/transformers/add_new_model) and some models that might be useful for understanding general patterns of the code in transformers:

 - [ViT](https://github.com/huggingface/transformers/tree/main/src/transformers/models/vit)
 - [ZoeDepth](https://github.com/huggingface/transformers/tree/main/src/transformers/models/zoedepth)
 - [DPT](https://github.com/huggingface/transformers/tree/main/src/transformers/models/dpt)

Conversion script should follow [mllama conversion script format](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mllama/convert_mllama_weights_to_hf.py)
 
Feel free to open a PR and ping me if you encounter any issues. I'd be happy to help!

> Feel free to open a PR and ping me if you encounter any issues. I'd be happy to help!

Thank you, I have already started working on this!
Support Constant Learning Rate with Cooldown
Author: LoserCheems
Pull request: no
Labels: name: Feature request

### Feature request

In `transformers.optimization` support `constant learning rate with cooldown` functions.


### Motivation

This method will implement that scaling experiments can be performed with significantly reduced compute and GPU hours by utilizing fewer but reusable training runs.
[SmolLM](https://huggingface.co/blog/smollm) had used this method to train a series of SOTA small language models.
Paper: [https://arxiv.org/pdf/2405.18392](https://arxiv.org/pdf/2405.18392)

### Your contribution

I've created a branch, I'm finishing the implementation of these functions, and intend to submit a PR.

Comments:
Enable pytest live log and show warning logs on GitHub Actions CI runs
Author: ydshieh
Pull request: yes

We want to have warnings being logged on GitHub Actions run.

(currently on `main`, only those given by `warnings.warn(...)` are saved when using `--make-reports`)

Comments:
run-slow: convnextv2

run-slow: convnextv2

This comment contains run-slow, running the specified jobs: ['models/convnextv2'] ...

run-slow: convnextv2

This comment contains run-slow, running the specified jobs: ['models/convnextv2'] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35912). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

See a run with the live log (for warnings)

https://github.com/huggingface/transformers/actions/runs/13008702695/job/36281380509

run-slow: convnextv2

This comment contains run-slow, running the specified jobs: ['models/convnextv2'] ...

run-slow: convnextv2

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/convnextv2']
quantizations: [] ...

run-slow: convnextv2

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/convnextv2']
quantizations: [] ...
ImportError: cannot import name 'NoneType' from 'types' on main in Python 3.9
Author: harupy
Pull request: no
Labels: name: bug

### System Info

Linux

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
docker run python:3.9 bash -c 'pip install git+https://github.com/huggingface/transformers && python -c "import transformers"'
```

Output:

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "/usr/local/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "/usr/local/lib/python3.9/site-packages/transformers/utils/__init__.py", line 27, in <module>
    from .chat_template_utils import DocstringParsingException, TypeHintParsingException, get_json_schema
  File "/usr/local/lib/python3.9/site-packages/transformers/utils/chat_template_utils.py", line 22, in <module>
    from types import NoneType
ImportError: cannot import name 'NoneType' from 'types' (/usr/local/lib/python3.9/types.py)
```

### Expected behavior

No import error

Comments:
https://github.com/huggingface/transformers/blob/f4f33a20a23aa90f3510280e34592b2784d48ebe/src/transformers/utils/chat_template_utils.py#L22

`types.None` only exists in Python >= 3.10.

https://docs.python.org/3/library/types.html#types.NoneType

This should be resolved in the latest patch, and apologies!
Revert checkpoint tmp dir
Author: SunMarc
Pull request: yes

# What does this PR do ? 

Reverts https://github.com/huggingface/transformers/pull/35580 and https://github.com/huggingface/transformers/pull/36094

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36112). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
The docstring doesn't match the file name in utils/check_table.py
Author: kkscilife
Pull request: no
Labels: name: bug

### System Info

![Image](https://github.com/user-attachments/assets/848f8d76-1b73-4294-84aa-83768bf37cc1)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

check file docstring

### Expected behavior

change into correct file name

Comments:
Yes, this seems incorrect - @kkscilife would you be willing to open a PR to fix the docstring?

> Yes, this seems incorrect - [@kkscilife](https://github.com/kkscilife) would you be willing to open a PR to fix the docstring?

Done in https://github.com/huggingface/transformers/pull/36108
fix bnb warning
Author: SunMarc
Pull request: yes

# What does this PR do?
Fixes https://github.com/huggingface/transformers/issues/35581, switched warning to info instead ! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36116). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

SG thanks !
[generate] shape checks in tests compatible with fixed-length caches (+ some minor fixes)
Author: gante
Pull request: yes

# What does this PR do?

Fixes a few test cases exposed by #33212 [e.g. some models are failing cache shape checks, if their default cache is not a dynamic cache]. Fix added in a separate PR to avoid making #33212 a huge PR :)

This PR:
1. Updates generate output checks to also work with fixed-length caches;
2. Tests with `output_attentions=True` now explicitly use `eager` attention (`sdpa` doesn't return the attentions, `eager` was being used implicitly with warnings);
3. (enabled by 1. and 2.) Adds a test regarding generate + extra outputs + compile, which was not being tested; 
4. Standardizes the inconsistent method to pull the cache from the model outputs in the generation loop;
5. (enabled by 1.) Deletes unnecessary overwrites.
6. [EDIT, added after PR reviews] updates variable names used in common tests, so we can easily understand what's going on.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35993). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Before diving into more details, I think it's best to separate the fix of copies in a separate PR.

https://github.com/huggingface/transformers/pull/36063

we can revert the changes (in this PR) to those files once the above PR is merged.

@ydshieh should be ready for a re-review.

Two notes:
1. `_check_outputs` and especially its inner functions should be much easier to follow. LMK if any part is unclear—documenting these general functions is important to future-proof our test base!
2. As I rewrote the test functions, I noticed some overwrites were not necessary (or at least could be simplified) ✂️
AutoformerForPrediction test add atol
Author: ivarflakstad
Pull request: yes

Add `atol` to AutoformerForPrediction test to fix failing test
> ValueError: Both 'rtol' and 'atol' must be either specified or omitted, but got no atol.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36017). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Refactor OPT model
Author: jiqing-feng
Pull request: yes

Hi @ArthurZucker  @SunMarc . I removed useless codes for opt model including:
1. Remove `cros_attention` because `key_value_states` has never been passed
2. Remove `self.is_decoder` because it is fixed to `True`

I also checked it by `RUN_SLOW=1 pytest tests/models/opt/test_modeling_opt.py` and the results are the same as main branch.

After this PR is merged, I will enable static cache on opt model.

Comments:
Just fyi and as additional context: iirc it's a relic from using BART code via copied from but it deviated after some time making it inherent the unnecessary code.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36101). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Okay found it here! https://github.com/huggingface/transformers/pull/35724/files#r1918254784
Fix nighlty CIs: missing atols
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Few missing atols

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35903). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Updated documentation and added conversion utility
Author: viktor-shcherb
Pull request: yes

# What does this PR do?

Documentation improvement on tiktoken integration + tiktoken conversion function.

<!-- Remove if not applicable -->

Fixes #34221 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

@ArthurZucker
@stevhliu


Comments:
@ArthurZucker could not properly run `make fixup` due to some dependency problems, I think should be all good now

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34319). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add support for 4D custom attention masks in GPT-2
Author: sambhavnoobcoder
Pull request: yes

## Problem Statement
Currently, GPT-2's attention mechanism only supports 2D attention masks, limiting its flexibility for advanced use cases like packed sequence processing. When users attempt to use 4D attention masks (shape [batch_size, num_heads, seq_length, seq_length]), the model raises dimension mismatch errors.

Issue #35290  demonstrates this limitation when trying to process packed sequences with custom attention patterns.

## Proposed Solution
Extend GPT-2's attention mechanism to properly handle both 2D and 4D attention masks while maintaining backward compatibility. This allows for:
- Direct support for packed sequence processing
- More flexible attention patterns
- Compatibility with existing 2D mask implementations

## Implementation Details
The changes focus on the GPT2Attention class, specifically:
1. Updated attention mask handling in the forward pass
2. Maintained compatibility with existing 2D attention masks
3. Preserved causal attention behavior when required

## Testing Strategy
Added comprehensive test suite (`test_modeling_4D_attention_mask.py`) that verifies:
- Shape compatibility with 4D masks
- Correctness of attention patterns
- Consistency between 2D and 4D mask results
- Causal attention preservation
- Batch processing consistency
- Edge cases (empty sequences, single tokens, maximum length)

## Test Results
All tests passed successfully. Screenshot of test results:
<img width="1242" alt="Screenshot 2025-01-06 at 3 49 52 AM" src="https://github.com/user-attachments/assets/6e48d191-7b5b-45ea-a3e8-278f02b41cbe" />

## Impact and Benefits
This enhancement:
1. Enables efficient packed sequence processing
2. Provides more flexibility in attention pattern design
3. Maintains backward compatibility
4. Improves model versatility without performance overhead

## Validation
- ✅ New test suite validates 4D mask functionality
- ✅ Backward compatible with existing 2D masks
- ✅ No performance regression

## Related Issues
Closes #35290  - Support for 4D attention masks in GPT-2

## Additional Notes
- No breaking changes introduced
- Existing model weights remain compatible
- Performance impact is negligible

requested reviewers - @ArthurZucker 

Comments:
@Rocketknight1 can you have a look at this as well , or perhaps tag the right maintainers for this ? i would give the comments on this a look right away , as well as open to any discussion on this issue as well . 

This looks okay to me, but since it affects core philosophy / modeling code, I'll need a second opinion from @arthurzucker!

Also, you can fix the failing tests in the CI by doing `pip install transformers[quality]` followed by `make fixup` in the repo directory, and then committing/pushing.

Done . now passes all tests .
Instructions to raise PR for addition of shared library files(.so) and .cpp files 
Author: vineel96
Pull request: no
Labels: name: Feature request

### Feature request

Hello @ArthurZucker  @zucchini-nlp @Rocketknight1 @gante,
I have an proposal for improvement of Mamba model for which I wanted to raise PR.  
**Method:** The code changes are made in c++ file. I use cython to integrate c++ code to python. For this I will be generating an shared library file (.so) which will have the code. I will import it as library in transformers python code(Ex: import library) and call API's (Ex: library.api_name() )  from it. 

I have **queries on how to raise a PR** for the above scenario:
1. In which folder path should I keep the cython c++ binding codes ?
2. How to provide the share library file (.so) file in the PR, In which location should I keep?

### Motivation

I will be detailing the motivation when I raise the PR for it. Now I need to instructions to raise the PR

### Your contribution

I will be submitting  a PR.

Comments:
 @ArthurZucker @zucchini-nlp @Rocketknight1 @gante, Any help regarding this?

We usually have kernels for CUDA/CPU here https://github.com/huggingface/transformers/tree/main/src/transformers/kernels, so I  believe mamba will also belong there

@zucchini-nlp , To bind c++ code with python this repo is using torch.utils.cpp_extension to load ".cpp" extension files and create shared file(.so) (https://github.com/huggingface/transformers/blob/main/src/transformers/models/mra/modeling_mra.py#L59 ) . But torch.utils.cpp_extension uses pybindings to bind c++ code with python. In our experiments, we found that **cython is faster than pybindings** when we call c++ code from python. So can we integrate using cython? or it is mandatory to follow what transformers repo uses by default?

Hey! We usually like to help contributors / want to avoid people having to compile anything. That's the same issue with `.so`  unless they are sufficienty abstract to work for all frameworks we avoid having them! 

For `mamba` we added a soft deps on https://github.com/alxndrTL/mamba.py for an example !
Deprecation Warning for `max_size` in `DetrImageProcessor.preprocess`
Author: HichTala
Pull request: no
Labels: name: WIP

name: Vision

name: Processing

### System Info

- `transformers` version: 4.47.0.dev0
- Platform: Linux-5.15.0-126-generic-x86_64-with-glibc2.31
- Python version: 3.11.0
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: 1.1.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 3090

### Who can help?

@amyeroberts, @qubvel
and I think @NielsRogge worked on it too ?

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import logging

import numpy as np
from transformers.models.detr.image_processing_detr import DetrImageProcessor

logging.basicConfig(level=logging.WARNING)
logger = logging.getLogger(__name__)

images = [np.ones((512, 512, 3))]
annotations = [{'image_id': [], 'annotations': []}]
size = {'max_height': 600, 'max_width': 600}

image_processor = DetrImageProcessor()
images = image_processor.preprocess(images, do_resize=True, do_rescale=False, size=size, annotations=annotations, format='coco_detection')
```

### Expected behavior

Hello!

I noticed that the `preprocess` method in the `DetrImageProcessor` class always passes `max_size` to the `resize` method,

https://github.com/huggingface/transformers/blob/4120cb257f03b834fb332e0b0ee6570245e85656/src/transformers/models/detr/image_processing_detr.py#L1445-L1447

 and that triggers a deprecation warning in `resize` method,

```bash
The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`.
```

https://github.com/huggingface/transformers/blob/4120cb257f03b834fb332e0b0ee6570245e85656/src/transformers/models/detr/image_processing_detr.py#L992-L997

I propose removing the unused `max_size` argument from the preprocess method since it is always `None`,

https://github.com/huggingface/transformers/blob/4120cb257f03b834fb332e0b0ee6570245e85656/src/transformers/models/detr/image_processing_detr.py#L1340

Would it be okay if I work on this and submit a pull request? I can try to see if the problem also occurs in other models.

Comments:
Hi @HichTala! Nice catch, would appreciate a PR!

Pinging @qubvel here
Fix CI issues 
Author: molbap
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fix oom issue in bamba generate. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35662). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
update tiktoken integ to use converted
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes https://github.com/huggingface/transformers/pull/34319#discussion_r1874545183 : we used `.tokenizer` instead of `.converted`! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36135). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.