Add support for Apple's Depth-Pro
Author: geetu040
Pull request: yes
Labels: name: New model

name: Vision

# What does this PR do?

Fixes #34020 

This PR adds Apple's Depth Pro model to Hugging Face Transformers. Depth Pro is a foundation model for zero-shot metric monocular depth estimation. It leverages a multi-scale vision transformer optimized for dense predictions. It downsamples an image at several scales. At each scale, it is split into patches, which are processed by a ViT-based (Dinov2) patch encoder, with weights shared across scales. Patches are merged into feature maps, upsampled, and fused via a DPT decoder.

Relevant Links
- Research Paper: [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://arxiv.org/pdf/2410.02073)
- Authors: [Aleksei Bochkovskii](https://arxiv.org/search/cs?searchtype=author&query=Bochkovskii,+A), [Amaël Delaunoy](https://arxiv.org/search/cs?searchtype=author&query=Delaunoy,+A), and others
- Implementation: [apple/ml-depth-pro](https://github.com/apple/ml-depth-pro)
- Models Weights: [apple/DepthPro](https://huggingface.co/apple/DepthPro)


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@amyeroberts, @qubvel


Comments:
I have implemented the foundational components of the model and manually loaded the weights to ensure that the architecture aligns with the original design and produces consistent output.

Below is a concise overview of the class hierarchy. I would greatly appreciate your feedback or any suggestions for improvements:

```
DepthProForDepthEstimation
├── depth_pro: DepthProModel
│   ├── encoder: DepthProEncoder
│   │   ├── patch_encoder: DepthProViT
│   │   │   ├── embeddings: DepthProViTEmbeddings
│   │   │   └── encoder: DepthProViTEncoder
│   │   ├── image_encoder: DepthProViT
│   │   │   ├── embeddings: DepthProViTEmbeddings
│   │   │   └── encoder: DepthProViTEncoder
│   ├── decoder: DepthProDecoder
│   └── fov_model: DepthProFOVModel
│       ├── encoder: DepthProViT
│       │   ├── embeddings: DepthProViTEmbeddings
│       │   └── encoder: DepthProViTEncoder
└── head: DepthProDepthEstimationHead
```

I have a couple of questions:

1. The `encoder: DepthProEncoder` outputs features processed at various scales, including hidden states from the intermediate layers of `ViTEncoder`. Currently, I use `BaseModelOutput`, returning all features in the `last_hidden_state` argument. Should I create a dedicated `ModelOutput` class for `DepthProEncoder`? If so, it should reside in the same file as the DepthPro classes since it is specific to them.

2. For handling the `FOV (Field of View)` output, would it be appropriate to create a class named `DepthEstimatorOutputWithFOV` in `transformers.modeling_outputs`, or should it also remain within the DepthPro context?

cc @pcuenca as well!

Hi @geetu040! Thanks for working on this model!

Regarding model outputs they should be written if you want to add a new argument or write better docs. In case of intermediate outputs you can store them in `BaseModelOutput.hidden_states`, for example [mllama](https://github.com/huggingface/transformers/blob/5e1fd4e204d81f2f66f8c164433e62ea5f4d0467/src/transformers/models/mllama/modeling_mllama.py#L1454-L1459) set default `output_hidden_states=True` and then select required hidden states from vision transformer.

@qubvel @pcuenca  Thanks, I have updated the code for hidden_states.

I still need an opinion with `fov` (field of view)
DepthPro returns the `predicted_depth` as well as the `fov` which is a scaler value.

The existing `DepthEstimatorOutput` class in [transformers/src/transformers/modeling_outputs.py](https://github.com/huggingface/transformers/blob/5e1fd4e204d81f2f66f8c164433e62ea5f4d0467/src/transformers/modeling_outputs.py#L1319) looks like this:
```py
class DepthEstimatorOutput(ModelOutput):
    loss: Optional[torch.FloatTensor] = None
    predicted_depth: torch.FloatTensor = None
    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
    attentions: Optional[Tuple[torch.FloatTensor, ...]] = None
```

Q1: Do I create a new class `DepthEstimatorOutputWithFOV` or update the existing class?
Q2: User should be given the option to turn the `FOV` on or off because calculating `FOV` requires extra processing. In this case should this parameter be a part of model initialization `DepthProForDepthEstimation(config, return_fov=True)` or should it be kept inside config.

Thanks @geetu040 

Q1:

```python
class DepthProDepthEstimatorOutput(DepthEstimatorOutput):
    fov: Optional[torch.FloatTensor] = None
```

This output can be returned in both cases: `fov=None` and not `None`.

Q2:

Yeah, this can be a parameter of the config, but also should be an argument in `forward` method to override the config parameter (similar to `output_hidden_states`)

Please, let me know if you have more questions!

> Yeah, this can be a parameter of the config, but also should be an argument in `forward` method to override the config parameter (similar to `output_hidden_states`)

This needs to be done during `__init__`,  because it requires `fov_model` (another vision transformer) to be initialized.

OK, got it! Then it should be done with config! And anyone can just load a model as following:

```python
model = DepthProForDepthEstimation(checkpoint, fov_model=True)
# or
model = DepthProForDepthEstimation(checkpoint, fov_model=False)
```

With such initialization `fov_model` param will be overridden in config

- currently an image is down-scaled to medium resolution (high / 2) and low resolution (high / 4)
- then patches are created from high, medium and low and concatenated.

I was wondering can we also give this option to the users to decide which scales to use, for example, a user tells in config to use these custom scales `image_scales=[0.6, 0.4, 0.3]`
- now an image will downscale to these 3 scales
- then patches are created from high and scaled images and concatenated.

@qubvel  I have looked into the code how this can be implemented, it is do-able and I can easily make this option available and I would prefer that, but I have to ask you as well, do you think this option should be given to the users?

Hi @geetu040, we try to avoid overcomplicated code with lots of parameters, the general rule is to get rid of different code paths / unused params that are not different across pretrained checkpoints. For this particular case, feel free to add it, but only in case it will not introduce extra complexity to the modeling code.

Hi @qubvel  I have a question about the image processor.

the source code from `apple/depth-pro` preprocesses the image in this sequence `normalize -> resize`, however in conventional image processor for `vit and dpt`, the sequence is `resize -> normalize`

this causes the two outputs to be slightly different from each other.

do you suggest I stay with the convention and ignore the minor difference in output or I make the implementation exactly like the source code, I am not very sure how to do this because the original `resize` function gives an error if it is simply moved above normalization code and if I use `torch.nn.funtional.interpolate` that is also not very optimal, it requires data conversions.

Here are the outputs
<details>

**Different in Outputs**

there is a slight difference, this happens because of how the image is pre-processed before being given to the model

Source code results
```
ic| depth: tensor([[0.9604, 0.9329, 0.8837,  ..., 3.0123, 2.9720, 2.9517],
                   [0.9210, 0.8995, 0.8605,  ..., 3.0148, 3.0120, 3.0106],
                   [0.8811, 0.8655, 0.8366,  ..., 3.0245, 3.0473, 3.0592],
                   ...,
                   [1.2283, 1.2263, 1.2225,  ..., 1.2698, 1.2818, 1.2881],
                   [1.2228, 1.2241, 1.2266,  ..., 1.2679, 1.2806, 1.2872],
                   [1.2167, 1.2223, 1.2333,  ..., 1.2655, 1.2757, 1.2810]])
ic| depth.shape: torch.Size([2268, 3024])
ic| focallength_px: tensor(3362.0200)
```

HF code results
```
ic| predicted_depth: [tensor([[0.9727, 0.9443, 0.8937,  ..., 3.0023, 2.9608, 2.9399],
                             [0.9320, 0.9097, 0.8693,  ..., 3.0045, 3.0006, 2.9987],
                             [0.8899, 0.8737, 0.8439,  ..., 3.0129, 3.0352, 3.0469],
                             ...,
                             [1.2393, 1.2373, 1.2334,  ..., 1.2805, 1.2934, 1.3001],
                             [1.2344, 1.2356, 1.2379,  ..., 1.2802, 1.2935, 1.3004],
                             [1.2286, 1.2341, 1.2447,  ..., 1.2788, 1.2892, 1.2947]])]
ic| fov: [tensor(3383.9839)]
```

---

**Difference in Output Image**

visually no difference in the 2 images

Input Image
![example](https://github.com/user-attachments/assets/a3892bfa-511e-4c71-8fe5-79ec3fed812b)

Source code results
![Figure_1](https://github.com/user-attachments/assets/48936816-0d69-49cc-a10a-ab59d9a3180e)

HF code results
![Figure_2](https://github.com/user-attachments/assets/1c3e14e0-2702-41aa-90df-5bb6cef3cdef)

</details>

Also how does the weight conversion work?

I have created the script for weight conversion, but when and who uploads that on huggingface? because I would need these converted weights for examples in docstring.

Hi @geetu040! Thanks for the questions with examples!

> do you suggest I stay with the convention and ignore the minor difference in output or I make the implementation exactly like the source code,

We usually reproduce input preprocessing as closely as possible. So its better to copy the code and add required modifications.

> I am not very sure how to do this because the original resize function gives an error if it is simply moved above normalization code and if I use torch.nn.funtional.interpolate that is also not very optimal, it requires data conversions.

Thats interesting, we might need to fix it in a separate PR. Resize should work with and without normalization. Can you create a separate issue, please?

> Also how does the weight conversion work?
I have created the script for weight conversion, but when and who uploads that on huggingface? because I would need these converted weights for examples in docstring.

First, you create all checkpoints in your personal HF hub account and list them here. Also in the code, you refer to your own checkpoints. When PR is going to be at the final stage we will move the checkpoint to Apple's arg account and then you can replace them in the code before merging.
P.S. for conversion script format please refer to `mllama` model script.


Let me know if you have any follow-up questions!

> Thats interesting, we might need to fix it in a separate PR. Resize should work with and without normalization. Can you create a separate issue, please?

I have created the issue here: https://github.com/huggingface/transformers/issues/34920

since this is a problem with `PIL`, can I use `torch interpolation` here instead? as I will be creating `ImageProcessorFast` as well that already uses `torch` and the original code also uses torch interpolation for resizing.

Thanks for opening the issue! Feel free to use torch interpolation for now. I will try to figure out the best solution for this problem. Thanks!

@qubvel another question regarding weight conversion script

Why do you insist on `mllama` script? the scripts from `vit`, `dinov2` and `dpt` look more familiar for `depth-pro`. Is it because the conventions followed in `mllama` are better and new?

@geetu040 yes, please use `mllama`-like format, we are going to use it for all new models

@qubvel should I create `DepthProForSemanticSegmentation` class as well? In the paper they don't discuss segmentation tasks or segmentation datasets and no weights are available for this, but someone might want to try this, but they can use same depth estimation class for segmentation as well, shouldn't it work?

Hey @geetu040! Let's have only `ForDepthEstimation` if there are no checkpoints for segmentation.

https://github.com/huggingface/transformers/blob/1ed1de2fecdb5fe82f67e4fe742417f737370324/tests/test_modeling_common.py#L767-L774

`cosine_similarity` doesnot work for tensors with just one item
- `tensor1 = [0.4]` and `tensor2 = [0.8]` or let it be any non-zero number, the result is always 0
- `tensor1 = [0.0]` or `tensor2 = [0.0]` and it always results in 1

I am getting this error on `fov` which is a 1-D tensor of length batch_size

@qubvel do you suggest, I make some changes in the `test_modeling_common.py`?

Hey @geetu040, I believe it's better to override this method in model-specific tests

@qubvel some of these tests are failing, I think they are not related to this PR, can you please confirm?

[ci/circleci: tests_generate](https://app.circleci.com/pipelines/github/huggingface/transformers/112842/workflows/fcdf3e2c-7bbb-446b-bc84-4b53967c3d80/jobs/1506354/parallel-runs/2)
```
FAILED tests/models/fuyu/test_modeling_fuyu.py::FuyuModelTest::test_prompt_lookup_decoding_matches_greedy_search - RuntimeError: The size of tensor a (2) must match the size of tensor b (0) at non-singleton dimension 1
```

[ci/circleci: tests_non_model](https://app.circleci.com/pipelines/github/huggingface/transformers/112842/workflows/fcdf3e2c-7bbb-446b-bc84-4b53967c3d80/jobs/1506358)
```
FAILED tests/utils/test_modeling_utils.py::ModelUtilsTest::test_from_pretrained_low_cpu_mem_usage_equal - AssertionError: 423.62109375 != 425.75 within 2 delta (2.12890625 difference) : using `low_cpu_mem_usage` should incur the same memory usage in both cases.
```

@geetu040, yes it seems they are unrelated to this PR

@qubvel Thanks for all the help!

This PR is complete and ready for review.

Failing tests are unrelated to this PR.

some minor fixes I realised
- `image_encoder` and `fov_encoder` uses image scaled to `patch_size` instead of `scaled_images_ratios[0]` as suggested in the paper
- `DepthProFeatureFusionStage` now returns `fused_hidden_states` for each `hidden_state`, just like `DPT`, as someone can make use of the intermediate `fused_hidden_state` 
- fixed the output shape in example for `DepthProModel`

Should be ready for review now, failing test seems unrelated to this PR

@qubvel Thanks for the detailed review. I have address all, except for a few where I have left a question or explanation
- [patch/batch transposition](https://github.com/huggingface/transformers/pull/34583#discussion_r1894173861)
- [using standard embeddings](https://github.com/huggingface/transformers/pull/34583#discussion_r1894162631)
- [torch.jit.trace](https://github.com/huggingface/transformers/pull/34583#discussion_r1894187870)

Hi @geetu040! Thanks for addressing the questions, I'm on holidays until Jan 3rd, and I'll do another review after that.

@qubvel I am tying to re-consider the architecture, while using the `ViT` as a backbone model that can be switched with other `ViT` variates.
It would require some changes in the code of `DinoV2`.
If there is anything I should be looking forward to, please let me know.

Hi @geetu040, regarding changes for DinoV2 it's actually depends on changes, if you provide a bit more context we can discuss it here 🤗 

As far as I see from the paper the standard ViT architecture should be usable for the purpose, so the idea is to construct patches from different scales, concat, and then pass them all together through the ViT, it would be nice if we could utilize `AutoModel` or `load_backbone` to avoid copy-pasting ViT code for this model. Let me know if there any specific issues that have to be solved to make it possible

> we could utilize `AutoModel` or `load_backbone` to avoid copy-pasting ViT code for this model

Agreed! I am working on it. I'll use `AutoModel`, since the backbone is expected to return `hidden_states` instead of `feature_maps` for `DepthPro`

> regarding changes for DinoV2 it's actually depends on changes, if you provide a bit more context we can discuss it here 🤗

I'll see that in detail, if it needs some changes when using the backbone

@qubvel I have updated the model to use the variates of vit using the `backbone_config` and it should be ready for review now

I have 2 questions now
1. [patch/batch transposition](https://github.com/huggingface/transformers/pull/34583#discussion_r1896842549) which is still unresolved, I have provided the update in the same thread
2. segmentation head: I have asked this [previously](https://github.com/huggingface/transformers/pull/34583#issuecomment-2503110979), and you [suggested](https://github.com/huggingface/transformers/pull/34583#issuecomment-2503528810) to leave it, but I still think I should add it. `DepthEsimationHead` is same for `DPT` and `DepthPro`, I can also create the `SegmentationHead` for `DepthPro` same as it is for `DPT`. It would just add few lines of code but it produces really good results for segmentations tasks, I believe someone might need that for experiments.

@qubvel a soft ping!

Hi @geetu040! Thanks for the ping and sorry for the delay, I have a few PRs in a line, so will try to review tomorrow / the day after tomorrow, thanks for your patience 🤗 

@qubvel Thanks for the review, I really appreciate your feedback.

---

Answering your questions ...

---

> 1. Do any of `patch_encoder`, `image_encoder`, or `fov_model`, which are initialized from the same config, share weights? If so, we should either `tie` those parameters or just initialize them as a single instance.

They are completely independent architectures and don't share any weights

---

> 2\. I have a warning message: `Some weights of DepthProForDepthEstimation were not initialized from the model checkpoint at geetu040/DepthPro and are newly initialized: `

I actually planned to put these in `DepthProPreTrainedModel.__keys_to_ignore_on_load_missing`, because even if they are loaded they are never used in the forward pass. I'll see if plain `ViT` works, but let me know if you think "keys_to_ignore" might be a solution.

---

> 3\. The DepthProEncoder forward method is too long.

Noted!

---

> 4\. The solution using `unfold` for the `split` function is excellent, but I am not sure the `merge` function effectively removes padding.

That's interesting.
If `padding==0`, the `merge` function is 3x faster than original code, and 1.5x slower otherwise.
I'll work on the padding operation or the overall function, considering the ONNX and compile/export constraints.

1. Ok, thanks for clarifying! I suppose architectures are the same, because they are initialized from the same config, but weights are not shared, right?
2. It's better to use proper architecture if possible
4. Interesting, thanks for benchmarking, we can actually have both functions and use one for `padding=0` case and double loop for another one (if you don't find anything better).

Thanks!

Hi @qubvel,

Since you mentioned compile/export, I wanted to try this locally.
To check compilation, I simply did `torch.compile(model)` and the input seems to work without any warning or error on different padding sizes.
Then for onnx export, I was following this [tutorial](https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html) and when I try to export the model, I get a bunch of warning messages.
<details>

code
```py
config = DepthProConfig(...)
model = DepthProForDepthEstimation(config)
dummy_inputs = torch.rand((2, 3, 256, 256))
out = model(dummy_inputs)

# Export the model
import torch.onnx
torch.onnx.export(
    model,                             # model being run
    dummy_inputs,                      # model input (or a tuple for multiple inputs)
    "custom_model.onnx",               # where to save the model (can be a file or file-like object)
    export_params=True,                # store the trained parameter weights inside the model file
    opset_version=14,                  # the ONNX version to export the model to
    do_constant_folding=True,          # whether to execute constant folding for optimization
    input_names = ['input'],           # the model's input names
    output_names = ['output'],         # the model's output names
    dynamic_axes={                     # variable length axes
        'input' : {0 : 'batch_size'},
        'output' : {0 : 'batch_size'}
    }
)
```

warnings
```
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:448: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if not (num_channels == self.config.num_channels):
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:453: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if min(self.scaled_images_ratios) * min(height, width) < self.config.patch_size:
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:287: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if height == width == patch_size:
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:485: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  n_patches_per_scaled_image = [len(i) for i in scaled_images]
/home/geetu/work/hf/transformers/src/transformers/models/dinov2/modeling_dinov2.py:164: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:531: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  out_size = torch_int(math.sqrt(image_encodings.last_hidden_state.shape[1]))
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:532: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  exponent_value = torch_int(math.log2(width / out_size))
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:307: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  size = torch_int(math.sqrt(seq_len))
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:322: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  sqrt_n_patches_per_batch = torch_int(math.sqrt(n_patches_per_batch))
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:325: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if n_patches == batch_size:
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:330: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if n_patches_per_batch < 4:
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:341: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  padding = min(out_size // 4, padding)
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:362: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if padding > 0:
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:375: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  *[torch.arange(index, index + out_size - padding * 2) for index in starting_indexes],
```

</details>

I have a few questions:
- Am I doing the compile/export correctly, or is there a some other guide you may refer me?
- Do I need to worry about these warnings and fix them all?
- Were you talking about these warnings when you said, "I am not sure the merge function effectively removes padding. First of all, compile/export do not work with this approach, and it might be a problem for ONNX export.", [here](https://github.com/huggingface/transformers/pull/34583#pullrequestreview-2567395939)?

Hi @geetu040! It depends on the warning, but such a warning sounds not that good
```
/home/geetu/work/hf/transformers/src/transformers/models/depth_pro/modeling_depth_pro.py:375: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  *[torch.arange(index, index + out_size - padding * 2) for index in starting_indexes],
  ```
  
To check compile compatibility it's better to use `fullgraph=True` option enabled, this way we can ensure there are no graph breaks and we can achieve better speedup. 

```python
compiled_model = torch.compile(model, fullgraph=True)
with torch.no_grad():
    compiled_mode = compiled_mode(**inputs)
```

Also, do not forget to pass inputs at least once, because actually compilation happens "just in time", not "ahead of time".

Thanks for looking into these issues 🤗 


@qubvel, the code is ready for review

Here is the change log, you may already see this through the commit messages, but here I reiterate:
- padding operation in merge_patches
	- I tried to optimize the code in this commit: https://github.com/huggingface/transformers/pull/34583/commits/83706b85b6382a12cda1544674e6f36e90626403 but that didnot solve the onnx problem and was slower than original implementation for padding > 0, thus I switched to using both the implementations depending on the padding value.
	- now compile (full_graph) and export (onnx) works safely
- moved the common code in `forward` pass to a new function `feature_extractor`, I may give it a better name later.
- added `config.use_mask_token` in `dinov2` to fix the mask token weight in backbone, and the changes are all backward compatible
- created new `DepthProFeatureUpsampleBlock` for modularity
- major changes in backbone logic
	- each of the encoder uses a config of its own and `DepthProConfig` uses the `sub_configs` attribute
	- rest of the logic, I tried to copy from `ColPaliConfig` which seemed most familiar
- reflected the changes in weight conversion script and tests
- the failing test is irrelevant to this PR

Hey @geetu040, thanks a lot for iterating and following up on the changes, it's super helpful 🤗 

I will review the code changes tomorrow, here are some comments based on follow up:

> - padding operation in merge_patches
	- I tried to optimize the code in this commit: https://github.com/huggingface/transformers/pull/34583/commits/83706b85b6382a12cda1544674e6f36e90626403 but that didnot solve the onnx problem and was slower than original implementation for padding > 0, thus I switched to using both the implementations depending on the padding value.
	- now compile (full_graph) and export (onnx) works safely

Awesome, thanks!

> - moved the common code in `forward` pass to a new function `feature_extractor`, I may give it a better name later.

The naming convention for function is a bit different, it's better to name something like `extract_features`, but I would also appreciate a bit more descriptive name as well.

> - added `config.use_mask_token` in `dinov2` to fix the mask token weight in backbone, and the changes are all backward compatible

Ok, I appreciate it's backward compatible, will review it carefully. Did you tried using ViT?

> - created new `DepthProFeatureUpsampleBlock` for modularity

Super nice, thanks!

> - major changes in backbone logic
	- each of the encoder uses a config of its own and `DepthProConfig` uses the `sub_configs` attribute
	- rest of the logic, I tried to copy from `ColPaliConfig` which seemed most familiar

Nice! Will check this out

> - reflected the changes in weight conversion script and tests
- the failing test is irrelevant to this PR

Awesome!



> Did you tried using ViT?

yes I tried that, but the architecture doesnot work, it is quite different from dinov2

@qubvel Thanks for the review again!

I have made all the changes and replied to these comments: https://github.com/huggingface/transformers/pull/34583#discussion_r1932167399, https://github.com/huggingface/transformers/pull/34583#discussion_r1932550653

> One more question left on my side is very long message for unused weights while specifying `use_fov_model=False`. I think we should come up on how to avoid it, otherwise make two different checkpoints (with and without fov head).

@qubvel does this look like a safe solution?
```py
class DepthProPreTrainedModel(PreTrainedModel):
    ...
    # a list of `re` patterns of `state_dict` keys that should be removed from the list of
    # unexpected keys we find (keys inside the checkpoint but not the model) and avoid unnecessary
    # warnings.
    _keys_to_ignore_on_load_unexpected = ['fov_model.*']
    ...
```
just add a line here and `PreTrainedModel` handles the rest. And if this doesnot look like the solution we are looking for, then I have the question: what is a use-case where we might use `_keys_to_ignore_on_load_unexpected`?

@qubvel I also replied to a comment here https://github.com/huggingface/transformers/pull/34583#discussion_r1935089316

And you can access the google colab notebook, which I don't think is necessary now, but this was the particular discussion https://github.com/huggingface/transformers/pull/34583#discussion_r1925492430, for which I wrote some code on [google colab](https://colab.research.google.com/drive/1YSLmQmvlhHYzQd1wwYVRWobXDdfZZ-w9#scrollTo=X5YZV8f8v5rP)

And also, reading through [your code](https://github.com/qubvel/transformers-notebooks/blob/main/notebooks/DepthPro_inference.ipynb), I think may be you are inferring and showing how the estimated-depth changes with and without fov, but that is not really the case. `raw predicted depth` doesnot use the fov at all, although the `post_processed depth` uses the fov, but your code is safe from that, since it compares the `post_processed depth` only.
And the code looks really nice, Thanks for that.

@qubvel 
I have opened a PR on [huggingface/documentation-images #429](https://huggingface.co/datasets/huggingface/documentation-images/discussions/429) to add another image for the docs, please see in `depth_pro.md` where I have used it, and that's different from the one you already added there.

I have completed the docs, tested slow tests, merged the latest code from main branch and am only left with the model card, which is outside this PR. Guess the PR is open for review again. Thanks.

Hey @geetu040,  thanks for the iteration!

`_keys_to_ignore_on_load_unexpected = ['fov_model.*']` seems to be legit, let's leave it this way!

Doc builder fails to build with the following error:

```python
Traceback (most recent call last):
  File "/usr/local/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.10/site-packages/doc_builder/commands/build.py", line 102, in build_command
    build_doc(
  File "/usr/local/lib/python3.10/site-packages/doc_builder/build_doc.py", line 371, in build_doc
    sphinx_refs = check_toc_integrity(doc_folder, output_dir)
  File "/usr/local/lib/python3.10/site-packages/doc_builder/build_doc.py", line 478, in check_toc_integrity
    raise RuntimeError(
RuntimeError: The following files are not present in the table of contents:
- model_doc/depth_pro
Add them to ../transformers/docs/source/en/_toctree.yml.
```

Multi GPU test fails with (we should add all layers that have residual connections to `_no_split_modules`)

```
self = DepthProPreActResidualLayer(
  (activation1): ReLU()
  (convolution1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1...g=(1, 1))
  (activation2): ReLU()
  (convolution2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
)
hidden_state = tensor([[[[-6.0393e-01,  7.8733e-01, -1.0852e+00,  ...,  8.8840e-01,
           -1.[267](https://github.com/huggingface/transformers/actions/runs/13075912051/job/36487986571#step:12:268)9e+00,  3.4131e-01],
          [...01,  ...,  1.3227e+00,
            1.6042e+00,  6.5902e-01]]]], device='cuda:1',
       grad_fn=<ConvolutionBackward0>)

    def forward(self, hidden_state: torch.Tensor) -> torch.Tensor:
        residual = hidden_state
        hidden_state = self.activation1(hidden_state)
    
        hidden_state = self.convolution1(hidden_state)
    
        if self.use_batch_norm:
            hidden_state = self.batch_norm1(hidden_state)
    
        hidden_state = self.activation2(hidden_state)
        hidden_state = self.convolution2(hidden_state)
    
        if self.use_batch_norm:
            hidden_state = self.batch_norm2(hidden_state)
    
>       return hidden_state + residual
E       RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
```

run-slow: depth_pro

This comment contains run-slow, running the specified jobs: ['models/depth_pro'] ...

> A few final comments! Waiting for model cards updated, tests fixed and comments addressed, then passing it to review for @ ArthurZucker

FYI: @ArthurZucker 

@qubvel I have updated the model card here: [geetu040/DepthPro](https://huggingface.co/geetu040/DepthPro)
- Please check the [assets folder](https://huggingface.co/geetu040/DepthPro/tree/main/assets), should these images go to [documentation-images](https://huggingface.co/datasets/huggingface/documentation-images/tree/main/transformers/model_doc)?
- I mistakenly added a space to the file name, that I have created to a PR to fix here: [#430](https://huggingface.co/datasets/huggingface/documentation-images/discussions/430). When fixed the image will be correctly rendered on the docs and model card pages.

Let me know if I need to change anything else anywhere.

@geetu040 thanks for the updates! Looks great, I suppose the final comments re code (on my side) are above 🤗 

>@qubvel I have updated the model card here: [geetu040/DepthPro](https://huggingface.co/geetu040/DepthPro)
Please check the [assets folder](https://huggingface.co/geetu040/DepthPro/tree/main/assets), should these images go to [documentation-images](https://huggingface.co/datasets/huggingface/documentation-images/tree/main/transformers/model_doc)?
I mistakenly added a space to the file name, that I have created to a PR to fix here: [#430](https://huggingface.co/datasets/huggingface/documentation-images/discussions/430). When fixed the image will be correctly rendered on the docs and model card pages.

Great model card! Thank you!
 - PR for space correction is merged
 - Re assets, if those images are not used in docs (depth_pro.md) then you can leave them in the repo

@qubvel, ready for review again.
- resolved above conversations
- model card updated (weights and docs)
- slow tests are passing locally
- merged branch main

run-slow: depth_pro

This comment contains run-slow, running the specified jobs: ['models/depth_pro'] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34583). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@qubvel everything is updated, a bunch of test are failing, though none related to this PR.

I need to address the latest changes from main

- I have updated the code to use the latest `BaseImageProcessorFast` class.
- Failing tests are unrelated.
- PR ready for review again 

@geetu040 please see https://github.com/huggingface/transformers/pull/34583#discussion_r1942987644 re adding `antialias`

@qubvel grouping in one loop was a great suggestion [here](https://github.com/huggingface/transformers/pull/34583#discussion_r1942727429), glad you noticed that. Grouping them twice was unnecessary, since the size of images did not change after the first group. Thanks for that!

Well this did speed up the fast processor and in fact, it's faster than slow processor now. So I am not skipping the test anymore. I was using `is_flanky` before because locally it does fail sometimes sporadically, but it seems to fail for other processors locally as well (sporadically). But it's working fine in the CI tests (atleast for the last 2/2 workflows), so I'll avoid flanky tag.

And everything is up-to-date now, ready for review again. Failing tests are unrelated.

Thanks for applying the changes! 🤗 We’re all set to proceed with the model.  

Before merging, there’s just one thing left: we usually transfer the checkpoint to the organization that released the original model, in this case - Apple, but only if you’re okay with that.  

If you're fine with it, we can go ahead with the transfer. However, before that we’ll need to:  
- Change the Hub repository name from **"DepthPro"** to **"depth-pro-hf"**  
- Update the checkpoint name in the model card snippet  
- Update the checkpoint names in the code  

Let us know how you’d like to proceed! 😊

> Apple, but only if you’re okay with that.

I am okay with that.

> If you're fine with it, we can go ahead with the transfer. However, before that we’ll need to:

I have updated the hub repository name: https://huggingface.co/geetu040/depth-pro-hf
and updated the checkpoint in model card and code

And Thank you @qubvel for all the fine detailed reviews.

Thanks, can you please also update repo_id to "apple" everywhere? Like "apple/depth-pro-hf"

> Thanks, can you please also update repo_id to "apple" everywhere? Like "apple/depth-pro-hf"

updated

@qubvel, `DepthProImageProcessingTest::test_fast_is_faster_than_slow` has failed in the last (2/5) runs, do you think I should put `is_flanky` decorator on it? It's guaranteed (with a very high probability) that it passes every time.

Yes, let's add `@is_flaky()`

> Yes, let's add `@is_flaky()`

I've updated the test and its working fine now

Thanks! woking on checkpoint transfer

[Checkpoint has been transferred](https://huggingface.co/apple/DepthPro-hf).

run-slow: depth_pro

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/depth_pro']
quantizations: [] ...

> [Checkpoint has been transferred](https://huggingface.co/apple/DepthPro-hf).

@qubvel Shouldn't the checkpoint be `apple/depth-pro-hf` instead of `apple/DepthPro-hf` as you suggested? The code uses `apple/depth-pro-hf` everywhere.

@geetu040 I followed the same [pattern for the model family](https://huggingface.co/collections/apple/depthpro-models-66fee63b2f0dc1b231375ca6). Let me check if there's anything we can do.

> @geetu040 I followed the same [pattern for the model family](https://huggingface.co/collections/apple/depthpro-models-66fee63b2f0dc1b231375ca6). Let me check if there's anything we can do.

Otherwise I can change the checkpoint in the code, and it would also require to be changed in the model card.

https://huggingface.co/apple/depth-pro-hf now redirects to `DepthPro-hf`. So code should work.

But good point about the model card, I'll change so it's less confusing. Edit: updated.

Thanks @pcuenca!

@geetu040 can you please update the code references? sorry for the back-and-forth changes.

Waiting for the slow CI to merge it then.

> @geetu040 can you please update the code references? sorry for the back-and-forth changes.

sure, updated!

run-slow: depth_pro

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/depth_pro']
quantizations: [] ...

@qubvel, the tests did not run in the last 2 attempts.
<details>
<summary> Error Log </summary>

```
Run PR_MERGE_SHA=$(git log -1 --format=%H)
  PR_MERGE_SHA=$(git log -1 --format=%H)
  if [ $PR_MERGE_SHA != $VERIFIED_PR_MERGE_SHA ]; then
    echo "The merged commit SHA is not the same as the verified one! Security issue detected, abort the workflow!";
    exit -1;
  fi
  shell: sh -e {0}
  env:
    HF_HOME: /mnt/cache
    TRANSFORMERS_IS_CI: yes
    OMP_NUM_THREADS: 8
    MKL_NUM_THREADS: 8
    RUN_SLOW: yes
    HF_HUB_READ_TOKEN: ***
    SIGOPT_API_TOKEN: ***
    TF_FORCE_GPU_ALLOW_GROWTH: true
    RUN_PT_TF_CROSS_TESTS: 1
    CUDA_VISIBLE_DEVICES: 0,1
    matrix_folders: models_depth_pro
    VERIFIED_PR_MERGE_SHA: 850bdaaad25826da72b87e9455296742bb83e331
The merged commit SHA is not the same as the verified one! Security issue detected, abort the workflow!
/__w/_temp/881ad002-6e8a-4896-968a-9a2b93d19337.sh: 4: exit: Illegal number: -1
Error: Process completed with exit code 2.
```

</details>

Looks like something is wrong with the workflow itself.

@geetu040 yes, smth wrong with CI, waiting for the team to fix it

run-slow: depth_pro


This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/depth_pro']
quantizations: [] ...

@geetu040, congratulations on getting the model merged! 🎉🚀
Fantastic work! Huge thanks for all your iterations to make the modeling code simpler, faster, and more export-friendly. It was a pleasure to support you in contributing this model, thank you!

Thanks @qubvel 

Hi!

I know this is merged already but I wanted to point out a small detail missing compared to Apple's implementation.

In the `load_rgb()` method in Apple's implementation, they check for the focal length in the EXIF data and extract it if it exists.

Reference: [Apple's implementation `load_rgb()` method](https://github.com/apple/ml-depth-pro/blob/b2cd0d51daa95e49277a9f642f7fd736b7f9e91d/src/depth_pro/utils.py#L100-L110)

Then later, although by default they predict the FOV if `use_fov_head == True`, they only use the predicted FOV to calculate the focal length if there was none in the EXIF data.

Reference: [Apple's implementation `depth_pro.py`](https://github.com/apple/ml-depth-pro/blob/b2cd0d51daa95e49277a9f642f7fd736b7f9e91d/src/depth_pro/depth_pro.py#L281-L283)

I think this would be good to add to Hugging Face's implementation since it removes one layer of estimation, lowering the margin of error.

Hi @Boulaouaney, thanks a lot for pointing this out! Would you like to propose a PR to update it?

Hi @qubvel and @Boulaouaney 
`FOV` value is used only by the `DepthProImageProcessor.post_process_depth_estimation`, can one not update the outputs like this?
```py
...
model = DepthProForDepthEstimation.from_pretrained("apple/DepthPro-hf", use_fov_model=False)
outputs = model(**inputs)

# use the infered fov from given image
outputs.field_of_view = torch.tensor([0.4])

post_processed_output = image_processor.post_process_depth_estimation(
    outputs, target_sizes=[(image.height, image.width)],
)
...
```

Hi @geetu040,

Yes, you're right. One could manually adjust the image processing if they know the focal length used to take the image. However, I still believe it would be more efficient to handle this automatically within the image processor. Automating this process ensures consistency and reduces the potential for human error. Additionally, it would streamline the workflow for users who may not be familiar with the technical details of image processing.


@qubvel yes, if necessary I can open a PR when I'm free to update this

@Boulaouaney I understand, so you are suggesting that we extract the FOV value as well along with the RGB values from the raw input image right?

I'm not sure if we need to read it in the image processor, because I'm not sure how consistent it is among image formats. Currently, image/metadata reading is the user's responsibility. However, we should clearly describe this in the documentation and the model card. Also, it's better to avoid modifying `output` and to make FOV an optional argument for the postprocessing method.

@geetu040 yes I am suggesting to extract the focal length in 35mm from the image's metadata if it exists and convert it to focal length in pixels `f_px` to be used for calculating the metric depth instead of the estimated value from the `fov_model`.


The logic to be added would be similar to apple's implementation:
```python
# Extract focal length at 35mm from exif data
f_35mm = img_exif.get(
    "FocalLengthIn35mmFilm",
    img_exif.get(
        "FocalLenIn35mmFilm", img_exif.get("FocalLengthIn35mmFormat", None)
    ),
)

# if extracted, convert it to pixels
if f_35mm is not None:
    f_px = f_35mm * np.sqrt(width**2.0 + height**2.0) / np.sqrt(36**2 + 24**2)
```

But I guess @qubvel 's solution to leave the metadata handling to the user during image loading would also be good. 

Ok, it would be nice to mention it in docs (`depth_pro.md`), feel free to open PR to clarify it, then we can update examples and model cards on our side.