Add Prompt Depth Anything Model
Author: haotongl
Pull request: yes
Labels: name: New model

name: Vision

name: run-slow

# What does this PR do?

This PR adds the **Prompt Depth Anything Model**. [Prompt Depth Anything](https://promptda.github.io/) builds upon [Depth Anything V2](https://depth-anything-v2.github.io/) and incorporates **metric prompt depth** to enable accurate and high-resolution metric depth estimation.

The implementation leverages [Modular Transformers](https://huggingface.co/docs/transformers/main/en/modular_transformers). The main file can be found [here](https://github.com/haotongl/transformers/blob/modeling_prompt_depth_anything/src/transformers/models/prompt_depth_anything/modular_prompt_depth_anything.py).


## Before submitting
- [ N/A] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). 
- [ ✅] Did you read the [contributor guideline] (https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ N/A] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ✅] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ✅] Did you write any new necessary tests?



Comments:
@NielsRogge @qubvel  @pcuenca  Could you help review this PR when you have some time? Thanks so much in advance! Let me know if you have any questions or suggestions. 😊

Hi @haotongl! Thanks for working on the model integration to transformers 🤗 I'm on holidays until Jan 3rd, and I'll do a review after that if it's still necessary.

Hi, @xenova @NielsRogge ! All suggestions have been addressed. Could you please take another look and provide any further suggestions, or go ahead and merge this PR? Thanks!

@qubvel @NielsRogge All suggestions have been addressed. Could you please take another look and provide any further suggestions, or go ahead and merge this PR? Thanks!

 Hi @qubvel, the issues from this round of reviews have been addressed. Could you please take a look?
 
The only issue is with padding in prompt depth anything image processing. Currently, padding is not supported in our image processor, as it may cause misalignment between the prompt depth and image features at the pixel level. I believe we can either remove the padding from the image processor or leave it as it is. The second option would only cause a slight shift.

> Hi @haotongl, thanks for the update, I think we are mostly done, great!
> 
> A few steps remain:
> 
> 1. fix torch.export test (it fail in my env)
> 
> ```
> (strict=True) SUBFAIL tests/models/prompt_depth_anything/test_modeling_prompt_depth_anything.py::PromptDepthAnythingModelIntegrationTest::test_export - torch._dynamo.exc.Unsupported: Observed exception
> (strict=False) SUBFAIL tests/models/prompt_depth_anything/test_modeling_prompt_depth_anything.py::PromptDepthAnythingModelIntegrationTest::test_export - NotImplementedError: Training is not implemented yet
> ```
> 
> 2. Add model cards for checkpoints on the HF Hub
> 3. Correct minor comments below (just nits)
> 
> And I will pass the model to the final review for Arthur Zucker.
> 
> I've also prepared example notebook, please have a look https://github.com/qubvel/transformers-notebooks/blob/main/notebooks/PromptDepthAnything_inference.ipynb

@qubvel Thank you for your efforts. I have fixed these steps. It seems that the torch.export test failure does not occur in the torch 2.5.1 and 2.4.1 environments, and it passes the checks from the transforms repository, so there doesn't seem to be any issue. Could you please share the final review for Arthur Zucker?

run-slow: prompt_depth_anything

Hi @haotongl, thanks for the update! Can you please merge the `main` branch so I can trigger slow tests in CI? Also it seems that model cards do not contain `transformers` snippets

> Hey @haotongl! I found the cause of the test failuire 🤗 see the corrections below. Also it seems docs like docs build failed, can you have a look?
> 
> Other than that looks nice 🤗
> 
> cc @ArthurZucker for review

@qubvel  Thank you! I am trying to fix it!

@haotongl Would you mind using "Update brunch" button to merge `main`?

> @haotongl Would you mind using "Update brunch" button to merge `main`?

@qubvel Yeah, I have finished "update branch".

The last commit is inconsistent with pattern of other models, can you please revert?

@qubvel 

> The last commit is inconsistent with pattern of other models, can you please revert?

Yeah, I reverted it. I had made the change because some tests reported errors, indicating that having two return statements is not allowed.

@qubvel 
Many thanks for your assistance!
I have found that the current test failures are due to issues in external models, rather than in our model (prompt depth anything). There are several errors originating from external libraries, and I am unable to reproduce them locally. Could you please provide some guidance on how to resolve these issues?

For reference, the tests I ran locally are as follows:
`RUN_SLOW=1 pytest -n 8 tests/models/prompt_depth_anything/test_modeling_prompt_depth_anything.py
`
The results were:
```
41 tests passed
109 tests skipped
145 warnings
Total time: 54.59 seconds
```

run-slow: prompt_depth_anything

This comment contains run-slow, running the specified jobs: ['models/prompt_depht_anything', 'models/prompt_depth_anything'] ...

~~Please, re-run modular conversion to fix repo-consistency, because the `modular_` file have been updated~~

Thanks

run-slow: prompt_depth_anything


This comment contains run-slow, running the specified jobs: ['models/prompt_depth_anything'] ...

Multi-gpu test failed with the following error
```python
        predicted_depth = self.head(hidden_states, patch_height, patch_width)
        if prompt_depth is not None:
            # denormalize predicted depth
            depth_min, depth_max = depth_min.squeeze(1), depth_max.squeeze(1)
>           predicted_depth = predicted_depth * (depth_max - depth_min) + depth_min
E           RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
```

I suppose can be fixed with 

```python
depth_min = depth_min.squeeze(1).to(predicted_depth.device)
depth_max = depth_max.squeeze(1).to(predicted_depth.device)
predicted_depth = predicted_depth * (depth_max - depth_min) + depth_min
```

> Multi-gpu test failed with the following error
> 
> ```python
>         predicted_depth = self.head(hidden_states, patch_height, patch_width)
>         if prompt_depth is not None:
>             # denormalize predicted depth
>             depth_min, depth_max = depth_min.squeeze(1), depth_max.squeeze(1)
> >           predicted_depth = predicted_depth * (depth_max - depth_min) + depth_min
> E           RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
> ```
> 
> I suppose can be fixed with
> 
> ```python
> depth_min = depth_min.squeeze(1).to(predicted_depth.device)
> depth_max = depth_max.squeeze(1).to(predicted_depth.device)
> predicted_depth = predicted_depth * (depth_max - depth_min) + depth_min
> ```

Thank you for your suggestion! I have fixed it.



run-slow: prompt_depth_anything

run-slow: prompt_depth_anything

This comment contains run-slow, running the specified jobs: ['models/prompt_depth_anything'] ...

I've fixed the document error.
run-slow: prompt_depth_anything
It seems that only members can run this test. @qubvel, could you please help with performing this check?


run-slow: prompt_depth_anything

Yes, only members can run slow tests

This comment contains run-slow, running the specified jobs: ['models/prompt_depth_anything'] ...

run-slow: prompt_depth_anything

This comment contains run-slow, running the specified jobs: ['models/prompt_depth_anything'] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35401). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> run-slow: prompt_depth_anything

Thank you! All checks have passed. @qubvel 

Thanks for fixing the tests! Waiting for the final review form @ArthurZucker 

Friendly ping @ArthurZucker 🤗 

@ArthurZucker Hey, just a friendly ping to see if you’ve had a chance to look at the PR. Appreciate your help when you get a moment! 

Super sorry i will get to this soon 😭 

A few small conflicts to fix

> Kudos it's super clean 🤗 Thanks a lot @haotongl and @qubvel Mostly missing a licence and good to go!

Thanks for your reviews! I am working on these suggestions.

run-slow: prompt_depth_anything

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/prompt_depth_anything']
quantizations: [] ...

Thanks for supporting! @qubvel @ArthurZucker 

Thanks for adding the model @haotongl! Can you please update model cards in HF repos to include the `transformers` code snippet?

> Thanks for adding the model @haotongl! Can you please update model cards in HF repos to include the `transformers` code snippet?

@qubvel  Yeah, could you please provide some example repos for reference?

Sure, here is an example of the model card for another depth estimation model https://huggingface.co/apple/DepthPro-hf
You can make smth similar to the model doc page - https://huggingface.co/docs/transformers/main/en/model_doc/prompt_depth_anything

@qubvel Thanks for your help! Should I open a new PR  to update the doc?

I mean for these models:

https://huggingface.co/depth-anything/prompt-depth-anything-vits-hf
https://huggingface.co/depth-anything/prompt-depth-anything-vitl-hf
https://huggingface.co/depth-anything/prompt-depth-anything-vits-transparent-hf

model cards have **not** a `transformers` library code snippet, but it should be similar to the one in [docs](https://huggingface.co/docs/transformers/main/en/model_doc/prompt_depth_anything)

```python
import torch
import requests
import numpy as np

from PIL import Image
from transformers import AutoImageProcessor, AutoModelForDepthEstimation

url = "https://github.com/DepthAnything/PromptDA/blob/main/assets/example_images/image.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("depth-anything/prompt-depth-anything-vits-hf")
model = AutoModelForDepthEstimation.from_pretrained("depth-anything/prompt-depth-anything-vits-hf")

prompt_depth_url = "https://github.com/DepthAnything/PromptDA/blob/main/assets/example_images/arkit_depth.png?raw=true"
prompt_depth = Image.open(requests.get(prompt_depth_url, stream=True).raw)
# the prompt depth can be None, and the model will output a monocular relative depth.

# prepare image for the model
inputs = image_processor(images=image, return_tensors="pt", prompt_depth=prompt_depth)

with torch.no_grad():
    outputs = model(**inputs)

# interpolate to original size
post_processed_output = image_processor.post_process_depth_estimation(
    outputs,
    target_sizes=[(image.height, image.width)],
)

# visualize the prediction
predicted_depth = post_processed_output[0]["predicted_depth"]
depth = predicted_depth * 1000 
depth = depth.detach().cpu().numpy()
depth = Image.fromarray(depth.astype("uint16")) # mm
```

https://huggingface.co/depth-anything/prompt-depth-anything-vits-hf
Hi, @qubvel , should it be like this?

Yes, this one looks good, thanks!