Add Depth Anything V2 Metric models
Author: bt2513
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Add Depth Anything V2 models for _metric_ depth estimations

Fixes #31972 

## Before submitting
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@NielsRogge @amyeroberts

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @NielsRogge, I've just opened a PR to add the models. I plan to actively work on it this week. As this is my first contribution, I'll rely on your guidance if needed. Thank you! :)

Hi @NielsRogge - I've adapted the `DepthAnythingConfig` to support _metric_ depth estimation by introducing an attribute that enables easy switching of the model configuration. I've also modified the `convert_depth_anything_to_hf.py` script to validate my results against the original repo, and things look good.

As this is my first time contributing to add a model, I'd greatly appreciate your guidance on what steps to take next to get this merged. Thank you!

> PR is looking very nice! Did you confirm all slow tests are passing (RUN_SLOW=yes pytest tests/models/depth_anything)?
> 
> In that case, we can also add a slow integration test for the metric depth estimation checkpoints, similar to [here](https://github.com/huggingface/transformers/blob/49928892d6491ff5a49c12cbc34695f6fa7ac0ed/tests/models/depth_anything/test_modeling_depth_anything.py#L248).

Good catch, I actually forgot to run these tests! And they are not passing ... I don't think the failures are related to my changes though, as they seem to be about the backbone initialization:

<img width="745" alt="Screenshot 2024-07-29 at 6 35 19 PM" src="https://github.com/user-attachments/assets/de7983b7-6e77-4658-81f3-d05057babc30">

I'll investigate this and try to fix the failure

If I am reproducing the error correctly, which I replicated with the following code by retracing what happens I believe:
```
from transformers import TimmBackbone, Dinov2Config, AutoBackbone, TimmBackboneConfig
import timm

image_size=32
patch_size=16
hidden_size=4
num_hidden_layers=2
num_attention_heads=2
intermediate_size=8
is_training=True
out_features=["stage1", "stage2"]
reshape_hidden_states=False
num_channels=3

config = Dinov2Config(
    image_size=image_size,
    patch_size=patch_size,
    num_channels=num_channels,
    hidden_size=hidden_size,
    num_hidden_layers=num_hidden_layers,
    num_attention_heads=num_attention_heads,
    intermediate_size=intermediate_size,
    is_training=is_training,
    out_features=out_features,
    reshape_hidden_states=reshape_hidden_states,
)

config.backbone = "resnet18"
config.use_pretrained_backbone = True
config.use_timm_backbone = True
config.backbone_config = None
# For transformer backbones we can't set the out_indices or just return the features
config.backbone_kwargs = {"out_indices": (-2, -1)}

# backbone = AutoBackbone.from_pretrained(
#         config.backbone,
#         use_timm_backbone=config.use_timm_backbone,
#         use_pretrained_backbone=config.use_pretrained_backbone,
#         **config.backbone_kwargs,
#     )

backbone_config = getattr(config, "backbone_config", None)
use_timm_backbone = getattr(config, "use_timm_backbone", None)
use_pretrained_backbone = getattr(config, "use_pretrained_backbone", None)
backbone_checkpoint = getattr(config, "backbone", None)
backbone_kwargs = getattr(config, "backbone_kwargs", None)
backbone_kwargs = {} if backbone_kwargs is None else backbone_kwargs

backbone_config = TimmBackboneConfig()
num_channels = backbone_config.num_channels
features_only = backbone_config.features_only
use_pretrained_backbone = backbone_config.use_pretrained_backbone
out_indices = backbone_config.out_indices
config = TimmBackboneConfig(
    backbone=config.backbone,
    num_channels=num_channels,
    features_only=features_only,
    use_pretrained_backbone=use_pretrained_backbone,
    out_indices=out_indices,
)

# TimmBackbone(config)

out_indices = config.out_indices if getattr(config, "out_indices", None) is not None else (-1,)
_backbone = timm.create_model(
        config.backbone,
        pretrained=config.use_pretrained_backbone,
        # This is currently not possible for transformer architectures.
        features_only=config.features_only,
        in_chans=config.num_channels,
        out_indices=out_indices,
    )

# _return_layers = {
#         layer["module"]: str(layer["index"]) for layer in _backbone.feature_info.get_dicts()
#     }
print(_backbone.feature_info.get_dicts())
```
The error is due because `_backbone.feature_info.get_dicts()` seems to return `[{'num_chs': 512, 'reduction': 32, 'module': 'layer4'}]`, and there is indeed no key "index" in the dict. Any guidance on how to fix this issue? I am not sure what this attribute should return ideally in [modeling_timm_backbone.py](https://github.com/huggingface/transformers/blob/f0bc49e7f61f74f055c47ad40e6010f57eed0b0b/src/transformers/models/timm_backbone/modeling_timm_backbone.py#L85)

If I comment out the timm backbone test (i.e. [these lines](https://github.com/huggingface/transformers/blob/main/tests/models/depth_anything/test_modeling_depth_anything.py#L220-L227)), the tests pass successfully :) 

> If I am reproducing the error correctly, which I replicated with the following code by retracing what happens I believe:
> 
> ```
> from transformers import TimmBackbone, Dinov2Config, AutoBackbone, TimmBackboneConfig
> import timm
> 
> image_size=32
> patch_size=16
> hidden_size=4
> num_hidden_layers=2
> num_attention_heads=2
> intermediate_size=8
> is_training=True
> out_features=["stage1", "stage2"]
> reshape_hidden_states=False
> num_channels=3
> 
> config = Dinov2Config(
>     image_size=image_size,
>     patch_size=patch_size,
>     num_channels=num_channels,
>     hidden_size=hidden_size,
>     num_hidden_layers=num_hidden_layers,
>     num_attention_heads=num_attention_heads,
>     intermediate_size=intermediate_size,
>     is_training=is_training,
>     out_features=out_features,
>     reshape_hidden_states=reshape_hidden_states,
> )
> 
> config.backbone = "resnet18"
> config.use_pretrained_backbone = True
> config.use_timm_backbone = True
> config.backbone_config = None
> # For transformer backbones we can't set the out_indices or just return the features
> config.backbone_kwargs = {"out_indices": (-2, -1)}
> 
> # backbone = AutoBackbone.from_pretrained(
> #         config.backbone,
> #         use_timm_backbone=config.use_timm_backbone,
> #         use_pretrained_backbone=config.use_pretrained_backbone,
> #         **config.backbone_kwargs,
> #     )
> 
> backbone_config = getattr(config, "backbone_config", None)
> use_timm_backbone = getattr(config, "use_timm_backbone", None)
> use_pretrained_backbone = getattr(config, "use_pretrained_backbone", None)
> backbone_checkpoint = getattr(config, "backbone", None)
> backbone_kwargs = getattr(config, "backbone_kwargs", None)
> backbone_kwargs = {} if backbone_kwargs is None else backbone_kwargs
> 
> backbone_config = TimmBackboneConfig()
> num_channels = backbone_config.num_channels
> features_only = backbone_config.features_only
> use_pretrained_backbone = backbone_config.use_pretrained_backbone
> out_indices = backbone_config.out_indices
> config = TimmBackboneConfig(
>     backbone=config.backbone,
>     num_channels=num_channels,
>     features_only=features_only,
>     use_pretrained_backbone=use_pretrained_backbone,
>     out_indices=out_indices,
> )
> 
> # TimmBackbone(config)
> 
> out_indices = config.out_indices if getattr(config, "out_indices", None) is not None else (-1,)
> _backbone = timm.create_model(
>         config.backbone,
>         pretrained=config.use_pretrained_backbone,
>         # This is currently not possible for transformer architectures.
>         features_only=config.features_only,
>         in_chans=config.num_channels,
>         out_indices=out_indices,
>     )
> 
> # _return_layers = {
> #         layer["module"]: str(layer["index"]) for layer in _backbone.feature_info.get_dicts()
> #     }
> print(_backbone.feature_info.get_dicts())
> ```
> 
> The error is due because `_backbone.feature_info.get_dicts()` seems to return `[{'num_chs': 512, 'reduction': 32, 'module': 'layer4'}]`, and there is indeed no key "index" in the dict. Any guidance on how to fix this issue? I am not sure what this attribute should return ideally in [modeling_timm_backbone.py](https://github.com/huggingface/transformers/blob/f0bc49e7f61f74f055c47ad40e6010f57eed0b0b/src/transformers/models/timm_backbone/modeling_timm_backbone.py#L85)

Thanks for the review @amyeroberts! 🙌 I'll address your comments in a bit :)
Any insights on the above? This currently prevents the slow tests for depth anything to pass, and I am not sure how to resolve it.  cc @NielsRogge 

Hi @NielsRogge @amyeroberts, I've updated the checkpoints in the tests. Once they are transferred from my account to the organization, things should be good I believe. Please let me know if you have any additional feedback :) 

@bt2513 Great - I'll let @NielsRogge let us know when the checkpoints have been updated :) 

Is the error [with the backbone](https://github.com/huggingface/transformers/pull/32126#issuecomment-2259145967) still an issue?  

> @bt2513 Great - I'll let @NielsRogge let us know when the checkpoints have been updated :)
> 
> Is the error [with the backbone](https://github.com/huggingface/transformers/pull/32126#issuecomment-2259145967) still an issue?

@amyeroberts Yes, it is! It's actually not resolved yet - my bad, I had this part commented out to ensure that the other tests passed. Could you please help me with this? I am not sure what the attribute should return ideally in [modeling_timm_backbone.py](https://github.com/huggingface/transformers/blob/f0bc49e7f61f74f055c47ad40e6010f57eed0b0b/src/transformers/models/timm_backbone/modeling_timm_backbone.py#L85), so it's hard for me to debug

> > @bt2513 Great - I'll let @NielsRogge let us know when the checkpoints have been updated :)
> > Is the error [with the backbone](https://github.com/huggingface/transformers/pull/32126#issuecomment-2259145967) still an issue?
> 
> @amyeroberts Yes, it is! It's actually not resolved yet - my bad, I had this part commented out to ensure that the other tests passed. Could you please help me with this? I am not sure what the attribute should return ideally in [modeling_timm_backbone.py](https://github.com/huggingface/transformers/blob/f0bc49e7f61f74f055c47ad40e6010f57eed0b0b/src/transformers/models/timm_backbone/modeling_timm_backbone.py#L85), so it's hard for me to debug

Actually, all the tests in [test_modeling_timm_backbone.py](https://github.com/huggingface/transformers/blob/main/tests/models/timm_backbone/test_modeling_timm_backbone.py) also fail for me when I run them. I'm not sure if this is an issue with my local setup, if I am missing something, or if something has changed
<img width="1070" alt="Screenshot 2024-08-02 at 9 19 56 PM" src="https://github.com/user-attachments/assets/64a0d9f4-ccc4-4e4b-855f-4bd2d3abfac2">
Do they pass for you @amyeroberts @NielsRogge? It's the same errors I get when running `test_modeling_depth_anything` (as mentioned in my previous comments), so if you don't encounter this issue, it might be that the tests are actually fine and this is just an issue with my setup

Thanks for transferring the weights, @NielsRogge! The only PR blocker now is the issue mentioned above. Can you help resolve this? 🙏 Once it's fixed, we should be able to merge 🥳
cc @amyeroberts

Hi @bt2513 it can be solved by running `pip install --upgrade timm`.

All tests pass for me.

> Hi @bt2513 it can be solved by running `pip install --upgrade timm`.
> 
> All tests pass for me.

Oh, thanks @NielsRogge! You're right, I thought I had done this, my bad 🤦🏻‍♂️
I believe everything looks good then :) I don't have merge permissions, so I am deferring to you for the next steps. Thanks!

Got it @amyeroberts, that makes sense. I just made the changes to address your feedback, please let me know if there are any other things you'd like me to change :) thanks!

Hi @amyeroberts @NielsRogge, bumping this in case you missed my last changes :) 

@bt2513 Looks great - thanks for iterating! Only thing left to do is to run the slow tests on our CI runners. Could you push a commit with `[run_slow] depth_anything` as a messsage? This should trigger the tests (I'll need to approve the work flow). We may need to update the integration test values a little as there can be slight numerical differences given hardware and envs 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32126). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts it looks like the `PR slow CI / Run all tests for the model (models/depth_anything, single-gpu) (pull_request)` job failed, but I can't see any error message when I click on details - except "This job failed". Should we just try to re-run it or is there anything else you can see that can help fix it? Please let me know :)

@bt2513 thanks, I reported it to team, the infrastructure is being updated. Will rerun when that's finished

Congrats on this amazing contribution @bt2513 :) would be great to share about it on social media, which we'll help amplify!