Add post_process_depth_estimation to image processors and support ZoeDepth's inference intricacies
Author: alex-bene
Pull request: yes
Labels: name: Vision

name: Processing

# What does this PR do?

This PR adds a `post_process_depth_estimation` method for the image processors of depth estimation models, similar to the `post_process_semantic_segmentation` methods for the segmentation models. Also, it updates the depth estimation pipeline to use the new `post_process_depth_estimation` method. Lastly, it adds full support for the ZoeDepth *special* inference (dynamically padded input + inference of the flipped of the input). A small update to the documentation is pending.

Fixes #30917 #32381 

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@NielsRogge @amyeroberts @Narsil

Comments:
I'm not sure who to tag about this, but I have added to the `image_transforms.py` file the function `colorize_depth`, which, given a depth prediction, generates a colored PIL image or numpy image. If this is not the correct place for this function let me know. @NielsRogge

Hey everyone, @NielsRogge @amyeroberts @Narsil, is there anything missing from this PR so that it can be merged?

@amyeroberts let me know if there's anything else that needs changing.

@amyeroberts kind reminder

Linking to the outstanding comment here, in case it's been lost in review, as it's always hard to find again in github: https://github.com/huggingface/transformers/pull/32550#discussion_r1757124860

Hey @amyeroberts , I finished up with the last pending comments; let me know if it seems ok.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32550). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @amyeroberts , in the docs, I have added a code block and a link inside a "tip" block that did not render markdown assets as usual, as seen in the image. Do you have a good idea of how to fix this?

<img width="821" alt="image" src="https://github.com/user-attachments/assets/a219ca4e-49a7-420d-9ad7-b6b653995f3c">

@alex-bene Tbh, I'm not sure. I'd suggest just removing the tip tags altogether -- the section follows on directly from the part above, so I don't think this will impact the message in the docs much 

> @alex-bene Tbh, I'm not sure. I'd suggest just removing the tip tags altogether -- the section follows on directly from the part above, so I don't think this will impact the message in the docs much

Hey @amyeroberts , I was thinking of doing something like this if it makes sense:
```html
<Tip>
<p>In the <a href="https://github.com/isl-org/ZoeDepth/blob/edb6daf45458569e24f50250ef1ed08c015f17a7/zoedepth/models/depth_model.py#L131">original implementation</a> ZoeDepth model performs inference on both the original and flipped images and averages out the results. The <code>post_process_depth_estimation</code> function can handle this for us by passing the flipped outputs to the optional <code>outputs_flipped</code> argument:</p>
<pre><code class="language-Python">&gt;&gt;&gt; with torch.no_grad():   
...     outputs = model(pixel_values)
...     outputs_flipped = model(pixel_values=torch.flip(inputs.pixel_values, dims=[3]))
&gt;&gt;&gt; post_processed_output = image_processor.post_process_depth_estimation(
...     outputs,
...     source_sizes=[image.size[::-1]],
...     outputs_flipped=outputs_flipped,
... )
</code></pre>
</Tip>
```

> > @alex-bene Tbh, I'm not sure. I'd suggest just removing the tip tags altogether -- the section follows on directly from the part above, so I don't think this will impact the message in the docs much
> 
> Hey @amyeroberts , I was thinking of doing something like this if it makes sense:
> 
> ```
> <Tip>
> <p>In the <a href="https://github.com/isl-org/ZoeDepth/blob/edb6daf45458569e24f50250ef1ed08c015f17a7/zoedepth/models/depth_model.py#L131">original implementation</a> ZoeDepth model performs inference on both the original and flipped images and averages out the results. The <code>post_process_depth_estimation</code> function can handle this for us by passing the flipped outputs to the optional <code>outputs_flipped</code> argument:</p>
> <pre><code class="language-Python">&gt;&gt;&gt; with torch.no_grad():   
> ...     outputs = model(pixel_values)
> ...     outputs_flipped = model(pixel_values=torch.flip(inputs.pixel_values, dims=[3]))
> &gt;&gt;&gt; post_processed_output = image_processor.post_process_depth_estimation(
> ...     outputs,
> ...     source_sizes=[image.size[::-1]],
> ...     outputs_flipped=outputs_flipped,
> ... )
> </code></pre>
> </Tip>
> ```

You can always try and see how it renders :) 

Yeah, it seems to have done the trick @amyeroberts -- I think we're good!

<img width="916" alt="image" src="https://github.com/user-attachments/assets/a69df46b-a9c0-4465-a333-ab7e7dec9b58">


@alex-bene Great! Could you resolve the conflicts? Once done I think we're good to merge! 

cc @qubvel 

@amyeroberts @qubvel I think we're done, but let me know if there's anything else pending.

@qubvel is it now ready to merge?

@qubvel @amyeroberts kind reminder

@alex-bene thanks for ping!

cc @ArthurZucker for final review

Ok thanks, merging as you answered 🤗 

Hi @alex-bene Thank you for the contribution!

There are 2 failing tests after this PR is merged to `main`:

(GLPN model)
```bash
FAILED tests/models/glpn/test_modeling_glpn.py::GLPNModelTest::test_pipeline_depth_estimation - AttributeError: 'GLPNImageProcessor' object has no attribute 'post_process_depth_estimation'

FAILED tests/models/glpn/test_modeling_glpn.py::GLPNModelTest::test_pipeline_depth_estimation_fp16 - AttributeError: 'GLPNImageProcessor' object has no attribute 'post_process_depth_estimation'
```

Could you take a look? Thanks in advance.

[job run page](https://app.circleci.com/pipelines/github/huggingface/transformers/108316/workflows/035f3ea5-9d3e-4a61-b9dc-9e454b9a7e74/jobs/1439527)



Hey @ydshieh , that's because this is a new model not available when we initially added the post processing interface for depth models. I'll try to make the necessary changes until ~tomorrow.

Hey @ydshieh , the fix for this is on #34413