Add OneFormer Model
Author: praeclarumjj3
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Adds the Code, Documentation, and Tests for OneFormer proposed in [OneFormer: One Transformer to Rule Universal Image Segmentation](https://arxiv.org/abs/2211.06220). I have also opened a [PR](https://huggingface.co/datasets/huggingface/documentation-images/discussions/11) to add the documentation images to `huggingface/documentation-images`.

I have also made changes to the `ImageSegmentationPipeline` to accommodate OneFormer.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten @NielsRogge

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @praeclarumjj3, thanks a lot for your PR. It's awesome OneFormer will be available in the library (we already have [MaskFormer](https://huggingface.co/docs/transformers/model_doc/maskformer) and plan to add Mask2Former as well).

I've got 2 main points for now:

## Backbones

However, there's no need to implement backbones from scratch again, as we've just added the `AutoBackbone` class, which allows to use frameworks like DETR, Mask R-CNN, and also OneFormer with all vision backbones available in the library. The idea is to add an `xxxBackbone` class to each vision model, see for instance [here](https://github.com/huggingface/transformers/blob/699e90437f984d69ad3c9b891dd2e9d0fc2cffe4/src/transformers/models/resnet/modeling_resnet.py#L434) for ResNet. 

Next, the framework (like OneFormer) can use the `AutoBackbone` class as shown [here](https://github.com/huggingface/transformers/blob/699e90437f984d69ad3c9b891dd2e9d0fc2cffe4/src/transformers/models/maskformer/modeling_maskformer.py#L1385) for MaskFormer. This allows to mix-and-match backbones with a given framework.

The plan is to next add `SwinBackbone`, `ConvNextBackbone`, as well as `NatBackbone` and `DinatBackbone` => which will make sure OneFormer can use them.

## Auto class

I doubt there's a need for an `AutoModelForUniversalSegmentation` class, as OneFormer is probably the only class which will ever be supported by it. It'd be great to make OneFormer work with our existing image segmentation pipeline (cc @Narsil). This pipeline supports instance, semantic and panoptic segmentation, and uses the appropriate postprocess method.

Will soon do a more in depth review! Thanks already for all your work.



> @praateekmahajan thank you for working on this! Seems like you already made very good progress, my main comments are:
> 
> * As Niels suggested, you can create and/or leverage the XXXBackbone classes. The SwinBackbone PR will be merged shortly so you can just focus on the DinatBackbone class.
> * The current code is CUDA dependent (correct me if I'm wrong). I took a look at the paper and the Pixel Decoder seems very similar to that of Mask2Former (also uses multi-scale deformable attention). Perhaps you could use their PyTorch implementation to get rid of the CUDA scripts, here is the [relevant Mask2Former code.](https://github.com/facebookresearch/Mask2Former/blob/main/mask2former/modeling/pixel_decoder/msdeformattn.py)
> * I think having a OneFormerForUniversalSegmentation class makes sense but we can add it to auto mapping for instance segmentation instead of creating a new mapping for simplicity.
> 
> I will do a detailed review once the custom CUDA scripts are cleaned up.
> 
> Thanks again :)

Thanks for the suggestions @alaradirik! I will work on using AutoBackbone classes everywhere. About the CUDA code, sure, the PyTorch code is already [there](https://github.com/praeclarumjj3/transformers/blob/cb9cba1bf6d0249401ffacfbe9eca54ba1c384c8/src/transformers/models/oneformer/modeling_oneformer.py#L1228), we just check for the presence of GPU. I will clean the CUDA files. Also, I believe you tagged the wrong person by mistake 😂.

> I think having a OneFormerForUniversalSegmentation class makes sense but we can add it to auto mapping for instance segmentation instead of creating a new mapping for simplicity.

I still think it's better to create a different `AutoMapping` class for OneFormer as it belongs to a whole new class of architecture which uses a single model for all three tasks. Is it possible for us to keep it? Hopefully, there will be follow-up works in the same direction as OneFormer's approach of training a single model.

> Thanks for the suggestions @alaradirik! I will work on using AutoBackbone classes everywhere. About the CUDA code, sure, the PyTorch code is already [there](https://github.com/praeclarumjj3/transformers/blob/cb9cba1bf6d0249401ffacfbe9eca54ba1c384c8/src/transformers/models/oneformer/modeling_oneformer.py#L1228),

Great, that makes things much easier then, and sorry about tagging the wrong person :)
> 
> I still think it's better to create a different `AutoMapping` class for OneFormer as it belongs to a whole new class of architecture which uses a single model for all three tasks. Is it possible for us to keep it? Hopefully, there will be follow-up works in the same direction as OneFormer's approach of training a single model.

MaskFormer and Mask2Former (in progress in another PR) also feature universal segmentation architectures and I agree that new research will likely leverage the same paradigm. In retrospect, creating an auto mapping for universal segmentation and adding MaskFormer and Mask2Former along with OneFormer might be better. @NielsRogge what do you think about this?



Hi @NielsRogge @alaradirik, I have the made all the suggested changes, please let me know if I missed anything. Only one thing remains: using Autobackbone for Dinat (will do after the PR for that is merged).

Also a reminder about merging this [PR](https://huggingface.co/datasets/huggingface/documentation-images/discussions/11) for documentation images :)

## Changes after Review

- [x] Replace Swin backbone file with AutoBackbone
- [x] Replace Dinat backbone file with AutoBackbone
- [x] Remove FeatureExtractor Class.
- [x] Remove CUDA dependency code.
- [x] Apply suggested changes to image segmentation `task_inputs` description.
- [x] Remove dataset info files and use json files hosted on hf_hub instead.


@praeclarumjj3 Thanks for adding this model! ⭐ 

@praeclarumjj3 @NielsRogge @sgugger Yes, I think it would be better to add a `OneFormerProcessor` that contains both the tokenizer and image processor, similar to e.g. [OwlViT](https://github.com/huggingface/transformers/blob/94f8e21c7095430caa01272e16a367a421822e1c/src/transformers/models/owlvit/processing_owlvit.py#LL63C5-L63C5). In particular because the text processing is, as far as I can tell, independent of the processing of the images and it ensures accessing, loading and saving of the processing objects (tokenizer & image processor) is consistent across models. 

Hi @NielsRogge @alaradirik @amyeroberts, I have made all the requested changes and added a new `OneFormerProcessor` class.

Sorry I actually pushed on this PR, I didn't mean to:

https://github.com/huggingface/transformers/pull/20851 (For some reason I could not create a PR on top of you PR)

This PR has become too massive to be merged safely. Could you split the model addition and the pipeline addition in two different PRs?

> This PR has become too massive to be merged safely. Could you split the model addition and the pipeline addition in two different PRs?

@NielsRogge do you mind taking care of it ?
Let's remove my commits from this branch and just ignore the pipeline, I will then rebase my own PR on top once this is merged.

Sure, I think @praeclarumjj3 can revert the pipeline commits since I don't have write access and then @sgugger can have a final review.

@NielsRogge, are you sure you don't have write access? If @Narsil managed to push on the branch I think we should all have write access. I think it would be nice to take care of this given that @praeclarumjj3 has already done a big amount of work on the PR :) 

Thanks a lot!

@praeclarumjj3 opened a PR here to revert the pipeline updates: https://github.com/praeclarumjj3/transformers/pull/1

Thanks for all your work! Merging now.

Hi @praeclarumjj3 Thank you for adding this model! 

There are a few examples in the docstrings failing the CI. For example,  in `OneFormerForUniversalSegmentation.forward`
```python
        >>> # you can pass them to feature_extractor for instance postprocessing
        >>> predicted_instance_map = feature_extractor.post_process_instance_segmentation(
```
the `feature_extractor` is not defined.

Would you like to make them fixed 🙏 ? If so, you can run the doctest like

(if you have some change in the branch, stage them first)

```python
python3 utils/prepare_for_doc_test.py src docs
```
then
```bash
python3 -m pytest -v --make-reports doc_tests_gpu --doctest-modules src/transformers/models/gptj/modeling_gptj.py::transformers.models.gptj.modeling_gptj.GPTJForSequenceClassification.forward -sv --doctest-continue-on-failure --doctest-glob="*.mdx"
```
and also
```bash
python3 -m pytest -v --make-reports doc_tests_gpu --doctest-modules src/transformers/models/oneformer/modeling_oneformer.py::transformers.models.oneformer.modeling_oneformer.OneFormerModel.forward -sv --doctest-continue-on-failure --doctest-glob="*.mdx"
```
After running the doctests, discard the change produced by `prepare_for_doc_test.py`, and see if you need more changes in the branch.

Don't hesitate if you have further question, or if you could not find time on this at this moment (our team will fix it then) 🙏 Thank you


Hi @ydshieh, thanks for pointing this out to me. I apologize for not fixing the docstrings in the original PR (missed the changes after changing the code in an older commit). I have opened a new PR with the inconsistencies fixed: #21215.

Please take a look and let me know if something's still broken. And thanks for letting me know about the doctests! ✌🏻