Add missing Maskformer dataclass decorator, add dataclass check in ModelOutput for subclasses
Author: rachthree
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # https://github.com/huggingface/transformers/issues/25504


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->
@amyeroberts 

Some additional notes:
* I tried finding a way to find all subclasses of `ModelOutput` that could be tested in one unittest to check to see if they were a `dataclass`, but this may not be possible without finding all files that use `ModelOutput` and importing each subclass. I instead opted for a check inside `ModelOutput` that happens if a subclass is instantiated.
* One problem I ran into was that `Pipeline` uses `ModelOutput` directly: https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/base.py#L933. This is not a true `dataclass`, so `dataclasses` utilities such as `is_dataclass` cannot be reliably used. The checks done in `ModelOutput.__post_init__` would not be executed because `__post_init__` is a `dataclass`-specific dunder. This makes me think there should be a separate `PipelineOutput` class that can be created from a `ModelOutput` to create a true `dataclass`, or a class method in `ModelOutput` that creates a true `dataclass` given arguments similar as ones for `OrderedDict`. However, I don't know what the potential impacts could be since I don't know all the inner workings of `transformers.`

Comments:
cc @amyeroberts 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25638). All of your documentation changes will be reflected on that endpoint.

Thank you for reviewing, @amyeroberts! How do you feel about my 2nd bullet point under the additional notes? Should I open another issue?

@rachthree For the second point, is it necessary that the checks in `__post_init__` are run within the pipeline code? Providing our model output classes inherit from `ModelOutput`, the checks will be performed when the classes are used, which I think is sufficient. We don't want to add an additional output class in this case - pipelines are essentially just glue which holds together the preprocessing, inference and postprocessing steps, so should handle the same objects. 

I'm not sure I got all the implications of your question however. If you think I've missed the point, let me know.

@amyeroberts My apologies for my late response, I was on travel for a bit. Could the workflow for my PR be approved? CI got triggered again once I pushed the commit. Thanks!

Regarding dataclasses, the checks are not necessary if those checks are specifically for a model's subclassed `ModelOutput` with the `@dataclass` decorator, but the reason why I'm suggesting an additional output class is that `output = ModelOutput(foo_inputs)` by itself is not a true `dataclass`. A new class that is a true `dataclass` can be created using `dataclasses.makedataclass`, perhaps in some new classmethod for `ModelOutput`. I originally was thinking something like `PipelineOutput.from_dict` if using a new class.

Should anyone try to use `dataclass` utilities such as `dataclass.fields` on a Pipeline's `ModelOutput`, those utilities will not function properly:

```python
from transformers.utils.generic import ModelOutput
import dataclasses

output = ModelOutput({"a": 1, "b": 2})

fields = dataclasses.fields(output)
```

Results in `TypeError: must be called with a dataclass type or instance`

@rachthree Let's just stick to the changes here in this PR for now. If pipeline consistently use `ModelOutput` directly, then it shouldn't cause an issue. If it does. then we can open a new, separate PR to address it. 

@rachthree Let me know if there's any other changes you'd like to push before I merge

@amyeroberts Totally fair, and didn't meant to add more dataclass-related changes to this PR. Leaving it for now sounds good to me. No more changes on my end, thanks!

@amyeroberts 
Hi, I found the same problem in the model: BiomedVLP-CXR-BERT-specialized.
transformer version: transformers-4.35.2

reproduce (code found in https://huggingface.co/microsoft/BiomedVLP-CXR-BERT-specialized):

> import torch
> from transformers import AutoModel, AutoTokenizer
> 
> url = "microsoft/BiomedVLP-CXR-BERT-specialized"
> tokenizer = AutoTokenizer.from_pretrained(url, trust_remote_code=True)
> model = AutoModel.from_pretrained(url, trust_remote_code=True)
> 
> text_prompts = ["There is no pneumothorax or pleural effusion",
>                 "No pleural effusion or pneumothorax is seen",
>                 "The extent of the pleural effusion is constant."]
> 
> tokenizer_output = tokenizer.batch_encode_plus(batch_text_or_text_pairs=text_prompts,
>                                                add_special_tokens=True,
>                                                padding='longest',
>                                                return_tensors='pt')
> embeddings = model.get_projected_text_embeddings(input_ids=tokenizer_output.input_ids,
>                                                  attention_mask=tokenizer_output.attention_mask)
> 
> sim = torch.mm(embeddings, embeddings.t())
> 

Error that I encountered:
TypeError: transformers_modules.microsoft.BiomedVLP-CXR-BERT-specialized.b59c09e51ab2410b24f4be214bbb49043fe63fc2.modeling_cxrbert.CXRBertOutput is not a dataclasss. This is a subclass of ModelOutput and so must use the @dataclass decorator.


Could you add the missing datacalss decorator to this model also?
Thank you so much!


Hi @lixiaoqingnnz, the model output is defined as [code on the hub](https://huggingface.co/microsoft/BiomedVLP-CXR-BERT-specialized/blob/b59c09e51ab2410b24f4be214bbb49043fe63fc2/modeling_cxrbert.py#L19). I suggest opening a PR on the repo to add the decorator there or opening a discussion asking for the decorator to be added by the repo maintainers 

@amyeroberts Thanks for your reply, I already added the decorated and the problem have been solved. Sorry that I bothered you cause I thought this repo maintainers is also you. 
Thanks anyway again!

@lixiaoqingnnz No worries - it wasn't a bother :)