Adapt PerceiverIO Multimodal class to work with arbitrary modalities
Author: stevenmanton
Pull request: yes

# What does this PR do?

The current codebase is excellent, but the multimodal classes are tightly coupled to the example in the paper where the modalities are video, audio, and binary class labels. This PR makes a few small changes to support arbitrary modalities, such as text and image.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@NielsRogge 

I'm not an experienced committer to this repo, so I'm very happy to take direction. My hope is to share the improvements I made to a wider audience and extend an already awesome package.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger can you take a look at this PR? I'm happy to make any modifications to get this through. These changes are required to use the multimodal classes on modalities other than those that are currently hard-coded.

Thanks for your PR! Could you clarify which errors you run into with the current implementation that would be solved with this PR?

The biggest issue I had was that the signature of `forward` for the various Preprocessors isn't consistent. This mean I couldn't use the `TextPreprocessor` within the `PerceiverMultimodalPreprocessor` class. There were a couple other issues as well (e.g. `dict` instead of `ModuleDict`.) I'm happy to take feedback to improve this PR. Thanks for all your help!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@NielsRogge any thoughts on these improvements?

Hi @NielsRogge, just checking in on this PR. I know you're probably super busy, so is there something I can do to make the review easier for you? I'm very happy to do what I can to incorporate feedback. I have additional changes I'd like to make (mostly around type hints), but I'm hoping to make these initial fixes first, which are more critical. Please let me know how I can help! Thanks again for all your work.

Hi, sorry for the late reply here. I'll take a look tomorrow 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@amyeroberts Could you have a look?

@stevenmanton I can see that some of the tests failing are unrelated to this PR. Can you rebase from main to make sure all upstream changes are included? 

Thanks for your contribution!