Add Perceiver IO
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR implements [Perceiver IO](https://arxiv.org/abs/2107.14795), by Google Deepmind.

Fixes #12996

It's a pretty cool piece of work: it applies a Transformer encoder to any kind of modality (images, text, audio, video) and for any problem (text classification, image classification, audio classification, video autoencoding, optical flow,...)! Perceiver is basically a BERT, ViT and Wav2Vec2 in one. However, the authors did apply the Perceiver on each problem separately, although I believe you could train a single (shared) encoder as backbone, and have different pre- and postprocessors for the different modalities.

The memory- and time requirements of the self-attention mechanism don't depend on the length of the inputs, as the inputs are only used for doing cross-attention. The bulk of computation happens on a set of latent variables, which are just randomly initialized at the beginning of training (i.e. `self.latents = nn.Parameter(torch.randn((num_latents, d_latents)))`).

Some demo notebooks to showcase what Perceiver can do:

- image classification & masked language modeling: [colab](https://colab.research.google.com/drive/1drKjC2EH8YvYAtIayUzmQ82A3KJkf-ka?usp=sharing)
- optical flow: [colab](https://colab.research.google.com/drive/1NE6BKj7JBlNgnntNWoyYOorsgOQCXy-u?usp=sharing)
- video autoencoding (and video classification): [colab](https://colab.research.google.com/drive/1V9BYnF4nK89NNcNOMw3431bw-Q6H8lsP?usp=sharing)

These colab notebooks are based on the original ones provided by Deepmind, in their (very clean) JAX/Haiku implementation which can be found [here](https://github.com/deepmind/deepmind-research/tree/master/perceiver).

cc @esceptico



Comments:
Thanks for the reviews, I've:

* removed the einops dependency, and replaced it by just PyTorch.
* removed the pre and postprocessors from the main init, they can still be imported using `from transformers.models.perceiver.modeling_perceiver import xxx`
* copied all arguments for the Perceiver model-specific outputs, to better reflect the existing philosophy
* fixed the tests on GPU
* renamed `PerceiverForImageClassification` to `PerceiverForImageClassificationLearned` (to better reflect that this one uses learned position embeddings, which is more in line with the other models, namely `PerceiverForImageClassification` and `PerceiverForImageClassificationConvProcessing`)

To do: 
- now, it's mainly about writing more docstrings.

Hi, reading this nice perceiver io implementation I came up with a question: It seems that in Perceiver the input is iteratively mixed with the latents via cross attention but in Perceiver IO it seems that the input is fed only once in the network. Does somebody know if I am right and the reason of doing it in this way? If I got it right, it seems that also in this implementation the input is fed only once...

Hi @tonibagur 
As mentioned in the paper:

>  The encode and processing stages re-use the basic structure of the original Perceiver. With the exception of the repeated encoder cross-attends. We found it simpler to omit these modules, as
 they increase compute without dramatically increasing performance, as reported in Appendix Table 6 in original Perceiver paper.

> Hi @tonibagur As mentioned in the paper:
> 
> > The encode and processing stages re-use the basic structure of the original Perceiver. With the exception of the repeated encoder cross-attends. We found it simpler to omit these modules, as
> > they increase compute without dramatically increasing performance, as reported in Appendix Table 6 in original Perceiver paper.

Thanks esceptico, I completely missed this paragraph :'