Efficientformer
Author: Bearnardd
Pull request: yes

This PR adds Efficientformer, a model that has similar latency as MobileNets, but achieves better accuracy on ImageNet. It is based on the closed PR: https://github.com/huggingface/transformers/pull/18296

Paper: https://arxiv.org/abs/2206.01191
Code and weights: https://github.com/snap-research/EfficientFormer

Fixes https://github.com/huggingface/transformers/issues/18041
## Who can review?
@alaradirik @NielsRogge 

Comments:
Hey @Bearnardd, thank you for working on this! Could you run `make fixup` to fix the failed style and code quality tests?

Also, type casting function arguments (e.g. `def something(arg1: torch.Tensor):`) causes errors if the type depends on a conditionally imported library (torch), you can see the failed test logs if you head over to the CI test details. Could you remove those from `test_modeling_efficientformer.py`?

Hi @alaradirik - thank you very much for the detailed review! I will address the changes shortly :) . I am  aware of the failing tests but I am not entirely sure how to count the number of expected attentions and hidden layers for this particular model since it does not have a "standard" transformer based architecture. Nevertheless I think that I will address the current comments and as the next step I will ask you some questions about expected attention and hidden outputs.

> Hi @alaradirik - thank you very much for the detailed review! I will address the changes shortly :) . I am aware of the failing tests but I am not entirely sure how to count the number of expected attentions and hidden layers for this particular model since it does not have a "standard" transformer based architecture. Nevertheless I think that I will address the current comments and as the next step I will ask you some questions about expected attention and hidden outputs.

Hey @Bearnardd, no problem at all! We define our own small model architecture within the `test_modeling_efficientformer.py` file as it is faster to test with a smaller dummy model. You would just need to check `num_hidden_layers` and `num_attention_heads` attributes of the test class to see the expected number of layers. It seems the model has the correct number of attention heads and hidden layers but doesn't return all of the outputs (attentions and hidden state outputs from all layers). 

If you are sure the implementation is correct and this is expected (in this case or other cases), you can always override the common tests within `test_modeling_efficientformer.py` by adding a method with the same name to the test class (`EfficientFormerModelTester`).


_The documentation is not available anymore as the PR was closed or merged._

Hi @NielsRogge @sgugger - could you take a look at the changes?

@NielsRogge I have applied the changes. Would you mind to do the review?

@sgugger thanks good catch with the resolved one. Actually I have checked that and `self.ab` is used in the `EfficientFormerSelfAttention` forward method so in fact it is needed. Moreover the original `EfficientFormer` code is based on the `levit` model and in the levit code there is a similar method.

Model is on the hub under the following [path](https://huggingface.co/Bearnardd/efficientformer-l1-300).

> Thanks for the explanation on the train method. Could you just make sure that caching this tensor this way does not add a key to the state dict of the model? In LeViT, the cache is a dictionary, not a tensor, so there is no problem.

@sgugger I have checked that and it does not add a key.

Thanks @Bearnardd !
@NielsRogge I'll let you have one last look and merge if you're happy :-)

Thank you so much for working on this @Bearnardd!