Add Neighborhood Attention Transformer (NAT) and Dilated NAT (DiNAT) models
Author: alihassanijr
Pull request: yes

# What does this PR do?

This PR adds [NAT](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer/blob/main/NAT.md) and [DiNAT](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer/blob/main/DiNAT.md) and their dependencies.

## Dependencies
- [NATTEN](https://github.com/SHI-Labs/NATTEN/) is the only new requirement. The models themselves are mostly in the same style as most `timm` models. They just require NATTEN to get the sliding window attention.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?
  - Yes, mostly boilerplate from similar models.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten @NielsRogge @amyeroberts @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20219). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20219). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20219). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20219). All of your documentation changes will be reflected on that endpoint.

In this instance, I'd actually prefer to rely on the extra dep (as long as it's properly set up as a soft dependency, which seems to be the case in the PR). We don't know how to maintain CUDA kernels anyway, so support will be a lot better if it's done elsewhere.

Hi @NielsRogge @sgugger 
Actually we're happy to do it either way, but the reason we packaged NATTEN as a pip package in the first place is to make installation easier, especially since we plan to upgrade it frequently.
Unlike Deformable Attention's extension, NATTEN it doesn't come with a fixed set of kernels. There's still improvements that we've planned ahead to NATTEN, especially adding new kernels to optimize latency. 
And just to confirm @sgugger 's comment, maintaining all the kernels in NATTEN might increase your wheel sizes, which I'm not sure if you want to do. The cpu-only wheels aren't too bad, but the ones with cuda wheels are up to 50MB.
And as far as testing CUDA kernels go, you'd need to have unit tests to check the backwards functions (gradcheck), and running those for all different use cases that call different kernels is just really time consuming (and we only pull it off by running it on 80GB A100 GPUs; it's so memory-intensive).

And yes, as @sgugger stated, it would work as a soft dependency; even imports aren't broken. But there's dummy calls to the package in case it's not available, that will raise an error only when the forward functions are called.

As for the torch tests, I only did those as a suggestion. I would personally recommend having a separate test for these models in general so that it doesn't get in the way. Additionally, knowing the torch build beforehand is better, since that way we can just specify a wheel URL and have it just install a lot faster.

I'll make the changes to the docs and run fix-copies again.

And yes, both models were cloned off of Swin; the architectures are somewhat similar.
The difference here is that there's a convolutional tokenizer and downsampler replacing the patch+embed and patch merging layers; and we like to keep tensors in shape B H W C, since NATTEN also expects the height and width axes to be unrolled, like a Conv2d. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20219). All of your documentation changes will be reflected on that endpoint.

Actually, I just noticed `transformers` doesn't come with wheels, right?
My previous statement about wheel sizes is irrelevant in that case.

However, I would shift more towards @sgugger 's point of view, since loading torch extensions at runtime becomes less and less reliable as extensions grow, and NATTEN already has twice the number of kernels compared to MSDeformAttn (excluding the templating that goes on in NATTEN). This would have the users wait up to 5 minutes before being able to use these models, and would affect reproducibility (because the torch build's cuda version doesn't necessarily match the system's, or the expected one for that matter).

FWIW, I've definitely seen libraries take one of three approaches: 
* either adding a C backend to their package for all custom operations and build wheels (`detectron2`, `mmcv`); 
* or just having soft dependencies to pip packages that already do that to avoid the hassle. It also doesn't create new issues with upgrades to CUDA or torch (which depending on their usage can break things) 
* And of course there's still the option of lazy loading (the way MSDeformAttn is handled right now), which is honestly a great alternative to both, but only as long as the kernels aren't being updated and compile time is relatively low.



@sgugger All done.
And sorry I didn't go with directly applying your changes, some of them needed a few more replacements so I figured I'd make the changes and commit all of them at once.

Thanks! You'll need a rebase on main to get the tests to pass as TensorFlow new release breaks the CI

Thanks for the reviews and feedback @sgugger @NielsRogge @amyeroberts .
Looking forward to contributing more in the future.