Simplify GPT-Neo local attention implementation
Author: finetunej
Pull request: yes
Labels: name: WIP

# What does this PR do?

The current local self attention implementation for gpt-neo follows the implementation in mtf, which makes it hard to understand. This pull request implements local self attention by changing the bias mask in global attention. Local self attention is a sliding window where each token can only attend to the previous window_size tokens. This implementation clearly reflects this.

[Measured performance](https://gist.github.com/finetuneanon/5b2186c3555b652f387c86160cd89b55) (apply just 330686a3c0520c9727fe5ebed385e708d0178d77 and 269c497be1691556c830f61fa8f90001c692722f on #11320 patched to use) shows no noticable difference between implementations with respect to speed or VRAM usage. The results of both implementations are also identical.

Fixes #11320

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Since this PR mostly removes code, no additional tests or documentation were written.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

gpt-neo: @patil-suraj


Comments:
I just found an issue with use_cache. Will fix it later.

Edit: Fixed.

unstale

I have used this version of the code quite extensively without issues and have found no difference in performance when doing evaluations. I have also just fixed the so far present warning about the local attention layer biases not being initialized from checkpoints.

> I have used this version of the code quite extensively without issues and have found no difference in performance when doing evaluations. I have also just fixed the so far present warning about the local attention layer biases not being initialized from checkpoints.

I want to second this.  For the past month, I have been [using](https://github.com/nostalgebraist/nostalgebraist-autoresponder/blob/ff2eafb1fbf6d670513f5d7b64ab99a739120945/stable_library_code/transformers/gpt_neo/modeling_gpt_neo.py) this version of the code in my bot -- which runs 24/7 and uses GPT-Neo 2.7B.  It's worked perfectly for me.

By comparison, the implementation currently on `master` [requires](https://github.com/nostalgebraist/nostalgebraist-autoresponder/blob/ff2eafb1fbf6d670513f5d7b64ab99a739120945/selector_model/selector_nn_neo.py#L45) counter-intuitive, inefficient manual padding to avoid OOM.

Sorry about the super late response.

@finetuneanon  @nostalgebraist 
This sounds awesome! And yeah, I agree, the current implem is not super readable. 
Did you guys run any memory/speed benchmarks comparing the two versions? If so could you please post the results and the script so that we could take a look?

Can provide those soon. Will close this PR though, because it's also included in #12106.

Performance evaluation results are available here:
https://github.com/huggingface/transformers/pull/12106#discussion_r650008259

I am reopening this pull request due to the splitting of that PR.

@patil-suraj is there a reason this hasn't been merged yet? I notice that this method is used in #12493

HI @finetuneanon, sorry to only come back to this now. 

This is a great solution, so let's go ahead with this!
The PR is in good shape already but needs some clean-up and we would also need to adapt tests.

Let me know if you want to continue working on this.

Thanks for taking over.