GPT-J
Author: StellaAthena
Pull request: yes

**This is a work-in-progress focused on reconciling styles and may break without warning. If you want to use GPT-J with the HF interface, you can do that by installing transformers from [here](https://github.com/finetuneanon/transformers/tree/gpt-j). The purpose of this PR is to make progress on converting that repo to the style HF prefers.**

# What does this PR do?

This is my attempt to reconcile #12106 with the HF style guidelines as described by @sgugger. The original PR was created by @finetuneanon and @kurumuz.

This implementation has not been thoroughly tested yet, but I wanted to get something out as a starting point for continuing the conversation before too much momentum is lost. I need to reread HF documentation a bit more to figure out the things that are wrong, or hopefully one of you lovely people can help me out.

For comparison, a frozen version of the code in the original PR can be found [here](https://github.com/finetuneanon/transformers/tree/c0dcc7fad45e9ac07cdff525cbe7fb0ff76a1304).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [Link](https://github.com/huggingface/transformers/pull/12106)
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten @patil-suraj @sgugger 

Comments:
The main thing I'm uncertain about is how to handle unimplemented functionality. GPT-J uses the same tokenizer as GPT-2, so I removed the tokenizer definition. Is that correct, or no? Relatredly, there were many types of modeling that GPT-J was not designed for, and @finetuneanon's PR just deleted the boilerplate for them. Is this correct?

> Also, sorry to ask this again, but could we not modify generation in this PR, since it seems it's not related to GPT-J.

Damn. It looks like I messed something up.... this was supposed to not include @finetuneanon's commits. I might close this and create a replacement PR with the correct commit history.

Mmm, I was wondering how this has been going. I would love to try a stable version of this!

Hey @sualehasif 

A stable version will be available in a week, stay tuned! 

> Damn. It looks like I messed something up.... this was supposed to not include @finetuneanon's commits. I might close this and create a replacement PR with the correct commit history.

@StellaAthena any idea when would you be adding a new PR? We are also running some experiments so maybe we could help.

@mittalpatel 

I'm taking over the PR. But feel free to post your findings :) 

In #12106 @finetuneanon reports the results of some evaluations of the ported model on EleutherAI’s evaluation harness. The numbers were a little lower than what we had found using the original implementation, but both he and I felt this was likely due to FP16. I can now confirm that the ported model achieves the same performance as the original model when evaluated in FP32. The absolute difference in performance on lambada, HellaSwag, PiQA, and Winogrande are all less than 0.5% when done in FP32

Cool, that's good to know.

@patil-suraj can you mark this as a draft, as it is not ready to merge in its current state?

> Hey @sualehasif
> 
> A stable version will be available in a week, stay tuned!

Hi, @patil-suraj thanks so much for working on this. Is there any progress on integration to huggingface transformers?

Just chiming in here: All of the .py files with dashes will not be importable :) So I'd suggest changing `gpt-j` to `gptj` or `gpt_j` in the .py file path names.

Any updates on this and any help required?

@patil-suraj What is the status of this?
I would really like to use this model, and I don't feel like messing around with forks to get this to work.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I would still love to see this happen.

> I would still love to see this happen.

This is going to happen any day now, see #13022