Add TF implementation of GPT-J
Author: stancld
Pull request: yes

# What does this PR do?

This PR adds a TensorFlow implementation of `GPT-J` models 

Fixes #15583


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@LysandreJik @patrickvonplaten 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15623). All of your documentation changes will be reflected on that endpoint.

Super exciting! cc @Rocketknight1 and @gante

Hey @stancld 👋  Thank you for adding much-needed TensorFlow love ❤️  I will be your TensorFlow reviewer, expect a review by tomorrow morning

Hey @stancld 👋  I noticed you haven't replied to my review -- are you still interested in continuing this PR? :) If so, is there any way I can be of help? 

Hello @gante, sorry for being silent for the last two weeks as I was busy with other projects. I'm gonna try to address your comments tomorrow O:] 

@gante Bit late but working on the PR now :]

> @gante Bit late but working on the PR now :]

lovely 👌  lmk if you need a hand with anything, let's push this beauty to the finish line

I have uploaded the TF weights https://huggingface.co/EleutherAI/gpt-j-6B/blob/main/tf_model.h5

@gante Do you know how to save the fp16 weights in TF ? We need those for the `float16` branch. 

> @gante Do you know how do you save the fp16 weights in TF ? We need those for the `float16` branch.

@patil-suraj negative 😬 @Rocketknight1, do you know how to do it?

Not easy to do in TF, unfortunately! Keras really wants you to do mixed precision, with full-precision weights. I don't think there's any "native" way to convert a `Model` to use `float16` weights except with `TFLite` stuff, or just manually converting the weight arrays yourself.

Okay, thanks for the answer.

> or just manually converting the weight arrays yourself.

so should we manually create fp16 weights for the `float16` branch ? Not sure if that affects anything in TF.

I'm not sure, basically - I don't know how you'd load them into the model class without converting them to float32. Keras has some options for forcing dtypes but I don't think they're very well-supported, so as long as we want the model in Keras and not raw TF then I don't really know how to do this. Can we just drop the `float16` branch for TF for now?

Gently pinging everyone here :) Is this PR good for merge ?

There are two outstanding issues that require light changes (@stancld):
1. https://github.com/huggingface/transformers/pull/15623#discussion_r826236349
2. https://github.com/huggingface/transformers/pull/15623#discussion_r826227556

Other than that, good to go IMO 👍 


> There are two outstanding issues that require light changes (@stancld):
> 
> 1. [Add TF implementation of GPT-J #15623 (comment)](https://github.com/huggingface/transformers/pull/15623#discussion_r826236349)
> 2. [Add TF implementation of GPT-J #15623 (comment)](https://github.com/huggingface/transformers/pull/15623#discussion_r826227556)
> 
> Other than that, good to go IMO 👍

@gante I'm gonna solve these issues now O:]

_The documentation is not available anymore as the PR was closed or merged._

@gante All the remaining issues/comments should be resolved now :] Thanks a lot for your guidance! O:]

@stancld amazing, thank you so much for this contribution! Adding these models is always a tough task, especially the last mile, so I appreciate your effort 🤗 and I'm sure the community does as well.

@patil-suraj @Rocketknight1 are you cool with me merging this PR? We won't be able to call `resize_token_embeddings` (and I have an action point to enable that), but other than that seems good to go!

Good to merge for me! Maybe just override the `resize_token_embeddings` method and raise `NotImplementedError` with a  message. But don't feel strongly about this.

(@Rocketknight1 approved on slack, merging now to get a boost on Friday afternoon good vibes)