Auto-translate GPTNeo to TensorFlow with GPT-4
Author: Rocketknight1
Pull request: yes

cc @gante @amyeroberts @ydshieh @sayakpaul

This is the preliminary result of auto-translating an entire module to TensorFlow using GPT-4.

The prompt I used was:

> 
> Hi GPT, can you translate this class from Hugging Face Transformers from PyTorch to TensorFlow for me?
> Some pointers:
> - When creating layers, please pass their attribute name as the name kwarg.
> - Retain any docstrings attached to methods like forward and translate them, even when the method is being renamed to call.
> - If the new class inherits from tf.keras.layers.Layer, it should accept **kwargs and pass these to super.__init__ . It should also be renamed by adding "TF" to the start of its name.
> - You don't need to add any extra imports, you can assume that any other functions or classes you call will be imported for you.
> - If the class calls other classes in the same module, you can assume that these have already been converted. Please add "TF" to the start of their name if required.

I'm going to experiment with auto-translating the tests and seeing how successful this port was. Right now it's a WIP and there are likely issues, but I've had a lot of success avoiding problems just by mentioning them in the prompt and telling GPT what to do in those situations!

Other things to go in the prompt:

- PyTorch Embedding layers support a `padding_idx` initialization arg that TF does not. If you want to exactly match PT's behaviour, you need to manually zero out those positions after embedding in TF.

Comments:
Shame Github doesn't have 🔥 as an available reaction

> Shame Github doesn't have 🔥 as an available reaction

@amyeroberts 🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥

<img width="666" alt="Screenshot 2023-03-24 185139" src="https://user-images.githubusercontent.com/2521628/227602659-692ebc96-a5d9-4b7b-9519-bac44916537a.png">


@Rocketknight1 aren't the commits going a bit too fast? 😑

I don't want to be a party pooper but this is really not a use case where I would trust any kind of language model output. LLMs tend to produce content that look good on the surface but are not rigorous and usually full of nasty little bugs (that's the reason why I personally stopped using Copilot) and we will completely miss them in such PRs that are usually adding one or several thousands of lines. I don't think our CI will catch all such bugs.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22365). All of your documentation changes will be reflected on that endpoint.

@sgugger Absolutely agree on the potential for small bugs, but in the case of model porting doesn't the CI test equivalence with the PT original? If the model accepts various inputs and always yields equivalent output to the PT version, I think it's probably "good enough" that most users shouldn't notice any issues, right?

Well for now those tests fail even before returning a diff between the PT and TF model :-p 

I'm still working on the prompt!!!!!

Ah ah, are you saying this should be the code exercise if we ever decide to open a Prompt Engineer position?

> Ah ah, are you saying this should be the code exercise if we ever decide to open a Prompt Engineer position?

That would be **THE** middleman between `PyTorch` and `TensorFlow`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.