# GPT-J

## Overview

The GPT-J model was released in the [kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax) repository by Ben Wang and Aran Komatsuzaki. It is a GPT-2-like

causal language model trained on [the Pile](https://pile.eleuther.ai/) dataset.

This model was contributed by [Stella Biderman](https://huggingface.co/stellaathena).

## Usage tips

- To load [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) in float32 one would need at least 2x model size

  RAM: 1x for initial weights and another 1x to load the checkpoint. So for GPT-J it would take at least 48GB

  RAM to just load the model. To reduce the RAM usage there are a few options. The `torch_dtype` argument can be

  used to initialize the model in half-precision on a CUDA device only. There is also a fp16 branch which stores the fp16 weights,

  which could be used to further minimize the RAM usage:

thon

>>> from transformers import GPTJForCausalLM

>>> import torch

>>> device = "cuda"

>>> model = GPTJForCausalLM.from_pretrained(

     "EleutherAI/gpt-j-6B",

     revision="float16",

     torch_dtype=torch.float16,

 ).to(device)

- The model should fit on 16GB GPU for inference. For training/fine-tuning it would take much more GPU RAM. Adam

  optimizer for example makes four copies of the model: model, gradients, average and squared average of the gradients.

  So it would need at least 4x model size GPU memory, even with mixed precision as gradient updates are in fp32. This

  is not including the activations and data batches, which would again require some more GPU RAM. So one should explore

  solutions such as DeepSpeed, to train/fine-tune the model. Another option is to use the original codebase to

  train/fine-tune the model on TPU and then convert the model to Transformers format for inference. Instructions for

  that could be found [here](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md)

- Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer. These extra

  tokens are added for the sake of efficiency on TPUs. To avoid the mismatch between embedding matrix size and vocab

  size, the tokenizer for [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) contains 143 extra tokens

  `
 
`, so the `vocab_size` of tokenizer also becomes 50400.

## Usage examples

The [`~generation.GenerationMixin.generate`] method can be used to generate text using GPT-J

model.

thon

>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-j-6B")

>>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")

>>> prompt = (

     "In a shocking finding, scientists discovered a herd of unicorns living in a remote, "

     "previously unexplored valley, in the Andes Mountains. Even more surprising to the "

     "researchers was the fact that the unicorns spoke perfect English."

 )

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(

     input_ids,

     do_sample=True,

     temperature=0.9,

     max_length=100,

 )

>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]

or in float16 precision:

thon

>>> from transformers import GPTJForCausalLM, AutoTokenizer

>>> import torch

>>> device = "cuda"

>>> model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", torch_dtype=torch.float16).to(device)

>>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")

>>> prompt = (

     "In a shocking finding, scientists discovered a herd of unicorns living in a remote, "

     "previously unexplored valley, in the Andes Mountains. Even more surprising to the "

     "researchers was the fact that the unicorns spoke perfect English."

 )

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

>>> gen_tokens = model.generate(

     input_ids,

     do_sample=True,

     temperature=0.9,

     max_length=100,

 )

>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GPT-J. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- Description of [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B).

- A blog on how to [Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker](https://huggingface.co/blog/gptj-sagemaker).

- A blog on how to [Accelerate GPT-J inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/gptj-deepspeed-inference).

- A blog post introducing [GPT-J-6B: 6B JAX-Based Transformer](https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/). 🌎

- A notebook for [GPT-J-6B Inference Demo](https://colab.research.google.com/github/kingoflolz/mesh-transformer-jax/blob/master/colab_demo.ipynb). 🌎

- Another notebook demonstrating [Inference with GPT-J-6B](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/GPT-J-6B/Inference_with_GPT_J_6B.ipynb).  

- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the 🤗 Hugging Face Course.

- [`GPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).

- [`TFGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).

- [`FlaxGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb).

**Documentation resources**

- [Text classification task guide](../tasks/sequence_classification)

- [Question answering task guide](../tasks/question_answering)

- [Causal language modeling task guide](../tasks/language_modeling)

## GPTJConfig

[[autodoc]] GPTJConfig

    - all

## GPTJModel

[[autodoc]] GPTJModel

    - forward

## GPTJForCausalLM

[[autodoc]] GPTJForCausalLM

    - forward

## GPTJForSequenceClassification

[[autodoc]] GPTJForSequenceClassification

    - forward

## GPTJForQuestionAnswering

[[autodoc]] GPTJForQuestionAnswering

    - forward

## TFGPTJModel

[[autodoc]] TFGPTJModel

    - call

## TFGPTJForCausalLM

[[autodoc]] TFGPTJForCausalLM

    - call

## TFGPTJForSequenceClassification

[[autodoc]] TFGPTJForSequenceClassification

    - call

## TFGPTJForQuestionAnswering

[[autodoc]] TFGPTJForQuestionAnswering

    - call

## FlaxGPTJModel

[[autodoc]] FlaxGPTJModel

    - __call__

## FlaxGPTJForCausalLM

[[autodoc]] FlaxGPTJForCausalLM

    - __call__