GPyT is a GPT2 model trained from scratch (not fine tuned) on Python code from Github. Overall, it was ~80GB of pure Python code, the current GPyT model is a mere 2 epochs through this data, so it may benefit greatly from continued training and/or fine-tuning.

Newlines are replaced by ``

Input to the model is code, up to the context length of 1024, with newlines replaced by ``

Here's a quick example of using this model:

from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("Sentdex/GPyT")
model = AutoModelWithLMHead.from_pretrained("Sentdex/GPyT")

# copy and paste some code in here
inp = """import"""

newlinechar = ""
converted = inp.replace("\n", newlinechar)
tokenized = tokenizer.encode(converted, return_tensors='pt')
resp = model.generate(tokenized)

decoded = tokenizer.decode(resp[0])
reformatted = decoded.replace("","\n")


Should produce:

``` import numpy as np import pytest

import pandas as pd

Downloads last month
Hosted inference API
This model can be loaded on the Inference API on-demand.