---
language:
  - en
tags:
  - FantasyGPT
widget:
  - "Legolas and Gimli advanced on the orcs, raising their weapons with a harrowing war cry."
  - "Hermione smiled at Harry."
  - "Ghost bared his teeth."
  - "Geralt drew his sword"
  - "Galadriel drew her sword"
  - "Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw."
  - "Harry leapt forward, dodging Fingolfin's wildfire and reaching for the ring."
---

GPT-2 (including tokenizer) trained **from scratch** on some of my favorite books (about 14M words in total).

It's only trained on an RTX 3090 for two hours, so don't take it seriously, just have fun!

- peak lr: 4e-4
- global batch size: 32
- weight decay: 0.01
- training steps: 20k
- warmup steps: 1k
- lr decay: cosine

Example usage:

```python
from transformers import AutoTokenizer, GPT2LMHeadModel

tokenizer = AutoTokenizer.from_pretrained('Geralt-Targaryen/FantasyGPT')
model = GPT2LMHeadModel.from_pretrained('Geralt-Targaryen/FantasyGPT')

input_text = ["Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw."]
input_tokenized = tokenizer(input_text, return_tensors='pt')

output = model.generate(inputs=input_tokenized.input_ids, max_new_tokens=256, do_sample=True, top_p=0.95, temperature=1)

print(tokenizer.decode(output[0]))
```

Sample output:

> Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw. But when it was done, it was a question not to be denied, it would be too late.
>
> And now it is the very last thing that has happened.
>
> Shadow looked down at the ceiling and saw that at once he was in prison. As he pulled himself upright, he looked at it, and he was looking down at the stone.
>
> A thin, black finger, with the exact same hook on the pommel. The edge of it was a piece of steel, and the tip of it, as it drew closer.
>
> He pulled his bow around. A foot from the bottom, with the same blunt. And the long, dark fingers. He could hear the voices of the dwarves, or hear their voices in the same tongue. He had not thought that they would hear voices, did not. His ears were filled with fear and despair. The goblins had become, the ones who had been on the edge of the mountains, were dying away in the distance. This was some battle the only battle they had left: the elves and Urgals, and the elves of the Northern Kingdoms.
>
> The goblins were dead; and the dragons they had driven back to the edge of Mirkwood were dead and buried in the earth that still controlled the land