v3

README.md

@@ -0,0 +1,55 @@
+---
+language:
+  - en
+tags:
+  - FantasyGPT
+widget:
+  - "Legolas and Gimli advanced on the orcs, raising their weapons with a harrowing war cry."
+  - "Hermione smiled at Harry."
+  - "Ghost bared his teeth."
+  - "Geralt drew his sword"
+  - "Galadriel drew her sword"
+  - "Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw."
+  - "Harry leapt forward, dodging Fingolfin's wildfire and reaching for the ring."
+---
+
+A tiny GPT-2 (51.5M params) trained **from scratch** on some of my favorite books (about 14M words in total).
+
+It's only trained on an RTX 3090 for two hours, so don't take it seriously, just have fun!
+
+- peak lr: 5e-4
+- global batch size: 32
+- weight decay: 0.01
+- training steps: 20k
+- warmup steps: 1k
+- lr decay: cosine
+
+Example usage:
+
+```python
+from transformers import AutoTokenizer, GPT2LMHeadModel
+
+tokenizer = AutoTokenizer.from_pretrained('Geralt-Targaryen/FantasyGPT')
+model = GPT2LMHeadModel.from_pretrained('Geralt-Targaryen/FantasyGPT')
+
+input_text = ["Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw."]
+input_tokenized = tokenizer(input_text, return_tensors='pt')
+
+output = model.generate(inputs=input_tokenized.input_ids, max_new_tokens=256, do_sample=True, top_p=0.95, temperature=1)
+
+print(tokenizer.decode(output[0]))
+```
+
+Sample output:
+
+> Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw. And then the knight was on his horse and had no helm or helm or a lance. He flew to his feet.
+>
+> The knight roared and stamped. At the top of the helm he fell on his horse, and the knight grabbed the bridle, and the knight fell to his knees. The knight slashed at his shield.
+>
+> The knight caught hold of the blade of the knight and twisted his left foot off into the saddle. There was a moment of stunned silence. His blade went through its scabbard, and he fell to his knees, his sword raised from his belt. The blood of the knight’s blood spilled down the hilt of the lance. His sword flashed silver, and the hilt of the steel gleamed like burnished steel.
+>
+> The knight turned to obey. As he slashed at his chest, its sheath sheared free with a clang. His own longsword bounced and fell into a fist on his thigh, shattering the steel with his staff. The shield came from a blow and knocked him against the face of his foe.
+>
+> The knight did not hesitate. He saw his sword before him as the lance stabbed through the helm. The knight screamed, and the knight saw it. The knight grabbed the shield and held it before his neck.
+>
+> The sword

config.json

@@ -0,0 +1,41 @@
+{
+  "_name_or_path": "gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 0,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 0,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 512,
+  "n_head": 8,
+  "n_inner": null,
+  "n_layer": 8,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 256,
+      "temperature": 1,
+      "top_p": 0.95
+    }
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.25.1",
+  "use_cache": true,
+  "vocab_size": 50257
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:035b30000fd1d08d3acac589cbdf84b7d63b61f64e2dbb7d234afa6a1f803bf7
+size 214328413

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "add_prefix_space": false,
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1024,
+  "name_or_path": "tokenizer",
+  "special_tokens_map_file": null,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}