IllaryiaStories-1M

A fine-tuned version of roneneldan/TinyStories-1M, trained exclusively on GPT-4 generated children's stories from the karpathy/tinystories-gpt4-clean dataset. The goal was to improve story quality — particularly named characters, dialogue coherence, and moral resolution — by removing the lower-quality GPT-3.5 stories present in the original training corpus.

---

Model Details

Property	Value
Base model	roneneldan/TinyStories-1M
Architecture	GPT-Neo (causal language model)
Parameters	~3.75M total
Context window	256 tokens
Model size	~48.6 MB
Tokenizer	EleutherAI/gpt-neo-125M
Training type	Full fine-tune (no LoRA)
Precision	fp16

---

Training Details

Hyperparameter	Value
Dataset	karpathy/tinystories-gpt4-clean
Training examples	500,000 stories
Epochs	2
Batch size	512
Learning rate	1e-4
LR scheduler	Cosine
Warmup ratio	0.05
Weight decay	0.01
Total steps	3,908

Training Curve

Step	Training Loss	Validation Loss
200	1.9994	1.9884
400	1.9828	1.9730
800	1.9699	1.9558
1200	1.9559	1.9426
1600	1.9436	1.9324
2000	1.9305	1.9244
2400	1.9235	1.9132
2800	1.9203	1.9067
3200	1.9106	1.9012
3600	1.9010	1.8973
3908	1.9073	1.8963

Validation loss remained consistently below training loss throughout, indicating clean generalisation with no overfitting. The final validation loss of 1.8963 compares favourably against the original TinyStories-1M, which achieves approximately 2.1–2.3 on comparable data.

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("EphAsad/IllaryiaStories-1M")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-125M")
model.eval()

prompt = "Once upon a time there was"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        input_ids,
        max_length=200,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Tip: Adjust temperature to taste. Values around 0.7–0.85 produce coherent, varied stories. Below 0.6 the model becomes repetitive; above 1.0 outputs become less coherent.

---

Sample Outputs

Prompt: Once upon a time there was

Once upon a time there was a girl called Mia. She was three years old and loved playing outside. One day, Mia went out to play outside with her friends. She asked her mom, "Can I play with you?" Her mom smiled and said, "Yes, let's play hide and seek!" Mia was so happy...

Prompt: In a small village there lived a

In a small village there lived a girl. She was three years old and she had a big, bright yellow bird. The bird liked to sing and play... The girl and the bird became best friends. They played together every day. And they lived happily ever after. The moral of the story is that even when we give away the gem, we can always be friends.

Prompt: One sunny morning, a rabbit

One sunny morning, a rabbit hopped over to the garden. He saw a big, round tomato in the grass... The moral of the story is that it is important to help others, because it's good to be kind to others.

---

Observed Improvements Over Base Model

• Named characters appear more consistently from the first sentence
• Dialogue is more naturally structured with attributions
• Moral resolution endings appear in the majority of generated stories — a clear GPT-4 data signature absent in the original
• Grammar is clean throughout, consistent with the base model's strength in this area

Known Limitations

• Stories begin to degrade in coherence around the 120–150 token mark — a capacity constraint of the 1M parameter architecture rather than a data issue
• Occasional repetition loops occur, particularly around action sequences
• Character tracking across multiple named characters is unreliable

---

Data

Training used rows 20,000–520,000 of karpathy/tinystories-gpt4-clean (500K stories), following the dataset's suggested split convention:

• Rows 0–9,999: test
• Rows 10,000–19,999: validation
• Rows 20,000+: train

The Karpathy clean dataset filters the original TinyStories corpus to GPT-4 generations only and applies Unicode normalisation, non-ASCII rejection, and whitespace cleanup — producing a significantly cleaner training signal than the mixed GPT-3.5/4 original.

---

Relation to TinyStories

This model is part of the TinyStories ecosystem (Eldan & Li, 2023), which demonstrated that language models well below 10M parameters can generate fluent, coherent English when trained on appropriately scoped data. IllaryiaStories-1M builds on that foundation with a higher-quality training corpus.

---

Citation

If you use this model, please also cite the original TinyStories paper:

@misc{eldan2023tinystories,
  title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
  author={Ronen Eldan and Yuanzhi Li},
  year={2023},
  eprint={2305.07759},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

---

License

MIT

Author

Zain Asad (EphAsad)