borisbat/dasllama-stories

dasLLAMA story models

The two story models behind dasllama.io's browser examples, as GGUF. dasLLAMA is the daslang inference engine (https://github.com/GaijinEntertainment/daScript, modules/dasLLAMA); the examples are examples/dasLLAMA/storyteller (stories15M writes, KittenTTS reads) and examples/dasLLAMA/storywish (you type the words, tinystories-instruct-27M writes a story that uses them, KittenTTS reads it). Both are plain llama architecture, so llama.cpp and anything GGUF-based runs them too.

Files

file	what	bytes	sha256
stories15M-Q8_0.gguf	llama2.c stories15M, Q8_0 (dim 288, 6 layers, 32K llama-2 vocab)	26671392	1a8385fbd5de38704f005ba3d9b881fc0c60431bbbf17ab36c665e0be93ffd32
tinystories-instruct-27M-Q8_0.gguf	tinystories-instruct-27M, Q8_0 - the served file	31467200	92c2b775070b76ee31b9921f7f339ec9fc0e08b54be0616418414d4eec65d282
tinystories-instruct-27M-f32.gguf	the same weights in f32 - the reference lane, requantize from here	118097504	c813200b37c5e306d1b50d049dd378128e0a8b0109cb7672e5b7ee8c9f9b9253
tinystories-instruct-27M.llama2c.bin	the same weights as a llama2.c v0 checkpoint (run.c loads it with tok4096.bin)	109742108	c125572d4ed4d6da53f39a8a2fb13c6c0f03fd47df08da76b4e28a38dd3bbc26
tok4096.model	the SentencePiece BPE vocabulary (4096 pieces, byte fallback)	64709	c8a97cbc920542eb49bd588cdb1223405569067a59cca4da6d17a3052e60fa9f
tok4096.bin	the same vocabulary in llama2.c's tokenizer layout	54249	a2188c02de8f12f3c76be9397d621b592acf2fb8cf5356fca6c6588bea3aa4e2

Both GGUFs carry llama.context_length as the models were trained: 256 for stories15M (the llama2c converter writes 128; the storyteller pins 256 itself) and 512 for tinystories-instruct-27M.

tinystories-instruct-27M

A 27M-parameter llama (dim 512, 8 layers, 8 heads, SwiGLU, RoPE, tied embeddings, 4096-token vocabulary) trained from scratch on the TinyStoriesInstruct corpus, so that a request written the corpus's way steers the story. The point of it: the published TinyStories-Instruct models are GPT-Neo, which GGUF engines do not run; this one is llama architecture with the same behaviour.

Prompt

The corpus's own layout - any subset of the fields, in any order, Story: last, then a blank line. Sample at temperature 0.8, top-k 40; stop at EOS. The corpus orders its fields at random, so the model sometimes appends a Summary: line after the story: cut at the first line that opens with a field name.

Features: Dialogue
Words: dragon, cake, moon
Story: 

Fields the corpus knows: Words: (a comma list), Summary: (a sentence), Features: (any of Dialogue, BadEnding, MoralValue, Twist, Foreshadowing, Conflict), Random sentence: (a sentence the story must contain).

dasllama-server --model tinystories-instruct-27M-Q8_0.gguf
llama-cli -m tinystories-instruct-27M-Q8_0.gguf -p $'Words: dragon, cake, moon\nStory: \n\n' --temp 0.8 --top-k 40

How well it follows

24 sampled stories per word triple (temperature 0.8, top-p 0.9), counting the stories that contain all three words; the official GPT-Neo TinyStories-Instruct-33M under the same test:

words	tinystories-instruct-27M	TinyStories-Instruct-33M
dog, ball, park	16 / 24	17 / 24
rabbit, carrot, garden	22 / 24	21 / 24
boat, fish, hat	15 / 24	16 / 24
dragon, cake, moon	6 / 24	3 / 24
robot, pizza, snow	2 / 24	3 / 24

Without a Words: line the same words appear in 0 to 1 of 24 stories. Common nouns land; rarer ones (robot, pizza) land one at a time. Validation loss 1.144 nats per token on the corpus's validation split under this vocabulary.

Words: dragon, cake, moon

Once upon a time there was a dragon who enjoyed cake very much. Every day, he would go flying around the world with a big cake in his cave. He would look up at the moon and remember how the moon made him feel. One day, the dragon was walking...

Recipe

• Data: roneneldan/TinyStoriesInstruct, the train file as published (2,476,533 examples, <|endoftext|>-separated), tokenized as-is with BOS before and EOS after each example: 683,848,778 tokens, 276 per example. The validation file (25,027 examples) is the held-out split.
• Tokenizer: SentencePiece BPE, 4096 pieces, trained on the first 150,000 examples with llama2.c's settings (byte fallback, identity normalization, digits split) - tok4096.model.
• Model: llama2.c's model.py, dim 512, 8 layers, 8 heads, 8 KV heads, hidden 1376 (multiple of 32), max sequence 512, no dropout, tied embeddings. 27.4M parameters.
• Training: llama2.c's train.py on one H100 SXM, bf16 autocast, torch.compile; AdamW (betas 0.9/0.95, weight decay 0.1), learning rate 5e-4 with 500 warmup steps and cosine decay to 0, gradient clip 1.0, 256 sequences of 512 tokens per step (131,072 tokens), 9500 steps = 1.25B tokens = 1.8 epochs, 15.5 minutes. Train loss 1.163, validation loss 1.144 at the end, still falling.
• Export: export.py --version 0 to the llama2.c checkpoint, llama.cpp's llama-convert-llama2c-to-ggml --copy-vocab-from-model tok4096.bin (98c4764b6) to f32 GGUF, llama-quantize Q8_0, llama.context_length set to 512.

The trainer, the data preparation and the hit-rate test are llama2.c's scripts with a data module for this corpus.

stories15M

Andrej Karpathy's llama2.c stories15M (dim 288, 6 layers, 6 heads, the llama-2 32K vocabulary), trained on TinyStories. Converted from karpathy/tinyllamas (stories15M.bin + tokenizer.bin) with llama.cpp's llama-convert-llama2c-to-ggml and quantized to Q8_0. The storyteller prompts it with Once upon a time and stops at BOS, the corpus's story separator. No instruction following: it writes what it wants.