StoryTOM-10M-te

StoryTOM-10M-te is a 10M-parameter Telugu base model from StoryTOM, the first release of TOM (Task-Oriented Models).

TOM = Task-Oriented Model: a model trained from scratch for a single task. TOM is unrelated to Theory of Mind (ToM).

About StoryTOM

StoryTOM models are trained only on children's stories, using the same stories in English and in Telugu translation. They were built for the research paper "Same Stories, Different Language: A Controlled Comparison of Tiny English and Telugu Language Models." This first release uses one standard model design for both languages so the two can be compared fairly. Future TOM releases will use designs chosen for their specific task.

These are base models: they continue stories but do not follow instructions. Instruction-following versions are planned.

StoryTOM models are small research models. They are not intended for use in products or for real-world decisions.

Model

A 10M-parameter Telugu base model trained from scratch on translated children's stories.


Parameters	11,175,040 total (9,864,320 non-embedding + 1,310,720 tied embedding)
Design	decoder-only Transformer: 8 layers, hidden size 320, 10 attention heads (head size 32), feed-forward 1280, pre-LayerNorm, GELU, rotary position embeddings, tied input/output embedding
Context	544 tokens
Tokenizer	SentencePiece, 4,096 pieces, trained on the Telugu stories
Format	base model: input = `<BOS>` + text; the model continues the story and ends it with `<EOS>`; no chat template

How to use

pip install torch safetensors sentencepiece huggingface_hub

from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("modeling_tinylm", hf_hub_download("10X-Technologies/StoryTOM-10M-te", "modeling_tinylm.py"))
modeling = importlib.util.module_from_spec(spec); spec.loader.exec_module(modeling)

model, tok = modeling.TinyLM.from_pretrained("10X-Technologies/StoryTOM-10M-te")              # Default model
print(model.generate_text(tok, "ఒకప్పుడు, లిల్లీ అనే ఒక చిన్న అమ్మాయి ఉండేది."))

Or with the included script: python generate.py --model 10X-Technologies/StoryTOM-10M-te --prompt "ఒకప్పుడు, లిల్లీ అనే ఒక చిన్న అమ్మాయి ఉండేది.". Sampling defaults (as in the paper's evaluation): temperature 0.8, top-p 0.95; --temperature 0 for greedy decoding. Optional downloads load the same way with subfolder=, for example TinyLM.from_pretrained("10X-Technologies/StoryTOM-10M-te", subfolder="seeds/seed-44").

Downloads

Folder		Note
`(main page)`	Default model: use this one	seed 43, 0.003 learning rate, 4 passes
`seeds/seed-44`	Optional: same model, different training seed (44)	4 passes
`seeds/seed-45`	Optional: same model, different training seed (45)	4 passes
`lr-tuning/lr-0.0015`	Optional, for research only: learning-rate test run (0.0015), used for the paper's appendix	1 pass, seed 42
`lr-tuning/lr-0.003`	Optional, for research only: learning-rate test run (0.003), used for the paper's appendix	1 pass, seed 42
`lr-tuning/lr-0.006`	Optional, for research only: learning-rate test run (0.006), used for the paper's appendix	1 pass, seed 42
`lr-tuning/lr-0.01`	Optional, for research only: learning-rate test run (0.01), used for the paper's appendix	1 pass, seed 42

Evaluation (Default model)

Validation: 21,795 story pairs. Test: 21,295 story pairs, scored once for the paper. Bits per token are within-language only (each language has its own tokenizer); bits per English character (BPEC) uses the same character count for both languages.

	Bits per token	Bits per English character
Validation	2.1386	0.4751
Test	2.1397	0.4752

Training

Data: TinyStories V2 (synthetic children's stories, English). The Telugu stories are machine translations by Gemma 4 E2B (20 stories translated with ChatGPT); translations failing a strict script filter (about 21%) were removed from both languages. The English and Telugu models see exactly the same stories in the same order.
Training set: 2,125,871 story pairs (1% of the pool held out as the test set); 4 passes, 32,928 steps, 256 stories per step.
Optimizer: AdamW (0.9, 0.95), weight decay 0.1, peak learning rate 0.003 (warmup 2%, cosine to 10%), gradient clipping 1.0, bf16, one NVIDIA A100-SXM4-40GB per run.
Seeds 43, 44 and 45 were trained; seed 43 is the Default model.

Limitations

Trained only on simple synthetic children's stories: the models know little else and can produce incoherent or repetitive text.
The Telugu data is machine-translated, so it carries the translation model's errors and style.
In the machine-translated training data, named male animals receive inconsistent gender agreement, and the models reproduce this pattern (see the paper, Section 5.3).
The tokenizers were trained before the test set was held out, so the test stories influenced the tokenizer vocabulary (but not model training).

License

License: StoryTOM models and tokenizers are released under the Apache 2.0 license. If you redistribute them or anything built from them, include the license and the NOTICE file.

Credits: trained on English stories from TinyStories (Eldan and Li, 2023; CDLA-Sharing-1.0) and on Telugu translations generated with Gemma 4 E2B (Apache 2.0); 20 translations were produced with ChatGPT.