III-TinyStories

III-TinyStories is a small autoregressive language model trained from scratch on the TinyStories dataset.

The model is primarily intended as an experimental testbed for the III architecture rather than as a production-ready general-purpose language model.

Model Details

Model Description

III is a custom decoder architecture that combines causal self-attention with gated causal convolutional layers.

The architecture includes:

• Hybrid attention and causal convolutional token mixing
• Causal dilated depthwise convolutions
• RMSNorm
• Query/key RMS normalization
• Rotary Position Embeddings (RoPE)
• SwiGLU-style MLP blocks
• KV caching for autoregressive generation
• Tied input/output token embeddings

The model was implemented as a custom Transformers architecture and can be used for autoregressive text generation.

• Developed by: KordAI
• Model type: Autoregressive causal language model
• Language: English
• Training dataset: roneneldan/TinyStories
• Training objective: Next-token prediction
• License: Not specified

Model Architecture

III does not use a conventional all-attention Transformer stack.

Each decoder layer uses either:

• causal self-attention, or
• a gated causal convolutional mixer,

depending on the layer configuration.

The attention blocks use:

• Multi-head QKV projection
• RMS-normalized queries and keys
• RoPE
• PyTorch scaled dot-product attention

The convolutional blocks use:

• Gated projections
• Depthwise causal convolutions
• Configurable receptive fields
• Configurable dilation

The model also implements custom cache handling for autoregressive generation.

Uses

Direct Use

This model is primarily intended for:

• Experimenting with small language model architectures
• Studying training behavior of hybrid sequence mixers
• Testing autoregressive generation
• Comparing the III architecture against conventional Transformer baselines
• Educational and research experimentation

Downstream Use

The model may be fine-tuned for small-scale experimental tasks, but its performance for downstream applications has not been comprehensively evaluated.

Out-of-Scope Use

This model should not be considered suitable for:

• Production applications
• Safety-critical applications
• Factual question answering
• Reliable knowledge retrieval
• High-stakes decision making
• Representing current world knowledge

The model was trained on TinyStories and therefore has a deliberately narrow training distribution.

Bias, Risks, and Limitations

The main limitations are a direct consequence of the training data and experimental nature of the model.

TinyStories contains short, synthetic stories with relatively simple language. As a result, the model should not be expected to have broad factual knowledge or the linguistic coverage of a larger general-purpose language model.

The model may:

• Generate incorrect or nonsensical statements
• Repeat patterns from the training distribution
• Produce grammatically unusual text
• Fail on topics outside the TinyStories distribution
• Exhibit behavior that does not generalize to larger and more diverse corpora

Training and evaluation results on TinyStories should therefore not be interpreted as evidence of general-purpose LLM capability.

Recommendations

Use this model primarily for experimentation and architecture research.

For meaningful comparisons between architectures, keep the tokenizer, training data, token budget, context length, optimizer, and other training conditions consistent.

How to Get Started with the Model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "KordAI/III-tinystories"

tokenizer = AutoTokenizer.from_pretrained(
    repo,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
)

model.eval()

prompt = "Once upon a time, there was a little boy named Tom."

inputs = tokenizer(
    prompt,
    return_tensors="pt",
)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        use_cache=True,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(
    tokenizer.decode(
        output_ids[0],
        skip_special_tokens=True,
    )
)