Aurelius-Llama-3.2M-v0.5

Part of The Aurelius TinyStories Collection, a specialized series of highly optimized, sub-10M parameter models trained exclusively on the TinyStories dataset. This collection explores the boundaries of grammatical coherence, narrative depth, and reasoning capacity at micro-scale.

This model utilizes a standard LLaMA-style causal architecture compiled directly into native weights. It requires no custom code, special configurations, or remote execution flags to run.

Model Details

• Collection: The Aurelius TinyStories Collection
• Version: 0.5
• Total Parameters (with tied embeddings): 3,221,952 ($\approx 3.22\text{M}$)
• Active Parameters (excluding embeddings): 2,435,520 ($\approx 2.43\text{M}$)
• Layers ($L$): 6
• Hidden Dimension ($d_{\text{model}}$): 192
• Intermediate Dimension ($d_{\text{ff}}$): 512 (SwiGLU)
• Attention Heads: 6 Query Heads / 3 Key-Value Heads (Grouped-Query Attention)
• Context Length ($L_{\text{seq}}$): 256 tokens (scalable to 512)
• Vocabulary Size ($V$): 4,096 (Custom Byte-Level BPE)

Evaluation & Performance

Trained on the synthesized prose of the TinyStoriesV2 dataset, the model achieves the following validation benchmarks:

• Validation Cross-Entropy Loss: 1.1078
• Validation Perplexity: 3.03

Important Disclaimer & Liability Limitation

This model is provided strictly "as is" and "with all faults," without warranty of any kind, express or implied.

• Experimental & Research Nature Only: This model is an experimental, micro-scale prototype developed strictly for educational, scientific, and academic benchmarking purposes. It is not designed, tested, or certified for deployment in production environments, commercial applications, or consumer-facing products.
• No Safety Alignment: While the pre-training dataset (TinyStoriesV2) is conceptually designed around simple, child-like narratives, this model has not undergone any safety tuning, RLHF, supervised fine-tuning (SFT) for alignment, or toxic content filtering. It can output unpredictable, nonsensical, or potentially inappropriate text. Consequently, under no circumstances should this model or its outputs be deemed safe, verified, or appropriate for children or general public interaction.
• User Assumption of Risk: Any output generated by this model is the result of statistical text completion and does not represent the views, opinions, or endorsements of the developers or hosting entities. The end-user assumes all liabilities and risks associated with running, testing, fine-tuning, or utilizing the model or any downstream text generated by it.
• Architectural and Trademark Clarification: The use of "Llama" in the model name refers solely to the underlying open-source mathematical architecture used to structure the neural network layers (such as RMSNorm, SwiGLU, and RoPE). This model is trained from scratch and is not affiliated with, endorsed by, or associated with Meta Platforms, Inc. or any of its affiliates.

Usage

Because this model has been compiled into standard native layers, you can load it using the standard Hugging Face transformers library with zero runtime dependencies.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "MultivexAI/Aurelius-Llama-3.2M-v0.5"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "Once upon a time, Lily saw"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs, 
    max_new_tokens=60, 
    temperature=0.7, 
    top_p=0.85, 
    do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))