Novi-Nano-Base

Novi-Nano-Base is a tiny causal language model trained from scratch by Novi-AI.

With just 1,258,560 parameters, Novi-Nano explores language modeling at an extremely small scale while remaining compatible with the Hugging Face Transformers ecosystem.

⚡ 1.26M parameters · 300M training tokens · 256-token context

Model Details

Architecture

Property	Value
Model type	Causal Language Model
Parameters	1,258,560
Vocabulary size	8,192
Context length	256
Embedding size	96
Layers	4
Attention heads	4
FFN size	384
Tensor type	F32

Training

Novi-Nano-Base was trained from scratch using approximately 300 million training tokens.

Training Statistics

Metric	Result
Training tokens	300,023,808
Best validation loss	5.418699
Final validation loss	5.418699
Final validation perplexity	225.5853

Tokenizer

Novi-Nano uses a custom tokenizer with a vocabulary size of 8,192 tokens.

The tokenizer was trained using data from:

• FineWeb-Edu
• FineWeb-HQ
• SmolLM-Cosmopedia

Intended Use

Novi-Nano-Base is primarily intended for:

• 🔬 Research and experimentation
• 🧪 Small-model language-model experiments
• 🎓 Educational purposes
• 🛠️ Fine-tuning experiments
• 💻 Lightweight local inference

As a base model, it is not specifically instruction-tuned for following user commands or acting as a conversational assistant.

Limitations

Novi-Nano-Base is an extremely small experimental language model.

Because of its size and short context window, it will have significant limitations compared with modern billion-parameter language models.

It may:

• Generate incoherent text
• Repeat phrases
• Produce factual errors
• Struggle with complex instructions
• Have limited world knowledge
• Perform poorly on reasoning tasks
• Lose context beyond its 256-token window

This model should be considered a research and experimentation model, rather than a production-ready general-purpose LLM.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Novi-AI/Novi-Nano-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Hello, my name is"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Project History

Novi AI follows the earlier AppleMind experiments, with Novi becoming the primary project for developing small language models.

AppleMind → Novi AI → Novi-Nano 🚀

Acknowledgements

Novi-Nano was built using the open-source machine-learning ecosystem and datasets made available by the community.

Special thanks to:

• Hugging Face 🤗
• FineWeb
• SmolLM
• Cosmopedia

License

This model is released under the Apache 2.0 license.

---

🧠 Novi AI

Small models. Big experiments.

Novi-Nano is intentionally tiny — exploring how far a language model can go with just a fraction of the parameters used by modern LLMs.