CodVa-1-Small-IT Model Card

---

Model Overview

CodVa-1-Small-IT is an instruction-tuned Large Language Model developed by Smilyai-labs — a team of high-school students passionate about AI research and development. This model is the instruction-tuned (IT) variant of our base model, CodVa-1-Small, fine-tuned to follow instructions, answer questions, and assist with coding and mathematical reasoning tasks.

---

Key Features

• 🧠 Custom Architecture — Built on our in-house architecture designed from the ground up to maximise performance on code and math benchmarks
• ⚡ Mixture of Experts (MoE) FFN — Sparse MoE feed-forward layers with shared experts, giving a large parameter count while keeping per-token compute low
• 🔄 Grouped Query Attention (GQA) — Reduces KV-cache memory footprint significantly compared to standard multi-head attention, enabling longer context at inference time
• 📐 RoPE with Extended Theta — Rotary positional embeddings with a high theta value (5,000,000) for improved length generalisation beyond the training context window
• 🔧 QK Norm — Per-head RMS normalisation on queries and keys for more stable attention across long sequences
• 💬 Instruction Tuned — Fine-tuned on a curated dataset of code, math, and general instruction-following examples to produce a helpful, chat-capable assistant

---

Architecture Details

Component	Configuration
Architecture	Custom Decoder-Only Transformer
Hidden Dimension	1536
Layers	28
Attention Heads	24 (Query) / 6 (KV)
Attention Type	Grouped Query Attention (GQA)
Positional Encoding	RoPE (θ = 5,000,000)
FFN Type	Dense SwiGLU + Sparse MoE
MoE Experts	16 routed + 2 shared
MoE Top-K	2 routed experts per token
MoE Hidden Dim	1024
MoE Frequency	Every 2 layers
Context Length	4096 tokens
Vocabulary Size	32,000 (+ special tokens)
Normalisation	RMSNorm throughout
QK Norm	✅ Enabled
Structural Bias	✅ Enabled (4 relation types)
Precision	BFloat16

---

Training Details

Pre-Training (CodVa-1-Small Base)

The base model was pre-trained from scratch on a large corpus of code and mathematical text, optimised for strong reasoning and programming capabilities.

Instruction Fine-Tuning (This Model)

CodVa-1-Small-IT was produced by supervised fine-tuning (SFT) of the base model on a curated instruction dataset.

Setting	Value
Fine-tuning Method	Supervised Fine-Tuning (SFT)
Dataset	Bc-AI/codva-it-data-v5.1
Sequence Length	4096 tokens
Optimizer	AdamW (8-bit where available)
Learning Rate	2e-5
LR Schedule	Cosine decay with warmup
Weight Decay	0.01
Gradient Clipping	1.0
Precision	BFloat16

---

Intended Use

CodVa-1-Small-IT is intended for:

• 💻 Code generation and completion — Writing functions, classes, scripts, and full programs across a range of programming languages
• 🐛 Code explanation and debugging — Understanding existing code, identifying bugs, and suggesting fixes
• 📐 Mathematical reasoning — Step-by-step problem solving, symbolic reasoning, and numerical computation
• 🤖 General instruction following — Answering questions, summarising text, and following natural language instructions
• 🧪 Research and experimentation — Exploring small-scale LLM behaviour, fine-tuning further, or using as a base for downstream tasks

---

Limitations

As with all language models — particularly smaller ones — there are important limitations to be aware of:

• Factual accuracy — The model may confidently produce incorrect information. Always verify outputs for any critical use case
• Context length — Performance may degrade on inputs approaching the 4096-token context limit
• Mathematical reliability — While the architecture is optimised for math, the model can still make arithmetic and reasoning errors, especially on complex multi-step problems
• Safety — This model has not undergone extensive red-teaming or safety fine-tuning. It should not be deployed in production applications without additional safety measures
• Scale — As a small model, it will be outperformed by larger models on complex tasks. It is best suited for lightweight deployment, experimentation, and research

---

About Smilyai-labs

Smilyai-labs is a team of high-school students who are passionate about AI research. We design and train our own model architectures from scratch, rather than fine-tuning existing open-source models, with the goal of learning every part of the deep learning stack — from architecture design and custom CUDA kernels to dataset curation and training infrastructure.

CodVa is our flagship model series, focused on code and mathematical reasoning.

We are students building real models. Feedback, collaboration offers, and questions are very welcome.

---

Citation

If you use CodVa-1-Small-IT in your research or projects, please consider citing or crediting the Smilyai-labs team:

@misc{codva1small,
  author       = {Smilyai-labs},
  title        = {CodVa-1-Small-IT: An Instruction-Tuned Code and Math Language Model},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/Smilyai-labs/CodVa-1-Small-IT}},
}

---

License

Please refer to the repository's license file for terms of use. If you intend to use this model for commercial purposes, please contact the Smilyai-labs team directly.