Heet24/model42M-v1

🐯 VaghLM-42M

A 42.6M Parameter GPT-Style Decoder-Only Transformer Built and Trained from Scratch

---

VaghLM (designated internally as Model42M-V1) is a lightweight, efficient, and educational GPT-style decoder-only language model built completely from scratch in PyTorch. Trained on over 1.5B tokens and aligned with high-quality structured instruction datasets, VaghLM is designed to deliver rapid, structured, and helpful responses on consumer hardware.

Model Card • Installation • Usage • Hugging Face Repository

---

📖 Table of Contents

1. Architecture Overview
2. Model Specifications
3. Key Features
4. Repository Structure
5. Installation
6. Quick Start
   • CLI Interactive Chat
   • Gradio Web Application
7. Python API Example
8. Training & Alignment Pipeline
9. Limitations & Future Roadmap
10. License & Citation
11. Acknowledgements

---

🏗️ Architecture Overview

VaghLM utilizes a classical decoder-only GPT architecture with modernized training optimizations such as pre-layer normalization and PyTorch's native Scaled Dot-Product Attention (SDPA).

graph TD
    UserQuery[User Prompt] --> Tokenizer[SentencePiece BPE Tokenizer]
    Tokenizer --> SpecTokens[Sequence: BOS + USER + tokens + ASSISTANT]
    SpecTokens --> Model[VaghLM-42M Block Layers]
    
    subgraph Blocks ["8 x Transformer Blocks"]
        Model --> LN1[LayerNorm]
        LN1 --> SDPA[Causal Multi-Head Self-Attention]
        SDPA --> Add1[Residual Connection]
        
        Add1 --> LN2[LayerNorm]
        LN2 --> GELU[FFN: Linear -> GELU -> Linear]
        GELU --> Add2[Residual Connection]
    end
    
    Add2 --> LNFinal[Final LayerNorm]
    LNFinal --> TiedHead[Tied LM Head Output Projection]
    TiedHead --> Sampler[Nucleus / Temp / Repetition Sampler]
    Sampler --> OutputToken[Next Token Prediction]

---

📊 Model Specifications

Parameter / Configuration	Value / Details
Model Name	VaghLM-42M (Model42M-V1)
Active Parameters (Tied)	42,615,808 (~42.6M)
Total Parameters (Untied)	59,000,000 (~59M)
Vocabulary Size	32,000
Context Length	2,048 tokens
Transformer Layers	8
Attention Heads	8
Embedding Dimension ($d_{model}$)	512
Feed-Forward Dimension ($d_{ff}$)	2,048
Weight Tying	Enabled (Input embeddings shared with output projection)
Positional Embeddings	Learned Absolute Positional Embeddings
Normalization	Pre-LayerNorm (standard nn.LayerNorm)

---

⚡ Key Features

• Built from Scratch: 100% PyTorch implementation of embeddings, attention, feed-forward blocks, and the sequence loss generator.
• BPE Tokenization: A custom BPE tokenizer trained using Google's SentencePiece library, with specialized tokens for structuring conversational turns (<user>, <assistant>, code, system, etc.).
• High-Performance Attention: Causal multi-head attention accelerated using PyTorch's scaled_dot_product_attention (compatible with FlashAttention kernel executions).
• Mixed-Precision Training: Integrated PyTorch Automatic Mixed Precision (autocast and GradScaler) for memory-efficient and stable training.
• Instruction-Aligned: Pre-aligned to follow user instructions using a curated dataset of structured definitions, coding guidelines, and logic reasoning splits.
• Interactive Gradio UI: Includes a feature-rich, dark-themed local Gradio chat application with streaming text, parameter adjusting sliders (temperature, top-k, top-p, repetition penalty), and example query templates.

---

📂 Repository Structure

VaghLM-42M/
├── assets/                     # Visual diagrams and assets (Mermaid models)
├── checkpoints/                # Saved weights (.pt files)
├── data/
│   └── tokenized/
│       └── all_tokens.bin      # Binary file of pre-tokenized training data (1.6B tokens)
├── examples/
│   ├── generate_example.py     # Simple, self-contained inference script
│   └── load_model.py           # Model inspection and validation script
├── inference/
│   └── generate.py             # CLI interactive chat application script
├── model/
│   ├── __init__.py
│   ├── attention.py            # Multi-Head Causal Self-Attention block
│   ├── config.py               # Hyperparameter configurations (VaghLMConfig)
│   ├── embeddings.py           # Learned Token + Positional embeddings
│   └── transformer.py          # Main VaghLM Transformer model implementation
├── tokenizer/
│   └── vocab/
│       ├── tokenizer.model     # SentencePiece BPE model file
│       └── tokenizer.vocab     # Vocabulary list file
├── training/
│   ├── __init__.py
│   ├── dataset.py              # NumPy memmap training dataset & dataloader loaders
│   └── trainer.py              # Core pretraining & evaluation training script
├── .gitignore                  # Git exclusion rules
├── CHANGELOG.md                # Release version history
├── CITATION.cff                # Citation metadata YAML
├── CODE_OF_CONDUCT.md          # Contributor Covenant Code of Conduct
├── CONTRIBUTING.md             # Developer contribution guidelines
├── data.md                     # Instruction dataset distribution & 500 seed examples
├── generate_data_md.py         # Script to validate and compile seed instruction examples
├── gradio_app.py               # Gradio chat web server script
├── INSTALL.md                  # Comprehensive install steps for Windows & Linux
├── LICENSE                     # MIT License
├── requirements.txt            # Python library dependencies
├── upload.py                   # Hugging Face hub folder upload script
└── USAGE.md                    # Detailed model execution & sampling controls guide

---

⚙️ Installation

To set up the workspace, clone the repository, activate a virtual environment, and install the dependencies:

# Clone the repository
git clone https://github.com/Heet24/VaghLM-42M.git
cd VaghLM-42M

# Create and activate virtual environment (Windows)
python -m venv venv
.\venv\Scripts\activate

# Create and activate virtual environment (Linux)
# python3 -m venv venv
# source venv/bin/activate

# Install requirements
pip install -r requirements.txt

For more detailed instructions, please check the Installation Guide.

---

🚀 Quick Start

1. Download Checkpoints

Before running inference, download the weights from Hugging Face:

from huggingface_hub import hf_hub_download
import os

os.makedirs("checkpoints", exist_ok=True)
hf_hub_download(repo_id="Heet24/model42M-v1", filename="finetuned_v2_best.pt", local_dir="checkpoints")

CLI Interactive Chat

You can chat with VaghLM directly from your terminal:

python inference/generate.py

Gradio Web Application

Launch the web interface locally to chat with parameter tuning sliders:

python gradio_app.py

Open http://127.0.0.1:7860 in your web browser.

---

🐍 Python API Example

Here is a simple example showing how to initialize the model and generate text programmatically:

import torch
import sentencepiece as spm
from model.config import VaghLMConfig
from model.transformer import VaghLM

# Load model configurations
config = VaghLMConfig()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load model
model = VaghLM(config)
checkpoint = torch.load("checkpoints/finetuned_v2_best.pt", map_location=device)
model.load_state_dict(checkpoint["model_state"])
model.to(device)
model.eval()

# Load Tokenizer
sp = spm.SentencePieceProcessor(model_file="tokenizer/vocab/tokenizer.model")

# Format prompt
prompt = "Explain Newton's third law of motion."
input_ids = [config.bos_id, config.user_id] + sp.encode(prompt) + [config.assistant_id]
x = torch.tensor([input_ids], dtype=torch.long, device=device)

# Simple greedy inference loop
generated = []
with torch.no_grad():
    for _ in range(150):
        logits, _ = model(x)
        next_token = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)
        if next_token.item() == config.eos_id:
            break
        generated.append(next_token.item())
        x = torch.cat([x, next_token], dim=1)

print("Response:", sp.decode(generated))

---

📈 Training & Alignment Pipeline

The model training pipeline is split into pretraining and instruction alignment stages:

flowchart TD
    RawData[Raw English Text Corpus] --> Tokenize[SentencePiece Tokenizer Train]
    Tokenize --> BinaryTokens[Convert Corpus to uint16 Binary File]
    BinaryTokens --> Pretraining[Pretraining Loop: Cosine LR Decay + Warmup]
    Pretraining --> PretrainedWeights[Base Model Weights: best_checkpoint.pt]
    
    PretrainedWeights --> InstructionTuning[Instruction Alignment: XML Prompts]
    InstructionTuning --> FinetunedWeights[Aligned Model Weights: finetuned_v2_best.pt]

1. Pretraining: Autoregressive next-token prediction trained on raw textual representations. The model learns syntax, facts, and code constructs.
2. Instruction Alignment: Fine-tuned on the validated instruction data layout. Key formatting rules (such as answering in structured bullet points, avoiding chat filler, and outputting definitions in the first sentence) are enforced.

---

⚠️ Limitations & Future Roadmap

Current Limitations:

• Small Capacity (42.6M): Susceptible to hallucinations on complex logic, long coding puzzles, or advanced math questions.
• Short Horizon: Optimized for outputs under 150-300 tokens. Longer outputs might loop or degrade.
• Static Positional Window: Absolute position embeddings constrain sequence attention strictly within the 2,048 context length.

Future Roadmap:

• Rotary Positional Embeddings (RoPE): Replace absolute position embeddings to improve long-range dependencies and context length extrapolation.
• SwiGLU Activation: Implement SwiGLU in the Feed-Forward Blocks to improve representative capacity per parameter.
• Grouped Query Attention (GQA): Optimize memory bandwidth and speed up multi-batch inference.
• Extended Instruction Scaling: Expand alignment datasets from 500 rows to 50k rows to support complex conversational workflows.

---

📄 License & Citation

This project is licensed under the MIT License - see the LICENSE file for details.

If you use this model or code in your research or applications, please cite it using:

@software{Rana_VaghLM-42M_A_42,
  author = {Rana, Heet},
  title = {{VaghLM-42M: A 42.6M Parameter Decoder-Only Transformer Trained From Scratch}},
  url = {https://github.com/Heet24/VaghLM-42M},
  version = {1.0.0},
  year = {2026}
}

---

🤝 Acknowledgements

• Inspired by Andrej Karpathy's nanoGPT for clean, hackable, and simple Transformer codebases.
• Optimized and trained using PyTorch native features, inspired by Meta's Llama and Google's Gemma repositories.