NanoMind-SLM-60M

NanoMind-SLM is a compact Llama-style decoder-only Transformer trained from random initialization on quality-filtered Python source code.

This project was built for learning and research into dataset filtering, tokenization, Transformer architecture, pretraining, evaluation, and small-language-model deployment.

Model Details

Property	Value
Parameters	18,808,704
Architecture	Llama-style decoder-only Transformer
Training tokens	62,531,072
Validation tokens	6,277,632
Vocabulary size	8,000
Context length	256
Weight size	71.8 MB
Framework	PyTorch

Evaluation Results

Metric	Result
Validation loss	2.1563
Perplexity	8.64
Syntax validity	5/10 (50%)
Evaluated validation tokens	1,024,000

Research Comparison

Model	Training tokens	Syntax validity
Baseline model	~31M	1/10
Earlier quality model	~31M	0/10
Final quality model	62.5M	5/10

Intended Use

NanoMind-SLM is intended for:

• Educational demonstrations
• Small-language-model research
• Python code-generation experiments
• Studying training and evaluation pipelines

Limitations

This is not a production coding assistant.

The model frequently generates logically incorrect code, incomplete docstrings, repeated statements, and invalid syntax. It should not be used for security-sensitive or production code without human review.

It is a custom PyTorch architecture and is not directly compatible with the standard Transformers AutoModel interface.

Files

• model.pt: trained model weights
• model.yaml: architecture configuration
• training.yaml: training configuration
• tokenizer/: 8,000-token BPE tokenizer
• nanomind_slm/: model implementation
• final_metrics.json: validation results
• syntax_results.json: generation evaluation
• requirements.txt: runtime dependencies

Source Code

Full training pipeline and documentation:

https://github.com/anujpal1/NanoMind-SLM