Veovesna SLM — Indonesian Small Language Model (101M)
Veovesna SLM is an experimental Indonesian-language small language model built from scratch using a custom decoder-only Transformer architecture in PyTorch.
The project explores the development of a compact language model for Indonesian text generation and context-based question answering (QA), including three answer types: SPAN, YESNO, and UNANSWERABLE.
The released checkpoint is Run B-Control, selected based on validation results. This repository provides the model weights, tokenizer, architecture implementation, inference script, configuration, and experiment documentation.
Research checkpoint: Veovesna SLM is an experimental model, not a production-ready general-purpose assistant. Its answers can be incomplete, inaccurate, or unsupported by the supplied context.
Model Overview
| Property | Value |
|---|---|
| Model name | Veovesna SLM |
| Model type | Decoder-only Transformer |
| Language | Indonesian |
| Architecture | Custom PyTorch implementation |
| Unique parameters | 101,202,432 (101.2M) |
| Training origin | Pretrained from random initialization |
| Tokenizer | Byte Pair Encoding (BPE) |
| Vocabulary size | 20,000 |
| Context length | 1,024 tokens |
| Hidden dimension | 768 |
| Transformer layers | 12 |
| Attention heads | 12 |
| Feed-forward dimension | 3,072 |
| Activation | GELU |
| Normalization | LayerNorm |
| Embeddings | Tied input/output embeddings |
| Weight format | Safetensors (FP32) |
| Primary checkpoint | Run B-Control |
The model uses learned token and positional embeddings, causal self-attention, residual connections, LayerNorm, and GELU feed-forward layers.
The input and output token embeddings share the same weight parameter.
The architecture is implemented in modeling_veovesna.py, with its configuration in config.json.
Intended Use
Veovesna SLM is intended for:
- Research and educational experimentation with small language models.
- Studying custom Transformer architecture implementations.
- Indonesian text-generation experiments.
- Context-based question-answering research.
- Exploring model training and evaluation under limited computational resources.
Limitations
This checkpoint is not intended for:
- Medical, legal, financial, or other high-stakes decisions.
- Production deployment without additional evaluation.
- Unverified factual question answering without supporting context.
- Applications requiring consistently reliable reasoning or factual accuracy.
The model has limited capacity compared with larger language models. It can generate hallucinations, produce incomplete answers, or return answers that are related to the context but do not match the expected answer.
Training Overview
Veovesna SLM was developed through several training stages.
1. Pretraining
The model architecture was initialized with randomly initialized parameters and pretrained on an Indonesian-focused text corpus.
The documented pretraining source families include:
Detailed source identification, preprocessing information, and licensing notes are available in docs/DATA_PROVENANCE.md.
2. Supervised Fine-Tuning
The pretrained checkpoint was subsequently fine-tuned for context-based question answering.
The QA pipeline used:
- IndoQA as an identified source.
- A curated and reviewed gold QA dataset.
- Context-based answer supervision with SPAN, YESNO, and UNANSWERABLE categories.
The final gold QA training dataset contained 3,501 reviewed records:
| Category | Records |
|---|---|
| SPAN | 3,387 |
| YESNO | 56 |
| UNANSWERABLE | 58 |
| Total | 3,501 |
3. Run B Experiments
Two experimental variants were trained from the same Run A SFT checkpoint.
| Experiment | Training data |
|---|---|
| Run B-Control | 3,501 gold QA records |
| Run B-Distill | 3,501 gold records + 40 human-approved synthetic examples |
The primary released checkpoint is Run B-Control. Run B-Distill is included in the experiment documentation as a comparison, not as the primary release.
Run B configuration:
| Setting | Value |
|---|---|
| Initial checkpoint | Run A SFT best.pt |
| Optimizer | Fresh AdamW |
| Learning rate | 1e-5 |
| Optimizer steps | 110 |
| Micro-batch size | 8 |
| Gradient accumulation | 4 |
| Effective batch size | 32 |
| Sampling target | 50% SPAN / 25% YESNO / 25% UNANSWERABLE |
| Evaluation steps | 30, 60, 90, 110 |
| Decoding | Greedy |
| Maximum generated tokens | 64 |
The raw datasets, QA records, and per-example evaluation predictions are not included in this repository.
Evaluation Results
The model was evaluated on separate validation and held-out test sets.
Validation Results
The validation set contained 445 examples:
- 429 SPAN
- 8 YESNO
- 8 UNANSWERABLE
| Metric | Run B-Control | Run B-Distill |
|---|---|---|
| NLL | 0.7183 | 0.7261 |
| Perplexity | 2.051 | 2.067 |
| Answerable F1 | 0.4085 | 0.4006 |
| Answerable EM | 0.2243 | 0.2128 |
| SPAN F1 | 0.3975 | 0.3894 |
| SPAN EM | 0.2098 | 0.1981 |
| YESNO EM | 1.0000 (8/8) | 1.0000 (8/8) |
| UNANSWERABLE exact refusal | 4/8 | 2/8 |
| UNANSWERABLE hallucinations | 4/8 | 6/8 |
Run B-Control performed better on the main aggregate validation metrics and was selected as the primary checkpoint.
Held-out Test Results
The held-out test set contained 451 examples:
- 435 SPAN
- 8 YESNO
- 8 UNANSWERABLE
| Metric | Run B-Control | Run B-Distill |
|---|---|---|
| NLL | 0.8215 | 0.8233 |
| Perplexity | 2.274 | 2.278 |
| Answerable F1 | 0.3549 | 0.3753 |
| Answerable EM | 0.2190 | 0.2302 |
| SPAN F1 | 0.3523 | 0.3753 |
| SPAN EM | 0.2138 | 0.2276 |
| YESNO EM | 0.5000 (4/8) | 0.3750 (3/8) |
| UNANSWERABLE exact refusal | 4/8 | 4/8 |
| UNANSWERABLE hallucinations | 4/8 | 4/8 |
| Unsupported answer rate | 0.2641 | 0.2754 |
The held-out test set was evaluated once after model selection and was not used for training or hyperparameter selection.
Evaluation Interpretation
The results do not establish a consistent overall benefit from adding 40 synthetic QA examples.
Run B-Control performed better on validation aggregate metrics and had slightly lower test NLL, perplexity, and unsupported-answer rate. Run B-Distill achieved higher SPAN F1 and EM on the held-out test set.
The YESNO and UNANSWERABLE subsets each contain only eight examples. Therefore, their category-level metrics are highly uncertain.
These results should be treated as preliminary research findings rather than statistically robust performance estimates.
Download and Run
1. Clone the Repository
Install Git and Git LFS, then clone the repository:
git lfs install
git clone https://huggingface.co/Rafraeger/Veovesna-SLM-101M
cd Veovesna-SLM-101M
Alternatively, download the repository files directly through the Hugging Face Files and versions page.
2. Install Dependencies
It is recommended to use a Python virtual environment.
python -m venv .venv
Activate the environment on Windows:
.venv\Scripts\activate
Install the dependencies:
python -m pip install -r requirements.txt
3. Run Inference
The repository includes a custom inference script. It does not currently implement the Hugging Face AutoModelForCausalLM or AutoTokenizer interfaces.
Example:
python inference.py --context "Jakarta adalah ibu kota Indonesia." --question "Apa ibu kota Indonesia?" --max-new-tokens 64 --device auto
The inference script uses the following prompt structure:
Konteks: {context}
Pertanyaan: {question}
Jawaban:
The script loads the model configuration, Safetensors checkpoint, and tokenizer assets from the repository.
The --device argument accepts auto, cpu, or an available PyTorch device such as cuda.
Note: The example demonstrates the intended inference interface. Generated answers are not guaranteed to be correct.
Repository Structure
Veovesna-SLM-101M/
│
├── README.md
├── config.json
├── inference.py
├── model.safetensors
├── modeling_veovesna.py
├── requirements.txt
│
├── tokenizer/
│ └── tokenizer.json
│
└── docs/
├── DATA_PROVENANCE.md
├── REPRODUCIBILITY.md
└── TRAINING_AND_EVALUATION.md
Main Files
| File | Description |
|---|---|
model.safetensors |
Released Run B-Control model weights |
config.json |
Model architecture and configuration |
modeling_veovesna.py |
Custom PyTorch Transformer implementation |
inference.py |
Local inference entry point |
tokenizer/tokenizer.json |
BPE tokenizer |
requirements.txt |
Runtime dependencies |
Documentation
For additional technical details:
Licensing and Dataset Attribution
No explicit open-source license is declared for the model repository at this time.
The pretraining and QA datasets originate from multiple sources with different licensing terms. Their individual licenses and redistribution conditions are not interchangeable, and the licensing status of trained model weights has not been conclusively established.
The repository does not grant an explicit blanket license for unrestricted commercial use, modification, or redistribution of the model weights.
Please consult the original dataset sources and their applicable terms before any reuse. Additional provenance and licensing information is documented in docs/DATA_PROVENANCE.md.
Acknowledgements
This project builds on publicly documented Indonesian-language dataset resources and open-source machine learning software.
Dataset creators and maintainers are acknowledged in the data provenance documentation.
Project Status
Veovesna SLM is an experimental research project. The released checkpoint and supporting implementation are provided to make the architecture, training experiments, and evaluation results available for inspection and further research.
Future improvements may include larger and more carefully curated datasets, expanded evaluation coverage, multiple training seeds, and further experiments on Indonesian-language model training.
- Downloads last month
- -