AG-dataScientist/tamil-llm-23m

Tamil LLM 23M

A 23-million-parameter decoder-only Transformer language model trained from scratch for Tamil text generation.

Model Details

• Architecture: Decoder-only Transformer
• Parameters: 23,033,856
• Language: Tamil
• Vocabulary Size: 32,000
• Tokenizer: Custom BPE tokenizer
• Context Length: 256 tokens
• Layers: 6
• Hidden Size: 384
• Attention Heads: 6
• Feed-Forward Size: 1,536

Training

The model was pretrained on approximately:

• 100,000 Tamil documents
• ~58 million tokens
• 90% training split
• 10% validation split

Pretraining Configuration

Parameter	Value
Batch Size	32
Learning Rate	3e-4
Weight Decay	0.1
Maximum Epochs	4
Sequence Length	256
Warmup Steps	500
Gradient Clipping	1.0

The best completed pretrained checkpoint achieved a validation loss of approximately 5.13.

Supervised Fine-Tuning

The pretrained model was further fine-tuned using a Tamil conversational dataset.

Current SFT experiment:

Parameter	Value
Training Examples	20
Batch Size	4
Learning Rate	2e-5
Epochs	5
Optimizer	AdamW
Weight Decay	0.01

SFT Loss

Epoch	Loss
1	9.5971
2	7.7903
3	6.1306
4	4.4205
5	2.8685

Tokenizer

The model uses a custom Tamil BPE tokenizer with a 32,000-token vocabulary.

Special tokens:

<pad>          0
<unk>          1
<bos>          2
<eos>          3
<sep>          4
<|user|>       5
<|assistant|>  6
<|system|>     7

The tokenizer is provided as:

tamil_bpe_tokenizer.json
Files
best_model.pt

Best pretrained model checkpoint.

sft_model.pt

Supervised fine-tuned model checkpoint.

tamil_bpe_tokenizer.json

Custom Tamil BPE tokenizer.

config.json

Model architecture configuration.

Limitations

The current SFT model is an experimental model trained with a small instruction dataset. Generation quality is currently limited and may produce repetitive or incoherent outputs.

The model should not be considered a production-ready general-purpose Tamil language model.

Training Environment

Training was performed using Google Colab GPU.

Intended Use

This model is intended for:

Research and experimentation
Tamil language-model development
Educational purposes
Studying Transformer pretraining and SFT

It is not intended for production applications without further training and evaluation.