Bija-5M

Bija-5M is a compact GPT-style causal language model trained from scratch as an end-to-end language-modeling project. The project covers tokenizer training, document-level data splitting, binary dataset creation, Transformer implementation, checkpointing, evaluation, Hugging Face export, and text generation.

This repository contains the Hugging Face export of the custom Bija architecture and is intended for experimentation, learning, and lightweight generation tests.

Model Summary

• Architecture: custom causal Transformer
• Parameters: about 5M
• Layers: 6 Transformer blocks
• Hidden size: 192
• Attention heads: 6
• Context length: 256
• Vocabulary: 12,000
• Tokenizer: custom byte-level BPE
• Framework: PyTorch + Transformers custom code

Training Data

Bija-5M-Fixed was trained on a mixed English-language corpus designed to expose a small language model to multiple writing styles and domains.

The four sources used were:

• TinyStories
• FineWeb-Edu
• English Wikipedia
• Everyday Conversations

The goal of this mixture was to improve general language modeling ability while keeping the pipeline compact, auditable, and reproducible.

Data Card

Dataset Summary

The training mixture combines narrative text, educational web text, encyclopedic writing, and conversational dialogue. This gives the model exposure to child-friendly stories, broad web language, factual exposition, and short dialogue-style turns.

Intended Use

This dataset mixture was created for:

• training a small experimental autoregressive language model from scratch;
• studying tokenizer design, corpus balancing, and evaluation methodology;
• learning how to build a complete language-model training pipeline.

It was not created for:

• safety-critical use;
• legal, financial, or medical advice;
• production deployment without additional evaluation;
• high-quality instruction following.

Tokenizer and Preprocessing

A custom 12K byte-level BPE tokenizer was trained on a balanced document sample from all four sources.

Tokenizer design choices:

• byte-level pre-tokenization;
• NFC Unicode normalization;
• byte-level decoder;
• minimum token frequency of 2;
• dedicated special tokens:
  • PAD
  • BOS
  • EOS
  • UNK
  • DOC

Each document was encoded in the form:

<BOS> <DOC> document tokens <EOS>

This preserves document boundaries and helps the model learn where documents begin and end.

Data Splits

Documents were assigned at the document level to train, validation, and test splits using a 98/1/1 ratio.

This design was chosen to:

• avoid document leakage across splits;
• keep evaluation cleaner;
• preserve coherent document boundaries.

The tokenized binaries were stored as uint16, which is sufficient for the 12,000-token vocabulary.

Corpus Statistics

Dataset	Train tokens	Validation tokens	Test tokens	Total tokens
TinyStories	39,197,674	394,381	408,002	40,000,057
FineWeb-Edu	39,194,433	427,776	377,913	40,000,122
Wikipedia	19,637,001	168,144	194,939	20,000,084
Conversations	441,997	5,076	3,626	450,699
Total	98,471,105	995,377	984,480	100,450,962

Training Sampling Distribution

Sampling probabilities were computed from the actual tokenized training files rather than planned corpus targets.

Dataset	Sampling probability
TinyStories	39.806%
FineWeb-Edu	39.803%
Wikipedia	19.942%
Conversations	0.449%

This was important because the conversation corpus was much smaller than originally intended, and using real token counts prevented severe oversampling.

Known Limitations

• The conversation dataset is very small relative to the other sources.
• The corpus is English-centric and not multilingual.
• Source domains are heterogeneous, so behavior varies across tasks.
• Small models trained on mixed corpora can still memorize repeated patterns.
• This dataset mixture was built for experimentation and learning, not benchmark-level production performance.

Bias, Risk, and Safety Notes

Because the corpus includes web, encyclopedic, story, and dialogue text, it may contain:

• factual inaccuracies;
• stylistic imbalance across domains;
• social or cultural bias present in source material;
• unsafe or low-quality conversational patterns.

Outputs from models trained on this data should be reviewed before real-world use.

Simple Prompt Example

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Betrayedchair24/BIJA-5M"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

prompt = "User: Hello\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(
    **inputs,
    max_new_tokens=60,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Notes

Bija-5M-Fixed improved over the original run by:

• balancing tokenizer training across domains;
• splitting data at the document level;
• computing sampling weights from real token counts;
• using a cleaner evaluation setup;
• exporting a reproducible Hugging Face-compatible model package.

How to Use

This repository uses custom Transformers code, so load it with trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Betrayedchair24/BIJA-5M"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

prompt = "User: Hello\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(
    **inputs,
    max_new_tokens=60,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Intended Use and Out-of-Scope Use

Intended Use

• educational projects;
• experimentation with small language models;
• lightweight text generation tests;
• studying custom Hugging Face model packaging.

Out-of-Scope Use

• safety-critical applications;
• autonomous decision-making;
• high-stakes factual assistance;
• deployment without additional red-teaming and evaluation.

License

This repository is released under the Apache License 2.0, declared in the metadata above as apache-2.0.

Author

Developed by Amitabh Dey (LinkedIn)