Anvaya-1B

Anvaya-1B is a 1,018,126,848-parameter dense decoder-only language model pretrained from random initialization as part of an educational systems project: pretraining a real billion-parameter model on two 16 GB NVIDIA Tesla T4 GPUs.

Current public milestone: 160M

Field	Value
Parameters	1,018,126,848
Checkpoint update	1225
Model-input tokens	160,563,200
Context	4096
Layers / width / heads	29 / 1536 / 24
FFN width	6144
Vocabulary	128,256
Training hardware	2× Tesla T4 16 GB
Distributed strategy	PyTorch FSDP FULL_SHARD
Public export dtype	FP16
Validation NLL	4.75070
Validation perplexity	115.67
Validation token accuracy	25.21%

This is an early / intentionally undertrained checkpoint, at about 10.7% of the planned 1.5B model-input-token run.

Architecture

Anvaya-1B uses the exact dense architecture from the public trainer: 29 decoder blocks, RMSNorm, dense causal PyTorch SDPA, half-split RoPE, a standard two-layer GELU MLP, and a tied input/output token embedding matrix. It is not an SCA checkpoint.

Training and provenance

Raw resumable checkpoint archive: https://www.kaggle.com/datasets/vijayendrad/dense-1b-do

Full training recipe and engineering documentation: https://github.com/VijayendraDwari/pretrain-1b-on-2xT4

The Kaggle checkpoint contains model, optimizer, GradScaler and distributed state. This Hugging Face repository contains model-only FP16 safetensors plus the reference architecture/tokenizer files.

Exact checkpoint validation

After clean resume from update 1225:

NLL             4.7507029250
Perplexity      115.6655602
Token accuracy  0.2520523071
Validation      131,072 tokens

These are next-token metrics on the project's fixed validation sample, not downstream benchmark results.

Loading

import os, torch
from transformers import AutoTokenizer
from modeling_anvaya import Anvaya1B

repo_id = "Vijayendra/Anvaya-1B"
revision = "160M"
model = Anvaya1B.from_pretrained(repo_id, revision=revision, token=os.environ.get("HF_TOKEN"), dtype=torch.float16, device="cuda")
tokenizer = AutoTokenizer.from_pretrained(repo_id, revision=revision, token=os.environ.get("HF_TOKEN"))

See inference_example.py for a small generation example. The reference model code intentionally stays close to the training implementation and does not implement a KV cache yet.

Tokenizer

Tokenizer family: meta-llama/Llama-3.2-1B; vocabulary 128,256; EOS 128001. Tokenizer files retain their upstream terms.

Data

Training uses tokenized FineWeb data derived from HuggingFaceFW/fineweb, revision v1.4.0, config CC-MAIN-2025-26. The historical corpus contains 1.485B train tokens and 15M validation tokens. The sampler uses deterministic 4096-token windows with replacement, so the reported token count is a model-input ledger rather than a unique-token count.

Planned milestone tags

• 160M — first public early checkpoint
• 250M
• 500M
• 1B
• 1.5B — planned final target

Limitations

Early / undertrained checkpoint; no downstream benchmark suite yet; no KV cache; not optimized for serving; not a compute-optimal 1B recipe; outputs may be inaccurate, incoherent, biased, or unsafe. Do not use it for high-stakes decisions.

Licensing

The original model/training code in the GitHub project is MIT-licensed. This initial weights release is marked license: other until a dedicated weights license is explicitly selected. Tokenizer files retain their upstream terms. Nothing here overrides third-party dataset/tokenizer terms.

Author

Vijayendra Dwari