ANLP Assignment 1 — Transformers from Scratch, Architectural Variants & BLT
This repo hosts the model checkpoints for a from-scratch (no nn.Transformer /
nn.MultiheadAttention) sequence-to-sequence Transformer trained to decode a
binary-cipher → plaintext mapping derived from the Brown corpus, across a
5-way architectural ablation (C1–C5).
Code: see the accompanying GitHub/submission repo (src/models/, src/train.py,
src/dataset.py, src/utils.py).
Files
| File | Config | Description |
|---|---|---|
C1.pt |
Base | Sinusoidal absolute PE + Multi-Head Attention (MHA) + LayerNorm + subword tokenizer |
C2.pt |
RoPE | Same as C1, but Rotary Positional Embeddings (RoPE) instead of sinusoidal PE |
C3.pt |
GQA | Same as C1, but Grouped-Query Attention (n_kv_heads=2) instead of MHA |
C4.pt |
RMSNorm | Same as C1, but RMSNorm instead of LayerNorm |
C5.pt |
BLT | Same as C1, but token-free Byte Latent Transformer (patch_size=4) instead of subword tokenization |
Each configuration changes exactly one component relative to the C1 baseline.
Architecture / training hyperparameters
| Parameter | Value |
|---|---|
d_model |
192 |
n_heads |
4 |
n_kv_heads (C3 only) |
2 |
d_ff |
384 |
n_layers |
4 |
patch_size (C5 only) |
4 |
| Batch size | 32 |
| Optimizer | Adam, lr 1e-3, linear warmup + cosine decay |
| Epochs | 25 |
max_gen_len (greedy decode) |
560 chars (C1–C4) / derived from MAX_BITS / patch_size (C5) |
| Seed | 42 |
Trained on an NVIDIA RTX 3060 GPU.
Dataset
Line-aligned binary-cipher → plaintext pairs derived from the Brown corpus. Sentences with cipher length ≤ 4424 bits (~553 chars, ≈ median sentence length) were kept (2500 total), split 80:10:10 into train/val/test (n_train=2000, n_val=250, n_test=250).
Results (test set, greedy decoding)
| Config | Bit-level Acc | Seq Acc | Avg Levenshtein | BLEU | ROUGE-L | Train Time (s) | Peak GPU Mem |
|---|---|---|---|---|---|---|---|
| C1 (base) | 0.9971 | 0.184 | 3.292 | 0.8866 | 0.9423 | 539.4 | 3916.6 MB |
| C2 (RoPE) | 0.9999 | 0.928 | 0.104 | 0.9960 | 0.9922 | 706.2 | 3917.5 MB |
| C3 (GQA) | 0.9940 | 0.100 | 6.048 | 0.8127 | 0.9047 | 651.6 | 3907.7 MB |
| C4 (RMSNorm) | 0.9975 | 0.216 | 2.904 | 0.8996 | 0.9481 | 717.2 | 4174.9 MB |
| C5 (BLT) | 0.9362 | 0.228 | 20.496 | – | – | 132.0 | 701.8 MB |
BLEU/ROUGE are not computed for C5, per the assignment spec ("for tokenized models only").
Takeaways: RoPE (C2) gives by far the largest quality improvement over sinusoidal PE (seq acc 0.184 → 0.928). RMSNorm (C4) is roughly on par with LayerNorm. GQA (C3) trades a small amount of accuracy for reduced K/V capacity. The token-free BLT model (C5) trains ~4–5x faster and uses ~5.6x less peak GPU memory than the tokenized configs, at the cost of accuracy — mainly due to difficulty terminating generation cleanly rather than poor content reconstruction.
Loading a checkpoint
Checkpoints are plain torch.save(state_dict, ...) files matching the
model classes defined in src/train.py / src/models/. Example:
import torch
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(repo_id="avi1o1/anlp-a1", filename="C2.pt")
state_dict = torch.load(ckpt_path, map_location="cpu")
# Rebuild the matching model architecture from src/train.py with the
# hyperparameters above (config="C2"), then:
# model.load_state_dict(state_dict)
Author: Aviral Gupta (2023111023)