aadith-warrier/anlp-a1-cipher-transformer

Cipher-to-plaintext Transformers (Advanced NLP, Assignment 1)

Encoder-decoder Transformers built from scratch (no nn.Transformer or nn.MultiheadAttention) that decrypt a position-periodic substitution cipher back to English. Five configurations differing from the C1 base by exactly one component each.

Tokenization is hand-written too: a from-scratch BPE over the raw bit string on the cipher side and over characters on the English side (C1-C4), and byte-level input with entropy-chosen patches for C5. data/bpe_*.json and data/entropy_ngram.npz are part of the checkpoint -- the weights are unusable without them.

Measured on the whole 1000-line test split (data/splits/test.txt) under greedy decoding: each line is tiled into windows, decoded, reassembled, and scored as a whole line. C5's BLEU and ROUGE-L read --, not 0 -- the assignment restricts them to tokenized models and C5 is byte-level.

Config	Seq. acc.	Lev. (mean)	Lev. (median)	BLEU	ROUGE-L	Bit acc.
C1 base	0.0160	0.0681	0.0607	0.7205	0.8511	0.7465
C2 RoPE	0.1040	0.0166	0.0129	0.8997	0.9504	0.8363
C3 GQA	0.0080	0.0982	0.0874	0.6438	0.8048	0.7356
C4 RMSNorm	0.0190	0.0745	0.0651	0.7081	0.8425	0.7441
C5 BLT	0.0020	0.1942	0.1844	--	--	0.9269

Cost

Config	Params	s/epoch	Peak GPU mem (GB)
C1 base	8,948,736	25.7	2.34
C2 RoPE	8,948,736	26.6	2.33
C3 GQA	7,764,480	26.1	2.32
C4 RMSNorm	8,943,104	25.3	2.16
C5 BLT	12,905,987	30.4	1.62

Files

c1/best.ckpt .. c5/best.ckpt   one checkpoint per configuration
data/bpe_plain.json            plaintext BPE vocabulary (C1-C4)
data/bpe_cipher.json           cipher BPE vocabulary (C1-C4)
data/entropy_ngram.npz         order-1 entropy estimator for C5's patcher
data/splits/                   the train/test partition behind every number

hf download aadith-warrier/anlp-a1-cipher-transformer --local-dir anlp-a1

Loading a checkpoint

import torch
from omegaconf import OmegaConf
from models.transformer import build_model

ck = torch.load('c1/best.ckpt', map_location='cpu', weights_only=False)
cfg = OmegaConf.create(ck['hyper_parameters'])   # the run's own config
model = build_model(cfg, 259, 2048)              # 259 for byte targets (C5)
model.load_state_dict({k[6:]: v for k, v in ck['state_dict'].items()
                       if k.startswith('model.')})

build_model is not in this repo: the snippet needs the assignment's src/ on PYTHONPATH. data/bpe_plain.json is the exact BPE vocabulary these checkpoints were trained against, and data/splits/ is the train/test partition, so the numbers above are reproducible.