JBalwaySUS/anlp-a1-transformer-ablation

ANLP Assignment 1 — Transformer Ablations C1–C5

Encoder–decoder Transformers built from basic PyTorch operations (no nn.Transformer or nn.MultiheadAttention), trained to map encrypted binary sequences to plaintext. C2–C5 each change exactly one component from the C1 base.

Config	Change	Test loss	Bit acc. (%)	Seq. acc. (%)	Levenshtein	BLEU	Peak GPU (MB)
C1	base	0.8883	71.99	1.40	46.16	0.3149	1338
C2	RoPE	0.0892	90.02	28.68	3.62	0.9009	1346
C3	GQA (1 KV head)	0.9995	71.54	1.05	51.89	0.2737	1337
C4	RMSNorm	1.0933	71.17	1.19	57.48	0.2504	1276
C5	BLT (token-free)	0.0098	99.60	65.88	0.77	n/a	1436

BLEU/ROUGE are omitted for C5, which has no tokenized representation to score over.

Contents

Each c1/–c5/ directory holds best.pt, metrics.json, predictions.json, and training_curves.png. C1–C4 share the two BPE tokenizers at the repository root; C5 is token-free and uses a fixed 256-byte alphabet plus PAD/BOS/EOS.

Loading a checkpoint

Checkpoints carry a self-describing architecture dict, so no external config is needed. Clone https://github.com/ your copy of the training code, then:

import torch
from huggingface_hub import hf_hub_download
from src.models.transformer import TransformerConfig, Seq2SeqTransformer

path = hf_hub_download("JBalwaySUS/anlp-a1-transformer-ablation", "c2/best.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=True)

config = TransformerConfig(**ckpt["architecture"])
model = Seq2SeqTransformer(
    source_vocab_size=1024, target_vocab_size=1024,
    source_pad_id=0, target_pad_id=0, config=config,
)
model.load_state_dict(ckpt["model"])
model.eval()

C5 uses ByteLatentTransformer from src.models.blt instead, and additionally needs the byte transition probabilities estimated from the training split.

Training

Every configuration shares d_model=64, d_ff=512, 4 heads, 4+4 layers, dropout 0.15, AdamW at lr 1e-3 with cosine decay, batch size 64, seed 42, AMP, and early stopping (patience 10, min-δ 1e-3). Data is split at document level before 256-character chunking.