RaunakSeksaria/anlp-a1-transformers

ANLP Assignment 1 - encoder-decoder transformer ablation

Five encoder-decoder transformers trained from scratch to map an XOR-encrypted binary sequence to English plaintext. Each configuration differs from the base by exactly one architectural component.

Config	Change from base	Checkpoint
C1	base: sinusoidal positions, multi-head attention, LayerNorm, subword	c1_best.pt
C2	RoPE instead of sinusoidal positions	c2_best.pt
C3	grouped-query attention instead of multi-head	c3_best.pt
C4	RMSNorm instead of LayerNorm	c4_best.pt
C5	token-free byte latent transformer instead of subword	c5_best.pt

Test results (greedy decoding)

Config	Bit acc	Seq acc	Levenshtein	BLEU	ROUGE-L
C1	0.8960	0.3203	2.9023	0.8645	0.9305
C2	0.9482	0.5316	1.5340	0.9215	0.9598
C3	0.8701	0.2129	4.5672	0.8120	0.9043
C4	0.8972	0.3164	2.9723	0.8609	0.9283
C5	0.9987	0.6574	0.6363

Loading a checkpoint

import torch
state = torch.load('c1_best.pt', map_location='cpu')
state['config']       # the hyperparameters used
state['model']        # the weights

Checkpoints also carry optimizer and scheduler state so training can resume. Model code is in the accompanying submission, not on the Hub.