AN5517/anlp-a1-transformers-run1

ANLP Assignment 1: Transformers from Scratch & Ablation Study

This repository contains trained model checkpoints and evaluation outputs for ANLP Assignment 1 across five controlled architectural configurations:

• C1 (Base): Sinusoidal Absolute PE + Multi-Head Attention (MHA) + Pre-LayerNorm + Scratch BPE
• C2 (RoPE): Rotary Positional Embedding (RoPE) + MHA + Pre-LayerNorm + Scratch BPE
• C3 (GQA): Sinusoidal Absolute PE + Grouped-Query Attention (GQA) + Pre-LayerNorm + Scratch BPE
• C4 (RMSNorm): Sinusoidal Absolute PE + MHA + RMSNorm + Scratch BPE
• C5 (BLT): Sinusoidal Absolute PE + MHA + Pre-LayerNorm + Token-Free Byte Latent Transformer

Experimental Results Summary

----------------------------------------------------------------------------------------------------------------
               ANLP ASSIGNMENT 1 — ABLATION STUDY RESULTS
----------------------------------------------------------------------------------------------------------------
Config   | Status     | Val Loss   | Seq Acc    | Bit Acc    | Levenshtein  | BLEU     | VRAM (GB)  | Time (s)  
----------------------------------------------------------------------------------------------------------------
C1       | SUCCESS    | 3.2377     | 0.0%       | 36.8%      | 843.3        | 0.03     | 5.30       | 1043.4    
C2       | SUCCESS    | 2.8871     | 0.0%       | 40.7%      | 641.0        | 0.13     | 5.36       | 1081.6    
C3       | SUCCESS    | 3.2527     | 0.0%       | 36.8%      | 844.5        | 0.01     | 5.35       | 999.9     
C4       | SUCCESS    | 3.2459     | 0.0%       | 36.9%      | 839.1        | 0.04     | 5.19       | 1009.3    
C5       | SUCCESS    | 0.0024     | 0.0%       | 0.0%       | 577.5        | N/A      | 0.87       | 182.4     
----------------------------------------------------------------------------------------------------------------