bala-skv/2025121002-anlp-assignment1

ANLP Assignment 1 — Custom Transformers and Byte Latent Transformer

This repository contains the pretrained checkpoints for Advanced Natural Language Processing Assignment 1.

Checkpoints

hf_models/
├── C1/
│   └── model.pt
├── C2/
│   └── model.pt
├── C3/
│   └── model.pt
├── C4/
│   └── model.pt
└── C5/
    └── model.pt

Configurations

Configuration	Positional Encoding	Attention	Normalization	Input Processing
C1	Sinusoidal	MHA	LayerNorm	Byte-BPE
C2	RoPE	MHA	LayerNorm	Byte-BPE
C3	Sinusoidal	GQA	LayerNorm	Byte-BPE
C4	Sinusoidal	MHA	RMSNorm	Byte-BPE
C5	Sinusoidal	MHA	LayerNorm	Simplified BLT

C1–C4 use 8-bit byte grouping followed by BPE source tokenization. C5 uses a token-free byte-level pipeline based on a simplified Byte Latent Transformer (BLT).

Common training setup

• Model dimension (d_model): 256
• Attention heads: 8
• Encoder layers: 4
• Decoder layers: 4
• Feed-forward dimension (d_ff): 1024
• Learning rate: 1e-4
• Batch size: 8
• Epochs: 30
• Maximum sequence length: 1024
• Optimizer: AdamW
• Random seed: 42
• Final decoding: greedy decoding

The final C1–C5 comparison uses 128-byte aligned dataset segmentation.

Byte-BPE preprocessing

For C1–C4:

binary ciphertext
      ↓
group every 8 bits
      ↓
byte values (0–255)
      ↓
BPE tokenization
      ↓
Transformer

The ciphertext/plaintext pairs are aligned. For segmented experiments, each aligned example is divided into 128-byte segments before tokenization.

C5 — Simplified BLT

C5 operates directly on bytes rather than using BPE:

ciphertext bits
      ↓
8-bit bytes
      ↓
byte patches
      ↓
local byte encoder
      ↓
global Transformer
      ↓
local byte decoder
      ↓
predicted bytes

The current implementation uses fixed 8-byte patches. The global Transformer therefore operates on patch representations instead of individual byte positions.

Checkpoint format

Each model.pt file is a PyTorch checkpoint containing:

• model state dictionary
• configuration name
• model dimensions
• vocabulary/output size
• maximum sequence length
• source tokenization
• segmentation setting
• training epoch
• training loss
• validation loss
• special-token IDs
• random seed

Loading a checkpoint

import torch

checkpoint = torch.load(
    "model.pt",
    map_location="cpu",
    weights_only=False,
)

print("Configuration:", checkpoint["config"])
print("Validation loss:", checkpoint["val_loss"])

model.load_state_dict(checkpoint["model_state_dict"])

The custom model architecture from the accompanying assignment source code is required to instantiate the model before loading the state dictionary.

Experiment tracking

Training runs were logged using Weights & Biases.

W&B project:
https://wandb.ai/skv-balasubramanian-iiit-hyderabad/anlp_assignment1

Assignment

Course: Advanced Natural Language Processing

Assignment: Transformers from Scratch, Architectural Variants, and Byte Latent Transformers (BLT)