ArihantRastogi20/anlp-a1-checkpoints

ANLP Assignment 1 — Roll 2023111012

Encoder–decoder transformer from basic PyTorch ops that maps encrypted binary sequences (brown_cipher.txt) to plaintext (brown_plain.txt), plus a five-way ablation (C1–C5).

Design highlights for this submission:

1. C5 uses fixed-size BLT patches (patch_size=8, concat → project; no entropy / dynamic boundaries).
2. From-scratch dual BPE trained on the train split only and stored as JSON (outputs/source_tokenizer.json, outputs/target_tokenizer.json).
3. Correct attention / metrics: mask fill with -inf; bit accuracy is reference-aligned; sacreBLEU + ROUGE for C1–C4.

Setup

source /home/arihantr/ANLP/.venv/bin/activate
cd /home/arihantr/ANLP/2023111012

# Dataset files are symlinked in the repo root:
#   brown_cipher.txt -> ../2023111015/brown_cipher.txt
#   brown_plain.txt  -> ../2023111015/brown_plain.txt

pip install -r requirements.txt   # if needed
wandb login                       # once; for online logging

Edit wandb_project / hf_repo in each configs/*.yaml before long runs.

Smoke test (quick sanity check)

cd /home/arihantr/ANLP/2023111012
source /home/arihantr/ANLP/.venv/bin/activate
python scripts/smoke_test.py

This trains/loads BPE artifacts, runs a forward+backward pass for C1–C5, and checks attention/metrics helpers.

WandB logging

Training always initializes WandB when the package is installed.

Mode	Command
Online (default after login)	export WANDB_MODE=online
Offline (local files under outputs/)	export WANDB_MODE=offline
Sync offline runs later	wandb sync outputs/wandb/offline-run-*

export WANDB_DIR=./outputs
export WANDB_MODE=online   # or offline
python -m src.train --config configs/c1_base.yaml

Logged fields include train/loss, val/loss, train/tokens_per_sec*, system/peak_memory_mb, and test metrics after eval.

Train one config

source /home/arihantr/ANLP/.venv/bin/activate
cd /home/arihantr/ANLP/2023111012

# Full train + greedy test eval
python -m src.train --config configs/c1_base.yaml

# Or split train / eval
python -m src.train --config configs/c1_base.yaml --train-only
python -m src.train --config configs/c1_base.yaml --eval-only

Configs:

Config	File	Change vs base
C1	configs/c1_base.yaml	Sinusoidal + MHA + LayerNorm + subword
C2	configs/c2_rope.yaml	RoPE
C3	configs/c3_gqa.yaml	GQA (n_kv_heads=2)
C4	configs/c4_rmsnorm.yaml	RMSNorm
C5	configs/c5_blt.yaml	Fixed-patch BLT (token-free)

Shared hparams: d_model=256, 4+4 layers, 8 heads, FFN 1024, batch 8, lr 3e-4, up to 60 epochs, patience 10.

Train all five (plug-and-play)

source /home/arihantr/ANLP/.venv/bin/activate
cd /home/arihantr/ANLP/2023111012
chmod +x scripts/run_all.sh

export WANDB_MODE=online          # or offline
export WANDB_DIR=./outputs
bash scripts/run_all.sh

# Or a subset:
bash scripts/run_all.sh c1_base c5_blt

After metrics exist:

python scripts/aggregate_results.py
# -> outputs/results_table.md and bar_*.png plots

Hugging Face checkpoints (optional)

export HF_TOKEN=hf_...
# set hf_repo in YAML, or:
python scripts/upload_checkpoints.py --repo <username>/anlp-a1-checkpoints

Outputs

Path	Contents
outputs/source_tokenizer.json	From-scratch ciphertext byte-BPE (vocab 512)
outputs/target_tokenizer.json	From-scratch plaintext BPE (vocab 1024)
outputs/checkpoints/<run>_best.pt	Best val checkpoint
outputs/<run>_history.json	Train/val curves, peak mem, throughput
outputs/<run>_metrics.json	Test metrics (greedy decode)
outputs/report_outline.md	Report skeleton / design decisions
Report.pdf	Short outline PDF (fill analysis after runs)

Repository layout

src/
├── models/
│   ├── attention.py    # SDPA, MHA, GQA (-inf mask)
│   ├── positional.py   # Sinusoidal, RoPE
│   ├── norm.py         # LayerNorm, RMSNorm
│   ├── transformer.py  # Encoder/Decoder, Seq2Seq
│   └── blt.py          # Fixed-patch LocalEncoder/Decoder + BLTSeq2Seq
├── dataset.py          # Dual BPE + byte loaders
├── train.py            # WandB training loop
└── utils.py            # Greedy decode + metrics
configs/                # c1..c5 YAML
scripts/                # smoke_test, run_all, aggregate, upload
outputs/

Links (fill after your runs)

• WandB: <paste workspace URL>
• Hugging Face: <paste repo URL>