anlp-a2-part2-sophia

Dense decoder-only transformer pretrained from scratch on browndw/human-ai-parallel-corpus (1x the training split) with a from-scratch sophia optimizer.

• params: 17,011,584
• trained on 41,680,896 tokens
• W&B run: https://wandb.ai/dunkronit-iiit-hyderabad/anlp-a2-part2/runs/sophia-781dd7da

Load with src.part2.model.Transformer.from_pretrained("DunkRonit/anlp-a2-part2-sophia") from the assignment repo.