ProjectScugnizz/scugnizz-llama-training

Scugnizz Llama-PCS Training (HF Jobs + DDP)

Parallel pretraining for ScugnizzDecoder (Llama-adjacent + PCS + RoPE) on Hugging Face Jobs.

Quick start

export HF_TOKEN=hf_...
export OUT_DIR=/mnt/rope-v2-training-results/runs/my-run
export TARGET_TOKENS=1000000000
export FLAVOR=a100x4
export TOKENS_PER_STEP=262144
export MODEL_SIZE=1.7b
export HUB_REPO_ID=ProjectScugnizz/scugnizz-llama-pcs
export HUB_PATH=training-runs/my-run

# scripts from Hub (recommended for collaborators)
export SCRIPT_SOURCE=hub:ProjectScugnizz/scugnizz-llama-training
bash run-hf-job.sh

Model sizes

MODEL_SIZE	~params
1b	1.0B
1.7b	1.7B
3b	3.0B

Architecture must match the starting weights.

Starting weights

Priority:

1. OUT_DIR/checkpoint_resume.pt or checkpoint_last.pt — full resume with optimizer
2. INIT_FROM — bucket/local path to model_final.pt or full checkpoint
3. INIT_HUB_REPO + INIT_HUB_FILE — download from Hub
4. OUT_DIR/model_final.pt — weights-only fallback

Examples:

# Continue an existing bucket run (automatic if checkpoints exist)
export OUT_DIR=/mnt/rope-v2-training-results/runs/pretrain-fineweb-llama-pcs-1.7b

# Start from Hub weights
export INIT_HUB_REPO=ProjectScugnizz/scugnizz-llama-pcs
export INIT_HUB_FILE=training-runs/pretrain-fineweb-llama-pcs-550m/model_final.pt
export INIT_STEP=2100

# Start from bucket file
export INIT_FROM=/mnt/rope-v2-training-results/runs/pretrain-fineweb-llama-pcs-1.7b/model_final.pt
export INIT_STEP=0
export NO_RESUME=1

Multi-GPU (DDP)

Uses torchrun automatically when the job has >1 GPU. Set global throughput with:

export TOKENS_PER_STEP=262144   # auto-computes GRAD_ACCUM per GPU count
# or set GRAD_ACCUM manually per GPU

Monitor GPUs:

hf jobs stats ProjectScugnizz/<job_id>

Checkpoints

Keep (optimizer resume): checkpoint_resume.pt, checkpoint_last.pt

Safe to delete: checkpoint_weights_last.pt

I/O performance: checkpoints write to /tmp/scugnizz-ckpts first; only resume/last copied to bucket. Defaults: WEIGHTS_SAVE_INTERVAL=0, SAVE_INTERVAL=100. Logs show (N inst) instantaneous tok/s.

export CKPT_LOCAL_DIR=/tmp/scugnizz-ckpts
export WEIGHTS_SAVE_INTERVAL=0
export SAVE_INTERVAL=100
bash cleanup-checkpoints.sh hf://buckets/ProjectScugnizz/rope-v2-training-results/runs/my-run

Files

File	Purpose
scugnizz-llama.py	Model + training loop (DDP)
job.sh	In-job entrypoint (pip install + torchrun)
run-hf-job.sh	Parametrized HF Jobs submitter
examples/	Example launch configs