anhdao69/M1_training

M1 (P1-M1-s429) — mid-epoch-1 training state for continuation on another server

Run: episodic-memory campaign arm M1 (R = 9 recent groups, B = 0, no episodic bank). Attempt P1-M1-s429-H2-a2, campaign c42b737b-9519-4be5-9886-c8a2c9c2db50. Trained from Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a on the full joint R2R + English RxR15 manifest (30,815 episodes, 3,128,624 steps).

Resume from checkpoint-3142/ (update 3,142 of 3,852 = epoch 1). checkpoint-3000/ is a fallback only. Both are complete full-state DeepSpeed ZeRO-2 checkpoints. Each holds:

• model weights (pytorch_model.bin);
• CPU-offloaded optimizer shards for 2 ranks;
• the LR scheduler;
• trainer_state.json, including the sampler position;
• rng_state_0/1.pth;
• navigation.json and resume_contract.json;
• the processor and tokenizer.

Invariants: do not change

Item	Value
GPUs / world size	exactly 2 H100 80GB (--nproc_per_node=2). Do not resume on 4 GPUs.
Gradient accumulation / global batch	4 / 8 (from configs/profile_h2.yaml)
DeepSpeed	configs/deepspeed_zero2_cpu_optimizer.json (ZeRO-2, optimizer on CPU)
Configs	configs/science_common.yaml, configs/arm_m1.yaml, configs/profile_h2.yaml. Resolved config SHA-256 0d7b1e9c…40b3, identical to the other server's own M1 launch.
Code	source/SimpleMemVLN-a340e13.tar.gz (exact training code; also GitHub anhdao69/SimpleMemVLN commit a340e13, branch c2_ablations). Model code is identical to c2_h2_speed e789a13.
Schedule	7,704-update cosine (warmup 232, min ratio 0.1, LR 5e-6), stopped at epoch 1 = update 3,852 (--stop-after-epochs 1)
Seed / sampler	429 / EpisodeLengthSampler; the episode order must equal data/r2r_rxr15_train.jsonl
Base model	Qwen3.5-4B snapshot 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Steps

# 0. download and verify every byte
hf download anhdao69/M1_training --local-dir M1_training      # or: huggingface-cli download ...
cd M1_training && sha256sum -c SHA256SUMS                       # must print only OK lines

# 1. exact code (use your existing uv environment from this repo's uv.lock)
mkdir -p /YOUR/src && tar -C /YOUR/src -xzf source/SimpleMemVLN-a340e13.tar.gz
SRC=/YOUR/src/a340e131b63c9ab7846690acdb369599462076c4

# 2. manifest with YOUR image root (only the rgb_path prefix changes; verified)
python3 tools/rebase_manifest.py --manifest data/r2r_rxr15_train.jsonl \
   --old-root /groups/yshang/an221229/data/JanusVLN_data --new-root /YOUR/JanusVLN_data \
   --out /YOUR/r2r_rxr15_train.rebased.jsonl --check-images

# 3. output dir: REAL copies (the trainer rejects symlinks) of the checkpoint and the logs
OUT=/YOUR/M1-continue/train && mkdir -p $OUT
cp -a checkpoint-3142 $OUT/
cp -a run/train/*.jsonl run/train/*.json $OUT/

# 4. point the copied checkpoint's resume contract at the rebased manifest
#    (verified, same-size, one-field change; original kept as resume_contract.json.orig)
python3 tools/rebase_resume_contract.py --checkpoint $OUT/checkpoint-3142 \
   --original-manifest data/r2r_rxr15_train.jsonl --rebased-manifest /YOUR/r2r_rxr15_train.rebased.jsonl

# 5. resume (same command as the original run, with your paths)
cd $SRC
export PYTHONPATH=$SRC/src:$SRC PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 OPENBLAS_NUM_THREADS=1 TOKENIZERS_PARALLELISM=false
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1      # CUDA_HOME / CXX as your env requires (TileLang needs C++20)
CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc_per_node=2 -m qwen_vl.train.train_qwen \
  --vln_config configs/episodic/science_common.yaml --output_config configs/episodic/arm_m1.yaml \
  --runtime_profile configs/episodic/profile_h2.yaml --deepspeed configs/deepspeed_zero2_cpu_optimizer.json \
  --manifest /YOUR/r2r_rxr15_train.rebased.jsonl \
  --model_name_or_path /YOUR/hf/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
  --output_dir $OUT --run-name P1-M1-s429-H2-a2 \
  --recovery-save-steps 250 --first-recovery-save-step 50 --stop-after-epochs 1 \
  --resume_from_checkpoint auto

What to expect: the trainer validates the resume contract (navigation, episode order, rebased manifest hash, world size 2, accumulation 4) and the checkpoint inventory before training. It then continues at update 3,143 and stops at the epoch boundary, update 3,852 (about 710 updates; ≈22–26 s/update on 2 H100s with 16 CPUs). It writes checkpoint-3852/ (full state, for epoch 2) and final/ (serving export). Recovery saves happen every 250 updates (3,250, 3,500, 3,750).

Resources measured on this run: per-GPU peak reserved ≈ 73 GiB on the longest episode. Host memory peaked at 262 GiB working set (startup gate), so give the job ≥ 512 GB RAM. 0 OOMs.

Contents

Path	What
checkpoint-3142/, checkpoint-3000/	complete training state (2 ranks)
data/r2r_rxr15_train.jsonl, data/manifest_info.json	exact training manifest (original paths), its SHA-256 and path-independent fingerprint_v1
configs/	the four config files used (convenience copies; the source tarball is authoritative)
source/SimpleMemVLN-a340e13.tar.gz	immutable training source snapshot
run/	launch manifest, resolved config, status, events, Slurm logs, per-allocation qualification reports, per-rank profile/exposure/microtime logs up to update 3,142
tools/	rebase_manifest.py, rebase_resume_contract.py
SHA256SUMS	SHA-256 of every file

History: trained on newton, first in interactive job 899055 (updates 1–2,975; checkpoint-stop before the time limit), then 916782 (updates 2,976–3,142, GPUs 0,1). Never trained elsewhere; the queued continuation here was cancelled so M1 cannot train twice. Training notes and decisions: reports/episodic_idea/09_ablation_runs.md (GitHub branch c2_ablations). Epoch 1 is a mid-schedule snapshot of the 2-epoch schedule (LR at update 3,852 ≈ 2.86e-6).