anhdao69/g3a1_training

G3A1 (G3 + A1) — mid-epoch training state for continuation on another 4 × H100 server

Run: G3A1 = G3 delayed native GDN-state reads (training-only state head) + A1 GeoAddress rank-8 Q/K adapters with the relation KL, fused at Qwen3.5-4B GDN decoder 24, online frozen FP32 VGGT-Omega-1B-512 teacher. Trained from Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a on the joint R2R + English RxR15 manifest (30,815 episodes, 3,128,624 steps). The schedule is a true one-epoch decay: 3,852 updates, warmup 116, cosine to 0.1 × 5e-6 (head LR 1e-4, adapter LR 5e-5, λ 0.1 for both objectives, ramp over 771 updates, now complete).

Resume from checkpoint-2100/: update 2,100 of 3,852 (54.5%), stopped cleanly by request at an update boundary. It is a complete DeepSpeed ZeRO-2 full-state checkpoint for 4 ranks:

• model weights (pytorch_model.bin, including the 4 GeoAddress tensors and the G3 state head);
• the 4 CPU-offloaded optimizer shards and mp_rank_00_model_states.pt;
• the LR scheduler, trainer_state.json (sampler position) and rng_state_0..3.pth;
• navigation.json, resume_contract.json and the tokenizer/processor;
• RECOVERY_COMPLETE.json, the inventory the trainer checks.

Training state at the stop (data/training_progress.json):

Updates done / left	2,100 / 1,752 (epoch 0.5452)
Episodes trained	16,800 exposures, all distinct, 0 repeats; 14,016 remaining in the epoch
LR at the stop	2.533e-6
Loss (mean of the last 100 updates)	0.155
G3 / A1 loss per pair (last 100 updates)	0.0139 / 0.348
CUDA OOMs	0

Invariants: do not change

Item	Value
GPUs / world size	exactly 4 H100 80GB (--nproc_per_node=4). The optimizer state is sharded for 4 ranks.
Accumulation / global batch	2 / 8
DeepSpeed	configs/deepspeed_zero2_cpu_optimizer.json (ZeRO-2, DeepSpeedCPUAdam)
Configs	configs/episodic/science_common.yaml + configs/vln_empty.yaml + configs/vln_memory_window8.yaml + configs/vggt/g3a1_joint_one_epoch.yaml + the runtime profile generated from configs/vggt/g3a1_newton_h4.yaml
Code	source/SimpleMemVLN-g3a1-877815e.tar.gz, the exact production code (byte-identical to git archive 877815e). It is not on GitHub; the full branch history is source/c2_vggt_omega_g3_a1.bundle (branch c2_vggt_omega_g3_a1, tip 260e6d8 = 877815e + implementation report).
Seed / sampler	429 / EpisodeLengthSampler; the episode order must equal data/r2r_rxr15_train.jsonl
Base model	Qwen3.5-4B snapshot 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Teacher	facebook/VGGT-Omega vggt_omega_1b_512.pt SHA-256 c02da418b18bb01d0392598d3f6147366bcde1bb70fd08a5e3bf7925b0667934 (gated; not included). The vggt-omega source tree hash b0e8d9d7… holds for any commit ≥ 299be1f (newton used main 48b23c8). These are the worker-2 G3/A1 teacher files.
Python stack	Transformers 5.11.0 (enforced) and the repo's uv.lock environment, i.e. the worker-2 venv that trained G3/A1

The trainer refuses to resume unless the checkpoint's resume_contract.json and navigation.json equal the resolved config (every field, including teacher file paths and loader settings) and the manifest SHA-256 matches. Only three things are server-specific:

• the image root in the manifest;
• the VGGT-Omega file paths;
• optionally, dataloader_num_workers, which is runtime-only: the worker count does not change the episode order or targets.

tools/prepare_resume.py changes exactly these. It proves that every other byte of meaning is unchanged and runs the trainer's own validation code before copying anything.

Steps (example values are the worker-2 paths from train/launch_a_omega_joint_e1.sh)

# 0. download and verify every byte
hf download anhdao69/g3a1_training --local-dir g3a1_training     # or: huggingface-cli download ...
cd g3a1_training && sha256sum -c --quiet SHA256SUMS && echo ALL_FILES_OK

# 1. exact code (use the venv that trained G3/A1 on worker-2: Transformers 5.11.0, DeepSpeed, FLA)
ROOT=/mnt/data/vmo-ai-task/anhdh35
mkdir -p $ROOT/src && tar -C $ROOT/src -xzf source/SimpleMemVLN-g3a1-877815e.tar.gz
SRC=$ROOT/src/SimpleMemVLN-g3a1-877815e
PY=$ROOT/SimpleMemVLN/.venv/bin/python
#    (optional git history: git clone -b c2_vggt_omega_g3_a1 source/c2_vggt_omega_g3_a1.bundle SimpleMemVLN-g3a1)

# 2. prepare the run dir (CPU only; copies + re-hashes 66 GB; must end with PREPARE_OK)
$PY tools/prepare_resume.py --bundle . --src $SRC --run-dir $ROOT/g3a1_joint_e1 \
   --image-root /YOUR/JanusVLN_data \
   --teacher-source $ROOT/third_party/vggt-omega \
   --teacher-weights $ROOT/models/vggt-omega/vggt_omega_1b_512.pt \
   --model $ROOT/cache/huggingface/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
   --dataloader-num-workers 8 --check-images
#    --dataloader-num-workers: 8 is what G3/A1 used on worker-2 (60 CPUs); omit it to keep newton's 2.
#    --copy-mode hardlink: skip the 66 GB copy when the run dir is on the same filesystem as the download.

# 3. resume inside a 4 x H100 allocation (>= 512 GB RAM; HOST_GUARD_GIB = 0.9 x RAM, default 691.2 for 768G)
export G1_CUDA_HOME=$ROOT/cuda-12.9.1 OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 MAX_JOBS=2
SRC=$SRC RUN_DIR=$ROOT/g3a1_joint_e1 \
MODEL=$ROOT/cache/huggingface/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
bash tools/resume_g3a1_h4.sh

tools/resume_g3a1_h4.sh runs the same command as the original launchers: the host-memory supervisor scripts.vln.run_episodic_gate, then torchrun with 4 ranks, --recovery-save-steps 100 and --resume_from_checkpoint auto. The default ENV_WRAPPER=$SRC/train/g1_omega_env.sh is the worker-2 environment: it sets CUDA_HOME from G1_CUDA_HOME, the venv on PATH, PYTHONPATH with third_party/vggt-omega, HF offline mode, and a node-local TORCH_EXTENSIONS_DIR for CPUAdam. On a different machine, pass ENV_WRAPPER=<script>: a script that cds to $G1_CODE_DIR, exports the same variables and ends with exec "$@".

What to expect

• Prepare prints "PREFLIGHT": "PASS". That means:

  • the serializer metadata equals the rebased navigation.json;
  • validate_resume_contract passes;
  • the teacher files hash to the trained teacher identity;
  • the copied checkpoint passes validate_complete, and latest_checkpoint selects it.

  It then prints PREPARE_OK. checkpoint-2100/REBASE_RECORD.json records every change, and the originals are kept as *.orig.

• Startup takes about 10 minutes (CPUAdam build on the first run on a node, then the 66 GB load). Expect SCHEDULE_GATE_PASS (3852, 116) on all 4 ranks and a TRAINING_CONTRACT line. The first logged update is 2,101: 'learning_rate': '2.531e-06', 'epoch': '0.5454', with a loss in the 0.1–0.3 range. The trainer trims the copied per-rank logs to update 2,100 and archives the rest under train/recovery-history/.

• Saves: recovery checkpoints at 2,200, 2,300, …, 3,800. Training ends at 3,852, the end of the epoch, with checkpoint-3852/ (full state) and final/.

• Speed: newton ran at 17.5–20 s/update with 4–8 CPUs and Lustre images. G3/A1 on worker-2 took about 14 h per full epoch (≈13 s/update). Expect roughly 6.5–10 h for the remaining 1,752 updates.

• Memory: peak GPU 67.9 GiB allocated / 76.6 GiB reserved on the longest RxR episode. Host memory peaked at 355 GiB (job total). 0 OOMs.

• Clean stop at any time: echo -n $SLURM_JOB_ID > $ROOT/g3a1_joint_e1/train/REQUEST_CHECKPOINT_AND_STOP saves a complete checkpoint at the next update and exits. To resume, re-run step 3; there is nothing to prepare again.

• Serving model after the epoch: run $PY -m scripts.vln.export_vggt_g1 --source <train>/final --output <serving_dir> --model <Qwen snapshot> from $SRC. It keeps the GeoAddress adapters and drops the G3 head (the A1 serving architecture).

Contents

Path	What
checkpoint-2100/	complete 4-rank training state (66 GB)
data/r2r_rxr15_train.jsonl	exact training manifest with newton image paths (data_root in manifest_info.json); same file as the M1_training bundle
data/manifest_info.json, data/training_progress.json	manifest SHA-256 and path-independent fingerprint_v1; progress, losses and remaining data at the stop
configs/	the configs used, plus A1's worker-2 profile a1_h4.yaml for reference (convenience copies; the source tarball is authoritative)
source/	SimpleMemVLN-g3a1-877815e.tar.gz (exact code), c2_vggt_omega_g3_a1.bundle (git history), source_manifest.sha256
run/train/	per-rank exposures/profile/microtimes logs for updates 1–2,100, which the resume needs
run/meta/, run/history/, run/logs/	trainer state, resolved config, selected episodes, recovery history, supervisor/launch/continuation logs
tools/	prepare_resume.py, preflight_resume.py, rebase_manifest.py, resume_g3a1_h4.sh
SHA256SUMS	SHA-256 of every file

History, all on newton (4 × H100):

• interactive job 926610 (evc104), updates 1–501, ended by a deadline checkpoint-stop;
• job 930602 (evc101), updates 502–2,100, ended by a user-requested clean stop.

It never trained anywhere else, and nothing on newton will continue it, so it cannot train twice. Implementation, tests and qualification gates: reports/vggt_five/G3A1/implementation_report.md and plans/vggt/G3A1_execution.md in the source.

Known caveat (from qualification): the export replay audit's R2R streaming-vs-offline max logit error was 1.0 against an atol of 0.75 on newton. A control with zeroed adapters shows the same excess, so this is native BF16 numerics on newton. Re-run the serving parity audit on the evaluation hardware.