darrenxuu/TrustMed-DoctorR1-training-state

TrustMed Doctor-R1 exact training-state recovery

This private repository is a disaster-recovery snapshot of doctorr1_final_20260902 at committed training step 12. It is not the smaller merged inference model. The snapshot contains the four-rank FSDP model, optimizer and extra-state shards, dataloader cursor, transactional experience snapshot, frozen train/validation data, launch gates, exact code snapshot, and an SHA-256 manifest/attestation.

Restore on a replacement four-GPU trainer

1. Authenticate the replacement machine to Hugging Face with read access to darrenxuu/TrustMed-DoctorR1-training-state.

2. Download the recovery tool:

   hf download darrenxuu/TrustMed-DoctorR1-training-state \
     recovery/hf_full_state_backup.py --local-dir /root/doctorr1-recovery
   

3. Download and byte-verify the complete snapshot into the original run ID:

   python3 /root/doctorr1-recovery/recovery/hf_full_state_backup.py restore \
     --repo-id darrenxuu/TrustMed-DoctorR1-training-state \
     --dest-run-dir /root/doctorr1_runs/doctorr1_final_20260902 \
     --step 12
   

4. Restore the exact RL code snapshot and provision the pinned runtime:

   tar -xzf /root/doctorr1_runs/doctorr1_final_20260902/recovery/step_12/mirror_rl_code.tar.gz -C /root
   bash /root/mirror_rl/scripts/setup_box.sh
   

5. Start the four service forwards and confirm the preflight sees patient, judge, embedding, and reranker endpoints. Then resume:

   RUN_ID=doctorr1_final_20260902 \
   RUN_MODE=resume TRAINING_PHASE=full TRAIN_GPUS=4 MICRO_BSZ=4 \
   WANDB_MODE=offline bash /root/mirror_rl/scripts/run_mirror_rl.sh
   

The restore command refuses a non-empty destination, verifies every payload file against recovery/step_12/backup_manifest.json, writes the checkpoint tracker to step 12, and atomically restores the step-12 experience snapshot. Use four training ranks: the saved FSDP state has world size 4.

Raw rollouts, reward details, and dialogue-bearing logs are intentionally not included because they are not needed for resume and may contain sensitive consultation text.