[tab] slurm-36297.out (plaintext)
2025-12-12 16:21:42,671	INFO scripts.py:1371 -- Did not find any active Ray processes.
HAS_NVLINK: 0 (detected 0 NVLink references)
Detected Head Node IP: 10.86.2.50
2025-12-12 16:21:47,264	INFO usage_lib.py:447 -- Usage stats collection is disabled.
2025-12-12 16:21:47,266	INFO scripts.py:919 -- Local node IP: 10.86.2.50
2025-12-12 16:21:59,834	SUCC scripts.py:963 -- --------------------
2025-12-12 16:21:59,836	SUCC scripts.py:964 -- Ray runtime started.
2025-12-12 16:21:59,837	SUCC scripts.py:965 -- --------------------
2025-12-12 16:21:59,838	INFO scripts.py:967 -- Next steps
2025-12-12 16:21:59,839	INFO scripts.py:970 -- To add another node to this Ray cluster, run
2025-12-12 16:21:59,839	INFO scripts.py:973 --   ray start --address='10.86.2.50:6379'
2025-12-12 16:21:59,840	INFO scripts.py:982 -- To connect to this Ray cluster:
2025-12-12 16:21:59,841	INFO scripts.py:984 -- import ray
2025-12-12 16:21:59,842	INFO scripts.py:985 -- ray.init(_node_ip_address='10.86.2.50')
2025-12-12 16:21:59,843	INFO scripts.py:997 -- To submit a Ray job using the Ray Jobs CLI:
2025-12-12 16:21:59,844	INFO scripts.py:998 --   RAY_API_SERVER_ADDRESS='http://10.86.2.50:8265' ray job submit --working-dir . -- python my_script.py
2025-12-12 16:21:59,845	INFO scripts.py:1007 -- See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html 
2025-12-12 16:21:59,846	INFO scripts.py:1011 -- for more information on submitting Ray jobs to the Ray cluster.
2025-12-12 16:21:59,847	INFO scripts.py:1016 -- To terminate the Ray runtime, run
2025-12-12 16:21:59,848	INFO scripts.py:1017 --   ray stop
2025-12-12 16:21:59,848	INFO scripts.py:1020 -- To view the status of the cluster, use
2025-12-12 16:21:59,849	INFO scripts.py:1021 --   ray status
2025-12-12 16:21:59,850	INFO scripts.py:1025 -- To monitor and debug Ray, view the dashboard at 
2025-12-12 16:21:59,851	INFO scripts.py:1026 --   10.86.2.50:8265
2025-12-12 16:21:59,852	INFO scripts.py:1033 -- If connection to the dashboard fails, check your firewall settings and network configuration.
Ray started. Waiting for Dashboard to be ready...
Dashboard is up!
2025-12-12 16:22:06,898	INFO cli.py:41 -- Job submission server address: http://10.86.2.50:8265
2025-12-12 16:22:08,093	SUCC cli.py:65 -- -------------------------------------------------------
2025-12-12 16:22:08,096	SUCC cli.py:66 -- Job 'raysubmit_V9F6seQV39Shz46N' submitted successfully
2025-12-12 16:22:08,097	SUCC cli.py:67 -- -------------------------------------------------------
2025-12-12 16:22:08,098	INFO cli.py:291 -- Next steps
2025-12-12 16:22:08,099	INFO cli.py:292 -- Query the logs of the job:
2025-12-12 16:22:08,100	INFO cli.py:294 -- ray job logs raysubmit_V9F6seQV39Shz46N
2025-12-12 16:22:08,101	INFO cli.py:296 -- Query the status of the job:
2025-12-12 16:22:08,102	INFO cli.py:298 -- ray job status raysubmit_V9F6seQV39Shz46N
2025-12-12 16:22:08,103	INFO cli.py:300 -- Request the job to be stopped:
2025-12-12 16:22:08,104	INFO cli.py:302 -- ray job stop raysubmit_V9F6seQV39Shz46N
2025-12-12 16:22:08,108	INFO cli.py:312 -- Tailing logs until the job exits (disable with --no-wait):
2025-12-12 16:22:07,989	INFO job_manager.py:568 -- Runtime env is setting up.
Running entrypoint for job raysubmit_V9F6seQV39Shz46N: python3 train.py --hf-checkpoint /fast/project/HFMI_SynergyUnit/tab_model/huggingface/Qwen3-0.6B --load /fast/project/HFMI_SynergyUnit/tab_model/huggingface/Qwen3-0.6B --ref-load /fast/project/HFMI_SynergyUnit/tab_model/huggingface/Qwen3-0.6B --rollout-function-path miles.rollout.sft_rollout.generate_rollout --prompt-data /fast/project/HFMI_SynergyUnit/tab_model/huggingface/openhermes2_5.parquet --input-key messages --apply-chat-template --rollout-shuffle --num-epoch 3 --rollout-batch-size 128 --global-batch-size 128 --loss-type sft_loss --calculate-per-token-loss --disable-compute-advantages-and-returns --debug-train-only --optimizer adam --lr 1e-5 --lr-decay-style constant --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 --use-wandb --wandb-project crowd-pilot-miles --wandb-team instant-uv --wandb-group qwen3-0.6b-sft-torch --train-backend fsdp --update-weight-buffer-size 536870912 --gradient-checkpointing --attn-implementation flash_attention_3 --train-env-vars {\"PYTORCH_CUDA_ALLOC_CONF\":\"expandable_segments:True\"} --actor-num-gpus-per-node 2 --use-dynamic-batch-size --max-tokens-per-gpu 9216 --actor-num-nodes 1 --actor-num-gpus-per-node 2 --colocate --use-fault-tolerance --dump-details /fast/project/HFMI_SynergyUnit/tab_model/huggingface/shared_data/qwen3-600M-fsdp-1116-noref/dump_details
Traceback (most recent call last):
  File "/fast/home/mihir.mahajan/Projects/miles/train.py", line 2, in <module>
    from sglang.srt.constants import GPU_MEMORY_TYPE_KV_CACHE, GPU_MEMORY_TYPE_WEIGHTS
ModuleNotFoundError: No module named 'sglang'
2025-12-12 16:22:15,132	ERR cli.py:73 -- ---------------------------------------
2025-12-12 16:22:15,133	ERR cli.py:74 -- Job 'raysubmit_V9F6seQV39Shz46N' failed
2025-12-12 16:22:15,134	ERR cli.py:75 -- ---------------------------------------
2025-12-12 16:22:15,135	INFO cli.py:88 -- Status message: Job entrypoint command failed with exit code 1, last available logs (truncated to 20,000 chars):
2025-12-12 16:22:07,989	INFO job_manager.py:568 -- Runtime env is setting up.
Running entrypoint for job raysubmit_V9F6seQV39Shz46N: python3 train.py --hf-checkpoint /fast/project/HFMI_SynergyUnit/tab_model/huggingface/Qwen3-0.6B --load /fast/project/HFMI_SynergyUnit/tab_model/huggingface/Qwen3-0.6B --ref-load /fast/project/HFMI_SynergyUnit/tab_model/huggingface/Qwen3-0.6B --rollout-function-path miles.rollout.sft_rollout.generate_rollout --prompt-data /fast/project/HFMI_SynergyUnit/tab_model/huggingface/openhermes2_5.parquet --input-key messages --apply-chat-template --rollout-shuffle --num-epoch 3 --rollout-batch-size 128 --global-batch-size 128 --loss-type sft_loss --calculate-per-token-loss --disable-compute-advantages-and-returns --debug-train-only --optimizer adam --lr 1e-5 --lr-decay-style constant --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 --use-wandb --wandb-project crowd-pilot-miles --wandb-team instant-uv --wandb-group qwen3-0.6b-sft-torch --train-backend fsdp --update-weight-buffer-size 536870912 --gradient-checkpointing --attn-implementation flash_attention_3 --train-env-vars {\"PYTORCH_CUDA_ALLOC_CONF\":\"expandable_segments:True\"} --actor-num-gpus-per-node 2 --use-dynamic-batch-size --max-tokens-per-gpu 9216 --actor-num-nodes 1 --actor-num-gpus-per-node 2 --colocate --use-fault-tolerance --dump-details /fast/project/HFMI_SynergyUnit/tab_model/huggingface/shared_data/qwen3-600M-fsdp-1116-noref/dump_details
Traceback (most recent call last):
  File "/fast/home/mihir.mahajan/Projects/miles/train.py", line 2, in <module>
    from sglang.srt.constants import GPU_MEMORY_TYPE_KV_CACHE, GPU_MEMORY_TYPE_WEIGHTS
ModuleNotFoundError: No module named 'sglang'