[terminal_command] TERMINAL ()
source /home/mihir.mahajan/Projects/crowd-pilot/.venv/bin/activate[mihir.mahajan@hai-login1.haicore.berlin:~/Projects/crowd-pilot] $ source /home/mihir.mahajan/Projects/crowd-pilot/.venv/bin/activate(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:~/Projects/crowd-pilot] $ queue(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:~/Projects/crowd-pilot] $  cd /fast/project/HFMI_SynergyUnit/mihir/logs/(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ lsfinetune_lora_4k.log  train(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ queue(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ lsfinetune_lora_4k_34569.log  finetune_lora_4k.log  train(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ tail -f finetune_lora_4k_34569.log/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]─ Entering Experiment qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_token… ─[18:47:19] INFO     Log directory is:                     local_scheduler.py:777                    /home/mihir.mahajan/.nemo_run/experim                                           ents/qwen3_30b_a3b_lora_finetune_crow                                           d_code_dataset_4k_tokens/qwen3_30b_a3                                           b_lora_finetune_crowd_code_dataset_4k                                           _tokens_1763920039/qwen3_30b_a3b_lora                                           _finetune_crowd_code_dataset_4k_token                                           s                                                           [18:47:19] Launching job                                       experiment.py:744           qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_t                             okens for experiment                                                            qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_t                             okens                                                                           INFO     Log directory is:                     local_scheduler.py:777                    /home/mihir.mahajan/.nemo_run/experim                                           ents/qwen3_30b_a3b_lora_finetune_crow                                           d_code_dataset_4k_tokens/qwen3_30b_a3                                           b_lora_finetune_crowd_code_dataset_4k                                           _tokens_1763920039/qwen3_30b_a3b_lora                                           _finetune_crowd_code_dataset_4k_token                                           s                                                                      INFO     Launched app:                                launcher.py:111                    local_persistent://nemo_run/qwen3_30b_a3b_lo                                    ra_finetune_crowd_code_dataset_4k_tokens-dgt                                    pkcwk3ws60                                                  ─ Waiting for Experiment qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_to… ─Experiment Status for qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_1763920039Task 0: qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens- Status: RUNNING- Executor: LocalExecutor- Job id: qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens-dgtpkcwk3ws60- Local Directory: /home/mihir.mahajan/.nemo_run/experiments/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_1763920039/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens           INFO     Waiting for job                              launcher.py:130                    qwen3_30b_a3b_lora_finetune_crowd_code_datas                                    et_4k_tokens-dgtpkcwk3ws60 to finish                                            [log=True]...                                               _4k_tokens/0 /fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you._4k_tokens/0   import pynvml  # type: ignore[import]_4k_tokens/0 I1123 18:47:22.056000 4037417 torch/distributed/run.py:657] Using nproc_per_node=2._4k_tokens/0 W1123 18:47:22.057000 4037417 torch/distributed/run.py:774] _4k_tokens/0 W1123 18:47:22.057000 4037417 torch/distributed/run.py:774] *****************************************_4k_tokens/0 W1123 18:47:22.057000 4037417 torch/distributed/run.py:774] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. _4k_tokens/0 W1123 18:47:22.057000 4037417 torch/distributed/run.py:774] *****************************************_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199] Starting elastic_operator with launch configs:_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   entrypoint         : nemo_run.core.runners.fdl_runner_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   min_nodes          : 1_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   max_nodes          : 1_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   nproc_per_node     : 2_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   run_id             : 4168_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   rdzv_backend       : c10d_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   rdzv_endpoint      : localhost:0_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   rdzv_configs       : {'timeout': 900}_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   max_restarts       : 0_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   monitor_interval   : 0.1_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   log_dir            : /home/mihir.mahajan/.nemo_run/experiments/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_1763920039/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens/nemo_run/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens-dgtpkcwk3ws60/torchelastic/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   metrics_cfg        : {}_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199]   event_log_handler  : null_4k_tokens/0 I1123 18:47:22.080000 4037417 torch/distributed/launcher/api.py:199] _4k_tokens/0 I1123 18:47:22.086000 4037417 torch/distributed/elastic/agent/server/api.py:869] [default] starting workers for entrypoint: python3_4k_tokens/0 I1123 18:47:22.087000 4037417 torch/distributed/elastic/agent/server/api.py:677] [default] Rendezvous'ing worker group_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523] [default] Rendezvous complete for workers. Result:_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   restart_count=0_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   master_addr=hai003.haicore.berlin_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   master_port=45485_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   group_rank=0_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   group_world_size=1_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   local_ranks=[0, 1]_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   role_ranks=[0, 1]_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   global_ranks=[0, 1]_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   role_world_sizes=[2, 2]_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   global_world_sizes=[2, 2]_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523]   event_log_handler=null_4k_tokens/0 I1123 18:47:22.195000 4037417 torch/distributed/elastic/agent/server/api.py:523] _4k_tokens/0 I1123 18:47:22.196000 4037417 torch/distributed/elastic/agent/server/api.py:685] [default] Starting worker group_4k_tokens/0 I1123 18:47:22.196000 4037417 torch/distributed/elastic/agent/server/local_elastic_agent.py:296] use_agent_store: True_4k_tokens/0 I1123 18:47:22.197000 4037417 torch/distributed/elastic/agent/server/local_elastic_agent.py:192] Environment variable 'TORCHELASTIC_ENABLE_FILE_TIMER' not found. Do not start FileTimerServer._4k_tokens/0 I1123 18:47:22.197000 4037417 torch/distributed/elastic/agent/server/local_elastic_agent.py:236] Environment variable 'TORCHELASTIC_HEALTH_CHECK_PORT' not found. Do not start health check._4k_tokens/0 [default1]:/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you._4k_tokens/0 [default0]:/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you._4k_tokens/0 [default1]:  import pynvml  # type: ignore[import]_4k_tokens/0 [default0]:  import pynvml  # type: ignore[import]^C(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ queue(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ queue(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ queue(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ queue(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ scancel 34572(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ queue(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ lsfinetune_lora_4k_34569.log  finetune_lora_4k_34584.log  finetune_lora_4k.logfinetune_lora_4k_34572.log  finetune_lora_4k_34585.log  train(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ cat finetune_lora_4k_34585.log | less(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ queue(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ lsfinetune_lora_4k_34569.log  finetune_lora_4k_34584.log  finetune_lora_4k_34587.log  trainfinetune_lora_4k_34572.log  finetune_lora_4k_34585.log  finetune_lora_4k.log(crowd-pilot) [mihir.mahajan@hai-login1.haicore.berlin:/fast/project/HFMI_SynergyUnit/mihir/logs] $ queue