[tab] /fast/project/HFMI_SynergyUnit/mihir/logs/finetune_lora_4k_34587.log (log)
/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
─ Entering Experiment qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_token… ─
[19:32:20] INFO     Log directory is:                     local_scheduler.py:777
                    /home/mihir.mahajan/.nemo_run/experim                       
                    ents/qwen3_30b_a3b_lora_finetune_crow                       
                    d_code_dataset_4k_tokens/qwen3_30b_a3                       
                    b_lora_finetune_crowd_code_dataset_4k                       
                    _tokens_1763922739/qwen3_30b_a3b_lora                       
                    _finetune_crowd_code_dataset_4k_token                       
                    s                                                           
[19:32:20] Launching job                                       experiment.py:744
           qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_t                  
           okens for experiment                                                 
           qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_t                  
           okens                                                                
           INFO     Log directory is:                     local_scheduler.py:777
                    /home/mihir.mahajan/.nemo_run/experim                       
                    ents/qwen3_30b_a3b_lora_finetune_crow                       
                    d_code_dataset_4k_tokens/qwen3_30b_a3                       
                    b_lora_finetune_crowd_code_dataset_4k                       
                    _tokens_1763922739/qwen3_30b_a3b_lora                       
                    _finetune_crowd_code_dataset_4k_token                       
                    s                                                           
           INFO     Launched app:                                launcher.py:111
                    local_persistent://nemo_run/qwen3_30b_a3b_lo                
                    ra_finetune_crowd_code_dataset_4k_tokens-d7d                
                    42vwtpqpjwd                                                 
─ Waiting for Experiment qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_to… ─


Experiment Status for 
qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_1763922739

Task 0: qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens
- Status: RUNNING
- Executor: LocalExecutor
- Job id: qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens-d7d42vwtpqpjwd
- Local Directory: /home/mihir.mahajan/.nemo_run/experiments/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_1763922739/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens

           INFO     Waiting for job                              launcher.py:130
                    qwen3_30b_a3b_lora_finetune_crowd_code_datas                
                    et_4k_tokens-d7d42vwtpqpjwd to finish                       
                    [log=True]...                                               
_4k_tokens/0 /fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
_4k_tokens/0   import pynvml  # type: ignore[import]
_4k_tokens/0 I1123 19:32:22.528000 4058528 torch/distributed/run.py:657] Using nproc_per_node=2.
_4k_tokens/0 W1123 19:32:22.529000 4058528 torch/distributed/run.py:774] 
_4k_tokens/0 W1123 19:32:22.529000 4058528 torch/distributed/run.py:774] *****************************************
_4k_tokens/0 W1123 19:32:22.529000 4058528 torch/distributed/run.py:774] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
_4k_tokens/0 W1123 19:32:22.529000 4058528 torch/distributed/run.py:774] *****************************************
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199] Starting elastic_operator with launch configs:
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   entrypoint         : nemo_run.core.runners.fdl_runner
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   min_nodes          : 1
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   max_nodes          : 1
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   nproc_per_node     : 2
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   run_id             : 2662
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   rdzv_backend       : c10d
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   rdzv_endpoint      : localhost:0
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   rdzv_configs       : {'timeout': 900}
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   max_restarts       : 0
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   monitor_interval   : 0.1
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   log_dir            : /home/mihir.mahajan/.nemo_run/experiments/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_1763922739/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens/nemo_run/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens-d7d42vwtpqpjwd/torchelastic/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   metrics_cfg        : {}
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199]   event_log_handler  : null
_4k_tokens/0 I1123 19:32:22.532000 4058528 torch/distributed/launcher/api.py:199] 
_4k_tokens/0 I1123 19:32:22.538000 4058528 torch/distributed/elastic/agent/server/api.py:869] [default] starting workers for entrypoint: python3
_4k_tokens/0 I1123 19:32:22.538000 4058528 torch/distributed/elastic/agent/server/api.py:677] [default] Rendezvous'ing worker group
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523] [default] Rendezvous complete for workers. Result:
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   restart_count=0
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   master_addr=hai003.haicore.berlin
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   master_port=34181
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   group_rank=0
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   group_world_size=1
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   local_ranks=[0, 1]
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   role_ranks=[0, 1]
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   global_ranks=[0, 1]
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   role_world_sizes=[2, 2]
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   global_world_sizes=[2, 2]
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523]   event_log_handler=null
_4k_tokens/0 I1123 19:32:22.661000 4058528 torch/distributed/elastic/agent/server/api.py:523] 
_4k_tokens/0 I1123 19:32:22.662000 4058528 torch/distributed/elastic/agent/server/api.py:685] [default] Starting worker group
_4k_tokens/0 I1123 19:32:22.662000 4058528 torch/distributed/elastic/agent/server/local_elastic_agent.py:296] use_agent_store: True
_4k_tokens/0 I1123 19:32:22.663000 4058528 torch/distributed/elastic/agent/server/local_elastic_agent.py:192] Environment variable 'TORCHELASTIC_ENABLE_FILE_TIMER' not found. Do not start FileTimerServer.
_4k_tokens/0 I1123 19:32:22.664000 4058528 torch/distributed/elastic/agent/server/local_elastic_agent.py:236] Environment variable 'TORCHELASTIC_HEALTH_CHECK_PORT' not found. Do not start health check.
_4k_tokens/0 [default1]:/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
_4k_tokens/0 [default1]:  import pynvml  # type: ignore[import]
_4k_tokens/0 [default0]:/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
_4k_tokens/0 [default0]:  import pynvml  # type: ignore[import]
_4k_tokens/0 [default1]:[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
_4k_tokens/0 [default1]:[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
_4k_tokens/0 [default1]:[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
_4k_tokens/0 [default1]:[W1123 19:32:45.307010279 ProcessGroupNCCL.cpp:981] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 api:1251] Disabling try_restore_best_ckpt restoration for adapters
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 nemo_logger:162] Experiments will be logged at /fast/project/HFMI_SynergyUnit/tab_model/mihir/logs/qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens/2025-11-23_19-32-51
_4k_tokens/0 [default0]:GPU available: True (cuda), used: True
_4k_tokens/0 [default0]:TPU available: False, using: 0 TPU cores
_4k_tokens/0 [default0]:HPU available: False, using: 0 HPUs
_4k_tokens/0 [default0]:[NeMo W 2025-11-23 19:32:51 nemo_logger:190] "update_logger_directory" is True. Overwriting tensorboard logger "save_dir" to /fast/project/HFMI_SynergyUnit/tab_model/mihir/logs/tb_logs
_4k_tokens/0 [default0]:[NeMo W 2025-11-23 19:32:51 nemo_logger:209] The Trainer already contains a ModelCheckpoint callback. This will be overwritten.
_4k_tokens/0 [default0]:[NeMo W 2025-11-23 19:32:51 nemo_logger:239] The checkpoint callback was told to monitor a validation value and trainer's max_steps was set to 1000. Please ensure that max_steps will run for at least 1 epochs to ensure that checkpointing will not error out.
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_strategy:460] Fixing mis-match between ddp-config & mcore-optimizer config
_4k_tokens/0 [default0]:You are using a CUDA device ('NVIDIA H100 80GB HBM3') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:449] Rank 0 has data parallel group : [0]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:455] Rank 0 has combined group of data parallel and context parallel : [0]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:460] All data parallel group ranks with context parallel combined: [[0], [1]]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:463] Ranks 0 has data parallel rank: 0
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:471] Rank 0 has context parallel group: [0]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:474] All context parallel group ranks: [[0], [1]]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:475] Ranks 0 has context parallel rank: 0
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:482] Rank 0 has model parallel group: [0, 1]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:483] All model parallel group ranks: [[0, 1]]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:492] Rank 0 has tensor model parallel group: [0, 1]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:496] All tensor model parallel group ranks: [[0, 1]]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:497] Rank 0 has tensor model parallel rank: 0
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:507] All expert model parallel group ranks: [[0, 1]]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:508] Rank 0 has expert model parallel rank: 0
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:532] Rank 0 has pipeline model parallel group: [0]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:544] Rank 0 has embedding group: [0]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:550] All pipeline model parallel group ranks: [[0], [1]]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:551] Rank 0 has pipeline model parallel rank 0
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:552] All embedding group ranks: [[0], [1]]
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:51 megatron_init:553] Rank 0 has embedding rank: 0
_4k_tokens/0 [default0]:[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
_4k_tokens/0 [default0]:[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
_4k_tokens/0 [default0]:[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
_4k_tokens/0 [default0]:[W1123 19:32:51.528323712 ProcessGroupNCCL.cpp:981] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
_4k_tokens/0 [default0]:----------------------------------------------------------------------------------------------------
_4k_tokens/0 [default0]:distributed_backend=nccl
_4k_tokens/0 [default0]:All distributed processes registered. Starting with 2 processes
_4k_tokens/0 [default0]:----------------------------------------------------------------------------------------------------
_4k_tokens/0 [default0]:
_4k_tokens/0 [default1]:[Gloo] Rank 1 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
_4k_tokens/0 [default0]:[Gloo] Rank 0 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:54 model_transform:67] Setting up ModelTransform for stage: fit
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:54 model_transform:70] Found model_transform attribute on pl_module
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:54 model_transform:73] Set model_transform to: <function _call_counter.<locals>.wrapper at 0x7fd10907e5f0>
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:54 base:373] Use preset vocab_size: 151936, original vocab_size: 151669, dummy tokens: 267.
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:56 megatron_strategy:472] Copying Trainer's 'max_steps' (1000) to LR scheduler's 'max_steps'.
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:56 num_microbatches_calculator:228] setting number of microbatches to constant 4
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:56 megatron_strategy:1045] Doing selective restore from RestoreConfig(path='/fast/project/HFMI_SynergyUnit/tab_model/data/checkpoints/nemo_converted_weights_qwen3-coder-30b-a3b-instruct', load_model_state=True, load_optim_state=False, load_artifacts=True)
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:56 pl:411] Loaded sharded_state_dict_metadata from checkpoint: {'distrib_optim_sharding_type': 'fully_sharded_model_space'}
_4k_tokens/0 [default1]:LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
_4k_tokens/0 [default0]:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
_4k_tokens/0 [default0]:[NeMo W 2025-11-23 19:32:56 serialization:184] DEPRECATED: Passing 'checkpoint_dir' as a Path object in load_common_state_dict will no longer be supported in a future release. Please pass it as a string instead.
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:32:57 pl:275] Using <megatron.core.dist_checkpointing.strategies.fully_parallel.FullyParallelLoadStrategyWrapper object at 0x7fd0b0639090> dist-ckpt load strategy.
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:18 pl:297] Global Checkpoint Load : Rank : 0 : Start time : 1763922777.025s : Time spent in load_checkpoint: 201.714s
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:18 megatron_strategy:1050] Restoring model weights from RestoreConfig(path='/fast/project/HFMI_SynergyUnit/tab_model/data/checkpoints/nemo_converted_weights_qwen3-coder-30b-a3b-instruct', load_model_state=True, load_optim_state=False, load_artifacts=True)
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:19 megatron_strategy:1058] Finished restoring from RestoreConfig(path='/fast/project/HFMI_SynergyUnit/tab_model/data/checkpoints/nemo_converted_weights_qwen3-coder-30b-a3b-instruct', load_model_state=True, load_optim_state=False, load_artifacts=True), cleaning up.
_4k_tokens/0 [default0]:
_4k_tokens/0 [default0]:  | Name   | Type     | Params | Mode 
_4k_tokens/0 [default0]:--------------------------------------------
_4k_tokens/0 [default0]:0 | module | GPTModel | 15.3 B | train
_4k_tokens/0 [default0]:--------------------------------------------
_4k_tokens/0 [default0]:15.3 B    Trainable params
_4k_tokens/0 [default0]:0         Non-trainable params
_4k_tokens/0 [default0]:15.3 B    Total params
_4k_tokens/0 [default0]:61,089.833Total estimated model params size (MB)
_4k_tokens/0 [default0]:1065      Modules in train mode
_4k_tokens/0 [default0]:0         Modules in eval mode
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.0.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.0.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.1.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.1.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.2.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.2.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.3.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.3.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.4.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.4.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.5.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.5.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.6.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.6.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.7.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.7.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo W 2025-11-23 19:36:20 nemo_logging:361] /fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:424: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=2` in the `DataLoader` to improve performance.
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.8.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.8.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.9.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.9.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.10.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.10.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.11.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.11.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.12.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.12.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.13.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.13.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.14.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.14.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.15.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.15.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.16.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.16.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.17.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.17.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.18.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.18.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.19.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.19.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.20.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.20.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.21.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.21.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.22.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.22.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.23.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.23.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.24.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.24.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.25.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.25.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.26.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.26.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.27.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.27.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.28.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.28.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.29.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.29.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.30.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.30.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.31.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.31.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.32.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.32.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.33.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.33.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.34.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.34.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.35.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.35.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.36.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.36.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.37.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.37.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.38.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.38.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.39.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.39.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.40.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.40.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.41.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.41.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.42.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.42.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.43.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.43.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.44.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.44.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.45.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.45.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.46.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.46.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.47.self_attention.linear_proj
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 lora:468] Adding lora to: module.decoder.layers.47.self_attention.linear_qkv
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 model_transform:97] After applying model_transform:
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:  | Name   | Type     | Params | Mode 
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:--------------------------------------------
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:0 | module | GPTModel | 15.3 B | train
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:--------------------------------------------
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:20.4 M    Trainable params
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:15.3 B    Non-trainable params
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:15.3 B    Total params
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:61,171.622Total estimated model params size (MB)
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:1545      Modules in train mode
_4k_tokens/0 [default0]:    
_4k_tokens/0 [default0]:0         Modules in eval mode
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 peft:250] Initializing model parallel
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 megatron_parallel:584]  > number of parameters on (tensor, pipeline) model parallel rank (0 ,0): 15292905472
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 megatron_parallel:587]  > number of trainable parameters: 20447232 (0.13% of total)
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 utils:673] Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=False, overlap_param_gather=False, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=True, check_for_large_grads=False, bucket_size=None, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_megatron_fsdp=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache=False, nccl_ub=False, fsdp_double_buffer=False, outer_dp_sharding_strategy='no_shard', disable_symmetric_registration=False, delay_wgrad_compute=False)
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 utils:694] Number of buckets for gradient all-reduce / reduce-scatter: 1
_4k_tokens/0 [default0]:    Params for bucket 1 (20447232 elements, 20447232 padded size):
_4k_tokens/0 [default0]:    	module.decoder.layers.46.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.34.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.31.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.18.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.45.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.42.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.40.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.40.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.25.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.37.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.32.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.29.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.20.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.13.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.10.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.6.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.2.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.42.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.26.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.23.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.16.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.7.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.44.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.36.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.27.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.21.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.17.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.16.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.8.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.4.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.2.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.1.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.43.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.41.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.24.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.20.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.5.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.47.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.38.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.34.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.20.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.19.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.44.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.40.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.24.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.21.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.3.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.0.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.45.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.32.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.31.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.29.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.13.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.10.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.46.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.30.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.36.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.33.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.22.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.20.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.14.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.11.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.3.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.43.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.37.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.23.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.17.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.16.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.5.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.4.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.37.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.28.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.15.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.12.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.9.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.45.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.41.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.27.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.24.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.0.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.45.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.35.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.26.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.23.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.16.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.10.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.36.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.33.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.19.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.6.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.28.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.8.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.39.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.38.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.34.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.25.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.47.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.28.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.23.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.18.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.15.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.12.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.9.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.5.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.32.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.29.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.22.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.13.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.10.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.8.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.7.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.6.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.38.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.27.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.9.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.38.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.32.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.31.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.26.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.14.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.11.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.4.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.4.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.0.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.40.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.35.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.18.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.3.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.31.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.47.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.36.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.34.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.13.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.8.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.1.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.42.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.35.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.22.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.1.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.42.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.37.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.6.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.5.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.2.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.35.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.43.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.46.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.43.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.39.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.33.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.30.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.22.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.41.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.27.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.25.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.25.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.17.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.15.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.12.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.2.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.47.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.28.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.26.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.18.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.15.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.12.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.9.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.0.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.30.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.21.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.19.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.3.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.1.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.44.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.39.self_attention.linear_proj.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.24.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.21.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.19.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.14.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.7.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.39.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.17.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.11.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.46.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.44.self_attention.linear_qkv.adapter.linear_out.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.41.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.33.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.30.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.29.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.14.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.11.self_attention.linear_proj.adapter.linear_in.weight
_4k_tokens/0 [default0]:    	module.decoder.layers.7.self_attention.linear_qkv.adapter.linear_in.weight
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 peft:254] Setting up optimizers
_4k_tokens/0 [default0]:[NeMo I 2025-11-23 19:36:20 utils:673] Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=None, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='delayed', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.98, adam_eps=1e-05, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather=False, overlap_param_gather_with_optimizer_step=False, use_megatron_fsdp=False, optimizer_cpu_offload=False, optimizer_offload_fraction=0.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=False, timers=None, config_logger_dir='')
_4k_tokens/0 [default1]:[rank1]: Traceback (most recent call last):
_4k_tokens/0 [default1]:[rank1]:   File "/home/mihir.mahajan/.local/share/uv/python/cpython-3.10.12-linux-x86_64-gnu/lib/python3.10/runpy.py", line 196, in _run_module_as_main
_4k_tokens/0 [default1]:[rank1]:     return _run_code(code, main_globals, None,
_4k_tokens/0 [default1]:[rank1]:   File "/home/mihir.mahajan/.local/share/uv/python/cpython-3.10.12-linux-x86_64-gnu/lib/python3.10/runpy.py", line 86, in _run_code
_4k_tokens/0 [default1]:[rank1]:     exec(code, run_globals)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo_run/core/runners/fdl_runner.py", line 72, in <module>
_4k_tokens/0 [default1]:[rank1]:     fdl_runner_app()
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/main.py", line 327, in __call__
_4k_tokens/0 [default1]:[rank1]:     raise e
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/main.py", line 310, in __call__
_4k_tokens/0 [default1]:[rank1]:     return get_command(self)(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/click/core.py", line 1485, in __call__
_4k_tokens/0 [default1]:[rank1]:     return self.main(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/core.py", line 719, in main
_4k_tokens/0 [default1]:[rank1]:     return _main(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/core.py", line 192, in _main
_4k_tokens/0 [default1]:[rank1]:     rv = self.invoke(ctx)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/click/core.py", line 1269, in invoke
_4k_tokens/0 [default1]:[rank1]:     return ctx.invoke(self.callback, **ctx.params)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/click/core.py", line 824, in invoke
_4k_tokens/0 [default1]:[rank1]:     return callback(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/main.py", line 691, in wrapper
_4k_tokens/0 [default1]:[rank1]:     return callback(**use_params)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo_run/core/runners/fdl_runner.py", line 68, in fdl_direct_run
_4k_tokens/0 [default1]:[rank1]:     fdl_fn()
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/api.py", line 237, in finetune
_4k_tokens/0 [default1]:[rank1]:     return train(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/api.py", line 136, in train
_4k_tokens/0 [default1]:[rank1]:     trainer.fit(model, data)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 538, in fit
_4k_tokens/0 [default1]:[rank1]:     call._call_and_handle_interrupt(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 46, in _call_and_handle_interrupt
_4k_tokens/0 [default1]:[rank1]:     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
_4k_tokens/0 [default1]:[rank1]:     return function(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 574, in _fit_impl
_4k_tokens/0 [default1]:[rank1]:     self._run(model, ckpt_path=ckpt_path)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 981, in _run
_4k_tokens/0 [default1]:[rank1]:     results = self._run_stage()
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1023, in _run_stage
_4k_tokens/0 [default1]:[rank1]:     self._run_sanity_check()
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1052, in _run_sanity_check
_4k_tokens/0 [default1]:[rank1]:     val_loop.run()
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 178, in _decorator
_4k_tokens/0 [default1]:[rank1]:     return loop_run(self, *args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 135, in run
_4k_tokens/0 [default1]:[rank1]:     self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 396, in _evaluation_step
_4k_tokens/0 [default1]:[rank1]:     output = call._call_strategy_hook(trainer, hook_name, *step_args)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 319, in _call_strategy_hook
_4k_tokens/0 [default1]:[rank1]:     output = fn(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/pytorch/strategies/megatron_strategy.py", line 829, in validation_step
_4k_tokens/0 [default1]:[rank1]:     out = self.model.validation_step(dataloader_iter, *args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 371, in validation_step
_4k_tokens/0 [default1]:[rank1]:     return self._step(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 458, in _step
_4k_tokens/0 [default1]:[rank1]:     return self.forward(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 308, in forward
_4k_tokens/0 [default1]:[rank1]:     microbatch_outputs = step()
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 1248, in __call__
_4k_tokens/0 [default1]:[rank1]:     return self.forward_backward_func(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/pipeline_parallel/schedules.py", line 600, in forward_backward_no_pipelining
_4k_tokens/0 [default1]:[rank1]:     output_tensor, num_tokens = forward_step(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/pipeline_parallel/schedules.py", line 400, in forward_step
_4k_tokens/0 [default1]:[rank1]:     output_tensor, loss_func = forward_step_func(data_iterator, model)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 523, in wrapped_forward_step_func
_4k_tokens/0 [default1]:[rank1]:     output_tensor = _forward_step(model, batch)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 858, in wrapped
_4k_tokens/0 [default1]:[rank1]:     return attr(*args)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py", line 693, in validation_step
_4k_tokens/0 [default1]:[rank1]:     return self.forward_step(batch)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py", line 666, in forward_step
_4k_tokens/0 [default1]:[rank1]:     return self.config.forward_step_fn(self, batch)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py", line 143, in gpt_forward_step
_4k_tokens/0 [default1]:[rank1]:     return model(**forward_args)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
_4k_tokens/0 [default1]:[rank1]:     return self._call_impl(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
_4k_tokens/0 [default1]:[rank1]:     return forward_call(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py", line 634, in forward
_4k_tokens/0 [default1]:[rank1]:     output_tensor = self.module(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
_4k_tokens/0 [default1]:[rank1]:     return self._call_impl(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
_4k_tokens/0 [default1]:[rank1]:     return forward_call(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/distributed/data_parallel_base.py", line 22, in forward
_4k_tokens/0 [default1]:[rank1]:     return self.module(*inputs, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
_4k_tokens/0 [default1]:[rank1]:     return self._call_impl(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
_4k_tokens/0 [default1]:[rank1]:     return forward_call(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/transformer/module.py", line 237, in forward
_4k_tokens/0 [default1]:[rank1]:     outputs = self.module(*inputs, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
_4k_tokens/0 [default1]:[rank1]:     return self._call_impl(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
_4k_tokens/0 [default1]:[rank1]:     return forward_call(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/models/gpt/gpt_model.py", line 395, in forward
_4k_tokens/0 [default1]:[rank1]:     return self._postprocess(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/models/gpt/gpt_model.py", line 562, in _postprocess
_4k_tokens/0 [default1]:[rank1]:     loss = self.compute_language_model_loss(labels, logits)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/models/common/language_module/language_module.py", line 157, in compute_language_model_loss
_4k_tokens/0 [default1]:[rank1]:     loss = fused_vocab_parallel_cross_entropy(logits, labels, self.model_comm_pgs.tp)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/fusions/fused_cross_entropy.py", line 148, in fused_vocab_parallel_cross_entropy
_4k_tokens/0 [default1]:[rank1]:     return _VocabParallelCrossEntropy.apply(vocab_parallel_logits, target, tp_group)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/autograd/function.py", line 576, in apply
_4k_tokens/0 [default1]:[rank1]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/fusions/fused_cross_entropy.py", line 104, in forward
_4k_tokens/0 [default1]:[rank1]:     calculate_predicted_logits(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 736, in compile_wrapper
_4k_tokens/0 [default1]:[rank1]:     return fn(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/fusions/fused_cross_entropy.py", line 25, in calculate_predicted_logits
_4k_tokens/0 [default1]:[rank1]:     @jit_fuser
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
_4k_tokens/0 [default1]:[rank1]:     return fn(*args, **kwargs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1241, in forward
_4k_tokens/0 [default1]:[rank1]:     return compiled_fn(full_args)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 384, in runtime_wrapper
_4k_tokens/0 [default1]:[rank1]:     all_outs = call_func_at_runtime_with_args(
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
_4k_tokens/0 [default1]:[rank1]:     out = normalize_as_list(f(args))
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 750, in inner_fn
_4k_tokens/0 [default1]:[rank1]:     outs = compiled_fn(args)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 556, in wrapper
_4k_tokens/0 [default1]:[rank1]:     return compiled_fn(runtime_args)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 584, in __call__
_4k_tokens/0 [default1]:[rank1]:     return self.current_callable(inputs)
_4k_tokens/0 [default1]:[rank1]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2716, in run
_4k_tokens/0 [default1]:[rank1]:     out = model(new_inputs)
_4k_tokens/0 [default1]:[rank1]:   File "/tmp/torchinductor_mihir.mahajan/tj/ctjz7gizv2gyy5qgzm66gfxvrkoizsyyos7w6jy4ztyvflbtyayt.py", line 235, in call
_4k_tokens/0 [default1]:[rank1]:     buf9 = empty_strided_cuda((4096, 16, 75968), (1215488, 75968, 1), torch.float32)
_4k_tokens/0 [default1]:[rank1]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 18.55 GiB. GPU 1 has a total capacity of 79.19 GiB of which 12.51 GiB is free. Including non-PyTorch memory, this process has 66.67 GiB memory in use. Of the allocated memory 59.22 GiB is allocated by PyTorch, and 4.00 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
_4k_tokens/0 [default0]:[rank0]: Traceback (most recent call last):
_4k_tokens/0 [default0]:[rank0]:   File "/home/mihir.mahajan/.local/share/uv/python/cpython-3.10.12-linux-x86_64-gnu/lib/python3.10/runpy.py", line 196, in _run_module_as_main
_4k_tokens/0 [default0]:[rank0]:     return _run_code(code, main_globals, None,
_4k_tokens/0 [default0]:[rank0]:   File "/home/mihir.mahajan/.local/share/uv/python/cpython-3.10.12-linux-x86_64-gnu/lib/python3.10/runpy.py", line 86, in _run_code
_4k_tokens/0 [default0]:[rank0]:     exec(code, run_globals)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo_run/core/runners/fdl_runner.py", line 72, in <module>
_4k_tokens/0 [default0]:[rank0]:     fdl_runner_app()
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/main.py", line 327, in __call__
_4k_tokens/0 [default0]:[rank0]:     raise e
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/main.py", line 310, in __call__
_4k_tokens/0 [default0]:[rank0]:     return get_command(self)(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/click/core.py", line 1485, in __call__
_4k_tokens/0 [default0]:[rank0]:     return self.main(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/core.py", line 719, in main
_4k_tokens/0 [default0]:[rank0]:     return _main(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/core.py", line 192, in _main
_4k_tokens/0 [default0]:[rank0]:     rv = self.invoke(ctx)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/click/core.py", line 1269, in invoke
_4k_tokens/0 [default0]:[rank0]:     return ctx.invoke(self.callback, **ctx.params)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/click/core.py", line 824, in invoke
_4k_tokens/0 [default0]:[rank0]:     return callback(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/typer/main.py", line 691, in wrapper
_4k_tokens/0 [default0]:[rank0]:     return callback(**use_params)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo_run/core/runners/fdl_runner.py", line 68, in fdl_direct_run
_4k_tokens/0 [default0]:[rank0]:     fdl_fn()
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/api.py", line 237, in finetune
_4k_tokens/0 [default0]:[rank0]:     return train(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/api.py", line 136, in train
_4k_tokens/0 [default0]:[rank0]:     trainer.fit(model, data)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 538, in fit
_4k_tokens/0 [default0]:[rank0]:     call._call_and_handle_interrupt(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 46, in _call_and_handle_interrupt
_4k_tokens/0 [default0]:[rank0]:     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
_4k_tokens/0 [default0]:[rank0]:     return function(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 574, in _fit_impl
_4k_tokens/0 [default0]:[rank0]:     self._run(model, ckpt_path=ckpt_path)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 981, in _run
_4k_tokens/0 [default0]:[rank0]:     results = self._run_stage()
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1023, in _run_stage
_4k_tokens/0 [default0]:[rank0]:     self._run_sanity_check()
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1052, in _run_sanity_check
_4k_tokens/0 [default0]:[rank0]:     val_loop.run()
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 178, in _decorator
_4k_tokens/0 [default0]:[rank0]:     return loop_run(self, *args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 135, in run
_4k_tokens/0 [default0]:[rank0]:     self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 396, in _evaluation_step
_4k_tokens/0 [default0]:[rank0]:     output = call._call_strategy_hook(trainer, hook_name, *step_args)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 319, in _call_strategy_hook
_4k_tokens/0 [default0]:[rank0]:     output = fn(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/pytorch/strategies/megatron_strategy.py", line 829, in validation_step
_4k_tokens/0 [default0]:[rank0]:     out = self.model.validation_step(dataloader_iter, *args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 371, in validation_step
_4k_tokens/0 [default0]:[rank0]:     return self._step(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 458, in _step
_4k_tokens/0 [default0]:[rank0]:     return self.forward(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 308, in forward
_4k_tokens/0 [default0]:[rank0]:     microbatch_outputs = step()
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 1248, in __call__
_4k_tokens/0 [default0]:[rank0]:     return self.forward_backward_func(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/pipeline_parallel/schedules.py", line 600, in forward_backward_no_pipelining
_4k_tokens/0 [default0]:[rank0]:     output_tensor, num_tokens = forward_step(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/pipeline_parallel/schedules.py", line 400, in forward_step
_4k_tokens/0 [default0]:[rank0]:     output_tensor, loss_func = forward_step_func(data_iterator, model)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 523, in wrapped_forward_step_func
_4k_tokens/0 [default0]:[rank0]:     output_tensor = _forward_step(model, batch)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/lightning/megatron_parallel.py", line 858, in wrapped
_4k_tokens/0 [default0]:[rank0]:     return attr(*args)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py", line 693, in validation_step
_4k_tokens/0 [default0]:[rank0]:     return self.forward_step(batch)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py", line 666, in forward_step
_4k_tokens/0 [default0]:[rank0]:     return self.config.forward_step_fn(self, batch)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py", line 143, in gpt_forward_step
_4k_tokens/0 [default0]:[rank0]:     return model(**forward_args)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
_4k_tokens/0 [default0]:[rank0]:     return self._call_impl(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
_4k_tokens/0 [default0]:[rank0]:     return forward_call(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py", line 634, in forward
_4k_tokens/0 [default0]:[rank0]:     output_tensor = self.module(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
_4k_tokens/0 [default0]:[rank0]:     return self._call_impl(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
_4k_tokens/0 [default0]:[rank0]:     return forward_call(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/distributed/data_parallel_base.py", line 22, in forward
_4k_tokens/0 [default0]:[rank0]:     return self.module(*inputs, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
_4k_tokens/0 [default0]:[rank0]:     return self._call_impl(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
_4k_tokens/0 [default0]:[rank0]:     return forward_call(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/transformer/module.py", line 237, in forward
_4k_tokens/0 [default0]:[rank0]:     outputs = self.module(*inputs, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
_4k_tokens/0 [default0]:[rank0]:     return self._call_impl(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
_4k_tokens/0 [default0]:[rank0]:     return forward_call(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/models/gpt/gpt_model.py", line 395, in forward
_4k_tokens/0 [default0]:[rank0]:     return self._postprocess(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/models/gpt/gpt_model.py", line 562, in _postprocess
_4k_tokens/0 [default0]:[rank0]:     loss = self.compute_language_model_loss(labels, logits)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/models/common/language_module/language_module.py", line 157, in compute_language_model_loss
_4k_tokens/0 [default0]:[rank0]:     loss = fused_vocab_parallel_cross_entropy(logits, labels, self.model_comm_pgs.tp)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/fusions/fused_cross_entropy.py", line 148, in fused_vocab_parallel_cross_entropy
_4k_tokens/0 [default0]:[rank0]:     return _VocabParallelCrossEntropy.apply(vocab_parallel_logits, target, tp_group)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/autograd/function.py", line 576, in apply
_4k_tokens/0 [default0]:[rank0]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/fusions/fused_cross_entropy.py", line 104, in forward
_4k_tokens/0 [default0]:[rank0]:     calculate_predicted_logits(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 736, in compile_wrapper
_4k_tokens/0 [default0]:[rank0]:     return fn(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/megatron/core/fusions/fused_cross_entropy.py", line 25, in calculate_predicted_logits
_4k_tokens/0 [default0]:[rank0]:     @jit_fuser
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
_4k_tokens/0 [default0]:[rank0]:     return fn(*args, **kwargs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1241, in forward
_4k_tokens/0 [default0]:[rank0]:     return compiled_fn(full_args)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 384, in runtime_wrapper
_4k_tokens/0 [default0]:[rank0]:     all_outs = call_func_at_runtime_with_args(
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
_4k_tokens/0 [default0]:[rank0]:     out = normalize_as_list(f(args))
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 750, in inner_fn
_4k_tokens/0 [default0]:[rank0]:     outs = compiled_fn(args)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 556, in wrapper
_4k_tokens/0 [default0]:[rank0]:     return compiled_fn(runtime_args)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 584, in __call__
_4k_tokens/0 [default0]:[rank0]:     return self.current_callable(inputs)
_4k_tokens/0 [default0]:[rank0]:   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2716, in run
_4k_tokens/0 [default0]:[rank0]:     out = model(new_inputs)
_4k_tokens/0 [default0]:[rank0]:   File "/tmp/torchinductor_mihir.mahajan/6d/c6dg6vqpqpl2mz7tet2vtgfywqerx2ffg4d7r7zrunwwlnravrjf.py", line 234, in call
_4k_tokens/0 [default0]:[rank0]:     buf9 = empty_strided_cuda((4096, 16, 75968), (1215488, 75968, 1), torch.float32)
_4k_tokens/0 [default0]:[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 18.55 GiB. GPU 0 has a total capacity of 79.19 GiB of which 12.38 GiB is free. Including non-PyTorch memory, this process has 66.80 GiB memory in use. Of the allocated memory 59.22 GiB is allocated by PyTorch, and 4.13 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
_4k_tokens/0 W1123 19:37:13.059000 4058528 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 4058544 closing signal SIGTERM
_4k_tokens/0 E1123 19:37:13.361000 4058528 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 1 (pid: 4058545) of binary: /fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/bin/python3
_4k_tokens/0 I1123 19:37:13.367000 4058528 torch/distributed/elastic/multiprocessing/errors/__init__.py:370] ('local_rank %s FAILED with no error file. Decorate your entrypoint fn with @record for traceback info. See: https://pytorch.org/docs/stable/elastic/errors.html', 1)
_4k_tokens/0 Traceback (most recent call last):
_4k_tokens/0   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/bin/torchrun", line 10, in <module>
_4k_tokens/0     sys.exit(main())
_4k_tokens/0   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
_4k_tokens/0     return f(*args, **kwargs)
_4k_tokens/0   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
_4k_tokens/0     run(args)
_4k_tokens/0   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
_4k_tokens/0     elastic_launch(
_4k_tokens/0   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 143, in __call__
_4k_tokens/0     return launch_agent(self._config, self._entrypoint, list(args))
_4k_tokens/0   File "/fast/home/mihir.mahajan/Projects/crowd-pilot/.uv_venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 277, in launch_agent
_4k_tokens/0     raise ChildFailedError(
_4k_tokens/0 torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
_4k_tokens/0 ============================================================
_4k_tokens/0 nemo_run.core.runners.fdl_runner FAILED
_4k_tokens/0 ------------------------------------------------------------
_4k_tokens/0 Failures:
_4k_tokens/0   <NO_OTHER_FAILURES>
_4k_tokens/0 ------------------------------------------------------------
_4k_tokens/0 Root Cause (first observed failure):
_4k_tokens/0 [0]:
_4k_tokens/0   time      : 2025-11-23_19:37:13
_4k_tokens/0   host      : hai003.haicore.berlin
_4k_tokens/0   rank      : 1 (local_rank: 1)
_4k_tokens/0   exitcode  : 1 (pid: 4058545)
_4k_tokens/0   error_file: <N/A>
_4k_tokens/0   traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
_4k_tokens/0 ============================================================
[19:37:15] INFO     Job                                          launcher.py:160
                    qwen3_30b_a3b_lora_finetune_crowd_code_datas                
                    et_4k_tokens-d7d42vwtpqpjwd finished: FAILED                
                                                                                
# The experiment was run with the following tasks: ['qwen3_30b_a3b_lora_finetune
# You can inspect and reconstruct this experiment at a later point in time using
experiment = run.Experiment.from_id("qwen3_30b_a3b_lora_finetune_crowd_code_data
experiment.status() # Gets the overall status                                   
experiment.logs("qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens") # Ge
experiment.cancel("qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens") # 
                                                                                
                                                                                
# You can inspect this experiment at a later point in time using the CLI as well
nemo experiment status qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_
nemo experiment logs qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_17
nemo experiment cancel qwen3_30b_a3b_lora_finetune_crowd_code_dataset_4k_tokens_