[content] /p/project1/envcomp/mihir/logs/eval/eval_miles_run/eval_miles_run_13057326.log (log)

Loading safetensors checkpoint shards:  69% Completed | 11/16 [00:16<00:07,  1.49s/it]

Loading safetensors checkpoint shards:  75% Completed | 12/16 [00:17<00:05,  1.45s/it]

Loading safetensors checkpoint shards:  81% Completed | 13/16 [00:19<00:04,  1.46s/it]

Loading safetensors checkpoint shards:  88% Completed | 14/16 [00:20<00:02,  1.47s/it]

Loading safetensors checkpoint shards:  94% Completed | 15/16 [00:22<00:01,  1.47s/it]

Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:22<00:00,  1.12s/it]

Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:22<00:00,  1.41s/it]

[2026-01-04 17:08:51 TP0] Load weight end. type=Qwen3MoeForCausalLM, dtype=torch.bfloat16, avail mem=23.75 GB, mem usage=14.31 GB.
[2026-01-04 17:08:51 TP2] Load weight end. type=Qwen3MoeForCausalLM, dtype=torch.bfloat16, avail mem=23.75 GB, mem usage=14.31 GB.
[2026-01-04 17:08:51 TP1] Load weight end. type=Qwen3MoeForCausalLM, dtype=torch.bfloat16, avail mem=23.68 GB, mem usage=14.31 GB.
[2026-01-04 17:08:51 TP3] Load weight end. type=Qwen3MoeForCausalLM, dtype=torch.bfloat16, avail mem=23.68 GB, mem usage=14.31 GB.
[2026-01-04 17:08:51 TP0] Using KV cache dtype: torch.bfloat16
[2026-01-04 17:08:51 TP1] KV Cache is allocated. #tokens: 619690, K size: 7.09 GB, V size: 7.09 GB
[2026-01-04 17:08:51 TP1] Memory pool end. avail mem=8.80 GB
[2026-01-04 17:08:51 TP2] KV Cache is allocated. #tokens: 619690, K size: 7.09 GB, V size: 7.09 GB
[2026-01-04 17:08:51 TP2] Memory pool end. avail mem=8.88 GB
[2026-01-04 17:08:51 TP0] KV Cache is allocated. #tokens: 619690, K size: 7.09 GB, V size: 7.09 GB
[2026-01-04 17:08:51 TP3] KV Cache is allocated. #tokens: 619690, K size: 7.09 GB, V size: 7.09 GB
[2026-01-04 17:08:51 TP0] Memory pool end. avail mem=8.88 GB
[2026-01-04 17:08:51 TP3] Memory pool end. avail mem=8.80 GB
[2026-01-04 17:08:52 TP2] Capture cuda graph begin. This can take up to several minutes. avail mem=8.37 GB
[2026-01-04 17:08:52 TP1] Capture cuda graph begin. This can take up to several minutes. avail mem=8.29 GB
[2026-01-04 17:08:52 TP3] Capture cuda graph begin. This can take up to several minutes. avail mem=8.29 GB
[2026-01-04 17:08:52 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=8.37 GB
[2026-01-04 17:08:52 TP0] Capture cuda graph bs [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160]

  0%|          | 0/24 [00:00<?, ?it/s]
Capturing batches (bs=160 avail_mem=8.25 GB):   0%|          | 0/24 [00:00<?, ?it/s][2026-01-04 17:08:54 TP2] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=128,N=192,device_name=NVIDIA_A100-SXM4-40GB.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2026-01-04 17:08:54 TP2] Using MoE kernel config with down_moe=False. Performance might be sub-optimal! Config file not found at /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=128,N=192,device_name=NVIDIA_A100-SXM4-40GB_down.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2026-01-04 17:08:54 TP0] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=128,N=192,device_name=NVIDIA_A100-SXM4-40GB.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2026-01-04 17:08:54 TP0] Using MoE kernel config with down_moe=False. Performance might be sub-optimal! Config file not found at /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=128,N=192,device_name=NVIDIA_A100-SXM4-40GB_down.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2026-01-04 17:08:54 TP1] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=128,N=192,device_name=NVIDIA_A100-SXM4-40GB.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2026-01-04 17:08:54 TP1] Using MoE kernel config with down_moe=False. Performance might be sub-optimal! Config file not found at /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=128,N=192,device_name=NVIDIA_A100-SXM4-40GB_down.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2026-01-04 17:08:55 TP3] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=128,N=192,device_name=NVIDIA_A100-SXM4-40GB.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2026-01-04 17:08:55 TP3] Using MoE kernel config with down_moe=False. Performance might be sub-optimal! Config file not found at /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=128,N=192,device_name=NVIDIA_A100-SXM4-40GB_down.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton

Capturing batches (bs=160 avail_mem=8.25 GB):   4%|▍         | 1/24 [00:02<01:02,  2.72s/it]
Capturing batches (bs=152 avail_mem=8.09 GB):   4%|▍         | 1/24 [00:02<01:02,  2.72s/it]
Capturing batches (bs=152 avail_mem=8.09 GB):   8%|▊         | 2/24 [00:02<00:28,  1.28s/it]
Capturing batches (bs=144 avail_mem=8.06 GB):   8%|▊         | 2/24 [00:02<00:28,  1.28s/it]
Capturing batches (bs=144 avail_mem=8.06 GB):  12%|█▎        | 3/24 [00:03<00:16,  1.30it/s]
Capturing batches (bs=136 avail_mem=8.06 GB):  12%|█▎        | 3/24 [00:03<00:16,  1.30it/s]
Capturing batches (bs=136 avail_mem=8.06 GB):  17%|█▋        | 4/24 [00:03<00:10,  1.89it/s]
Capturing batches (bs=128 avail_mem=8.03 GB):  17%|█▋        | 4/24 [00:03<00:10,  1.89it/s]
Capturing batches (bs=128 avail_mem=8.03 GB):  21%|██        | 5/24 [00:03<00:07,  2.51it/s]
Capturing batches (bs=120 avail_mem=8.02 GB):  21%|██        | 5/24 [00:03<00:07,  2.51it/s]
Capturing batches (bs=120 avail_mem=8.02 GB):  25%|██▌       | 6/24 [00:03<00:05,  3.14it/s]
Capturing batches (bs=112 avail_mem=7.99 GB):  25%|██▌       | 6/24 [00:03<00:05,  3.14it/s]
Capturing batches (bs=112 avail_mem=7.99 GB):  29%|██▉       | 7/24 [00:03<00:04,  3.76it/s]
Capturing batches (bs=104 avail_mem=7.99 GB):  29%|██▉       | 7/24 [00:03<00:04,  3.76it/s]
Capturing batches (bs=104 avail_mem=7.99 GB):  33%|███▎      | 8/24 [00:03<00:03,  4.31it/s]
Capturing batches (bs=96 avail_mem=7.96 GB):  33%|███▎      | 8/24 [00:03<00:03,  4.31it/s] 
Capturing batches (bs=96 avail_mem=7.96 GB):  38%|███▊      | 9/24 [00:04<00:03,  4.74it/s]
Capturing batches (bs=88 avail_mem=7.95 GB):  38%|███▊      | 9/24 [00:04<00:03,  4.74it/s]
Capturing batches (bs=88 avail_mem=7.95 GB):  42%|████▏     | 10/24 [00:04<00:02,  5.11it/s]
Capturing batches (bs=80 avail_mem=7.92 GB):  42%|████▏     | 10/24 [00:04<00:02,  5.11it/s]
Capturing batches (bs=80 avail_mem=7.92 GB):  46%|████▌     | 11/24 [00:04<00:02,  5.42it/s]
Capturing batches (bs=72 avail_mem=7.91 GB):  46%|████▌     | 11/24 [00:04<00:02,  5.42it/s]
Capturing batches (bs=72 avail_mem=7.91 GB):  50%|█████     | 12/24 [00:04<00:02,  5.72it/s]
Capturing batches (bs=64 avail_mem=7.89 GB):  50%|█████     | 12/24 [00:04<00:02,  5.72it/s]
Capturing batches (bs=64 avail_mem=7.89 GB):  54%|█████▍    | 13/24 [00:04<00:01,  5.79it/s]
Capturing batches (bs=56 avail_mem=7.88 GB):  54%|█████▍    | 13/24 [00:04<00:01,  5.79it/s]
Capturing batches (bs=56 avail_mem=7.88 GB):  58%|█████▊    | 14/24 [00:04<00:01,  5.93it/s]
Capturing batches (bs=48 avail_mem=7.85 GB):  58%|█████▊    | 14/24 [00:04<00:01,  5.93it/s]
Capturing batches (bs=48 avail_mem=7.85 GB):  62%|██████▎   | 15/24 [00:05<00:01,  6.00it/s]
Capturing batches (bs=40 avail_mem=7.84 GB):  62%|██████▎   | 15/24 [00:05<00:01,  6.00it/s]
Capturing batches (bs=40 avail_mem=7.84 GB):  67%|██████▋   | 16/24 [00:05<00:01,  6.10it/s]
Capturing batches (bs=32 avail_mem=7.81 GB):  67%|██████▋   | 16/24 [00:05<00:01,  6.10it/s]
Capturing batches (bs=32 avail_mem=7.81 GB):  71%|███████   | 17/24 [00:05<00:01,  5.27it/s]
Capturing batches (bs=24 avail_mem=7.80 GB):  71%|███████   | 17/24 [00:05<00:01,  5.27it/s]
Capturing batches (bs=24 avail_mem=7.80 GB):  75%|███████▌  | 18/24 [00:05<00:01,  4.54it/s]
Capturing batches (bs=16 avail_mem=7.78 GB):  75%|███████▌  | 18/24 [00:05<00:01,  4.54it/s]
Capturing batches (bs=16 avail_mem=7.78 GB):  79%|███████▉  | 19/24 [00:06<00:01,  4.52it/s]
Capturing batches (bs=12 avail_mem=7.77 GB):  79%|███████▉  | 19/24 [00:06<00:01,  4.52it/s]
Capturing batches (bs=12 avail_mem=7.77 GB):  83%|████████▎ | 20/24 [00:06<00:00,  4.79it/s]
Capturing batches (bs=8 avail_mem=7.74 GB):  83%|████████▎ | 20/24 [00:06<00:00,  4.79it/s] 
Capturing batches (bs=8 avail_mem=7.74 GB):  88%|████████▊ | 21/24 [00:06<00:00,  5.02it/s]
Capturing batches (bs=4 avail_mem=7.73 GB):  88%|████████▊ | 21/24 [00:06<00:00,  5.02it/s]
Capturing batches (bs=4 avail_mem=7.73 GB):  92%|█████████▏| 22/24 [00:06<00:00,  5.24it/s]
Capturing batches (bs=2 avail_mem=7.71 GB):  92%|█████████▏| 22/24 [00:06<00:00,  5.24it/s]
Capturing batches (bs=2 avail_mem=7.71 GB):  96%|█████████▌| 23/24 [00:06<00:00,  5.37it/s]
Capturing batches (bs=1 avail_mem=7.70 GB):  96%|█████████▌| 23/24 [00:06<00:00,  5.37it/s]
Capturing batches (bs=1 avail_mem=7.70 GB): 100%|██████████| 24/24 [00:07<00:00,  4.55it/s]
Capturing batches (bs=1 avail_mem=7.70 GB): 100%|██████████| 24/24 [00:07<00:00,  3.42it/s]
[2026-01-04 17:08:59 TP0] Registering 2328 cuda graph addresses
[2026-01-04 17:09:00 TP1] Capture cuda graph end. Time elapsed: 7.88 s. mem usage=0.70 GB. avail mem=7.60 GB.
[2026-01-04 17:09:00 TP2] Capture cuda graph end. Time elapsed: 7.92 s. mem usage=0.70 GB. avail mem=7.67 GB.
[2026-01-04 17:09:00 TP0] Capture cuda graph end. Time elapsed: 7.95 s. mem usage=0.70 GB. avail mem=7.67 GB.
[2026-01-04 17:09:00 TP3] Capture cuda graph end. Time elapsed: 8.07 s. mem usage=0.70 GB. avail mem=7.60 GB.
[2026-01-04 17:09:00 TP0] max_total_num_tokens=619690, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=4096, context_len=40960, available_gpu_mem=7.67 GB
[2026-01-04 17:09:01] INFO:     Started server process [617098]
[2026-01-04 17:09:01] INFO:     Waiting for application startup.
[2026-01-04 17:09:01] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-01-04 17:09:01] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-01-04 17:09:01] INFO:     Application startup complete.
[2026-01-04 17:09:01] INFO:     Uvicorn running on http://localhost:30000 (Press CTRL+C to quit)
[2026-01-04 17:09:02] INFO:     127.0.0.1:56218 - "GET /model_info HTTP/1.1" 200 OK
[2026-01-04 17:09:02 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2026-01-04 17:09:02] INFO:     127.0.0.1:56232 - "POST /generate HTTP/1.1" 200 OK
[2026-01-04 17:09:02] The server is fired up and ready to roll!
[2026-01-04 17:09:10] INFO:     127.0.0.1:56238 - "GET /v1/models HTTP/1.1" 200 OK
wandb: WARNING `resume` will be ignored since W&B syncing is set to `offline`. Starting a new run with run id 13032802.
wandb: Tracking run with wandb version 0.23.1
wandb: W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Run data is saved locally in /p/project1/envcomp/mahajan1/miles/tab-model-eval/eval_logs/13032802/wandb/offline-run-20260104_170910-13032802
wandb: Detected [openai] in use.
wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
Using server at http://localhost:30000/v1
Launching sglang server:
  /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/bin/python -m sglang.launch_server --model-path /p/scratch/envcomp/mihir/huggingface/Qwen3-30B-A3B --host localhost --port 30000 --context-length 40960 --mem-fraction-static 0.75 --tp-size 4
Waiting for server at http://localhost:30000 ...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server is up.

Filtered dataset:
  Valid test cases: 5
  Skipped (too long): 0

Running 5 test cases with concurrency=64 ...

  0%|          | 0/5 [00:00<?, ?it/s][2026-01-04 17:09:11 TP0] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 2, 
[2026-01-04 17:09:11 TP0] Prefill batch, #new-seq: 2, #new-token: 4096, #cached-token: 219, token usage: 0.01, #running-req: 0, #queue-req: 3, 
[2026-01-04 17:09:12 TP0] Prefill batch, #new-seq: 4, #new-token: 4096, #cached-token: 5537, token usage: 0.01, #running-req: 1, #queue-req: 0, 
[2026-01-04 17:09:12 TP0] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.02, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:09:12 TP0] Prefill batch, #new-seq: 1, #new-token: 2297, #cached-token: 0, token usage: 0.02, #running-req: 4, #queue-req: 0,