[content] /p/project1/envcomp/mihir/logs/eval/eval_miles_run/eval_miles_run_13057326.log (log)
[2026-01-04 17:11:18] INFO:     127.0.0.1:60330 - "GET /v1/models HTTP/1.1" 200 OK
wandb: WARNING `resume` will be ignored since W&B syncing is set to `offline`. Starting a new run with run id 13032802.
wandb: Tracking run with wandb version 0.23.1
wandb: W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Run data is saved locally in /p/project1/envcomp/mahajan1/miles/tab-model-eval/eval_logs/13032802/wandb/offline-run-20260104_171118-13032802
wandb: Detected [openai] in use.
wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
Using server at http://localhost:30000/v1
Launching sglang server:
  /p/project1/envcomp/mahajan1/miles/tab-model-eval/.venv/bin/python -m sglang.launch_server --model-path /p/scratch/envcomp/mihir/huggingface/Qwen3-30B-A3B --host localhost --port 30000 --context-length 40960 --mem-fraction-static 0.75 --tp-size 4
Waiting for server at http://localhost:30000 ...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server not ready yet (All connection attempts failed); retrying...
Server is up.

Filtered dataset:
  Valid test cases: 5
  Skipped (too long): 0

Running 5 test cases with concurrency=64 ...

  0%|          | 0/5 [00:00<?, ?it/s][2026-01-04 17:11:19 TP0] Prefill batch, #new-seq: 1, #new-token: 1356, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2026-01-04 17:11:19 TP0] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.00, #running-req: 1, #queue-req: 3, 
[2026-01-04 17:11:19 TP0] Prefill batch, #new-seq: 3, #new-token: 4096, #cached-token: 5319, token usage: 0.01, #running-req: 1, #queue-req: 1, 
[2026-01-04 17:11:19 TP0] Prefill batch, #new-seq: 2, #new-token: 4096, #cached-token: 218, token usage: 0.02, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:19 TP0] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.02, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:19 TP0] Prefill batch, #new-seq: 1, #new-token: 1152, #cached-token: 0, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:20 TP0] Decode batch, #running-req: 5, #token: 16853, token usage: 0.03, cuda graph: True, gen throughput (token/s): 3.41, #queue-req: 0, 
[2026-01-04 17:11:20] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:20 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:20] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:20] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:20 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:20 TP0] Prefill batch, #new-seq: 1, #new-token: 98, #cached-token: 6134, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:20] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:20 TP0] Prefill batch, #new-seq: 1, #new-token: 68, #cached-token: 1287, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:21] INFO:     127.0.0.1:60394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:21 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:21 TP0] Decode batch, #running-req: 4, #token: 16799, token usage: 0.03, cuda graph: True, gen throughput (token/s): 279.00, #queue-req: 0, 
[2026-01-04 17:11:21] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:21 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:21] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:21] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:21 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 6231, token usage: 0.03, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:21 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:21] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:21 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1354, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:21 TP0] Decode batch, #running-req: 5, #token: 16824, token usage: 0.03, cuda graph: True, gen throughput (token/s): 330.22, #queue-req: 0, 
[2026-01-04 17:11:21] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:21 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:21] INFO:     127.0.0.1:60394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:21 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:21] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:21 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:22] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:22] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:22 TP0] Decode batch, #running-req: 5, #token: 16294, token usage: 0.03, cuda graph: True, gen throughput (token/s): 378.70, #queue-req: 0, 
[2026-01-04 17:11:22 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:22 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 6231, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:22] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:22 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1354, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:22] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:22 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:22] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:22 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:22 TP0] Decode batch, #running-req: 4, #token: 16839, token usage: 0.03, cuda graph: True, gen throughput (token/s): 303.79, #queue-req: 0, 
[2026-01-04 17:11:22] INFO:     127.0.0.1:60394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:22 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:23] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:23 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 6231, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:23] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:23 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:23] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:23 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:23 TP0] Decode batch, #running-req: 5, #token: 16813, token usage: 0.03, cuda graph: True, gen throughput (token/s): 328.72, #queue-req: 0, 
[2026-01-04 17:11:23] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:23 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1354, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:23] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:23 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:23] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:23 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:23 TP0] Decode batch, #running-req: 5, #token: 16844, token usage: 0.03, cuda graph: True, gen throughput (token/s): 364.32, #queue-req: 0, 
[2026-01-04 17:11:23] INFO:     127.0.0.1:60394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:23 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:24] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:24 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 6231, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:24] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:24 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:24] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:24 TP0] Decode batch, #running-req: 5, #token: 15811, token usage: 0.03, cuda graph: True, gen throughput (token/s): 380.90, #queue-req: 0, 
[2026-01-04 17:11:24] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:24 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:24 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1354, token usage: 0.03, #running-req: 5, #queue-req: 0, 
[2026-01-04 17:11:24] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:24 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 6222, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:24] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:24 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1566, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:25 TP0] Decode batch, #running-req: 5, #token: 16873, token usage: 0.03, cuda graph: True, gen throughput (token/s): 337.96, #queue-req: 0, 
[2026-01-04 17:11:25] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:25 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:25] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:25 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 6222, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:25] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:25 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1354, token usage: 0.03, #running-req: 4, #queue-req: 0, 
[2026-01-04 17:11:25] INFO:     127.0.0.1:60394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:25] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK

 20%|██        | 1/5 [00:06<00:25,  6.45s/it][2026-01-04 17:11:25 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.03, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:25 TP0] Decode batch, #running-req: 4, #token: 16424, token usage: 0.03, cuda graph: True, gen throughput (token/s): 318.38, #queue-req: 0, 
[2026-01-04 17:11:25] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:25 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 7090, token usage: 0.03, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:25] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:25 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 6231, token usage: 0.03, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:25 TP0] Decode batch, #running-req: 4, #token: 16494, token usage: 0.03, cuda graph: True, gen throughput (token/s): 367.46, #queue-req: 0, 
[2026-01-04 17:11:26] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1355, token usage: 0.03, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:26] INFO:     127.0.0.1:60394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.03, #running-req: 3, #queue-req: 0, 
[2026-01-04 17:11:26] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK

 40%|████      | 2/5 [00:07<00:09,  3.15s/it][2026-01-04 17:11:26 TP0] Decode batch, #running-req: 3, #token: 15448, token usage: 0.02, cuda graph: True, gen throughput (token/s): 325.42, #queue-req: 0, 
[2026-01-04 17:11:26] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 6231, token usage: 0.02, #running-req: 2, #queue-req: 0, 
[2026-01-04 17:11:26] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 82, #cached-token: 1289, token usage: 0.02, #running-req: 2, #queue-req: 0, 
[2026-01-04 17:11:26 TP0] Decode batch, #running-req: 3, #token: 15462, token usage: 0.02, cuda graph: True, gen throughput (token/s): 279.84, #queue-req: 0, 
[2026-01-04 17:11:26] INFO:     127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.02, #running-req: 2, #queue-req: 0, 
[2026-01-04 17:11:27] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:27 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 1354, token usage: 0.02, #running-req: 2, #queue-req: 0, 
[2026-01-04 17:11:27 TP0] Decode batch, #running-req: 3, #token: 15435, token usage: 0.02, cuda graph: True, gen throughput (token/s): 274.97, #queue-req: 0, 
[2026-01-04 17:11:27] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK

 60%|██████    | 3/5 [00:08<00:04,  2.17s/it][2026-01-04 17:11:27] INFO:     127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:27 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.02, #running-req: 1, #queue-req: 0, 
[2026-01-04 17:11:27 TP0] Decode batch, #running-req: 2, #token: 9386, token usage: 0.02, cuda graph: True, gen throughput (token/s): 257.39, #queue-req: 0, 
[2026-01-04 17:11:27] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK

 80%|████████  | 4/5 [00:08<00:01,  1.48s/it][2026-01-04 17:11:27 TP0] Decode batch, #running-req: 1, #token: 8240, token usage: 0.01, cuda graph: True, gen throughput (token/s): 277.05, #queue-req: 0, 
[2026-01-04 17:11:27] INFO:     127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-04 17:11:27 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 8191, token usage: 0.01, #running-req: 0, #queue-req: 0, 
[2026-01-04 17:11:28 TP0] Decode batch, #running-req: 1, #token: 8204, token usage: 0.01, cuda graph: True, gen throughput (token/s): 140.19, #queue-req: 0, 
[2026-01-04 17:11:28] INFO:     127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK

100%|██████████| 5/5 [00:09<00:00,  1.15s/it]
100%|██████████| 5/5 [00:09<00:00,  1.85s/it]
wandb: 
wandb: Run history:
wandb:                                eval_step ▁
wandb:        unit-test-v1/num_samples_per_task ▁
wandb:  unit-test-v1/total_exact_match_avg_at_n ▁
wandb: unit-test-v1/total_exact_match_pass_at_n ▁
wandb:        unit-test-v1/total_judge_avg_at_n ▁
wandb:       unit-test-v1/total_judge_pass_at_n ▁
wandb:            unit-test-v1/total_test_cases ▁
wandb: 
wandb: Run summary:
wandb:                                eval_step 3000
wandb:        unit-test-v1/num_samples_per_task 10
wandb:  unit-test-v1/total_exact_match_avg_at_n 0.4
wandb: unit-test-v1/total_exact_match_pass_at_n 2
wandb:        unit-test-v1/total_judge_avg_at_n 0.4
wandb:       unit-test-v1/total_judge_pass_at_n 2
wandb:            unit-test-v1/total_test_cases 5
wandb: 
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /p/project1/envcomp/mahajan1/miles/tab-model-eval/eval_logs/13032802/wandb/offline-run-20260104_171118-13032802
wandb: Find logs at: ./eval_logs/13032802/wandb/offline-run-20260104_171118-13032802/logs
[2026-01-04 17:11:28] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2026-01-04 17:11:30] Gracefully exiting... Remaining number of requests 0. Remaining requests remaining_rids=[].

==================================================
--- Evaluation Complete ---
==================================================
Total Test Cases: 5
Total Judge Pass At N: 2
Total Judge Avg At N: 40.00%
Total Exact Match Pass At N: 2
Total Exact Match Avg At N: 40.00%
Evaluations output file: data/eval/output/13032802/qwen1.7B_lora_bs_8_13032802_3000_evaluations.jsonl
Shutting down sglang server ...
Done
LoRA adapter dir: /p/scratch/envcomp/mihir/huggingface/shared_data/13032802/checkpoints/iter_0004000/, Step: 4000
generating..