Repository: ggml-org/llama.cpp
Pull request: benches : add eval results

Add 8xAIME25 eval for `gpt-oss-120b` high reasoning performed on DGX Spark

Final result: `93.75% +/- 0.24%`

Changed files:
filename: benches/dgx-spark/aime25_openai__gpt-oss-120b-high_temp1.0_20251109_094547.json
patch: @@ -0,0 +1,6 @@
+{
+  "chars": 2296.1916666666666,
+  "chars:std": 986.051306946325,
+  "score": 0.925,
+  "score:std": 0.26339134382131846
+}
\ No newline at end of file

filename: benches/dgx-spark/run-aime-120b-t8-x8-high.log
patch: @@ -0,0 +1,11 @@
+nohup: ignoring input
+Running with args Namespace(model='openai/gpt-oss-120b', reasoning_effort='high', sampler='chat_completions', base_url='http://localhost:8066/v1', eval='aime25', temperature=1.0, n_threads=8, debug=False, examples=None)
+
+Running the following evals: {'aime25': <gpt_oss.evals.aime_eval.AIME25Eval object at 0xe8638c0857c0>}
+Running evals for the following models: {'openai/gpt-oss-120b-high': <gpt_oss.evals.chat_completions_sampler.ChatCompletionsSampler object at 0xe8638c085340>}
+  0%|          | 0/240 [00:00<?, ?it/s]  0%|          | 1/240 [02:55<11:38:24, 175.33s/it]  1%|          | 2/240 [03:48<6:51:24, 103.72s/it]   1%|▏         | 3/240 [05:47<7:16:38, 110.54s/it]  2%|▏         | 4/240 [06:44<5:51:23, 89.34s/it]   2%|▏         | 5/240 [07:53<5:21:29, 82.08s/it]  2%|▎         | 6/240 [17:13<15:53:25, 244.47s/it]  3%|▎         | 7/240 [25:07<20:41:12, 319.62s/it]  3%|▎         | 8/240 [25:10<14:06:28, 218.91s/it]  4%|▍         | 9/240 [27:13<12:06:34, 188.72s/it]  4%|▍         | 10/240 [30:33<12:16:51, 192.22s/it]  5%|▍         | 11/240 [32:45<11:03:23, 173.82s/it]  5%|▌         | 12/240 [39:34<15:31:54, 245.24s/it]  5%|▌         | 13/240 [44:17<16:11:28, 256.78s/it]  6%|▌         | 14/240 [51:54<19:55:10, 317.30s/it]  6%|▋         | 15/240 [52:26<14:26:56, 231.18s/it]  7%|▋         | 16/240 [1:00:47<19:26:26, 312.44s/it]  7%|▋         | 17/240 [1:00:52<13:37:59, 220.09s/it]  8%|▊         | 18/240 [1:09:47<19:24:30, 314.73s/it]  8%|▊         | 19/240 [1:15:59<20:22:08, 331.80s/it]  8%|▊         | 20/240 [1:19:18<17:51:04, 292.11s/it]  9%|▉         | 21/240 [1:23:46<17:19:23, 284.77s/it]  9%|▉         | 22/240 [1:27:49<16:29:35, 272.36s/it] 10%|▉         | 23/240 [1:31:20<15:18:04, 253.85s/it] 10%|█         | 24/240 [1:33:42<13:12:51, 220.24s/it] 10%|█         | 25/240 [1:34:30<10:03:44, 168.49s/it] 11%|█         | 26/240 [1:35:12<7:45:29, 130.51s/it]  11%|█▏        | 27/240 [1:41:05<11:40:47, 197.40s/it] 12%|█▏        | 28/240 [1:41:52<8:58:11, 152.32s/it]  12%|█▏        | 29/240 [1:50:38<15:29:30, 264.31s/it] 12%|█▎        | 30/240 [1:58:51<19:25:22, 332.97s/it] 13%|█▎        | 31/240 [2:06:54<21:57:01, 378.09s/it] 13%|█▎        | 32/240 [2:11:15<19:49:00, 342.98s/it] 14%|█▍        | 33/240 [2:15:26<18:07:41, 315.27s/it] 14%|█▍        | 34/240 [2:28:06<25:40:44, 448.76s/it] 15%|█▍        | 35/240 [2:36:39<26:38:46, 467.93s/it] 15%|█▌        | 36/240 [2:50:14<32:24:40, 571.96s/it] 15%|█▌        | 37/240 [2:58:42<31:10:55, 552.98s/it] 16%|█▌        | 38/240 [3:02:52<25:55:32, 462.04s/it] 16%|█▋        | 39/240 [3:07:01<22:13:39, 398.11s/it] 17%|█▋        | 40/240 [3:08:19<16:47:08, 302.14s/it] 17%|█▋        | 41/240 [3:17:10<20:29:05, 370.58s/it] 18%|█▊        | 42/240 [3:17:43<14:49:16, 269.48s/it] 18%|█▊        | 43/240 [3:25:41<18:10:02, 331.99s/it] 18%|█▊        | 44/240 [3:37:25<24:08:45, 443.50s/it] 19%|█▉        | 45/240 [3:47:01<26:11:16, 483.47s/it] 19%|█▉        | 46/240 [3:48:23<19:33:53, 363.06s/it] 20%|█▉        | 47/240 [3:53:08<18:12:17, 339.57s/it] 20%|██        | 48/240 [4:10:36<29:26:13, 551.95s/it] 20%|██        | 49/240 [4:10:53<20:46:01, 391.42s/it] 21%|██        | 50/240 [4:17:30<20:45:27, 393.30s/it] 21%|██▏       | 51/240 [4:29:11<25:29:12, 485.46s/it] 22%|██▏       | 52/240 [4:34:22<22:36:56, 433.07s/it] 22%|██▏       | 53/240 [4:47:51<28:21:08, 545.82s/it] 22%|██▎       | 54/240 [4:52:15<23:50:38, 461.50s/it] 23%|██▎       | 55/240 [4:54:52<19:00:49, 370.00s/it] 23%|██▎       | 56/240 [4:55:15<13:35:13, 265.84s/it] 24%|██▍       | 57/240 [5:00:00<13:48:46, 271.73s/it] 24%|██▍       | 58/240 [5:00:48<10:20:53, 204.69s/it] 25%|██▍       | 59/240 [5:03:49<9:55:38, 197.45s/it]  25%|██▌       | 60/240 [5:06:15<9:06:35, 182.20s/it] 25%|██▌       | 61/240 [5:09:56<9:37:40, 193.64s/it] 26%|██▌       | 62/240 [5:13:47<10:07:58, 204.94s/it] 26%|██▋       | 63/240 [5:14:29<7:40:38, 156.15s/it]  27%|██▋       | 64/240 [5:17:05<7:37:56, 156.11s/it] 27%|██▋       | 65/240 [5:30:00<16:36:50, 341.78s/it] 28%|██▊       | 66/240 [5:39:18<19:39:03, 406.57s/it] 28%|██▊       | 67/240 [5:42:53<16:46:13, 348.98s/it] 28%|██▊       | 68/240 [5:48:27<16:27:58, 344.64s/it] 29%|██▉       | 69/240 [5:55:10<17:11:40, 361.99s/it] 29%|██▉       | 70/240 [6:00:11<16:14:13, 343.85s/it] 30%|██▉       | 71/240 [6:08:12<18:04:28, 385.02s/it] 30%|███       | 72/240 [6:21:30<23:44:30, 508.75s/it] 30%|███       | 73/240 [6:24:19<18:52:02, 406.72s/it] 31%|███       | 74/240 [6:38:51<25:11:28, 546.31s/it] 31%|███▏      | 75/240 [6:47:02<24:17:30, 530.00s/it] 32%|███▏      | 76/240 [6:48:41<18:15:08, 400.66s/it] 32%|███▏      | 77/240 [6:52:16<15:36:35, 344.76s/it] 32%|███▎      | 78/240 [7:08:27<23:58:08, 532.65s/it] 33%|███▎      | 79/240 [7:12:16<19:45:25, 441.78s/it] 33%|███▎      | 80/240 [7:18:40<18:51:06, 424.16s/it] 34%|███▍      | 81/240 [7:30:02<22:09:27, 501.68s/it] 34%|███▍      | 82/240 [7:33:07<17:50:48, 406.64s/it] 35%|███▍      | 83/240 [7:41:45<19:11:43, 440.15s/it] 35%|███▌      | 84/240 [7:44:50<15:44:45, 363.37s/it] 35%|███▌      | 85/240 [7:46:50<12:30:50, 290.65s/it] 36%|███▌      | 86/240 [7:51:41<12:25:57, 290.64s/it] 36%|███▋      | 87/240 [7:54:04<10:28:02, 246.29s/it] 37%|███▋      | 88/240 [8:02:06<13:22:58, 316.96s/it] 37%|███▋      | 89/240 [8:04:11<10:53:13, 259.56s/it] 38%|███▊      | 90/240 [8:07:54<10:21:03, 248.42s/it] 38%|███▊      | 91/240 [8:15:49<13:05:58, 316.50s/it] 38%|███▊      | 92/240 [8:16:13<9:24:07, 228.70s/it]  39%|███▉      | 93/240 [8:16:52<7:00:40, 171.70s/it] 39%|███▉      | 94/240 [8:39:28<21:22:17, 526.97s/it] 40%|███▉      | 95/240 [8:42:43<17:12:44, 427.34s/it] 40%|████      | 96/240 [8:45:54<14:16:08, 356.73s/it] 40%|████      | 97/240 [8:59:10<19:24:13, 488.49s/it] 41%|████      | 98/240 [8:59:51<13:58:02, 354.10s/it] 41%|████▏     | 99/240 [9:01:08<10:36:32, 270.87s/it] 42%|████▏     | 100/240 [9:06:27<11:05:41, 285.30s/it] 42%|████▏     | 101/240 [9:06:28<7:43:55, 200.25s/it]  42%|████▎     | 102/240 [9:07:39<6:11:00, 161.31s/it] 43%|████▎     | 103/240 [9:15:25<9:37:25, 252.89s/it] 43%|████▎     | 104/240 [9:30:57<17:14:58, 456.61s/it] 44%|████▍     | 105/240 [9:35:16<14:53:51, 397.27s/it] 44%|████▍     | 106/240 [9:43:12<15:39:45, 420.79s/it] 45%|████▍     | 107/240 [9:52:02<16:45:14, 453.49s/it] 45%|████▌     | 108/240 [10:05:21<20:25:45, 557.16s/it] 45%|████▌     | 109/240 [10:05:47<14:28:49, 397.93s/it] 46%|████▌     | 110/240 [10:19:14<18:48:09, 520.69s/it] 46%|████▋     | 111/240 [10:24:24<16:23:32, 457.46s/it] 47%|████▋     | 112/240 [10:26:40<12:50:01, 360.95s/it] 47%|████▋     | 113/240 [10:29:11<10:30:36, 297.92s/it] 48%|████▊     | 114/240 [10:32:22<9:18:10, 265.80s/it]  48%|████▊     | 115/240 [10:33:11<6:58:38, 200.95s/it] 48%|████▊     | 116/240 [10:38:50<8:20:55, 242.38s/it] 49%|████▉     | 117/240 [10:45:18<9:46:26, 286.07s/it] 49%|████▉     | 118/240 [10:50:44<10:05:33, 297.82s/it] 50%|████▉     | 119/240 [10:57:07<10:52:15, 323.43s/it] 50%|█████     | 120/240 [11:05:17<12:26:41, 373.35s/it] 50%|█████     | 121/240 [11:16:15<15:10:02, 458.85s/it] 51%|█████     | 122/240 [11:26:20<16:28:25, 502.59s/it] 51%|█████▏    | 123/240 [11:29:47<13:27:14, 413.97s/it] 52%|█████▏    | 124/240 [11:34:33<12:06:12, 375.63s/it] 52%|█████▏    | 125/240 [11:35:10<8:45:30, 274.18s/it]  52%|█████▎    | 126/240 [11:56:22<18:09:23, 573.36s/it] 53%|█████▎    | 127/240 [12:05:56<18:00:20, 573.63s/it] 53%|█████▎    | 128/240 [12:20:20<20:33:12, 660.64s/it] 54%|█████▍    | 129/240 [12:27:46<18:23:19, 596.39s/it] 54%|█████▍    | 130/240 [12:32:48<15:31:21, 508.01s/it] 55%|█████▍    | 131/240 [12:39:47<14:34:29, 481.37s/it] 55%|█████▌    | 132/240 [12:51:17<16:19:10, 543.99s/it] 55%|█████▌    | 133/240 [12:55:07<13:21:53, 449.66s/it] 56%|█████▌    | 134/240 [13:00:11<11:57:18, 406.02s/it] 56%|█████▋    | 135/240 [13:13:04<15:02:59, 516.00s/it] 57%|█████▋    | 136/240 [13:21:15<14:41:18, 508.44s/it] 57%|█████▋    | 137/240 [13:21:25<10:16:34, 359.17s/it] 57%|█████▊    | 138/240 [13:23:16<8:04:05, 284.76s/it]  58%|█████▊    | 139/240 [13:42:58<15:32:01, 553.68s/it] 58%|█████▊    | 140/240 [13:50:52<14:43:06, 529.87s/it] 59%|█████▉    | 141/240 [13:59:02<14:14:46, 518.04s/it] 59%|█████▉    | 142/240 [14:01:23<11:01:08, 404.78s/it] 60%|█████▉    | 143/240 [14:06:23<10:03:40, 373.41s/it] 60%|██████    | 144/240 [14:15:00<11:06:32, 416.59s/it] 60%|██████    | 145/240 [14:33:48<16:37:26, 629.96s/it] 61%|██████    | 146/240 [14:34:38<11:54:11, 455.87s/it] 61%|██████▏   | 147/240 [14:40:38<11:01:51, 427.01s/it] 62%|██████▏   | 148/240 [14:45:33<9:54:25, 387.67s/it]  62%|██████▏   | 149/240 [14:52:54<10:11:51, 403.42s/it] 62%|██████▎   | 150/240 [14:54:05<7:35:51, 303.90s/it]  63%|██████▎   | 151/240 [15:08:14<11:33:01, 467.21s/it] 63%|██████▎   | 152/240 [15:09:48<8:41:10, 355.35s/it]  64%|██████▍   | 153/240 [15:11:16<6:38:50, 275.06s/it] 64%|██████▍   | 154/240 [15:13:14<5:27:02, 228.16s/it] 65%|██████▍   | 155/240 [15:31:49<11:39:52, 494.03s/it] 65%|██████▌   | 156/240 [15:36:30<10:02:15, 430.18s/it] 65%|██████▌   | 157/240 [15:39:06<8:01:26, 348.03s/it]  66%|██████▌   | 158/240 [15:43:07<7:11:25, 315.68s/it] 66%|██████▋   | 159/240 [15:44:02<5:20:47, 237.63s/it] 67%|██████▋   | 160/240 [15:45:25<4:15:08, 191.36s/it] 67%|██████▋   | 161/240 [15:49:51<4:41:18, 213.65s/it] 68%|██████▊   | 162/240 [15:50:53<3:38:24, 168.01s/it] 68%|██████▊   | 163/240 [15:55:02<4:06:58, 192.45s/it] 68%|██████▊   | 164/240 [16:01:56<5:27:57, 258.91s/it] 69%|██████▉   | 165/240 [16:04:16<4:39:02, 223.24s/it] 69%|██████▉   | 166/240 [16:12:22<6:12:21, 301.92s/it] 70%|██████▉   | 167/240 [16:12:48<4:26:49, 219.30s/it] 70%|███████   | 168/240 [16:14:11<3:34:03, 178.38s/it] 70%|███████   | 169/240 [16:19:51<4:28:31, 226.93s/it] 71%|███████   | 170/240 [16:30:26<6:47:39, 349.42s/it] 71%|███████▏  | 171/240 [16:33:38<5:47:29, 302.17s/it] 72%|███████▏  | 172/240 [16:37:45<5:23:27, 285.40s/it] 72%|███████▏  | 173/240 [16:39:41<4:22:02, 234.67s/it] 72%|███████▎  | 174/240 [16:40:51<3:23:41, 185.18s/it] 73%|███████▎  | 175/240 [16:41:44<2:37:52, 145.72s/it] 73%|███████▎  | 176/240 [16:45:50<3:07:20, 175.63s/it] 74%|███████▍  | 177/240 [16:48:47<3:04:52, 176.07s/it] 74%|███████▍  | 178/240 [16:53:08<3:28:21, 201.63s/it] 75%|███████▍  | 179/240 [16:57:23<3:41:12, 217.59s/it] 75%|███████▌  | 180/240 [17:00:38<3:30:48, 210.80s/it] 75%|███████▌  | 181/240 [17:11:31<5:37:51, 343.59s/it] 76%|███████▌  | 182/240 [17:14:47<4:49:10, 299.15s/it] 76%|███████▋  | 183/240 [17:26:16<6:35:17, 416.10s/it] 77%|███████▋  | 184/240 [17:30:52<5:49:19, 374.28s/it] 77%|███████▋  | 185/240 [17:34:16<4:56:02, 322.96s/it] 78%|███████▊  | 186/240 [17:51:24<8:01:03, 534.52s/it] 78%|███████▊  | 187/240 [17:51:28<5:31:34, 375.36s/it] 78%|███████▊  | 188/240 [17:55:31<4:50:57, 335.71s/it] 79%|███████▉  | 189/240 [17:59:11<4:15:45, 300.88s/it] 79%|███████▉  | 190/240 [18:01:07<3:24:44, 245.68s/it] 80%|███████▉  | 191/240 [18:02:37<2:42:23, 198.86s/it] 80%|████████  | 192/240 [18:04:20<2:16:06, 170.13s/it] 80%|████████  | 193/240 [18:11:49<3:18:47, 253.77s/it] 81%|████████  | 194/240 [18:18:32<3:48:46, 298.40s/it] 81%|████████▏ | 195/240 [18:24:22<3:55:26, 313.92s/it] 82%|████████▏ | 196/240 [18:29:19<3:46:36, 309.02s/it] 82%|████████▏ | 197/240 [18:34:10<3:37:24, 303.36s/it] 82%|████████▎ | 198/240 [18:35:26<2:44:47, 235.42s/it] 83%|████████▎ | 199/240 [18:41:17<3:04:29, 269.99s/it] 83%|████████▎ | 200/240 [18:47:09<3:16:28, 294.72s/it] 84%|████████▍ | 201/240 [18:56:54<4:08:09, 381.77s/it] 84%|████████▍ | 202/240 [19:03:40<4:06:23, 389.04s/it] 85%|████████▍ | 203/240 [19:10:56<4:08:33, 403.07s/it] 85%|████████▌ | 204/240 [19:11:45<2:58:00, 296.69s/it] 85%|████████▌ | 205/240 [19:13:46<2:22:25, 244.16s/it] 86%|████████▌ | 206/240 [19:19:15<2:32:42, 269.48s/it] 86%|████████▋ | 207/240 [19:23:32<2:26:09, 265.76s/it] 87%|████████▋ | 208/240 [19:28:23<2:25:42, 273.22s/it] 87%|████████▋ | 209/240 [19:36:18<2:52:32, 333.94s/it] 88%|████████▊ | 210/240 [19:43:21<3:00:13, 360.47s/it] 88%|████████▊ | 211/240 [19:44:01<2:07:54, 264.62s/it] 88%|████████▊ | 212/240 [19:52:54<2:40:59, 344.98s/it] 89%|████████▉ | 213/240 [19:57:08<2:22:55, 317.60s/it] 89%|████████▉ | 214/240 [20:06:35<2:50:07, 392.61s/it] 90%|████████▉ | 215/240 [20:09:09<2:13:47, 321.08s/it] 90%|█████████ | 216/240 [20:18:29<2:37:03, 392.64s/it] 90%|█████████ | 217/240 [20:19:23<1:51:32, 291.00s/it] 91%|█████████ | 218/240 [20:21:04<1:25:51, 234.14s/it] 91%|█████████▏| 219/240 [20:27:53<1:40:13, 286.34s/it] 92%|█████████▏| 220/240 [20:32:29<1:34:28, 283.44s/it] 92%|█████████▏| 221/240 [20:34:05<1:11:53, 227.01s/it] 92%|█████████▎| 222/240 [20:36:33<1:01:03, 203.51s/it] 93%|█████████▎| 223/240 [20:37:01<42:45, 150.89s/it]   93%|█████████▎| 224/240 [20:37:50<32:03, 120.23s/it] 94%|█████████▍| 225/240 [20:43:28<46:24, 185.60s/it] 94%|█████████▍| 226/240 [20:45:55<40:36, 174.02s/it] 95%|█████████▍| 227/240 [20:46:36<29:03, 134.13s/it] 95%|█████████▌| 228/240 [20:49:03<27:33, 137.81s/it] 95%|█████████▌| 229/240 [21:04:52<1:09:54, 381.27s/it] 96%|█████████▌| 230/240 [21:06:50<50:22, 302.26s/it]   96%|█████████▋| 231/240 [21:06:57<32:02, 213.59s/it] 97%|█████████▋| 232/240 [21:24:35<1:02:17, 467.19s/it] 97%|█████████▋| 233/240 [21:28:13<45:46, 392.40s/it]   98%|█████████▊| 234/240 [21:28:29<27:55, 279.27s/it] 98%|█████████▊| 235/240 [21:30:37<19:29, 233.92s/it] 98%|█████████▊| 236/240 [21:46:31<30:00, 450.00s/it] 99%|█████████▉| 237/240 [22:08:52<35:52, 717.38s/it] 99%|█████████▉| 238/240 [22:23:53<25:44, 772.40s/it]100%|█████████▉| 239/240 [22:29:52<10:48, 648.24s/it]100%|██████████| 240/240 [22:39:48<00:00, 632.85s/it]100%|██████████| 240/240 [22:39:48<00:00, 339.95s/it]
+Writing report to /tmp/aime25_openai__gpt-oss-120b-high_temp1.0_20251109_094547.html
+{'chars': np.float64(2296.1916666666666), 'chars:std': np.float64(986.051306946325), 'score': np.float64(0.925), 'score:std': np.float64(0.26339134382131846)}
+Writing results to /tmp/aime25_openai__gpt-oss-120b-high_temp1.0_20251109_094547.json
+Writing all results to /tmp/aime25_openai__gpt-oss-120b-high_temp1.0_20251109_094547_allresults.json
+[{'eval_name': 'aime25', 'model_name': 'openai__gpt-oss-120b-high_temp1.0_20251109_094547', 'metric': 0.925}]