RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
Authors: Meng-Hao Guo
Jiajun Xu
Yi Zhang
Jiaxi Song
Haoyang Peng
Yi-Xuan Deng
Xinzhi Dong
Kiyohiro Nakayama
Zhengyang Geng
Chen Wang
Bolin Ni
Guo-Wei Yang
Yongming Rao
Houwen Peng
Han Hu
Gordon Wetzstein
Shi-min Hu
Venue: ICML 2025 poster (2025)
Keywords: Evaluation
Benchmark
Large Language Model
Multimodal Model
Reasoning.
Primary area: general_machine_learning->evaluation
TL;DR: An olympiad-level multi-disciplinary benchmark for complex reasoning evaluation

Abstract: Reasoning stands as a cornerstone of intelligence, enabling the synthesis of existing knowledge to solve complex problems. Despite remarkable progress, existing reasoning benchmarks often fail to rigorously evaluate the nuanced reasoning capabilities required for complex, real-world problemsolving, particularly in multi-disciplinary and multimodal contexts. In this paper, we introduce a graduate-level, multi-disciplinary, EnglishChinese benchmark, dubbed as Reasoning Bench (RBench), for assessing the reasoning capability of both language and multimodal models. RBench spans 1,094 questions across 108 subjects for language model evaluation and 665 questions across 83 subjects for multimodal model testing. These questions are meticulously curated to ensure rigorous difficulty calibration, subject balance, and cross-linguistic alignment, enabling the assessment to be an Olympiad-level multidisciplinary benchmark. We evaluate many models such as o1, GPT-4o, DeepSeek-R1, etc. Experimental results indicate that advanced models perform poorly on complex reasoning, especially multimodal reasoning. Even the top-performing model OpenAI o1 achieves only 53.2% accuracy on our multimodal evaluation. Data and code are made publicly available athttps://evalmodels.github.io/rbench/

Bench:
Graduate-level Milti-disyyllnnary Benchmarks for LLM & MILM Complex Reasoning Evaluation
Meng-Hao Guo
Jiajun Xu Vi Zhang' Jiaxi Song'
Haoyang Peng' Yi-Xuan Deng' Xinzhi Dong
Kiyohiro Nakayama²
Zhengyang Geng Chen Wang Bolin Ni Guo-We Bei Yang
Yongming Rao 15 Houwen Peng"
Han Hu
Gordon Wetzstein Shi-Min Hu teal
Abstract
Reasoning stands as a cornerstone of intelligence, enabling the synthesis of existing knowledge to solve complex problems. Despite remarkable progress, existing reasoning benchmarks often fail rigorously evaluate the nuanced reasoning capa- bilities required for complex, real-world problem- solving, particularly in multi-disciplinary and multimodal contexts. In this paper, we intro- duce a graduate-leve multi-diiscplinnay. English- Chinese benchmark, dubbed as Reasoning Bench (RBench), for assessing the reasoning capabil- ity of both language and multimodal models. RBench spans 1,094 questions across 108 sub- jects for language model evaluation and 665 ques- tions across 83 subjects for multimodal model testing. These questions are meticulously curated to ensure rigorous difficulty calibration, subject balance, and cross-linguiie moos-lingu alignment, enabling the assessment be Olympiad-level multi- disciplinary benchmark. We evaluate many mod- els such as ol, GPT-4o, DeepSeek-R1, etc. Ex- perimental results indicate that advanced models perform poorly on complex reasoning, especially multimodal reasoning. Even the top-performing model OpenAl ol achieves only 53.2% 5% accuracy our multimodal evaluation. Data and code are made publicly available here.
MMUU
X: BaallTT
76.2
6AB
Accuracy (%)
23.6
01-20141217
GPT-4c
DeepSeckRR
01-20241217
GPT-40
Figure I. Top-1 accuracy comparison of different models on MMLU, MMMU, and RBench. RBench poses a greater chal- lenge to current models.
Reasoning, the systematic process of synthesizing knowl- edge to solve novel problems, lies at the heart of intelligence. Yet, as foundation models grow increasingly sophisticated, existing benchmarks fail to comprehensively assess their complex reasoning capabilities. As shown in the above quote, before equipping foundation models with reasoning skills, we should first define goals for them by establishing reliable evaluation assess their reasoning capabilities.
As noted in (Kahneman, 2011) and (Wei et al., 2022), re- alizing system-l, a.k.a., quick and intuitive thinking and system-1l, a.k.a., slow and deliberate reasoning raises dis- tinet requirements on foundation models. Similarly, assess- ing quick thinking and complex reasoning requires sub- stantially different assessment methods. On the one hand, evaluating system-I needs evaluate the knowledge and memory, which requires collecting various daily conver- sations and knowledge-based questions e-g., concept and common sense questions. On the other hand, evaluating system-II requires evaluating complex reasoning skills. requires gathering a diverse range of reasoning questions, such as analytical and deductive ones, which is more chal-
1. Introduction
"Setting goals is the first step in turning the invisi- ble into the visible." -Tony Robbins
Joint project lead, 'Tsinghua University. "ttanford University, 2 "CMU, "University Pennsylvania, *Tencent Hunyuan X, FFitten. Correspondence to: Shi-Min Hu <shimin tsinghua.cdu.cn>>
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
benchmarks examples. MMLU (Hendrycks al., 2021) is comprehensive benchmark for multi-discipline under- standing, which has served as a critical guide for the develop- ment of foundation models in recent years. However, consid- ering the current level of model intelligence, this benchmark is close to saturation (c. ol (OpenAI, 2024b) has achieved 92.3% accuracy on it). Besides, dose not take multimodal- ity and multilingualism into consideration, which is also crit- ical for ideal reasoning test. MMMU (Yue et al., 2024a) is holistic evaluation for multimodal reasoning tests. With the launch of ol (OpenAT, 2024b), this benchmark is also close to saturation. Also, it cannot be used to evaluate lan- guage models and ignores multilingual testing. We show the comparison of MMLU, MMMU, and RBench in Fig. I and Tab. 1. Frontiermath (Glazer al., 2024) collects some challenging problems specifically designed for advanced mathematical reasoning evaluation, which indicates that current models still exhibit weaknesses in mathematical rea- soning. However, it falls short in comprehensiveness and multilingual testing. This also applies to Omni-Math (Gao et al., 2024) and AIME (OpenAI, 2024b), both of which serve as benchmarks focused on employing mathematical olympiad challenges.
Table 1. Comp. denotes comprehensiveenss. saturation rep- resents ol (OpenAl, 2024b) performance on this benchmark. It reflects the challenge that the benchmark poses to advanced mod- els.
Name
Comp.
ol Saturation
Language
MMLU
0.923
AIME@ @ 2024
X 0.744
en
RBench-T T
0.690
en & zh
MMMU V 0.782
en en & zh
RBench-M
V
0.532
lenging to collect and filter than the former. In this paper, we focus on building reliable complex reasoning benchmark for both large language models (LLMs) and multimodal large language models (MLLMs).
How can we design an ideal assessment for complex reason- ing? We believe following four properties are critical.
Comprehensiveness. Evaluating the intelligence of foundation models is akin to evaluating human intel- ligence. We cannot focus on just one aspect, such as mathematics. A comprehensive evaluation is essential.
In this paper, goal is to build a benchmark Bencch that aligns with the four properties we proposed for evaluating the reasoning abilities of intelligent models. To achieve that, we follow more than 100 college courses from 19 de- partments at Tsinghua University and collect challenging problems from their exams, textbooks, quizzes, homework, etc. After multiple rounds of rigorous screening by experts and models, we finally select 1,094 questions spanning 108 subjects for language models reasoning test, and 665 ques- tions covering 83 subjects for multimodal models reasoning test. We will present the detailed screening process in the Sec 2. After building the KBench benchmark, we test the reasoning capabilities of various powerful proprietary mod- els such as ol (OpenAI, 2024b), GPT-4o (OpenAI, 2024a), Gemini (Team et al., 2023), Claude (Anthropic, 2024a), and open-sourcee models such as Llama 3 (Touvron et al., 2023), Qwen 2.5 (Yang et al., 2024), etc. From experiments, our observations and findings are summarized as follows:
Difficulty. A meaningful evaluation should exhibit the capability to effectively discriminate between the performance of different models and provide valuable insights for guiding model improvement. At present, foundation models are developing rapidly, and some simple benchmarks have been saturated and cannot pro- vide guidance and discrimination for advanced models.
Multimodality. We live in a multimodal world, con- stantly processing various visual and linguistic signals. Therefore, an ideal benchmark should be designed to assess both LLMs and MLLMs.
Multilingualism. We believe that performing com- plex reasoning is more challenging than understanding multiple languages. A model with robust complex rea- soning skills should be capable of solving reasoning problems across different languages. This is like for human expert, he she will not lose the ability to ad- dress problems due to language changes. Thus, assess- ing model performance on equally difficult questions across languages is essential. It will provide insight into whether the model has genuinely learned to reason or merely overfitting to a specific language.
With the emergence of advanced models like ol, exist- ing multidisciplinary evaluations have nearly reached saturation. Besides, solely relying on math problems, e.g., mathematical olympiad problems, may bring bias in model evaluation. Therefore, the community needs challenging multi-disciplinary benchmarks to guide foundational models in enhancing their reasoning abil- ities, and the goal of RBench is to address it.
While there have been attempts to create an ideal reasoning benchmark, to the best of our knowledge, existing bench- marks cannot incorporate all four of these key properties simultaneously. Here, we take some widely used reasoning
We illustrate from three dimensions - expert scor- ing, model scoring, and model thinking time- - that RBench is more complex benchmark with higher re-
2

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
quirements for model reasoning compared to existing
multidisciplinary benchmarks MMLU and MMMU.
Step Define a list of collected disciplines
Step 2 Experts collect and select reasoning questions
Multimodal complex reasoning remains challenging. Despite rapid advances, models lag behind text-based reasoning. For instance, GPT-4o scores 53.6% on text but only 33.7% in multimodal reasoning on RBench.
KQ
RQ
Chain of Thought (CoTT) can enhance reasoning abili- ties in most chat models, such as GPT-4o. However, for reasoning models like ol-mini, CoT does not have the same effect. This may be because reasoning models inherently build CoT, making explicit CoT ineffective.
Step
Digitize the thelquestions
Same text-only questions
Some multimodal questions
Models maintain high consistency in answering Chi- nese and English questions of equal difficulty, exceed- ing 70% for most models, demonstrating strong cross- lingual reasoning capabilities.
Step 4: ol model rescreen based on reasoning difficulty
Step 5: The third round screens for completeness, repetition and ambiguily
-2000
AQ
Foundation models perform differently across disci- plines. Specifically, GPT-4o achieves 30.4%-68.3%% accuracy across various fields.
CQ
> 2000
2. RBench
R-Bench
Step 6: Constructing
options and translations
In this section, we will thoroughly introduce the construc- tion process of RBench. The entire process involves mul- tiple steps such as data collection, filtering and improving. The overall pipeline is illustrated in Fig. 2.
R-Bench -T
R-Bench -T(ch)
R-Bench -M
x R-Bench
-M[zh)
2.1. Data collection
Before gathering reasoning questions, we conducted an investigation of the curriculum systems of graduate and undergraduate students across different departments at Tsinghua University. Based our survey, we obtained collection list covering over 100 courses across 19 depart- ments, which is shown in Fig. 2 Step 1.
Figure 2. Pipeline of building RBench. The process divided into six steps. which are detailed in Sec. 2. The funnel represents screening. We always filter out the blue ball preserve the brown one. In Step 2, KQ and RQ denote knowledge-based questions and reasoning- hasssning-haeed hased questions, respectively. In Step 4, < 2000 indicates that the reasoning tokens of o1 are less than 2000. Finally, in Step 5, AQ and CQ represent ambiguous questions and clear questions, respectively. -T indicates text-only testing for LLMs. -M means multimodal testing. zh represents the Chinese version.
After acquiring a collection list, we recruit senior under- graduates and graduate students from different departments as experts to provide reasoning question-answer pairs. We recruit total of 51 experts, with at least two participants from each department, to help collect and filter ques- tions. During the collection process, we mainly focus controlling the following key aspects: 1) The questions should align with the collection list we provide. 2) The professional expertise should filter out "knowledge-basee" based" questions-those that rely solely on memory rather than reasoning, such as concept-definition questions. Simulta- questions
After the two steps above, we collected a total of 10,270 questions. Among them, 7,163 questions, which do not include images, are designated for testing language models, while the remaining 3,107 questions, containing images, allocated for testing multimodal models.
2.2. Data digitization
neously, experts should retain reasoning- and ensure they present sufficient degree of difficulty. 3) All questions should have corresponding answers that can be automatically verified. In this collecting process, we ex- clude proof-based questions, as current automated methods cannot verify the correctness of proofs. The process above shown in Fig. Step2. 2.
After initially collecting the questions, we find that the col- lected questions are in messy format, including pictures, screenshots, text, etc. In addition, the summary question files provided by different experts are also different, includ- ing pdf, word, excel, etc. Therefore, we need to organize and digitize this data.
3

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
Examples for language models reasoning ability test
Major: computer science; Subject: data structure.
Major: math;
Subject: complex variable function.
In the undireeted graph G=(VE) where (=1...334.566677 and (1..)))) (1.3), (2.3). (4,5), (3.6), (4,7). (5,7), how many different spanning forests does the graph contain? Note: the edge of two spanning forests are different, they are considered diffcrent spanning forests.
English
Chinese
Consider the polynomial [[=)= 55 How many zero points (counting multiplicities) does p have in the armulus 1<2]?2?
+ =
A: 5 B:3
C:2 A:All other answers are incorrect
C:2 D
1: 8:9 C: 10 D: 1: E: incorrect
E:4 F:6
E:4 F:6
A:All other
Examples for multimodal models reasoning ability test
Major: mechanical engineering:
Subject: theoretical mechanles.
English
Chinase
Scre
Jop
3ume
Jap
The width of the brick clamp is 25 om, and the rods and point G, with dimensions shown the figure. Suppose the weight of the brick is brick along the centerline the brick clamp. The coefficient friction berween the brick clamp the brick
hinged
1J GCED & G $ # R Ff ie tk Q=120N #
@ 0 120N and ferce that
AM F-0.5, * #.
be
Determine maximum value of distance required lift the brick using the clamp.
A:9 B:1l cm
15
D:
E:25cm
F: 11
A B: 11 E: 25 em
C: 15
D. 20 cm
F: All other answers are incorroct
Figure 3. Some examples in RBench. These examples show that RBench is multidiseiplinary. multimodal, and multilingual. As shown in the figure, the problems RBench are complex and cannot solved by quick thinking, which shows that RBench focuses on deep reasoning problems rather than knowledge problems, such as conceptual problems.
To do so, we recruit a data annotation team of about 20 people. They are responsible for organizing, digitizing, checking, and compiling all the questions into Excel sheets. The questions used for language models organized in the following format:
2.3. Data filtering
As shown in Fig. 2, the funnelss in steps 2, 4, and represent three different rounds of data filtering. These three rounds of screening represent expert screening, model-based filtering, and manual review.
"Department- Subject Question (text) Answer (text) Original Question (text, screenshots, photos, etc.) Original Answer (text, screenshots, photos, etc.)".
Expert-screening. As mentioned in Sec. 2.1, we recruit experts from different departments to provide questions for us. They primarily rely on their professional knowledge to filter out "knowledge-basee based" questions while retaining "reasoning- based" questions.
As for questions designed for multimodal models, they are organized into the following format:
"Department Subject Question (text) Answer (text) Question Images- Original Question (text, screenshots, pho- tos etc.) Original Answer (text, screenshots, photos etc.)".
Moreenncr.,,ing. OpenAl ol (OpenAI, 2024b) is widely used reasoning model. When we call its API, returns the number of reasoning tokens, which, to some extent, reflects the difficulty of the question. In this round of screening, we mainly focus on the difficulty of reasoning. We filter out the questions with less than 2,000 reasoning tokens to ensure that our RBench is a benchmark for rea- soning evaluation.
In this process, we utilize tools such as GPT-4o and Mathpix for OCR processing, followed by manual proofreading to ensure it is correct. After the data team organizes the data, perform double-check on the OCR results.

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
Inoeganie Chemisity
Anslytical Mechunics
Chemistry
Plysical Chenisity
Chemical Thameddynaiis
Fundamental Plysics
Math
Physics
Electionic
Compater
Acroepace
4.8%
4.7%
4,4%
3.7%
3,7% 3 9.%%
2.7%
2.15
Organic Clamislyy
Chemiical Reactisn Kinctios
Computational Chamistry
Chemisrry
4.8%
4.7%
4.1%
3.9%
3,8%
3,5%
3,21
Optics
Flectmolnnamics
Automation
1.0%
5%
2.75
Stataties
5.4%
Mechamical
4,8%
2.6%
Materall
5.9%
3,8%
Civil
Diology
7.3%
14.7%
4.4%
9%
1.5%
Vehicle
6.6%
1.5%
Physics E.
Chemical
0.9%
Micrddectonic
Economiiss
0.5%
1%
7.5%
10.2%
Archisscture
Environment
9.4% 17.2%
D S9C
9.2%%
(a) Statistics of RBeench-T
(b) Statistics of 3BBnnchMM
Figure 4. According to statisties on RBench, the benchmark spans 19 departments, including mathematics, physies, biology, computer science, and chemistry, covering over 100 subjects such as Inorganic Chemistry, Chemical Reaction Kinetics, and Electromagnctism, It features 1,094 questions designed for testing language models and 665 questions specifically tailored for evaluating multimodal reasoning capabilities. For detailed list of subjects, please refer to the appendix.
Manual review. Our manual review focuses on whether the question conditions are complete, whether the questions are repeated, whether the questions are ambiguous, and the balance of subjects.
2.5. Overview of XBench
After completing the aforementioned steps, we develop RBench, graduate-level, multi-discipline, multilingual benchmark designed to evaluate complex reasoning capa- bilities for both language and multimodal models. Fig. 3 illustrates several examples from RBench, clearly highlight- ing its above distinctive features.
Checking for completeness, repetition, and ambiguity re- quire multiple rounds of thorough review by different in- dividuals to eliminate ambiguities, along with the use of duplication detection tools to avoid repetition. As for the balance check, to reduce testing bias from subject imbal- ance, we limit the number of questions per subject to a maximum of 50 by filtering out excess.
RBench can be divided into four sub-benchmarks: RBench- T and RBench-T(zhh for language model evaluation, RBench-M and RBench-M(zh) for multimodal model eval- uation. Here, RBench-T denotes RBench using text-only questions in English for LLM evaluation, whereas RBench- T(zh) represents RBench using text-only questions in Chi- nese for LLM evaluation. Likewise, the other two notations follow the same naming convention.
2.4. Conducting options and translations
In order to enable answers to be evaluated automatically and accurately, we convert all questions such as analyti- cal, fillinntheebllan,, and multiple-choice questions into the single-choice question format. We use GPT-4o to construct 5 options for each question and add an option "All other answers are incorreet", which equips each question with 6 candidate answers. Then, we check the options multiple times to ensure the correctness of our construction. Further- more, we manually adjust the options to ensure a sufficient numerical gap between them, thereby avoiding errors caused by numerical approximations.
We conduct statistical analysis on RBenchwith the results presented in Fig. 4. It presents the RBench-T statistics for text-only questions used in evaluating the reasoning capabil- ities of language models. RBench-T spans 18 departments, including mathematics, biology, chemistry, computer sci- ence, electronic engineering, and others. It encompasses over 108 subjects, such calculus, number theory, analytic geometry, ordinary differential equations, and functional analysis, and comprises a total of 1,094 questions. Fig. 4 also presents the statistics of XBench-M, which evaluates the reasoning capabilities of multimodal models. RBench- M incorporates a diverse set of question types requiring both textual and visual inputs. It covers departments, such as physies, biology, architecture, and economics, and includes 83 subjects, such as thermodynamics, molecular biology, structural design, and microeconomics, including total of 665 questions. It is worth noting that we provide English and Chinese versions for all questions.
Besides, in order enable RBench acquire the multi- lingual property. we manually constructed Mnglish-Chinese translations for each question. During the translation pro- cess, we utilize tools like GPT-4o. Each question is meticu- lously reviewed and refined by three experts fluent in both English and Chinese to ensure correctness and clarity.
5

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
Table Comparison of reasoning requirements for problems in RBench-T I and MMLU via expert and ol voting.
Table 4. The average thinking time of ol on 30 randomly selected samples from different benchmarks. TT denotes thinking time.
RBench-T win
MMLU win
Tie
MMLU
RBBnch-T
MMMU
RBench-M
Expert voting ol voting
85.94%
10.62% 20.00%
3.44%
TT
13.5s
98.2s (7.3x)
20.38
91.7s (4.5x)
76.67%
3.33%
Table 3. Comparison of reasoning requirements for problems in RBench-M and MMMU via expert and ol voting.
3.2. Evaluating reasoning capability of different models
RBench-M M win
MMMU win
Tie
We employ 3Bench-T to assess the reasoning capabilities of various LLMs such as ol (OpenAI, 2024b), GPT-4o (Ope- AAI, 2024a), DeepSeek-R1 (AI, 2025), Gemini (Team et al., 2024), Claude3.5 (Anthropic, 2024b), Qwen2.5 (Yang et al.,
Expert voting ol voting 83.33%
76.88%
15.94% 13.33%
7.19%
3.33%
2024), Llama3 (Dubey et al., 2024), etc, in both English and Chinese settings. The evaluation involves utilizing API calls and deploying open-source models locally. For API calls, we utilize the official interfaces with default hyperpa- rameters. For open-source models, we deploy their weights locally using vLLM (Kwon et al., 2023), setting the tem- perature to O while keeping all other parameters at their default values. The evaluation was conducted using the tools provided by OpenCompass (Contributors, 2023). In all tests, the CoT prompt is used by default. For details on the specific prompts, please refer to appendix. In the results shown in Tab. 5, we found that models designed for reasoning tasks, such as ol, outperform chat models like GPT-4o in complex reasoning. Besides, there remains a significant gap in complex reasoning between open-source models and commercial models.
3. Experiments
After developing RBench, we utilize it to assess the com- plex reasoning capabilities of various LLMs and MLLMs, including both open-source models such as Llama and close- source models such as GPT-4o. Firstly, we aim to demon- strate that RBench is a benchmark for complex reasoning through expert scoring (user study) and ol model scoring. Then, we evaluate the reasoning capabilities models with and without CoT prompting under zero-shot setting. Fi- nally, we analyze the experiments and summarize observa- tions and findings from the experimental process.
3.1. Reasoning comparison with other benchmarks
To illustrate that RBench is benchmark designed to eval- uate reasoning ability, employ two methods: expert scoring and reasoning model scoring.
Moreover, we utilize RBench-M evaluate the reason- ing capabilities of various MLLMs, including ol (OpenAI, 2024b), GPT-4o (OpenAI, 2024a), Claude 3.5 (Anthropic, 2024b), Qwen2.5-VL. (Yang et al., 2024), and InternVL. 2.5 (Chen et al., 2024), etc, across both English and Chinese languages. The evaluation also involves utilizing API calls and deploying open- sour-s models locally. For API calls, we utilize the official interfaces with default hyperparam- eters. For open-source models, we deploy their weights locally using VLMEvallit (Duan et al., 2024), setting the temperature to 0 while keeping all other parameters at their default values. In all tests, the CoT prompt is used by de- fault. For details on the specific prompts, please refer to our appendix. We draw three conclusions from the results in Tab. 6. First, we found that models perform worse in multi- modal complex reasoning compared to reasoning in a purely linguistic environment. Second, the reasoning model ol still demonstrates outstanding performance in multimodal complex reasoning evaluation. Third, the gap between open- source and closed-source models is even more pronounced in multimodal complex reasoning.
We conducted expert scoring through user studies. To be specific, we randomly selected 30 questions from RBench-T and another 30 questions from MMLU and presented them to experts for pairwise comparisons to determine which question required more reasoning skills solve. We con- structed similar experiments using the same settings be- tween RBench-M and MMMU.
For reasoning model scoring, we adopted two approaches. On one hand, we used the ol model to determine which question required more reasoning ability based on the num- ber of reasoning tokens (reasoning time); on the other hand, we asked the ol model to directly compare the two questions and determine which one required more reasoning.
The results are shown in Tab. 2,3 and 4. The results indicate that both ol 's judgment and the experts' judgment consider RBench to require significantly higher reasoning ability compared to MMLJU and MMMU.
6

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
Table Performance comparison various models on XBench-- in zero-shot settings with COT. The table is divided by middle line: API-based models are listed above line, while open-source models shown below. 'zh' indicates the Chinese version, values the table represent the Top-1 accuracy, in %.
Table 6. Performance comparison of various models on RBench- M in zero-shot settings with CoT. The table is divided by middle line: API-hased models are listed above the line, while open-source models are shown below. 'zh' indicates the Chinese version. The values in the table represent the Top-1 accuracy, in %.
Model Name
RBench-T RBench- T(zh)
Model Name
RBench-M RBench-Moe
o1-20241217
69.0
70.1
ol 01-20241217
53.2
55.0
Gemini-2.0-flashttiinn
68.4
67.5
Claude3.5-sonnette 1022
39.7
38.3
Doubaol .pp0220.55000222250111
62.0
63.4
Doubaol..prro-0225012
37.9
42.4
ol-preview @ 20240912
62.3
62.6
GPT-40-202411200
33.4
33.2
ol-mini @ 20240912
64.0
59.9
Gemini-1.5-Pro
35.5
35.9
Drubaa-pro-2222412155
60.7
60.8
Qwen2-VL-72B VL-72B
25.1
25.7
Claude3.5-sonnettee
57.5
57.0
Qwen2- VL-n2-
19.6
22.3
GPT-44-20241120
53.6
51.6
LLaVA-OneVisionn
23.8
23.5
MixiMax-Text-01
53.8
53.6
DeepSeek-VEs V1.2
21.8
24.4
GLM--Zero-Preview
53.6
48.6
1amaa322--1B-Instruee
20.0
18.6
ERNIIE4.0-SK-Lateed
39.7
50.1
IntemVL-2..588
15.9
17.1
Deepseek-R1
61.2
59.3
Deepseek-V3
59.6
56.6
Qwen3-235BBA22B
58.0
58.4
Table 7. Assessing the performance impact of CoT across different
Qwen3-32B
52.3
54.3
models on RBench-T.
Frren..5-22-Insstuee
53.7
52.0
JJama-3.3-70B-Instiie
49.5
47.6
Model Name
CoT(%)
w/o CoT(%%
Qwen2.5-32B-Insteee
50.8
49.9
ol-mini 20240912
21-min1@202409122
64.0
64.0
Gemma-2-27b-it
36.0
38.9
GPT-40-20241 120
53.6
51.5
Phi-4-14B
55.3
47.3
LLAMA333-70B-BB
49.5
47.4
Phi-3-14B
29.5
24.4
50.8
44.6
Owen3-8B
47.5
45.9
Bren2.5-7B-Instruee
43.6
42.6
IntemmLM3-8B-Inntiee
41.1
45.8
Qwen2.5-7B-Inste
43.6
44.5
GLM-4-9b-chat
25.6
32.4
focused models such as ol-mini. We conjecture that this discrepancy arises because reasoning models inherently uti- lize CoT-like mechanisms, leading the explicit addition of CoT redundant and ineffective.
(lama-3.1-8B-Insse
26.1
23.6
24.2
24.0
3.3. Observations and findings
Consistency between English and Chinese questions. As shown in Fig. 5, we tested the consistency of the same question across different languages on RBench-T. can be observed that most models, such as ol, Noubaol.5pre and GPT-4o, exhibit certain degree of consistency across dif- ferent languages. This suggests that foundation models have already demonstrated a certain level of intelligence, enabling them to perform reasoning on problems of the same diffi- culty different linguistic environments. However, these models are not perfect and still require further improvement in this aspect. This consistency reflects the extent which the model overfits different languages. Therefore, hope future models will focus more on learning how to reason rather than merely fitting to specific languages.
Multimodal reasoning remains challenging for current models. We compared the performance the same model on RBench-T and RBench-M. For example, ol achieved 69.0% on RBench-T T but only 53.2% on KBench-M. The same situation also occurs in other models, such GPT-4o. It indicates that the model's capability in language reasoning significantly surpasses its ability in multimodal reasoning. Therefore, key focus of research in the recent future will be how to transfer linguistic intelligence to the multimodal domain.
The effect of COT. In Tab. 7, we tested the effect of CoT on five models. As seen in the table, most models benefit from CoT, but it has no impact on ol-mini. The results indi- cate that CoT enhances the performance chat models like GPT-4o. However, it has no notable impact on reasoning-
Models show significant performance variation across disciplines. Fig. shows the performance of GPT-4o in

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
leveraged across diverse fields such as writing, coding, edu- cation, healthcare, finance, and more, serving source for providing intelligence. Now, foundation models have become an essential part of our daily work and life.
suncy of English Chinas quattions cene deffccuity
12.5%
12.4%
692%
To build a high-quality foundation model, we believe that five key aspects are essential: pre-training (Vaswani et al., 2017: Raffel al., 2020; Sun al., 2023; Radford et al., 2021: 2018), supervised fine-tuning (Ouyang al., 2022; Liu et al., 2024b; Pareja et al., 2024; Li et al., 2024; Zhu et al., 2023; Taori et al., 2023), preference optimiza- tion (Rafailov et al., 2024; Schulman et al., 2017; Lightman et al., 2023; Pal et al., 2024; Azar et al., 2024))test-time enhancement (Brown et al., 2020; Wei et al., 2022; OpenAl, 2024b; DeepSeek, 2024; Dong et al., 2022), and trustwor- thy evaluation (Chiang et al., 2024; Li et al., 2023; Chen et al., 2021; Jain et al., 2024; Yu et al., 2023). Reliable evaluation plays a crucial role in revealing models' weak- nesses and shortcomings, guiding further optimization and improvement, which is also the focus of this paper.
Duad apeestenn hotmmit
Pwent SwwzA1338ee tar
(a Praeotoott
poezA tortee
Greate 2u2t
devez faroot
Iooea all
Figure 5. The performance of different models on questions of the same difficulty Chinese and English.
Accurazy Differast Deparieens
Averaee: 33,6%
4.2. Evaluation for foundation models
Evaluating the intelligence of foundation models is mul- tifaceted and complex challenge, akin to assessing human intelligence. Researchers have introduced various evalu- ation benchmarks, broadly categorized into fast-thinking assessments aak.a., system-I evaluation (Chiang al., 2024; Dubois et al., 2024; Zheng et al., 2023; Lin et al., 2021; 2024; Lu al., 2022; Yu et al., 2023; Li et al., 2023), which requires the foundation models to memorize exten- sive knowledge and retrieve efficiently, and slow-thinking assessments (a.k.a, system- evaluation (Hendrycks et al., 2021; Yue et al., 2024a; Lu et al., 2023; Wang et al., 2024a; Liu al., 2024c; Chen et al., 2021; Jimenez et al., 2023), which emphasizes the complex reasoning skills of founda- tion models. RBench focuses on the latter.
51.% 542%
35.9%
Accemnn
Cumporr Morial
Ood' tiblle
Figure 6. GPT-4o on RBench-T across different departments, which shows large variation among different disciplines..
MMLU (Hendrycks et al., 2021) is the pioneer in reasoning evaluation, which proposes a multi-discipline understanding test. After that, lots of multi culti-discciline benchmarks (Rein et al., 2023; Wang et al., 2024b; Yue et al., 2024b) at the undergraduate or graduate level are proposed for reasoning assessment. However, with the rapid development of intelli- gent models (OpenAI, 2024b), these benchmarks are close to saturation. Besides, several studies (Glazer et al., 2024; Gao et al., 2024; Lu et al., 2023; Wang et al., 2024a) assess reasoning ability through complex mathematical problems such as mathematical olympiad challenges, which bring challenges and guidance to current foundation models. How- ever, only guiding the model to improve its mathematical reasoning skills appears to be limited. In this paper, target is to build a reliable benchmark for LLM and MILLM reasoning evaluation, which matches the comprehensive- ness of MMLU (Hendrycks et al., 2021) while achieving
different areas of the RBench-T benchmark. From the figure, it can be observed that the performance varies significantly across different domains, with range reaching 37.9%. This suggests that if we want to improve the reasoning ability of models, we should take a comprehensive approach rather than focusing solely on improvements in a single subject, such as mathematics.
4. Related Work
4.1. Foundation models
With the emergence of ChatGPT (OpenAI, 2022), founda- tion models (Ouyang et al., 2022; Touvron et al., 2023; Yang et al., 2024; Jiang et al., 2023; Team et al., 2024; Zeng et al., 2022; Bi et al., 2024; Liu et al., 2024a; Cai et al., 2024; Anthropic, 2024a; Wu et al., 2024) are increasingly being
8

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
the difficulty of mathematical olympiad questions.
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseek Ilm: Scaling open-source language models with longtermism. 2024.
5. Conclusion
arXiv preprint arXiv:2401.0290
In this paper, we proposed RBench, a graduate-level multi- disciplinary, multilingual benchmark for both LLM and MLLM reasoning evaluation, which has coverage similar to MMLU and MMMU while reaching the difficulty of mathematical competitions such as AIME@ 2024. We eval- uated multiple closed-source and open-source models such as OpenAl ol, GPT-4o, DeepSeek-RI etc, on &Bench and observed both the progress and limitations of current mod- els in reasoning. Later, we will make the data and code available, hoping to provide guidance and insight for the development of foundation models.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H., Fan, Q. Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q.. He, C., Hu, Y., Huang, T., Jiang. T., Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W., Li, Y., Liu, H., Liu, J., Hong, J., Liu, K., Liu, K., Liu, X., Lv. C., Lv, H., Lv, K., Ma. L.. Ma, R., Ma, Z., Ning, W., Ouyang, L., Qiu, J., Qu, Y., Shang, F,, Shao, Y., Song, D., Song, Z., Sui, Z., Sun, P., Sun, Y., Tang, H., Wang, B., Wang, G., Wang. J., Wang, J., Wang, R., Wang. Y., Wang, Z., Wei, X., Weng, Q. Wu, F,, Xiong, Y. Xu, C., Xu, R., Yan, H., Yan, Y., Yang, X., Ye, H., Ying, H., Yu, J., Yu, J., Zang, Y., Zhang, C., Zhang, L., Zhang, P., Zhang, P., Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang, W., Zhang, X., Zhang, X., Zhao, H., Zhao, Q., Zhao, X., Zhou, F., Zhou, Z.. Zhuo, J., Zou, Y., Qiu. X., Qiao, Y., and Lin, D. Internim2 technical report, 2024.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (project No. 62495061, 623B2057), the Research Grant of Tsinghua-Teneent Joint Laboratory for Internet Innovation Technology.
Impact Statement
RBench provides actionable insights into model reasoning gaps, especially in multimodal and pross-lingual contexts, and helps guide the development of more generalizable and robust reasoning models. Moreover, our benchmark en- ables researchers and practitioners to diagnose and compare reasoning abilities across a broad range of disciplines and modalities, supporting not only academic progress but also downstream applications in education, healthcare, law, and science. All data and code are publicly released to foster transparency and reproducibility, making R-Bench a valu- able asset for the community.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021.
Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv: wwXhv:241..05277 2024.
References
AI, D. Deepseck-rl:
Incentivizing reasoning capabil- ity Iims via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonza- lez, J. E., et al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint urXhv:2403.04132,, 2024.
Anthoopic. The claude model family: Opus, sonnet, haiku. http://wwwwwwwwoooc.coo////////e 3-family, 2024a.
Contributors, O. Opencompass: A universal evaluation platform for foundation models. https://qithit com/
Anthropic Introducing 2024b.
claude 3.5 sonnet.
2023.
DeepSeek.
Deepseek-rl release. hutps:/appi- docs. wwwsdddepeeek.co/ew//eeeee
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
2024.
Dong, Q., Li, L.., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Liu, T., et al. A survey on in-context learning. arXiv preprint arXiv:2301.000 2022.
9

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang. P., Wang. J., et al. Vlmevalkit: An open-source toolkit for evaluating large mulli-modaliity models. In Proceedings of the 32nd ACM international conference on multimedia, pp. 1 11198- 11201, 2024.
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-hench: Benchmarking multimodal llms with gener- ative comprehension. arXiv preprint arXtv:2307.16123, 2023.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint urXiv:2408.03326. 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Ilama herd of models. arXiv preprint arXiv:2407.21788, 2024.
Lightman, H.,, Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20055) 2023.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04473 2024.
Lin, B. Y., Deng, Y., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L., and Choi, Y. Wild- bench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.0477 2024.
Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q., Li, L.,, Ma, C., Chen, L., Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXh::2109.07958 2021.
Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F.. Denain, J.-S., Ho, A., Santos, E. d. O., et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:24 1,0472, 2024.
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao C., Dengr, C., Ruan, C., Dai, D.. Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-exper language model. arXiv preprint AKKv:24055044334 2024a
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardtt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https:// d= 33333/333m3 Jm.. wwwnnreviee.net/eeee Ka
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing, Advances neural information processing systems, 36, 2024b.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint 203i::2403.079774. 2024.
Liu, J., Liu, H., Xiao, L., Wang. Z., Liu, K., Gao, S., Zhang, W., Zhang, S.. and Chen, K. Are your Ilms capable of stable reasoning? arXiv preprint arXiv:2412.13147. 2024c.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. 1., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint 10.06253 2023.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507-2521, 2022.
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- cls resolve real-world github issues? arXiv preprint arXiv:2310.0677 2023.
Lu, P. Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310002255,
Kahneman, D. Thinking, fast and slow. Farror, Straus and Giroux, 2011.
Kwon, W., Li, Z., Zhuang, S., Sheng. Y., Zheng, L., Yu, C. H.. Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
OpenAI.
Introducing 2022.
chatgpt.
OpenAl. Gpt-4o system card. http://opend..co/indeeggg 4o-system-cardl, 2024a.
10

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
OpenAl.
Learning
reason with Ilms.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following Ilama model, 2023.
http:///gneniiicoo/////e/leaaaie llms/, 2024b.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 15:27730-277744 2022.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint 2xi::2322./8800 2023.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang. S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of pref- erence optimisation with dpo-positive. arXiv preprint arXiv:2402. 13228, 2024.
Pareja, A., Nayak, N. S., Wang, H., Killamsetty, K., Su- dalairaj, S., Zhao, W. Han, S., Bhandwaldar, A., Xu, G., Xu, K., et al. Unveiling the secret recipe: A guide for supervised fine-tuning small llms. arXiv preprint arXiv:2412. 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S.. and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpps///p neerseiiidiinnnnnneeeooeeeo cc cc/sapee_filee/ppprr//////// 3 1ff422456476e1111000001144444433 pdf.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, 1. Improving language understanding by generative pre- training. 2018.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Rafailov, R., Sharma, A., Mitchell, E.,, Manning, D., Er-
Wang, K., Pan, J., Shi, W., Lu, Z., Zhan, M., and Li, H. Mea- suring multimodal mathematical reasoning with math- vision dataset. arXiv preprint arXiv:2402. 14804, 2024a.
mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., al. Mmlu-pro. A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574, 2024b.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (140):1-67, 2020.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of of-thouugh prompting elicits reasoning in large language models. Advances neural information processing systems, 35:24824 24837, 2022.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-prood proof q&a benchmark. arXiv preprint arXiv:231 12022, 2023.
Wu, 7., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al. Deepseek-v12: Mixture- of-experts lasion-la models for advanced multi- modal understanding, arXiv preprint 242 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.0600 1707.06347 2017.
Sun, Q., Yu, Q. Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J., Huang, T., and Wang, X. Emu: Genera- tive pretraining in multimodality. In The Twelfth Interna- tional Conference on Learning Representations, 2023.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprint 151 2024.

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z.,, Wang, X., and Wang, I.. Mm-vet: Evaluating large multi- modal models for integrated capabilities, arXiv preprint arXiv:2308. 02490, 2023.
Appendix.
A.1. Response Example
Here, we present our question and the output of the Ope- nAl's model in the box below. The model carried out step-by-step and detailed reasoning based on the question and finally arrived the correct answer.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., al. Mmmu: A massive multi multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 9556-9567, 2024a.
Question and the response by ol
Answer the following single choice question. The last line of your response should be of the following format: 'ANSWER: SLETTER (without quotes) where LETTER is one of Options(e-g. ABCDEF). Think step by step before answering. Question: A sudden environmental change causes recessive allele to become completely lethal. How many generations will it take for its frequency decrease from 10-2 to 10 10-3?
Yuc, X.. Zheng, T., Ni, Y., Wang. Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.00 00000, 2024b.
one of
Zeng, A.. Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, 7., Xu, Y., Zheng, W., Xia, X., et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. 2022.
A. 950
B. 850
C. All other answers are incorrect
D. 900
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang. Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
E. 800
F. 920
Answer:
### Response by solve this, use the fact that for a fully recessive lethal allele (with initial frequency qo), its frequency in the next generation is given by:
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Ga
co++1
+ qr.
Define R., 1. Then from the recurrence above:
Rot1
4a+1
1 + Jn
4n
+
4n
RS +
Thus, R, = Ro + n. Since Ro to decrease from 0.01 0.001:
1 and we want Za.
Rg ã.ol 100.
We need 1 1000, R, 1000. Vn
Hence 1000 = 100 + n, giving n = 900.
ANSWER: D
A.2. CoT Prompt
Here, we present the format of our CoT prompt in following box. As we can see, our CoT prompt mainly uses "Think step by step",, which shows that even simple prompt still
12

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
has positive effect most chat models.
System Prompt for 3Bench
Answer the following single-choice question. The line your response should be of the following format: 'ANSWER: SLETTER' (without quotes) where LETTER, is of the Options (e.g- of ABCDEF). Think step by step before answering. Question: (Question Input}
A. {OptionA]
B. [OptionB]
C. (OptionC]
D. (OptionD)
[OplionE]
(OptionF)
Answer:
### Example:
Answer the following single choice question. The last line of your response should be of the following format: 'ANSWER: SLETTER' (without quotes) where LETTER is ABCDEF). Think step by step before answering. Question: Consider a CMOS inverter driving wire of length L. In the initial design, the on-resistance of the inverter is equal to the total resistance of the wire, the source-drain capacitance of the inverter is equal to the total capacitance of the wire, and the total delay of the inverter and wire is tp. Now, the devices are scaled down using Constant Field Scaling, while the wire is ideally scaled down. Assuming the wire can be modeled using a lumped parameter model, answer the following questions in first-order approximation:
of Options(e-g-
of
(1) Assuming the wire is local wire, and the scaling factors for both process and supply voltage are 2, express the total delay after scaling terms tp.
(2) Now assume the wire is global, and the length of the wire increases inversely with the process scaling, with scaling factors for both process and supply voltage being 2, express the total delay after scaling in terms of tp.
A. (1) 1/3tp (2) 35/6tp
B. (1) 1/2tp (2) 38/6tp
All other answers are incorrect
D. (1) 3/4tp (2) 36/6tp
E. (1) 5/6tp (2) 39/6tp
F. (1) 2/3tp (2) 37 37/6tp
Answer:
A.3. Specific subject distribution
We present the specific subject distributions of RBench-T and MBBnch-M in Table and Table respectively. It be observed that RBench has broad coverage, making it difficult to improve performance on RBench by overfitting to specific subjects.
13

---

RBench: Graduate-level Multi-disciplinar Benchmarks for LLM & MILM Complex Reasoning Evaluation
Table 8: Distribution of Courses by Discipline in RBench-T
Discipline
Specific Subject
Count
Fluid Mechanics
33
Civil Engincering
Structural Mechanics
Surveying
Principles of Process Transport Principles of Chemical Engineering
Chemical Engineering
15
Analytical Chemistry
Advanced Mathematical Economies
Econometrics
3
Intermediate Financial Theory
Economics
Financial Engineering
Game Theory and Mechanism Design
Intermediate Microeconomics
Principles of Accounting
Time Series Analysis
Soil Science
Genetics
16
Biology
Physiology
Biochemistry
15
Heredity
Mathematical Methods in Physics
10
Electromagnetics
11
Optics
23
Quantum Mechanics
Physics
Analytical Mechanics
Electrodynamics
Thermodynamics and Statistical Physics
General Relativity
Basie Physies
29
Group Theory
Mechanics of Materials
Structural Mechanics of Aircraft
Theoretical Mechanics
Aerospace
Fluid Mechanics and Aerodynamics
15
Optimal Control
29
Propulsion Principles and Thermal Fluid Basics
Digital Large Scale Integrated Circuits
20
Microelectronies
Analog Circuits
Signals and Systems
20
Automation
Operations Research
15
Automatic Control Theory
17
Communication and Network
15
Principles of Analog Circuits
4
Electromagnetic Fields and Waves
Fundamentals of Solid State Physics Digital Signal Processing Stochastic Processes
12
Electronic Engincering
Continued on next page
14

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
Table 8 C from previous page
Discipline
Specific Subject
Count
Solid State Physics
Applied Stochastic Processes
26
Fluid Mechanics
11
Principles and Interface Technology of Single Chip Microcomputer
Mechanical Design
Theory of Machines
4
Electromechanical Transmission and Control
4
Mechanical Engineering
Electrical and electronic Technology
Mechanical Vibration
Hydraulic and Pneumatic Transmission
Engineering Thermodynamics
Mechanics of Materials
Theoretical Mechanies
Data Structure
24
Combinatorial Mathematics
Numerical Analysis
Cryptography
17
Computer Science
Automata
Principles of Computer Organization
2
Compilation Principles
Computer Network
11
Operating System
Computer Architecture
Inorganic Chemistry
50
Chemical Thermodynamics
48
Chemical Reaction Kinetics
20
Introduction to Computational Chemistry
Chemistry
Quantum Chemistry
Physical Chemistry
22
Organic Chemistry
Complex Analysis
Analytic Geometry
26
Advanced Calculus
Number Theory
22
Matrix Analysis
36
Partial Differential Equations
29
Mathematics
Mathematical Analysis
Stochastic Differential Equations
Functional Analysis
14
Ordinary Differential Equations
Differential Geometry
Topology
Thermodynamics and Statistical Physics Nuclear Radiation Physics and Detection
Physies Engineering
20
Quantum and Statistics
14
Physical Properties of Materials
Materials
Fundamentals of Materials Science
23
Continued next page
15

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
Table 8 S from previous page
Discipline
Specific Subject
Count
Materials Analysis and Characterization
Finite Element Analysis Basics
Principles of Automotive Power System
Automotive Electronics and Control
Vehicle Engineering
Fundamentals of Control Engineering
Theory of Automobile
3
Automobile Construction
Discrete Mathematics
12
Probability Theory
42
Introduction to Bayesian Statistics
Statistics
Reliability Data and Survival Analysis
Statistical Inference
Principles of Environmental Engineering Science and Engineering
Environment
Water Treatment Engineering Environmental Chemistry
12
16

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
Table 9: Distribution of Courses by Discipline in RBench-M
Discipline
Specific Subject
Count
Materials Mechanics
26
Fluid Mechanics and Acrodynamics
2
Theoretical Mechanics
Aerospace
Aircraft Structural Mechanics
2
Propulsion Principles and Thermal Fluids
2
Optimal Control
2
Digital VLSI
Analog Circuits
Integrated Circuits
Digital Electronics Fundamentals
8
Analog Electronics Fundamentals
6
Transport Process Principles
Chemical Engineering
Chemical Principles
Physical Chemistry
12
Chemical Thermodynamics
Chemical Reaction Kinetics
Chemistry
Inorganic Chemistry
Organic Chemistry
17
Physical Chemistry
2
Data Structures
4
Combinatorics
Discrete Mathematics
Theory of Automata
7
Computer Science
Operating Systems
6
Compilers
4
Computer Architecture
3
Cryptography
Computer Networks
Analytical Mechanics
25
Optics
6
Physics
Electrodynamics
9
Electromagnetism
Basic Physics
Analog Circuit Principles
17
Signals and Systems
3
Digital Signal Processing
Electrical Engineering
Communication and Networks
Electromagnetic Fields and Waves
Solid State Physics
I
Complex Analysis
8
Analytic Geometry
2
Probability Theory
5
Stochastic Processes
3
Mathematics
Analysis
Probability and Statisties
1
Mathematical Analysis
5
Statistics
6
Continued next page
17

---

RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MILM Complex Reasoning Evaluation
Table 9 D- from previous page
Discipline
Specific Subject
Count
Topology
Water Treatment Engineering
4
Environmental Science and Engineering Principles
9
Environmental Monitoring
6
Environmental Engineering
Water Pollution Control Project
4
Environmental Chemistry
1
Solid Waste Treatment and Disposal
2
Genetics
Biology
Biochemistry
Materials Mechanics
37
Quantum and Statistical Mechanics
5
Material Science
Material Analysis and Characterization
Basic Material Science
Operations Research
2
Principles of Accounting
3
Economics
Financial Engineering
2
Intermediate Financial Theories
3
Game Theory and Mechanism Design
8
Electrical and Electronics Technology
5
Mechanical Design
24
Electromechanical Transmission and Control
Hydraulic and Pneumatic Transmission
9
Mechanical Engineering
Mechanical Vibrations
7
Fluid Mechanics
7
Principles of Mechanies
Theoretical Mechanics
11
Civil Engineering
Structural Mechanics
33
Engineering Physics
Engineering Mechanics
23
Automation
Automatic Control Theory
25
Fluid Mechanics
34
Engineering Control Basics
8
Finite Element Analysis Basics
5
Vehicle Engineering
Automotive Electronics and Control
5
Automobile Construction
5
Advanced Heat Transfer
9
Automotive Power System Principles
2
Architecture
Structural Engineering
18


Reviews:
[{"review_id": "djWnDXLWDs", "reviewer": "Reviewer_KtMe", "summary": "This paper introduces R-Bench, a new benchmark designed to evaluate complex reasoning capabilities in both LLMs and MLLMs. The benchmark contains questions in two languages: English and Chinese. There are 1,094 questions in 108 subjects\nfor textual evaluation and 665 questions in 83 subjects for multimodal evaluation. The authors conduct comprehensive experiments of current LLMs and MLLMs.  Several key observations are presented for further development.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, which are crucial to improving the quality of this work. The followings are detailed response to your concerns.\n\n### Q1: \nDespite the broad subject coverage, the total number of collected questions, 1,094 for textual and 665 for multimodal, is relatively limited. \n\n### A1: \nWe were indeed aware of the issue regarding the number of test questions when designing R-Bench. We would like to address this concern from following three aspects:\n\n1. In fact, we originally collected over 15,000 candidate questions—comparable to benchmarks like MMLU and MMMU. However, our rigorous and multi-stage filtering process eliminated a large portion of these samples to ensure high quality, fairness and difficulty. This strict curation results in a smaller final dataset, but we believe it contributes to a more reliable evaluation. In future versions of R-Bench, we plan to include more questions that require advanced reasoning and offer broader topic coverage to further improve the benchmark.\n\n2. If we consider a typical human examination paper to contain around 20 questions, then R-Bench's 1,094 questions would be equivalent to approximately 55 such exams, while the 665-question subset corresponds to about 33 exams. Compared to human-oriented standardized tests such as the ACT (American College Test), SAT (Scholastic Assessment Test), GRE (Graduate Record Examination), and China's GAOKAO, R-Bench significantly increases the number of test questions and the overall coverage.\n\n3. Based on the evaluation results, although R-Bench contains only 1,094 and 665 samples for language and multimodal model testing respectively, the outcomes align well with our understanding of the reasoning capabilities of different models. For example, the OpenAI o1, o1-preview and o1-mini models demonstrate stronger performance than GPT-4o and DeepSeek R1 exhibits stronger reasoning ability than DeepSeek V3. \n\nIn addition, the results also indicate that most models still require significant improvements in handling complex reasoning problems. This provides new insights and guidance for the future development of reasoning-capable models.\n\n### Q2: \nThe evaluation strategy seems to be too straightforward. As the dataset targets difficult reasoning questions, the authors are expected to provide an evaluation strategy for the reasoning rather than only the answer.\n\n\n### A2: \nDuring the development of R-Bench, we also considered incorporating evaluation methods beyond final outcome, such as process-based evaluation. However, due to the inherent uncertainty in reasoning processes, it was difficult to identify a reliable way to assess intermediate steps consistently. As a result, we chose to reflect the focus on reasoning primarily through our question selection process. We applied rigorous filtering involving both domain experts and intelligent models to ensure that the majority of questions emphasize reasoning over knowledge, rather than being heavily knowledge-dependent. In future work, we plan to explore alternative evaluation methodologies as well. We believe your suggestion provides a valuable direction for improving our benchmark.\n\n\n### Q3: \nThe error analysis part provides very few valuable insights. The observations in Section 3.3 could deliver more fine-grained observations into the reasoning process of GPT-4o or o1.\n\n### A3:\nWe agree that it can help improve the quality of our work. We have conducted an analysis of some error examples; however, due to format and length limitations during the rebuttal stage, we are unable to include them here. In our analysis of GPT-4o and o1's errors, we observed that most failures occurred during the reasoning process. These errors stem from various sources, such as calculation mistakes, flawed reasoning strategies, and perception errors (in the case of multimodal models). Notably, the models rarely failed due to a lack of factual knowledge, indicating that they have generally mastered knowledge at a graduate level. In the revision, we will include an error analysis section in Section 3.3 and the supplementary materials to present our findings and insights. We believe this will strengthen our work, and we sincerely appreciate your professional and helpful feedback.\n\n### Q4:   \nDo authors have any plans to expand the dataset size?\n### A4:\nYes, we do have plans to expand the size of the dataset. Our expansion will not be limited to multi-disciplinary reasoning problems; we also plan to extend towards more general-purpose reasoning tasks — for example, complex reasoning scenarios in daily life such as path planning.In future work, we hope to scale the dataset to around 5,000 high-quality questions in future work.\n\n### Q5: \nFor other comments or suggestions\n\n### A5: \nDue to the length limit of the rebuttal, we apologize for not being able to give a detailed response. We will follow your two insightful suggestions in the revision to improve the quality of our work.", "claims_and_evidence": "Yes. The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Strengths of the benchmark:\n1. The benchmark dataset is of high quality. The authors make great efforts to ensure the difficulty of the dataset. \n2. The benchmark contains data newly collected from college courses. \nWeaknesses of the benchmark and its evaluation method:\n1. The scope of the benchmark is very comprehensive, with 108 subjects in the dataset. However, compared with the coverage of the subjects, the total number of the collected questions seems rather limited, with only 1094 and 665 questions for textual and multimodal questions.  For example, in Table 8, there are many sub-subjects with questions less than 5. This could lead to randomness when evaluating.\n2. The evaluation strategy seems to be too straightforward. As the dataset targets difficult reasoning questions, the authors are expected to provide an evaluation strategy for the reasoning rather than only the answer. Since the problems are all sourced from college courses, the knowledge required when reasoning could also be difficult. Therefore, the model could obtain a wrong answer due to a lack of specific knowledge required in reasoning instead of a poor reasoning capability. Simply assessing the final answer omits this situation totally.", "theoretical_claims": "No theoretical claims are introduced.", "experimental_designs_or_analyses": "Weaknesses:\n1. The error analysis part provides very few valuable insights. The observations in Section 3.3 could deliver more fine-grained observations into the reasoning process of GPT-4o or o1. How these models make mistakes is a good topic to discuss in the error analysis section. For example, does the model obtain a wrong answer due to a deduction error, knowledge error, or other error types?", "relation_to_broader_scientific_literature": "The related models and benchmarks have been fully discussed in the related work.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Weakness:\n1. The dataset size seems limited compared with MMLU and MMMU. For example, MMMU contains 11.5K samples for multimodal questions while this dataset contains only over 600 samples.", "other_comments_or_suggestions": "1. More data samples of different subjects could be placed in the appendix to give the readers a better view of the dataset.\n2. Table 1 could add another column demonstrating the dataset size.", "questions_for_authors": "1. I think the authors should clarify how to better evaluate the model prediction. For example, how can errors caused by knowledge be considered and how can the evaluation contain more aspects?\n2. Do authors have any plans to expand the dataset size?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vhTXOya3yl", "reviewer": "Reviewer_Gu7t", "summary": "This work proposes R-Bench, a graduate-level multidisciplinary, multilingual benchmark for both LLM and MLLM reasoning evaluation, which has coverage similar to MMLU and MMMU while reaching the difficulty of mathematical competitions such as AIME@2024. The authors evaluate multiple closed-source and open-source models on R-Bench and then observe both the progress and limitations of current models in reasoning.\n\n## update after rebuttal\n\nThe authors' response address all my concerns. I decide to maintain my score (which is already high) and acknowledge that the contributions of this paper merit publication.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, which are crucial to improving the quality of this work.  In addition, thank you for your positive evaluation of our work, which has been a great source of encouragement for us. The followings are detailed response to your concerns.\n\n### Q1:\nThe related work in Section 4 should be placed earlier in the paper to assist readers in understanding the context.\n\n### A1:\nIn the revised version, we will move the Related Work section to follow the Introduction and add more analysis and discussion to highlight the advantages of R-Bench compared to existing benchmarks.\n\n### Q2:\n In the Introduction section, authors should express \"Comprehensiveness\" more accurately.\n\n### A2:  \nWe want to build R-Bench as a comprehensive benchmark, aiming to cover as many subjects as possible that reflect various aspects of human intelligence such as economics and chemistry. We also hope to provide a more precise or definitive characterization of comprehensiveness. However, it is challenging to offer a clear-cut definition of this concept. \n\nIn the revised version, we will attempt to describe this property using more concrete language — for example, by stating that R-Bench includes over 100 subjects or by proposing a quantifiable definition of comprehensiveness Thank you again for your suggestion which will help us make the paper more rigorous overall.\n\n### Q3: \nWhy should an ideal assessment for complex reasoning emphasize \"difficulty\"? In most cases, difficulty and ease are relative concepts, and it is unclear how to determine whether a problem is difficult. \n\n### A3:\nIndeed, in real-world scenarios, simple problems may outnumber difficult ones. However, in our context, difficulty refers to the level of challenge a model faces when solving a problem.\n\nJust as humans encounter exams of varying difficulty at different educational stages—elementary school, middle school, university—these assessments guide individual development by signaling what knowledge to acquire and which direction to pursue. Similarly, for foundation models, evaluation benchmarks serve as developmental milestones.\n\nFor instance, prior to December 2022 (before the release of ChatGPT), the focus was on whether large models could memorize and reproduce a broad range of knowledge. At that time, MMLU provided a meaningful developmental direction and was considered challenging for models. As model capabilities improved with the release of GPT-4o, Claude 3.5, and others, MMLU became increasingly saturated, i.e., it started to be perceived as easy, prompting the development of more difficult benchmarks like MMLU-Pro.\n\nNow, with the advent of advanced reasoning models such as o1, even MMLU-Pro is showing signs of saturation. This creates a demand for more difficult benchmarks to continue guiding model development.\n\nBased on the above, we define difficulty in terms of the saturation level of current state-of-the-art models. This is exactly what our \"o1 saturation\" metric in Table 1 aims to capture — a quantifiable measure of how much room there is for improvement for top-performing models.\n\nIn this sense, R-Bench not only points to the future direction of model development — emphasizing complex reasoning — but also presents immediate challenges for today’s best models, none of which have yet saturated R-Bench.\n\n### Q4: \nIn Section 3.1, the authors do not provide an explanation for the result. More high-level discussion on the underlying reasons for this outcome is necessary.\n\n### A4:\nThank you for pointing out this issue. In the revised version, we will provide more details about Section 3.1 experiments, including the specific implementation, the voting template used during the user study, and additional explanation and analysis of the results. We believe these additions will help improve the quality of our work and make the methodology clearer and more accessible to readers.\n\n### Q5: \nSection 3.3 shows significant performance variation across disciplines. However, I am unsure whether this experiment is fair. How do the authors ensure that the difficulty of questions in economics and physics is consistent? \n\n### A5: \nThank you for your insightful comment. Indeed, it is challenging to ensure that the difficulty level is consistent across different subjects. In the next version, we will make the description in Section 3.3 more rigorous—for example, by revising the subsection title to emphasize that all subjects still require improvement and none have reached a perfect state. Additionally, we plan to include an analysis of error cases across different subjects under this section, to help readers better understand the types of mistakes models make in each subject.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed benchmark in this work makes sense for the reasoning problems in the literature.", "theoretical_claims": "This paper mainly focuses on benchmark and its construction process, and therefore does not provide theoretical claims.", "experimental_designs_or_analyses": "Yes. This work conducts extensive experiments to validate the value of their benchmark.", "relation_to_broader_scientific_literature": "The benchmark proposed in this paper can provide better support for the research of LLM and MLLM.", "essential_references_not_discussed": "No, I believe the related work is thoroughly discussed.", "other_strengths_and_weaknesses": "Strengths: This paper is well-written and the motivation is clearly defined. The proposed R-benchmark is highly important and critical for reasoning tasks. \n\nWeaknesses: The related work in Section 4 should be placed earlier in the paper to assist readers in understanding the context. The authors have not thoroughly discussed the superiority of R-benchmark compared to other benchmarks.", "other_comments_or_suggestions": "1. Why have the authors placed the related work in Section 4? I believe it would be more appropriate to present the related work in Section 2. \n2. In the Introduction section, the authors should express \"**Comprehensiveness**\" more accurately. For example, economics and chemistry are also subjects that reflect human intelligence, and therefore, we should also evaluate models' performance in these areas.", "questions_for_authors": "1. Why should an ideal assessment for complex reasoning emphasize \"**difficulty**\"? In most cases, difficulty and ease are relative concepts, and it is unclear how to determine whether a problem is difficult. If a benchmark overly emphasizes difficulty, it may overlook the model's performance on simpler problems, whereas in real-world scenarios, simple problems may outnumber difficult ones. I hope the authors can provide a deeper discussion on this matter.\n2. In Section 3.1, the authors conduct experiments comparing reasoning abilities with other benchmarks. Although this experiment validates the stronger reasoning ability of R-benchmark, the authors do not provide an explanation for the result. More high-level discussion on the underlying reasons for this outcome is necessary. \n3. Section 3.3 shows significant performance variation across disciplines. However, I am unsure whether this experiment is fair. How do the authors ensure that the difficulty of questions in economics and physics is consistent? If the difficulty across disciplines is not uniform, for instance, if economics questions are particularly difficult while physics questions are relatively easier, then this finding would be meaningless.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RBmHX53OZP", "reviewer": "Reviewer_PuZC", "summary": "This paper proposes R‑Bench, a benchmark designed to evaluate complex reasoning in language and multimodal models. The dataset spans a wide range of subjects and includes more than 1,000 text-based and 665 multimodal questions. The questions are carefully selected and filtered to ensure that they require deep reasoning rather than simple recall, and are provided in both English and Chinese to test cross-linguistic capabilities. Experiments on various LLMs and MLLMs show that even state‑of‑the‑art models achieve only moderate accuracy, with multimodal reasoning posing a greater challenge than text-only tasks. This benchmark not only highlights current limitations across disciplines and modalities but also provides valuable insights and guidance for future improvements in foundation models’ reasoning skills.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, which are crucial to improving the quality of this work. The followings are detailed response to your concerns.\n\n### Q1:\n\nIn the conclusion, authors claim R-Bench achieves competition-level difficulty, but this is not fully supported by the evidence provided.\n\n### A1:\n\nIndeed, it is difficult to claim that the difficulty of multidisciplinary questions is directly comparable to that of competition problems. \nIn our paper, we aimed to illustrate that for current SOTA models such as o1 and DeepSeek R1, their performance on R-Bench is lower than on AIME@2024. For example, o1 and DeepSeek R1 achieved 74.4% and 79.8% accuracy on AIME@2024, while their scores on R-Bench were 69.0% and 61.2%. This observation served as part of the motivation for our claim.\n\nBesides,  in revised version, we will refine our claim and introduce a clearer qualification: specifically, for current advanced reasoning models such as o1, R-Bench achieves competition-level accuracy comparable to contests such as AIME@2024. \n\n### Q2: \nThe paper does not provide a thorough discussion or comparison of MMMU-pro.\n### A2: \nIn revised version, we will include comparisons with MMMU-Pro in Tables 3 and 4.  In rebuttal stage, we conduct o1 voting and o1 thinking time comparison experiments.\n\nWe randomly sample 30 questions in MMMU-Pro and use o1 model  to compare them with questions in R-Bench-M.The voting results were 22:7:1 (win:loss:tie) in favor of R-Bench, showing a significant preference for R-Bench questions in terms of reasoning demand. In addition, for questions in R-Bench-M, the OpenAI o1 model required an average of 91.7s  to generate a response, whereas for questions in MMMU-Pro, it required only 28.1s on average. This further suggests that R-Bench-M demands more reasoning capabilities.\n\nBesides, we will expand the discussion of MMMU-Pro in related work.\n\n### Q3: \nThe benchmark does not report the performance of human experts, leaving an important baseline unexplored.\n\n### A3: \nWe agree that human expert performance is an important baseline. However, establishing such a baseline is extremely challenging. The main difficulty lies in recruiting domain experts from different fields to solve thousands of high-difficulty questions, which is practically demanding. Moreover, a single test run is not statistically meaningful; reliable baselines require averaging over 5 runs on thousands of questions.\n\nTherefore, we plan to conduct this baseline on 3–5 selected subjects, such as computer science, to better reflect the gap between models and human experts.\n\n### Q4:\nIn Section 3.1, 30 questions is a small sample size that limits the reliability of the experimental results.\n\n### A4: \nFor the reasoning model scoring, we extended the number of evaluated questions from 30 to 300. The experimental results were consistent with those reported in the paper. In terms of textual reasoning, R-Bench takes approximately 7 times longer than MMLU. For multimodal reasoning, R-Bench requires roughly 4 times the reasoning time compared to MMMU. We will incorporate this result in the revised version to make our experimental results more convincing. \n\n### Q5: \nThe failure patterns of LLMs and MLLMs are underexplored.\n\n### A5: \nWe have conducted an analysis of some error examples; however, due to format and length limitations during the rebuttal stage, we are unable to include them here. In our analysis of GPT-4o and o1's errors, we observed that most failures occurred during the reasoning process. These errors stem from various sources, such as calculation, flawed reasoning strategies, and perception errors (for multimodal models). Notably, the models rarely failed due to a lack of  knowledge, indicating that they have generally mastered knowledge at a graduate level. In the revised version, we will include an error analysis in Section 3.3 and Appendix to present our findings.\n\n### Q6:\nHow was the 2,000 reasoning token threshold set? What's the pre-filter distribution and o1 cost? \n### A6: \nThe 2,000-token threshold was a heuristic decision. We found that 2,000 tokens served as a reasonable threshold to help distinguish \"reasoning-oriented\" questions from \"knowledge-based\" ones. In the revised version, we will include a distribution histogram to illustrate the original reasoning token distribution.. The cost of API call for Openai o1 is about 6,000 USD.\n\n### Q7:\nWhat criteria guided the manual adjustments to ensure sufficient numerical gaps? Additionally, was “all answers are incorrect” ever used as the correct answer? \n### A7: \nWe believe that the other answer options should differ from the correct answer by at least 10% or 0.5. This is a heuristic guideline to prevent models from being penalized due to minor approximation errors during the reasoning process.\nRegarding the second question, some \"all answers are incorrect\" options were used as the correct answer.", "claims_and_evidence": "In the conclusion, the authors claim that the benchmark achieves competition-level difficulty comparable to contests such as AIME@2024. However, this assertion is not fully substantiated by the evidence presented in the paper. Although the dataset is derived from graduate-level materials, there remains a gap between the difficulty of graduate-level content and that of olympiad-level challenges.The question samples provided in Fig. 3 indicates that the problems do not approach the complexity expected of olympiad-level tasks. Consequently, the claim appears overstated and lacks clear justification.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A. No theories involved in this paper.", "experimental_designs_or_analyses": "The experiment design is solid.", "relation_to_broader_scientific_literature": "The development of such a broad and difficult benchmark thus fills a gap in the literature, providing a tool that can more comprehensively evaluate both the retrieval of knowledge and the capacity for complex reasoning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is written in a clear and straightforward manner, with all key steps of data collection and benchmark construction thoroughly documented.\n2. The experimental evaluation is comprehensive, testing a wide range of models. The observed shortcomings in multimodal reasoning capabilities align with findings from previous studies.\n3. The benchmark’s design—spanning a broad knowledge base and requiring deep reasoning—provides a robust testbed for models. Its multilingual and multimodal versions strengthens its utility and relevance.\n\nWeaknesses:\n1. The paper does not provide a thorough discussion or comparison of MMMU-pro, a more robust evolution of MMMU. Including MMMU-pro in experiments (e.g., in Tables 2–4) would strengthen the analysis.\n2. The benchmark does not report the performance of human experts, leaving an important baseline unexplored.\n3. In Section 3.1, only 30 questions are sampled for the win rate comparison, which is a small sample size that limits the reliability of the experimental results.\n4. The failure patterns of LLMs and MLLMs are underexplored. More detailed categorization of errors, such as lacking knowledge, wrong  reasoning steps can provide valuable insights into developing more capable models.", "other_comments_or_suggestions": "1. In Appendix A.3, there are two \"table 8\" in line 654.\n2. The caption of figure 6 is confusing. I suggest the authors rewrite it.", "questions_for_authors": "1. In the model screening process described in Section 2.3, how was the 2000 reasoning token threshold determined? Can you provide details on the distribution of reasoning tokens in the dataset before filtering in Step 4? Can you also estimate the computational cost of prompting o1 with the entire dataset to obtain the reasoing token number?\n2. Regarding Section 2.4, can you elaborate on how the answer options were constructed? Specifically, on line 257 you mention that \"we manually adjust the options to ensure a sufficient numerical gap between them.\" What criteria or standards guide these adjustments? Additionally, when including the \"all answers are incorrect\" option, were there cases where this was intended to be the ground truth answer?\n3. How do human experts perform in the benchmark?\n4. In the conclusion, you claim that R‑Bench reaches the difficulty level of olympiad-level competitions. However, given that the dataset is derived from undergrad and graduate courses, which are generally more focused on in-depth knowledge rather than competition-level challenge, can you provide further justification or evidence to support this claim?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T87ZbhvSB6", "reviewer": "Reviewer_rJWX", "summary": "This paper proposes a new benchmark called R-Bench, with features of Comprehensiveness, Difficulty, Multimodality and Multilingualism.\nThe paper also conducted various experiments on current mainstream LLMs and MLLMs using R-Bench.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the selfless efforts and constructive comments for improving the quality of this work. The followings are detailed response to your concerns.\n\n### Q1: \nThe paper doesn't fully explore whether R-Bench needs more reasoning abilities than current benchmark. \n### A1: \nEvaluating the reasoning demands of a dataset on models is inherently challenging, and there is no standard quantitative metric for doing so. In this work, we employed both expert scoring and model-based evaluation to assess the reasoning requirements of our dataset.\n\nRegarding the dataset biases, we understand this as referring to biases across different subjects. To address it, we conducted experiments focusing only on 50 mathematical questions from both R-Bench and MMLU to control for subject-specific variation in reasoning demands. The results showed that the average number of reasoning tokens for R-Bench was 6,867.1, while for MMLU it was 1,011.3, indicating a difference of approximately 6.8 times.\n\nIn addition, we asked the OpenAI o1 model to vote on which question required more reasoning. The voting results were 43:5:2 (win:loss:tie) in favor of R-Bench, showing a significant preference for R-Bench questions in terms of reasoning demand.\n\n### Q2: \nMore detailed breakdown of failure modes would be helpful when evaluating LLMs/MLLMs.\n\n### A2: \nWe have conducted analysis of some error examples. Due to format and length limitations during the rebuttal stage, we are unable to include them here. In our analysis of GPT-4o and o1's errors, we observed that most failures occurred during the reasoning process. These errors stem from various sources, such as calculation, flawed reasoning strategies, and perception errors. Notably, the models rarely failed due to a lack of knowledge, indicating that they have generally mastered knowledge at a graduate level. In the revised version, we will include an error analysis section to present our findings.\n\n### Q3: \nFor Table 7, which analyze the effect of CoT, it would be more convincing to include additional reasoning models beyond o1-mini.\n\n### A3: \nThank you for your constructive feedback. Due to experimental cost, we only added DeepSeek R1 without CoT. It scored 60.7% on R-Bench, just 0.5% lower than the CoT-enabled version (61.2%). We observed that this performance drop is smaller than that of chat models, where the decrease often exceeds 1% or even 2%.\n\nThis is an interesting observation, and we plan to deeply explore CoT's impact across more reasoning and chat models in future work.\n\n### Q4: \nIn Section 3.1, Paragraph 2: Is pairwise comparison sufficient to assess overall benchmark reasoning demand?\n\n### A4: \nAs far as we know,  there is no clearly defined metric for assessing reasoning abilities, whether for humans or foundation models. A single metric, such as the expert pairwise comparison, is one dimension, but it is not sufficiently accurate to determine which benchmark demands higher reasoning capabilities. Our goal is to provide a comprehensive assessment, presented in Tables 2–4, to reflect, across multiple dimensions, that R-Bench imposes higher reasoning demands compared to other benchmarks, both in terms of expert scores and model performance.\n\n### Q5: \nFor Section 3.1, Paragraph 3, how these two aspects impact o1 voting in Table 3?  (eg. using a weighted sum?)\n### A5: \nWe apologize for the confusion caused by Section 3.1, Paragraph 3 in our manuscript. The o1 voting and reasoning tokens are separate and presented in Table 3 and Table 4, respectively. For Table 3, we randomly paired one R-Bench and one MMMU question, then asked o1 which required more reasoning. R-Bench scored 1 point if selected, otherwise MMMU. This was repeated for 30 pairs to compute win rates.\n\n### Q6: \nSection 3.1, Paragraph 3: Reasoning tokens and reasoning time are not linearly correlated. Additionally, longer R-Bench problems may naturally yield more tokens, making the comparison potentially unfair.\n\n### A6: \nTo strengthen our results, we conducted a reasoning tokens experiment using 50 math questions from both R-Bench and MMLU. OpenAI o1 generated an average of 6,867.1 tokens for R-Bench and 1,011.3 for MMLU—a 6.8× difference. While problem length may play a role, we believe the gap mainly reflects R-Bench’s higher reasoning demands.\n\n### Q7:  \n\nFor Section 3.3: How to compute Consistency and Accuracy in Figure 5&6?\n\n### A7: \n\nFigure 5 shows models' consistency on English and Chinese versions of identical difficult questions. For a question, if both the English and Chinese versions of the question are answered correctly or incorrectly, we increment the consistency variable c = c + 1, Finally, consistency = c / total questions.\n\nFor the accuracy in Figure 6, to obtain this accuracy, we grouped the questions by different departments. We then separately calculated GPT-4o's accuracy on the questions from each department to generate Figure 6.", "claims_and_evidence": "Yes. All claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes. The evaluation criterias make sense to evaluate the authentic abilities of LLMs/MLLMs.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "Yes. I've checked the soundness/Validity of experimental designs and analyses.", "relation_to_broader_scientific_literature": "The paper proposed a new benchmark to evaluate LLMs and MLLMs, which is based on prior benchmarks like MMLU and MMMU. R-Bench made improvements on Comprehensiveness and Multilingualism.", "essential_references_not_discussed": "Essential paper are cited and discussed in the paper.", "other_strengths_and_weaknesses": "Strengths: \n1. This paper is well-structured and easy to follow for readers.\n2. This paper is comprehensive and explain the methodology with great details, e.g. Section 2 shows a lot details in describing the process of data collection.\n3. The benchmark proposed in this paper is both multi-disciplinary and multi-lingual, incorporating the strengths of existing benchmarks.\n\nWeakness: \n1. The paper doesn't fully explore whether R-Bench needs more reasoning abilities than current benchmark. In this paper, authors use thinking tokens/ thinking time to judge, but they haven't taken dataset biases etc into account.\n2. This paper focus heavily on the Accuracy in the benchmark. More detailed breakdown of failure modes would be helpful when evaluating LLMs/MLLMs.", "other_comments_or_suggestions": "For Tab.7 and Section 3.3, which are discussing The effect of CoT,as mentioned in your paper above, do more test on Reasoning Models may be better. (In Tab.7, the only Reasoning Model is o1-mini.)", "questions_for_authors": "1.For Section 3.1, Paragraph 2: Is the pairwise comparison approach sufficiently accurate to determine which benchmark (considered in its entirety) demands higher reasoning capabilities?\n\n2.For Section 3.1, Paragraph 3: You mentioned two aspects, on the one hand based on the reasoning tokens and on the other hand using o1 to judge correctly. My question is how these two aspects impact o1 voting in Table 3?(eg. using a weighted sum?)\n\n3.For Section 3.1, Paragraph 3: Reasoning tokens and reasoning time cannot be simply considered to have a linear relationship. (When calling the API service, the GPU memory bandwidth and computing power of the corresponding service will affect the generation speed.) Furthermore, If the length of the R-Bench problem is longer than the baseline problem, the generated reasoning tokens will most likely increase, so the a comparison might not be fair.\n\n4.For Section 3.3: How to compute Consistency and Accuracy in Figure 5&6?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes R-Bench, which is a reasoning benchmark targeting graduate-level tasks across multiple disciplines and languages. The benchmark is designed for both LLM and MLLM evaluation. It achieves coverage similar to MMLU and MMMU, while introducing more challenging problems inspired by math competitions. The authors test a series of open-/closed-source models, which provides key insights into current reasoning performance and limitations.

During the initial review phase, all four reviewers acknowledged that the paper offers meaningful insights to the foundation model research community. At the same time, they raised several concerns, regarding the scale and construction of the dataset, as well as comparisons with more recent baselines. Following the rebuttal, all reviewers submitted acknowledgments and reached a consensus to recommend acceptance.

AC has carefully reviewed the paper and the reviewers’ comments, and agrees with the overall recommendation. The authors are encouraged to incorporate the reviewers’ constructive feedback in the final version to further strengthen the clarity and impact of this work.
Decision: Accept (poster)