CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery
Authors: Xiaoshuai Song
Muxi Diao
Guanting Dong
Zhengyang Wang
Yujia Fu
Runqi Qiao
Zhexu Wang
Dayuan Fu
Huangxuan Wu
Bin Liang
Weihao Zeng
Yejie Wang
Zhuoma GongQue
Jianing Yu
Qiuna Tan
Weiran Xu
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
evaluation
computer science
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) have demonstrated significant potential in advancing various fields of research and society. However, the current community of LLMs overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first multilingual (English, Chinese, French, German) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 10K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities. Our project homepage is available at https://csbench.github.io/.

Published as a conference paper at ICLR 2025
CS-BENCH: A COMPREHENSIVE BENCHMARK FOR LARGE LANGUAGE MODELS TOWARDS COMPUTER SCI- ENCE MASTERY
Xiaoshuai Song, Muxi Diao; Guanting Dong! Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, Weihao Zeng. Yejie Wang, Zhuoma GongQue, Jianing Yu, Qiuna Tan, Weiran Xu Beijing University of Posts and Telecommunications, Beijing, China (songxiaoshuai, dmx, xuweiran } @ bupt.edu.cn
ABSTRACT
Large language models (LLMs) have demonstrated significant potential in advanc- ing various fields of research and society. However, the current community of LLMs overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first multi- lingual (English, Chinese, French, German) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 10K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, conduct comprchensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplemen- tation and CS-specific reasoning. Further cross pross-capabilityy experiments show high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a comerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities. Our project homepage is available at httpsss//chencchhithubb...
INTRODUCTION
Large language models (LLMs), exemplified by ChatGPT (OpenAl 2022) have emerged a signif- icant milestone in artificial intelligence (AI), demonstrating potential far beyond natural language
processing (NLP) (Zhao et al. 2023: Chang et al. 2024). These models are increasingly impacting
diverse fields including education, industry, and scientific research (Guha et al.] [023] Guo et al.
2023 Xiao al.) 2023 Huang et al.) 2023a Zhou et al. 2023a] Zhao et al. 2024 [Zhang et al.
2024a). As computer science (CS) continues to be the cornerstone of modern information technology.
from the advent of electronic computers to today's ss Al advancements (Denning 2000) Campbell-Kelly
al. 2023). key challenge lies in enabling LLMs to better utilize CS knowledge to serve humanity
in the evolving intelligent era (Donadel et al. 2024
Murtuza 2024 Marques et al. 2024).
Understanding the performance of LLMs in computer science is fundamental to the research and application of LLMs within the field. Despite studies like MMLU and C-Eval (Hendrycks et al.
2021a Huang al.) [203b] Liu et al. 2023a Li et al. 2024 Gu et al., 2024) covering wide range of fields including CS, their broad scope implies that CS is merely a component within the multiple categories of science and engineering, overlooking the importance of thoroughly evaluating the
CS field. Moreover, such evaluation result can further guide the development of LLMs, offering
Equual Contribution
"ooresponniig Authors, and Weiran Xu the primary one.

---

Published as a conference paper at ICLR 2025
Central Processing Unit Main Memory
Eustr
Bius
Oagus Nyston
Haear Loe
Soath theen
Temporal and Spatiol Complexity of Algorithms
Stack and Queue
Huffman Tree
Logic Unit Instruction Addressing
String
Minimum Sponning
Tree Algonithvn
Cache
Sysions
Exception and Interrupt Mechanism
Sturage
Frers
Linked Lists
Binary Tree
I/0 Interface CPU Architecture
Instruction Cycle and Pipeline
Dato Reprrnnticc
Computee
Organization
Data Stracture and
Algeztilim
Graph
Sesrching
KMP Pottern Motching Algorithn
Hash Search
Depth-First Search
Red-Black Tree
Multiprocessor Concept
Overvion
Sarting
Quick Sort
Concurrency and Asynchronous
Overvion
Operatingg System m
CS-Bench
Netwark
Arctuleetare
Overnicw and
Bandwidth
Throughput
Virtual Machine Keree Subaystem
Physkca Layer
Round-Trip Time
Batch Processing System
Deadlock
Proctsso
Computer
Daw Lisk Layse
OSI Seven-Layer Model
Process and Thread Processor Scheduling I/O Spooling
Memmar sere
Nrrwork Laver
Network NetworkInterffun Interface Gord
Disk Scheduling
World Wide Web LAN and WAN
Virtual Memory Management
IPv4 and IPv6
UDP and TCP
Process Synchronization and Mutual Exclusion
Router
HTTP
Mobile IP
CS- ES-Beench
Sukject
Cagatas
Lengagge
Spil
Agrnttin
Madgee
Fegisk
Knowledge
6152
Valid
944
Comguner Netek
2488
Awurdea
Chiase
Coepater Oe piristike
1624
Slank
1012
Prench
2419
Reasening
Test
N732
Operatin Syrten
2164
Opin-mnk4
755
Gereaa
2419
Figure 1: Overview diagram and statistics of CS-Bench.
practical insights to advance the corresponding capabilities. Recently, a series of studies have devoted
on actively assessing and analyzing the capabilities of LLMs in mathematics, coding, and logical
reasoning (Frieder et al. 2023 Collins et al. 2023] Wu et al. 2023 Yuan et al. 2023bb Dong et al.
2024 Liu al. [024 Zhang et al. 2024bb Lin al. 2024 Liu et al. 2023b Saparov et al.) 2023 Xu
et al. 2023 Wu et al. 2024). Unfortunately, efforts on LLMs in cross cross-capabilityy evaluation is quite
scarce. Considering the intersection of computer science with coding, mathematics, and reasoning abilities, we have grounds to believe that cross-cas coss-capability research and analysis in CS can effectively propel the comprehensive development of the LLM community. Here, we are particularly interested in two research questions for evaluating LLMs' proficiency in computer science field:
RQI: How do LLMs perform the field of computer science and what are the challenges and potential directions for improvement?
RQ2: What are the relationship between the abilities of LLMs in computer science, mathematics, and code programming?
As the bedrock for exploration, we introduce CS-Bench, the first benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench features high-quality, diverse task forms, varying capacities, and quadrilingual evaluation. Firstly, CS- CS-Bench comprises approximately 10K carefully curated test items spanning 26 sections across 4 key CS domains. Unlike conventional benchmarks consisting solely of multiple-choice (MC) questions (Hendrycks et al.) 2021a] Huang al. 2023b Li al. 2024), CS-Bench includes 4 tasks: multiple-choice, assertion, fill-in-the-blee blank (FITB), ), and open-ended, to better simulate real-world scenarios and assess the robustness of LLMs different task formats. In addition to knowledge-type questions assessing LLMs mastery of CS knowledge, reasoning type questions further evaluate LLMs' ability to apply CS knowledge for reasoning. Lastly, by supporting evaluation in English, Chinese, French, and German, CS-Bened allows assessment of LLMSs ability to tackle CS challenges in various language contexts.
In response to RQ1, we evaluate over 30 mainstream L.LMs on CS-Bench. Our main findings are: (1) CS-Bench effectively differentiates the capabilities of LLMs in the CS field while also posing challenges to the best-performing GPT-40/OpenAl-ol. (2) LLMs exhibit a consistent logarithmic growth pattern in scale and linear growth pattern in scores on the CS-Bench. By establishing the scale-score fitting function, smaller models can be used to predict and guide the development of larger-scale models. (3) Further error type analysis indicates that the primary reason for the limited performance of LLMs is the lack of domain knowledge, and the CS-specific reasoning is difficult to achieve merely by enhancing general reasoning abilities, necessitating targeted reinforcement.
2

---

Published as a conference paper at ICLR 2025
In response to RQ2, we perform detailed analysis of the relationship of General LLMs' ability in three domains: mathematics, coding, and computer science, as well as the performance of code- and math-specific expert LLMs on CS-Bench. We observe consistent trends in the overall performance of the general LLMs across CS-Bench and scores in benchmarks related to mathematics and coding, indicating a strong correlation between LLM'' computer science proficiency and its mathematical and programming abilities. Furthermore, despite decline in general capabilities, some expert LLMs still exhibit improvements in certain areas of CS, such data structures and algorithms, with more pronounced knowledge and reasoning capabilities evident in supplementary maallrr-scale models.
To summarize, our contributions are as follows:
We introduce CS-Bench, the first benchmark dedicated evaluate the performance of LLMSs in the field of computer science. CS-Bench supports four languages, covers four key areas with 26 subfields, and includes diverse range of task formats.
Utilizing CS Bench, we conduct a comprchensive evaluation of mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for fallures in existing L.LMs and highlight directions for improvement.
We conduct exploratory experiments on LLMs' cross-ability and find a strong relationship between their CS proficiency and mathematical and programming abilities. Moreover, the expertise in mathematics and programming expert LLMs can improve performance specific CS subfields.
2
CS-BENCH
2.1
DESIGN PRINCIPLE
The objective of CS-Bench is to robustly assess the knowledge and reasoning capabilities of LLMs in different linguistic contexts within the field of computer science. To this end, our benchmark adheres to the following guidelines: (1) Coverage of key domains: covers key areas of CS with (2) Díverse task forms: questions vary in format to simulate diverse real-world user queries. (3) CS-specitic reasoning: it evaluates CS logical and arithmetic reasoning in addition to CS knowledge. (4) Multilinguality support: it supports assesses LLMs performance in different language environments. Based on these criteria, CS-Bench focuses on quadrilingual evaluation in English, Chinese, French, and German, covering four domains: Data Structure and Algorithm (DSA), Computer Organization (CO), Computer Network (CN), and Operating System (OS). Twenty-six fine-grained subfields, diverse task forms, and divisions of knowledge and reasoning are further developed to enrich the dimensions of assessment and simulate real-world scenarios.
finer subfields for specificity.
2.2 DATA COLLECTION
Data Sources. Diverse data sources key to achieving the sample diversity of CS-Bench. Our raw data originates from three sources: (I) Com- puter science-related questions obtained from pub- licly available online channels, such professional exams and practice tests. (2)Knowledee-type ques- tions obtained through the initial manual extraction and subsequent adaptation of blog articles from var- ious computer-related websites. (3) Construction of teaching materials and examination papers authorized by the authors' institutions. The latter two cate-
Table 1: Comparison of perplexity (PPL)
across evaluation datasets. The PPI. of En- glish and Chinese datasets is calculated on Llama2-7B and Qwen .5-7B. "MC" denotes
multiple-choice, and "ALL" denotes all tasks.
English Dataset
PPL
Chinese Dataset
PPL
TruthfulQA (MC) MMLU
7.73
C-Eval
11.47
9.54
CMMLU
13.62
CS-Bench (MC)
11.86
CS-Bench (MC)
13.31
CS-Bench (ALL)
13.3
CS- (ALL)
16.95
gories constitute the vast majority (over 70%) of the data, and these data are not directly exposed on the internet, effectively reducing the likelihood of LLMS encountering these questions during pre-training. We compare the perplexity (Jelinek et al. 1977) of models on CS-Bench English and Chinese datasets with several other prominent evaluation datasets in Table T In both English and Chinese, the perplexity of CS- Bench is comparable to or even higher than that of other datasets, further indicating the high quality of CS-Bench samples and the rarity of data leakage instances.
Mooe collection and processing procedures can he found in Appendix
3

---

Published as a conference paper at ICLR 2025
Data Processing. The data processing relies on team composed of five members, each holding bachelor's s degree in computer science and receiving appropriate compensation. Initially, we parse questions and answers for each sample from the data sources either automatically manually. Subsequently, we manually label questions with knowledge-type or reasoning-yy tags depending on whether in-depth reasoning and calculation to collect explanations from the data sources whenever possible; otherwise, we handle them through cross-annotation and verification among team members. We first construct Chinese data, then translate it into other languages using GPT-4, supplemented by manual checks, to create multilingual data. Finally, we conduct thorough manual checks on the entire dataset to ensure quality. As this benchmark pertains to objective knowledge and reasoning in the field of computer science, the annotation content not influenced by regional or cultural differences among annotators.
required. For reasoning-type questions, attempt
Statistics. CS-Bench is an evaluation benchmark supporting quadrilingual assessment, encompass- ing a total of 26 subfields across 4 domains, with cumulative total of 9676 samples. These samples encompass various task formats including multiple-choice, assertion, ended questions. Besides, CS-Bench assesses both knowledge-type and higher-order reasoning-type questions, with each reasoning question accompanied by an explanation. To validate the effectiveness of models, we randomly sample 10% of the data for validation, using the remaining 90% for testing. The statistics of CS- CS-Bench are shown in Figure I with detailed exposition provided in Appendix C
and open-
3 EXPERIMENT
3.1 EXPERIMENTAL SETUP
Evaluation Protocols. Due to the diverse task formats CS-Bench, we first design question templates for each task type. For comprehension tasks (MC and Assertion), we use regex to match LLM's predictions and then calculate their accuracy against the ground-truth answers. For generation tasks (FITB and Open-ended), due to the diversity of ground-truth answers, LLM's predictions by GPT-4 using standard answers in CS-Bench as references. In detail, FITB questions are scored as either 0 or 1, while the score range for Open-ended questions is 1-10, which is then linearly mapped to a range of 1.1 to 1. Finally, scores are weighted based on the quantity of each type to derive the ultimate overall score. It worth emphasizing that while using GPT-4 for scoring generation tasks may introduce a certain threshold for evaluation, its primary purpose is to simulate diverse task formats in real-world scenarios. Therefore, we encourage isolating comprehension tasks from CS- Be-Bencch to facilitate automatic evaluation with no need for GPT-4. We provide the details of the evaluation setup in Appendix D where we also verify the validity of GPT-4 scoring through its consistency with manually scored results, and the effect of different scoring models.
score
Models. For open-source models, we selected Gemma-2B/7B (Team et al. 2024), Llama2-
7B/13B/70B (Touvron et al. 2023), Llama3-8B/70B (Meta 2024). Llama3.1 1 1-40SB (ubeyy al.
2024). ChatGLM3-6B (THUDM (203). Baichuan2 (v2.0))7B/13B (Yang et al. (023). InternLM2-
7B/20B (Cai et al., 2024). Qwen I.. /BB4////4BB/22B//// 10B (Alibabal 2024). Mistral-7B (v0.2)
(Tiang et al. 2023), Mixtral-8x77 7B (v0.1) Diang et al.) 2024). Mistral-Large- 123B (MistralAl 2024). and DeepSeckLLM-
(Bi et al. 2024). For closed-source commercial models, we utilized PaLM-2 2-almm2-chat-bisom (Anil et al. 2023). C (Anthropic 2023), Claude-3 (opus) (Anthropic 2024a). Claude-3.5 (Sonnet) (Anthropic 2024b), as well as GPT-3.5, GPT-4 (0125 version) (Achiam al. 2023). GPT-4o ciatraction-tolooinng
2024a) and -oarnAA---mmmm/pevieee (0912 ver-
sion) (OpenAl 2024b To ensure the instruction following abilities, we employ the official chat or
instruction-tuned versions for all models. Details on these models are provided in Appendix D.4
3.2
MAIN RESULTS
Table presents the overall results of all foundation models directly answering questions under the zero-shot setting 3 In summary, the overall scores of models range from 39.86% to 72.29%. demonstrating CS Bench's effectiveness in distinguishing between the abilities of various models
Duue to the unique reasoning form of OpenAl-ol, we exclude it from Section 3.2
analyze it in Section 3.3
Due space constraints, the results and analyses in other languages are provided in Appendix P.3
4

---

Published as a conference paper at ICLR 2025
Table 2: Zero-shot scores (%) of LLMs across domains on CS-Bench (EN), where "Klg" denotes knowledge-tyye -apgeeetteeeeeeeo "Rng" denotes reasoning- typp,, and "Avg" denotes Average. The random scores are weighted as follows: 25% for MC, 50% for Assertion, 0% for FITB, and 10% for Open-ended The
highest scores for open-source and closed-source LLMs is marked in green and blue respectively.
Model
Data Struc & Algo
Computer Organization
Computer Network
Operating System
Overall
Kig
Rng
Avg
Klg
Rng
Avg
Kig
Rag
Avg
Klg
Rng
Avg
Klg
Rag
Avg
Random
28.04
24.63
26.65
26.57
25.24
26.13
26.34
22.49
24.98
29.06
24.23
27.27
27.4
24.12
26.2
Open-source IJM (Seatesc (OR)
Gemma-28
43.20
45.
26.38
32
35.39
27.59
5-1B
58.76
36.56
49.72
52.31
33.88
46.23
52.70
33.97
46.11
40.03
38.52
39.17
51.18
35.70
45.54
ChaGGLM3-6
51.10
34.08
44.17
48.11
32.75
43.04
51.15
32.66
44.64
43.57
37.03
41.14
+88.63
34.07
43.33
Llama2-7B
51.51
32.61
43.82
48.89
31.82
43.26
46.72
30.75
41.10
41.04
26.26
35.55
47.15
30.48
41.08
DeppeckLLMM-7
28.94
45.23
52.09
52.43
38.06
50.87
31.11
Buichunn--7B
53.11
34.95
45.72
45.10
38.67
42.98
51.26
34.27
45.28
43.47
33.63
39.82
48.29
35.33
43.57
Gemma: 7B
59,53
35.18
49.62
49.97
33.27
44.46
60.87
37.09
$2.50
48.67
34.25
43.31
54.90
35.02
47.56
1.
59,90
35.28
49.88
55,21
42.73
51.09
61.56
43.02
55.04
52.01
78
47.47
57.5
40.08
$1.05
IntemLmm2-7B
59.57
40.92
51.98
50.94
58.31
39.77
Misml-78
63.24
34 86
$1.69
57.52
38
$1.30
61.48
44.92
55.65
51.66
43.79
48.73
58.63
40:44
52.01
Llama3-8B
66.25
37.29
54.46
55.38
40.67
50.53
62.21
53.02
$8.98
55.26
49.34
53.06
59.75
44.97
54.37
Open-source
(Sculet-108)
Lhama2-13B
51.74
35.00
44.93
51.81
36.
46.66
53.03
37
47.74
48.12
32.36
42.27
51.31
35.16
45.54
Baichuan-130 130
54.82
33.39
46.10
50.50
39. 59.52
46.88
55.87
42.21
$1.06
48.44
34.73
43.35
52.53
37.44
47.03
Qwenl.5-14B
64,95
46.74
57.54
60.06
45. 15.55
55.28
68.66
52.91
63.12
56.56
51.48
54,67
62.79
49.18
57.83
mm2emLm2:200
64.13
50.88
60.81
Qwenl.5 5-32BB
69.70
$1.19
62.17
67.63
52.91
62.78
69.23
58.74
65.54
60.06
56.21
58.63
66.87
54.72
62.45
Mistral-&x7BB
70.94
40.50
58.55
66. 88
42.06
58.70
67.49
52.86
62.34
57.56
$1.65
55.37
65.91
46,66
58.90
DeepseelilLMM.3
69.70
44.17
59.31
63.59
39.15
55.53
69.04
50.25
62.43
57,86
50.11
54.98
68.23
45.96
58.22
Llmmu2-70R
64.28
41.51
55.01
56.35
40.85
51,24
61.99
43.07
55.33
51,79
41.15
47,84
58.73
41.68
52.52
Llama3-708
75.72
53.03
66.48
71.45
$1.09
64.74
74.78
63.02
70.64
63.77
58.08
61.65
71.65
56.26
66.08
Qwenl .5-728
72.71
50.69
63.75
69.28
54.12
64,28
71.97
66.73
70.13
63.96
59.62
62.35
69.63
57.75
65.31
Qwent 15-101
65.16
64.82
60.91
Mistral-Lage-122 123B
79:13
59.82
71.15
74.21
63.76
70.76
77. 189
68. 68.19
74.54
66.92
66
66.61
74.84
61.37
71,03
Llama3. 1-4058
77.63
58.85
69.99
76.57
57.58
70.31
78.50
69. 90
75.47
68.86
64.75
67.33
75.64
62.81
70.96
Closed Sourree LLM
70.07
57.41
63.81
41.91
65.11
64.85
44.01
Claxde-2.1
68.39
44.54
59.68
62.09
50.24
58.18
66.5%
.52.81
61.74
53.93
50.55
52.67
62.97
49.42
58.04
Clinde-3-
77.53
52.25
67.24
72.53
64.12
69.76
75.08
68.69
72.83
64.36
62. 80
63.78
72.57
61.75
68.63
GPT:335
78.53
56
70:73
75.40
59.33
70.06
77.38
67
73.95
74.85
62.66
70.41
Claude-3.5- Sonmmmc
77.16
58.07
69.39
75.13
61. 76
70.72
77.92
71 44.46
75.65
69.55
64.75
67.76
75.13
63.97
71.07
GPT-4o
81.51
80
71.86
75.60
58.61
70.00
80.57
71.76
77.47
69.35
68.65 68
69.10
76.95
64.15
72.29
Overall
Liara 73
Liama 120
GFF35
Mistral
Liama3-700
05
D5A kzowlcdgg
Openeeded
MC knoweddge
Benning Scat Ph
63
65
CD
ro3001ng
Assentio
20
g
CD
Acsention
30
Kaowledge Scare (80
40
53
00
70
05 (a) Comparison between domains
DIEA
Openendad (b) Comparison beween tooks
MC
Figure/ 2: knowledge-type and reasoning- type scores across all models.
The distribution of
Figure 3: Comparison of representative LLMs' scores across different domains and tasks.
in the field of CS while also posing significant challenges to the best-performing existing models. Subsequently, we conduct a comprehensive analysis of the experimental results from various aspects including Foundation Models, Knowledge & Reasoning, Domains, and Task Formats.
Firstly, the closed-source models Claude3. 3/G// represent the highest standard on CS-Bench, with a proficiency exceeding 70%. Secondly, leading open-source models (Mistral-Large-I1 Largee-BBBB-- and Llama3. 1-405B) have significantly narrowed the gap between open-sourge and closed-source models. On one hand, both models exhibit a substantial increase in parameters; the other hand, current closed-souree models are tending towards balance between performance and efficiency, exemplified by GPT-4o-mini and Gemini Flash. Thirdly, newer models demonstrate significant improvements compared earlier versions. For example,
Comparison between Foundation Models.
5

---

Published as a conference paper at ICLR 2025
Type
Model
Score
Reasoning Completion Tokens
GPT-4o
76.95
8/6.7
Knowledge
OpenAl-ol-mini
77.60 (+0.65)
251.0/269.06 (x31.03)
OpenAl-ol-presiens
83.61 (46.66)
518.49 545.37 (x62.9)
77.9
GPT-4o
64.15
56.43
Reasoning
OpenAl-ol-mini
76.12 (+11.97)
500.1 346,7/ (x9.69)
OpenAT-ol-presieew
80,98 (+16.33
1106.43/ 1198.8 (x21.224)
72.5
Kaswelddge
200-
Reasoring
GPT-4o
72.29
26.08
Doneall
Overall
OpenAl-ol-miní
77.06 (+4.77)
341.8/ 370.29 (x114.2)
67.3
Opeus--ol-pressew
82,65 (+10.36)
732.79/ 783.54 (x30.001)
CponAtol OpenAlol-greiin prenioo
QpesAlui-man
Comparison between OpenAl-ol and GPT-4o. Reason- ing Tokens refer to tokens used for internal reasoning in OpenAl- ol; Completion Tokens refer all tokens consumed both inter- nally and in external output. "-" and "x" indicate performance gains and token consumption of OpenAT-ol over GPT-4o.
Figure 4:
Number Completion 'lekens (Logartthmiel
Figure 5: The relationship be- tween CS-Bench scores and the number of tokens consumed.
among models with scales below 10B, Llama3-8B performs the best, rivaling previous much larger- scale models and even surpassing Llama2-70B, indicating significant potential for compression in model parameters (Delétang et al. 2024p. Lastly, while performance variations exist among models of different families at the same scale, models within the same family continue improve with increasing scale on CS- Bench (see detailed scale analysis in Section 3.4
Comparison of Knowledge and Reasoning. Overall, all models perform worse on reasoning (average 45.60% compared to knowledge scores (average 60.61 %)) indicating that reasoning poses a greater challenge to LLMS compared positive correlation between reasoning scores and knowledge scores. However, this correlation is not absolute. For instance, PaLM-2 has a higher knowledge score but lower reasoning score compared to Claude-2.1, showing PaLM-2's weakness in applying knowledge. Furthermore, more powerful LLMs demonstrate a stronger ability to use knowledge for reasoning compared to weaker LLMs. This reflected in the much lower reasoning scores of weaker models relative their knowledge scores. However, as the model's capability increases, the growth in reasoning scores is more pronounced than that of knowledge scores, gradually bridging the gap between knowledge and reasoning abilities.
knowledge. As shown in Figure 2 there is strong
Comparison between Domains. First, regarding knowledge scores in Table 2 and Figure 3 (a). models generally perform best DSA and worst OS, which we attribute mainly to differences in the scale of pretraining data and the varying learning capabilities induced by model size. Second, the demand for reasoning ability varies across different domains, as evidenced by the gap between knowledge and reasoning scores. A notable example is GPT-4o, which shows close knowledge and reasoning scores in OS, while exhibiting extreme differences in DSA, with the highest and lowest scores, respectively. We further explore LLMs' performance in fine-coursed subfields in Appendix and explore the impact of Code and Math abilitics on different CS domains Section 3.5
Comparison between Tasks. As shown in Figure 3 (b) and Table 14 given the varying initial random scores, LLMs generally performs best on Assertion questions (average 63. 14% across all models), followed by MC questions (average 55.45 45.45%). Open-ended questions (average 49.3%) and performs worst on FITB questions (average 41 11.58%). However, the variation in task format sensitivity highly pronounced in weaker models, while stronger models can mitigate the disparities caused by different task formats, exhibiting robustness. For instance, Llama2-7B scores only 26. 19% on Open-ende ended reasoning but 60.61% on Assertion reasoning, whereas GPT-4 scores comparably on both Open-ender reasoning (68. 94%) and Assertion reasoning (67 (67.6%))
3.3 OPENAI-O1 RESULT
Compared to general models' reasoning processes, OpenAl-ol models think before they answer, forming a long internal chain of thought through reasoning tokens. As shown in Figure 4 OpenAl-ol significantly improves reasoning performance. While OpenAl-ol-ma -mni nearly matches GPT-4o on knowledge-type questions, it boosts scores by 11.97 points reasoning problems. Further- more. OpenAl-ol --reviewww achieves substantial progress in both knowledge and reasoning questions. However, this performance gain comes at the expense of high token consumption. For instance,

---

Published as a conference paper at ICLR 2025
dt
prodet
6-4tut amownn-rng only
5-chot ancvon-enly
70-
10
5 <-cbee COT
Overall 50
50-
Liazzz
Usmaa
Ouennt Pitting
Gemaaa
Lamaa Fiting
Inaern
Duent Obernodd
40
DongsecALA
Liunzz Ulserved Scuus
Owent 1,08 Preduction
14
Parumeter Scale (Linear)
32
72
110
24
Farameter Scale (Logurithmic)
72 110
1002
75
02
50
189143
100
(a)
(8)
(6)
Figure 6: (a) The performance of LLMs at different parameter scales. (b) The scale-score fitting curve of Qwenl.5 5 and Llama2 series. (c) Comparison of models under O-shot answer-only, 5-shot answer-only, and 5-shot COT settings.
OpenAI-ol-pree -prevvew consumes an average of nearly 1.2K tokens on reasoning questions, 21.24 times that of GPT-4o. Figureß) visualizes the relationship between model performance and completion token consumption. For reasoning questions, performance roughly increases logarithmically with the number of tokens, while the gains from token expenditure are lower for knowledge-tye type questions. We conduct a case study in Appendix E.7 comparing the responses of GPT-4o and OpenAl-ol.
3.4 QUALITATIVE ANALYSIS
Relationship between Scores and Model Scales. To investigate how model performance varies with parameter size, we examine several model families and plot the results in Figure (a). Models within the same family consistently improve as parameter size increases, but performance gains diminish with larger models. For instance, the score in Qwen ..5 improves by 16. 1..19 from 0.5B 7B, by 7.11% from 14B 72B, and by only 2.66% from 72B to 110B. Figure 6 (b) shows that when the parameter scale grows exponentially. the score increases approximately linearly. indicating a logarithmic scale pattern in the CS field. Given the substantial computational resources required for large-scale models, we aim to establish the relationship between model scales and scores to predict the performance of larger-scale models by fitting smaller-scale model scores. Due to space limitations, the specific design and implementation of the fitting function are provided in Appendix E.2 Overall, we fit the functions of Llama2 and Qwen series based on models ranging from 7B to 70/72B. We validate the fitting function Qwen-1.5 1.5 1 10BB, where the predicted value (67.83%) closely matches the actual value (7,55,), enabling further predictions for theoretical models, even up to 1000B.
Comparison between Zero-shot, Few-shot and COT Prompting. To investigate the impact of few-shot prompts and chain of thought (COT (Wei et al. 2022 ) model performance, we evaluate model's performance under 5-shot answer-only (AO) and 5-shot COT prompts in Figure where the prompt samples are sampled from the validation set and match the domain of the test questions. Given that codel-generated results under 0-shot COT often don't it adhere to specific formats, making regular matching difficuut, omit O-shot COT experiments, similar to C-Eval. Additionally, for Open-ended questions, since the answers include detailed explanations, 5-shot COT the same as 5-shot AO. For all tested models, the 5-shot prompts show improvement compared to 0-shot, with average increases of 1. 1.47% for 5-shot AO and 2.00% for 5-shot COT, respectively. Moreover, the efficacy of few-shot prompts in bringing improvements appears models such as GPT-3.5 and GPT-4, owing to their superior in-context learning capabilities.
pronounced in some robust
Analysis of Error Types. To delve into the roots LLMs' failure on CS-Bench and offer pathways toward improvement, we acquire the solution process of model errors under 5-shot COT, and utilize GPT-4 to categorize each error type in MC questions in Figure 7 It should be emphasized that models may cause joint errors, resulting in more than one error type assigned to single answer. In general, from Llama2-7B 7B all the way to GPT-4, the total number errors continues to decrease for both knowledge- -tyde and reasoning- -ype questions. For knowledge-type questions, both single concept errors and concept confusion show a decreasing trend. Initially, some completely wrong concepts transitioning partially erroneous ones and subsequently being eliminated, thus exhibiting initial
7

---

Published as a conference paper at ICLR 2025
1Jama-
Dama2
Jamat.
Liamaz
1Jama7-
Uxma2-
10.00%
14.0%
1647%
50.30%
16 00%
Complete cinceptual seror
Concepi releted arnor
Partially concspitual arnor Conceptual confacton
Logical reas reasening srree
G114
13.23%
7.50%
GMS4
34.11%
7.3095.00%
Mochomatioal reasoning ecnor
Types of wrong knowledgo-based questions
(b) Types wrong reasoning-based questions
Figure 7: The proportion of different error types varies by models for multiple-choice questions.
MATH
0.9458
Ovenl Lanal OWELD Shorl Oven 50verL 0994
GSMBK
Stom (N)
HumeniSral
Chatbot Arena
0.039
(3) MATEI 3.44)
C.vall
0.7943
0.7175
DamesDDi p-11011
T00ml
Scor th1
MIF? -C- Seec
Avcropo:.5339
0.0019
09al
0.3
0.4
0.5
0.5
0.7
0.9
1.0
Peasson corriation confucient (P)
(pm0.92)
Figure The consistency of model perfor- mance between CS-Bench and other bench- marks. Model scores from other benchmarks sourced from their official papers/websiiee or the OpenLLM Leaderboard.
Figure 8: The score changes on CS-Bench as LLM's Math/Code score increases. denotes Pear- son correlation coefficient. We obtain the scores on Math/Code datasets from (Alibaba, 2024).
rise followed by a decline partial concept errors. For reasoning-type questions, we observe that significant portion of errors still fall under the category of chowledge-basee mistakes. While stronger models have evidently reduced arithmetic reasoning errors for reasoning inaccuracies, there hasn't t been much change in logic reasoning errors specific the CS field. Our analysis highlights that reinforcing CS knowledge concepts is the most direct and effective approach to improving L.LMs' performance in the field of CS. Furthermore, significant improvements in CS reasoning performance are challenging to achieve solely by enhancing general reasoning abilities and mathematical reasoning, necessitating CS CS-specific reinforcement. More details be found in
3.5 WHAT'S S THE RELATIONSHIP BETWEEN CS, MATH AND CODE ABILITIES OF LLMs?
To explore the relationship between CS proficiency and the mathematical and coding capabilities of models, we investigate (1) the performance of general LLMs across the fields of Math, Code, and CS, and (2) performance of LLMs specialized in Code and Math within the field of CS.
Exploration on General Models. In Figure g we illustrate how the models' performance on CS-Bench varies with increasing scores on the Math datasets (GSM8K (Cobbe et al., 2021). MATH (Hendrycks al. (202bb) and Code datasets (HumanEval (Chen et al.) (022). MBPP (Austin et al. 2021)) The overall trend in CS-Bench performance closely aligns with changes in Math and Code scores, as indicated by a Pearson correlation coefficient (Cohen et al. 2009) exceeding 0.9. Besides the general enhancement of diverse competencies that superior models typically bring, consider this evidence suggests strong correlation between CS proficiency and skills in Math and Code.
We leave the analysis from the model representation perspective in Appendix E.5
8

---

Published as a conference paper at ICLR 2025
Table 3: The performance of the Math-expert LLMs on CS-Bench (EN), We use blue to emphasize areas where the expert LLMs improve compared to the Chat LLMs.
Model
Type
DSA
CO
CN
OS
All
Klg
Rug
Klg
Rog
Kig
Rng
Klg
Rog
Kig
Rog
Avg.
Intern
Chat
59.57
40.92
58.83
37.94
62,65
40.60
50.94
39.29
58.31
39.77
51.56
InteraLM Math 7B Ving et 2024 2024
Math
60.23
31.56
50.56
38.61
55,93
44.47
47.69
43.85
53.64
39.41
48.45
Deepseekd J.M-78
Chat
56.42
28.94
52.09
52.48
52.43
31.41
41,66
31.98
50.87
31.11
43.67
MatiieekMMtt--nssturttte. Shao al.
Math
63.98
34.82
55.13
61.26
42.69
56.68
39.67
50.49
Llama2-138 13B
Chat
51.74
35.00
51.81
36.18
53.03
37.99
18.12
32.36
51.31
35.46
45.51
MAmmmoTH-1338 Yoe al 2023
Math
50.84
28. 26
46.16
34,61
51.39
30:45
34,94
32.64
46.20
31.32
40.78
Llama2-70B
41.51
56.35
40.85
51.79
41.15
58.73
41.68
WizardMath- 70B
Lav et al. 2023a
Math
60.17
28.67
56.41
34,91
58.52
41.51
47.01
42.53
55.77
36.67
48.82
Table 4: The performance of the Code-expert LLMs on CS-Bench (EN).
Model
Type
DSA
Kig
Rng
Klg
Rng
Klg
Rng
Klg
Rng
Klg
Rng
Avg
Llama2.7B
Chat
51.51
31.82
47.15
CodeLlama-778 Rozière al N024
Code
58.90
36.15
45.46
36.24
52.87
26.23
44.35
25.33
50.36
31.09
43.35 34
Dolphander:78 7B Wang
Code
50.13
36.47
34:71
34.36
41.78
23.92
40.03
28.35
41.40
30.82
37.54
WizanCodder 715 (Lao et al.
30235
Code
47.42
33.58
35.54
37.09
41.17
26.03
40.88
30.60
41.02
31.73
37.63
LIama2-130 13B
Chat
51.74
35.00
51.81
36,18
53.03
37.99
48.12
32.36
51.31
35.46
45,54
CodeL 13B (Roziere et al. 2024
Code
59.87
34.17
44.96
35.82
51.56
35.83
43.28
34.56
49.84
35.08
44.47
WizandCoder- 135 [Llb et al
2023b
Code
50,50
32.98
38.69
35.27
43.42
28.34
40.88
34.29
43.27
32.59
39.38
Another piece of evidence is the consistency between CS-Bench and other benchmarks, including non-CS scientific benchmarks (GPQA (Rein al. 2023 specialized benchmarks (Tool-use ssoo--use-evll (Alibaba 2024), T-Eval (Chen et al. 2024). L-eval (An et al., (023), BBH (Suzgun et al.) (022)), and general benchmarks (Chatbot Arena (Zheng et al., 2023). C-Eval(Huang et al., 2023b)). The results are shown in Figure 9 We observe that CS-Bench and agents, tool usage, and other fields have a low correlation (.8.. whereas their correlation with benchmarks involving long texts, multi-step reasoning, and general tasks is moderate (Pt). In contrast, the P-values for CS-Bench and benchmarks in mathematics/coding are all above 0.9, reaching as high as 0.96 in MBPP, which strongly supports the assertion that the model's CS capabilities mathematical abilities. Additionally, we find that Chatbot Arena, BBH, and L-eval include subsets related mathematics and coding, resulting their correlation with CS-Bench being higher than that of other non-Code/Math benchmarks.
closely related to its coding and
Next, we examine models with inconsistent patterns between CS and Math/Code. In the Math domain, Qwenl.5-7B outperforms Llama2-70B in both GSM8K and MATH, yet in CS-Bench, Llama2-700 surpasses Qwenl.5-277 1.5-BB. In the Code domain, Mixtral-8 x 7B performs better than Qwenl.5-32B on HumanEval and MBPP, whereas the opposite is observed on CS-Bench. Given the NI.P community's s sustained focus on the Code and Math domains, some recently released models have been trained on a large amount data in these domains, leading to smaller-scale models outperforming much larger-scale ones (e.g., liven1.5-7D surpassing Llama2-70B). However, in the CS domain, due to insufficient attention and training data, even excellent small-scale models struggle to surpass much larger-scale models. This also indicates that CS-Bench has not been overfitted during LLM pretraining, making it a fairer benchmark for measuring model performance differences.
Exploration on Expert Models. We present the results of the Math and Code expert LLMs in Tables 3 and Compared to general Chat LLMs, expert LLMs usually sacrifice other abilities to boost proficiency in Math or Code, which is reflected in the lower overall performance of most expert LLMs. Therefore, we are more concerned with identifying the specific aspects of CS where Math and Code models show improvement. Regarding mathematics, Interni. Baatnnmm--Mahhh7 improves Internl.m² 7B's performance in CO, CN, and OS reasoning tasks, while DeepseekMath exhibits significant improvements across all domains. According to (Shao et al. 2024). DeepseekMath effectively maintains general knowledge and reasoning ability during specialization. Conversely, MAammo TH and WizardMath perform poorly due to just fine-tuning limited mathematical datasets, resulting in a significant decline in general knowledge and reasoning. The score changes in LLMs suggest that OS is most closely linked to mathematies, followed by CO, and lastly DSA and CN.
In terms of Code, many Code models show significant improvements in DSA (especially knowledge) and OS (especially reasoning), such Cade and Dolpheoder. This indicates that the disciplines

---

Published as a conference paper at ICLR 2025
of DSA and OS are more closely related to code, thus enhancing knowledge and reasoning abilities in these directions, while CO and CN have lower relevance, leading to a decrease in scores. Finally, we observe that the enhancement brought about by small-scale expert LLMs compared to larger-scale L.MMs is more pronounced (see Codel_lama-75e supplementary need for specific knowledge and reasoning capabilities in small-scale L.Mss, whereas large-scale LLMs already encompass a greater breadth of knowledge and stronger reasoning abilities, resulting in diminishing gains from further training in specific domains.
WizardCoder- BBB133B) We attribute this to the
4 RELATED WORK
Exploration of LLMs in Computer Science. Given the powerful capabilittes of LLMs, recent research has explored their potential applications across various industries and scientific fields,
including finance (Zhao al. 2024), autonomous driving (Huang et al., 2023a| Zhou et al. 2024).
robotics (Yuan et 2023a, Xiao et al., [023] Wang et al. [202aa). medicine (Zhou et al. 2023a
Vaid et al. 2024), and chemistry (Guo et al. 2023] Zhang et al. 2024a). Currently, studies exploring LLMs in the field of computer science fall into two main categories. The first category includes broad
evaluation benchmarks covering various fields, such as MMLU (Hendrycks et al.) (02 CMMLU
(Li et al. 2024). C-Eval (Huang al. 2023b), Xiezhi (Gu et al. 2024), and M3KE (Liu et al. 2023a). However, computer science constitutes only small fraction of these benchmarks, accounting for less than 5% and lacking detailed CS CS-specific analysis. The second category focuses solely on exploring specific applications of LLMs within computer science, such as network topology (Donadel et al. 2024), cybersecurity (Ferrag et al., 2024; Murtuza 2024), and software engineering (Marques et al. 2024 Dipongkor 2024). Nonetheless, there has been a persistent lack of comprehensive evaluation of Mss foundational knowledge and reasoning abilities in computer science. To address this gap, we propose CS-Bench and conduct thorough evaluation of LLMs, providing guidance for understanding and improving their performance in the CS field.
Evaluation of LLMs' Capabilities. Evaluating and understanding the capabilities of LLMs is
major focus within the NLP community. Researchers have extensively explored the capabilities of
LLMs including planning (Huang et al. [2024), multilingual processing (Lai et al.) 2023 Bang et al.
2023), instruction following (Zhou et al. 2003bb Wang et al., 20236), cross-domain generalization
(Wang et al.) 2023a| Song al.) 2023 Wang et al. (024bb and safety (Mazeika et al. 2024 Diao
cct al., 2024; Mou et al.,
2025). Recently, there has been growing interest in LLMs abilities in
mathematics (Frieder et al. 2023 Collins et al. 2023 Wu et al. 2023 Yuan et al.
2023b Dong et al.
2024 [iu et al. 2024)) code programming (Luo et al. 2023b Rozière et al. 2024 Wang et al. 2024c
Zhang et al. | 2024b][Lin et al.] 2024), and logical reasoning (Liu al. 2023b (Saparvv al. 2023
Xu et al. 2023 Wu et al. (2024). While individual capabilities have been well-studied, research on
their integrated application and interrelationships remains sparse. Different from (Dong et al.) 2024]
which investigates interactions between abilities during the supervised fine-tuning phase, we choose computer science as our research context. Given that computer science inherently integrates coding, mathematics, and reasoning, we utilize CS-Bench in this paper to deeply explore the relationship between LLMs' performance in computer science and their mathematical and coding abilities, aiming to advance cross-capability research and integrated analysis of LLM abilities.
5 CONCLUSION
In this work, we introduce CS-Bench the first benchmark specifically designed to systematically analyze the knowledge and reasoning capabilities of mainstream LLMs in the field of computer science. Our evaluation of over 30 models highlights that even the top-performing GPT-40/OpenAlloo has significant room for improvement in computer science. Further score-scale experiments and error type analyses provide directions for enhancing LLMs in the field. Moreover, our investigation into the relationship between computer science, mathematics, and coding demonstrates their close interconnections and provides valuable insights into LLMs' cross-abilities and applications.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.088774 2023.
Zhipu AI, 2024. URL https: https://zhipua, wwwpaaii.nn//////eeeeeeemeee
Alibaba. Introducing qwen1 - qwen, 2024. URL
bottchubb.bbboo...o og/
qwen 57
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, arXiv preprint arXiv:2307. 1088, 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arXiv preprint arXiv:2305.10440 2023.
Anthropic. Introducing claude 2.1, 2023. URL httpss//www..arrrrppc.cooo////// claude-2-1
Anthropic. Introducing the next generation of claude, 2024a. URI/tttp::/www.athheeg
com Com/news//e hamm/news/cllaude clweeelladdde-ee family
Anthropic. Introducing claude 3.5 sonnet, 2024b. URL news/
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Baidu. Wenxinyiyan online, 2023. URL httts:://yyyan htwp:/yyyaaaaa...ooo.oo baidu.cod.........
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint wrXiv:2401.029544 2024.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024.
Martin Campbell-Kelly, William Aspray, Jeffrey R Yost, Honghong Tinn, and Gerardo Con Díaz. Computer: history of the information machine. Routledge, 2023.
Yupeng Chang, Xu Wang, Jindong Wang. Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45,2000
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.0337 2021.
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. T-eval: Evaluating the tool utilization capability of large language models step by step. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9510-9529, 2024.
11

---

Published as a conference paper at ICLR 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing,
pp. 1-4, 2009.
Katherine M. Collins, Albert Q. Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B. Tenenbaum, William Hart, Timothy Gowers, Wenda Li, Adrian Weller, and Mateja Jamnik. Evaluating language models for mathematics through interactions, 2023.
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression, 2024.
Peter Denning. Computer science: The discipline. Encyclopedia of computer science, 32(1):9-23, 2000.
Muxi Diao, Rumei Li, Shiyang Liu, Guogang Liao, Jingang Wang, Xunliang Cai, and Weiran Xu. Seas: Self-evolving adversarial safety optimization for large language models. arXiv preprint vrXiv:2408.02632. 2024.
Atish Kumar Dipongkor. Towards interpreting the behavior of large language models on software engineering tasks. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, pp. 257-257, 2024.
Denis Donadel, Francesco Marchiori, Luca Pajola, and Mauro Conti. Can Ilms understand computer networks? towards virtual system administrator, 2024.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 27X1v:240..21783, 2024.
Mohamed Amine Ferrag, Fatima Alwahedi, Ammar Battah, Bilel Cherif, Abdechakour Mechri, and Norbert Tihanyi. Generative and large language models for cyber security: All insights you need, 2024.
Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, and Julius Berner. Mathematical capabilities of chatgpt, 2023.
Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, Qianyu He, Rui Xu, Wenhao Huang, Jingping Liu, Zili Wang, Shusen Wang, Weiguo Zheng, Hongwei Feng, and Yanghua Xiao. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation, 2024.
Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi. Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Ree Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun lyer, and Zehua Li. Legalbench: collaboratively built benchmark for measuring legal reasoning large language models, 2023.
Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. What can large language models do in chemistry? a comprehensive benchmark eight tasks, 2023.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset, 2021
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Ilm agents: A survey, 2024.
Yu Huang, Yue Chen, and Zhu Li. Applications of large scale foundation models for autonomous driving. arXiv preprint arXiv:2311.12144 2023a.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lx, Yikai Zhang. Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models, 2023b.
Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. Perplexity-a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1)) S63-S63, 1977.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint urXiv:2310.06825, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.044 004088 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E, Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023.
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning, 2023.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifci Yang. Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin, Cmmlu: Measuring massive multitask language understanding in chinese, 2024.
Jiayi Lin, Hande Dong, Yutao Xie, and Lei Zhang. Scaling laws behind code understanding model, 2024.
Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng. Peiyi Zhang, Qingging Lyu, Xiaowen Su, Qun Liu, and Deyi Xiong. M3ke: A massive multi-level multi-subjeet knowledge evaluation benchmark for chinese large language models, 2023a.
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4, 2023b.
Wentao Liu, Hanglei Hu, Jie Zhou, Yuyang Ding, Junsong Li, Jiayi Zeng, Mengliang He, Qin Chen, Bo Jiang, Aimin Zhou, and Liang He. Mathematical language models: A survey, 2024.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin. and Daxin Jiang. Wizardcoder: Empowering code large language models with cvol-instruct, 2023b.
Nuno Marques, Rodrigo Rocha Silva, and Jorge Bernardino. Using chatgpt in software requirements engineering: A comprehensive review. Future Internet, 16(6):180, 2024.
13

---

Published as a conference paper at ICLR 2025
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang. Norman Mu, Elham Sakhaee, Nathamiel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024.
Meta. Introducing meta Ilama 3: The most capable openly available IIm to date, 2024. URL
https://ai. meta. com / www.//otaaaaa.ooooooooooooooooaa 1aaaalllaaa222 3/
MistralAI. Large enough mistral -i- frontier ai in your hands, 2024. URL https://mistrre ai/ aa/ning www.aatttttaaaaaaeaaaaaaaee
Yutao Mou, Shikun Zhang, and Wei Ye. Sg-bench: Evaluating Ilm safety generalization across diverse tasks and prompt types. Advances in Neural Information Processing Systems, 17:123032-123054 2025.
Shariq Murtuza. Sentinels of the stream: Unleashing large language models for dynamic packet classification in software defined networks position paper, 2024.
OpenAI. Introducing chatgpt, 2022. URL http:////ppeenaa.ccm/mneeeee////
OpenAl. Hello gpt-4o, 2024a. URL /ttpss//poeeaaa.ccmm/nneee//////
OpenAl. Openai ol hub - openai, 2024b. [RLL/tttpp::/peaai..omm
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petry, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprini arXi::2311.
2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong. Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024.
Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. Testing the general deductive reasoning capacity of large language models using ood examples, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning open language models, 2024.
Xiaoshuai Song, Keqing He, Pei Wang, Guanting Dong, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, and Weiran Xu. Large language models meet open-world intent discovery and recognition: An evaluation of chatgpt, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint 10X.1:2210..92211, 2022.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifree, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
THUDM. Chatglm3 series: Open bilingual chat 1lms, 2023. THUDM/Chat.I.M]
URL https: attps://github. hwtphb.ttttut..../gg com/
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0928 2023.
Akhil Vaid, Joshua Lampert, Juhee Lee, Ashwin Sawant, Donald Apakama, Ankit Sakhuja, Ali Soroush, Denise Lee, Isotta Landi, Nicole Bussola, et al. Generative large language models are autonomous practitioners of wrdsecce-based medicine. arXiv preprint arXiv:2401.02855 2024.
14

---

Published as a conference paper at ICLR 2025
Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, et al. Large language models for robotics: Opportunities, challenges, and perspectives. arXiv preprint arXir:2401.04334, 2024a.
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang. Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao. Yue Zhang, and Xing Xie. On the robustness of chatgpt: An adversarial and out-of-distribution perspective, 2023a.
Pei Wang, Keqing He, Yejic Wang, Xiaoshuai Song, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, and Weiran Xu. Beyond the known: Investigating Ilms performance on out-of- domain intent detection, 2024b.
Shenzhi Wang and Yaowei Zheng. Llamaa-8b-chinese-chaa (revision 6622a23), 2024, URL https //huggingface. co/shenzh/ wang htt///aamaaaa 8B Chinese Chat
Yejie Wang, Keqing He, Guanting Dong, Pei Wang, Weihao Zeng, Muxi Diao, Yutao Mou, Mengdi Zhang, Jingang Wang, Xunliang Cai, and Weiran Xu. Dolphcoder: Echo-locating code large language models with diverse and multi-objective instruction tuning, 2024c.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248477 2022.
Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. An empirical study on challenging math problem solving with gpt-4, 2023.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks, 2024.
Xuan Xiao, Jiahang Liu, Zhipeng Wang, Yanmin Zhou, Yong Qi, Qian Cheng, Bin He, and Shuo Jiang. Robot learning in the of foundation models: A survey, arXiv preprint arXiv:2311.14379 2023.
Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? a comprehensive evaluation and beyond, 2023.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint 2rXiv:2309.103300 2023.
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma. Jiawei Hong, Kuikun Liu, Ziyi Wang. Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin. Internim-man math: Open math large language models toward verifiable reasoning, 2024.
Mingze Yuan, Peng Bao, Jiajia Yuan, Yunhao Shen, Zifan Chen, Yi Xie, Jie Zhao, Yang Chen, Li Zhang, Lin Shen, et al. Large language models illuminate a progressive pathway to artificial healthcare assistant: A review. arXiv preprint arXiv:2311.01911 2023a.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023b.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su. and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, 2023.
15

---

Published as a conference paper at ICLR 2025
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-1306 An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
Qiang Zhang, Keyang Ding, Tianwen Lyv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang. Xiang Zhuang, Zeyuan Wang, Ming Qin, Mengyao Zhang, Jinlu Zhang, Jiyu Cui, Renjun Xu, Hongyang Chen, Xiaohui Fan, Huabin Xing, and Huajun Chen. Scientific large language models: A survey on biological and chemical domains, 2024a.
Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. Unifying the perspectives of nlp and software engineering: A survey on language models for code, 2024b.
Huaqin Zhao, Zhengliang Liu, Zihao Wu, Yiwei Li, Tianze Yang, Peng Shu, Shaochen Xu, Haixing Dai, Lin Zhao, Gengchen Mai, et al. Revolutionizing finance with Ilms: An overview of applications and insights. arXiv preprint arXiv:2401. 1144, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang. Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models, 2023.
Lianmin Zheng, Ying Sheng, Wei-Lin Chiang, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Chatbot arena: Benchmarking ilms in the wild with elo ratings. Blog post, May, 2023.
Hongjaan Zhou, Boyang Gu, Xinyu Zou, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Xian Wu, al. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint arXiv:231 1.05/ 12, 2023a.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follovingg evaluation for large language models, 2023b.
Ziqi Zhou, Jingyue Zhang, Jingyuan Zhang, Boyue Wang. Tianyu Shi, and Alaa Khamis. In-context learning for automated driving scenarios. arXiv preprint arXiv:2405.044 0415, 2024.
ACKNOWLEDGEMENTS
This work was partially supported by:
The State Key Laboratory of Massive Personalized Customization System and Technology (No. H&&C-MPC-203-00--070
The State Grid Technology Project 65700-20241623AA-11111 on "Research on Active Semantic Discovery Technology Based on SG-CIM and Its Application in Power Grid Equipment Supply Chain Optimization".
China Unicom Software Research Institute under the "Framework Agreement for Seven Model Technology Research and Application Demonstration Projects (Software Development for Govern- ment Enterprise Content Generation) from 2024 to 2025" (No. 5500331818
The National Natural Science Foundation of China (NSFC Nos. 62076031 and 62076036).
Zhongguancun Academy.
16

---

Published as a conference paper at ICLR 2025
Appendix
CONTENTS
Introduction
2
CS-Bench
2.1
Design Principle
2.2
Data Collection
3
Experiment
3.1
Experimental Setup
Main Results
3.3
OpenAl-O1 Result
Qualitative Analysis
3,5
What's the Relationship between CS, Math, and Code abilities of LLMs?
8
Related Work
10
Conclusion
10
Limitations and Dataset Bias
19
Broaden Impact
19
More Details on CS-Bench
20
C.I Details of Data Collection and Processing,
20
Detailed Design Motivation and Statistics CS-Bench
20 ................................
Distribution of Word Lengths
22
CS-Bench Examples
22
D
More Details Experiment Setup
26
D.I Details of Template for Each Task Format
26
D.2 Details of GPT-4 Scoring
26
D.3 Details of Interence Implementation
27
D.4 Details of the Models being Evaluated
27
More Details on Experiment
31
Details of Model Performance
31
Scale-Score Fitting Function for CS-Bench
33
E.3
Model performance CS-Bench across different languages
................................
34
Case Study of Error Types
36
Analysis from Model Representation)
40
17

---

Published as a conference paper at ICLR 2025
E,6 Exploration for Improving CS Performance
41
Case Study of OpenAl-ol
42
18

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND DATASET BIAS
In this paper, we introduce CS-Bench, providing comprehensive evaluation of LLMS and exploring the relationships between model capabilities. However, there still some limitations to this paper.
(1) Coverage Limitations: Although CS-Bench has made significant strides in comprehensiveness of CS evaluations compared to existing work, given the breadth of computer science, our evaluations cannot cover the entire scope of computer science knowledge. Furthermore, our assessment content focuses university- level content, examining LLM's mastery of basic subjects in computer science, rather than specific computer science-related research scenarios.
(2) Evaluation Limitations: In the CS-Bench evaluation experiments, we employ GPT- scoring to assess generative tasks such as fill-in-the-bland and open-ended tasks. This might lead to certain evaluation thresholds and costs. However, such issues only constitute about 20% of CS-Bench. Additionally, we provide an evaluation scheme that separates comprehension tasks from CS-Bench, allowing for automatic evaluations without the need for GPT-4. We also explore the effect of different scoring models and find that even slightly inferior models produce score rankings consistent with GPT-4, while significantly reducing costs.
(3) Language Limitations: CS-Bench are primarily focused on Chinese, French, German, and English language environments, ensuring comprehensive and in-depth evaluations in these language environments. However, for other environments, its support and coverage are relatively weak, and further optimization and improvement are needed.
The main biases of CS-Bench can be categorized into two aspects:
(1) Difficulty Level: Overall, the benchmark reflects a university-level difficulty.
(2) Task Focus: It emphasizes knowledge- and reasoning- based questions rather than specific real- world production scenarios in computer science.
The impact of difficulty on LLM evaluation is: lower difficulty tends to narrow the score differences between models, while higher difficulty amplifies these differences. is worth noting that although CS-Bench maintains an overall university-level difficulty, the diversity of difficulty across questions remains significant, ranging from simple definitions in data structures to challenging computer network application problems. This is reflected in the model score range of 39.86% to 72.29%, demonstrating CS-Bench's is effectiveness in distinguishing LLMs' capabilities computer science and presenting challenges even for the best-performing models.
Regarding task scenarios, a key motivation for CS-Bench is that "some prior work has explored LLMs in specific applications such as cybersecurity and software engineering." However, stated in Section 4 there has been a lack of comprehensive evaluations of L.MSs' fundamental knowledge and reasoning capabilities in computer science. Besides, considering the extensive research already conducted by the LLM community on coding abilities, we view programming as a separate focus area. Therefore, CS-Bench does not include programming questions but instead focuses on the relationship between CS capabilities and coding abilities.
B BROADEN IMPACT
Societal Impact. CS-Bench is anticipated play a significant role the field of computer sci- ence. LLMs, trained and evaluated with the aid of CS-Bench, can enhance the work efficiency of relevant professionals, enabling them to complete computer-related tasks, such as code review, error detection, and algorithm optimization, more quickly and accurately. Although this might result in the disappearance of some repetitive jobs, could also create new career opportunities. education, the CS-Bench dataset can serve explaining complex computer science concepts and techniques, and also enabling students to better understand and master this knowledge through practice. However, we should also be cautious of potential risks associated with ES--Bech, such as exam cheating by students in the CS field, which requires management mechanisms and logging to prevent misuse.
the realm of effective teaching tool, assisting teachers in better
Ethics Statement. We ensure adherence to applicable laws and ethical guidelines during the process of data collection, annotation, and usage, providing adequate compensation all crowd
19

---

Published as a conference paper at ICLR 2025
workers. As this benchmark pertains to objective knowledge and reasoning in the field of computer science, the annotation content is not influenced by regional or cultural differences among annotators. Moreover, our dataset does not contain any personally identifiable information or offensive content. The authenticity and accuracy of CS-Bench have heen thoroughly verified, providing a reliable basis for evaluating L.MMs. CS-Bench is intended solely for academic and research purposes. Any commercial use or other misuse deviating from this purpose is strictly prohibited. We urge all users respect this provision maintain the integrity and ethical use of this valuable resource.
C MORE DETAILS ON CS-BENCH
In we introduce the details of data collection and processing for CS-Bench. In we provide a detailed explanation of the design motivation and statistics for CS-Bench. In we present the distribution of question and answer lengths for each task in CS-Bench. In provide example for each type under each dimension of CS- JS-Bench.
C.I DETAILS DATA COLLECTION AND PROCESSING.
Our data mainly comes from three sources: 1. Public channels providing computer science-related questions, such as professional exams and practice tests (5) 2. Knowledge-basee questions manually extracted and adapted by professionals from various Srademic--ermitter CS-related blog posts 6 3. Questions constructed from non-public teaching materials and exam papers authorized by the author's affiliated institution. For resources sourced from the internet, we manually extract the knowledge points and questions. For physical materials, we use Optical Character Recognition (OCR) to obtain the data. We then ensure the accuracy of the collected data through manual cross-checking. The data collection was carried out by team of five students with bachelor's degrees in computer science. We provided each person with adequate wages, significantly above the local minimum wage standard. Based on whether the questions require in-depth reasoning and computation, we label each question as either knowledge-type or reasoning-type. Additionally, we tag each instance with domain and task type labels. For English data, used GPT-4 to translate the Chinese instances into English, French, and German, followed by manual verification.
C.2 DETAILED DESIGN MOTIVATION AND STATISTICS OF CS-
We elaborate the design motivation of CS-Bench and statisties under each dimension follows.
Evaluation Content. To ensure comprehensive coverage of fundamental and critical areas in computer science, we select the four most foundational and prevalent domains within the field of computer science the core content of the CS-Bench dataset. These four domains are as follows: Data Structure and Algorithm, investigating data organization and algorithmic efficiency; Computer Organization, focusing on hardware composition and foundational system operation; Computer Network, involving the analysis of network communication and data transmission; Operating System, delving into system resource management and process control. As depicted in Figure I0 (a), these four disciplines exhibit roughly uniform distribution. Furthermore, we subdivide the disciplines into 26 granular chapters, allowing CS-Bench to furnish more nuanced evaluation outcomes for models and provide comprchensive guidance for model refinement. We summarize these chapters in Table 5
Task Format. To better simulate the diverse forms of problems encountered in the real world, we introduce assertion, fill-n--hhe-blank, and open-ended questions addition to multiple-choice questions. Specifically, multiple-choice and assertion questions correspond to understanding tasks in CS, while fill-in-the-blank and open-ended questions correspond to generation tasks in CS. Although assessing generation tasks using GPT-4 incurs certain costs, it is important to emphasize that this component represents only a minority ie--in-thhhbaan 10.67%. 77,, open-ended: 7.81%) whereas comprehension tasks relying on rule-based scoring constitute the majority (multiple-choice: 61.22%,
'c.g., http://gthh..aoccccceeeoo////o/r ec.g. http://wwwwiwiionoooo/////////// http:///wwp////wwccc...ee//// https://hhanlan.zhhhu.cc
http::/wwww.cnblgg..cmm
20

---

Published as a conference paper at ICLR 2025
Table Summary of 26 fine-grained subfields of CS-Bench.
Chalpter
Main Content
Subject
Question Number
Overview
Concepts and elements data structure, Temporal and spatial complexity...
DSA
168
Lincar List
Lincar tables, Sequential tables and Linked lists...
DSA
276
Stack, Qucue,and Array
Shared stack, Circle queue, Arrays.Spocial matrices...
DSA
352
String
Concept operation strings, Pattern matching strings...
Tree
Binary trees, Traversal of frees ans forests, Hullmun tree...
DSA
428
Graph
Concepts graphs. Traversals graphs, Application of graphs... Sequential seurch. Half-spiit search, Chunked seurch. Red black tree, B-tree and B+ tree. Hash search...
DSA
368
Searching
DSA
316
Insert Sorting. Swap Sorting, Selection Sorting, Merge Sorting. Heap Sorting. Merge Sorting, Cardinality Sorting, External Sorting Algorithms...
Sorting
DSA
356
Overview
Hardware performance indicators computers...
CO
Data Representation and Operation
Number system and encoding, Representation and oporation of fixed-point mu- mhers and flouting-pe point numbers...
CO
436
Storage System Instruction System
Main Memory, Exteral Memory, Cache Memory, Virtual Memory...
CO
448
Instruction format, Instruction addressing methed, CTSC and RISC... Fumctions
CO
312
Instruction execution process, path, CPU hard wiring design and micro programmming Exception and inter- rupt mechanisms, Instruction pipelines, multiprocssorr concept...
hus
Central Processing Unit
CO
488
Bus
Overview of the bus, Bus arbitration, Bus operation and timing, Bus standards...
CO
268
Inpat/Output System
PO interfaces and methods...
CO
312
Concepts, compositions, functions computer networks, Architecture and reference models of computer networks...
Overview and Architecture
CN
296
F'mddamentals Communication Theory, Transmisiio Media Physicul
Physical Layer
Layer Devices...
CN
328
Data frumes, Ertur control, Fllo comtrol and Reliable transmission, Media ess control, Local and wide area networks, and data link layer devices...
Layer
Overview of octwork layer functions. Routing algorithms, IPv4 and IPv6, Roa- ting protoculs, multicast, The services provided by the transport layer. UDP and TCP protocols... system DNS, FTP protocol, World
Network Layer
CN
Router...
Transport Layer
CN
364
Network application model, Domain Wide Web, and HTTP..
Application Layer
CN
408
Concopts of operating systems, Development and classification of operating
Overview
OS
systems, Operational mechanisms architecture openating systems... Processes and threuds, Schechling of processers, Synchrvnization and mutual exclusion processes. Deadlock issues...
Processes and Threads
0S
700
Memory Management
and methods viriual memory management..
Concept memory management, Concepi of virtual memory management,
08
432
File Management
systems, Organization and management of disks... 1/O devices and control methods, I/O core subsystem, Buffer management..
OS
332
InpotiOuupput Management
0S
368
7.01
JUN
(1003)
$5.424
(2418)
(ARLE)
(CRC4)
(1938)
27 183
(218))
(2419)
GARLS)
Multiple
Knowledge
English
CN
Assertios
Open-end
Beasoning
Chinnse
German
(a)
(c) Cognition
(4) Language
Figure 10: The quantity and proportion of each type in different dimensions on CS-Bench.
assertion: 20.3%). Therefore, if resources limited, we recommend considering the independent use of understanding tasks from CS-Bench for evaluation purposes.
Knowledge / Reasoning. The design goal CS-Bench is not only to assess the mastery of knowledge in the field of CS but also evaluate the model's ability reason using CS knowledge. Therefore, each dataset is labeled with "knowledge" or "reasoning" corresponding to simple questions requiring knowledge recall and challenging questions necessitating knowledge inference, respectively, As shown in Figure 10 (c), knowledge-base questions account for 63.58% while reasoning-base questions account for 36.42%.
Language. To assess the ability of LLMs in addressing CS problems in various linguistic environ- ments, and to adapt CS- CS-Bench for the evaluation wider range of LLMs, CS-Bench comprises quadrilingual (English, Chinese, French, and German) data, with each language accounting for 25 %
21

---

Published as a conference paper at ICLR 2025
The other data is obtained through translation by GPT-4, followed by manual verification of processed Chinese data.
C.3 DISTRIBUTION OF WORD LENGTHS
We compute the distributions of word lengths for questions and answers in CS-Bench (English) across various task formats, as illustrated in Figure II For Multiple-Choier questions, the question length includes both the question itself and the four options. Since Multiple-Choice and Assertion questions are comprehension tasks, the answers consist of only one character (A/B///D or True/False). For generation tasks, fill-in-the-blant answers are relatively short, with an average word length of approximately 2, whereas Open-ended questions typically yield longer answers they entail detailed explanatory processes.
Varta Clica
Avsrriic
Rkaddheasc
Operaedd
2.25
Ma
No
1.5
30
60
150
170
30
45
1.5
30
20
40
120
1:0
250
100
132
Vedicnl
Sin Meckern
0.3
Mi
20
6.5
10
15
Figure 11: Question and answer lengths of each task format in CS-Bench (English).
C.4 CS-BENCH EXAMPLES
We present samples of knowledge and reasoning types in Table samples from various domains in Table samples different task formats Table & and samples from different languages in Table
Table 6: Examples of knowledge-type and reasoning-type
Type
Example
Question:
The three fundamental elements of data structure include ().
A: Logical structure, storage structure, operations data.
B: Logical structure, algorithm design, program implementation.
C: Data types, data storage, data manipulation.
Knowledge
D: Data Definition, Data Implementation. Data Manipulation.
Answer:
Analysis:
None
Question:
The time complexity a certain algorithm is O(n?), indicating that the algorithm's 's O.
A: The problem size is O(n²).
B: Execution time equals O(n²).
C: The execution time is directly proportional O(n²).
D: The problem size is directly proportional O(z²).
Reasoning
Answer:
C
Analysis:
The time complexity is O(n²) )), which means the time complexity T(n) satisfics T(n) & c (where e proportionality constant), that is, T(n) = O The time complexity T(n) is a function of the problem size n, and the problem size remains n, not n²
22

---

Published as a conference paper at ICLR 2025
Table Examples samples in different domains.
Domain
Example
Question:
The correct statement about data structures is Q.
A: The logical structure of data is independent of its storage structure.
B: The storage structure of data is independent of its logical structure.
C: The logical structure of data uniquely determines its storage structure.
Data Structure and Algorithm
D: data structure determined solely by its logical structure storage structure.
Answer:
A
Analysis:
The logical structure of data is approached from the perspective of practical problems, using only abstract expressions and is independent of the various choices of data storage methods. The storage structure of data is the mapping of the logical structure on a computer, and it cannot exist independently of the logical structure. Data structure includes three essential elements, all of which are indispensable.
Question:
complete computer system should include O. A: Arithmetic Logic Unit (ALU), Memory. Control Unit
B: Peripheral devices and host computer
C: Host and Application
Computer Organization
D: The accompanying hardware devices and software systems
Answer:
Analysis:
A is component of the computer host, while B and C only involve parts of the computer system and are both incomplete.
Question:
The most hasic function of computer networks is ().
Data Communication
B: Resource Sharing
C: Distributed Processing
D: Information Synthesis Processing
Computer Network
Answer:
Analysis:
The functions of computer networks include: data communication, resource sharing, distributed processing, integrated information processing, load balancing, enhancing reliability, etc. However, the most fundamental function data communication, which also the basis for realizing other functions.
Question:
Among the following options, ( is not an issue of concern for the operating system.
A: Manage bare-metal computers
B: Design and provide interface between programs and hardware systems
C: Manage computer system resources
D: Compiler for High-Level Programming Languages
Operating System
Answer:
D
Analysis:
The operating system manages computer software/hardware resources, expands the bare machine to provide a more powerful extended machine, and acts as an intermediary between the user and the hardware. Clearly, the compiler for high-level programming languages is concem of the operating system. The essence of compiler set of program instructions that stored in the computer.
23

---

Published as a conference paper at ICLR 2025
Table 8: Examples of different task formats.
Type
Example
Question:
Given that the storage space for circular queue the array A[21]. with front pointing the position before the head element and rear pointing to the tail element, assuming the current values of front and rear are 8 and 3, respectively, the length of the queue is ().
A:
B:
Multiple Choice
C:
D: 17
Answer:
Analysis:
The length of the queue is (rear front + maxsize) maxsize = (rear front 21) % 21 16. This situation is the same as when front points to the current clement and rear points to the next clement after the last clement in the queuc.
Question:
In directed graph with vertices, the degree of cach vertex reach up 2n.
Answer:
False.
Assertion
Analysis:
In directed graph, the degree of vertex is equal to the sum of its in-degree and outdegree. In directed graph with vertices, any given vertex have most pair of oppositely directed edges connecting it with each of other vertices.
Question:
sequential list length n, when deleting the ith S S ri) element, 0 elements need to be moved forward.
Fill-in- the-blank
Answer:
n-i
Analysis:
The elements from ali+1] to a(n] need to be moved forward by one position, involving the movement of (e(i+1)+1=n-> elements.
Question:
Given that the 9th level of a complete binary tree has 240 nodes, how many nodes does the entire complete binary tree have? How many leaf nodes are there?
Answer:
a complete binary tree, 9th level full, then the number of nodes = (9- = 256.
However, currently, there are only 240 nodes on the 9th level, indicating that the 9thlevel not full and is the last level. Levels I to 8 are full, so the total number of nodes = 2 + 240 495. Since the 9th level the last level, all nodes the 9th level nodes. Moreover, the parents of the 240 nodes the 9th level are on the 8th level, with the number parents being 120, which means there are 120 branch nodes on the Sth level, and the rest are leaf nodes. Therefore, the number of leaf nodes on the 8th level is 2(8-1) 120 = 8, Consequently, the total number of leaf nodes 8 240 248.
Open-ended
leaf
Analysis:
None
24

---

Published as a conference paper at ICLR 2025
Table 9: Examples of different languages.
Type
Example
Question:
For a linear list with sequential storage, the operation with time complexity of O(1)
should he ()
A: Sort elements in ascending order.
B: Remove the i-th (1 Σ S vi) element.
C: Change the value of the i-th element (1 S i S vt).
English
D: Insert element after the i-th (1 n) element.
Answer:
Analysis:
The time complexity for sorting n elements is at least O(n) (when initially ordered). and typically (n log, n) O(n²). Options B and D are clearly incorrect. Sequential lists support random access by index.
Question:
A: T W
B:
(1 <
n) パ
C: Er
<
n) A
Chinese
D: ii < "
Answer:
Analysis:
xin H1 O(n loga n) m))
EX 100 4
Question:
Pour une liste linéaire stockée manière séquentielle, l'opération dont complexité temporelle al algorithmique est O(1) devrait être ().
A: Trier éléments dans l'ordre croissant.
B: Supprimer le i-ème Σ Σ r2) élément.
C: Modifier la valeur du ième élément I < n1.
D: Insérez un nouvel élément après le i-ème (1 i vz) élément.
French
Answer:
C
Analysis:
La complexité temporelle pour trier n éléments est au minimum (n)) (lorsque liste est initialement ordonnée), généralement O log, n.) ou O(n²). Les options B et sont manifestement incorrectes. Une liste séquentielle supporte l'accès aléatoire par indice.
Question:
Für sequentiell gespeicherte lineare Listen, welche Operation hat eine algorithmische Zeitkomplexität von {(1) O.
A: Sortiere n Elemente der GröBe nach von klein nach groB.
B: Lösche das i-te (1 < < n)) Element.
C: Andere den Wert des i-ten Elements (1 < n).
D: Füge nach dem i-ten < m) Element ein Element ein.
German
Answer:
Analysis:
Die Zeitkomplexitä für das Sortieren von Elementen beträgt mindestens O(n) (wenn sie anfänglich sortiert sind) und normalerweise O(n log, n)) oder O(n²). und sind offensichtlich falsch. Eine sequentielle Liste unterstützt den zufälligen Zugriff nach Index.
25

---

Published as a conference paper at ICLR 2025
D MORE DETAILS ON EXPERIMENT SETUP
DnI, we present the question templates used to prompt models for each type of task. In D.2 we show the prompts used for GPT-4 score models' answers fill-n-tthe-blank and open-ended questions, and validate the effectiveness of GPT-4's automatic scoring through consistency experiments with human scoring. We also explore the effect of different scoring models. In [.3 we detail the experimental environment used to implement model inference. In D.4 we introduce all the evaluated model families.
D.1 DETAILS OF TEMPLATE FOR EACH TASK FORMAT
We present the templates for querying LLMs with various question formats in Table
Table 10: Prompt Templates for asking various questions LLMs.
Type
Prompt Template
This is multiple-choice question. Please read the question carefully and choose the correct answer. Question: < <Question>>
Which one of the following options is correct? Options:
Multiple Choice
(A) <A>
(B) <B>
(C) <C>
(D) <D>
Please provide the answer to this question directly (a single letter):
This is true/false question. Please determine whether the following statement is true or false. Statement: <Question>>
Assertion
Please give the answer directly (true or false):
You are professor proficient in computer science. This is a fill-in-the-blas question.
Fill-in- the-blank
Give answers the following question without explanation repeating it. Question: <Question>> Answer:
This is subjective Question: <Queettio>>> Please provide brief answer to this question:
Open-ended
D.2 DETAILS OF GPT-4 SCORING
GPT-4 Scoring Prompt. In Table 13 present the prompts utilized to instruct GPT-4 in scoring the outputs of LLMs in CS generation tasks, encompassing both FITB and Open-ended questions.
Consistency between GPT-4 Scoring and Manual Scoring. assess the effectiveness of GPT-4 scoring in evaluating L.M responses, we conduct a consistency experiment between GPT-4 prediction scores and manual scores. For Fill-in-the-lle blankl and Open-ended types, we randomly sample 100 instances from the GPT-4 scoring samples and employ three human annotators to score these predicted results. These three annotators hold bachelor's degrees in computer science. Their scoring criteria were consistent with the evaluation standards provided to GPT-4 and can be found in Table [3 In Table [1] we report the consistency scores among human annotators (measured by Cronbach's alpha), well the consistency scores between the average human annotation scores and GPT-4 scoring (measured by Pearson correlation coefficient). The excellent consistency between human and GPT-4 scores validates the effectiveness of GPT-4 scoring.
Table Consistency between GPT-4 scoring and human scoring.
Consistency
Type
Annotation Count
Human-Gin GPT4
Fill-in-the-blank Open-ended
100
0.808
0.9311
100
0.9494
0.9751
26

---

Published as a conference paper at ICLR 2025
Effect of Different Scoring Models To investigate the impact of different models acting judges, as well as the relationship between the tested model and the judge's 's capabilities on the robustness of the evaluation process, first use the Claude family to construct set of models with varying abilities: Claude3-Haike < Claude3-Sonnet < Claude3-Opus < Sanude..--Sonett (following the official Claude capability ranking (Anthropic (002bb). Next, we pair each models respectively as the judge and the tested model. The results are as shown in Table 12
Table 12:
Performance of tested models under different scoring models.
Claude3-Jaiku (Judge)
Sauuee--oonee Claude3-Opel Claude3.5 S-Sonnet (Judge)
GPT-4
Model
(Judge)
(Judge)
(Judge)
Claude3-Haiku (Test)
64.61
64.56
63.22
62.45
60.69
Claude3-Sonnet (Test)
64.33
63.75
62.13
62.05
60.17
Claude3-Opus (Test)
71.74
71.29
70.46
69.97
68.63
Blaude3.5-Somme (Test)
73.97
73.46
72.43
72.27
71.01
Firstly, when all Claude models act judges, their scoring of the tested models aligns consistently with GPT-4. This effectively demonstrates the robustness of the evaluation process, as consistent ranking results are obtained regardless of whether the tested model's performance is inferior or superior to that of the judge model (Claude3-Haiku and Claude3.5 3.00cs representing the two extremes of judge model capabilities). We believe that long as a model possesses a certain level of discernment, it can serve as a judge, without needing to outperform all tested models, Even the least capable model, Slaude3-Haikee can yield score rankings consistent with GPT-4 during evaluation, while costing only $0.25 1.25 per million tokens for input/output, respectively, significantly reducing evaluation costs. Secondly, we observe that stronger models tend to be more stringent in their scoring. This is reflected in the assessment scores for all tested models, consistently following the order: < Claude3
Claude3- Haiku < Claude3-Sonee because more capable evaluation models can identify deeper levels of error. Lastly, find that the models score fairly and do not tend to give themselves disproportionately high scores (as seen when the same model acts both judge and test model).
< Sauudee.5-Sonee. We believe this occurs
D.3 DETAILS OF INFERENCE IMPLEMENTATION
For all open-source models, we utilize cluster with 8 NVIDIA A100-80GB GPUs to run the inference, and we use vLL.M (Kwon al., 2023) for inference acceleration, applying the corre- sponding chat templates and the same paper-parrmtters: batch size=1, temperature=0,, top-p=1.0,, and moxetokens=2048. For all closed-source models with API access, we also adopt the generation scheme with temperature=0, and simply run the inference with CPUs, which typically completes within a day. During the evaluation of GPT-4, we also applied the setting of temperature=O. To avoid error bias, conducted the experiments times and took the average of the scores. For models supporting web search or tool calls, we disable these features to ensure a fair comparison.
D.4 DETAILS OF THE MODELS BEING EVALUATED
Gemma (Tea et al. 2024) is a family of lightweight, open models from Google, built from the same research and technology used create the Gemini models. They are text-to-text, decoder-only large language models, available in English, with open weights, pre-trained variants, and instruction- tuned variants. The Gemma model excels academic benchmarks in language understanding, reasoning, and security. Gemma publishes models in two sizes (2 billion and 7 billion parameters).
Llama2 Touvron et al. 2023) is an upgraded version of Llama developed by MetaAI. It uti- lizes more robust data cleaning and mixing techniques, and up samplee sources closest to factual information, which can enhance knowledge and reduce hallucinations. Additionally, employs Grouppd-Queet Attention technology to lessen reliance on memory.
Llama3 Meta 2024 is the latest generation of large language models developed by MetaAL. The training dataset for Llama 3 is seven times larger than that used for Llama 2, with the amount of code included being four times that of Llama 2. Compared to previous versions of the model, it has seen a tremendous enhancement in reasoning, code generation, and instruction following capabilities.
27

---

Published as a conference paper at ICLR 2025
Table 13: Scoring Prompts for Fill-in-the-blank and Open-ended Questions.
Type
Prompt Template
teaching assistant. As TA. your task is to grade the fill-in-the til-in-thh-blanle the-klank assignments of computer science students. You will see the standard answer for each question (these answers are verified and completely correct), and you need to score the students' answers based on this. If the student's 's answer conveys the same mcaning as the standard answer or other answers (different formats also considered correct), then award point; if not, then 0 points.
You
Fill-in- the-blank
Question: <question>
Standard Answer:
Other Answers:
Student Response: <predict.output>
Score (U or 1):
You are now serving as a teaching assistant. this role, your task to grade the subjective homework assignments of computer science students. You will be presented with the standard answers for each question (which are verified and completely correct), and you must use these to score the students' responses. The grading scale ranges from I 10 points, with 10 being the highest and being the lowest. When grading, please take into consideration the accuracy, relevance, completeness, and depth thought of the answers. Scores should be assigned based on the following #criteria::
First Tier: 1-3 points
Accuracy: The answer contains several fundamental errors, showing limited understanding. Relevance: The answer has low relevance to the question and standard answer, with most content straying from the requirements. Completeness: The answer omits multiple key points, failing cover the main aspects of the question.
Second Tier: 4-6 points
Accuracy: There are some errors in the answer, although most of the basic concepts are understood correctly.
Relevance: The answer is generally relevant the question and standard answer, but some content does fully conform to the requirements. Completeness: The answer fairly complete, but lacks some important details or certain key points are not fully claborated.
Open- ended
Third Tier: 7-8 points
Accuracy: The answer almost entirely correct, with only very minor errors. Relevance: The answer is highly relevant to the question and standard answer, focused and with almost no deviation from the topic. Completeness: The answer comprehensive and detailed, covering all key aspects very well.
Fourth Tier: 9-10 points
Accuracy: The answer is free of any errors, demonstrating deep understanding and precise grasp of the issue.
Relevance:
The answer is in complete accordance with the requirements, strictly aligned with the question and standard answer, without any deviation. Completeness: The answer is structured rigorously, logically organized, and systemati- cally covers all aspects of the question.
When assigning score, please first make a preliminary assessment accuracy based on the student's 's answer compared to the standard answer. Then, consider the relevance and completeness to determine the final score. Ensure that each point awarded is based fair and justified comprehensive evaluation.
Grading Guide:
Llama3.1-405B (Dubey et al. 2024) is an auto-regressive language model developed by MetaAI, utilizing optimized transformer architecture. Trained over 15 trillion tokens, it demonstrates excellent flexibility, control, and advanced capabilities. Llama3. 1405B is the largest openly available foundation model released by MetaAT.
28

---

Published as a conference paper at ICLR 2025
(Wagg& & Zheng, 2024 is instruction-tune tutoond--dddddd- language model for Chinese and English users with various abilities such as roleplaying and tool-using built upon the Meta-Llama-3- 8B-Instruct model.
Llama3-Chinese
ChatGLM3 ZZng et al. 2022 is a next-generation conversational pre-trained model jointly released by Zhipu Al and KEG Lab of Tsinghua University. ChatGLM3-6B adopts a newly designed Prompt format, in addition to regular multi-tury dialogue. It also natively supports complex scenarios such as function call, code interpretation.
Baichuan2 Yang et al. 2023 is large-scale multilingual model developed by Baichuan Company. It adopts several advanced techniques in its design and training process, including Rotary Position Embedding, a novel position encoding technique, SwiGLU activation function, and memory efficient attention mechanism. Compared with Baichuanl, its performance has been greatly improved.
InternLM2 (Cai et al. 2024 is open-source large-scale language model developed by Shanghai AI Laboratory. This model has good processing ability for ultra long texts and adopts COOL RLHF technology. It solves human preference conflicts through conditional reward model and performs multiple rounds of online RLHF to improve the model's ' alignment ability.
Qwen1.5 (Bai al. 2023) is family of language models developed by Alibaba. It has features such as SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Qwen 1.5 series models have strong basic capabilities including language understanding.
Mistral-7B Jiang et al. 2023), 7-billion-parameter language model designed for superior perfor- mance and efficiency, which is developed by Mistral AI. Mistral leverages Packet Query Attention (GQA) for faster inference, combined with Sliding Window Attention (SWA) to efficiently process sequences of arbitrary length while reducing inference costs.
Mixtral-8x77 Jiang et al. 2024) Sparse Mixture of Experts (SMoE) language model de- veloped by Mistral AI. Its architecture is the same as that of the Mistral 7B, except that each layer consists of 8 feedforward blocks (i.e., experts). Mixtral has demonstrated exceptional abilities in math, code generation, and tasks that require multilingual understanding.
Mistral-Large- 123B M 2024) is a language model developed by Mistral AI with 128K context window, supporting dozens of languages including French, German, Spanish, Italian, Russian, Chinese, Japanese, and Korean, well over 80 programming languages. It is designed for single-node inference with long-context applications in mind its size of 123 billion parameters allows at large throughput single node.
DMM (Bi et al. 2024) is family of models released by DeepSeek-Al, and its core architecture borrows from the Llama model. This family of models employs Multi-Head Attention (MHA) and Group Query Attention (GQA) techniques, which significantly enhance their performance and efficiency. Furthermore, DeepSeekLLM demonstrates strong bilingual capabilities in both Chinese and English.
PaLM-2 Anil al. 2023) is the nigher-performance successor to PaLM released by Google, which differs in terms of dataset mixing. Compared to the first-generation PaLM version, it uses a smaller model but performs more training calculations. It also relies more diverse pre-training targets.
Claude2 Claude2. (Anthropic 2023) and Claude3 (Anthropic) 2024a) are AI models developed by Anthropic, showcasing advanced language understanding and generation capabilities. Utilizing the constitutional AI framework, Claude models are designed to ensure helpfulness and trustworthiness. Souude-3.5-Sonee (Anthropic 2024b) is the first product in the Claude-3.5 model series developed by Anthropic. It demonstrates significant improvements in understanding nuance, humor, and complex instructions, and excels generating high-quality content with natural, relatable tone. It possesses
29

---

Published as a conference paper at ICLR 2025
exceptional coding capabilities, allowing it to independently write, edit, and execute code, along with advanced reasoning and troubleshooting skills. Additionally, is Anthropic's most powerful visual model to date.
GPT GPT-3.5 OpenAA 2022), GPT-4 (Achiam et al. 2023) and GPT-4o (OpenAl 2024a), released by OpenAI, are part of the GeT-series models enhanced by three-stage reinforcement learning with human feedback (RLHF) algorithm. This algorithm not only improves the models' ability to follow instructions but also significantly reduces the generation of harmful or toxic content. Additionally, GPT-4 supports image inputs and achieves human-level performance on various benchmarks. GPT-4o, the latest model developed by OpenAI, boasts powerful real-time reasoning, language interaction, and multimodal capabilities.
OpenAI-ol (OpenAl 2024b) a new large language model developed by OpenAl, trained with reinforcement learning handle complex reasoning. It can produce long internal chain of thought before responding to the user, and ol refines its chain of thought and improves its strategies by learning to recognize and correct mistakes, break down complex steps into simpler ones, and adopt alternative approaches when the current method fails. It currently has two versions: ol op-preieew, which has strong reasoning capabilities and broad world knowledge, and ol-mini, lightweight version with faster reasoning speed.
GLM-4 AI 2024 is a new generation base large model developed by Zhipu AL It has strong tool calling and multi-mot capabilities, well strong mathematical reasoning ability and code generation ability.
ERNIE (aidu 2023) ERNIE3.5 and ERNIE4 are large language models developed by Baidu. ERNIE3.5 is capable of processing text data in multiple languages and has good understanding and representation ability for entities and relationships in text. Ernie 4 has adopted more advanced knowl- edge graph information and more advanced knowledge integration technology, further improving the performance of the model.
30

---

Published as a conference paper at ICLR 2025
E MORE DETAILS ON EXPERIMENT
we present detailed performance of the models on CS-Bench (EN), including the leaderboard, task formats, and domains. In E.2 we describe and validate the design of the scale-score fitting function. In [ we compare models' performance across different contexts, with a focus on evaluating and analyzing the model's performance on CS-Bench (CN). E.4 we conduct case studies to better understand the specific details of the models' failures on CS Bench. In E.5. We further analyze and interpret the relationship phenomena among Code, Math, and CS abilities from internal representations and data characteristics. In E.6 based on our findings, we explore and experiment with several specific approaches to enhance CS capabilities. F.7 from OpenAl-ol and compare them with GPT-4o.
In
we analyze examples
E.1
DETAILS OF MODEL PERFORMANCE
The Leaderboard on CS-Bench. We visualize the results of LLMS on CS-Bench in Figure
OpenAl-o 1oann--o--preview..
80
OpenAI-ol-ming
Score (90 CS S-Bench (EN)
GPT-4g laude:3
Mistral-Large: 123
70
Qwen1 1-11-155181
GPT4
Llama3-70B
Claude3
Qwen 1.5-728
Qwen1 55-328
60
Qwen1.5-148
55-14
Mixtral-8x71
DeepsockLLM-67B
Claude2
PaLM2E
Llama3-8B 8B
InternLm2-200
GPT:3.5
Mistral-7B
Llama2-70B
InternI.22 m2-7B
50
Qwen1.5-7B
Cemma-7B
Paichyan-13B
Qwen1.5.4DB
Liama
ChatGi M3-6B
DeepseexL
Baichuan2-7B
wen 5551.88
Gemma-
Llama2-7B
2B
13B
32B
70B
110B
405B
Closed-Souree Source LLM
Model Scale (Billion Parameters)
Figure 12: The leaderboard of LLMs on CS-Bench (EN).
Detailed Performance on Each Task Format. We present models' performance on four types of
tasks in Table 14 and visualize the results in Figure 13
Chaidtacs-d
Baldwann2 Coemma776 77 Oweeg 5/B
Ioteminn Misteal-1 -1 Limsssi-ee
Pa1M2
Cauade-L5
Despondds Tt
Limmit--8 1M
Manx2-72n
Mistra-aaggg
G2733.5
Cleado 2.1
Openatioz-man
Opiaxiaded
Opss-saded
ztendedge
Acardia
Ausrtio
TRAN PUTTE
Assiction
10 20
50 21
Amerten
PCD scoeledge
Acsertoo
reesoting
Opsn-endad
Karwindje
Small spcn-
Large open-source
models
Figure 13: Performance of various LLMs for each ability dimension about task formats.
Detailed Performance Each Subfield. In Figure 14 we visualize the models' knowledge and reasoning performance across the four domains of CS-Bench. Subsequently, we focus on the models' performance in 26 fine-grained subfields. Table 15 presents the results of eight representative models.
31

---

Published as a conference paper at ICLR 2025
Table 14: Zero-shot scores (%) of LLMs across question formats on CS- (EN).
Model
culliplechooce
Assettion
Fill-in-lie-blnkk
Open-cnded
All
Kig
Rng
Avg
Kig
Rug
Avg
Klg
Rng
Arg
Kig
Rog
Avg
Kig
Rug
Avg
Random
25.00
25.00
25,00
50,00
50.00
50.00
0.00
0.00
0,00
10.00
10,00
10.00
27.4
24.12
26.20
Open-source
(Scale <. 10B)
Gemma-2B
46.87
25.85
38.74
52.58
48.48
51.64
34,12
7.55
27.68
49.60
26.02
32.96
46.89
27.59
39.86
Qwent ,548
53.00
15.47
46.22
56.84
58.55 59
57.24
29.41
11:32
25.02
55.40
28.58
36.47
51.18
35,70
45.54
Cha6GLM3-6
33.07
31.76
5.66
34.07
Liana2-78
47.00
28. 06
39.67
56.84
60.61
57.70
23.53
5.66
19.20
63.80
26.19
37.25
47.15
30.48
41.08
DospscckLLMM77 7B
50.19
28.05
41.63
60.49
58.59
60.06
31.76
13:21
27.26
59.50
28.67
37.83
50.87
31.11
43.67
Baichuan2-7B
47:51
35.27
42.77
57.14
59.60
57.70
32.94
7.55
26.78
52.10
26.90
34.40
18.29
35.33
13.57
Gemman7B
56.70
33.07
47.56
58.05
57.94
38.82
15.09
33.06
54.90
47.66
Qwent 5-7B
59.90
40.08
52.23
58,97
56.57
58.42
18.24 24
16.98
33.08
69.00
35.75
45.71
57.34
40.08
$1.05
IntemL.m2-78
59.26
19.48
51.61
60.49
55.56
59.36
45.88
15.09
38.41
69.00
39.03
47.84
58.31
39.77
51.56
Mistral-7B
50.51
1.ama3-88
61.81
46.09
55.73
61.62
63.80
38.24
11.32
31.71
67.60
41.33
06
59.75
44.97
54.37
Open-source LLM (Scale >> 10B)
1.aam2--188
50.06
33.87
43.79
55.93
56.57
56.08
44,71
22:
19.36
62.00
29.65
16
51.31
35,46
45
Baichuan-138
53.00
37.68
47.07
58.66
57.49
31.30
31.15
47.03
Qwent
64.62
50.70
59.23
62.61
59 60
61.92
$1.76
28. 30
46.07
70.60
43.44
51 44
62. 79
49. 18
57 83
Inteml Lm2.208
62.20
43.69
55.04
61.09
62.63
61.44
$1.18
24.53
44.72
62.20
36.02
45. 19
60.81
43.66
54.56
Qwenl 5-3228
65.71
22.46
54.72
Miixtral-Sx77B
66.28
47.09
58.85
67.78
56.57
65.22
58.24
26.42
50.52
71.00
45.93
53.30
65.91
46.66
53.90
DccpseokLLM-675
66.92
15.29
58.55
65.96
63.64
63.43
$1.71
28.30
18.30
67.20
42.57
49.51
65.23
45.96
58.22
Lluma2-700 70B
58.88
42.28
52,46
61.09
59.60
60.75
51.18
16.98
42.88
63,80
34,96
43.44
58.73
41.68
52.52
Llama3-70B
57.52
67.59
63.53
37.74
57.27
71.65
56.36
5-772
72.03
60.32
67.50
70.52
66.67
69.64
55,29
28.
48.74
73.00
$2.30
58.3 39
69.63
57.75
65.31
@wen1,5-1108
74.33
62.73
69 84
73.25
68
71 M8
57.06
33. 96
51 46
75.20
60.000
64.47
71.98
60.91
67 97.95
Misral-I 12g-1138
78.29
66.13
73.58
72.22
60.00
55.52
74
64.37
71.08
Llama3,1-4055
79.57
63.93
73.52
72.04
65.66
70.58
65. 29
56.00 60
63.18
70.80
58.58
62.17
75 64
62.81
70.96
Cloved-source LLM
PaLM-2
65.91
43.66
57.30
66.36
62.77
65.54
56.52
29.79
50,04
64..4
35.64
44.12
64.85
44.01
57.26
Claude-2.1
63.47
46.89
57.05
66.87
67.68
66.066
49.41
24.53
43.38
72.40
55.84
60.71
62.97
49.42
58.04
Clande33-pus
73.82
61.32
68.98
73.56
70.71
72.91
62.94
37.74
56.83
76.73
66.11
69:23
72.57
61.75
68.63
GPT:3.5
63.35
41.48
54.89
68.39
63.64
67.30
48.83
24.53
42.93
68
42.65
50.11
63.04
43.45
55.91
GPT-4
77.27
61.18
74.85
Claude-3.5-
77.14
64.33
72.18
73.25
61.65
71.28
69:11
58.49
66.76
73.60
64.16
66.94
75.13
63.97
71.07
GPT-40
80.08
63.73
73:75
75.68
72.73
75.01
64.71
41.51
$9.08
75.20
69.47
71.16
76.95
64.15
72.29
Special Reasoning LLM
OpenAl-ol-aint SpenAl-ol-preveww
80.46
77. 56
79.34
76.29
75.76
76.17
67.06
71 71:70
68
75 60
74
73.55
77 60
76.12
77 77.06
87.99
83.57
86.2
79.33
77.78
78.98
72.35
73.58
72.65
80.60
76.73
77.87
83.61
80.98
82.65
Chaidlaca-a dB
Balchaan2
Inteminna
Mitri-8a77 Deepens
Owanl.5-18
PMlM2 (2733
Caude-)
Cauate-.5
Mtskedd-11 Lians
Oreel
un/
Openkloo--man
Despensd-77 73
Owi213377
Lans-3-20
Elaaad 1-455
Claude221
GP74
Openal-si-gneedoe
06 rouusizg
DSA Stondedg
05 2aaa
2N
CID
40
60
DO
CU
czontedge
dovidedg
karwladge
anzaleige
(b) Small open-sourre oe models
(b) Large ogen-source models
(4) Close-tource models
Figure 14: Performance of various LLMs for each ability dimension about CS domains.
Firstly, we can observe significant variations in scores across different subfields within the same domains for the models. Taking the DSA domain as an example. Llama2-70B scores range from 45.44% to 76.67% across different chapters (average 56.93%). while GPT-3.5 scores range from 55.17% to 80.00% (average 60.67%). Secondly, the performance of different models the same subfield is generally consistent compared to the average scores. For instance, all models perform above the average scores in the "Overview" and "Stack, Queue, and Array" subfields of DSA but below average in the "Tree" and "Graph" subfields. These detailed scores allow us to understand which content poses greater challenges for the models and provides guidance for improving the models' performance in computer science by strengthening these weaker subfields.
We further observe that although the overall scores of models from the same family increase with scale, not all chapters follow this pattern. As shown in Figure 15 the Llama2 series exhibits trend
32

---

Published as a conference paper at ICLR 2025
Table 15: Detailed scores models on fine- tne-grained subfields.
Llama2-78
Mixtral-8:2B
GPT-33
Data Structure and Algoritiu
Overview
36.67
51.11
59.44
68.06
73.33
68.06
71.11
74.17
Stack, Queue, and Array
49
50.91
57 57.40
$7.66
58.96
71.95
61.43
76.49
String
76.67
66.67
76.67
66.67
70,00
80.00
80.00
70.00
Tioe
Graph
43.80
37.47
45.44
65.70
34,56
68.23
56.96
68.61
Scarching
51.29
$2.00
61.14
60.57
$4.86
56.71
58.14
74,86
Sorting
30.52
37.27
56.10
52,08
54.55
71.56
62.21
74.68
Average
60.67
Computer Organization
Overview
51.20
61.40
61,60
76.40
68.20
80.20
73.20
$1.80
Representation Operation
Storage System
41 M1
46.10
58.00
61.70
53.60
68.10
56.20
68.50
Instraction System
51.76
53.68
57.79
$9.56
53.82
75.74
65.29
80.44
Processing
Bus
60.70
59.12
61.40
66.32
47.37
71.75
66.49
73.33
Input/Otput System
37.58
35.48
29.19
52.42
44.03
52/12
35.48
$8.23
Average
44.70
48.17
$1.44
60.20
51,20
65.91
56.29
70.66
Computer Network
Overview and Architecture
$2.15
48.31
$8.77
62,77
58.15
68.62
57.23
69.18
Physical Layer
42.11
17.61
52.25
57.89
53.52
65.77
54,51
69.01
Layer
Network Layer
38.40
48.78
58.47
62.37
65.19
75.57
62.98
77.48
Transport Layer
42.95
48.72
66.28
70.77
63.46
81.79
61.54
86.79
Application Layer
Average
42.50
48.25
56.41
62.81
39.04
71.12
60.17
74.37
Operating System
Overview
74
Procesees Threads
34.14
12.61
13.57
55.73
50.83
63.57
47.58
66.83
Memory Management
31.63
42.04
$2.04
51.02
53.67
60.71
51.02
70.41
File Management
40.00
19.34
57.37
54.87
55.66
61.97
56.32
64.08
Inpot/Ourput Managementt
Average
36.08
42.29
48.60
55.58
53.58
61.44
48.96
65.88
Overall
41.08
45.54
52.52
$8.90
$4.37
66.08
55.91
70.41
50
50-
Lai
QuieRAnE Arro)
Stoop
Trans
Graph
Searchung Sorting
Ceuthoon:
Hepersor Openddie
Sterage
Hyslem Processan
9ystmn
Data Straemir and Algorittm
(b) Computer Organization
and
Laye
Layer
Layer
Laste
and Thnads
Masspsoeet
Maragezsnn
Computer Natwork
(4) Openaing Systum
Figure 15: The performance of the Llama2 series models in each subfield.
of scores increasing with scale most subfields (17 out of 26 subfields); however, there are some exceptions. For instance, Llama2-7B performs exceptionally well in the "string" chapter of DSA, while Llama2-133 excels in the "Data Representation and Operation" chapter of CO, surpassing the performance of Llama2-70B.
E.2 SCALE-SCORE FITTING FUNCTION FOR CS-BENCH
To enhance CS performance, large-scale models are often utilized: however, these models demand more computational resources for both training and deployment inference. Therefore, it is desirable
33

---

Published as a conference paper at ICLR 2025
70
70
Overall Sco78 (%)
50-
50
30
30
Piiting (0.=0.09 6222.12)
1Juma2 Fitting Curve 16,-0.54 0,
Qwenl Ground/Tratb Score
1JumaZ Ground-Truth Score
10
10
Qwen1 55-108 Prodiction
Owee1 5
1Jama2-708 Prediction
7B
146
328
720
1106
78
328
728 Parameter (Linear)
1103
Parameter Scale (Logaritmicc
(a)
(b)
Figure 16: The logarithmic scale-score performance and scale-score fitting curve of Qwen 1.5 and Ilama2 series.
establish relationship between model scale and CS performance, enabling the prediction of theoretically larger models' scores on CS-Bench based on the performance of smaller-scale models. The established fitting function should adhere the following criteria:
1. The score should monotonically increase with the increase in model scale, approaching 0 as the scale approaches 0, and approaching (100%) the scale approaches infinity.
2. As illustrated in Figure 16 (a), when the model scale varies exponentially, the score should exhibit an approximately linear trend.
3. Due to variations in performance and change slopes among different model families at the same scale, the fitting function needs to incorporate mondel-family-speeiff hyperparameters.
Guided by these criteria, we experiment with various functions and find the following function to satisfy the conditions and work best:
Score 1 01logio(22 Scale 1)
(I)
Where 0, and 02 are hyperparameters specific to the model family. To validate the effectiveness of function, we estimate hyperparameters based on the minimum mean square error on small-scale models and predict performance scores on larger-scale models. For the Qwen1.5 family, we use models of 14, 32, and 72B to predict the LOB model's performance. For the Llama2 series, we predict the 70B model's performance based on 7B and 13B. As depicted in Figure (b), for Qwen1.5 10B, the predicted score (67.83%) closely matches the true value (67.95%). For Llama2-70B, with only reference data points, the predicted score (55.08%) deviates from the true value (52.52%) by only 2.56%.
E.3 MODEL PERFORMANCE ON CS- ACROSS DIFFERENT LANGUAGES
We first compare the performance of multilingual models in different languages within CSBench, as shown in Figure 17, It is observed that the Llama2 and Llama3 series show significant decline in performance for languages other than English, such as Llama3-8B, In contrast, GPT-4o maintains good balance across multiple languages. Next, we select the Chinese model case outside of English, conducting in-depth tests on Chinese-oriented models and comparing their performance differences between Chinese and English.
Performance on CS-Bench (CN). We assess models that support Chinese CS-Bench (CN). The foundation models include the LLama3 and GPT-4 series, which are not specifically optimized for Chinese, well as Chinese-orienter open-source models, including ChatGLM. Baichuan2, ImternLm2, Qwen 1.5 and Haanee-chnese series. We also evaluate Chinese-oriented closed-source models, including GLM-4 and ERNIE-3.5/4,, Details of these models provided in Appendix D.4
34

---

Published as a conference paper at ICLR 2025
70
Chipese
Prench
00
German
English
50
40
30
LJamm2-718
Liammai--ts
Figure 17: Comparison models different languages on CS-Bench. Due to the Ilama2 series not supporting Chinese, we ignore their results in CS-Bench (CN).
Table 16: Zero-shot scores (%) of LLMs across domains on CS- Be-Bench (CN), where "Klg" represents knowledge-typee type, "Rng" represents reasoning-lyne type, and "Avg" represents Average.
Model
Data Stroe & Algo
Computer Organication
Computer Network
Operating System
Overall
Kig
Rng
Avg
Kig
Rag
Avg
Kig
Rag
Avg
Klg
Rng
Avg
Klg
Rng
Avg
Random
28.04
21.63
26.65
26.57
25.21
26.13
26.34
22.19
21.98
29.06
24.23
27.27
27.4
21.12
26.20
Open-source
M5-6B
M3-68
41.74
32.48
31.97
44.177
34.91
41.05
49.02
32.31
43.14
43.02
32.
35.98
44.67
33,09
40.45
Baichuan2-78 7B
42.04
31.51
33.75
44.93
37.88
42.61
50.74
31.11
43.83
42.18
34.07
39.16
45.27
33.47
40.97
Imtem1.m2-77
34.54
39.61
Qwent
37.71
57.62
Llana3-88
50.47
29.68
42.01
50.81
36.30
46.03
56.09
42.21
51.21
52.01
38. 38.85
47.12
52.46
36.61
46.69
1Jama Chi--heeee
49.20
33.72
42.90
54.99
33.09
47.77
58.77
48.59
55.19
55.58
41
50.20
54. 84
39.17
49.13
Open-source
(Scole
100)
Baichuan2:155 13B
48.83
34.68
43.07
54.18
36,00
48.18
$5.11
39.85
49.74
49.19
40.27
45.88
52.10
37.63
46.83
Qwen1.5-14p 1-BB
51.47
48.81
50.39
64.43
46.85
58.63
68.69
55.18
63.94
69.58
56.59
64.76
63.78
51.81
59.42
IntemLm2-220B
51.97
38,03
46.30
58.36
45.76
54.20
60.60
50.50
57.05
58.70
45.66
53.86
57.59
44.85
52.95
Qwent
Llamaá-708
53.28
55.41
54.15
67.97
49.58
61.91
71.07
61.81
67.81
65.29
57.36
62.35
64. 86
56. 18
61. .10
5-728 --78
58.16
57,02
55 66
70.28
52.91
64.55
75.25
66.23
72.08
74.12
19
70.06
69.73
58.52
65 64.64
Cloved-
GFT:3
54.15
39.63
48.24
60,86
43.27
55.06
64.29
48.89
58.87
56.36
84
50.22
59.27
42.96
53.33
GPT-4
60.03
60.28
60.13
77.60
60.24
71.88
73.50
72.86
73.27
71.46
65.60
69.29
71.06
64.80
68.78
GPT-to
61.67
66.45
63.62
78.86
55.32
71.10
78.61
74.17
77.05
72.66
69.94
71.67
73.46
66.69
71.00
53.16
55.62
57.13
74.56
58.73
69.34
74.68
65.16
71.33
72.13
63.37
68.94
70.28
60.63
66.77
ERNIE-4
57.92
62.33
39.72
78.24
64.18
73.60
76.27
69.74
73.97
75.84
69.54
73.54
72.49
66.36
70.26
Table 17: Zero-shot scores (%) of LLMs across task formats CS-Bench (CN).
Moddl
Muliiple choice
Assertion
Fill-in-th-
Open ended
Overall
Klg
Rng
Avg
Kig
Rug
Avg
Kig
Ring
Avg
Kig
Rag
Arg
Kig
Rag
Avg
Random
25.00
25 00
25.00
50000
50,00
50.00
0.00
0.00
0.00
10.00
10.00
10.00
27.4
24.12
26.20
Open-source LLM (Scale (08)
ChaiGLM3-6t
22
Baichuan2- 7B
64.96
32:26
40.05
53.80
56.57
54.43
29.41
13.21
25.48
47.20
27.52
33.31
45:27
33.47
40.97
Internl.m2.70 m.
$1.09
40.08
46.83
59.88
55.56
58.89
44.12
18.87
38.00
60:50
33.27
41.37
52.71
39.61
47.94
Qweni_55.B 59.64
48.50
55.33
60.79
50.51
58.44
42.35
15.09
35.74
58.20
30.35
38.54
57.62
43.79
52.59
$3,26
35.67
57.00
16.98
49.60
29.47
36.61
Saanu3--B--hhine
55.43
40.08
49.49
59.57
56.57
58.88
42.94
16.98
36.64
55.66
30.62
37.97
54.84
39.17
49.13
Openssource LLM (Scale 108)
Baichuan2-13B
47.22
Owent 5-14B
67.82
57.72
63.91
65.05
56.57
63.11
43.53
24.53
38.92
63.80
34.96
43.44
63.78
51.81
59.42
Intern m2.208
58.49
46.89
54.00
59.57
54.55
59.42
47.06
26.42
42.05
67.00
35,40
44.69
$7.59
44.85
$2.95
Qwent 5:328
71.26
68.74
70.28
64.74
63.64
64.49
51.76
28.30
46.07
63.40
42.04
48.32
66.77
61.35
64.80
Llama3-708
60.32
63.822
66.57
66.34
59.24
33.96
52.35
40.71
56.11 56.18
Qwenl S-72B
72.11
67.74
70.60
72.34
55.56
68.51
$1.71
28,30
48.30
63.80
34.96
43.44
69.73
58.52
65.64
Closed-sowre LLM
59:27
GPT-4
73.31
67.13
70.92
72.04
67.68
71.04
62.35
60.38
61.57
60.40
54.16
56.00
71.06
64,80
68.78
GPT-40
75.92
69.33
73.37
73.36
68.69
72.68
62.94
50.94
60.03
70.20
62.92
65 06
73.45
66.
71
ERNIE-3.5
72.24
63.71
68
69.30
61.62
67.55
63.91
50.94
60.76
70.40
51.95
70.28
60.63
66.77
ERNIE-4
73.55
70.35
72.31
72.34
56.57
68.74
70.00
67.92
69.50
68.10
58.32
61.28
72.19
66.36
70.26
35

---

Published as a conference paper at ICLR 2025
English
70
Chinese
Score (%)
20
ClanCLM3 Buicharn Buchoat
Owend 5 InteraCar2
Liana3
Istemdan 208
GPT33
Qweet. 148
Qweul. 328
Oweel. 728
Llama8
CPS4
CPT4
68
7B
188
75
86
738
Figure 18: Comparison of models in English and Chinese on CS-Bench.
As shown in Table 16 and Table 17 the scores of these models on (S-Beech((C) range from 40.45% 70.26%. Despite not being specifically optimized for Chinese, GPT-4o still achieves the best performance. Among the Chinese-oriented models, ERNIE-4 outperforms GPT-4, achieving performance close GPT-4o. Additionally, ERNIE-335 and GL.M44 score similarly, slightly lower than GPT-4's performance in Chinese. Notably, Allama3-8B-chineee surpasses Llama3-8B by 2.44%, highlighting the importance of adapting models to specific languages. We further compare the performance of the models on CS-Bench(EN) and CS- CS-Bench(CN) in Figure [8] Compared to English, the GPT and Llama3 series, which are not optimized for Chinese, perform worse on Chinese context. For instance, Llama3-8P experiences a decrease of 7.68% on Chinese, and Llama3-70B drops by 4.38%. Although some Chinese-oriented models also show slight decreases performance in the Chinese context, such as Interl.m2-200B, the decline is much less significant than that of the Llama3 series. Moreover, the Qwen1 series even demonstrates improved performance on Chinese tasks. Finally, observe that larger models within the same family are less affected by different languages, as reflected in Rhichuan2-7/133 Internlm2-7/20B, and Llama3-8/705 8/70B.
E.4 CASE STUDY OF ERROR TYPES
We first introduce the error types of knowledge-type questions and reasoning-type questions in Table 18 and Table [9] To facilitate a better understanding of each error type, we provide examples of cach error type made by GPT-3.5 in knowledge-basee and reasoning- based questions in Table20 and 21 respectively. Additionally, Table 22 presents several examples that contain multiple error types.
Table 18: Error types for knowledge-type questions.
Error Type
Explanation
Complete
For a certain piece of knowledge or concept, the model is completely unaware of it or has misunderstood all of it.
conceptual error
Partial
For a certain piece of knowledge concept, the model has only grasped part of it or has misunderstood part of its content.
conceptual error
Conceptual confusion
For multiple pieces of knowledge concepts, the model has incorrectly matched their names with their attributes.
Table 19: Error types for reasoning-tyy questions.
Error Type
Explanation
Concept- related emror
The model misunderstands certain pieces of knowledge, causing incorrect answers, This includes the three types of knowledge-basee emoor mentioned above.
Logical reasoning error
The model made logical reasoning errors when answering computer sciencc-related questions.
Mathematical reasoning error
During the response process, errors in numerical calculations, symbolic calculations, and other computational processes caused the model produce incorrect answers.
36

---

Published as a conference paper at ICLR 2025
Table 20: Example of knowledge-type question error types. We use red to mark errors in the output and blue indicate the reasons for the errors.
Error Type
Example
Question:
When two sequential stacks share same one-dimensional array space, how are their stack bottoms and stack tops set?"
A: Two stacks with the same bottom end, extending in opposite directions from the top.
B: Two stacks with tops the same end and bottoms extending in opposite directions.
C: The bottom and top of the stack are at the same end.
D: Two stacks with bottoms at opposite ends and tops extending towards the middle.
Answer:
D
Complete conceptual
Wrong Output:
When two sequential stacks share same one-dimensional array space, their stack bottoms and stack tops are set as follows: Two stacks with tops at the same end and bott- oms extending in opposite directions. This means that the top of one stack and the bottom of the other stack are at the same end of the array, while their bottoms extend in opposite directions. This allows for efficient utilization of the array space for both stacks. The correct answer (B), Answer: (B) Two stacks with tops at the same end and bottoms extending in opposite directions.
error
Error Details:
In GPT-3.5's response, it was mentioned that the top of stacks are the end, while the bottom the stack extends opposite directions. This reflects that GPT-3.5 completely not familiar with the implementation of shared stacks.
Question:
What are the main functions of the network layer?
A: Data Transparent Transmission
B: Power Management
C: Signal Amplification
D: Data Compression
Answer:
A
Partial
Wrong Output:
conceptual error
The main functions the network layer are data transparent transmission and data compression. The network layer is responsible for routing and forwarding data packets between different networks, ensuring that data is delivered to the correct destination. It does not handle power management signal amplification. The correct answers A. and D.
Error Details:
GPT-3.5 correctly answered that transparent data transmission is a function of the network layer. but at the same time incorrectly pointed out that data compression is the main function of the network layer. witch indicating incorrect understanding of certain aspects of network layer functionality.
Question:
What are the basic steps simple selection sort? A: Select the element with the smallest key from the elements to be sorted and swap it with the first element. B: Insert elements into the sorted sequence in order.
C: Compare adjacent elements in sequence and swap them.
D: Use divide and conquer method sort elements.
Answer:
D
Conceptual confusion
Wrong Output:
The basic steps of simple selection sort are: (A) Select the element with the smallest key from the elements be sorted and swap it with the first element. (C) Compare adjacent elements in sequence and swap them. The correct answer is A and C.
Error Details:
The answers output by GPT-3.5 are A and C, and it is claimed that the basic steps of simple selection sorting are select the smallest element and exchange with the first
element, compare adjacent elements and exchange them. This is obviously incorrect because comparing adjacent elements and swapping them is a Bubble Sort This indicates that GPT confuses bubble sorting and selection sorting.
37

---

Published as a conference paper at ICLR 2025
Table 21: Example of reasoning-tys question error types.
Error Type
Example
Question:
In the topological ordering of the directed graph if vertex precedes vertex Vj, then the following situation cannot occur: O
A: G contains an are Vi, Vj>.
B: There is a path from Vi to Vj in G.
C: There is
<Vi, Vj> in G. There is path from Vj to in G.
Answer: D
Concept- related error
Wrong Output:
If vertex Vi precedes vertex Vj in the topological ordering of the directed graph G, then it that there directed path from Vi to Vj G. Therefore, the tuation where there is no are <Vi, Vja cannot occur. The correct answer (C) There is no are
<Vi, Vj> in G. Error Details:
The sequence obtained topological sorting has before j. GPT-3.5 chose C in the answer, but according to the topological sorting rule, placing before does not necessarily indicate that two nodes can be directly connected. GPT-3.5 does not under- stand the meaning of topological sorting.
Question:
In the arryy representation of a linear list with elements, the operations with a time complexity of O(1) are I. Accessing the ith (1 S r) node and finding the immediate node after the last node. III.
predecessor the ith (2 < S n) node. II. Inserting Deleting the first node. IV. Sequentially outputting the values these elements.
A:T
B: II, III
C: II
Logical reasoning
D: II.
Answer: C
сттоr
Wrong Output:
The operations with time complexity of O(1) in the array representation of linear list with elements are: I. Accessing the ith (1
n) node and finding the immediate predecessor the ith(2 S S n) node. II. Inserting a new node after the node, III.
Deleting the first node. Therefore, the correct answer is (D) I, II, III. Answer: D
Error Details:
For item in the question, after deleting the first element of the array, the n-1 elements need to moved forward, resulting a time complexity o (n). However, GPT-3.5 considers it be o(1), which is a GPT-3.5 inference error.
Question:
In a computer with an 8-bit adder, the machine numbers of signed integers X and y are represented in two's 's complement, with [x] complement F5H and [y] complement = 7EH. If the calculation X y performed in this adder,the carry-in information for the lower bit and the overflow flag OF after the operation are respectively
A: 1,1 B: 1,0 C: 0,1 D: 0,0
Answer: C
Wrong Output:
First, let's find the two's complement of y. The two's complement of number is obtained
Mathematical reasoning
by inverting all the and adding 1. [y] complement 7EH Inverting all the bits:
10000001. Adding 10000010.Now, we can perform the subtraction X y using the 8-bit adder. [X] complement = FSH [y] complement = 10000010. Adding X and the two's complement of y: 111 10101 + 10000010 1011001 11. The result is 101 101 11, which is the two's complement representation of the result of X y. To determine the carry-in information for the lower bit, we need to look the carry-out from the lower bit. this case, the carry-out is 1, so the carry-in for the lower bit is also To determine the overflow flag (OF), need to compare the carry-in and carry-out from the sign bit. In this case, the carry-in and carry-out from the sign bit are both 0, the overflow flag is O. Error Details: The complement of x and (- y) is calculated incorrectly.
38

---

Published as a conference paper at ICLR 2025
Table 22: Examples containing multiple types of errors.
Error Type
Example
Question:
Given that the system uses 32-bit physical address and 48-bit virtual address, with page size of 4KB and a page table entry size of 8B. Assuming the system employs pure paging, it will use a (--level page table, and page offiett O bits.
A: 3, 12
B: 3, 14
C: 3, 12
14
Answer:
C
Wrong Output:
To determine the number of levels in the page table, need to calculate the number of bits required to represent the virtual address space. The virtual address space bits. and the page size is 4KB, which is equivalent to 212 bytes. Therefore, the num- ber of pages in the virtual address space is 2 / 22 2 pages. Since cach page table entry is bytes,
Logical reasoning & Mathematical reasoning error
calculate the number of entries in each level of the page table. At the first level, there are 2 entries, which can be represented by 36 bits. At the second level, each entry points a page table at next level, so there are 2 entries. At the third level, each entry points to a page table at the next level, so there are 2 entries. Therefore, we need 3 levels of page tables. The page offset is the number of bits required to represent the offset within a page. Since the page size is 4KB, which is equivalent 2 bytes, the bits. Therefore, the correct answer (A)
page offset
Error Details:
The number of page table items that can he accommodated on each page is 4KB/8B 512=2", while GPT-3.5 did not mention calculations and directly stated that the first level page table has 2 items; According the GPT-3.5 calculation, if there the first level page table, there should be a 36/36=first level page table instead a third level page table, resulting in a mathematical inference error. From the process of calculating the number of page table levels in GPT-3.5, it can be seen that its logical reasoning path is not correct.
2 items
Question:
The following network connecting devices that operate the data link layer
A: Repeaters Hubs
B: Hubs and Bridges
C: Network bridges and local area network switches
D: Hubs and Local Area Network (LAN) Switches
Answer:
Complete
C
conceptual
Wrong Output:
етror
The correct answer is (D) Hubs and Local Area Network (LAN) Switches. Repeaters operate the physical layer, while hubs and bridges operate the data link layer. Local Area Network (LAN) switches also operate at the data link layer. Therefore, the correct combination of network connecting devices that operate at the data link layer is hubs and
Conceptual confusion
LAN switches.
Error Details:
GPT-3.5 is unaware that the bridge is network link device operating at the data link layer, indicating lack knowledge about bridge. Meanwhile, GPT-3.5 believes that the Hub a device at the data link layer, indicating that confuses physical layer devices with data link layer devices.
39

---

Published as a conference paper at ICLR 2025
E.5
ANALYSIS FROM MODEL REPRESENTATION
00
20-
20
20
20
20
-20
-10
-50
-60
-75
-50
-25
20
90
70
-75
-50
-25
50
-75
-30
-25
26
60
76
-70
-50
-25
60
76
(el Charbot Arena
[b) MATH
(6)
CS-Beach
Id:
MBPP
Figure 19: Representation Visualization of General and Expert Models.
To analyze the reasons behind capability correlations, we first explore the relationships between different abilities of general models at the representation level. We extract the last hidden layer of Llama3-88 Baataa3---innttrue instutttctt and obtain representations for different domains through average pooling. We then calculate the average cosine distance between CS-Bench reasoning-yy ttpee representations and other data representations, where smaller distances indicate greater similarity.
As shown in Table 23 We find that the simi- larity of representations between different abili- ties/domains generally aligns with the observed capability correlations, especially in the areas of code and math. This explains the correlation between CS skills and math and coding abilities at the representation level. Additionally, due to BBH containing various reasoning data, the dis- tance between CS-Bench and BBH is relatively small, which is consistent with the analysis in Section 3.5
Table 23:
Representation Distance between CS-
Bench and Different Datasets.
Dataset
Domain code
Distance
MBPP
0.4645
MATH
math
0.5395
BBH
reasoning
0,5494
T-Eval
tool
0.6310
Chatbot Arena (first turn)
chat 0,5555
GPQA
science 0.5775
L-eval
long context
0.6409
Next, we analyze from the perspective of training data characteristics. intuitively find that some questions in CS-Bench be understood mathematical problems in computer sci- ence context, while the main role of code is to supplement the model's knowledge of data struc- tures and logical reasoning abilities. This data-level correlation explains the expert model exper- imental results Table 3 and 4 specifically why math and code expert models help improve CS capabilities in certain areas, even when their general abilities weakened. For exam- ple: Q1: If the data part is 3800B and the maximum fragment size is 1420B, what the total length in bytes Q2: The height h of a binary tree with 1025 nodes is ().
the third fragment? ? and
Finally, we analyze the relationship between general models and math & code expert mod- els in computer science representations. We select the chat model Jlama2-7B-Chaa math ex- pert model MAmmoTH, and code expert model CodeLlama, all based on Llama2-78 -B-baee to observe their average cosine distances for the same data points in CS-Bench.
Table 24: Representation Distance of Different Models on CS-Bench.
Model pair
Distance
between Chat and Math model between Chat and Code model
0,4645
0,5395
between Math and Code model
0.5494
Table 24 show that expert models' representa-
extent compared to the Chat model, especially the Code model, which can be attributed to Code's unique data patterns. Further representation visualization in Figure T9 reveals that in the general dataset Chathot-arena, the features of the three models do not significantly separate. However, in MATH and CS-Bench datasets, the Code model forms a distinct feature cluster, while Math and Chat models' representations remain similar, as state before: many questions in CS can be viewed as mathematical problems in CS context. Lastly, in the MBPP dataset, the representations of the three models clearly differentiate and form their own clusters.
tions shift to
40

---

Published as a conference paper at ICLR 2025
E.6 EXPLORATION FOR IMPROVING CS PERFORMANCE
Although the specific methods to enhance LLM performance in CS are not the primary focus of this paper, we are eager to explore potential solutions based on our findings. These include capability transfer, inference frameworks, integrating external knowledge sources, incorporating symbolic reasoning systems, and targeted fine-tuning.
demonstrate that math and code expert models can improve CS performance certain areas. Therefore, in scenarios where CS data is scarce, training on math or code tasks can be leveraged to transfer and enhance CS capabilities.
Capability Transfer.
The capability analysis experiments in Section 3.5
Inference Frameworks. The experiments in Section 3.3 show that compared to GPT-4o, OpenAl- ol significantly improves the reasoning scores on CS-Bench. This suggests that combining LLMs with reasoning frameworks can be expected to enhance reasoning capabilities. However, this approach comes with the trade-off of increased token consumption during inference.
Integrating External Knowledge Sources. The error type analysis in Section i.4jieenssssss knowledge gaps as the main reason for LLM failures in the CS field. To address this, we implemented a RAG framework that uses Wikipedia an external knowl-
Table 25: Changes in model scores after incorporating RAG.
Model
Knowledge
Reasoning
Overall
GPT-3,5-turk
63,04
43.45
55.91
GPT-3.5-turo (+RAG)
64.92(+1.88) (88))
48.68(-5.23)
59(+3.09)
Llama3-8B
59.75
44.97
54.37
Llama3-8B(+RAG)
66.11(+6.36)
50.64(+5.67)
60.47(+6.1)
edge source. The process involves extracting key terms from the question, retrieving relevant information from Wikipedia, and summarizing it to provide supplemental knowledge, which is then integrated into the context for answering the question. The results are shown in Table 25 Both knowledge and reasoning scores of the model have improved significantly, aligning with the findings in error analysis: the primary errors in knowledge and reasoning questions are knowledge-rel related issues. Performance is expected to be further enhanced when combined with a more advanced RAG framework or more suitable knowledge sources.
Incorporating Symbolic Reasoning Systems. To enable joint reasoning between language and symbolic sys- tems, we developed a Python API as a symbolic reasoning tool. This allows the LLM generate Python code when it identifies a need for sym- bolic reasoning. The generated code is executed using the Python tool, and the results are reintegrated into the LLM's context for further output generation. The results are shown in Table 26 Since symbolic reasoning focuses on improving reasoning tasks, we observe that there is little improve- ment in the model's knowledge capabilities, but there is significant enhancement in its reasoning performance.
Table 26: Changes in model scores after incorporating Python Tool.
Model
Knowledge
Reasoning
Overall
gpt-4o Ppython Tool)
76.95
64.15
72.29
77.57(+0.5)
71.86(+7.71)
75.42(+3.13)
Fine-tuning, The error analysis also highlighted the need for specialized fine-tuning for CS. Due to the scareity of fine-tuning data in the CS domain, we randomly sampled 10% of the En Llama3-Sh-Instmeet (+SFT)
Table 27: Changes in model scores after CS fine-tuning.
Model
Knowledge
Reasoning
Overall
Llama3-$b-Instrun
66.89
43.73
58.07
71.19(+1.3)
49.88(+6.15)
63.07(+5)
glish data from CS-Bench as a mini-
test set. The remaining data was converted into SFT training data and used to fine-tune Llama3-8B- Instruct. As shown in Table 27 despite the challenges posed by the difficulty of CS tasks and the limited size of the training set, the model's knowledge and reasoning capabilities improved after fine-tuning. We expect that fine-tuning with larger amount of domain-specific data could further enhance the LLM's performance in CS.
Through these discussions, we aim to provide insights and directions for the future development of LLMs in the field of computer science.
41

---

Published as a conference paper at ICLR 2025
E.7 CASE STUDY OF OPENAI-Ol
We present the responses of GPT-4o, OpenAl-ol-man -miin, and -peeel--l-prreveew to the same questions
in Table 28 29 and 30 and analyze them in detail follows.
Example 1. In this example, the models GPT-4o, OpenAl-ol-mini, and OpenAl-ol-previie each scored 0.2, 0.3, and 0.2 respectively, but none of them provided the correct answer. GPT-4o had correct understanding of the IEEE 754 standard concept but incorrectly identified the fraction (last 23 bits) as 1000 0000 0000 0000 0000 0000 (24 bits), which led to the wrong result. OpenAl-ol-mee also had correct understanding of the IEEE 754 standard concept but made error in identifying the e fraction (last 23 bits), resulting in an incorrect outcome. Similarly, OpenAl-ol-preview made a mistake in the interpretation of the fractional part, leading a correct understanding of the IEEE 754 standard; however, they made errors in identifying the last 23 bits of the fraction, which led incorrect results.
incorrect result. All three models had
Example 2. In this example, the models GPT-4o, OpenAl-ol-mint and OpenAl-ol ---evvvve each scored 0.9, 1.0, and 0.3 respectively. GPT-4o and OpenAI-ol-ming answered the question correctly, while OpenAI-ol- --------vevvwww did not provide the correct answer. Open OpenAl-ol-par Al-o1-prevvit -preview showed a detailed calculation process, contained several critical errors that led to an incorrect result. The model used an incorrect throughput formula. The maximum throughput of the pure ALOHA protocol is 18 39.4% of the channel capacity, but the model directly used the full bandwidth of kb/s for calculations. This resulted in overestimation of the maximum number of stations.
Example 3. In this example, the models GPT-4o, OpenAI-ol- -mini, and OpenAl-ol-prevvie each scored 0.3, 0.3, and 0.7 respectively, but none of them provided the correct answer. GPT-4o and OpenAI-ol-1ae Amiii- used incorrect methods to calculate the total number of nodes. The models incorrectly calculated the number of nodes from level 0 to level 8 as 2 should have 2 - = 255 nodes because the levels are counted starting from O, which led to incorrect conclusions. OpenAl-ol-previeen
511. The 9th level
correctly calculated total number of nodes but lacked an analysis of the leaf nodes at level 8 when calculating the number of leaf nodes. Although the answer was not fully complete, arrived at the correct result.
42

---

Published as a conference paper at ICLR 2025
Table 28: An example where GPT-4o, OpenAI-ol-min -inii, and OpenAL-ol-prewiew all failed.
Type
Content
Question:
Explain which type of number (normalized number, denormalized number, inlinity. 0) the Following IEEE 754 single precision value represents. represents specific value, please provide
Question
that value: 0b1111 IIII 1000 0000 0000 0000 0000 0000.
Reference Answer:
Since the exponent field of the number is all ones and the mantissa field is zeco, with the sign bit being one, it represents negative infinity.
Output:
The given JEEE 754 single-precision value is:
0b1111
0000 0000
To determine what this represents, let's break it down:
The first is sign bit. next are exponent. The remaining are fraction
mantissa).
the given value:
The sign (finst
indicating negative number.
The exponent (next bits) is 1111 1111, which is 255 in decimal. 0000
GPT-4o
hits) According to the TEFE 754 standard: An exponent of 255 (all ones) with any non non-zero fraction indicates NaN (Not a Number). An exponent of 255 zero fraction woold indicate infinity. determined by the sign In this the fraction nob-zero starts with 1000). Therefore, this specific combination represents NaN (Not a Number).
fraction
the IEEE 754 single-precision value 0b1111 1111 1000 OC00 on00 0000 0000 0000 represents NaN Number).
Score:
0.2
Output:
The IERF. 754 single-precision value 0bl IIIIII 10000000 00000000 00000000 represents NaN (Not
Number).
Explanation:
Sign Bit (negative)
OpenAI-ol- mini
Exponent:
11111111 (255), which is the maximum value.
Fraction Mantissa):
all zeros)
In JEEE 754: When the
255 and the fraction is not all zeros, the value is classified as NaN. If the exponent were 255 and the fraction were all zeros, would represent infinity.
Since the fraction
zeros, value NaN, indicating undefined unrepresentable value.
Score:
0.3
Output:
The IEEE 754 single precision binury value
Breaking down into IEEE 754 format:
Sign bit (S): The first bit 'I' indicating the sign (though NaNs, generally ignored).
Expooeent bits (E): The next 8 bits are '11111111.
Fraction/Mantisa (FF): remaining
Analysis:
Expoment Analysis: Exponent bits are '11111111' (E = 255), which is the maximum possible value for the
expoment [EEE 754, When the expoment all (E 255), indicates either infinity NaN.
2. Fraction Analysis: Fraction bits are not zeros (F of 0). Specifically, the most significant bit of the fraction the
OpenAI-ol- preview
255 and F o the value represents 3. Type of NaN: Quiet NaN vs. Signaling NaN: 1f the most signilicant bit of the fraction (also known as the quiet fraction, Signaling
When
Number (NaN).
any non NaN (sNaN). In this case, the most significant bit of the fraction is 'I', so it's Quiet NaN
Quiet NaN (gNaN).
bit
Conclusion:
cingle-precision The IEFE 754 single precision value represents Quiet NaN (Not Number). This special value is used represent undefined or morepresentable numerical results, such as the result of 0/00 or the square root of negative number.
Answer:
It represents quiet NaN (Not Number) in [EEE 754 single- preccsseeisiio format.
Score:
0.2
43

---

Published as a conference paper at ICLR 2025
Table 29; An example where GPT-4o and OpenAl-ol-mine succeeded, OpenAL-ol-prrviiew failed.
Type
Content
Question:
group of N stations share Sökh/s pure ALOHA chamnel, with each station outputting 1000 hit frame on average every 100s,
the previous frame has finished transmitting. What the maximum value of N?
Question
Reference Answer:
For the pure ALOHA protocol. the channel utilization is 0,184, the availablc bandwidth 0.184x56Kb's. The bundwidth required by each station is 1001/100- 10b/s. Therefore, the maximum value of is 1030.
Output:
To determine the maximum number of stations N that can share a 56 kb/s pure ALOHA channel efficiently, we need consider the throughput of the pure ALOHA protocol. The maximum throughput for pure ALOHA is 18.4%, 2
First, let's calculate the frame generation rate for one station:
station generates
every seconds. Therefore,
rale per station
1000 10 hps.
1007
the total maximon channel capacity:
GPT-4o
The channel cupacity is 56,000 bps. With the maximum throughput efficiency of 18.45. etfective throughput
56, 000 0.184 10, 304 bps.
Now, find the maximum number of stations N:
- Each station contribures 10 bps to the load. - The maximum number of stations can be calculated by dividing the effective throughput by the load per station: N 10. 20.
1, 030.4.
10 sains
Since N must be an integer, the maximum value of N 1,030. value
Score:
0.9
Output:
To determine the maximum numher of stations N that share 56 kivss Al JOHA channel, follow these steps:
1. Calculate Frame Transmission Time (T): T b0,000 bass R 0.01786 scconds (17 77.86 ms)
1000
Determine the Offered Load (G): Each station sends a frame every 100 secomds, so the load per station per
TO0 500g Total
load
stations:
x G;
OpenAI-ol- mini
Apply Pure ALOHA Throughpur: maximum throughpur for pure ALOHA SaL
0.184. Throughput: s000 2e Solving N 5608 S000
Conclusion: The maximum number of stations N that can effectively share the 56 khis pure Al ALOHA channel is approximately 1030.
Score:
1.0
Output:
To determine the maximum number of stations N that can share the 56 khis pure ALOHA channel without exceading the maximum acceptable load, follow these steps:
1. Calculate the Frame Transmission Time (Time)
Thmme Chasnel
Fiunc
1000 bus
sec
Determine
per Station (Gani))
Fach station generates frames at average rate of: Aestin
100 we
The offered load station in frames per frame time is: Gasiin Asaion
Timms
Tixy
5,000
3. Calculate the Total Offered Load (G)
N stations, total offered load is: N Ganio
5,700
OpenAI-ol- preview
Apply the Pure ALOHA Throughput Formula In ALOHA, the maximum throughput when the offered load G 0.5 frames frame time: Guax 0.5
Setting the total offered load equal the maximum: N
= 0.5
Solve
N = 0.5 x 600 8,0
Conclusion:
The maximum number of stations NN that can share the channel without exceeding the optimal load for pure ALOHA is 2,800 stations.
Answer: 2800
Score:
0.3
44

---

Published as a conference paper at ICLR 2025
Table 30: An example where GPT-4o and OpenAl-ol-mini failed, OpenAAlol--rreveew succeeded.
Type
Content
Question:
Given that the 9th level of complete binary tree has 240 nodes, how many nodes does the entire complete binary
How many leaf nodes there?
Reference Answer:
In complete hinary tree, the 9th level is full, then the number of nodes 2d 256. However, currenty. there are only 240 nodes on the 9th level, indicating that the 9th level not full and the last level. Lovels the
Question
dull, total number nodes =2 9th level ure leaf nodes. Moreover, the parents of the 240 nodes on the 9th level are on the 8th level, with the number parents being
last
nodes on the
on level, Therefore, the number of leaf nodes on the Sth level is 2 120 & Consequently, the total number of leaf nodes 240
means
248.
Output:
a complete hinary tree, determine the total number of nodes in the tree and the number of leaf nodes.
2
level nodes. this,
to
First, let's confirm that the given information fits structure complete binary tree. The 9th level should have 2 = 512 nodes. Since 240 nodes do this pallern, implies the free is incomplete that level, hence not truly complete. Süll, if this were theoretical exercise, we might infer the tree is complete up until level 8 and partially filled at
I. Calculating the Total Number of Nodes: - Levels through are fully filled, with 2 + 2 + ... + 2 nodes. 511. Adding 240 nodes from the level: 511 240 751.
GPT-4o
sum nodes from So, the total number of nodes in the tree is 751.
to
Calculating Number Leaf Nodes: In a complete binary tree, the leaf modes are typically the nodes at the last level partially filled level. Here, the 9th level only has 240 nodes. These are leaf modes.
So, there are 240 leaf nodes.
complete binary
nodes leaf nodes.
Score:
0.3
Output:
In a complete binary tree where the 9th level has 240 nodes: Total Nodes; The lirst
containing 2 nodes the 9th level gives total of 511 210 751 nodes. Leaf Nodes: All 240 nodes at 9th level are leaf nodes.
are fully
nodes. Aiddnn
OpenAI-ol- mini
Summary: Total Nodes: 751 Leaf Nodes: 240
Score:
0.3
Output:
To determine the total number of nodes in the complete binary tree and the number of leal nodes, we can follow
these steps:
1. Calculate the total nodes from levels to 8:
complete binury tree, levels are fully filled. number the ront.
level doubles starting
Total nodes from level to level Ed
256 255 nodes
2. Add the nodes level 9:
OpenAI-ol- preview
1t's given that there are 240 nodes at level
Total nodes the 255 (nodes from levels 1-8) 240 (nodes level 495 nodes
3. Determine the number of leaf nodes:
complete hinury tree, nodes at
Therefore, number leaf equal
the number of nodes at level 9.
Number of leaf nodes: 240
Answer:
The complete hinary tree hus 495 nodes in total, and there are 240 leaf nodes.
Score:
0.7
45


Reviews:
[{"review_id": "rSrHapAh75", "reviewer": "Reviewer_xmLP", "summary": "Develops CS Bench, a benchmark dataset of CS expertise that tests the ability of LLMs in computer science, as well as the relationship between CS math and code programming ability.  The benchmark measures 4 languages and 26 sufields with 9676 samples. To test this they evaluated using 30 LLMs.\n\nThe datasets were constructed using professional exams and practice tests widely available on the internet; extraction from blog articles of questions and answers; teaching materials and exams from the authors institutions.  For scoring, the authors use autoraters (GPT4) for genration tasks and  for comprehension tasks use questions where they can score with a regex", "questions": "What does your dataset not include for testing compettency in CS?\nWhat level does this test? The level is notably absent from the paper -- but staging this relative to expertise (high school, undergrad, grad, etc) is extremely useful. \nWhat is the breakdown of problems in the benchmark to expertise?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The strength of the paper is the dataset . This type of dataset is extremely valuable and I especially appreciate the sourcing of exam questions from universities and the knowledge transfer from blogs across specialized fields with such a large corpus of questions.  The final version of the paper should explain exactly where the questions came from, including giving the class and level at the university so that the difficult level is clear. \n\nI'd also call out the thoroughness of the analysis. The different correlations noted, eg math and CS ability and so forth, are interesting as is the relationship between performance and tokens across scales. None of these are unexpected but they are nice things to see.", "weaknesses": "The first sentence \"Computer Science (CS) stands as a testament to the intricacies of human intelligence, profoundly advancing the development of artificial intelligence and modern society\"  is a bit much and some of the corresponding language needs to be toned down. Could we please not philosphize about the importance of specific fields (especially those were we are practitioners) and instead focus on the task that the paper accomplishes -- building a novel eval set and using it.\n\nOther than this, the generative questions that require autorating to score are less convincing than the regex. The authors are right to emphasize that people should focus on this dataset if they can, but it does make me wonder whether the generative questions could be downgraded even further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "taE8Z9tUHa", "reviewer": "Reviewer_7dpp", "summary": "The paper introduces a benchmark of 10k test samples that pertain to Computer Science specific knowledge.", "questions": "Q1. What would the key contributions of the benchmark be to the field at large, as it stands today?\n\nQ2. Can the authors evaluate the value of the data through one of several Value-of-Data approaches including perplexity of training on the benchmark, FT-ing on the benchmark or other approaches?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. Commendable that 70% of the benchmark is from private source not already on the intrernet.\n\n2. The paper has a somewhat thorough review of several LLMs on where they stand in the benchmark.", "weaknesses": "1. It is not clear what the benchmark aims to achieve or improve. LLMs are already good at MCQ, open-ended, FITB and T/F questions. There is no coding in the benchmark. The authors need to discuss more clearly why this is pivotal and important to the field.\n\n2. Scoring on the benchmark appears to be ad-hoc with regular expression matching, instead of measuring with existing techniques.\n\n3. The benchmark lacks a fingerprint to prevent training. THis would be required to protect the integrity of the benchmark.\n\n4. The paper indicates that this may help evaluate or improve reasoning though it is unclear how?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hc9VOHFQsI", "reviewer": "Reviewer_g1yj", "summary": "This paper introduces CS-Bench, a comprehensive, multilingual benchmark designed to evaluate the computer science knowledge and reasoning abilities of Large Language Models (LLMs). Comprising approximately 10,000 meticulously curated samples across 26 subfields within four key CS areas (Data Structures and Algorithms, Computer Organization, Computer Networks, and Operating Systems), CS-Bench supports evaluation in English, Chinese, French, and German, and includes diverse task formats such as multiple-choice, assertion, fill-in-the-blank, and open-ended questions. The authors conduct an extensive evaluation of over 30 mainstream LLMs on CS-Bench, analyzing their performance relative to model scale and investigating the reasons for model failures. They further explore the relationship between LLM capabilities in computer science, mathematics, and coding, observing strong correlations and evaluating the performance of specialized expert LLMs on the benchmark. The findings suggest a logarithmic growth relationship between model scale and CS performance, highlight the challenge of CS-specific reasoning for LLMs, and reveal the potential for expert LLMs to improve performance in certain CS subfields. CS-Bench aims to serve as a valuable resource for assessing and improving the diverse reasoning capabilities of LLMs in the context of computer science.", "questions": "Reducing Reliance on GPT-4 for Evaluation: The dependence on GPT-4 for scoring generative tasks raises validity and reproducibility concerns. While you explored alternative scoring models and human evaluation, these were limited. Could you elaborate on the feasibility of using alternative automatic metrics like BLEU, ROUGE, or METEOR, perhaps in conjunction with domain-specific constraints or regular expressions? Could a smaller, more accessible model be fine-tuned specifically for evaluating CS-related generative tasks? This would improve reproducibility and lower the barrier for other researchers to use CS-Bench. Also, provide more details on the human evaluation: the number of annotators, their expertise level, and the specific guidelines they followed. Calculating inter-annotator agreement (e.g., using Cohen's kappa) would strengthen the validation of both human and GPT-4 scoring.\n\nDeepening the Analysis of Cross-Capability Relationships: The observed correlations between CS, math, and coding performance are interesting, but the analysis lacks depth. Could you investigate why these correlations exist? Are certain model architectures better suited for integrating these skills? Does the training data composition play a significant role? Analyzing the internal representations of CS concepts in general vs. expert LLMs could provide valuable insights. Consider using techniques like probing tasks or representational similarity analysis to understand how different models learn and represent CS knowledge. This would move beyond simple correlations to provide a more mechanistic understanding of cross-capability transfer.\n\nGenerating Actionable Insights for Improvement: The error analysis identifies knowledge gaps and reasoning challenges, but it doesn't offer concrete solutions. How can we leverage these insights to improve LLM performance on CS tasks? Could you explore targeted fine-tuning on CS-specific datasets, curriculum learning strategies, or methods for incorporating external knowledge sources (e.g., knowledge graphs, symbolic reasoning systems)? Providing concrete examples of how these techniques could be applied would significantly enhance the paper's practical value.\n\nAddressing Benchmark Biases: What are the potential biases introduced by the composition of CS-Bench (e.g., university-level focus, source selection)? How might these biases affect the evaluation of different LLMs? Consider expanding the benchmark to include tasks from different levels of expertise (e.g., introductory CS, advanced topics, real-world applications) and diversifying the sources to mitigate potential biases. Discuss the limitations of CS-Bench in representing the full spectrum of CS knowledge and skills. A critical self-reflection on the benchmark's limitations would strengthen the paper.\n\nImproving Visualization and Clarity: Some figures, particularly the radar charts, are difficult to interpret. Consider using alternative visualizations (e.g., bar charts, scatter plots) that more clearly communicate the key findings. Streamline the writing to improve conciseness and clarity. Focus on conveying the key messages effectively and avoid unnecessary jargon. Ensure that the figures are clearly labeled and explained in the captions. A more polished presentation would enhance the paper's impact and accessibility.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Focus on Reasoning: The inclusion of both knowledge-based and reasoning-based questions is a good step towards a more comprehensive evaluation of LLMs' understanding of CS concepts.\n\nExtensive Evaluation: Testing numerous LLMs, both open-source and closed-source, across various scales provides a good overview of current capabilities and highlights areas where models struggle. The scale-score analysis is also a useful contribution.\n\nComprehensive Benchmark: The breadth of CS topics covered, along with the multilingual aspect and diverse task formats (multiple-choice, assertion, fill-in-the-blank, open-ended), makes this a valuable contribution. Creating such a benchmark is a substantial undertaking.\n\nError Analysis: The analysis of error types provides some insights into the reasons for model failures and points towards potential directions for improvement.", "weaknesses": "Writing and Presentation Could Be Improved: The writing is occasionally dense and could be more concise and clear. Some figures, especially the radar charts, are difficult to interpret and don't effectively communicate the key findings. A more visual and intuitive presentation of the results would enhance readability and impact.\n\nLimited Insight into Cross-Capability Relationships: While the paper shows correlations between CS, math, and coding abilities, it doesn't delve deeply into why these correlations exist. More in-depth analysis is needed to understand the underlying factors driving these relationships. Do specific architectural choices or training data characteristics explain the observed correlations? The analysis of expert LLMs is a step in the right direction, but it lacks depth. How do the internal representations of CS concepts differ between general and expert LLMs?\n\nLack of Actionable Insights for Improvement: The error analysis, while pointing towards knowledge gaps and reasoning challenges, doesn't offer concrete suggestions for how to address these limitations. Beyond general recommendations like \"knowledge supplementation\" and \"CS-specific reasoning,\" what specific techniques or strategies could be employed to improve LLM performance on CS tasks? Exploring methods like curriculum learning, incorporating external knowledge sources, or fine-tuning on specific CS datasets could provide more actionable insights.\n\nLimited Discussion of Benchmark Biases: All benchmarks have biases. What are the potential biases in CS-Bench, and how might these biases influence the results? For example, the reliance on university-level content might not fully reflect the diversity of CS tasks in real-world applications. Addressing this limitation by including tasks from different levels of expertise or from specific industry applications would strengthen the benchmark's generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D4sis3zj9A", "reviewer": "Reviewer_Rfxy", "summary": "This paper introduces CS-Bench, the most comprehensive and detailed benchmark for testing computer science knowledge and reasoning that exists to date. It contains over 10,000 items over 26 subfields of computer science, uses 4 different response types (multi choice, assertion, fill-in-the-blank, and open-ended responding), and is presented in 4 languages (English, Chinese, German, French). The authors study over 30 LLMs, including both open- and closed-source models. They provide extensive analysis of performance across the benchmark and its subdomains, and conduct post-hoc studies of how performance scales with model size and the kinds of errors that LLMs make on CS-Bench.", "questions": "1. What do the authors suggest to avoid the issue of contamination? \n2. Have the authors considered using answer ordering permutation, number switching, orquestion negations to reduce the risk of contamination? By number switching, I mean dynamically changing the numbers in questions so that the answers are different but the fundamental relationships continue to hold. By question negations, I mean asking the opposite question by including a negative particle, such that the question has a different answer.\n3. What is the utility of CS-Bench when compared to approaches in which LLMs are situated in an environment with access to a terminal and tasked to solve problems?\n4. Can the authors provide further evidence that the interlingual translations using GPT-4 are sound and accurate?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "In the already saturated LLM benchmark market, this paper has a number of stand-out strengths. First, it is highly comprehensive. Unlike other benchmarks which seek generality at the cost of precision and depth, CS-Bench focuses on a single domain and contains thousands of in-depth testing items from it. This is an improvement on existing computer science benchmarks like MMLU. Second, it is multilingual. This facilitates, among other things, the generalisability of deeper CS knowledge and reasoning skills when changing the surface level features of the language the system is operating in. This will also allow users to measure the degree of contamination, since (ex hypothesi) a considerable amount of LLM training data will be from English-language sources. Third, the authors include multiple response types, ranging from multiple choice to open ended responding. This facilitates researchers using CS-Bench to probe CS knowledge and reasoning skills with more nuance. Fourth, the results on logarithmic scaling laws is intriguing. Finally, the extensive error analysis is commendable and often not seen in benchmark papers such as this. It demonstrates again that CS Bench can be used to produce a much richer picture of the knowledge and reasoning skills of LLMs in this domain than can be gleaned by simply aggregating task performance.", "weaknesses": "There are a few weaknesses with this paper. First, CS-Bench is static and openly available online. This means that it is possible that LLM developers can train directly on the benchmark, leading to contamination. While we as a community hope that developers wouldn't be so mal-intentioned, it is an increasingly common phenomenon. Dynamic benchmarks (e.g., DynaBench) in which testing items are produced on the fly are able to sidestep such problems.\n\nSecond, true tests of computer science skills, particularly for software engineering and development, come from situating models in a coding environment, with access to a terminal. The authors might consider comparing their approach to this more embodied approach to evaluation.\n\nFinally, I am sceptical of the use of GPT-4 for translating the benchmark from Chinese to other languages. While this is certainly scalable, I fear that the translations will be poor, particularly for the most complex questions. The authors' note that some manual checking was done, but I would like to see a more robust measure of translation quality, perhaps through asking a sample of independent native speakers of English, French, and German to check a random sample of questions and answers for soundness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)