CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

## Abstract

Abstract Computer Science (CS) stands as a testament to the intricacies of human intelligence, profoundly advancing the development of artificial intelligence and modern society. However, the current community of large language models (LLMs) overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first bilingual (Chinese-English) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 5K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs’ capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs’ diverse reasoning capabilities. The CS-Bench data and evaluation code are available at https://github.com/csbench/csbench .

## 1 Introduction

Serving as the cornerstone of the modern information revolution, the significance of computer science (CS) extends from the early days of electronic computers to today’s advancements in artificial intelligence (AI) [, ]. As a new milestone in AI, large language models (LLMs) [, ] represented by ChatGPT [] and GPT-4 [] are not limited to the natural language processing (NLP) community, showing vast potential in fields including education, industry, and science [, , , , , , ].
However, enabling LLMs to effectively utilize computer science knowledge and serve humanity more efficiently is one of the key challenges on the path to the future intelligent era [, , ].

Understanding the performance of LLMs in computer science is fundamental to the research and application of LLMs within the field.
Despite studies like MMLU and C-Eval [, , , , ] covering a wide range of fields including CS, their broad scope implies that CS is merely a component within the multiple categories of science and engineering, overlooking the importance of thoroughly evaluating the CS field. Moreover, such evaluation result can further guide the development of LLMs, offering practical insights to advance the corresponding capabilities. Recently, a series of studies have devoted on actively assessing and analyzing the capabilities of LLMs in mathematics, coding, and logical reasoning [, , , , , , , , , , , ]. Unfortunately, efforts on LLMs in cross-capability evaluation is quite scarce. Considering the intersection of computer science with coding, mathematics, and reasoning abilities, we have grounds to believe that cross-capability research and analysis in CS can effectively propel the comprehensive development of the LLM community. Here, we are particularly interested in two research questions for evaluating LLMs’ proficiency in computer science field:

*RQ1: How do LLMs perform in the field of computer science and what are the challenges and potential directions for improvement?*

*RQ2: What are the relationship between the abilities of LLMs in computer science, mathematics, and code programming?*

As the bedrock for exploration, we first propose CS-Bench, the first benchmark dedicated to evaluating the performance of LLMs in the field of computer science. CS-Bench features high-quality, diverse task forms, varying capacities, and bilingual evaluation. Firstly, CS-Bench comprises approximately 5,000 carefully curated test items spanning 26 sections across 4 key CS domains. Diverging from conventional benchmarks consisting solely of multiple-choice (MC) questions [, , ], CS-Bench includes 4 tasks: multiple-choice, assertion, fill-in-the-blank (FITB), and open-ended, to better simulate real-world scenarios and assess the robustness of LLMs to different task formats.
In addition to knowledge-type questions assessing LLMs’ mastery of CS knowledge, reasoning-type questions further evaluate LLMs’ ability to apply CS knowledge for reasoning.
Lastly, by supporting bilingual evaluation in Chinese and English, CS-Bench enables the appraisal of LLMs’ adeptness in addressing CS challenges across different language contexts.

Figure: Figure 1: Overview diagram and statistics of CS-Bench.
Refer to caption: x1.png

In response to RQ1, we evaluate over 30 mainstream LLMs on CS-Bench.
Our main findings are:
(1) CS-Bench effectively differentiates the capabilities of LLMs in the CS field while also posing significant challenges to the best-performing GPT-4/ GPT-4o.
(2) LLMs exhibit a consistent logarithmic growth pattern in scale and a linear growth pattern in scores on the CS-Bench. By establishing the scale-score fitting function, smaller models can be used to predict and guide the development of larger-scale models.
(3) Further error type analysis indicates that the primary reason for the limited performance of LLMs is the lack of domain knowledge, and the CS-specific reasoning is difficult to achieve merely by enhancing general reasoning abilities, necessitating targeted reinforcement.

In response to RQ2, we perform a detailed analysis of the relationship of General LLMs’ ability in three domains: mathematics, coding, and computer science, as well as the performance of code- and math-specific expert LLMs on CS-Bench. We observe consistent trends in the overall performance of the general LLMs across CS-Bench and scores in benchmarks related to mathematics and coding, indicating a strong correlation between LLM’s computer science proficiency and its mathematical and programming abilities. Furthermore, despite a decline in general capabilities, some expert LLMs still exhibit improvements in certain areas of CS, such as data structures and algorithms, with more pronounced knowledge and reasoning capabilities evident in supplementary smaller-scale models.

To summarize, our contributions are as follows:

- •
We introduce CS-Bench, the first benchmark dedicated to evaluate the performance of LLMs in the field of computer science. CS-Bench supports both Chinese and English, covers four key areas with 26 subfields, and includes a diverse range of task formats.
- •
Utilizing CS Bench, we conduct a comprehensive evaluation of mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvement.
- •
We conduct exploratory experiments on LLMs’ cross-ability and find a strong relationship between their CS proficiency and mathematical and programming abilities. Moreover, the expertise in mathematics and programming of expert LLMs can improve performance in specific CS subfields.

## 2 CS-Bench

### 2.1 Design Principle

The objective of CS-Bench is to robustly assess the knowledge and reasoning capabilities of LLMs in different linguistic contexts within the field of computer science.
To this end, our benchmark adheres to the following guidelines:
(1) Coverage of key domains: it covers key areas of CS with finer subfields for specificity.
(2) Diverse task forms: questions vary in format to simulate diverse real-world user queries.
(3) CS-specific reasoning: it evaluates CS logical and arithmetic reasoning in addition to CS knowledge.
(4) Multilinguality support: it supports assesses LLMs’ performance in different language environments.
Based on these criteria, CS-Bench focuses on bilingual evaluation in Chinese and English, covering four domains: Data Structure and Algorithm (DSA), Computer Organization (CO), Computer Network (CN), and Operating System (OS). Twenty-six fine-grained subfields, diverse task forms, and divisions of knowledge and reasoning are further developed to enrich the dimensions of assessment and simulate real-world scenarios.

### 2.2 Data Collection

**Table 1: Comparison of perplexity (PPL) across evaluation datasets. The PPL of English and Chinese datasets is calculated on Llama2-7B-base and Qwen1.5-7B-base, respectively. “MC” denotes multiple-choice, and “ALL” denotes all tasks.**
| English Dataset | PPL | Chinese Dataset | PPL |
| --- | --- | --- | --- |
| TruthfulQA (MC) [] | 7.73 | C-Eval [] | 11.47 |
| MMLU [] | 9.54 | CMMLU [] | 13.62 |
| CS-Bench (MC) | 11.86 | CS-Bench (MC) | 13.31 |
| CS-Bench (ALL) | 13.3 | CS-Bench (ALL) | 16.95 |

#### Data Sources.

Diverse data sources are key to achieving the sample diversity of CS-Bench. Our raw data originates from three sources:
(1) Computer science-related questions obtained from publicly available online channels, such as professional exams and practice tests(^1^11e.g., [https://github.com/CodePanda66/CSPostgraduate-408](https://github.com/CodePanda66/CSPostgraduate-408)).
(2)Knowledge-type questions obtained through the initial manual extraction and subsequent adaptation of blog articles from various computer-related websites(^2^22e.g., [https://www.wikipedia.org/](https://www.wikipedia.org/),[https://www.cnblogs.com/](https://www.cnblogs.com/), [https://www.csdn.net/](https://www.csdn.net/)).
(3) Construction of teaching materials and examination papers authorized by the authors’ institutions.
The latter two categories constitute the vast majority (over 70%) of the data, and these data are not directly exposed on the internet, effectively reducing the likelihood of LLMs encountering these questions during pre-training.
We compare the perplexity [] of models on CS-Bench and several prominent evaluation datasets in Table 1. In both English and Chinese, the perplexity of CS-Bench is comparable to or even higher than that of other datasets, further indicating the high quality of CS-Bench samples and the rarity of data leakage instances.

#### Data Processing.

The data processing relies on a team composed of five members, each holding a bachelor’s degree in computer science and receiving appropriate compensation. Initially, we parse questions and answers for each sample from the data sources either automatically or manually.
Subsequently, we manually label questions with knowledge-type or reasoning-type tags depending on whether in-depth reasoning and calculation are required.
For reasoning-type questions, we attempt to collect explanations from the data sources whenever possible; otherwise, we handle them through cross-annotation and verification among team members. We first construct Chinese data, then translate it into English using GPT-4, supplemented by manual checks, to create English data. Finally, we conduct thorough manual checks on the entire dataset to ensure quality.
As this benchmark pertains to objective knowledge and reasoning in the field of computer science, the annotation content is not influenced by regional or cultural differences among annotators.

#### Statistics.

CS-Bench is an evaluation benchmark supporting bilingual assessment, encompassing a total of 26 subfields across 4 domains, with a cumulative total of 4838 samples. These samples encompass various task formats including multiple-choice, assertion, fill-in-the-blank, and open-ended questions. Besides, CS-Bench assesses both knowledge-type and higher-order reasoning-type questions, with each reasoning question accompanied by an explanation. To validate the effectiveness of models, we randomly sample 10% of the data for validation, using the remaining 90% for testing. The statistics of CS-Bench are shown in Figure 1, with detailed exposition provided in Appendix C.

## 3 Experiment

### 3.1 Experimental Setup

#### Evaluation Protocols.

Due to the diverse task formats in CS-Bench, we first design question templates for each task type. For comprehension tasks (MC and Assertion), we use regex to match LLM’s predictions and then calculate their accuracy against the ground-truth answers. For generation tasks (FITB and Open-ended), due to the diversity of ground-truth answers, we score LLM’s predictions by GPT-4 using standard answers in CS-Bench as references. In detail, FITB questions are scored as either 0 or 1, while the score range for Open-ended questions is 1-10, which is then linearly mapped to a range of 0.1 to 1. Finally, scores are weighted based on the quantity of each type to derive the ultimate overall score. It is worth emphasizing that while employing GPT-4 for scoring generation tasks may introduce a certain threshold for evaluation, its primary purpose is to simulate diverse task formats in real-world scenarios. Therefore, we encourage isolating comprehension tasks from CS-Bench to facilitate automatic evaluation with no need for GPT-4.
We provide the details of the evaluation setup in Appendix D, where we also verify the validity of GPT-4 scoring through its consistency with manually scored results.

#### Models.

We evaluate nearly 30 models in different sizes from 12 model families. For open-source models, we selected Gemma-2B/7B [], Llama2-7B/13B/70B [], Llama3-8B/70B [], ChatGLM3-6B [], Baichuan2 (v2.0)-7B/13B [], InternLM2-7B/20B [] , Qwen1.5-4B/7B/14B/72B/110B [], Mistral-7B (v0.2) [], Mixtral-8$\times$7B (v0.1) [], and DeepSeekLLM-7B/67B []. For closed-source commercial models, we utilized PaLM-2 (palm-2-chat-bison) [], Claude-2.1 [], Claude-3 (opus) [], as well as GPT-3.5, GPT-4 (0125 version) [] and GPT-4o []. To ensure the instruction-following abilities, we employ the official chat or instruction-tuned versions for all models. Details on these models are provided in Appendix D.4.

### 3.2 Main Results

**Table 2: Zero-shot scores (%) of LLMs across domains on CS-Bench (EN), where “Klg” denotes knowledge-type, “Rng” denotes reasoning-type, and “Avg” denotes Average. The random scores are weighted as follows: 25% for MC, 50% for Assertion, 0% for FITB, and 10% for Open-ended. The highest scores for open-source and closed-source LLMs is marked in green and blue respectively.**
| Model | Data Struc & Algo | Computer Organization | Computer Network | Operating System | Overall |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg |  |
| Random | 28.04 | 24.63 | 26.65 | 26.57 | 25.24 | 26.13 | 26.34 | 22.49 | 24.98 | 29.06 | 24.23 | 27.27 | 27.4 | 24.12 | 26.2 |
| Open-source LLM (Scale < 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Gemma-2B | 56.76 | 23.44 | 43.20 | 47.69 | 30.18 | 41.92 | 45.22 | 26.38 | 38.59 | 37.79 | 31.32 | 35.39 | 46.89 | 27.59 | 39.86 |
| Qwen1.5-4B | 58.76 | 36.56 | 49.72 | 52.31 | 33.88 | 46.23 | 52.70 | 33.97 | 46.11 | 40.03 | 38.52 | 39.47 | 51.18 | 35.70 | 45.54 |
| ChatGLM3-6B | 51.10 | 34.08 | 44.17 | 48.11 | 32.73 | 43.04 | 51.15 | 32.66 | 44.64 | 43.57 | 37.03 | 41.14 | 48.63 | 34.07 | 43.33 |
| Llama2-7B | 51.51 | 32.61 | 43.82 | 48.89 | 31.82 | 43.26 | 46.72 | 30.75 | 41.10 | 41.04 | 26.26 | 35.55 | 47.15 | 30.48 | 41.08 |
| DeepseekLLM-7B | 56.42 | 28.94 | 45.23 | 52.09 | 32.48 | 45.62 | 52.43 | 31.41 | 45.03 | 41.66 | 31.98 | 38.06 | 50.87 | 31.11 | 43.67 |
| Baichuan2-7B | 53.11 | 34.95 | 45.72 | 45.10 | 38.67 | 42.98 | 51.26 | 34.27 | 45.28 | 43.47 | 33.63 | 39.82 | 48.29 | 35.33 | 43.57 |
| Gemma-7B | 59.53 | 35.18 | 49.62 | 49.97 | 33.27 | 44.46 | 60.87 | 37.09 | 52.50 | 48.67 | 34.23 | 43.31 | 54.90 | 35.02 | 47.66 |
| Qwen1.5-7B | 59.90 | 35.28 | 49.88 | 55.21 | 42.73 | 51.09 | 61.56 | 43.02 | 55.04 | 52.01 | 39.78 | 47.47 | 57.34 | 40.08 | 51.05 |
| InternLm2-7B | 59.57 | 40.92 | 51.98 | 58.83 | 37.94 | 51.94 | 62.65 | 40.60 | 54.89 | 50.94 | 39.29 | 46.61 | 58.31 | 39.77 | 51.56 |
| Mistral-7B | 63.24 | 34.86 | 51.69 | 57.52 | 38.67 | 51.30 | 61.48 | 44.92 | 55.65 | 51.66 | 43.79 | 48.73 | 58.63 | 40.44 | 52.01 |
| Llama3-8B | 66.25 | 37.29 | 54.46 | 55.38 | 40.67 | 50.53 | 62.21 | 53.02 | 58.98 | 55.26 | 49.34 | 53.06 | 59.75 | 44.97 | 54.37 |
| Open-source LLM (Scale > 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Llama2-13B | 51.74 | 35.00 | 44.93 | 51.81 | 36.18 | 46.66 | 53.03 | 37.99 | 47.74 | 48.12 | 32.36 | 42.27 | 51.31 | 35.46 | 45.54 |
| Baichuan-13B | 54.82 | 33.39 | 46.10 | 50.50 | 39.52 | 46.88 | 55.87 | 42.21 | 51.06 | 48.44 | 34.73 | 43.35 | 52.53 | 37.44 | 47.03 |
| Qwen1.5-14B | 64.95 | 46.74 | 57.54 | 60.06 | 45.58 | 55.28 | 68.66 | 52.91 | 63.12 | 56.56 | 51.48 | 54.67 | 62.79 | 49.18 | 57.83 |
| InternLm2-20B | 66.72 | 38.21 | 55.11 | 58.38 | 39.82 | 52.26 | 64.13 | 50.35 | 59.28 | 53.51 | 46.43 | 50.88 | 60.81 | 43.66 | 54.56 |
| Qwen1.5-32B | 69.70 | 51.19 | 62.17 | 67.63 | 52.91 | 62.78 | 69.23 | 58.74 | 65.54 | 60.06 | 56.21 | 58.63 | 66.87 | 54.72 | 62.45 |
| Mistral-8$\times$7B | 70.94 | 40.50 | 58.55 | 66.88 | 42.06 | 58.70 | 67.49 | 52.86 | 62.34 | 57.56 | 51.65 | 55.37 | 65.91 | 46.66 | 58.90 |
| DeepseekLLM-67B | 69.70 | 44.17 | 59.31 | 63.59 | 39.15 | 55.53 | 69.04 | 50.25 | 62.43 | 57.86 | 50.11 | 54.98 | 65.23 | 45.96 | 58.22 |
| Llama2-70B | 64.28 | 41.51 | 55.01 | 56.35 | 40.85 | 51.24 | 61.99 | 43.07 | 55.33 | 51.79 | 41.15 | 47.84 | 58.73 | 41.68 | 52.52 |
| Llama3-70B | 75.72 | 53.03 | 66.48 | 71.45 | 51.09 | 64.74 | 74.78 | 63.02 | 70.64 | 63.77 | 58.08 | 61.65 | 71.65 | 56.36 | 66.08 |
| Qwen1.5-72B | 72.71 | 50.69 | 63.75 | 69.28 | 54.12 | 64.28 | 71.97 | 66.73 | 70.13 | 63.96 | 59.62 | 62.35 | 69.63 | 57.75 | 65.31 |
| Qwen1.5-110B | 73.11 | 53.58 | 65.16 | 73.65 | 54.18 | 67.23 | 75.36 | 70.75 | 73.74 | 64.55 | 65.27 | 64.82 | 71.98 | 60.91 | 67.95 |
| Closed-source LLM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PaLM-2 | 70.07 | 38.98 | 57.41 | 63.81 | 41.91 | 56.59 | 65.11 | 49.43 | 59.59 | 60.41 | 45.96 | 55.22 | 64.85 | 44.01 | 57.26 |
| Claude-2.1 | 68.39 | 44.54 | 58.68 | 62.09 | 50.24 | 58.18 | 66.58 | 52.81 | 61.74 | 53.93 | 50.55 | 52.67 | 62.97 | 49.42 | 58.04 |
| Claude-3 | 77.53 | 52.25 | 67.24 | 72.53 | 64.12 | 69.76 | 75.08 | 68.69 | 72.83 | 64.36 | 62.80 | 63.78 | 72.57 | 61.75 | 68.63 |
| GPT-3.5 | 71.34 | 39.22 | 58.27 | 60.78 | 42.97 | 54.91 | 65.27 | 52.16 | 60.66 | 54.42 | 39.01 | 48.69 | 63.04 | 43.45 | 55.91 |
| GPT-4 | 78.53 | 59.36 | 70.73 | 75.40 | 59.21 | 70.06 | 77.38 | 67.64 | 73.95 | 67.21 | 64.40 | 66.16 | 74.85 | 62.66 | 70.41 |
| GPT-4o | 81.51 | 57.80 | 71.86 | 75.60 | 58.61 | 70.00 | 80.57 | 71.76 | 77.47 | 69.35 | 68.68 | 69.10 | 76.95 | 64.15 | 72.29 |

Figure: Figure 2: The distribution of knowledge-type and reasoning-type scores across all models.
Refer to caption: x2.png

Table 2 presents the overall results of all foundation models directly answering questions under the zero-shot setting (^3^33Due to space constraints, the results and analysis on CS-Bench (CN) are provided in Appendix E.3.). In summary, the overall scores of models range from 39.86% to 72.29%, demonstrating CS-Bench’s effectiveness in distinguishing between the abilities of various models in the field of CS while also posing significant challenges to the best-performing existing models.
Subsequently, we conduct a comprehensive analysis of the experimental results from various aspects including Foundation Models, Knowledge & Reasoning, Domains, and Task Formats.

Figure: Figure 4: (a) The performance of LLMs at different parameter scales. (b) The scale-score fitting curve of Qwen1.5 and Llama2 series. (c) Comparison of models under 0-shot answer-only, 5-shot answer-only, and 5-shot COT settings.
Refer to caption: x4.png

#### Comparison between Foundation Models.

Firstly, the closed-source models GPT-4/GPT-4o represent the highest standard on CS-Bench, being the only two models exceeding 70% proficiency.
Secondly, the disparity between the leading open-source and closed-source models is not significant. Notably, premier open-source models such as Qwen1.5-110B and Llama3-70B have surpassed previously strong closed-source models like GPT-3.5 and Claude-2.1, drawing close to Claude-3 in performance.
Thirdly, newer models demonstrate significant improvements compared to earlier versions. For example, among models with scales below 10B, Llama3-8B performs the best, rivaling previous much larger-scale models and even surpassing Llama2-70B, indicating significant potential for compression in model parameters [].
Lastly, while performance variations exist among models of different families at the same scale, models within the same family continue to improve with increasing scale on CS-Bench (see detailed scale analysis in Section 3.3).

#### Comparison of Knowledge and Reasoning.

Overall, all models perform worse on reasoning (average 44.63%) compared to knowledge scores (average 60.52%), indicating that reasoning poses a greater challenge to LLMs compared to knowledge. As shown in Figure 3, there is a strong positive correlation between reasoning scores and knowledge scores. However, this correlation is not absolute. For instance, PaLM-2 has a higher knowledge score but a lower reasoning score compared to Claude-2.1, showing PaLM-2’s weakness in applying knowledge. Furthermore, more powerful LLMs demonstrate a stronger ability to use knowledge for reasoning compared to weaker LLMs. This is reflected in the much lower reasoning scores of weaker models relative to their knowledge scores. However, as the model’s capability increases, the growth in reasoning scores is more pronounced than that of knowledge scores, gradually bridging the gap between knowledge and reasoning abilities.

#### Comparison between Domains.

First, regarding knowledge scores in Table 2 and Figure 3 (a), models generally perform best in DSA and worst in OS, which we attribute mainly to differences in the scale of pretraining data and the varying learning capabilities induced by model size. Second, the demand for reasoning ability varies across different domains, as evidenced by the gap between knowledge and reasoning scores. A notable example is GPT-4o, which shows close knowledge and reasoning scores in OS, while exhibiting extreme differences in DSA, with the highest and lowest scores, respectively. We further explore LLMs’ performance in fine-coursed subfields in Appendix E.1 and explore the impact of Code and Math abilities on different CS domains in Section 3.4.

#### Comparison between Tasks.

As shown in Figure 3 (b) and Table 13, given the varying initial random scores, LLMs generally performs best on Assertion questions (average 63.11% across all models), followed by MC questions (average 54.92%), Open-ended questions (average 49.1%), and performs worst on FITB questions (average 41%). However, the variation in task format sensitivity is highly pronounced in weaker models, while stronger models can mitigate the disparities caused by different task formats, exhibiting robustness. For instance, Llama2-7B scores only 26.19% on Open-ended reasoning but 60.61% on Assertion reasoning, whereas GPT-4 scores comparably on both Open-ended reasoning (68.94%) and Assertion reasoning (67.68%).

### 3.3 Qualitative Analysis

#### Relationship between Scores and Model Scales.

To investigate how the performance of models varies with the increase in parameter size, we examine several model families and plot the results in Figure 4 (a). It can be observed that although different families exhibit distinct performances, models within the same family consistently show improvement as the parameter size increases. However, as the model parameter size continues to increase, the performance gains from scaling diminish, resulting in diminishing returns in efficiency. For instance, the score in Qwen1.5 improves by 16.19% from 0.5B to 7B, by 7.11% from 14B to 72B, and by only 2.66% from 72B to 110B.
Additionally, as shown in Figure 4 (b), when the parameter scale grows exponentially, the score approximately increases linearly. This indicates that in the CS field, the model’s performance also follows a logarithmic scale pattern. Given the substantial computational resources required for large-scale models, we aim to establish the relationship between model scales and scores to predict the performance of larger-scale models in the CS field by fitting smaller-scale model scores. Due to space limitations, the specific design and implementation of the fitting function are provided in Appendix E.2. Overall, we fit the functions of Llama2 and Qwen1.5 series based on models ranging from 7B to 70/72B. We validate the fitting function on Qwen-1.5 110B, where the predicted value (67.83%) closely matches the actual value (67.95%), enabling further predictions for theoretical models, even up to 1000B.

Figure: Figure 5: The proportion of different error types varies by models for multiple-choice questions.
Refer to caption: x5.png

#### Comparison between Zero-shot, Few-shot and COT Prompting.

To investigate the impact of few-shot prompts and chain of thought (COT []) on model performance, we evaluate model’s performance under 5-shot answer-only (AO) and 5-shot COT prompts in Figure 4 (c), where the prompt samples are sampled from the validation set and match the domain of the test questions. Given that model-generated results under 0-shot COT often don’t adhere to specific formats, making regular matching difficult, we omit 0-shot COT experiments, similar to C-Eval. Additionally, for Open-ended questions, since the answers include detailed explanations, 5-shot COT is the same as 5-shot AO.
For all tested models, the 5-shot prompts show improvement compared to 0-shot, with average increases of 1.47% for 5-shot AO and 2.00% for 5-shot COT, respectively.
Moreover, the efficacy of few-shot prompts in bringing improvements appears more pronounced in some robust models such as GPT-3.5 and GPT-4, owing to their superior in-context learning capabilities.

#### Analysis of Error Types.

To delve into the roots of LLMs’ failure on CS-Bench and offer pathways toward improvement,
we acquire the solution process of model errors under 5-shot COT, and utilize GPT-4 to categorize each error type in MC questions in Figure 5. It should be emphasized that models may cause joint errors, resulting in more than one error type assigned to a single answer. In general, from Llama2-7B all the way to GPT-4, the total number of errors continues to decrease for both knowledge-type and reasoning-type questions.
For knowledge-type questions, both single concept errors and concept confusion show a decreasing trend. Initially, some completely wrong concepts transitioning to partially erroneous ones and subsequently being eliminated, thus exhibiting an initial rise followed by a decline in partial concept errors.
For reasoning-type questions, we observe that a significant portion of errors still fall under the category of knowledge-based mistakes. While stronger models have evidently reduced arithmetic reasoning errors for reasoning inaccuracies, there hasn’t been much change in logic reasoning errors specific to the CS field.
Our analysis highlights that reinforcing CS knowledge concepts is the most direct and effective approach to improving LLMs’ performance in the field of CS. Furthermore, significant improvements in CS reasoning performance are challenging to achieve solely by enhancing general reasoning abilities and mathematical reasoning, necessitating CS-specific reinforcement. More details can be found in E.4.

### 3.4 What’s the Relationship between CS, Math, and Code abilities of LLMs?

To explore the relationship between CS proficiency and the mathematical and coding capabilities of models, we investigate (1) the performance of general LLMs across the fields of Math, Code, and CS, and (2) the performance of LLMs specialized in Code and Math within the field of CS.

#### Exploration on General Models.

In Figure 6, we illustrate how the models’ performance on CS-Bench varies with increasing scores on the Math datasets (GSM8K [], MATH []) and Code datasets (HumanEval [], MBPP []). We observe that the overall trend in CS-Bench performance closely aligns with changes in Math and Code scores, as indicated by a Pearson correlation coefficient [] exceeding 0.9. Besides the general enhancement of diverse competencies that superior models typically bring, we consider this evidence to suggest a close correlation between CS proficiency and abilities in Math as well as Code.

Figure: Figure 6: The score changes on CS-Bench as LLM’s Math/Code score increases. $p$ denotes Pearson correlation coefficient. We obtain the scores on Math/Code datasets from [].
Refer to caption: x6.png

Next, we examine models with inconsistent patterns between CS and Math/Code. In the Math domain, Qwen1.5-7B outperforms Llama2-70B in both GSM8K and MATH, yet in CS-Bench, Llama2-70B surpasses Qwen1.5-7B. In the Code domain, Mixtral-8$\times$7B performs better than Qwen1.5-32B on HumanEval and MBPP, whereas the opposite is observed on CS-Bench. Given the NLP community’s sustained focus on the Code and Math domains, some recently released models have been trained on a large amount data in these domains, leading to smaller-scale models outperforming much larger-scale ones (e.g., Qwen1.5-7B surpassing Llama2-70B). However, in the CS domain, due to insufficient attention and training data, even excellent small-scale models struggle to surpass much larger-scale models. This also indicates that CS-Bench has not been overfitted during LLM pretraining, making it a fairer benchmark for measuring model performance differences.

**Table 3: The performance of the Math-expert LLMs on CS-Bench (EN). We use blue to emphasize areas where the expert LLMs improve compared to the Chat LLMs.**
| Model | Type | DSA | CO | CN | OS | All |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Klg | Rng | Klg | Rng | Klg | Rng | Klg | Rng | Klg | Rng | Avg |  |  |
| InternLm2-7B | Chat | 59.57 | 40.92 | 58.83 | 37.94 | 62.65 | 40.60 | 50.94 | 39.29 | 58.31 | 39.77 | 51.56 |
| InternLM-Math-7B [] | Math | 60.23 | 31.56 | 50.56 | 38.61 | 55.93 | 44.47 | 47.69 | 43.85 | 53.64 | 39.41 | 48.45 |
| DeepseekLLM-7B | Chat | 56.42 | 28.94 | 52.09 | 32.48 | 52.43 | 31.41 | 41.66 | 31.98 | 50.87 | 31.11 | 43.67 |
| DeepSeekMath-Instruct-7B [] | Math | 63.98 | 34.82 | 55.13 | 39.64 | 61.26 | 42.16 | 45.29 | 42.69 | 56.68 | 39.67 | 50.49 |
| Llama2-13B | Chat | 51.74 | 35.00 | 51.81 | 36.18 | 53.03 | 37.99 | 48.12 | 32.36 | 51.31 | 35.46 | 45.54 |
| MAammoTH-13B [] | Math | 50.84 | 28.26 | 46.16 | 34.61 | 51.39 | 30.45 | 34.94 | 32.64 | 46.20 | 31.32 | 40.78 |
| Llama2-70B | Chat | 64.28 | 41.51 | 56.35 | 40.85 | 61.99 | 43.07 | 51.79 | 41.15 | 58.73 | 41.68 | 52.52 |
| WizardMath-70B [] | Math | 60.17 | 28.67 | 56.41 | 34.91 | 58.52 | 41.51 | 47.01 | 42.53 | 55.77 | 36.67 | 48.82 |

**Table 4: The performance of the Code-expert LLMs on CS-Bench (EN).**
| Model | Type | DSA | CO | CN | OS | All |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Klg | Rng | Klg | Rng | Klg | Rng | Klg | Rng | Klg | Rng | Avg |  |  |
| Llama2-7B | Chat | 51.51 | 32.61 | 48.89 | 31.82 | 46.72 | 30.75 | 41.04 | 26.26 | 47.15 | 30.48 | 41.08 |
| CodeLlama-7B [] | Code | 58.90 | 36.15 | 45.46 | 36.24 | 52.87 | 26.23 | 44.35 | 25.33 | 50.36 | 31.09 | 43.34 |
| Dolphcoder-7B [] | Code | 50.13 | 36.47 | 34.71 | 34.36 | 41.78 | 23.92 | 40.03 | 28.35 | 41.40 | 30.82 | 37.54 |
| WizardCoder-7B [] | Code | 47.42 | 33.58 | 35.54 | 37.09 | 41.17 | 26.03 | 40.88 | 30.60 | 41.02 | 31.73 | 37.63 |
| Llama2-13B | Chat | 51.74 | 35.00 | 51.81 | 36.18 | 53.03 | 37.99 | 48.12 | 32.36 | 51.31 | 35.46 | 45.54 |
| CodeLlama-13B [] | Code | 59.87 | 34.17 | 44.96 | 35.82 | 51.56 | 35.83 | 43.28 | 34.56 | 49.84 | 35.08 | 44.47 |
| WizardCoder-13B [] | Code | 50.80 | 32.98 | 38.69 | 35.27 | 43.42 | 28.34 | 40.88 | 34.29 | 43.27 | 32.59 | 39.38 |

#### Exploration on Expert Models.

We present the results of the Math and Code expert LLMs in Tables 3 and 4. Compared to general Chat LLMs, expert LLMs usually sacrifice other abilities to boost proficiency in Math or Code, which is reflected in the lower overall performance of most expert LLMs. Therefore, we are more concerned with identifying the specific aspects of CS where Math and Code models show improvement.
Regarding mathematics, InternLm-Math-7B improves InternLm2-7B’s performance in CO, CN, and OS reasoning tasks, while DeepseekMath exhibits significant improvements across all domains. According to [], DeepseekMath effectively maintains general knowledge and reasoning ability during specialization. Conversely, MAammoTH and WizardMath perform poorly due to just fine-tuning on limited mathematical datasets, resulting in a significant decline in general knowledge and reasoning. The score changes in LLMs suggest that OS is most closely linked to mathematics, followed by CO, and lastly DSA and CN.
In terms of Code, many Code models show significant improvements in DSA (especially knowledge) and OS (especially reasoning), such as CodeLlama and Dolphcoder. This indicates that the disciplines of DSA and OS are more closely related to code, thus enhancing knowledge and reasoning abilities in these directions, while CO and CN have lower relevance, leading to a decrease in scores.
Finally, we observe that the enhancement brought about by small-scale expert LLMs compared to larger-scale LLMs is more pronounced (see CodeLlama-7B/13B, WizardCoder-7B/13B). We attribute this to the supplementary need for specific knowledge and reasoning capabilities in small-scale LLMs, whereas large-scale LLMs already encompass a greater breadth of knowledge and stronger reasoning abilities, resulting in diminishing gains from further training in specific domains.

## 4 Related Work

#### Exploration of LLMs in Computer Science.

Given the powerful capabilities of LLMs, recent research has explored their potential applications across various industries and scientific fields, including finance [], autonomous driving [, ], robotics [, , ], medicine [, ], and chemistry [, ].
Currently, studies exploring LLMs in the field of computer science fall into two main categories.
The first category includes broad evaluation benchmarks covering various fields, such as MMLU [], CMMLU [], C-Eval [], Xiezhi [], and M3KE []. However, computer science constitutes only a small fraction of these benchmarks, accounting for less than 5% and lacking detailed CS-specific analysis.
The second category focuses solely on exploring specific applications of LLMs within computer science, such as network topology [], cybersecurity [, ], and software engineering [, ].
Nonetheless, there has been a persistent lack of comprehensive evaluation of LLMs’ foundational knowledge and reasoning abilities in computer science. To address this gap, we propose CS-Bench and conduct a thorough evaluation of LLMs, providing guidance for understanding and improving their performance in the CS field.

#### Evaluation of LLMs’ Capabilities.

Evaluating and understanding the capabilities of LLMs is a major focus within the NLP community. Researchers have extensively explored the capabilities of LLMs including planning [], multilingual processing [, ], instruction following [, ], and cross-domain generalization [, , ]. Recently, there has been growing interest in LLMs’ abilities in mathematics [, , , , , ], code programming [, , , , ], and logical reasoning [, , , ]. While individual capabilities have been well-studied, research on their integrated application and interrelationships remains sparse. Different from [], which investigates interactions between abilities during the supervised fine-tuning phase, we choose computer science as our research context. Given that computer science inherently integrates coding, mathematics, and reasoning, we utilize CS-Bench in this paper to deeply explore the relationship between LLMs‘ performance in computer science and their mathematical and coding abilities, aiming to advance cross-capability research and integrated analysis of LLM abilities.

## 5 Conclusion

In this work, we introduce CS-Bench, the first benchmark specifically designed to systematically analyze the knowledge and reasoning capabilities of mainstream LLMs in the field of computer science. Our evaluation of over 30 models highlights that even the top-performing GPT-4o has significant room for improvement in computer science.
Further score-scale experiments and error type analyses provide directions for enhancing LLMs in the field. Moreover, our investigation into the relationship between computer science, mathematics, and coding demonstrates their close interconnections and provides valuable insights into LLMs’ cross-abilities and applications.

## Appendix A Limitations

In this paper, we introduce CS-Bench, providing a comprehensive evaluation of LLMs and exploring the relationships between model capabilities. However, there are still some limitations to this paper.

(1) Coverage Limitations: Although CS-Bench has made significant strides in comprehensiveness of CS evaluations compared to existing work, given the breadth of computer science, our evaluations cannot cover the entire scope of computer science knowledge. Furthermore, our assessment content focuses on university-level content, examining LLM’s mastery of basic subjects in computer science, rather than specific computer science-related research scenarios.

(2) Evaluation Limitations: In the CS-Bench evaluation experiments, we employ GPT-4 scoring to assess generative tasks such as fill-in-the-blank and open-ended tasks. This might lead to certain evaluation thresholds and costs. However, such issues only constitute about 20% of CS-Bench. Additionally, we provide an evaluation scheme that separates comprehension tasks from CS-Bench, allowing for automatic evaluations without the need for GPT-4.

(3) Language Limitations: CS-Bench are primarily focused on Chinese and English-dominated language environments, ensuring comprehensive and in-depth evaluations in these two language environments. However, for other non-Chinese and English language environments, its support and coverage are relatively weak, and further optimization and improvement are needed.

## Appendix B Broaden Impact

#### Societal Impact.

CS-Bench is anticipated to play a significant role in the field of computer science. LLMs, trained and evaluated with the aid of CS-Bench, can enhance the work efficiency of relevant professionals, enabling them to complete computer-related tasks, such as code review, error detection, and algorithm optimization, more quickly and accurately. Although this might result in the disappearance of some repetitive jobs, it could also create new career opportunities. In the realm of education, the CS-Bench dataset can serve as an effective teaching tool, assisting teachers in better explaining complex computer science concepts and techniques, and also enabling students to better understand and master this knowledge through practice.

#### Ethics Statement.

We ensure adherence to applicable laws and ethical guidelines during the process of data collection, annotation, and usage, providing adequate compensation to all our crowd workers. As this benchmark pertains to objective knowledge and reasoning in the field of computer science, the annotation content is not influenced by regional or cultural differences among annotators. Moreover, our dataset does not contain any personally identifiable information or offensive content. The authenticity and accuracy of CS-Bench have been thoroughly verified, providing a reliable basis for evaluating LLMs. CS-Bench is intended solely for academic and research purposes. Any commercial use or other misuse deviating from this purpose is strictly prohibited. We urge all users to respect this provision to maintain the integrity and ethical use of this valuable resource.

## Appendix C More Details on CS-Bench

In C.1, we provide a detailed explanation of the design motivation and statistics for CS-Bench. In C.2, we present the distribution of question and answer lengths for each task in CS-Bench. In C.3, we provide a case example for each type under each dimension of CS-Bench.

### C.1 Detailed Design Motivation and Statistics of CS-Bench

We elaborate on the design motivation of CS-Bench and statistics under each dimension as follows.

#### Evaluation Content.

To ensure comprehensive coverage of fundamental and critical areas in computer science, we select the four most foundational and prevalent domains within the field of computer science as the core content of the CS-Bench dataset. These four domains are as follows: Data Structure and Algorithm, investigating data organization and algorithmic efficiency; Computer Organization, focusing on hardware composition and foundational system operation; Computer Network, involving the analysis of network communication and data transmission; Operating System, delving into system resource management and process control. As depicted in Figure 7 (a), these four disciplines exhibit a roughly uniform distribution. Furthermore, we subdivide the disciplines into 26 granular chapters, allowing CS-Bench to furnish more nuanced evaluation outcomes for models and provide comprehensive guidance for model refinement. We summarize these chapters in Table 5.

**Table 5: Summary of 26 fine-grained subfields of CS-Bench.**
| Chatpter | Main Content | Subject | Question Number |
| --- | --- | --- | --- |
| Overview | Concepts and elements of data structure, Temporal and spatial complexity… | DSA | 84 |
| Linear List | Linear tables, Sequential tables and Linked lists… | DSA | 138 |
| Stack, Queue,and Array | Shared stack, Circle queue, Arrays,Special matrices… | DSA | 176 |
| String | Concept and operation of strings, Pattern matching of strings… | DSA | 66 |
| Tree | Binary trees, Traversal of trees ans forests, Huffman tree… | DSA | 214 |
| Graph | Concepts of graphs, Traversals of graphs,Application of graphs… | DSA | 184 |
| Searching | Sequential search, Half-split search, Chunked search, Red-black tree, B-tree<br>and B+ tree, Hash search… | DSA | 158 |
| Sorting | Insert Sorting, Swap Sorting, Selection Sorting, Merge Sorting, Heap Sorting,<br>Merge Sorting, Cardinality Sorting, External Sorting Algorithms… | DSA | 178 |
| Overview | Hardware and performance indicators of computers… | CO | 112 |
| Data Representation and<br>Operation | Number system and encoding, Representation and operation of fixed-point nu-<br>mbers and floating-point numbers… | CO | 218 |
| Storage System | Main Memory, External Memory, Cache Memory, Virtual Memory… | CO | 224 |
| Instruction System | Instruction format, Instruction addressing method, CISC and RISC… | CO | 156 |
| Central Processing Unit | Functions of CPU, Instruction execution process, CPU internal bus and data<br>path, CPU hard wiring design and micro programming, Exception and inter-<br>rupt mechanisms, Instruction pipelines, and multiprocessor concepts… | CO | 244 |
| Bus | Overview of the bus, Bus arbitration, Bus operation and timing, Bus standards… | CO | 134 |
| Input/Output System | I/O interfaces and methods… | CO | 156 |
| Overview and Architecture | Concepts, compositions, functions of computer networks, Architecture and<br>reference models of computer networks… | CN | 148 |
| Physical Layer | Fundamentals of Communication Theory, Transmission Media and Physical<br>Layer Devices… | CN | 164 |
| Data Link Layer | Data frames, Error control, Flow control and Reliable transmission, Media acc-<br>ess control, Local and wide area networks, and data link layer devices… | CN | 316 |
| Network Layer | Overview of network layer functions, Routing algorithms, IPv4 and IPv6, Rou-<br>ting protocols, IP multicast, Mobile IP, Router… | CN | 300 |
| Transport Layer | The services provided by the transport layer, UDP and TCP protocols… | CN | 182 |
| Application Layer | Network application model, Domain name system DNS, FTP protocol, World<br>Wide Web, and HTTP… | CN | 204 |
| Overview | Concepts of operating systems, Development and classification of operating<br>systems, Operational mechanisms and architecture of operating systems… | OS | 166 |
| Processes and Threads | Processes and threads, Scheduling of processors, Synchronization and mutual<br>exclusion of processes, Deadlock issues… | OS | 350 |
| Memory Management | Concept of memory management, Concept of virtual memory management,<br>and methods of virtual memory management… | OS | 216 |
| File Management | File systems, Organization and management of disks… | OS | 166 |
| Input/Output Management | I/O devices and control methods, I/O core subsystem, Buffer management… | OS | 184 |

Figure: Figure 7: The quantity and proportion of each type in different dimensions on CS-Bench.
Refer to caption: x7.png

#### Task Format.

To better simulate the diverse forms of problems encountered in the real world, we introduce assertion, fill-in-the-blank, and open-ended questions in addition to multiple-choice questions. Specifically, multiple-choice and assertion questions correspond to understanding tasks in CS, while fill-in-the-blank and open-ended questions correspond to generation tasks in CS. Although assessing generation tasks using GPT-4 incurs certain costs, it is important to emphasize that this component represents only a minority (fill-in-the-blank: 10.67%, open-ended: 7.81%), whereas comprehension tasks relying on rule-based scoring constitute the majority (multiple-choice: 61.22%, assertion: 20.3%). Therefore, if resources are limited, we recommend considering the independent use of understanding tasks from CS-Bench for evaluation purposes.

#### Knowledge/Reasoning.

The design goal of CS-Bench is not only to assess the mastery of knowledge in the field of CS but also to evaluate the model’s ability to reason using CS knowledge. Therefore, each dataset is labeled with “knowledge” or “reasoning”, corresponding to simple questions requiring knowledge recall and challenging questions necessitating knowledge inference, respectively. As shown in Figure 7 (c), knowledge-based questions account for 63.58%, while reasoning-based questions account for 36.42%.

#### Language.

To assess the ability of LLMs in addressing CS problems in various linguistic environments, and to adapt CS-Bench for the evaluation of a wider range of LLMs, CS-Bench comprises bilingual Chinese-English data, with each language accounting for 50%. The English data is obtained through translation by GPT-4, followed by manual verification of processed Chinese data.

### C.2 Distribution of Word Lengths

Due to CS-Bench containing both English and Chinese languages, we separately compute the distributions of word lengths for questions and answers in CS-Bench (English) and CS-Bench (Chinese) across various task formats, as illustrated in Figure 8 and Figure 9. For Multiple-Choice questions, the question length includes both the question itself and the four options. Since Multiple-Choice and Assertion questions are comprehension tasks, the answers consist of only one character (A/B/C/D or True/False). For generation tasks, Fill-in-the-blank answers are relatively short, with an average word length of approximately 2, whereas Open-ended questions typically yield longer answers as they entail detailed explanatory processes.

Figure: Figure 8: Question and answer lengths of each task format in CS-Bench (English).
Refer to caption: x8.png

Figure: Figure 9: Question and answer lengths of each task format in CS-Bench (Chinese).
Refer to caption: x9.png

### C.3 CS-Bench Examples

We present samples from various domains in Table 6, samples of different task formats in Table 7, samples of knowledge and reasoning types in Table 8, and samples from different languages in Table 9.

**Table 6: Examples of samples in different domains.**
| Domain | Example |
| --- | --- |
| Data<br>Structure<br>and<br>Algorithm | Question:<br>The correct statement about data structures is ().<br>A: The logical structure of data is independent of its storage structure.<br>B: The storage structure of data is independent of its logical structure.<br>C: The logical structure of data uniquely determines its storage structure.<br>D: The data structure is determined solely by its logical structure and storage structure.<br>Answer:<br>A<br>Analysis:<br>The logical structure of data is approached from the perspective of practical problems, using only abstract expressions and is independent of the various choices of data storage methods. The storage structure of data is the mapping of the logical structure on a computer, and it cannot exist independently of the logical structure. Data structure includes three essential elements, all of which are indispensable. |
| Computer<br>Organization | Question:<br>A complete computer system should include ().<br>A: Arithmetic Logic Unit (ALU), Memory, Control Unit<br>B: Peripheral devices and host computer<br>C: Host and Application<br>D: The accompanying hardware devices and software systems<br>Answer:<br>D<br>Analysis:<br>A is a component of the computer host, while B and C only involve parts of the computer system and are both incomplete. |
| Computer<br>Network | Question:<br>The most basic function of computer networks is ().<br>A: Data Communication<br>B: Resource Sharing<br>C: Distributed Processing<br>D: Information Synthesis Processing<br>Answer:<br>A<br>Analysis:<br>The functions of computer networks include: data communication, resource sharing, distributed processing, integrated information processing, load balancing, enhancing reliability, etc. However, the most fundamental function is data communication, which is also the basis for realizing other functions. |
| Operating<br>System | Question:<br>Among the following options, () is not an issue of concern for the operating system.<br>A: Manage bare-metal computers<br>B: Design and provide an interface between user programs and hardware systems<br>C: Manage computer system resources<br>D: Compiler for High-Level Programming Languages<br>Answer:<br>D<br>Analysis:<br>The operating system manages computer software/hardware resources, expands the bare machine to provide a more powerful extended machine, and acts as an intermediary between the user and the hardware. Clearly, the compiler for high-level programming languages is not a concern of the operating system. The essence of a compiler is a set of program instructions that are stored in the computer. |

**Table 7: Examples of different task formats.**
| Type | Example |
| --- | --- |
| Multiple<br>Choice | Question:<br>Given that the storage space for a circular queue is the array A[21], with front pointing to the position before the head element and rear pointing to the tail element, assuming the current values of front and rear are 8 and 3, respectively, the length of the queue is ().<br>A: 5<br>B: 6<br>C: 16<br>D: 17<br>Answer:<br>C<br>Analysis:<br>The length of the queue is (rear - front + maxsize) % maxsize = (rear - front + 21) % 21 = 16. This situation is the same as when front points to the current element and rear points to the next element after the last element in the queue. |
| Assertion | Question:<br>In a directed graph with n vertices, the degree of each vertex can reach up to 2n.<br>Answer:<br>False.<br>Analysis:<br>In a directed graph, the degree of a vertex is equal to the sum of its in-degree and outdegree. In a directed graph with n vertices, any given vertex can have at most one pair of oppositely directed edges connecting it with each of the other n-1 vertices. |
| Fill-in-<br>the-blank | Question:<br>In a sequential list of length n, when deleting the ith ($1\leq i\leq n$) element, () elements need to be moved forward.<br>Answer:<br>n-i<br>Analysis:<br>The elements from a[i+1] to a[n] need to be moved forward by one position, involving the movement of n-(i+1)+1=n-i elements. |
| Open-ended | Question:<br>Given that the 9th level of a complete binary tree has 240 nodes, how many nodes does<br>the entire complete binary tree have? How many leaf nodes are there?<br>Answer:<br>In a complete binary tree, if the 9th level is full, then the number of nodes =<br>$2^{(9-1)}$ = 256. However, currently, there are only 240 nodes on the 9th level, indicating that the 9thlevel is not full and is the last level. Levels 1 to 8 are full, so the total number of<br>nodes = $2^{8}+240=495$. Since the 9th level is the last level, all nodes on the 9th level are leaf nodes. Moreover, the parents of the 240 nodes on the 9th level are on the 8th level, with the number of parents being 120, which means there are 120 branch nodes on the 8th level, and the rest are leaf nodes. Therefore, the number of leaf nodes on the 8th level is $2^{(8-1)}-120=8$. Consequently, the total number of leaf nodes = 8 + 240 = 248.<br>Analysis:<br>None |

**Table 8: Examples of knowledge-type and reasoning-type.**
| Type | Example |
| --- | --- |
| Knowledge | Question:<br>The three fundamental elements of data structure include ().<br>A: Logical structure, storage structure, operations on data.<br>B: Logical structure, algorithm design, program implementation.<br>C: Data types, data storage, data manipulation.<br>D: Data Definition, Data Implementation, Data Manipulation.<br>Answer:<br>A<br>Analysis:<br>None |
| Reasoning | Question:<br>The time complexity of a certain algorithm is $O(n^{2})$, indicating that the algorithm’s ().<br>A: The problem size is $O(n^{2})$.<br>B: Execution time equals $O(n^{2})$.<br>C: The execution time is directly proportional to $O(n^{2})$.<br>D: The problem size is directly proportional to $O(n^{2})$.<br>Answer:<br>C<br>Analysis:<br>The time complexity is $O(n^{2})$, which means the time complexity $T(n)$ satisfies $T(n)\leq c*n^{2}$ (where $c$ is a proportionality constant), that is, $T(n)=O(n^{2})$. The time complexity $T(n)$ is a function of the problem size $n$, and the problem size remains $n$, not $n^{2}$. |

**Table 9: Examples of different languages.**
| Type | Example |
| --- | --- |
| English | Question:<br>For a linear list with sequential storage, the operation with a time complexity of O(1)<br>should be ().<br>A: Sort n elements in ascending order.<br>B: Remove the i-th ($1\leq i\leq n$) element.<br>C: Change the value of the i-th element ($1\leq i\leq n$).<br>D: Insert a new element after the i-th ($1\leq i\leq n$) element.<br>Answer:<br>C<br>Analysis:<br>The time complexity for sorting n elements is at least O(n) (when initially ordered), and typically $\mathcal{O}(n\log_{2}n)$ or $\mathcal{O}(n^{2})$. Options B and D are clearly incorrect. Sequential lists support random access by index. |
| Chinese | Question:<br>{CJK*}UTF8gbsn对于顺序存储的线性表，其算法时间复杂度为O(1)的运算应该是()。<br>A: {CJK*}UTF8gbsn将n个元素从小到大排序<br>B: {CJK*}UTF8gbsn删除第i ($1\leq i\leq n$)个元素<br>C: {CJK*}UTF8gbsn改变第i ($1\leq i\leq n$)个元素的值<br>D: {CJK*}UTF8gbsn在第i ($1\leq i\leq n$)个元素后插入个新元素<br>Answer:<br>C<br>Analysis:<br>{CJK*}UTF8gbsn对n个元素进行排序的时间复杂度最小也要O(n)（初始有序时）通常为<br>{CJK*}UTF8gbsn$\mathcal{O}(n\log_{2}n)$或$\mathcal{O}(n^{2})$。B和D显然错误。顺序表支持按序号的随机存取方式。 |

## Appendix D More Details on Experiment Setup

In D.1, we present the question templates used to prompt models for each type of task. In D.2, we show the prompts used for GPT-4 to score models’ answers to fill-in-the-blank and open-ended questions, and validate the effectiveness of GPT-4’s automatic scoring through consistency experiments with human scoring. In D.3, we detail the experimental environment used to implement model inference. In D.4, we introduce all the evaluated model families.

### D.1 Details of Template for Each Task Format

We present the templates for querying LLMs with various question formats in Table 10.

**Table 10: Prompt Templates for asking various questions to LLMs.**
| Type | Prompt Template |
| --- | --- |
| Multiple<br>Choice | This is a multiple-choice question. Please read the question carefully and choose the<br>correct answer. Question: <Question><br>Which one of the following options is correct? Options:<br>(A) <A><br>(B) <B><br>(C) <C><br>(D) <D><br>Please provide the answer to this question directly (a single letter): |
| Assertion | This is a true/false question. Please determine whether the following statement is true<br>or false. Statement: <Question><br>Please give the answer directly (true or false): |
| Fill-in-<br>the-blank | You are a professor proficient in computer science. This is a fill-in-the-blank question.<br>Give answers to the following question without explanation or repeating it.<br>Question: <Question><br>Answer: |
| Open-ended | This is a subjective Question: <Question><br>Please provide a brief answer to this question: |

### D.2 Details of GPT-4 Scoring

#### GPT-4 Scoring Prompt.

In Table 12, we present the prompts utilized to instruct GPT-4 in scoring the outputs of LLMs in CS generation tasks, encompassing both Fill-in-the-blank and Open-ended questions.

#### Consistency between GPT-4 Scoring and Manual Scoring.

To assess the effectiveness of GPT-4 scoring in evaluating LLM responses, we conduct a consistency experiment between GPT-4 prediction scores and manual scores. For Fill-in-the-blank and Open-ended types, we randomly sample 100 instances from the GPT-4 scoring samples and employ three human annotators to score these predicted results. In Table 11, we report the consistency scores among human annotators (measured by Cronbach’s alpha), as well as the consistency scores between the average human annotation scores and GPT-4 scoring (measured by Pearson correlation coefficient). The excellent consistency between human and GPT-4 scores validates the effectiveness of GPT-4 scoring.

**Table 11: Consistency between GPT-4 scoring and human scoring.**
| Type | Annotation Count | Consistency |  |
| --- | --- | --- | --- |
| Human-GPT4 | Human-Human |  |  |
| Fill-in-the-blank | 100 | 0.808 | 0.9311 |
| Open-ended | 100 | 0.9494 | 0.9751 |

**Table 12: Scoring Prompts for Fill-in-the-blank and Open-ended Questions.**
| Type | Prompt Template |
| --- | --- |
| Fill-in-<br>the-blank | You are now a teaching assistant. As a TA, your task is to grade the fill-in-the-blank<br>assignments of computer science students.<br>You will see the standard answer for each question (these answers are verified and<br>completely correct), and you need to score the students’ answers based on this.<br>If the student’s answer conveys the same meaning as the standard answer or other<br>answers (different formats are also considered correct), then award 1 point; if not,<br>then 0 points.<br>Question: <question><br>Standard Answer: <correct_answer><br>Other Answers: <other_answers><br>Student Response: <predict_output><br>Score (0 or 1): |
| Open-<br>ended | You are now serving as a teaching assistant. In this role, your task is to grade the<br>subjective homework assignments of computer science students. You will be presented<br>with the standard answers for each question (which are verified and completely correct),<br>and you must use these to score the students’ responses. The grading scale ranges from 1<br>to 10 points, with 10 being the highest and 1 being the lowest. When grading, please take<br>into consideration the accuracy, relevance, completeness, and depth of thought of the<br>answers. Scores should be assigned based on the following *criteria*:<br>First Tier: 1-3 points<br>Accuracy: The answer contains several fundamental errors, showing limited understanding.<br>Relevance: The answer has low relevance to the question and standard answer, with most<br>content straying from the requirements.<br>Completeness: The answer omits multiple key<br>points, failing to cover the main aspects of the question.<br>Second Tier: 4-6 points<br>Accuracy: There are some errors in the answer, although most of the basic concepts are<br>understood correctly.<br>Relevance: The answer is generally relevant to the question and standard answer, but<br>some content does not fully conform to the requirements.<br>Completeness: The answer is fairly complete, but lacks some important details or certain<br>key points are not fully elaborated.<br>Third Tier: 7-8 points<br>Accuracy: The answer is almost entirely correct, with only very minor errors.<br>Relevance: The answer is highly relevant to the question and standard answer, focused<br>and with almost no deviation from the topic.<br>Completeness: The answer is comprehensive and detailed, covering all key aspects<br>very well.<br>Fourth Tier: 9-10 points<br>Accuracy: The answer is free of any errors, demonstrating a deep understanding and<br>precise grasp of the issue.<br>Relevance: The answer is in complete accordance with the requirements, strictly aligned<br>with the question and standard answer, without any deviation.<br>Completeness: The answer is structured rigorously, logically organized, and systemati-<br>cally covers all aspects of the question.<br>Grading Guide: When assigning a score, please first make a preliminary assessment of<br>accuracy based on the student’s answer compared to the standard answer. Then, consider<br>the relevance and completeness to determine the final score. Ensure that each point<br>awarded is based on a fair and justified comprehensive evaluation. |

### D.3 Details of Inference Implementation

For all open-source models, we utilize a cluster with 8 NVIDIA A100-80GB GPUs to run the inference, and we use vLLM [] for inference acceleration, applying the corresponding chat templates and the same hyper-parameters: batch size=1, temperature=0, top-p=1.0, and max_tokens=2048. For all closed-source models with API access, we also adopt the generation scheme with temperature=0, and simply run the inference with CPUs, which typically completes within a day. During the evaluation of GPT-4, we also applied the setting of temperature=0. To avoid error bias, we conducted the experiments 3 times and took the average of the scores. For models supporting web search or tool calls, we disable these features to ensure a fair comparison.

### D.4 Details of the Models being Evaluated

#### Gemma

[] is a family of lightweight, open models from Google, built from the same research and technology used to create the Gemini models. They are text-to-text, decoder-only large language models, available in English, with open weights, pre-trained variants, and instruction-tuned variants. The Gemma model excels on academic benchmarks in language understanding, reasoning, and security. Gemma publishes models in two sizes (2 billion and 7 billion parameters) .

#### Llama2

[] is an upgraded version of Llama developed by MetaAI. It utilizes more robust data cleaning and mixing techniques, and up-samples sources closest to factual information, which can enhance knowledge and reduce hallucinations. Additionally, it employs Grouped-Query Attention technology to lessen reliance on memory.

#### Llama3

[] is the latest generation of large language models developed by MetaAI. The training dataset for Llama 3 is seven times larger than that used for Llama 2, with the amount of code included being four times that of Llama 2. Compared to previous versions of the model, it has seen a tremendous enhancement in reasoning, code generation, and instruction following capabilities.

#### Llama3-Chinese

[] is an instruction-tuned language model for Chinese and English users with various abilities such as roleplaying and tool-using built upon the Meta-Llama-3-8B-Instruct model.

#### ChatGLM3

[] is a next-generation conversational pre-trained model jointly released by Zhipu AI and KEG Lab of Tsinghua University. ChatGLM3-6B adopts a newly designed Prompt format, in addition to regular multi-turn dialogue. It also natively supports complex scenarios such as function call, code interpretation.

#### Baichuan2

[] is a large-scale multilingual model developed by Baichuan Company. It adopts several advanced techniques in its design and training process, including Rotary Position Embedding, a novel position encoding technique, SwiGLU activation function, and memory efficient attention mechanism. Compared with Baichuan1, its performance has been greatly improved.

#### InternLM2

[] is an open-source large-scale language model developed by Shanghai AI Laboratory. This model has good processing ability for ultra long texts and adopts COOL RLHF technology. It solves human preference conflicts through a conditional reward model and performs multiple rounds of online RLHF to improve the model’s alignment ability.

#### Qwen1.5

[] is a family of language models developed by Alibaba. It has features such as SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Qwen 1.5 series models have strong basic capabilities including language understanding.

#### Mistral-7B

[], a 7-billion-parameter language model designed for superior performance and efficiency, which is developed by Mistral AI. Mistral 7B leverages Packet Query Attention (GQA) for faster inference, combined with Sliding Window Attention (SWA) to efficiently process sequences of arbitrary length while reducing inference costs.

#### Mixtral-8 × \times × 7B

[] is a Sparse Mixture of Experts (SMoE) language model developed by Mistral AI. Its architecture is the same as that of the Mistral 7B, except that each layer consists of 8 feedforward blocks (i.e., experts). Mixtral has demonstrated exceptional abilities in math, code generation, and tasks that require multilingual understanding.

#### DeepSeekLLM

[] is a family of models released by DeepSeek-AI, and its core architecture borrows from the Llama model. This family of models employs Multi-Head Attention (MHA) and Group Query Attention (GQA) techniques, which significantly enhance their performance and efficiency. Furthermore, DeepSeekLLM demonstrates strong bilingual capabilities in both Chinese and English.

#### PaLM-2

[] is the higher-performance successor to PaLM released by Google, which differs in terms of dataset mixing. Compared to the first-generation PaLM version, it uses a smaller model but performs more training calculations. It also relies on more diverse pre-training targets.

#### Claude

Claude2.1[] and Claude3 [] are AI models developed by Anthropic, showcasing advanced language understanding and generation capabilities. Utilizing the constitutional AI framework, Claude models are designed to ensure helpfulness and trustworthiness.

#### GPT

GPT-3.5 [], GPT-4 [] and GPT-4o [], released by OpenAI, are part of the GPT-series models enhanced by a three-stage reinforcement learning with human feedback (RLHF) algorithm. This algorithm not only improves the models’ ability to follow instructions but also significantly reduces the generation of harmful or toxic content. Additionally, GPT-4 supports image inputs and achieves human-level performance on various benchmarks. GPT-4o, the latest model developed by OpenAI, boasts powerful real-time reasoning, language interaction, and multimodal capabilities.

#### GLM-4

[] is a new generation base large model developed by Zhipu AI. It has strong tool calling and multi-modal capabilities, as well as strong mathematical reasoning ability and code generation ability.

#### ERNIE

[] ERNIE3.5 and ERNIE4 are large language models developed by Baidu. ERNIE3.5 is capable of processing text data in multiple languages and has a good understanding and representation ability for entities and relationships in text. Ernie 4 has adopted more advanced knowledge graph information and more advanced knowledge integration technology, further improving the performance of the model.

## Appendix E More Details on Experiment

In E.1, we present detailed performance of the models on CS-Bench (EN), including the leaderboard, task formats, and domains. In E.2, we describe and validate the design of the scale-score fitting function. In E.3, we evaluate models’ performance on CS-Bench (CN) and compare the differences in performance between the English and Chinese contexts. In E.4, we conduct case studies to better understand the specific details of the models’ failures on CS-Bench.

### E.1 Details of Model Performance

#### The Leaderboard on CS-Bench (EN).

We visualize the results of LLMs on CS-Bench (EN) in Figure 10.

Figure: Figure 10: The leaderboard of LLMs on CS-Bench (EN).
Refer to caption: x10.png

#### Detailed Performance on Each Task Format.

We present models’ performance on four types of tasks in Table 13 and visualize the results in Figure 11.

**Table 13: Zero-shot scores (%) of LLMs across question formats on CS-Bench (EN).**
| Model | Multiple-choice | Assertion | Fill-in-the-blank | Open-ended | All |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg |  |
| Random | 25.00 | 25.00 | 25.00 | 50.00 | 50.00 | 50.00 | 0.00 | 0.00 | 0.00 | 10.00 | 10.00 | 10.00 | 27.4 | 24.12 | 26.20 |
| Open-source LLM (Scale < 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Gemma-2B | 46.87 | 25.85 | 38.74 | 52.58 | 48.48 | 51.64 | 34.12 | 7.55 | 27.68 | 49.60 | 26.02 | 32.96 | 46.89 | 27.59 | 39.86 |
| Qwen1.5-4B | 53.00 | 35.47 | 46.22 | 56.84 | 58.59 | 57.24 | 29.41 | 11.32 | 25.02 | 55.40 | 28.58 | 36.47 | 51.18 | 35.70 | 45.54 |
| ChatGLM3-6B | 47.51 | 33.07 | 41.92 | 58.97 | 60.61 | 59.34 | 31.76 | 5.66 | 25.43 | 53.80 | 28.94 | 36.25 | 48.63 | 34.07 | 43.33 |
| Llama2-7B | 47.00 | 28.06 | 39.67 | 56.84 | 60.61 | 57.70 | 23.53 | 5.66 | 19.20 | 63.80 | 26.19 | 37.25 | 47.15 | 30.48 | 41.08 |
| DeepseekLLM-7B | 50.19 | 28.06 | 41.63 | 60.49 | 58.59 | 60.06 | 31.76 | 13.21 | 27.26 | 59.80 | 28.67 | 37.83 | 50.87 | 31.11 | 43.67 |
| Baichuan2-7B | 47.51 | 35.27 | 42.77 | 57.14 | 59.60 | 57.70 | 32.94 | 7.55 | 26.78 | 52.40 | 26.90 | 34.40 | 48.29 | 35.33 | 43.57 |
| Gemma-7B | 56.70 | 33.07 | 47.56 | 58.05 | 57.58 | 57.94 | 38.82 | 15.09 | 33.06 | 58.20 | 33.36 | 40.67 | 54.90 | 35.02 | 47.66 |
| Qwen1.5-7B | 59.90 | 40.08 | 52.23 | 58.97 | 56.57 | 58.42 | 38.24 | 16.98 | 33.08 | 69.60 | 35.75 | 45.71 | 57.34 | 40.08 | 51.05 |
| InternLm2-7B | 59.26 | 39.48 | 51.61 | 60.49 | 55.56 | 59.36 | 45.88 | 15.09 | 38.41 | 69.00 | 39.03 | 47.84 | 58.31 | 39.77 | 51.56 |
| Mistral-7B | 57.34 | 39.68 | 50.51 | 62.61 | 54.55 | 60.77 | 53.53 | 16.98 | 44.66 | 67.40 | 42.39 | 49.75 | 58.63 | 40.44 | 52.01 |
| Llama3-8B | 61.81 | 46.09 | 55.73 | 64.44 | 61.62 | 63.80 | 38.24 | 11.32 | 31.71 | 67.60 | 41.33 | 49.06 | 59.75 | 44.97 | 54.37 |
| Open-source LLM (Scale > 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Llama2-13B | 50.06 | 33.87 | 43.79 | 55.93 | 56.57 | 56.08 | 44.71 | 22.64 | 39.36 | 62.00 | 29.65 | 39.16 | 51.31 | 35.46 | 45.54 |
| Baichuan-13B | 53.00 | 37.68 | 47.07 | 58.66 | 53.54 | 57.49 | 35.88 | 16.98 | 31.30 | 59.80 | 31.15 | 39.58 | 52.53 | 37.44 | 47.03 |
| Qwen1.5-14B | 64.62 | 50.70 | 59.23 | 62.61 | 59.60 | 61.92 | 51.76 | 28.30 | 46.07 | 70.60 | 43.45 | 51.44 | 62.79 | 49.18 | 57.83 |
| InternLm2-20B | 62.20 | 43.69 | 55.04 | 61.09 | 62.63 | 61.44 | 51.18 | 24.53 | 44.72 | 67.20 | 36.02 | 45.19 | 60.81 | 43.66 | 54.56 |
| Qwen1.5-32B | 70.63 | 57.92 | 65.71 | 63.53 | 62.63 | 63.32 | 53.53 | 22.64 | 46.04 | 73.20 | 48.76 | 55.95 | 66.87 | 54.72 | 62.45 |
| Mixtral-8$\times$7B | 66.28 | 47.09 | 58.85 | 67.78 | 56.57 | 65.22 | 58.24 | 26.42 | 50.52 | 71.00 | 45.93 | 53.30 | 65.91 | 46.66 | 58.90 |
| DeepseekLLM-67B | 66.92 | 45.29 | 58.55 | 65.96 | 63.64 | 65.43 | 54.71 | 28.30 | 48.30 | 67.20 | 42.57 | 49.81 | 65.23 | 45.96 | 58.22 |
| Llama2-70B | 58.88 | 42.28 | 52.46 | 61.09 | 59.60 | 60.75 | 51.18 | 16.98 | 42.88 | 63.80 | 34.96 | 43.44 | 58.73 | 41.68 | 52.52 |
| Llama3-70B | 73.95 | 57.52 | 67.59 | 69.91 | 63.64 | 68.48 | 63.53 | 37.74 | 57.27 | 72.00 | 53.98 | 59.28 | 71.65 | 56.36 | 66.08 |
| Qwen1.5-72B | 72.03 | 60.32 | 67.50 | 70.52 | 66.67 | 69.64 | 55.29 | 28.30 | 48.74 | 73.00 | 52.30 | 58.39 | 69.63 | 57.75 | 65.31 |
| Qwen1.5-110B | 74.33 | 62.73 | 69.84 | 73.25 | 67.68 | 71.98 | 57.06 | 33.96 | 51.46 | 75.20 | 60.00 | 64.47 | 71.98 | 60.91 | 67.95 |
| Closed-source LLM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PaLM-2 | 65.91 | 43.66 | 57.30 | 66.36 | 62.77 | 65.54 | 56.52 | 29.79 | 50.04 | 64.47 | 35.64 | 44.12 | 64.85 | 44.01 | 57.26 |
| Claude-2.1 | 63.47 | 46.89 | 57.05 | 66.87 | 67.68 | 67.06 | 49.41 | 24.53 | 43.38 | 72.40 | 55.84 | 60.71 | 62.97 | 49.42 | 58.04 |
| Claude-3 | 73.82 | 61.32 | 68.98 | 73.56 | 70.71 | 72.91 | 62.94 | 37.74 | 56.83 | 76.73 | 66.11 | 69.23 | 72.57 | 61.75 | 68.63 |
| GPT-3.5 | 63.35 | 41.48 | 54.89 | 68.39 | 63.64 | 67.30 | 48.82 | 24.53 | 42.93 | 68.00 | 42.65 | 50.11 | 63.04 | 43.45 | 55.91 |
| GPT-4 | 77.27 | 62.32 | 71.48 | 75.38 | 67.68 | 73.62 | 61.18 | 43.40 | 56.87 | 77.40 | 68.94 | 71.43 | 74.85 | 62.66 | 70.41 |
| GPT-4o | 80.08 | 63.73 | 73.75 | 75.68 | 72.73 | 75.01 | 64.71 | 41.51 | 59.08 | 75.20 | 69.47 | 71.16 | 76.95 | 64.15 | 72.29 |

Figure: Figure 11: Performance of various LLMs for each ability dimension about task formats.
Refer to caption: x11.png

**Table 14: Detailed scores of models on fine-grained subfields.**
| Content | Llama2-7B | Llama2-13B | Llama2-70B | Mixtral-8$\times$7B | Llama3-8B | Llama3-70B | GPT-3.5 | GPT-4 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Data Structure and Algorithm |  |  |  |  |  |  |  |  |
| Overview | 56.67 | 51.11 | 59.44 | 68.06 | 73.33 | 68.06 | 71.11 | 74.17 |
| Linear List | 34.48 | 44.83 | 53.45 | 58.62 | 53.45 | 65.52 | 55.17 | 67.24 |
| Stack, Queue, and Array | 49.61 | 50.91 | 57.40 | 57.66 | 58.96 | 71.95 | 61.43 | 76.49 |
| String | 76.67 | 66.67 | 76.67 | 66.67 | 70.00 | 80.00 | 80.00 | 70.00 |
| Tree | 32.78 | 36.33 | 45.78 | 47.89 | 35.67 | 57.11 | 40.33 | 60.56 |
| Graph | 43.80 | 37.47 | 45.44 | 65.70 | 54.56 | 68.23 | 56.96 | 68.61 |
| Searching | 51.29 | 52.00 | 61.14 | 60.57 | 54.86 | 56.71 | 58.14 | 74.86 |
| Sorting | 30.52 | 37.27 | 56.10 | 52.08 | 54.55 | 71.56 | 62.21 | 74.68 |
| Average | 46.98 | 47.07 | 56.93 | 59.66 | 56.92 | 67.39 | 60.67 | 70.83 |
| Computer Organization |  |  |  |  |  |  |  |  |
| Overview | 51.20 | 61.40 | 61.60 | 76.40 | 68.20 | 80.20 | 73.20 | 81.80 |
| Data Representation and Operation | 27.95 | 38.72 | 38.46 | 50.51 | 39.74 | 50.38 | 45.64 | 57.44 |
| Storage System | 41.80 | 46.10 | 58.00 | 61.70 | 53.60 | 68.10 | 56.20 | 68.50 |
| Instruction System | 51.76 | 53.68 | 57.79 | 59.56 | 53.82 | 75.74 | 65.29 | 80.44 |
| Central Processing Unit | 41.93 | 42.66 | 53.67 | 54.50 | 51.65 | 62.75 | 51.74 | 74.86 |
| Bus | 60.70 | 59.12 | 61.40 | 66.32 | 47.37 | 71.75 | 66.49 | 73.33 |
| Input/Output System | 37.58 | 35.48 | 29.19 | 52.42 | 44.03 | 52.42 | 35.48 | 58.23 |
| Average | 44.70 | 48.17 | 51.44 | 60.20 | 51.20 | 65.91 | 56.29 | 70.66 |
| Computer Network |  |  |  |  |  |  |  |  |
| Overview and Architecture | 52.15 | 48.31 | 58.77 | 62.77 | 58.15 | 68.62 | 57.23 | 69.08 |
| Physical Layer | 42.11 | 47.61 | 52.25 | 57.89 | 53.52 | 65.77 | 54.51 | 69.01 |
| Data Link Layer | 32.35 | 41.06 | 42.35 | 57.12 | 50.61 | 59.62 | 60.23 | 63.94 |
| Network Layer | 38.40 | 48.78 | 58.47 | 62.37 | 65.19 | 75.57 | 62.98 | 77.48 |
| Transport Layer | 42.95 | 48.72 | 66.28 | 70.77 | 63.46 | 81.79 | 61.54 | 86.79 |
| Application Layer | 47.61 | 55.00 | 60.34 | 65.91 | 63.30 | 75.34 | 64.55 | 79.89 |
| Average | 42.60 | 48.25 | 56.41 | 62.81 | 59.04 | 71.12 | 60.17 | 74.37 |
| Operating System |  |  |  |  |  |  |  |  |
| Overview | 39.74 | 40.65 | 48.57 | 65.32 | 60.65 | 69.87 | 51.82 | 68.31 |
| Processes and Threads | 34.14 | 42.61 | 43.57 | 55.73 | 50.83 | 63.57 | 47.58 | 66.82 |
| Memory Management | 31.63 | 42.04 | 52.04 | 51.02 | 53.67 | 60.71 | 51.02 | 70.41 |
| File Management | 40.00 | 49.34 | 57.37 | 54.87 | 55.66 | 61.97 | 56.32 | 64.08 |
| Input/Output Management | 34.88 | 36.83 | 41.46 | 50.98 | 47.07 | 51.10 | 38.05 | 59.76 |
| Average | 36.08 | 42.29 | 48.60 | 55.58 | 53.58 | 61.44 | 48.96 | 65.88 |
| Overall | 41.08 | 45.54 | 52.52 | 58.90 | 54.37 | 66.08 | 55.91 | 70.41 |

Figure: Figure 12: Performance of various LLMs for each ability dimension about CS domains.
Refer to caption: x12.png

#### Detailed Performance on Each Subfield.

In Figure 12, we visualize the models’ knowledge and reasoning performance across the four domains of CS-Bench. Subsequently, we focus on the models’ performance in 26 fine-grained subfields. Table 14 presents the results of eight representative models.

Firstly, we can observe significant variations in scores across different subfields within the same domains for the models. Taking the DSA domain as an example, Llama2-70B scores range from 45.44% to 76.67% across different chapters (average 56.93%), while GPT-3.5 scores range from 55.17% to 80.00% (average 60.67%).
Secondly, the performance of different models in the same subfield is generally consistent compared to the average scores. For instance, all models perform above the average scores in the “Overview” and “Stack, Queue, and Array” subfields of DSA but below average in the “Tree” and “Graph” subfields. These detailed scores allow us to understand which content poses greater challenges for the models and provides guidance for improving the models’ performance in computer science by strengthening these weaker subfields.

We further observe that although the overall scores of models from the same family increase with scale, not all chapters follow this pattern.
As shown in Figure 13, the Llama2 series exhibits a trend of scores increasing with scale in most subfields (17 out of 26 subfields); however, there are some exceptions. For instance, Llama2-7B performs exceptionally well in the “string” chapter of DSA, while Llama2-13B excels in the “Data Representation and Operation” chapter of CO, surpassing the performance of Llama2-70B.

Figure: Figure 13: The performance of the Llama2 series models in each subfield.
Refer to caption: x13.png

### E.2 Scale-Score Fitting Function for CS-Bench

Figure: Figure 14: The logarithmic scale-score performance and scale-score fitting curve of Qwen1.5 and Llama2 series.
Refer to caption: x14.png

To enhance CS performance, large-scale models are often utilized; however, these models demand more computational resources for both training and deployment inference. Therefore, it is desirable to establish a relationship between model scale and CS performance, enabling the prediction of theoretically larger models’ scores on CS-Bench based on the performance of smaller-scale models. The established fitting function should adhere to the following criteria:

1. The score should monotonically increase with the increase in model scale, approaching 0 as the scale approaches 0, and approaching 1 (100%) as the scale approaches infinity.

2. As illustrated in Figure 14 (a), when the model scale varies exponentially, the score should exhibit an approximately linear trend.

3. Due to variations in performance and change slopes among different model families at the same scale, the fitting function needs to incorporate model-family-specific hyperparameters.

Guided by these criteria, we experiment with various functions and find the following function to satisfy the conditions and work best:

$$ $\text{Score}=1-\frac{1}{\theta_{1}log_{10}(\theta_{2}\cdot\text{Scale}+1)+1}$ (1) $$

Where $\theta_{1}$ and $\theta_{2}$ are hyperparameters specific to the model family.
To validate the effectiveness of the function, we estimate hyperparameters based on the minimum mean square error on small-scale models and predict performance scores on larger-scale models. For the Qwen1.5 family, we use models of 7, 14, 32, and 72B to predict the 110B model’s performance. For the Llama2 series, we predict the 70B model’s performance based on 7B and 13B. As depicted in Figure 14 (b), for Qwen1.5 110B, the predicted score (67.83%) closely matches the true value (67.95%). For Llama2-70B, with only two reference data points, the predicted score (55.08%) deviates from the true value (52.52%) by only 2.56%.

**Table 15: Zero-shot scores (%) of LLMs across domains on CS-Bench (CN), where “Klg” represents knowledge-type, “Rng” represents reasoning-type, and “Avg” represents Average.**
| Model | Data Struc & Algo | Computer Organization | Computer Network | Operating System | Overall |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg |  |
| Open-source LLM (Scale < 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Random | 28.04 | 24.63 | 26.65 | 26.57 | 25.24 | 26.13 | 26.34 | 22.49 | 24.98 | 29.06 | 24.23 | 27.27 | 27.4 | 24.12 | 26.20 |
| ChatGLM3-6B | 41.74 | 32.48 | 37.97 | 44.07 | 34.91 | 41.05 | 49.02 | 32.31 | 43.14 | 43.02 | 32.86 | 35.98 | 44.67 | 33.09 | 40.45 |
| Baichuan2-7B | 42.04 | 31.51 | 37.75 | 44.93 | 37.88 | 42.61 | 50.74 | 31.11 | 43.83 | 42.18 | 34.07 | 39.16 | 45.27 | 33.47 | 40.97 |
| InternLm2-7B | 41.97 | 34.54 | 38.95 | 55.77 | 38.67 | 50.13 | 60.05 | 41.86 | 53.65 | 50.94 | 44.07 | 48.39 | 52.71 | 39.61 | 47.94 |
| Qwen1.5-7B | 49.13 | 37.71 | 44.48 | 60.86 | 44.48 | 55.46 | 60.90 | 45.68 | 55.54 | 58.38 | 48.24 | 54.61 | 57.62 | 43.79 | 52.59 |
| Llama3-8B | 50.47 | 29.68 | 42.01 | 50.81 | 36.30 | 46.03 | 56.09 | 42.21 | 51.21 | 52.01 | 38.85 | 47.12 | 52.46 | 36.61 | 46.69 |
| Llama3-8B-Chinese | 49.20 | 33.72 | 42.90 | 54.99 | 33.09 | 47.77 | 58.77 | 48.59 | 55.19 | 55.58 | 41.10 | 50.20 | 54.84 | 39.17 | 49.13 |
| Open-source LLM (Scale > 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Baichuan2-13B | 48.83 | 34.68 | 43.07 | 54.18 | 36.00 | 48.18 | 55.11 | 39.85 | 49.74 | 49.19 | 40.27 | 45.88 | 52.10 | 37.63 | 46.83 |
| Qwen1.5-14B | 51.47 | 48.81 | 50.39 | 64.43 | 46.85 | 58.63 | 68.69 | 55.18 | 63.94 | 69.58 | 56.59 | 64.76 | 63.78 | 51.81 | 59.42 |
| InternLm2-20B | 51.97 | 38.03 | 46.30 | 58.36 | 45.76 | 54.20 | 60.60 | 50.50 | 57.05 | 58.70 | 45.66 | 53.86 | 57.59 | 44.85 | 52.95 |
| Qwen1.5-32B | 55.89 | 56.70 | 56.22 | 67.74 | 60.00 | 65.19 | 70.33 | 66.83 | 69.10 | 72.40 | 62.03 | 68.55 | 66.77 | 61.35 | 64.80 |
| Llama3-70B | 53.28 | 55.41 | 54.15 | 67.97 | 49.58 | 61.91 | 71.07 | 61.81 | 67.81 | 65.29 | 57.36 | 62.35 | 64.86 | 56.18 | 61.70 |
| Qwen1.5-72B | 58.16 | 52.02 | 55.66 | 70.28 | 52.91 | 64.55 | 75.25 | 66.23 | 72.08 | 74.12 | 63.19 | 70.06 | 69.73 | 58.52 | 65.64 |
| Closed-source LLM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-3 | 54.15 | 39.63 | 48.24 | 60.86 | 43.27 | 55.06 | 64.29 | 48.89 | 58.87 | 56.36 | 39.84 | 50.22 | 59.27 | 42.96 | 53.33 |
| GPT-4 | 60.03 | 60.28 | 60.13 | 77.60 | 60.24 | 71.88 | 73.50 | 72.86 | 73.27 | 71.46 | 65.60 | 69.29 | 71.06 | 64.80 | 68.78 |
| GPT-4o | 61.67 | 66.45 | 63.62 | 78.86 | 55.32 | 71.10 | 78.61 | 74.17 | 77.05 | 72.66 | 69.94 | 71.67 | 73.46 | 66.69 | 71.00 |
| GLM-4 | 58.12 | 58.37 | 58.22 | 74.03 | 59.49 | 69.24 | 71.65 | 70.21 | 71.14 | 73.31 | 67.14 | 71.06 | 69.55 | 63.75 | 67.44 |
| ERNIE-3.5 | 58.16 | 55.62 | 57.13 | 74.56 | 58.73 | 69.34 | 74.68 | 65.16 | 71.33 | 72.13 | 63.37 | 68.94 | 70.28 | 60.63 | 66.77 |
| ERNIE-4 | 57.92 | 62.33 | 59.72 | 78.24 | 64.18 | 73.60 | 76.27 | 69.74 | 73.97 | 75.84 | 69.54 | 73.54 | 72.49 | 66.36 | 70.26 |

**Table 16: Zero-shot scores (%) of LLMs across task formats on CS-Bench (CN).**
| Model | Multiple-choice | Assertion | Fill-in-the-blank | Open-ended | Overall |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg | Klg | Rng | Avg |  |
| Random | 25.00 | 25.00 | 25.00 | 50.00 | 50.00 | 50.00 | 0.00 | 0.00 | 0.00 | 10.00 | 10.00 | 10.00 | 27.4 | 24.12 | 26.20 |
| Open-source LLM (Scale < 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| ChatGLM3-6B | 45.21 | 34.07 | 40.90 | 54.41 | 48.48 | 53.05 | 23.53 | 11.32 | 20.57 | 43.80 | 25.22 | 30.68 | 44.67 | 33.09 | 40.45 |
| Baichuan2-7B | 44.96 | 32.26 | 40.05 | 53.80 | 56.57 | 54.43 | 29.41 | 13.21 | 25.48 | 47.20 | 27.52 | 33.31 | 45.27 | 33.47 | 40.97 |
| InternLm2-7B | 51.09 | 40.08 | 46.83 | 59.88 | 55.56 | 58.89 | 44.12 | 18.87 | 38.00 | 60.80 | 33.27 | 41.37 | 52.71 | 39.61 | 47.94 |
| Qwen1.5-7B | 59.64 | 48.50 | 55.33 | 60.79 | 50.51 | 58.44 | 42.35 | 15.09 | 35.74 | 58.20 | 30.35 | 38.54 | 57.62 | 43.79 | 52.59 |
| Llama3-8B | 53.26 | 35.67 | 46.45 | 56.23 | 59.60 | 57.00 | 42.35 | 16.98 | 36.20 | 49.60 | 29.47 | 35.39 | 52.46 | 36.61 | 46.69 |
| Llama3-8B-Chinese | 55.43 | 40.08 | 49.49 | 59.57 | 56.57 | 58.88 | 42.94 | 16.98 | 36.64 | 55.60 | 30.62 | 37.97 | 54.84 | 39.17 | 49.13 |
| Open-source LLM (Scale > 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Baichuan2-13B | 52.11 | 39.48 | 47.22 | 59.57 | 51.52 | 57.73 | 40.00 | 16.98 | 34.42 | 43.40 | 27.08 | 31.88 | 52.10 | 37.63 | 46.83 |
| Qwen1.5-14B | 67.82 | 57.72 | 63.91 | 65.05 | 56.57 | 63.11 | 43.53 | 24.53 | 38.92 | 63.80 | 34.96 | 43.44 | 63.78 | 51.81 | 59.42 |
| InternLm2-20B | 58.49 | 46.89 | 54.00 | 59.57 | 54.55 | 58.42 | 47.06 | 26.42 | 42.05 | 67.00 | 35.40 | 44.69 | 57.59 | 44.85 | 52.95 |
| Qwen1.5-32B | 71.26 | 68.74 | 70.28 | 64.74 | 63.64 | 64.49 | 51.76 | 28.30 | 46.07 | 63.40 | 42.04 | 48.32 | 66.77 | 61.35 | 64.80 |
| Llama3-70B | 66.03 | 60.32 | 63.82 | 66.57 | 65.66 | 66.36 | 58.24 | 33.96 | 52.35 | 59.00 | 40.71 | 46.09 | 64.86 | 56.18 | 61.70 |
| Qwen1.5-72B | 72.41 | 67.74 | 70.60 | 72.34 | 55.56 | 68.51 | 54.71 | 28.30 | 48.30 | 63.80 | 34.96 | 43.44 | 69.73 | 58.52 | 65.64 |
| Closed-source LLM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-3 | 57.98 | 42.48 | 51.98 | 65.05 | 61.62 | 64.27 | 54.71 | 24.53 | 47.39 | 56.60 | 36.81 | 42.63 | 59.27 | 42.96 | 53.33 |
| GPT-4 | 73.31 | 67.13 | 70.92 | 72.04 | 67.68 | 71.04 | 62.35 | 60.38 | 61.87 | 60.40 | 54.16 | 56.00 | 71.06 | 64.80 | 68.78 |
| GPT-4o | 75.92 | 69.33 | 73.37 | 73.86 | 68.69 | 72.68 | 62.94 | 50.94 | 60.03 | 70.20 | 62.92 | 65.06 | 73.46 | 66.69 | 71.00 |
| GLM-4 | 73.68 | 69.76 | 72.16 | 68.09 | 57.58 | 65.69 | 55.03 | 47.17 | 53.12 | 68.00 | 52.92 | 57.36 | 69.55 | 63.75 | 67.44 |
| ERNIE-3.5 | 72.24 | 63.71 | 68.94 | 69.30 | 61.62 | 67.55 | 63.91 | 50.94 | 60.76 | 70.40 | 51.95 | 57.38 | 70.28 | 60.63 | 66.77 |
| ERNIE-4 | 73.55 | 70.35 | 72.31 | 72.34 | 56.57 | 68.74 | 70.00 | 67.92 | 69.50 | 68.40 | 58.32 | 61.28 | 72.49 | 66.36 | 70.26 |

### E.3 Performance of Models on CS-Bench (Chinese)

We assess models that support Chinese on CS-Bench (CN). The foundation models include the LLama3 and GPT-4 series, which are not specifically optimized for Chinese, as well as Chinese-oriented open-source models, including ChatGLM, Baichuan2, InternLm2, Qwen1.5 and llama3-chinese series. We also evaluate Chinese-oriented closed-source models, including GLM-4 and ERNIE-3.5/4. Details of these models are provided in Appendix D.4.

As shown in Table 15 and Table 16, the scores of these models on CS-Bench(CN) range from 40.45% to 70.26%. Despite not being specifically optimized for Chinese, GPT-4o still achieves the best performance. Among the Chinese-oriented models, ERNIE-4 outperforms GPT-4, achieving performance close to GPT-4o. Additionally, ERNIE-3.5 and GLM-4 score similarly, slightly lower than GPT-4’s performance in Chinese. Notably, Llama3-8B-chinese surpasses Llama3-8B by 2.44%, highlighting the importance of adapting models to specific languages.
We further compare the performance of the models on CS-Bench(EN) and CS-Bench(CN) in Figure 15. Compared to English, the GPT and Llama3 series, which are not optimized for Chinese, perform worse on Chinese context. For instance, Llama3-8B experiences a decrease of 7.68% on Chinese, and Llama3-70B drops by 4.38%. Although some Chinese-oriented models also show slight decreases in performance in the Chinese context, such as InterLm2-20B, the decline is much less significant than that of the Llama3 series. Moreover, the Qwen1.5 series even demonstrates improved performance on Chinese tasks. Finally, we observe that larger models within the same family are less affected by different languages, as reflected in Baichuan2-7/13B, Internlm2-7/20B, and Llama3-8/70B.

Figure: Figure 15: Comparison of models in different languages on CS-Bench.
Refer to caption: x15.png

### E.4 Case Study of Error Types

We first introduce the error types of knowledge-type questions and reasoning-type questions in Table 17 and Table 18. To facilitate a better understanding of each error type, we provide examples of each error type made by GPT-3.5 in knowledge-based and reasoning-based questions in Table19 and 20, respectively. Additionally, Table 21 presents several examples that contain multiple error types.

**Table 17: Error types for knowledge-type questions.**
| Error Type | Explanation |
| --- | --- |
| Complete<br>conceptual error | For a certain piece of knowledge or concept, the model is completely unaware of it<br>or has misunderstood all of it. |
| Partial<br>conceptual error | For a certain piece of knowledge or concept, the model has only grasped part of it<br>or has misunderstood part of its content. |
| Conceptual<br>confusion | For multiple pieces of knowledge or concepts, the model has incorrectly matched<br>their names with their attributes. |

**Table 18: Error types for reasoning-type questions.**
| Error Type | Explanation |
| --- | --- |
| Concept-<br>related error | The model misunderstands certain pieces of knowledge, causing incorrect<br>answers, This includes the three types of knowledge-based errors mentioned above. |
| Logical<br>reasoning error | The model made logical reasoning errors when answering computer science-related<br>questions. |
| Mathematical<br>reasoning error | During the response process, errors in numerical calculations, symbolic calculations,<br>and other computational processes caused the model to produce incorrect answers. |

**Table 19: Example of knowledge-type question error types. We use red to mark errors in the output and blue to indicate the reasons for the errors.**
| Error Type | Example |
| --- | --- |
| Complete<br>conceptual<br>error | Question:<br>When two sequential stacks share the same one-dimensional array space, how are their stack bottoms and stack tops set?"<br>A: Two stacks with the same bottom end, extending in opposite directions from the top.<br>B: Two stacks with tops at the same end and bottoms extending in opposite directions.<br>C: The bottom and top of the stack are at the same end.<br>D: Two stacks with bottoms at opposite ends and tops extending towards the middle.<br>Answer:<br>D<br>Wrong Output:<br>When two sequential stacks share the same one-dimensional array space, their stack bottoms and stack tops are set as follows: Two stacks with tops at the same end and bott-<br>oms extending in opposite directions. This means that the top of one stack and the bottom of the other stack are at the same end of the array, while their bottoms extend in<br>opposite directions. This allows for efficient utilization of the array space for both<br>stacks. The correct answer is (B). Answer: (B) Two stacks with tops at the same end<br>and bottoms extending in opposite directions.<br>Error Details:<br>In GPT-3.5’s response, it was mentioned that the top of two stacks are at the same end,<br>while the bottom of the stack extends in opposite directions. This reflects that GPT-3.5<br>is completely not familiar with the implementation of shared stacks. |
| Partial<br>conceptual<br>error | Question:<br>What are the main functions of the network layer?<br>A: Data Transparent Transmission<br>B: Power Management<br>C: Signal Amplification<br>D: Data Compression<br>Answer:<br>A<br>Wrong Output:<br>The main functions of the network layer are data transparent transmission and data<br>compression. The network layer is responsible for routing and forwarding data packets between different networks, ensuring that data is delivered to the correct<br>destination. It does not handle power management or signal amplification. The correct answers are A and D.<br>Error Details:<br>GPT-3.5 correctly answered that transparent data transmission is a function of the network<br>layer, but at the same time incorrectly pointed out that data compression is the main<br>function of the network layer, witch indicating an incorrect understanding of<br>certain aspects of network layer functionality. |
| Conceptual<br>confusion | Question:<br>What are the basic steps of simple selection sort?<br>A: Select the element with the smallest key from the elements to be sorted<br>and swap it with the first element.<br>B: Insert elements into the sorted sequence in order.<br>C: Compare adjacent elements in sequence and swap them.<br>D: Use the divide and conquer method to sort elements.<br>Answer:<br>D<br>Wrong Output:<br>The basic steps of simple selection sort are: (A) Select the element with the<br>smallest key from the elements to be sorted and swap it with the first element.<br>(C) Compare adjacent elements in sequence and swap them. The correct answer is A<br>and C.<br>Error Details:<br>The answers output by GPT-3.5 are A and C, and it is claimed that the basic steps of<br>simple selection sorting are to select the smallest element and exchange it with the<br>first element, and to compare adjacent elements and exchange them. This is obviously<br>incorrect because comparing adjacent elements and swapping them is a Bubble Sort This indicates that GPT confuses bubble sorting and selection sorting. |

**Table 20: Example of reasoning-type question error types.**
| Error Type | Example |
| --- | --- |
| Concept-<br>related<br>error | Question:<br>In the topological ordering of the directed graph G, if vertex Vi precedes vertex<br>Vj, then the following situation cannot occur: ()<br>A: G contains an arc <Vi, Vj>.<br>B: There is a path from Vi to Vj in G.<br>C: There is no arc <Vi, Vj> in G.<br>D: There is a path from Vj to Vi in G.<br>Answer:<br>D<br>Wrong Output:<br>If vertex Vi precedes vertex Vj in the topological ordering of the directed graph G,<br>then it means that there is a directed path from Vi to Vj in G. Therefore, the si-<br>tuation where there is no arc <Vi, Vj> in G cannot occur. The correct answer is (C)<br>There is no arc <Vi, Vj> in G.<br>Error Details:<br>The sequence obtained by topological sorting has i before j. GPT-3.5 chose C in the<br>answer, but according to the topological sorting rule, placing i before j does not<br>necessarily indicate that two nodes can be directly connected. GPT-3.5 does not under-<br>stand the meaning of topological sorting. |
| Logical<br>reasoning<br>error | Question:<br>In the array representation of a linear list with n elements, the operations with a<br>time complexity of O(1) are () I. Accessing the ith ($1\leq i\leq n$) node and finding<br>the immediate predecessor of the ith ($2\leq i\leq n$) node. II. Inserting a new node<br>after the last node. III. Deleting the first node. IV. Sequentially outputting the values of these n elements.<br>A:I<br>B: II, III<br>C: I, II<br>D: I, II, III<br>Answer:<br>C<br>Wrong Output:<br>The operations with a time complexity of O(1) in the array representation of a linear<br>list with n elements are: I. Accessing the ith ($1\leq i\leq n$) node and finding the immediate predecessor of the ith($2\leq i\leq n$) node. II. Inserting a new node<br>after the last node. III. Deleting the first node. Therefore, the correct answer is (D) I, II, III. Answer: D<br>Error Details:<br>For item II in the question, after deleting the first element of the array, the next<br>n-1 elements need to be moved forward, resulting in a time complexity of o (n).<br>However, GPT-3.5 considers it to be o(1), which is a GPT-3.5 inference error. |
| Mathematical<br>reasoning<br>error | Question:<br>In a computer with an 8-bit adder, the machine numbers of signed integers X and y are represented in two’s complement, with [x] complement = F5H and<br>[y] complement = 7EH. If the calculation X - y is performed in this adder,the<br>carry-in information for the lower bit and the overflow flag OF after the operation are respectively ().<br>A: 1,1 B: 1,0 C: 0,1 D: 0,0<br>Answer: C<br>Wrong Output:<br>First, let’s find the two’s complement of y. The two’s complement of a number<br>is obtained by inverting all the bits and adding 1. [y] complement = 7EH<br>Inverting all the bits: 10000001. Adding 1: 10000010.Now, we can perform the<br>subtraction X - y using the 8-bit adder. [X] complement = F5H [y] complement =<br>10000010. Adding X and the two’s complement of y: 11110101 + 10000010 = 101101111.<br>The result is 10110111, which is the two’s complement representation<br>of the result of X - y. To determine the carry-in information for the lower bit, we<br>need to look at the carry-out from the lower bit. In this case, the carry-out is 1,<br>so the carry-in for the lower bit is also 1. To determine the overflow flag (OF), we<br>need to compare the carry-in and carry-out from the sign bit. In this case, the<br>carry-in and carry-out from the sign bit are both 0, so the overflow flag is 0.<br>Error Details:<br>The complement of x and (- y) is calculated incorrectly. |

**Table 21: Examples containing multiple types of errors.**
| Error Type | Example |
| --- | --- |
| Logical<br>reasoning error<br>&<br>Mathematical<br>reasoning error | Question:<br>Given that the system uses a 32-bit physical address and a 48-bit virtual address, with<br>a page size of 4KB and a page table entry size of 8B. Assuming the system employs pure<br>paging, it will use a ()-level page table, and the page offset is () bits.<br>A: 3, 12<br>B: 3, 14<br>C: 3, 12<br>D: 4, 14<br>Answer:<br>C<br>Wrong Output:<br>To determine the number of levels in the page table, we need to calculate the number of<br>bits required to represent the virtual address space. The virtual address space is 48<br>bits, and the page size is 4KB, which is equivalent to $2^{12}$ bytes. Therefore, the num-<br>ber of pages in the virtual address space is $2^{48}$ / $2^{12}$ = $2^{36}$ pages. Since each<br>page table entry is 8 bytes, we can calculate the number of entries in each level of the<br>page table. At the first level, there are $2^{36}$ entries, which can be represented by 36 bits. At the second level, each entry points to a page table at the next level, so there are $2^{36}$ entries. At the third level, each entry points to a page table at the next level,<br>so there are $2^{36}$ entries. Therefore, we need 3 levels of page tables. The page offset<br>is the number of bits required to represent the offset within a page. Since the page<br>size is 4KB, which is equivalent to $2^{12}$ bytes, the page offset is 12 bits. Therefore,<br>the correct answer is (A) 3, 12.<br>Error Details:<br>The number of page table items that can be accommodated on each page is 4KB/8B<br>= 512=$2^{9}$,<br>while GPT-3.5 did not mention calculations and directly stated that the first<br>level page table has $2^{36}$ items; According to the GPT-3.5 calculation, if there are $2^{36}$<br>items in the first level page table, there should be a 36/36=first level page table<br>instead of a third level page table, resulting in a mathematical inference error.<br>From the process of calculating the number of page table levels in GPT-3.5, it can be<br>seen that its logical reasoning path is not correct. |
| Complete<br>conceptual error<br>&<br>Conceptual<br>confusion | Question:<br>The following network connecting devices that operate at the data link layer are ()<br>A: Repeaters and Hubs<br>B: Hubs and Bridges<br>C: Network bridges and local area network switches<br>D: Hubs and Local Area Network (LAN) Switches<br>Answer:<br>C<br>Wrong Output:<br>The correct answer is (D) Hubs and Local Area Network (LAN) Switches. Repeaters<br>operate at the physical layer, while hubs and bridges operate at the data link layer.<br>Local Area Network (LAN) switches also operate at the data link layer. Therefore,<br>the correct combination of network connecting devices that operate at the data link<br>layer is hubs and LAN switches.<br>Error Details:<br>GPT-3.5 is unaware that the bridge is a network link device operating at the data link<br>layer, indicating a lack of knowledge about the bridge. Meanwhile, GPT-3.5 believes that<br>the Hub is a device at the data link layer, indicating that it confuses<br>physical layer devices with data link layer devices. |

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: What does your dataset not include for testing compettency in CS?
What level does this test? The level is notably absent from the paper -- but staging this relative to expertise (high school, undergrad, grad, etc) is extremely useful. 
What is the breakdown of problems in the benchmark to expertise?
soundness: 3
strengths: The strength of the paper is the dataset . This type of dataset is extremely valuable and I especially appreciate the sourcing of exam questions from universities and the knowledge transfer from blogs across specialized fields with such a large corpus of questions.  The final version of the paper should explain exactly where the questions came from, including giving the class and level at the university so that the difficult level is clear. 

I'd also call out the thoroughness of the analysis. The different correlations noted, eg math and CS ability and so forth, are interesting as is the relationship between performance and tokens across scales. None of these are unexpected but they are nice things to see.
summary: Develops CS Bench, a benchmark dataset of CS expertise that tests the ability of LLMs in computer science, as well as the relationship between CS math and code programming ability.  The benchmark measures 4 languages and 26 sufields with 9676 samples. To test this they evaluated using 30 LLMs.

The datasets were constructed using professional exams and practice tests widely available on the internet; extraction from blog articles of questions and answers; teaching materials and exams from the authors institutions.  For scoring, the authors use autoraters (GPT4) for genration tasks and  for comprehension tasks use questions where they can score with a regex
weaknesses: The first sentence "Computer Science (CS) stands as a testament to the intricacies of human intelligence, profoundly advancing the development of artificial intelligence and modern society"  is a bit much and some of the corresponding language needs to be toned down. Could we please not philosphize about the importance of specific fields (especially those were we are practitioners) and instead focus on the task that the paper accomplishes -- building a novel eval set and using it.

Other than this, the generative questions that require autorating to score are less convincing than the regex. The authors are right to emphasize that people should focus on this dataset if they can, but it does make me wonder whether the generative questions could be downgraded even further.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: Q1. What would the key contributions of the benchmark be to the field at large, as it stands today?

Q2. Can the authors evaluate the value of the data through one of several Value-of-Data approaches including perplexity of training on the benchmark, FT-ing on the benchmark or other approaches?
soundness: 3
strengths: 1. Commendable that 70% of the benchmark is from private source not already on the intrernet.

2. The paper has a somewhat thorough review of several LLMs on where they stand in the benchmark.
summary: The paper introduces a benchmark of 10k test samples that pertain to Computer Science specific knowledge.
weaknesses: 1. It is not clear what the benchmark aims to achieve or improve. LLMs are already good at MCQ, open-ended, FITB and T/F questions. There is no coding in the benchmark. The authors need to discuss more clearly why this is pivotal and important to the field.

2. Scoring on the benchmark appears to be ad-hoc with regular expression matching, instead of measuring with existing techniques.

3. The benchmark lacks a fingerprint to prevent training. THis would be required to protect the integrity of the benchmark.

4. The paper indicates that this may help evaluate or improve reasoning though it is unclear how?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Reducing Reliance on GPT-4 for Evaluation: The dependence on GPT-4 for scoring generative tasks raises validity and reproducibility concerns. While you explored alternative scoring models and human evaluation, these were limited. Could you elaborate on the feasibility of using alternative automatic metrics like BLEU, ROUGE, or METEOR, perhaps in conjunction with domain-specific constraints or regular expressions? Could a smaller, more accessible model be fine-tuned specifically for evaluating CS-related generative tasks? This would improve reproducibility and lower the barrier for other researchers to use CS-Bench. Also, provide more details on the human evaluation: the number of annotators, their expertise level, and the specific guidelines they followed. Calculating inter-annotator agreement (e.g., using Cohen's kappa) would strengthen the validation of both human and GPT-4 scoring.

Deepening the Analysis of Cross-Capability Relationships: The observed correlations between CS, math, and coding performance are interesting, but the analysis lacks depth. Could you investigate why these correlations exist? Are certain model architectures better suited for integrating these skills? Does the training data composition play a significant role? Analyzing the internal representations of CS concepts in general vs. expert LLMs could provide valuable insights. Consider using techniques like probing tasks or representational similarity analysis to understand how different models learn and represent CS knowledge. This would move beyond simple correlations to provide a more mechanistic understanding of cross-capability transfer.

Generating Actionable Insights for Improvement: The error analysis identifies knowledge gaps and reasoning challenges, but it doesn't offer concrete solutions. How can we leverage these insights to improve LLM performance on CS tasks? Could you explore targeted fine-tuning on CS-specific datasets, curriculum learning strategies, or methods for incorporating external knowledge sources (e.g., knowledge graphs, symbolic reasoning systems)? Providing concrete examples of how these techniques could be applied would significantly enhance the paper's practical value.

Addressing Benchmark Biases: What are the potential biases introduced by the composition of CS-Bench (e.g., university-level focus, source selection)? How might these biases affect the evaluation of different LLMs? Consider expanding the benchmark to include tasks from different levels of expertise (e.g., introductory CS, advanced topics, real-world applications) and diversifying the sources to mitigate potential biases. Discuss the limitations of CS-Bench in representing the full spectrum of CS knowledge and skills. A critical self-reflection on the benchmark's limitations would strengthen the paper.

Improving Visualization and Clarity: Some figures, particularly the radar charts, are difficult to interpret. Consider using alternative visualizations (e.g., bar charts, scatter plots) that more clearly communicate the key findings. Streamline the writing to improve conciseness and clarity. Focus on conveying the key messages effectively and avoid unnecessary jargon. Ensure that the figures are clearly labeled and explained in the captions. A more polished presentation would enhance the paper's impact and accessibility.
soundness: 3
strengths: Focus on Reasoning: The inclusion of both knowledge-based and reasoning-based questions is a good step towards a more comprehensive evaluation of LLMs' understanding of CS concepts.

Extensive Evaluation: Testing numerous LLMs, both open-source and closed-source, across various scales provides a good overview of current capabilities and highlights areas where models struggle. The scale-score analysis is also a useful contribution.

Comprehensive Benchmark: The breadth of CS topics covered, along with the multilingual aspect and diverse task formats (multiple-choice, assertion, fill-in-the-blank, open-ended), makes this a valuable contribution. Creating such a benchmark is a substantial undertaking.

Error Analysis: The analysis of error types provides some insights into the reasons for model failures and points towards potential directions for improvement.
summary: This paper introduces CS-Bench, a comprehensive, multilingual benchmark designed to evaluate the computer science knowledge and reasoning abilities of Large Language Models (LLMs). Comprising approximately 10,000 meticulously curated samples across 26 subfields within four key CS areas (Data Structures and Algorithms, Computer Organization, Computer Networks, and Operating Systems), CS-Bench supports evaluation in English, Chinese, French, and German, and includes diverse task formats such as multiple-choice, assertion, fill-in-the-blank, and open-ended questions. The authors conduct an extensive evaluation of over 30 mainstream LLMs on CS-Bench, analyzing their performance relative to model scale and investigating the reasons for model failures. They further explore the relationship between LLM capabilities in computer science, mathematics, and coding, observing strong correlations and evaluating the performance of specialized expert LLMs on the benchmark. The findings suggest a logarithmic growth relationship between model scale and CS performance, highlight the challenge of CS-specific reasoning for LLMs, and reveal the potential for expert LLMs to improve performance in certain CS subfields. CS-Bench aims to serve as a valuable resource for assessing and improving the diverse reasoning capabilities of LLMs in the context of computer science.
weaknesses: Writing and Presentation Could Be Improved: The writing is occasionally dense and could be more concise and clear. Some figures, especially the radar charts, are difficult to interpret and don't effectively communicate the key findings. A more visual and intuitive presentation of the results would enhance readability and impact.

Limited Insight into Cross-Capability Relationships: While the paper shows correlations between CS, math, and coding abilities, it doesn't delve deeply into why these correlations exist. More in-depth analysis is needed to understand the underlying factors driving these relationships. Do specific architectural choices or training data characteristics explain the observed correlations? The analysis of expert LLMs is a step in the right direction, but it lacks depth. How do the internal representations of CS concepts differ between general and expert LLMs?

Lack of Actionable Insights for Improvement: The error analysis, while pointing towards knowledge gaps and reasoning challenges, doesn't offer concrete suggestions for how to address these limitations. Beyond general recommendations like "knowledge supplementation" and "CS-specific reasoning," what specific techniques or strategies could be employed to improve LLM performance on CS tasks? Exploring methods like curriculum learning, incorporating external knowledge sources, or fine-tuning on specific CS datasets could provide more actionable insights.

Limited Discussion of Benchmark Biases: All benchmarks have biases. What are the potential biases in CS-Bench, and how might these biases influence the results? For example, the reliance on university-level content might not fully reflect the diversity of CS tasks in real-world applications. Addressing this limitation by including tasks from different levels of expertise or from specific industry applications would strengthen the benchmark's generalizability.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. What do the authors suggest to avoid the issue of contamination? 
2. Have the authors considered using answer ordering permutation, number switching, orquestion negations to reduce the risk of contamination? By number switching, I mean dynamically changing the numbers in questions so that the answers are different but the fundamental relationships continue to hold. By question negations, I mean asking the opposite question by including a negative particle, such that the question has a different answer.
3. What is the utility of CS-Bench when compared to approaches in which LLMs are situated in an environment with access to a terminal and tasked to solve problems?
4. Can the authors provide further evidence that the interlingual translations using GPT-4 are sound and accurate?
soundness: 4
strengths: In the already saturated LLM benchmark market, this paper has a number of stand-out strengths. First, it is highly comprehensive. Unlike other benchmarks which seek generality at the cost of precision and depth, CS-Bench focuses on a single domain and contains thousands of in-depth testing items from it. This is an improvement on existing computer science benchmarks like MMLU. Second, it is multilingual. This facilitates, among other things, the generalisability of deeper CS knowledge and reasoning skills when changing the surface level features of the language the system is operating in. This will also allow users to measure the degree of contamination, since (ex hypothesi) a considerable amount of LLM training data will be from English-language sources. Third, the authors include multiple response types, ranging from multiple choice to open ended responding. This facilitates researchers using CS-Bench to probe CS knowledge and reasoning skills with more nuance. Fourth, the results on logarithmic scaling laws is intriguing. Finally, the extensive error analysis is commendable and often not seen in benchmark papers such as this. It demonstrates again that CS Bench can be used to produce a much richer picture of the knowledge and reasoning skills of LLMs in this domain than can be gleaned by simply aggregating task performance.
summary: This paper introduces CS-Bench, the most comprehensive and detailed benchmark for testing computer science knowledge and reasoning that exists to date. It contains over 10,000 items over 26 subfields of computer science, uses 4 different response types (multi choice, assertion, fill-in-the-blank, and open-ended responding), and is presented in 4 languages (English, Chinese, German, French). The authors study over 30 LLMs, including both open- and closed-source models. They provide extensive analysis of performance across the benchmark and its subdomains, and conduct post-hoc studies of how performance scales with model size and the kinds of errors that LLMs make on CS-Bench.
weaknesses: There are a few weaknesses with this paper. First, CS-Bench is static and openly available online. This means that it is possible that LLM developers can train directly on the benchmark, leading to contamination. While we as a community hope that developers wouldn't be so mal-intentioned, it is an increasingly common phenomenon. Dynamic benchmarks (e.g., DynaBench) in which testing items are produced on the fly are able to sidestep such problems.

Second, true tests of computer science skills, particularly for software engineering and development, come from situating models in a coding environment, with access to a terminal. The authors might consider comparing their approach to this more embodied approach to evaluation.

Finally, I am sceptical of the use of GPT-4 for translating the benchmark from Chinese to other languages. While this is certainly scalable, I fear that the translations will be poor, particularly for the most complex questions. The authors' note that some manual checking was done, but I would like to see a more robust measure of translation quality, perhaps through asking a sample of independent native speakers of English, French, and German to check a random sample of questions and answers for soundness.

Meta-review: The paper "CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery" introduces a novel and extensive benchmark, CS-Bench, designed to evaluate the performance of Large Language Models (LLMs) in the domain of computer science. CS-Bench comprises approximately 10,000 meticulously curated test samples across 26 subfields within four key areas of computer science, supporting multilingual evaluation in English, Chinese, French, and German. The benchmark encompasses various task formats, including multiple-choice, assertion, fill-in-the-blank, and open-ended questions. The authors evaluate over 30 mainstream LLMs using CS-Bench, revealing insights into the relationship between model scale and performance, identifying reasons for model failures, and exploring the correlation between LLM capabilities in computer science, mathematics, and coding. The paper also discusses potential improvements, such as knowledge supplementation and CS-specific reasoning, and introduces measures to mitigate data contamination risks.

#### Contributions
1. **CS-Bench Benchmark**:
   - A comprehensive, multilingual benchmark for evaluating LLMs in computer science, covering diverse subfields and task formats. CS-Bench is the first of its kind and addresses a significant gap in the LLM evaluation landscape.

2. **Error Analysis and Improvement Directions**:
   - Detailed analysis of error types, identifying knowledge gaps and reasoning challenges as key areas for improvement. The paper explores potential solutions, including knowledge supplementation, symbolic reasoning integration, and targeted fine-tuning.

3. **Cross-Capability Analysis**:
   - Exploration of the correlation between CS, mathematics, and coding capabilities, demonstrating that specialized models in mathematics and coding can enhance performance in specific CS subfields.

4. **Mitigation of Data Contamination**:
   - Introduction of dynamic data generation strategies to maintain a private subset of the benchmark, reducing the risk of data contamination.




#### Weaknesses
1. **Clarity and Presentation**:
   - Some reviewers noted that the writing and visualization could be improved, particularly the radar charts, which were found to be dense and difficult to interpret (g1yj). The authors have addressed this by replacing the radar charts with bar charts in the revised manuscript.
   - The paper's narrative occasionally includes over-philosophical statements, which could be toned down for a more focused presentation on the benchmark's contributions (xmLP).

2. **Depth of Analysis**:
   - While the paper demonstrates correlations between CS, mathematics, and coding capabilities, it lacks a deeper analysis of the underlying factors driving these relationships. Reviewers suggested exploring architectural or training data characteristics to explain these correlations (g1yj).
   - The error analysis, while comprehensive, does not offer concrete strategies for improving LLM performance beyond general recommendations (g1yj). The authors have since provided exploratory experiments on knowledge supplementation, symbolic reasoning, and fine-tuning, but further specificity could enhance the paper's practical value.

3. **Benchmark Biases**:
   - The paper acknowledges potential biases, such as the focus on university-level content, but does not fully explore how these biases might influence results or limit the benchmark's generalizability (g1yj, 7dpp). The authors have expanded the discussion in the revised appendix, but a more critical reflection on the benchmark's limitations could strengthen the paper.

4. **Evaluation Methodology**:
   - The reliance on GPT-4 for scoring generative tasks raises concerns about validity and reproducibility (g1yj). The authors have explored alternative metrics and fine-tuned smaller models for scoring, but further validation of these alternatives could improve confidence in the benchmark's evaluation framework.
   - The use of GPT-4 for multilingual translations was questioned, with concerns about translation quality (Rfxy). The authors have provided evidence of translation quality through back-translation and human evaluation, but caution is advised when interpreting results across languages.

5. **Data Contamination**:
   - The static nature of CS-Bench makes it vulnerable to data contamination, as LLM developers could train directly on the benchmark (Rfxy). The authors have implemented dynamic generation strategies for a private subset of the data, but this may not fully mitigate contamination risks for the public dataset.

Decision: Accept (Poster)