OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models (2024)

Abstract: Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favour of functional programming (FP), e.g., HumanEval and MBPP. To address this, ❶ our study introduces a pioneering OOP-focused benchmark, featuring 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods. ❷ We propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k metric. ❸ Our evaluation of 23 leading large language models (LLMs), including both general and code-specialized models, reveals three key insights: 1) pass@o offers a more relevant and comprehensive assessment for OOP code generation; 2) Despite excelling in FP, code-specialized LLMs like WizardCoder lag in OOP compared to models like ChatGPT; 3) The poor performance of all advanced LLMs on our OOP benchmark highlights a critical need for improvements in this field. Our benchmark and scripts are publicly released at: https: //github.com/alphadl/OOP-eval.

heading: 1 Introduction
text: Large language models (LLMs, Ouyang et al., 2022a; Touvron et al., 2023), consisting of billions or even trillions of parameters’ Transformer blocks (Vaswani et al., 2017), have emerged like mushrooms after the rain, especially since the emergence of ChatGPT1. In comparison to small models, LLMs exhibit stronger generalization and reasoning capabilities (Wei et al., 2022). Currently, LLMs are playing a crucial role in various tasks, e.g., code generation (Chen et al., 2021; Li et al., 2022; Roziere et al., 2023), language understanding (Zhong et al., 2023), human-computer interac-
∗Corresponding authors. 1https://chat.openai.com
tion (Tolomei et al., 2023; Moslem et al., 2023), and translation (Peng et al., 2023; Lu et al., 2023).
The process of code generation entails crafting code in a suitable programming language from natural language descriptions of problems or requirements, aiming to effectively solve the problems or fulfill the requirements. Given that hiring professional programmers to write code consumes a significant amount of human and material resources, the importance of automated programming becomes particularly evident. Currently, the question of how to use the rising LLMs to generate more accurate automated programming codes based on problems or requirements stated by actual natural language has become an important research topic (Liu et al., 2023; Zhong and Wang, 2023). In the research process, code generation evaluation is crucial. Code generation evaluation not only needs to objectively and impartially reflect the current performance of LLMs in programming but also should disclose the shortcomings in LLM programming to further enhance its potential.
[Importance of OOP] According to the Novem-
13619
ber programming language rankings by TIOBE 2, four out of the top five programming languages are OOP, which reflects the importance of OOP languages. OOP is centred on designing code around data or objects rather than organizing it based on functionality and logic (Stroustrup, 1988; Stefik and Bobrow, 1985). OOP focuses more on the programming paradigm of class and object (Wegner, 1990). Functions are commonly referred to as methods in OOP.
[Motivation] However, existing code generation evaluation benchmarks primarily focus on the evaluation of FP, and lack the evaluation of relevant concepts and features of OOP, e.g., class, inheritance, encapsulation methods, etc. If using existing benchmarks in Table 1 for evaluation can only show the performance of LLMs in FP, it fails to reflect their potential in OOP, as illustrated in Figure 1.
[OOP benchmark and metric] Considering the limitations of current code generation evaluation FP benchmarks and the widespread use of the Python programming language, we propose the first OOP evaluation benchmark based on Python. OOP benchmark consists of 431 Python programs, covering key concepts and features of OOP, including class, inheritance, encapsulation methods, etc. Furthermore, to prevent the issue where LLMs may not generate concepts and features of OOP, we have optimized the pass@k (Kulal et al., 2019; Chen
2https://www.tiobe.com/tiobe-index/
et al., 2021) metric by matching key points in natural language with key points in the programming language, i.e., the class names and private function names, etc, for natural language requirements are matched with the class names and private function names, etc., in the programming language. Our main contributions are summarized as follows:
1. We construct and release the first OOP evaluation benchmark, which encompasses concepts and features of OOP, e.g., class, polymorphism, encapsulation methods, etc.
2. We devise a new metric pass@o based on conventional pass@k, tailored for the OOP code generation task, by matching key points in natural language and programming language.
3. We extensively evaluated our OOP with 23 advanced LLMs, demonstrating that i) there is still significant room for improving the OOP tasks, ii) our benchmark could serve as a robust and fair indicator that helps the community quantify LLMs’ OOP performance.

heading: 2 Related work
text: Code Evaluation Benchmark In the early days of LLMs, researchers from Google and OpenAI launched artificial handwritten code evaluation benchmarks, namely MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021), respectively.
MBPP and HumanEval are currently the mainstream code generation evaluation benchmarks, but both of them are based on the Python programming language. Subsequently, MultiPL-MBPP (Cassano et al., 2023) and MultiPL-HumanEval (Cassano et al., 2023) expanded upon these two benchmarks by translating the Python programming language into eighteen other programming languages, e.g., Java, C++, PHP, etc, to evaluate the performance of LLMs across others programming languages. Additionally, HumanEval-X (Zheng et al., 2023) incorporated multiple test cases into the HumanEval benchmark. Apart from the extensions made to these two benchmarks, other benchmarks like CodeApex (Fu et al., 2023) and ODEX (Wang et al., 2022) exhibit distinctive features across different natural languages and task types. ClassEval (Du et al., 2023) focuses on class tasks. Unlike existing code evaluation benchmarks, our proposed OOP benchmark primarily focuses on the concepts and features of OOP, e.g., class, inheritance, etc. These works are summarized in Table 1.
Code Evaluation Metrics Existing evaluation metrics can be broadly categorized into two types: dynamic evaluation metrics and static evaluation metrics. Dynamic evaluation metrics evaluate the executability of generated codes by using test cases, with pass@k (Kulal et al., 2019; Chen et al., 2021) serving as the primary representative. The calculation process for pass@k is shown in Appendix A. Additionally, this category of metrics includes n@k (Li et al., 2022). Static evaluation metrics calculate BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), Codescore (Dong et al., 2023) and CodeBLEU (Ren et al., 2020) among manually written examples and generated programs. However, these code evaluation metrics do not specifically focus on evaluating the concepts and features of OOP. Therefore, we further optimized the pass@k metric based on the evaluation benchmark for OOP.

heading: 3 Evaluation Framework

heading: 3.1 Overview
text: Existing code generation benchmarks in Table 1 for are confined to FP and do not involve essential concepts and features of OOP. We take the frequently used benchmarks, HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) in Table 1, as examples. They primarily evaluate the capabilities of LLMs in FP. The detailed descriptions of Hu-
given prompt and implement the corresponding requirements within the functions. However, ChatGPT does not generate the private functions named private_Shortest_subarray outlined in the red box.)
manEval and MBPP are provided in Appendix B. If we use existing benchmarks in Table 1 for evaluation, it does not show the capability of LLMs in OOP, as illustrated in Figure 1, that is, the seemingly decent LLMs (on FP tasks) perform relatively worse on OOP tasks. In addition, existing code generation evaluation metrics primarily use pass@k to evaluate the executability of the generated code. However, using the pass@k metric can not reflect whether LLMs generate concepts and features related to OOP, as illustrated in Figure 2. Therefore, pass@k can not objectively and fairly reflect the OOP capabilities of LLMs.
As a result, we established an OOP benchmark and proposed the evaluation metric pass@o for OOP. The process for constructing the OOP benchmark is illustrated in Figure 3.

heading: 3.2 Building OOP Benchmarks
text: Data Filtering. The training data for current LLMs mostly comes from the internet. If we directly evaluate LLMs using existing OOP data from
the web, it would not reflect the OOP capabilities of LLMs. Therefore, we first rigorously selected 500 natural language description-based problems or requirements based on Python from platforms like LeetCode 3, open-source repositories on GitHub 4, Stack Overflow 5, and Codewars 6. These 500 questions or requirements only are limited to FP and do not involve concepts and features related to OOP.
Human Rewritten. Subsequently, we manually rewrite the collected 500 questions or requirements by adhering to the following rules:
1. Designing, based on the problems or requirements, with relevant OOP concepts and features, e.g., class names, inheritance name (i.e., parent class name), encapsulation methods name (i.e., public function name and private function names), etc.
2. Related problems or requirements are implemented within the public function and private function of the class while ensuring the encapsulation of that implementation.
3https://leetcode.com/ 4https://github.com/ 5https://stackoverflow.com/ 6https://www.codewars.com/
3. Convert the variables associated with problems or requirements into class attribute variables, ensuring that these variables are accessible in both public and private functions.
4. If the implementation of problems or requirements is placed within the private function of the class, it is necessary to design a corresponding public function for access.
5. The rewritten OOP relevant problems or requirements can be successfully implemented and accessed through objects.
Following the five rules mentioned above, we conducted a standardized rewriting of the 500 Python-based problems or requirements.
Case Design. Finally, we designed corresponding test cases to evaluate OOP. We obtained 431 samples of OOP, as shown in Figure 3. The specific construction details of OOP are provided in Appendix C.
Level Classification. Given the difficulty nature of programming, we divided the designed OOP benchmark into three levels: Simple-level OOP, Moderate-level OOP, and Difficult-level OOP, as shown in Figure 7.
Simple-level OOP has 77 program samples, and includes only class, and public function. Moderatelevel OOP builds upon simple-level OOP by adding attribute variables and private functions, and has 179 program samples. Nevertheless, the difficultlevel OOP is based on the Simple-level of OOP, and adds inheritance, polymorphism and other related concepts and features of OOP. There are a total of 175 program samples for difficult-level OOP. Although private functions are not involved in the difficulty level, the problems or requirements in difficult-level OOP are more complex and varied. Using such a level of classification, we can not only evaluate the performance of existing LLMs in OOP but also analyze the shortcomings of LLMs, which allows us to better unearth the potential of LLMs in OOP. Using this approach makes it more convenient for us to improve the OOP performance of LLMs.
3.3 Evaluation Metrics Pass@o To evaluate whether LLMs generate concepts and features related to OOP, i.e., generated subclass name, parent class name, private function name and public function name, etc, in the programming language, we proposed a pass@o metric based on OOP. The pass@o metric adds keyword points matching between natural language with programming language based on the pass@k, i.e.,
α = n∑
i=1
f (Xi) ,
wheref(Xi) = (1)   1, if utf (Xi) passed and m∑ j xj∃Xi
0, otherwise
,
pass@o := E Problems
[ 1− ( n−α o ) ( n o ) ] , (2)
In Eq. (1), n represents the number of code generations for a given problem; Xi represents the i-th generated program code; α represents the quantity of n generated codes passing tests and matches; ut (·) denotes the unit test function; m represents the number of keyword points in the current prompt; and xj represents the j-th keyword points in natural language. In Eq. (2), o ≤ n.
The pass@o metric not only optimizes the limitations of pass@k evaluation but also objectively and fairly reflects the OOP performance of LLMs.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Evaluated LLMs In the OOP task, we conduct experiments on 23 mainstream LLMs. These models include both general LLMs, i.g., ChatGPT (Ouyang et al., 2022b; OpenAI, 2023), Llama2 (Touvron et al., 2023), InternLm (Team, 2023a), MPT (Team, 2023b), DeepSeek (Team, 2024), Falcon (Almazrouei et al., 2023), Qwen (Bai et al., 2023), Yi 7 and code-specialized LLMs, e.g., CodeLlama (Roziere et al., 2023), WizardCoder (Luo et al., 2023), StarCoder (Li et al., 2023), as shown in Table 6. The details description of 24 LLMs are shown in Appendix D.
Parameter Settings. In the experiment, we followed the settings on Llama2 (Touvron et al., 2023), configuring the temperature to 0.1 and 0.8 for code generation. The remaining parameters (top − p = 0.95, n = 200, o ≤ n), consistently remained unchanged. We evaluate the OOP benchmark on eight NVIDIA A100 GPUs using the vllm (Kwon et al., 2023) 0.2.1.post1 framework 8.
Metrics. In terms of evaluation metrics, we use for pass@k and the proposed pass@o metrics.

heading: 4.2 Overall Evaluation Result
text: The evaluation results of the LLMs with temperatures set to 0.1 and 0.8 are presented in Table 2, respectively. From the experimental results, We have obtained the following conclusions:
The OOP capabilities of the existing LLMs fall far short of the ideal state In Table 2, we can observe that LLMs with strong coding capabilities (e.g., WizardCoder-15b, CodeLlama-7b-Python, and CodeLlama-13b, achieved scores of 58.12, 40.48, and 35.07, respectively, in the HumanEval code leaderboard 9), exhibit performance in OOP benchmarks that falls significantly short of the ideal state. WizardCoder-15b, CodeLlama-7b-Python, and CodeLlama-13b scored 3.02, 1.27, and 2.92, respectively, on the OOP benchmark at pass@1. Their scores on pass@100 were also 10.01, 14.07, and 13.11, respectively. Even the current ChatGPT model with strong general capabilities scores 15.69
7https://01.ai/cn 8https://github.com/vllm-project/vllm 9https://huggingface.co/spaces/bigcode/
bigcode-models-leaderboard
scores, we use a temperature of 0.1 and top-p=0.95. The best results are highlighted in black bold; Red indicates the differences evaluated using the pass@o and pass@k metrics; Underlined indicates the maximum disparities evaluated between pass@o and pass@k metrics; Gray indicates models with a larger number of parameters.)
on pass@1 and 59.80 on pass@100. The results indicate that the untapped potential of existing LLMs in OOP has not been fully explored.
Limitations of pass@k evaluated OOP The scores from Table 2 indicate that using pass@k does not objectively reflect the OOP performance of LLMs, e.g., the WizardCoder-15b model achieves scores of 6.83, 28.10, and 29.41 using pass@k, while its scores drop to 3.02, 9.50, and 10.01 when using pass@o. The evaluation scores of other LLMs using pass@o in Table 2 showed a decline, once again proving the limitations of pass@k in evaluation OOP.
In addition, we also observed a significant phenomenon, e.g., when evaluated using pass@k, Qwen-14b (score 26.28) scored lower than WizardCoder-15b (score 28.10) on pass@80. However, when evaluated using pass@o, Qwen-14b (score 10.58) scored higher than WizardCoder-15b (score 9.50) on pass@80. Analyzing the experimental results of Qwen-14b and WizardCoder-15b, we observed that when evaluated using pass@o, Qwen-14b outperforms WizardCoder-15b in terms
of the ability to correctly generate OOP concepts and feature keywords, as illustrated in Figure 4. It also reiterates that pass@k cannot objectively and fairly reflect the evaluation results of OOP.
A larger model scale does not necessarily perform better on pass@1 In Table 2, when evaluated using pass@o, CodeLlama-34b scores 1.58 on pass@1, whereas CodeLlama-13b scores 2.92 on pass@1. Additionally, CodeLlama-13bPython scores 1.03 on pass@1, while the corresponding CodeLlama-7b-Python scores 1.27 on pass@1. However, CodeLlama-7b scores 1.20 on pass@1, which is lower than the score achieved by CodeLlama-13b. The scores of CodeLlama-7b, CodeLlama-13b, CodeLlama-34b, CodeLlama-7bPython, and CodeLlama-13b-Python on pass@1 indicate that a larger model scale does not necessarily result in the highest scores on pass@1.

heading: 4.3 Different-level Evaluation Results
text: Following the classification of OOP benchmarks in Section 3.2, we conducted evaluations for three levels of OOP benchmarks, and the results are pre-
sented in Tables 3, 4, and 5, respectively. we have drawn the following conclusions:
LLMs perform better at the simple-level OOP compared to the moderate-level and difficultlevel OOP From the simple-level OOP evaluation results in Table 3, we can see that the evaluation results using pass@k and pass@o are the same. It also indicates that LLMs can comprehend the fundamental concepts and features of OOP, e.g., class, and encapsulation methods (i.e., public function). However, in Tables 4 and 5, LLMs exhibit a weaker understanding of concepts and features related to OOP, e.g., encapsulation methods (i.e., private function), inheritance, and polymorphism, and are unable to generate corresponding code accurately. Detailed descriptions are in Appendix E.
ChatGPT has large gap in moderate-level usage using pass@k and pass@o From the results in Table 4, We observe that with pass@k evaluation, ChatGPT scores are 51.71, 83.30, and 83.67, but with pass@o evaluation, ChatGPT only achieves scores of 2.53, 51.54, and 54.78. We analyzed the moderate-level OOP results for ChatGPT, and found that its understanding of private functions is relatively poor. When evaluated using pass@k, a total of 5551 codes can pass the test cases correctly. However, when evaluated using pass@o, only 272
codes can successfully pass the test cases. Among them, 5279 codes fail to match the pass@o criteria. Upon careful examination, we found that all these 5279 codes resulted from errors generated by private functions To further validate the authenticity of the experimental results, we randomly selected prompts corresponding to three error results. Subsequently, we input prompts of the erroneous results into the web version of ChatGPT for code generation, as illustrated in Figure 13, 14 and 15. We found that the code generated by online ChatGPT 10 is also private function error.
ChatGPT outperforms moderate-level in difficult-level evaluation results According to the evaluation results from Tables 3 and 4, we observe that the performance of ChatGPT at the difficult level is stronger than at the moderate level. At the difficult-level OOP, ChatGPT scores are 19.70, 71.83, and 73.37, whereas at the moderate-level OOP, ChatGPT scores are only 2.53, 51.54, and 54.78.

heading: 5 Discussion
text: In this section, we will explore the reasons behind the generally lower scores of LLMs in OOP, as well as the applicability of the Chain-of-Thoughts
10https://chat.openai.com/
Attribute variable Private function Public function Class
CodeLlam a-34b 29752 51 36950 8069 此处键入公式 ChatGPT 62325 3090 72219 72222 All 70800 36000 86200 86200
a public function name; and “def __init__” represents the indicator for attribute variables name. Moreover, In our OOP benchmark, the LLM should ideally generate at least 86,200 “class”, 36,000 “def __” or “def _”, 86,200 “def”, and 70,800 “def __init__”.)
(CoT) method to OOP.
Why LLMs score lower in OOP benchmarks? We use the experimental results of ChatGPT and CodeLlama-34b on pass@1 as examples for analysis. As we instruct LLMs to generate relevant class names, private function names, public function names, etc., We conducted searches using simple keywords, e.g., “class”, “def _”, “def __”, “def __init__”, and “def” on both CodeLlama-34b and ChatGPT results. A detailed description of the retrieval process is provided in Appendix F. We compiled and analyzed the distribution of retrieval “class”, “def _”, “def __”, “def __init__”, and “def”, as shown in Figure 5, concluding that: 1) Weak knowledge, e.g., class, encapsulation methods, etc, of OOP in LLMs; 2) LLMs particularly lack cognition of private functions; 3) There is a certain degree of gap between CodeLlama-34b and ChatGPT. Specific example is shown in Figure 11.
The applicability of CoT in OOP. Taking CodeLlama-13b, StarCoder, and WizardCoder-15b as examples, we respectively incorporate the fewshot, zero-shot CoT, and few-shot CoT methods to validate whether CoT approaches demonstrate applicability in OOP, as shown in Table 7. We observed a significant improvement in the scores
of LLMs in OOP when using the few-shot approach, e.g., CodeLlama-13b achieved scores of 14.50, 48.13, and 49.85 using the few-shot method, representing improvements of 396.58%, 294.51%, and 280.24%, respectively, compared to the zeroshot method. In Table 7, we also observe that CodeLlama-13b achieves scores of 1.33, 13.31, and 14.62 in zero-shot CoT, but its score at pass@1 is lower at 2.92 compared to zero-shot. Additionally, StarCoder scores 0.25, 6.58, and 7.07 in zeroshot CoT, which are lower than StarCoder scores in zero-shot at 1.26, 10.05, and 10.88, respectively. The scores of the CodeLlama-13b, StarCoder, and WizardCoder-15b models on few-shot CoT are also lower than their scores on few-shot. We analyzed the experimental results of zero-shot and zero-shot CoT and found that using the CoT method introduces an illusion to the model, preventing it from directly generating the corresponding code, as illustrated in Figure 12. Therefore, it is necessary to integrate the concepts and features of OOP to design appropriate CoT strategies in order to enhance the effectiveness of generating OOP by LLMs. Appendix G provides detailed prompts for few-shot, zero-shot CoT, and few-shot CoT.

heading: 6 Conclusion
text: In this paper, we propose the first OOP evaluation benchmark based on Python, consisting of 431 Python programs, encompassing key concepts and features of OOP, e.g., class, encapsulation methods, etc. Simultaneously, we propose the evaluation metric pass@o for the OOP benchmark. pass@o improves upon the limitations of pass@k by matching keyword points between natural language with program language. We evaluate 23 mainstream LLMs using the proposed OOP benchmark and pass@o metric. Experimental results show that the current OOP of LLMs is far from ideal, which also reveals that LLMs have room for further improvement. Furthermore, Existing LLMs have a certain gap with ChatGPT in OOP. Moreover, we also investigate that applying some of the current improvement strategies directly to the OOP benchmark does not show significant improvement. In the future, we need to further strengthen the OOP knowledge of LLMs, especially regarding private functions. At the same time, we also hope that more researchers can contribute to the advancement of research in OOP.
Limitations
Our OOP benchmark has several limitations: (1) Our proposed OOP benchmark is based on the Python programming language and does not cover other OOP languages. (2) Given the incorporation of crucial concepts like polymorphism and inheritance in the OOP benchmark, it does not specifically address challenges associated with more intricate scenarios, e.g., multiple inheritance and overloading. (3) While OOP languages hold a significant share, non-OOP languages, e.g., C and Go languages, also play irreplaceable roles. In future work, we plan to consider expanding the OOP benchmark to cover a broader spectrum. Additionally, we encourage researchers to explore the potential of LLMs through evaluations based on the OOP benchmark.
Ethics Statement
We take ethical considerations very seriously. This paper focuses on establishing benchmarks for OOP to analyze the performance of existing LLMs. Our research reveals that existing LLMs fall far short of ideal performance in OOP. We conducted experiments on open and publicly available LLMs and accurately and objectively report the findings and conclusions of this paper. Therefore, we believe that this study does not raise ethical concerns.

heading: Acknowledgements
text: This work is supported in part by the STI 2030—Major Projects (No. 2021ZD0201405), National Natural Science Foundation of China (Grant No. U23A20318 and 62276195), the Special Fund of Hubei Luojia Laboratory under Grant 220100014, and the CCF-Zhipu AI Large Model Fund OF 202224.

heading: B Limitations of HumanEval and MBPP benchmarks
text: Existing HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) benchmarks primarily focus on FP to evaluate the programming capabilities of LLMs, as illustrated in Figure 6.
scores, we use a temperature of 0.1 and top-p=0.95. Red indicates the differences evaluated using the pass@o and pass@k metrics; Underlined indicates the maximum disparities evaluated between pass@o and pass@k metrics; Gray indicates models with a larger number of parameters.)
green color indicates the added few-shot content.)

heading: C Detailed construction process of OOP.
text: In the process of establishing the OOP benchmark, we hired a total of nine fourth-year undergraduate computer science students. Among them, two students were involved in the data collection process, four students participated in the rewriting process, and two students contributed to the use case construction phase, as shown in Figure 3.
During the data collection process, problems or requirements described in Non-English natural language are translated using the Google API, followed by manual verification. In the use case construction phase, we begin by inputting the rewritten prompt into ChatGPT to generate the corresponding code. Subsequently, the generated code is used for input testing. Finally, the output results are saved along with the input tests to serve as test cases. However, the code generated by ChatGPT may not always be correct, requiring manual inspection and correction. During the process of building the benchmark for OOP, we spent a total of $200.
scores, we use a temperature of 0.1 and top-p=0.95. Red indicates the differences evaluated using the pass@o and pass@k metrics; Underlined indicates the maximum disparities evaluated between pass@o and pass@k metrics; Gray indicates models with a larger number of parameters.)

heading: D The details of 23 LLMs
text: We have selected a total of 23 mainstream LLMs, including both code-specialized models and general models, e.g., ChatGPT (Ouyang et al., 2022b; OpenAI, 2023): ChatGPT was released by OpenAI in November 2022 and has been widely recognized for its astonishing conversational generation capabilities. In March 2023, OpenAI released ChatGPT 4.0. In our experiments, we chose to use ChatGPT 3.5 (gpt-3.5-turb) to explore its OOP. GPT_BigCode (Allal et al., 2023): GPT_BigCode, derived from the BigCode project, is a 1.12 billion parameter model trained on subsets of Java, JavaScript, and Python from The Stack. CodeLlama (Roziere et al., 2023): CodeLlama is a series of large-scale code language models based on Llama2 that offers state-of-the-art performance in open modeling, function completion, support for large input contexts, and zero-shot instruction following capabilities for programming tasks. CodeLlama includes the base model (CodeLlama), the Python specialized model (CodeLlama-Python), and the instruction-following model (CodeLlamaInstruct), each available with 7b, 13b, and 34b parameters. In our experiments, we selected the base models with 7b, 13b, and 34b parameters, as well as the Python-specialized models with 7b and 13b parameters. WizardCoder (Luo et al., 2023): WizardCoder is a model fine-tuned using the Evol-Instruct (Xu et al., 2023) method based on CodeLlama. WizardCoder includes the base model and the Python specialized model (WizardCoder-Python). The base model comes in 1b, 3b, and 15b variants, while the Python specialized model is available in 7b, 13b, and 34b. In our experiments, we selected the 15b version of the base model. StarCoder (Li et al., 2023): StarCoderBase is trained on The Stack (v1.2) 11 data in the GitHub repository. The StarCoder model is fine-tuned based on the StarCoderBase model. Llama2 (Touvron et al., 2023): The Llama2 model was released by the Meta team in July 2023. 11https://huggingface.co/datasets/bigcode/ the-stack
scores, we use a temperature of 0.1 and top-p=0.95. Red indicates the differences evaluated using the pass@o and pass@k metrics; Underlined indicates the maximum disparities evaluated between pass@o and pass@k metrics; Gray indicates models with a larger number of parameters.)
Model name Size Years Open-source Task type
Falcon 7b, 40b 2023 " General DeepSeek 7b, 67b 2023 " General Llama2 7b, 13b, 70b 2023 " General Yi 6b, 34b 2023 " General InternLm 7b 2023 " General MPT 7b 2023 " General Qwen 7b, 14b, 72b 2023 " General ChatGPT N/A 2023 % General GPT_BigCode 1.12b 2023 " Code-specialized CodeLlama 7b, 13b, 34b 2023 " Code-specialized CodeLlama-Python 7b, 13b 2023 " Code-specialized StarCoder 15b 2023 " Code-specialized WizardCoder 15b 2023 " Code-specialized
Table 6: Overview of the Evaluated Models.
Llama2 is a large language model (LLM) that has undergone pre-training and fine-tuning, with a range of parameters from 7 billion to 70 billion. In our experiments, we selected models with 7b, 13b, and 70b parameters. InternLm (Team, 2023a): InternLM encompasses models designed for practical scenarios. The InternLM model includes both a base model and a chat model with 7b and 20b parameters. In our experiments, we selected the base model with 7b parameters.
MPT (Team, 2023b): The MPT model is a decoder-style transformer trained by MosaicML. In our experiments, we selected the base model with 7b parameters.
DeepSeek (Team, 2024): DeepSeek is an LLM based on the power-law scaling, encompassing models with 7b and 67b parameters. In our experiments, we opted to utilize the foundational models with 7b and 67b parameters.
Falcon (Almazrouei et al., 2023): The Falcon series models are primarily trained on diverse and high-quality corpora assembled from web data, including the 7b, 40b, and 180b parameter models. In our experiments, we opted to use models with 7b and 40b parameters.
Qwen (Bai et al., 2023): The Qwen model is a large language model based on the Transformer architecture, trained on a vast and diverse dataset for pre-training. The dataset encompasses a wide range of types, including extensive web text, professional books, code, and more. During our experiments, we selected the base models with 7b, 14b, and 72b parameters.
The blue color indicates guiding the model to generate code step by step using the CoT approach.)
Yi 12: The Yi series models are developed as bilingual language models with a focus on Chinese and English. Yi models are trained on a 3T multilingual corpus and demonstrate promising prospects in language understanding, common sense reasoning, and reading comprehension. In our experiments, we selected models with 6 billion and 34 billion parameters.
We use 23 mainstream code-specialized and general models with the aim of better illustrating
12https://01.ai/cn
the performance of existing LLMs in OOP. The overview of the evaluated models is presented in Table 6.

heading: E Analysis of results
text: In simple-level OOP of Table 3, ChatGPT scored 37.34 at pass@1. However, in the difficultlevel and Moderate-level OOP, ChatGPT scored only 19.70 and 2.53 at pass@1, respectively. CodeLlama-13b scored 16.21 at pass@1 in the simple-level OOP. In the difficult-level and
Moderate-level OOP, CodeLlama-13b scored only 0.00 and 0.00 at pass@1, respectively. Additionally, WizardCoder-15b scored 16.79 at pass@1 in the simple-level OOP., while in the difficultlevel and Moderate-level OOP, it scored only 0.00 and 0.00 at pass@1, respectively. It indicates that LLMs can comprehend and execute simple class, and public functions. However, their understanding of private functions, inheritance, and polymorphism is relatively weak. It also provides us with room for improvement.

heading: F Detailed description of the retrieval process
text: During the retrieval process, we first search for the class class and attribute variables def __init__. Subsequently, we replace def __init__ in the generated code snippets with <endoftext>, and finally, we search for private functions def _ and def __. Using this approach helps prevent the inadvertent retrieval of attribute variables as private functions
during the search for private functions. The process of searching for public functions def follows a similar method.

heading: G Details of using the CoT strategy.
text: zero-shot CoT. We incorporate "Let’s think step by step" on top of the zero-shot, enabling LLMs to stepwise infer and thus complete the entire code generation process, as shown in Figure 8. few-shot. We randomly selected three samples from MBPP (Austin et al., 2021), but these three samples are limited to functions and do not involve relevant concepts and features of OOP. Subsequently, we manually re-write the selected three samples into examples of OOP based on the five major principles. Finally, the constructed samples were integrated into zero-shot to form a few-shot, as shown in Figure 9. few-shot CoT. On the foundation of a few-shot, we first instruct the LLMs to generate corresponding steps based on the question and then proceed
few-shot and zero-shot prompting strategies. (Red indicates decline, while blue indicates increase.)
step by step to complete the entire code generation process, as shown in Figure 10.

heading: Future Work
text: In the future, we need to further strengthen the OOP knowledge of LLMs, especially regarding private functions. At the same time, we also hope that more researchers can contribute to the advancement of research in OOP. Limitations
Our OOP benchmark has several limitations: (1) Our proposed OOP benchmark is based on the Python programming language and does not cover other OOP languages. (2) Given the incorporation of crucial concepts like polymorphism and inheritance in the OOP benchmark, it does not specifically address challenges associated with more intricate scenarios, e. g. , multiple inheritance and overloading. (3) While OOP languages hold a significant share, non-OOP languages, e. g. , C and Go languages, also play irreplaceable roles. In future work, we plan to consider expanding the OOP benchmark to cover a broader spectrum. Additionally, we encourage researchers to explore the potential of LLMs through evaluations based on the OOP benchmark