DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning (2024)

Abstract: Code Large Language Models (Code LLMs) have demonstrated outstanding performance in code-related tasks. Several instruction tuning approaches have been proposed to boost the code generation performance of pre-trained Code LLMs. In this paper, we introduce a diverse instruction model (DolphCoder) with self-evaluating for code generation. It learns diverse instruction targets and combines a code evaluation objective to enhance its code generation ability. Our model achieves superior performance on the HumanEval and MBPP benchmarks, demonstrating new insights for future code instruction tuning work. Our key findings are: (1) Augmenting more diverse responses with distinct reasoning paths increases the code capability of LLMs. (2) Improving one’s ability to evaluate the correctness of code solutions also enhances their ability to create it. Our code is released at https://github.com/prisnlp/DolphCoder.

heading: 1 Introduction
text: Code pre-trained models have achieved remarkable progress in the era of large language models (LLMs), such as Codex (Chen et al., 2021), AlphaCode (Li et al., 2022), and PaLM-Coder (Chowdhery et al., 2022b). Code-related tasks are also the key factors in evaluating the capability of LLMs. Numerous code LLMs have been proposed, including closed-source models (Chen et al., 2021; Li et al., 2022; OpenAI, 2023) and open-source models (Li et al., 2023b; Rozière et al., 2023). They perform expensive pre-training using substantial amounts of code data and display impressive performance.
In contrast to these pre-trained code LLMs, another lightweight paradigm of enhancing code capability is instruction tuning using relatively small
* Equal contribution. † Corresponding author.
high-quality code-related data. For example, Code Alpaca (Chaudhary, 2023) employs a similar selfinstruct method as Alpaca (Taori et al., 2023) to generate code instructions via OpenAI’s ChatGPT1. Further, WizardCoder (Luo et al., 2023a) introduces a more complicated Evol-Instruct method (Xu et al., 2023a) which evolves existing instruction data to generate more complex and diverse datasets. Instead, OctoPack (Muennighoff et al., 2023) and Magicoder (Wei et al., 2023a) construct code instructions by mining existing code corpus. All of these methods enhance the performance of the open-source Code LLMs.
However, these methods have two weaknesses: (1) They take the only golden answer but ignore the diversity of answers in code generation. We find that augmenting more diverse responses using different system prompts increases the code capability of LLMs. (2) Current models generate plausible code snippets in terms of grammar and logic but are unable to identify subtle errors, such as corner cases and wrong input/output formats. It has no guarantee that temperature sampling will consistently produce accurate answers over time. We suppose that LLMs are capable of generating correct solutions while struggling to discriminate correct from incorrect ones. Improving one’s ability to evaluate the correctness of code also enhances their ability to create it.
Inspired by the two insights, we introduce a diverse instruction model (DolphCoder) with selfevaluating for code generation. Specifically, we use Code Llama-python as our base model and obtain evolved instruction data following WizardCoder. Then motivated by rejection sampling (Touvron et al., 2023b; Yuan et al., 2023a) and ORCA (Mukherjee et al., 2023), we use different system prompts to generate diverse answers via ChatGPT. After removing low-quality and similar data using
1https://openai.com/blog/ChatGPT
4706
heuristic rules (Luo et al., 2023a; Di et al., 2023a), we perform supervised fine-tuning on the remaining instruction data. Further, we explore whether improving one’s ability to evaluate code helps generate it. We propose a self-evaluate multi-task learning framework by adding a code evaluation objective to the traditional instruction fine-tuning task. We find training the model for both code generation and code evaluation benefits the code capability.
Our key contributions are summarized as follows:
1. We introduce a diverse instruction model (DolphCoder) with self-evaluating for code generation. It learns diverse instruction targets and combines a code evaluation objective to enhance its code generation ability.
2. DolphCoder outperforms strong open-source code LLMs by a large margin, including CODELLAMA-INSTRUCT, OctoCoder, and WizardCoder.

heading: 2 Method
text: In this section, we elaborate on the methodological details of DolphCoder. As shown in Figure 1, DolphCoder has two training stages: (1) The first is Diverse Instruction Tuning (DIT) with multiple chain-of-thought answers to the same instruction. (2) The second is Multi-Objective Tuning (MOT) of combining the code generation task and code evaluation task both in the form of a natural language generation task.

heading: 2.1 Diverse Instruction Tuning
text: We follow the Evol-Instruct technique (Xu et al., 2023a; Luo et al., 2023a) to construct our training corpus. Based on the Code Alpaca dataset 2, we iteratively evolve the programming problems in this dataset via in-depth evolving to obtain new instructions.3 For each instruction, then we use different system prompts to query ChatGPT and obtain diverse targets. These system prompts aim at augmenting user instructions and task descriptions to give more code solutions with diverse reasoning paths. We display our system prompts in Figure 2. We find more diverse answers can increase the code capability of LLMs (see Section 4.2) and argue that different styles of code solutions provide more supervised signals for the model. Similar to Luo et al. (2023a); Di et al. (2023a), we also use several heuristic rules to remove low-quality and similar data. Finally, we get a diverse code instruction dataset with a size of 510k and utilize Code Llama as the foundation LLM to finetune.

heading: 2.2 Multi-Objective Instruction Tuning
text: We discover that the current instruction models produce both correct and incorrect code solutions when randomly sampled. We argue that LLMs can generate correct solutions while struggling to discriminate correct from incorrect ones. Therefore, we explore whether improving one’s ability to evaluate code helps generate it.
2https://huggingface.co/datasets/sahil2801/CodeAlpaca20k
3Since the original datasets or code are not released, we reproduce the evolve procedure following WizardLM (Xu et al., 2023a) and modify the evolve prompts according to the original WizardCoder paper.
We sample 5k instructions from the above dataset and use our model in the first stage to generate 100 answers using temperature sampling. Next, we use GPT-44 to verify the correctness of the deduplicated generated answers in terms of grammar, logic and efficiency. In our preliminary experiments, We encounter difficulties in assessing the correctness of code using ChatGPT or other existing LLMs. We also consider the use of compiled signals from the code executor but find most generated code is grammatically correct. Previous work (Liu et al., 2023a; Shen et al., 2023a) use existing unit tests in the training set, which is not applicable to our situation. We leave out more evaluation methods to future work. Finally, we obtain the code evaluation dataset with a size of
4https://openai.com/GPT-4
370k.5 We formulate the code evaluation task in the form of a natural language generation task as shown in Figure 1. We hope its similar training form can provide multiple meaningful supervision signals to the original code generation model. In our experiments, we find it difficult to balance the two code generation and code evaluation tasks because the model always tends to overfit the code generation objective. Therefore, we use a multistep training paradigm where we first finetune the above model as an evaluator and then as a generator. Specifically, we finetune the DIT model using the code evaluation dataset for 1 epoch and then continue training on the diverse instruction data for 100 steps. We find training more steps will not get further improvements. Experiment results demonstrate that the multi-step training stably enhances its code generation ability (shown in Figure 6).

heading: 3 Experiments

heading: 3.1 Benchmarks
text: In this paper, we focus on two of the most widely used benchmarks in the field of code generation.
• HumanEval (base)6 and HumanEval+ (plus). HumanEval (base) is a widely used benchmark proposed by OpenAI for code synthesis. It consists of 164 handcrafted programming problems, with an average of 9.6 test cases allocated to each one for correctness checking. Further, Liu et al. (2023b) find that
5We remove failed GPT-4 requests and responses without <passed> or <not passed> results.
6https://github.com/openai/human-eval
test cases in the benchmark may be insufficient and propose HumanEval+ (plus) powered by the EvalPlus framework to obtain 80× test cases.
• MBPP (base) (Austin et al., 2021) and MBPP+ (plus). MBPP (base) is also a code synthesis benchmark offering a set of 500 crowd-sourced Python programming problems covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. EvalPlus also provides the MBPP+ (plus) benchmark which expands by 35x test cases.
We perform n-gram matching de-duplication between our training datasets and the benchmarks to prevent data leakage. For all the experiments, we use greedy decoding and report the pass@1 metric. The inference prompt template is shown in Figure 4 following WizardCoder. To keep a fair comparison, we use the same EvalPlus 7 framework to compute metrics.

heading: 3.2 Baselines
text: In this paper, we categorize the baseline models into the following two types. (1) Closed-source models: We have specifically incorporated OpenAI’s GPT-3.5 and GPT-4, which are developed privately by leading technology companies, demonstrating the current state-of-the-art in LLM proficiency. (2) Open-source models: Several opensource LLMs have been made available to the AI community, although their performance generally lags behind the closed-source models a lot. As part of our research, we incorporate a significant number of these open-source models as our baselines, which include CodeGen (Nijkamp et al., 2023), CodeT5+ (Wang et al., 2021), StarCoder, CODELLAMA, OctoCoder and WizardCoder series.

heading: 3.3 Implementation Details
text: Data generation For Diverse Instruction data, we devise six system prompts shown as Figure 2, inclusive of a blank one, and prompt GPT-3.5-turbo to generate more diverse responses. Inspired by CodeFuse (Di et al., 2023b), we use heuristic rules to filter out low-quality data and deduplicate data based on the test set. The specific rules are as follows:
7https://evalplus.github.io/leaderboard.html
• Filtering low-quality data
1. Filter data with instruction length less than 10 words or greater than 1000 words;
2. Filter data with output length less than 80 words;
3. Filter out data with invalid markdown format, such as: code blocks not closed;
4. Filter data with more than 2048 tokens;
• Filtering data similar to test dataset
1. Filter data containing any function name from the test dataset.
2. Using NLTK to remove stop words and punctuation from the docstring of HumanEval, obtain the core words such as "sort array prime", etc. Filter data containing more than 40% of the core words from the test dataset.
For Multi-Objective Instruction data, we extract 5000 instructions from the Diverse Instruction data and subsequently generate 100 responses for each instruction in 0.5 temperature and 0.95 topp with the model we get after diverse instruction tuning. To ascertain the correctness of each code solution, we prompt GPT-4 to evaluate whether this code solution can meet the instruction requirement. And then they will be classified to either ’passed’ or ’not passed’. The prompt we used to evaluate is as Figure 3.
Training In our study, we utilize the CODELLAMA-PYTHON-7B and CODELLAMA-PYTHON-13B as our foundational models. We train DIT for 3 epochs and MOT for 1 epoch of code evaluation data and 100
steps of code generation data. During the training phase, we establish a global batch size of 512 and a sequence length of 2048, with a learning rate initialized at 5e-6 and a warmup fraction of 15%. After the warmup, the learning rate decays following a cosine schedule. We employ the Adam optimizer, with beta1 and beta2 parameters set at 0.9 and 0.95 respectively. And to ensure training stability, we incorporate gradient clipping with a value set at 1.0, a technique designed to prevent excessive gradient escalation that could lead to numerical instability or model divergence.

heading: 3.4 Main Results
text: The primary outcomes of our proposed method in comparison with the baselines are illustrated in Table 1. We conduct the following comparisons: (1) Generally, Our proposed method significantly outperforms all the previous methods in different sizes of model parameters. (2) Compared to our
base model (CODELLAMA), DolphCoder shows significant improvements in both HumanEval(+) and MBPP(+). In detail, DolphCoder-7b has a 25 percentage point increase on HumanEval and a 7.3 percentage point increase on MBPP. (3) As we compare our model DolphCoder with the recent baseline model WizardCoder-7b, which is built on the same base model (CODELLAMA), DolphCoder still consistently outperforms it across all benchmarks and model sizes. (4) DolphCoder-7b even outperforms CODELLAMA-PYTHON 34b, demonstrating the efficiency of using small-size LLMs. These results indicate the generalizability and robustness of our method.

heading: 4 Analysis

heading: 4.1 Ablation Study
text: To investigate the characteristics of the main components in DolphCoder, we conduct ablation ex-
periments in Table 2. From the results, we have the following observations: (1) Both DIT and MOT contribute to the performance improvement. Specifically, on the CODELLAMA-PYTHON-7b base, DIT yields an improvement of 7.9 pp on HumanEval and 4.8 pp on MBPP, compared to the baseline evolving instruction approach. MOT results in an improvement of 5.5 pp and 4.8 pp respectively on HumanEval and MBPP. Moreover, we observe that the MOT training data constructed by GPT-4 contains approximately 20% error noise, which severely limits the upper-bound performance of the MOT method in this work. A more detailed discussion can be found in Appendix B. (2) The combination of DIT and MOT yields further benefits. DolphCoder-7b exhibits an enhancement of 12.8 pp on HumanEval and 5.5 pp on MBPP, demonstrating the relatively orthogonal relationship between these two methods. These results demonstrate the effectiveness of our proposed methods.

heading: 4.2 Effect of Diverse Instruction Tuning
text: To further explore the source of the model’s improvement brought by DIT, we test the model’s performance under different sampling ratios which is shown in Table 3. The sampling ratio represents the number of system prompts. As the sampling ratio increases, the model’s performance on all indicators gradually increases, which proves that using more diverse responses can enhance the model’s performance.
In addition, we extract code snippets from the responses and check whether the implementation of these codes is different. For ease of statistics, we only focus on Python-related code. Specifically, we use AST to parse each piece of code and calculate the similarity between codes, then remove duplicates. We count the average number of unique code
solutions left for each code task as an indicator of code diversity. Table 3 shows that as the sampling ratio increases, the number of different code solutions corresponding to the same code instruction also gradually increases and the model’s performance on HumanEval improves too. Moreover, we observe that the increase in code diversity is not linear. Specifically, when we increase the sampling ratio from 3 to 6, we only observe marginal performance gains which we argue that the DIT data may contain unnecessary redundancy. Concurrent work (Lu et al., 2023b; Liu et al., 2023c) explore more complicated diversity-based compression methods for general instruction fine-tuning. We leave it to future work.

heading: 4.3 Effect of Multi-Objective Instruction Tuning
text: To explore how the code evaluation capability and code generation capability benefit each other, we evaluate them in the MOT training stage. Specifically, we evaluate the model’s code evaluation capability by sampling 40 answers for each of the 164 test questions in HumanEval and classifying their correctness using the given golden unit tests. The experimental results are shown in Figure 5.
From the results, we observe that there exists a strong association between the evaluation and generation capabilities of the model. As the model continues to train on the evaluation task, its evaluation capability keeps improving and gradually stabilizes. However, this process impairs the model’s code generation capability, potentially due to catastrophic forgetting caused by the multi-step training. After the model undergoes the generator training process, the model’s code generation capability is restored and the pass@1 metric surpasses the limit
of DIT model performance. Meanwhile, its evaluation capability significantly decreases. We suppose that the improvement in code generation capability is achieved through the transformation of the evaluation capability, but these two capabilities are challenging to coexist.
To further scrutinize the impact of the MOT, we compare the metrics of pass@1 and pass@10 to ascertain whether the improvements stem from the model’s heightened preference for the correct response. From Table 4, we find that the DolphCoder outperforms the DIT Model across all metrics. The improvement of MOT on the pass@1 greedy decoding indicator is the largest, with an increase of 4.9%. Compared with pass@1, the improvement on pass@10 is significantly reduced to only 1.8%. This implies that the impact of the MOT does not focus on enhancing the model’s capability to gener-
ate robust solutions. Instead, it primarily augments the model’s ability to distinguish responses, leading to a higher preference for the correct answer.

heading: 4.4 Analysis of Key Designs
text: System Prompt Ablation We obtain more diverse training data by transforming the system prompt. In order to explore whether the system prompt should still be used during training and inference, we conduct experimental verification. Table 5 shows the effect of the system prompts during the training and inference process. From the results, we observe that, compared to training with system prompts, training without the system prompt results in a significant decline. This implies that assigning completely identical code instructions to different answers is not a good training method. When comparing whether to use the system prompt during
inference, we find that erasing the system prompt results in a significant improvement compared to using the system prompt. This improvement may stem from the model’s freedom to determine the most suitable inference path.
Multi-Task Training Method Ablation We also explore different multi-task training methods as shown in Figure 6. The results show that both MOT variants significantly outperform the DIT baseline and the multi-step training setting gets superior performance compared to the mix training one. We also find the multi-step training gets a more stable training process to balance the two code generation and code evaluation tasks.

heading: 5 Related Work

heading: 5.1 Instruction Fine-Tuning
text: Large Language Models (LLMs) initially have shown notable zero-shot performance in various domains (Brown et al., 2020; Wu et al., 2021; Hou et al., 2024; Dong et al., 2023b; Zhou et al., 2024; Wu et al., 2023; Song et al., 2023). In order to further explore his potential, Large language models (LLMs) experience the Instruction fine-tuning (IFT) stage, which enhances their capability to accomplish tasks and adhere to human instructions. The term IFT is broadly used here to encompass a range of sequence-to-sequence fine-tuning applications (Zhou et al., 2023b; Yuan et al., 2023c; Cheng et al., 2023b; Zhang et al., 2024a). T5 (Raffel et al., 2023) was one of the first models to explore this approach, training on multiple supervised text-to-text tasks. Recent studies have delved into the multitask instruction-based fine-tuning of pre-trained LLMs, aiming to improve their innate ability to perform various downstream NLP tasks effectively, such as FLAN (Wei et al., 2022), T0 (Sanh et al., 2022), and UnifiedQA (Khashabi et al., 2020), further expanded the task scope to improve the overall generalization ability of LMs.
Following the notable successes of proprietary LLMs, particularly ChatGPT, a growing emphasis has been placed on employing Instruction FineTuning (IFT) to better align LLMs with human intentions, as underscored by the research of (Brown et al., 2020; Ouyang et al., 2022). A prevalent observation from these investigations is the significant enhancement in model performance attainable through fine-tuning on a more diverse dataset.
Additionally, IFT has proven its effectiveness for Large Language Models (LLMs) across various
domains, such as mathematical reasoning (Cobbe et al., 2021; Hendrycks et al., 2021; Yuan et al., 2023b; Chen et al., 2024; Yue et al., 2023; Gou et al., 2024; Li et al., 2023a; Yue et al., 2024) and code generation (Chaudhary, 2023; Luo et al., 2023b; Wei et al., 2023b), as well as in Information Extraction (IE) (Lu et al., 2022; Xu et al., 2023c; Zhao et al., 2023; Cheng et al., 2023c; Wang et al., 2023a; Zhang et al., 2024b; Li et al., 2024), Information Retrieval (IR) (Zhu et al., 2024a; Liu et al., 2024), and Spoken Language Understanding (SLU) (Hoscilowicz et al., 2024; Zhu et al., 2024c,b; Yin et al., 2024; Cheng et al., 2023a, 2024; Dong et al., 2023a). By leveraging their interactional features, researchers have adeptly applied SFT-refined LLMs to formulate commands that link with external tools, markedly enhancing the handling of a range of downstream applications (Shen et al., 2023c; Yao et al., 2023b; Song et al., 2024; Yao et al., 2023a; Fu et al., 2024).
Several studies have adopted alternative approaches; for instance, Taori et al. (2023); Zhou et al. (2023a); Wang et al. (2023b); Lu et al. (2023a) have employed a self-guided technique, utilizing large language models to generate expansive training datasets. Chiang et al. (2023) have focused on training their model using user-shared conversations from ShareGPT.com. Meanwhile, Dong et al. (2024) investigate the data composition challenge across different IFT capabilities in LLMs. Furthermore, Xu et al. (2023b) introduced the EvolInstruct method, which enhances existing instructional data to cultivate more complex and diversified datasets.

heading: 5.2 Large Language Models for Code
text: Large language models are pre-trained on trillions of tokens following the scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020), which demonstrates remarkable achievements across a broad spectrum of tasks and such an amount of text data is often a diverse composite with a non-negligible part of code (Zhang et al., 2023). Pioneered by Codex, researchers have also found continual pretraining on code to significantly benefit language models’ performance on code. For example, Chowdhery et al. (2022a) post-trains PaLM on 7.8B additional code tokens to get PaLM-Coder and Rozière et al. (2023) train LLaMA 2 (Touvron et al., 2023a) on more than 500B code tokens to acquire Code Llama, which leads open-source models to a new height. For supervised fine-tuning, many works
utilize larger, more capable teacher models to synthesize instruction data to finetune small language models (Mitra et al., 2023; Luo et al., 2023a; Di et al., 2023b). Instead, Muennighoff et al. (2023); Wei et al. (2023a) construct code instructions by mining existing code corpus, which are orthogonal to our method. Since code-supervised signals are easily collected by compiling and running them, reinforcement learning becomes another important branch. Numerous works have attempted to utilize reinforcement learning with feedback information offered by compilation and other sources. PanGuCoder2 (Shen et al., 2023b) introduces a ranking loss based on unit tests, which helps to align with the capable code model deeply. However, these methods rely on human-annotated unit tests, which limits the application in the practical scenario.

heading: 6 Conclusion and Future Work
text: In this paper, we investigate two fine-tuning methods to improve the LLMs’ performance on code generation. We first introduce a response augmentation strategy by using different ChatGPT system prompts to increase the diversity of code solutions. We find different chain-of-thought reasoning paths improve performance. Then, We adopt a multistep training approach that combines traditional code generation and code evaluation objectives. We find improving one’s ability to evaluate the correctness of code also enhances their ability to create it. For future work, we aim to explore the effect of our methods on the larger foundation models and parameter-efficient fine-tuning mechanisms. We also plan to increase the accuracy of evaluation signals via other ways like automatic unit tests and reinforcement learning methods.

heading: 7 Limitations
text: Our limitations are there-fold: (1) We only explore our method on the 7B/13B base models due to the computation cost. More experiments on the larger models and other code models should be conducted to confirm our conclusion. (2) We only use GPT-4 to evaluate the quality of generated code solutions. The performance of GPT-4 is still poor and limits the performance of our proposed method. More precise and open-source evaluation models should be explored in future work. (3) There is still room for optimization in our training data. For example, we find continually increasing the number of system prompts only gets marginal performance
gains. Diversity-based compression methods (Lu et al., 2023b; Liu et al., 2023c) may be valuable while the number of system prompts is large.

heading: 8 Broader Impacts
text: Similar to the other LLMs, our DolphCoder could also generate unethical, harmful, or misleading information, which is not considered in our work. Future research to address the ethical and societal implications is needed. DolphCoder is also susceptible to hallucination in ungrounded generation use cases due to its smaller size. This model is solely designed for research settings, and its testing has only been carried out in such environments. It should not be used in downstream applications, as additional analysis is needed to assess potential harm or bias in the proposed application.

heading: 9 Acknowledgement
text: We thank all anonymous reviewers for their helpful comments and suggestions. This work was partially supported by the State Key Laboratory of Massive Personalized Customization System and Technology (No. H&C-MPC-2023-02-07(Q), State Grid Technology Project (5700-202416236A-1-1-ZN) “Research on active semantic discovery technology based on SG-CIM and its application in power grid equipment supply chain optimization” and the National Natural Science Foundation of China (NSFC No.62076031 and No.62076036).

heading: A Case Study
text: Table 6 presents comparative examples of DolphCoder and WizardCoder. In this case, the models are required to determine whether an array can become a non-decreasing sequence after applying a circular right shift operation. From the case, we can observe that WizardCoder-13b generates an obvious logical error, where it only checks if the sequence is non-decreasing without performing any right shift operation. In contrast, DolphCoder accurately simulates the circular right shift operation and correctly identifies the termination condition. Furthermore, it is worth noting that DolphCoder considers more robust boundary input cases, which can be attributed to its training on a code evaluation task and a more diverse training dataset.

heading: B Effect of Code Evaluation Capability of GPT-4
text: In our study, we utilize GPT-4 to create a training dataset for code evaluation. However, there is a concern regarding whether GPT-4 has the ability to generate a perfect golden label. To obtain the answer we test it on HumanEval. Specifically, we use the DIT model to generate multiple candidate answers for each question. Subsequently, we
leverage GPT-4 to evaluate each of these candidate answers. And then we select the code solutions
that GPT-4 identified as correct. Since the test cases are available in HumanEval, we can get the
binary <passed> or <not passed> accuracy of GPT4 which is just 79.4%, which indicates that the train data we constructed through GPT-4 may contain about 20% noise. The noise limits the upper-bound performance of the MOT method in this work. Additionally, we report the average pass@1 results before and after filtering by GPT-4, as shown in Table 7. We can observe that the pass@1 before filtering is 56.6% which is generated by the DIT model directly at 0.2 temperature. Then we calculate the pass@1 after golden filtering is 72.6% where we use golden unit tests to filter wrong solutions. We consider this percentage of 72.6% is a theoretical upper bound of GPT-4. We also report pass@1 of MOT and GPT-4 Filtering. We summarize key insights as follows: (1) GPT-4 can not perfectly perform code evaluation (62.7% vs 72.6%). (2) Our MOT can achieve a slightly worse performance than the GPT-4 Filtering (59.9% vs 62.7%). (3) GPT-4 evaluation capability limits the upper bound of MOT. How to improve automatic code evaluation is essential to future work.

heading: Future Work
text: How to improve automatic code evaluation is essential to future work