CatCode: A Comprehensive Evaluation Framework for LLMs On the Mixture of Code and Text

Authors: zhenru  lin
yiqun  yao
yang  yuan
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 03:34:45

Abstract: Large language models (LLMs) such as ChatGPT are increasingly proficient in understanding and generating a mixture of code and text. Evaluation based on such *mixture* can lead to a more comprehensive understanding of the models' abilities in solving coding problems. However, in this context, current evaluation methods are either limited in task coverage or lack standardization. To address this issue, we propose using category theory as a framework for evaluation. Specifically, morphisms within a code category can represent code debugging and transformation, functors between two categories represent code translation, and functors between a code category and a natural language category represent code generation, explanation, and reproduction. We present an automatic evaluation framework called **CatCode** (**Cat**egory **Code**) that can comprehensively assess the coding abilities of LLMs, including ChatGPT, Text-Davinci, and CodeGeeX.
Large language models (LLMs) are increasingly proficient in understanding and generating a mixture of code and text. Evaluation based on such *mixture* can lead to a more comprehensive understanding of the models' abilities in solving coding problems. However, current evaluation methods are either limited in task coverage or lack standardization. To address this issue, we propose to apply category theory as math abstraction for code-related evaluation. Specifically, morphisms within a code category can represent code debugging and transformation, functors between two categories represent code translation, and functors between a code category and a natural language category represent code generation and explanation. We present an automatic evaluation framework called **CatCode** (**Cat**egory *Code*) that can assess the coding abilities of various ChatGPT-like LLMs in a *comprehensive* and *standard* way, and further support *composite* task evaluation. The code can be found in https://github.com/scorpio-nova/CatCode.

Main paper: 
 introduction: 
 The success of large language models (LLMs) as programming assistants has been widely acknowledged, with their higher proficiency demonstrated in various coding tasks such as code generation (Li et al., 2022; Svyatkovskiy et al., 2020) , code explanation (MacNeil et al., 2022) , and code translation (Zhu et al., 2022) . However, what truly sets the recent LLMs apart is their ability, akin to ChatGPT-like models, to comprehend and align with human intents by processing a mixture of natural language and code. This unique attribute significantly lowers the entry barrier for users, leading to their widespread adoption and notable achievements.To assess the coding ability of LLMs, numerous efforts have been made by researchers. However, current evaluation methods are either limited in task coverage or lack standardization. Match-based automatic frameworks such as CodeBLEU (Ren et al., 2020) rely primarily on similarity scores to evaluate the quality of code. However, such frameworks often fail to capture the nuances of code functionality and meaning. Execution-based evaluation methods, for example, MBXP (Athiwaratkun et al., 2022) and MultiPL-E (Cassano et al., 2023) , can evaluate the function accuracy of code, but they primarily focus on code generation and translation tasks. Task-based evaluation frameworks like CodeXGLUE (Lu et al., 2021) offer a comprehensive approach but lack standardization due to completely different datasets, task formulations, and APIs. Consequently, we still lack an evaluation framework to adapt to the context of a mixture of natural language and code, and there is a need to establish a comprehensive evaluation framework that not only supports diverse and novel task definitions but also provides a standardized approach.We aim to establish a comprehensive theoretical framework that can be open-sourced and applied to essential coding tasks, providing standardized automatic evaluation metrics. It is a challenging task without theoretical guidance. Therefore, we seek a theory that can effectively express the structural aspects of code, language, and their interrelationships. In this context, we find category theory, a branch of mathematics, to be particularly suitable for describing relationships between various elements.By utilizing category theory's mathematical abstractions, we can gain insights into the relationships among different programming languages (PLs) and natural languages (NLs).We consider PLs and NLs as categories, and functionally equivalent programs as objects, and leverage functors and morphisms to capture the object relations within and across these categories. This gives a unified framework for describing functional equivalence, which not only works within a single PL, but also among different PLs, and even between PLs and NLs. By learning the morphisms within a programming language category, the model can grasp the similarities and differences between code snippets. Additionally, by acquiring knowledge of the functors between categories, the model can understand the relationship between different programming languages and natural languages.Based on the categorical perspective above, we build a standardized evaluation framework (see Figure 1 ). It is standardized in terms of data definition, task formulation, and APIs. This framework can be extended to many code-related tasks as long as a categorical definition is given. In our experiments, we give some examples of common code-related tasks, and assess models such as ChatGPT and CodeGeeX for their capabilities in identifying functional equivalence in code, performing code translation, generating code explanations, and reconstructing code based on explanations. We have observed that these models still struggle to differentiate between the concepts of "functional equivalence" and "similarity" in code.While they demonstrated relatively satisfactory performance in code translation, maintaining functional equivalence between code and its corresponding explanations remains a persistent challenge.Our main contributions can be listed as follows:Reproduction functor explanation functor translation functorEnglish python • We introduce CatCode, a novel evaluation perspective for code-related tasks based on category theory, which provides a comprehensive framework that encompasses a wide range of code-related task formulations. • We present a standardized automatic evaluation platform within the CatCode framework, that offers a quantitative assessment of the coding abilities of Language Models (LLMs) and can adapt to various datasets and models, which will be publicly available. • We evaluate competitive LLMs, including ChatGPT, StarCoder and CodeGeeX, providing insights into their strengths and limitations in understanding the mixture of NL and code.A B C A' B' C' A'' B'' C'' Java A B C Java python A B C A'' B'' C'' Java A B C Java English A B C A' B' C' Java A B C 
methods: 
 Generally, it is difficult to achieve both comprehensiveness and standardization. In Section 2.1, we show that the categorical perspective, with its emphasis on generalization and abstraction, offers a valuable approach to achieving comprehensiveness in dealing with the mixture of code and natural language. In Section 2.2, we discuss the significance of standardization and outline strategies for achieving it.Category theory studies how structures arising in different areas can be described with a common language. It was originally developed to transport ideas from one branch of mathematics to another, e.g. from topology to algebra. Category Theory has been applied to different fields, ranging from functional programming to logic, software design, and linguistics (Spivak, 2014; Ahrens & Wullaert, 2022; Fong & Spivak, 2018; Bradley, 2018) . Here we provide the basic definitions used in our work and explain their applications to describe code and natural language. -Associativity: If f : a → b, g : b → c, and h : c → d then h • (g • f ) = (h • g) • f .-Identity: For every object x, there exists a morphism id x : x → x called the identity morphism for x, such that for every morphism f : a → b, we have id b• f = f = f • id a .Application 2.1. (Category, Object, Morphism of Code and NL). For describing the mixture of code and natural languages, we first regard them as objects from different categories. We consider the code categories that contain all executable code in a certain language:Figure 3 : Standardized evaluation platform. The central pipeline offers a consistent approach for all evaluations. Behind the pipeline, we provide a variety of functions to automatically conduct the most important steps. With our platform released, the pipeline can easily accommodate novel datasets, tasks, and models by following the instructions outlined alongside the grey lines.C Java , C Python , ... We define the natural language categories which contain all description/explanation/summarization of code etc.: C English , C Chinese , ... For simplicity, let's call them C 1 , C 2 , ... In a category, there are infinitely many different objects. We name these objects o 1 , o 2 , ... Each object contains infinitely many programs, with the same running outcome for each valid input.We use this definition because we focus on the functional equivalence of code function. For example, one may have slightly different implementations of quick sorts, but they represent exactly the same function.1 Based on the definition of objects, we define the morphism between two objects, as "necessary edits to convert a function to another". In particular, we define the self-morphism for each object, as "edits that do not change the functionality of the program". Definition 2.2. (Functor). A functor F from a category C to a category D, written as F : C → D, maps an object x ∈ Ob(C) to F(x) ∈ Ob(D); as well as a morphism f : x → y in Hom(C) to F( f ) : F(x) → F(y) in D, such that the following two properties hold:-For every object x in C, F(id x ) = id F(x) ;-For all morphisms f : x → y and g : y → z, F(g• f ) = F(g) • F( f ).Application 2.2. (Functor of Mixture of Code and NL). We define the functor from C 1 to C 2 , to be the transform from one language to another language, but with the same functionality.When it applies to two code categories, it usually represents code translation, and we define it as a "translation functor". When the functor is between a code category and a natural language category, it may have many possible meanings, for instance, we can define "generation functor" from NL to PL that generate code solution to a problem description, "explanation functor" from PL to NL that explains a piece of code, and "reproduction functor" from NL to PL that generates code snippets based on code descriptions.To show the practical use of categorical framework and support the community, we opensource our evaluation framwork2 . Figure 3 . shows the streamlined process of our standardized evaluation platform. Standardization within the framework is achieved through a focus on three critical aspects: data definition, task formulation, and APIs.The original datasets may consist of a mixture of code and natural languages, so a clear data definition is crucial. The code can be in different languages (Java, Python, JavaScript, etc.) , and on different levels (single-line, function-level, file-level). Natural language can be in different relationships with the code (problem descriptions, code comments, demands, etc.). By establishing a standardized data definition, we ensure compatibility and facilitate the comparison and integration of different datasets and tasks. For implementation, we use a "data formatter" to rearrange code based on the definition of objects and categories. Additionally, we provide "predefined morphisms" and implement them based on JavaTransformer (Rabin et al., 2019) to automatically apply morphisms to code objects, which makes it easy for data augmentation.We use a categorical perspective to formulate diverse code-related tasks using the unified language of objects, morphisms, and functors. A good task formulation allows for a more generalized and flexible approach to defining more complex code problems, enabling the inclusion of a wide variety of code-related tasks and comparing their similarities and differences. Based on the task formulation, we carefully decide what parts of the code and NL should be the model input, ensuring a cleaner setting. Then we use "prompt selector" to select a suitable prompt for task description, and use "input constructor" to combine the data and prompt as task input.APIs After the data is ready, we focus on the standardization of model APIs and test APIs. Standardized APIs promote transparency, fairness, and efficiency in the evaluation. By defining a clear and consistent set of APIs, we have integrated OpenAI models into the evaluation process and can test them using parallel requests. For postprocessing the model's output, we enable the extraction and filtering of plain text from the code-text mixture. For test APIs, we provide a "pairing test" API for evaluating the model's answer based on natural language, and an "execution test" API, which connects to Mxeval (Athiwaratkun et al., 2022) for compiling and running the tests for a given code. 
use category framework for evaluation: 
 In this section, we exhibit some experimental examples of how to use our platform. Meanwhile, we explore the following three types of tasks: discriminant task, generation task, and composite task, that correspond to those illustrated in Step 3 of Figure 1 . (Related tasks: code explanation, code summarization, code generation, code reproduction)→ Categorical perspective: Can the model preserve the functional equivalence after applying an explanation functor and then a reproduction functor?We conduct three experiments from categorical aspects accordingly. For a common setting, we use three multilingual datasets: HumanEval-X (Zheng et al., 2023) , MBXP (Athiwaratkun et al., 2022) , and MathQA (Amini et al., 2019; Athiwaratkun et al., 2022) throughout the three experiments, and use five models: Text-Davinci-003, gpt3.5-turbo, Llama-2-7b-chat- 1-neq morphism ! 𝑎 ! 𝑏 ̅ 𝑐 ≁ 𝑎 ! 𝑎 ! " 𝑎 ! # 𝑎 $ 1-e q m o rp h is m 𝑑 = 2 𝑑 = 1 𝑑 = 1 ∃ global morphism 𝑑 ≫ 1 𝒞Code Objects We conduct the experiments within the PL category and define "the function with the same running outcome for every valid input" as the same object.The general category perspectives only define morphisms between two code snippets, but morphisms do not have distance information. Considering fine-grained evaluation, we are curious about the model's ability to modify or debug the code, which corresponds to the local scale; and to write an equivalent new solution to a coding problem, which is of a global scale.Object Distance To test the model on both local and global scales, we define the "distance" between two codes. As illustrated in Figure 4 . In a PL Category C, we use ∼ to express functional equivalence. ā = {x | x ∈ C, x ∼ a} means the set of all code that has the same functionality as a. In this category, ā and c are two different objects. Within ā, all code instances are functionally equivalent, e.g. a i and a j . We can apply predefined morphisms to the objects. For example, we can apply 1-step equivalent morphisms(1-eq morphism) on a i to get new program instances still in ā, or apply 1-step nonequivalent morphism(1neq morphism) to get new instance in another equivalent class b. Distance d represents the minimal number of morphisms needed to transform one program to another given a predefined morphism set. For example, Applying two different self-morphism on a i returns a 1 i and a 2 i , then we need at least two steps to transform from a 1 i to a 2 i . Those are local-scale equivalence. If it is hard to transform a i to a j within just a few steps(e.g., 1 or 2 steps as illustrated here), we treat them as global-scale equivalence.Dataset For local morphism, we extract the Java functions within HumanEval-X, MBXP, and MathQA datasets. We perform the 9 local morphisms based on AST transformations.1. Variable Renaming (VR): rename a variable, with a new name randomly chosen from all variable names in the dataset, or use "var N " for simplicity 2. Boolean Exchange (BE): propagate the exchange of "true" and "false" throughout the method and preserve the same logic 3. Loop Exchange (LE): exchange f or loops and while loops 4. Switch To If (SI): replace a switch statement with an equivalent if statement 5. Unused Statement(US): insert an unused string declaration to a randomly selected basic block in a method 6. Reorder Condition (RC): write the condition in reverse order (e.g., change i < n to n > i) 7. Permute Statement(PS): swap two independent statements (i.e. without data or control dependence) 8. Modify Condition(MC): change the binary operation in the condition (e.g., change i < n to i ≤ n) 9. Remove Else(RE): delete the else branch of the if-else statement The first 7 morphisms are adopted from JavaTransform (Rabin et al., 2019) that conducts functionally equivalent transformations, which means they are self-morphisms. Other than self-morphisms, we add 2 morphisms: ModifyCondition and RemoveElse, that change the program's function.For global morphisms, since the three datasets above do not contain multiple solutions to the same problem using the same PL, we complement with code from the test split of Code Contest (Li et al., 2022) dataset.For dataset statistics and the detailed preprocess procedure, please refer to Table 2 in the Appendix.We evaluate Text-Davinci and ChatGPT. We input (code, code) pairs from the datasets or generated by morphisms, and ask the model to answer whether they are functionally equivalent.Evaluation We collect the model's yes/no responses and calculate the average precision for different morphisms.The results are demonstrated in Figure 4 . Overall, Text-Davinci and ChatGPT make a tie. Locally, ChatGPT is better at identifying nonequivalent morphisms, and worse at equivalent ones. Globally, Text-Davinci is better at identifying global nonequivalent morphisms. Note that a random guess leads to a precision of 50%, so both models behave just slightly better than random guesses for identifying global equivalence.We investigate the model's ability to perform code translation by utilizing functors between two programming language categories. To isolate the influence of natural language, we exclude problem descriptions and code comments, forcing the models to focus solely on translating code from one programming language (PL) category to another. The input prompt may contain a request for translation using natural language (e.g., "translate the below Java code to Python code").Datasets We conduct evaluations on the HumanEval-X, MathQA, and MBXP datasets. The input category is Java, while the output categories are JavaScript and Python.We evaluate five models: Text-Davinci, ChatGPT, CodeGeeX, Llama2 and StarCoder. For CodeGeeX, code translation is supported inherently, so we simply input the Java code without additional text. For the other models, we provide prompts that request the model to do the translation.We extract the functions from the model's responses and assess their correctness using the Pass@1 rates of execution-based tests.The results are depicted in Figure 5 . Among the three models, ChatGPT performs the best, exhibiting a slight advantage over Text-Davinci. With regards to the datasets, ChatGPT and Code explanation and code generation tasks have been conducted separately in previous work. However, from a categorical perspective, we can make a combination. We define the explanation functor as the functor that maps an object from a PL category to an NL category, with a precise description of the code's function. We define the reproduction functor as the functor from an NL category to a PL category, that uses an explanation object in NL to reproduce the code of the same function in a PL category. If the reproduced code is functionally the same as the original code, we reckon the model preserves the functional equivalence after applying an explanation functor and then a generation functor.Datasets We conduct the evalution on three datasets: HumanEval-X, MathQA, and MBXP.For consistency, we select Java as the programming language (PL) category for all evaluations.We evaluate Text-Davinci, ChatGPT, Llama2 and StarCoder, prompting each model twice. In the first prompt, we ask the models to describe the code using natural language, including the precise function name, arguments, and return type, and provide sufficient information to reproduce the code. In the second prompt, we provide the model's explanation and ask it to translate the description back into code.Evaluation To evaluate the correctness of the reproduced code, we extract the functions from the model's second responses and assess them using the Pass@1 rates of execution-based tests.The results are depicted in Figure 6 . On the same datasets and with the same two models, the average pass@1 rate is about 30% lower compared to that of the translation functor. This indicates the model has significant information loss in the explain-reproduce process. As models evolve, scores on high-quality datasets tend to saturate (HumanEval 90%). Should we abandon HumanEval, and call for massive efforts to construct new ones? We believe it's not the only way. With CatCode, we can easily define composite tasks, and make the evaluation effective again. The construction of composite tasks is mostly automatic, greatly reducing the labor for collecting new datasets.Moreover, real-world applications often require model to handle those composite tasks. For example, developers may need to use the model to write new features, and synchronize code updates between Android and iOS versions. To this end, we propose and experiment on a new task "repair morphism + translation functor" using the CodeScope dataset. We use the code repair subset from the CodeScope dataset (Yan et al., 2023) and add a step for the model to translate from C++ to C after debugging C programs. Results on GPT-4 show that pass@1 drops from 26/60 in the original task to 18/60 in the composite task, which shows that the composite task is more difficult.Based on our experiments, we have found that both Text-Davinci and ChatGPT models are capable of identifying the impact of one or two local morphisms on code function. However, identifying global morphisms proves to be challenging for these models. In terms of code translation, both ChatGPT and Text-Davinci models outperform other models. They demonstrate accurate reproduction of basic syntax, variable names, and numerical values. Nevertheless, when dealing with more intricate program structures and aligning data types across different programming language categories, these models encounter obstacles that hinder more precise translation.It is important to note that maintaining functional equivalence between code and natural language presents a more formidable challenge compared to code translation. 
coding abilities of large language models: 
 Research on large language models trained on code has made significant strides. CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo et al., 2020) incorporated program Abstract Syntax Trees (AST) and data flow information to enhance code structure and context understanding. The Encoder-Decoder architecture in CodeT5 (Wang et al., 2021) and PLBART (Ahmad et al., 2021) supports multi-task learning, aiding code translation and generation. More recently, the emergence of Codex (Chen et al., 2021) , ChatGPT, and StarCoder (Li et al., 2023) has brought GPT-based models into the mainstream for content generation tasks, including coding. These models have shown impressive performance across various domains, but their proprietary nature, limited access to data, or lack of model checkpoints have made it challenging to thoroughly evaluate and understand their capabilities and limitations. Consequently, there is a need to develop a comprehensive framework to evaluate these black-box models.Evaluation of code models is diverse, encompassing match-based, task-based, and executionbased frameworks. The match-based approach, exemplified by CodeBLEU (Ren et al., 2020) , considers n-gram matches and AST similarity but may overlook functional equivalence in varying solutions. Task-based evaluations, like CodeXGLUE (Lu et al., 2021) , offer comprehensive assessments across coding tasks but lack uniformity in data and task settings, limiting scalability. Execution-based evaluation, used in benchmarks like MBXP (Athiwaratkun et al., 2022) and MultiPL-E (Cassano et al., 2023) , judges code correctness through test cases.Our work mainly focuses on providing a new framework for unifying the task-based evaluation. While our work also considers test cases from execution-based benchmarks, our focus extends beyond correctness evaluation to include code morphism, PL to NL functor aspects, and the composition of different tasks, for example the code reproduction task can be viewed as the combination of code explanation and generation tasks.Moreover, code-related models are found to be sensitive to slight edits in the prompts, therefore to evaluate their robustness is crucial. Previous work applies semantics-preserving transformations on code docstrings and program, for better training and evaluation of models (Wang et al., 2022; Henkel et al., 2022) . Apart from using transformations based on AST edits to ensure accuracy, we further investigate the models' robustness by providing a finer-grained characterization based on the edit distance of correct and incorrect edits. 
conclusion: 
 Our contributions include introducing CatCode as a novel evaluation perspective based on category theory, which enables a comprehensive and mathematically abstract approach to evaluate LLMs that excel in understanding and generating a mixture of code and text.Based on categorical perspectives, we present a standardized automatic evaluation platform, which is adaptive to new datasets, tasks, and models. We evaluate competitive LLMs to provide valuable insights and find out current models' deficiency in recognizing functionally equivalent code and preserving information of code function between the code and its explanation. We open-source our platform, hoping to contribute to the comprehensive and standardized evaluation for LLMs by offering a categorical perspective to deal with the mixture of code and text. We hope that CatCode inspires further research and development in the field, leading to LLMs that can effectively assist developers in more complex coding tasks and contribute to advancements in artificial intelligence and software engineering as a whole. 
discussion: 
 While we strive to apply categorical perspectives to offer a comprehensive and standardized way of evaluation, we find some limitations and room for future work. First, we use prompts to instruct the models to act as certain functors, but prompts that express similar meanings will lead to different results, we have not investigated which prompts are more suitable for the model or which models are more robust to different prompts.In this paper, our experiments primarily focus on objects, morphisms, functors, and their combinations. For future work, more powerful concepts and properties in category theory can be used for more complex and composite coding tasks. For example, we can further investigate the setting of composing many morphisms and many functors, to further explore the commutative law and isomorphism properties of categories. 
 
 Appendix: Davinci ✓ ✓ ✓ ✓ ChatGPT ✓ ✓ ✓ ✓ CodeGeeX ✓ LLaMa2-7b ✓ ✓ ✓ ✓ StarCoder ✓ ✓ ✓ ✓Table 1 : Baseline Models and their supported evaluation tasks.• Only the Java split of HumanEval-X, MBXP, and MathQA datasets are used. The local morphism is applied by JavaTransformer by AST edits at the function level. To use JavaTransformer, code must compile successfully and contain a single function. We filter out code in the Mxeval datasets that are generated by the model and cannot compile or contain multiple functions.• Due to the large size of the Code Contest dataset, only the test set is used. Considering that the maximum token length of Text-Davinci is 2048, we filter out solutions with a length greater than 500. This ensures that the code pair length remains ≤ 1000, allowing for additional tokens for prompt content and response. After filtering, 97 problems remain, and a problem may have multiple correct and incorrect solutions in different languages. We utilize the Java and Python splits of this dataset.We utilize the following baseline models by making API calls to their official endpoints: Text-Davinci-003 (referred to as Davinci), ChatGPT, CodeGeeX. The default hyperparameters of these models are used, with the exception of setting max token to 500 when an early stop of the answer is observed.Table 1 provides an overview of the tasks supported by each model. Since Text-Davinci-003 and ChatGPT support various types of text inputs and outputs, they are capable of performing all tasks. On the other hand, the CodeGeeX API does not support natural language output, so we only employ it in translation functor experiments.To assess the performance of the model, we conduct two types of tests: pairing test and execution-based test.The pairing test is utilized for morphism identification. The task involves comparing two code snippets and requesting the model to explicitly answer "True" or "False", along with providing a comparison. For automatic evaluation, we extract the True" or "False" answer.The precision score is used to evaluate the model's ability to identify different equivalence classes. For equivalent objects, the precision is calculated as Precision(eq) = TP TP+FN , while for nonequivalent objects, the precision is calculated as Precision(neq) = TN TN+FP . The execution-based test is employed for both translation and reproduction experiments. We extract the functions from the model's responses and evaluate their correctness using the Pass@1 rates of execution-based tests. This test ensures that the translated/reproduced code produces the same expected results as the original code. 3 : Morphism types with the three worst average precision scores of each dataset. As an example for notations, BE-VR (66.67) stands for a 2-eq morphism of "Boolean Exchange" and "Variable Renaming" with a precision score of 66.67%. solely of Java code. The constructed pairs are generated through sampling. It is worth noting that certain morphisms can have multiple outputs for a given input. For instance, the Variable Renaming morphism can be applied to all variables in a function, resulting in multiple outputs equal to the number of variables. To balance the number of outputs across different morphism types, random sampling is employed when there are more than two applicable morphism types. For each code sample, two types of morphisms are first sampled, followed by sampling one output for each morphism type.Which morphism types are harder to be identified, and are they hard across datasets? We calculate the average precision scores for 1-eq, 2-eq and 1-neq local morphisms for ChatGPT answers, as shown in Table 3 .The following observations can be made:• For one morphism: "Unused Statements", "Modify Condition" and "Boolean Exchange" are particularly difficult to identify. One possible reason for this difficulty is that these morphisms involve subtle changes or transformations in the code that may not be easily recognizable based on the surrounding context alone. It requires a more nuanced understanding of the code logic and structure to detect these morphisms• For 2-eq morphisms: The observation that "Unused Statements" remains challenging when combined with other morphisms suggests that the presence of multiple morphisms in the code can further complicate the identification task.• Dataset-specific difficulties: The results indicate that the difficulty of identifying "Unused Statements" persists across datasets, suggesting that this morphism type poses inherent challenges in code comprehension. On the other hand, the specific difficulty in identifying "Modify Condition" in the MathQA dataset may be attributed to the nature of the dataset itself, which focuses on math problem solvers.Data statistics. We use the filtered data in Table 2 as input, i.e. 159, 953, and 97 Java Snippets for HumanEval-X, MBXP, and MathQA, respectively. Each model generates a Python object and a JavaScript object for each Java object. During test phase, since the Mexeval dataset is missing 3 test cases in the Java split of HumanEval (TaskID: 32, 38, 50), we use 156, 953, and 97 Java functions to calculate the pass@1 rates for HumanEval-X, MBXP, and MathQA respectively.Results statistics. The pass@1 score of different models on different datasets is shown in fails to pass some of the test cases. Additionally, both models also encounter errors related to symbol identification, such as "CannotFindSymbol" and "TypeError". These errors are primarily compilation errors.It's worth further investigating the causes behind these failure types and exploring strategies to improve the models' performance, especially in handling test cases and resolving symbolrelated errors.In this Section, we analyze some classical cases from our three experiments.Refer to Figure 7 and Figure 8 . Those cases show the model can identify the literate differences between the code pairs but fails to clearly interpret their functional difference.According to the previous statistics, most translation errors happen in the compilation stage.We are also concerned about if there are no compile errors, when will the model make mistakes. Figure 9 shows such a case where some information about the numerical data type is missing when translation.Figure 10 shows a case where information loss and hallucination happen during explanation and reproduction. This is often the case when the model deals with the MathQA dataset. To improve the performance of the model, using few-shot prompts to illustrate how to explain the code may be useful.We can also extend the definition of the object to be the set of code with not only the same outcome, but also with the same time or space complexity, but due to the difficulties of automatically evaluating the complexity, this can be extended for future work.The code can be found in https://github.com/scorpio-nova/CatCode.https://huggingface.co/datasets/THUDM/humaneval-x 


Decision: This paper presents CatCode, which uses the category theory to design a framework for evaluation of coding tasks. The benefit of this framework is to unify coding benchmarks of different formats, and provide a flexible way to compose new tasks.

The proposed evaluation framework is novel and interesting, and the authors made great efforts to communicate with reviewers during the rebuttal and discussion period. I think this work can be valuable to the conference, and thus recommend acceptance. I encourage the authors to revise the paper and include the results and discussion they promised to add in the author response.

Reviews: review: paper_summary: paper_summary: In this paper, the authors introduce a novel evaluation framework called CatCode, which is based on category theory principles. CatCode offers a comprehensive and mathematically abstract approach to evaluating the performance of large language models (LLMs) in tasks involving the understanding and generation of a mixture of code and natural language text.

One of the key contributions of this work is the development of a standardized automatic evaluation platform that is adaptive to new datasets, tasks, and models. The authors evaluate ChatGPT, Text-Davinci and CodeGeeX model using the proposed CatCode framework, providing valuable insights into the current strengths and limitations of these models. Notably, they identify a significant deficiency in the ability of existing models to recognize functionally equivalent code and to preserve information about code functionality when generating explanations or descriptions of the code.
strength_weakness: strength_weakness: I don't have any major reason to reject this paper but there are a few areas of improvements

- It's important to study if model's ability to handle different transformations improve across the board. However, authors don't present any correlation among performance across transformations. If a model does well on X (variable renaming), does it also do well on (loop exchange)? I think such analysis will add a lot of value to this work.  
- Code transformations are extensively studied under robustness areas in code generation space [1] so it's important to connect this paper to robustness literature. 




References: 
[1] Wang, Shiqi, et al. "ReCode: Robustness Evaluation of Code Generation Models." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
questions: questions: - I know it's a naive question but it's an important one. How do you ensure that transformations are applied correctly? Have you tried executing code post AST transformation and obtained the same execution accuracy? 

- Given that both Text-Davinci, ChatGPT are OpenAI models and hence are most likely trained on similar sources of data, I am not surprised to see similar performance on multiple tasks. It would be interesting to conduct this analysis with some models other than OpenAI. In particular, can you run experiments with one additional model from Anthropic, Mixtral, Llama models for broader coverage?
paper_summary: paper_summary: This paper discusses methods to evaluate the abilities of LLMs in solving coding problems that involve a combination of code and text. The authors introduce a novel evaluation perspective for code-related tasks. The key idea is to incorporate category theory, which aims to provide a comprehensive and mathematically abstract framework to evaluate LLMs, encompassing a wide range of code-related tasks. An advantage of the proposed framework is its adaptability to new datasets, tasks, and models, making it a potential standardized automatic evaluation platform. This paper evaluates competitive LLMs to identify current models' deficiencies in recognizing functionally equivalent code and preserving information about code function between the code and its explanation.
strength_weakness: strength_weakness: * The idea of incorporating category theory is interesting and unique. However, the merit of incorporating category theory for the evaluation was not strongly supported by the experiments (if my understanding is correct. See the questions to authors section. I am willing to increase my score if I have misunderstood and the authors convince me in the rebuttal.)

* The authors may need to show results of simple baseline methods without applying the proposed method and compare the results under fair conditions. Otherwise, it is hard to understand the actual benefit of using the proposed method.
questions: questions: * In Section 3.2.2, __"For local morphism, we extract the Java functions within HumanEval-X, MBXP, and MathQA datasets. We perform the 9 local morphisms based on AST transformations."__ and __"For global morphisms, since the three datasets above do not contain multiple solutions to the same problem using the same PL, we complement with code from the test split of Code Contest(Li et al., 2022) dataset."__
If I am not mistaken, it does not seem to provide clear numbers of questions for the local morphism task in the three datasets. This is also the case for the global morphism task conducted in this paper. Please provide more details on the evaluation settings for Section 3.2 so that readers can reproduce the experiments.


* Three different experimental results, namely Morphism Identification, Translation Functor, and Explanation Functor, conducted in this paper were presented in Sections 3.2.3, 3.3.3, and 3.4.3. However, the benefit of using the proposed method, specifically incorporating category theory, is not clearly described, particularly for Sections 3.3.3 and 3.4.3. Could you explain how readers can interpret the benefit of the proposed method from these results?

* In conclusion, the authors mention, __"Based on categorical perspectives, we present a standardized automatic evaluation platform, which is adaptive to new datasets, tasks, and models."__  However, I am not sure if this paper clearly demonstrates the ability to adapt to new tasks, as it mainly shows the adaptation ability for datasets and models.
paper_summary: paper_summary: This paper proposes an automated evaluation framework called CatCode, designed to assess the capabilities of various large-scale language models, such as ChatGPT, in solving programming problems. The framework leverages category theory as a mathematical abstraction to evaluate code-related tasks and introduces concepts such as code debugging and transformation, code translation, code generation, and interpretation. This approach is comprehensive and standardized, allowing for the evaluation of mixed-code tasks of different types.

In terms of quality, the paper provides clear definitions and detailed explanations, enabling readers to comprehend and apply the CatCode framework.

In terms of originality, the paper introduces a novel evaluation framework that addresses limitations within existing evaluation methods by utilizing the principles of category theory.
strength_weakness: strength_weakness: Relies only on existing small datasets for validation, requiring further experimental validation and refinement of the framework, such as in the case of Codescope.

The framework introduces a difference in the Morphism Identification task compared to previous mainstream code evaluation tasks, while the other two tasks remain consistent or a combination of previous tasks.

Comments: title: More details on the questions concerned
Thanks for additional expriments
comment: Dear reviewer, 

We would like to first thank you again for your constructive comments and valuable suggestions! Since we are nearly at the end of the discussion phase, we wish to clarify and expand on a few points based on your queries.

1. Regarding question 1 on the experimental details for the morphism identification task:

   - Numbers of questions for the morphism identification task

     (1) Local morphism task of the three datasets:

     | question type  | Humaneval | MBXP | MathQA |
     | ----- | --------- | ---- | ------ |
     | 1-eq  | 318       | 1896 | 3237   |
     | 2-eq  | 159       | 953  | 1615   |
     | 1-neq | 173       | 1010 | 1418   |
     | total | 650       | 3859 | 6270   |

     (2) For the global morphism task in the `CodeContest` dataset, there are 97 #global-eq and 366 #global-neq questions.

     (3) In Appendix A, we provide more details about processing the question for fair evaluation(e.g.removing code comments that may leak some information for identification), and the filtering policy to make the number of different morphism types more balanced. 

   - More reproduction details: In the appendix, we provide (1) a link to anonymized code and (2) 5 pages in Appendices A&B detailing the reproduction steps and experimental statistics. We acknowledge that the main text was less detailed which may let you confused. We are also willing to update the main text with those important data statistics and reproducibility-related information as you suggested :)

2. For question 2: We want to point out that CatCode is the first paper to highlight the "task unification" and "task composition" in the field of code-related LLM evaluation. The benefits of the CatCode are that

   - **For LLMs**: The paper underscores the importance of recognizing functional equivalence under varying lexical similarities—a critical challenge in coding. It also illustrates how LLMs might struggle with composite tasks despite performing well on singular tasks.
   - **For new tasks**: As elaborated in the Rebuttal, category theory provides the **capacity to define new challenging tasks** E.g. morphism-identification, composite code reproduction task in the paper; debug+translate task for CodeScope; and can support more new tasks in the future (Please refer to rebuttal for Q1 for detailed elaboration and more examples).
   - **For existing datasets**: The unified math abstraction and standard pipeline can **adapt existing datasets to those new tasks automatically and systematically**. This can alleviate high-quality datasets from saturation, and reduce human labor to create new datasets for new tasks.
   - The above benefits are unique contributions to supplement previous evaluations, specifically there are the case for two classical previous work

       1. CodeXGLUE (Shuai Lu *et al.*, 2021) offers a comprehensive benchmark. From its leaderboard (https://microsoft.github.io/CodeXGLUE/), we can see researchers following on single tasks, but there's currently no following work that conducts all the tasks to obtain the overall performance. It's partly due to the complexity brought by the lack of standardness: they use almost completely different datasets, data formats, and pipelines for different tasks (ref: the first table in https://github.com/microsoft/CodeXGLUE). **Our work addresses this by introducing unified task definitions and a standard pipeline, making it easier for researchers to follow.**

       2. CodeScope(W Yan *et al.*, 2024) is more standard in the model API and data format, but they don't further build the connection among coding tasks or include any composite tasks (https://github.com/WeixiangYAN/CodeScope). With people increasingly relying on LLMs to conduct composite coding tasks (e.g. We may ask LLMs to generate the code, debug the code, and write docstrings in the same dialog), and we've found by experiments that the composite tasks are hard. Therefore, **the category framework fills this gap, facilitating composite task definition and adapting datasets to them in an automatic way.**

      In summary, our contributions are **both theoretical**—providing a unified mathematical abstraction for understanding the relationships among coding tasks—**and practical**, by standardizing the evaluation pipeline to enable automatic and systematic composite task assessments.

Again, we sincerely appreciate your insightful questions and feedback, which certainly help refine our work on quality and clarity. We hope our responses adequately address your queries, and are eager to address any further questions or concerns you may have :)
Dear Reviewer,

Thank you very much for your supportive feedback and for recognizing the efforts made to address your questions. 

We deeply appreciate your valuable feedback and the opportunity to refine our work further :)
Thanks for answering my questions and conducting additional experiments. As indicated by my initial score, I think it's a good paper so I will keep my score with accept decision.
Dear Reviewer,

Thank you for acknowledging the response and for considering an increase in your evaluation score! We deeply appreciate your valuable feedback and the opportunity to refine our work further.

We are committed to ensuring the highest quality of our research and welcome any additional suggestions you might have that could further improve our paper.

Thank you once again for your constructive critique and your acknowledgment of our work :)
The author addressed some of my concerns, and I am willing to increase the score from 6 to 7.