CatCode: A Comprehensive Evaluation Framework for LLMs On the Mixture of Code and Text
Authors: Zhenru Lin
Yiqun Yao
Yang Yuan
Venue: COLM (2024)
Keywords: Category Theory,LLM,Code,Evaluation
TL;DR: A standard, unified and scalable framework that can support the evaluation for complex coding tasks

Abstract: Large language models (LLMs) such as ChatGPT are increasingly proficient in understanding and generating a mixture of code and text. Evaluation based on such *mixture* can lead to a more comprehensive understanding of the models' abilities in solving coding problems. However, in this context, current evaluation methods are either limited in task coverage or lack standardization. To address this issue, we propose using category theory as a framework for evaluation. Specifically, morphisms within a code category can represent code debugging and transformation, functors between two categories represent code translation, and functors between a code category and a natural language category represent code generation, explanation, and reproduction. We present an automatic evaluation framework called **CatCode** (**Cat**egory **Code**) that can comprehensively assess the coding abilities of LLMs, including ChatGPT, Text-Davinci, and CodeGeeX.
Large language models (LLMs) are increasingly proficient in understanding and generating a mixture of code and text. Evaluation based on such *mixture* can lead to a more comprehensive understanding of the models' abilities in solving coding problems. However, current evaluation methods are either limited in task coverage or lack standardization. To address this issue, we propose to apply category theory as math abstraction for code-related evaluation. Specifically, morphisms within a code category can represent code debugging and transformation, functors between two categories represent code translation, and functors between a code category and a natural language category represent code generation and explanation. We present an automatic evaluation framework called **CatCode** (**Cat**egory *Code*) that can assess the coding abilities of various ChatGPT-like LLMs in a *comprehensive* and *standard* way, and further support *composite* task evaluation. The code can be found in https://github.com/scorpio-nova/CatCode.

Published as a conference paper at COLM 2024
CatCode: A Comprehensive Evaluation Framework for LLMs On the Mixture of Code and Text
Zhenru Lin
Yiqun Yao Beijing Academy of Artificial Intelligence yqyaoêbaai ac.
III, Tsinghua University 1zr22@mails. tsinghua.edis weighua.edu.cc cn
Yang Yuan"
IIIS, Tsinghua University
Shanghai Artificial Intelligence Laboratory Shanghai Qi Zhi Institute
yuanyang@tsinghua. edu. cn
Abstract
Large language models (LLMs) are increasingly proficient in understanding and generating mixture of code and text. Evaluation based on such mixture can lead to more comprehensive understanding of the models' abilities in solving coding problems. However, current evaluation methods are either limited in task coverage or lack standardization. propose to apply category theory as math abstraction for code-related eval- uation. Specifically, morphisms within a code category can represent code debugging and transformation, functors between two categories represent code translation, and functors between code category and a natural lan- guage category represent code generation and explanation. We present an automatic evaluation framework called CatCode (Category Code) that can assess the coding abilities of various ChatGl'T-like LLMs in comprehensive and standard way, and further support composite task evaluation. The code can be found at https:/ / /githuu..coo htabbcco////eec hetp://oooooooo/Calle CatCodee.
address this issue, we
1 Introduction
The success of large language models (LLMs) as programming assistants has been widely acknowledged, with their higher proficiency demonstrated in various coding tasks such as code generation (Li et al., 2022 Svyatkovskiy et al. 2020), code explanation (MacNeil eet al. 2022), and code translation (Zhu et al. 2022). However, what truly sets the recent LLMs apart is their ability, akin to ChatGP Tike models, to comprehend and align with human intents by processing mixlure of natural language and code. This unique attribute significantly lowers the entry barrier for users, leading to their widespread adoption and notable achievements.
To assess the coding ability of LLMs, numerous efforts have been made by researchers. How- ever, current evaluation methods are either limited in task coverage or lack standardization. Match-based automatic frameworks such as CodeBLEU (Ren et al. (0200) rely primarily on similarity scores to evaluate the quality of code. However, such frameworks often fail to capture the nuances of code functionality and meaning. Execution-based evaluation methods, for example, MBXP (Athiwaratkun et al. (2022) and MultiPL.E (Cassano et al., 2023), can evaluate the function accuracy of code, but they primarily focus on code genera- fion and translation tasks. Task-based evaluation frameworks like CodeXGLUE (Lu et al., 2021) offer comprehensive approach but lack standardization due to completely different datasets, task formulations, and APIs. Consequently, we still lack an evaluation framework to adapt to the context of mixture of natural language and code, and there is need to
* *Corresponding author.

---

Published as a conference paper at COLM 2024
Step 1:
Construct Category
Step Categorical Task Formulation
Step 3:
Model Evaluation
Reformat the code datasets
Use categorical perspective to define coding tasks with mixture of natural language and code
Evaluate the model's ability by morphisms and functors morphism identification
by objects and apply
different morphisms
Reformat datasets based on the definition of different objects
Original Tasks
Categorical Perspective
(A.A)
(A. (A.C)
Code detection
object
object C
Defect detection
Object morphisms
Code repuir
model as fuuncto
kmbke
Cyrna
Cryshen
Code translation Docstring translation
Translation functor
model the combination of e'eppannatio & reproduction functor Cjusa
Applying morphisms for data augmentation
Code explanation
Explanation functor
Code summarization
Congitin
(new)
wff sorghisn
Code generation Code reproduction
Generation functor Reproduction functor
B'
( mow) C
Figure 1: The overall evaluation framework. We category perspectives to reorganize and transform data, formulate different coding tasks, and conduct model evaluations.
establish comprehensive evaluation framework that not only supports diverse and novel task definitions but also provides standardized approach.
We aim to establish comprehensive theoretical framework that can be open-sourced and applied essential coding tasks, providing standardized automatic evaluation metrics. It is challenging task without theoretical guidance. Therefore, we seek theory that can effectively express the structural aspects of code, language, and their interrelationships. In this context, we find category theory, a branch of mathematics, to be particularly suitable for describing relationships between various elements.
By utilizing category theory's mathematical abstractions, we can gain insights into the relationships among different programming languages (PLs) and natural languages (NLs). We consider PLs and NLs as categories, and functionally equivalent programs as objects, and leverage functors and morphisms to capture the object relations within and across these categories. This gives unified framework for describing functional equivalence, which not only works within a single PL, but also among different PLs, and even between PLs and NL.s. By learning the morphisms within programming language category, the model can grasp the similarities and differences between code snippets. Additionally, by acquiring knowledge of the functors between categories, the model can understand the relationship between different programming languages and natural languages.
Based on the categorical perspective above, we build standardized evaluation framework (see [igure It is standardized in terms of data definition, task formulation, and APIs. This framework can be extended to many code-related tasks as long as categorical definition is given. In our experiments, we give some examples of common code-related tasks, and assess models such as ChatGPT and CodeGeeX for their capabilities in identifying functional equivalence in code, performing code translation, generating code explanations, and recon- structing code based on explanations. We have observed that these models still struggle to differentiate between the concepts of "functional equivalence" and "similarity" in code. While they demonstrated relatively satisfactory performance in code translation, maintain- ing functional equivalence between code and its corresponding explanations remains persistent challenge.
Our main contributions can be listed as follows:
2

---

Published as a conference paper at COLM 2024
JAVA
SELF-MORPHISM
MORPHISM
TRANSLATION ATION FUNCTOR
B
EXPLANATION FUNCTOR
REPRODUCTION FUNCTOR
ENGLISH
PYTHON
Figure 2: Categorical framework for mixture of code and NL. A, and represent different objects, A and A" represent the equivalent object of A in other categories.
We introduce CatCode, novel evaluation perspective for code-related tasks based on category theory, which provides a comprehensive framework that encompasses a wide range of code-related task formulations.
We present a standardized automatic evaluation platform within the CatCode framework, that offers a quantitative assessment of the coding abilities of Language Models (LLMs) and can adapt to various datasets and models, which will be publicly available.
We evaluate competitive LLMs, including ChatGPT, StarCoder and CodeGeeX, providing insights into their strengths and limitations in understanding the mixture of NL and code.
Methods
Generally, it is difficult achieve both comprehensiveness and standardization. In Sec- tion 2.1 we show that the categorical perspective, with its emphasis on generalization and abstraction, offers a valuable approach achieving comprehensiveness the mixture of code and natural language. In Section 2.2 we discuss the significance of standardization and outline strategies for achieving it.
dealing with
2.1 Comprehensive Categorical Perspective
Category theory studies how structures arising in different areas can be described with common language. It was originally developed to transport ideas from one branch of mathematics to another, e.g. from topology algebra. Category Theory has been applied
to different fields, ranging from functional programming to logic, software design, and LingusstccsSpiiak 2014; Ahrens & Wullaert 2022 Fong & Spivak 2018 Bradley 2018). Here provide the basic defimitions used in work and explain their applications to describe code and natural language. Figure illuustatas those concepts in a comprehensive way.
Definition 2.1. (Category, Object, Morphism). A category C consists of of objects Ob(C) and a set of morphisms Home b), from to b for every a, b E Ob(C). Given two morphisms f E Home (a, b) E Homc(b,c) c), we define their composition as g E Home ca. The composition of morphisms is governed by two axioms:
Asssoiiativit:: If f
b,g b c, and h - then h (g f) (ho of
Identity: For every object x, there exists a morphism idx : x + called the identity morphism for x, such that for every morphism f - b, we have id, f= id
Application 2.1. (Category, Object, Morphism of Code and NL). For describing the mixture of code and natural languages, we first regard them as objects from different categories. We consider the code categories that contain all executable code in certain language:
3

---

Published as a conference paper at COLM 2024
New Dutasct
New Tisk
Neew
Modil
Reformat: the data
Formulone: cotegonical per quertive fermulate the
1. Preproossc enhhed prompss inpuis
2. Porgrocees: extnect code/NL from osipt
Adeptation
pinogeiie and ohjonts (2. Augmariation sclect moephians to apply)
2. Scrialia:: mis and foom the task lpptt
Standandized Pipeline
raw data
data definition
categorical data
model API
test
formulation
oulput
API
results
Platform
data fommatter
prospp selector
parillet
puiling lcil (NL)
Provided
roquers
Functions
predefiree
input
code
execution
merphiems
(code)
Figure 3: Standardized evaluation platform. The central pipeline offers consistent approach for all evaluations. Behind the pipeline, we provide a variety of functions to automatically conduct the most important steps. With our platform released, the pipeline can easily accommodate novel datasets, tasks, and models by following the instructions outlined alongside the grey lines.
Cjavar Cpythwn ... We define the natural language categories which contain all descrip- tion/expo expllnatiio//summaizaaa them C₁, C2
of code etc.: CFnglish CChineser ... For simplicity, let's call
In a category, there are infinitely many different objects. We name these objects 01, 02. Each object contains infinitely many programs, with the same running outcome for each valid input. We use this definition because we focus on the functional equivalence of code function. For example, one may have slightly different implementations of quick sorts, but they represent exactly the same function.
Based on the definition of objects, we define the morphism between two objects, as "neces- sary edits to convert a function to another". In particular, we define the self-morphism for each object, as "edits that do not change the functionality of the program".
Definition 2.2. (Functor). A functor from a category to a category D, written as F:C + D, maps an object E Ob(C)) to F(x) E Ob(D); as well morphism f f:x y in (C) to F(f) : F(x) + F(y) in D, such that the following two properties hold:
For every object in Fidx) id F(x)
For all morphisms f : y and g z = F(f)
Application 2.2. (Functor of Mixture of Code and NL). We define the functor from C1 to Cz, to be the transform from one language to another language, but with the same functionality. When it applies to two code categories, it usually represents code translation, and we define as a "translation functor". " When the functor is between a code category and can define
natural language category, it may have many possible meanings, for instance, "generation functor" from NL to PL that generate code solution to a problem description, "explanation functor" from PL to NL that explains piece of code, and "reproduction functor" from NL to PL that generates code snippets based on code descriptions.
2.2 Standardized Evaluation Platform
To show the practical use of categorical framework and support the community, we open- source our evaluation framwork?] Figure 3 shows the streamlined process of our stan- dardized evaluation platform. Standardization within the framework achieved through focus on three critical aspects: data definition, task formulation, and APls.
We can also extend the definition of the object be the set of code with not only the same outcome, but also with the same time or space complexity, but due the difficulties of automatically evaluating the complexity, this can be extended for future work.
2The code can be found in https::///thhb http//ggtthhh////// wtt.ccco/ooocoooooca/aa////////e /CaCCoee.
4

---

Published as a conference paper at COLM 2024
Data definition The original datasets may consist of mixture of code and natural lan- guages, so a clear data definition is crucial. The code can be in different languages (Java, Python, JavaScript, etc.), and on different levels (single-line, function-level, file-level), Nat- ural language can be in different relationships with the code (problem descriptions, code comments, demands, etc.). By establishing standardized data definition, ensure com- patibility and facilitate the comparison and integration of different datasets and tasks. For implementation, we use "data formatter" to rearrange code based on the definition of objects and categories. Additionally, we provide "predefined morphisms" and implement them based on JavaTransformen (Rabin al. 2019) to automatically apply morphisms to code objects, which makes it easy for data augmentation.
Task formulation We use a categorical perspective formulate diverse code-related tasks using the unified language of objects, morphisms, and functors. A good task formulation allows for more generalized and flexible approach to defining more complex code prob- lems, enabling the inclusion of wide variety of code-related tasks and comparing their similarities and differences. Based on the task formulation, we carefully decide what parts of the code and NL should be the model input, ensuring cleaner setting. Then use "prompt selector" " to select a suitable prompt for task description, and use "input constructor" to combine the data and prompt task input.
APIs After the data is ready, we focus on the standardization of model APIs and test APIs. Standardized APIs promote transparency, fairness, and efficiency in the evaluation. By defining a clear and consistent set of APIs, we have integrated OpenAl models into the evaluation process and can test them using parallel requests. For postprocessing the model's output, we enable the extraction and filtering of plain text from the code-text mixture. For test APIs, we provide a "pairing test" API for evaluating the model's answer based on natural language, and an "execution test" API, which connects to Mxeval (Athwaaratkun et 2022) for compiling and running the tests for a given code.
3 Experiments
3.1 Use category framework for evaluation
In this section, we exhibit some experimental examples of how to use our platform. Mean- while, we explore the following three types of tasks: discriminant task, generation task, and composite task, that correspond to those illustrated in Step of Figure
Discriminant task: Can the model exactly capture code meanings and identify simi- lar/different code?
(Related tasks: code detection, defect detection, code repair)
+ Categorical perspective: Can the model identify the self-morphisms and other morphisms within the same code category?
Generation task: Can the model translate code between programming languages?
(Related tasks: code translation)
+ Categorical perspective: Can the model accurately perform code translation functor?
Composite task: Can the model reproduce the code after equivalent transformations?
(Related tasks: code explanation, code summarization, code generation, code reproduction)
+ Categorical perspective: Can the model preserve the functional equivalence after apply- ing explanation functor and then reproduction functor?
We conduct three experiments from categorical aspects accordingly. For a common setting, use three multilingual datasets: LalmanEval-X(Zheerl et al. 2023), (thiwaraiaa ee al.) 2022), and MathQA(Amini et al. 2019 Athiwaratkun et al. 2022) throughout the three experiments, and use live models: Text- Davinci-003, gpt3.5-turbo, .lamm-2-7b-chat-
5

---

Published as a conference paper at COLM 2024
100
Prociston())
60
plobal morpoet
coupttion
dj
80
d=1
10
1-90
1-neq
globai-oq
global-nea
CharGPT
Figure Morphism Identification Experiment, "1", "2" and "global" stands for the distance of the code. "Eq" and "neq" indicates w whether the morphism is self-morphism. (Left) An illustration of morphisms and the definition of object distance. (Right) Comparison of Text-Davinci and ChatGPT for morphism identification.
hf, starchat-bela (Text-Davinci, ChatGPT, Llama2 and StarCoder for short accordingly) as common baseline models. For more detailed experimental settings and interesting results about specific model performance, please refer to Appendix and B
3.2 Experiment 1: Morphism Identification Within Code Category
3.2.1 Categorical Perspective Settings
Code Objects We conduct the experiments within the PL category and define "the function with the same running outcome for every valid input" as the same object.
Code Morphism The general category perspectives only define morphisms between two code snippets, but morphisms do not have distance information. Considering fine-grained evaluation, are curious about the model's ability to modify or debug the code, which corresponds to the local scale; and to write an equivalent new solution to a coding problem, which is of global scale.
Object Distance To test the model on both local and global scales, we define the "distance" between two codes. As illustrated in Figure 4 In a PL Category functional equivalence. ă = {x E C, af means the set of all code that has the same functionality a. In this category, and are two different objects. Within a, all code instances are functionally equivalent, e.g. a; and aj. We can apply predefined morphisms to the objects. For example, we can apply 1-step equivalent morphisms(1 morphism) on a; to get new program instances still in a, or apply 1-step nonequivalent morphism(1- neq morphism) get new instance in another equivalent class b. Distance represents the minimal number of morphisms needed to transform one program to another given predefined morphism set. For example, Applying two different self-morphism on a; returns at and a7, then we need least two steps to transform from a! to 47. Those are local-scale equivalence. If it is hard to transform to aj within just few steps(e.g., or 2 steps as illustrated here), we treat them global-scale equivalence.
use to express
3.2.2 Implementation
Dataset For local morphism, we extract the Java functions within HumanBval-X, MBXP, and MathQA datasets. We perform the 9 local morphisms based on AST transformations.
Variable Renaming (VR): rename a variable, with a new name randomly chosen from all variable names in the dataset, or use "varN" " for simplicity
Boolean Exchange (BE): propagate the exchange of "true" and "false" throughout the method and preserve the same logic
3. Loop Exchange (LE): exchange for loops and while loops
4. Switch To If (SI): replace switch statement with an equivalent if statement
Unused Statement (US): insert an unused string declaration randomly selected basic block in method

---

Published as a conference paper at COLM 2024
6. Reorder Condition (RC): write the condition in reverse order (e.g., change < n to
Permute Statement (PS): swap two independent statements (i.e. without data or control dependence)
Modify Condition ( change the binary operation in the condition (e.g,, change <
to i S n)
9. Remove (RE): delete the else branch of the if-else statement
The first 7 morphisms are adopted from Java Transform Rabin et al., 2019) that conducts functionally equivalent transformations, which means they are self-morphisms. Other than self-morphisms, we add 2 morphisms: Modi Modi F tion and RemoveElse that change the program's 's function.
For global morphisms, since the three datasets above do not contain multiple solutions to the same problem using the same PL, we complement with code from the test split of Code Contest(Li et al. 2022) dataset.
For dataset statistics and the detailed preprocess procedure, please refer to Table 2 in the Appendix.
Models We evaluate Text-Davinci and ChatGPT. We input (code, code) pairs from the datasets generated by morphisms, and ask the model to answer whether they are func- tionally equivalent.
Evaluation We collect the model's yes/no responses and calculate the average precision for different morphisms.
3.2.3 Results
The results are demonstrated in Figure Overall, Text-Davinci and ChatGPT make a tie. Locally, ChatGPT better identifying nonequivalent morphisms, and worse equivalent ones. Globally, Text-Davinci is better at identifying global nonequivalent morphisms. Note that random guess leads to a precision of 50%, so both models behave just slightly better than random guesses for identifying global equivalence.
3.3 Experiment 2: Translation Functor Between Different PL Categories
3.3.1 Categorical Perspective Settings
We investigate the model's ability to perform code translation by utilizing functors between two programming language categories. To isolate the influence of natural language, we exclude problem descriptions and code comments, forcing the models focus solely on translating code from one programming language (PL) category to another. The input prompt may contain a request for translation using natural language (e.g., "translate the below Java code to Python code").
3.3.2 Implementation
Datasets We conduct evaluations on the HumanEval-X, MathQA, and MBXP datasets. The input category is Java, while the output categories are JavaScript and Python.
Models We evaluate five models: Text-Davinci, ChatGPT, CodeGeeX, Llama2 and StarCoder. For CodeGeeX, code translation is supported inherently, so we simply input the Java code without additional text. For the other models, we provide prompts that request the model to do the translation.
Evaluation We extract the functions from the model's responses and assess their correctness using the Pass@1 rates of execution-based tests.
3.3.3 Results
The results are depicted Figure 5, Among the three models, ChatGPT performs the best, exhibiting slight advantage over Texi-Davinci. With regards to the datasets, ChatGPT and
7

---

Published as a conference paper at COLM 2024
100
100
70
Ruaddr (90
30
10
avarage
Humantivll
MatroA
averge
TextDevin ChanGPT CodaGlack BI Liama----7 dh-b chast-t Starchatt-oe
TestCedeccl ChatoPT Gode
Figure 5: Comparison of performance for the model as translation functor between different PL categories. (Left) Translate from Java to JavaScript (Right) Translate from Java to Python.
Text-Davine achieve nearly perfect translation accuracy on MathQA, whose data consists of functions with internal variable initialization, and without any input arguments, loops, or conditional statements. This indicates ChatGPT'ss and Text-Davinci's proficiency in reproducing exact numerical values and handling simple program structures.
3.4 Experiment 3: Explanation Functor and Reproduction Functor Between PL and NL Categories
3.4.1
Categorical Perspective Settings
Code explanation and code generation tasks have been conducted separately in previous work. However, from categorical perspective, we can make a combination. We define the explanation functor as the functor that maps an object from PL category with precise description of the code's function. We define the reproduction functor as the functor from an NL category to reproduce the code of the same function in PL category. If the reproduced code is functionally the same as the original code, we reckon the model preserves the functional equivalence after applying explanation functor and then generation functor.
NL category,
PL category, that uses an explanation object in NL
3.4.2 Implementation
Datasets We conduct the evalution on three datasets: HumanEval-X, MathQA, and MBXP For consistency, we select Java as the programming language (PL) category for all evalua- tions.
Models We evaluate Text-Davinci, ChatGPT, Llama2 and StarCoder, prompting each model twice. In the first prompt, we ask the models to describe the code using natural language, including the precise function name, arguments, and return type, and provide sufficient in- formation to reproduce the code. In the second prompt, provide the model's explanation and ask it to translate the description back into code.
Evaluation To evaluate the correctness of the reproduced code, extract the functions from the model's second responses and assess them using the Pass@1 rates of execution- tests.
3.4.3 Results
The results are depicted in F On the same datasets and with the same two models, the average pass@1 rate about 30% lower compared to that the translation functor. This indicates the model has significant information loss in the explain-reproduce process.
8

---

Published as a conference paper at COLM 2024
100.0
80.0
Passth1 (95)
20.0
0.0
HumanEval
MathQA
MEXP
TextDavincl
CharGPT
1ama-2-7h-chat hf
Starchat-bon berchatt-era
Figure 6: Comparison of model performance for the model as the combination of explanation functor and reproduction functor between PL and NL categories.
3.4.4 Additional Notes on the composite task
As models evolve, scores on high-quality datasets tend to saturate (HumanBval 90%). Should we abandon HumanEval, and call for massive efforts to construct new ones? We believe it's not the only way. With CatCode, the evaluation effective again. The construction of composite tasks is mostly automatic, greatly reducing the labor for collecting
easily define composite tasks, and make
datasets.
Moreover, real-world applications often require model to handle those composite tasks. For example, developers may need to use the model to write new features, and synchronize code updates between Android and iOS versions. To this end, we propose and experiment on new task "repair morphism + translation functor" using the CodeScope dataset. We use the code repair subset from the CodeScope dataset(Yan Yan et 2023) and add a step for the model to translate from C++ C after debugging C Programs. Results on GPT-4 show that pass@1 drops from 26/ 60 in the original task to 18/60 in the composite task, which shows that the composite task is more difficult.
3.5 Summary
Based on our experiments, we have found that both Text-Davinci and ChatGPT models are capable of identifying the impact of one or two local morphisms on code function. However, identifying global morphisms proves to be challenging for these models. In terms of code translation, both ChatGPT and Text-Davinci models outperform other models. They demonstrate accurate reproduction of basic syntax, variable names, and numerical values. Nevertheless, when dealing with more intricate program structures and aligning data types across different programming language categories, these models encounter obstacles that hinder more precise translation.
It is important to note that maintaining functional equivalence between code and natural language presents a more formidable challenge compared to code translation.
Related Work
4.1 Coding Abilities of Large Language Models
Research on large language models trained on code has made significant strides. CodeBERT (Feng et al. 2020) and GraphCodeBERT (Guo et al. 2020) incorporated program Abstract Syntax Trees (AST) and data flow information to enhance code structure and context un- derstanding. The Encoder-Decoded architecture in Code15 (Wang et al.) 2021 and PLBART supports multi-task learning, aiding code translation and generation. More recently, the emergence of Codex (Chen et al. 2021), ChatGPT, and StarCoder et al., 2023) has brought GPT-based models into the mainstream for content generation tasks, in- cluding coding. These models have shown impressive performance across various domains,
(Ahmad et

---

Published as a conference paper at COLM 2024
but their proprietary nature, limited access to data, or lack of model checkpoints have made challenging to thoroughly evaluate and understand their capabilities and limitations. Consequently, there is need to develop comprehensive framework to evaluate these black-box models.
4.2
Code Model Evaluation
Evaluation of code models is diverse, encompassing match-based, task-based, and execution- based frameworks. The match-based approach, exemplified by CodeBLEU et al., 2020), considers n-gram matches and AST similarity but may overlook functional equivalence in varying solutions. Task-based evaluations, like CodeXGLUUELL al. 2021), , offer compre- hensive assessments across coding tasks but lack uniformity in data and task settings, limit- ing scalability. Execution-based evaluation, used in benchmarks like MBXP et al. (022) and MultiPL-E((assaan et al., 2023), judges code correctness through test cases. Our work mainly focuses on providing new framework for unifying the task-based eval- uation. While our work also considers test cases from execution-based benchmarks, our focus extends beyond correctness evaluation include code morphism, PI. NL functor aspects, and the composition of different tasks, for example the code reproduction task can be viewed as the combination of code explanation and generation tasks.
Moreover, code-related models are found to be sensitive to slight edits in the prompts, therefore to evaluate their robustness is crucial. Previous work applies semantics remantiss-preserving transformations code docstrings and program, for better training and evaluation of models Wang et al., 2022 Henkel et al. 2022). Apart from using transformations based on AST edits to ensure accuracy, further investigate the models' robustness by providing finer-grained characterization based on the edit distance of correct and incorrect edits.
5 Conclusion
Our contributions include introducing CatCode as novel evaluation perspective based on category theory, which enables a comprehensive and mathematically abstract approach to evaluate LLMs that excel in understanding and generating a mixture of code and text. Based on categorical perspectives, we present a standardized automatic evaluation platform, which is adaptive to new datasets, tasks, and models. We evaluate competitive LLMs to provide valuable insights and find out current models' deficiency in recognizing functionally equivalent code and preserving information of code function between the code and its explanation. We open-source our platform, hoping to contribute to the comprehensive and standardized evaluation for LLMs by offering categorical perspective to deal with the mixture of code and text. We hope that CatCode inspires further research and development in the field, leading to LLMs that can effectively assist developers in more complex coding tasks and contribute to advancements in artificial intelligence and software engineering as whole.
Discussion
While we strive to apply categorical perspectives to offer a comprehensive and standardized way of evaluation, we find some limitations and room for future work. First, we prompts to instruct the models to act as certain functors, but prompts that express similar meanings will lead to different results, we have not investigated which prompts more suitable for the model or which models are more robust to different prompts.
In this paper, our experiments primarily focus objects, morphisms, functors, and their combinations. For future work, more powerful concepts and properties in category theory can be used for more complex and composite coding tasks. For example, can further investigate the setting of composing many morphisms and many functors, to further explore the commutative law and isomorphism properties of categories.
10

---

Published as a conference paper at COLM 2024
References
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 2655-2668. Association for Computational Linguistics, 2021. doi: 10.18653 maa.22222aaaammaaiiiaaaaaaaaaa
URL https
//doi.org/18, 8653/v1/2021 naacl main. 211
Benedikt Ahrens and Kobe Wullaert. Category theory for programming. CoRR, hbs//220.00155,, 2022. doi: 10.48550/ 10.448550/rXiv.22000.122 arXiv.200.001259 URL https: 0Tg/10.48550)
arXiv.2209.01259
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. arv preprint ar arXiio1905513319,
2019. 2019.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. Multi-lingual evaluation of code generation models. CoRR, abs 22/./2220..118886 2022. doi: 10.48550 25550//2Xiv.2200146668 URL /tRps:::s///ooiorrr org/10 48550/ 2210 14868
Tai-Danae Bradley. What is applied category theory? arxiv preprint arXiv:1809.05923, 2018, 2018.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. JEEE Transactions on Software Engineering, pp. 1-17, 2023. ISSN 0098-5589, 1939-3520, 2326-3881. doi: 10.1109/TSE.2023 3267446. Uthhttts://ieeexporrr ieee. www..eee......cccceeee///e1177
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL URLLittps:///xxii.orr//bb/// 03374
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Lin- jun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. ar preprint arXiv:2002.08155, 2020.
Brendan Fong and David I Spivak. Seven sketches in compositionality: An invitation to applied category theory. arXiv preprint arXiv:1803.0533 2018.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. arxiv preprint trXiv:2009.08366 2020.
11

---

Published as a conference paper at COLM 2024
Jordan Henkel, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. Semantic robustness of models of source code. In 2022 IEEE International Conference Software Analysis, Evolution and Reengineering (SANER), pp. 526-537. IEEE, 2022.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muh- tasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Ku- nakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378(6624): wwwssceieecc..cccecee..e .ew.l.rg///ooe prrg/doi/abs.
1092-1097, 2022. doi: 10.1126/ science.abql158 URL https: https://wwssccencee.... 10.11155/cieece..bbb/
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint orXiv:2102.04664, 2021.
Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. Generating diverse code explanations using the GPT-3 large language model. In Jan Vahrenhold, Kathi Fisler, Matthias Hauswirth, and Diana Franklin (eds.), ICER 2022: ACM Conference on International Computing Education Research, Lugano and Virtual Event Swvitzerland, August 7 11, 2022, Volume 2, pp. 37-39. ACM, 2022. doi: 10.1145 3544280. URL http::/s:///do.. orw.aa..rrrr/... 1040/.1115355511 000445/5511100 3544280
Md. Rafiqul Islam Rabin, Ke Wang, and Mohammad Amin Alipour. Testing neural programs. CoRR, /bbs//990..0111.. 2019. URL http::/aaaxxw...rrg///////////// htt.://rg////////g//// 10711
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: method for automatic evaluation of code synthesis. CoRR, abs9/2009.10299 2020. URL
10297
David Spivak. Category Theory for the Sciences. MIT Press, 2014. ISBN 9780262228144 URL https://mitpreest: t. /ttp:///oooo///ttttttttttterrr-e sciencess
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode code generation using transformer. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (eds.), ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8- 13, 2020, pp. 1433-1443. ACM, 2020. doi: 10.1145/ /3383333338100000 URL 00/////01111
compose:
https::/doi., orgg 10.10111 15511115338888.44444
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. Recode: Robustness evalua- tion of code generation models. arXiv preprint ar XrXi::2212. 10264, 2022.
12

---

Published as a conference paper at COLM 2024
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-awwre unified pre-trained encoder-decoder models for code understanding and generation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 8696-8708, Association for Computational Linguistics, 2021. doi: 10.18653/v1/2000 emnlp-main.6855 6855. URL http:ttt//ioooorreet/
18653/v1/202/ .ww.:21meemmmpmmiin... 685
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Shuiguang Deng, and Hari Sundaram. Codescope: An execution- based multilingual multitask multidimensional benchmark for evaluating ilms on code understanding and generation, 2023.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang. and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023.
Ming Zhu, Karthik Suresh, and Chandan K Reddy. Multilingual code snippets training for program translation. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp. 11783-11790, 2022.
13

---

Published as a conference paper at COLM 2024
Appendix
The appendix is divided into three main sections, each focusing on distinct aspects of the study. These sections are labeled as follows:
A. Implementation Details: This section covers the implementation process, including the GitHub link our code, dataset details, data filtering, model specifications, and evaluation metrics, providing information on how to reproduce based on our standard evaluation pipeline.
B. Experimental Details: This part introduces the experimental settings and more experi- mental results in detail. We further discuss the model's performance on different morphism types in the identification experiment, and the common error types in the translation and reproduction experiments.
C. Case Study: This section offers some intuitive examples of the experiments. Through analysis of those examples, we can find some limitations of current models, prompts, and test scripts.
A Implementation Details
We have uploaded the code GitHub: https:/ hhpps://gitie http:/ggtthhhhoooo cwto.ooooc/eeeeeeee hom///oppioonoa//CC /otteCee A README md file is provided, offering step-by-step instructions on how to use the evaluation platform and replicate the entire evaluation process in a standardized way.
In this section, we will provide additional details regarding the reproduction process, including the dataset split, data filtering, models used, and evaluation metrics.
A.1 Dataset Details
The following datasets were used in our evaluation:
HumanEval-X4B human-crafted dataset with 164 problems, correct solutions in 5 dif- ferent languages, and test cases. It is benchmark for evaluating code generation and translation models. We use the Java split of HumanEval-X input for applying mor- phism, code translation, and code explanation. Although Mxeval provide multilingual solutions, they are generated by models and do not make a distinction between correct and wrong solutions. In contrast, HumanEval-X provides correct solutions, making it more suitable complement.
MBXP: benchmark for evaluating code generation models. It covers over 10 program- ming languages and is generated using scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. It's dataset similar to HumanEval-X but does not ensure the correctness of the transpiled code.
MathQA: dataset for evaluating math problem solvers. All functions in this dataset have no input argument and return a number. The functions first initialize variables, then do some calculations, and do not use any loops or conditional statements. Compared to HumanEval-X and MBXP, the code in MathQA is simpler in structure and functionality,
Code Contest: a large-scale competitive multi-language programming dataset used for training AlphaCode(Li et al. 2022). It consists of programming problems, test cases in the form of paired inputs and outputs, and both correct and incorrect solutions. Since the aforementioned datasets do not provide multiple solutions within the same programming language, Code Contest serves as a valuable complement by offering multiple solutions to the same problem, representing global equivalence and non-equivalence
Data filtering
14

---

Published as a conference paper at COLM 2024
Model
Morphism Translation Functor
Explanation Functor
Reproduction Functor
Davinci
ChatGPT
CodeGeeX
LLAMM2-7b
StarCoder
Table 1: Baseline Models and their supported evaluation tasks.
Only the Java split of HumanBval-X, MBXT, and MathQA datasets are used. The local morphism is applied by ravarransfrrmrr by AST edits at the function level. To use Javalransffrmer, code must compile successfully and contain a single function. We filter out code in the Mxeval datasets that are generated by the model and cannot compile or contain multiple functions.
Due to the large size of the Code Contest dataset, only the test set is used. Considering that the maximum token length of Text-Davinci is 2048, we filter out solutions with length greater than 500. This ensures that the code pair length remains I 1000, allowing for additional tokens for prompt content and response. After filtering, 97 problems remain, and a problem may have multiple correct and incorrect solutions different languages. We utilize the Java and Python splits of this dataset.
A.2 Model details
We utilize the following baseline models by making API calls to their official endpoints: Text- Davinci-003 (referred to Davinci), ChatGPT,, CodeGeeX. The default hyperparameters of these models are used, with the exception of setting max token to 500 when an early stop of the answer is observed.
Table Tablepproo provides an overview of the tasks supported by each model. Since Text-Davinci- 003 and ChatGPT support various types of text inputs and outputs, they are capable of performing all tasks. On the other hand, the CodeGeeX API does not support natural language output,
only employ it in translation functor experiments.
A.3 Evaluation Metrics
To assess the performance of the model, we conduct two types of tests: pairing test and execution-based test.
The pairing test is utilized for morphism identification. The task involves comparing two code snippets and requesting the model to explicitly answer "True" or "False", along with providing a comparison. For automatic evaluation, we extract the True" or "False" answer. The precision score is used to evaluate the model's ability to identify different equivalence TP
classes. For equivalent objects, the precision is calculated as Precision(eq) TP+EN while for nonequivalent objects, the precision is calculated as Precision TN+FP
TN
The execution-baat -based test is employed for both translation and reproduction experiments. We extract the functions from the model's responses and evaluate their correctness using the Pass@1 rates of execution-base code produces the same expected results as the original code.
tests. This test ensures that the translated /reproduced
B Experimental Details
B.1 Morphism Identification
Data statistics. Table the statistics of the raw data, filtered data, and constructed pairs for the morphism identification experiment. The filtering strategy, described in Sec- tion A.1 involves removing comments and docstrings, resulting in filtered data consisting
15

---

Published as a conference paper at COLM 2024
Dataset
#after filter
#raw data
#equivalent pair #nonequivalent pair
Humaneval-X
159
164
477
173
MBXP
953
974
2849
1010
MathQA
1734
1881
4852
1418
Code Contest
97
/ 164
97
366
Table 2: Dataset statistics for morphism identification experiment.
HvmanEval
MathQA
MBXP
BE-VR (66.67)
MC (45.66)
BE (33.33)
US (74.74)
RC-US (85.71)
PS-US (69.57)
LE-RC (78.57)
PS-US (87.84)
RC-US (70.39)
Table 3:
Morphism types with the three worst average precision scores of each dataset. As an example for notations, BE-VR (66.67) stands for 2-eq morphism of "Boolean Exchange" and "Variable Renaming" with a precision score of 66.67%.
solely Java code. The constructed pairs are generated through sampling. It worth noting that certain morphisms can have multiple outputs for a given input. For instance, the Vabiable Renaming morphism can be applied to all variables a function, resulting in multiple outputs equal to the number of variables. To balance the number of outputs across different morphism types, random sampling is employed when there are more than two applicable morphism types. For each code sample, two types of morphisms are first sampled, followed by sampling one output for each morphism type.
Discussion of different morphism types.
Which morphism types are harder to be identified, and are they hard across datasets? We calculate the average precision scores for 1-eq, 2-eq and 1-neq local morphisms for ChatGPT answers, as shown in Table 3
The following observations be made:
"Unused Statements", "Modify Condition" and "Boolean Exchange" are particularly difficult to identify. One possible reason for this difficulty is that these morphisms involve subtle changes or transformations in the code that may not be eas- ily recognizable based on the surrounding context alone. It requires more nuanced understanding of the code logic and structure to detect these morphisms
For one morphism:
For 2-eq morphisms: The observation that "Unused Statements" remains challenging when combined with other morphisms suggests that the presence of multiple morphisms in the code can further complicate the identification task.
Dataset-specific difficulties: The results indicate that the difficulty identifying "Unused Statements" persists across datasets, suggesting that this morphism type poses inherent challenges in code comprehension. On the other hand, the specific difficulty in identifying "Modify Condition" in the MathQA dataset may be attributed the nature of the dataset itself, which focuses on math problem solvers.
B.2 Translation Functor
Data statistics. We use the filtered data in Table input, i.e. 159, 953, and 97 Java Snippets for HumanEval-X, MBXP, and MathQA, respectively. Each model generates Python object and a JavaScript object for each Java object. During test phase, since the Mexeval dataset is missing test cases in the Java split of HumanEval (TaskID: 32, 38, 50), we use 156, 953, and 97 Java functions to calculate the pass@1 rates for HumanEval-X. MBXP, and MathQA respectively.
Results statistics. The pass@1 score of different models on different datasets is shown in Table Overall, ChatGPT has relatively higher translation ability.
16

---

Published as a conference paper at COLM 2024
HumanEval
MathQA
MBXP
Model
Avg.
Java->Py
Java->JS
Java->Py
Java-SJS
Java->Py
Java-SJS
Text-Davinci 80.8
79.5
99.4
99.1
66.6
64.4
81.6
ChatGPT 83.0
85.8
99.2
99.4
69.0
72.1
84.8
CodeGeeX 24.5
20.5
70.5
72.2
29.2
32.8
41.6
Table 4: Translation pass@1 score (%) statistics, corresponding the Figure 5 (Left). The models translate Java code to two languages Python (Py) and JavaScript (JS), on three datasets. The average (Avg.) is the average score over the six scores.
Model
HumanEval
MatiQA
MBXP
Java->Py
Java->JS
Jnva->Py
Jnva->JS
Iavac>Py
Java->JS
Unknown (10)
ReferenzeeErr, (24)
Unknown (9)
Test Failure (10)
Unknown(1
Referencefir. (285)
Davinci
NameErr. (7)
Test Failure (18)
IndentationEir. (1)
ReferencoErr. (2)
NamoEir, (47)
Test Failure (174)
Test Failure (2)
Typelin, (5)
Namelier. (1)
Syntaxlirr. (2)
Referencelirr. (26)
Typelimr. (41)
Unknown (14)
Referene:Err (21)
Unknown
Test Fallure (9)
Unknown (145)
ReferenceErr. (247)
ChatGPT
NamelIm.
Fuilure (17)
Syntaxfirr (4)
SyntaxEim:
Namefirr
Failure (101)
SyntaxErr. (2)
SyntaxErr.
Nametir.
(0)
SynlaxErr. (15)
SyntaxErr.
Unknown (50)
Test Failure Referencelir. (330)
Test Failure (394)
Unknown (175)
ReferenceEir. (559)
CodeGooX
NameErr. (31)
ReferenesErr (26)
NameErr. (107)
RefereneeErr. (75)
NameEm (47)
Test Fallure (289)
Test Failure (10)
TypeFFrr. (6)
Unknown (38)
SyntaxFrr. (11)
ReferenceFerr, (26)
TypeFrr. (55)
Table 5: Most frequent failure types of translation functor. The data in the table cell is written as ErrorType (frequency), and Err. is short for Error. Specifically, the "Unknown" type only exists Python and "Test Failure" represents the code that complies successfully but fails to pass some of the test cases. Other types of errors in this table are primarily caused by compilation errors.
Discussion of failure types. Table represents the most frequent failure types of translation functor on three datasets. We observe that most errors are compilation errors rather than failures at corner test cases.
These errors are often caused by type mismatches between different programming languages. For instance, when translating a list in Java, there are multiple potential types in Python, including list, tuple, and List from the typing module.
The occurrence of "NameError" and "ReferenceError" indicates an unfaithful translation of variable names by the model or mismatched function input arguments across different languages for the same problem. This limitation inherent the current evaluation setup.
Overall, the models have demonstrated the ability to perform correct translations in most cases between different programming languages. The errors mainly arise from type align- ment issues between programming languages and local variable definitions. These errors are typically easy to debug for programmers if provided with relevant information. Future work involve passing error messages to the model and allowing it to debug itself.
B.3 Explanation and Reproduction Functor
Data statistics. Same as Appendix B.2 we use 156, 953, and 97 Java functions calculate the pass@1 rates for HumanEval-X, MBXP, and MathQA separately.
Results statistics. Table shows the pass@1 rates of different models. TextDavinci is better at MathQA, while ChatGPT is better at HumanEval and MBXP datasets. The pass@1 score is significantly lower compared to translation functor experiments, indicating that it's more difficult task.
Discussion of failure types. 7 the most frequent failure types of translation functor the three datasets. Based the results, it appears that both models encounter similar types of failures across the different evaluation tasks. The "Test Failure" type is par- ticularly prominent, indicating that the reproduced code, although compiling successfully,
17

---

Published as a conference paper at COLM 2024
Model
HumanEval
MathQA
MBXP
Avg.
TextDavinci
33.0
75.5
55.2
54.6
ChatGPT
35.9
69.1
57.1
54.0
Llama-2-7b-chat--tf
25.9
59.4
45.4
43.5
Starchat-beta
21.8
45.4
40.7
36.0
Table 6: Pass@1 score (%) statistics of the reproduced code, corresponding to the Figure (Right).
Model
HvaanEval
MathQA
MBXP
CannofFindSymbol (47)
Test Failure (350)
Test Failure (181)
TextDavinci
TypeError (38)
Uninitialized Variable 224JaanotFindSymbol (164)
Test Failure (22)
CannofFiddSmboll (18)
TypeError (27)
CannotFindSymbol (46)
Test Failure (401)
ConnotFindSymbol (137)
ChatGPT
TypeError (39)
DuplicateDeclaattin (60)
Test Failure (126)
Test Failure (17)
SyntuxError (44)
UnexpectedEOF (70)
Table 7: Most frequent failure types of the reproduced code. The data in the table cell is written as pwoorypeffreuueccy).. Specifically, the "Test Failure" type represents the code that complies successfully but fails to pass some of the test cases. Other types of errors in this table are primarily caused by compilation errors.
fails to pass of the test cases. Additionally, both models also encounter errors related to symbol identification, such as "CannotFiidSymbool" and "Typelirror", These errors are primarily compilation errors.
It's worth further investigating the causes behind these failure types and exploring strategies to improve the models' performance, especially in handling test cases and resolving symbol- related errors.
C Case Study
In this Section, we analyze some classical cases from our three experiments.
C.1
Morphism Identification
Refer to Figure 7 and Figure Those cases show the model can identify the literate differences between the code pairs but fails to clearly interpret their functional difference.
C.2 Translation Functor
According to the previous statistics, most translation errors happen in the compilation stage. We are also concerned about if there are no compile errors, when will the model make mistakes. Figure 9 shows such case where some information about the numerical data type is missing when translation.
C.3 Explanation Functor and Reproduction Functor
Figure shows case where information loss and hallucination happen during explanation and reproduction. This often the case when the model deals with the MathQA dataset. To improve the performance of the model, using few-shot prompts to illustrate how to explain the code may be useful.
18

---

Published as a conference paper at COLM 2024
Prompt Text:
Do the two codes below have the same output for cach input? In your first sentence, answer "Truc" or "False". Then points out all the differences of the codes.
HumanEval/154.
LoopEnehange- Cewpechhannn-RerreeCCCCC ReorderCondition
public boolean eycpatiemCheoo((Srigee a, String b) public booleun VycputtemmChhecllSttigg a, String b) int = b.lengthO:
int = b.length();
String put
String pat b by for (int = 0; = a.lnngh()) i++)
for (int = 0; <= a.length() i++)
for (int j 0:j (:
if (a.substtiigg D.gpuaaapttattttsssssinggggg
i
+ ())
while ( <-
return true;
(a.substrnng i Dequuls(pat.
substring(.J
return true;
return false;
j++
}
}
retur false;
ChatGPT Responsc
Falsc. Differences:
In code A, the while loop is used to iterate through the "I+1" possible subpatterns of 'a' that have length TT' In code B, the for loop used instead.
code B, the condition for the for loop is ij <= I, which means it checks 'J' is less than or equal to T. This result in the last possible subpattem being ignored hecause 'J' will stop at 'I" and not check anything after it. Therefore, the corect condition should be jj< I',
sequuals(patsuusstinnge. j D))). checks if the substring of 'a' starting from index 'i' with a length of" equals the corresponding substring of pat' In code A, the condition is similar, except uses 'a.substring(i, + instead. Both conditions correct, but they differ slightly in their expression.
B, the condition compuring substrings (a.substring(.,
HumanEval/129
LoogExchang-RRorreCCnittti
ChatGPT Response
False. The codes have differences their inner loops and conditions inside the loops. Code A uses while loop to iterate through the columns of cach row, whercas Code B uses two for loops to iterate through both and columns of the grid.
Code also checks whether a cell contains a before checking neighbors, while Code simply iterates through every cell regardless of its value.
Finally, there is error in Code B where it checks the neighbor to the bottom incorrectly: it should use ii + instead of 1''+11 +1..
Figure 7: Two failed cases of ChatGPT in the morphism identification experiment. For "HumanEval/152" ChatGPT fails to interpret the loop execution condition correctly. For "HumanEval/ 129" ChatGPT reckon equivalent math expression as different. To improve the model's reasoning ability, possible way is to utilize the method of Chain of Thoughts(CooT) to construct better prompts and give the model some time to think before making a decision.
19

---

Published as a conference paper at COLM 2024
Prompt Text:
Do the two codes below have the same output for each input? In your first sentence, answer "True" or "False". Then points out all the differences of the two codes.
HluaanEval/9
BooleanExchanger VariableRenaming
public skjkasdkddiia Integer ls)
public int maxe for (int var
vari
int maxx
For (int (ss)
maxx)
if maxx)
boolean isPrime 10 t- 1J:
boolean isPrime I;
for (intj 2:j Math.sqr(() 15j++)
for (lnt) = 2:j Math sqrt()) 1:3H+)
--jjjjj- 0)
in( 0)
isPrime trae;
isPrime fale::
break;
break;
if ((sPrrim))
if (isPrime) [
return sUms:
retum stam::
ChatGPT Response
False.
Differences:
code A, the 'isPrime' boolean variable assigned to 166 1Y' which is equivalent to Whereas code B, it
ussigned to i f- 1'.
code A, the condition for updating the 'maxx' variable 'if (1isPrime)) which the number prime. Whereas code it iff (isPrime)" which means the number is prime.
Inside the loop of the 'isPrime' check, code sets 'isPrime' 'true' if factor of found and breaks of the loop. This incorrect since it sets 'isPrime' true even when factor found, whereas should set it to false because has factors other than and itself. Code B correctly sets 'isPrime' to "false" in this case.
Figure 8: failed case of ChatGPT in the morphism identification experiment. ChatGPT understands the meaning of the word "isPrime" and points out the logical error. Although logical enror exists natural language, the code function is still the same. ChatGPT fails to find that code A is still equivalent to B.
20

---

Published as a conference paper at COLM 2024
Prompt Text:
Translate the below Java code to JavaScript code. The function header is function isPerfeetSquareef)) (. You should only
unswer the JavaScript function body without further comments and explanations.
MBXP/803
Java Code (from the dataset)
JavaScript Code (by TextDavinci)
public static Boolean isPerfecttquarecint {
if = 0)
(n 0)
return false;
return false;
let sqrt Math.sqrt(n)::
int (int) Math.sqri(n); return sgrt sqrt m.;
return sqrt sqrt
Test Cases
arg00 10;
var isPerfeeSqqaroo(rr(00)
vur vO - fulse;
i((vO))
throw EErorr at 1thh assert statement. Value -1 1 strigggtr)
var arg10= 36;
isPerfocttSquarccrgg11
var vl true;
if( (oomppprr v)))
throw 'Eroor at 2th assert statement. Value ++ =* .Stn.ssrr)giff(())
var arg20 14;
(sPerfcctSquareenn221]::
vur fulse;
if( compprrr) v2)) 'Eroor
'Error at 3th assert statement. Value ="+ stinsstriggffy(x)
Execution Results
dhrow Error 3th assert
Figure 9: A notable failure case occurred during the translation functor experiment with Text-Davinci. Text-Davinci fails consider data type conversions during translation. This issue becomes evident when examining the three test cases provided in Mxeval, as the execution encounters an error the third case. However, were not for the incorrect test scripts generated by the model in Mxeval, the translated code should have been identified as incorrect in the first case. This particular case highlights the importance enhancing the correctness of model-generated execution test scripts for more accurate test of code.
21

---

Published as a conference paper at COLM 2024
Prompt Text:
(Explanation task) Please describe the following code written using natural language. Your description should include the exact function name, its arguments, and the return type. Additionally, provide enough details such as variable initialization to allow someone reproduce the code based on your explanation.
(Reproduction task) Translate the below code description to Java code: <Explanation>
Mat1QA/1014
Java Code (from the dataset)
Reproduced Code (by Text-Davinci)
double nO 2:
public static double problem() {
double nl = 4;
double n0 2.0;
double n2
double nl 7.0;
double n3 3;
double n2 57.0;
double n4 2:
double n3 3.0;
double n5
double n4 6.0;
double 10;
double n5 63.0;
double n7 3;
double n6 5.0;
double n8 14;
double n7 9.0;
double 2;
double n8 206.0;
double n10 3;
double n9 5.0;
double Math (owh,, n10); double t1 Math math.min(n3, n0);
double n10 8.0;
double t0 Math.pow(n9, Math.min(n3, 5));
return t0 + t1;
double t1 Math.pow(nl0,, Math.min(nO, S)); double answer= tl;
return answer;
Explanation (by Text-Davinci)
This is public static method called 'problem() that does not take any arguments and returns double value. The method initializes variables to n10 with double values and assigns numbers. It then uses the Math.pow( method to use the variables n9 and n10 as the base for the power and the Math.mink method to the variables n3 n0 the second parameter. It then assigns these two values, 10 and t1, as addends a sum and returns the answer, double value. Exccution Results
the specified
Exception test case did not pass.
Figure 10: A failure happen in the explanation and reproduction functor experiment. Text- Davinci doesn't explain the code with any numerical details, and it makes up some numbers during reproduction.
22


Reviews:
[{"review_id": "2Yh4r5esTg", "reviewer": "Reviewer_hN4m", "summary": "In this paper, the authors introduce a novel evaluation framework called CatCode, which is based on category theory principles. CatCode offers a comprehensive and mathematically abstract approach to evaluating the performance of large language models (LLMs) in tasks involving the understanding and generation of a mixture of code and natural language text.\n\nOne of the key contributions of this work is the development of a standardized automatic evaluation platform that is adaptive to new datasets, tasks, and models. The authors evaluate ChatGPT, Text-Davinci and CodeGeeX model using the proposed CatCode framework, providing valuable insights into the current strengths and limitations of these models. Notably, they identify a significant deficiency in the ability of existing models to recognize functionally equivalent code and to preserve information about code functionality when generating explanations or descriptions of the code.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the novelty and main contributions of our approach, including the \"principled method\", \"standardized evaluation platform\", and \"granular data on which transformations are hard to detect for LLMs\". We address the questions as follows:\n### Q1: Is there a \"correlation among performance across transformations\"?\n- This is an insightful question! To think from a correlation aspect is what our framework aims to advocate. For similar morphisms, they're correlated. If a model does well on X (variable renaming), it may not be guaranteed to promote Y (loop exchange) performance because those are two independent morphisms. \n- Here's an interesting relevant fact: if a model does well in \"debug then translate\", it may likely do well at \"translate then debug\". (It's nontrivial because they're debugging in different languages. The intuition is that the PL is isomorphic to the Turing Machine under our categorical definition, therefore sharing some structural invariance information). Similarly, why Dall-E can generate many avocado chair images even if they don't exist in the dataset can also be explained by Category Theory. \n### Q2: Connect this paper to robustness literature.\nWe appreciate the advice of including robustness literature as relevant work. We will incorporate references to \"ReCode\"(Wang et al., 2023) and other literature like \"Semantic Robustness of Models of Source Code\" (J Henkel et al.,2022), etc. in the next version!\n### Q3: How to ensure transformation correctness. Does code post-AST transformation obtain the same execution accuracy?\nYes, we ensure theoretical accuracy based on the definitions of these edits. The correctness of AST transformations can be detected by a tree isomorphism detection algorithm. We also verified them with test cases in the datasets, and obtained the same execution accuracy. \n### Q4: Conduct this analysis with models other than OpenAI. In particular ... an additional model from Anthropic, Mixtral, Llama models for broader coverage?\nThanks for the valuable suggestions to improve our work coverage! We've conducted experiments on `Llama2-7b-chat-hf` with key results: (1) Global morphism identification:23% accuracy (2) Translation Java->JavaScript: 59.3% Pass@1 rate on average(HumanEval:38.5, MBXP:43.8, MathQA:95.5) (3) Reproduction: 43.5% Pass@1 on average(HumanEval:25.6, MBXP:45.4, MathQA: 59.4). We will update the results for Mixtral, WizardCoder, StarCoder for a more thorough comparison :)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a novel and principled approach to evaluating LLMs in the context of code-related tasks, offering a standardized evaluation platform and highlighting important areas for improvement in current models. The proposed approach can be extended to different programming languages and datasets. \n- Authors provide granular data about which transformations are hard to be detected by the current SOTA models, guiding community to improve performance in target research areas.", "reasons_to_reject": "I don't have any major reason to reject this paper but there are a few areas of improvements\n\n- It's important to study if model's ability to handle different transformations improve across the board. However, authors don't present any correlation among performance across transformations. If a model does well on X (variable renaming), does it also do well on (loop exchange)? I think such analysis will add a lot of value to this work.  \n- Code transformations are extensively studied under robustness areas in code generation space [1] so it's important to connect this paper to robustness literature. \n\n\n\n\nReferences: \n[1] Wang, Shiqi, et al. \"ReCode: Robustness Evaluation of Code Generation Models.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.", "questions_to_authors": "- I know it's a naive question but it's an important one. How do you ensure that transformations are applied correctly? Have you tried executing code post AST transformation and obtained the same execution accuracy? \n\n- Given that both Text-Davinci, ChatGPT are OpenAI models and hence are most likely trained on similar sources of data, I am not surprised to see similar performance on multiple tasks. It would be interesting to conduct this analysis with some models other than OpenAI. In particular, can you run experiments with one additional model from Anthropic, Mixtral, Llama models for broader coverage?", "extra_scores": {}}, {"review_id": "2X2eM8F7qQ", "reviewer": "Reviewer_kBTR", "summary": "This paper discusses methods to evaluate the abilities of LLMs in solving coding problems that involve a combination of code and text. The authors introduce a novel evaluation perspective for code-related tasks. The key idea is to incorporate category theory, which aims to provide a comprehensive and mathematically abstract framework to evaluate LLMs, encompassing a wide range of code-related tasks. An advantage of the proposed framework is its adaptability to new datasets, tasks, and models, making it a potential standardized automatic evaluation platform. This paper evaluates competitive LLMs to identify current models' deficiencies in recognizing functionally equivalent code and preserving information about code function between the code and its explanation.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work's importance and novelty. Your queries help refine our approach.\n### Q1: The benefit of the method & ability to adapt to new tasks\nThe key benefit of our method is its capacity to define novel, challenging tasks, and adapt existing datasets to the tasks without human labor. \n### From the experiments:\n1. Sec3.3.2: **Challenging New Task**\n    - Inspiration: A fundamental question of category theory is object relationships, leading to natural thinking of the task. \n    - Key results: Experiment 1 **reveals a significant gap in models' lexical versus functional understanding**, (even advanced models like ChatGPT show ~50% accuracy at global morphism, just random guessing).\n2. Sec3.3.4: **Composite Task**\n    - Relation to Category Theory: The unified math abstraction, and the associativity property of Morphisms and functors help us define composite tasks.\n    - Key insights for future work: As models evolve, scores on high-quality datasets tend to saturate (HumanEval~90%). Should we abandon HumanEval, and call for massive efforts to construct new ones? We believe it's not the only way! **With CatCode, we can easily define composite tasks, and make the evaluation effective again. The construction of composite tasks is mostly automatic, greatly reducing the labor for collecting new datasets** (and may prevent test set contamination). \n### Potentials for Automatically Adapting New Tasks\n1. **New composite task**: We propose and experiment on a new task \"repair morphism + translation functor\" using the CodeScope dataset. (Ref:Reviewer#VBwZ-Q1)\n    - Practical Application: This task can automate cross-platform consistency, such as synchronizing bug fixes or features between Android and iOS versions of an app.\n2. **Future potential tasks**\n    - Other morphism+translation functor (harder than pure morphism/translation, run execution to eval)\n    - Self-morphism+summarization functor (produce many summarizations with the same meaning, can be judged by GPT without human-written ground truth), etc. We'd open-source the tool for transforming single tasks into composite tasks.\n### Q2: No clear numbers of questions for the local morphism task in the three datasets... details to reproduce the experiments\nPlease refer to Table 2 in Appendix A for the number of questions. In the appendix, we provide (1) link to anonymized code (2) 5 pages in Appendices A&B detailing the reproduction steps. We'll add more details in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper tackles an important evaluation aspect for code-related tasks.\n* This paper introduces a novel evaluation method for code-related tasks based on the category theory.", "reasons_to_reject": "* The idea of incorporating category theory is interesting and unique. However, the merit of incorporating category theory for the evaluation was not strongly supported by the experiments (if my understanding is correct. See the questions to authors section. I am willing to increase my score if I have misunderstood and the authors convince me in the rebuttal.)\n\n* The authors may need to show results of simple baseline methods without applying the proposed method and compare the results under fair conditions. Otherwise, it is hard to understand the actual benefit of using the proposed method.", "questions_to_authors": "* In Section 3.2.2, __\"For local morphism, we extract the Java functions within HumanEval-X, MBXP, and MathQA datasets. We perform the 9 local morphisms based on AST transformations.\"__ and __\"For global morphisms, since the three datasets above do not contain multiple solutions to the same problem using the same PL, we complement with code from the test split of Code Contest(Li et al., 2022) dataset.\"__\nIf I am not mistaken, it does not seem to provide clear numbers of questions for the local morphism task in the three datasets. This is also the case for the global morphism task conducted in this paper. Please provide more details on the evaluation settings for Section 3.2 so that readers can reproduce the experiments.\n\n\n* Three different experimental results, namely Morphism Identification, Translation Functor, and Explanation Functor, conducted in this paper were presented in Sections 3.2.3, 3.3.3, and 3.4.3. However, the benefit of using the proposed method, specifically incorporating category theory, is not clearly described, particularly for Sections 3.3.3 and 3.4.3. Could you explain how readers can interpret the benefit of the proposed method from these results?\n\n* In conclusion, the authors mention, __\"Based on categorical perspectives, we present a standardized automatic evaluation platform, which is adaptive to new datasets, tasks, and models.\"__  However, I am not sure if this paper clearly demonstrates the ability to adapt to new tasks, as it mainly shows the adaptation ability for datasets and models.", "extra_scores": {}}, {"review_id": "MuW4LNoYwS", "reviewer": "Reviewer_VBwZ", "summary": "This paper proposes an automated evaluation framework called CatCode, designed to assess the capabilities of various large-scale language models, such as ChatGPT, in solving programming problems. The framework leverages category theory as a mathematical abstraction to evaluate code-related tasks and introduces concepts such as code debugging and transformation, code translation, code generation, and interpretation. This approach is comprehensive and standardized, allowing for the evaluation of mixed-code tasks of different types.\n\nIn terms of quality, the paper provides clear definitions and detailed explanations, enabling readers to comprehend and apply the CatCode framework.\n\nIn terms of originality, the paper introduces a novel evaluation framework that addresses limitations within existing evaluation methods by utilizing the principles of category theory.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the quality, originality, comprehensiveness, and standardness of our work.\n### Q1: Relies on small datasets ... Further experimental validation ... Such as in the case of CodeScope\nIn response to the reviewer's concerns, we conduct a \"code-repair morphism+translation functor\" task on the CodeScope dataset. \n- We used the `code_repair` subset and added a step for the model to translate C++ ->C after debugging. Results on `GPT-4` show that **pass@1 drops** from 26/60 in C++ to 18/60 in C. The task is more complex and has real-world applications, such as synchronizing code updates between Android and iOS versions.\n\n**Dataset Justification:**\n-  Dataset Size: Our Morphism Identification task uses ~10k test samples, and other tasks use ~2k samples each, which are substantial for code evaluation, comparable to CodeScope's average of ~1.7k samples/task.\n-  Framework Advantage: Our framework enables the augmentation of datasets by morphisms and the composition of functors. This can make datasets larger, apply to new tasks (Sec3.3.2), and new composite tasks (Sec3.3.4).\n### Q2: Integration and Innovation in Task Design\nWe appreciate the positive feedback on our new morphism identification, and provide further explanation for the other two:\n- **Translation functor task**: It shows how traditional tasks fit within this framework. The advantage of our framework is that it **embraces all mainstream coding tasks** :) instead of excluding or replacing them. This supports further composition with other morphisms and functors (e.g. The CodeScope task above).\n- **Composite functor task**: Combining various coding tasks is **Non-trivial**, because previous evaluation benchmarks use different datasets, different data fields, and independent evaluation pipelines for each task (e.g.CodeXGLUE). Our framework has the advantage of **systematically and automatically enabling task composition** (Please refer to Reviewer#kBTR_Q1 for details).\n\nCategory Theory Framework Advantages in task design:\n- **Math abstraction**: Offers unified data definitions for various coding tasks, providing a conceptual basis for evaluating code performance.\n- **Comprehensive perspective**: Many studies have found that models pre-trained on code (e.g.DeepSeek-Coder, CodeLlama) perform better on mathematics. Coding tasks also have inter-relations. The category theory framework can serve as a good **starting point for understanding the correlations among various coding tasks**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Provides a comprehensive and standardized evaluation framework, facilitating a better understanding of the capabilities of large-scale language models in solving programming problems.\n\nUtilizes the principles of category theory to address limitations in existing evaluation methods, showcasing innovation.", "reasons_to_reject": "Relies only on existing small datasets for validation, requiring further experimental validation and refinement of the framework, such as in the case of Codescope.\n\nThe framework introduces a difference in the Morphism Identification task compared to previous mainstream code evaluation tasks, while the other two tasks remain consistent or a combination of previous tasks.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents CatCode, which uses the category theory to design a framework for evaluation of coding tasks. The benefit of this framework is to unify coding benchmarks of different formats, and provide a flexible way to compose new tasks.

The proposed evaluation framework is novel and interesting, and the authors made great efforts to communicate with reviewers during the rebuttal and discussion period. I think this work can be valuable to the conference, and thus recommend acceptance. I encourage the authors to revise the paper and include the results and discussion they promised to add in the author response.
Decision: Accept