Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance (2024)

Abstract: This paper revisits recent code similarity evaluation metrics, particularly focusing on the application of Abstract Syntax Tree (AST) editing distance in diverse programming languages. In particular, we explore the usefulness of these metrics and compare them to traditional sequence similarity metrics. Our experiments showcase the effectiveness of AST editing distance in capturing intricate code structures, revealing a high correlation with established metrics. Furthermore, we explore the strengths and weaknesses of AST editing distance and prompt-based GPT similarity scores in comparison to BLEU score, execution match, and Jaccard Similarity. We propose, optimize, and publish an adaptable metric that demonstrates effectiveness across all tested languages, representing an enhanced version of Tree Similarity of Edit Distance (TSED).

heading: 1 Introduction and Related Work
text: In the fields of natural language processing and software engineering, code generation tasks are gaining more and more attention. Assessing the quality of generated code is now critically important, but we still lack evaluation methods other than traditional statistical sequence evaluation methods. Widely used semantic evaluation metrics like BLEU score and Jaccard similarity rely on statistical characteristics, overlooking the intricate grammatical structures and logical relationships inherent in complex programming languages.
However, recent developments in the NLP field paved the way for novel evaluation metrics which we explore in this study. For one, the staggering number of powerful large language models (LLMs) such as GPT-3.5/4 (Achiam et al., 2023) revolutionized the NLP landscape and led to noteworthy advancements in the realm of code review and evaluation (Wang et al., 2023; Tang et al., 2024). Another recent study introduced the novel TSED metric and
used it to evaluate text-to-SQL tasks (Song et al., 2023). For this study, we take advantage of these developments to (1) prompt the GPT-4 model to generate similarity scores for code, and (2) expand on the TSED metric.
We utilize these two different metrics (GPT and TSED) to evaluate the structural similarity of different programming languages and how they relate to execution matches. Furthermore, we address how these metrics are correlated to semantic similarity metrics like the BLEU score. Finally, we investigate some limitations of these metrics by delving into the impact of TSED’s penalty weight of tree operations on evaluation accuracy and exploring the stability of outputs from the GPT LLMs.
As a result, we have these 3 contributions from this research: (a) we propose and publish a new tool for 48 programming languages1, (b) we discuss 2 recent evaluation metrics and 2 traditional metrics and compare them via correlation coefficient, recall to execution match, (c) we discuss the unstable nature of GPT similarity scoring and the ways to optimize TSED.

heading: 2 Approaches

heading: 2.1 TSED on Programming Languages
text: Applying the TSED evaluation method, initially designed for SQL analysis, we have undergone modifications to extend its applicability to various programming languages. The fundamental TSED approach, illustrated in Figure 1, encompasses AST parsing, AST Editing Distance Calculation, and normalization, closely resembling the methodology outlined in the original paper. However, we have made modifications to both the AST parsing and normalization.
Code Parsing: Parsing in the domain of programming languages involves parsing raw code
1https://github.com/Etamin/TSED
38
text into its associated AST. This parsing underscores the complexity of interpreting various programming constructs and converting them into a structured grammar tree representation.
We use tree-sitter2 as our AST parser which is based on GLR(generalized left-to-right rightmost), a powerful parsing algorithm commonly found in the literature (Latif et al., 2023; Tomita, 1991; Clem and Thomson, 2021).
Tree Distance Computation: For calculating tree edit distance as ∆, we utilize the same function as outlined in the TSED paper, which is APTED(All Path Tree Edit Distance) algorithm (Pawlik and Augsten, 2015, 2016). Considering G1 as predicted code’s AST and G2 as AST from ground-truth:
∆(G1, G2) = min ops
n∑
i=1
w(opi) (1)
Here, ops is a sequence of edit operations transforming G1 into G2, with w(opi) as the cost for the ith operation.
Normalization: Normalization of tree edit distances accounts for the complexity of the code by considering the maximum number of nodes between two trees, and we add a ramp function to avoid some extreme situations:
TSED = max{1− δ MaxNodes(G1, G2) , 0} (2)
This provides a metric for structural similarity comparison of programming code, enabling a nuanced analysis beyond mere syntactic comparison.

heading: 2.2 GPT Structure Similarity
text: Between 2020 and 2023, OpenAI introduced the GPT-3/3.5 and GPT-4 models, showcasing remark-
2https://tree-sitter.github.io/tree-sitter/
able reasoning capabilities and achieving state-ofthe-art performance across numerous tasks (Brown et al., 2020). Our approach involves utilizing prompts to elicit the model’s output regarding the structural similarity between two code segments, resulting in a score on a scale from 0 to 1. A score of 1 indicates identical structures, while 0 signifies complete dissimilarity. Despite its effectiveness, this metric operates as a black box, leaving us unaware of the specific calculations performed by GPT or whether it consistently employs the same metric. From various research papers, we’ve observed that these LLMs tend to produce more unstable results with each iteration (Tian et al., 2023; Liu et al., 2023).
Given 2 Java code paragraphs, please generate a similarity score from 0 to 1 (to three decimal places), by grammar parsing structure. Answer with a format like [[0.777]]. =====Code 1===== [Java code snippet 1] =====Code 2===== [Java code snippet 2] =====End=====
This prompt above is designed to calculate and return a similarity score between two Java code snippets based on their grammatical structure. The similarity score ranges from 0 to 1, with three decimal places of precision. A score of 1 indicates identical grammatical structures, while a score of 0 indicates completely different structures. The output format [[0.777]] facilitates easy extraction and post-processing of the score.

heading: 3 Research Questions and Targets
text: RQ1: Can TSED be used in more programming languages? We investigate the adaptability of AST Edit Distance which is a generalized version of TSED, exploring its effectiveness in languages like Python and Java to assess its applicability for code similarity analysis. RQ2: How are TSED and GPT similarity correlated to semantic similarity and execution match? We assess the correlation between these different metrics to understand their respective contributions in evaluating code similarity across multiple programming languages. RQ3: What are the limits of these metrics? We assess the stability of GPT-based similarity output and analyze how parameters, particularly operation weights (delete, insert, rename), influence TSED.

heading: 4 Experiments

heading: 4.1 General Setup
text: In this study, our primary objective is to apply the theoretical framework to a diverse range of programming languages. To achieve this, we aim to identify executable datasets and evaluate them using predefined metrics. The experimental setup comprises two key tasks: firstly, expanding the application of TSED and GPT similarity to additional programming languages, followed by exploring the correlation between these metrics. Subsequently, we seek to assess the stability of GPT scoring and examine the impact of various parameters on the TSED metric. This structured approach allows us to comprehensively investigate the adaptability, correlations, and stability of the chosen metrics across a spectrum of programming languages.

heading: 4.2 Evaluation Metrics
text: • BLEU Score is calculated as the geometric mean of the modified precision scores for various ngram lengths, providing a concise and standardized similarity measurement between the generated and reference text (Papineni et al., 2002).
• Jaccard Similarity is a measure of similarity between two sets and is calculated by dividing the size of the intersection of the sets by the size of their union, offering a quantitative assessment of the degree of overlap between the sets’ elements.
• Execution Match Execution Match pertains to the consistency in execution outcomes between
generated code and its corresponding ground truth, evaluating the equivalence in practical functionality. 1 in Execution match means they have the same execution results, and 0 means different.
• GPT Similarity mentioned in the Section 2.2
• TSED mentioned in the Section 2.1.

heading: 4.3 Datasets
text: Although the execution match metric is infrequently employed in programming code-related datasets, its prominence has increased in recent years. Our comparative analysis involved assessing datasets from various papers, considering factors such as dataset sizes, programming languages, and executables. As highlighted in Table 1, the MBXP dataset encompasses 13 different languages, serving as a function-level benchmark that effectively evaluates programming paragraphs. However, the MBXP dataset includes ground-truth solutions for only 7 languages, with C# omitted due to compilation issues. Additionally, we consider the CoderEval dataset to facilitate a comparison between Python and Java code generation, leveraging its longer test samples, results are in the appendix.
In the Bash-Shell scenarios, we reproduce results and conduct a comparative analysis using the InterCode dataset. Notably, we identify the SPIDER dataset within InterCode and establish it as a baseline. SPIDER, previously evaluated in comparison to the TSED paper, is a substantial human-labeled dataset for the text-to-SQL task. This dataset encompasses databases with intricate join solutions across diverse domains (Yu et al., 2018).

heading: 5 Results

heading: 5.1 Similarity Results
text: As we analyze the results presented in Table 2, our experiment demonstrates the effective performance of TSED and GPT similarity in evaluating the MBXP dataset across all 6 programming languages. No instances of parsing or scoring generation failures were observed, confirming the robustness of these metrics across languages.
Moreover, TSED shows a commendable correlation ranging from 0.6 to 0.8 with BLEU score and Jaccard similarity, as illustrated in Figure 2. Additionally, TSED exhibits a strong correlation with GPT similarity, especially in Java and Python during the CoderEval test, as depicted in Figure 3, underscoring its sensitivity to code structure. We employ thresholding to establish a prediction-toexecution match. If the metric value exceeds the threshold T , we assign the prediction as 1; otherwise, it is set to 0. The optimal threshold values are determined through enumeration to achieve the best match results. Based on their F1/Accuracy match to the Execution match, both TSED and GPT similarity exhibit higher accuracy compared to semantic metrics in Table 3. Notably, GPT similarity demonstrates a slightly superior F1 score and TSED gives good results on accuracy.
Java
Python
ChatGPT CodeGen PanGu

heading: 5.2 Stability of GPT Scoring
text: To understand how unstable GPT scoring is, we execute the GPT-4 Similarity scoring five times on identical prediction sets, we establish the initial result as a baseline to assess differences through statistical indicators such as Mean Squared Error (MSE) or Mean Absolute Error (MAE) in comparison to the first scoring. Table 4 demonstrates that GPT scoring exhibits limited stability in the context of code similarity evaluation.

heading: 5.3 Parameter optimization of TSED
text: We can configure the penalty weight of 3 operations in tree distance computing: Delete, Insert, and Rename. Figure 4 which is from a test for the MBXP/Java dataset shows is ‘Insert’ has a sweet spot of 0.8. ’Delete’ and ’Rename’ operations just keep them in 1.0 penalty weight as the best choice. But we need to keep in mind it can be different in other programming languages.

heading: 5.4 Efficiency
text: The table 5 illustrates the computational time (in ms) required by each programming language tested, including TSED, BLEU score, Jaccard similarity, and GPT 3.5 Score. Our findings indicate that the performance of TSED is comparable to the BLEU score, with significantly lower computational time compared to GPT-3.5. This suggests that TSED is
indeed efficient enough to be applied at scale.

heading: 6 Conclusion
text: In this paper, we applied TSED to more programming languages, compared GPT similarity and TSED to semantic metrics, and checked representation to execution match. Then we discuss limitations about the stability of GPT scoring and the penalty parameters of TSED.
Limitations
While our study provides valuable insights into code similarity assessment using TSED and GPTbased metrics, it is essential to acknowledge certain limitations. Firstly, the generalizability of our findings may be influenced by the specific datasets and programming languages employed in our analysis. Additionally, the stability of GPT-based similarity metrics, as highlighted in our results, poses a limitation in terms of consistent and reliable code assessments. Furthermore, variations in the interpretation and definition of similarity metrics across different studies may introduce inherent biases. Lastly, the effectiveness of TSED metrics may be contingent upon the quality of the employed parsers and the fine-tuning of penalty parameters. These limitations underscore the need for caution when extrapolating our results to diverse contexts and emphasize the necessity for further research to address these challenges.
Ethics Statement
Our research adheres to ethical standards, prioritizing integrity and respect for all involved parties. We ensured data privacy, obtained informed consent
where applicable, and maintained transparency in our methodologies. The study was conducted with the utmost consideration for ethical guidelines and the welfare of participants, upholding the principles of fairness, accountability, and academic integrity throughout the research process.

heading: Acknowledgment
text: This research was funded in whole, or in part, by the Luxembourg National Research Fund (FNR), grant references NCER22/IS/16570468/NCERFT and BRIDGES2021/IS/16229163/LuxemBERT. We extend our heartfelt appreciation to our collaborator, BGL BNP PARIBAS, for their invaluable support and special thanks to Saad Ezzini from Lancaster University for his advisory contributions.

heading: A Additional Experiment Details
text: A.1 Parser Comparison The ANTLR3 (ANother Tool for Language Recognition) tool, serving as a distinct AST parser compared to tree-sitter, demonstrated notable differences. Following our evaluation using identical settings for TSED metrics, as Figure 5 shows, it became evident that the correlation with other metrics was inferior to the original solutions. This experiment underscores the crucial role of parser performance in the computation procedure, highlighting the significance of selecting an appropriate parser for accurate and reliable code similarity assessments.
A.2 Other experiment results Due to space constraints, a subset of experimental data is provided in the appendix. A comprehensive evaluation of CoderEval and InterCoder is detailed in Table 6, while specific original sample data from the MBXP dataset is presented in Table 7.
CoderEval, designed for class-level code generation tasks, proves to be a challenging test. Utilizing Pass@10 data as a test sample, TSED demonstrates a robust correlation with semantic indicators in both Java and Python languages. Additionally, a noteworthy correlation is observed between TSED and GPT Similarity.
In the case of InterCoder, we confirm that TSED calculations extend to Bash scripts. Also, the correlation in Figure 6 between TSED to semantic metrics is acceptable, the GPT score doesn’t have a good correlation to others. We also replicate the performance of the SPIDER dataset, noting differences from the original paper but not to a significant extent.
Despite the notably low semantic similarity between the MBXP built-in samples and the ground
3https://www.antlr.org/
truth, a relatively high execution match is observed. We acknowledge this disparity and plan to address it through optimization in future research endeavors.

heading: B Case Studies
text: B.1 A. Low BLEU, but high TSED
### Code Paragraph 1 int result = 0;
for(int i = 0; i < n; i++) { result = n * (7 * n - 5) / 2;
}
return result; }
} ### Code Paragraph 2 int jacobsthalNumber = 1;
for(int i = 2; i <= n; i++){ jacobsthalNumber =
↪→ jacobsthalNumber + (n ↪→ - i) * (i - 1);
} return jacobsthalNumber;
} }
In the provided code snippets, both segments involve loops for performing calculations, which contributes to their high structural similarity. However, the semantic similarity is relatively low due to the significant disparity in variable names, which occupy a considerable portion of the tokens. Despite the differences in semantics, the BLEU score, a metric commonly used for evaluating text similarity, yields a score of 0.359, indicating some level of similarity. In contrast, the Tree Similarity of Edit Distance (TSED) metric, which accounts for structural differences, produces a higher score of 0.8, highlighting the effectiveness of TSED in capturing structural similarities even when semantic differences exist.
B.2 BLEU and TSED similar
### Code Paragraph 1 def max_of_two(a, b):
if a > b: return a
else: return b
### Code Paragraph 2 def max_of_two(a, b):
return max(a, b)
Both the BLEU score (0.408) and the TSED (0.444) score suggest that while the two code snippets achieve somehow similar functionality, they do so using different structural approaches.
Token Similarity: Despite structural differences, the two snippets share many common tokens, leading to a moderate BLEU score. This reflects their similar semantic intent and consistent use of basic programming constructs.
Structural Difference: The TSED score highlights the structural changes. The first snippet uses
explicit control flow statements (if-else), while the second snippet uses a built-in function (max). This difference is significant enough to lower the structure similarity score but not so drastic as to suggest entirely different functionalities.
Overall, these scores indicate that the snippets are functionally similar (performing the same task) with some commonality in token usage but exhibit distinct structural approaches in their implementations. This case illustrates how different coding styles and structures can be used to achieve the same outcome in programming.
B.3 Let ChatGPT explain in detail
Prompt: Given 2 Java code paragraphs, please generate a similarity score from 0 to 1 (to three decimal places), by grammar parsing structure. Answer with a format like [[0.777]]. And give us 3 dimensions of similarity in integer score 1 5, which is “functional”, “tree structure”, and “semantical”; Then give the reason of the score you think if it’s high or low.
int result = 0; for(int i = 0; i < n; i++) {
result = n * (7 * n - 5) / 2; }
return result; }
} int jacobsthalNumber = 1;
for(int i = 2; i <= n; i++){ jacobsthalNumber =
↪→ jacobsthalNumber + (n ↪→ - i) * (i - 1);
} return jacobsthalNumber;
} }
Here is the similarity score and the three dimensions of similarity for the given code snippets: [[0.875]] Functional Similarity: 4 Tree Structure Similarity: 5 Semantical Similarity: 3 Explanation:
• Functional Similarity (4): Both code snippets perform calculations within a loop structure, albeit with different mathematical operations. They both iterate over the variable i to calculate a result based on the value of n.
• Tree Structure Similarity (5): The overall structure of both code snippets is highly similar, with both containing a loop structure followed by a calculation and a return statement.
• Semantical Similarity (3): While the overall structure is similar, the semantics differ significantly. The variable names (result vs. jacobsthalNumber) and the mathematical operations performed within the loop differ, leading to a moderate level of semantic dissimilarity.

heading: Future Work
text: We acknowledge this disparity and plan to address it through optimization in future research endeavors