Improving Accuracy of LLM-based Code Clone
Detection Using Functionally Equivalent Methods
Ryutaro Inoue, Yoshiki Higo
Osaka University, Japan, {ry-inoue, higo}@ist.osaka-u.ac.jp

Abstract—A code clone is a code snippet identical or similar toanother in the source code. The presence of code clones causes thespread of bugs, which means that efﬁcient code clone detection
and appropriate refactoring are necessary. Code clone detection
using large language models (in short, LLMs) is more accurate
than conventional tools that do not use LLMs for code clones
with low syntactic similarity. However, even for LLM-based clone
detection tools, detecting such code clones is still difﬁcult, andthere is room for improvement. In this study, we improved theaccuracy of LLM-based code clone detection through ﬁne-tuning
using FEMPDataset. The results showed that our ﬁne-tuning
improved the accuracy of code clone detection.
Index Terms—code clone, large language model, ﬁne tuning,
functionally equivalent methods

I. I NTRODUCTION
A code clone is a code snippet identical or are similar toanother in the source code [1]. Changing the text of a code
clone requires consistent changes occationally; unintentionalinconsistent changes can lead to defects [2]. Thus, the presenceof code clones can signiﬁcantly prevent source code changes
and impair the maintainability of software systems. Therefore,
detecting clones efﬁciently and refactoring them appropriately
is necessary.
Many code clone detection tools have been developed before
now. Well-known tools include CCFinder [3], NiCad [4], Oreo[5], and NIL [6], which use lexical analysis and various
metrics for detection. Those tools tend to be highly accurate
in detecting clones with high syntactic similarity, while theytend to be less accurate in detecting clones with low syntacticsimilarity.
In contrast, code clone detection with large language models(in short, LLMs) has demonstrated superior accuracy for code
clones with low syntactic similarity compared to conventional tools. LLMs have shown promising results in various
ﬁelds, including natural language processing, and have gainedwidespread attention [7]. ChatGPT, which can use models suchas GPT [8], has reached 100 million users in just two months
since its launch. Additionally, Meta’s Llama2 [9] and codellama [10], a ﬁne-tuned model of Llama2, have also beendeveloped.
Dou et al. used several LLMs for code clone detection,comparing their performance with conventional tools [11].
They found that conventional tools like NiCad and Oreo
struggled to detect code clones with low syntactic similarity.
On the other hand, LLM-based detection techniques, including
GPT-3.5-turbo, GPT-4-turbo, and Llama2, achieved higherdetection accuracy for code clones with low syntactic similarity than conventional tools. However, the detection accuracyof code clones with low syntactic similarity by GPT-3.5turbo and GPT-4 is not sufﬁciently high. Additionally, Llama2
inaccurately identiﬁed many pairs of code snippets that are notcode clones as code clones.
This study aims to enhance the detection accuracy ofcode clones with low syntactic similarity through ﬁne-tuning
LLMs. The LLMs targeted in this study are GPT-3.5-turbo,
Llama2-Chat-7B, and CodeLlama-7B-Instruct. Fine-tuning ofGPT-3.5-turbo has been executed using OpenAI’s API, adjusting hyperparameters such as epoch numbers and batchsizes as necessary. LLMs require substantial GPU memory
and processing time for ﬁne-tuning. For this reason, ﬁnetuning of Llama2-Chat-7B and CodeLlama-7B-Instruct has
been executed using techniques such as Lora [12] and ZeRO[13] to reduce GPU memory consumption. FEMPDataset [14],a dataset of functionally equivalent but structurally diverse
Java methods, has been used for ﬁne-tuning and evaluation.
This dataset has been divided into training, validation, andtesting blocks to assess improvements in detection accuracy.
Section II outlines the deﬁnition of clones, previous researchon LLM-based code clone detection, and the experimental
dataset. Section III describes the methodology of this study.
Section IV presents our experimental results. Section V discusses those ﬁndings, and Section VI concludes the study,
highlighting future challenges and directions.
II. P REPARATIONThis section describes the deﬁnition of code clones, previous
research on LLM-based code clone detection, and the dataset
we use in this study.
A. Code Clone Classiﬁcation
A code clone is a code snippet identical or similar to anotherin the source code [1]. Code clones are created in the sourcecode for various reasons such as reusing existing code orreimplementing similar functions [15]. A pair of code snippetsthat are identical or similar to each other is called a clone pair.
1) Code Clone: Roy et al. classiﬁed clones into four types
based on their similarity [16].
Type-1(T1): Identical code snippets except for variations inwhitespace, layout, and comments.
Type-2(T2): Syntactically identical code snippets except forvariations in identiﬁers, literals, types, whitespace, layout,and comments.

Type-3(T3): Similar code snippets with further differencessuch as changed, added, or removed statements, andvariations in identiﬁers, literals, types, whitespace, layout,and comments.
Type-4(T4): Two or more code snippets that perform thesame computation but are implemented by different syntactic variants.
B. FEMPDataset
FEMPDataset [14] is a dataset of functionally equivalentmethods with different structure1 . Functionally equivalentmethod pairs in FEMPDataset are classiﬁed as T4 in the above
categories. FEMPDataset extracts candidates for functionallyequivalent method pairs by using mutual execution of test
cases and then collects truly functionally equivalent methods
by visually determining whether the candidates have the samefunctionality.
C. LLMs(large language models)
Large-scale Language Models (LLMs) are language models
trained on a large corpus. There has been a signiﬁcant increasein the models’ scale and the data volume for training.
D. Fine-tuning techniques of LLMs
LLMs have many parameters, requiring substantial GPU
memory for ﬁne-tuning. This subsection discusses several
techniques to reduce GPU memory consumption through ﬁnetuning.
1) Lora (Low-Rank Adaptation): Lora (Low-Rank Adaptation) [12] is a technique that reduces the number of parameters
updated through ﬁne-tuning, thereby enabling ﬁne-tuning withfewer resources. Lora approximates the differences between
the parameters before and after ﬁne-tuning with a low-rankmatrix, reducing the number of parameters and enabling efﬁcient learning. The size of the low-rank matrix is determinedby specifying the hyperparameter r, which reduces the numberof parameters as r decreases.
2) ZeRO (Zero Redundancy Optimizer): ZeRO (Zero Redundancy Optimizer) [13] is a technique that minimizes theGPU memory required per GPU for training by leveraging
multiple GPUs.
During ﬁne-tuning, it is necessary to maintain and calculate
information on the dynamic change of learning rate, gradients,
and weights on the GPU. ZeRO allows each GPU to makeparameter changes for speciﬁc layers of the LLMs and thencombines the results to change all the parameters of theLLMs. By assigning each GPU to make parameter changes
for speciﬁc layers, ZeRO reduces the GPU memory required
per GPU compared to conventional methods.
E. Indicators of Detection Accuracy
Recall, Precision, and Accuracy are used as performance
metrics for code clone detection and LLMs evaluation. The
meaning and calculation formulas of those metrics are asfollows.
1 The dataset is available at https://github.com/YoshikiHigo/FEMPDataset

Recall: Percentage of method pairs determined to be clonesout of method pairs that are clones.
TPRecall =TP + FN
Precision: Percentage of method pairs detected as clones thatare clones.
TPP recision =TP + FP
Accuracy: Percentage of method pairs that the detection
system has correctly identiﬁed.
TP + TN
Accuracy =TP + FP + FN + TN
TP (True Positive), FP (False Positive), FN (False Negative),and TN (True Negative) are as follows.
TP: Number of method pairs that are detected as clones outof the methods that are clones.
FP: Number of method pairs that are detected as clones outof the methods that are not clones.
FN: Number of method pairs that are not detected as clones
out of the methods that are clones.
TN: Number of method pairs that are not detected as clones
out of the methods that are not clones.
F. Previous Research on Code Clone Detection using LLMs
We introduce the study by Dou et al. as previous researchon code clone detection by LLMs [11]. Their study compared
the performance of existing tools without LLMs and LLMs
using a single prompt as input based on BigCloneBench [17].
The results show that existing tools have higher recall forclones with high syntactic similarity, while LLMs have higher
recall for clones with low syntactic similarity.
Additionally, GPT-3.5-turbo and GPT-4-turbo show higher
detection rates than existing tools in detecting Type-4 clones
with low syntactic similarity. However, the detection accuracy
is not sufﬁciently high, and there is room for improvement.
Llama2-chat-7B shows high recall but low precision, recognizing almost all method pairs as clone pairs. Therefore, thisstudy aims to improve the accuracy of code clone detection
by LLMs through ﬁne-tuning.
III. E XPERIMENTThis section describes the experimental methodology.
Target LLMs in the experiment are as follows.
• GPT-3.5-turbo• Llama2-Chat-7B• CodeLlama-7B-Instruct
The experiment aims to improve the detection accuracy ofclones with low syntactic similarity through ﬁne-tuning. We
experimented with the following steps.
STEP1: Fine-tuning
Executing ﬁne-tuning of LLMs using FEMPDataset.
STEP2: Clone Detection
Giving method pairs of test data in FEMPDataset toLLMs before and after ﬁne-tuning, and answers are got
as either ”Yes” or ”No.”STEP3: Performance Evaluation
Aggregating the answers of LLMs before and after ﬁnetuning and comparing their performance.
A. Fine-tuning

0.69

Precision

0.74
0.68

Accuracy

FEMPDataset is used for ﬁne-tuning. The dataset is divided
into training, validation, and testing data for ﬁne-tuning. Finetuning is executed using training and validation data, andperformance evaluation is executed using testing data. The
number of method pairs for each data is shown in Table I.
Fine-tuning of GPT-3.5-turbo is executed using OpenAI
API. Fine-tuning of Llama2-Chat-7B and CodeLlama-7BInstruct is executed using Lora and ZeRO to reduce GPUmemory consumption. The hyperparameter r of Lora, asexplained in Section II-D, is set to 2. Fine-tuning is executedby two GPUs, Quadro RTX 8000 and Tesla V100.
B. Clone detection (LLMs execution)

1) Getting Method Pairs
Getting functionally equivalent method pairs and functionally inequivalent ones from FEMPDataset. Thosepairs are targets of clone detection to check whether thetarget LLMs can recognize clone pairs correctly.
2) Creating Prompts
Converting each method pair into prompts asking whether
it is a clone pair .
3) Getting Answers
Inputting the prompts to LLMs before and after ﬁnetuning to get either ”Yes” or ”No” answers.
1) Dataset for Evaluation: LLMs performance is evaluatedusing the testing data of the divided FEMPDataset.
2) Prompts: The prompts consist of two roles: ”system”
and ”user.” In the system role, the answers are instructed as”Yes” or ”No.” In the user role, the prompts are instructed todetermine whether the two methods are a clone pair or not.
C. Performance Evaluation
We aggregate the answers from LLMs and calculate thethree values of Recall, Precision, and Accuracy to evaluate
the performance. We used 219 method pairs in FEMPDataset
as test data for evaluating performance.

N UMBER OF METHOD PAIRS FOR EACH DATA
Validation data
132
88
220

0.20

0.40

0.60

fine-tuned GPT-3.5-Turbo

0.84

0.81
0.76
0.80

1.00

GPT-4-turbo

Fig. 1. Comparison of GPT-3.5-turbo and GPT-4-turbo performanceIV. E XPERIMENTAL R ESULTS
This section describes the results of the experiment. The
comparisons before and after ﬁne-tuning of GPT-3.5-turbo,
Llama2-Chat-7B, and CodeLlama-7B-Instruct were executedusing the testing data of FEMPDataset.

Recall, Precision, and Accuracy of GPT-3.5-turbo before
and after ﬁne-tuning and GPT-4-turbo are shown in Table II.
After ﬁne-tuning, GPT-3.5-turbo shows signiﬁcant improvement in Precision. Recall remains almost the same. This resultindicates that ﬁne-tuning has enabled the correct classiﬁcationof non-clone pairs.
Comparing GPT-4-turbo and ﬁne-tuned GPT-3.5-turbo, theﬁne-tuned GPT-3.5-turbo has higher Accuracy.
B. Evaluation of Llama2-Chat-7B
Recall, Precision, and Accuracy of Llama2-Chat-7B before
and after ﬁne-tuning are shown in Table III. A graphicalsummary of the above results is shown in Figure2.
Before ﬁne-tuning, Llama2-Chat-7B recognized all functionally equivalent method pairs and functionally inequivalent
ones as clone pairs. However, after ﬁne-tuning, Llama2Chat-7B can correctly classify some functionally inequivalentmethod pairs. Although Recall has declined, Precision andAccuracy have also improved. This result indicates that theperformance has improved compared to Llama2-Chat-7B before ﬁne-tuning.

TABLE IIE VALUATION OF GPT-3.5- TURBO , FINE - TUNED GPT-3.5- TURBO , AND
GPT-4- TURBO
model
GPT-3.5-turbo
ﬁne-tuned GPT-3.5-turbo
GPT-4-turbo

TABLE I
Train data2
1,081
674
1,755

0.00

GPT-3.5-Turbo

0.91

A. Evaluation of GPT-3.5-turbo

LLMs before and after ﬁne-tuning are executed with thefollowing steps.

Clone method pairs
Non-clone method pairs
Total

0.84
0.83

RecallTest data
12990219

2 We excluded one non-clone method pair from the train data that exceeded

4,096 tokens at the prompt to reduce the GPU resources required.

Recall
0.84
0.83
0.91

Precision
0.69
0.84
0.74

Accuracy
0.68
0.81
0.76

TABLE IIIE VALUATION OF FINE - TUNED L LAMA 2-C HAT-7B AND L LAMA 2-C HAT-7B
model
Llama2-Chat-7B
ﬁne-tuned Llama2-Chat-7B

Recall1.00
0.78

Precision
0.60
0.66

Accuracy
0.60
0.63

1.00

Recall0.78

Precision

0.60

Accuracy

0.60
0.63

0.66

0.00

0.20

0.40

Llama2-Chat-7B

0.60

0.80

1.00

fine-tuned LLaMA2-Chat-7B

Fig. 2. Comparison of Llama2-Chat-7B performance

0.51

Recall0.73
0.71

Precision

0.85
0.59

Accuracy

0.77
0.00

0.20

0.40

CodeLlama-7B-Instruct

0.60

0.80

1.00

fine-tuned CodeLlama-7B-Instruct

Fig. 3. Comparison of CodeLlama-7B-Instruct performance

Recall, Precision, and Accuracy of CodeLlame-Instruct-7B
before and after ﬁne-tuning are shown in Table IV. A graphicalsummary of the above results is shown in Figure 3.
All Recall, Precision, and Accuracy have improved.
The performance has improved signiﬁcantly compared toCodeLlama-7B-Instruct before ﬁne-tuning.
V. D ISCUSSION
As a result of ﬁne-tuning GPT-3.5-turbo, Llama2-Chat7B, and CodeLlama-7B-Instruct, the improvement of code
clone detection accuracy was conﬁrmed for all three language
models. This result suggests that ﬁne-tuning is effective incode clone detection using LLMs.
The effect of ﬁne-tuning depends on the type of pre-trained
data and the performance of the model before ﬁne-tuning.
CodeLlama is a model that learned about programs basedon Llama2, and Llama2-Chat is a model that learned about
dialogue in chat format. CodeLlama-7B-Instruct shows a moresigniﬁcant improvement in performance before and after ﬁnetuning than Llama2-Chat-7B. This result suggests that the pretrained data affects the effect of ﬁne-tuning.
VI. C ONCLUSION
In this study, we attempted to improve the accuracy ofcode clone detection by LLMs through ﬁne-tuning using

TABLE IVE VALUATION OF FINE - TUNED C ODE L LAMA -7B-I NSTRUCT AND
C ODE L LAMA -7B-I NSTRUCTRecall
0.51
0.73

Precision
0.71
0.85

ACKNOWLEDGEMENTS
This research was supported by JSPS KAKENHI Japan(JP21K18302, JP21H04877, JP22H03567, JP22K11985).
R EFERENCES

C. Evaluation of CodeLlama-7B-Instruct

model
CodeLlama-7B-Instructﬁne-tuned CodeLlama-7B-Instruct

FEMPDataset. As a result of ﬁne-tuning, all models in theexperiment improved in code clone detection accuracy.
The following three points are raised as future research
directions.
Adding models of the experiment
We believe conducting similar experiments for models
not covered in this study (e.g., models focused on sourcecode) will allow for performance comparison between
models.
Performance evaluation using other benchmarks
We believe executing similar experiments for otherdatasets than FEMPDataset would provide a more detailed evaluation of ﬁne-tuning performance.
Improvement of Prompt
In this study, we did not improve the prompts. We believe
improving the prompts using technologies such as Chainof thought [18] will improve performance.

Accuracy
0.590.77[1] I. Baxter, A. Yahin, L. Moura, M. Sant’Anna, and L. Bier, “Clonedetection using abstract syntax trees,” in Proc. ICSM, 1998, pp. 368–377.
[2] M. Mondal, C. Roy, and K. Schneider, “A Fine-Grained Analysis on
the Inconsistent Changes in Code Clones,” in 2020 IEEE ICSME, 2020,pp. 220–231.
[3] T. Kamiya, S. Kusumoto, and K. Inoue, “CCFinder: a multilinguistic
token-based code clone detection system for large scale source code,”IEEE Trans. Software Engineering, vol. 28, no. 7, pp. 654–670, 2002.
[4] C. Roy and J. Cordy, “NICAD: Accurate Detection of Near-Miss Intentional Clones Using Flexible Pretty-Printing and Code Normalization,”in 2008 IEEE ICPC, 2008, pp. 172–181.
[5] V. Saini, F. Farmahinifarahani, Y. Lu, P. Baldi, and C. Lopes, “Oreo:detection of clones in the twilight zone,” in ESEC/FSE 2018, 2018, p.
354 ‒ 365.
[6] T. Nakagawa, Y. Higo, and S. Kusumoto, “NIL: large-scale detection oflarge-variance clones,” in ESEC/FSE 2021, 2021, p. 830 ‒ 841.
[7] W. X. Zhao et al., “A Survey of Large Language Models,” 2023.
[8] OpenAI, “GPT-4 Technical Report,” 2023.
[9] H. Touvron et al., “Llama 2: Open Foundation and Fine-Tuned ChatModels,” 2023.
[10] B. Rozière et al., “Code Llama: Open Foundation Models for Code,”2023.
[11] S. Dou et al., “Towards Understanding the Capability of Large Language
Models on Code Clone Detection: A Survey,” 2023.
[12] E. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,”in ICLR 2022, 2022.
[13] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “ZeRO: Memoryoptimizations Toward Training Trillion Parameter Models,” in SC20,2020, pp. 1–16.
[14] Y. Higo, “Dataset of Functionally Equivalent Java Methods and Its
Application to Evaluating Clone Detection Tools,” IEICE Trans. Inf.
& Syst., 02 2024.
[15] C. Roy and J. Cordy, “A Survey on Software Clone Detection Research,”
School of Computing TR 2007-541, pp. 3–7, 01 2007.
[16] C. Roy, J. Cordy, and R. Koschke, “Comparison and evaluation of code
clone detection techniques and tools: A qualitative approach,” Scienceof Computer Programming, vol. 74, no. 7, pp. 470–495, 2009.
[17] J. Svajlenko, J. Islam, I. Keivanloo, C. Roy, and M. Mia, “Towards aBig Data Curated Benchmark of Inter-project Code Clones,” Proc. 30thICSME, pp. 476–480, 09 2014.
[18] J. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large
Language Models,” 2022.