CLEVER: Combining Code Metrics with Clone Detection for
Just-In-Time Fault Prevention and Resolution in Large Industrial
Projects
Mathieu Nayrolles

Abdelwahab Hamou-Lhadj

La Forge Research Lab, Ubisoft
mathieu.nayrolles@ubisoft.com

ECE Department, Concordia University
wahab.hamou-lhadj@concordia.ca

ABSTRACT

1

Automatic prevention and resolution of faults is an important research topic in the field of software maintenance and evolution.
Existing approaches leverage code and process metrics to build
metric-based models that can effectively prevent defect insertion ina software project. Metrics, however, may vary from one project toanother, hindering the reuse of these models. Moreover, they tend to
generate high false positive rates by classifying healthy commits asrisky. Finally, they do not provide sufficient insights to developers
on how to fix the detected risky commits. In this paper, we propose
an approach, called CLEVER (Combining Levels of Bug Prevention
and Resolution techniques), which relies on a two-phase process forintercepting risky commits before they reach the central repository.
When applied to 12 Ubisoft systems, the results show that CLEVER
can detect risky commits with 79% precision and 65% recall, which
outperforms the performance of Commit-guru, a recent approachthat was proposed in the literature. In addition, CLEVER is able
to recommend qualitative fixes to developers on how to fix risky
commits in 66.7% of the cases.

Automatic prevention and resolution of faults is an important research topic in the field of software maintenance and evolution.
Effective approaches can help reduce significantly the cost of maintenance of software systems, while improving their quality. A particular line of research focuses on the problem of preventing the
introduction of faults by detecting risky commits (commits thatmay potentially introduce faults in the system) before they reachthe central code repository. We refer to this as just-in-time fault
detection/prevention [18].
There exist techniques that aim to detect risky commits (e.g.,[2, 5, 40]), among which the most recent approach is the one proposed by Rosen et al. [36]. The authors developed an approach anda supporting tool, Commit-guru, that relies on building models
from historical commits using code and process metrics (e.g., code
complexity, the experience of the developers, etc.) as main features.
These models are used to classify new commits as risky or not.
Commit-guru has been shown to outperform previous techniques(e.g., [18, 23]).
However, Commit-guru and similar tools suffer from a numberof limitations. First, they tend to generate high false positive rates
by classifying healthy commits as risky. The second limitation isthat they do not provide recommendations to developers on how to
fix the detected risky commits. They simply return measurementsthat are often difficult to interpret by developers. In addition, they
have been mainly validated using open source systems. Their effectiveness when applied to industrial systems has yet to be shown.
In this paper, we propose an approach, called CLEVER (Combining Levels of Bug Prevention and Resolution techniques), that relies
on a two-phase process for intercepting risky commits before theyreach the central repository. The first phase consists of building
a metric-based model to assess the likelihood that an incoming
commit is risky or not. This is similar to existing approaches. The
next phase relies on clone detection to compare code blocks extracted from suspicious risky commits, detected in the first phase,with those of known historical fault-introducing commits. Thisadditional phase provides CLEVER with two apparent advantages
over Commit-guru. First, as we will show in the evaluation section,
CLEVER is able to reduce the number of false positives by relying
on code matching instead of mere metrics. The second advantage isthat, with CLEVER, it is possible to use commits that were used tofix faults introduced by previous commits to suggest recommendations to developers on how to improve the risky commits at hand.
This way, CLEVER goes one step further than Commit-guru (andCCS CONCEPTS• Software and its engineering → Software testing and debugging; Maintaining software; • Information systems → Expert
systems;

KEYWORDS
Defect Predictions, Fault Fixing, Software Maintenance, Software
Evolution
ACM Reference Format:
Mathieu Nayrolles and Abdelwahab Hamou-Lhadj. 2018. CLEVER: Combining Code Metrics with Clone Detection for Just-In-Time Fault Prevention and Resolution in Large Industrial Projects. In MSR ’18: MSR ’18:
15th International Conference on Mining Software Repositories , May 28–
29, 2018, Gothenburg, Sweden. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3196398.3196438
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
MSR ’18, May 28–29, 2018, Gothenburg, Sweden© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-5716-6/18/05. . . $15.00
https://doi.org/10.1145/3196398.3196438

INTRODUCTION

MSR ’18, May 28–29, 2018, Gothenburg, Sweden
similar techniques) by providing developers with a potential fix for
their risky commits.
Another important aspect of CLEVER is its ability to detect risky
commits not only by comparing them to commits of a single project
but also to those belonging to other projects that share common
dependencies. This is important in the context of an industrial
setting where software systems tend to have many dependencies
that make them vulnerable to the same faults.
CLEVER was developed in a context of a research project between Concordia University and Ubisoft. Ubisoft is one of theworld’s largest video game development companies specializing in
the design and implementation of high-budget video games. Ubisoft
software systems are highly coupled containing millions of files andcommits, developed and maintained by more than 8,000 developers
scattered across 29 locations in six continents.
We tested CLEVER on 12 major Ubisoft systems. The results
show that CLEVER can detect risky commits with 79% precision
and 65% recall, which outperforms the performance of Commitguru (66% precision and 63% recall) when applied to the samedataset. In addition, 66.7% of the proposed fixes were accepted by atleast one Ubisoft software developer, making CLEVER an effective
and practical approach for the detection and resolution of risky
commits.
The remaining parts of this paper are organised as follows. In
Section 2, we present related work. Sections 3, 4 and 5 are dedicatedto describing the CLEVER approach, the case study setup, and thecase study results. Then, Sections 6 and 7 present the threats tovalidity and a conclusion accompanied with future work.

2

RELATED WORK

Our approach, CLEVER, is related to two research areas: defectprediction and patch generation.

2.1

File, Module, and Risky Change Prediction

Existing studies for predicting risky changes within a repository
rely mainly on code and process metrics. As discussed in the introduction section, Rosen et al. [36] developed Commit-guru a toolthat relies on building models from historical commits using codeand process metrics (e.g., code complexity, the experience of thedevelopers, etc.) as the main features. There exist other studies thatleverage several code metric suites such as the CK metrics suite [5]or the Briand’s coupling metrics [2]. These metrics have been used,with success, to predict defects as shown by Subramanyam et al.
[40] and Gyimothy et al. [11].
Further improvements to these metrics have been proposed by
Nagappan et al. [29, 31] and Zimmerman et al. [46, 47] who usedcall graphs as the main artifact for computing code metrics with astatic analyzer.
Nagappan et al. proposed a technique that uses data mined fromsource code repository such as churns to assess the quality of achange [30]. Hassan et al and Ostrand et al used past changes anddefects to predict buggy locations [12], [33]. Their methods rely on
various heuristics to identify the locations that are most likely tointroduce a defect. Kim et al [22] proposed the bug cache approach,which is an improved technique over Hassan and Holt’s approach

Mathieu Nayrolles and Abdelwahab Hamou-Lhadj[12]. Rahman and Devanbu found that, in general, process-based
metrics perform as good as code-based metrics [35].
Other studies that aim to predict risky changes use the entropyof a given change [13, 41] and the size of the change combined withfiles being changed [18].
These techniques operate at different levels of the systems andmay require the presence of the entire source code. In addition, thereliance of metrics may result in high false positives rates. We needa way to validate whether a suspicious change is indeed risky. In
this paper, we address this issue using a two-phase process thatcombines the use of metrics to detect suspicious risky changes, andcode matching to increase the detect accuracy. As we will show
in the evaluation section, CLEVER reduces the number of false
positives while keeping good recall. In addition, CLEVER operates
at commit-time for preventing the introduction of faults before
they reach the code repository. Through interactions with Ubisoft
developers, we found that this integrates well with the workflow
of developers.

2.2

Automatic Patch Generation

One feature of CLEVER is the ability to propose fixes that can help
developers correct the detected risky commit. This is similar inprinciple to the work on automatic patch generation. Pan et al.
and Kim et al. proposed two approaches that extract and apply fix
patterns [20, 34]. Pan et al. identified 27 patterns and were able tofix 45.7% - 63.6% of bugs using one of the proposed patterns. The
patterns found by Kim et al. are mined from human-written patches
and were able to successfully generate patches for 27 out of 119
bugs. The tool by Kim et al., named PAR, is similar to the second
part of CLEVER where we propose fixes. Our approach also mines
potential fixes from human-written patches found in the historical
data. In our work, we do not generate patches, but instead propose
known patches to developers for further assessment. It has also
been shown that patch generation is useful in understanding anddebugging the causes of faults [42].
Despite the advances in the field of automatic patch generation,
this task remains overly complex. Developers expect from tools highquality patches that can be safely deployed. Many studies proposeda classification of what is considered an acceptable quality patch
for an automatically generated patch to be adopted in industry [7,24, 25].

3

THE CLEVER APPROACH

Figures 1, 2 and 3 show an overview of the CLEVER approach,which consists of two parallel processes.
In the first process (Figures 1 and 2), CLEVER manages events
happening on project tracking systems to extract fault-introducing
commits and commits and their corresponding fixes. For simplicity
reasons, in the rest of this paper, we refer to commits that are used
to fix defects as fix-commits. We use the term defect-commit to mean
a commit that introduces a fault.
The project tracking component of CLEVER listens to bug (or issue) closing events of Ubisoft projects. Currently, CLEVER is tested
on 12 large Ubisoft projects. These projects share many dependencies. We clustered them based on their dependencies with the aim to
improve the accuracy of CLEVER. This clustering step is importantCombining Code Metrics With Clone Detection For Faults Prevention and Resolution
in order to identify faults that may exist due to dependencies, while
enhancing the quality of the proposed fixes.

In the second process (Figure 3), CLEVER intercepts incoming
commits before they leave a developer’s workstation using theconcept of pre-commit hooks. A pre-commit hook is a script that is
executed at commit-time and it is supported by most major code
versioning systems such as Git. There are two types of hooks: clientside and server-side. Client-side hooks are triggered by operations
such as committing and merging, whereas server-side hooks run on
network operations such as receiving pushed commits. These hooks
can be used for different purposes such as checking compliance withcoding rules, or the automatic execution of unit tests. A pre-commit
hook runs before a developer specifies a commit message.
Ubisoft’s developers use pre-commit hooks for all sorts of reasons
such as identifying the tasks that are addressed by the commit athand, specifying the reviewers who will review the commit, andso on. Implementing this part of CLEVER as a pre-commit hook
is an important step towards the integration of CLEVER with theworkflow of developers at Ubisoft. The developers do not have todownload, install, and understand additional tools in order to use
CLEVER.
Once the commit is intercepted, we compute code and process
metrics associated with this commit. The selected metrics are discussed further in Section 3.2. The result is a feature vector (Step 4)that is used for classifying the commit as risky or non-risky.
If the commit is classified as non-risky, then the process stops,
and the commit can be transferred from the developer’s workstation
to the central repository. Risky commits, on the other hand, arefurther analysed in order to reduce the number of false positives(healthy commits that are detected as risky). We achieve this by
first extracting the code blocks that are modified by the developer
and then compare them to code blocks of known fault-introducing
commits.

3.1

Clustering Projects

We cluster projects according to their dependencies. The rationaleis that projects that share dependencies are most likely to containdefects caused by misuse of these dependencies. In this step, theproject dependencies are analysed and saved into a single NoSQL
graph database as shown in Figure 2. A node corresponds to a
project that is connected to other projects on which it depends.
Dependencies can be external or internal depending on whether
the products are created in-house or supplied by a third-party. For
confidentiality reasons, we cannot reveal the name of the projects
involved in the project dependency graph. We show the 12 projects
in yellow color with their dependencies in blue color in Figure 4. In
total, we discovered 405 distinct dependencies. Dependencies canbe internal (i.e. library developed at Ubisoft) or external (i.e. libraryprovided by third parties). The resulting partitioning is shown in
Figure 5.
At Ubisoft, dependencies are managed within the frameworkof a single repository, which makes their automatic extraction
possible. The dependencies could also be automatically retrieved if
the projects use a dependency manager such as Maven.

MSR ’18, May 28–29, 2018, Gothenburg, Sweden

Once the project dependency graph is extracted, we use a clustering algorithm to partition the graph. To this end, we choose theGirvan–Newman algorithm [10, 32], used to detect communities by
progressively removing edges from the original network. Insteadof trying to construct a measure that identifies the edges that are
the most central to communities, the Girvan–Newman algorithm
focuses on edges that are most likely “between” communities. Thisalgorithm is very effective at discovering community structure inboth computer-generated and real-world network data [32]. Other
clustering algorithms can also be used.

3.2

Building a Database of Code Blocks of
Defect-Commits and Fix-Commits

To build our database of code blocks that are related to defectcommits and fix-commits, we first need to identify the respective
commits. Then, we extract the relevant blocks of code from thecommits.
3.2.1 Extracting Commits: CLEVER listens to issue closing
events happening on the project tracking system used at Ubisoft.
Every time an issue is closed, CLEVER retrieves the commit thatwas used to fix the issue (the fix-commit) as well as the one that
introduced the defect (the defect-commit). To link fix-commits
and their related issues we implemented the well-known SZZ
algorithm presented by Kim et al. [21].
3.2.2 Extracting Code Blocks: Algorithm 1 presents an overview
of how to extract blocks. This algorithm receives as arguments, thechangesets and the blocks that have been previously extracted.
Then, Lines 1 to 5 show the f or loop that iterates over the changesets. For each changeset (Line 2), we extract the blocks by callingthe extract_blocks (Chanдeset cs) function. In this function, we
expand our changeset to the left and to the right in order to have acomplete block.
Data: Chanдeset[] changesets;
Block[] prior_blocks;
Result: Up to date blocks of the systems
1 for i ← 0 to size_o f chanдesets do
2
Block[] blocks ← extract_blocks (chanдesets);3for j ← 0 to size_o f blocks do
4
write blocks[j];
5
end
6 end
7 Function extract_blocks (Chanдeset cs)
8
if cs is unbalanced riдht then9cs ← expand_le f t (cs);10else if cs is unbalanced le f t then11cs ← expand_riдht (cs);
12
end
14
return txl_extract_blocks (cs);
Algorithm 1: Overview of the Extract Blocks Operation
As depicted by the diff below (not from Ubisoft), changesets contain only the modified chunk of code and not necessarily complete
blocks.

MSR ’18, May 28–29, 2018, Gothenburg, Sweden

Mathieu Nayrolles and Abdelwahab Hamou-Lhadj

Figure 1: Managing events happening on project tracking systems to extract defect-introducing commits and commits thatprovided the fixes

Figure 2: Clustering by dependency

Figure 3: Classifying incoming commits and proposing fixes@@ -315,36 +315,6 @@int initprocesstree_sysdep(ProcessTree_T **reference) {mach_port_deallocate(mytask,task);}}- if (task_for_pid(mytask, pt[i].pid,- &task) == KERN_SUCCESS) {mach_msg_type_number_t
count;
task_basic_info_data_ttaskinfo;
Therefore, we need to expand the changeset to the left (or right)
to have syntactically correct blocks. We do so by checking theblock’s beginning and ending with parentheses algorithms [3].

3.3

Building a Metric-Based Model

We adapted Commit-guru [36] for building the metric-based model.
Commit-guru uses a list of keywords proposed by Hindle et al.
[14] to classify commit in terms of maintenance, feature or fix.
Then, it uses the SZZ algorithm to find the defect-commit linked to
the fix-commit. For each defect-commit, Commit-guru computes

the following code metrics: la (lines added), ld (lines deleted), nf(number of modified files), ns (number of modified subsystems),nd (number of modified directories), en (distriubtion of modified
code across each file), lt (lines of code in each file (sum) before thecommit), ndev (the number of developers that modifed the files
in a commit), age (the average time interval between the last andcurrent change), exp (number of changes previously made by theauthor ), rexp (experience weighted by age of files (1 / (n + 1))),
sexp (previous changes made by the author in the same subsystem),loc (total number of modified LOC across all files), nuc (number of
unique changes to the files). Then, a statistical model is built using
the metric values of the defect-commits. Using linear regression,Commit-guru is able to predict whether incoming commits are risky
or not.
We had to modify Commit-guru to fit the context of this study.
First, we used information found in Ubisoft’s internal project tracking system to classify the purpose of a commit (i.e., maintenance,feature or fix). In other words, CLEVER only classifies a commit as adefect-commit if it is the root cause of a fix linked to a crash or a bugin the internal project tracking system. Using internal pre-commit
hooks, Ubisoft developers must link every commit to a given task

Combining Code Metrics With Clone Detection For Faults Prevention and Resolution

Figure 4: Dependency Graph

Figure 5: Clusters

MSR ’18, May 28–29, 2018, Gothenburg, Sweden

MSR ’18, May 28–29, 2018, Gothenburg, Sweden#ID. If the task #ID entered by the developer matches a bug or crash
report within the project tracking system, then we perform theSCM blame/annotate function on all the modified lines of code for
their corresponding files on the fix-commit’s parents. This returns
the commits that previously modified these lines of code and are
flagged as defect-commits. Another modification consists of theactual classification algorithm. We did not use linear regression but
instead the random forest algorithm [43, 44]. The random forest
algorithm turned out to be more effective as described in Section 5.
Finally, we had to rewrite Commit-guru in GoLang for performanceand internal reasons.

3.4

Comparing Code Blocks

Each time a developer makes a commit, CLEVER intercepts it using
a pre-commit hook and classifies it as risky or not. If the commit isclassified as risky by the metric-based classifier, then, we extract thecorresponding code block (in a similar way as in the previous phase),and compare it to the code blocks of historical defect-commits. If
there is a match, then the new commit is deemed to be risky. A
threshold α is used to assess the extent beyond which two commits
are considered similar.
To compare the extracted blocks to the ones in the database, weresort to clone detection techniques, more specifically, text-based
clone detection techniques. This is because lexical and syntactic
analysis approaches (alternatives to text-based comparisons) would
require a complete program to work, i.e., a program that compiles.
In the relatively wide-range of tools and techniques that exist to
detect clones by considering code as text [8, 16, 17, 27, 28, 45], wechose the NICAD clone detector because it is freely available andhas shown to perform well [6].
NICAD can detect Type 1, 2 and 3 software clones [19]. Type 1
clones are copy-pasted blocks of code that only differ from eachother in terms of non-code artifacts such as indentation, whitespaces, comments and so on. Type 2 clones are blocks of code that aresyntactically identical except literals, identifiers, and types that canbe modified. Also, Type 2 clones share the particularities of Type1 about indentation, whitespaces, and comments. Type 3 clones
are similar to Type 2 clones in terms of modification of literals,
identifiers, types, indentation, whitespaces, and comments but also
contain added or deleted code statements.
The problem with the current implementation of NICAD is that it
only considers complete Java, C#, and C files. We improved NICAD
to process blocks that come from commit-diffs. This is because thecurrent version of NICAD can only process syntactically correct
code and commit-diffs are, by definition, snippets that represent
modified regions of a given set of files.
By reusing NICAD, CLEVER can detect Types 3 software clones[19]. Type 3 clones can contain added or deleted code statements,
which make them suitable for comparing commit code blocks. In
addition, NICAD uses a pretty-printing strategy from where statements are broken down into several lines [38]. This functionality
allowed us to detect Segments 1 and 2 as a clone pair, as shown by
Table 1, because only the initialization of i changed. This specificexample would not have been marked as a clone by other tools wetested such as Duploc [8].

Mathieu Nayrolles and Abdelwahab Hamou-Lhadj
The extracted, pretty-printed, normalized filtered blocks aremarked as potential clones using a Longest Common Subsequence(LCS) algorithm [15]. Then, a percentage of unique statements canbe computed and, given the threshold α, the blocks are marked asclones.

3.5

Classifying Incoming Commits

As discussed in Section 3.3, a new commit goes through the metricbased model first (Steps 1 to 4). If the commit is classified as nonrisky, we simply let it through, and we stop the process. If thecommit is classified as risky, however, we continue the process with
Steps 5 to 8 in our approach.
One may wonder why we needed to have a metric-based modelin the first place. We could have resorted to clone detection as themain mechanism. The main reason for having the metric-basedmodel is efficiency. If each commit had to be analysed against
all known signatures using code clone similarity, then, it would
have made CLEVER more time consuming. We estimate that, in an
average workday (i.e. thousands of commits), if all commits had
to be compared against all signatures on the same cluster we used
for our experiments it would take around 25 minutes to process acommit with the current processing power dedicated to CLEVER.
In comparison, it takes, in average, 3.75 seconds with the current
two-step approach.

3.6

Proposing Fixes

An important aspect in the design of CLEVER is the ability toprovide guidance to developers on how to improve risky commits.
We achieve this by extracting from the database the fix-commit
corresponding to the top 1 matching defect-commits and present it
to the developer. We believe that this makes CLEVER a practical
approach. Developers can understand why a given modification
has been reported as risky by looking at code instead of simple
metrics as in the case of the studies reported in [18, 36].
Finally, using the fixes of past defects, we can provide a solution,in the form of a contextualised diff, to developers. A contextualised
diff is a diff that is modified to match the current workspace of thedeveloper regarding variable types and names. In Step 8 of Figure 3,
we adapt the matching fixes to the actual context of the developer
by modifying indentation depth and variable name in an effort to
reduce context switching. We believe that this would make it easierfor developers to understand the proposed fixes and see if it applies
in their situation.
All the proposed fixes will come from projects in the same cluster
as the project where the risky commit is. Thus, developers haveaccess to fixes that should be easier to understand as they comefrom projects similar to theirs inside the company.

4

CASE STUDY SETUP

In this section, we present the setup of our case study in terms of
repository selection, dependency analysis, comparison process andevaluation measures.

4.1

Project Repository Selection

In collaboration with Ubisoft developers, we selected 12 major software projects (i.e., systems) developed at Ubisoft to evaluate theCombining Code Metrics With Clone Detection For Faults Prevention and Resolution

MSR ’18, May 28–29, 2018, Gothenburg, Sweden

Table 1: Pretty-Printing Example
Segment 1
for (i = 0;
i >10;
i++)

Segment 2 Segment 3
for (
for (
i = 1;
j = 2;
i >10;
j >100;i++)j++)Total Matches
Total Mismatches

S1 & S2
1
0
1
1
3
1

S1 & S3
1
0
0
0
1
3

S2 & S3
1
0
0
0
1
3

effectiveness of CLEVER. These systems continue to be actively
maintained by thousands of developers. Ubisoft projects are organized by game engines. A game engine can be used in the development of many high-budget games. The projects selected for thiscase study are related to the same game engine. For confidentiality
and security reasons, neither the names nor the characteristics ofthese projects are provided. We can however disclose that the sizeof these systems altogether consists of millions of files of code,
hundreds of millions of lines of code and hundreds of thousands ofcommits. All 12 systems are AAA videos games.

4.2

Project Dependency Analysis

Figure 4 shows the project dependency graph. As shown in Figure
4, these projects are highly interconnected. A review of each cluster shows that this partitioning divides projects in terms of theirhigh-level functionalities. For example, one cluster is related to aparticular given family of video games, whereas the other cluster
refers to another family. We showed this partitioning to 11 experienced software developers and ask them to validate it. They all
agreed that the results of this automatic clustering are accurate
and reflects well the various project groups of the company. The
clusters are used for decreasing the rate of false positive. In addition,
fixes mined across projects but within the cluster are qualitative asshown in our experiments.

Figure 6: Process of Comparing New Commits

To build the database that we can use to assess the performanceof CLEVER, we use the same process as discussed in Section 3.2.
We retrieve the full history of each project and label commits as
defect-commits if they appear to be linked to a closed issue using
the SZZ algorithm [21]. This baseline is used to compute the precision and recall of CLEVER. Each time CLEVER classifies a commit
as risky; we can check if the risky commit is in the database of
defect-introducing commits. The same evaluation process is usedby related studies [1, 9, 18, 23, 26].

that we have three commits from two projects as presented by
Figure 6. At time t 1 , commit c 1 in project p1 introduces a defect.
The defect is experienced by a user that reports it via an issue i 1
at t 2 . A developer fixes the defect introduced by c 1 in commit c 2
and closes i 1 at t 3 . From t 3 we know that c 1 introduced a defectusing the process described in Section 4.3. If at t 4 , c 3 is pushed top2 and classified by the metric-based classifier as risky, we extractc 3 blocks and compare them with the ones of c 1 . If c3 and c1 are amatch after preprocessing, pretty-printing and formatting, then c 3
is classified as risky by CLEVER and c 2 is proposed to the developer
as a potential solution for the defect introduced in c 3 .
While this example explains the processes of CLEVER it does
not encompasse all the cases. Indeed, the user that experiences
the defect can be internal (i.e. another developer, a tester, . . . ) or
external (i.e. a player). In addition, many other projects receive
commits in parallel and they are all to be compared with all the
known signatures.

4.4

4.5

4.3

Building a Database of Defect-Commits
and Fix-Commits

Process of Comparing New Commits

Because our approach relies on commit pre-hooks to detect risky
commits, we had to find a way to replay past commits. To do so,
we cloned our test subjects, and then created a new branch called
CLEVER. When created, this branch is reinitialized at the initial stateof the project (the first commit), and each commit can be replayed asthey have originally been. For each commit, we store the time taken
for CLEVER to run, the number of detected clone pairs, and thecommits that match the current commit. As an example, suppose

Evaluation Measures

Similar to prior work (e.g., [18, 41]), we used precision, recall, andF1 -measure to evaluate our approach. They are computed using TP(true positives), FP (false positives), FN (false negatives), which aredefined as follows:• TP is the number of defect-commits that were properly classified by CLEVER• FP is the number of healthy commits that were classified by
CLEVER as risky

MSR ’18, May 28–29, 2018, Gothenburg, Sweden• FN is the number of defect introducing-commits that were
not detected by CLEVER• Precision: TP / (TP + FP)• Recall: TP / (TP + FN)• F1 -measure: 2 × (precision × recall )/(precision + recall )It is worth mentioning that, in the case of defect prevention, false
positives can be hard to identify as the defects could be in the code
but not yet reported through a bug report (or issue). To address
this, we did not include the last six months of history. Following
similar studies [4, 18, 36, 39], if a defect is not reported within six
months then it is not considered.

5

CASE STUDY RESULTS

In this section, we show the effectiveness of CLEVER in detecting
risky commits using a combination of metric-based models andclone detection. The main research question addressed by this casestudy is: Can we detect risky commits by combining metrics and code
comparison within and across related Ubisoft projects, and if so, what
would be the accuracy?
The experiments took nearly two months using a cluster of six 12
3.6 Ghz cores with 32GB of RAM each. The most time consuming
part of the experiment consists of building the baseline as eachcommit must be analysed with the SZZ algorithm. Once the baseline
was established, the model built, it took, on average, 3.75 seconds
to analyse an incoming commit on our cluster.
In the following subsections, we provide insights on the performance of CLEVER by comparing it to Commit-guru [36] alone, i.e.,
an approach that relies only on metric-based models. We chose
Commit-guru because it has been shown to outperform other techniques (e.g., [18, 23]). Commit-guru is also open source and easy to
use.

5.1

Performance of CLEVER

When applied to 12 Ubisoft projects, CLEVER detects risky commits with an average precision, recall, and F1-measure of 79.10%,
a 65.61%, and 71.72% respectively. For clone detection, we used athreshold of 30% . This is because Roy et al. [37] showed throughempirical studies that using NICAD with a threshold of around 30%,
the default setting, provides good results for the detection of Type 3
clones. When applied to the same projects, Commit-guru achieves
an average precision, recall, and F1-measure of 66.71%, 63.01% and64.80%, respectively.
We can see that with the second phase of CLEVER (clone detection) there is considerable reduction in the number of false positives(precision of 79.10% for CLEVER compared to 66.71% for Commitguru) while achieving similar recall (65.61% for CLEVER compared
to 63.01% for Commit-guru).

5.2

Analysis of the Quality of the Fixes
Proposed by CLEVER

In order to validate the quality of the fixes proposed by CLEVER,
we conducted an internal workshop where we invited a number ofpeople from Ubisoft development team. The workshop was attended
by six participants: two software architects, two developers, one
technical lead, and one IT project manager. The participants have
many years of experience at Ubisoft.

Mathieu Nayrolles and Abdelwahab Hamou-Lhadj
The participants were asked to review 12 randomly selected
fixes that were proposed by CLEVER. These fixes are related to one
system in which the participants have excellent knowledge. We
presented them with the original buggy commits, the original fixes
for these commits, and the fixes that were automatically extractedby CLEVER. We asked them the following question “Is the proposed
fix applicable in the given situation?” for each fix.
The review session took around 50 minutes. This does not include
the time it took to explain the objective of the session, the setup,the collection of their feedback, etc.
We asked the participants to rank each fix proposed by CLEVER
using this scheme:• Fix Accepted: The participant found the fix proposed by
CLEVER applicable to the risky commit.
• Unsure: In this situation, the participant is unsure about
the relevance of the fix. There might be a need for more
information to arrive to a verdict.
• Fix Rejected: The participant found the fix is not applicableto the risky commit.
Table 2 shows answers of the participants. The columns referto the fixes proposed by CLEVER, whereas the rows refer to theparticipants that we denote using P1, P2, . . . , P6. As we can see fromthe table, 41.6% of the proposed fixes (F1, F3, F6, F10 and F12) havebeen accepted by all participants, while 25% have been accepted byat least one member (F4, F8, F11). We analysed the fixes that were
rejected by some or all participants to understand the reasons.
F 2 was rejected by our participants because the region of thecommit that triggered a match is a generated code. Although thisgenerated code was pushed into the repositories as part of bug fixing
commit, the root cause of the bug lies in the code generator itself.
Our proposed fix suggests to update the generated code. Because
the proposed fix did not apply directly to the bug and the question
we ask our reviewers was “Is the proposed fix applicable in the given
situation?” they rejected it. In this occurrence, the proposed fix was
not applicable.
F 4 was accepted by two reviewers and marked as unsure by
the other participants. We believe that this was due the lack ofcontext surrounding the proposed fix. The participants were unable
to determine if the fix was applicable or not without knowing what
the original intent of the buggy commit was. In our review session,
we only provided the reviewers with the regions of the commits thatmatched existing commits and not the full commit. Full commits
can be quite lengthy as they can contain asset descriptions andgenerated code, in addition to the actual code. In this occurrence,
the full context of the commit might have helped our reviewers
to decide if F 4 was applicable or not. F 5 and F 7 were classified as
unsure by all our participants for the same reasons.
F 8 was rejected by four of participants and accepted by two. The
participants argued that the proposed fix was more a refactoring
opportunity than an actual fix.
F 12 was marked as unsure by all the reviewers because the code
had to do with a subsystem that is maintained by another team andthe participants felt that it was out of scope of this session.
After the session, we asked the participants two additional questions: Will you use CLEVER in the future? and What aspects of
CLEVER need to be improved?

Combining Code Metrics With Clone Detection For Faults Prevention and Resolution

MSR ’18, May 28–29, 2018, Gothenburg, Sweden

Table 2: Workshop results
P1
P2
P3
P4
P5
P6

F1
Accepted
Accepted
Accepted
Accepted
Accepted
Accepted

F2
Rejected
Rejected
Rejected
Rejected
Rejected
Rejected

F3
Accepted
Accepted
Accepted
Accepted
Accepted
Accepted

F4
Accepted
UnsureUnsureUnsureAccepted
UnsureF5
UnsureUnsureUnsureUnsureUnsureUnsureF6
Accepted
Accepted
Accepted
Accepted
Accepted
Accepted

All the participants answered the first question favourably. They
also proposed to embed CLEVER with Ubisoft’s quality assurance
tool suite. The participants reported that the most useful aspects of
CLEVER are:• Ability to leverage many years of historical data of interrelated projects, hence allowing development teams to share
their experiences in fixing bugs.
• Easy integration of CLEVER into developers’ work flow
based on the tool’s ability to operate at commit-time.
• Precision and recall of the tool (79% and 65% respectively)
demonstrating CLEVER’s capabilities to catch many defects
that would otherwise end up in the code repository. For thesecond question, the participants proposed to add a feedback
loop to CLEVER where the input of software developers
is taken into account during classification. The objective
is to reduce the number of false negatives (risky commitsthat are flagged as non-risky) and false positives (non-risky
commits that are flagged as risky). The feedback loop mechanism would work as follows: When a commit is misclassified
by the tool, the software developer can choose to ignore
CLEVER’s recommendation and report the misclassified commit. If the fix proposition is not used, then, we would givethat particular pattern less strength over other patterns automatically.
We do not need manual input from the user because CLEVER
knows the state of the commit before the recommendation and after.
If both versions are identical then we can mark the recommendation
as not helpful. This way, we can also compensate for human error(i.e., a developer rejecting CLEVER recommendation when thecommit was indeed introducing a defect. We would know this byusing the same processes that allowed us to build our database of
defect-commits as described in Section 3.2. This feature is currently
under development.
It is worth noting that Ubisoft developers who participated to thisstudy did not think that CLEVER fixes that were deemed irrelevant
were a barrier to the deployment of CLEVER. In their point of view,
the performance of CLEVER in terms of classification should makea significant impact as suspicious commits will receive extended
reviews and/or further investigations.
We are also investigating the use of adaptive learning techniques
to improve the classification mechanism of CLEVER. In addition tothis, the participants discussed the limitation of CLEVER as to its inability to deal with automatically generated code. We are currently
working with Ubisoft’s developers to address this limitation.

F7
UnsureUnsureUnsureUnsureUnsureUnsure5.3

F8
Rejected
Rejected
Rejected
Accepted
Rejected
Accepted

F9
Rejected
Rejected
Rejected
Rejected
Rejected
Rejected

F10
Accepted
Accepted
Accepted
Accepted
Accepted
Accepted

F11
Accepted
Accepted
Accepted
Accepted
Accepted
Accepted

F12
UnsureUnsureUnsureUnsureUnsureUnsureDeployment of CLEVER at Ubisoft

CLEVER will soon be rolled out at Ubisoft. It will be made available
to thousands of developers across various divisions. Our researchteam provided on-site training of this new tool. In addition, Ubisoft
developed an instructional video to support the launch of CLEVER
and raise awareness about the tool. Our research team is currently
monitoring the use of CLEVER at Ubisoft to evaluate its adoption(usage, barriers, etc.).

6

DISCUSSION

In this section, we share the lessons learned, discuss the limitationsof CLEVER, and present threats to validity of our study.

6.1

Lessons Learned

6.1.1 Understanding the industrial context: Throughout the design of CLEVER, we made many design decisions that were triggered by the discussions we had with Ubisoft developers. Some of
the key decisions that we made included having CLEVER operate
on clusters of inter-related systems and combining metric-basedand code matching techniques into a two-phase approach. These
decisions were not only critical in obtaining an improved accuracy,
but also in proposing effective fixes that guide developers. From ourinteractions with Ubisoft developers, it was also important for us tocome up with a solution that integrates well with the workflow of
Ubisoft developers. This motivated the use of commit-time and theintegration of CLEVER with Ubisoft version control systems. The
key lesson here is the importance of understanding the industrial
context by working with the company’s development teams. This
collaboration is also an enabler for the adoption of tools, developedin the context of research projects.
6.1.2 Leveraging an iterative process: Throughout this research
project, we followed an iterative and incremental process. The
results of each iteration were presented to Ubisoft developers for
feedback. Adjustments were made as needed, before the subsequent
iteration started. This process was not only helpful in keeping the
project on track, but also in producing “quick wins” as a way ofshowing practical results from each iteration. Examples of such“quick wins” include the creation of the defect introduction landscape at Ubisoft in terms of number defects, time to fix defects
and time to discover defects organization wide. In addition, thecomputed clusters of similar projects turned out to be useful for
upper management in order to organize collaborations between
teams belonging to different projects.

MSR ’18, May 28–29, 2018, Gothenburg, Sweden
6.1.3 Communicating effectively: During the development of
CLEVER, we needed to constantly communicate the steps of our research to developers and project owners. It was important to adopt
a communication strategy suitable to each stakeholder. For example,
in our meetings with management, we focused more on the abilityof CLEVER to improve code quality and reduce maintenance costs
instead of the technical details of the proposed approach. Developers, on the other hand, were interested in the potential of CLEVER
and its integration with their work environment.
6.1.4 Underestimating the time needed for full deployment of
CLEVER:. Part of our mandate was to develop a working tool. It took
a tremendous amount of time and effort to develop a production
tool version of CLEVER, which required a lot of engineering work.
We recognize that we underestimated the complexity of this task.
Examples of deliverables we had to produce include automating
the acquisition of new commits, presenting the recommendations
to the developers, building grammars for various programming
languages, creating APIs that interact with any types of clientsystems, authentication, and authorization of end-users, etc. Overall,
the machine learning code represents less than 5% of our code base.
The lesson here is to manage expectations and to better estimate
the project time and effort from an end to end perspective, and not
only the research part.

6.2

Limitations

Mathieu Nayrolles and Abdelwahab Hamou-Lhadj
Finally, part of the analysis of the CLEVER proposed fixes that
we did was based on manual comparisons of the CLEVER fixes withthose proposed by developers with a focus group composed of experienced engineers and software architects. Although, we exercised
great care in analysing all the fixes, we may have misunderstoodsome aspects of the commits.
In conclusion, internal and external validity have both beenminimized by choosing a set of 12 different systems, using input
data that can be found in any programming languages and version
systems (commits and changesets).

7

CONCLUSION

In this paper, we presented CLEVER (Combining Levels of Bug
Prevention and Resolution Techniques), an approach that detects
risky commits (i.e., a commit that is likely to introduce a bug)with an average of 79.10% precision and a 65.61% recall. CLEVER
combines code metrics, clone detection techniques, and project
dependency analysis to detect risky commits within and acrossprojects. CLEVER operates at commit-time, i.e., before the commits
reach the central code repository. Also, because it relies on code
comparison, CLEVER does not only detect risky commits but also
makes recommendations to developers on how to fix them. We
believe that this makes CLEVER a practical approach for preventing
bugs and proposing corrective measures that integrate well withthe developer’s workflow through the commit mechanism. CLEVER
is still in its infancy and we expect it to be available this year to
thousands of developers.
As future work, we want to build a feedback loop between theusers and the clusters of known buggy commits and their fixes. If a
fix is never used by the end-users, then we could remove it fromthe clusters and improve our accuracy. We also intend to improve
CLEVER to deal with generated code. Moreover, we will investigate
how to improve the fixes proposed by CLEVER to add contextual
information to help developers better assess the applicability of thefixes.

We identified two main limitations of our approach, CLEVER, which
require further studies.
CLEVER is designed to work on multiple related systems. Applying CLEVER to a single system will most likely be less effective.
The two-phase classification process of CLEVER would be hinderedby the fact that it is unlikely to have a large number of similar bugs
within the same system. For single systems, we recommend theuse of metric-based models. A metric-based solution, however, may
turn to be ineffective when applied across systems because of the
difficulty associated with identifying common thresholds that are
applicable to a wide range of systems.
The second limitation we identified has to do with the fact that
CLEVER is designed to work with Ubisoft systems. Ubisoft uses
C#, C, C++, Java and other internally developed languages. It is
however common to have other languages used in an environment
with many inter-related systems. We intend to extend CLEVER toprocess commits from other languages as well.

We are thankful to the software development team at Ubisoft for
their participations to the study and their assessment of the effectiveness of CLEVER. We are also thankful to NSERC (Natural
Sciences and Engineering Research Concil of Canada) which financed part of this research.

6.3

REFERENCES

Threats to Validity

The selection of target systems is one of the common threats tovalidity for approaches aiming to improve the analysis of softwaresystems. It is possible that the selected programs share common
properties that we are not aware of and therefore, invalidate ourresults. Because of the industrial nature of this study, we had towork with the systems developed by the company.
The programs we used in this study are all based on the C#,
C, C++ and Java programming languages. This can limit the generalization of the results to projects written in other languages,
especially that the main component of CLEVER is based on code
clone matching.

ACKNOWLEDGMENTS[1] Bhattacharya, P. and Neamtiu, I. 2011. Bug-fix time prediction
models: can we do better? Proceeding of the international conferenceon mining software repositories (New York, New York, USA), 207–
210.
[2] Briand, L. et al. 1999. A unified framework for coupling measurement in object-oriented systems. IEEE Transactions on Software Engineering. 25, 1 (1999), 91–121. DOI:https://doi.org/10.1109/32.748920.
[3] Bultena, B. and Ruskey, F. 1998. An Eades-McKay algorithm forwell-formed parentheses strings. Information Processing Letters. 68,
5 (1998), 255–259.

Combining Code Metrics With Clone Detection For Faults Prevention and Resolution

MSR ’18, May 28–29, 2018, Gothenburg, Sweden[4] Chen, T.-h. et al. 2014. An Empirical Study of Dormant Bugs
Categories and Subject Descriptors. Proceedings of the international
conference on mining software repository 82–91.

[19] Kapser, C. and Godfrey, M.W. 2003. Toward a Taxonomy ofClones in Source Code: A Case Study. International workshop on
evolution of large scale industrial software architectures 67–78.

[5] Chidamber, S. and Kemerer, C. 1994. A metrics suite for object
oriented design. IEEE Transactions on Software Engineering. 20, 6(Jun. 1994), 476–493. DOI:https://doi.org/10.1109/32.295895.

[20] Kim, D. et al. 2013. Automatic patch generation learned fromhuman-written patches. Proceedings of the international conferenceon software engineering 802–811.

[6] Cordy, J.R. and Roy, C.K. 2011. The NiCad Clone Detector. Proceedings of the international conference on program comprehension
219–220.

[21] Kim, S. et al. 2006. Automatic Identification of Bug-Introducing
Changes. Proceedings of the international conference on automatedsoftware engineering 81–90.

[7] Dallmeier, V. et al. 2009. Generating Fixes from Object Behavior
Anomalies. Proceedings of the international conference on automatedsoftware engineering 550–554.

[22] Kim, S. et al. 2007. Predicting Faults from Cached History.
Proceedings of the international conference on software engineering
489–498.

[8] Ducasse, S. et al. 1999. A Language Independent Approachfor Detecting Duplicated Code. Proceedings of the international
conference on software maintenance 109–118.

[23] Kpodjedo, S. et al. 2010. Design evolution metrics for defectprediction in object oriented systems. Empirical Software Engineering. 16, 1 (Dec. 2010), 141–175. DOI:https://doi.org/10.1007/
s10664-010-9151-7.

[9] El Emam, K. et al. 2001. The prediction of faulty classes using
object-oriented design metrics. Journal of Systems and Software.
56, 1 (Feb. 2001), 63–75. DOI:https://doi.org/10.1016/S0164-1212(00)00086-8.
[10] Girvan, M. and Newman, M.E.J. 2002. Community structure insocial and biological networks. Proceedings of the National Academyof Sciences. 99, 12 (Jun. 2002), 7821–7826. DOI:https://doi.org/10.
1073/pnas.122653799.
[11] Gyimothy, T. et al. 2005. Empirical validation of object-oriented
metrics on open source software for fault prediction. IEEE Transactions on Software Engineering. 31, 10 (Oct. 2005), 897–910. DOI:https://doi.org/10.1109/TSE.2005.112.

[24] Le Goues, C. et al. 2012. A systematic study of automatedprogram repair: Fixing 55 out of 105 bugs for $8 each. Proceedingsof the international conference on software engineering 3–13.
[25] Le, X.-B.D. et al. 2015. Should fixing these failures be delegated to automated program repair? Proceedings of the international
symposium on software reliability engineering 427–437.
[26] Lee, T. et al. 2011. Micro interaction metrics for defect prediction. Proceedings of the european conference on foundations ofsoftware engineering (New York, New York, USA), 311–231.
[27] Manber, U. 1994. Finding similar files in a large file system.
Proceedings of the usenix winter 1–10.

[12] Hassan, A. and Holt, R. 2005. The top ten list: dynamic fault
prediction. Proceedings of the international conference on software
maintenance 263–272.

[28] Marcus, A. and Maletic, J. 2001. Identification of high-level
concept clones in source code. Proceedings international conferenceon automated software engineering 107–114.

[13] Hassan, A.E. 2009. Predicting faults using the complexity ofcode changes. Proceedings of the international conference on software
engineering 78–88.

[29] Nagappan, N. and Ball, T. 2005. Static analysis tools as early
indicators of pre-release defect density. Proceedings of the international conference on software engineering (New York, New York,USA), 580–586.

[14] Hindle, A. et al. 2008. What do large commits tell us?: a taxonomical study of large commits. Proceedings of the international
workshop on mining software repositories (New York, New York,USA), 99–108.
[15] Hunt, J.W. and Szymanski, T.G. 1977. A fast algorithm forcomputing longest common subsequences. Communications of theACM. 20, 5 (May 1977), 350–353. DOI:https://doi.org/10.1145/359581.
359603.
[16] Johnson, J.H. 1993. Identifying redundancy in source codeusing fingerprints. Proceedings of the conference of the centre foradvanced studies on collaborative research 171–183.

[30] Nagappan, N. and Ball, T. 2005. Use of relative code churnmeasures to predict system defect density. Proceedings of the international conference on software engineering 284–292.
[31] Nagappan, N. et al. 2006. Mining metrics to predict component failures. Proceeding of the international conference on software
engineering (New York, New York, USA), 452–461.
[32] Newman, M.E.J. and Girvan, M. 2004. Finding and evaluating
community structure in networks. Physical Review E. 69, 2 (Feb.
2004), 026113. DOI:https://doi.org/10.1103/PhysRevE.69.026113.

[17] Johnson, J.H. 1994. Visualizing textual redundancy in legacy
source. Proceedings of the conference of the centre for advanced studies
on collaborative research 32.

[33] Ostrand, T. et al. 2005. Predicting the location and numberof faults in large software systems. IEEE Transactions on Software
Engineering. 31, 4 (Apr. 2005), 340–355. DOI:https://doi.org/10.1109/
TSE.2005.49.

[18] Kamei, Y. et al. 2013. A large-scale empirical study of just-intime quality assurance. IEEE Transactions on Software Engineering.
39, 6 (Jun. 2013), 757–773. DOI:https://doi.org/10.1109/TSE.2012.70.

[34] Pan, K. et al. 2008. Toward an understanding of bug fix patterns. Empirical Software Engineering. 14, 3 (Aug. 2008), 286–315.
DOI:https://doi.org/10.1007/s10664-008-9077-5.

MSR ’18, May 28–29, 2018, Gothenburg, Sweden

Mathieu Nayrolles and Abdelwahab Hamou-Lhadj[35] Rahman, F. and Devanbu, P. 2013. How, and why, process
metrics are better. Proceedings of the international conference on
software engineering 432–441.

[42] Tao, Y. et al. 2014. Automatically generated patches as debugging aids: a human study. Proceedings of the international symposium
on foundations of software engineering 64–74.

[36] Rosen, C. et al. 2015. Commit guru: analytics and risk predictionof software commits. Proceedings of the joint meeting on foundationsof software engineering (New York, New York, USA), 966–969.

[43] Tin Kam Ho 1995. Random decision forests. Proceedings ofthe international conference on document analysis and recognition
278–282.

[37] Roy, C. and Cordy, J. 2008. NICAD: Accurate Detection of NearMiss Intentional Clones Using Flexible Pretty-Printing and CodeNormalization. 2008 16th iEEE international conference on program
comprehension 172–181.

[44] Tin Kam Ho 1998. The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence. 20, 8 (1998), 832–844. DOI:https://doi.org/10.1109/
34.709601.

[38] Roy, C.K. 2009. Detection and Analysis of Near-Miss Software
Clones. Queen’s University.

[45] Wettel, R. and Marinescu, R. 2005. Archeology of code duplication: recovering duplication chains from small duplication
fragments. Proceedings of the seventh international symposium on
symbolic and numeric algorithms for scientific computing 63–71.

[39] Shivaji, S. et al. 2013. Reducing Features to Improve CodeChange-Based Bug Prediction. IEEE Transactions on Software Engineering. 39, 4 (2013), 552–569.
[40] Subramanyam, R. and Krishnan, M. 2003. Empirical analysis of
CK metrics for object-oriented design complexity: implications forsoftware defects. IEEE Transactions on Software Engineering. 29, 4(Apr. 2003), 297–310. DOI:https://doi.org/10.1109/TSE.2003.1191795.
[41] Sunghun Kim, S. et al. 2008. Classifying Software Changes:
Clean or Buggy? IEEE Transactions on Software Engineering. 34, 2(Mar. 2008), 181–196. DOI:https://doi.org/10.1109/TSE.2007.70773.

[46] Zimmermann, T. and Nagappan, N. 2008. Predicting defects
using network analysis on dependency graphs. Proceedings of theinternational conference on software engineering (New York, New
York, USA), 531.
[47] Zimmermann, T. et al. 2007. Predicting Defects for Eclipse.
Proceedings of the international workshop on predictor models insoftware engineering 9.