Improving Code Review Effectiveness through
Reviewer Recommendations
Patanamon Thongtanunam* , Raula Gaikovina Kula† , Ana Erika Camargo Cruz* ,Norihiro Yoshida* , Hajimu Iida*†*

Nara Institute of Science and Technology,Japan{patanamon-t,camago,yoshida}@is.naist.jp,
iida@itc.naist.jp

Osaka University, Japanraula-k@ist.osaka-u.ac.jp

ABSTRACTa support tool [7] to accommodate a wide range of knowledgeof developers in large scale projects.
Recently, a reviewer recommendation tool (called Review
Bot) has been proposed by Balachandran [3]. This tool isdeveloped to reduce human effort for modern peer code review
in the industrial setting of VMware. The recommendation
algorithm of this tool uses a source code change history, which
considers line-by-line code modification history. Appropriate
reviewers are distinguished as having recently examined thesame line in the history. Also, time prioritization was used
to improve the performance.
The Review Bot’s algorithm seems best suited to thoseprojects with frequent changes of source code. However, we
believe that this granularity of code change history can belimited. Since modern peer code review is relatively new,most projects that adopted this technique would alreadybe in the maintenance phase. This makes most of codechange history unavailable. Additionally attributes such as
its large-scale codebase as well as the many diverse global
development teams may be influencing factors. Therefore, it
is challenging to find appropriate reviewers for those projects
with lacking of code changes history.
In this paper, we propose a novel recommendation algorithm, called File Path Similarity (FPS) algorithm. The
algorithm determines similarity of reviews based on the location of changed file (file path). Our key assumption is thatfiles that are located in similar file paths would be managed
and reviewed by similar experienced expert code reviewers.
The motivation behind this is that in most large systems,
like the Linux kernel, the directory structure loosely mirrors
the system architecture and files with similar functions areusually located in the same or near directories [4].
The performance of FPS algorithm was evaluated using
three distributed Open Source Software (OSS) projects: Android Open Source Project (AOSP), OpenStack, and Qt. In
particular, we addressed the following research questions as
a guideline: RQ1.) Does FPS outperform the Review Bot’s
algorithm?, and RQ2.) Does a recent modification history
also influence the accuracy of the algorithms?.
We measured the accuracy of the algorithms for the top-1,
top-3 and top-5 recommendations at different time priorities.
Our findings confirmed that our FPS algorithm (accurate up
to 77.97%) outperformed Review Bot (accurate up to 38.9%).
The results suggest that in an distributed environment, theanalysis at the directory structure level is more effective thanat source code level. We believe that this study opens new in-

Effectively performing code review increases the quality ofsoftware and reduces occurrence of defects. However, thisrequires reviewers with experiences and deep understandingsof system code. Manual selection of such reviewers can bea costly and time-consuming task. To reduce this cost, we
propose a reviewer recommendation algorithm determining
file path similarity called FPS algorithm. Using three OSS
projects as case studies, FPS algorithm was accurate up
to 77.97%, which significantly outperformed the previous
approach.

Categories and Subject Descriptors
D.2.7 [Software Engineering]: Distribution, Maintenance,and Enhancement

General Terms
Algorithm

Keywords
Peer Code Review; Recommendation System; Open Source
Software; Software Quality;

1.

INTRODUCTION

Distributed software development, especially with global
collaboration through online tools, is increasingly used insoftware firms [8]. This has led to the birth of a modern peer
code review process [2]. Modern peer code review is a toolsupported review that currently used in both industrial andOpen Source Software (OSS) projects. This is an effective
technique for defect detection, which requires reviewers withexperience and a deep understanding of the related systemcode [1]. However, in a much larger community and relatively
impersonal landscape of distributed software development,
manual selection of an appropriate reviewer can be a laboredand time-consuming task [6]. Thus, there is a growing need ofPermission to make digital or hard copies of all or part of this work for personal orclassroomusetois makegranteddigital
withoutorfeehard
provided
thatofcopies
not madeor work
distributed
Permission
copiesall orarepartof thisforfor profit or commercial advantage and that copies bear this notice and the full citation
personal
orclassroomuseisgrantedwithout
fee
providedthat
copiesareon the first page. Copyrights for components of this work owned by others than ACM
not madeor distributed
forwithprofitor iscommercial
and thatcopiesmust
be honored.
Abstracting
creditpermitted. Toadvantage
copy otherwise,or republish,topostthison noticeservers andor totheredistributeto lists,
specific
permission
and/ortoa
bear
full citation
onrequires
the firstpriorpage.
To copyotherwise,
fee.
Requesttopermissions
from Permissions@acm.org.
republish,
post on servers
or to redistribute to lists, requires prior specificpermission and/or
CHASE’14,
June 2a–fee.
June 3, 2014, Hyderabad, India
CHASE ’14,
JuneACM2 – June
3, 2014, Hyderabad, India
Copyright
2014
978-1-4503-2860-9/14/06...$15.00
http://dx.doi.org/10.1145/2593702.2593705
Copyright 14 ACM 978-1-4503-2860-9/14/06 ...$15.00.

119Approveralgorithm returns a list of the top-k candidates ordered bytheir file path similarity scores.

Verifier

Code Reviewer

Algorithm 1 RecommendReviewers(Rn , k)AuthorModified
files

Modified
Modified
files
files
Modified
Modified
files
Modified
files
Modified
files
files

Review Request

Review Request

Approver accepted

List of Review Requests

1: candidates ← list()
2: pastReviewList ← getP astReviews(Rn )3: m ← 0
4: for Review Rp : pastReviewList do
5:score ← F P S(Rn , Rp , m)
6:
for Reviewer r : getReviewers(Rp ) do
7:
candidates[r] ← candidates[r] + score8:
end for
9:m←m+1
10: end for
11: candidates.sort()
12: return candidates[0 : k]Modified
files

Code
Repository

Verifier accepted

Figure 1: Code Review Process of Gerrit (Simplified Version)sights into reviewer recommendation systems for distributed
software projects.

2.

MODERN PEER CODE REVIEWIn line 2, a list of past reviews of Rn is retrieved. The
past reviews in this list are the reviews that closed by either
being accepted or rejected, before the creation date of Rn .
This list is sorted by the creation date of the past reviewsin reverse chronological order. In line 3, the value of m isinitialized to zero. This variable help us to score past reviews
when time prioritization is considered; so that as its value
increases, the past review being scored is older and a minor
score is given. In lines 4-10, the FPS function is iterated
for every past review. In lines 6-8, for each past review, itsreviewers are retrieved and assigned as candidates. Theirscores are increased by the FPS score of this past review. In
line 11, the list of candidates is sorted in descending order
based on their scores. Line 12 returns the top k candidates
with the highest score from the sorted list.
The calculation of FPS function is described in Equation
1. This calculates a score of a past review (Rp ) from anaverage of similarity of every file in Rp (fp ) comparing withevery file in Rn (fn ). The Files function returns a set of filepaths of the input review. The Similarity(fn , fp ) function
measures the similarity between fp and fn , using Equation 2.
The averaged similarity score is prioritized by m and δ value.
Same as time prioritization of the Review Bot’s algorithm,
the δ parameter is a time prioritization factor ranging (0, 1].
When δ = 1, the time prioritization is not considered.
P
Similarity(fn , fp )A modern peer code review refers to a software inspection
method that is 1) informal (contrast to traditional method),2) tool-based, and 3) currently used in both industrial and
OSS development [2]. Since practitioners of this method cancoordinate asynchronously through a review tool, the adoption to distributed software development would be effortless.
The tool that our case study projects use is Gerrit1 The
process of this tool is shown Fig. 1. It begins with an author
creating a patch and submitting a set of new or modified files
as a review request to the system. This request is then added
to a awaiting list to be reviewed. An author can manuallyassign reviewers to reduce the waiting time. However, theassigned reviewers also can ultimately decide whether toreview the request, based on their experience and interests [9].
In Gerrit, the reviewers (i.e. Code reviewers, Approvers
and Verifiers) can examine the patch of a review request
with different permissions. Code reviewers can give opinion
and comments. Approvers mainly determine the quality
and impact of the changes. Verifiers are responsible for
integration testing, which is normally automated by Gerrit.
Code reviewers can be anyone in the project while approvers
and verifiers are experienced reviewers designated by project
leads. In the final step of the review process, a patch canbe merged into the projects only when the review request is
accepted by at least one approver and one verifier.
A code review will be effective if an author requests appropriate reviewers. However, without information about
available reviewers in Gerrit, it is difficult for the authors
to select those reviewers. Therefore, an automatic reviewer
recommendation system should allow code reviews to be
performed with less time and effort.

FPS(Rn , Rp , m) =fn ∈Files(Rn ),fp ∈Files(Rp )Similarity(fn , fp ) =3.

REVIEWER RECOMMENDATION|Files(Rn )| × |Files(Rp )|× δ m (1)commonPath(fn , fp )max(Length(fn ), Length(fp ))(2)

In Equation 2, the commonPath(fn , fp ) function counts
the number of common directory and/or file name that appear in both file path from the beginning. This count isbased on the assumption that files, which are under thesame directory, would have the similar function. Thus, thefirst directory of file paths is compared firstly. Then, theother components of file path are compared respectively. For
example, suppose fn is /src/camera/video/a.java and fp
is /src/camera/photo/a.java. The common path will be/src/camera and the commonPath(fn , fp ) returns 2. The
count of commonPath(fn , fp ) can be formularized as Equa-

Based on the assumption, FPS algorithm selects candidates from reviewers who had examined files with similar
directory paths. Furthermore, this algorithm also uses timeprioritization in the same way as the Review Bot’s algorithm.
A detail of FPS algorithm is described in Algorithm 1. Thisalgorithm takes two inputs: a new review request (Rn ) andthe number of top candidates (k) to be recommended. The
1

AOSP: https://android-review.googlesource.com, and
OpenStack: https://review.openstack.org, and
Qt: https://codereview.qt-project.org/

120

tion 3, where the values of i and j are initial from 0.
commonPath(fn , fp , i + 1,


j + 1) + 1, if fn [i] = fp [j]commonPath(fn , fp , i, j) = 0otherwise(3)
Continuing to the previous example, the Similarity(fn , fp )returns the value of common path number normalized by the2= 0.5
maximum length of these two file path, i.e. max(4,4)

4.

mendations. For the test, we determined Top-5 accuracy ofboth algorithms where δ = 1 for every number of reviews.
The recommendation accuracy of both algorithms were plotted in Fig. 2. In these charts, the x-axis represents thenumber of reviews N that the algorithms had recommended.
The y-axis represents the accuracy calculated corresponding
to that number of reviews. We took a one-side statistical
test with null hypothesis H0 : No statistical difference inthe recommendation accuracy percentages of both algorithms
and the greater case for alternative hypothesis. We used apaired-wise Mann-Whitney significance test since the data
did not follow a normal distribution. From the tests against
all three projects, H0 can be rejected as their all p-values
are less than significant value, i.e. α < 0.05.
For RQ2: Does a recent modification history also
influence the accuracy of the algorithms?, the resultsin Table 2 interestingly show that FPS algorithm without
time prioritization factor (δ = 1) achieved the highest recommendation accuracy. This was similar for the Review
Bot’s algorithm. Neither of time prioritization factors cansignificantly increase the accuracy. This is surprising asprevious work did suggest a time prioritization (δ = 0.8)
yielded the best results. This finding indicates that in thehistories of these projects, an approver who would examine
the next review request was not necessary to be the recent
one. The reasons behind this phenomena could be interesting
future avenues of research. Possible future avenues could be
studying a review selection of reviewers along with the code
evolution as well as tangled code changes investigation to
improve our results.
According to the preliminary results, we were able toanswer our research questions as follows: For RQ1, we statistically showed that our FPS algorithm outperformed the Bot
Review algorithm. For RQ2, we found that, in distributed
projects environment, the time prioritization factor did not
improve results.

EXPERIMENTThe effectiveness of FPS algorithm and Review Bot algorithm were measured using the same method as as the Review
Bot research [3], i.e. Top-k accuracy. For N review requests,
the Top-k accuracy can be calculated using Equation 4. A
recommendation is correct when at least one of the top k
recommended reviewers actually examined a review request.
#correct top-k recommendations
×100%
N
(4)
The review histories of the case study projects were obtained from Hamasaki et. al [5]. For the experiment, weselected the reviews that a) are already closed ; b) have atleast one approver; and c) have at least one other file besides
the commit message file. Table 1 summarizes the data sets
used in our study.
Top-k Accuracy =Table 1: Summary of Data Sets

Study Period#Reviews#Approvers#Files

AOSP
10/08 - 01/12
5,126
51
26,840OpenStack
07/11 - 05/12
6,586
64
12,275Qt
05/11 - 05/12
23,810128
78,400We evaluated the recommendation algorithms by performing recommendations for every review in chronological order.
In this study, both algorithms recommended candidates for areview request by selecting only approvers who had examineda review request in the past. This is because this kind ofreviewers is the most important reviewers in Gerrit system.
We measured the Top-1, Top-3, and Top-5 accuracy of thealgorithms with different δ values of time prioritization factor.
Table 2 lists an accuracy of both algorithms using the timeprioritization factors, δ = {1, 0.8, 0.6, 0.4, 0.2}.

4.1

4.2

Discussion

Limitation of Review Bot’s algorithm: As we conjectured,
the Review Bot’s algorithm had a poor performance due tothe lack of change lines histories. In examining the data
sets, we found that the line change histories of these projects
were relatively limited. For example, 80.62% of the files in
AOSP were modified only once. This was similar for both
OpenStack and Qt.
Limitation of FPS algorithm: For the poor performance of
our FPS algorithm in Qt, the cause can be from the file pathcomparison of function commonPath(fn , fp , i, j) in Equation
3. We addressed this point because this calculation was
based on our assumption which possibly did not correspondto directory structure of Qt. We plan to explore this toimprove our algorithm in our future work. Furthermore,other unknown factors also can be involved for larger scale
project. Closer case by case analysis as well as peer review
interviews will be needed to better understand and improve
our method. We also need to investigation other large scale
projects for confirmation.
Repeating Recommended Reviewers: From the observation,
we found that some reviewers had examined a lot of review
requests. Since both algorithms recommend reviewer using
historical data, it is possible that those reviewers would
be frequently recommended. Consequently, they would beburdened with a huge number of assigned review requests.

Experimental Results

For RQ1: Does FPS outperform the Review Bot’s
algorithm?, the results indicate that FPS algorithm works
more effectively than the Review Bot’s algorithm. For AOSP,FPS algorithm had the highest recommendation accuracy of
77.12 % while the Review Bot’s algorithm had the highest
recommendation accuracy of 29.30 %. For OpenStack, FPS
algorithm had the highest recommendation accuracy of 77.97% while the Review Bot’s algorithm had the highest recommendation accuracy of 38.90 %. However, for the relatively
larger project of Qt, neither of the two algorithms accurately
recommended reviewers. They only had the recommendation
accuracies of about 27-36%.
We used statistical test to determine if the recommendation
accuracy of FPS algorithm is higher than the recommendation accuracy of the Review Bot’s algorithm for all recom-

121Table 2: Top-k accuracy of FPS algorithm and Review Bot’s algorithm with different time prioritization factors (δ)δ=1
21.38 %
29.15 %
29.30 %
22.94 %
35.76 %
38.78 %
18.64 %
26.18 %
27.18 %

Review Bot’s Algorithm
δ = 0.8
δ = 0.6
δ = 0.4
21.63 % 21.71 % 21.77 %29.17 % 29.17 % 29.20 %29.30 % 29.30 % 29.30 %23.23 % 23.19 % 23.20 %35.76 % 35.68 % 35.55 %38.90 % 38.82 % 38.89 %18.70 % 18.72 % 18.71 %26.19 % 26.16 % 26.16 %27.18 % 27.17 % 27.19 %Accuracy (%)

60

1000

2000

3000

4000

5000

20

20
0

FPS Algorithm
Review Bot's Algorithm

00

FPS Algorithm
Review Bot's Algorithm

δ = 0.2
44.50 %
29.20 %
29.30 %
23.19 %
35.53 %
38.89 %
18.75 %
26.16 %
27.19 %

8080

δ = 0.2
21.30 %
63.38 %
70.80 %
34.04 %
61.66 %
71.71 %
8.88 %
18.46 %
24.36 %

40

Accuracy (%)

60
40
20

Accuracy (%)

80

Qt

FPS Algorithm
δ = 0.8
δ = 0.6
δ = 0.4
40.50 % 39.21 % 38.29 %67.66 % 66.56 % 64.83 %75.30 % 74.17 % 72.36 %36.47 % 35.18 % 34.73 %63.09 % 62.45 % 62.10 %73.28 % 72.85 % 72.62 %11.64 % 10.21 %9.45 %
21.39 % 20.15 % 19.27 %27.25 % 26.07 % 25.34 %60

OpenStack

Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5

δ=1
41.81 %
69.84 %
77.12 %
38.32 %
67.57 %
77.97 %
13.02 %
28.82 %
36.86 %

40

AOSP

Top-k

0

1000

2000

3000

4000

5000

Number of Reviews (N)Number of Reviews (N)(a) AOSP(b) OpenStack

6000

FPS Algorithm
Review Bot's Algorithm

0

Project

0

5000

10000

15000

20000

Number of Reviews (N)(c) QtFigure 2: Top-5 accuracy of FPS algorithm and Review Bot’s algorithm with time prioritization factor (δ = 1)

Thus, considering workload balancing would reduce tasks ofthese potential reviewers as well as the number of awaiting
reviews.

5.

[2] A. Bacchelli and C. Bird. Expectations, outcomes, andchallenges of modern code review. In Proc. ICSE ’13,
pages 712–721, 2013.
[3] V. Balachandran. Reducing Human Effort andImproving Quality in Peer Code Reviews using
Automatic Static Analysis and Reviewer
Recommendation. In Proc. ICSE ’13, pages 931–940,
2013.
[4] E. T. Barr, C. Bird, P. C. Rigby, A. Hindle, D. M.
German, and P. Devanbu. Cohesive and Isolated
Development with Branches. In Proc. FASE ’12, pages
316–331, 2012.
[5] K. Hamasaki, R. G. Kula, N. Yoshida, C. C. A. Erika,
K. Fujiwara, and H. Iida. Who does what during a Code
Review ? An extraction of an OSS Peer Review
Repository. In Proc. MSR’ 13, pages 49–52, 2013.
[6] E. Kocaguneli, T. Zimmermann, C. Bird, N. Nagappan,
and T. Menzies. Distributed development consideredharmful? In Proc. ICSE ’13, pages 882–890, 2013.
[7] A. Mockus and J. Herbsleb. Expertise Browser: aquantitative approach to identifying expertise. In Proc.
ICSE ’02, pages 503–512, 2002.
[8] N. Ramasubbu and R. K. Balan. Globally Distributed
Software Development Project Performance : An
Empirical Analysis. In Proc. ESEC/FSE ’07, pages
125–134, 2007.
[9] P. C. Rigby and M.-A. Storey. Understanding broadcast
based peer review on open source software projects. In
Proc. ICSE ’11, pages 541–550, 2011.

CONCLUSION & FUTURE WORKSIn this study, we have proposed a recommendation algorithm using file path similarity for the modern peer code
review process. The results indicate that our proposed FPS
algorithm effectively recommend reviewers in two of thethree OSS projects. This algorithm also significantly outperform the existing algorithm (i.e. Review Bot’s algorithm) inthese projects. Additionally, we found that the used of timeprioritization was surprisingly not appropriate for the recommendation algorithms in distributed projects environment.
Our future work will concentrate on explore more insight
into projects, especially large scale projects to improve thealgorithm. Other evaluations will also be considered tomeasure the performance of the recommendation algorithms
in other aspect besides the accuracy. At the same time, wewill consider ways to balance the workload of reviewers to
help reviewers and reduce the number of awaiting reviews.

ACKNOWLEDGMENTS
We are thankful to Dr. Mike Barker from NAIST for his
valuable suggestions and discussions.

6.

REFERENCES[1] A. Aurum, H. Petersson, and C. Wohlin. State-of-the-art:software inspections after 25 years. Software Testing,
Verification and Reliability, 12(3):133–154, Sept. 2002.

122