2016 IEEE International Conference on Software Maintenance and Evolution

Search-Based Peer Reviewers Recommendation inModern Code Review
Ali Ouni, Raula Gaikovina Kula, Katsuro Inoue
Department of Computer Science, Osaka University, Japan
Email: {ali,raula-k,inoue}@ist.osaka-u.ac.jp
Abstract—Code review is of primary importance in modern
software development. It is widely recognized that peer review isan efﬁcient and effective practice for improving software quality
and reducing defect proneness. For successful review process,peer reviewers should have a deep experience and knowledge withthe code being reviewed, and familiar to work and collaborate
together. However, one of the main challenging tasks in modern
code review is to ﬁnd the most appropriate reviewers for
submitted code changes. So far, reviewers assignment is stilla manual, costly and time-consuming task. In this paper, we
introduce a search-based approach, namely RevRec, to provide
decision-making support for code change submitters and/or
reviewers assigners to identify most appropriate peer reviewers
for their code changes. RevRec aims at ﬁnding reviewers tobe assigned for a code change based on their expertise andcollaboration in past reviews using genetic algorithm (GA). We
evaluated our approach on a benchmark of three open-source
software systems, Android, OpenStack, and Qt. Results indicatethat RevRec accurately recommends code reviewers with up to
59% of precision and 74% of recall. Our experiments provide
evidence that leveraging reviewers expertise from their priorreviews and the socio-technical aspects of the team work andcollaboration is relevant in improving the performance of peer
reviewers recommendation in modern code review.

Although MCR tools provide efﬁcient and automated techniques to support the code review process, still a signiﬁcantamount of human effort involved. In typical software projects,
author of a code change need to invite/assign reviewers mainly
based on their expertise with the changed ﬁles and previous
review collaborations, in order for their change to be merged[5], [9]. One of the main challenges in code review is to ﬁnd
the most appropriate reviewers for pending reviews in a timely
manner. Inappropriate reviewers assignment may lead to aninaccurate, time consuming and non effective review process.
Identifying appropriate peer reviews is a non-trivial
decision-making task for developers. If a patch affects several
modules in the project, then generally the review should be
performed by several peers of each affected module. For
example, in VMware, most of the projects require at least two
independent reviewers for every commit [10]. Moreover, sincea ﬁle could be edited by multiple developers, and reviewed bymultiple reviewers, it is difﬁcult to ﬁnd and assign appropriate
reviewers if the number of ﬁles involved and the changes
within them are large.
In code review, the reviewers expertise with the codefragments being reviewed is crucial to ensure time effective
and high quality review. Recent studies showed that whenreviewers have a priori knowledge of the context and thecode, they complete reviews more quickly and provide morevaluable feedback to the author [8], [11]. On the other hand, as
code review is basically a human process involving personaland social aspects, thus the socio-technical factor plays an
extremely improtant role in ﬁnding peer reviewers [9], [12].
Ultimately, efﬁcient reviewers recommendation tools are
essential to provide quality review and reduce the time taken
for the review process. Unfortunately, so far, the ﬁeld ofpeer reviewers recommendation in MCR is still in its infancy,
and there has been little effort in building automatic recommendation techniques. The ﬁrst attempts in addressing thisproblem, [5], [13], and [14], formulated the peer reviewers
recommendation problem from a single perspective to ﬁnd andrank candidate reviewers based on their experience with thecode being reviewed. Reviewers are ranked and recommended
in an independent manner, neglecting the socio-technical factor
related to the relationships between review contributors which
is a crucial aspect that affects the review quality as pointed
out by many researchers and practitioners [9], [12], [15]–[17].
In this paper, we propose a novel approach, namely RevRec,that formulates the peer code reviewers recommendation prob-

I. I NTRODUCTION
Software code review is a disciplined engineering practicethat has been commonly employed for several years in bothindustrial development and the open-source software (OSS)
community [1]. Peer code review is a manual inspection of acode change, i.e., patch, by third-party developers, before it iscommitted to the project’s code base, in order to detect andcorrect potential defects and ensure quality software [2], [3].
Modern code review (MCR) has become a vital and essential practice in contemporary software development [4], [5].
Taking the Linus’s Law that [6]:“many eyes make all bugs shallow”,
the OSS community have incorporated MCR, where developers utilize dedicated tools that facilitate the code review
process, e.g., Gerrit1 , Codestriker2 , and ReviewBoard3 . It is
widely recognized that peer code review is a valuable andeffective practice that can be applied to software development
at all stages of the life cycle, to improve software quality, decrease defect-proneness, share knowledge and increase
learning through rich communication [3], [7], [8].
1 https://code.google.com/p/gerrit/
2 http://codestriker.sourceforge.net/
3 http://www.reviewboard.org/

978-1-5090-3806-0/16 $31.00 © 2016 IEEE
DOI 10.1109/ICSME.2016.65

367

lem as a combinatorial search-based optimization problem.
The aim is to ﬁnd appropriate reviewers for a given patch
based on their expertise with the patch ﬁles and their priorreview collaboration (co-review) rate with the review request
submitter. To this end, we used genetic algorithm (GA) [18], inorder to explore this large search space of possible reviewers
combinations. Indeed, the search space is not determined onlyby the number of reviewers in a project, but also with thenumber of ﬁles, modules and contributors in the project.
The reviewer expertise refers to the frequency and recencyof reviews performed by a reviewer to the patch’s ﬁles ormodule. The frequency is a count of the number of participatedreview comments, while the recency refers to the time spansince the most recent review on each ﬁle. Indeed, as expertise
may change over time, we consider both reviews frequency
and recency as primary factors to capture the reviewers expertise. Second, the concept of review collaboration refers thenumber of times the candidate reviewers previously reviewed
for a code author. Indeed, as the code review process is mainly
a human process, we thus leverage the socio-technical aspectsof the teamwork and collaboration which has been proved asimportant factor to the review quality and efﬁciency [9], [12],[19]. Indeed, complex combinatorial decision problems, suchas peer reviewers recommendation, are best suited to searchbased software engineering (SBSE) [20].
The main contributions of the paper can be summarized asfollows:











   
Refactoring of smart-types defined in DSL* MuranoType and MuranoObjectParameterType smart types
were merged into a single smart type because their functionality overlap
to a large degree. New smart type is called MuranoObjectParameter* Other smart types were renamed to have the same name pattern:
ThisParameterType -> ThisParameter,
InterfacesParameterType -> InterfacesParameter* For MuranoObjectInterface instead of saying obj.data().propertyName
the syntax now is obj.properties.propertyName
Change-Id: I3c925d1ba1a4ac0864987377a3e90c6f166823a7(a) Summary of the submitted code change.
,QFOXGHGLQź

5HSO\«Owner

3DWFK6HWV  ź

'RZQORDGź 

Reviewers     

!

 "
Project

# $% 

Branch% #

Topic%  Updated(# &#% ')!( *   +,!

*   *-  . , (

*-   (b) Involved people in the review process.
2SHQ$OO

,,  #4 
 %%##   $#$  ,'/$%  /  ,'/$  ,'/0'%  $ #$ #0'--1%  $ #$#/"## 0'%  $ #$ / &#$# / 0'%  $ #$ / &#$#'#/ &0'2%  $ #$, 0'%  $ #$'/#%0'2%  $ #$' 3/, #0'%  $ #$' 3/  0'%  $$% / "/%  0'%  $$#'#%$ 0'-1%  $$#'#%$# /%  0'%  $##$$$#'#%$#/ 0' !

1) We introduce a search-based formulation for the peer
code reviewers recommendation problem. Our approach,
RevRec, aims at ﬁnding, among a large list of reviewers,
an appropriate set of reviewers, to review a code change,
by leveraging reviewers expertise and collaboration.
2) We evaluate our approach on a benchmark of threeopen-source projects, Android, OpenStack, and Qt using
Gerrit code review. We report the results of an empirical
study on an implementation of our approach with a comparison with available state-of-the-art techniques on peer
reviewers recommendation. Results indicate that RevRecsigniﬁcantly outperforms three existing techniques by
accurately recommending code reviewers with up to
59% of precision and 74% of recall.
3) We present the results of a second empirical study
that serves as ‘sanity check’ to compare our GAbased approach with existing search techniques including simulated annealing (SA), particle swarm optimization (PSO), and random search (RS) for solving the
problem.

(c) Changed ﬁles in the submitted code change.

Fig. 1: A Gerrit-based review example4 from the OpenStackproject (code change #283653).
II. BACKGROUND AND MOTIVATION
A. Modern code review
Recently, the lightweight modern code review process has
been adopted by many open source (e.g., Android, Qt, OpenStack) and industry (e.g., Google, Microsoft, Cisco) projects.
In contrast to the formal peer review, the modern peer review
process is a tool-based management system. It entails contributors submitting patches of code that need to be reviewed andapproved before they are committed to the code base.
To give the reader perspective, Figure 1 presents an example
to describe the typical review process using the Gerrit tool
based-code review within the OpenStack. The ﬁgure shows theGerrit interface of the code change #2836534 which is related
to some refactoring operations. This code change refers to a

The rest of the paper is structured as follows. Section II
describes necessary background. Section III introduces our
RevRec approach for reviewers recommendation. Section IV
describes the design of the empirical study we employ to evaluate our approach, while Section V presents and discusses theobtained results. Section VI describes the threats to validity.
Section VII outlines the related work. Finally, in Section VIII,
we conclude and describe our future research directions.

4 https://review.openstack.org/#/c/283653/, For privacy reasons, we havehide some details from the interface, and used alias instead of the original
developer names.

368

C. Genetic algorithms

patch that is being reviewed before it is merged into the codebase which follows four main phases.

Genetic Algorithms (GA) [18] are computer algorithms thatsearch for good solutions to a problem from among a large
number of possible solutions. These computational paradigms
were inspired by the mechanics of natural evolution, including
survival of the ﬁttest, reproduction, and mutation.
GAs begin with a set of random population of candidate
solutions, also called individuals or chromosomes. Each individual of the population is evaluated by a ﬁtness functionthat determines a quantitative measure of its ability to solve
the target problem. The exploration of the search space isachieved by the evolution of candidate solutions using selection and genetic operators such as crossover and mutation.
The selection operator ensures selection of individuals in thecurrent population proportionally to their ﬁtness values, so thatthe ﬁtter an individual is, the higher the probability is thatit be allowed to transmit its features to new individuals by
undergoing crossover and/or mutation operators.
A new population is created from individuals of an old population in hope of getting a better population. Solutions which
are chosen to form new solutions (offspring) are selectedaccording to their ﬁtness. The more suitable the solutions are
the bigger chances they have to reproduce. This process isrepeated until some condition is satisﬁed. The result of GA(the best solution found) is the ﬁttest individual produced
along all generations.

1) Submission phase - Owner (Stephan) creates achange and submits it for a code review. At this stage,
the review is labeled ‘open’. Open reviews can beaccessible by any reviewer to make review comments.
2) Review phase - Reviewers (Alex and David) start
to make comments on the code, which may lead to
additional change revisions. In this case, the change
consisted of 6 revised patch sets.
3) Testing and Veriﬁcation phase - Running concurrently
in the review phase, reviewers Jack and Michel, run
tests and verify that the change will not cause failure
in other parts of the code base. In this example, both
reviewers are ‘bot’ applications that perform the tests
automatically.
4) Decision Phase - Once satisﬁed with the state of thepatch and all the testing and veriﬁcation have been completed, reviewers Paul and David approve the patch.
Both reviewers have higher roles in the project, thus areable to approve this review. As shown, both reviewers
need to score either +1 or +2 to approve the change.
Negative scores such as -1 or -2 wwill lead to thechange being rejected. The change is then successfully
merged into the codebase. It is then labelled as ‘merged’.
Rejected changes are labeled as ‘abandoned’.

III. R EV R EC : S EARCH - BASED P EER R EVIEWERSR ECOMMENDATION

From the example in Figure 1, due to the nature of OSS,
we ﬁnd that attracting reviewers to be assigned to a review
can be tedious, especially in large projects. Each reviewer has
different roles, expertise, experience, and collaboration which
are critical factors for review to be completed in a timely
and efﬁcient manner. For instance, ﬁnding a reviewer who
has had experience involving the same ﬁles in that particular
modules of the code base, and preferably familiar with thecode author.Also, a reviewer who is able to approve a codechange is critical in order for the change to be successfully
merged. To help with this tedious task of selecting appropriate
reviewers, we propose a recommendation technique to explore
and search for possible reviewers to perform a given review.

In this section, we describe our approach, RevRec, forrecommending appropriate reviewers for code changes. Then,
we describe our formulation of the peer reviewers recommendation problem as a search-based optimization problem.
A. Approach overview
Our approach aims at supporting review request submitterto ﬁnd and invite appropriate reviewers in order to reduce
the time taken for the review process and provide quality
review. Figure 2 presents an overview of the RevRec approach.
RevRec takes as input a review request which consists of apatch, i.e., a set of changed ﬁles submitted by a developer,
and the history of completed code reviews recorded fromthe project’s review tool, e.g., Gerrit. As output, RevRec
recommends a set of peer reviewers that are most appropriate
to review the submitted change. Our approach used genetic
algorithm to ﬁnd the best set of reviewers based on two
heuristics (1) the reviewers expertise with the submitted patchﬁles, and (2) the reviewers collaboration with the review
request submitter.
1) Expertise. The reviewers expertise is one of the main
factors for a successful code review. It is widely acceptedthat if the reviewers have a priori knowledge of the code,
they complete reviews more quickly and effectively [8],[11].
2) Collaboration. Each reviewer has his own collaboration
sub-team, i.e., social network, within the project. It is

B. Search-Based Software Engineering
Search-Based Software Engineering (SBSE) consists of theapplication of a computational search to solve optimization
problems in software engineering [21]. The term SBSE wascoined by Harman and Jones in 2001, and the goal of the ﬁeld
is to move software engineering problems from human-based
search to machine-based search, using a variety of techniquesfrom the metaheuristic search and evolutionary computation
paradigms [21], [22]. SBSE provides best practice in formulating a software engineering problem as a search problem,by deﬁning a suitable solution representation, ﬁtness function,and solution change operators. Indeed, there are a multitudeof search algorithms ranging from single to multi-objective
techniques that can be applied to solve that problem [20].

369Each reviewer ri has (i) his own expertise with the ﬁles Fp ,and (ii) his review collaboration history with the developer d,i.e., the review request submitter, and also with the rest ofother co-reviewers Rp \ ri in the project.
Our approach consists of the two following components:

common that, in large projects, same reviewer(s) frequently review the patches from a particular developer.
Moreover, it is common that sub-groups of reviewers
use to collaborate together, i.e., co-review, based on
repositories or modules. In practice, the reviewer(s) andthe code author communicate more frequently with eachother than with other peers, which may form a friendship[9], [19]. On the other hand, code review can be asource of conﬂict, as a code author may consider areviewer’s rejection or critique of his code to be unfair
and become offended. Thus code authors often invite orassign reviewers from their peers that they used to workwith [9].

Reviewer expertise model (RevRecRE ). Each reviewer ri
has a degree of expertise with each of the patch ﬁles Fp thatcan be represented as a vector. Each vector’s dimension isassociated with its relative ﬁle f pj , and denoted by:
Eri = {Exp(ri ,fp1 ) , ..., Exp(ri ,fpt ) }where Exp(ri ,fpj ) denotes the expertise of the reviewer ri withthe ﬁle fpj .
Formally, the expertise of the reviewer ri with the patchﬁles Fp is denoted as E(ri , Fp ) and calculated as follows:

Review Request

Code Review
Submitter

Search-based Reviewers
Recommendation(Genetic Algorithm)

Recommended
ReviewersE(ri , Fp ) =Exp(ri , fpj )(1)∀fpj ∈Fp
Reviews history

where Exp(ri , fpj ) refers to the expertise of the reviewer ri
with the ﬁle fpj ∈ Fp .
To calculate Exp(ri , fpj ), we collect all previous reviews
Rev performed by ri that are (i) closed, i.e., marked as“Merged” or “Abandoned”, and (ii) created before the patch
P . For each ﬁle fpj in the patch P , we check if similar ﬁlesare previously reviewed in each collected review in Rev. Our
similarity measure is based on ﬁle path similarity. For each ﬁle,we ﬁrst split its path into components using ﬁle separator (i.e.,slash character) as a delimiter. Each component represents amodule, directory, package or ﬁle in the system. Thereafter,
we use a camel case splitter to break down each component
to its constituent tokens. Then, the ﬁle path similarity measure(Sim) represents the common tokens between two ﬁles fa andfb based on the Jaccard similarity, as follows:

Fig. 2: RevRec overview.
We believe the peer reviewers recommendation problem
is well suited for applying a search-based technique to aid
developers ﬁnd appropriate reviewers to their patches. Indeed,
the number of reviewers combinations is not determined onlyby the number of reviewers in a project, but also with thenumber of ﬁles, modules and contributors in the project. Even,if a code change consists of one single ﬁle, there may be
several reviewers that may have expertise with it.
Problem complexity. Finding the appropriate reviewers fora patch is not an obvious task for reviewer assigners as thenumber of possible reviewers can be very large causing a
combinatorial problem. The search space tends to be enormousas the number of possible reviewers combinations is equals atk!
× p, where N counts theleast to N = nk × p = n!(n−k)!
number of different possibilities of how a set of k reviewers
can be identiﬁed from a given set of n reviewers working on
the project, and p is a count of the number of ﬁles in the
project. Note that the order of the reviewers is not considered.
Indeed, the search space is prohibitively large for an exhaustive approach. Therefore, we consider a meta-heuristic searchand use a genetic algorithm to ﬁnd a near-optimal solution
representing the most appropriate set of reviewers.

Sim(fa , fb ) =This section gives deﬁnitions and characteristics of the peer
reviewers recommendation problem.
It is assumed that for an existing software system S, there
is a set of developers, D = {d1 , ..., dn }, and set of reviewers
R = {r1 , ..., rm }. The software system consists of a set of q
source ﬁles, F = {f1 , ..., fq }. A patch P , i.e., code change, issubmitted by a developer d ∈ D, and itself consists of a set oft changed ﬁles Fp = {fp1 , ..., fpt }, and denoted by P d, Fp .
The set of possible peer reviewers for the patch P is denotedby:
R = {r , ..., r }1(2)

where tokens(fi ) is the function that returns all tokens in aﬁle path fi as described above.
Our expertise model combines two main aspects. For eachpreviously reviewed ﬁle, the reviewer expertise combines thereview comments (1) frequency, and (2) recency. These two
aspects are calculated as follows.
1) Comments frequency (CF ). The CF of the reviewer ri
for a ﬁle f is a simple count of the number of commentsof all its similar ﬁles F having a Sim score greater thana given threshold k, to be set up by the review request
submitter. Formally, CF is given as follows.

B. Problem formulationp|tokens(fa ) ∩ tokens(fb )|∈ [0, 1]
|tokens(fa ) ∪ tokens(fb )|CF (ri , f ) =comments(ri , fj )(3)

∀fj ∈F

where comments(ri , fj ) is a count of the number ofreview comments participated by the reviewer ri for theﬁle fj .
2) Comments recency (cr). The measure cr is a weightthat reﬂects the recency, i.e., freshness, of the reviewkwhere Rp ⊆ R, and 1 ≤ k ≤ |R|.

370

comments on the ﬁle f by the reviewer ri . The weight
cr is given as follow.

St 

Tf
cr(ri ,f ) = 1 −∈ [0, 1]T

(4)

Exp(ri , f ) = cr(ri ,f ) × CF (ri , f )In the following, we describe our SBSE model for RevRec.
To adopt a seach-based technique, a set of essential elements
should be deﬁned including the solution representation, ﬁtness
function, computational search algorithm, selection, genetic
operators, and termination criteria [21].
Solution representation. A solution Rp , i.e., set of peer
reviewers, for a patch P , can be represented in an array ofsize equals to the total number of current reviewers working
on the project, numReviewers. Each element from this arraydenotes whether a reviewer is recommended for the codechange (‘1’) or not (‘0’). For example, Figure 4 encodes a
candidate solution for the review example of the OpenStackproject #283653 sketched in Figure 1. For sake of simplicity,
we assume that OpenStack has only ten reviewers. Thissolution example indicates that the 2nd , 4th , 5th , 7th , and the10th reviewers (Paul, Michel, David, Alex, and Jack,
respectively) are recommended for this code change submitted
by Stephan.

E(ri , Fp )(6)

Reviewer collaboration model (RevRecRC ). Each reviewer
ri may have a review collaboration RC with both (i) the
review request submitter, and/or (ii) the rest of candidate reviewers Rp \ri based on past reviews. The review collaboration
forms a social network within the project. This network isrepresented as a graph G = (V, E) where the vertices V
represent the people (code authors and reviewers) and theedges E represent the collaboration measure between themas a count of the number of exchanged review comments
during their past co-reviews. For a patch P , the code author
and the candidate reviewers can be represented as a weighted,
undirected sub-graph Gp = (Vp , Ep ).
Formally, let Rp a candidate set of reviewers recommended
for a patch P submitted by a code author d, then, we calculate
the reviewers collaboration, RC, from the sub-graph Gp =(Vp , Ep ) based on two aspects: the sub-graph connectivity,
and the sum of weights on the edges (comments count), as
follows:

|Ep |
e pi
×
|Vp | × (|Vp | − 1)/2 ∀e ∈E
pi

59

C. Genetic Algorithm Adaptation

where E(ri , Fp ) is given by Equation 1.

RC(d, Rp ) =  

18

Fig. 3: Example of review collaboration sub-graph.

(5)

|Rp |

126



Finally, the expertise of a (set of) reviewer(s) Rp to be
invited/assigned for the patch P having a set of ﬁles Fp isdenoted as RE and calculated as follows:∀ri ∈Rp

59

45

Then, Exp(ri , f ) is deﬁned as follows.

RE(Rp , Fp ) =


46

26

where Tf is a count of the number of calendar days since
the most recent comment in all similar ﬁles F , and T
is a count of number of calendar days since the wholeproject is created. That is, if the most similar (or same)ﬁle is commented at the day the patch P is submitted,
then cr = 1.



41

32

 



0

1

    

0

1



  

1

0

 

1



    

0

0

1

Fig. 4: Solution representation.
Constraints. Each solution has to comply with a set of
constraints. The ﬁrst constraint considered in RevRec, is thatthe recommended list should contain at least one approver,i.e., core reviewer. This constraint, could be either activatedor inactivated by the review request submitter. Moreover, oursolutions should satisfy two other constraints, the minimum
and the maximum number of recommended reviewers, minRev and maxRev, respectively. The two latter constraints arechecked by simply calculating the sum of the ‘1’ elementsin the solution’s array. RevRec uses these three constraints
as conﬁguration parameters to be set up by the developer
according to the complexity and the size of his code change.
Computational search algorithm. As a search method,
we employed a widely used metaheuristic algorithm, namely
genetic algorithm (GA) [18].
GA starts by creating an initial population of solutions. A
solution is a set of peer reviewers. In the ﬁrst generation, thesesolutions are generated randomly. Then in an iterative process,every iteration produces a new generation of solutions derived
from the previous ones. For each iteration, GA pushes the“ﬁttest” candidates to the next generation (elitism), and thenGA generates the rest of the solutions composing the next
generation by combining/modifying existing solutions using

(7)

p

|E |

where the coefﬁcient |Vp |×(|Vpp |−1)/2 ∈ [0, 1] reﬂects the
sub-graph connectivity (ideally equals to 1 if the graph iscomplete), and epi is the weight on the edge ei , i.e., a countof the number of review comments between each pair ofcontributors (reviewers and code authors).
To illustrate the collaboration model, let us consider theexample sketched in Figure 3, where a patch is submitted bya developer d = Stephan, and reviewed by a set of reviewers Rp = {Paul, Michel, David, Alex}. Then, thereviewer collaboration score RC is calculated from the col9× 285 = 256.5.
laboration sub-graph as follows, RC = 5∗4/2

371

ranking technique is as follows. Top k solutions of the last
population obtained in the last iteration of GA are copied ina single pool. Then, the rank of each reviewer corresponds to
his frequency count in the pool. That is, reviewers that arerecommended in many solutions are ranked ﬁrst.

crossover and/or mutation operators. The ﬁtness of a solution
is computed using a function implementing the heuristics
stated earlier (reviewers expertise and collaboration).
Fitness function. The ﬁtness function quantiﬁes the qualityof each individual in the current population. To evaluate theﬁtness of each candidate solution we employed a ﬁtness function that combines the reviewers expertise (RevRecRE ) andreviewers collaboration (RevRecRC ). To calculate the ﬁtnessof a candidate solution Rp for a patch P d, Fp  submitted bya code author d and consists of a set of Fp ﬁles, we use thefollowing function:
F itness(Rp , P ) = α × RE(Rp , Fp ) + β × RC(d, Rp )IV. E MPIRICAL EVALUATIONIn this section, we present the results of our evaluation forthe proposed approach. The aim of this study is to investigate
the effectiveness of the RevRec approach in providing peer reviewers recommendation solutions. We conduct two empiricalstudies to compare our approach with (1) existing state-ofthe-art approaches in peer reviewers recommendation, and (2)existing metaheuristic search techniques.

(8)

where the functions RE(ri , Fp ) and RC(d, Rp ) are given by
Equations 6 and 7, respectively, and α + β = 1.
Note that both RE and RC values are normalized in therange [0,1] using min-max normalization based on all solutionsin the current population.
Selection. The selection process is based on ﬁtness. Solutions that are evaluated with higher values (ﬁtter) will
most likely be selected to reproduce, whereas, those withlow values will be discarded. In RevRec, we used roulettewheel selection [23], combined with elitism, where a numberof ﬁttest solutions are copied without changes to the new
population, so the best solutions found will not be lost.
Roulette-wheel is a simple method of implementing ﬁtnessproportionate selection. It is conceptually equal to giving eachindividual a slice of a circular roulette wheel equal in area tothe individual’s ﬁtness. The wheel is spun n times, and oneach spin, the individual under wheel’s marker is selected to
be in the pool of parents for the next generation [23].
Genetic operators. We deﬁned our genetic operators
crossover and mutation for RevRec as follows.
Crossover. We employ a single random cut-point crossover.
This operator is performed by generating a random number k
between 0 and numReviewers −1. Then, it exchanges the subsequences before and after k between two parent individuals
to create two offsprings. We perform crossover with a certain
probability.
Mutation. Our mutation changes the new offspring by
ﬂipping bits from 1 to 0 or from 0 to 1. Mutation can occur ateach bit position in the string with some probability. It aims at
preventing falling all solutions in the population into a local
optimum of solved problem.
Termination conditions. The population converges wheneither 90% of the solutions in the population have the sameﬁtness value or the number of evaluation functions is greaterthan a ﬁxed number, maxEval. Then the best (ﬁttest) solution
is returned by RevRec, based on its ﬁtness value.

A. Research Questions
We design our experiments to address three research questions.
RQ1 . How accurate is RevRec in recommending peer
reviewers for code changes?
RQ2 . What are the effects of each of the reviewers expertise and collaboration on the accuracy of RevRec?
RQ3 . How does GA compared to random search (RS) andother popular search algorithms, SA and PSO?
B. Context Selection
Studied systems. We evaluate our approach on a benchmark[24] of three well-known open-source systems, Android5 ,OpenStack6 , and Qt7 . Android is a free software stack fora wide range of mobile devices led by Google. OpenStack is
software platform for cloud computing, controling large poolsof compute, storage, and networking resources throughout adatacenter. Qt is a comprehensive cross-platform frameworkand development tools that is widely used for developing
application software. All collected reviews are closed, i.e.,marked as “Merged” or “Abandoned”, and contain at least one
ﬁle. Table I summarizes the statistics of the three systems.
TABLE I: Studied systems statistics.
SystemAndroid
OpenStack
Qt

Period studied#Reviews#Reviewers#Files

2008-10 ∼ 2012-01
2011-07 ∼ 2012-05
2011-05 ∼ 2012-05

5,126
6,586
23,81094
82
202

26,84016,95378,401

We selected these three systems for our evaluation because
they range from medium to large-sized open-source projects,
which have been actively developed for more than ﬁve years,
and have been extensively studied in the software review
literature [7], [13], [14], [24].
State-of-the-art techniques. To evaluate the efﬁciency of ourapproach we compare it with available state-of-the-art techniques, cHRev, REVFINDER, and ReviewBot, in order to investigate what improvements such an approach will bring. The

D. Reviewers ranking
Although our approach recommends a set of candidate
reviewers to co-review a submitted patch, each has his expertise with some particular (or all) ﬁles and used to co-review
with some (or all) other peers, RevRec provides a second
component to rank all candidate reviewers in the project. Our

5 https://source.android.com/
6 http://www.openstack.org/
7 http://qt-project.org/

372

the multiplicative inverse of the rank of the ﬁrst true positive
reviewer. The mean reciprocal rank is the average of thereciprocal ranks of results for a sample of recommendation list.
Given a reviewers recommendation lists R, the score MRR is
calculated as follows:

cHRev approach [14] is based on a reviewer expertise model,
generated from completed reviews, that combines a quantiﬁcation of review comments and their recency. REVFINDER [13]is based on the past reviews of ﬁles with similar names andpaths to build an expertise model. ReviewBot [5] is based on
the modiﬁcation history of source code using static analysis
tools to ﬁnd experienced reviewers.
Moreover, we compare our GA-based approach with random search (RS) [25] as a ‘sanity check’ and also with existing
search techniques including simulated annealing (SA) [26]and particle swarm optimization (PSO) [27] for solving the
problem, as they are the most popular techniques in solving
software engineering problems [20], [22].

MMR =We evaluate our approach on the dataset obtained fromour three studied systems as described in Table I. Due tothe stochastic nature of the used search algorithms, they may
give slightly different results in each run. To cutter with thisstochastic nature, we repeat the simulation run 31 times ineach case, and report the median value as commonly used inmetaheuristic algorithms [28].
Our evaluation process is as follows. For each system, wetake the most recent 1,000 reviews, in their chronologicalorder, as test reviews. For each test review Tr , we consider
its actual reviewers, that are known, as the ground truth.
Thereafter, we execute RevRec on Tr by deriving the expertiseand collaboration models from all reviews performed before Tr
was created. Similarly, cHRev, REVFINDER, and ReviewBot
are evaluated with the same process.
To answer RQ1 , we used three common evaluation metrics
for recommendation systems, the precision, recall, and mean
reciprocal rank (MRR) [5], [13], [14], [29]. For each test
review Tr , of each studied system, we calculate the precisionand recall as follows.

recall@k =TPTP + FPTP
TP + FN

rank(r)(11)|R|

where rank(r) is the rank score of the ﬁrst reviewer in therecommendation list r. The higher is the MRR score, the betteris the recommendation approach.
To answer RQ2 , we investigate the effectiveness ofeach of the components of RevRec, the reviewers expertise(RevRecRE ), and the reviewers collaboration (RevRecRC ). To
assess the effectiveness of RevRecRE , we set up α = 1 and
β = 0 in the ﬁtness function (Equation 8), and inversely for
the RevRecRC component (i.e., α = 0 and β = 1). To thisend, we use our evaluation metrics precision@k, recall@k,
and MRR.
To answer RQ3 , we compare the effectiveness of GAagainst SA, PSO and RS in terms of precision and recall.
This evaluation is a sanity check to make sure that we used
an appropriate search algorithm. Moreover, if the proposed
formulation does not allow an intelligent search algorithm
to outperform random search, then the proposed formulation
is not appropriate. For all algorithms, we use the same formulation given in Section III-B. The parameters setting andstatistical tests used are described in sections IV-D and IV-E,
respectively.

C. Method Analysis

precision@k =∀r∈R

D. Parameter Setting
The initial population/solution of GA, SA, PSO, and RS
are completely random. The stopping criterion is when the
maximum number of function evaluations maxEval, set to8,000, is reached. After several trial runs of the simulation,
the parameter values were ﬁxed. Indeed, there are no generalrules to determine these parameters, and thus we set the
combination of parameter values by trial-and-error, a methodthat is commonly used by the SBSE community [28], [30].
For replicability, we report in Table II the parameter settingsof GA, SA and PSO. For RS, we switched off the individual
selection based on ﬁtness value in our GA adaptation.
For our experiments, the parameters minRev is set to 1,
and maxRev is set to 10. We set the parameters α and β
to 0.5 as default parameters. Note that RevRec allows the

(9)
(10)

where TP (True Positive) corresponds to the number of topk reviewers recommended by the approach and also actual
reviewers; FP (False Positive) corresponds to the number of
top-k reviewers recommended by the approach, but not actual
reviewers; FN (False Negative) corresponds to the numberof actual reviewers, that are not among the top-k reviewers
recommended by the approach.
We calculate these metrics with different k values, 1, 3,
5, 10 and auto. The auto refers to the size of the bestsolution (set of reviewers) recommended by RevRec, auto ∈[minRev, maxRev] of size n which is automatic by RevRec.
To ensure fair comparison, we select the top n reviewers foreach approach.
In addition to the precision and recall, we calculate theMean Reciprocal Rank (MRR) as the average of reciprocal
ranks of true positive reviewers in a recommendation list.
The reciprocal rank of a reviewers recommendation list is

TABLE II: Parameter settings used for GA, SA and PSO.

373

Algorithm

Parameter

Value

GA

population size
crossover probability
mutation probability
number of crossing points
selection

200
0.85
0.1
1
Roulette selectionSA

initial temperatureﬁnal temperature
cooling coefﬁcientnumber of iterations

100
0.157
0.98
25

PSO

number of particles in a swarm
acceleration coefﬁcient c1
acceleration coefﬁcient c2

200
2
2

TABLE III: The precision and recall results achieved by RevRec, cHRev, REVFINDER, and ReviewBot, for each system.
precision@k

Systemrecall@kk

RevReccHRev

REVFINDER

ReviewBot

RevReccHRev

REVFINDER

ReviewBot

Android

auto
1
3
5
10

0.52
0.58
0.47
0.39
0.34

0.38
0.50
0.35
0.30
0.26

0.24
0.34
0.25
0.22
0.18

0.17
0.21
0.17
0.12
0.09

0.54
0.38
0.51
0.61
0.71

0.42
0.27
0.50
0.61
0.65

0.38
0.18
0.39
0.48
0.54

0.18
0.11
0.19
0.29
0.38

OpenStack

auto
1
3
5
10

0.54
0.59
0.51
0.43
0.36

0.41
0.48
0.42
0.38
0.31

0.26
0.32
0.27
0.25
0.21

0.19
0.24
0.20
0.16
0.11

0.56
0.41
0.54
0.61
0.74

0.40
0.31
0.39
0.52
0.66

0.31
0.15
0.29
0.37
0.46

0.22
0.12
0.20
0.32
0.39

Qt

auto
1
3
5
10

0.46
0.49
0.45
0.41
0.34

0.39
0.45
0.40
0.37
0.31

0.26
0.30
0.27
0.21
0.16

0.18
0.22
0.19
0.13
0.09

0.51
0.41
0.50
0.59
0.65

0.45
0.33
0.47
0.52
0.60

0.31
0.14
0.27
0.35
0.43

0.22
0.9
0.16
0.24
0.30

TABLE IV: The mean reciprocal rank (MRR) of the approaches RevRec, cHRev, REVFINDER, and ReviewBot, foreach studied system.

review submitter to easily set these parameters according to
his preferences.
E. Statistical Test Methods
We used the Wilcoxon signed rank test in a pairwise fashion[31], [32] in order to detect signiﬁcant performance differences
between the search algorithms GA, SA, PSO, and RS undercomparison. The Wilcoxon test does not require that the data
sets follow a normal distribution since it operates on values’ranks instead of operating on the values themselves. We set
the conﬁdence limit, α, at 0.01.
Moreover, we investigate the effect size using Cliff’s delta(d) [33]. The effect size is considered: (1) negligible (N) if| d |< 0.147, (2) small (S) if 0.147 ≤| d |< 0.33, (3) medium(M) if 0.33 ≤| d |< 0.474, or (4) large (L) if | d |≥ 0.474.
The goal of these tests is to assess whether such comparedresults are statistically better than each other, to cope with thestochastic nature of such analyzed data [28].

System

RevReccHRev

REVFINDER

ReviewBot

Android
OpenStack
Qt

0.69
0.63
0.54

0.65
0.58
0.43

0.60
0.55
0.31

0.25
0.30
0.22

(auto), i.e., the default set of recommended reviewers without
ranking. Indeed, this an interesting feature of the RevRec approach to recommend a set (number) of reviewers, as reviewer
assigners are unlikely to look through a long recommendation
list, and manually ﬁnd reviewers combination.
Table IV reports the Mean Reciprocal Rank (MRR) reﬂecting the overall ranking performance. As shown in the table,
RevRec achieves a MRR score of at least 0.54 for all thebenchmark systems. This indicated that our approach provides
a higher chance of recommending appropriate reviewers in theﬁrst ranks, than cHRev, REVFINDER, and ReviewBot.
This superiority of RevRec has an actionable ﬁnding supporting the claim that combining reviewers expertise andreviewers social relationships is beneﬁcial for accurately recommending peer reviewers.

V. S TUDY R ESULTS
This section reports the analysis of the results for the threeresearch questions formulated in Section IV-A.
A. RQ1 : Accuracy of the proposed approach
Table III reports the precision@k and recall@k of ourapproach RevRec compared to cHRev, REVFINDER, andReviewBot, for each studied system. The experiment showsthat RevRec achieves a precision@1 at 0.58, 0.59 and 0.49
for Android, OpenStack and Qt, respectively. Lower precision
scores are obtained with top-10 but still higher than 0.34
for all systems. For the three systems, we observe that ourapproach RecRec achieved signiﬁcantly superior precision
values compared to cHRev, REVFINDER and ReviewBot.
In terms of recall@k, RevRec achieved the top-10 recall
scores at 0.71, 0.74 and 0.60, for Android, OpenStack andQt, respectively. The lowest recall scores were at k = 1, but
still higher than 0.51 in worst case for Qt project. Clearly, therecall results achieved by RevRec are considerably superior
than cHRev, REVFINDER and ReviewBot. We also observethat the accuracy performance of RevRec is consistent acrossthe three studied systems.
Furthermore, results show that RevRec achieved signiﬁcantly superior precision and recall results when k is automatic

B. RQ2 : Effectiveness of each of the reviewers expertise andcollaboration
To investigate the effectiveness and the contribution ofeach of the reviewers expertise and reviewers collaboration
heuristics, we assess the accuracy of each of them individuallywith k = auto. Figure 5 reports the results of each individualcomponent RevRecRE and RevRecRC . We observe that the
RevRecRE component performs slightly better than RevRecRC
in terms of precision, recall and MRR. Clearly, both heuristics
are lower than the RevRec formulation. Interestingly, we
found that based only on social relationships (collaboration)
without reviewers expertise, we are able to recommend code
reviewers with an acceptable precision and recall score, morethan 0.31 and 0.39, respectively. Thus, based on the resultsof Figure 5, we can conclude that combining both expertiseand collaboration is an appropriate formulation of the code
reviewers recommendation problem, as it improves precision,
recall and MRR.

374

0.6

0.37

0.4

0.39
0.31

0.3
0.2

0.42

0.4

0.38

0.69

0.51

0.63

0.45

0.6

0.38 0.39
0.32

0.3
0.2

0.1

RevRec
RevRec^RE
RevRec^RC

0.8

0.56

0.54

0.5

0.46
0.41
0.37

recall6 precision6 

0.5
0.4

0.6

0.54

0.52

RevRec
RevRec^RE
RevRec^RC

0.7

MRR

RevRec
RevRec^RE
RevRec^RC

0.7

0.49

0.55

0.53

0.58

0.54
0.42

0.4

0.35

0.2

0.1

0.0

0.0

0.0
ndroidentack(a) Precision.

5t

ndroidpentackndroid5t(b) Recall.

pentack5t(c) MRR.

Fig. 5: The precision, recall and MRR results achieved by each of the RevRec, RevRecRE and RevRecRC .
Android

An important observation to highlight from the results of
Figure 5 and Tables III and IV is the importance of social
relationships in peer code review. Results show that RevRecRE
achieves comparable results to cHRev for all studied systems,
but considerably better performance is achieved by RevRec
when the reviewers collaboration is incorporated in the search
process. This ﬁnding provides evidence that code reviewers
assignment is inﬂuenced by social relationships between code
authors and reviewers. That is, code authors are likely to be
more familiar and invite their code review collaborators morethan their non-code review collaborators.
In more details, we studied the intersection between therecommendation lists identiﬁed by each of RevRecRE and
RevRecRC , individually in each of the three systems. We
observed that more than 68% of common reviewers arerecommended by both expertise and collaboration models.
This indicates that the peer code review process is forminga review expertise between code review collaborators.

OpenStack

Qt

0.6

0.6

0.6

0.5

0.5

0.5

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2
GA

PSO

RS

SA

0.2
GA

PSO

RS

SA

GA

PSO

RS

SA

(a) Precision.
Android

C. RQ3 : Performance of GA

OpenStack
0.6

0.6

0.5

0.5

0.5

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2
GA

Figure 6 reports the boxpots, while Table V reports the
Wilcoxon signiﬁcance tests and the Cliff ’s delta (d) effectsize test to compare each of GA, PSO, SA and RS in terms ofprecision and recall. Each algorithm is executed 31 times andthe default output is reported, without reviewers ranking (forthis reason we do not consider the MRR metric). Clearly, GA,
PSO and SA signiﬁcantly outperform random search (RS) in
all the three systems with a large effect size. This provides
evidence that our problem formulation is appropriate, as anintelligent search is required to explore the search space.
From the results, we observe that GA achieves signiﬁcantly
better results than SA in terms of precision and recall withlarge effect size in all the cases (two metrics × three systems).
Moreover, GA achieved signiﬁcantly better results than PSO
with large effect size, except in 2 out of 6 cases the effect size
was medium (the precision with Android, and the recall with
OpenStack). Furthermore, SA turns out to be remarkably lower
than both GA and PSO in all metrics and all systems. Morespeciﬁcally, we observed that for smaller systems (Android
and OpenStack), the performance of SA in terms of precisionand recall is better than larger systems (Qt), as can be seen in
the boxplots of Figure 6. This results indicate that populationbased algorithms (GA and PSO) are more appropriate than

Qt

0.6

PSO

RS

SA

0.2
GA

PSO

RS

SA

GA

PSO

RS

SA

(b) Recall.

Fig. 6: Boxplots of precision an recall results achieved by GA,
PSO, SA and RS through 31 independent runs.
local search algorithms (SA) for the reviewers recommendation problem.
Based on these results we can conjecture that GA achieves
signiﬁcantly better results than SA and PSO, in all the studied
systems for the code reviewers recommendation problem.
VI. T HREATS TO VALIDITY
This section discusses threats to the validity of our study.
Construct threat to validity can be related to the set ofcorrect reviewers, i.e., the ground truth, to calculate precision and recall. During the code review process, sometimes
reviewers are assigned to a code change but they do notcontribute to it. This may due to several reasons including
his current workload, availability or the social relationshipwith the code change submitter. To mitigate this threat, weconsidered the ground truth as the set of reviewers who areassigned and contributed by at least one comment, as ourcollaboration graph is based on the number of comments

375

TABLE V: Statistical tests results of GA vs. PSO, SA and RS.
Precision

RecallSystemGAvsp-valuedp-valuedAndroid

PSO
SA
RS

<0.01
<0.01
<0.01

0.43 (M)0.74 (L)0.96 (L)

<0.01
<0.01
<0.01

0.52 (L)0.92 (L)0.96 (L)OpenStack

PSO
SA
RS

<0.01
<0.01
<0.01

0.88 (L)0.96 (L)0.96 (L)

<0.01
<0.01
<0.01

0.40 (M)0.94 (L)0.96 (L)

Qt

PSO
SA
RS

<0.01
<0.01
<0.01

0.58 (L)0.96 (L)0.96 (L)

<0.01
<0.01
<0.01

0.66 (L)0.96 (L)0.96 (L)N: negligible

S: smallnotable limitation of these works is that the socio-technicalaspect of the code review process is not considered.
Human aspects in peer code review. It is widely agreed
that code review is a complex process involving personal andsocial aspects [17]. Fagan ﬁrst proposed software inspection
as a vigorous and systematic peer review activity to ensure
the quality of software [36]. Due to the volunteer nature of
OSS developers and the peer review structure, studies suchas Rigby et al. [11], [37], Bosu et al. [9], [15] Baysal etal. [12], Bird et al. [38], and Yang et al. [16] found thatdifferent human factors and socio-technical issues inﬂuence
the OSS peer review, motivating the need for a peer review
recommendation system.
Baysal and her collegues conducted several studies to investigate non-technical and human aspects in code review.
They found that organizational, personal and participation
dimensions inﬂuence the review process [12], [39]. Moreover,
Baysal et al. found that the review process can be sensitive
due to its nature of dealing with people’s egos [40].
Bosu et al. [9] studied different aspects of peer impression
formation, and found that there is a high level of trust,reliability, perception of expertise, and friendship between
OSS peers who have participated in code review for a periodof time. Also developers reputation is an important aspect toreceive quicker ﬁrst feedback, and core developers are more
likely to have their code changes accepted into the projectcodebase [41].

M: medium

L: large

participated between two reviewers in their previous reviews.
To accurately measure the level of reviewers collaboration,
we have to consider other types of socio-technical interactions
between them, e.g., exchanged emails, collaboration in related(sub-)projects, etc.
Internal threats to validity can be related to the stochastic
nature of the metaheuristic algorithms [20]. To mitigate thisthreat, we used the Wilcoxon test and Cliff’s delta effectsize over 31 independent runs of each algorithm [28]. The
choice of the best weights, α and β, between our components,
RevRecRE and RevRecRC , can affect the the overall results.
Although we set the average (α = β = 0.5), we plan toempirically investigate several combinations.
External threats to validity can be related to the generalizability of our results. Although we empirically evaluatedour approach on three large size open-source systems fromdifferent application domains, Android, OpenStack, and Qt,
we cannot claim that the same results would be achieved
with other projects or other periods of time. Moreover, ourempirical evaluation is based on open-source projects using
the Gerrit code review tool, theretofore we cannot generalize
our results on other code review tools. As future work, we plan
to extend our evaluation on other open-source and industrial
projects using different code review platforms.

VIII. CONCLUSION AND FUTURE WORKIn this paper, we introduced a new search-based approach,namely RevRec, for peer code reviewers recommendation. Our
approach uses genetic algorithm to ﬁnd the appropriate peer
reviewers to be invited/assigned to review code change basedon their expertise and their collaboration in past reviews. The
proposed approach is evaluated on a benchmark of three open
source systems. Results provide evidence that formulatingcode reviewers recommendation based on a combination ofreviewers expertise and their socio-technical relationships ismore accurate and effective in practice. Moreover, statisticalanalysis of the obtained results indicate that GA is more efﬁcient in solving the peer reviewers recommendation problem.
As future work, we plan to extend our approach to incorporate the reviewers workload in the recommendation task,
and consider the number of recommended reviewers based on
the code change size and complexity. Moreover, we plan toinvestigate and integrate the commit history in our expertisemodel. Another interesting research direction that interest us
is to analyze the rich information embodied in the review
comments in order to better understand the social relationships
between code authors/reviewers to improve the reviewers
recommendation accuracy.

VII. R ELATED WORK
Related work to this study could be divided into two maincategories, (i) peer reviewers recommendation systems, and(ii) studies on social and human aspects in peer review.
Peer code reviewers recommendation. Balachandran ﬁrst
suggested to use review bot tool, as a recommendation system
to reduce human effort and improve review quality in an global
industrial setting, where face-to-face meetings are not possible[5]. Later, Patanamon et al. [13] adapted and improved therecommendation system to assist with the tool-based modern peer review process. The authors used the ﬁle review
history to recommend the best reviewer. Recently, Zanjani
et al. [14] proposed a reviewer recommendation approachbased on a reviewer expertise model, generated from pastreviews, that combines a quantiﬁcation of review comments
and their recency. Anvik et al. [34] used machine learningtechniques to recommend developers to ﬁx new bug reports.
Xia et al. [35] used bug report and developer information torecommend developers to resolve bugs. However, the most

ACKNOWLEDGMENT
This work was supported by Japan Society for the Promotion of Science, Grant-in-Aid for Scientiﬁc Research (S)JP25220003.

376

R EFERENCES[20] M. Harman, S. A. Mansouri, and Y. Zhang, “Search-based software
engineering: Trends, techniques and applications,” ACM Computing
Surveys (CSUR), vol. 45, no. 1, p. 11, 2012.
[21] M. Harman and B. F. Jones, “Search-based software engineering,”
Information and Software Technology, vol. 43, no. 14, pp. 833–839,
2001.
[22] M. Harman, “The current state and future of search based software
engineering,” in Future of Software Engineering (FOSE), 2007, pp. 342–
357.
[23] M. Mitchell, An Introduction to Genetic Algorithms. Cambridge, MA,
USA: MIT Press, 1996.
[24] X. Yang, R. Gaikovina Kula, N. Yoshida, and H. Iida, “Mining themodern code review repositories: A dataset of people, process andproduct,” in 13th Working Conference on Mining Software Repositories(MSR), 2016.
[25] F. J. Solis and R. J.-B. Wets, “Minimization by random search techniques,” Mathematics of operations research, vol. 6, no. 1, pp. 19–30,
1981.
[26] S. Kirkpatrick, “Optimization by simulated annealing: Quantitative studies,” Journal of statistical physics, vol. 34, no. 5-6, pp. 975–986, 1984.
[27] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in International Conference on Neural Networks, 1995, pp. 1942–1948.
[28] A. Arcuri and L. Briand, “A practical guide for using statistical tests
to assess randomized algorithms in software engineering,” in 33rd
International Conference on Software Engineering (ICSE), 2011, pp.
1–10.
[29] I. Avazpour, T. Pitakrat, L. Grunske, and J. Grundy, “Dimensions andMetrics for Evaluating Recommendation Systems,” in Recommendation
Systems in Software Engineering. Springer Berlin Heidelberg, 2014,
ch. 10, pp. 245–273.
[30] A. E. Eiben and S. K. Smit, “Parameter tuning for conﬁguring and analyzing evolutionary algorithms,” Swarm and Evolutionary Computation,
vol. 1, no. 1, pp. 19–31, 2011.
[31] J. Demšar, “Statistical comparisons of classiﬁers over multiple data sets,”
The Journal of Machine Learning Research, vol. 7, pp. 1–30, 2006.
[32] J. Cohen, Statistical power analysis for the behavioral sciences. Academic press, 1988.
[33] N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinalquestions.” Psychological Bulletin, vol. 114, no. 3, p. 494, 1993.
[34] J. Anvik, L. Hiew, and G. C. Murphy, “Who should ﬁx this bug?” in28th International Conference on Software Engineering (ICSE), 2006,
pp. 361–370.
[35] X. Xia, D. Lo, X. Wang, and B. Zhou, “Accurate developer recommendation for bug resolution,” in 20th Working Conference on Reverse
Engineering (WCRE), 2013, pp. 72–81.
[36] M. E. Fagan, “Design and code inspections to reduce errors in program
development,” IBM Systems Journal, vol. 15, no. 3, pp. 182–211, 1976.
[37] P. C. Rigby, D. M. German, and M.-A. Storey, “Open source software
peer review practices: a case study of the apache server,” in 30th
international conference on Software engineering, 2008, pp. 541–550.
[38] C. Bird, D. Pattison, R. D’Souza, V. Filkov, and P. Devanbu, “Latentsocial structure in open source projects,” in 16th ACM SIGSOFTInternational Symposium on Foundations of Software Engineering, ser.
SIGSOFT ’08/FSE-16, 2008, pp. 24–35.
[39] O. Baysal, O. Kononenko, R. Holmes, and M. W. Godfrey, “Theinﬂuence of non-technical factors on code review,” in 20th Working
Conference on Reverse Engineering (WCRE), 2013, pp. 122–131.
[40] O. Baysal and R. Holmes, “A qualitative study of mozillas process
management practices,” David R. Cheriton School of Computer Science,
University of Waterloo, Waterloo, Canada, Tech. Rep. CS-2012-10, 2012.
[41] A. Bosu and J. C. Carver, “Impact of developer reputation on code
review outcomes in oss projects: An empirical investigation,” in 8th
ACM/IEEE International Symposium on Empirical Software Engineeringand Measurement, 2014, pp. 33:1–33:10.

[1] “Ieee standard for software reviews,” IEEE Std 1028-1997, pp. i–37,
March 1998.
[2] A. F. Ackerman, P. J. Fowler, and R. G. Ebenau, “Software inspections
and the industrial production of software,” in Symposium on Software
Validation: Inspection-testing-veriﬁcation-alternatives, 1984, pp. 13–40.
[3] A. Ackerman, L. Buchwald, and F. Lewski, “Software inspections: aneffective veriﬁcation process,” IEEE Software, vol. 6, no. 3, pp. 31–36,
may 1989.
[4] P. C. Rigby and C. Bird, “Convergent contemporary software peer review
practices,” in 9th Joint Meeting on Foundations of Software Engineering(ESEC/FSE), 2013, pp. 202–212.
[5] V. Balachandran, “Reducing human effort and improving quality in peer
code reviews using automatic static analysis and reviewer recommendation,” in 35th International Conference on Software Engineering (ICSE),
2013, pp. 931–940.
[6] E. S. Raymond, The Cathedral and the Bazaar, 1st ed. Sebastopol,
CA, USA: O’Reilly & Associates, Inc., 1999.
[7] S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “The impact ofcode review coverage and code review participation on software quality:
A case study of the qt, vtk, and itk projects,” in 11th Working Conferenceon Mining Software Repositories (MSR), 2014, pp. 192–201.
[8] A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges ofmodern code review,” in 35th International Conference on Software
Engineering (ICSE), Piscataway, NJ, USA, 2013, pp. 712–721.
[9] A. Bosu and J. C. Carver, “Impact of peer code review on peer impression formation: A survey,” in International Symposium on Empirical
Software Engineering and Measurement, 2013, pp. 133–142.
[10] P. C. Rigby and D. M. German, “A preliminary examination of code
review processes in open source projects,” University of Victoria, Tech.
Rep. DCS-305-IR, 2006.
[11] P. C. Rigby and M.-A. Storey, “Understanding broadcast based peer
review on open source software projects,” in 33rd international conference on Software engineering (ICSE), New York, New York, USA, may
2011, p. 541.
[12] O. Kononenko, O. Baysal, L. Guerrouj, Y. Cao, and M. W. Godfrey,“Investigating code review quality: Do people and participation matter?”in International Conference on Software Maintenance and Evolution(ICSME), 2015, pp. 111–120.
[13] T. Patanamon, T. Chakkrit, G. K. Raula, Y. Norihiro, I. Hajimu, and
M. Ken-ichi, “Who Should Review My Code? A File Location-Based
Code-Reviewer Recommendation Approach for Modern Code Review,”
in 22nd International Conference on Software Analysis, Evolution, andReengineering (SANER), 2015.
[14] M. Zanjani, H. Kagdi, and C. Bird, “Automatically recommending peer
reviewers in modern code review,” IEEE Transactions on Software
Engineering, vol. PP, no. 99, 2015.
[15] A. Bosu and J. C. Carver, “How do social interaction networks inﬂuencepeer impressions formation? A case study,” in Open Source Software:
Mobile Open Source Technologies - 10th International Conference onOpen Source Systems (OSS), 2014, pp. 31–40.
[16] X. Yang, N. Yoshida, R. G. Kula, and H. Iida, “Peer review social
network (person) in open source projects,” IEICE TRANSACTIONS on
Information and Systems, vol. 99-D, no. 3, pp. 661–670, 2016.
[17] J. Cohen, E. Brown, B. DuRette, and S. Teleki, Best kept secrets of peer
code review. Smart Bear, 2006.
[18] J. H. Holland, Adaptation in natural and artiﬁcial systems: an introductory analysis with applications to biology, control, and artiﬁcial
intelligence. U Michigan Press, 1975.
[19] M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens, “Modern code
reviews in open-source projects: Which problems do they ﬁx?” in 11th
Working Conference on Mining Software Repositories (MSR), ser. MSR
2014, 2014, pp. 202–211.

377