Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Improving Code Review Effectivenessthrough Reviewer Recommendations
Patanamon Thongtanunam* andRaula Gaikovina Kula†, Ana Erika Camargo Cruz*,
Norihiro Yoshida*, Hajimu Iida**Nara Institute of Science and Technology (NAIST)†Osaka University, Japan

2/6/2014

1

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Introduction• Code Review : A source code inspection performed bydevelopers other than the author• Supported Tools: Gerrit, ReviewBoad, etc.
• A patch reviewed by developers with related knowledge will

have high quality and low defects??

Author
2/6/2014

A new Patch(Software contribution)

ReviewerProject’s
Code Repository

Introduction > Reviewer Recommendation > Evaluation > Summary2

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Reviewer Assignment Problem• It is difficult and time-consuming to find an appropriate

reviewers in large-scale projects
Who isappropriate toreview my code?

e.g. Globally distributed software development

2/6/2014

Introduction > Reviewer Recommendation > Evaluation > Summary3

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Previous Work
• Review Bot’s algorithm[1]• For industrial setting of VMware• Selects reviewers who have reviewed in the same section ofcode change.
• Required a long history of code changes in code review

Code change

R1

R2
Past Reviews

R3

R4 (New Review)

Reviewer Candidates = Reviewers ([R3, R1])

[1] V. Balachandran, “Reducing Human Effort and Improving Quality in Peer Code Reviews using Automatic Static Analysis and Reviewer Recommendation,” inProc. ICSE’13, 2013, pp. 931–940.

2/6/2014

Introduction > Reviewer Recommendation > Evaluation > Summary4

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Reviewer Recommendation Algorithm• Propose File Path Similarity (FPS) algorithm• Select reviewers by calculating similarity of file paths
comparing with file paths of previous reviews.
Proj
Core
…

UI
…

…
…

…
…

• Files with similar functions areusually located under the sameor near directories. [2]• Reviewers having similar
knowledge of system would
have reviewed similar file.

[2] Ivan T. Bowman, Richard C. Holt, and Neil V. Brewster. Linux as a case study: Its extracted software architecture. In Proceedings ICSE ’99, pp. 555–563, 1999.

2/6/2014

Introduction > Reviewer Recommendation > Evaluation > Summary5

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Example of FPS algorithm
Past Reviews
Review History

Review R1
Files- video/resource/a.xml
Reviewers
A

Review R2
Files- video/src/x.java- video/src/y.java
Reviewers

FPS	A

B

FPS	

Score of past reviewers propagate to reviewers

2/6/2014

1A= FPS(R3,R1) + FPS(R3,R2) = 0.1 + 0.5 = 0.6

2

B= FPS(R3,R1) = 0.1

Review R3
Files- video/src/a.java- video/src/b.java
Reviewers?

?

Introduction > Reviewer Recommendation > Evaluation > Summary?

6and interests [9].
The calculation of FPS function is described in Equation
review requestwiththe highest score from the sorted list.
ers, Approvers
1.
This
calculates a score of a past review (Rp ) from an
can give opinion
The
calculationof FPS function is described in Equation
review request
averageofsimilarityof every file in Rp (fp ) comparing withmine the quality
1.
This
calculates
ascoreof a pastthroughreview(RpRecommendations) from an (CHASE 2014)
Improving
Code
Review Effectiveness
Reviewer
an give opinion
every
fileinR(f).
TheFilesfunction
returnsa set of filenn
responsible foraverageofsimilarity
ofevery
fileinR(f)comparing
withppne the quality
paths
oftheinput
review.
The
Similarity(f,f)
functionnp
mated by Gerrit.
every file in Rn (fn ). The Files function returns a set of file
responsible formeasures the similarity between fp and fn , using Equation 2.
while approvers
paths
of the input review. The Similarity(fn , fp ) function
ated by Gerrit.
Theaveragedsimilarity score is prioritized by m and value.
nated by projectmeasuresthesimilarity
between fp and fn , using Equation 2.
while approvers
Sameastimeprioritizationof the Review Bot’s algorithm,
ess, a patch canTheaveragedsimilarity
score
isprioritized by m and value.
ated by project
theparameterisatimeprioritization
factor ranging (0, 1].
eview request
is PathSame
astimeprioritizationoftheReview
Bot’salgorithm,•FileSimilarity
scoreofnew
review
Randreview
Rp	n is notpast
ss, a patch canWhen=1,thetimeprioritizationconsidered.
verifier.
the parameter is a time prioritization factor ranging (0, 1].
view request isP
hor requests apWhen = 1, the time prioritization
is not considered.
erifier.
Similarity(fn , fp )ormation about
fn 2Files(RP n ),or requests apfp 2Files(Rp )Similarity(fn , fp )for the authorsmrmation about
FPS(R,R,m)=⇥(1)
np
f2Files(R),nnomatic reviewer|Files(R)|⇥|Files(R)|p
fp 2Files(Rp ) nor the authors
me reviews to beFPS(R⇥(1)
n , Rp , m) =matic reviewer|Files(Rn )| ⇥ |Files(Rp )|Function:
reviews to• Similarity
becommonPath(fn , fp )Similarity(fn , fp ) =(2)max(Length(f),Length(f))
npcommonPath(fn , fp )ATION
Similarity(fn , fp ) =(2)max(Length(fnn),, fLength(fIn Equation 2, the commonPath(fp )) countsp ) function
m selects candiATION
the number of common directory and/or file name that aples with similar
In Equation 2, the commonPath(fn , fp ) function counts
pearin ,fboth
file path
fromPrefix
the function
beginning.
Thiscountis
m selects candi• commonPath(fLongestCommon
returns
numberof apdirectory/filem also uses time
the numbercommondirectory
and/or filename
thatn p) :ofbasedon
theassumption
thatfiles, which are under thees with similarnames thatappearfrom
thebeginningof both
file path
Bot’s algorithm.
pearin both
filepath fromthebeginning. This count issame directory, would have the similar function. Thus, thealso uses time
gorithm 1. This
based on the assumption that files, which are under the
first directory of file paths is compared firstly. Then, theBot’s algorithm.
equest (Rn ) andsamedirectory,
would have the similarfunction. Thus,commonPath(“video/src/a.java”,= 2theother components
of file path are “video/src/b.java”)
compared respectively.
Forgorithm 1. Thisommended. The
first directory of file paths is compared firstly. Then, theexample, suppose f is /src/camera/video/a.java and fquest (Rn ) andother components of nfile path are compared respectively. Forp
is /src/camera/photo/a.java.
Thecommon>pathwill bemmended.
Introduction> Reviewer> Evaluation
Summaryand 2/6/2014The
7
example, suppose
fnRecommendation
is /src/camera/video/a.java
and fp/src/camera and the commonPath(fn , fp ) returns 2. The
is /src/camera/photo/a.java. The common path will beandcount of commonPath(fn , fp ) can be formularized as Equa-FPS score

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Evaluation• Accuracy (same as the review bot research)
Number of Correct Top-k recommendation	

Top-k Accuracy = 	Total number of reviews	

• Study Projects• Using Gerrit code review

2/6/2014

Projects

Study Period*# of Reviews# of Files

AOSP

Oct 2008 – Jan 2012(~2 years)

5,126

26,840OpenStackJul 2011 – May 2012(~1 year)

6,586

16,953Qt

May 2011 – May 2012(~1 year)

23,81078,401*Study period started from the 1st year of using peer code review.

Introduction > Reviewer Recommendation > Evaluation > Summary

8

80

80

Accuracy (%)

60

2000

3000

4000

Number of Reviews (N)100(a) AOSP

5000

20201000FPS Algorithm
Review Bot's Algorithm

00FPS Algorithm
Review Bot's Algorithm
0Top-5 Accuracy (%)

40

Accuracy (%)

60
40
20

Accuracy (%)

80

Results

40.50 % 39.21 % 38.29 % 21.30 %21.38 % 21.63 % 21.71 % 21.77 % 44.50 %67.66 % 66.56 % 64.83 % 63.38 %29.15 % 29.17 % 29.17 % 29.20 % 29.20 %75.30 % 74.17 % 72.36 % 70.80 %29.30 % 29.30 % 29.30 % 29.30 % 29.30 %36.47 % 35.18 % 34.73 % 34.04 %22.94 % 23.23 % 23.19 % 23.20 % 23.19 %ImprovingCode2014)63.09% 62.45% Review
62.10 Effectiveness% 61.66 % through35.76 % Reviewer
35.76 %Recommendations
35.68 % 35.55 %(CHASE
35.53 %
73.28 % 72.85 % 72.62 % 71.71 %38.78 % 38.90 % 38.82 % 38.89 % 38.89 %11.64 % 10.21 %9.45 %
8.88 %
18.64 % 18.70 % 18.72 % 18.71 % 18.75 %21.39 % 20.15 % 19.27 % 18.46 %26.18 % 26.19 % 26.16 % 26.16 % 26.16 %27.25 % 26.07 % 25.34 % 24.36 %27.18 % 27.18 % 27.17 % 27.19 % 27.19 %60Qt

41.81 %
69.84 %
77.12 %
38.32 %
67.57 %
77.97 %
13.02 %
28.82 %
36.86 %

40OpenStack

Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5

0

1000

2000

3000

4000

5000

Number of Reviews (N)(b) OpenStack

6000

FPS Algorithm
Review Bot's Algorithm

0AOSP

0

5000

10000

15000

20000

Number of Reviews (N)(c) Qt77.97
Figure 2:77.12
Top-5 accuracyof FPS algorithm and Review Bot’s algorithm with time prioritization factor ( = 1)

38.78WORKS
36.865. CONCLUSION& FUTURE
50
29.3
27.18
In this study, we have proposed a recommendation
algo-• FPS algorithm significantly[3] V. Balachandran. Reducing Human E↵ort andImproving Quality in Peer Code Reviews using
Automatic Static Analysis and Reviewer
rithm using file path similarity for the modern peer code reRecommendation. In Proc. ICSE ’13, pages 931–940,
view process. The results indicate that our proposed FPS al2013.
0gorithm e↵ectively
recommend reviewers in two of the three[4]E.
T. Barr, C. Bird, P. C. Rigby, A. Hindle, D. M.
OSS projects. ThisalgorithmOpenStackalso significantly outperform
AOSP
Qt
German, and P. Devanbu. Cohesive and Isolated
the existing algorithm (i.e. Review Bot’s algorithm) in theseDevelopment with Branches. In Proc. FASE ’12, pages
projects. Additionally, we found that the used of time priFPS
algorithm
Review
Bot'salgorithm
316–331, 2012.
oritization was surprisingly not appropriate for the recom[5] K. Hamasaki, R. G. Kula, N. Yoshida, C. C. A. Erika,
mendation algorithms in distributed projects environment.
K. >Fujiwara,and H.>Iida.
Who does what during a 9
Introduction>Reviewer
Recommendation
Evaluation
Summary
Our
futurework
willconcentrate
onexplore
moreinsight
2/6/2014
Code Review ? An extraction of an OSS Peer Review
into projects, especially large scale projects to improve theRepository. In Proc. MSR’ 13, pages 49–52, 2013.
algorithm. The other impact factors such as directory structure will also be investigated. At the same time, we will
[6] E. Kocaguneli, T. Zimmermann, C. Bird, N. Nagappan,

outperformed Review Bot’salgorithm

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)
Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

Reviewer Assignment Problem

Reviewer Recommendation Algorithm• It is difficult and time-consuming to find an appropriate• Propose File Path Similarity (FPS) algorithm• Select reviewers by comparing file paths with previous
reviews

reviewers in large-scale projects

Proj

Who isappropriate toreview my code?

• Files with similar functions areusually located in the same orCore
UI
near directories. [1]• Reviewers having similar…
…
…
…
knowledge of system would
Table 2: Top-k accuracy of FPS algorithm and Review Bot’s algorithm with di↵erent time prioritization factors ( )have reviewed similar files.
… …

Summary

Projecte.g. Globally distributed software development

AOSP

2/6/2014

Introduction > Reviewer Recommendation > Evaluation > Summary

3
Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)

2/6/2014

OpenStackQtEvaluationFPS Algorithm
Review Bot’s Algorithm=1= 0.8= 0.6= 0.4= 0.2=1= 0.8= 0.6
= 0.4
= 0.2
Top-1
41.81 % 40.50 % 39.21 % 38.29 % 21.30 %21.38 % 21.63 % 21.71 % 21.77 % 44.50 %Top-3
69.84 % 67.66 % 66.56 % 64.83 % 63.38 %29.15 % 29.17 % 29.17 % 29.20 % 29.20 %Top-5
77.12 % 75.30% 74.17 Recommendation% 72.36 % 70.80 % > Evaluation
29.30 % 29.30>%Summary
29.30 % 29.30 % 29.30 %Introduction> Reviewer
Top-1
38.32 % 36.47 % 35.18 % 34.73 % 34.04 %22.94 % 23.23 % 23.19 % 23.20 % 23.19 % 6
ImprovingCode2014)Top-3
67.57 % 63.09% 62.45% Review
62.10 Effectiveness% 61.66 % through35.76 % Reviewer
35.76 %Recommendations
35.68 % 35.55 %(CHASE
35.53 %
Top-5
77.97 % 73.28 % 72.85 % 72.62 % 71.71 %38.78 % 38.90 % 38.82 % 38.89 % 38.89 %Top-1
13.02 % 11.64 % 10.21 %9.45 %
8.88 %
18.64 % 18.70 % 18.72 % 18.71 % 18.75 %Top-3
28.82 % 21.39 % 20.15 % 19.27 % 18.46 %26.18 % 26.19 % 26.16 % 26.16 % 26.16 %Top-536.86 % 27.25 % 26.07 % 25.34 % 24.36 %27.18 % 27.18 % 27.17 % 27.19 % 27.19 %Top-kResults# of Files• Review Bot’s algorithm[1]Oct 2008 – Jan 2012(~2 years)

5,126

26,840• For industrial setting of VMware

OpenStackJul 2011 – May 2012(~1 year)

6,586

16,953

23,81078,401

2/6/2014

R1R2
Past Reviews

R3*Study period started from the 1st year of using peer code review.

Introduction > Reviewer Recommendation > Evaluation > Summary

R4 (New Review)

Reviewer Candidates = Reviewers ([R3, R1])[1] V. Balachandran, “Reducing Human Effort and Improving Quality in Peer Code Reviews using Automatic Static Analysis and Reviewer Recommendation,” inProc. ICSE’13, 2013, pp. 931–940.

2/6/2014

2/6/2014

Introduction > Reviewer Recommendation > Evaluation > Summary

42000

3000

4000

100(a) AOSP

5000

80
60
2040

Accuracy (%)

6080
1000

FPS Algorithm
Review Bot's Algorithm
01000

2000

3000

4000

5000

6000

Number of Reviews (N)(b) OpenStack

FPS Algorithm
Review Bot's Algorithm
05000

10000

15000

20000

Number of Reviews (N)(c) Qt77.97
Figure 2:77.12Top-5 accuracyof FPS algorithm and Review Bot’s algorithm with time prioritization factor ( = 1)

• FPS algorithm significantly

38.78WORKS
36.86
CONCLUSION& FUTURE[3] V. Balachandran. Reducing Human E↵ort and50
29.3
Improving Quality in Peer Code Reviews using
27.18
In this study, we have proposed a recommendation
algoAutomatic Static Analysis and Reviewer
rithm using file path similarity for the modern peer code reRecommendation. In Proc. ICSE ’13, pages 931–940,
view process. The results indicate that our proposed FPS al2013.
0gorithm e↵ectively
recommend reviewers in two of the three[4] E. T. Barr, C. Bird, P. C. Rigby, A. Hindle, D. M.
OSS projects. ThisalgorithmOpenStackalso significantly outperform
AOSP
Qt
German, and P. Devanbu. Cohesive and Isolated
the existing algorithm (i.e. Review Bot’s algorithm) in theseDevelopment with Branches. In Proc. FASE ’12, pages
projects. Additionally, we found that the used of time priFPSwas
algorithm
Review
Bot'sforalgorithm
316–331, 2012.
oritization
surprisingly not
appropriate
the recom[5] K. Hamasaki, R. G. Kula, N. Yoshida, C. C. A. Erika,
mendation algorithms in distributed projects environment.
K. >Fujiwara,and H.>Iida.
Who does what during a 10
Introduction> onReviewer
Recommendation
Evaluation
Summary
Our future work
will concentrate
explore moreinsight
2/6/2014
Code Review ? An extraction of an OSS Peer Review
into projects, especially large scale projects to improve theRepository. In Proc. MSR’ 13, pages 49–52, 2013.
algorithm. The other impact factors such as directory structure will also be investigated. At the same time, we will
[6] E. Kocaguneli, T. Zimmermann, C. Bird, N. Nagappan,consider ways to balance the workload of reviewers to helpand T. Menzies. Distributed development consideredreviewers and reduce the number of awaiting reviews.
harmful? In Proc. ICSE ’13, pages 882–890, 2013.
[7] a. Mockus and J. Herbsleb. Expertise Browser: aquantitative approach to identifying expertise. In Proc.
ACKNOWLEDGMENTSICSE ’02, pages 503–512, 2002.
We are thankful to Dr. Mike Barker from NAIST for his[8] N. Ramasubbu and R. K. Balan. Globally Distributed
valuable suggestions and discussions.
Software Development Project Performance : An
Empirical Analysis. In Proc. ESEC/FSE ’07, pages
6. REFERENCES
125–134, 2007.
[1] A. Aurum, H. Petersson, and C. Wohlin.
[9] P. C. Rigby and M.-A. Storey. Understanding
State-of-the-art: software inspections after 25 years.
broadcast based peer review on open source software
Software Testing, Verification and Reliability,
5.

940

Accuracy (%)

20FPS Algorithm
Review Bot's Algorithm0

Number of Reviews (N)AOSP• Selects reviewers who have reviewed in the same section ofMay 2011 – May 2012
Qtcode change.
(~1 year)60
20# of Reviews0

Study Period*Top-5 Accuracy (%)

Projects0• Using Gerrit code review

Previous Work0• Study Projects

Improving Code Review Effectiveness through Reviewer Recommendations (CHASE 2014)40

Number of Correct Top-k recommendation!
Total number of reviews!

Accuracy (%)

Top-k Accuracy = !

80• Accuracy (same as the review bot research)outperformed Review Bot’s
algorithm

Introduction > Reviewer Recommendation > Evaluation > Summary10