22nd IEEE International Conference on Software Analysis,
Evolution, and Reengineering
Montréal, Quebec, Canada
March 3 - 6, 2015

Who Should Review My Code?
A File Location-Based Code-Reviewer Recommendation
Approach for Modern Code Review

Patanamon (Pick)
Chakkrit
Raula G. KulaThongtanunam Tantithamtavorn

patanamon-t@is.naist.jp

Norihiro
YoshidaHajimu Iida

@pamonKen-ichiMatsumoto1Modern Code Review is a tool-based, and used regularly
in practice nowadays at industrial and OSS projects

2Modern Code Review is a tool-based, and used regularly
in practice nowadays at industrial and OSS projects

GerritCode Review

Rietveld2Modern Code Review is a tool-based, and used regularly
in practice nowadays at industrial and OSS projects

GerritCode Review

Rietveld2Modern Code Review

Developer3Modern Code Review
GerritCode Review
Code change

Developer3Modern Code Review
GerritCode Review
Code change

Developer3Modern Code Review
Can you review mycode?
GerritCode Review
Code change

Developer

Invite Reviewers

Reviewers3Modern Code Review
Can you review mycode?
GerritCode Review
Code change

Developer

Coding stylepassed :)There’s abug!

Invite Reviewers

Reviewers
3

Code-Reviewer Assignment Problem
GerritCode Review
Code change

Developer

Invite Reviewers

4Code-Reviewer Assignment Problem
Who should review
my code?
GerritCode Review
Code change

New Developer

Invite Reviewers

4Code-Reviewer Assignment Problem
Who should review
my code?
GerritCode Review
Code change

New Developer

Invite Reviewers

4RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

5RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

Studied Projects

5

RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

Studied Projects
Sample Reviews

357

363

378

363

5

RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

Studied Projects
Sample Reviews

357

363

378

363

#Reviews w/
Assignment Problem

10%

5%

30%

4%

5

RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

Studied Projects
Sample Reviews

357

363

378

363

30

13 Days Longer

9 Days Longer

6 Days Longer

10

20

18 Days Longer

0

0

10

20

Reviewing Time(Days)

Reviewing time (Days)

30

Reviewing time (Days)#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem pro

Android

Android

OpenStackOpenStack

Qt

Qt

LibreOfficeLibreOffice

30
20
10

viewing time (Days)

Review
w/ Assignment
Review
w/o Assignment
Problem
eviews with code−reviewer assignment
problem
Reviews
withoutcode−rev
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems5

RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

Studied Projects
Sample Reviews

357

363

378

363

30

13 Days Longer

9 Days Longer

6 Days Longer

10

20

18 Days Longer

0

0

10

20

Reviewing Time(Days)

Reviewing time (Days)

30

Reviewing time (Days)#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem pro

Android

Android

OpenStackOpenStack

Qt

Qt

LibreOfficeLibreOffice30

viewing time (Days)

Review
w/ Assignment
Review
w/o Assignment
Problem
eviews with code−reviewer assignment
problem
Reviews
withoutcode−rev
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems

10

20

Reviews with code-reviewer assignment problem
significantly take 12 days longer to approve a code change.
5

RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

6RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

6RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

6RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

6RQ1: How do reviews with code-reviewer
assignment problem impact reviewing time?

A code-reviewer recommendation tool will be useful in distributed
software development to speed up the code review process.
6

RevFinder: A File Location-based Code-Reviewer
Recommendation Approach7

RevFinder: Overview
R1

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files- src/devices/screen.java

Code-Reviewers
Reviewed by John

8

RevFinder: Overview
R1

R3 (New)

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John?

8

RevFinder: Overview
R1

R3 (New)

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John?

Find reviewers who have reviewed
the most similar file paths

RevFinder
8

RevFinder: Overview
R1

R3 (New)

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John?

RevFinder
8

RevFinder: Overview
R3 (New)

R2

R1
Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by Pat

Reviewed by John?

CompareCompare(FilePath_R2,FilePath_R3) = 0.3
RevFinder
8

RevFinder: Overview
R1

R3 (New)

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John?

CompareCompare(FilePath_R2,FilePath_R3) = 0.3
Compare(FilePath_R1,FilePath_R3) = 0.6

RevFinder
8

RevFinder: Overview
R1

R3 (New)

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John?

Compare(FilePath_R2,FilePath_R3) = 0.3
Compare(FilePath_R1,FilePath_R3) = 0.6

RevFinder
9

RevFinder: Overview
R1

R3 (New)

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John?

Compare(FilePath_R1,FilePath_R3) = 0.6“Pat should review R3”

RevFinder
9

RevFinder: Overview
R1

R3 (New)

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John?

Compare(FilePath_R1,FilePath_R3) = 0.6“Pat should review R3”

RevFinder
9

RevFinder: FilePath Comparison Techniques
4 String Comparison Techniques
Longest Common Prefix

Longest Common Suffix

Longest Common Substring

Longest Common
Subsequence
10

RevFinder: FilePath Comparison Techniques
4 String Comparison Techniques
Longest Common Prefix

Longest Common Suffix

Longest Common Substring

Longest Common
Subsequence
10

RevFinder: FilePath Comparison Techniques
4 String Comparison Techniques
Longest Common Prefix

Longest Common Suffix

Combine(Borda Count)Longest Common Substring

Longest Common
Subsequence

Recommendation List
10

RevFinder: FilePath Comparison Techniques
4 String Comparison Techniques
Longest Common Prefix

Longest Common Suffix

Combine(Borda Count)Longest Common Substring
Voting Ranks
Longest Common
Subsequence

Recommendation List
10

RevFinder: FilePath Comparison Techniques
4 String Comparison Techniques
Longest Common Prefix
Bubble-up
correct reviewers
Longest Common Suffix

Combine(Borda Count)Longest Common Substring
Voting Ranks
Longest Common
Subsequence

Recommendation List
10

Empirical Evaluation

Studied Projects
Period~4 years~1year~1 year~2 year#Reviews

5,126

6,586

23,810

6,523#Reviewers

94

82

202

63

11

Empirical Evaluation

Studied Projects
Period~4 years~1year~1 year~2 year#Reviews

5,126

6,586

23,810

6,523#Reviewers

94

82

202

63

Baseline Approach: ReviewBot (Balachandran ICSE’13)
R3 (New)
R1

R2

Pat

John

“Recommend reviewers who havereviewed the same line of code changes”
11Empirical Evaluation

Studied Projects
Period~4 years~1year~1 year~2 year#Reviews

5,126

6,586

23,810

6,523#Reviewers

94

82

202

63

Baseline Approach: ReviewBot (Balachandran ICSE’13)
R3 (New)
R1

R2

Pat

John

John
should review
R3“Recommend reviewers who havereviewed the same line of code changes”
11Empirical Evaluation

12

Empirical Evaluation
Accuracy“RQ2: Does RevFinder accurately
recommend reviewers?”

12

Empirical Evaluation
Accuracy

Ranking Performance“RQ2: Does RevFinder accurately
recommend reviewers?”“RQ3: Does RevFinder provide a
better ranking of correct reviewers?”

12

Empirical Evaluation
Accuracy

Ranking Performance“RQ2: Does RevFinder accurately
recommend reviewers?”“RQ3: Does RevFinder provide a
better ranking of correct reviewers?”

12

Empirical Evaluation
Accuracy

Ranking Performance“RQ2: Does RevFinder accurately
recommend reviewers?”“RQ3: Does RevFinder provide a
better ranking of correct reviewers?”

Measure: Top-k Accuracy

12Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly
recommend with a Top-k recommendation.

13

Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly
recommend with a Top-k recommendation.

Top-5 Recommendation

13

Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly
recommend with a Top-k recommendation.

Actual Reviewer

Top-5 Recommendation

13

Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly
recommend with a Top-k recommendation.
Correct!
Actual Reviewer

Top-5 Recommendation

13

Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly
recommend with a Top-k recommendation.

Actual Reviewer

Top-10 Accuracy

Correct!

RevFinder
90%

ReviewBot

87%

86%
67.5%

74%

69%
41%

45%

29%

28%

22.5%

10%

0%
Top-5 Recommendation

RevFinder can correctly recommended 69%-87% of reviews with a top-10
recommendation. RevFinder is 4 times more accurate than ReviewBot.
13

Empirical Evaluation
Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

Measure: Top-k Accuracy

14

Empirical Evaluation
Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% ofreviews with a top-10 recommendation.

14

Empirical Evaluation
Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% ofreviews with a top-10 recommendation.

14

Empirical Evaluation
Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% ofreviews with a top-10 recommendation.

Measure: Mean Reciprocal
Rank (MRR)

14

Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach
can recommend.

15

Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach
can recommend.

ActualReviewer

vsRecommendation Lists

15

Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach
can recommend.
MRR
= 1/3

ActualReviewer

vsRecommendation Lists

15

Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach
can recommend.
MRR
= 1/3

MRR
= 1/1

ActualReviewer

vsRecommendation Lists

15

Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach
can recommend.
MRR
= 1/3

Rank ofCorrect
Reviewers
Rank

MRR
= 1/1
Android

200
100
50

ActualReviewer

Qt

LibreOffice

●

15

●

5

●
●
●
●
●
●
●

1

●

RevFinder

ReviewBot

RevFinder ReviewBot

vsOpenStack

RevFinder

ReviewBot

RevFinder

ReviewBot

RevFinder ReviewBot
RevFinder ReviewBot
Approach

RevFinder

ReviewBot

RevFinder ReviewBot

MRR
RevFinder

0.6

0.55

0.31

0.40

ReviewBot

0.25

0.30

0.22

0.07

Recommendation Lists

RevFinder recommended the correct code-reviewers with a
median rank of 2 - 8. The code-reviewers ranking of RevFinder
is 3 times better than that of ReviewBot.

15

Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach
can recommend.
MRR
= 1/3

Rank ofCorrect
Reviewers
Rank

MRR
= 1/1
Android

200
100
50

ActualReviewer

Qt

LibreOffice

●

15

●

5

●
●
●
●
●
●
●

1

●

RevFinder

ReviewBot

RevFinder ReviewBot

vsOpenStack

RevFinder

ReviewBot

RevFinder

ReviewBot

RevFinder ReviewBot
RevFinder ReviewBot
Approach

RevFinder

ReviewBot

RevFinder ReviewBot

MRR
RevFinder

0.6

0.55

0.31

0.40

ReviewBot

0.25

0.30

0.22

0.07

Recommendation Lists

RevFinder recommended the correct code-reviewers with a
median rank of 2 - 8. The code-reviewers ranking of RevFinder
is 3 times better than that of ReviewBot.

15

Empirical Evaluation
Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% ofreviews with a top-10 recommendation.

Measure: Mean Reciprocal
Rank (MRR)

16

Empirical Evaluation
Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% ofreviews with a top-10 recommendation.

Measure:Mean Reciprocal
On average,
RevFinder
can recommend
Ranka (MRR)
reviewers withmedian rank of 4.

16

Empirical Evaluation
Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% ofreviews with a top-10 recommendation.

Measure:Mean Reciprocal
On average,
RevFinder
can recommend
Ranka (MRR)
reviewers withmedian rank of 4.

We can use a similarity of previously reviewed file paths torecommend code-reviewers for new code changes.
16Summary17

Code-Reviewer Assignment Problem
Who should review
my code?
GerritCode Review
Code change

New Developer

Invite Reviewers

4Code-Reviewer Assignment Problem
Who should review
my code?
GerritCode Review
Code change

New Developer

Invite Reviewers

4How do reviews with code-reviewer assignment
problem impact reviewing time?

Code-Reviewer Assignment Problem
Who should review
my code?

GerritCode Review

Code change

New Developer

Invite Reviewers

Studied Projects
Sample Reviews

357

363

378

363

#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem pro
30

13 Days Longer

9 Days Longer

6 Days Longer

10

20

18 Days Longer

0

0

10

20

Reviewing Time(Days)

Reviewing time (Days)

30

Reviewing time (Days)

4

Android

Android

OpenStackOpenStack

Qt

Qt

LibreOfficeLibreOffice30

viewing time (Days)

Review
w/ Assignment
Review
w/o Assignment
Problem
eviews with code−reviewer assignment
problem
Reviews
withoutcode−rev
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems

10

20

Reviews with code-reviewer assignment problem
significantly take 12 days longer to approve a code change.
5How do reviews with code-reviewer assignment
problem impact reviewing time?

Code-Reviewer Assignment Problem
Who should review
my code?
GerritCode Review

Studied Projects
Sample Reviews

Code change

New Developer

357

363

378

363

30

9 Days Longer

6 Days Longer

20

13 Days Longer

10

Reviewing time (Days)

18 Days Longer

0

0

10

20

Reviewing Time(Days)

30

Reviewing time (Days)

30

Android Android
OpenStackOpenStack
Qt
Qt
LibreOfficeLibreOffice
Review
w/ Assignment
Review
w/o Assignment
Problem
Reviews with code−reviewer
assignment
problem
Reviews
withoutcode−revie
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems

4

Android

20
10

5

0

10

20

Reviews with code-reviewer assignment problem
significantly take 12 days longer to approve a code change.

0

Reviewing time (Days)

Invite Reviewers

30

Reviewing time (Days)#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem proble

OpenStackAndroid
Studied systems

Qt
OpenStackStudieHow do reviews with code-reviewer assignment
problem impact reviewing time?

Code-Reviewer Assignment Problem
Who should review
my code?

RevFinder: Overview
GerritCode Review

Studied Projects
Sample Reviews

Code change

New Developer

Reviewed by Pat

378

363

30
20

R3 (New)
13 Days Longer

9 Days Longer

6 Days Longer

10

Reviewing time (Days)

18 Days Longer

0

10

20

Reviewing Time(Days)

30

Reviewing time (Days)

30

Android Android
OpenStackOpenStack
Qt
Qt
LibreOfficeLibreOffice
Review
w/ Assignment
Review
w/o Assignment
Problem
Reviews with code−reviewer
assignment
problem
Reviews
withoutcode−revi
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems

04

Android- src/devices/screen.java

10
0

10

Changed Files

20

Reviews with code-reviewer assignment problem
significantly take 12 days longer to approve a code change.

20

Reviewing time (Days)

30

Reviewing time (Days)

R2

Invite Reviewers

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers

363#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem proble

0

R1

357

Changed Files

5

OpenStackQt
Android
OpenStack- src/video/recorder.java
Studied systems

Code-Reviewers
Reviewed by John

StudieCode-Reviewers?

Find reviewers who have reviewed
the most similar file paths

RevFinder
7

How do reviews with code-reviewer assignment
problem impact reviewing time?

Code-Reviewer Assignment Problem
Who should review
my code?
GerritCode Review

Studied Projects
Sample Reviews

Code change

New Developer

357

363

378

363

30

9 Days Longer

30

Reviewing time (Days)

30

Android Android
OpenStackOpenStack
Qt
Qt
LibreOfficeLibreOffice
Review
w/ Assignment
Review
w/o Assignment
Problem
Reviews with code−reviewer
assignment
problem
Reviews
withoutcode−revi
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems

4

20
10

5

0

10

20

Reviews with code-reviewer assignment problem
significantly take 12 days longer to approve a code change.

Android

OpenStackAndroid
Studied systems

RevFinder: Overview
R1

R3 (New)

R2

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers
Reviewed by Pat

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John?

Find reviewers who have reviewed
the most similar file paths

6 Days Longer

20

13 Days Longer

10

Reviewing time (Days)

18 Days Longer

0

0

10

20

Reviewing Time(Days)

0

Reviewing time (Days)

Invite Reviewers

30

Reviewing time (Days)#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem proble

RevFinder
7

Qt
OpenStackStudieHow do reviews with code-reviewer assignment
problem impact reviewing time?

Code-Reviewer Assignment Problem
Who should review
my code?

Empirical Evaluation
GerritCode Review

Studied Projects
Sample Reviews

Code change

New Developer

357

363

378

363

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% of
Changed Files
Files
reviews
withChanged
a top-10
recommendation.
ChangedFiles
Code-Reviewers

Code-Reviewers

30

Reviewing time (Days)

30

20
10
0

10

6 Days Longer

30
20
10
0

5

OpenStackAndroid

Qt
OpenStackStudied systems“Does RevFinder provide
a betterranking of correct reviewers?”

R3 (New)- src/video/recorder.java

0

Reviewing time (Days)

30
20
10
0

4

- src/devices/screen.java

9 Days Longer

Reviews with code-reviewer assignment problem
significantly take 12 days longer to approve a code change.

StudieMeasure:Mean Reciprocal
On average,
RevFinder
can recommend
Ranka (MRR)
reviewers withmedian rank of 4.

Review History- src/video/camera.java- src/video/player.java
Code-Reviewers

13 Days Longer

Android Android
OpenStackOpenStack
Qt
Qt
LibreOfficeLibreOffice
Review
w/ Assignment
Review
w/o Assignment
Problem
Reviews with code−reviewer
assignment
problem
Reviews
withoutcode−revi
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems“Does RevFinder accurately
RevFinder: Overview
recommend reviewers?”
R2

20

Reviewing Time(Days)

Android

R1

18 Days Longer

Ranking Performance

Invite Reviewers

Reviewing time (Days)

Accuracy

Reviewing time (Days)#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem proble

Reviewed by John?
We canuse a similarity
of previously reviewed file paths torecommend code-reviewers for new code changes.

Reviewed by Pat

Find reviewers who have reviewed
the most similar file paths

15

RevFinder
7

How do reviews with code-reviewer assignment
problem impact reviewing time?

Code-Reviewer Assignment Problem
Who should review
my code?
GerritCode Review

Studied Projects
Sample Reviews

Code change

New Developer

357

363

378

363

30
20

30

Reviewing time (Days)

30

20
10

5

0

20
10
4

Reviewed by Pat

R3 (New)

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John

OpenStackAndroid

Empirical Evaluation

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers

Qt
OpenStackStudied systems

Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% ofreviews with a top-10 recommendation.

?

Find reviewers who have reviewed
the most similar file paths

6 Days Longer

Reviews with code-reviewer assignment problem
significantly take 12 days longer to approve a code change.

RevFinder: Overview
R2

9 Days Longer

Android Android
OpenStackOpenStack
Qt
Qt
LibreOfficeLibreOffice
Review
w/ Assignment
Review
w/o Assignment
Problem
Reviews with code−reviewer
assignment
problem
Reviews
withoutcode−revi
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems

Android

R1

13 Days Longer

10

Reviewing time (Days)

18 Days Longer

0

0

10

20

Reviewing Time(Days)

0

Reviewing time (Days)

Invite Reviewers

30

Reviewing time (Days)#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem proble

StudieMeasure:Mean Reciprocal
On average,
RevFinder
can recommend
Ranka (MRR)
reviewers withmedian rank of 4.

We can use a similarity of previously reviewed file paths torecommend code-reviewers for new code changes.
RevFinder
7

15

How do reviews with code-reviewer assignment
problem impact reviewing time?

Code-Reviewer Assignment Problem
Who should review
my code?
GerritCode Review

Studied Projects
Sample Reviews

Code change

New Developer

357

363

378

363

30
20

30

Reviewing time (Days)

30

20
10

5

0

20
10
4

Reviewed by Pat

R3 (New)

Changed Files

Changed Files- src/devices/screen.java- src/video/recorder.java

Code-Reviewers

Code-Reviewers

Reviewed by John

OpenStackAndroid

Empirical Evaluation

Review History

Changed Files- src/video/camera.java- src/video/player.java
Code-Reviewers

Qt
OpenStackStudied systems

Accuracy

Ranking Performance“Does RevFinder accurately
recommend reviewers?”“Does RevFinder provide a better
ranking of correct reviewers?”

On average, RevFinder can correctly
Measure:Top-k Accuracy
recommend
reviewers
for 79% ofreviews with a top-10 recommendation.

?

Find reviewers who have reviewed
the most similar file paths

6 Days Longer

Reviews with code-reviewer assignment problem
significantly take 12 days longer to approve a code change.

RevFinder: Overview
R2

9 Days Longer

Android Android
OpenStackOpenStack
Qt
Qt
LibreOfficeLibreOffice
Review
w/ Assignment
Review
w/o Assignment
Problem
Reviews with code−reviewer
assignment
problem
Reviews
withoutcode−revi
Reviews
with Problem
code−reviewer
assignment
problem
Studied systems
Studied systems

Android

R1

13 Days Longer

10

Reviewing time (Days)

18 Days Longer

0

0

10

20

Reviewing Time(Days)

0

Reviewing time (Days)

Invite Reviewers

30

Reviewing time (Days)#Reviews w/
10%
5%
30%
4%
Assignment ProblemReviews with
Reviewscode−reviewer
with code−reviewer
assignmentassignment
problem problem
Reviews without
Reviewscode−reviewer
without code−reviewer
assignmentassignment
problem proble

StudieMeasure:Mean Reciprocal
On average,
RevFinder
can recommend
Rank
reviewers with a (MRR)
median rank of 4.

We can use a similarity of previously reviewed file paths torecommend code-reviewers for new code changes.
RevFinder
7

15