ReDA: A Web-based Visualization Tool for
Analyzing Modern Code Review Dataset
Patanamon Thongtanunam∗ , Xin Yang∗ , Norihiro Yoshida† , Raula Gaikovina Kula‡ , Ana Erika Camargo Cruz,
Kenji Fujiwara∗ , Hajimu Iida∗∗ Nara Institute of Science and Technology, Japan† Nagoya University, Japan‡ Osaka University, Japan{patanamon-t, kin-y, camargo, kenji-f}@is.naist.jp, iida@itc.naist.jp

yoshida@ertl.jp

raula-k@ist.osaka-u.ac.jp

Abstract—ReDA(http://reda.naist.jp/) is a web-based visualization tool for analyzing Modern Code Review (MCR) datasetsfor large Open Source Software (OSS) projects. MCR is acommonly practiced and lightweight inspection of source codeusing a support tool such as Gerrit system. Recently, mining
code review history of such systems has received attention
as a potentially effective method of ensuring software quality.
However, due to increasing size and complexity of softwares
being developed, these datasets are becoming unmanageable.
ReDA aims to assist researchers of mining code review data byenabling better understand of dataset context and identifying
abnormalities. Through real-time data interaction, users canquickly gain insight into the data and hone in on interestingareas to investigate. A video highlighting the main features canbe found at: http://youtu.be/ fEoTRRas0U
Keywords—Modern Code Review, Mining software repository,
Visualization toolI.

I NTRODUCTION

Software code review refers to inspection of the source codeby developers, other than the author, who have appropriate
knowledge of the software. Code review is an establishedmethod for reducing defects and improving software quality.
Nowadays, Modern Code Review (MCR) [1], an informal,
lightweight and tool-based code review methodology has
been widely used and put into development regularly in bothindustrial and OSS projects. Many tools such as Gerrit1 ,ReviewBoard2 , Rietveld3 are available to support MCR. ReDA
focus on Gerrit which supports code reviews for projects using
Git code repository system. Gerrit facilitates a review of codechanges and management of patchsets (set of changes) whichdevelopers expect to merge into the code repository.
Due to the increasing use of MCR in several large OSS
projects, massive volumes of code review data have beenrecorded. Their histories have been captured and these datasets
are publicly available online. Extracting knowledge from thesedatasets has produced promising research with the goal of
improving the software quality and software development
process. Recently, many studies have used code review datasets
to understand and improve both review effort [2]–[5] and review
quality [6], [7]. However, a raw code review dataset is generally
imperfect since the data collection process in each support tool
1 https://code.google.com/p/gerrit/
2 https://www.reviewboard.org/
3 https://code.google.com/p/rietveld/

varies in methodology, accuracy, and degree of automation[8]. Reliability of the data depends on the variety of inputs
and operations, for instance, input from default values of thesystems, mistaken input of users, and data missing or removed
data (cleaned up). As Kalliamvakou et. al. [9] reported thatwhile publicly available data from support tools is a rich sourceon mining, various potential perils should also be taken into
consideration. Therefore, to avoid such perils, researchers must
understand the context and quality of a dataset to decide whatfurther data pre-processing and cleansing is required.
To facilitate researchers understanding of a code review
dataset, we developed a prototype of a data visualization tool
named ReDA – Review Data Analyzer. ReDA is a lightweightand interactive web-based visualization tool that enables users
to quickly understand large and complex code review data.
ReDA leverages both review activity and human factors forthe analysis by extracting proprieties available from Gerrit andvisually presenting them to users. Throughout this paper, we
illustrate the use of ReDA using review history from Android
Open Source Project (AOSP) captured from October 2008 to
January 2012 with 11,632 reviews [10].
Related Work. It is generally accepted that visualization
can help developers to understand and discover knowledge
from a software repository. Minelli et al. [11] mined software
repositories of apps and presents the mined data to understandevolution of apps. Yongpisanpop et al. [12] also presents thatinteracting with big data in a bugs reporting system throughvisualization encourage developers to provide feedback as arequest for quality improvement. While bug reporting is closely
related to MCR, prior to ReDA there was no visualization tool
available explicitly for MCR.
In this paper, we demonstrate how ReDA assists researchers
in three ways: 1) Extracting and defining the dataset, 2) Showing
basic statistical analyses, and 3) Indicating interesting andpotentially major problems in the project.
II.

R E DA OVERVIEW

ReDA is a web application to enable distributed platform
independent access and a familiar, easy to use interface. It is
implemented with HTML5 and JavaScript for the front-end
and python Django for the server backend. Figure 1 shows anarchitectural overview of ReDA. MCR repository datasets (e.g.
from Gerrit system) are exported in JSON and imported to
ReDA database. From the database, ReDA can easily access

(a)	

Visualization
using d3.js	

Review History
Database	

Properties
Extractions	

Properties Dataset	Fig. 1: An architectural overview of ReDA(b)	Fig. 3: An overview of Activity Statistic page(a)	B. Activity Statistic

(b)	

(c)	

(d)	

(e)	

Fig. 2: An overview of Review Statistic pageand extract properties of the project and then visualizes themusing d3.js4 which is a javascript library for manipulating dataand graphically presents results. ReDA also provides summariesof the extracted properties.
To support researchers comprehensive understanding thedatasets, ReDA provides and visualizes the properties ofprojects from three MCR perspectives: 1) Review perspective
shown in Review Statistic page, 2) Process perspective shownin Activity Statistic page, and 3) Human perspective shownin Contributor Activities page.
A. Review Statistic
The goal of the Review Statistic page is to present an
overview of reviews that AOSP developers have performed. An
overview of Review Statistic page is shown in Fig. 2. On thispage, users can find a large graph on the top Fig. 2(a) showing
how many reviews have been created per week. The four small
graphs below Fig. 2(b)-(e) show the distribution of properties
which are the number of reviewers, the number of patchsets,
number of modified files and number of comments, respectively.
These graphs are interactive. Users can focus the data on any
graph by dragging a slider over any desired section. Then, every
graph will be changed corresponding to the selection area. For
example, if users want to see distributions of the metrics ofreviews created in 2011, they can drag over the Number ofReviews graph as the blue area shown in Fig.2(a). Then, thegraphs of Fig.2(b)-(e) will show the distributions corresponding
to the selected portion. Multiple graphs can be selected so users
can compare and understand relationships among properties indifferent areas of the data.
4 http://d3js.org/

The goal of Activity Statistic page is to present the overallof activity in the process of MCR. Figure 3 shows an overview
of this page where the large graph on the top Fig. 3(a) shows
the number of activities performed in Gerrit on a daily basis.
The small graph below Fig. 3(b) is for focusing and zoomingon activities history in a specific period. ReDA extracts theactivities in this process into seven different types: 1) Create
review, 2) submit patch, 3) code review, 4) verify, 5) mergechanges, 6) reject changes, and 7) give a comment. For zooming
detail, users can drag a selection over the graph of Fig. 3(b).
Then, the graph of Fig. 3(a) will display the selected portion.
The exact number of activities in each day will be shown inthe description box at the top right when the cursor is hoveringover the graph 3(a). ReDA also shows the date for Android
release version5 via a dashed line to show how these activities
are before/after the release date. For example, as shown in
Fig. 3(a), the number of activities before releasing the Froyo
version is higher than the activities after the release.
C. Contributor Activities
The goal of Contributor Activities page is to present the
viewpoint of human in MCR. Figure 4 shows an overview ofthis page where Fig. 4(a) shows contributors graph, Fig. 4(b)describes chart legend and Fig. 4(c) is the visualizing menu forthis graph. In Fig. 4(a), ReDA plots a bubble chart where eachbubble represents individual contributor. For each contributor,
ReDA extracts the number of four main activities: 1) Create
Review, 2) code review, 3) verify and 4) comment. The size of
bubble represents the total number of their contributions. The
larger bubbles present the more contributions. The color of the
bubble represents the combination of activity types that they
have done. The color reference is described in the panel in Fig.
4(b). For instance, a blue bubble means they did all four typesof contributions, and an orange bubble means they only did acode review. Users can see the exact number of contributions byhovering the cursor over the desired bubble. Then, a description
box will appear next to the cursor. Moreover, users can filter thecontributors by selecting the minimum number of contributions
at the drop-down box in Fig. 4(c). When the number is selected,
the graph will be automatically re-loaded.
5 based on http://en.wikipedia.org/wiki/Android (operating system)

(c)	

(b)	

(a)	

(a) Unusual number of created reviews in Review Statistic Page

Fig. 4: An overview of Contributors Activities page

ReDA can separate contributors into groups to view different
aspects. For example shown in Fig. 4(a), the contributors aregrouped by the activity they have done the most. Users canselect the type of group at the menu at Fig. 4(c). This canvisualize the relationship between activities they have done
(represented by color) and the type of grouping. For example,
from Fig. 4(a), we found that most of contributors, who createa review as the activity they have done the most, had no verify
activity (yellow and pink bubbles).
III.

I LLUSTRATIVE U SAGE S CENARIO

In this section, we demonstrate the use of ReDA to identify
abnormalities in the dataset and characteristics of activities in
AOSP. Moreover, we examined the dataset according to thefindings to uncover their causes.
At the Review Statistic page, the number of reviews being
created in the end of 2011 is markedly higher than otherperiods over the time as shown in Fig. 5(a). Similarly, theActivity Statistic page shows that in the same period, thenumber of created reviews is unusually high as shown in Fig.
5(b). In particular, the number of reviews created is high on
23/12/2010 and the number of reviews abandoned is high on
29/12/2010. Following this evidence, we investigate the causeof this phenomenon using the following two questions:
Q1:
Q2:

What reviews were created and abandoned in thisperiod?
Is there a review in the dataset similar to thesereviews?

To answer Q1, we first filtered and downloaded the dataset
in the Reviews Statistic page by dragging over the highest
bar on the Number of Review graph. We found that from 374
reviews6 created on 23/12/2011, there are 356 reviews (98%)
that were abandoned on 29/12/2010. Moreover, these reviews
were created by the same developer at consecutive times. These
reviews are the changes for supporting Nvidia Tegra family.
The properties’ values of these reviews can be summarized
as #Reviewers ≤ 1, #Patchset = 1, #Modified
files > 0, and #Comment = 1. We scanned through theircomments to find the reason of creating and abandoning thesereviews. Unfortunately, there is no explanation from the owner.
6 Result for Q1: http://reda.naist.jp/static/findings/ReDA finding 1.htm(b) Unusual number of activities in Activity Statistic Page

Fig. 5: An abnormality identified using ReDA

Only automatically generated messages recorded that thesechanges were abandoned. We can answer Q1 that the reviews
were created and abandoned in this period are those without
performing code review.
To better understand this phenomenon and answer Q2, weselected reviews that have the same metric values for the wholecaptured period. We simply selected these reviews by dragging
a selection over the bars on the small four graphs in the Review
Statistic page. There are 1,305 reviews7 returned and 924 (89%)
of them are abandoned. We then grouped reviews that occurredwith the same pattern as found in Q1 i.e. reviews created by
the same developer at consecutive time. We then manuallyscanned through the comments to find an explanation. For the
result, we can identify 633 reviews that have the same pattern
including reviews found in Q1. Moreover, we found that thisphenomenon happens more than once a month. From theircomments we found that 18% of these reviews are abandoned
because these changes were already fixed, had a new version,or the code is obsolete; and 11% of them are abandoned dueto the mistakes of developers such as a lack of Git and Gerritknowledge, writing a bad commit message, and accidentally
uploading the changes. However, the rest of these reviews
have no explanation. We conjecture that the cause of thesereviews can be one of the causes we have found previously. For
instance, it is possible that developers accidentally uploaded
the changes if we consider that the creation of reviews is
continuous (the difference of creation time is in seconds). A
further investigation is needed to explain this clearly. At thisstage, we can answer Q2 that there are reviews that are similar
to reviews we found in Q1 and they occasionally occurred overthe project.
Besides the findings we have discussed, we also discovered
other abnormalities and characteristics. In Table I, we summarizes the findings that we have identified using ReDA. This
7 Result for Q2: http://reda.naist.jp/static/findings/ReDA finding 2.htm

TABLE I: Summary of findings are discovered using ReDA.
F1:

Summary
10% of all reviews were created not for code review.

F2:

Review history in AOSP is
missing.

F3:

AOSP developers usually docode review on weekdays.
Most of main contributors of
AOSP are the members of
Google and Android teams.

F4:

Evidences in ReDA
In Review Statistic and Activity Statistic page, there are reviews created andabandoned with unusual numbers at theend of year 2010.
In Review statistic page, there are reviews that do not contain changes (Nomodified file).
In Number of Reviews graph of Review
Statistic page, there is no reviews created from July 2011 to January 2012.
In Activity Statistic page, the numberof activities are periodically low.
In Contributor Activities, most of large
bubbles have blue color.

table describes evidences shown in ReDA and findings fromthese evidences.
IV.

E XPERIENCE WITH R E DA.

From our experience, we mined MCR dataset of AOSP to
better understand human aspects. We identified main contributors by profiling from code review history [2] and analyzed
important roles in MCR using social network analysis (SNA)[3].
From our studies, we found that main contributors are AOSP
and Google people. This result is corroborated by finding F4of ReDA described in Table I. Additionally, we also used
the Contributors Activities page validating the result of SNA,
concretely when we separated a group of verifier and nonverifier. According to this, we believe that ReDA provides
evidence that can be investigated to improve the code review
process in the future.
V.

S UMMARY & F UTURE W ORK

In this paper, we present ReDA a visualization tool forunderstanding code review datasets generated through MCR
support tools. We demonstrate the use of ReDA on a large,
complex OSS project and find code review abnormalities. Using
ReDA, we found that there is noise and incorrect information
hidden in the dataset. This shows that researchers must be awareand avoid this peril and carefully refine the dataset before usingit.
Future work. We will develop ReDA to provide realtime code review data “dashboard” system. This would enable
users to monitor current status of code reviews and quickly
indicate problems in the project and code review process. Other
important aspects of code reviews will also be developed to
expand code review viewpoints. Furthermore, ReDA is not
limited to only researchers. It is also useful for industrial project
managers to monitor projects and quickly identify problems;or perform lessons learned for process improvement activity
or for use in satisfying CMMI process quality requirements.
We will develop a portable version of ReDA as well as makeit compatible for other MCR support tools.

Findings
There are 924 reviews that were created and abandoned without
performing code review. There are 55% of these reviews wereabandoned without discussion. Furthermore, some of them werecreated by mistakes such as uploading changes that already fixed,
accidentally uploading changes.
There are 217 reviews that have no modified file. From their commit
messages, we found that 93% of them are branch merging requests
which are not related code reviews.
We found that on this period, Git and Gerrit servers were down fora while. This was reported in Google forum [13] by Jean-Baptiste
Queru, the software engineer for Google AOSP.
We found that the days with low number of activities are usually
Saturday and Sunday.
We found that most of large blue bubbles have email domain
google.com and android.com. These contributors also have a numberof activities higher than the others in the same group of the firstyear of contribution.

ACKNOWLEDGMENTS
We are thankful to Assoc. Prof. Daniel Port for his valuablesuggestions and constructive comments.
R EFERENCES[1] A. Bacchelli and C. Bird, “Expectations, Outcomes, and Challenges ofModern Code Review,” in Proc. of ICSE’13, 2013, pp. 712–721.
[2] R. G. Kula, C. C. A. Erika, N. Yoshida, K. Hamasaki, K. Fujiwara,
X. Yang, and H. Iida, “Using Profiling Metrics to Categorise Peer Review
Types in the Android Project,” in Proc. of ISSRE’12, 2012, pp. 146–151.
[3] X. Yang, R. G. Kula, C. C. A. Erika, N. Yoshida, K. Hamasaki,
K. Fujiwara, and H. Iida, “Understanding OSS Peer Review Roles
in Peer Review Social Network (PeRSoN),” in Proc. APSEC’12, 2012,
pp. 709–712.
[4] V. Balachandran, “Reducing Human Effort and Improving Quality inPeer Code Reviews using Automatic Static Analysis and Reviewer
Recommendation,” in Proc. ICSE’13, 2013, pp. 931–940.
[5] P. Thongtanunam, R. G. Kula, A. E. C. Cruz, N. Yoshida, andH. Iida, “Improving Code Review Effectiveness through Reviewer
Recommendations,” in Proc. of CHASE’14, 2014, pp. 119–122.
[6] S. Mcintosh, Y. Kamei, B. Adams, and A. E. Hassan, “The Impact ofCode Review Coverage and Code Review Participation on Software
Quality Categories and Subject Descriptors,” in Proc. of MSR’14, 2014,
pp. 192–201.
[7] M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens, “Modern Code
Reviews in Open-Source Projects : Which Problems Do They Fix ?
Categories and Subject Descriptors,” in Proc. of MSR’14, 2014, pp.
202–211.
[8] A. Mockus, “Engineering Big Data Solutions,” in Proc. of FOSE’14,
2014, pp. 85–99.
[9] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German,
and D. Damian, “The Promises and Perils of Mining GitHub Categories
and Subject Descriptors,” in Proc. of MSR’14, 2014, pp. 92–101.
[10] K. Hamasaki, R. G. Kula, N. Yoshida, C. C. A. Erika, K. Fujiwara, andH. Iida, “Who does what during a Code Review ? An extraction of an
OSS Peer Review Repository,” in Proc. MSR’13, 2013, pp. 49–52.
[11] R. Minelli and M. Lanza, “SAMOA – A Visual Software Analytics
Platform for Mobile Applications,” in Proc. of ICSM’13, 2013, pp.
476–479.
[12] P. Yongpisanpop, H. Hata, and K. Matsumoto, “Bugarium: 3d Interaction
for Supporting Large-Scale Bug Repositories Analysis,” in Proc. of ICSE
Companion’14, 2014, pp. 500–503.
[13] J.-B.
Queru,“GoogleGroupsDisccusion.”[Online].
Available: https://groups.google.com/forum/#!searchin/android-contrib/
gerrit$20down/android-contrib/m5FPkdI3ImQ/vL9 61Dof6AJ