DOI:10.1587/transinf.2015EDP7261
Publicized:2015/11/27
This advance publication article will be replaced by the finalized version
after proofreading.

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

1

PAPER

Peer Review Social Network (PeRSoN) in Open Source Projects
Xin YANG†a) , Nonmember, Norihiro YOSHIDA††b) , Member, Raula GAIKOVINA KULA†††c) , Nonmember,and Hajimu IIDA†d) , Member

SUMMARY Software peer review is regarded as one of the most important approaches to preserving software quality. Due to the distributed
collaborations in Open Source Software (OSS) development, the review
techniques and processes conducted in OSS environment differ from thetraditional review method that based on formal face-to-face meetings. Unlike other related works, this study investigates peer review processes of
OSS projects from the social perspective: communication and interactionin peer review by using social network analysis (SNA). Moreover, the relationship between peer review contributors and their activities is studied.
We propose an approach to evaluating contributors’ activeness and social
relationship using SNA named Peer Review Social Network (PeRSoN). We
evaluate our approach by empirical case study, 326,286 review comments
and 1,745 contributors from three representative industrial OSS projects
have been extracted and analyzed. The results indicate that the social network structure influences the realistic activeness of contributors significantly. Based on the results, we suggest our approach can support project
leaders in assigning review tasks, appointing reviewers and other activities
to improve current software processes.
key words: Peer Review, Open Source Software, Social Network Analysis

1. Introduction
Software peer review refers to the code inspections by developers, rather than the authors themselves. It can be regarded as one of the most important activities to guarantee
the quality of software products [1] [2]. Software projects
adopt peer review for two principal reasons: reducing defects and saving development cost. The traditional code review (a.k.a code inspection) was established 30 years ago.
Code inspection requires experienced reviewers meet anddiscuss the source code written by other developers [3] [4].
Peer review is not only an indicator of the quality of sourcecode but also signifies a healthy organization. Stable andgrowing development communities always hope the experienced developers could share their knowledge with newmembers. For example, developers share knowledge when
they perform code review activities [5].
Recently, the peer review process of Open Source Software (OSS) varies from the traditional industrial setting.
One main reason is most OSS projects are geographically
distributed, whereas traditional industry projects take theform of gathering developers in the same room. The modern OSS peer review applies a broadcasting method to announce code review tasks and locate appropriate reviewers.
However, we find that only a few of studies have focused on
the code review process.
Our objective is establishing a model of OSS code review community and a set of quantitative measures to describe the code review process from both technical metrics
and non-technical metrics. Another motivation comes fromthe importance of the human factor in software development. The human factor has been researched from the diversity of different cultures and the rise of globally distributed
projects [6].
To understand how contributors work and communicate together, we need to investigate the structure of code review community. In this work, we present PeRSoN, which
is a construction of social networks from peer review activities, which is based on our previous work [7] [8]. We
categorize contributors into different role groups based on
their authorities, and we define the review activity as any
contribution in the review process. We use the PeRSoN toevaluate two research questions: RQ1. Which contributorrole is the most important in the peer review community?
and RQ2.What is the relationship between contributors’ activities and their network position?.
We applied PeRSoN to three large-scale OSS projects:
Android Open Source Project (AOSP), Qt and OpenStackby case study. The results of analysis addressed our researchquestions and gave hints about the relationships among OSS
peer review contributor roles, their activities and their network structure. Our main findings can be summarized as
two points. First, the contributors who have the verification
authority are the most important (most central) role in thereview community (see Table 8). Second, a strong linear relationship exists between activities of the contributors who
have verification authority and their network positions (seeTable 9). The main contribution of this work can be summarized as follows:†

Graduate School of Information Science, NAIST
Graduate School of Information Science, Nagoya University†††
Graduate School of Information Science and Technology, Osaka Universitya) E-mail: kin-y@is.naist.jpb) E-mail: yoshida@ertl.jpc) E-mail: raula-k@ist.osaka-u.ac.jpd) E-mail: iida@itc.naist.jp††• We established a novel approach to extract data for social network analysis from our raw dataset that minedfrom code review repository.
• This study investigated the importance of OSS peer review contributor roles and their review activities by using social network analysis.
• We could evaluate the performance of review contribu-

Copyright c 200x The Institute of Electronics, Information and Communication Engineers

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x2

Role
Contributor
AuthorReviewer
Committer
Approver
Verifier

Table 1 Contributor Roles in the Peer Review Process .
Definition
A contributor represents a participant who takes part in the code review process.
An author represents the contributor who submits a patchset and the owner of the review report belonged to this patchset.
A reviewer represents the contributor who reviews the patchset to find defect and bug inside.
A committer represents the contributor who has the authority to commit the patchset into the code repository.
A approver is an experienced reviewer who reviews and approves the patchsets by checking whether the patchset changes follows
the best practices that have been established by the project fits the project’s stated purpose and the existing architecture.
A verifier handles building, testing and verifying the patchset and decides whether it is suitable for merging into
the source code. In many OSS projects, verifiers can be automated tools.

Fig. 1The Review Process of Change #17768 in AOSPThe article is organized as follows. Section 2 describes
code review in software engineering as our research background. In Section 3, we present PeRSoN as our approachthat studies code review process from the social aspect. We
also detail the steps of the approach and the metrics we used.
Section 4 describes our experiment design, research questions, data processing, and results. Section 5 discuss theimplications and validity based on our findings. In Section
6, we lay out those important related work and the relationship with this study. Section 7 describes the conclusion ofthis paper and the future plan of our study.

of face-to-face inspection meeting (e.g., Google uses Gerrit† in AOSP, and Microsoft uses CodeFlow as their review
tools†† ). Many large-scale OSS projects have adopted webbased peer review. They also have claimed that peer review
is an important quality assurance technique in their projects[10] [11] [12]. During the early time of OSS projects, most
OSS projects assign review task and retrieve feedback using
mailing list [13]. Currently, OSS projects adopt light-weightpeer review tools instead of using mailing-list. Applying
peer review tools, the status of code review can be tracked
easily and review contributors can participate in peer review
freely. The mechanism being used in OSS projects for peer
review provides benefits to communities like sharing knowledge and experience among contributors.

2. Background

3. Peer Review Social Network (PeRSoN)

In the past decade, OSS as a dynamic software development
manner has been adopted by many software organizations.
Unlike traditional industry projects, OSS code is accessible
for patch contributions. With the rise of OSS projects, regular code inspection also has been modified to cater for OSS
projects development. It is not feasible for developers to always have communication or collaboration directly. As aresult, it is hard to search appropriate reviewers to performthe high-quality review because of lacking the informationof contributors. Based on these difficulties, OSS peer review applies a broadcasting method to announce and search
the appropriate reviewer for particular source code [9].
To facilitate these OSS peer review style, many software projects even industrial projects have adopted peer review tools. These companies perform code review instead

3.1 Process and Networks

tors from their personal activeness, behaviors, and social relationship using our approach.

Peer Review Process.
The peer review process represents the process to perform code review related activities by code authors and reviewers to guarantee the reliable software. In our study,
contribution activity refers to the activities carried out by
contributors, which means both authors of code change andreviewers. The primary contribution activities include submission of new patchset, revision of patchset, code review,
review approval, review verification, review discussion andothers. Based on the activities records in code review sys†††

https://code.google.com/p/gerrit/
http://goo.gl/5zk0wF

YANG et al.: PEER REVIEW SOCIAL NETWORK (PERSON) IN OPEN SOURCE PROJECTS

3
Table 2Centrality Measures and Social Implication.

Centrality Measures
DegreeBetweenness
Closeness

Social Implication
Activity
Control
Independencetem, we calculate the number of activities then separate contributors into different roles by their activities. Also, we introduce the definitions of contributor roles that will be usedthroughout this paper in Table 1.
Figure 1 is a practical example of review process. This
figure represents the review process of Change #17768 of
AOSP† . The first step of review is commit phase that author
submit new patchset and the system will notify reviewers byemail. Then in review phase, reviewers will perform reviews
based on this patchset (Change #17768). Every contributor
can perform review in a project, but only those reviewers
who have the authority of Approval or Verification(Always
chosen from core members, or experienced and active reviewers) can determine whether this change can be merged(Some projects also use bots to build and test the patchset as
Verifiers). In this case, three different reviewers have performed reviews. Mark Gross has reviewed but still needsomeone with approval authority to approve it. As a result,David Turner has approved this patchset, and JBQ (JeanBaptiste Queru) has verified it. The final step is integration
phase, system or particular core members will merge the approved and verified patchset into project code repository. In
this example, JBQ has merged this patchset to the repository.
PeRSoN Definition.
Our approach uses social network described as a graphnetwork called PeRSoN. PeRSoN is a social network constructed by peer review dataset, which a vertex represents a
review contributor and an edge represents a review activity
happened between two review contributors (e.g., in Figure 1,
Mark, David, and JBQ performed code review for Bruce’s
patchset, and then they left comments as feedback to everyone who have contributed in these reviews). In this study,
we define the network model as a undirected and weighted
network.
We assume that a contributor ci has a set of reviews Rci .
A review r in a patch set has a set of contributors including
both authors and reviewers {c1 , c2 , ... , cn }. A PeRSoN
edge e is formed when two contributors (e.g., ci and c j ) aremembers of the same review. Formally, e(ci , c j ) exists if
ci ∈ Rci and c j ∈ Rc j where Rci ∩ Rc j , ∅.
Network Measures
Our model can be used to describe more complex distribution characteristics of the contributors. In our approach,
we evaluate the reviewers network by using the three standard centrality measures of Degree, Betweenness and Closeness based on the definitions from Freeman [14] as below:• Degree Centrality.
Degree Centrality indicates the number of edges that a†

https://goo.gl/8uFZiv

Fig. 2

An Example of Contributors Evaluation Using PeRSoN

vertex has, A vertex (contributor) is defined as ck anda(ci , ck ) = 1 if ci and ck are connected, otherwise 0.
Degree Centrality of ck is defined as C D (ck ):C D (ck ) =nXa(ci , ck )i=1

• Betweenness Centrality.
Betweenness Centrality of a given vertex indicates thenumber of shortest paths from all vertices to all othervertices that pass through this vertex. We define gi j
= the number of edges from vertex ci to vertex c j , andgi j (ck ) = the number of edges from vertex ci to vertex c jthat passing through ck . Then calculate the probability
gi j (ck ). Betweenness Centrality of ck is definedbi j =
gi j
as C B (ck ):C B (ck ) =XnX

bi j (ck )i< j• Closeness Centrality.
Closeness Centrality of one vertex indicates the inverseof its farness. Farness indicates the sum of the distance between this vertex to all other vertices. We define d(ci , ck ) = the distance (number of edges) linking
ci and ck . Then Closeness Centrality of ck is defined as
CC (ck )−1 :CC (ck )−1 =nXd(ci , ck )i=1

Freeman also suggested that each centrality measures
have social implications as shown in Table 2 [14]. First,
Degree centrality implies activity degree, a vertex with ahigh degree in the network suggests this person should be
active and enthusiastic. Second, betweenness centrality implies control. A vertex with high betweenness centrality acts
as a bridge among other vertices in social networks. Third,
closeness centrality implies independence of a vertex. A
vertex with low closeness represents that this people is independent and far away from all other people.
Figure 2 is an example of how we evaluate the performance of contributors from their human factor and social
aspect using our approach. Here we simplified the networks
by ignoring the weight of edges. However, in the practical

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

4
Table 3Field
reviewId
authorId
reviewerId
lastUpdate
writtenOn
message

Field and Definition in Comment History.
Definition
ID of the review report.
The author who created the patch.
The contributors who reviewed this patch.
The timestamp of last update in this review.
the time when this review comment is created.
the content of this review comment.

experiment, we calculate the weight of edges. The vertices
represent contributors and edges represent the review activities between contributors. We perform social network analysis for this network by calculating the centrality measuresfor each contributor. After we calculated the centrality distribution, some observation can be summarized as follows:c5 has the highest degree in this network, which indicatedhe/she is the most active contributors; c1 has the highest betweenness and it represents that he/she work as a bridge inthe community, and he/she could be a potential bottleneck,which may cause problems to the process; c0 has the lowest closeness which means he/she contribute as individual or
rarely collaborate with others.
Due to the importance of social aspect study on peer
review, we introduce the following research questions that
we mentioned in Section 1.
RQ1 Which contributor role is the most important in thepeer review community?
RQ2 What is the relationship between contributors’ activities and their network position?
As we introduced in Section 3.1, social network analysis
provides the data constructed from the social relationshipbetween people. We want to investigate who are the mostimportant contributors and is there a relationship between
review authority and social network position in peer review
community (RQ1). Moreover, whether realistic activities
of contributors correlate with their social network position(RQ2). To address these research questions, we introduce
the details of our approach in next subsection.
3.2 Approach
We now introduce the main steps to our approach as threesteps; 1.) Dataset mining, 2.) Network generation and roleclassification and 3.) Metrics analysis.
Dataset Mining. We used a dataset from our previous
work [15]†.
To extract raw peer review dataset we apply Gerrit official API to obtain raw dataset. Gerrit code review systemprovides an REST-like API for users †† . Users can accessand gain the raw dataset through HTTP for their use. The
raw dataset is stored in the form of JSON files. For each review in peer review system, the JSON dataset has a unique†
The dataset is available to download at http://sdlab.
naist.jp/reviewmining/††https://goo.gl/PsKwFx

Fig. 3An Illustrative Example of Network Generation

ChangeID. The JSON dataset includes the following features: The review information with reviewers’ comments,
the updates history of patches, and the details of when andhow they have been merged into the source code or be abandoned. We created scripts using Python to download the raw
dataset by using official API. The primary functions of thescripts are extracting useful data out from raw dataset andstore this useful refined dataset into a database. In detail,we extracted the data from the comments history of each review reports in raw dataset. Some important data we used
is described as Table 3. In next paragraph, we will introduce how we use these data to create the social network andinvestigate the potential relationship between contributors.
Network Generation and Role Classification. We regard the review as a communication channel between codesubmitters and reviewers. Every review is a time of discussion or feedback between patch authors and reviewers.
Using the history dataset of each review, we extract the connection of all the contributors in peer review communities.
The steps how we extract the review connection is introduced as follows: First, it was assumed that all the review
related activities of the contributors in a review process arerecorded in the comments on the review reports, such as an
author has submitted a new patch or another reviewer has
reviewed a patch and given his/her comments based on thepatch change. This step is shown in Figure 3(a).
Second, to form the network, all contributors participating in the same review were connected, which indicatesthe contributors who work in the same review have used thesame communication channel as a team work, as shown in
Figure 3(b).
Finally, we perform role classification by contributors’
activities from their comments (e.g., An approval comment
from a contributor can be regarded that this contributor has
the authority to approve a patch). We separate contributors by their different activities and authorities. More details
about classification will be introduced in Section 4.3.
Metrics Analysis. The analysis of this study includes analysis of the network metrics. Specifically, we use
the standard centrality measures: degree, betweenness andcloseness to obtain contributors’ network position. Then weYANG et al.: PEER REVIEW SOCIAL NETWORK (PERSON) IN OPEN SOURCE PROJECTS

5
Table 4

Basic Information of AOSP, OpenStack and Qt.

Period# of Comments# of Contributors# of Reviewers# of Approvers# of Verifiers# of Vertices# of Edges

AOSP
2009/01∼
2011/06
42449
1086
451
99
111
808
15429

OpenStack2011/07∼2012/06
64793
426
165
86
29
379
55301

Qt
2011/07∼2012/06
219044
620
207
201
117
558
150017compare the centralities between different contributor rolegroups to investigate which group is most important. We
then use statistical analysis to find correlations between thecontributors centrality measures and their activities.
4. Empirical Case Study
We evaluate our approach by applying PeRSoN network to
three real world OSS projects that using Gerrit as code review system. These three projects require strict code review
mechanism, which means every code change must be reviewed first, then be commit to project code repository. Another reason for choosing these projects is that they adopt
code review system instead of a mailing list, which bringmore convenience when collecting the data.
4.1 Experiment Setup
At first, we studied the peer review process using Gerrit environment in three projects. We found that Gerrit systemmanages contributors by providing different authorities. For
example, normal contributors (not core members) can review code, but they have no permission to make the final
decision of change. While core members can both review
and judge a change. Because high-authority contributors
have more responsibility to the quality of change, we first
hypothesize the high-authority contributors are more important to the review system as follow.
H1 The contributors who have the highest authorities
in code review play the most important role in code review
community.
We address RQ1 by accepting H1, and we introduce
how we compare the difference between high review authorities contributors and other people in statistical analysis
in Section 4.4. Complementary to H1, we add a H0 as anull hypothesis that indicates the contributors who have thehighest review authorities and other contributors are fromthe same distribution.
We address RQ2 by calculating the correlation of contributors’ activities and their network positions for different
contributor groups, which separated by the roles. We use
degree, closeness and betweenness to measure contributors’
network positions, and then compare their activities.

4.2 PeRSoN Generation
Datasets. AOSP (Android Open Source Projects) is an industrial open source project that developing software products for mobile device† . OpenStack project provides cloudservices platform †† . Only a few of verifiers are human, but
the primary verification works are done by Continuous Integration (CI) tool. Qt Project is an application frameworkthat mainly use to develop graphical user interface ††† . Qt
uses the term Sanity Review instead of Verify in its Gerrit
system.
Several necessary data in change information table isneeded to generate PeRSoN. e.g., Change-Id represents thereview report identification, a unique Hash code generated
by Git. Uploaded and Updated represents the timestamps
when this report created and when is the latest update.
When a patchset is under review, the author who has
submitted this patchset could still update and fix this patchset. A review report could include more than one patchsets
if the author upload revisions. In Gerrit, contributors cancheck the status for current patchset such as who has reviewed or who will verify it. As a result, the comments
record all the reviewers who take part in the review are important. In this study, PeRSoN was generated by R †††† andigraph package. The statistical analysis of this study was
performed by R.
Generated Networks. Applied the network generating method mentioned in 3.2 and in Figure 3, the social networks have been generated from these projects separately.
The basic information of these social networks is shownas Table 4. In this study, the dataset of AOSP covers 2.5
years but OpenStack and Qt have only one year. Because
projects have different periods, the review dataset was separated into smaller samples. We split the dataset by one
month, three months and six months to observe how the networks of AOSP evolved through time.
In experiment period, we found that results of every
six months are most obvious and evident, and then we decided to divide dataset by every six months. We also generate PeRSoN by the whole dataset, and the number of vertices and edges are shown as Table 4. We include all theparticipants in our networks as vertices, and all the review
comments between contributors as edges. We can found thateach project has a different size in networks.
From the information in Table 4, it is easy to findout the common reviewers group is larger than the verifiersgroup. This observation complies with the Onion Model of
OSS development [16] that considers the core members as
the most important and the smallest group. It can be supposed that the smallest group of peer review, the verifiers
should also play the most important roles in code review†

http://source.android.com/
http://www.openstack.org/†††
http://qt-project.org/††††
http://www.r-project.org/††

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

6
Table 5 Summary of Distributions for VAC, VC, AC, and C contributors
in AOSP
VAC
VC
AC
C
Min
5
4
1
1
1st Qu.
46
15
73
DegreeMedian
108
29
24
6
Mean
251
56
25
13
3rd Qu.
293
49
31
12
Max
3349
439
79
779
Min
0.00000.00000.00000.0000
1st Qu.
0.00020.00000.00000.0000Median0.00360.00000.00000.0000Betweenness
Mean
0.01760.00170.00060.0002
3rd Qu. 0.0111
0.00240.00010.0000Max
0.65090.01550.00650.0440Min
0.34170.28370.31940.0024
1st Qu.
0.42740.34380.40890.3328Closeness
Median0.44740.40990.41990.4033Mean
0.44550.38840.40940.3708
3rd Qu. 0.4683
0.42630.4226
0.4121Max
0.67930.47380.44730.5176

Table 6 Summary of Distributions for VAC, AC, and C contributors in
OpenStackprocess. To observe results easily, we separated contributors into different roles from the activities of their review
comments that can be regarded as the history of their activities.

Table 7in Qt

4.3 Contributor Roles Classifications
We classified the roles of the contributors by the activities
they performed during the code review. We extracted contributors’ activities by mining their review comments. Administrators or team leaders in a project can set the review
rules, such as evaluating code changes by a different scoreor approving code change with certain review authorities.
Each project may have their rules and own authorities assignment. In this study, we investigated the review rules ofthree projects to extract review activities from the comment
history. Moreover, we classify contributors roles more detailto gain a better understanding of the relationship between
contributor groups.
As a result, a new classification method using activities
are proposed and applied. Based on our previous work related to contributor classifications [7] [17] [18], we apply aclassification method based on the activity types. Each contributor is labeled based on their roles, such as V as Verification, A as Approval, C as Code-Review. We investigate contributors from all seven combinations of V, A, C, VA, VC,
AC, and VAC. For example, If one contributor has activities
records of review and verification, he will be noted as VC.
A contributor only contributed by approval without other activities, he will be noted as A. If a contributor has Verification activities also did code review and approved patchsets,
who takes part in everything can be noted as VAC.
4.4 Results
RQ1: Most Important Contributors. We address RQ1
by comparing the distributions of contributors’ network positions. In peer review process, verification should be theDegreeBetweenness

ClosenessMin
1st Qu.
MedianMean
3rd Qu.
Max
Min
1st Qu.
MedianMean
3rd Qu.
Max
Min
1st Qu.
MedianMean
3rd Qu.
Max

VAC
263
995
1497
2665
2604
20050
0.00000.00060.00330.03370.01280.62900.51080.56020.57530.5966
0.60820.8873

AC
9
106
277
429
650
2273
0.00000.00000.00020.00110.00120.01390.47250.50990.52320.5306
0.54980.6087

C
1
11
29
69
78
1016
0.00000.00000.00000.00000.00000.00600.35100.48090.49930.49660.51010.5745

Summary of Distributions for VAC, VC, A, and C contributors

DegreeBetweenness

ClosenessMin
1st Qu.
MedianMean
3rd Qu.
Max
Min
1st Qu.
MedianMean
3rd Qu.
Max
Min
1st Qu.
MedianMean
3rd Qu.
Max

VAC
8
363
776
1204
1420
12010
0.00000.00000.00000.00020.00000.00530.50140.51190.52130.52830.53690.6545

VC
7
48
124
314
346
4194
0.00000.00000.00000.00000.00000.00060.50180.50530.50820.51230.5134
0.5919

AC
2
6
36
45
82
124
0.00000.00000.00000.00000.00000.00000.50040.50180.5036
0.50390.50590.5073

C
2
6
20
70
68
764
0.00000.00000.00000.00000.00000.00010.50090.50230.5036
0.50610.50710.5351highest authority in the peer review as it was the last stageof change before its merge or abandoned. Moreover, from
Freeman’s study, we know in a network structure, the centrality measures imply the importance of each node. As aresult, the most important contributors indicate they havethe highest centralities. In this study, three standard centrality measures have been used, and all of them have different social implications. From the observations of our previous work based on AOSP, which we separated contributor
roles into Verifier and Non-Verifier, we created cumulativegraphs of contributors’ frequency, and we observed: Verifiers’ degree and betweenness increased over time while
non-verifiers did not change too much. Verifiers have a relatively greater degree and betweenness than non-verifiers.
However, we did not find any relationship between Verifiers
and non-verifiers in terms of closeness.
Also, we separate contributors by more detail way as
we mentioned in Section 4.1. The classification results

YANG et al.: PEER REVIEW SOCIAL NETWORK (PERSON) IN OPEN SOURCE PROJECTS

7
Table 8 Comparison of Three Centrality Measures Distributions For Different Role Groups in AOSP,
OpenStack and Qt. V Represents Verification, A Represents Approval, C Represents Code-Review, And
The Following Number Indicate The Amount Of Reviewers In This Group (e.g., VC (26) represents that
26 contributors have performed Verification and also Code-Review in the current project). Each Row
Presents The Hypothesis Being Tested (e.g., VAC ∼ VC), The One-Side Alternative Hypothesis (i.e., >)And The p-Value (With (*) Indicate That The Alternative Hypothesis Is Accepted.)

Projects
AOSP
OpenStack

Qt

Comparison
VAC (80) ∼ VC (26)
VAC (80)∼ AC (16)
VAC (80)∼ C (451)
VAC (26) ∼ AC (69)
VAC (26) ∼ C (165)
VAC (116) ∼ AC (72)
VAC (116)∼ A (13)VAC (116)∼ C (207)

Degree>, p = 5.091e - 06, (*)>, p = 9.292e - 07, (*)>, p < 2.2e - 16, (*)>, p = 9.292e - 07, (*)>, p < 2.2e - 16, (*)>, p = 1.447e - 14, (*)>, p = 1.939e - 08, (*)>, p < 2.2e - 16, (*)

like follows: AOSP has four different contributor roles as
VAC, VC, AC, and C. From the observations we found:
VAC group has greater median value than other contributors’ groups, and the interquartile range of VAC group is
wider than other groups in terms of degree and betweenness but not in closeness (see Table 5). OpenStack has threegroups: VAC, AC, and C. The results show that the VAC
group has greater median than other groups, and same as
AOSP, the interquartile range of VAC group is wider thanother groups in degree and betweenness but not in closeness (see Table 6). Qt has four groups: VAC, AC, A and C.
The observation shows, Qt’s VAC group has greater medianvalue than other groups in degree and closeness but not obviously in betweenness, and the interquartile range of VAC
group is wider than any other group (see Table 7).
To compare the different centrality distributions among
VAC and other roles, we applied a Wilcoxon-Mann-Whitney
test to evaluate H1 [19]. We adopt Wilcoxon-MannWhitney because that we found the population has only one
variable (each centrality) but with more than two levels (different roles), and we assume that each role group have independent centrality distribution and not affect each other.
All three project have tested by comparing the VAC rolewith other groups existing in each project. VAC role with
VC, AC and C has proved in AOSP. The null hypothesis H0
that related to H1 is that VAC and other roles come fromthe same distribution. H0 can be rejected, and H1 can be
accepted if VAC > VC, VAC > AC and VAC > C, the pvalue of all comparison is below the significant threshold of
0.05. The results of p-value are given in Table 8 are all below 0.05. The null hypothesis is rejected, and the one-sided
alternative hypothesis is accepted, the true location shift isgreater than 0. For AOSP, OpenStack and Qt, The most active Verifier (VAC) have significantly higher centrality thanother contributors.
As mentioned above, RQ1 can be addressed that themost active Verifier (VAC) are the most important (central)role in the review process.
RQ2: Activities and Network Position. We address
RQ2 by calculating the correlation of contributors’ activities
and their network positions. We use Spearman because we

Closeness>, p = 8.751e - 07, (*)>, p = 2.897e - 05, (*)>, p < 2.2e - 16, (*)>, p = 2.897e - 05, (*)>, p < 2.2e - 16, (*)>, p = 1.536e - 11, (*)>, p = 2.356e - 08, (*)>, p < 2.2e - 16, (*)

Table 9sure.

Betweenness>, p = 2.416e - 05, (*)>, p = 2.059e - 05, (*)>, p < 2.2e - 16, (*)>, p = 2.059e - 05, (*)>, p < 2.2e - 16, (*)>, p = 1.096e - 09, (*)>, p = 2.645e - 07, (*)>, p < 2.2e - 16, (*)

Correlation (Spearman) of VAC Activity And Centrality MeaProjects
AOSP
OpenStack
Qt

Degree
0.952
0.964
0.953

Betweenness
0.789
0.992
0.884

Closeness
0.485
0.868
0.795

take the measurements from ordinal scales, while Pearson
correlation is more appropriate for the measurement takenfrom interval scale. The correlations between the activities of different contributors and their centrality measures inthree projects have been calculated. We calculated the correlation for each role group, but we only found a strong relationship in VAC contributors. The results in Table 9 showsthat in OpenStack and Qt, activities of VAC have a strong
linear relationship to all centralities. In AOSP, activities of
VAC have a strong linear relationship between degree andbetweenness, but not in closeness.
As a result, we addressed RQ2 by analyzing contributors’ activities and their centrality measures that indicate
their network position. We found that most active verifiers(VAC), the relationship between their activities and their
network position had a strong positive correlation. Except
closeness centrality is unusual in AOSP, all the results showthat a strong positive correlation exists between with centralities and contributors’ activities. However, we did not founda strong relationship for other contributors’ role groups.
5. Discussion
5.1

Implications

The results of the case study show that the network metrics(centrality) can be used to provide useful information about
users roles based on their review activities. In RQ1, we findthat verifiers and not the approvers are the most importantroles of a review process. In RQ2, we find that again contributors that are verifiers (VAC) have a significant correlation with all other network measures.
To further understand the reviewer roles, additionallystudied the evolution of role types and network positions

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x8

Fig. 4

Correlations for Three Centrality Measures Evolution.

over time. As depicted in Figure 4, the relationship between
the three centrality measures is compared, to estimate correlation coefficients over time. This figure shows that strong
positive correlation exists between verifiers’ betweenness
and closeness, and also between their degree and betweenness. We used this results to categorizing several exception
cases of contributors’ centralities. Based on the positive correlation between verifiers’ betweenness and closeness, it is
impossible to find a verifier has high betweenness, but low
closeness or a verifier has low betweenness but high closeness. Exception cases have been analyzed in which:• A verifier had high-degree and low-closeness. A verifier has high-degree means he or she performed more
activities than other people, and the low-closenessmeans he or she is far away from the network center.
• A verifier had low-degree and high-closeness. A verifier has low-degree means he or she performed few activities, and the high-closeness means he or she is close
to the network center.
We manually checked dataset to find special cases we
described above. We found a core member from AOSP,
who has a high degree and a low closeness (comparing
with median value). This contributor has contributed a lot
in Approvals and Verifications but participated only in kernel/common project. While we found another maintainer in
AOSP who has low degree but high closeness, we investigated that this contributor contributed very limited times
but participated in many projects (e.g., platform/build, platform/external/qemu, platform/external/clearsilver, and platform/system/core).
In addition, the correlation results (see Figure 4) andthe exception cases to create Table 10 were combined.
From which the following observations can be made: First,
verifiers with high-closeness (close to the network center)may have high-betweenness (more control). Second, verifiers with high-degree (more contribution) may have highbetweenness (more control). However, results show that no
significant correlation exists between verifiers’ degree (con-Table 10

Exception Cases For Verifiers.

Low
DegreeLow
Closeness

Low
BetweennessHigh
Degree—Active/ Far away—High
Closeness

Inactive/ Central

—

—

High
Betweenness

—

—

—

tribution activity) and closeness (network position). Finally,
from these findings, several suggestions for the exception
cases can be provided:• Verifiers with high-degree and low-closeness may be
experts in specialized fields because they perform more
activity and have few connections with other members
or work teams. Therefore, if they result to be specialists, the important review requests can be suggested tosend to them.
• Verifiers with low-degree and high-closeness may contribute less; however, being close to the network center,they may represent key figures tied to many other people. So, verifiers as high authorities can be suggested
not to remove them or change their roles thoughtlessly.
Based on the analysis of detecting particular cases, certain contributors can be found in their different behaviors.
5.2 Threats to Validity
This study researched three large-scale OSS projects andproved that the verifiers in peer review process are the mostimportant contributors. We discuss the limitations of thisstudy as the following:• Dataset Period. Modern peer review process usingcode review tools is a new technique for OSS projects,

YANG et al.: PEER REVIEW SOCIAL NETWORK (PERSON) IN OPEN SOURCE PROJECTS

9

different from many bug tracking systems that havebeen used for more than ten years. In this study, review
dataset was extracted from three projects, the longest
period (AOSP) are still less than three years. Because
AOSP server has shut down for six months, the data
structures have been changed after that. As a result,
our study based on the dataset has relatively short term
period.
• Community Size. The community size can be regarded as an important factor in the project’s development. Centrality measures are dependent on network
size, which is presumably changing over time. The
three projects have the different size that may affect theanalysis of results. This approach needs to be evaluatedby more projects with the different size.
• Measures of SNA. Due to our limited knowledge, inthis study centrality measures were introduced into thestudy. More measures need to be introduced such ascomponent, k-core, in-degree/out-degree, modularity.
Also, the relationship between actors and time phases
can be improved to analyze the network evolution in
many different situations.
6. Related Work
Prior work related to this study could be divided into two
aspects: Studies on OSS community and OSS peer review;
Studies on the social aspect of software engineering. We
provide these related work as following two parts.
Raymond referred to the different structures and processes of industry software and OSS as Cathedral andBazaar [20]. Rigby et al. examined Apache Server Project
for two techniques and created several metrics similar to traditional inspection experiments to find an efficient and effective OSS review technique [13]. Rigby et al. also havestudied the broadcast nature of OSS peer review, which is
totally different with traditional method [9]. Balachandransuggested use review-bot to reduce human effort and improve review quality [21].
Bird et al. extracted and studied the potential structure
for latent sub-communities in OSS projects using SNA [22].
Zanetti et al. adopted SNA to predict bugs into valid and invalid as the bug triage approach of OSS projects [23]. Kwak
et al. have studied social networks based on social networking media such as Twitter [24].
The main difference between our study and related
works above is we study code review from social aspect
while traditional study are mainly from the technical perspective only. Moreover, based on the prior studies thatstudied the human and social aspect of software engineering, we found the value and importance to perform study
from human and social perspective.
7. Conclusion and Future work
The motivation of using the approach of Social Network

Analysis to research OSS peer review process comes fromthe distributed construction of OSS community and humanfactors in software development. OSS projects, especiallyindustry-leaded OSS projects need developers to contributeenthusiastically. As proposed human factor should affect
OSS review process, SNA approach was applied into thiscase study has researched three OSS projects. Then reviewers can be classified into several role groups with significant
differences. The results show there is a strong correlation
between the activities of most important contributors andtheir network positions. Network measures distributions ofcontributors can be used for evaluating contributors’ activeness. For example, project managements can identify contributors who are enthusiastic but in a specialized field, andcontributors who are in important network position but unenthusiastic.
In our future plan, we plan to apply our approach
to more software projects. Since different projects havedifferent review processes and different review techniques,
we will apply this social network based approach to otherprojects to prove the usefulness of our approach. We believe our approach will identify the main factors influencing
the review communities, and what are the main differences
between the review techniques and processes.
• We plan to establish a set of metrics that could measure the activeness and social relationship of each review contributor in a review community, from both thesocial aspect and technical aspect. We could investigate the experienced and active contributors and assign
them to the important task and higher authority in certain fields.
• In order to establish healthy communities with thehigh-productivity and good relationship, we plan toinvestigate more networks from different projects.
Comparing the difference between network between
projects will give hints about the different patterns ofreview networks. Based on these patterns, we can suggest what kind of network structures are more suitablefor certain cases.
• All of this analysis could help to detect the weakness or
potential risk in the development process. In the end,
the results could help project managements to monitor
the communication and collaboration networks amongdevelopers to control the quality of the software product from human and social aspects.
Acknowledgement
This work was supported by JSPS KAKENHI Grant Numbers 26730036.
References[1] A. Ackerman, L. Buchwald, and F. Lewski, “Software inspections:an effective verification process,” Software, IEEE, vol.6, pp.31–36,
1989.

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

10[2] A.F. Ackerman, P.J. Fowler, and R.G. Ebenau, “Software inspections and the industrial production of software,” Proc. of a
symposium on Software validation: inspection-testing-verificationalternatives, pp.13–40, 1984.
[3] M. Fagan, “Advances in software inspections,” Software Engineering, IEEE Transactions on, vol.SE-12, no.7, pp.744–751, 1986.
[4] M.E. Fagan, “Design and code inspections to reduce errors in program development,” IBM Systems Journal, vol.15, no.3, pp.182–
211, 1976.
[5] P.C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp.202–212, 2013.
[6] D.E. Damian and D. Zowghi, “An insight into the interplay between
culture, conflict and distance in globally distributed requirements negotiations,” System Sciences, 2003. Proceedings of the 36th Annual
Hawaii International Conference on, pp.10–pp, IEEE, 2003.
[7] X. Yang, R.G. Kula, C.C.A. Erika, N. Yoshida, K. Hamasaki, K. Fujiwara, and H. Iida, “Understanding oss peer review roles in peer review social network (person),” Proc. of APSEC 2012, pp.709–712,
2012.
[8] X. Yang, “Social network analysis in open source software peer review,” Proc. of FSE 2014, pp.820–822, 2014.
[9] P.C. Rigby and M.A. Storey, “Understanding broadcast based peer
review on open source software projects,” Proc. of ICSE 2011,
pp.541–550, 2011.
[10] A. Mockus, R.T. Fielding, and J. Herbsleb, “A case study of open
source software development: the apache server,” Proc. of ICSE
2000, pp.263–272, 2000.
[11] S. Breu, R. Premraj, J. Sillito, and T. Zimmermann, “Informationneeds in bug reports: improving cooperation between developers
and users,” Proceedings of the 2010 ACM conference on Computer
supported cooperative work, pp.301–310, 2010.
[12] G. Jeong, S. Kim, T. Zimmermann, and K. Yi, “Improving code
review by predicting reviewers and acceptance of patches,” Researchon Software Analysis for Error-free Computing Center Tech-Memo(ROSAEC MEMO 2009-006), 2009.
[13] P.C. Rigby, D.M. German, and M.A. Storey, “Open source software
peer review practices: a case study of the apache server,” Proc. of
ICSE 2008, pp.541–550, 2008.
[14] L.C. Freeman, “Centrality in social networks conceptual clarification,” Social Networks, vol.1, no.3, pp.215–239, 1978-1979.
[15] K. Hamasaki, R.G. Kula, N. Yoshida, A. Cruz, K. Fujiwara, andH. Iida, “Who does what during a code review? datasets of oss peer
review repositories,” Proc. of MSR 2013, pp.49–52, 2013.
[16] K. Crowston and J. Howison, “The social structure of free and open
source software development,” First Monday, vol.10, no.2, 2005.
[17] R.G. Kula, A.E.C. Cruz, N. Yoshida, K. Hamasaki, K. Fujiwara,
X. Yang, and H. Iida, “Using profiling metrics to categorise peer
review types in the android project,” Proc. of ISSRE 2012, pp.146–
151, 2012.
[18] P. Thongtanunam, X. Yang, N. Yoshida, R.G. Kula, A.E. Camargo Cruz, K. Fujiwara, and H. Iida, “Reda: A web-based visualization tool for analyzing modern code review dataset,” Proc. of
ICSME 2014, pp.605–608, 2014.
[19] D.A. Wolfe and M. Hollander, “Nonparametric statistical methods,”
Nonparametric statistical methods, 1973.
[20] E.S. Raymond, The Cathedral and the Bazaar, 1st ed., O’Reilly &Associates, Inc., Sebastopol, CA, USA, 1999.
[21] V. Balachandran, “Reducing human effort and improving quality inpeer code reviews using automatic static analysis and reviewer recommendation,” Proc. of ICSE 2013, pp.931–940, 2013.
[22] C. Bird, D. Pattison, R. D’Souza, V. Filkov, and P. Devanbu, “Latentsocial structure in open source projects,” Proc. of FSE 2008, pp.24–
35, 2008.
[23] M.S. Zanetti, I. Scholtes, C.J. Tessone, and F. Schweitzer, “Categorizing bugs with social networks: a case study on four open source
software communities,” Proc. of ICSE 2013, pp.1032–1041, 2013.

[24] H. Kwak, C. Lee, H. Park, and S. Moon, “What is twitter, a social
network or a news media?,” Proceedings of the 19th International
Conference on World Wide Web, pp.591–600, 2010.

Xin Yang
is a PhD student in the Graduate School of Information Science at Nara Institute of Science and Technology (NAIST). He
received his M.E. from NAIST in 2013. His research interests are in Empirical Software Engineering, Mining Software Repository, Software
Peer Review, and Social Network Analysis. He
is a student member of ACM.

Norihiro Yoshidareceived his BE fromthe Kyushu Institute of Technology in 2004 andhis Master and PhD from Osaka University in2006 and 2009, respectively. He is an associate
professor at Nagoya University. Before joining
Nagoya University in 2014, he was an assistantprofessor at the Nara Institute of Science andTechnology from 2010. His research interests
include program analysis and software development environment. He is a member of the IEEE,the IEEE Computer Society, and the ACM.

Raula Gaikovina Kulais a currently Research Assistant Professor at Software Engineering Lab, Osaka University. In 2013, he
graduated with a PhD from Nara Institute of Science and Technology. He is currently an active member of the IEEE Computer Society andACM. His research interests include repository
mining, peer review, API & software reuse andsoftware visualizations.

Hajimu Iida
received his B.E., M.E., andDr. of Eng. degrees from Osaka University in
1988, 1990, and 1993, respectively. From 1991
to 1995, he worked for the Department of Information and Computer Science, Faculty of Engineering Science, Osaka University as a research
associate. Since 1995 he has been with the
Graduate School of Information Science, Nara
Institute of Science and Technology, Japan. His
current position is a Professor of the Laboratoryof Software Design and Analysis. His researchinterests include modeling and analysis of software and development process.