A Large Scale Study of Programming Languages
and Code Quality in Github
Baishakhi Ray, Daryl Posnett, Vladimir Filkov, Premkumar Devanbu{bairay@, dpposnett@, filkov@cs., devanbu@cs.}ucdavis.edu
Department of Computer Science, University of California, Davis, CA, 95616, USA

ABSTRACT

1.

What is the effect of programming languages on software quality? This question has been a topic of much debate for a very longtime. In this study, we gather a very large data set from GitHub(729 projects, 80 Million SLOC, 29,000 authors, 1.5 million commits, in 17 languages) in an attempt to shed some empirical lighton this question. This reasonably large sample size allows us to use
a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static v.s. dynamic typing, strong v.s. weaktyping on software quality. By triangulating findings from different methods, and controlling for confounding effects such as teamsize, project size, and project history, we report that language design does have a significant, but modest effect on software quality.
Most notably, it does appear that strong typing is modestly better
than weak typing, and among functional languages, static typing is
also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It
is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as
project size, team size, and commit size. However, we hasten to
caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preferenceof certain personality types for functional, static and strongly typed
languages.

A variety of debates ensue during discussions whether a given
programming language is “the right tool for the job". While someof these debates may appear to be tinged with an almost religious
fervor, most people would agree that a programming language can
impact not only the coding process, but also the properties of theresulting artifact.
Advocates of strong static typing argue that type inference willcatch software bugs early. Advocates of dynamic typing may argue
that rather than spend a lot of time correcting annoying static typeerrors arising from sound, conservative static type checking algorithms in compilers, it’s better to rely on strong dynamic typing tocatch errors as and when they arise. These debates, however, havelargely been of the armchair variety; usually the evidence offeredin support of one position or the other tends to be anecdotal.
Empirical evidence for the existence of associations between code
quality programming language choice, language properties, and usage domains, could help developers make more informed choices.
Given the number of other factors that influence software engineering outcomes, obtaining such evidence, however, is a challenging task. Considering software quality, for example, there are
a number of well-known influential factors, including source code
size [8], the number of developers [29, 3], and age/maturity [13].
These factors are known to have a strong influence on software
quality, and indeed, such process factors can effectively predict defect localities [25].
One approach to teasing out just the effect of language properties, even in the face of such daunting confounds, is to do acontrolled experiment. Some recent works have conducted experiments in controlled settings with tasks of limited scope, with students, using languages with static or dynamic typing (based on experimental treatment setting) [11, ?, 15]. While type of controlled
study is “El Camino Real" to solid empirical evidence,another opportunity has recently arisen, thanks to the large number of open
source projects collected in software forges such as GitHub.
GitHub contains many projects in multiple languages. These
projects vary a great deal across size, age, and number of developers. Each project repository provides a historical record from which
we extract project data including the contribution history, projectsize, authorship, and defect repair. We use this data to determine
the effects of language features on defect occurrence using a variety
of tools. Our approach is best described as mixed-methods, or triangulation [7] approach. A quantitative (multiple regression) study
is further examined using mixed methods: text analysis, clustering, and visualization. The observations from the mixed methods
largely confirm the findings of the quantitative study.

Categories and Subject Descriptors
D.3.3 [PROGRAMMING LANGUAGES]: [Language Constructs
and Features]

General Terms
Measurement, Experimentation, Languages

Keywords
programming language, type system, bug fix, code quality, empirical research, regression analysis, software domain

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
FSE’14 November 16âĂŞ22, 2014, Hong Kong, China
Copyright 2014 ACM 978-1-4503-3056-5/14/11 ...$15.00.

INTRODUCTION

In summary, the main features of our work are as follows.
• We leverage a categorization of some important features ofprogramming languages that prior knowledge suggests areimportant for software quality (strong v.s. weak typing, dynamic v.s.. static typing, memory managed v.s. unmanaged,
and scripting vs. compiled) to study their impact on defectquality.
• We use multiple regression to control for a range of different
factors (size, project history, number of contributors, etc.)
and study the impact of the above features on defect occurrence. The findings are listed under RQ1 and RQ2 in Section 3.
• We use text analysis and clustering methods to group projects
into domains of application, and also the defects into categories of defects; we then use heat maps to study relationships of project types and defect types to programming languages. The findings from this study (RQ3 and RQ4 in Section 3) are consistent with statistical results.
While the use of regression analysis to deal with confounding
variables is not without controversy, we submit that a couple offactors increase the credibility of our results: a fairly large sample
size, and use of mixed methods to qualitatively explore and largely
confirm the findings from the regression model.

2.

METHODOLOGY

Here, we describe the languages and GitHub projects that wecollected, and the analysis methods we used to answer our researchquestions.

2.1

Study Subjects

To understand whether choice of programming languages have
any impact on software quality, we choose the top 19 programming languages from GitHub. We disregard CSS, Shell script, andVim script as they are not considered to be general purpose languages. We further include TypeScript, a typed superset of
JavaScript. Then, for each of the studied languages we retrieve top 50 projects that are primarily written in that language.
Table 1 shows top three projects in each language, based on their
popularity. In total, we analyze 850 projects spanning 17 different
languages.

2.2

Data Collection

To retrieve the top programming languages and their corresponding projects from GitHub, we used GitHub Archive [?], a databasethat records all public GitHub activities. The archive logs eighteen
different GitHub events including new commits, fork events, pull
request, developers’ information, and issue tracking of all the open
source GitHub projects on an hourly basis. The archive data is uploaded to Google BigQuery [?] service to provide an interface forinteractive data analysis.
Identifying top languages. The top languages in GitHub aremeasured by first finding the number of open source GitHub projects
developed in each language, and then choosing the top languages
with the maximum number of projects. However, since multiple
languages are often used to develop a project, assigning a single
language to a project is difficult. GitHub Linguist [9] can measure such a language distribution of a GitHub project repository.
Since languages can be identified by the extension of a project’s
source files, GitHub Linguist counts the number of source files withdifferent extensions. The language with the maximum number ofTable 1: Top three projects in each language
Language

ProjectsC
C++C#
Objective-C
Go
Java
CoffeeScript
JavaScript
TypeScript
Ruby
Php
PythonPerl
ClojureErlangHaskellScala

linux, git, php-src
node-webkit, phantomjs, mongo
SignalR, SparkleShare, ServiceStack
AFNetworking, GPUImage, RestKit
docker, lime, websocketd
storm, elasticsearch, ActionBarSherlock
coffee-script, hubot, brunchbootstrap, jquery, node
bitcoin, litecoin, qBittorrent
rails, gitlabhq, homebrew
laravel, CodeIgniter, symfony
flask, django, reddit
gitolite, showdown, rails-dev-box
LightTable, leiningen, clojurescript
ChicagoBoss, cowboy, couchdb
pandoc, yesod, git-annex
Play20, spark, scalasource files is assigned as primary language of the project. GitHub
Archive stores this information. We aggregate projects based on
their primary language. Then we select the top languages having
maximum number of projects for further analysis as shown in Table 1.
Retrieving popular projects. For each selected language, we
retrieve the project repositories that are primarily written in that
language. We then count the number of stars associated with that
repository. The number of stars relate to how many people areinterested in that project [?]. Thus, we assume that stars indicate the popularity of a project. We select the top 50 projects ineach language. To ensure that these projects have a sufficient development history, we filter out the projects having less than 28
commits, where 28 is the first quartile commit number of all the
projects. This leaves us with 729 projects. Table 1 shows the top
three projects in each language. This includes projects like Linux,
mysql, android-sdk, facebook-sdk, mongodb, python, ruby sourcecode etc.
Retrieving project evolution history. For each of these 729
projects, we downloaded the non merged commits, along with thecommit logs, author date, and author name using the command:
git log -no-merges -numstat. The numstat flag shows
the number of added and deleted lines per file associated with eachcommit. This helps us to compute code churn and the number offiles modified per commit. We also retrieve the languages associated with each commit from the extensions of the modified files.
Note that, one commit can have multiple language tags. For eachcommit, we calculate its commit age by subtracting its commit datefrom the first commit of the corresponding project. We also calculate some other project related statistics, including maximum commit age of a project and the total number of developers; we usethem as control variables in our regression model as discussed in
Section 3 . We further identify the bug fix commits made to individual projects by searching for error related keywords: ‘error’,‘bug’, ‘fix’ , ‘issue’, ‘mistake’, ‘incorrect’, ‘fault’, ‘defect’ and‘flaw’ in the commit log using a heuristic similar to that in Mockus
and Votta [19].
Table 2 summarizes our data set. Since a project may use multiple languages, the second column of the table shows the total number of projects that use a certain language at some capacity. We
further exclude some languages from a project that have fewer than
20 commits in that language, where 20 is the first quartile value ofthe total number of commits per project per language. For example,
we find 220 projects that use more than 20 commits in C. This ensures that the studied languages have significant activity within the
projects. In summary, we study 729 projects developed in 17 lan-

Table 2: Study Subjects
Project Details
Language

Total Commits#Authors

SLOC(KLOC)

Period#Commits#Insertion(KLOC)#Bug Fixes#Insertion(KLOC)CC++C#
Objective-C
Go
Java
CoffeeScript
JavaScript
TypeScript
Ruby
Php
PythonPerl
ClojureErlangHaskell
Scala

220
149
77
93
54
141
92
432
96
188
109
286
106
60
51
55
55

13,769
3,831
2,275
1,643
659
3,340
1,691
6,754
789
9,574
4,862
5,042
758
843
847
925
1,260

22,418
12017
2,231
600
591
5,154
260
5,816
18,363
1,656
3,892
2,438
86
444
2484
837
1,370

1/1996 to 2/2014
8/2000 to 2/2014
6/2001 to 1/2014
7/2007 to 2/2014
12/2009 to 1/2014
11/1999 to 2/2014
12/2009 to 1/2014
2/2002 to 2/2014
3/2011 to 2/2014
1/1998 to 1/2014
12/1999 to 2/2014
8/1999 to 2/2014
1/1996 to 2/2014
9/2007 to 1/2014
05/2001 to 1/2014
01/1996 to 2/2014
04/2008 to 1/2014

447,821
196,534
135,776
21,645
19,72887,12022,500118,318
14,987122,023
118,664
114,200
5,483
28,35331,39846,08755,69675,30845,97027,7042,400
1,589
19,093
1,134
33,13465,9105,804
16,164
9,033
471
1,461
5,001
2,922
5,262

182,568
79,31250,6897,089
4,423
35,1286,312
39,2502,443
30,47847,19441,946
1,903
6,022
8,129
10,362
12,95020,121
23,995
8,793
723
269
7,363
269
8,676
8,970
1,649
5,139
2,984
190
163
1,970
508
836

Summary

729

28,94880,657

1/1996 to 2/2014

1,586,333

318,360

566,198

92,618guages with 18 years of parallel evolution history. This includes 29
thousand different developers, 1.58 million commits, and 566,000
bug fix commits.

2.3

Categorizing Languages

We define language classes based on several properties of thelanguage that have been thought to influence language quality [11,12, 15], as shown in Table 3 . The Programming Paradigm indicates whether the project is written in a procedural, functional, or
scripting language. Compile Class indicates whether the project isTable 3: Different Types of Language Classes
Language
ClassesCategories

Languages

Programming
Paradigm

Procedural

C, C++, C#, Objective-C,
Java, Go
CoffeeScript, JavaScript,
Python, Perl, Php, Ruby
Clojure, Erlang, Haskell,
Scala

Scripting
Functional
Compilation
Class

StaticDynamic

Type Class

Strong

Weak

Memory Class

BugFix Commits#Projects

ManagedUnmanaged

C, C++, C#, Objective-C,
Java, Go, Haskell, Scala
CoffeeScript, JavaScript,
Python, Perl, Php, Ruby,
Clojure, Erlang
C#, Java, Go, Python, Ruby,
Clojure, Erlang, Haskell,
Scala
C,
C++,
Objective-C,
CoffeeScript, JavaScript,
Perl, Php
OthersC, C++, Objective-C

statically or dynamically typed.
Type Class classifies languages based on strong and weak typing,based on whether the language admits type-confusion. We consider
that a program introduces type-confusion when it attempts to interpret a memory region populated by a datum of specific type T1, as
an instance of a different type T2 and T1 and T2 are not related byinheritance. We classify a language as strongly typed if it explicitly
detects type confusion and reports it as such. Strong typing could
happen by static type inference within a compiler (e.g., with Java),
using a type-inference algorithm such as Hendley-Milner [?, ?], orat run-time using a dynamic type checker. In contrast, a language

is weakly-typed if type-confusion can occur silently (undetected),and eventually cause errors that are difficult to localize. For example, in a weakly typed language like JavaScript adding a string
to a number is permissible (e.g., ‘5’ + 2 = ‘52’), while such an operation is not permitted in strongly typed Python. Also, C andC++ are considered weakly typed since, due to type-casting, one
can interpret a field of a structure that was an integer as a pointer.
Finally, Memory Class indicates whether the language requires
developers to manage memory. We treat Objective-C as unmanaged, though Objective-C follows a hybrid model, because
we observe many memory errors in Objective-C codebase, as
discussed in RQ4 in Section 3.

2.4

Identifying Project Domain

We classify the studied projects into different domains basedon their features and functionalities using a mix of automated andmanual techniques. The projects in GitHub come with projectdescriptions and Readme files that describe their features.
First, we used Latent Dirichlet Allocation(LDA) [4], a well-known
topic analysis algorithm, on the text describing project features.
Given a set of documents, LDA identifies a set of topics where each
topic is represented as probability of generating different words.
For each document, LDA also estimates the probability of assigning that document to each topic.
Table 4: Characteristics of Domains
DomainName

DomainCharacteristics

ExampleProjects

Total
Proj

Application(APP)

end user programs.

bitcoin, macvim

120

Database(DB)

sql and nosql
databases

mysql, mongodb

43

CodeAnalyzer(CA)

compiler, parser
interpreter etc.

ruby, php-src

88

Middleware(MW)

Operating Systems,
Virtual Machine, etc.

linux, memcached

48

Library(LIB)

APIs, libraries etc.

androidApis,
opencv

175

Framework(FW)

SDKs, plugins

ios sdk, coffeekup

206

Other
(OTH)

-

Arduino, autoenv

49

We detect distinct 30 domains (i.e. topics) and estimate theprobability of each project belonging to these domains. For example, LDA assigned the facebook-android-sdk project to the following topic with high probability: (0.042 ∗ f acebook + 0.010 ∗swank/slime+0.007∗f ramework+0.007∗environments.+
0.007 ∗ transf orming). Here, the text values are the topics andthe numbers are their probability of belonging to that domain. For
clarity, we only show the top 5 domains. Since such auto-detecteddomains include several project-specific keywords, such as facebook, swank/slime as shown in the previous example, it is hard to
identify the underlying common functionalities. Hence, we manually inspect each of the thirty domains to identify project-nameindependent, domain-identifying keywords. Manual inspection helps
us in assigning a meaningful name to each domain. For example,
for the domain described earlier, we identify the keywords framework, environments, and transforming to call it development framework. We manually rename all the thirty auto-detected domains insimilar manner and find that the majority of the projects fall undersix domains: Application, Database, CodeAnalyzer, Middleware,
Library, and Framework. We also find that some projects like “online books and tutorials”, “scripts to setup environment”, “hardware programs” etc. do not fall under any of the above domains
and so we assign them to a catchall domain labeled as Other. This
classification of projects into domains was subsequently checkedand confirmed by another member of our research group. Table 4summarizes the identified domains resulting from this process. In
our study set, the Framework domain has the greatest number ofprojects (206), while the Database domain has the fewest numberof projects (43).

2.5

Categorizing Bugs

While fixing software bugs, developers often leave important information in the commit logs about the nature of the bugs; e.g., why
the bugs arise, how to fix the bugs. We exploit such information tocategorize the bugs, similar to Tan et al. [16, 26]. First, we categorize the bugs based on their Cause and Impact. Root Causes arefurther classified into disjoint sub-categories of errors—Algorithmic,
Concurrency, Memory, generic Programming, and Unknown. The
bug Impact is also classified into four disjoint sub-categories: Security, Performance, Failure, and other unknown categories. Thus,
each bug fix commit has a Cause and a Impact type. For example, a Linux bug corresponding to the bug fix message: “return
if prcm_base is NULL.... This solves the following crash" 1 wascaused due to a missing check (programming error), and impact
was crash (failure). Table 5 shows the description of each bug category. This classification is performed in two phases:(1) Keyword search. We randomly choose 10% of the bug-fix
messages and use a keyword based search technique to automatically categorize the messages with potential bug types. We use thisannotation, separately, for both Cause and Impact types. We chosea restrictive set of keywords and phrases as shown in Table 5. For
example, if a bug fix log contains any of the keywords: deadlock,race condition or synchronization error, we infer it is related to theConcurrency error category. Such a restrictive set of keywords andphrases help to reduce false positives.
(2) Supervised classification. We use the annotated bug fix logs
from the previous step as training data for supervised learning techniques to classify the remainder of the bug fix messages by treatingthem as test data. We first convert each bug fix message to a bag-ofwords. We then remove words that appear only once among all of
the bug fix messages. This reduces project specific keywords. We
1https://lkml.org/lkml/2012/12/18/102

also stem the bag-of-words using standard natural language processing (NLP) techniques. Finally, we use a well-known supervised classifier: Support Vector Machine(SVM) [27] to classify thetest data.
To evaluate the accuracy of the bug classifier, we manually annotated 180 randomly chosen bug fixes, equally distributed acrossall of the categories. We then compare the result of the automaticclassifier with the manually annotated data set. The following tablesummarizes the result for each bug category.
precision

recallPerformance
Security
FailureMemoryProgramming
Concurrency
Algorithm

70.00%
75.00%
80.00%
86.00%
90.00%
100.00%85.00%

87.50%
83.33%
84.21%
85.71%
69.23%
90.91%
89.47%

Average

83.71%

84.34%The result of our bug classification is shown in Table 5. In the
Cause category, we find most of the bugs are related to generic programming errors (88.53%). Such high proportion is not surprisingbecause it involves a wide variety of programming errors including
incorrect error handling, type errors, typos, compilation errors, incorrect control-flow, and data initialization errors. The rest 5.44%
appears to be incorrect memory handling; 1.99% is concurrency
bugs, and 0.11% is algorithmic errors. Analyzing the impact of thebugs, we find 2.01% are related to security vulnerability; 1.55%
is performance errors, and 3.77% causes complete failure to thesystem. Our technique could not classify 1.04% of the bug fix messages in any Cause or Impact category; we classify these with the
Unknown type.

2.6

Statistical Methods

We use regression modeling to describe the relationship of a setof predictors against a response. In this paper, we model the number of defective commits against other factors related to software
projects. All regression models use negative binomial regression,
or NBR to model the non-negative counts of project attributes suchas the number of commits. NBR is a type of generalized linear
model used to model non-negative integer responses. It is appropriate here as NBR is able to handle over-dispersion, e.g., cases
where the response variance is greater than the mean [5].
In our models we control for several language per-project dependent factors that are likely to influence the outcome. Consequently, each (language, project) pair is a row in our regression andis viewed as a sample from the population of open source projects.
We log-transform dependent count variables as it stabilizes the variance and usually improves the model fit [5]. We verify this bycomparing transformed with non transformed data using the AIC
and Vuong’s test for non-nested models [28].
To check that excessive multi-collinearity is not an issue, we
compute the variance inflation factor (VIF) of each dependent variable in all of the models. Although there is no particular value of
VIF that is always considered excessive, we use the commonly usedconservative value of 5 [5]. We check for and remove high leverage points through visual examination of the residuals vs leverageplot for each model, looking for both separation and large values of
Cook’s distance.
We employ effects, or contrast, coding in our study to facilitate
interpretation of the language coefficients [5]. Effects codes differ from the more commonly used dummy, or treatment, codes thatcompare a base level of the factor with one or more treatments.
With effects coding, each coefficient indicates the relative effectTable 5: Categories of bugs and their distribution in the whole dataset

Cause

Bug Type

Bug Description

Search keywords/phrases

count%countAlgorithm (Algo)
Concurrancy (Conc)algorithm
deadlock, race condition, synchronization error.

606
11111

0.11
1.99

Memory (Mem)algorithmic or logical errors
multi-threading or multi-processing
related issues
incorrect memory handling

30437

5.44

Programming (Prog)

generic programming errors

memory leak, null pointer, buffer overflow, heap
overflow, null pointer, dangling pointer, double
free, segmentation fault.
exception handling, error handling, type error,
typo, compilation error, copy-paste error, refactoring, missing switch case, faulty initialization, default value.

495013

88.53

Security (Sec)correctly runs but can be exploitedby attackers
correctly runs with delayed response
crash or hangbuffer overflow, security, password, oauth, ssl

11235

2.01

optimization problem, performance

8651

1.55

reboot, crash, hang, restart

21079

3.77

5792

1.04

ImpactPerformance (Perf)
Failure (Fail)
Unknown (Unkn)

not part of the above seven categoriesof the use of a particular language on the response as comparedto the weighted mean of the dependent variable across all projects.
Since our factors are unbalanced, i.e., we have different numbers ofprojects in each language, we use weighted effects coding, whichtakes into account the scarcity of a language. This method has beenused previously in software engineering to study the impact of pattern roles on change proneness [24]. As with treatment coding, it
is still necessary to omit a factor to compute the model. Since theuse of effects codes compares each level to the grand mean, however, we can easily extract the missing coefficient from the othermodel coefficients, or more practically, we simply re-compute thecoefficients using a different level of the factor as a base [5].
To test for the relationship between two factor variables we use
a Chi-Square test of independence [17]. After confirming a dependence we use Cramer’s V, an r × c equivalent of the phi coefficientfor nominal data, to establish an effect size [6].

3.

RESULTS

Prior to analyzing language properties in more detail, we begin
with a straightforward question that directly addresses the core ofwhat some fervently believe must be true, namely:ance. The remaining coefficients are significant and either positiveor negative. For those with positive coefficients we can expect thatthe language is associated with, ceteris paribus, a greater numberof defect fixes. These languages include C, C++, JavaScript,
Objective-C, Php, and Python. The languages Clojure,
Haskell, Ruby, Scala, and TypeScript, all have negative
coefficients implying that these languages are less likely than the
average to result in defect fixing commits.

NULL
log commits
log agelog sizelog devs
language

Df

Deviance

1
1
1
1
16

36986.03
42.70
12.25
48.22
242.89Resid.
Df
1113
1112
1111
1110
1109
1093

Resid.
Dev
38526.51
1540.481497.781485.53
1437.30
1194.41

Pr(>Chi)0.00000.00000.00050.00000.0000One should take care not to overestimate the impact of language
on defects. While these relationships are statistically significant,the effects are quite small. In the analysis of deviance table above
we see that activity in a project accounts for the majority of explained deviance. Note that all variables are significant, that is, all
Table 6: Some languages induce fewer defects than other languages.

RQ1. Are some languages more defect prone than others?
We evaluate this question using an NBR model with languages
encoded with weighted effects codes as predictors for the numberof defect fixing commits. The model details are shown in Table 6.
We include some variables as controls for factors that will clearly
influence the number of defect fixes. Project age is included asolder projects will generally have a greater number of defect fixes.
Trivially, the number of commits to a project will also impact the
response. Additionally, the number of developers who touch a
project and the raw size of the project are both expected to growwith project activity.
The sign and magnitude of the Estimate in the above model relates the predictors to the outcome. The first four variables arecontrol variables and we are not interested in their impact on theoutcome other than to say, in this case, that they are all positive,
as expected, and significant. The language variables are indicator, or factor, variables for each project. The coefficient compares
each language to the grand weighted mean of all languages in all
projects. The language coefficients can be broadly grouped into
three general categories. The first category are those for which thecoefficient is statistically insignificant and the modeling procedure
could not distinguish the coefficient from zero. These languages
may behave similarly to the average or they may have wide vari-Response is the number of defective commits.Languages are coded withweighted effects coding so each language is compared to the grandmean. AIC=10673, BIC=10783, Log Likelihood = -5315, Deviance=1194,
Num. obs.=1114
Defective Commits Model
Coef.
Std. Err.
(Intercept)−1.93(0.10)∗∗∗
log commits
2.26(0.03)∗∗∗
log age
0.11(0.03)∗∗
log size
0.05(0.02)∗
log devs
0.16(0.03)∗∗∗
C
0.15(0.04)∗∗∗
C++
0.23(0.04)∗∗∗
C#
0.03(0.05)
Objective-C
0.18(0.05)∗∗∗
Go−0.08(0.06)
Java−0.01(0.04)
CoffeeScript
0.07(0.05)JavaScript
0.06(0.02)∗∗TypeScript−0.43(0.06)∗∗∗
Ruby
−0.15
(0.04)∗
Php
0.15(0.05)∗∗∗
Python0.10(0.03)∗∗
Perl−0.15(0.08)
Clojure−0.29(0.05)∗∗∗
Erlang−0.00(0.05)
Haskell−0.23(0.06)∗∗∗
Scala
−0.28
(0.05)∗∗∗∗∗∗
p < 0.001, ∗∗ p < 0.01, ∗ p < 0.05

of the factors above account for some of the variance in the numberof defective commits. The next closest predictor, which accounts
for less than one percent of the total deviance, is language. Allother controls taken together do not account for as much devianceas language.
Although we expressed the impact of language in this model asa percentage of deviance, we hasten to remind the reader that although interpretation is similar, in a rough sense, to a percentageof the total variance explained in an ordinary least squares regression, it is not accurate to say that the measures are synonymous [5].
About the best we can do is to observe that it is a small affect. We
can read the coefficients as the expected change in the log of the
response for a one unit change in the predictor with all other predictors held constant; i.e., for a coefficient βi , a one unit change
in βi yields an expected change in the response of eβi . For thefactor variables, this expected change is normally interpreted as
a comparison to a base factor. In our case, however, the use ofweighted effects coding allows us to compare this expected change
to the grand mean, i.e., the average across all languages. Thus, if,
for some number of commits, a particular project developed in an
average language had four defective commits, then the choice to
use C++ would mean that we should expect one additional buggy
commit since e0.23 × 4 = 5.03. For the same project, choosing Haskell would mean that we should expect about one fewer
defective commit as e−0.23 × 4 = 3.18. The accuracy of thisprediction is dependent on all other factors remaining the same, achallenging proposition for all but the most trivial of projects. All
observational studies face similar limitations and we address thisconcern in more detail in section 5.
Result 1: Some languages have a greater association withdefects than other languages, although the effect is small.
In the remainder of this paper we expand on this basic result
by considering how different categories of application, defect, andlanguage, lead to further insight into the relationship between languages and defect proneness.
Software bugs usually fall under two broad categories: (1) Domain Specific bug: specific to project functionalities and do notdepend on the underlying programming language. For example,
we find a bug fix in Linux with log message: “Fix headset mic support for Asus X101CH". The bug was due to a missing functionality [23] in Asus headset2 and less to do with language feature. Previous researches term these errors as Software Component bug [16,26]. (2) Generic bug: more generic in nature and have less to dowith project functionalities. For example, type-error, concurrency
error etc.
Consequently, it is reasonable to think that the interaction of application domain and language might impact the number of defects
within a project. Moreover some languages are believed to excel atsome tasks more so than others, e.g., C for low level work, or Javafor user applications. A less than ideal language for an application
might lead to a greater number of defects. To compare error proneness of different languages we ideally should ignore the domain
specific bugs and focus only on the generic ones since they are
more likely to depend on the programming language feature. However, since a domain specific bug may also arise due to a generic
programming error, it is difficult to separate the two. A possible
workaround is to study the defect proneness of languages while
controlling the domain. To this end, we would like to confirm spe2
https://bugs.launchpad.net/ubuntu/+source/
linux/+bug/1169138

cific interactions between language and domain with respect to defects within projects. Statistically, this goal poses challenges forregression. Interaction terms in a regression model might yieldsome insight, however, with 17 languages across 7 domains thiswill yield an excessive number of terms and a challenging, andmost likely spurious, interpretation.
Given this, we first consider testing for the dependence between
domain and language usage within a project, using a Chi-Square
test of independence. Unfortunately, out of 119 cells in our dataset, 46, i.e. 39%, are below the value of 5. This exceeds the recommendation that no more than 20% of the counts should be below
5 [17]. We include the value here for completeness3 , however, thelow strength of association of 0.191 as measured by Cramer’s V,
suggests that although there is likely some relationship between domain and language in our data set, including domain in regression
models is unlikely to lead to meaningful models.
One option to address this concern would be to remove languages or combine domains, however, our data here presents noclear choices. Alternatively, we could combine languages; thischoice leads to a related but slightly different question.
RQ2. Which language properties relate to defects?
Rather than considering languages individually, we aggregate
them by language class, as described in Section 2.3, and analyze
the relationship between defects and language class. Broadly, eachof these properties divides languages along some line that is often
discussed in the context of errors, drives user debate, or have been
the subject of prior work. To arrive at the six factors in the model
we combined all of these factors across all of the languages in ourstudy.
Ideally, we would want to include each of the separate properties
in the regression model so that we can assert with some assurance
that a particular property is responsible for particular defects. Unfortunately, however, the properties are highly correlated and models with all properties are not stable. To avoid this issue we model
the impact of the six different language classes, which result fromcombining the language properties, on the number of defects while
controlling for the same basic covariates that we used in the modelin RQ1.
Table 7: Functional languages have a smaller relationship to defects
than other language classes where as procedural languages are either
greater than average or similar to the average. Language classes arecoded with weighted effects coding so each language is compared to the
grand mean. AIC=10419, Deviance=1132, Num. obs.=1067
Defective Commits(Intercept)
log commits
log agelog sizelog devs
Functional-Static-Strong-Managed
Functional-Dynamic-Strong-Managed
Proc-Static-Strong-Managed
Script-Dynamic-Strong-Managed
Script-Dynamic-Weak-Managed
Proc-Static-Weak-Unmanaged∗∗∗
p < 0.001, ∗∗ p < 0.01, ∗ p < 0.05

−2.13
0.96
0.07
0.05
0.07
−0.25
−0.17
−0.06
0.001
0.04
0.14

(0.10)∗∗∗(0.01)∗∗∗(0.01)∗∗∗(0.01)∗∗∗(0.01)∗∗∗(0.04)∗∗∗(0.04)∗∗∗(0.03)∗(0.03)(0.02)∗(0.02)∗∗∗

As with language, we are comparing language classes with theaverage behavior across all languages. The model is presented inTable 7. As with the previous model, the first four variables arecontrol variables and we are not interested in their impact on theoutcome other than to say, in this case, that, with the exceptionof size, they are all positive and significant. their It’s clear that
3

Chi-Squared value of 243.6 with 96d.f. and p = 8.394e − 15

Script-Dynamic-Strong-Managed class has the smallest
magnitude coefficient. The coefficient is insignificant implying thatthe z-test for the coefficient cannot distinguish the coefficient fromzero. Given the magnitude of the standard error, however, we canreasonably assume that the lack of significance is not related towide variance or insufficient data, rather, it is because the behavior of languages in this class is very close to the average behavior
across all languages. We confirm this by recoding the coefficient
using Proc-Static-Weak-Unmanaged as the base level andemploying treatment, or dummy coding that compares each language class with the base level. In this case, Script-Dynamic-Strong-Managed is significantly different with p = 0.00044.
We note here that while choosing different coding methods affects
the coefficients and z-scores, the models are identical in all otherrespects. When we change the coding we are rescaling the coefficients to reflect the comparison that we wish to make [5]. Comparing the other language classes to the grand mean, Proc-Static-Weak-Unmanaged languages are more likely to induce defects.
This implies that either weak typing or memory management issues contribute to greater defect proneness as compared with otherprocedural languages.
Among scripting languages we observe a similar relationship between weak and strong typing. This is some evidence that weak vsstrong typing is more likely responsible for this difference as opposed to memory management, we cannot state this conclusively
given the correlation between factors. It is possible that among
script languages strong versus weak typing is driving the relationship and among procedural languages, memory management is driving the difference. However, as a group, strongly typed languages
are less error prone than average while the weakly typed languages
are more error prone than the average. The contrast between staticand dynamic typing is also visible in functional languages.
The functional languages as a group show a strong difference
from the average. Compared to all other language types, both
Functional-Dynamic-Strong-Managed and Functional-Static-Strong-Managed languages show a smaller relationship with defects. Statically typed languages have substantiallysmaller coefficient yet both functional language classes have thesame standard error. This is strong evidence that functional static
languages are less error prone than functional dynamic languages,
however, the z-tests only test whether the coefficients are different
from zero. In order to strengthen this assertion we recode the model
as above using treatment coding and observe that the Functional-Static-Strong-Managed language class is significantly
less defect prone than the Functional-Dynamic-Strong-Managed language class with p = 0.034.
NULL
log commits
log agelog sizelog devs
Lang. Class

Df

Deviance

1
1
1
1
5

31634.32
51.04
50.82
31.11
95.54

Resid. Df
1066
1065
1064
1063
1062
1057

Resid. Dev
32995.23
1360.911309.871259.05
1227.941132.40

Pr(>Chi)0.00000.00000.00000.00000.0000As with the relationship between language and defects, the relationship between language class and defects is based on a small
effect. The deviance explained is shown in the anova table above
and is similar, although smaller, to the deviance related to language
and, consequently, has a similar interpretation.
Having discussed the relationship between language class anddefects, we revisit the question of application domain.As before we
ask whether domain has an interaction with language class. Does
the choice of, e.g., a functional language, have an advantage fora particular domain? For this pair of factors, the contingency table confirms to assumptions. As above, a Chi-Square test for the

relationship between these factors and the project domain yields
a value of 99.0494 and df = 30 with p = 2.622e − 09 allowing us to reject the null hypothesis that the factors are independent.
Cramer’s-V yields a value of 0.133, a weak level of association.
Consequently, although there is some relation between domain andlanguage, there is only a weak relationship between domain andlanguage class.
Result 2: There is a small but significant relationship between language class and defects. Functional languages havea smaller relationship to defects than either procedural or
scripting languages.
It is somewhat unsatisfying that we do not observe a strong association between language, or language class, and domain withina project. An alternative way to view this same data is to aggregate
defects over all languages and domains, disregarding the relationship to projects. Since we cannot view this data as independent
samples, we do not attempt to analyze it statistically, rather we takea descriptive, visualization based approach.
We define Defect Proneness as the ratio of bug fix commits overtotal commits per language per domain. Figure 1 illustrates the interaction between domain and language using a heat map, where
the defect proneness increases from lighter to darker zone. We investigate which language factors influence defect fixing commits
across a collection of projects written across a variety of languages.
This leads to the following research question:
RQ3. Does language defect proneness depend on domain?
A first glance at Figure 1(a) reveals that defect proneness of thelanguages indeed depends on the domain. For example, in the
Middleware domain JavaScript is most defect prone (31.06%
defect proneness). This was little surprising to us since JavaScript is typically not used for Middleware domain. On a closer
look, we find that JavaScript has only one project, v8 (Google’s
JavaScript virtual machine), in Middleware domain that is responsible for all the errors.
Also, Scala is most defect prone in the Application domain
with a defect density (number of bug fix commit over total commit) of 52.67%. However, a single project zipkin, with defect density of 80%, contribute most of defects. Similarly, Perl and C
are the two top most buggy languages in Database domain with defect proneness 58.63% and 61.47% respectively. However, we findthat mysql project is solely responsible for such high defect count,where overall defect density of mysql is 70.25%.
Thus, it turns out that variation of defect density of the domains
with languages may be an attribute of individual projects. To verify this, we re-evaluate domain-language interaction after ignoring
the outliers. We filter out the projects that have defect density below 10 percentile and above 90 percentile. Figure 1(b) shows theresult. Note that, the outliers’ effects are also controlled in all our
regression models as we filter them out as high leverage points, as
discussed in Section 2.6.
The variation of defect proneness and languages per domain ismuch subdued in the new heat map. The remaining variation comesfrom the inherent defect proneness of the languages, as we haveseen in RQ1. To ensure this, we measure the pairwise rank correlation (Spearman Correlation [30]) between the language defectproneness for each domain with the Overall. For all the domains,the correlation is positive, and p-values are significant (< 0.01) except for Database domain. This shows that w.r.t. defect proneness,language ordering in each domain (except Database) is correlated

Language

bug_pcent60
40
20ApplicatCD
ion odeAnalyzer atabase

FramewLiork braryCC++C#
Objective−C
GoJava
Coffeescript
Javascript
Typescript
Ruby
Php
PythonPerl
ClojureErlangHaskellScalabug_pcentLanguage

C
C++C#
Objective−C
GoJava
Coffeescript
Javascript
Typescript
Ruby
Php
PythonPerl
ClojureErlangHaskell
Scala
MiddlewOare verall

40
30
20
10

Applicat

ion

CodeA

D
nalyze atabaser

FramewLiork brary

MiddlewOare verall

Domain

Domain(a) Variation of defect proneness across languages for a given domain(b) Variation of defect proneness across languages for a given domain
after removing the outliers

Figure 1: Interaction between language’s defect proneness and domain
Each cell in the heat map represents defect proneness of a language (row header) for a given domain (column header). The ‘Overall’ column represents defect proneness of alanguage over all the domains. The cells with white cross mark indicate null value, i.e. no commits were made corresponding to that cell.

SpearmanCorr.
p-value

APP

CA

DB

FW

LIB

MW

0.71

0.56

0.30

0.76

0.90

0.46

0.00

0.02

0.28

0.00

0.00

0.09

Result 3: There is no general relationship between domain
and language defect proneness.
We have shown that different languages induce a larger numberof defects and that this relationship is not only related to particularlanguages but holds for general classes of languages, however, wefind that the type of project doesn’t mediate this relationship to alarge degree. We now turn our attention to categorization of the
response. We want to understand how language relates to specifickinds of defects and how this relationship compares to the moregeneral relationship that we observe. We divide the defects into
categories as described in Table 5 and ask the following question:
RQ4. What is the relation between language & bug category?
We use a mixed-method analysis to understand the relation between languages and bug categories. First, using a descriptive,
visualization based approach similar to RQ3, we study the relation between bug categories and language class. A heat map (seeFigure 2) shows aggregated defects over language classes and bug
types and illustrate an overall relationship between language classand the bug categories.
To understand the interaction between bug categories with individual languages, we use a separate NBR model, as discussed in
RQ1, for each bug category. For each of these models we use thesame control factors as RQ1 as well as languages encoded withweighted effects as predictors, with the number of defect fixing
commits belong to that category as a response.
The results are shown in Table 8. For these models we present
only a subset of the anova coefficients as the distribution of deviance explained is very similar to the language model presented
in Table 6 and its associated anova table. Most importantly,thenumber of commits still commands the largest proportion of thedeviance, and the deviance explained by the remaining control variables is substantially smaller than that explained by the language.
The overall deviance for each model is substantially smaller,
however, and the proportion explained by language for a specificdefect type is similar in magnitude for most of the categories. We
interpret this relationship to mean that language has a greater impact on specific categories of bugs, than it does on bugs overall.
In the following section, we elaborate the result for bug categories with significant bug count as reported in Table 5. However,
our conclusion generalizes for all categories.
script−dynamic−weak−managed
Language Class

with the overall language ordering. Hence, domain has less significance when it comes to defect proneness of the languages.

script−dynamic−strong−managedproc−static−weak−unmanagedproc−static−strong−managedfunc−static−strong−managedfunc−dynamic−strong−managed

Algo

Concu Failure
rrencyMemory Perform Progra Security
ance mming

Bug Type

Figure 2: Relation between bug categories and language class
Each cell represents percentage of bug fix commit out of all bug fix commits per
language class (row header) per bug category (column header). The values arenormalized column wise.

Programming Errors. Generic programming errors account foraround 88.53% of all bug fix commits and occur in all the language
classes. The regression analysis draws a similar conclusion as of
RQ1 (see Table 6) since programming errors represent the majorityof the studied fixes. All languages incur programming errors suchas faulty error-handling, faulty object and variable definitions, incorrect data initialization, typos, etc.. Still, some programming errors are more language-specific. For example, we find 122 runtime
errors in JavaScript that are not present in TypeScript. In
contrast, TypeScript has more type related errors, since TypeScript compiler flags them during development. Thus, althoughgeneric programming errors occur in all languages, some kinds relate to specific language features.
Memory Errors. Memory errors count for 5.44% of all the
bug fix commits. The heat map in Figure 2 shows a strong relationship between Proc-Static-Weak-Unmanaged class andmemory errors. This is obvious as languages with unmanaged
memory type are known for memory bugs. Regression analysis
in Table 8 also confirms that languages with unmanaged memory
type, e.g.,C, C++, and Objective-C introduce more memory

Table 8: While the impact of language on defects varies across defect category, language has a greater impact on specific categories than it does on
defects in general. For all models above the deviance explained by language type has p < 3.031e − 07.
Memory(Intercept)−7.45(0.45)∗∗∗
2.32(0.12)∗∗∗
log commits
log age
0.31(0.15)∗
log size
0.00(0.09)
log devs
0.18(0.10)
1.81(0.11)∗∗∗
C
C++
1.14
(0.10)∗∗∗
C#−0.05(0.17)
1.47(0.15)∗∗∗
Objective-C
Go
0.00(0.25)
Java
0.59(0.14)∗∗∗−0.36(0.23)
CoffeeScript
JavaScript−0.11(0.10)TypeScript−1.32(0.40)∗∗
Ruby−1.12(0.20)∗∗∗−0.63(0.17)∗∗∗
Php
Python−0.44(0.14)∗∗
Perl
0.41(0.36)−0.41(0.18)∗
Scala
Clojure−1.16(0.27)∗∗∗
Erlang−0.53(0.23)∗−0.22(0.20)
Haskell
AIC
3033.55
BIC
3143.90-1494.77
Log Likelihood
Deviance
905.381114
Num. obs.
Residual Deviance (NULL)
8508.6
Language Type Deviance
587.2
∗∗∗
p < 0.001, ∗∗ p < 0.01, ∗ p < 0.05

−8.19
2.60
0.37
−0.23
0.25
0.58
1.18
0.93
0.52
1.73
0.97
−1.64
−0.12
−2.15
−0.82
−1.64
−0.21
−0.86
0.73
0.10
0.76
−0.17

errors than the average with statistical significance. Among themanaged languages, Java has significantly more memory errors
than the average, though its regression coefficient is less than theunmanaged languages. Though Java has its own garbage collector, having memory leak is not surprising since unused objectreferences often prevents the garbage collector from reclaiming thememory [14]. In fact, we notice 28.89% of all the memory errors
in Java comes from a memory leak. In terms of effect size, language has a larger impact on memory defects than all other cause
categories.
Concurrency Errors. 1.99% of total bug fix commits are related to Concurrency Errors. The heat map shows that Proc-Static-Weak-Unmanaged class dominates concurrency errors. In this class, C and C++ introduce 19.15% and 7.89% of theConcurrency errors, and they are distributed across the projects.
The regression result of Table 8 also confirms that with significant
p-value. Language classes Proc-Static-Strong-Managed
and Functional-Static-Strong-Managed are also in thedarker zone in the heat map. The major languages that contributeconcurrency errors from these classes are Go, C#, and Scala andtheir regression coefficient is also statistically significant. These
results confirm, in general static languages produce more concurrency errors than others. Among the dynamic languages, only Erlang is more prone to concurrency errors. The regression analysis also shows that projects written in dynamic languages like
CoffeeScript, TypeScript, Ruby, and Php have fewer concurrency errors (note statistically significant negative coefficients,
in Table 8).
race
deadlock
SHM
MPI

C
63.11
26.55
28.78
0

C++
41.46
43.36
18.24
2.21

C#
77.7
14.39
9.36
2.16

Java
65.35
17.08
9.16
3.71

Scala
74.07
18.52
8.02
4.94

Go
92.08
10.89
0
1.98

Erlang
78.26
15.94
0
10.14

A textual analysis based on word-frequency of the bug fix messages suggests that most of the concurrency errors occur due to race

Concurrency(0.75)∗∗∗(0.21)∗∗∗(0.24)(0.16)(0.17)(0.22)∗∗(0.18)∗∗∗(0.24)∗∗∗(0.29)(0.30)∗∗∗(0.22)∗∗∗(0.55)∗∗(0.17)(0.98)∗(0.30)∗∗(0.35)∗∗∗(0.23)(0.86)(0.25)∗∗(0.30)(0.29)∗∗(0.32)
2234.292344.63-1095.14
666.411114
3426.5157.03

−7.65
2.04
0.74
−0.07
0.46
0.51
0.49
−0.30
−0.04
0.55
0.14
−0.26
−0.02
−1.34
−0.24
0.27
0.25
−0.16
−0.16
−0.69
0.73
−0.31

Security(0.56)∗∗∗(0.15)∗∗∗(0.18)∗∗∗(0.11)(0.14)∗∗∗(0.17)∗∗(0.15)∗∗(0.22)(0.23)(0.27)∗(0.18)(0.26)(0.12)(0.41)∗∗(0.20)(0.19)(0.16)(0.43)(0.22)(0.26)∗∗(0.22)∗∗∗(0.26)
3411.57
3521.92-1683.79
921.501114
3372.5
61.40

−6.41
2.03
0.14
0.30
−0.03
0.67
1.15
−0.02
1.16
−0.38
0.32
0.00
−0.16
−0.34
−0.34
−0.61
−0.07
−0.40
−0.91
−0.53
0.65
−0.38

Failure(0.40)∗∗∗(0.11)∗∗∗(0.13)(0.08)∗∗∗(0.10)(0.12)∗∗∗(0.11)∗∗∗(0.16)(0.15)∗∗∗(0.24)(0.13)∗(0.19)(0.09)(0.07)∗∗∗(0.16)∗(0.17)∗∗∗(0.13)(0.40)(0.19)∗∗∗(0.19)∗∗(0.17)∗∗∗(0.19)
4188.714299.06-2072.36
1070.601114
6071.5303.8

condition, deadlock, or incorrect synchronization, as shown in thetable above. In all the languages, race condition is most frequent,
ranging from 41% in C++ to 92% is Go. The enrichment of race
condition errors in Go is likely because the Go is distributed with arace-detection tool that may advantage Go developers in detecting
races. Deadlocks are also noteworthy, ranging from 43.36% in C++
to 10.80% in Go. The synchronization errors are mostly related tomessage passing (MPI) or shared memory operation (SHM). Erlang and Go use MPI [10, 1] (which do not require locking ofshared resources) for inter-thread communication, which explains
why these two languages do not have any SHM related errors likelocking, mutex etc. In contrast, projects in the other languages
use SHM primitives for communication and can thus have lockingrelated errors.
Security and Other Impact Errors. Around 7.33% of all the
bug fix commits is related to Impact errors. Among them Erlang, C, C++, and Go produce more security errors than average,
as confirmed by the regression model (Table 8). The regression alsosuggests that projects written in TypeScript and Clojure areless likely to introduce a security error than average (Figure 2).
From the heat map we also see that Static languages are in
general more prone to failure and performance errors, followed
by Functional-Dynamic-Strong-Managed languages. In
the later category, Erlang is more prone to induce a failure (positive regression coefficient with statistical significance). The analysis of deviance results confirm that language is strongly associatedwith failure impacts. While security errors are the weakest among
the categories, with respect to the residual deviance of the model,
the deviance explained by language is still quite strong.
Result 4: Defect types are strongly associated with languages; Some defect type like memory error, concurrency errors also depend on language primitives. Language mattersmore for specific categories than it does for defects overall.

4.

RELATED WORK

Prior work on programming language comparison falls in three
categories: (1) Controlled Experiment: For a given task, developers are monitored while programming in different languages. Researchers then compare outcomes such as development effort andprogram quality. Hanenberg et al. [11] compared static vs. dynamic
typing by monitoring 48 programmers for 27 hours while developing a parser program. They found no significant difference incode quality between the two. However, dynamic type-based language have shorter development time. Their study was conductedwith undergraduate students in a lab setting with custom-designed
language and IDE. Our study, by contrast is a field study of popular software applications. While we can only indirectly (and post
facto) control for confounding factors using regression, we benefit from much larger sample sizes, and more realistic, widely-usedsoftware. We find that statically typed languages in general are lessdefect prone than the dynamic types, and that strong typing is better
than weak typing in the same regard. The effect sizes are modest;
it could be reasonably argued that they are visible here precisely
because of the large sample sizes.
Harrison et al. [12] compared C++, a procedural language, with
SML, a functional language, finding no significant difference in total number of errors, although SML has higher defect density thanC++. SML was not represented in our data, which however, suggest that functional languages are generally less defect prone thanprocedural languages. Another line of work primarily focuses oncomparing development effort across different languages [15, 21].
However, they do not analyze language defect proneness.
(2) Surveys: Meyerovich et al. survey developers’ views of programming languages, to study why some languages are more popular than others[18]. They report strong influence from non-linguistic
factors: prior language skills, availability of open source tools, andexisting legacy systems. Such factors also arise in our findings:
we confirm that availability of external tools also impact software
quality; Go has lot more concurrency bugs related to race condition
due to its race condition detection tool (see RQ4 in Section 3).
(3) Repository Mining: Bhattacharya et al. [2] study four projects
developed in both C and C++ and find that the software components developed in C++ are in general more reliable than C. We
find that both C and C++ are more defect prone than average defectproneness of all the studied languages, although C++ has a higher
regression coefficient (0.23) than C (0.15) (see Table 6). However,for certain bug types like concurrency errors, C is more defect prone
than C++ (see RQ4 in Section 3).

5.

THREATS TO VALIDITY

We recognize few threats to our reported results. First, to identify bug fix commits we did not check the bug database; instead
we rely on the keywords that developers often use to indicate abug fix commit. Our choice was deliberate. We wanted to capture
the issues that developers continuously face in an ongoing development process, not just the reported bugs. However, such choicepossesses threats of over estimation. Our categorization of the domains is subjected to interpreter’s bias, although another memberof our group verified the categories. Also, our effort to categorize alarge number of bug fix commits could potentially raise some questions. Especially, the categorization can be tainted by the initialchoice of keywords. Also, the descriptiveness of commit logs varyacross the projects. To mitigate the threat, we evaluate our classification against manual annotation as discussed in Section 2.5.
To interpret the language classes in Section 2.3, we make certain assumptions based on how a language property is most com-monly used, as reflected in our data set. For instance, we classify
Objective-C as unmanaged memory type, although it may follow a hybrid memory model. Similarly, we annotate Scala asfunctional and C# as procedural language, although they support
both procedural and functional design [20, 22].We do not distinguish object-oriented languages (OOP) in this work as there is noclear distinction between pure OOP languages and procedural languages.The difference largely depends on programming style. We
categorize C++ as weakly typed because a memory region of a certain type can be treated differently using pointer manipulation [?].
However, depending on the compiler some C++ type errors canbe detected in compile time. We further exclude TypeScriptfrom our language classification model (see Table 3 and Table 7);
TypeScript is intended to be used as a static, strongly typed language. However, in practice, we notice that developers often (for50% of the variables, and in all the TypeScript-using projects
in our dataset) use any type, a catch-all union type, and thus makes
TypeScript dynamic and weak.
Finally, we associate defect fixing commits to language properties, although they could reflect reporting style or other developer
properties. Availability of external tools or libraries may also impact the extent of bugs associated with a language.

6.

CONCLUSION

We have presented a large scale study of language type and use,
as it relates to software quality. The Github data we used is characterized by its complexity and the variance along multiple dimensions of language, language type, usage domain, amount of code,
sizes of commits, and the various characteristics of the many issue
types.
The large GitHub dataset sample-size allows a mixed-methods
study of the effects of language, while controlling for a number ofconfounds. Through a combination of regression modeling, text
analytics, and visualization, we have examined the interactions oflanguage, domain, and defect type. The data indicates functional
languages are better than procedural languages; it suggests thatstrong typing is better than weak typing; that static typing is better
than dynamic; and that managed memory usage is better than unmanaged. Further, that the defect proneness of languages in general
is not associated with software domains. Also, languages are more
related to individual bug categories than bugs overall.
On the other hand, even large datasets become small and insufficient when they are sliced and diced many ways simultaneously,
i.e. when the underlying connectivity between variables is rich. The
implications are that the more dependent variables there are, themore difficult it becomes (vis-a-vis the amount of data available) toanswer questions about a specific variable’s effect on any outcome
where interactions with other variables exist. Hence, we are unable
to quantify the specific effects of language type on usage. Additional methods such as surveys could be helpful here. Addressingthese challenges remains for future work.

7.

ACKNOWLEDGEMENTS

We thank Sameer Khatri for cross checking domain categorization. We acknowledge support from the National Science Foundation under Grants No. CCF-1247280 and CCF-1446683 and from
AFOSR award FA955-11-1-0246.

8.

REFERENCES[1] J. Armstrong, R. Virding, C. Wikström, and M. Williams.
Concurrent programming in erlang. 1993.

[2] P. Bhattacharya and I. Neamtiu. Assessing programming
language impact on development and maintenance: A study
on c and c++. In Proceedings of the 33rd International
Conference on Software Engineering, ICSE ’11, pages
171–180, New York, NY, USA, 2011. ACM.
[3] C. Bird, N. Nagappan, B. Murphy, H. Gall, and P. Devanbu.
Don’t touch my code!: examining the effects of ownershipon software quality. In Proceedings of the 19th ACM
SIGSOFT symposium and the 13th European conference onFoundations of software engineering, pages 4–14. ACM,
2011.
[4] D. M. Blei. Probabilistic topic models. Communications of
the ACM, 55(4):77–84, 2012.
[5] J. Cohen. Applied multiple regression/correlation analysis
for the behavioral sciences. Lawrence Erlbaum, 2003.
[6] H. CRAMÉR et al. Mathematical methods of statistics.
Princeton University Press, 1946.
[7] S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian.
Selecting empirical methods for software engineeringresearch. In Guide to advanced empirical software
engineering, pages 285–311. Springer, 2008.
[8] K. El Emam, S. Benlarbi, N. Goel, and S. N. Rai. The
confounding effect of class size on the validity ofobject-oriented metrics. Software Engineering, IEEE
Transactions on, 27(7):630–650, 2001.
[9] GitHub. Linguist: https://github.com/github/linguist.
[10] Google. http://golang.org/doc/effective_go.
html#concurrency.
[11] S. Hanenberg. An experiment about static and dynamic typesystems: Doubts about the positive impact of static typesystems on development time. In Proceedings of the ACM
International Conference on Object Oriented Programming
Systems Languages and Applications, OOPSLA ’10, pages
22–35, New York, NY, USA, 2010. ACM.
[12] R. Harrison, L. Smaraweera, M. Dobie, and P. Lewis.
Comparing programming paradigms: an evaluation offunctional and object-oriented programs. Software
Engineering Journal, 11(4):247–254, 1996.
[13] D. E. Harter, M. S. Krishnan, and S. A. Slaughter. Effects ofprocess maturity on quality, cycle time, and effort in software
product development. Management Science, 46(4):451–466,
2000.
[14] M. Jump and K. S. McKinley. Cork: dynamic memory leakdetection for garbage-collected languages. In ACM
SIGPLAN Notices, volume 42, pages 31–38. ACM, 2007.
[15] S. Kleinschmager, S. Hanenberg, R. Robbes, É. Tanter, and
A. Stefik. Do static type systems improve the maintainabilityof software systems? an empirical study. In Program
Comprehension (ICPC), 2012 IEEE 20th International
Conference on, pages 153–162. IEEE, 2012.
[16] Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Havethings changed now? An empirical study of bug

characteristics in modern open source software. In ASID ’06:
Proceedings of the 1st workshop on Architectural and system
support for improving software dependability, October 2006.
[17] J. P. Marques De Sá. Applied statistics using spss, statistica
and matlab. 2003.
[18] L. A. Meyerovich and A. S. Rabkin. Empirical analysis ofprogramming language adoption. In Proceedings of the 2013
ACM SIGPLAN international conference on Object oriented
programming systems languages & applications, pages
1–18. ACM, 2013.
[19] A. Mockus and L. G. Votta. Identifying reasons for softwarechanges using historic databases. In ICSM ’00: Proceedingsof the International Conference on Software Maintenance,
page 120. IEEE Computer Society, 2000.
[20] M. Odersky, L. Spoon, and B. Venners. Programming inscala. Artima Inc, 2008.
[21] V. Pankratius, F. Schmidt, and G. Garretón. Combining
functional and imperative programming for multicore
software: an empirical study evaluating scala and java. In
Proceedings of the 2012 International Conference on
Software Engineering, pages 123–133. IEEE Press, 2012.
[22] T. Petricek and J. Skeet. Real World Functional
Programming: With Examples in F# and C#. Manning
Publications Co., 2009.
[23] A. A. Porter and L. G. Votta. An experiment to assess
different defect detection methods for software requirements
inspections. In Proceedings of the 16th International
Conference on Software Engineering, ICSE ’94, pages
103–112, Los Alamitos, CA, USA, 1994. IEEE Computer
Society Press.
[24] D. Posnett, C. Bird, and P. Dévanbu. An empirical study on
the influence of pattern roles on change-proneness.
Empirical Software Engineering, 16(3):396–423, 2011.
[25] F. Rahman and P. Devanbu. How, and why, process metrics
are better. In Proceedings of the 2013 International
Conference on Software Engineering, pages 432–441. IEEE
Press, 2013.
[26] L. Tan, C. Liu, Z. Li, X. Wang, Y. Zhou, and C. Zhai. Bug
characteristics in open source software. Empirical Software
Engineering, 2013.
[27] V. Vapnik. The nature of statistical learning theory. springer,
2000.
[28] Q. Vuong. Likelihood ratio tests for model selection andnon-nested hypotheses. Econometrica: Journal of the
Econometric Society, pages 307–333, 1989.
[29] E. J. Weyuker, T. J. Ostrand, and R. M. Bell. Do too manycooks spoil the broth? using the number of developers toenhance defect prediction models. Empirical Software
Engineering, 13(5):539–559, 2008.
[30] J. H. Zar. Significance Testing of the Spearman Rank
Correlation Coefficient. Journal of the American Statistical
Association, 67(339):578–580, 1972.