# Utility of CK Metrics in Predicting Size of Board-Based Software Games

Nosheen Sabahat, Ali Afzal Malik, Farooque Azam

► To cite this version:

Nosheen Sabahat, Ali Afzal Malik, Farooque Azam. Utility of CK Metrics in Predicting Size of Board-Based Software Games. Mehran University Research Journal of Engineering and Technology, 2017, 36 (4), pp.975-986. hal-01705539

HAL Id: hal-01705539

https://hal.science/hal-01705539

Submitted on 9 Feb 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Utility of CK Metrics in Predicting Size of Board-Based Software Games

NOSHEEN SABAHAT*, ALI AFZAL MALIK**, AND FAROOQUE AZAM*

RECEIVED ON 05.02.2017 ACCEPTED ON 21.02.2017

## ABSTRACT

Software size is one of the most important inputs of many software cost and effort estimation models. Early estimation of software plays an important role at the time of project inception. An accurate estimate of software size is, therefore, crucial for planning, managing, and controlling software development projects dealing with the development of software games. However, software size is unavailable during early phase of software development. This research determines the utility of CK (Chidamber and Kemerer) metrics, a well-known suite of object-oriented metrics, in estimating the size of software applications using the information from its UML (Unified Modeling Language) class diagram. This work focuses on a small subset dealing with board-based software games. Almost sixty games written using an object-oriented programming language are downloaded from open source repositories, analyzed and used to calibrate a regression-based size estimation model. Forward stepwise MLR (Multiple Linear Regression) is used for model fitting. The model thus obtained is assessed using a variety of accuracy measures such as MMRE (Mean Magnitude of Relative Error), Prediction of x(PRED(x)), MdMRE (Median of Relative Error) and validated using K-fold cross validation. The accuracy of this model is also compared with an existing model tailored for size estimation of board games. Based on a small subset of desktop games developed in various object-oriented languages, we obtained a model using CK metrics and forward stepwise multiple linear regression with reasonable estimation accuracy as indicated by the value of the coefficient of determination (R² = 0.756). Comparison results indicate that the existing size estimation model outperforms the model derived using CK metrics in terms of accuracy of prediction.

**Key Words:** Accuracy Measures, Chidamber and Kemerer Metrics, Game Sizing, Multiple Linear Regression, Open Source, Project Management, Project Planning, Simple Linear Regression, Software Size Estimation, K-fold Cross Validation

## 1. INTRODUCTION

A number of attributes can be used to describe and judge various aspects of software. Some of these attributes (e.g. quality) are external while others (e.g. complexity) are internal. Software size

is classified as an internal attribute [1]. Once the software has been developed, its size can be easily measured by using any of a number of different applications called code counters [2]. The real challenge however, is

Corresponding Author (E-Mail: nosheen.sabahat@ceme.nust.edu.pk)

* Department of Computer Engineering, National University of Sciences & Technology, Islamabad.

** Department of Computer Science, National University of Computer & Emerging Sciences, Lahore.
estimating the size of software before its development. Unfortunately, software size is unavailable during early phases of software development. So we need to develop a size estimation model that accurately predicts the size of desktop-based software games.

Estimation of software size is challenging but the rewards for accurate prediction of size are extremely high. This is primarily because software size is one of the key inputs of many effort and cost estimation models [3]. Some of these models, such as the widely used COCOMO-II (Constructive Cost Model) [4], use both direct size measures e.g. SLOC (Source Lines of Code) [5]) as well as indirect functional size measures such as function points [6]. An accurate estimate of either SLOC or the programming-language independent function points is, therefore, crucial in planning and managing software development projects.

Software development projects using the object-oriented paradigm [7] use class diagrams [8] for modeling purposes. Analysis class diagrams are used to model the problem space while design class diagrams model the solution space [9]. Apart from being used as a tool for improving understanding of the problem and solution spaces, these class diagrams can also be used as a tool for estimating the size of the final software application [10].

Application design information, whether obtained from the class diagrams or extracted by reverse engineering [11] code written in an object-oriented programming language, is a good candidate for being used as an input to a size estimation model. This is primarily because design is the last phase before implementation. Therefore, a wealth of information is available at the end of the design phase. This research attempts to take advantage of this fact.

In this work, we describe the derivation and validation of a regression-based model developed to predict the size of board-based software games. This model uses the

design information provided by CK metrics [12]. It is a suit of most validated and most reliable metrics. It is used to evaluate object-oriented design [13] as input and generates an estimate of software size as output. A dataset comprising around 60 open source board-based software games is used to calibrate this model. Once calibrated, this model is assessed using different accuracy measures and is validated using K-fold cross validation [14]. The accuracy of this model is also compared to that of an existing size estimation model.

The next section discusses relevant past work in this area. Section-III presents an overview of the CK metrics and section-IV illustrates their calculation via a small case study. Our research methodology is described in detail in section-V which also summarizes the results of model assessment and validation and presents a comparison within existing size estimation model. Threats to validity of this research are discussed in section-VI. Section-VII highlights our major contributions and finally section-VIII proposes some directions for further work in this area.

## 2. LITERATURE REVIEW

Researchers have explored different ways of estimating software size using information contained in the class diagrams of object-oriented systems. One of the earliest works in this area was done by Mišic and Tešic [15]. They used the number of classes and the number of methods of classes in the class diagram to predict the size of a software system via OLS (Ordinary Least Squares) regression. Since, we do not use number of methods and number of classes for size estimation, we only adapted OLS regression technique.

More recently, Harizi [16] also proposed a software size estimation model that relied on various parameters of a class diagram. However, along with such parameters, we support a more comprehensive set of well-known suite of
object-oriented CK metrics, in estimating the size of
software applications using the information from its UML
class diagram. Zhou et. al. [10] looked at the accuracy of
code size estimation approaches empirically using the
information contained in UML class diagrams to predict
the size of object-oriented systems. Different modeling
techniques were used to check the accuracy of models
and it was finally concluded that object-oriented metrics
can be used to estimate the size of software. However,
their findings are based on Java systems only. Whereas
our work validates the findings using systems developed
in various object-oriented programming languages such
as C#, C++, Python and Java etc.

A slightly different approach was adopted by Costagliola
et. al. [17]. This approach used class points - a functional
size measure similar to function points but specifically
designed to estimate the size of object-oriented
applications. First unadjusted and then adjusted class
points were calculated. The size of the software was
predicted on the basis of these class points. Since
calculation of class points do not add any significance to
this work, we did not use them.

Another attempt in this area was made by Tan et. al.
[18]. They proposed a size estimation method for
information systems. Multiple linear regression was
used to obtain a size estimation model that relied on
information present in the class diagrams (such as
relationships/associations between classes, attributes
of classes) to feed the independent variables (i.e.
inputs). It was concluded that using class diagram
metrics provides better results in predicting the size of
the data intensive systems. However, this approach
attempts to estimate the size of information systems
only which the authors refer to as database application
that supports business processes. Moreover, it relies
only on the information of association and attributes
of classes from UML class diagram.

Various other metrics have also been used to estimate the
size of an object-oriented system such as class diagram
related metrics [19], object-oriented function points [20],
fast&serious class points [21] etc. All above mentioned
approaches use various class diagram related metrics only
to estimate the size of software. The assessment of model
accuracy and validation of most size estimation
approaches have not been carried out. Secondly the data
set is limited to few systems hence statistical conclusions
were hard to determine.

This study focuses on the size estimation of object-
oriented board-based software games using the design
information provided by CK metrics. These well-known
metrics are used to build a regression-based model which
is assessed for accuracy, validated using K-fold cross
validation, and compared to an existing size estimation
model.

3. OVERVIEW OF CK METRICS SUITE

Chidamber et. al. [22] proposed a set of metrics in 1991
specifically for object-oriented software programs. It was
improved in 1994 [12] and is now used widely. This suite
consists of 6 metrics viz. WMC (Weighted Methods/
Class), DIT (Depth of Inheritance Tree), NOC (Number of
Children), CBO (Coupling Between Objects), RFC
(Response For a Class) and LCOM (Lack of Cohesion in
Methods). The values of these metrics are determined for
each class in an object-oriented software. Brief
descriptions of these metrics [12] are given below:

**Weighted Methods Per Class:** In its simplest form, WMC is a count of all methods in a class. It provides an indication of the complexity of a class.

**Depth of Inheritance Tree:** DIT refers to the “depth of inheritance of the class”. In case of multiple inheritance, DIT is “the maximum length from the node to the root of
the tree". As a class gets deeper in the hierarchy, it inherits
more methods and variables.

**Number of Children:** NOC refers to the “number of immediate subclasses subordinated to a class in a class hierarchy” [18]. Unlike DIT, it measures the breadth of the class hierarchy.

**Coupling between Object Classes:** CBO for a class is defined as “a count of the number of other classes to which it is coupled”. Two classes are said to be coupled “when methods declared in one class use methods or instance variables defined by the other class”.

**Response for a Class:** RFC refers to “a set of methods that can potentially be executed in response to a message received by an object of that class”.

**Lack of Cohesion in Methods:** LCOM”provides a measure of the relative disparate nature of methodsin the class”. It indicates how closely local methods are related to local instance variables in a class. In other words, it is a number of disjoint set of local methods [23].

# 4. CASE STUDY

Consider a board game called "X and O". The objective of this two-player game is to win by getting three X's or O's in a row. This game is played on a three by three game board. The players take turns in placing X's and O's on the board till one of them wins or the board gets filled. The rules of the game are listed below:

## Game Start:

* Any player can start the game.

## Moves:

* The players choose their signs as X or O and then move accordingly.

* A player can place X or O at any empty place of the grid on his turn.

## Draw:

* The game is said to be drawn if none of the player succeeds in getting three in a row and the board gets filled.

## Win:

* The game is won by the player who gets three in a row first.

A software version of this game was developed using Java (row 57 of Table 2) and made available on an open source repository. Fig. 1 shows the design class diagram of the software version of this game. This diagram was reverse engineered from the source code of this game using the Understand tool [24]. The same tool was also used to extract the values of the CK metrics for the design of this game. Table 1 depicts the values of CK metrics for each of the design classes shown in Fig. 1. The last row of this table sums up the values of each CK metric for all individual classes. These sums are then used as inputs for our size estimation model.

# 5. RESEARCH METHODOLOGY

This research extends our earlier work on the size estimation of board-based software games [25,26]. In our earlier work, we defined eight potential predictors to estimate the size of desktop-based board games viz. NRUL (Number of Rules), NPLY (Number of Players), ANIM (Animation), 3DVI (3D Visualization), COPP (Computer Opponent), MSKI (Multi Skills), NTVA (Number of Type of Variants) and MGOP (Miscellaneous Game Options) as input of our model. In this work, we present the derivation, assessment, and validation of a new size estimation model based on CK metrics. The accuracy of this new design-based model is compared with the accuracy of an early size estimation model proposed in one of our earlier works [25]. Fig. 2 illustrates our research methodology. It depicts the major steps carried out in this research.
<figure><img src="image_2.png" /><figcaption>FIG. 1. DESIGN CLASS DIAGRAM OF THE GAME "X AND O" USING 'UNDERSTAND'TOOL [24]</figcaption></figure>

<table><caption>TABLE 1. CK METRICS FOR THE GAME "X AND O"</caption><thead><tr><th>Classes</th><th>LCOM</th><th>DIT</th><th>CBO</th><th>NOC</th><th>RFC</th><th>WMC</th></tr></thead><tbody><tr><td>Engine PANEL</td><td>75</td><td>2</td><td>0</td><td>0</td><td>5</td><td>5</td></tr><tr><td>Engine.Noughts and Crosses</td><td>82</td><td>2</td><td>3</td><td>0</td><td>16</td><td>16</td></tr><tr><td>Engine.Noughts and Crosses GameSide</td><td>100</td><td>1</td><td>1</td><td>0</td><td>2</td><td>2</td></tr><tr><td>Engine.settings</td><td>89</td><td>1</td><td>0</td><td>0</td><td>18</td><td>18</td></tr><tr><td>EntryPoint.Boot</td><td>0</td><td>1</td><td>0</td><td>0</td><td>1</td><td>1</td></tr><tr><td>EntryPoint.Boot.main.(Anon_1)</td><td>0</td><td>1</td><td>2</td><td>0</td><td>1</td><td>1</td></tr><tr><td>Engine.NoughtsAndCrosses.initComponents.(for_loop_5).(for_loop_6).(Anon_1)</td><td>0</td><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td></tr><tr><td>Engine.NoughtsAndCrosses.initComponents(Anon_2)</td><td>0</td><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td></tr><tr><td>Total</td><td>346</td><td>10</td><td>8</td><td>0</td><td>45</td><td>45</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>FIG. 2. BASIC STRUCTURE OF METHODOLOGY</figcaption></figure>
## 5.1 Extraction of CK Metrics

The same dataset (comprising 67 board-based games) that was used for calibrating our previous model [25] was used as a starting point. However, games that were not developed using an object-oriented programming language were pruned from the original dataset. As shown in Table 2, this pruning resulted in a slightly smaller subset comprising 59 games all of which were developed using an object-oriented programming language.

The values of the six CK metrics (i.e. LCOM, DIT, CBO, NOC, RFC, and WMC) for each of these games were extracted using a commercial static analysis tool called Understand [24]. This tool provides the values of the six CK metrics for each class. The values for the entire game are obtained by summing up the values of each class used in the game. SLOC (Source Lines of Code) for each game were counted by using USC-CSSE Unified Code Count tool [27]. From Table 2 we can see that most of the games have single SLOC whereas few games have multiple SLOC. This is because those games are developed in more than one language and hence all those languages contribute to the FP (Function Points).

QSM's language-specific GF (Gearing Factors) [28] are used for converting the SLOC to the programming-language independent FP. The average values of the language-specific GFs provided in the QSM FP Languages Table 2 [29] are used for this conversion. Similarly we have mentioned GF for multiple languages in the table. Using multiple GF we converted multiple SLOC into FP.

## 5.2 Analysis of CK Metrics

First, SLR [30] was used to determine the predictive strength of each CK metric individually. Later, forward stepwise MLR [31] was used to obtain our size estimation model which uses FP as the response variable and the six CK metrics as the six predictor variables. Regression analyses were done using the IBM SPSS tool [32].

Table 3 shows the results of SLR. It lists the predictors in descending order of predictive strength. As is evident from these results, WMC is the strongest individual predictor of software size (FP) with $R^2$ value of 0.709 whereas DIT is the weakest. The size estimation model obtained using MLR is presented in the next subsection.

## 5.3 Model Fitting by MLR

Before building the size estimation model, outliers had to be eliminated. Cook's distance [33] was used for this purpose. Four games were identified as outliers and hence eliminated from the dataset since their Cook's distance was greater than ((4/n)*3) where n represents the total number of games.

The data set consisting of 55 games now (after eliminating the four outliers) was used to build the final size estimation model. Forward stepwise MLR was used for model fitting. This technique is helpful in discovering the set of significant predictor variables which best explains the relationship between the response variable (FP) and the independent/predictor variables (LCOM, NOC, DIT, RFC, CBO, and WMC).

In this technique, the model is gradually built by adding one independent variable at a time. At each step, the value of coefficient of determination ($R^2$) is assessed to check if the recently added variable increased the value of $R^2$ or not. Besides this, only those combinations of predictors are selected in which all predictors are statistically significant at ±-value of 0.05. Moreover, the presence of multicollinearity among the predictors is also checked using the value of VIF (Variance Inflation Factor) [34]. In our case, all of the predictors in the MLR models obtained had VIF values close to 1. The presence of multicollinearity was, therefore, ruled out.
<table><caption>TABLE 2. DATA SET</caption><thead><tr><th>No.</th><th>Games</th><th>Language</th><th>SLOC</th><th>GF</th><th>FP</th><th>LCOM</th><th>DIT</th><th>CBO</th><th>NOC</th><th>RFC</th><th>WMC</th></tr></thead><tbody><tr><td>1.</td><td>3D Tic Tac Toe</td><td>C#</td><td>2851</td><td>54</td><td>52.796</td><td>2310</td><td>15</td><td>475</td><td>2</td><td>782</td><td>380</td></tr><tr><td>2.</td><td>C# Tic Tac Toe</td><td>C#</td><td>531</td><td>54</td><td>9.833</td><td>91</td><td>1</td><td>20</td><td>0</td><td>50</td><td>32</td></tr><tr><td>3.</td><td>Chess Bin</td><td>C#</td><td>2312</td><td>54</td><td>42.815</td><td>641</td><td>4</td><td>134</td><td>0</td><td>275</td><td>149</td></tr><tr><td>4.</td><td>Clu</td><td>C#</td><td>8388</td><td>54</td><td>155.333</td><td>1554</td><td>6</td><td>417</td><td>0</td><td>1001</td><td>659</td></tr><tr><td>5.</td><td>Connect 4 ab</td><td>C#</td><td>521</td><td>54</td><td>9.648</td><td>168</td><td>1</td><td>17</td><td>0</td><td>43</td><td>25</td></tr><tr><td>6.</td><td>Dot Chess</td><td>C#</td><td>3155</td><td>54</td><td>58.426</td><td>1620</td><td>27</td><td>503</td><td>12</td><td>1069</td><td>363</td></tr><tr><td>7.</td><td>Draughts Master</td><td>C#</td><td>1196</td><td>54</td><td>22.148</td><td>831</td><td>15</td><td>152</td><td>3</td><td>400</td><td>174</td></tr><tr><td>8.</td><td>DraughtsDotNet</td><td>C#/JS/ASP</td><td>12353/3174/73</td><td>54/47/51</td><td>297.723</td><td>8355</td><td>289</td><td>1665</td><td>113</td><td>5519</td><td>1677</td></tr><tr><td>9.</td><td>Four in a Row</td><td>C#</td><td>659</td><td>54</td><td>12.204</td><td>403</td><td>2</td><td>53</td><td>0</td><td>165</td><td>75</td></tr><tr><td>10.</td><td>Grid</td><td>C#</td><td>3004</td><td>54</td><td>55.63</td><td>286</td><td>1</td><td>102</td><td>13</td><td>203</td><td>110</td></tr><tr><td>11.</td><td>Learn Connect 4</td><td>C#</td><td>3729</td><td>54</td><td>69.056</td><td>991</td><td>10</td><td>155</td><td>7</td><td>674</td><td>321</td></tr><tr><td>12.</td><td>LudoDP</td><td>C#</td><td>2103</td><td>54</td><td>38.944</td><td>577</td><td>2</td><td>85</td><td>3</td><td>204</td><td>105</td></tr><tr><td>13.</td><td>Ludo.net</td><td>C#</td><td>11355</td><td>54</td><td>55.056</td><td>10032</td><td>175</td><td>1930</td><td>46</td><td>4915</td><td>2065</td></tr><tr><td>14.</td><td>Multi Reversi</td><td>C#</td><td>1843</td><td>54</td><td>34.13</td><td>92</td><td>1</td><td>26</td><td>0</td><td>41</td><td>32</td></tr><tr><td>15.</td><td>Othello Reversi</td><td>C#</td><td>2302</td><td>54</td><td>42.63</td><td>1549</td><td>17</td><td>243</td><td>10</td><td>664</td><td>290</td></tr><tr><td>16.</td><td>Quixo 3D</td><td>C#</td><td>3227</td><td>54</td><td>59.759</td><td>1772</td><td>56</td><td>495</td><td>19</td><td>1095</td><td>443</td></tr><tr><td>17.</td><td>RAE TTT</td><td>C#</td><td>825</td><td>54</td><td>15.278</td><td>249</td><td>9</td><td>79</td><td>2</td><td>200</td><td>76</td></tr><tr><td>18.</td><td>Reversi</td><td>C#</td><td>2464</td><td>54</td><td>45.63</td><td>612</td><td>6</td><td>131</td><td>0</td><td>200</td><td>110</td></tr><tr><td>19.</td><td>Reversi 2008</td><td>C#</td><td>1434</td><td>54</td><td>26.556</td><td>1063</td><td>9</td><td>160</td><td>2</td><td>317</td><td>148</td></tr><tr><td>20.</td><td>Scrabble for wpf</td><td>C#</td><td>3170</td><td>54</td><td>58.704</td><td>3588</td><td>39</td><td>648</td><td>26</td><td>1495</td><td>652</td></tr><tr><td>21.</td><td>Scrabble Nights</td><td>C#</td><td>2976</td><td>54</td><td>55.111</td><td>1604</td><td>12</td><td>358</td><td>2</td><td>543</td><td>288</td></tr><tr><td>22.</td><td>Sharp Checkers</td><td>C#</td><td>1588</td><td>54</td><td>29.407</td><td>1011</td><td>3</td><td>259</td><td>7</td><td>467</td><td>224</td></tr><tr><td>23.</td><td>Snake board game</td><td>C#</td><td>1442</td><td>54</td><td>26.704</td><td>179</td><td>3</td><td>60</td><td>0</td><td>92</td><td>56</td></tr><tr><td>24.</td><td>Sr Chess for WPF</td><td>C#</td><td>8318</td><td>54</td><td>154.037</td><td>2625</td><td>30</td><td>776</td><td>9</td><td>1388</td><td>716</td></tr><tr><td>25.</td><td>Stoopid Chess</td><td>C#</td><td>2796</td><td>54</td><td>51.778</td><td>1026</td><td>5</td><td>209</td><td>1</td><td>336</td><td>192</td></tr><tr><td>26.</td><td>TTT Network</td><td>C#</td><td>1444</td><td>54</td><td>26.741</td><td>257</td><td>2</td><td>66</td><td>0</td><td>88</td><td>52</td></tr><tr><td>27.</td><td>Blockem</td><td>C++</td><td>8811</td><td>50</td><td>176.22</td><td>1705</td><td>12</td><td>154</td><td>4</td><td>518</td><td>449</td></tr><tr><td>28.</td><td>Brutal Chess</td><td>C++</td><td>5987</td><td>50</td><td>119.74</td><td>1849</td><td>18</td><td>123</td><td>15</td><td>673</td><td>438</td></tr><tr><td>29.</td><td>Ethereal Chess</td><td>C++</td><td>14771</td><td>50</td><td>295.42</td><td>1643</td><td>0</td><td>59</td><td>0</td><td>816</td><td>816</td></tr><tr><td>30.</td><td>Gamebit (Chess)</td><td>C++</td><td>6463</td><td>50</td><td>129.26</td><td>2173</td><td>35</td><td>317</td><td>22</td><td>782</td><td>558</td></tr><tr><td>31.</td><td>Hotel</td><td>C++/C#</td><td>5452/7323</td><td>50/54</td><td>244.651</td><td>2913</td><td>31</td><td>546</td><td>13</td><td>1929</td><td>1009</td></tr><tr><td>32.</td><td>Maxit</td><td>C++</td><td>674</td><td>50</td><td>13.48</td><td>387</td><td>3</td><td>12</td><td>0</td><td>55</td><td>54</td></tr><tr><td>33.</td><td>Nero Othello</td><td>C++</td><td>3881</td><td>50</td><td>77.62</td><td>830</td><td>16</td><td>22</td><td>5</td><td>469</td><td>328</td></tr><tr><td>34.</td><td>Pentobi</td><td>C++/Python</td><td>17721/316</td><td>50/24</td><td>367.587</td><td>5662</td><td>86</td><td>561</td><td>54</td><td>2178</td><td>1526</td></tr><tr><td>35.</td><td>Piskvork</td><td>C++/Pascal</td><td>6670/124</td><td>50/91</td><td>134.763</td><td>30</td><td>0</td><td>0</td><td>0</td><td>5</td><td>5</td></tr><tr><td>36.</td><td>Pouet Chess</td><td>C++/Python</td><td>14272/352</td><td>50/24</td><td>300.287</td><td>3934</td><td>56</td><td>404</td><td>43</td><td>3602</td><td>1393</td></tr><tr><td>37.</td><td>Snakes &amp; Ladders</td><td>C++</td><td>621</td><td>50</td><td>12.429</td><td>307</td><td>6</td><td>8</td><td>0</td><td>122</td><td>101</td></tr><tr><td>38.</td><td>The Genius</td><td>C++</td><td>3473</td><td>50</td><td>69.46</td><td>638</td><td>8</td><td>21</td><td>0</td><td>156</td><td>140</td></tr><tr><td>39.</td><td>Tsubo</td><td>C++</td><td>8002</td><td>50</td><td>160.04</td><td>4167</td><td>110</td><td>458</td><td>75</td><td>4187</td><td>1086</td></tr><tr><td>40.</td><td>Win Chess</td><td>C++</td><td>9014</td><td>50</td><td>180.28</td><td>900</td><td>0</td><td>57</td><td>0</td><td>52</td><td>52</td></tr><tr><td>41.</td><td>Capa Chess</td><td>Java</td><td>5498</td><td>53</td><td>103.736</td><td>1172</td><td>31</td><td>154</td><td>7</td><td>189</td><td>175</td></tr><tr><td>42.</td><td>Cluedo</td><td>Java</td><td>11188</td><td>53</td><td>211.094</td><td>5069</td><td>194</td><td>556</td><td>28</td><td>1150</td><td>981</td></tr><tr><td>43.</td><td>Domination</td><td>Java</td><td>6323</td><td>53</td><td>119.302</td><td>2125</td><td>64</td><td>142</td><td>7</td><td>788</td><td>594</td></tr><tr><td>44.</td><td>Dots and Boxes</td><td>Java</td><td>879</td><td>53</td><td>16.585</td><td>630</td><td>20</td><td>26</td><td>0</td><td>96</td><td>96</td></tr><tr><td>45.</td><td>EBF Checkers</td><td>Java</td><td>5371</td><td>53</td><td>101.34</td><td>2505</td><td>120</td><td>191</td><td>15</td><td>804</td><td>574</td></tr><tr><td>46.</td><td>Get Four</td><td>Java</td><td>515</td><td>53</td><td>9.717</td><td>304</td><td>25</td><td>38</td><td>0</td><td>70</td><td>70</td></tr><tr><td>47.</td><td>GoBang (5 in row)</td><td>Java</td><td>1863</td><td>53</td><td>35.151</td><td>620</td><td>39</td><td>67</td><td>1</td><td>184</td><td>160</td></tr><tr><td>48.</td><td>Java Checkers</td><td>Java</td><td>695</td><td>53</td><td>13.113</td><td>341</td><td>11</td><td>14</td><td>0</td><td>88</td><td>88</td></tr><tr><td>49.</td><td>Java Frittle Chess</td><td>Java</td><td>1170</td><td>53</td><td>22.075</td><td>1241</td><td>47</td><td>79</td><td>4</td><td>258</td><td>200</td></tr><tr><td>50.</td><td>Java Tic Tac Toe</td><td>Java</td><td>467</td><td>53</td><td>8.811</td><td>92</td><td>21</td><td>19</td><td>0</td><td>49</td><td>49</td></tr><tr><td>51.</td><td>JO Chess</td><td>Java</td><td>3886</td><td>53</td><td>73.321</td><td>2858</td><td>83</td><td>198</td><td>6</td><td>357</td><td>297</td></tr><tr><td>52.</td><td>Jscrabble</td><td>Java</td><td>4895</td><td>53</td><td>92.358</td><td>3715</td><td>161</td><td>335</td><td>20</td><td>694</td><td>597</td></tr><tr><td>53.</td><td>KludgopolB</td><td>Java</td><td>14461</td><td>53</td><td>272.849</td><td>5407</td><td>260</td><td>704</td><td>43</td><td>1720</td><td>1138</td></tr><tr><td>54.</td><td>Scotland Yard</td><td>Java</td><td>1171</td><td>53</td><td>22.094</td><td>653</td><td>19</td><td>36</td><td>2</td><td>115</td><td>105</td></tr><tr><td>55.</td><td>Two Player Chess</td><td>Java</td><td>1525</td><td>53</td><td>28.774</td><td>1283</td><td>37</td><td>94</td><td>6</td><td>306</td><td>204</td></tr><tr><td>56.</td><td>Tyrannus (chess)</td><td>Java</td><td>3013</td><td>53</td><td>56.849</td><td>1581</td><td>58</td><td>129</td><td>0</td><td>298</td><td>298</td></tr><tr><td>57.</td><td>X and O</td><td>Java</td><td>267</td><td>53</td><td>5.038</td><td>346</td><td>10</td><td>8</td><td>0</td><td>45</td><td>45</td></tr><tr><td>58.</td><td>Scrabble 3D</td><td>Pascal</td><td>14326</td><td>91</td><td>157.429</td><td>3689</td><td>104</td><td>119</td><td>0</td><td>857</td><td>659</td></tr><tr><td>59.</td><td>GlParchis (Ludo)</td><td>C++/Python</td><td>4667/2747</td><td>50/24</td><td>208.478</td><td>1375</td><td>28</td><td>40</td><td>11</td><td>686</td><td>481</td></tr></tbody></table>
Table 4 summarizes the results of applying forward stepwise MLR on the data set. Each row in this table represents an MLR model. A check mark (✓) represents inclusion while a dash (-) indicates exclusion of the corresponding predictor variable. Using different combinations of predictors, a total of six models were obtained in which all predictors were statistically significant at α-value of 0.05. It is important to note that all of these six models had only two predictor variables. This is because none of the MLR models obtained using more than two predictors had all statistically significant predictors (at α-value of 0.05). The best model (highlighted in bold in Table 4) has an R² value of 0.756. The two predictors it uses are CBO and WMC. The following Equation (1), in which EST_FP (Estimated

<table><caption>TABLE 3. SLR RESULTS USING CK METRICS</caption><thead><tr><td>No.</td><td>Predictors</td><td>R2</td></tr></thead><tbody><tr><td>1.</td><td>WMC</td><td>0.709</td></tr><tr><td>2.</td><td>RFC</td><td>0.508</td></tr><tr><td>3.</td><td>LCOM</td><td>0.460</td></tr><tr><td>4.</td><td>NOC</td><td>0.332</td></tr><tr><td>5.</td><td>CBO</td><td>0.210</td></tr><tr><td>6.</td><td>DIT</td><td>0.196</td></tr></tbody></table>

Function Points) denotes the estimated value of the response variable (FP), formalizes this model:

## 5.4 Model Assessment and Validation

The accuracy of an estimation model must be formally assessed using different accuracy measures. In this research, we have used the two de-facto standards i.e. MMRE [35] and PRED(x) [35-36]. MMRE is defined as the average of all MREs calculated as a result of regression. On the other hand, PRED(x) is defined as, “the percentage of relative error deviation that lies within x”. The most commonly used values of x in literature are 25 and 30 with little difference in results [37]. We use PRED(25) in our work (i.e. x = 25) which is a more stringent metric than PRED(30). According to Conte et. al. [37] for a model to have reasonably good estimation accuracy, MMRE ≤ 0.25 and PRED(25) ≥ 0.75.

Despite the fact that MMRE and PRED(x) are the de-facto standards of estimation accuracy, they have been criticized due to their limitations [38]. Therefore, to overcome these limitations, we have used additional accuracy metrics as well i.e. MdMRE, MMER (Mean Magnitude of Error Relative) to estimate, MBRE (Mean of Balanced Relative Error), and MIBRE (Mean of Inverted Balanced Relative Error).

<table><caption>TABLE 4. MLR RESULTS USING CK METRICS</caption><thead><tr><th>No.</th><th>LCOM</th><th>DIT</th><th>CBO</th><th>NOC</th><th>RFC</th><th>WMC</th><th>R<sup>2</sup></th><th>VIF</th></tr></thead><tbody><tr><td>1.</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>0.756</td><td>CBO = 2.091<br>WMC = 2.091</td></tr><tr><td>2.</td><td>✓</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.501</td><td>LCOM = 3.094<br>DIT = 3.094</td></tr><tr><td>3.</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>0.550</td><td>LCOM = 2.429<br>RFC = 2.429</td></tr><tr><td>4.</td><td>-</td><td>✓</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>0.732</td><td>LCOM = 4.715<br>WMC = 4.715</td></tr><tr><td>5.</td><td>-</td><td>✓</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>0.270</td><td>DIT = 1.336<br>CBO = 1.336</td></tr><tr><td>6.</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>0.733</td><td>RFC = 6.427<br>MC = 6.427</td></tr></tbody></table>
Table 5 summarizes the values of these different accuracy metrics for our CK metrics-based size estimation model. The formulas for calculating these accuracy metricsare also given in Table 5. The values of these accuracy metrics

clearly indicate that this size estimation model does not have a reasonably good estimation accuracy. MMRE and other error-related means are more than 0.25.Similarly, PRED(25) is much less than 0.75.

<table><caption>TABLE 5. VALUES OF ACCURACY METRICS</caption><thead><tr><th>No.</th><th>Metric</th><th>Based On</th><th>Formula</th><th>Value</th></tr></thead><tbody><tr><td>1.</td><td>MMRE</td><td>MRE<sub>i</sub> = <sup>|Y<sub>i</sub> - Ŷ<sub>i</sub>|</sup>&frasl;<sub>Y<sub>i</sub></sub></td><td>MMRE = <sup>1</sup>&frasl;<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> MRE<sub>i</sub></td><td>0.69</td></tr><tr><td>2.</td><td>MdMRE</td><td>MRE<sub>i</sub> = <sup>|Y<sub>i</sub> - Ŷ<sub>i</sub>|</sup>&frasl;<sub>Y<sub>i</sub></sub></td><td>MdMRE = MEDIAN (i=1, N) [MRE<sub>i</sub>]</td><td>0.32</td></tr><tr><td>3.</td><td>MMER</td><td>MRE<sub>i</sub> = <sup>|Y<sub>i</sub> - Ŷ<sub>i</sub>|</sup>&frasl;<sub>ŝ<sub>i</sub></sub></td><td>MMER = <sup>1</sup>&frasl;<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> MER<sub>i</sub></td><td>0.56</td></tr><tr><td>4.</td><td>MBRE</td><td>BRE<sub>i</sub> = <sup>|ŝ<sub>i</sub> - Y<sub>i</sub>|</sup>&frasl;<sub>ŝ<sub>i</sub></sub>, Ŷ<sub>i</sub> - Y<sub>i</sub> &lt; 0<br>BRE<sub>i</sub> = <sup>|ŝ<sub>i</sub> - Y<sub>i</sub>|</sup>&frasl;<sub>Y<sub>i</sub></sub>, Ŷ<sub>i</sub> - Y<sub>i</sub> &ge; 0</td><td>MBRE = <sup>1</sup>&frasl;<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> BRE<sub>i</sub></td><td>0.34</td></tr><tr><td>5.</td><td>MIBRE</td><td>IBRE<sub>i</sub> = <sup>|ŝ<sub>i</sub> - Y<sub>i</sub>|</sup>&frasl;<sub>Y<sub>i</sub></sub>, Ŷ<sub>i</sub> - Y<sub>i</sub> &lt; 0<br>IBRE<sub>i</sub> = <sup>|ŝ<sub>i</sub> - Y<sub>i</sub>|</sup>&frasl;<sub>ŝ<sub>i</sub></sub>, ŝ<sub>i</sub> - y<sub>i</sub> &ge; 0</td><td>MIBRE = <sup>1</sup>&frasl;<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> IBRE<sub>i</sub></td><td>0.91</td></tr><tr><td>6.</td><td>PRED(25)</td><td>MRE<sub>i</sub> = <sup>|Y<sub>i</sub> - Ŷ<sub>i</sub>|</sup>&frasl;<sub>Y<sub>i</sub></sub></td><td>PRED(x) = <sup>1</sup>&frasl;<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> {<sup>1 if MRE<sub>i</sub> &ge; x</sup>&frasl;<sub>0 Otherwise</sub>}</td><td>0.36</td></tr></tbody></table>

<table><caption>TABLE 6. K-FOLD CROSS VALIDATION RESULTS</caption><thead><tr><th>K</th><th>Data Points</th><th>Size Estimation Model</th><th>R2</th><th>MMRE</th><th>MdMRE</th><th>MMER</th><th>MBRE</th><th>MIBRE</th><th>PRED (0.25)</th></tr></thead><tbody><tr><td>1.</td><td>01-11</td><td>17.6531 + (0.2676 * WMC) + (-0.1251 * CBO)</td><td>0.73</td><td>0.18</td><td>0.12</td><td>0.19</td><td>0.18</td><td>0.15</td><td>0.63</td></tr><tr><td>2.</td><td>12-22</td><td>25.3752 + (0.2582 * WMC) + (-0.1264 * CBO)</td><td>0.73</td><td>1.34</td><td>1.30</td><td>0.52</td><td>1.34</td><td>0.52</td><td>0.09</td></tr><tr><td>3.</td><td>23-33</td><td>26.4727 + (0.2508 * WMC) + (-0.1370 * CBO)</td><td>0.74</td><td>1.84</td><td>1.32</td><td>0.49</td><td>1.84</td><td>0.49</td><td>0.27</td></tr><tr><td>4.</td><td>34-44</td><td>14.3837 + (0.2948 * WMC) + (-0.0816 * CBO)</td><td>0.78</td><td>0.53</td><td>0.36</td><td>0.47</td><td>0.53</td><td>0.34</td><td>0.18</td></tr><tr><td>5.</td><td>45-55</td><td>8.7118 + (0.2676 * WMC) + (-0.1508 * CBO)</td><td>0.78</td><td>0.35</td><td>0.19</td><td>2.26</td><td>0.35</td><td>0.32</td><td>0.54</td></tr><tr><td colspan="3">Average</td><td></td><td>0.84</td><td>0.65</td><td>0.78</td><td>0.84</td><td>0.36</td><td>0.34</td></tr></tbody></table>
The terms $Y$ and $\hat{Y}_i$ in Table 5, denote the actual and the predicted values respectively obtained after regression.

K-fold cross validation [14] was used to validate this model. The data set comprising 55 games was divided into 5 random subsets (i.e. $K = 5$), each subset comprising 11 games placed randomly. In each iteration, 4 folds were used to train the model and the remaining 5th fold was used to validate the model. Table 6 summarizes the results of this K-fold cross validation exercise. The last row of Table 6 shows the average values of accuracy metrics. As indicated by these average values, the performance of the model is not up to the mark.

## 5.5 Comparison of Results

In this subsection, the accuracy of the design-based size estimation model obtained using CK metrics is compared with that of an early size estimation model proposed in one of our earlier works [25]. The early size estimation model uses four inputs (i.e. number of game rules, number of players, animation, and miscellaneous game options) and produces an estimate of FP as output. Table 7 shows a comparison of the accuracy of these two models by depicting values of different accuracy metrics. This comparison clearly indicates that the model built using

CK metrics has much lower accuracy than the early size estimation model proposed earlier. The early size estimation model has higher values of $R^2$ and PRED(25) and lower values for MMRE and other error-related means.

A comparison of the results of K-fold cross validation shown in Table 8 also corroborates the same conclusion. The early size estimation model clearly outperforms the design-based size estimation model built using CK metrics. This may appear counterintuitive since more information is present at the time of design. It must be kept in mind, however, that these two models use different predictors. It is quite possible that the predictors of the early size estimation model (for instance, number of rules) capture crucial information about the requirements which may have been missed by the CK metrics.

## 6. THREATS TO VALIDITY

A couple of factors may have an impact on the validity of our results. The first of these is related to the comparison of the size estimation model obtained using CK metrics and the size estimation model proposed earlier. The earlier size estimation model was calibrated using 65 open source board-based games. Since all of these games were not

<table><caption>TABLE 7. COMPARISON OF ACCURACY METRICS</caption><thead><tr><td>No.</td><td>Metric</td><td>CK Metrics Model</td><td>EarlySize Estimation Model [25]</td></tr></thead><tbody><tr><td>1.</td><td>R2</td><td>0.75</td><td>0.91</td></tr><tr><td>2.</td><td>MMRE</td><td>0.69</td><td>0.24</td></tr><tr><td>3.</td><td>MdMRE</td><td>0.32</td><td>0.18</td></tr><tr><td>4.</td><td>MMER</td><td>0.56</td><td>0.22</td></tr><tr><td>5.</td><td>MBRE</td><td>0.34</td><td>0.32</td></tr><tr><td>6.</td><td>MIBRE</td><td>0.91</td><td>0.21</td></tr><tr><td>7.</td><td>PRED(25)</td><td>0.36</td><td>0.75</td></tr></tbody></table>

<table><caption>TABLE 8. COMPARISON OF K-FOLD CROSS VALIDATION RESULTS</caption><thead><tr><td>No.</td><td>Metric</td><td>CK Metrics Model</td><td>EarlySize Estimation Model [25]</td></tr></thead><tbody><tr><td>1.</td><td>MMRE</td><td>0.84</td><td>0.23</td></tr><tr><td>2.</td><td>MdMRE</td><td>0.65</td><td>0.18</td></tr><tr><td>3.</td><td>MMER</td><td>0.78</td><td>0.25</td></tr><tr><td>4.</td><td>MBRE</td><td>0.84</td><td>0.34</td></tr><tr><td>5.</td><td>MIBRE</td><td>0.36</td><td>0.21</td></tr><tr><td>6.</td><td>PRED(25)</td><td>0.34</td><td>0.71</td></tr></tbody></table>
developed using object-oriented languages, the exact
same dataset could not be used for building a size
estimation model based on the CK metrics. A slightly
smaller subset comprising games developed using only
object-oriented programming languages was used instead.

The second factor is related to measuring the values of CK metrics. In this research, we have used a single static analysis tool i.e. Understand [24]. Since different static analysis tools may use slightly different conventions for calculating the values of the CK metrics, a change of tool may lead to a slight variance in results.

7. CONCLUSIONS

This paper has presented the results of an investigation
conducted to determine the utility of CKmetrics in
estimating software size. A dataset comprising around 60
open source board-based desktop games was used as a
starting point. The values of CK metrics of each game
were extracted using a commercial static analysis tool.
These values were then used for model fitting using
forward stepwise multiple linear regression. The fitted
model was assessed via commonly used accuracy
parameters and validated using K-fold cross validation.
Assessment and validation results indicate that the
accuracy of this model does not meet the recommended
standards suggesting that CK metrics alone may have
little utility in estimating software size.

The prediction accuracy of this model was also compared
to that of an existing size estimation model developed
specifically for estimating the size of board-based games.
The existing model relies on information available at the
time of inception while CK metrics, on the other hand, are
available only after the completion of detailed design.
Therefore, even though one may expect the model using
CK metrics to be more accurate, comparison results
indicate that the existing model outperforms the model
based on CK metrics with respect to prediction accuracy.
This unexpected behavior is, perhaps, the result of not
capturing crucial game requirements such as the number
of rules of the game.

8. FUTURE WORK

This work can be extended in a variety of ways. So far, we
have focused on a small subset dealing with just board-
based desktop games. Other types of games (e.g. card-
based) may also be considered. Similarly, games made for
other platforms (e.g. mobile) may also be explored. Thirdly,
the utility of other suites of object-oriented metrics may
be investigated. Last but not the least, a hybrid-model
using a combination of object-oriented design-based
software metrics and requirements-based metrics may be
built to get the best of both worlds.

ACKNOWLEDGEMENTS

Authors are thankful to National University of Sciences
& Technology, Islamabad, Pakistan, and FAST-National
University of Computer & Emerging Sciences, Lahore,
Pakistan, for their joint collaboration in the research work.

REFERENCES

[1] Fenton, N.E., and Pfleeger, S.L., "Software Metrics: A Rigorous and Practical Approach", 2nd Edition, PWS Publishing Co., pp. 435-445, Boston USA, 1998.

[2] QSM Code Counters, http://www.qsm.com/resources/
code-counters/, [Last Accessed: 30th December 2016].

[3] Pfleeger, S.L., Wu, F., and Lewis, R., "Software Cost Estimation and Sizing Methods: Issues and Guidelines", RAND Corporation, pp. 1-7, California USA, 2005.

[4] Boehm, B.W., Abts, C., Brown, A.W., Chulani, S., Clark, B.K., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B., "Software Cost Estimation with COCOMO-II", Prentice Hall, New Jersey USA, 2000.

[5] Laird, L.M., and Brennan, M.C., "Software Measurement and Estimation: A Practical Approach", John Wiley & Sons, New Jersey, USA, 2006.

[6] Albrecht, A.J., "Measuring Application Development", Proceedings of the IBM Applications Development Joint SHARE/ GUIDE Symposium, pp. 83-92, California, USA, 1979.

[7] Meyer, B., "Object-Oriented Software Construction", Prentice Hall, Volume. 2, pp. 331-410. New York USA, 1988.

[8] Booch, G., "Object-Oriented Analysis and Design", Addison-Wesley, California USA, 1980.

[9] Blaha, M., and Rumbaugh, J., "Object-Oriented Modeling and Design with UML", Pearson Education, Upper Saddle River, USA, 2005.
[10] Zhou, Y., Yang, Y., Xu, B., Leung, H. and Zhou, X., "Source Code Size Estimation Approaches for Object-Oriented Systems from UML Class Diagrams: A Comparative Study", Information and Software Technology, Volume 56, No. 2, pp. 220-237, 2014.

[11] Chikofsky, E.J. and Cross, J.H., "Reverse Engineering and Design Recovery: A Taxonomy", IEEE Software, Volume 7, No. 1, pp. 13-17, 1990.

[12] Chidamber, S.R. and Kemerer, C.F., "A Metrics Suite for Object-Oriented Design", IEEE Transactions on Software Engineering, Volume 20, No. 6, pp. 476-493, 1994.

[13] Chidamber, S.R., Darcy, D.P. and Kemerer, C.F., "Managerial Use of Metrics for Object-Oriented Software: An Exploratory Analysis", IEEE Transactions on Software Engineering, Volume 24, No. 8, pp. 629-639, 1998.

[14] Picard, R.R., Cook, R.D., "Cross-Validation of Regression Models", Journal of the American Statistical Association, Volume 79, No. 387, pp. 575-83, 1984.

[15] Mišic, V.B., Tešic, D.N., "Estimation of Effort and Complexity: An Object-Oriented Case Study", Journal of Systems and Software, Volume 41, No. 2, pp. 133-143, 1998.

[16] Harizi, M., "The Role of Class Diagram in Estimating Software Size", International Journal of Computer Applications, Volume 44, No. 5, pp. 31-33, 2012.

[17] Costagliola, G., Ferrucci, F., Tortora, G., Vitiello, G., "Class point: An Approach for the Size Estimation of Object-Oriented Systems", IEEE Transactions on Software Engineering, Volume 31, No. 1, pp.52-74, 2005.

[18] Tan, H.B.K., Zhao, Y. and Zhang, H., "Conceptual Data Model-based Software Size Estimation for Information Systems", ACM Transactions on Software Engineering and Methodology, Volume 19, No. 2, pp. 1-37, 2009.

[19] Antoniol, G., Lokan, C., Fiutem, R., "Object-Oriented Function Points: An Empirical Validation", Empirical Software Engineering, Volume 8, No. 3, pp. 225-254, 2003.

[20] Antoniol, G., Lokan, C., Caldiera, G., and Fiutem, R., "A Function Point-like Measure for Object-Oriented Software", Empirical Software Engineering, Volume 4, No. 3, pp. 263-287, 1999.

[21] Carbone, M., Santucci, G., "Fast&&Serious: A UML Based Metric for Effort Estimation", Proceedings of the 6th International ECOOP Workshop on Quantitative Approaches in Object-Oriented Software Engineering, pp. 35-44, 2002.

[22] Chidamber, S.R. and Kemerer, C.F., "Towards a Metrics Suite for Object-Oriented Design", Proceedings of 6th OOPSLA Conference, ACM, pp. 197-211, 1991.

[23] Bakar, N.S., "The Analysis of Object-Oriented Metrics in C++ Programs". Lecture Notes on Software Engineering, Volume 4, No. 1, pp. 48, 2016.

[24] Understand tool, https://www.scitools.com/, [Last Accessed: 25th December 2016].

[25] Sabahat, N., Malik, A.A., Azam, F., "A Size Estimation Model for Desktop Games", Submitted to IEEE Access, 2017.

[26] Sabahat, N., Malik, A.A., Azam, F., "Size Estimation of Open Source Board-based Software Games", Presented at the 9th IEEE International Conference on Open Source Systems and Technologies, 2015.

[27] UCC, Unified Code Counter, http://sunset.usc.edu/uccWP/, [Last Accessed: 30th October 2016].

[28] QSM Gearing Factors, "A Flexible Sizing Approach, Quantitative Software Management. Inc. http://www.qsm.com/Blog/Gearing%20Factors.pdf, [Last Accessed: 30th December 2016].

[29] QSM, Function Point Languages Table Version 4.0, Quantitative Software Management, http://www.qsm.com/resources/function-pointlanguages-table/index.html, [Last Accessed: 30th December 2016].

[30] Weisberg, S., "Applied Linear Regression", Wiley, 1980.

[31] Fritz, M., and Berger, P.D., "Improving the User Experience Through Practical Data Analytics", Morgan Kaufmann, pp. 239-269, 2015.

[32] IBM Corp. Released 2013. IBM SPSS Statistics for Windows, Version 22.0, IBM Corp, New York, 2013.

[33] Cook, R.D., "Detection of Influential Observations in Linear Regression", Technometrics, Volume 19, pp.15-18, 1977.

[34] O'Brien, R.M., "A Caution Regarding Rules of Thumb for Variance Inflation Factors", Quality and Quantity, Volume 41, No.5, pp.673-690, 2007.

[35] Foss, T., Stensrud, E., Kitchinham, and B., Myrtveit, I., "A Simulation Study of the Model Evaluation Criterion MMRE", IEEE Transactions on Software Engineering, Volume 29, No. 11, pp. 985-95, 2003.

[36] Port, D., and Korte, M., "Comparative Studies of the Model Evaluation Criterions MMRE and PRED in Software Cost Estimation Research", Proceedings of the Second ACMIEEE International Symposium on Empirical Software Engineering and Measurement, 2008.

[37] Conte, S.D., Dunsmore, H.E., and Shen, V.Y., "Software Engineering Metrics and Models", BenjaminCummings Publishing Co.,Redwood, 1986.

[38] Myrtveit, I., Stensrud, E., and Shepperd, M., "Reliability and Validity in Comparative Studies of Software Prediction Models", IEEE Transactions on Software Engineering, Volume 31, No. 5, pp. 380-391, 2005.