# Opportunity Costs in Free Open-Source Software

Siim Karus

► To cite this version:

Siim Karus. Opportunity Costs in Free Open-Source Software. 15th IFIP International Conference on Open Source Systems (OSS), May 2019, Montreal, QC, Canada. pp.139-150, 10.1007/978-3-030-20883-7_13. hal-02305700

HAL Id: hal-02305700

https://inria.hal.science/hal-02305700

Submitted on 4 Oct 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Opportunity Costs in Free Open-Source Software

Siim Karus¹[0000-0002-1029-7122]

¹ University of Tartu, 51009 Tartu, Estonia
siim.karus@ut.ee

**Abstract.** Opportunity cost is a key concept in economics to express the value one misses out on when choosing one alternative over another. This concept is used to explain rational decision making in a scenario where multiple mutually exclusive alternative choices can be made. In this paper, we explore this concept in the realm of open-source software. We look at the different ways for measuring the cost and these can be used to support decisions involving open-source software. We review literature on opportunity cost use in decision support in software development process. We explain how the opportunity cost analysis in the realm of open-source software can be used for supporting architectural decisions within software projects. We demonstrate that different measures of costs can be used to mitigate problems (and maintenance complexity) arising from the use of open source software, allowing for better planning of both closed-source commercial and open-source community projects alike.

**Keywords:** code churn, what-if analysis, scenario analysis, coding effort, alternative cost, opportunity cost, software cost.

## 1 Introduction

A large subdivision of software engineering research is focused on software estimation. Software estimation is concerned with devising and validating models that can be used to predict or estimate some aspect of software. These estimation models can be used for detecting trends in software development or to support decision making process or planning processes in software development. As such, the aim of many of these models is to provide decision makers with comparable values for different alternatives. Despite much effort put into evaluating and improving the accuracy of different models [1], the applicability of these models for actual comparison of alternatives is often overlooked. Hereby, we aim to collect the different approaches to measure cost in software development and review the approaches' applicability for comparing alternative actions.

In order to accomplish the set task, we perform an analytical review of objective comparison options. That is, we give emphasis on data-driven options instead of subjective expert (or community) opinion or evaluation. As a limitation, the options are only readily applicable on open source software, which makes relevant data available. We start by reviewing options for estimating the cost of software development. This is then complemented with an overview of literature reporting opportunity cost or alternative cost use in software engineering. In particular, we look for the following aspects:
1. What is the measure of the cost used?

2. Is the estimation model objective? That is: can the estimation be made solely based on the data?

3. Does the literature confirm the generalisability of the estimation model on different projects?

4. Does the literature report the confidence and accuracy of the model?

The first aspect is for classification of the approaches and identification of different measures of cost used in software engineering. The second and the third aspect describe the models' independence from specialized knowledge of experts that might not be available or have comparable skill level to the experts used in the original studies. As such, objective data-based models that have been evaluated on multiple projects are preferred due to their wider applicability and higher theoretical reproducibility. The last aspect evaluates whether the model can be used for comparison of alternatives in general. It is important to know the accuracy of the estimations in order to assess, whether two estimations are significantly different from one another. If two estimations differ less than the estimation error at given confidence level (or, to a limit, to given p-value), then the estimations can not be considered significantly different from each other.

The literature overview is followed by examples of how decision making in or involving free open-source software setting can benefit from the comparison of alternatives based on the estimation models. We highlight the challenges of extending the process of comparing alternatives to closed-source commercial software development (as opposed to FOSS) and list the main limitations of opportunity cost analysis. Finally, we conclude the study by listing opportunities for the future.

## 2 Measures of cost

Cost is an amount that has to be paid or given up in order to get something [2]. In business, cost is usually a monetary valuation of (1) effort, (2) material, (3) resources, (4) time and utilities consumed, (5) risks incurred, and (6) opportunity forgone in production and delivery of a good or service [2]. In software projects, cost can be measured using several units. In commercial environments, the value of different goods is made comparable by introducing a conversion rate for all goods into a common good - currency. Consequently, currency is commonly used to evaluate costs.

### 2.1 Monetary value

Even though the monetary value is most commonly used for cost measurement, it has many variations. A contemporary good cost estimation model is supposed to take into account any development and delivery effort until a release of the software [3]. Sometimes the monetary cost is also considered to include sales effort and expenses, efforts to adjust to changes, maintenance efforts, operational efforts and expenses, brand-related turnover, or other expenses or revenue opportunities. As such, the different monetary cost estimation methods can lead to very different estimations and need to be evaluated separately.
In free open-source software development, the monetary cost is usually of negligible interest, as the product is not sold for profit. In some extreme cases the projects may be sponsored, in which case the monetary dimension comes to play as the product needs to keep the sponsors interested and the sponsorship is an opportunity to allow higher expenses in the project.

## 2.2 Effort (person-hours)

As noted before, cost is usually measured in monetary evaluation. However, cost can be measured using other metrics as well. As the cost of workforce is highly volatile, a proxy of monetary cost, effort in worktime (usually person-hours or person-months) is used instead [1]. This measure is much more stable as the cost is measured directly and therefore does not differ as much in time and space as currency [3]. Nevertheless, such cost is highly dependent on the individual assigned to the task, as different developers can need different amount of time to complete the same task. To make the situation even more complicated, reliable measuring of time spent on task is very difficult. As software development requires planning and substantial mental work, it is not limited to a location or registered by automatic means. Consequently, time measurements are often based on self-reporting of the developers, which is not very accurate.

## 2.3 Time (calendar)

The release cycle and release timings have become an important aspect of software development. It is the need to publish early that competes with the time available for development and testing [4,5]. Consequently, the value measured by the cost can be the earliness of the next release. The cost does not have to be linear or even monotonic in relation to calendar time as certain dates or weekdays can be considered more valuable or less valuable than others.

## 2.4 Size

Sometimes the amount of code is used as a measure of cost. The size reflects the amount of effort put into developing the software – at least in the actual writing effort. Nevertheless, sometimes smaller size can be more valuable as it means less code to go through and to understand. As such, the size of the software can have a positive or negative value depending on the evaluator's objective.

## 2.5 Code Churn

Code maintenance effort is often evaluated based on code churn. Whilst code churn can be measured in several different units from modules and files to tokens and symbols, it is most commonly measured in lines of code (LOC). Code churn in lines of code is calculated as the sum of the number of lines of code added, modified and deleted between two revisions of a software module [6]. Some studies only count lines that are
written in certain programming language and affect program logic. However, all source code needs to be maintained.

In addition to providing an indicator of code maintenance effort, code churn has also been shown to be correlated with software defects [7,8]. In those cases, code churn is usually aggregated over a certain period of time (commonly a year [9,10,11]).

Naturally, code churn estimates over a certain period of time show how stable is the code-base. Projects with stable code-base can be considered more mature as there is less on-going development and fewer bug fixing going on. Thus, in situations where high reliability and low maintenance costs are important, projects with low code churn estimates would be generally preferable. Of course, this is on the premise that the stability of the project is not confused with the abandonment of the project.

Code churn is often estimated based on object-oriented metrics of the project [9,12,13]. Nevertheless, process-based metrics and organisational metrics have been shown to provide far better results [14,15,16].

## 2.6 Other cost measures

In addition to these more common cost measures, sometimes the relevant and substantial costs can be of very specific or difficult to measure kinds. For example, costs of brand, morale or public opinion can be detrimental. When discussing technical debt, effects towards monetary cost are always to be considered together with potentially longer term effects on morale, productivity, risk, and quality [17]. The practical limitation for uniform opportunity cost assessment is that some of these costs (e.g. morale) are very difficult to quantify. In those cases, the opportunity cost evaluations will need to be multi-dimensional in order to take account of all the different costs and scales they can be measured on.

# 3 Opportunity cost in FOSS

The comparison of alternative actions requires the valuation of the outcomes from making a decision. The basis for such rational value-based decision-making in economics is opportunity cost. **An opportunity cost (also known as alternative cost)** is a benefit, profit, or value of something that must be given up to acquire or achieve something else [2]. That is, an opportunity cost can be expressed as:

$$ \text{Opportunity cost} = \text{value of an option not chosen} - \text{value of chosen option.} $$

In rational value-based decision-making, the chosen option should always be the highest-valued option. As such, one can look at decision-making as an optimisation process.

Even though the formula is simple, the evaluation of alternatives rarely is. The main complexity stems from the fact that the “value” of an alternative is often difficult to define precisely. The cost estimation models are generally focusing just a few of the measures of the true value of an option (e.g. some models try to estimate monetary gain or loss, others focus on delivery times), which might not even be directly relatable to one-another. Consequently, if multiple factors apply the optimal decision can be found
<table><caption>**Table 1.** Search results and topics. Search results show 'number of results from keywords "opportunity cost" "software"' + 'number of results from keywords "alternative cost" "software"'.</caption><thead><tr><th>Database</th><th>IEEE Xplore</th><th>ACM Digital Library</th><th>ScienceDirect</th><th>Total</th></tr></thead><tbody><tr><td>Search results</td><td>13+4</td><td>8+3</td><td>51+11</td><td>90</td></tr><tr><td>Software Engineering articles</td><td>10</td><td>6</td><td>19</td><td>35</td></tr><tr><td>Relevant articles</td><td>4</td><td>2</td><td>2</td><td>8</td></tr><tr><td>Topics</td><td>Maintenance, testing, planning</td><td>Testing, planning</td><td>Testing, planning</td><td></td></tr><tr><td>Measures</td><td>Monetary, effort</td><td>Monetary</td><td>Monetary</td><td></td></tr><tr><td>Data-based articles</td><td>1</td><td>0</td><td>0</td><td>1</td></tr></tbody></table>

using Pareto optimisation [18]. This also means that the best option may have sub-optimal value in some respect. For example, faster delivery date may be more important than lower monetary expense for some projects.

We searched three major repositories of research relating to software engineering: IEEE Xplore<fnref n="1" />, ACM Digital Library<fnref n="2" /> and ScienceDirect<fnref n="3" />. As keywords, we used "opportunity cost" and "alternative cost". The search was refined by adding a keyword "software" to both searches to eliminate the bulk of studies not relating to software. Nevertheless, the results still contained largely studies that did not discuss topics relating to software development as studies discussing opportunity cost in other fields commonly listed the software packages used for opportunity cost estimation and comparison. In summary, the search from the databases yielded 90 results with only 35 relating to software development processes<fnref n="4" />. The 55 results were rejected based on the review of titles and abstracts, which revealed these to be discussing opportunity costs in agriculture, medicine, (non-software) economics, or other fields. The remaining 35 papers were manually reviewed to filter out papers where opportunity costs were discussed only in reference (e.g. as future work, introduction or only in abstract) or did not relate to decision-making in any way (e.g. algorithm optimisation). This left us with eight papers (four from journals, four from proceedings with the earliest from 1990 and latest from 2016) relevant to our current focus. The review also identified additional related keyword "options analysis", but searching using this keyword did not reveal additional articles of relevance to our study. The statistics of the search results can be found in Table 1.

We found three studies discussing opportunity costs in the planning of software development activities: [19], [20], and [21]. All of these studies were conducted in com-

<footnote n="1">¹ https://ieeexplore.ieee.org/</footnote>

<footnote n="2">² https://dl.acm.org/</footnote>

<footnote n="3">³ https://www.sciencedirect.com/</footnote>

<footnote n="4">⁴ We also looked into Google Scholar search, but found its tools for filtering the results too limiting for practical use. The original search keywords would get about 40,000 hits due to the popularity of "opportunity cost" in many fields. Unfortunately, Google Scholar does not provide means to limit search to specific fields and by adding exclusion keywords for more frequent non-related fields, we could limit the search results down to about 4000 before hitting the limitations of Google Scholar query complexity.</footnote>
mercial environment and had monetary cost included in the analysis. Spinola et al. defined indexes that can be used for balancing effort with monetary cost when assessing resource (developers) reallocation options [19]. Whilst the paper shows a nice example of how to derive the indexes needed for cost computation and comparison, the practical application of the analysis process would require expert input in order to identify appropriate index values. Similarly, Cai et al. proposes a framework that can be used to justify architectural decisions by minimising the risks and monetary costs of potential changes [21]. Özogul at al., on the other hand, presents a case study of a formula used to evaluate alternative investments in a hospital information system development and operation [20]. The investment value estimation is based on sub-estimates from experts or past experience.

Papers relating to software testing were the most common among the software engineering papers on opportunity cost: 4 out of 8 studies looked at aspects of testing. Two of these were trying to answer the question of when should one stop testing and release the software: [4] and [5]. Both of these looked at monetary cost balance of the cost of testing (and fixing bugs found during testing) and the cost of resolving bugs in released software. In [4], the authors try to improve a previously developed model for assessing the optimal time to stop testing by adding measures to handle uncertainty of the values of some components of the model. In overall, their model is robust but rather simplified with only the efficiency of the testing team (that is, how fast do they detect bugs) left to the experts to evaluate. [5] offers a more detailed model for calculating the costs and introduces the concept of patching to the model. The other two articles look into prioritization of tests in order to balance testing expenses with the risks from missed bugs [22], and finding the balance between the number of automated test cases and the number of human test executions [23].

Finally, one paper discussed opportunity cost in software maintenance: [24]. In there, the authors were looking at the opportunity costs from the inner-company client's perspective. They concluded that in their case study the costs for the IT department were not affected by the time bugs remained open (the "lead time"). However, they did find that it does influence the opportunity costs of the users of the information systems and could lead to sub-optimal distribution of costs within a company. As the analysis was performed on actual historic data, it does not rely on the availability and skill of experts, which makes it stand out of all the papers on opportunity cost reviewed here.

Even though “opportunity cost” is a term rarely used in current software engineering research, a properly reported study of “cost estimation” can be used for assessing opportunity costs as well. This is an advantage as “cost estimation” is a popular topic with a combined total of over more than 2000 search hits in the three databases used in this study.

## 4 Opportunity cost in decision making

Opportunity cost can be used in several different decision-making scenarios. For example, it is useful for picking the platforms or libraries for use in a project potentially avoiding the need to reverse such decisions (e.g. reverting from Angular JS due to
maintenance costs incurred by its fast pace of changes); or evaluation of options for
refactoring a project's component and/or dependency structure. Hereby we give exam-
ples on how opportunity cost analysis can be used in these scenarios depending on the
available cost measure.

4.1 Choosing a product

Let us assume a scenario where a project manager needs to choose, which software
library or framework to use in the project. There are several approaches to solving this
task depending on the aim of the project. Given two or more alternative options, the
project manager can make a preliminary choice of one library that seems most suitable
(alternatively, he could choose to not use a third party library). Based on the choice, the
project manager can calculate the costs of adopting alternative libraries. If the oppor-
tunity cost of adopting an alternative library is favourable, the project manager can
change the choice to become the better alternative.

The monetary opportunity cost should be easy to understand – positive opportunity cost means that the monetary gains from adopting the alternative library outweigh the gains from the current choice. Accordingly, negative monetary opportunity cost favours current choice.

The opportunity cost in effort (person-hours) is a bit more complex to act upon. As effort is by its nature a loss (the team spends effort) and the opportunity cost is defined as difference of gains, the opportunity cost in effort is the negative of the difference of the team's efforts. Thus, a positive opportunity cost in effort means that the alternative requires less work than the current choice. This means that if the project is mostly voluntary, the minimisation of effort might be preferred (thus, the choice should be changed if opportunity cost is positive). In some cases, both the effort and monetary costs need to be considered together as separate dimensions. This can be a case if there are license or infrastructure costs linked to the adoption of libraries. We highlight that the comparison is not linear as the difference in monetary cost can be insignificant if the budget is high or the monetary cost difference can be limiting due to budget limitations. Thus, the monetary cost component tends to behave in a stair-like fashion rather than linear fashion.

The opportunity cost in time is even more complicated to calculate, as the value of
the timing of actions is rarely monotonic. For example, optimal release opportunities
tend to depend on the weekly and monthly cycles. In addition, special external events
or deadlines from clients may make certain periods (e.g. releasing consumer-oriented
services ahead of major holidays) more favourable than others. The same applies to the
team members' ability to react to other team members' changes in the source code.

As mentioned in the description of the measure of size, size can be either a positive or a negative attribute of the software. Consequently, it needs to be handled accordingly when calculating the opportunity cost.

As code churn is an indicator of maintenance effort, it can provide interesting in-
sights into the quality and development stage of the library. A high code churn estima-
tion will indicate that the library will be actively developed or maintained. A low esti-
mation means that the module is either stable or abandoned. Consequently, when look-
ing to adopt a more stable library, one would prefer libraries with lower code churn
estimation (code churn is a negative feature). If one would like to use more changing
and agile libraries, high code churn would be preferred (code churn is a positive fea-
ture). In general, commercial companies focused on reliability are more interested in
using stable libraries and frameworks in their projects [25].

This example of choosing a library or a framework can be extended to a scenario of
choosing a software application from multiple alternatives. For example, software with
lower estimated code churn can be expected to be more stable than other alternatives.

## 4.2 Refactoring options

Software component maintainability can be an important factor when deciding how to refactor the software. The refactoring options can include replacement of a component integration of a component into other source code, or preservation and continued maintenance of the component. Using a framework for assessing a module's or component's quality (e.g. via a method proposed by Upadhyay et al. [26]), will give general quality assessment on the module, but does not give an accurate assessment of the cost of replacing or integrating the component as the maintenance of software components is highly dependent on one-another (as shown by Mari et al. [27]). Understanding this, we can devise a following method for assessing the alternative cost of a software module.

Let us assume that a project manager or architect is wondering whether certain sections of the software should be replaced or removed from the project. We can find out, whether removing (or replacing) the modules has positive effect on the maintenance of the software by simulating the removal (or replacing) of the module and estimating the opportunity cost in maintenance effort.

From the general definition of opportunity cost, we construct the definition of opportunity cost of maintaining a software module. Opportunity cost or alternative cost of not having a module *m* in project *p* (*A*<sub>*m*</sub><sup>*p*</sup>) is the difference between the cost of maintaining or developing the project without the module (*E*<sub>*m*</sub><sup>*p*</sup>) and the cost of maintaining or developing the project with the module (*C*<sup>*p*</sup>).

$$A_m^p = E_m^p - C^p$$

It is clear from the definition, that in order to calculate the opportunity cost of maintenance of the project without a module, we need to know both the cost of developing the project with and without a module. These two costs never occur in the same project at the same time, which rules out using actual data for calculating the opportunity cost. Therefore, we need to use an oracle that would tell how much the development in one or another scenario would cost. The part of the oracle can be performed by a cost estimation model, which allows reproducible objective analysis of options, or by experts (or a combination of experts and cost models), which is less generalizable.
Having found out that the opportunity cost of removing the module is significantly<fnref n="5" /> negative, we know that we should keep the module. If the opportunity cost would have turned up significantly positive, you would know that the module is safe to remove. If the opportunity cost is near zero, the removal of the module is likely to have little to no effect on the maintainability of the project. Similarly, one can simulate any refactoring option and calculate corresponding opportunity cost. Therefore, it would be possible to evaluate all refactoring options before taking action.

As the possible simulations are not limited by anything but the oracle’s ability to
sense the changes, the same process can be used for more accurate evaluation of imple-
mentation alternatives. The oracle has another limitation – one needs to know whether
the oracle’s accuracy is below or above the significance threshold (confidence) of the
developer. Thus, only estimation models with published error and confidence (or p-
value) can be used as an oracle.

4.3 Commercial applications

In commercial applications, the monetary cost of the project becomes more relevant than in free software. The opposite is often true for the effort as the availability of workforce is less volatile. Simplistically, the monetary value can be assumed to stem from effort alone [1]. In practice, this can be an oversight as depending on the project, the infrastructure costs or even public image can have higher impact on the monetary costs than developer salary. Commercial evaluation also needs to take into account inflation, which affects monetary costs [20].

On the other hand, a more rigid structure of commercial projects will allow more
precise cost estimation methods to be used for opportunity cost estimation. This can
lead to much more accurate opportunity cost analysis and better options for finding the
optimal behaviour. In this study, we focus on free software as the data on free software
has better availability and lends itself to easier replication of the study.

5 Limitations of cost estimation

An important factor that was highlighted by Özogul et al. is that the value of the meas-
urement unit changes in time [20]. As they were studying the monetary costs, they re-
solved that limitation by calculating the present value of any future values. They could
do that on the assumption of knowing the approximate average inflation rate for the
prices in the studied currency. In general, it is not that simple to compare values or costs
from different time periods. This also applies to code churn as the efficiency of software
developers is increasing due to the introduction of frameworks, higher level languages
and design-time code generation. However, we are currently not aware of any standard
methods or discounting the effects of such advancements in code churn analysis.

<footnote n="5">⁵ Significantly means that the probability of being mistaken is less than acceptable by your confidence threshold. Commonly confidence of 0.95 or higher is used meaning the probability of being mistaken is 5% or less.</footnote>
As all estimations, cost estimation are not ideal and exact. Thus, one needs to con-
sider the limitations of the accuracy of cost estimation models used. We highlighted
that as a requirement for any cost estimation model to be generally useable as it is pos-
sible to identify significant cost differences only with models that have known error
ranges and confidence levels. Not understanding and controlling for the significance of
difference can easily lead to invalid conclusions. As in our experience similar software
modules often have similar cost estimations, this can lead to project managers switching
software modules and components too often (even when there is no significant benefit
of cost) causing more cost to incur due to instability of the project architecture.

6 Conclusions and future perspectives

Opportunity cost is a key concept in economics when describing rational decision-making. A literature search of opportunity cost in software in three larger software-related research databases found only a handful of relevant results. Therefore, it is clear that the value-based decision making rationales in software engineering are not well established. Even more, we found that only 1 of 8 papers demonstrated opportunity cost calculation without relying on expert opinion. Only 1 out of 8 papers claimed that the approach used was tested on more than 2 projects and none of the studies claimed verification from more than 2 experts when expert opinion was needed. As such, the generalisability of the current studies is weak. None of the studies reported the accuracy and confidence of the proposed methods.

On positive note, we found that there is at least one cost estimation method that can
be used for opportunity cost analysis. Furthermore, the identified method uses a model
that does not require any expert input – the estimations were based solely on the data
available in a source code repository history. This allows the approach to be used even
in situations where experts with sufficient competence are not available without need-
lessly exposing the estimation process to a potential human error. The technique pre-
sented in this paper shows how the effects of decisions made during software develop-
ment process can be simulated. More specifically, we offer an approach for better iden-
tification of the drivers of software coding effort in open source software projects. As
such, we cater for the future-oriented needs of software development analysts as de-
fined by Buse et al. [28]. As an added benefit of using easy-to-use end-user-oriented
analysis platform, our analysis technique can be easily deployed without deep under-
standing of statistics, databases or software source code management systems.
The obvious future perspective is the need for more studies of opportunity cost model-
ling in software engineering. Some of the scarcity of opportunity cost modelling can be
due to the performance of the estimation models, which we have seen to be sufficient
for limited success. Improving the estimation abilities of the models can lead to signif-
icantly better and more detailed what-if analysis.

7 Acknowledgement

This work was supported by the Estonian Research Council (grant IUT20-55).
8 References

1. Magne Jørgensen and Martin J. Shepperd, "A Systematic Review of Software Development Cost Estimation," *IEEE Transactions on Software Engineering*, vol. 33, no. 1, pp. 33-53, 2007.

2. BusinessDictionary. [Online]. http://www.businessdictionary.com/

3. P. Pandey, "Analysis of the Techniques for Software Cost Estimation," in *Proceedings of 2013 Third International Conference on Advanced Computing and Communication Technologies*, Rohtak, India, 2013, pp. 16-19.

4. Siddhartha R. Dalal and Colin L. Mallows, "Some graphical aids for deciding when to stop testing software," *IEEE Journal on Selected Areas in Communications*, vol. 8, no. 2, pp. 169-175, 1990.

5. PK Kapur and AK Shrivastava, "Release and testing stop time of a software: A new insight," in *Proceedings of 4th International Conference on Reliability, Infocom Technologies and Optimization*, Noida, India, 2015.

6. Gregory A. Hall and John C. Munson, "Software evolution: code delta and code churn," *Journal of Systems and Software*, vol. 54, no. 2, pp. 111-118, 2000.

7. John C. Munson and Sebastian G. Elbaum, "Code Churn: A Measure for Estimating the Impact of Code Change," in *Proceedings of International Conference on Software Maintenance (ICSM)*, 1998, pp. 24-31.

8. Nachiappan Nagappan and Thomas Ball, "Use of relative code churn measures to predict system defect density," in *Proceedings of the International Conference on Software Engineering*, 2005, pp. 284-29.

9. Mie Mie Thet Thwin and Tong-Seng Quah, "Application of neural networks for software quality prediction using object-oriented metrics," *Journal of Systems and Software*, vol. 76, no. 2, pp. 147-156, 2005.

10. C. van Koten and A. R. Gray, "An application of Bayesian network for predicting object-oriented software maintainability," *Information & Software Technology*, vol. 48, no. 1, pp. 59-67, 2006.

11. Siim Karus and Marlon Dumas, "Predicting Coding Effort in Projects Containing XML," in *Proceedings of 16th European Conference on Software Maintenance and Reengineering (CSMR)*, 2012, pp. 203-212.

12. Wei Li and Sallie Henry, "Object-Oriented Metrics Which Predict Maintainability," *The Journal of systems and software*, vol. 23, no. 2, pp. 111-122, November 1993.

13. Yuming Zhou and Baowen Xu, "Predicting the maintainability of open source software using design metrics," *Wuhan University Journal of Natural Sciences*, vol. 13, no. 1, pp. 14-20, 2008.

14. Foyzur Rahman and Premkumar Devanbu, "How, and Why, Process Metrics Are Better," in *Proceedings of the 2013 International Conference on Software Engineering*, San Francisco, CA, USA, 2013, pp. 432-441.

15. Siim Karus and Marlon Dumas, "Code churn estimation using organisational and code metrics: An experimental comparison," *Information and Software Technology*, vol. 54, no. 2, pp. 203-211, 2012.
16. Nachiappan Nagappan, Brendan Murphy, and Victor R. Basili, "The influence of organizational structure on software quality: an empirical case study," in *Proceedings of 30th International Conference on Software Engineering (ICSE)*, Leipzig, Germany, 2008, pp. 521-530.

17. Edith Tom, Aybüke Aurum, and Richard Vidgen, "An exploration of technical debt," *Journal of Systems and Software*, vol. 86, no. 6, pp. 1498-1516, 2013.

18. Andrzej P Wierzbicki, "A mathematical basis for satisficing decision making," *Mathematical modelling*, vol. 3, no. 5, pp. 391-405, 1982.

19. Mauro de Mesquita Spinola, Marcelo Schneck de Paula Pessoa, and Antonio Carlos Tonini, "The Cp and Cpk Indexes in Software Development Resource Relocation," in *Portland International Center for Management of Engineering and Technology*, Portland, USA, 2007, pp. 2431-2439.

20. C. Okan Özogul, E. Ertugrul Karsak, and Ethem Tolga, "A real options approach for evaluation and justification of a hospital information system," *Journal of Systems and Software*, vol. 82, no. 12, pp. 2091-2102, 2009.

21. Yuanfang Cai and Kevin J Sullivan, "A value-oriented theory of modularity in design," in *Proceedings of the seventh international workshop on Economics-driven software engineering research*, St. Louis, USA, 2005, pp. 1-4.

22. Amanda Schwartz and Hyunsook Do, "Cost-effective regression testing through Adaptive Test Prioritization strategies," *Journal of Systems and Software*, vol. 115, pp. 61-81, 2016.

23. Rudolf Ramler and Klaus Wolfmaier, "Economic perspectives in test automation: balancing automated and manual testing with opportunity cost," in *Proceedings of the 2006 international workshop on Automation of software test*, Shanghai, China, 2006, pp. 85-91.

24. Taizan Chan, "Beyond productivity in software maintenance: factors affecting lead time in servicing users' requests," in *Proceedings of International Conference on Software Maintenance, 2000, 2000*, pp. 228-235.

25. Steven Raemaekers, Arie van Deursen, and Joost Visser, "Measuring software library stability through historical version analysis," in *In Proceedings of the 28th IEEE International Conference on Software Maintenance (ICSM)*, Trento, Italy, 2012, pp. 378-387.

26. Nitin Upadhyay, BharatM. Despande, and VishnuP. Agrawal, "Towards a Software Component Quality Model," *Advances in Computer Science and Information Technology. In series Communications in Computer and Information Science*, vol. 131, pp. 398-412, 2011.

27. M. Mari and N. Eila, "The impact of maintainability on component-based software systems," in *Proceedings of 29th Euromicro Conference*, 2003, pp. 25-32.

28. Raymond P. L. Buse and Thomas Zimmermann, "Information Needs for Software Development Analytics," in *Proceedings of the 34th International Conference on Software Engineering*, Zurich, Switzerland, 2012, pp. 987-996.