# AHP Modelling and Sensitivity Analysis for Evaluating the Criticality of Software Programs

André Librantz, Fábio Santos, Cleber Gustavo Dias, Adriana Cunha, Ivanir

Costa, Mauro De Mesquita Spinola

## ► To cite this version:

André Librantz, Fábio Santos, Cleber Gustavo Dias, Adriana Cunha, Ivanir Costa, et al., AHP Modelling and Sensitivity Analysis for Evaluating the Criticality of Software Programs. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2016, Iguassu Falls, Brazil. pp.248-255, 10.1007/978-3-319-51133-7_30. hal-01615750

HAL Id: hal-01615750

https://inria.hal.science/hal-01615750

Submitted on 12 Oct 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# AHP Modelling and Sensitivity Analysis for Evaluating the Criticality of Software Programs

André Felipe Henriques Librantz¹,*, Fábio Cosme Rodrigues dos Santos¹,²,
Cleber Gustavo Dias¹, Adriana Cristina Aipp da Cunha¹, Ivanir Costa¹, and
Mauro de Mesquita Spinola³

¹ Nove de Julho University - UNINOVE,São Paulo, Brazil
*librantz@uninove.br

² Companhia de Saneamento Básico do Estado de São Paulo, São Paulo, Brazil

³ Universidade de São Paulo – USP, São Paulo, Brazil

**Abstract.** In this paper the application of the analytic hierarchy process method (AHP) combined with the sensitivity analysis was proposed for evaluating the criticality of software programs and verify priority ranking stability. Results pointed that the proposed decision model could be implemented to help the decision-making process of classifying software programs, regarding their risk priority.

**Keywords:** Multicriteria Decision-making · Risk assessment · Software

## 1 Introduction

In today's business and operational environments, multiple organizations routinely work collaboratively in pursuit of a common mission, creating a degree of programmatic complexity that is difficult to manage effectively. Success in these distributed environments demands collaborative management that effectively coordinates task execution and risk management activities among all participating groups [1]. In order to reduce errors caused by complex problems, mainly in distributed environment, Information Technology (IT) organizations require that the relevant activities risk management should be successful [1,2]. The management of risks is a central issue in the planning and management of any venture. In the field of software, Risk Management is a critical discipline. The process of risk management embodies the identification, analysis, planning, tracking, controlling, and communication of risk [3].

The strategy to be adopted for risk management of complex systems requires a comprehensive and full view, so that the uncertainties of these process can be managed with structured techniques and decision-making can be made in different areas in the organization, so that risks can be identified, prioritized and mitigated, [4,5,6]. In this context, the analytic hierarchy process (AHP), a well-known multicriteria decision method has been used for risk assessment, forecasting benchmarking, resource allocation in several segments, such as manufacturing systems, financial systems, governmental, information technology, trying
to reduce subjective judgement errors and increase the decision reliability [7,8,9]. AHP was also used for evaluating the risk in software projects [10].

Nevertheless, from your knowledge, this method was not used for evaluating the risk in software programs, in which management control is shared by multiple people from different organizations [1]. Therefore, the purpose of this article is to apply the AHP modelling combined to sensitivity analysis to the process of evaluating the risk priority of this complex system.

## 2 Risks Key drivers

A systemic risk assessment is based on a small set of factors, called drivers, which strongly influence the eventual outcome or result. This set of drivers can be used to assess the program's current strengths and weaknesses, and forms the basis for the subsequent risk analysis. The Software Engineering Institute (SEI) risk management research that cataloged sources of risk in software development, system acquisition and operational security. The result of our analysis was the development of a common structure, or framework, for classifying a set of drivers that influence a program's outcome. As listed in Table 1, the driver framework comprises six categories:

<table><caption>Table 1. Risk Key drivers categories [1]</caption><thead><tr><td>Risk factor</td><td>Concept</td><td>Corroborating researches</td></tr></thead><tbody><tr><td>Objectives</td><td>Related to items like product,</td><td>[1,2,4,11]</td></tr><tr><td>Preparation</td><td>Cost and schedule focuses on the processes and plans required to achieve objectives</td><td>[1,2,4,11]</td></tr><tr><td>Execution</td><td>Focused on assembling, organizing, and overseeing the assets required to bring that plan to life.</td><td>[1,2,4,11]</td></tr><tr><td>Environment</td><td>Involves items like enterprise, organizational, and political conditions to support the completion of program activities.</td><td>[1,2,4]</td></tr><tr><td>Resilience</td><td>Involves capacity and capability to identify and manage potential events and changing circumstances.</td><td>[1,4]</td></tr><tr><td>Result</td><td>Refers to the correctness and completeness of the software intensive system or system of systems that is being developed</td><td>[1,4]</td></tr></tbody></table>
## 3 Analytic Hierarchy Process (AHP)

AHP Saaty [12] is a multicriteria selection method that is applied to the solution of complex problems that can have multiple objectives that affect decision-making [10,13], making it possible to evaluate qualitative and quantitative criteria simultaneously according to the judgments and importance attributed to each criterion and alternative by the decision-makers, resulting in a classification of alternatives. Generally, the process can be divided into three steps.

1. Decompose the problem into a hierarchy structure. In this step, the problem is decomposed into criteria and subcriteria, defining a decision hierarchy, as depicted in Figure 1.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Decision Model Scheme</figcaption></figure>

2. Construct the pairwise comparison matrix using the Saaty scale importance [13].

The comparison between pairs is carried out by means of specific software or electronic spreadsheet programs. The process is done through decision matrix A, which calculates the partial results of weights of each criterion, as follows:

$$v_i(A_j), j = 1, \dots, n \qquad (1)$$

where A_j is the weight of an alternative relative to criterion i.

In order to interpret and give relative weights to each criterion, it is necessary to normalize the previous comparison matrix. To do so, the following expression
is used:

$$ \sum_{i=1} \nu_i(A_j) = 1, \text{ for } j = 1, \dots, n \qquad (2) $$

where n is the criterion number, sub-criterion or alternative to be compared.

The judgments made by those involved in the judging process are evaluated by means of a consistency calculation. Firstly, it is necessary to obtain the maximum value of the eigenvector for each matrix through the following equation:

$$ \lambda = (\sum_{i \in \kappa} C_{i\kappa}^{-1})/n \qquad (3) $$

where n is the number of criteria. Index of consistency (CI) is calculated by:

$$ CI = \frac{\lambda - n}{n - 1} \qquad (4) $$

The consistency ratio (CR) is calculated by the following equation:

$$ CR = \frac{CI}{RI(n)} \qquad (5) $$

RI(n) is a fixed value based on the number of criteria, as presented in Saaty [12]. If CR <= 0.1, the degree of consistency is satisfactory, but if CR > 0.1, serious inconsistencies may exist, and the AHP may not yield meaningful results [12].

Next, the sums of partial results of each criterion are calculated by the following expression:

$$ \nu_i(A_j) = \frac{a_{ij}}{\sum_{i=1}^{n} a_{ij}}, \text{ for } j = 1, \dots, n \qquad (6) $$

3. Calculate the priority weights of alternatives according to the pairwise comparison matrix: For that, the priorities vectors of each alternative i relative to criterion Ck are calculated with the following expression:

$$ \nu_k(A_i) = \frac{\sum_{j=1}^{n} \nu_j(A_j)}{n}, \text{ for } j = 1, \dots, n \qquad (7) $$

After this, the weight of each criterion Ck and its impact on each of the alternatives is calculated using the following equation:

$$ W_i(C_j) = \frac{C_{ij}}{\sum_{i=1}^{n} C_{ij}}, \text{ for } j = 1, \dots, m \qquad (8) $$

where m is the value of criteria at the same level. The priority vector is obtained by:

$$ w_i(C_i) = \frac{\sum_{j=1}^{n} w_j(C_j)}{m}, \text{ for } i = 1, \dots, m \qquad (9) $$

Finally the evaluation of values of each alternative after normalization is obtained by equation 7:

$$ f(A_i) = \sum_{j=1}^{n} w_j(C_j) * \nu_i(A), \text{ for } j = 1, \dots, n \qquad (10) $$

where n is the number of alternatives.
## 4 Sensitivity Analysis

This approach involves changing the weight values and calculating the new solution. The method, also known as One-at-a-time (OAT), works by incrementally changing one parameter at a time, calculating the new solution and graphically presenting how the global ranking of alternatives changes. In this method, the global weights are a linear function depending on the local contributions [14]. Given this property, the global priorities of alternatives can be expressed as a linear function of the local weights. Furthermore, if only one weight $w_i$ is changed at a time, the priority $P_i$ of alternative $A_i$ can be expressed as a function of $w_i$ using the following formula:

$$P_i = \frac{P_{i''} - P_{i'}}{w'' - w'} (w_i - w_{i'}) + P_{i'} \quad (11)$$

where $P_{i''}$ and $P_{i'}$ are the priority values for $w_{i''}$ and $w_{i'}$, respectively.

## 5 Numerical Application

In this section the proposed decision model was applied to evaluate the risk level of hypothetical software programs, as follows: the implementation of BI and CRM solutions (PROG 1); Solution Development and Embedded Systems Trading (PROG 2) and Computer Cloud Deployment in the organizations (PROG 3). Table 2 shows the normalized weights assigned to each risk driver obtained from the expert judgements. In this table one can see that the preparation (c2)

<table><caption>Table 2. Normalized Pairwise Comparison Matrix</caption><thead><tr><th>Risk drivers</th><th>Objectives</th><th>Preparation</th><th>Execution</th><th>Environment</th><th>Resilience</th><th>Result</th><th>Weights</th><th>Priority</th></tr></thead><tbody><tr><th>Objectives</th><td>0.278</td><td>0.404</td><td>0.313</td><td>0.278</td><td>0.278</td><td>0.071</td><td>0.2702</td><td>2</td></tr><tr><th>Preparation</th><td>0.278</td><td>0.404</td><td>0.438</td><td>0.5</td><td>0.5</td><td>0.643</td><td>0.4603</td><td>1</td></tr><tr><th>Execution</th><td>0.056</td><td>0.058</td><td>0.063</td><td>0.056</td><td>0.056</td><td>0.071</td><td>0.0597</td><td>4</td></tr><tr><th>Environment</th><td>0.056</td><td>0.045</td><td>0.063</td><td>0.056</td><td>0.056</td><td>0.071</td><td>0.0576</td><td>5</td></tr><tr><th>Resilience</th><td>0.056</td><td>0.045</td><td>0.063</td><td>0.056</td><td>0.056</td><td>0.071</td><td>0.0576</td><td>6</td></tr><tr><th>Result</th><td>0.045</td><td>0.045</td><td>0.063</td><td>0.056</td><td>0.056</td><td>0.071</td><td>0.0576</td><td>3</td></tr><tr><th>CR=0.075</th><td>Σ=1</td><td>Σ=1</td><td>Σ=1</td><td>Σ=1</td><td>Σ=1</td><td>Σ=1</td><td>Σ=1</td><td></td></tr></tbody></table>

is the more critical risk driver. Based on these results, the software programs were analyzed. Table 3 shows the decision matrix and the results obtained for the alternatives.

Once obtained the decision matrix, it was possible to evaluate the score and classify the software programs according to their risk priority. The final classification is shown in Figure 2, giving the following order, from first (more critical) to last: PROG 1 (0.4487), PROG 2 (0.29579) and PROG 3 (0.2721). The
<table><caption>**Table 3.** Decision matrix</caption><thead><tr><th>Alternative</th><th>Objectives</th><th>Preparation</th><th>Execution</th><th>Environment</th><th>Resilience</th><th>Result</th></tr></thead><tbody><tr><td>PROG 1</td><td>0.09</td><td>0.15</td><td>0.04</td><td>0.04</td><td>0.05</td><td>0.07</td></tr><tr><td>PROG 2</td><td>0.09</td><td>0.15</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.02</td></tr><tr><td>PROG 3</td><td>0.09</td><td>0.15</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.01</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Final Evaluation of the Alternatives.</figcaption></figure>

PROG 1 presents a higher risk level, when compared to the PROG 2 AND PROG 3, which have very close values. Alternatively, the weights for each risk driver assigned to the alternatives can be compared against each other graphically as shown in Figure 3.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Driver Risks Weights Assigned to the Alternatives.</figcaption></figure>
## 5.1 Sensitivity Analysis

Figure 4 illustrates how the alternatives perform with respect to the risk driver “objectives”. One can see that by shifting the current value (27%) to 100%, there is no change in ranking. Similarly, when shifting the value of this driver to zero,

<figure><img src="image_5.png" /><figcaption>Fig. 4. Numerical Incremental Sensitivity Analysis.</figcaption></figure>

it does not result in any changes in the rank. This behavior can also be observed for the other risk drivers. Overall, based on the sensitivity analysis, it can be concluded that the final decision is consistent and reliable.

## 6 Conclusions

In this work, The AHP technique combined to sensitivity analysis was applied to evaluate software programs risk level. The results show that PROG 3 is less critical, as it presents lower values for the main risk drivers.

The sensitivity analysis performed in this study showed that changes in current values do not lead to ranking changes showing that the decision process was well-conducted, being useful for decision-makers. Moreover, as knowing which risk factors is more critical, the decision maker can more effectively focus his/her attention to that one in a given multicriteria decision problem.

Finally, hybrid decision models, such as Fuzzy AHP (F-AHP) and Fuzzy TOPSIS techniques may also be developed on the basis of this model.

## References

1. Alberts, C.J., Dorofee, A.J.: A Framework for Categorizing Key Drivers of Risk. Tech. rep., DTIC Document (2009)

2. dos Santos, L.R., others: Fatores de Risco nas Alianças em Projetos de TI: Estudo de Casos no Banco Central do Brasil. Revista de Administração 49(1), 217 (2014)
3. Mead, N.R.: Requirements Prioritization Case Study Using AHP. Software Engineering Institute 2008 (2006)

4. Davies, A., Brady, T.: Organisational Capabilities and Learning in Complex Product Systems: Towards Repeatable Solutions. Research Policy 29(7), 931–953 (2000)

5. Benta, D., Podean, I.M., Mircean, C.: On Best Practices for Risk Management in Complex Projects. Informatica Economica 15(2), 142 (2011)

6. Corral-Quintana, S., Legna-de la Nuez, D., Verna, C.L., Hernández, J.H., de Lara, D.R.M.: How to Improve Strategic Decision-making in Complex Systems when Only Qualitative Information is Available. Land Use Policy 50, 83–101 (2016)

7. Ishizaka, A., Labib, A.: Review of the Main Developments in the Analytic Hierarchy Process. Expert systems with applications 38(11), 14336–14345 (2011)

8. Syamsuddin, I.: Multicriteria Evaluation and Sensitivity Analysis on Information Security. arXiv preprint arXiv:1310.3312 (2013)

9. Funo, K.A., Muniz Junior, J., Marins, F.A.S.: Risk Factors in Aerospace Supply Chain: Qualitative and Quantitative Aspects. Production 23(4), 832–845 (2013)

10. Göl, H., Çatay, B.: Third-party Logistics Provider Selection: Insights from a Turkish Automotive Company. Supply Chain Management: An International Journal 12(6), 379–384 (2007)

11. Rodríguez, A., Ortega, F., Concepción, R.: A Method for the Evaluation of Risk in IT Projects. Expert Systems with Applications 45, 273–285 (2016)

12. Saaty, T.L.: Decision Making with the Analytic Hierarchy Process. International journal of services sciences 1(1), 83–98 (2008)

13. Neves, R.B., Pereira, V., Costa, H.G.: Auxílio Multicritério à Decisão Aplicado ao Planejamento e Gestão na Indústria de Petróleo e Gás. SciELO Brasil (2013)

14. Chen, H., Kocaoglu, D.F.: A Sensitivity Analysis Algorithm for Hierarchical Decision Models. European Journal of Operational Research 185(1), 266–288 (2008)