# An Empirical Examination of the Consistency Ratio in the Analytic Hierarchy Process (AHP)

Valery Lukinskiy, Vladislav Lukinskiy, Boris Sokolov, Darya Bazhina

► To cite this version:

Valery Lukinskiy, Vladislav Lukinskiy, Boris Sokolov, Darya Bazhina. An Empirical Examination of the Consistency Ratio in the Analytic Hierarchy Process (AHP). IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2021, Nantes, France. pp.477-485, 10.1007/978-3-030-85914-5_51 . hal-03897864

HAL Id: hal-03897864

https://inria.hal.science/hal-03897864

Submitted on 14 Dec 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# An Empirical Examination of the Consistency Ratio in the Analytic Hierarchy Process (AHP)

Valery Lukinskiy¹,*, Vladislav Lukinskiy¹, Boris Sokolov², Darya Bazhina¹

¹ National Research University Higher School of Economics (HSE University), St. Petersburg, Russia

² Saint Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, St. Petersburg, Russia

lukinskiy@mail.ru

**Abstract.** Papers related to solving practical problems in all areas of economic activity use the AHP methodology from the 1980s, proposed on the basis of studying a limited number of tangible objects. One possible option to overcome this situation could be a new approach based on probability theory and mathematical statistics. We used information obtained from processing 292 matrices collected between 2014 and 2020 and reflecting expert judgements. We were able to compare a particular expert's judgement with an array of expert judgements from previous studies to refine the computational operations of the AHP. It was found that the probability estimates of the consistency indices for expert judgements differ significantly from the probability estimates for the generated values; e.g., for 6×6 matrices, they differ by a factor of 20. The results of our study do not confirm the need to revise expert judgements if the consistency ratio (C.R.) is greater than 0.10, and confirm the need for more example calculations for tangible objects and for new dependencies to estimate intangible objects based on artificial intelligence models and methods.

**Keywords:** Expert Judgements, Consistency Ratio, Tangible Objects

## 1 Introduction

Ivanov and Sokolov [1] emphasize that during the supply chain (SC) planning horizon, different structural elements (decision-makers, processes, products, control variables, constraints, goals, perturbations, and so on) are involved in the decision making for SC planning. Dolgui et al. [2] highlight that different problems need different solutions, and no one technique can be universally applied to the same effect. According to Felice [3], one of the most commonly used multi-criteria decision making methods is the Analytic Hierarchy Process (AHP). Csató [4] notes that in real-world applications the judgements of decision-makers may be inconsistent: for example, alternative A is two times better than alternative B, alternative B is three times better than alternative C, but alternative A is not six times better than alternative C. Sarraf and McGuire [5] note that the consistency ratio (C.R.) is the ratio of the consistency
index to the average random index for the same order matrix. A C.R. of 0.10 or less is considered acceptable.

In works related to practical tasks (the multi-objective facility layout problem, selection of automated guided vehicles for handling materials, supplier assessment, selection of the most suitable managers for projects, design operations in distribution centres, calculating a supply chain performance index, inventory classification, making sustainable investment decisions, and many others), the value C.R. ≤ 0.1 is used to assess matrix consistency [6-20]. Table 1 provides information from some of these sources in more detail.

<table><caption>**Table 1.** Assessing matrix consistency in works related to solving practical problems.</caption><thead><tr><td>Reference</td><td>Main purpose of the study</td><td>Remarks</td></tr></thead><tbody><tr><td>[8]</td><td>The paper proposes a model for supplier assessment</td><td>As CR &lt; 0.10, the degree of consistency is satisfactory. If the value is higher, that would indicate that the PCM must be revised</td></tr><tr><td>[11]</td><td>This paper proposes a tool for the decision-making process of selecting the most suitable managers for projects</td><td>If CR is higher than 0.10, then the judgment matrix is inconsistent and judgments should be reviewed and improved</td></tr><tr><td>[17]</td><td>AHP and VIKOR techniques are used to evaluate the supplier</td><td>If CR ≤ 0.1, the judgement matrix is acceptable; otherwise, it is considered inconsistent</td></tr><tr><td>[20]</td><td>The aim of this paper is to provide a framework for sustainable SC development</td><td>The value of CR must be lower than 0.10 for an acceptable level of consistency</td></tr></tbody></table>

A number of papers question the need to revise expert opinions if C.R. > 0.1. Falsini et al. [21] outline that in the original version of AHP, indeed, if C.R. is greater than 0.10 the decision maker traditionally should not tolerate the error and should reject the analysed matrix. In a business environment, problems connected to this limitation are clearly identifiable: every iteration of the same step implies a cost in terms of time and money. Ishizaka and Labib [22] note that for all consistency checking, some questions remain: what is the cut-off rule to declare my matrix inconsistent? Should this rule depend on the size of the matrix? The results of Ishizaka and Siraj [23] experimentally invalidate the threshold of C.R. < 0.1 and suggest a much higher threshold of acceptance. Out of 50 AHP participants, 5 did not report their level of inconsistency in the given questionnaire. Out of the remaining 45 participants, only 8 participants passed the widely accepted criterion of C.R. < 0.1. Chao et al. [24] indicate that inconsistency of PCM exists naturally in many sport activities, thus the ranking should depend on the existing matches in spite of the inconsistencies. Abastante et al. [25] declare that the C.R. threshold has to be suggested by the analyst depending on the complexity of the problem. Brunelli [26] emphasizes that not all the facets of pairwise comparison matrices can be fruitfully analyzed by means of randomly generated matrices. This suggests increasing the use of empirical preferences in the study of preference relations. According to Grzybowski and Starczewski [27], expert systems are designed to assist decision-makers and they are not substitutes for them.
Expert systems do not have such human capabilities as feelings, responsibility or simply concern about risk. Thus the final decision about the rejection or acceptance of the PCMs should belong to the DMs.

Thus, works related to practical problem solving in all areas of economic activity use C.R. ≤ 0.1 to assess matrix consistency [28, 29] in spite of the results of studies on the development of AHP. It should be noted that the value C.R. ≤ 0.1 was derived from a study on a limited number of tangible objects. One possible way to overcome this situation is to take a new approach based on probability theory and mathematical statistics.

This paper presents a probabilistic approach for evaluating matrix consistency, which, compared to the deterministic approach, allows establishing quantitative relationships between expert and generated matrices. We used information obtained from processing 292 matrices collected between 2014 and 2020 and reflecting expert judgements. The obtained results do not confirm the need to revise expert judgements if C.R. is greater than 0.10, calls for many more example calculations for tangible objects and a search for new dependencies to estimate intangible objects based on artificial intelligence models and methods.

The rest of the paper is organised as follows. The results of designing a methodology for assessing matrix consistency are presented in Section 2. Section 3 provides the directions for future research.

## 2 Conducting a Probabilistic Assessment of Matrix Consistency

Despite considerable progress in the study and use of the analytic hierarchy process (AHP), some aspects remain under-researched. One of them is the problem of assessing the difference between generated and expert (expert-generated) matrices for tangible and intangible objects.

The choice of a particular criterion value z, equal to a certain value of the consistency ratio C.R., is of fundamental importance, as it affects the rest of the calculation procedure according to the AHP. According to Saaty [28, 29], if the calculated value of C.R. is less than or equal to z, the matrix is considered consistent; in this case, the priority vector wᵢ and the consistency index C.I. can be used for decision making. If C.R. > z, then experts must reconsider some of the judgements in the matrix.

Table 2 shows the statistical characteristics used to determine the parameters of the distribution laws chosen to approximate the C.I. samples of expert and generated matrices. The samples of the first type are a summary of the material presented in various sources (International Journal of Production Research, International Journal of Production Economics, Expert Systems with Applications, European Journal of Operational Research, Decision Support Systems, Omega, and others) and in our own research. We used information obtained from processing 292 matrices collected between 2014 and 2020 and reflecting expert judgements. Samples of the second type were generated on the basis of generated matrices of dimensions n×n [28, 29].
<table><caption>**Table 2.** Statistical characteristics and distribution law parameters for expert and generated C.I. samples.</caption><thead><tr><th rowspan="2">Matrix</th><th rowspan="2">Type of sample</th><th rowspan="2">Sample size</th><th colspan="2">Statistical characteristics</th><th rowspan="2">Distribution law</th><th colspan="2">Parameters of distribution</th></tr><tr><th>x̄</th><th>σ</th><th>m</th><th>b<sub>m</sub></th></tr></thead><tbody><tr><td rowspan="2">3 × 3</td><td>1</td><td>76</td><td>0.050</td><td>0.037</td><td>Weibull</td><td>1.35</td><td>0.73</td></tr><tr><td>2</td><td>100</td><td>0.382</td><td>0.517</td><td>Weibull</td><td>0.73</td><td>0.53</td></tr><tr><td rowspan="2">4 × 4</td><td>1</td><td>60</td><td>0.160</td><td>0.140</td><td>Weibull</td><td>1.15</td><td>0.95</td></tr><tr><td>2</td><td>600</td><td>0.894</td><td>0.620</td><td>Weibull</td><td>1.45</td><td>0.91</td></tr><tr><td rowspan="2">5 × 5</td><td>1</td><td>86</td><td>0.445</td><td>0.240</td><td>Rayleigh</td><td>2.00</td><td>0.89</td></tr><tr><td>2</td><td>600</td><td>1.124</td><td>0.540</td><td>Truncated normal</td><td></td><td></td></tr><tr><td rowspan="2">6 × 6</td><td>1</td><td>57</td><td>0.561</td><td>0.405</td><td>Weibull</td><td>1.40</td><td>0.91</td></tr><tr><td>2</td><td>600</td><td>1.225</td><td>0.450</td><td>Normal</td><td></td><td></td></tr><tr><td rowspan="2">7 × 7</td><td>1</td><td>13</td><td>0.315</td><td>0.251</td><td>Weibull</td><td>1.25</td><td>0.93</td></tr><tr><td>2</td><td>600</td><td>1.354</td><td>0.343</td><td>Normal</td><td></td><td></td></tr></tbody></table>

Analysis of the statistical data in Table 2 shows that most of the distribution functions of the first and second type of samples can be approximated by the Weibull distribution [30]

$$F(x) = 1 - \exp\left[-\left(\frac{b_m \cdot x}{x}\right)^m\right], \qquad (1)$$

where *m* and *b*<sub>*m*</sub> are, respectively, shape and scale parameters, and *x* is an argument (consistency index C.I.).

The parameters are calculated using a method of moments. One can also use tables [30] where the values *m* and *b*<sub>*m*</sub> are listed depending on a coefficient of variation *v*. For instance, for an expert sample of the first type (3 × 3 matrix) at *v* = 0.74, we will get *m* = 1.35, *b*<sub>*m*</sub> = 0.73. Then the probability of consistency at *z* = 0.05 and, consequently, C.I. = 0.026 will amount to

$$F(C.I. = 0.026) = 1 - \exp\left[-\left(\frac{0.73 \cdot 0.026}{0.050}\right)^{1.35}\right] = 0.237$$

Table 3 shows the results of the *F*(*C.I.*) distribution functions calculated for the first and second type samples.
<table><caption>**Table 3.** Probabilistic assessment of expert $F_1(\text{C.I.})$ and generated $F_2(\text{C.I.})$ matrices.</caption><thead><tr><th>Matrix</th><th>C.I.</th><th>C.R. ≤ z</th><th>$F_1(\text{C.I.})$</th><th>$F_2(\text{C.I.})$</th><th>$F_1(\text{C.I.})/F_2(\text{C.I.})$</th></tr></thead><tbody><tr><td rowspan="3">3 × 3</td><td>0.026*</td><td>0.050*</td><td>0.237</td><td>0.1500</td><td>1.58</td></tr><tr><td>0.052</td><td>0.100</td><td>0.600</td><td>0.2300</td><td>2.61</td></tr><tr><td>0.100</td><td>0.193</td><td>0.900</td><td>0.350</td><td>2.57</td></tr><tr><td rowspan="3">4 × 4</td><td>0.050</td><td>0.056</td><td>0.281</td><td>0.0140</td><td>20.07</td></tr><tr><td>0.072*</td><td>0.080*</td><td>0.313</td><td>0.0230</td><td>13.61</td></tr><tr><td>0.100</td><td>0.112</td><td>0.423</td><td>0.0350</td><td>12.09</td></tr><tr><td rowspan="2">5 × 5</td><td>0.110*</td><td>0.100*</td><td>0.070</td><td>0.0270</td><td>2.59</td></tr><tr><td>0.200</td><td>0.180</td><td>0.140</td><td>0.0400</td><td>3.50</td></tr><tr><td rowspan="2">6 × 6</td><td>0.125*</td><td>0.100*</td><td>0.101</td><td>0.0052</td><td>19.42</td></tr><tr><td>0.200</td><td>0.160</td><td>0.187</td><td>0.0080</td><td>23.37</td></tr><tr><td rowspan="2">7 × 7</td><td>0.135*</td><td>0.100*</td><td>0.270</td><td>0.0002</td><td>1350</td></tr><tr><td>0.200</td><td>0.149</td><td>0.420</td><td>0.0004</td><td>1050</td></tr></tbody></table>

*Extreme values of C.I. for this sample

An analysis of Table 3 (excluding 7×7 matrices due to the small sample size of the first type) shows that

* probability estimates $F_1(\text{C.I.})$ corresponding to the limit value of C.I. ranged from 0.07 (5×5 matrix) to 0.313 (4×4 matrix), i.e., more than 31% of the expert matrices are consistent;

* for the generated matrices, the $F_2(\text{C.I.})$ values ranged from 0.15 (3×3 matrix) to 0.0052 (6×6 matrix), i.e., almost 30 times lower;

* $F_1(\text{C.I.})/F_2(\text{C.I.})$ ratio varies widely from 1.58 (3×3 matrix) to 19.42 (6×6 matrix).

A possible reason for these contradictions is that the average C.I. of the generated matrices is chosen as the random consistency index R.I., which is generated based on a uniform distribution without observing the transitivity requirements between the row and column judgements. Therefore, making the correct decision regarding the limits of C.R. ≤ 0.1 values based on the generated matrices requires additional verification of the tangible matrices.

Let's consider a possible approach to solving this problem by comparing the areas of five geometric figures: a circle, triangle, square, diamond, and rectangle [31]. The priority vectors calculated based on expert opinion ($w_i$) and actual data on the areas of geometric figures ($w_{i\_ST}$) are shown in Table 4 [31].

We consulted three experts, the results of whose calculations are also shown in Table 4. In addition to these indicators, we calculated the average values of absolute deviations according to the formula

$$ \Delta = \frac{\sum_{i=1}^{n} |w_i - w_{i\_ST}|}{n}, \qquad (2) $$
where $w_i$ is the value of the i-th element of the priority vector obtained by the experts (priorities from comparisons); $w_{i\_ST}$ is the value of the i-th element of the priority vector obtained from actual data on the areas of geometric figures (actual relative size).

<table><caption>**Table 4.** Results of calculations of priority vectors $w_i$ and average values of absolute deviations $\Delta$.</caption><thead><tr><th rowspan="2">Figures</th><th rowspan="2">$w_{i\_ST}$</th><th colspan="4">Priority vector, $w_i$</th></tr><tr><th>Saaty [31],<br>C.R.=0.005</th><th>Expert 1,<br>C.R.=0.036</th><th>Expert 2,<br>C.R.=0.036</th><th>Expert 3,<br>C.R.=0.179</th></tr></thead><tbody><tr><td>Circle</td><td>0.471</td><td>0.457</td><td>0.485</td><td>0.555</td><td>0.539</td></tr><tr><td>Triangle</td><td>0.050</td><td>0.049</td><td>0.052</td><td>0.041</td><td>0.030</td></tr><tr><td>Square</td><td>0.234</td><td>0.257</td><td>0.233</td><td>0.185</td><td>0.227</td></tr><tr><td>Diamond</td><td>0.149</td><td>0.150</td><td>0.152</td><td>0.140</td><td>0.143</td></tr><tr><td>Rectangle</td><td>0.096</td><td>0.087</td><td>0.078</td><td>0.079</td><td>0.061</td></tr><tr><td>Average deviation, $\Delta$</td><td></td><td>0.0096</td><td>0.0076</td><td>0.034</td><td>0.027</td></tr></tbody></table>

The following conclusions can be drawn from the analysis of Table 4:

* when C.R. values are equal, the average values of absolute deviations of evaluations of the first and second experts differ by a factor of almost 4.5 ($\Delta = 0.0076$ and $\Delta = 0.034$);

* for the third expert (C.R. = 0.179), the average value of absolute deviation is $\Delta = 0.027$, which is less than the average value of the absolute deviation for the second expert ($\Delta = 0.034$), for whom C.R. = 0.036;

* the value of the absolute deviation of the estimates made by the expert [31] is $\Delta = 0.0096$, which is higher than the value for the first expert ($\Delta = 0.0076$).

To illustrate the proposed approach, let us consider a case where a manufacturing company is predicted to purchase 1,000 units of five component parts, the number of which is proportional to the actual data on the areas of geometric figures $w_{i\_ST}$ (Table 4). Given the uncertainty of demand, four experts, whose estimates are proportional to the priority vectors $w_i$ (Table 4), were engaged to clarify the volume of purchases of each of the nomenclatures.

The results of the calculation of the discrepancies of the expert and predicted values are given in Table 5. For example, Table 5 shows that if the procurement was based on the judgement of Expert 1, the total number of shortage and surplus units would be 38 units. Taking into account the €10 penalty for each deficit or surplus unit, the additional cost to the company would be €380. The corresponding calculations for the other experts are shown in Table 5, from which it follows that the forecast variant based on the priority vector of Expert 1 (C.R. = 0.036) is the preferable option.

The paradoxical nature of these results calls for many more example calculations for tangible objects and a search for new dependencies to estimate intangible objects based on artificial intelligence models and methods.
<table><caption>**Table 5.** Evaluating the results of a product purchase decision.</caption><thead><tr><th rowspan="2">Component parts</th><th rowspan="2">w<sub>ST</sub></th><th colspan="4">Discrepancies between predicted and expert estimates</th></tr><tr><th>Saaty [31], C.R.=0.005</th><th>Expert 1, C.R.=0.036</th><th>Expert 2, C.R.=0.036</th><th>Expert 3, C.R.=0.179</th></tr></thead><tbody><tr><td>Number 1</td><td>471</td><td>14</td><td>14</td><td>84</td><td>68</td></tr><tr><td>Number 2</td><td>50</td><td>1</td><td>2</td><td>9</td><td>20</td></tr><tr><td>Number 3</td><td>234</td><td>23</td><td>1</td><td>49</td><td>7</td></tr><tr><td>Number 4</td><td>149</td><td>1</td><td>3</td><td>9</td><td>6</td></tr><tr><td>Number 5</td><td>96</td><td>9</td><td>18</td><td>17</td><td>35</td></tr><tr><td>Discrepancies, in units</td><td></td><td>48</td><td>38</td><td>168</td><td>136</td></tr><tr><td>Discrepancies, in euros</td><td></td><td>480</td><td>380</td><td>1680</td><td>1360</td></tr></tbody></table>

## 3 Conclusion

A probabilistic approach for evaluating matrix consistency has been proposed, which, compared to the deterministic approach, allows establishing quantitative relationships between expert and generated matrices. Analysis of the results from processing 292 matrices shows that the probabilistic estimates of consistency indices (in the case of expert judgements) differ significantly from the probabilistic estimates of consistency indices for the generated values. For example, for 6×6 matrices, the estimates differ by a factor of 20. The results of our study do not confirm the need to revise expert judgements if C.R. is greater than 0.10.

The main directions for further research are the following:

1. The matrix generation algorithm should be refined with the introduction of artificial intelligence elements, which can be identified from the analysis of tangible object matrices.

2. Collection, systematisation, and statistical analysis of expert judgements should be continued in order to obtain representative samples of matrices. They could then be compared with artificial (generated) matrices, which would provide reliable multi-criteria estimates.

## Acknowledgments

The research described in this paper is partially supported by state research 0073-2019-0004.

## References

1. Ivanov, D., Sokolov, B.: Structure dynamics control approach to supply chain planning and adaptation. International Journal of Production Research 50(21), 6133-6149 (2012)
2. Dolgui, A., Ivanov, D., Sokolov, B.: Ripple effect in the supply chain: an analysis and recent literature. International Journal of Production Research 56(1-2), 414-430 (2018)

3. Felice, F.: Research and applications of AHP/ANP and MCDA for decision making in manufacturing. International Journal of Production Research 50(17), 4735-4737 (2012)

4. Csató, L.: A characterization of the Logarithmic Least Squares Method. European Journal of Operational Research 276, 212-216 (2019)

5. Sarraf, R., McGuire, M.: Integration and comparison of multi-criteria decision making methods in safe route planner. Expert Systems With Applications 154, 113399 (2020)

6. Singh, S., Singh, V.: Three-level AHP-based heuristic approach for a multi-objective facility layout problem. International Journal of Production Research 49(4), 1105-1125 (2011)

7. Maniya, K., Bhatt, M.: A multi-attribute selection of automated guided vehicle using the AHP/M-GRA technique. International Journal of Production Research 49(20), 6107-6124 (2011)

8. Dai, J., Blackhurst, J.: A four-phase AHP-QFD approach for supplier assessment: a sustainability perspective. International Journal of Production Research 50(19), 5474-5490 (2012)

9. Salgado, E., Salomon, V., Mello, C.: Analytic hierarchy prioritisation of new product development activities for electronics manufacturing. International Journal of Production Research 50(17), 4860-4866 (2012)

10. Azizi, M., Azizpour, M.: A BOCR structure for privatisation effective criteria of Iran newsprint paper industry. International Journal of Production Research 50(17), 4867-4876 (2012)

11. Varajão, J., Cruz-Cunha, M.: Using AHP and the IPMA Competence Baseline in the project managers selection process. International Journal of Production Research 51(11), 3342-3354 (2013)

12. Quezada, L., López-Ospina, H.: A method for designing a strategy map using AHP and linear programming. Int. J. Production Economics 158, 244-255 (2014)

13. Luthra, S., Mangla, S., Xu, L., Diabat, A.: Using AHP to evaluate barriers in adopting sustainable consumption and production initiatives in a supply chain. Int. J. Production Economics 181, 342-349 (2016)

14. Vieira, J., Toso, M., Silva, J., Ribeiro, P.: An AHP-based framework for logistics operations in distribution centres. International Journal of Production Economics 187, 246-259 (2017)

15. D'Inverno, G., Carosi, L., Romano, G., Guerrini, A.: Water pollution in wastewater treatment plants: An efficiency analysis with undesirable output. European Journal of Operational Research 269, 24-34 (2018)

16. Katiyar, R., Meena, P., Barua, M., Tibrewala, R., Kumar, G.: Impact of sustainability and manufacturing practices on supply chain performance: Findings from an emerging economy. International Journal of Production Economics 197, 303-316 (2018)

17. Awasthi, A., Govindan, A., Gold, S.: Multi-tier sustainable global supplier selection using a fuzzy AHP-VIKOR based approach. International Journal of Production Economics 195, 106-117 (2018)

18. Ishizaka, A., Lolli, F., Balugani, E., Cavallieri, R., Gamberini, R.: DEASort: Assigning items with data envelopment analysis in ABC classes. International Journal of Production Economics 199, 7-15 (2018)

19. Vega, D., Vieira, J., Toso, E., Faria, R.: A decision on the truckload and less-than-truckload problem: An approach based on MCDA. International Journal of Production Economics 195, 132-145 (2018)
20. Mastrocinque, E., Ramírez, F., Honrubia-Escribano, A., Pham, D.: An AHP-based multi-criteria model for sustainable supply chain development in the renewable energy sector. Expert Systems With Applications 150, 113321 (2020)

21. Falsini, D., Fondi, F., Schiraldi, M.: A logistics provider evaluation and selection methodology based on AHP, DEA and linear programming integration. International Journal of Production Research 50 (17), 4822–4829 (2012)

22. Ishizaka, A., Labib, A.: Review of the main developments in the analytic hierarchy process. Expert Systems with Applications 38, 14336–14345 (2011)

23. Ishizaka, A., Siraj, S.: Are multi-criteria decision-making tools useful? An experimental comparative study of three methods. European Journal of Operational Research 264, 462–471 (2018)

24. Chao, X., Kou, G., Li, T., Pen, Y.: Jie Ke versus AlphaGo: A ranking approach using decision making method for large-scale data with incomplete information. European Journal of Operational Research 265, 239–247 (2018)

25. Abastante, F., Corrente, S., Greco S., Ishizaka, A., Lami, I.: A new parsimonious AHP methodology: Assigning priorities to many objects by comparing pairwise few reference objects. Expert systems with applications 127, 109–120 (2019)

26. Brunelli, M.: A study on the anonymity of pairwise comparisons in group decision making. European Journal of Operational Research 279, 502–510 (2019)

27. Grzybowski, A., Starczewski, T.: New look at the inconsistency analysis in the pairwise-comparisons-based prioritization problems. Expert Systems with Applications 159, 113549 (2020)

28. Saaty, T.: The Analytic Hierarchy Process. McGraw-Hill, New York (1980)

29. Saaty, T.: Fundamentals of Decision Making and Priority Theory with the Analytical Hierarchy Process. RWS Publications, Pittsburgh, PA, USA (1994)

30. Weibull, W.: Fatigue testing and analysis of results. Pergamon press, Oxford, London (1961)

31. Saaty, T.: Neurons the decision makers, Part I: The firing function of a single neuron. Neural Networks 86, 102–114 (2017)