# Trilevel Multi-criteria Decision Analysis Based on Three-Way Decision

Chengjun Shi, Yiyu Yao

► To cite this version:

Chengjun Shi, Yiyu Yao. Trilevel Multi-criteria Decision Analysis Based on Three-Way Decision. 4th International Conference on Intelligence Science (ICIS), Feb 2021, Durgapur, India. pp.115-124, 10.1007/978-3-030-74826-5_10 . hal-03741709

HAL Id: hal-03741709

https://inria.hal.science/hal-03741709v1

Submitted on 1 Aug 2022

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Trilevel Multi-Criteria Decision Analysis based on Three-Way Decision

Chengjun Shi, Yiyu Yao

Department of Computer Science, University of Regina,
Regina, Saskatchewan, Canada S4S 0A2
Email: {csn838, Yiyu.Yao}@uregina.ca

**Abstract.** Based on the principles of three-way decision as thinking in threes, this paper proposes a framework of trilevel multi-criteria decision analysis (3LMCDA). The main motivations and contributions are a new understanding of multi-criteria decision analysis at three levels: 1) single-criterion-level analysis with respect to individual criteria, 2) multi-criteria-level aggregation by pooling results from many criteria, and 3) multi-methods-level ensemble of different multi-criteria decision analysis methods. A full understanding of multi-criteria decision analysis requires understandings at the three levels. While the existing studies have been focused on the first two levels, new efforts at the third level may enable us to take advantages of many different multi-criteria decision analysis methods.

**Keywords:** Three-way decision · Multi-criteria decision-making · Trilevel thinking · Trilevel analysis

## 1 Introduction

Multiple criteria decision making (MCDM), or multiple criteria decision analysis (MCDA), deals with the evaluation and comparison of a set of decision alternatives with respect to a set of multiple, possibly conflicting, criteria. A basic task is to sort or rank the set of alternatives according to the set of criteria [7, 9, 16, 17]. There are extensive studies on models and methods of MCDM, for example, Weighted Sum Model (WSM) [5], Weighted Product Model (WPM) [3,12], Analytic Hierarchy Process (AHP) [15], Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) [7], ELimination Et Choix Traduisant la REalité (ELECTRE) [2], and many more. A review of these studies brings our attention to two issues, which motivates the present study.

First, some studies (e.g., ELECTRE) start with constructing a set of rankings according to the evaluations of individual attributes and then build an aggregate ranking based on the set of rankings. On the other hand, the majority of studies (e.g., TOPSIS) directly construct a ranking based on an aggregate evaluation of the evaluations of the set of criteria. A lack of consideration of rankings according to individual criteria perhaps needs a closer examination, as the individual rankings reveal some kinds of local properties that may be useful in MCDA. Second,
many methods have been proposed and existing methods are continuously being refined in order to avoid some of their difficulties. Although there is no general agreement on the existence of the best or the most superior method, research efforts are constantly being made in search of such a method. It might be the time to think in a different direction. Instead of searching for new methods, we may search for ways to combine existing methods. Since each method produces one ranking, a set of methods produces a set of rankings. The set of rankings may be aggregated into one ranking, in the same way that the set of rankings obtained from individual criteria can be aggregated into one ranking.

The main objective of this paper is to propose a framework of MCDA to address the two issues. For this purpose, we make use of the principles and ideas of a theory of three-way decision (3WD). The theory of 3WD, proposed by Yao [20, 21, 23], is about thinking, working, and processing in threes. That is, we divide a whole into three parts and understand the whole by studying the three parts, as well as their interactions and dependencies. Trilevel thinking is a special mode of three-way decision [22], involving interpretations and understandings of a problem at three levels. To apply principles of trilevel thinking, we identify three levels in MCDA: 1) single-criterion analysis, 2) multi-criteria analysis, and 3) multi-methods analysis. This enables us to build a framework of trilevel multi-criteria decision analysis (3LMCDA), based on the results from three-way decision in general and trilevel analysis in specific.

There are several studies on combining three-way decision and MCDM. Jia and Liu [8] introduced a three-way decision model for solving MCDM problems. Based on the notions of loss functions, they provided algorithms for constructing three-way decision rules. Liu et al [10] proposed a new three-way decision model with intuitionistic fuzzy numbers to solve MCDM problems, in which they used a different kind of loss functions. Liu and Wang [11] suggested a three-way decision model with intuitionistic uncertain linguistic variables to solve multi-attribute group decision-making problem, which takes the advantages of intuitionistic fuzzy sets and uncertain linguistic variables. Zhan et al [25] applied three-way decision into outranking approach and built a new three-way multi-attribute decision making model. Wang et al [19] combined three-way decision and MCDM in a hesitant fuzzy information environment. Cui et al [4] discussed a method for deriving the attribute importance based on a combination of three-way decision and AHP method. In this paper, we take a different direction of research for applying three-way decision to MCDA by proposing a framework of trilevel MCDA (i.e., 3LMCDA). Instead of focusing on a particular method, we look at a conceptual model that enables us to unify existing studies in a common setting. In particular, we look at rankings generalized at three levels. The single-criterion level focuses on individual criteria locally, the multi-criteria level considers a set of criteria globally but from the angle of a particular MCDA method, the multi-methods level looks at MCDA from multiple angles given by different MCDA methods. The trilevel framework provides a much richer understanding than any of the three levels.
This paper is organized as follows. In Section 2, we review the basic concepts of three-way decision and trilevel thinking. In Section 3, we discuss a framework of trilevel multiple criteria decision analysis (3LMCDA). We briefly explain, through examples, some basic issues and tasks at each level. In Section 4, we summarize the key points of the new framework and point out future research topics.

## 2 Three-Way Decision and Trilevel Analysis

The fundamental philosophy of three-way decision is thinking in threes or triadic thinking [20-22]. A triad of three things is a basic pattern and structure used for understanding and problem solving [24]. Thinking in threes appears everywhere. It is flexible and widely applicable. We look at a problem based on its three components; we search for a full understanding of an information-processing system through three levels of understanding (i.e., theory, algorithms, and implementation); we build an explanation from three perspectives. By considering three things, we avoid the over-simplification of thinking in twos and, at the same time, the complication of thinking in fours or more.

Yao [21] proposed a trisecting-acting-outcome (TAO) model of three-way decision. The basic idea is to divide a whole into three parts and to approach the whole through the three parts. There are three related components of the TAO model. Trisecting is to construct a trisection from a whole, acting is to devise and apply strategies on the three parts of the trisection, and outcome is a desirable result from a combination of trisecting and acting. A useful class of trisections is the type of trisections consisting of three levels. It gives rise to thinking in three levels or trilevel thinking [22]. A triad of three levels normally consists of a top level, a middle level, and a bottom level. There are three basic tasks of trilevel thinking: (i) interpretation of the three levels, (ii) relationships between the three levels, and (iii) analysis to be done at each of the three levels, as well as at two adjacent levels and, simultaneously, at all three levels. Concrete examples of trilevel structures include three levels of abstraction, three levels of scope and scale, and three levels of complexity. In a trilevel abstraction, the top level is the most general level, the bottom level is the most detailed and specific one, and the middle level is between the two. Levels of scope and scale determine the different types of research questions. A lower level is at a narrower scope and scale and may focus on localized issue. A higher level deals with problems from a wider scope and scale and may focus on some global issues. Levels of complexity aims to decompose a problem into sub-problems with decreasing complexity. The solutions to or the results of a simpler sub-problem become supporting materials to the next higher level.

Typically, in a trilevel structure, a higher level guides and determines its lower level, and a lower level supports its higher level. There are three possible modes of processing of a trilevel, namely, top-down, bottom-up, and middle-out [22]. From the bottom to the top, details are extracted into abstract information. Redundant or less relevant information is removed. The problems become more
general and more abstract. The results obtained at a lower level provide hints and support for the work at a higher level. In the top-down direction, the procedure is reversed. Goals or tasks at a higher level control the concrete ideas at a lower level. In the middle-out mode, the middle level is guided by the top and is supported by the bottom at the same time. That is, the middle level is developed in the context of both the top and bottom levels.

The support-and-control relationships of the levels are useful for learning, understanding, and processing. Trilevel thinking and analysis are generally applicable to a wide range of domains. For any problem, once we can identify and separate three levels, we can immediately apply the results of trilevel analysis. In the rest of the paper, we will demonstrate this method by considering the problem of multi-criteria decision analysis.

# 3 A Trilevel Framework of Multi-Criteria Decision Making

We apply a trilevel thinking model to the problem of MCDM and construct a novel trilevel analysis framework in this section. The following three levels are reasonable to be considered: the bottom single-criterion level, the middle multi-criteria level, and the top multi-methods level. We start from the basic notions of MCDM problem, analyze it from three levels by reviewing existing studies, and discuss the significance and prospects of each level.

## 3.1 Trilevel thinking in multi-criteria decision analysis

A multi-criteria decision problem is a complex decision analysis problem. The main task is to find the optimal decision among a set of decision alternatives, or to rank the alternatives, by comparing their performances under a multiple criteria environment.

**Definition 1.** A multi-criteria decision table is a triplet $MCDT = \{A, C, p\}$, where A is a finite and non-empty set of $n$ alternatives $A = \{a_1, a_2, \dots, a_n\}$, C is a finite and non-empty set of $m$ criteria $C = \{c_1, c_2, \dots, c_m\}$, and $p$ is a real-valued mapping function, i.e., $p: A \times C \rightarrow \mathbb{R}$.

The performance function $p$ assigns a real number to each alternative on every criterion. For example, $a_i \in A$, $p_j(a_i)$ is a real number and indicates $a_i$'s performance on criterion $c_j$. An assumption is made that a larger real number reflects a better performance.

A trilevel framework (3LMCDA) is built as depicted in Figure 1. From the bottom to the top, the three levels are explained as follows:

(i) The single-criterion level ranks alternatives on a single criterion. It provides detailed information of alternatives on every single criterion. The local ranking of alternatives is important at this level. Existing studies at this level include outranking relations, utility-based preference, etc.
(ii) The multi-criteria level ranks alternatives on a set of multiple criteria. It considers globally the ranking or the ordering of alternatives according to their total performance. Relevant methods are weighted models, AHP, TOPSIS, ELECTRE, and many more.

(iii) The multi-methods level ranks alternatives from a set of multiple MCDM methods. It considers strengths and drawbacks of existing approaches at the middle level. The ultimate goal is to construct a ranking based on an ensemble of results from multiple MCDM methods.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Trilevel framework of MCDA (3LMCDA)</figcaption></figure>

The connections between levels are clearly shown in Figure 1. The bottom-up mode is denoted by the solid lines with arrows. The top-down mode is denoted by the dashed lines with arrows.

We use an example to explain the three levels in a bottom-up mode. Suppose that we have two alternatives $a_1, a_2 \in A$ and a set of criteria $C = \{c_1, c_2, c_3\}$, the problem is to rank them on the three criteria. A more specific sub-task is to compare them on each criterion individually. This is what the bottom single-criterion level focuses on. However, the results might not be consistent on the three criteria. A possible case is that $a_1$ is better than $a_2$ on $\{c_1, c_2\}$ but worse on $c_3$. A middle multi-criteria level serves to solve the conflicts by integrating results from the bottom level. This is usually done by a particular MCDM method. For example, ELECTRE is such a method that calculates several indices based on the comparison results at the single-criterion level. The role of the top multi-methods level is to rank based on an ensemble of results from a set of MCDM methods, taking into considerations of advantages and disadvantages of each method. The rankings produced by different MCDM methods are normally inconsistent. There is no agreement on which method produces the best ranking. Thus, multiple MCDM methods provide various angles to solve the ranking problem. To take advantages from a set of methods, an ensemble approach is needed.

The structure of the proposed trilevel analysis framework (3LMCDA) is consistent to the interpretations of trilevel thinking. It reflects three levels of ab-
straction, three levels of scope and scale, and three levels of complexity. In the following sections, we briefly outline several approaches with respect to the trilevel model.

## 3.2 Single-Criterion Level

The single-criterion level concentrates on ranking alternatives' performances locally based on one criterion. In the definition of a multi-criteria decision table, the assumption is that alternatives' performances are described in a numerical value. Given two alternatives $a_1, a_2 \in A$, there are three basic results from a comparison of $p_j(a_1)$ and $p_j(a_2)$: less than (<), equal to (=), and greater than (>).

The ranking of $a_1$ and $a_2$ is decided by the following rules. If $p_j(a_1) < p_j(a_2)$, then we say that $a_1$ is worse than $a_2$, denoted by $a_1 \prec_{c_j} a_2$; if $p_j(a_1) = p_j(a_2)$, then $a_1$ is considered as good as $a_2$ with respect to $c_j$, denoted by $a_1 \sim_{c_j} a_2$; if $p_j(a_1) > p_j(a_2)$, then we believe that $a_1$ performs better than $a_2$ on $c_j$, denoted by $a_1 \succ_{c_j} a_2$. For a better representation, we consider comparing $(p_j(a_1)-p_j(a_2))$ with 0 and the decision rules are given by:

$$ \left\{ \begin{array}{l} a_1 \prec_{c_j} a_2 \iff (p_j(a_1) - p_j(a_2)) < 0, \\ a_1 \sim_{c_j} a_2 \iff (p_j(a_1) - p_j(a_2)) = 0, \\ a_1 \succ_{c_j} a_2 \iff (p_j(a_1) - p_j(a_2)) > 0. \end{array} \right. \qquad (1) $$

In a decision-theoretic setting, it is possible to interpret the values $p_j(a_i)$ as utilities. The relations may be viewed as preference relations and indifference relations [6].

Roy proposed an outranking approach [14] to decide the relationship between two alternatives at the single-criterion level. The meaning of outranking is “at least as good as” or “not worse than”. Roy’s approach is built by introducing two thresholds and four binary relations. The outranking relation can be considered as an union of other three relations, namely, indifferent with, strictly preferred to, and hesitation between the former two.

**Definition 2.** Given a multi-criteria decision table, $c_j \in C$ is a criterion, $a_1, a_2 \in A$ are two alternatives. An outranking approach introduces two thresholds $\alpha_j$ and $\beta_j$ ($0 < \alpha_j < \beta_j$). Four binary relations are defined as follows:

* Outranking relation S: $a_1S_{c_j}a_2 \iff (p_j(a_1) - p_j(a_2)) \geq -\alpha_j$,

* Indifference relation I: $a_1I_{c_j}a_2 \iff |(p_j(a_1) - p_j(a_2))| \leq \alpha_j$,

* Strict preference relation P: $a_1P_{c_j}a_2 \iff (p_j(a_1) - p_j(a_2)) > \beta_j$,

* Hesitation relation Q: $a_1Q_{c_j}a_2 \iff \alpha_j < (p_j(a_1) - p_j(a_2)) < \beta_j$.

The binary relations in Definition 2 are defined based on one specific criterion and rely on the alternatives' values on the criterion. The outranking relation is equivalent to a union of the other three relations. That is, if $a_1S_{c_j}a_2$ holds, then one and only one of the three statements $a_1I_{c_j}a_2$, $a_1P_{c_j}a_2$, or $a_1Q_{c_j}a_2$ must hold.

The results of the single-criterion level will be aggregated and combined at the middle level.
### 3.3 Multi-Criteria Level

The middle level sets up a task to aggregate results from the bottom level and generate a global ranking of alternatives according to a set of criteria. The individual comparisons on each criterion at the bottom level make contribution to the middle level.

The weighted sum model is a classical way to synthesize information from a set of criteria. It considers the importance of criteria by introducing a weight vector $W = \{w_1, w_2, \dots, w_m\}$. The vector $W$ denotes the relative importance of the criteria set. The following requirements should be satisfied: (i) there is no negative importance: $\forall w_j \in W, w_j \ge 0$, (ii) the summation of all weights equals to 1: $\sum_{j=1}^{m} w_j = 1$. To compare two alternatives $a_1, a_2$, the difference is calculated by:

$$ \Delta^{\text{WSM}}(a_1, a_2) = \sum_{j=1}^{m} w_j \times (p_j(a_1) - p_j(a_2)). \qquad (2) $$

In Equation (2), $(p_j(a_1) - p_j(a_2))$ calculates the difference value between $a_1$ and $a_2$ on $c_j$, $w_j$ is used to produce a weighted difference value. All of the weighted differences are added together. We have the following decision rules at the multi-criteria level:

$$ \left\{ \begin{array}{l} a_1 \prec_C a_2 \iff \Delta^{\text{WSM}}(a_1, a_2) < 0, \\ a_1 \sim_C a_2 \iff \Delta^{\text{WSM}}(a_1, a_2) = 0, \\ a_1 \succ_C a_2 \iff \Delta^{\text{WSM}}(a_1, a_2) > 0. \end{array} \right. \qquad (3) $$

In general, we can also replace 0 by setting up a pair of thresholds.

We consider the ELECTRE method, as an example, to illustrate an aggregation method at the middle. ELECTRE calculates a concordance index and a discordance index to decide the ranking of $a_1$ and $a_2$ at the multi-criteria level. The concordance index measures how sufficiently the set of criteria supports “$a_1$ outranks $a_2$”, denoted by $con(a_1Sa_2)$, while the discordance index is determined by a specific criterion indicates the degree of opposition to “$a_1$ outranks $a_2$”, denoted by $disc(a_1Sa_2)$. When concordance degree is high enough and discordance degree is low enough at the same time, we say that the assertion $a_1Sa_2$ is validated.

In ELECTRE I, the concordance index is computed by the weights associated with the criteria on which $a_1$ outranks $a_2$ out of the total weights:

$$ \mathrm{con}(a_1Sa_2) = \frac{1}{\sum_{j=1}^{m} w_j} \sum_{j:a_1S_{c_j}a_2} w_j. \qquad (4) $$

If the weight vector $W$ satisfies $\sum_{j=1}^{m} w_j = 1$, then Equation (4) is equivalent to $con(a_1Sa_2) = \sum_{j:a_1S_{c_j}a_2} w_j$. The discordance index $disc(a_1Sa_2)$ is defined as:

$$ \mathrm{disc}(a_1Sa_2) = \max_{j=1,2,\dots,m} \{p_j(a_2) - p_j(a_1)\}. \qquad (5) $$
It takes the greatest one of the difference values $(p_j(a_2) - p_j(a_1))$ on all $c_j$. The discordance index indicates the utmost degree of opposition.

To test the concordance degree and discordance degree, two thresholds $\delta$ and $v$ are introduced. The threshold $\delta$ is a given concordance level and usually satisfies $\delta \in (0.5, 1]$. If $\text{con}(a_1Sa_2) \ge \delta$, we say that it passes the concordance test. Similarly, $v$ is a given veto threshold. If $\text{disc}(a_1Sa_2) \le v$, it passes the non-discordance test, because the discordant criterion is not powerful enough to override $a_1Sa_2$. The assertion $a_1Sa_2$ is valid if and only if both of the tests have been passed.

## 3.4 Multi-Methods Level

Many MCDA methods have been proposed and applied in various domains and have been verified to be practically useful and effective. At the same time, most of them put their efforts at the middle level. Several studies [1, 16–18] reviewed the main ideas of common methods and summarized their relative advantages and disadvantages. These MCDA methods rank the set of alternatives in numerous and different ways. This may impose a difficulty for a decision maker when selecting a particular method among them.

At the top level, the idea of ensemble is a process of learning from a set of multiple methods and improving the model quality by considering multiple points of view. We simply discuss one of the possible ways. Given two alternatives $a_1, a_2$, we consider $a_1 \succ a_2$ to be a hypothesis and treat a MCDA method as an agent who votes on the hypothesis. A three-way voting decision is defined as follows.

**Definition 3.** Given a multi-criteria decision table, a finite set of MCDM methods is denoted by $T$. In order to determine whether $a_1$ is ranked ahead of $a_2$, a hypothesis is written as $h: a_1 \succ a_2$. Each method $t_k \in T$ makes one and only one of the three voting decisions according to the rules:

*Support:* $t_k$ admits that $a_1 \succ a_2$,

*Neutral:* $t_k$ neither admits $a_1 \succ a_2$ nor admits $a_1 \prec a_2$,

*Oppose:* $t_k$ admits $a_1 \prec a_2$.

Aggregating the voting results from the set of methods may help us rank the given pairs of alternatives.

Mohammadi and Rezaei [13] considered the ensemble problem from another angle. They proposed a novel approach to producing an ensemble ranking by modeling the problem of aggregating rankings as a new multi-criteria decision-making. In their approach, each method is treated as a criterion and the weight of it is determined by the half-quadratic theory.

# 4 Conclusion and Future Work

Trilevel thinking, namely, a special mode of three-way decision, is a powerful idea for intelligent data analysis. Inspired by the principles of trilevel thinking, we have proposed a framework of trilevel multi-criteria decision analysis
(3LMCDA). The 3LMCDA consists of three levels: a single-criterion level, a multi-criteria level, and a multi-methods level. Each of the three levels considers a different type of questions and contributes uniquely to MCDA. The single-criterion level analysis ranks alternatives based on individual criteria, the multi-criteria level aggregates the set of rankings from individual criteria into one ranking (which is typically given by one MCDA), and multi-methods level is an ensemble of multiple different MCDA methods. As we move up, we gradually take a more global view, namely, single criterion, multiple criteria and single method, and multiple criteria and multiple methods. This new view, offered by 3LMCDA, may be worthy of further research efforts.

The conceptual 3LMCDA model offers many possible future research opportunities. By following the same idea, it is possible to look at other trilevel models. With the 3LMCDA model, one may unify results from existing studies. It may also be necessary to examine detailed interpretations of MCDA methods and experimental implementations within the 3LMCDA model, in order to demonstrate its practical values. New methods for combining three-way decision and multi-criteria decision-making may be investigated in light of 3LMCDA.

## Acknowledgement

We would like to thank Professor Mihir Chakraborty and Professor Zhongzhi Shi for encouraging us to write this paper. We thank reviewers for their constructive and valuable comments. This work was partially supported by an NSERC (Canada) Discovery Grant.

## References

1. Aruldoss, M., Lakshmi, T.M., Venkatesan V.P.: A survey on multi criteria decision making methods and its applications. American Journal of Information Systems 1, 31-43 (2013).

2. Benayoun, R., Roy, B., Sussman, B.: ELECTRE: Une méthode pour guider le choix en présence de points de vue multiples. Note de travail 49, SEMA-METRA International, Direction Scientifique (1966).

3. Bridgman, P.W.: Dimensional analysis. Yale University Press. New Haven, CT (1922).

4. Cui, X., Yao, J.T., Yao, Y.Y.: Modeling user-oriented attribute importance with the three-way decision theory. In: Bello, R., Miao, D., Falcon, R., Nakata, M., Rosete, A., Ciucci, D. (eds) Rough Sets. IJCRS 2020. LNCS, vol. 12179, pp. 122-136. Springer, Cham (2020).

5. Fishburn, P.C.: Additive utilities with incomplete product sets: Applications to priorities and assignments. Operations Research Society of America 15, 537-542 (1967).

6. Fishburn, P.C.: Nonlinear preference and utility theory. Baltimore: Johns Hopkins University Press (1988).

7. Hwang, C.L., Yoon, K.: Methods for multiple attribute decision making. In: Multiple Attribute Decision Making. LNEMS, vol. 186, pp. 58-191. Springer, Berlin, Heidelberg (1981).
8. Jia, F., Liu, P.D.: A novel three-way decision model under multiple-criteria environment. Information Sciences 471, 29-51 (2019).

9. Kaliszewski, I., Miroforidis, J., Podkopaev, D.: Interactive multiple criteria decision making based on preference driven evolutionary multiobjective optimization with controllable accuracy. European Journal of Operational Research 216, 188-199 (2012).

10. Liu, P.D., Wang, Y.M., Jia, F., Fujita, H.: A multiple attribute decision making three-way model for intuitionistic fuzzy numbers. International Journal of Approximate Reasoning 119, 177-203 (2020).

11. Liu, P.D., Wang, H.Y.: Three-way decisions with intuitionistic uncertain linguistic decision-theoretic rough sets based on generalized Maclaurin symmetric mean operators. International Journal of Fuzzy Systems 22, 653-667 (2020).

12. Miller, D.W., Starr, M.K.: Executive decisions and operations research. Prentice-Hall, Inc. Englewood Cliffs, NJ, USA (1969).

13. Mohammadi, M., Rezaei, J.: Ensemble ranking: aggregation of rankings produced by different multi-criteria decision-making methods. Omega 96, 102254 (2020).

14. Roy, B.: The outranking approach and the foundations of ELECTRE methods. In: Bana e Costa, C.A. (eds) Readings in Multiple Criteria Decision Aid, pp. 155-183. Springer, Berlin, Heidelberg (1990).

15. Saaty, T.L.: What is the analytic hierarchy process? In: Mitra, G., Greenberg, H.J., Lootsma, F.A., Rijkaert, M.J., Zimmermann, H.J. (eds) Mathematical Models for Decision Support. NATO ASI Series (Series F: Computers and Systems Sciences), vol 48, pp. 109-121. Springer, Berlin, Heidelberg (1988).

16. Triantphyllou, E.: Multi-criteria Decision Making Methods: A Comparative Study. Springer, Boston, MA (2000).

17. Umm-e-Habiba, Asghar, S.: A survey on multi-criteria decision making approaches. In: International Conference on Emerging Technologies, ICET, pp. 321-325 (2009).

18. Velasquez, M., Hester, P.T.: An analysis of multi-criteria decision making methods. International journal of operations research 10, 56-66 (2013).

19. Wang, J.J, Ma, X.L., Xu, Z.S., Zhan, J.M.: Three-way multi-attribute decision making under hesitant fuzzy environments. Information Sciences 552, 328-351 (2021).

20. Yao, Y.Y.: Three-way decisions and cognitive computing. Cognitive Computation 8, 543-554 (2016).

21. Yao, Y.Y.: Three-way decision and granular computing. International Journal of Approximate Reasoning 103, 107-123 (2018).

22. Yao, Y.Y.: Tri-level thinking: models of three-way decision. International Journal of Machine Learning and Cybernetic 11, 947-959 (2019).

23. Yao, Y.Y.: Set-theoretic models of three-way decision. Granular Computing (2020).

24. Yao, Y.Y.: The geometry of three-way decision. Applied Intelligence (2021), https://doi.org/10.1007/s10489-020-02142-z.

25. Zhan, J.M., Jiang, H.B., Yao, Y.Y.: Three-way multi-attribute decision-making based on outranking relations. IEEE Transactions on Fuzzy Systems (2020) DOI: 10.1109/TFUZZ.2020.3007423