# Using a Combined Intuitionistic Fuzzy Set-TOPSIS Method for Evaluating Project and Portfolio Management Information Systems

Vassilis C. Gerogiannis, Panos Fitsilis, Achilles D. Kameas

► To cite this version:

Vassilis C. Gerogiannis, Panos Fitsilis, Achilles D. Kameas. Using a Combined Intuitionistic Fuzzy Set-TOPSIS Method for Evaluating Project and Portfolio Management Information Systems. 12th Engineering Applications of Neural Networks (EANN 2011) and 7th Artificial Intelligence Applications and Innovations (AIAI), Sep 2011, Corfu, Greece. pp.67-81, 10.1007/978-3-642-23960-1_9 . hal-01571455

HAL Id: hal-01571455

https://inria.hal.science/hal-01571455

Submitted on 2 Aug 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Using a Combined Intuitionistic Fuzzy Set-TOPSIS
Method for Evaluating Project and Portfolio
Management Information Systems

Vassilis C. Gerogiannis¹,², Panos Fitsilis¹,², and Achilles D. Kameas²

¹ Project Management Department, Technological Education Institute of Larissa, 41110, Larissa, Hellas

{gerogian, fitsilis}@teilar.gr

² Hellenic Open University, 167 R. Feraiou & Tsamadou str, 26222, Patras, Hellas
kameas@eap.gr

**Abstract.** Contemporary Project and Portfolio Management Information Systems (PPMIS) have embarked from single-user, single-project management systems to web-based, collaborative, multi-project, multi-functional information systems which offer organization-wide management support. The variety of offered functionalities along with the variation among each organization needs and the plethora of PPMIS available in the market, make the selection of a proper PPMIS a difficult, multi-criteria decision problem. The problem complexity is further augmented since the multi stakeholders involved cannot often rate precisely their preferences and the performance of candidate PPMIS on them. To meet these challenges, this paper presents a PPMIS selection/evaluation approach that combines TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) with intuitionistic fuzzy group decision making. The approach considers the vagueness of evaluators’ assessments when comparing PPMIS and the uncertainty of users to judge their needs.

**Keywords:** Project and Portfolio Management Information Systems, Multi-Criteria Decision Making, Group Decision Making, TOPSIS, Intuitionistic Fuzzy Sets.

# 1 Introduction

Research studies [1] present that increasing organizational requirements for the management of the entire life-cycle of complex projects, programs and portfolios motivate the further exploitation of Project and Portfolio Management Information System (PPMIS) from enterprises of any type and size. PPMIS have embarked from stand-alone, single-user, single-project management systems to multi-user, multi-functional, collaborative, web-based and enterprise-wide software tools which offer integrated project, program and portfolio management solutions, not limited to scope, budget and time management/control. Modern PPMIS support, through a range of
features, processes in all knowledge areas of the “Project Management Body of Knowledge” [2], by covering an expansive view of the “integration management” area that includes alignment and control of multi-project programs and portfolios. The market of PPMIS is rapidly growing and includes many commercial software products offering a number of functionalities such as time, resource and cost management, reporting features and support for change, risk, communication, contract and stakeholder management. Interested readers are referred to [3] where detailed information is given for 24 commercial leading PPMIS.

This variety of offered functionalities along with the variation among each organization needs and the plethora of PPMIS in the market, make their evaluation a complicate multi-criteria decision problem. The problem is often approached in practice by ad hoc procedures based only on personal preferences of users or any marketing information available [3], [4]. Such an approach may lead to a final selection that does not reflect the organization needs or, even worse, to an unsuitable PPMIS. Therefore, the use of a robust method from the multi-criteria decision making (MCDM) domain can be useful to support PPMIS selection. Review studies [5], [6], [7] reveal that the Analytic Hierarchy Process method (AHP) and its extensions have been widely and successfully used in evaluating several types of software packages (e.g., MRP/ERP systems, simulation software, CAD systems and knowledge management systems).

Although AHP presents wide applicability in evaluating various software products, little work has been done in the field of evaluating PPMIS. For example, in [8] the authors admit that their work is rather indicative with main objective to expose a representative case for illustrating the PPMIS selection process and not to create a definitive set of criteria that should be taken into account in practice. This lack of applicability of AHP in the PPMIS selection problem domain can be attributed to the fact that, despite its advantages, the method main limitation is the large number of pairwise comparisons required. The time needed for comparisons increases geometrically with the increase of criteria and alternatives involved, making AHP application practically prohibitive for complicate decisions, such as the selection of a PPMIS. As a response to this problem, in the recent past [9], we presented an approach for evaluating alternative PPMIS that combines group-based AHP with a simple scoring model. This group-AHP scoring model, although practical and easy to use, does not consider the vagueness or even the unawareness of users, when they evaluate their preferences from a PPMIS by rating their requirements. Also the approach does not take into account the uncertainty of experts, when they judge the performance of alternative PPMIS on the selected criteria, expressed as user requirements.

These ambiguities in evaluation of software systems and the incomplete available information expose the need to adopt a fuzzy-based approach [10]. Fuzzy-based methods provide the intuitive advantage to utilize, instead of crisp values, linguistic terms to evaluate performance of the alternatives and criteria weights. A fuzzy-based approach can be even more beneficial when it is combined with other decision making techniques. For example, fuzzy AHP [11] is proposed to handle the inherent imprecision in the pairwise comparison process, while fuzzy TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) [12] can be used to jointly consider both positive (benefit/functional oriented) and negative (cost/effort oriented) selection
criteria. Fuzzy-based MCDM techniques have been used to select various types of
software products (see, for example, [10], [13], [14]), but in the relevant literature
there is lack of a structured fuzzy-based approach for the selection of PPMIS under
uncertain knowledge.

The main objective of this paper is to present such an approach that involves both
users and evaluators (decision makers) in the decision making process and tries to
exploit the interest/expertise of each one in order to strengthen the final evaluation
results. This is achieved by aggregating all weights of criteria (requirements) and all
ratings of performance of the alternative systems, as they are expressed, by individual
stakeholders, in linguistic terms. The approach is based on intuitionistic fuzzy sets, an
extension of fuzzy sets proposed by Atanassov [15], which has been applied in a
variety of decision making problems [16], [17], [18], [19], [20]. An intuitionistic
fuzzy set includes the membership and the non-membership function of an element to
a set as well as a third function that is called the hesitation degree. This third function
is useful to express the lack of knowledge and the hesitancy concerning both
membership and non-membership of an element to a set. Expression of hesitation is
particularly helpful for both decision makers and users when they select a software
product for an organization such as, in our case, a PPMIS. On one hand, decision
makers often cannot have a full knowledge upon all functionalities included in the
newest version of each candidate system. Thus, they base their ratings only on
experience from using previous system versions as well by referencing to system
assessments which can be found in products survey reports. On the other hand, users
are often unfamiliar with how a PPMIS can support project management processes
tasks and, therefore, cannot precisely express which tasks require more to be
supported by a PPMIS. The presented approach mainly utilized the method proposed
in [18] which combines intuitionistic fuzzy sets with TOPSIS for supporting the
supplier selection problem. The advantage of this combination in case of PPMIS
evaluation is that we can distinguish between benefit criteria (e.g.,
functionalities/tasks supported by the PPMIS) and cost criteria (e.g., effort for system
customisation and price for ownership). The PPMIS that is closest to the positive to
ideal solution and most far from the negative ideal solution could be probably the
most appropriate PPMIS to cover the organization needs.

The rest of the paper is organized as follows. In section 2, we present the aspects
of the PPMIS evaluation problem and we justify how, in our case study, the PPMIS
selection criteria were determined. In section 3, we briefly discuss the basic concepts
of intuitionistic fuzzy sets. Section 4 presents the detailed description of the approach
and Section 5 presents the conclusions and future work.

## 2 Evaluation of PPMIS and Selection Criteria

Empirical studies [21] demonstrate that a number of project managers indicate a
strong impact of PPMIS upon successful implementation of their projects, while
others do not. The PPMIS selection process is usually supported by referencing to
market surveys [3], [4], [22] or by considering the users' perceptions and satisfaction
from a PPMIS usage [23]. Detailed assistance in evaluating PPMIS is provided by
evaluation frameworks which propose to consider an extensive list of system characteristics. These characteristics can be either functional or process oriented selection criteria. NASA, for example, in the past has convened a working group to evaluate alternative PPMIS for NASA's departments, upon a number of functional requirements. In the group's report [24] thirteen clusters of functional requirements are identified. Each cluster further includes a set of functional features and, in total, 104 functional criteria are identified to be evaluated. This vast number of criteria prevents decision makers from using a typical hierarchical MCDM approach like AHP.

As far as process oriented evaluation is concerned, evaluators may utilize as reference the set of criteria offered by a conceptual software architecture for PPMIS, like, for example, is the M-Model [25]. The M-Model was used in [3] to evaluate commercial PPMIS according to project phases/tasks and corresponding required functionality (Table 1). Each PPMIS was evaluated according to the extent that it offers the required functionality and the overall support for the corresponding project phase/task was specified with a “4-stars” score.

<table><caption>**Table 1.** Evaluation criteria (source: [3])</caption><thead><tr><td>Phase/Task</td><td>Required Functionality</td></tr></thead><tbody><tr><td>1. Idea Generation / Lead Management (IGLM)</td><td>Creativity Techniques, Idea / Project Classification, Lead Management (Mgmt.), Project Status / Project Process Mgmt.</td></tr><tr><td>2. Idea Evaluation (IE)</td><td>Estimation of Effort, Resource Needs Specification, Risk Estimation, Profitability Analysis, Project Budgeting, Offer Mgmt.</td></tr><tr><td>3. Portfolio Planning (PP1)</td><td>Organizational Budgeting, Project Assessment, Project Portfolio Optimization, Project Portfolio Configuration</td></tr><tr><td>4. Program Planning (PP2)</td><td>Project Templates, Resource Master Data, Resource Assignment Workflow, Resource Allocation</td></tr><tr><td>5. Project Planning (PP3)</td><td>Work Breakdown Structure Planning, Scope/ Product Planning, Network Planning, Scheduling, Resource Leveling, Risk Planning, Cost Planning</td></tr><tr><td>6. Project Controlling (PC1)</td><td>Change Request Mgmt., (Travel) Expense Mgmt., Timesheet, Cost Controlling, Meeting Support</td></tr><tr><td>7. Program Controlling (PC2)</td><td>Status Reporting, Deviation / Earned Value Analysis, Quality Controlling, Versioning, Milestone Controlling</td></tr><tr><td>8. Portfolio Controlling (PC3)</td><td>Performance Measurement, Dashboard, Organizational Budget Controlling</td></tr><tr><td>9. Program Termination (PT1)</td><td>Knowledge Portal, Competence Database / Yellow Pages, Project Archiving, Searching</td></tr><tr><td>10. Project Termination (PT2)</td><td>Invoicing, Document Mgmt., Supplier & Claim Mgmt.</td></tr><tr><td>11. Administration/Configuration (AC)</td><td>Workflow Mgmt., Access Control, Report Development, Form Development, User-Defined Data Structures, MS Office Project Interface, Application Programming Interface, Offline Usage</td></tr></tbody></table>
### 3 Basic Concepts of Intuitionistic Fuzzy Sets

Before proceeding to describe how the PPMIS selection problem was tackled, we briefly introduce some necessary introductory concepts of intuitionistic fuzzy sets (IFS). An IFS A in a finite set X can be defined as [15]:

$$A = \{< x, \mu_A(x), v_A(x) > | x \in X\}$$

where $\mu_A : X \to [0,1]$, $v_A : X \to [0,1]$ and $0 \le \mu_A(x) + v_A(x) \le 1 \quad \forall x \in X$. $\mu_A(x)$ and $v_A(x)$ denote respectively the degree of membership and non-membership of x to A. For each IFS A in X, $\pi_A(x) = 1 - \mu_A(x) - v_A(x)$ is called the hesitation degree of whether x belongs to A. If the hesitation degree is small then knowledge whether x belongs to A is more certain, while if it is great then knowledge on that is more uncertain. Thus, an ordinary fuzzy set can be written as:

$$\{< x, \mu_A(x), 1 - \mu_A(x) > | x \in X\}$$

In the approach we will use linguistic terms to express: i) the importance of decision stakeholders (users/decision makers), ii) judgements of decision makers on the performance of each PPMIS and iii) perceptions of users on the importance of each selection criterion. These linguistic terms can be transformed into intuitionistic fuzzy numbers (IFNs) in the form of [$\mu(x)$, $v(x)$]. For example, an IFN [0.50,0.45] represents membership $\mu = 0.5$, non-membership $v = 0.45$ and hesitation degree $\pi = 0.05$. In the approach we will also use addition and multiplication operators of IFNs. Let $a1 = (\mu_{a1}, v_{a1})$ and $a2 = (\mu_{a2}, v_{a2})$ be two IFNs. Then these operators can be defined as follows [15], [27], [28]:

$$a1 \oplus a2 = (\mu_{a1} + \mu_{a2} - \mu_{a1} \cdot \mu_{a2}, v_{a1} \cdot v_{a2})$$

$$a1 \otimes a2 = (\mu_{a1} \cdot \mu_{a2}, v_{a1} + v_{a2} - v_{a1} \cdot v_{a2})$$

$$\lambda \cdot a1 = (1 - (1 - \mu_{a1})^{\lambda}, v_{a1}^{\lambda}), \quad \lambda > 0$$

### 4 Evaluation of PPMIS with Intuitionistic Fuzzy Sets and TOPSIS

In this section we will describe how an intuitionistic fuzzy MCDM method was applied with the overall goal to select the most appropriate PPMIS system for the Hellenic Open University (HOU) (www.eap.gr). HOU is a university that undertakes various types of national and international R&D projects and programs, particularly in the field of lifelong adult education. The university does not maintain an integrated project/portfolio management infrastructure. In order to increase project management effectiveness and productivity, the management of HOU has decided to investigate the adoption of a collaborative PPMIS. The Department of Project Management (DPM) at the Technological Education Institute of Larissa in Greece was appointed to
act as an expert and aid this decision making process. Three experts $D_1, D_2$ and $D_3$ (decision makers) from DPM, with an average of seven years teaching/professional experience in using PPMIS, were involved in this process, aiming to identify HOU requirements from a PPMIS and to select an appropriate system that will cover these requirements. Three project officers/managers $U_1, U_2$ and $U_3$ (users) from the HOU site were also involved in the decision making. These persons have high expertise in contract management, multi-project coordination and planning of R&D projects and portfolios, but they present low experience in systematically using PPMIS. It should be mentioned here that the presented approach mainly utilized the method presented in [18] to handle a hypothetical supplier selection problem with five alternative suppliers and four selection criteria. In the current paper we validate the method in an actual context and we show its applicability with an extensive set of selection criteria. In addition, we show how sensitivity analysis can be applied to evaluate the influence of criteria weights on the final selection results.

The application of the approach for selecting an appropriate PPMIS for the case organization has been conducted in eight steps presented as follows.

**Step 1: Determine the weight of importance of decision makers and users.** In this first step, the expertise of both decision makers and users was analysed by specifying corresponding weights. In a joint meeting, the three decision makers $D_1, D_2, D_3$ agreed to qualify their experience in using PPMIS as “Master”, “Proficient” and “Expert”, respectively. The three users $U_1, U_2, U_3$ also agreed that their level of expertise in managing large projects can be characterized as “Master”, “Proficient” and “Expert”, respectively. These linguistic terms were assigned to IFNs by using the relationships presented in Table 2 between values in column 1 and values in column 3. In general, if there are $l$ stakeholders in the decision process, each one with a level of expertise rated equal to the IFN [$\mu_k, v_k, \pi_k$], the weight of importance of $k$ stakeholder can be calculated as [18]:

$$ \lambda_k = \frac{\left(\mu_k + \pi_k\left(\frac{\mu_k}{\mu_k + v_k}\right)\right)}{\sum_{k=1}^{l}\left(\mu_k + \pi_k\left(\frac{\mu_k}{\mu_k + v_k}\right)\right)} \quad (1) $$

where $\lambda_k \in [0,1]$ and $\sum_{k=1}^{l} \lambda_k = 1$.

<table><caption>**Table 2.** Linguistic terms for the importance of stakeholders and criteria</caption><thead><tr><th>Level of Stakeholder Expertise (1)</th><th>Importance of Selection Criteria (2)</th><th>IFN (3)</th></tr></thead><tbody><tr><td>Master</td><td>Very Important (VI)</td><td>[0.90,0.10]</td></tr><tr><td>Expert</td><td>Important (I)</td><td>[0.75,0.20]</td></tr><tr><td>Proficient</td><td>Medium (M)</td><td>[0.50,0.45]</td></tr><tr><td>Practitioner</td><td>Unimportant (U)</td><td>[0.35,0.60]</td></tr><tr><td>Beginner</td><td>Very Unimportant (VU)</td><td>[0.10,0.90]</td></tr></tbody></table>
According to eq. (1), the weights of decision makers were calculated as follows: $\lambda_{D1} = 0.406$, $\lambda_{D2} = 0.238$, $\lambda_{D3} = 0.356$. Since users were assigned to the same linguistic values, their weights were respectively the same: $\lambda_{U1} = 0.406$, $\lambda_{U2} = 0.238$, $\lambda_{U3} = 0.356$.

### Step 2: Determine the level of support provided by each alternative PPMIS.

Though there is a large number of available PPMIS, decision makers were queried to express their general opinion on ten commercial PPMIS which in market survey results [4] are characterised as leaders and challengers in this segment of enterprise software market. Five from these systems were excluded for two reasons: because they do not have presence in the national market and because decision makers were persuaded that their usage was inappropriate for the specific case, mainly due to lack of technical support and non-availability of training services. This first-level screening resulted in a list of five powerful PPMIS with strong presence (i.e., technical/training support) in the national market. For confidentiality reasons and aiming at avoiding the promotion of any software package, we will refer to these PPMIS as $A_1, A_2, A_3, A_4$ and $A_5$.

<table><caption>**Table 3.** Linguistic terms for rating the performance of PPMIS</caption><thead><tr><td>Level of Performance/Support</td><td>IFN</td></tr></thead><tbody><tr><td>Extremely high (EH)</td><td>[1.00,0.00]</td></tr><tr><td>Very very high (VVH)</td><td>[0.90,0.10]</td></tr><tr><td>Very high (VH)</td><td>[0.80,0.10]</td></tr><tr><td>High (H)</td><td>[0.70,0.20]</td></tr><tr><td>Medium high (MH)</td><td>[0.60,0.30]</td></tr><tr><td>Medium (M)</td><td>[0.50,0.40]</td></tr><tr><td>Medium low (ML)</td><td>[0.40,0.50]</td></tr><tr><td>Low (L)</td><td>[0.25,0.60]</td></tr><tr><td>Very low (VL)</td><td>[0.10,0.75]</td></tr><tr><td>Very very low (VVL)</td><td>[0.10,0.90]</td></tr></tbody></table>

In order to evaluate the candidate PPMIS in a manageable and reliable way, decision makers rated the performance of each system with respect to the criteria previously identified. Each decision maker was asked to carefully rate the support provided by each system on each of the 11 criteria (project phases/tasks) presented in Table 1. In addition to these 11 “positive” (benefit oriented) criteria, two “negative” (cost oriented criteria) was decided to be included in the list. These are the total price for purchasing/ownership (PO) and the effort required to customise/configure the PPMIS (CC). Thus, 13 criteria in total were adopted. All decision makers provided a short written justification for every rating they gave in linguistic terms. For ratings they used the linguistic terms presented in Table 3. Decision makers were also asked to cross-check their marks, according to the corresponding “4-stars” scores, as they are listed for each tool in [3]. Due to space limits, Table 4 presents ratings given by the three decision makers to the five PPMIS for the first three of the 13 criteria.

Based on these ratings and the weights of decision makers, the aggregated intuitionistic fuzzy decision matrix (AIFDM) was calculated by applying the intuitionistic fuzzy weighted averaging (IFWA) operator [28]. The basic steps of the
IFWA operator are that it first weights all given IFNs by a normalized weight vector, and then aggregates these weighted IFNs by addition. Each result derived by using the IFWA operator is an IFN. If $A = \{A_1, A_2, ..., A_m\}$ is the set of alternatives and $X = \{X_1, X_2, ..., X_n\}$ is the set of criteria, then AIFDM $R$ is an $m \times n$ matrix with elements IFNs in the form of $r_{ij} = [\mu_{A_i}(x_j), v_{A_i}(x_j), \pi_{A_i}(x_j)]$, where $i = 1, 2, ..., m$ and $j = 1, 2, ..., n$. By considering weights $\lambda_k (k = 1, 2, ..., l)$ of $l$ decision makers, elements $r_{ij}$ of the AIFDM can be calculated using IFWA as follows:

$$r_{ij} = \text{IFWA}_{\lambda}(r_{ij}^{(1)}, r_{ij}^{(2)}, ..., r_{ij}^{(l)}) = \lambda_1 r_{ij}^{(1)} \oplus \lambda_2 r_{ij}^{(2)} \oplus \lambda_3 r_{ij}^{(3)} \oplus \dots \oplus \lambda_l r_{ij}^{(l)} \quad (2)$$
$$= \left[ 1 - \prod_{k=1}^{l} (1 - \mu_{ij}^{(k)})^{\lambda_k}, \prod_{k=1}^{l} (v_{ij}^{(k)})^{\lambda_k}, \prod_{k=1}^{l} (1 - \mu_{ij}^{(k)})^{\lambda_k} - \prod_{k=1}^{l} (v_{ij}^{(k)})^{\lambda_k} \right]$$

<table><caption>**Table 4.** The ratings of the alternative PPMIS (excerpt)</caption><thead><tr><th rowspan="2">Criteria</th><th rowspan="2">Decision Makers</th><th colspan="5">PPMIS</th></tr><tr><th>A<sub>1</sub></th><th>A<sub>2</sub></th><th>A<sub>3</sub></th><th>A<sub>4</sub></th><th>A<sub>5</sub></th></tr></thead><tbody><tr><td rowspan="3">IGLM</td><td>D<sub>1</sub></td><td>VH</td><td>VH</td><td>H</td><td>MH</td><td>H</td></tr><tr><td>D<sub>2</sub></td><td>H</td><td>VH</td><td>MH</td><td>H</td><td>H</td></tr><tr><td>D<sub>3</sub></td><td>H</td><td>H</td><td>H</td><td>H</td><td>MH</td></tr><tr><td rowspan="3">IE</td><td>D<sub>1</sub></td><td>H</td><td>M</td><td>VH</td><td>M</td><td>M</td></tr><tr><td>D<sub>2</sub></td><td>MH</td><td>M</td><td>H</td><td>H</td><td>H</td></tr><tr><td>D<sub>3</sub></td><td>M</td><td>MH</td><td>H</td><td>MH</td><td>H</td></tr><tr><td rowspan="3">PP1</td><td>D<sub>1</sub></td><td>MH</td><td>H</td><td>VVH</td><td>VH</td><td>VH</td></tr><tr><td>D<sub>2</sub></td><td>MH</td><td>MH</td><td>VH</td><td>MH</td><td>VH</td></tr><tr><td>D<sub>3</sub></td><td>MH</td><td>MH</td><td>H</td><td>H</td><td>VH</td></tr></tbody></table>

<table><caption>**Table 5.** Aggregated intuitionistic fuzzy decision matrix (excerpt)</caption><thead><tr><th></th><th>A<sub>1</sub></th><th>A<sub>2</sub></th><th>A<sub>3</sub></th><th>A<sub>4</sub></th><th>A<sub>5</sub></th></tr></thead><tbody><tr><td rowspan="3">IGLM</td><td>0.746</td><td><strong>0.769</strong></td><td>0.679</td><td>0.663</td><td>0.668</td></tr><tr><td>0.151</td><td><strong>0.128</strong></td><td>0.220</td><td>0.236</td><td>0.231</td></tr><tr><td>0.104</td><td><strong>0.103</strong></td><td>0.101</td><td>0.101</td><td>0.101</td></tr><tr><td rowspan="3">IE</td><td>0.615</td><td>0.538</td><td>0.746</td><td>0.591</td><td>0.631</td></tr><tr><td>0.282</td><td>0.361</td><td>0.151</td><td>0.306</td><td>0.265</td></tr><tr><td>0.103</td><td>0.101</td><td>0.104</td><td>0.103</td><td>0.104</td></tr><tr><td rowspan="3">PP1</td><td><strong>0.600</strong></td><td>0.644</td><td>0.826</td><td>0.728</td><td>0.800</td></tr><tr><td><strong>0.300</strong></td><td>0.254</td><td>0.128</td><td>0.166</td><td>0.100</td></tr><tr><td><strong>0.100</strong></td><td>0.101</td><td>0.046</td><td>0.106</td><td>0.100</td></tr></tbody></table>

An excerpt of the AIFDM for the case problem is shown in Table 5. The matrix IFNs were calculated by substituting in eq. (2) the weights of the three ($l=3$) decision makers ($\lambda_{D1} = 0.406$, $\lambda_{D2} = 0.238$, $\lambda_{D3} = 0.356$) and the IFNs ($\mu_{ij}^{(k)}, v_{ij}^{(k)}, \pi_{ij}^{(k)}$) produced by using the relationships of Table 3 (i.e., these IFNs correspond to ratings given by the $k$ decision maker on each system $A_i$ ($i=1, 2, ..., 5$) with respect to each criterion $j$ ($j=1, 2, ..., 13$)). For example, in Table 5, the IFN [0.769, 0.128, 0.103], shown in bold, is the aggregated score of PPMIS $A_2$ on criterion IGLM (Idea Generation/Lead
Mgmt.), while the IFN [0.600, 0.300, 0.100], also shown in bold, is the aggregated score of PPMIS $A_i$ on criterion PP1 (Portfolio Planning).

**Step 3: Determine the weights of the selection criteria.** To analyse users' requirements from a PPMIS we disseminated to the three users/members of HOU a structured questionnaire, asking them to evaluate the 13 selection criteria and express their perceptions on the relative importance of each one criterion with respect to the overall performance and benefits provided from a candidate PPMIS. Each of the 3 users was requested to answer 13 questions by denoting a grade for the importance of each criterion in a linguistic term, as it is shown in column 2 of Table 2. Opinions of users $U_1, U_2$ and $U_3$ on the importance of the criteria are presented in columns of Table 6 entitled with the label “Users”. These opinions are assigned to corresponding IFNs by using the relationships between values in column 2 and values in column 3 of Table 2.

<table><caption>**Table 6.** Importance values and weights of the criteria</caption><thead><tr><th rowspan="2">Criteria</th><th colspan="3">Users</th><th colspan="3">Weights</th></tr><tr><th>$U_1$</th><th>$U_2$</th><th>$U_3$</th><th>$\mu$</th><th>$\nu$</th><th>$\pi$</th></tr></thead><tbody><tr><td>IGLM</td><td>VI</td><td>I</td><td>M</td><td>0.779</td><td>0.201</td><td>0.019</td></tr><tr><td>IE</td><td>M</td><td>VI</td><td>I</td><td>0.734</td><td>0.236</td><td>0.031</td></tr><tr><td>PP1</td><td>M</td><td>VI</td><td>VI</td><td>0.808</td><td>0.184</td><td>0.008</td></tr><tr><td>PP2</td><td>VI</td><td>VI</td><td>VI</td><td>0.900</td><td>0.100</td><td>0.000</td></tr><tr><td>PP3</td><td>I</td><td>VI</td><td>VI</td><td>0.855</td><td>0.133</td><td>0.013</td></tr><tr><td>PC1</td><td>M</td><td>VI</td><td>VI</td><td>0.808</td><td>0.184</td><td>0.008</td></tr><tr><td>PC2</td><td>M</td><td>VI</td><td>I</td><td>0.734</td><td>0.236</td><td>0.031</td></tr><tr><td>PC3</td><td>M</td><td>M</td><td>VI</td><td>0.718</td><td>0.263</td><td>0.018</td></tr><tr><td>PT1</td><td>I</td><td>VI</td><td>VI</td><td>0.855</td><td>0.133</td><td>0.013</td></tr><tr><td>PT2</td><td>VI</td><td>M</td><td>I</td><td>0.797</td><td>0.183</td><td>0.020</td></tr><tr><td>AC</td><td>VI</td><td>I</td><td>I</td><td>0.828</td><td>0.151</td><td>0.021</td></tr><tr><td>PO</td><td>VI</td><td>VI</td><td>M</td><td>0.823</td><td>0.171</td><td>0.007</td></tr><tr><td>CC</td><td>I</td><td>M</td><td>VI</td><td>0.787</td><td>0.189</td><td>0.023</td></tr></tbody></table>

The IFWA operator was also used to calculate the weights of criteria by aggregating the opinions of the users. Let $w_j^{(k)} = (\mu_j^{(k)}, \nu_j^{(k)}, \pi_j^{(k)})$ be the IFN assigned to criterion $j$ ($j = 1,2,...,n$) by the $k$ user ($k = 1,2,...,l$). Then the weight of $j$ can be calculated as follows:

$$ w_j = \text{IFWA}_\lambda(w_j^{(1)}, w_j^{(2)}, \dots, w_j^{(l)}) = \lambda_1 w_j^{(1)} \oplus \lambda_2 w_j^{(2)} \oplus \lambda_3 w_j^{(3)} \oplus \dots \oplus \lambda_l w_j^{(l)} \quad (3) \\ = \left[ 1 - \prod_{k=1}^{l} (1 - \mu_j^{(k)})^{\lambda_k}, \prod_{k=1}^{l} (\nu_j^{(k)})^{\lambda_k}, \prod_{k=1}^{l} (1 - \mu_j^{(k)})^{\lambda_k} - \prod_{k=1}^{l} (\nu_j^{(k)})^{\lambda_k} \right] $$

Thus, a vector of criteria weights is obtained $W = [w_1, w_2, \dots, w_j]$, where each weight $w_j$ is an IFN in the form $[\mu_j, \nu_j, \pi_j]$ ($j = 1,2,...,n$). In the case problem, substituting in eq. (3) the weights of three users ($\lambda_{U1} = 0.406, \lambda_{U2} = 0.238, \lambda_{U3} = 0.356$) and using IFNs which correspond to linguistic values of Table 6 yielded the criteria weights shown in the columns of the same table entitled with the label “Weights”.
**Step 4: Compose the aggregated weighted intuitionistic fuzzy decision matrix.** In this step, the aggregated weighted intuitionistic fuzzy decision (AWIFDM) matrix $R'$ is composed by considering the aggregated intuitionistic fuzzy decision matrix (i.e., table *R* produced in step 2) and the vector of the criteria weights (i.e., table *W* produced in step 3). Step 4 is necessary to synthesize the ratings of both decision makers and users. In particular, the elements of the AWIFDM can be calculated by using the multiplication operator of IFSs as follows:

$$R \otimes W = \{< x, \mu_{A_i}(x) \cdot \mu_W(x), v_{A_i}(x) + v_W(x) - v_{A_i}(x) \cdot v_W(x) > | x \in X\} \quad (4)$$

$R'$ is an *m* x *n* matrix composed with elements IFNs in the form of $r'_{ij} = [\mu_{A_W}(x_j), v_{A_W}(x_j), \pi_{A_W}(x_j)]$, where $\mu_{A_W}(x_j)$ and $v_{A_W}(x_j)$ are values derived by eq. (4) and $\pi_{A_W}(x) = 1 - v_{A_i}(x) - v_W(x) - \mu_{A_i}(x) \cdot \mu_W(x) + v_{A_i}(x) \cdot v_W(x)$.

In the case problem, substituting in eq. (4) the IFNs of Table 5 (table *R*) and IFNs of Table 6 (table *W*) yielded the IFNs of the AWIFDM (table $R'$), an excerpt from which is presented in Table 7. For example, in Table 7, the IFN [0.599, 0.304, 0.097], shown in bold, is the aggregated weighted score of PPMIS $A_2$ on criterion IGLM (Idea Generation/Lead Mgmt.), while the IFN [0.485, 0.429, 0.086], also shown in bold, is the aggregated score of PPMIS $A_1$ on criterion PP1 (Portfolio Planning).

**Step 5: Compute the intuitionistic fuzzy positive ideal solution and the intuitionistic fuzzy negative ideal solution.** In order to apply TOPSIS, the intuitionistic fuzzy positive ideal solution (IFPIS) $A^*$ and the intuitionistic fuzzy negative ideal solution (IFNIS) $A^-$ have to be determined. Both solutions are vectors of IFN elements and they are derived from the AWIFDM matrix as follows. Let *B* and *C* be benefit and cost criteria, respectively. Then $A^*$ and $A^-$ are equal to:

<table><caption>**Table 7.** Aggregated weighted intuitionistic fuzzy decision matrix (excerpt)</caption><thead><tr><th></th><th>A<sub>1</sub></th><th>A<sub>2</sub></th><th>A<sub>3</sub></th><th>A<sub>4</sub></th><th>A<sub>5</sub></th></tr></thead><tbody><tr><td rowspan="3">IGLM</td><td>0.581</td><td><strong>0.599</strong></td><td>0.529</td><td>0.517</td><td>0.520</td></tr><tr><td>0.322</td><td><strong>0.304</strong></td><td>0.377</td><td>0.390</td><td>0.386</td></tr><tr><td>0.097</td><td><strong>0.097</strong></td><td>0.094</td><td>0.094</td><td>0.094</td></tr><tr><td rowspan="3">IE</td><td>0.451</td><td>0.395</td><td>0.547</td><td>0.433</td><td>0.463</td></tr><tr><td>0.451</td><td>0.512</td><td>0.351</td><td>0.470</td><td>0.438</td></tr><tr><td>0.098</td><td>0.094</td><td>0.102</td><td>0.097</td><td>0.099</td></tr><tr><td rowspan="3">PP1</td><td><strong>0.485</strong></td><td>0.520</td><td>0.667</td><td>0.588</td><td>0.646</td></tr><tr><td><strong>0.429</strong></td><td>0.392</td><td>0.289</td><td>0.320</td><td>0.266</td></tr><tr><td><strong>0.086</strong></td><td>0.088</td><td>0.044</td><td>0.093</td><td>0.088</td></tr></tbody></table>

$$A^* = (\mu_{A^*W}^*(x_j), v_{A^*W}^*(x_j)) \quad \text{and} \quad A^- = (\mu_{A^-W}^*(x_j), v_{A^-W}^*(x_j)) \quad \text{where} \quad (5)$$

$$\mu_{A^*W}^*(x_j) = ((\max_i \mu_{A_W}^*(x_j) | j \in B), (\min_i \mu_{A_W}^*(x_j) | j \in C))$$
$$v_{A^*W}(x_j) = ((\min_i v_{A_W}(x_j) | j \in B), (\max_i v_{A_W}(x_j) | j \in C))$$

$$\mu_{A^*W}(x_j) = ((\min_i \mu_{A_W}(x_j) | j \in B), (\max_i \mu_{A_W}(x_j) | j \in C))$$

$$v_{A^*W}(x_j) = ((\max_i v_{A_W}(x_j) | j \in B), (\min_i v_{A_W}(x_j) | j \in C))$$

In the case problem, $B = \{IGLM, IE, PP1, PP2, PP3, PC1, PC2, PC3, PT1, PT2, AC\}$ and $C = \{PO, CC\}$. To obtain IFPIS and IFNIS, eq. (5) was applied on the IFNs of the AWIFDM decision matrix. The IFPIS and IFNIS were determined as follows:

$A^* = ([0.599, 0.304, 0.097], [0.547, 0.351, 0.102], [0.667, 0.289, 0.044], [0.692, 0.215, 0.093], [0.667, 0.235, 0.099], [0.602, 0.307, 0.090], [0.564, 0.334, 0.102], [0.532, 0.378, 0.090], [0.657, 0.244, 0.099], [0.641, 0.288, 0.072], [0.562, 0.338, 0.100], [0.476, 0.437, 0.087], [0.446, 0.459, 0.095])

$A^- = ([0.517, 0.390, 0.094], [0.395, 0.512, 0.094], [0.485, 0.429, 0.086], [0.536, 0.372, 0.092], [0.614, 0.284, 0.102], [0.539, 0.373, 0.088], [0.424, 0.481, 0.095], [0.445, 0.468, 0.087], [0.525, 0.377, 0.097], [0.513, 0.391, 0.096], [0.435, 0.468, 0.097], [0.613, 0.296, 0.091], [0.605, 0.293, 0.101])

**Step 6: Calculate the separation between the alternative PPMIS.** Next, the separation measures $S_i^*$ and $S_i^-$ were calculated for each candidate system $A_i$ from IFPIS and IFNIS, respectively. As a distance measure, the normalized Euclidean distance was adopted, since it has been proved to be a reliable distance measure that takes into account not only membership and non-membership but also the hesitation part of IFNs [29]. For each alternative these two separation values can be calculated as follows:

$$S^* = \sqrt{\frac{1}{2n} \sum_{j=1}^{n} [(\mu_{A_W}(x_j) - \mu_{A^*W}(x_j))^2 + (v_{A_W}(x_j) - v_{A^*W}(x_j))^2 + (\pi_{A_W}(x_j) - \pi_{A^*W}(x_j))^2]} \\ S^- = \sqrt{\frac{1}{2n} \sum_{j=1}^{n} [(\mu_{A_W}(x_j) - \mu_{A^*W}(x_j))^2 + (v_{A_W}(x_j) - v_{A^*W}(x_j))^2 + (\pi_{A_W}(x_j) - \pi_{A^*W}(x_j))^2]} \quad (6)$$

By utilizing equations (6), the positive and negative separation for the five alternatives were calculated, shown in columns (1) and (2) of Table 8, respectively.

<table><caption>**Table 8.** Separation measures and relative closeness coefficient of each PPMIS</caption><thead><tr><th>PPMIS</th><th>S<sup>*</sup> (1)</th><th>S<sup>-</sup> (2)</th><th>C<sup>*</sup> (3)</th></tr></thead><tbody><tr><td>A<sub>1</sub></td><td>0.076</td><td>0.074</td><td>0.495</td></tr><tr><td>A<sub>2</sub></td><td>0.091</td><td>0.074</td><td>0.448</td></tr><tr><td>A<sub>3</sub></td><td>0.041</td><td>0.116</td><td>0.737</td></tr><tr><td>A<sub>4</sub></td><td>0.069</td><td>0.074</td><td>0.520</td></tr><tr><td>A<sub>5</sub></td><td>0.088</td><td>0.085</td><td>0.490</td></tr></tbody></table>
**Step 7: Determine the final ranking of PPMIS.** The final score of each system was derived by calculating the corresponding relative closeness coefficient with respect to the intuitionistic fuzzy ideal solution. For each alternative $A_i$, the relative closeness coefficient $C_i^*$ with respect to the IFPIS is defined as follows:

$$C_i^* = \frac{S_i^-}{S_i^* + S_i^-} \qquad (7)$$

where $0 \le C_i^* \le 1$. Eq. (7) was used to calculate these coefficients (final scores) listed in column (3) of Table 8. The alternative PPMIS were ranked in a descending order of these scores as $A_3 > A_4 > A_1 > A_5 > A_2$, from where it can be deduced that alternative $A_3$ is the most dominant PPMIS for the present case study.

<table><caption>**Table 9.** Sensitivity analysis results</caption><thead><tr><th rowspan="2">Exp.</th><th rowspan="2">Criteria</th><th colspan="5">Criteria Weights</th><th rowspan="2">Ranking</th></tr><tr><th>A<sub>1</sub></th><th>A<sub>2</sub></th><th>A<sub>3</sub></th><th>A<sub>4</sub></th><th>A<sub>5</sub></th></tr></thead><tbody><tr><td>1</td><td>w<sub>1-13</sub> = [0.10,0.90]</td><td>0.497</td><td>0.447</td><td>0.730</td><td>0.514</td><td>0.494</td><td>A<sub>3</sub> > A<sub>4</sub> > A<sub>1</sub> > A<sub>5</sub> > A<sub>2</sub></td></tr><tr><td>2</td><td>w<sub>1-13</sub> = [0.35,0.60]</td><td>0.500</td><td>0.450</td><td>0.728</td><td>0.518</td><td>0.494</td><td>A<sub>3</sub> > A<sub>4</sub> > A<sub>1</sub> > A<sub>5</sub> > A<sub>2</sub></td></tr><tr><td>3</td><td>w<sub>1-13</sub> = [0.50,0.45]</td><td>0.499</td><td>0.449</td><td>0.729</td><td>0.517</td><td>0.494</td><td>A<sub>3</sub> > A<sub>4</sub> > A<sub>1</sub> > A<sub>5</sub> > A<sub>2</sub></td></tr><tr><td>4</td><td>w<sub>1-13</sub> = [0.75,0.20]</td><td>0.498</td><td>0.448</td><td>0.729</td><td>0.516</td><td>0.494</td><td>A<sub>3</sub> > A<sub>4</sub> > A<sub>1</sub> > A<sub>5</sub> > A<sub>2</sub></td></tr><tr><td>5</td><td>w<sub>1-13</sub> = [0.90,0.10]</td><td>0.497</td><td>0.447</td><td>0.730</td><td>0.514</td><td>0.494</td><td>A<sub>3</sub> > A<sub>4</sub> > A<sub>1</sub> > A<sub>5</sub> > A<sub>2</sub></td></tr><tr><td>6</td><td>w<sub>1</sub> = [0.90,0.10], w<sub>2-13</sub> = [0.10,0.90]</td><td>0.674</td><td>0.712</td><td>0.389</td><td>0.247</td><td>0.278</td><td>A<sub>2</sub> > A<sub>1</sub> > A<sub>3</sub> > A<sub>5</sub> > A<sub>4</sub></td></tr><tr><td>7</td><td>w<sub>2</sub> = [0.90,0.10], w<sub>1,3-13</sub> = [0.10,0.90]</td><td>0.388</td><td>0.147</td><td>0.910</td><td>0.291</td><td>0.458</td><td>A<sub>3</sub> > A<sub>5</sub> > A<sub>1</sub> > A<sub>4</sub> > A<sub>2</sub></td></tr><tr><td>8</td><td>w<sub>3</sub> = [0.90,0.10], w<sub>1,2,4-13</sub> = [0.10,0.90]</td><td>0.152</td><td>0.257</td><td>0.909</td><td>0.595</td><td>0.763</td><td>A<sub>3</sub> > A<sub>5</sub> > A<sub>4</sub> > A<sub>2</sub> > A<sub>1</sub></td></tr><tr><td>9</td><td>w<sub>4</sub> = [0.90,0.10], w<sub>1,3,5-13</sub> = [0.10,0.90]</td><td>0.294</td><td>0.171</td><td>0.896</td><td>0.597</td><td>0.804</td><td>A<sub>3</sub> > A<sub>5</sub> > A<sub>4</sub> > A<sub>2</sub> > A<sub>1</sub></td></tr><tr><td>10</td><td>w<sub>5</sub> = [0.90,0.10], w<sub>1,4,6-13</sub> = [0.10,0.90]</td><td>0.650</td><td>0.413</td><td>0.516</td><td>0.349</td><td>0.477</td><td>A<sub>1</sub> > A<sub>3</sub> > A<sub>5</sub> > A<sub>2</sub> > A<sub>4</sub></td></tr><tr><td>11</td><td>w<sub>6</sub> = [0.90,0.10], w<sub>1,5,7-13</sub> = [0.10,0.90]</td><td>0.333</td><td>0.656</td><td>0.821</td><td>0.601</td><td>0.320</td><td>A<sub>3</sub> > A<sub>2</sub> > A<sub>4</sub> > A<sub>1</sub> > A<sub>5</sub></td></tr><tr><td>12</td><td>w<sub>7</sub> = [0.90,0.10], w<sub>1,6,8-13</sub> = [0.10,0.90]</td><td>0.521</td><td>0.158</td><td>0.850</td><td>0.473</td><td>0.819</td><td>A<sub>3</sub> > A<sub>5</sub> > A<sub>1</sub> > A<sub>4</sub> > A<sub>2</sub></td></tr><tr><td>13</td><td>w<sub>8</sub> = [0.90,0.10], w<sub>1,7,9-13</sub> = [0.10,0.90]</td><td>0.487</td><td>0.683</td><td>0.318</td><td>0.784</td><td>0.718</td><td>A<sub>4</sub> > A<sub>5</sub> > A<sub>2</sub> > A<sub>1</sub> > A<sub>3</sub></td></tr><tr><td>14</td><td>w<sub>9</sub> = [0.90,0.10], w<sub>1,8,10-13</sub> = [0.10,0.90]</td><td>0.426</td><td>0.363</td><td>0.885</td><td>0.651</td><td>0.210</td><td>A<sub>3</sub> > A<sub>4</sub> > A<sub>1</sub> > A<sub>2</sub> > A<sub>5</sub></td></tr><tr><td>15</td><td>w<sub>10</sub> = [0.90,0.10], w<sub>1,9,11-13</sub> = [0.10,0.90]</td><td>0.400</td><td>0.250</td><td>0.881</td><td>0.197</td><td>0.703</td><td>A<sub>3</sub> > A<sub>5</sub> > A<sub>1</sub> > A<sub>2</sub> > A<sub>4</sub></td></tr><tr><td>16</td><td>w<sub>11</sub> = [0.90,0.10], w<sub>1,10,12-13</sub> = [0.10,0.90]</td><td>0.593</td><td>0.603</td><td>0.885</td><td>0.717</td><td>0.211</td><td>A<sub>3</sub> > A<sub>4</sub> > A<sub>2</sub> > A<sub>1</sub> > A<sub>5</sub></td></tr><tr><td>17</td><td>w<sub>12</sub> = [0.90,0.10], w<sub>1,11,13</sub> = [0.10,0.90]</td><td>0.821</td><td>0.791</td><td>0.464</td><td>0.592</td><td>0.196</td><td>A<sub>1</sub> > A<sub>2</sub> > A<sub>4</sub> > A<sub>3</sub> > A<sub>5</sub></td></tr><tr><td>18</td><td>w<sub>13</sub> = [0.90,0.10], w<sub>1,12</sub> = [0.10,0.90]</td><td>0.818</td><td>0.806</td><td>0.908</td><td>0.606</td><td>0.170</td><td>A<sub>3</sub> > A<sub>1</sub> > A<sub>2</sub> > A<sub>4</sub> > A<sub>5</sub></td></tr></tbody></table>

**Step 8: Sensitivity analysis.** Sensitivity analysis is concerned with ‘what-if’ kind of scenarios to determine if the final answer is stable to changes (experiments) in the inputs, either judgments or weights of criteria. In the present case, sensitivity analysis was performed by examining the impact of criteria weights (i.e., the weights of users’ requirements from a PPMIS) on the final ranking. Of special interest was to see if criteria weights’ changes alter the order of the alternatives. 18 experiments were conducted in a similar way with the approach presented in [30]. The details of all experiments are shown in Table 9, where $w_1, w_2, \dots, w_{13}$ denote respectively the weights of criteria IGLM, IE, PP1, PP2, PP3, PC1, PC2, PC3, PT1, PT2, AC, PO, CC. In experiments 1-5, weights of all criteria were set equal to [0.10,0.90], [0.35,0.60], [0.50,0.45], [0.75,0.20] and [0.90,0.10], respectively. These IFNs correspond to the linguistic terms VU, U, M, I and VI, respectively (see Table 2). In
experiments 6-18, the weight of each of the 13 criteria was set equal to the highest IFN [0.90,0.10], one by one, and the weights of the rest of criteria were set all equal to the lowest IFN [0.10,0.90]. The results show that PPMIS $A_1$ remains the dominant alternative in 14 out of the 18 experiments (this represents a clear “majority” equal to 77.77%). PPMIS $A_1$ was first in 2/18 experiments, namely in exp. 10 and in exp. 17, where the highest weights were assigned, respectively, to criterion PP3 (project planning) and criterion PO (total price for purchasing/ownership). System $A_2$ had the highest score in exp. 6, where the highest weight was assigned to criterion PP1 (portfolio planning), while system $A_4$ had the highest score in exp. 13, where the highest value was assigned to the weight of PC3 (portfolio controlling).

It should be noted that in the presented approach we have used the IFNs proposed in [18] to represent the linguistic terms of Table 2 and Table 3. Sensitivity analysis on the final ranking can be easily performed by changing these IFN values. In addition, further generalization of the approach requires the use of a parameterised form of the hesitation degree. This can be performed in two ways: i) by following a Positive-Confidence or a Negative-Confidence approach [17], and ii) by utilizing interval-valued intuitionistic fuzzy numbers [19]. We have plans to investigate these two solutions in a future research.

# 5 Conclusions

The paper presented, through a real case study, an approach that applied a group-based multi criteria decision making (MCDM) method for the evaluation and final selection of an appropriate Project and Portfolio Management Information System (PPMIS). The approach utilized a method that jointly synthesizes intuitionistic fuzzy sets and TOPSIS [18]. The benefit from this combination in a PPMIS selection is twofold: First, the selection approach actively involves decision makers and PPMIS users in the decision making process and aggregates their opinions to support agreement upon the final selection. Second, the approach considers that they both express their judgments under inherent uncertainty. More significantly, the approach handles adequately the degree of indeterminacy that characterizes decision makers and users in their evaluations. This is very important when an organization needs to decide upon the selection of any new, multi-functional information system, as in our case is a suitable PPMIS, since decision makers often cannot have full knowledge of the extend that each candidate system will support the user requirements. System users, on the other hand, can be unfamiliar with the processes supported by the required system, and thus, they cannot judge with certainty the importance of their needs.

The presented approach not only validated the method, as it was originally defined in [18], in a new application field that is the evaluation of PPMIS (where other application examples of MCDM methods are rather limited in the literature), but also considered a more extensive list of benefit and cost oriented criteria, suitable for PPMIS selection. In addition, final results were verified by applying sensitivity analysis. We should mention that the method underlying computations are not transparent to the problem stakeholders which utilise linguistic terms to state
evaluations/preferences. Actually, we implemented the method in a spreadsheet program that helps to effectively and practically apply the method with a variety of inputs.

The study raises several issues that could spark further research. For example, an interesting idea could be to validate the approach applicability in addressing the selection of other types of software packages. We are now investigating the selection of e-learning management systems for the case organization (i.e., the Hellenic Open University). In addition, treating more with uncertainties would further strengthen the proposed approach in deriving more precise results. Therefore, we have plans to examine more powerful models in the same domain, such as the interval-valued intuitionistic fuzzy sets [19], [20].

## References

1. Liberatore, M. J., Pollack-Johnson, B.: Factors Influencing the Usage and Selection of Project Management Software. IEEE Transactions on Engineering Management, 50(2), 164-174 (2003)

2. Project Management Institute: A Guide to the Project Management Body of Knowledge. 4th edition, Project Management Institute (PMI), Newtown Square (2008)

3. Meyer, M.M., Ahlemann, F.: Project Management Software Systems. 6th edition, Business Application Research Center, Würzburg, Germany (2010)

4. Stang, D. B.: Magic Quadrant for IT Project and Portfolio Management. Gartner RAS Core Research Note, Gartner Research (2010)

5. Jadhav, A. S., Sonar, R. M.: Evaluating and Selecting Software Packages: a Review. Information and Software Technology, 51(3), 555-563 (2009)

6. Vaidya, O. S., Kumar, S.: Analytic Hierarchy Process: an Overview of Applications. European Journal of Operational Research, 169(1), 1-29 (2006)

7. Ho, W.: Integrated Analytic Hierarchy Process and its Applications – A Literature Review. European Journal of Operational Research, 186(1), 211-218 (2008)

8. Ahmad, N., Laplante, P.: Software Project Management Systems: Making a Practical Decision using AHP. In: 30th Annual IEEE/NASA Software Engineering Workshop, pp. 76-84, IEEE Press (2006)

9. Gerogiannis, V.C., Fitsilis, P., Voulgaridou, D., Kirytopoulos, K.A., Sachini, E.: A Case Study for Project and Portfolio Management Information System Selection: a Group AHP-Scoring Model Approach. International Journal of Project Organisation and Management, 2(4), 361-381 (2010)

10. Bozdag, C.E., Kahraman, C., Ruan, D.: Fuzzy Group Decision Making for Selection among Computer Integrated Manufacturing Systems. Computers in Industry, 51(1), 3-29 (2003)

11. Chang, D. Y.: Application of Extent Analysis Method on Fuzzy AHP. European Journal of Operational Research, 95(3), 649-655 (1996)

12. Chen, C. T.: Extensions of the TOPSIS for group decision-making under fuzzy environment. Fuzzy Sets and Systems, 114(1), 1-9 (2000)

13. Cochran, J.K., Chen, H.N.: Fuzzy Multi-Criteria Selection of Object-Oriented Simulation Software for Production System Analysis. Computers & Operations Research, 32(1), 153-168 (2005)

14. Lin, H.Y., Hsu, P.Y., Sheen, G.J.: A Fuzzy-based Decision-Making Procedure for Data Warehouse System Selection. Expert System with Applications, 32(3), 939-953 (2007)

15. Atanassov, K.T.: Intuitionistic Fuzzy Sets. Fuzzy Sets and Systems, 20(1), 87-96 (1986)
16. De, S.K., Biswas, R., Roy, A.R.: An Application of Intuitionistic Fuzzy Sets in Medical Diagnosis. Fuzzy Sets and Systems, 117(2), 209--213 (2001)

17. Wang, P.: QoS-aware Web Services Selection with Intuitionistic Fuzzy Set under Consumer's Vague Perception. Expert Systems with Applications, 36(3), 4460--4466 (2009)

18. Boran, F.E., Genc, S. Kurt, M, Akay, D.: A Multi-Criteria Intuitionistic Fuzzy Group Decision Making for Supplier Selection with TOPSIS Method. Expert Systems with Applications, 36(8), 11363--11368 (2009)

19. Park, J.H., Park, I.Y., Kwun, Y.C., Tan, X.: Extension of the TOPSIS Method for Decision Making Problems under Interval-Valued Intuitionistic Fuzzy Environment. Applied Mathematical Modelling, 35(5), 2544--2556 (2011)

20. Chen, T.Y. Wang, H.P., Lu, Y.Y.: A Multicriteria Group Decision-Making Approach based on Interval-Valued Intuitionistic Fuzzy Sets: a Comparative Perspective. Expert Systems with Applications, 38(6), 7647--7658 (2011)

21. Raymond, L., Bergeron, F.: Project Management Information Systems: an Empirical Study of their Impact on Project Managers and Project Success. International Journal of Project Management, 26(2), 213--220 (2008)

22. Project Management Institute: Project Management Software Survey. Project Management Institute (PMI), Newtown Square (1999)

23. Ali, A.S.B., Money, W.H.: A Study of Project Management System Acceptance. In: Proceedings of the 38th Hawaii International Conference on System Sciences, pp. 234--244, IEEE Press (2005)

24. Gomez, D.C., Alexander, A., Anderson, D. Cook, D. Poole, K., Findlay, O.: NASA Project Management Tool Analysis and Recommendations White Paper. Technical report, Project Management Tool Working Group, NASA Glenn Research Center, available on line at: http://km.nasa.gov/pdf/54927main_pm-tool-paper.pdf (2004)

25. Ahlemann, F.: Towards a Conceptual Reference Model for Project Management Information Systems. International Journal of Project Management, 27(1), 19--30 (2009).

26. Business Application Research Center (BARC), Project Management Software Systems Report, http://www.pm-software-report.com

27. Xu, Z.S., Yager, R.R.: Some Geometric Aggregation Operators based on Intuitionistic Fuzzy Sets. International Journal of General Systems, 35(4), 417--433 (2006)

28. Xu, Z.S.: Intuitionistic Fuzzy Aggregation Operators. IEEE Transactions on Fuzzy Systems, 15(6), 1179--1187 (2007)

29. Szmidt, E., Kacprzy K.J.: Distances between Intuitionistic Fuzzy Sets. Fuzzy Sets and Systems, 114 (3), 505--518 (2000)

30. Awasthi, A., Chauhan, S.S., Goyal, S.K.: A Fuzzy Multicriteria Approach for Evaluating Environmental Performance of Suppliers. International Journal of Production Economics, 126(2), 370--378 (2010)