# A Combined Method for Evaluating Criteria When Selecting ERP Systems

David L. Olson, Björn Johansson, Rogério De Carvalho

► To cite this version:

David L. Olson, Björn Johansson, Rogério De Carvalho. A Combined Method for Evaluating Criteria When Selecting ERP Systems. 5th Working Conference on Research and Practical Issues of Enterprise Information Systems (CONFENIS), Oct 2011, Aalborg, Denmark. pp.64-74, 10.1007/978-3-642-28827-2_5. hal-01483872

HAL Id: hal-01483872

https://inria.hal.science/hal-01483872

Submitted on 6 Mar 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Combined Method for Evaluating Criteria when Selecting ERP Systems

David L. Olson<fnref n="1" />, Björn Johansson<fnref n="2" />, and Rogério Atem de Carvalho<fnref n="3" />

**Abstract.** There are many benefits offered by integrated enterprise computer systems. There are a growing number of options available to obtain such management information system support. A major problem when selecting Enterprise Information Systems, in special ERP systems, is how to deal with the great diversity of options as well as the number of criteria used to evaluate each alternative. There is an implicit tradeoff between cost and system functionality. Total cost of ownership (TCO) is in itself very difficult to calculate accurately, and needs to be considered in light of other criteria. Published criteria for ERP selection decisions in a variety of contexts are reviewed. We also present a method which integrates a multicriteria rating strategy based on the Simple MultiAttribute Rating Theory (SMART) with the meta-method Prepare-Identify-Rate-Compare-Select (PIRCS) framework for driving the selection process. The method is demonstrated with a general ERP selection decision, but is meant as a framework that can be applied with whatever criteria decision makers deem important in the context of their specific decision.

**Keywords:** ERP selection process, multiple criteria selection, decision criteria.

## 1 Introduction

Organizations can benefit a great deal from integrated enterprise systems, obtaining increased data accuracy through single-source databases, more efficient operations through business process reengineering, and reduced information technology payroll. The number of options is increasing, beyond top-of-the-line vendor systems such as SAP and Oracle, through more moderately priced vendors such as Microsoft and Lawson [1], to application service providers offering rental of enterprise computing. However, there is risk involved, especially for small businesses [2], [3]. In specific countries, such as China [4], Brazil [5], and elsewhere [6], there are additional local forms of ERP. One option that has become viable in the past decade is the open source alternative providing free software from various business models [7]. There are many enterprise system options available. When selecting an ERP option there is a general tradeoff between functionality and cost, although total cost of ownership (TCO) is a complex matter that defies accurate calculation [8]. This paper reviews

<footnote n="1">¹Department of Management, University of Nebraska, Lincoln, NE 68588-0491
Dolson3@unl.edu</footnote>

<footnote n="2">²Department of Informatics, School of Economics and Management, Lund University, Ole Römers väg 6, SE-223 63 Lund, Sweden, bjorn.johansson@ics.lu.se</footnote>

<footnote n="3">³Federal Center for Technological Education of Campos (CEFET Campos), R. Dr. Siqueira
273, CEP 28030-130, Campos/RJ, Brazil, ratem@cefetcampos.br</footnote>
criteria that have been published in the literature with respect to selection of enterprise resource planning (ERP) system. It also demonstrates how the meta-method PIRCS (prepare, identify, rate, compare, and select) [9] can be implemented through the simple multiattribute rating theory (SMART) [10].

## 2 ERP Selection Criteria

Many papers have dealt with selection among alternative means of obtaining ERP systems. Baki and Çaki [6] reviewed criteria considered by prior studies in manufacturing firms, and conducted a survey of 55 Turkish manufacturing companies concerning the importance of these criteria, adding references, consultancy, implementation time, and software methodology to the criteria used by the prior studies. Baki and Çaki used a 1-5 Likert scale, the mean of which is reported in the last column. A rating of 1 indicated lowest possible importance and a rating of 5 indicated highest possible importance (see Table 1).

<table><caption>**Table 1.** Comparative Criteria in ERP Selection in Manufacturing Firms</caption><thead><tr><th>Criteria</th><th>[11]</th><th>[12]</th><th>[13]</th><th>[14]</th><th>[15]</th><th>Mean</th></tr></thead><tbody><tr><td>Fit with allied organizations</td><td></td><td></td><td></td><td></td><td>*</td><td>4.79</td></tr><tr><td>Cross module integration</td><td></td><td></td><td></td><td>*</td><td>*</td><td>4.72</td></tr><tr><td>Compatibility with other systems</td><td></td><td>*</td><td></td><td></td><td></td><td>4.28</td></tr><tr><td>References</td><td></td><td></td><td></td><td></td><td></td><td>4.24</td></tr><tr><td>Vision</td><td>*</td><td></td><td></td><td>*</td><td></td><td>4.22</td></tr><tr><td>Functionality</td><td>*</td><td>*</td><td></td><td>*</td><td>*</td><td>4.15</td></tr><tr><td>System reliability</td><td></td><td></td><td></td><td></td><td>*</td><td>4.08</td></tr><tr><td>Consultancy</td><td></td><td></td><td></td><td></td><td></td><td>4.06</td></tr><tr><td>Technical aspects</td><td>*</td><td>*</td><td>*</td><td>*</td><td></td><td>4.01</td></tr><tr><td>Implementation time</td><td></td><td></td><td></td><td></td><td></td><td>3.94</td></tr><tr><td>Vendor market position</td><td></td><td></td><td></td><td>*</td><td>*</td><td>3.87</td></tr><tr><td>Ease of customization</td><td></td><td>*</td><td></td><td></td><td>*</td><td>3.84</td></tr><tr><td>Software methodology</td><td></td><td></td><td></td><td></td><td></td><td>3.83</td></tr><tr><td>Fit with organization</td><td></td><td></td><td></td><td></td><td>*</td><td>3.83</td></tr><tr><td>Service & support</td><td>*</td><td></td><td>*</td><td></td><td>*</td><td>3.77</td></tr><tr><td>Cost</td><td>*</td><td>*</td><td>*</td><td>*</td><td>*</td><td>3.65</td></tr><tr><td>Vendor domain knowledge</td><td></td><td></td><td>*</td><td></td><td></td><td>3.46</td></tr></tbody></table>

Baki and Çaki analyzed their data for differences between organizations that adopted MRP or MRP-II systems versus those who had not. They found no statistically significant difference between these two groups. Their inference was that prior exposure was not an important factor. The results show that all factors had some positive importance (as 3 would indicate neutrality), but Table 1 indicates that external fit (such as supply chain linkage) and software factors tend to be rated higher than organizational factors such as fit, service and support, and cost.

Open source ERP software is attractive for all small organizations. Three studies were found giving criteria for this domain. Criteria considered varied when selecting open source ERP as can be seen in Table 2.
<table><caption>**Table 2.** Open Source ERP Software Selection Criteria</caption><thead><tr><th>Criteria</th><th>[16]</th><th>[17]</th><th>[18]</th></tr></thead><tbody><tr><td>Technology</td><td>Technical requirements</td><td>Complexity of technology<br>East of database administration</td><td>Database migration</td></tr><tr><td>BPR</td><td>Business drivers</td><td>Ease of business logic implementation</td><td>Synchrorizing modules to workflow</td></tr><tr><td>User interface</td><td></td><td>Ease of presentation layer implementation</td><td>User friendly interfaces</td></tr><tr><td>Administration</td><td></td><td>Ease of administration</td><td>Integration with 3<sup>rd</sup> party software</td></tr><tr><td>Cost</td><td>Cost drivers</td><td></td><td></td></tr><tr><td>Others</td><td>Flexibility<br>Scalability<br>Business specific</td><td>Ease of service exposure<br>Resource utilization</td><td>User support</td></tr></tbody></table>

Benroider and Koch [19] sampled 138 small or medium sized organizations in Austria who had selected an ERP system about the criteria they used in their decisions. Small or medium sized was defined on the basis of the number of employees, based upon European Community standards. Large vendors were considered by almost all subjects, but a bit over 47 percent considered smaller ERP vendors. SAP was selected by nearly 70 percent of the samples, and small vendors by a little over 23.3 percent. There was a bias for larger organizations to select SAP. Delphi analysis was used to identify criteria deemed important. Benroider and Koch only reported criteria that had a strong relationship to organization size (focusing on those rated very important to SMEs). SMEs emphasized software adaptability and flexibility and shorter implementation time more than large organizations. Both SMEs and large organizations rated good support and process improvement as very important.

Other studies have looked at specific ERP selection contexts. Table 3 demonstrates further diversity of criteria proposed for consideration in the specific context of outsourcing (or ASP provider selection):

<table><caption>**Table 3.** ERP Selection Criteria for Outsourcing ERP</caption><thead><tr><th>Study</th><th>Context</th><th>Criteria</th></tr></thead><tbody><tr><td>[20]</td><td>Application service providers</td><td>Customer service<br>Reliability, availability, scalability<br>Integration<br>Total cost<br>Security<br>Service level</td></tr><tr><td>[21]</td><td>Outsourcing</td><td>Market leadership<br>Functionality<br>Quality<br>Price<br>Implementation speed<br>Link with other systems<br>International orientation</td></tr></tbody></table>
## 3 Modeling ERP Selection

There are a number of selection models presented in the literature. Conjoint analysis is used in marketing to determine the relative importance of product characteristics to potential clients. Keil et al. [22] conducted conjoint analysis to ERP selection, using software characteristics and implementation attributes. That study received 126 completed responses of 7 software package profiles from MIS managers of large organizations (see Table 4). The study modeled manager likelihood of recommending system acquisition using multiple regression, with a model adjusted R² of 0.506.

<table><caption>**Table 4.** Results of Keil et al.'s Conjoint Analysis</caption><thead><tr><td>Attribute</td><td>Effect</td><td>t-value</td><td>P&lt;0.01</td><td>P&lt;0.001</td></tr></thead><tbody><tr><td>Software Reliability</td><td>0.464</td><td>20.34</td><td>Yes</td><td>Yes</td></tr><tr><td>Software Functionality</td><td>0.457</td><td>20.03</td><td>Yes</td><td>Yes</td></tr><tr><td>Software Cost</td><td>-0.253</td><td>-11.08</td><td>Yes</td><td>Yes</td></tr><tr><td>Implementation Ease of Customization</td><td>0.129</td><td>5.67</td><td>Yes</td><td>Yes</td></tr><tr><td>Software Ease of Use</td><td>0.073</td><td>3.19</td><td>Yes</td><td>No</td></tr><tr><td>Implementation Vendor Reputation</td><td>0.007</td><td>0.29</td><td>No</td><td>No</td></tr><tr><td>Implementation Ease</td><td>0.000</td><td>0.01</td><td>No</td><td>No</td></tr></tbody></table>

The results indicate predominance of software factors over implementation factors. Only ease of customization was significant among the implementation factors. In this study, cost was found significant, but not as significant as software reliability and functionality. The subject firms were large. That set of ERP users can be expected to focus on getting the ERP system working. While cost is important, it's unpredictability would naturally be subsidiary to the necessity of obtaining required information system support. Ease of customization was significant, which indicates consideration of long-term life cycle cost. Ease of use was significant at a lower degree, while vendor reputation and ease of implementation were not significant. These last three factors relate to the impact of the system on the organization. MIS managers in the Keil et al. study placed less emphasis on these factors.

A second type of model using criteria is for decision maker selection. Among these models are analytic hierarchy process (AHP) [4], [23] and multiattribute utility theory, to include simple multiattribute rating theory (SMART) [10]. AHP has been the most widely used method in evaluating various aspects of ERP. Ahn and Choi [24] did so in a group context in South Korea, Salmeron and Lopez to evaluate ERP maintenance [25], Kahraman et al. [21] to consider ERP outsourcing, and Onut and Efendigil to ERP selection in Turkey [26]. The related analytic network process [27] was used by Ayağ and Özdemir [28] and Kirytopoulos et al. [29], allowing for feedback relationships. Olson and Wu [30] applied SMART along with data envelopment analysis to consider information system risk. One model for ERP selection used the criteria in Table 5 to compare alternative ERP vendors. That study [4] provided a thorough analysis of criteria starting with fundamental objectives for both system software factors and vendor factors, adding evaluation items at a third level, and identifying constraints reflecting means. The methodology was presented in a group decision making context. The hierarchy consisted of: factors, attributes, evaluation items and means.
<table><caption>**Table 5.** Value Analysis Hierarchy [4]</caption><thead><tr><th>Factors</th><th>Attributes</th><th>Evaluation items</th><th>Means</th></tr></thead><tbody><tr><td rowspan="5">System software</td><td>Total costs</td><td>Price<br>Maintenance<br>Consultant expenses<br>Infrastructure costs</td><td>Project budget<br>Annual maintenance budget<br>Infrastructure budget</td></tr><tr><td>Implementation time</td><td></td><td>Duration<br>Project management</td></tr><tr><td>Functionality</td><td>Module completion<br>Function fitness<br>Security</td><td>Necessary module availability<br>Currency, language, site issues<br>Permission management<br>Database protection</td></tr><tr><td>User friendliness</td><td>Ease of operation<br>Ease of learning</td><td>Guidebook<br>Online learning, help</td></tr><tr><td>Flexibility</td><td>Upgrade ability<br>Ease of integration<br>Ease of in-house development</td><td>Common programming language<br>Platform independence<br>Ease of integration</td></tr><tr><td rowspan="3">Vendor factors</td><td>Reliability</td><td>Stability<br>Recovery ability</td><td>Automatic data recovery<br>Automatic data backup</td></tr><tr><td>Reputation</td><td>Scale of vendor<br>Financial condition<br>Market share</td><td>Financial stability<br>Provision of reference sites</td></tr><tr><td>Technical capability</td><td>R&D ability<br>Technical support<br>Implementation</td><td>Upgrade service<br>Diverse product line<br>Implementation experience<br>Adequate number of engineers<br>Cooperation with partners<br>Domain knowledge</td></tr></tbody></table>

Another AHP model was applied to selecting an ERP system specific to clothing industry suppliers [31]. Criteria were selected based upon discussion with three such suppliers, as well as literature reviews. Criteria were:

* Cost

* Functionality

* Implementation approach

* Support

* Organizational credibility

* Experience

* Flexibility

* Customer focus

* Future strategy.

Cost benefit analysis was conducted for the first criterion, while AHP was used to generate a synthesis value for the other eight criteria. The ratio of synthesis value to normalized costs was used to rank alternatives.

ANP was applied to benchmarking and selecting ERP systems [32], applying the approach to an actual selection decision (see Table 6).
<table><caption>**Table 6.** System and Vendor Selection Criteria [32]</caption><thead><tr><td>System Factors</td><td>Vendor Factors</td></tr></thead><tbody><tr><td>Functionality</td><td>Market share</td></tr><tr><td>Strategic fitness</td><td>Financial capability</td></tr><tr><td>Flexibility</td><td>Implementation ability</td></tr><tr><td>User friendliness</td><td>R&D capability</td></tr><tr><td>Implementation time</td><td>Service support</td></tr><tr><td>Total costs</td><td></td></tr><tr><td>Reliability</td><td></td></tr></tbody></table>

Kahraman et al. [21] applied fuzzy modeling to a form of AHP for evaluation of selecting an outsourced ERP alternative. Table 7 shows the detailed criteria used by Kahraman et al. through two levels.

<table><caption>**Table 7.** AHP Hierarchy [21]</caption><thead><tr><td>Top Level Criteria</td><td>Second Level Criteria</td></tr></thead><tbody><tr><td>Market Leadership</td><td>Relevant technology<br>Innovative business process<br>Competitive position</td></tr><tr><td>Functionality</td><td>Consumer preference<br>Functional capability<br>Compatibility with third party</td></tr><tr><td>Quality</td><td>Reliability<br>Security<br>Information Quality<br>Configuration</td></tr><tr><td>Price</td><td>Service cost<br>Operating cost<br>Set-up cost</td></tr><tr><td>Implementation speed</td><td>Performance<br>Usability<br>Training</td></tr><tr><td>Interface with other systems</td><td>Data share<br>Compatibility with the system<br>Multi-level user<br>Flexibility</td></tr><tr><td>International orientation</td><td>National CRM<br>Web applications</td></tr></tbody></table>
# 4 The PIRCS and SMART Frameworks

PIRCS was proposed for free/open source ERP systems (FOS-ERP) [9]. PIRCS can be understood as a *meta-method*, given that it is composed by a series of procedures that should be adapted for specific purposes, according to the adopter's software evaluation culture and specific needs.

PIRCS is completely compatible with the simple multiattribute rating theory (SMART) model [10]. Olson [33] presented a SMART analysis of an ERP selection decision considering the seven criteria found on Table 3, taken from [20]. It is clear that there are many ways to approach incorporation of multiple criteria in ERP selection models. We have tried to demonstrate the importance of context with respect to the selection of these criteria. In the next section we present a model that aims at demonstrating context such as size of organization as important.

## 4.1 Demonstration Model for Small Business ERP Selection

It is possible to include many criteria, but it has been argued that a limited number of independent and equally scaled criteria will include the bulk of the relative importance [34].

The PIRCS framework for evaluation of ERP alternatives consists of the following steps:

* **Prepare**: define requirements, establish positioning strategy, identify attributes and constraints on the decision, and measures of attributes to be considered.

* **Identify**: Use searches to identify alternative ERP options and their characteristics.

* **Rate**: Establish the utility (value) of each attribute on each alternative.

* **Compare**: Apply multicriteria methods, such as AHP or SMART.

* **Select**: Consider the comparison analysis from the prior step and make the decision.

The focus of this paper is to demonstrate the use of SMART as a means to implement the Compare step.

We assume the context of a small business considering the criteria and options given in Table 8. The criteria would be identified in the **Prepare** step, and the options in the **Identify** step. The rating entries would be established in the third step, **Rate**. Criteria used here include cost, time, and robustness [9], as well as the most significant criteria identified for SMEs [19]. Ratings are scaled on a 0-1 range, with 0 indicating worst possible performance, and 1 indicating best possible performance. The assignment of these values should be done in the context of the organization, reflecting values of organizational decision makers. Table 8 shows the value matrix from the PIRCS Rate step, which is input to the SMART analysis. The entries in Table 8 are of course demonstrative. Cost values reflect best estimates of total life cycle costs. While OSS without support would be free with respect to software acquisition, there would be costs of implementing as well as training users.
<table><caption>**Table 8.** Value Matrix</caption><thead><tr><th></th><th>Cost</th><th>Time</th><th>Flexibility</th><th>Robustness</th><th>Support</th></tr></thead><tbody><tr><td>Large vendor</td><td>0.2</td><td>0.3</td><td>0.1</td><td>1.0</td><td>1.0</td></tr><tr><td>Customize vendor</td><td>0.0</td><td>0.0</td><td>0.8</td><td>0.7</td><td>0.5</td></tr><tr><td>Mid-size vendor</td><td>0.4</td><td>0.6</td><td>0.5</td><td>0.5</td><td>0.6</td></tr><tr><td>OSS with support fees</td><td>0.7</td><td>0.9</td><td>0.6</td><td>0.8</td><td>0.7</td></tr><tr><td>OSS without support</td><td>0.6</td><td>0.6</td><td>0.5</td><td>0.4</td><td>0.0</td></tr></tbody></table>

The fourth step is to **Compare**. Using the SMART approach, this includes identification of relative weights of importance (scale has been removed by identifying value ratings on 0-1 scales for all attributes). Use of swing weighting would begin by ordering criteria by importance, then assigning the most important criterion a value of 100. The other criteria are assessed in turn on the basis of: if the most important criterion was swung from its worst possible state to its best possible state, how relatively important would the next criterion be worth when swung from its worst possible state to its best possible state. Standardized weights are generated by dividing each assessed relative weighting by the sum of these relative weightings. Our demonstrative developed weights are presented in Table 9:

<table><caption>**Table 9.** Swing Weighting</caption><thead><tr><th>Criteria by order</th><th>Relative weighting</th><th>Standardized weighting (/320)</th></tr></thead><tbody><tr><td>Time</td><td>100</td><td>0.312</td></tr><tr><td>Robustness</td><td>80</td><td>0.250</td></tr><tr><td>Support</td><td>70</td><td>0.219</td></tr><tr><td>Cost</td><td>40</td><td>0.125</td></tr><tr><td>Flexibility</td><td>30</td><td>0.094</td></tr><tr><td>SUM</td><td>320</td><td>1.000</td></tr></tbody></table>

The **Select** step should be done judgmentally, by the organization's decision maker. The SMART analysis should be viewed in terms of decision support (not letting the model make the final decision). However, the PIRCS framework and SMART analysis will provide decision makers with a systematic means to consider important factors and provide greater confidence in the decision. Here the **Select** output would multiply the ratings in Table 8 by the weights in Table 9, yielding the relative scores shown in Table 10.

<table><caption>**Table 10.** Alternative Relative Scores</caption><thead><tr><th>Alternative</th><th>Score</th></tr></thead><tbody><tr><td>OSS with support fees</td><td>0.778</td></tr><tr><td>Large vendor</td><td>0.597</td></tr><tr><td>Mid-size vendor</td><td>0.541</td></tr><tr><td>ASP</td><td>0.446</td></tr><tr><td>OSS without support</td><td>0.409</td></tr><tr><td>Customize vendor</td><td>0.360</td></tr></tbody></table>
The implication is that the relatively moderate ratings over all attributes for the OSS with support fees option led to total value greater than that of the large vendor (which did very well on robustness and support, but very poorly on the other three criteria). The mid-size vendor was moderate on all criteria, but turned out to be dominated by the OSS with support fees option in the assumed context.

# 5 Conclusions and Future Research

There are many criteria that can be important in the selection of ERP systems. We have tried to show that the context in which such decisions are made is important. While there have been many studies of this matter, there is not universal agreement by any means. Furthermore, each individual organization should be expected to find various criteria critical while other criteria may be more important for other organizations. The ERP environment is also highly dynamic. In the 1990s, ERP was usually only feasible for large organizations. That is changing.

A business case for evaluation of software systems of any type is challenging. Cost estimates involve high levels of uncertainty, and benefits are usually in the realm of pure guesswork. A sound analytic approach is called for, especially given the large price tags usually present in ERP systems. There is a need for a method that can consider expected monetary impact along with other factors, to include risk elements such as project time and system robustness, as well as relatively subjective elements of value such as flexibility and availability of support. The PIRCS process and SMART multiattribute analysis offer a means to systematically evaluate ERP software proposals.

Multiattribute analysis has studied decision making under tradeoffs for a long time. It is quite robust, and can support consideration of a varying number of criteria. It usually is the case that for a specific decision, a relatively small number of criteria matter. If nothing else, the simple fact that if there are seven other criteria more important, the highest relative importance an eighth criterion could have is 0.125, with a high likelihood of a much lower weight [35]. This paper had the purpose of describing how the PIRCS framework could support the critical process of ERP software alternative evaluation, along with multiattribute analysis to consider the inevitable trade-offs that are encountered in such decisions. The demonstration of the combination of the PIRCS framework and the SMART analysis shows that the huge diversity of different options on ERP systems can be managed. The combined framework has a huge potential when it comes to deal with context related factors when making a selection on what option to implement. It also concisely shows tradeoffs among criteria being considered. However, the framework is dependent on that relevant and correct data on every option is possible to have. This is especially true with respect to life-cycle cost, which is very difficult to predict for most organizations, which hopefully do not have to repeat ERP selection decisions very often.

Future research is important in understanding what criteria are important in particular contexts. For instance, free open source ERP systems are emerging, broadening the market for enterprise system support.
References

1. Olson, D.L., Kesharwani, S. Enterprise Information Systems: Contemporary Trends and Issues. Singapore: World Scientific, 2010.

2. Poba-Nzaou, P., Raymond, L., Fabi, B. Adoption and Risk of ERP Systems in Manufacturing SMEs: A Positivist Case Study, Business Process Management Journal, 14(4), 530-550 (2008).

3. Kirytopoulos, K., Voulgaridou, D., Panopoulos, D., Leopoulosm V. Project Termination Analysis in SMEs: Making the Right Call, International Journal of Management & Decision Making 10(1/2), 69-90 (2009).

4. Wei, C.-C., Chien, C.-F., Wang, M.-J. An AHP-Based Approach to ERP System Selection, International Journal of Production Economics 96, 47-62 (2005).

5. De Carvalho, R.A. Free/Open Source Enterprise Resources Planning, In Jatinder Gupta, N.D., Rashid, M.A., Sharma, S.K.. (Org.). Handbook of Research on Enterprise Systems. Hershey, USA: Information Science Reference v. 32-44 (2009).

6. Baki, B., Çaki, K. Determining the ERP Package-Selecting Criteria: The Case of Turkish Manufacturing Companies, Business Process Management Journal 11(1), 75-86 (2005).

7. Johansson, B., de Carvalho, R.A. Management of Requirements in ERP Development: A Comparison Between Proprietary and Open Source ERP, ACM Symposium of Applied Computing March 8-12, Honolulu, Hawaii, 1605-1609 (2009).

8. Kabassi, K., Virvou, V. A Knowledge-Based Software Life-Cycle Framework for the Incorporation of Multicriteria Analysis in Intelligent User Interfaces, IEEE Transactions on Knowledge and Data Engineering 18(9), 1265-1277 (2006).

9. De Carvalho, R.A. Issues on Evaluating Free/Open Source ERP Systems, in International Federation for Information Processing, v. 205, Research and Practical Issues of Enterprise Information Systems, Tjoa, A.M., Xu, L., S Chaudhry, S. eds. Boston: Springer 667-675 (2006).

10. Edwards, W. Social Utilities, The Engineering Economist 6, 119-129 (1971).

11. Hecht, B. Choose the Right ERP Software, Datamation 43(3), 56-58 (1997).

12. Brewer, G. On the Road to Successful ERP, Instrumentation & Control Systems 73(5), 49-58 (2000).

13. Rao, S.S. Enterprise Resource Planning: Business Needs and Technologies, Industrial Management & Data Systems100(2), 81-88 (2000).

14. Verville, J., Hallingten, A. An Investigation of the Decision Process for Selecting an ERP Software: The Case of ESC, Management Decision 40(3), 206-216 (2002).

15. Kumar, V., Maheshwari, B., Kumar, U. An Investigation of Critical Management Issues in ERP Implementation: Empirical Evidence from Canadian Organizations, Technovation 23, 793-807 (2003).

16. Reuther, D. Critical Factors for Enterprise Resources Planning System Selection and Implementation Projects within Small to Medium Enterprises, International Engineering Management Conference 851-855 (2004).

17. Rittammanart, N., Wongyued, W., Dailey, M.N. ERP Application Development Frameworks: Case Study and Evaluation, Proceedings of ECTI-CON 173-176 (2008).

18. Baharum, Z., Ngadiman, M.S., Baron, H. Critical Factors to Ensure the Success of OS-ERP Implementation Based on Technical Requirement Point of View, Third Asia International Conference on Modelling & Simulation 419-424 (2009).

19. Benroider, E., Koch, S. ERP Selection Process in Midsize and Large Organizations, Business Process Management Journal 7(3), 251-257 (2001).

20. Ekanayaka, Y., Currie, W.L., Selsikas, P. Evaluating Application Service Providers, Benchmarking: An International Journal 10(4), 343-354 (2003).
21. Kahraman, C., Beskese, A., Kaya, I. Selection Among ERP Outsourcing Alternatives Using a Fuzzy Multi-Criteria Decision Making Methodology, International Journal of Production Research 48(2), 547-566 (2009).

22. Keil, M., Tiwana, A. Relative Importance of Evaluation Criteria for Enterprise Systems: A Conjoint Study, Information Systems Journal 16, 237-262 (2006).

23. Saaty, T.L. A Scaling Method for Priorities in Hierarchical Structures, Journal of Mathematical Psychology 15, 234-281 (1977).

24. Ahn, B.S., Choi, S.H. ERP system selection using a simulation-based AHP approach: A case of Korean homeshopping company, Journal of the Operational Research Society 59(3), 322-330 (2008).

25. Salmeron, J.L., Lopez, C. A multicriteria approach for risks assessment in ERP maintenance, Journal of Systems & Software 83(10), 1941-1953 (2010).

26. Onut, S., Efendigil, T. A theoretical model design for ERP software selection process under the constraints of cost and quality: A fuzzy approach, Journal of Intelligent & Fuzzy Systems 21(6), 365-378 (2010).

27. Saaty, T.L. The Analytic Network Process: Decision Making with Dependence and Feedback. Pittsburgh, PA: RWS Publications (1996).

28. Ayağ, Z., Özdemir, R.G. An intelligent approach to ERP software selection through fuzzy ANP, International Journal of Production Research 45(10, 2169-2194 (2007).

29. Kirytopooulos, K., Voulgaridou, D., Panopoulos, D., Leopoulos, V. Project termination analysis in SMEs: Making the right call, International Journal of Management & Decision Making 10(1/2), 69-90 (2009).

30. Olson, D.L., Wu, D.D. Multiple criteria analysis for evaluation of information system risk, Asia-Pacific Journal of Operational Research 28(11), 25-39 (2011).

31. Ünal, C., Güner, M.G. Selection of ERP Suppliers Using AHP Tools in the Clothing Industry, International Journal of Clothing Science and Technology 21(4), 239-251 (2009).

32. Perçin, S. Using the ANP Approach in Selecting and Benchmarking ERP Systems, Benchmarking: An International Journal 15(5), 630-649 (2008).

33. Olson, D.L. Evaluation of ERP outsourcing, Computers & Operations Research 34(12), 3715-3724 (2007).

34. Olson, D.L. Decision Aids for Selection Problems. New York: Springer (1996).

35. Miller, G. The Magical Number Seven Plus or Minus Two: Some Limits on Our Capacity for Processing Information, Psychological Review 3(2), 81-97 (1956).