# A Structured Methodology to Implement Performance Measurement Systems in Collaborative Networks

María-José Verdecho, Raúl Rodríguez-Rodríguez, Juan-José Alfaro-Saiz

► To cite this version:

María-José Verdecho, Raúl Rodríguez-Rodríguez, Juan-José Alfaro-Saiz. A Structured Methodology to Implement Performance Measurement Systems in Collaborative Networks. 12th Working Conference on Virtual Enterprises (PROVE), Oct 2011, São Paulo, Brazil. pp.151-158, 10.1007/978-3-642-23330-2_17. hal-01570005

HAL Id: hal-01570005

https://inria.hal.science/hal-01570005

Submitted on 28 Jul 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Structured Methodology to Implement Performance Measurement Systems in Collaborative Networks

María-José Verdecho¹, Raúl Rodríguez-Rodríguez¹, Juan-José Alfaro-Saiz¹

¹ Department of Business Organization, CIGIP (Research Centre on Production Management and Engineering), Universitat Politècnica de València, Camino de Vera, s/n, 46022, Valencia, Spain [mverdecho, jalfaro, raurodro]@cigip.upv.es

**Abstract.** Collaboration is one of the most commonly used strategies in the business environment. Enterprises that collaborate need reliable and efficient performance information. However, many collaborative enterprises fail to implement a common performance measurement system in an adequate manner due to the lack of sound mechanisms that connect all the elements within a performance measurement system. The purpose of this paper is to provide a methodology based on a multi-criteria decision method that aids to implement the different elements that compose a performance measurement system for collaborative networks. With this methodology enterprises have a tool to define all the aspects involved in the performance measurement system and to implement it. This approach has been applied to a collaborative enterprise network belonging to the automotive sector.

**Keywords:** performance measurement systems, collaborative networks, performance management.

## 1 Introduction

Collaboration is one of the business models that enterprises adopt to remain competitive. Companies collaborating share the responsibility of exchanging common planning, management, execution, and performance measurement information [1]. In addition, collaboration requires the commitment of human, financial or technical resources to build and sustain a more competitive business model [2]. For that reason, it is necessary to manage the evolution of the collaborative relationship by defining performance measurement systems (PMS) that aid to collect the necessary information in order to manage the activities of the collaborative association. In the literature, there has been an increasing interest in the development of PMS for inter-organizational contexts [3-6] and recently some PMS were developed for specific collaborative contexts [7-9]. This is because, as collaborative relationships present specific characteristics that need to be managed such as trust, commitment, cooperation, etc. [10-11]. Then, PMS developed for inter-organizational contexts should present specific characteristics to be useful for these contexts. One of this characteristic is related to the need that the design and implementation phases of PMS are coordinated in order to build them adequately. In this sense, decision makers usually deal with other important issues such as the prioritization of performance
objectives in order to know which the most strategic objectives are and, therefore, in which aspects enterprises should be focused and allocate resources. Despite the importance of these activities, the inter-organizational PMS developed in the literature present some limitations for providing weights [12]. Then, PMS should incorporate methods for solving these issues in order to get an adequate PMS design and implementation.

In order to prioritize objectives, it is useful to define weights for the different objectives which can be stated as a multi-criteria problem involving different actors. In the same vein, structuring and consolidating data may also be solved as a multi-criteria problem. Therefore, multi-criteria methods can contribute to the elaboration and establishment of the performance measurement systems. The main goal of this paper is to propose a novel methodology to design and implement PMS in collaborative networks by using a multi-criteria decision-making approach. With this methodology, enterprises will have a tool that describes the steps to be followed by all the collaborating enterprises in order to define, reach agreement and manage the evolution of the collaborative enterprise network. Then, this methodology is developed to manage the performance of the association of enterprises instead of the individual enterprises. Evolution is managed by analyzing the performance reached by the whole association at the different periods and deriving, defining and implementing actions plans when performance is not achieved.

The structure of this paper is as follows. First, a literature review of inter-organizational PMS and multi-criteria decision analysis methods are presented focusing attention on the Analytic Hierarchy Process (AHP) method. Then, the methodology proposed to implement performance measurement systems is presented. Next, a case study of a collaborative enterprise network belonging to the automotive sector is described. Finally, conclusions are exposed.

## 2 Literature Review

Collaborative enterprise networks involve that different partners are working together for mutual benefits. Therefore, the definition of PMS for managing its activity is a task that has to be made by all the collaborative partners. There are several PMS developed in the literature for inter-organizational contexts such as the works [3-9]. These PMS are useful structures for defining and structuring performance measurement information from the strategic level to the process level by deploying linkages among the performance elements. However, they lack of mechanisms to prioritize the performance measurement elements data so that performance can be analyzed under an assessment making preference to those aspects that are more relevant for the sustainability of the collaborative enterprise network.

On the other side, there are many multi-criteria methods. In fact, multi-criteria methods are classified into two groups [13]: Multiple Objective Decision Methods (MODM) and Multiple Attribute Decision Methods (MADM). The main difference between both methods is that in MODM, decision makers need to reach multiple objectives that are in conflict of interest and the number of alternatives is elevated. However, MADM problems focus on obtaining a preference ranking on the
alternatives characterized by multiple attributes that are in conflict and the number of
alternatives is limited. In our problem, the number of performance objectives is
limited and discrete so that MADM techniques are to be used. There are several
MADM methods such as Multiple Attribute Utility Theory, (MAUT) [14], ELECTRE
(ELimination Et Choix Traduisant la REalité) [15], PROMETHEE (Preference
Ranking Organization METhod for Enrichment Evaluations) [16] and the Analytic
Hierarchy Process, AHP [17].

AHP aims at integrating different measures into a single assessment for ranking
decision alternatives which is the case of our problem. Also, ranking of objectives
involves both tangible and intangible aspects and AHP deals with both types of
aspects. This is because overall assessment should not only consider quantitative
performance data but also some other criteria that are critical for successful
partnerships and are not directly quantifiable, e.g. trust and commitment [18]. In
addition, AHP has been used for many applications involving performance
measurement criteria such as selecting a supplier [19-21], selecting performance
indicators for supply chain management [22], evaluating performance of IT
department in the manufacturing industry in Taiwan [23], and selecting ERP systems
in textile industry [24]. However, there is not an application of AHP for supporting
PMS design and implementation within inter-organizational contexts. For this reason,
the purpose of the remaining of this paper is to present a methodology that uses AHP
method to support the prioritization of the objectives within a PMS and the
composition of an assessment for the whole collaborative network that fills this
research gap.

3 The Proposed Methodology to Support the Implementation of PMS in Collaborative Contexts

In [19], the COL-PMS framework for managing performance within collaborative
contexts is presented. The COL-PMS framework integrates the social side of
collaboration within an integrated and solid PMS structure. The main purpose of this
PMS is to support the decision-making process of the enterprises and entities that
collaborate. For that reason, it is necessary that the PMS considers two levels: inter-
organizational level (where collaboration takes place) and individual enterprise level.
Fig. 1 shows the phases of the methodology proposed to implement the COL-PMS
framework by using AHP which is composed of seven phases. In the phase 1, the
performance elements of the PMS are defined. In this work, the PMS structure used is
the COL-PMS but other PMS structures may be used. Then, the AHP method is
applied to build a model (phase 2). The AHP method structures the decision problem
in a hierarchy of levels. These levels are linked by unidirectional dependence
relationships. In the upper level of the hierarchy, it is defined the ultimate goal of the
decision problem. Then, the criteria that contribute to achieve the goal stand in the
second level. In the next levels, intermediate sub-criteria and attributes that compose
the hierarchical structure are located. Finally, in the last level, the decision
alternatives are established. Using levels allows decision makers to focus on a small
set of decisions [17]. The AHP method provides relative weights to each element
within a level depending on its contribution to an element linked to it that is located
on the immediate upper level. In our case, as we use the AHP model to obtain the
weights of the performance objectives, we will have three levels (see Fig. 2): PMS
vision (main goal), perspectives (criteria) and, finally, performance objectives
(alternatives).

<figure><img src="image_2.png" /><figcaption>Fig. 1. Methodology for implementing a PMS by using AHP</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. AHP model</figcaption></figure>

Once the model is built, it is needed to make pairwise comparisons in each level using the fundamental scale of Saaty [17], in order to obtain the pairwise comparison matrices (phase 3). After pairwise comparisons matrices are completed, the local priorities of the compared elements (priority vector) are calculated and consistency of judgements checked following the procedure described in [17]. Once all priority vectors are obtained and consistency verified, the final weights for the alternatives are calculated (phase 4). For that purpose, priorities of decision alternatives are combined
together with the sets of priorities of the criteria. Then, in phase 5, it is performed a sensitive analysis to check how changes in the local weights of one of the criteria or alternatives affect the final priorities previously obtained. The purpose of this phase is to verify that the solution obtained as the preferred alternative is robust enough. In case that the solution is not robust, it is needed to go back into the phase 3 to analyze the pairwise comparison matrices defined. In phase 6, the data regarding the performance indicators is collected. Finally, phase 7, it is obtained the overall performance evaluation by multiplying the priority of every performance objective (given by the normalized priority) and the value reached in its corresponding performance indicator.

## 4 Case Study

The methodology has been applied to a collaborative enterprise network belonging to the automotive industry in Spain which is composed raw material suppliers, design centres and manufacturing plants (of a plastic injection-moulding leading group) working for main OEMs. Due to the success of a new product launched, and the new market needs regarding this new product, the corporation pursued to change the policy and orientation towards collaborative working in order to agree the terms, commitment, money investments and resource and information sharing. One of the components of this product is critical as the technology and know-how required to design it and manufacture it is not a core capability of the manufacturer therefore collaborative working seem reasonable.

The first phase of the methodology consists of the definition of the PMS elements. Managing directors of the different enterprises assessed were in charge of the decision making of the different phases of the methodology. It took several meetings to reach an agreement on the strategic aspects of the relationship as the enterprises come with different backgrounds and understand in a different way the main components for the business vision. Once the global performance elements were defined, the different enterprises performed some adjustments into their performance indicators at the local level to accommodate all the relevant performance information coherently. Table 1 shows the performance elements (objectives and KPIs) defined for the collaborative enterprise network level. It consists of fourteen objectives and KPIs (divided into the five performance perspectives) which seems a reasonable number. These KPIs have been defined by the enterprises based on the objectives that they want to reach. It took two meetings to define these KPIs.

In the phase 3, in order to fill in the pairwise comparison matrices it was necessary to establish an initial meeting for explaining the AHP method. As this phase involves decision makers of all the enterprises, several possibilities can be used to obtain a final solution such as voting, aggregation methods on individual preferences or reaching consensus [17]. The companies agreed to use the aggregation on individual preferences if the initial judgement was equal or similar for all of them. However, if there was a large difference of judgement, this point needed to be dealt with in meetings for reaching consensus or a closer point of view. In order to reach consensus, the managing directors discuss the preferences until a better understanding
is achieved. Therefore, a combined solution was applied in order to get an approximate common view of the problem. Then, enterprises meet several times in order to fill in the pairwise comparison matrices. In other cases, if consensus is not reached, other methods can be used such as voting.

<table><caption>**Table 1.** Performance elements of the collaborative enterprise network.</caption><thead><tr><th>Perspect.</th><th>Objectives</th><th>KPIs</th></tr></thead><tbody><tr><td rowspan="2">Financial</td><td>FO1 Increase sales</td><td>KPI1 = sales (monthly)</td></tr><tr><td>FO2 Increase high quality product margins</td><td>KPI2 = average of high quality products margin variation (monthly)</td></tr><tr><td rowspan="3">Customer</td><td>CPO1 Increase market share</td><td>KPI3 = market share variation (quarterly)</td></tr><tr><td>CPO2 Increase customer loyalty</td><td>KPI4 = number of old customer purchases/ total purchases (quarterly)</td></tr><tr><td>CPO3 Increase customer satisfaction</td><td>KPI5 = number of claims/complaints (weekly)</td></tr><tr><td rowspan="4">Process</td><td>PO1 Decrease production lead time</td><td>KPI6 = production lead time variation (monthly)</td></tr><tr><td>PO2 Decrease new product development lead time</td><td>KPI7 = new product development lead time variation (per project)</td></tr><tr><td>PO3 Implement initiatives for production performance improvement</td><td>KPI8 = number of performance improvement initiatives implemented (semester)</td></tr><tr><td>PO4 Decrease non-conforming parts</td><td>KPI9 = number of non-conformance parts send to customer (weekly) (by customer)</td></tr><tr><td rowspan="2">Learning & Growth</td><td>LGO1 Increase innovation capability</td><td>KPI10 = Number of innovative products proposed (annual)</td></tr><tr><td>LGO2 Improve engineering knowledge</td><td>KPI11 = number of common engineering development training sessions performed (semester)</td></tr><tr><td rowspan="3">Collaboration</td><td>CO1 Increase commitment</td><td>KPI12 = Number of agreements failed (monthly)</td></tr><tr><td>CO2 Increase coordination</td><td>KPI13 = number of times that information is sent delayed (monthly)</td></tr><tr><td>CO3 Maintain equity</td><td>KPI14 = meetings to discuss perception of equity after agreement (quarterly)</td></tr></tbody></table>

In the phase 4, the weights of the objectives are obtained. Results showed that the most important objectives representing more than 70% of the total weight were: FO1 Increase sales (with normalized weight of 0,16), CPO1 Increase market share (0,12), FO2 Increase high quality product margins (0,11), CPO3 Increase customer satisfaction (0,09), PO4 Decrease non-conforming parts (0,09), LGO1 Increase innovation capability (0,08) and CO3 Maintain equity (0,08). It can be observed that the relevant objectives belong to all the performance perspectives but the importance of the perspectives differ, being the financial perspective the most relevant followed by the customer perspective. It is to be noted that the objective CO3 Maintain equity reaches a weight of 0,08. This fact notes the importance of social climate aspects for sustaining collaboration in the long time.
In the last phase it is obtained the final results. The analysis showed that performance was mainly achieved by some of the most relevant objectives (those objectives with highest weight). However, financial objectives were not accomplished in the desired level (accomplished around 20%) and decision makers have to reassess what is expected in current market conditions. Another objective that is not achieved is the decreasing of non-conforming parts (accomplished around 50%). For those objectives that have not reached the expected results, actions plans are to be developed which allows reassessing the current targets or reaching them properly. It has to be noted that performance achievement was only reached at the 60% what was under the initial expectations (70%). However, results showed that performance measurement implementation has provided performance knowledge to the collaborative network as well as a tool for monitoring performance from now on.

From an information systems point of view, it has to be said that a web-based tool is being built in order to make possible to implement the system. Some of the next steps involve completing the tool and testing it, training further the people that are going to use it, move the data into the application and feed the system with updated data in order to analyze how the performance of the collaborative enterprise network evolves.

## 5 Conclusions

The literature counts with inter-organizational PMS that are useful structures for defining performance measurement systems. However, they lack of mechanism to prioritize the performance measurement elements data. This paper introduces a structured methodology supported by a multi-criteria method to aids to implement PMS for collaborative networks aiming to fill this research gap. Adaptations of this methodology can be made to manage other inter-enterprise relationships such as supply chains, etc. Further research work will involve the full development of the web tool in order to not only provide the storage of data and calculation of the results but also deals with the report analysis settings. In addition, it is needed to further testing the web tool and observe its evolution as well as the evolution of the PMS and the performance of the collaborative enterprise network in the long time. Other interesting research line is the validation of the proposed methodology in other contexts as well as the adaptation of other PMS and MADM techniques.

**Acknowledgments.** This work has been developed within the framework of a research project funded by the Polytechnic University of Valencia, titled "Design and Implementation of Performance Measurement Systems within Collaborative Contexts for aiding the Decision-making Process", reference PAID-06-08-3206.

## References

1. Anthony, T. Supply chain collaboration: success in the new internet economy [online]. ASCET. http://anthony.ASCET.com.
2. Bowersox, D.J., Closs, D.J., Stank, T.P.: How to master cross-enterprise collaboration. Supply Chain Management Review, 7, 4, 18-27 (2003).

3. Brewer, P.C., Speh, T.W.: Using the Balanced ScoreCard to measure supply chain performance. Journal of Business Logistics. 21, 1, 75-93 (2000)

4. Bititci, U.S., Mendibil, K., Martinez, V., Albores, P.: Measuring and managing performance in extended enterprises. Intern. J. of Operations & Production Management. 25, 4, 333-353 (2005)

5. Folan, P.; Browne, J.: Development of an Extended Enterprise Performance Measurement System. Production Planning and Control, 16, 6, 531-544 (2005)

6. Alfaro, JJ., Ortiz, A., Rodríguez, R.: Performance measurement system for Enterprise Networks. International Journal of Productivity and Performance Management, 56, 4, 305-334 (2007)

7. Romero, D., Galeano, N., Molina, A.: A conceptual Model for Virtual Breeding Environments Value System. Establishing the Foundation of Collaborative Networks. Ed. L. Camarinha-Matos, H. Afsarmanesh, P. Novais, C. Analide. Springer. (2007)

8. Msanjila, S.S., Afsarmanesh, H.: Trust analysis and assessment in virtual organization breeding environments. International Journal of Production Research, 46, 5, 1253-1295. (2008)

9. Westphal I, Thoben KD, Seifert M. "Measuring collaboration performance in virtual organizations". In Establishing the Foundation of Collaborative Networks, Luis Camarina-Matos, Hamideh Afsarmanesh, Paulo Novais, Cesar Analide, eds. Boston: Springer, 2007.

10. Verdecho, M.J., Alfaro J.J., Rodriguez-Rodriguez, R.: Foundations for collaborative performance measurement. Production Planning and Control. 20, 3, 193-205 (2009).

11. Verdecho, M.J., Alfaro J.J., Rodriguez-Rodriguez, R. (2010). COL-PMS: a collaborative performance measurement system. In: Ortiz Bas, A., Franco, R.D., Gómez Gasquet, P. (eds.) BASYS 2010, IFIP AICT vol. 322, pp. 56-67. Springer, Heidelberg (2010).

12. Yüksel, I., Dagdeviren, M.: Using the fuzzy analytic network process (ANP) for Balanced Scorecard (BSC): A case study for a manufacturing firm, Expert Systems with Applications 37, 1270-1278 (2010).

13. Barba-Romero, S., Pomerol, JC. Decisiones Multicriterio: fundamentos teóricos y utilización práctica. Editorial: Universidad de Alcalá de Henares. Alcalá de Henares (1997).

14. Fishburn, PC.: Independence in utility theory with whole product sets. Operations Research, 13, 28-45 (1965).

15. Roy, B. Classement et choix en présence de points de vue multiples: la méthode ELECTRE. Revue Française d'Informatique et de Reserche Opérationnelle, 8, 57-75 (1968).

16. Brans, JP., Vincke, PH., Mareschal, B. How to select and how to rank projects: the PROMETHEE method. European Journal of Operational Research, 24, 228-238 (1986).

17. Saaty, T.L. The Analytic Hierarchy Process. McGraw-Hill, New York (1980)

18. Mohr, J. Spekman, R. Characteristics of partnership success: Partnership attributes, communication. Strategic Management Journal, 15, 2: 135-152 (1994).

19. Verdecho, M.J., Alfaro J.J., Rodriguez-Rodriguez, R. A multi-criteria approach to select suppliers based on performance. In: Ortiz Bas, A., Franco, R.D., Gómez Gasquet, P. (eds.) BASYS 2010, IFIP AICT vol. 322, pp.47-55 Springer, Heidelberg (2010).

20. Masella, C. & Rangone, A. A contigent approach to the design of vendor selection systems for different types of co-operative customer/ supplier relationships. International Journal of Operations & Production Management, 20, 1, 70-84 (2000).

21. Ghodspour SH, O'Brien, C. A decision support system for supplier selection using an integrated analytic hierarchy process and linear programming. International Journal of Production Economics, 56-57; 199-212 (1998).

22. Bhagwat R, Sharma, MK. Performance measurement of supply chain management using the analytical hierarchy process. Production Planning & Control, 18, 8; 666-680 (2007).

23. Lee AHI, Chen WC, Chang CJ. A fuzzy AHP and BSC approach for evaluating performance of IT department in the manufacturing industry in Taiwan. Expert Systems with Applications, 34, 1: 96-107 (2008).

24. Cebeci U. Fuzzy AHP-based decision support system for selecting ERP systems in textile industry by using balanced scorecard, Expert Systems with Applications, 36; 8900-8909 (2009).