# A Collaborative Decision Support Method to Design Performance Evaluation Systems in CNOs

Rui Pinto Ferreira, António Lucas Soares

► To cite this version:

Rui Pinto Ferreira, António Lucas Soares. A Collaborative Decision Support Method to Design Performance Evaluation Systems in CNOs. 11th IFIP WG 5.5 Working Conference on Virtual Enterprises (PRO-VE), Oct 2010, Saint-Etienne, France. pp.561-568, 10.1007/978-3-642-15961-9_67 hal-01055946

HAL Id: hal-01055946

https://inria.hal.science/hal-01055946

Submitted on 25 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Collaborative Decision Support Method to Design Performance Evaluation Systems in CNOs

Rui Pinto Ferreira², António Lucas Soares<fnref n="1" />,²,

¹ INESC Porto, Campus da FEUP, Rua Dr. Roberto Frias, 378, 4200-465 Porto, Portugal

² DEI, FEUP, University of Porto, Rua Dr. Roberto Frias, sn 4200-465 Porto, Portugal
[rui@omegadata.pt; asoares@inescporto.pt]

**Abstract.** This paper describes an approach to the design of performance management models in collaborative networks, from a constructivist approach. The general approach is outlined and the part concerning a multi-criteria decision support method is explained. The paper finishes with an illustration of a scenario for test and validation of the method.

**Keywords:** Performance management; collaborative networks; constructivism.

## 1 Introduction

The complexity of network evaluation arises from at least four problems mentioned by [1]: (i) the choice of appropriate evaluation criteria and indicators; (ii) network evaluation should be approached on different levels of analysis; (iii) network structural properties may involve outcomes that are also potential inputs in network processes; and (iv) there are multiple stakeholders involved who will have multiple interests. Managing collaborative networks of organizations (CNO) clearly involves a set of multi-criteria problems. In particular, understanding what are the objectives of the individual members of the network as well as the global network objectives, for the different types of collaborative structures, is an important research topic. It is difficult to prove the advantages of CNOs, but according to [2], the ability to measure the performance of a collaborative network as a whole, as well as the individual performance of each of its members, could represent an important boosting element for the wide acceptance of the paradigm. The collective construction of the performance evaluation (or more broadly, management) model, for a given situation and time-frame is, at least, as important as the use of the performance management system. In this paper, an approach to the design of network performance information models is described. The *pmColNet*<fnref n="1" /> approach adopts a constructivist perspective, providing the methodological and IT tools for the joint development of contextual and situational models fostering collaboration and negotiation. The paper focus on the multi-criteria decision support method for the negotiated agreement of the model.

<footnote n="1">¹ This paper describes results of the pmColNet project funded by the Fundação para a Ciência e Tecnologia under the contract PTDC/GES/71482/2006.</footnote>
## 2 The pmColNet approach

Nowadays, we assist three major transformations in performance measurement: (i) "from performance measurement to performance management"; (ii) "from individual to collaborative performance measurement"; and (iii) "from lagging to leading performance management". Performance management is then defined as: "the use of performance measurement information to effect positive change in organizational culture, systems and processes, by helping to set agreed-upon performance goals, allocating and prioritising resources, informing managers to either confirm or change current policy or programme directions to meet those goals, and sharing results of performance in pursuing those goals".

A supporting performance management system would include the following key elements [3]: (i) a structured methodology to design the performance measurement system; (ii) a structured management-process for using performance measurement information to help make decisions, set performance goals, allocate resources, inform management, and report success; (iii) a set of requirements specifications of the necessary electronic tools for data gathering, processing and analysis; (iv) theoretical guidelines on how to manage through measures (performance management systems are used to apply the information and knowledge arising from performance measurement systems); and (v) a review process to ensure that measures are constantly updated to reflect changes in strategy and/or market conditions.

The pmColNet approach addresses the point (i) of the above requirements, as applied to CNOs, by providing models and methods to: (i) develop and manage conceptualizations (models) of performance management systems, both individually and in collaboration through information and knowledge management tools and (ii) negotiate a network performance information model through a multi-criteria decision support method. The fundamental premise in pmColNet is that the socio-economic environment in which CNOs operate changes increasingly faster, meaning that CNOs need to constantly adapt to their specific contexts. This results in manifestations of CNOs that either result in an increasing rate of formation/dissolution (joint projects, partnerships, virtual enterprises, etc.) or, when operating in a more long-term, need to reassess goals and objectives more often. Consequently, CNOs performance models are required to be designed/adapted more frequently than before, be it when a new short-term oriented CNO is formed or when a long-term oriented one recognizes the need to reassess the existing performance model.

Roughly speaking, and from an information and knowledge management point of view<fnref n="2" />, the pmColNet approach has two moments (see figure 1): firstly the elaboration of an individual *i* (the organization as member of the network) performance information model and secondly the negotiation and synthesis of a shared network N performance information model. Both moments require the adequate tools to support the individual and collective processes of domain conceptualization, information organisation and retrieval. A reference performance information model (*R_PInfoModel*) was developed, aimed at being used as a structured "catalogue" of perspectives, criteria and indicators to be selected for a performance management information model. From the reference model, each network member *i* derives its own

<footnote n="2">2 This part of the pmColNet approach is only outlined here.</footnote>
performance information model (*i_PInfoModel*). The final network performance information model (*N_PInfoModel*), for a given time frame and network composition and structure, is a subset of the reference model and is achieved by negotiation under multi-criteria decision support method described in section 3. The *r_PInfoModel* considers aspects such as network typology, stakeholders, the framework (objectives/tasks/periods and ranges of evaluation time), evaluation levels, perspectives, criteria, sub-criteria and indicators.

<figure><img src="image_2.png" /><figcaption>Fig. 1. The *pmColNet* approach</figcaption></figure>

## 3 The pmColNet collaborative decision support method

The multi-criteria decision support model and method is at the core of the collaborative construction and negotiation phase of the *pmColNet* approach. In the beginning of this phase each of the members of the collaborative network should have conceptualised their individual view of the performance management information model. In the following paragraphs the fundamental components of the method are described.
**HPFA.** *pmColNet* decision-support model is structured along a very simple Hierarchical Framework for Performance Analysis (HFPA) (see figure 2). Criteria are forms of aggregation; they represent a common feature within different indicators. Within a specific perspective, a performance element is an indicator or a criterion or an objective. But, within two different perspectives, an element can be an objective for perspective A and an indicator for perspective B. In real life applications, hierarchical performance elements classification is sometimes subjective, that's why it follows a specific perspective. A perspective is just a specific materialisation, for a given situation, of the hierarchical performance elements.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** HPFA used in the *pmColNet* approach</figcaption></figure>

**Visual pairwise judgment.** Multi-criteria decision making is based on comparing different alternatives. *pmColNet* will use a visual rule to attain preference intensity, based on MCGDSS [4]. *pmColNet* users will simply click and drag a bar, starting at "Equal importance", dropping it at the exact point they feel the visual relation correctly manifests their importance judgement.

**Aggregating preferences.** Partners will rank HFPA alternatives using a 5 degree visual judgment. Using eigenvectors of the reciprocal matrices obtained, alternatives will be ranked. Nevertheless, final decision will ultimately be delivered with a final partner meeting, where interaction and negotiation will prevail over the method. *pmColNet* will promote vote power definition and negotiation. If this feature is not used, partners will be considered as having equal vote power.

The decision support method is outlined in table 1. The method is designed with a special emphasis in partner interaction and “decision overriding”. This means that *pmColNet* results, as indicating in a certain way, or using certain performance elements may be contradicted by network members applying any different method at any phase. *pmColNet* has the ability to incorporate this “manual” network decision and carry on to the next phase. In fact, every network decision will be supported by the structured *pmColNet* suggestion and by an unstructured partner’s meeting, where the partners can freely discuss and achieve a different path of the structured proposal. The Analytic Hierarchy Process (AHP) is the multiple criteria decision support method used in the *pmColNet* approach. The method has some limitations, but they
are minimised in our approach. Rank reversal problems can be minimized, because in a real network analysis partners will include only the relevant alternatives. Each alternative will be at least relevant for one particular network partner. Reducing the amplitude of the scale, for instance by using 1 to 5, over 1 to 9, inconsistency problems are also reduced. Because all the network partners are experts in the particular situation, 1 to 5 scale is perfectly adapted. Despite of the proposed pairwise verbal judgements, which are common to AHP, Macbeth or even Electre methods, and the prescriptive results, *pmColNet* definitely focus on interaction and commitment agreement, accordingly with its constructivist perspective. In other words, the prescriptive results regarding objectives, criteria and indicators should never be taken as final, but as a starting point for discussion.

<table><caption>**Table 1.** Steps of the *pmColNet* multi-criteria decision support model.</caption><thead><tr><th>Description</th><th>Method/characteristics</th></tr></thead><tbody><tr><td><strong>Phase A – Contextualisation</strong><br>I. Initial setup<br>II. Partner's individual performance model<br>III. Network typology<br>IV. Defining vote power</td><td>Web form/ PROMETHEE GDSS framework;<br>Tree structure; Private folders; HFPA;<br>Commitment level</td></tr><tr><td><strong>Phase B – Partner's proposals</strong></td><td>HFPA</td></tr><tr><td><strong>Phase C – Measuring preference intensity</strong></td><td>Visual pairwise judgment; AHP; Veto threshold</td></tr><tr><td><strong>Phase D – Negotiating and achieving a commitment package</strong><br>I. Network global commitment level<br>II. Defining failure and success<br>III. Levels of analysis<br>IV. Performance relationships<br>V. Index construction</td><td>Weighted Arithmetic Mean; Index construction</td></tr><tr><td><strong>Phase E – Monitoring performance</strong><br>I. Normalising indicators and achieving a dashboard/ tableau de bord<br>II. Ideal performance – TOPSIS<br>III. Relationship between individual and network performance model</td><td>Normalisation; Topsis</td></tr></tbody></table>

## 4 A scenario for test and validation

Two detailed scenarios were set up to demonstrate and validate the pmColNet approach. The first one deals with a CNO aimed at building a logistic planning information system and the second with an industry cluster with several strategic cooperative agreements, to collectively design, produce and distribute their textile goods. In this section some relevant aspects of the former are described.

Three organisations will start this year a new collaborative project: a Business School (BS); a Technology SME (SME); and an Industry Association (ASSOC).
These three organisations are used to collaborate in past projects and have achieved their objectives in a satisfactory way. Nevertheless, in spite of the institutional trust they deposit in each other, they wish to improve their collaborative network efficiency, starting to monitor their performance through the *pmColNet* system.

### Phase A - I, II, III and IV (see table 1)

In the initial phase of the method, each network member estimates the effort allocated to the network activities. They also build an individual performance evaluation model (individual HPFA) using the *r_PInfoModel* and the pmColNet collaborative modelling tool. The commitment level of the member towards the network is also defined at this stage (e.g., % allocated research hours or % of allocated investment). Type of inter-organisational governance: strategic cooperative agreements - "contractual business networks based on joint multi-party strategic control, with the partners collaborating over key strategic decisions and sharing responsibilities for performance outcomes." [5].

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Scenario #1: BS *individual* HFPA (*i_PInfoModel*)</figcaption></figure>

Type of network regarding its objective: Technological/functional synergies network - a network of organisations that collaborate to obtain functional synergies in support chain value areas: such as R&D, marketing and logistics [6]. Network main objective: As said before, the project final objective is building a logistic planning information system adapted to the scenario industry. All partners agreed on that.

### Phase B - Partner's proposals (see table 1)
A partner can propose several different alternatives for each of the network main objectives. If he does so, he must evaluate the several alternatives obtaining an ordered list according to its preferences and using the visual pairwise judgments and aggregation methods mentioned before. Each HFPA's element may have a description field and indicators must have additional scaling description, so that every partner correctly understands the proposal.

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /><figcaption>**Figure 4.** Scenario #1: SME network HFPA proposal and pairwise judgement</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 5.** Scenario#1 extract of the network HFPA (N_PInfoModel) showing the final</figcaption></figure>
global priorities for a distribution of vote power of BS=0.50, SME=0.17 and ASSOC=0.33

Phase C - *Measuring preference intensity* (see table 1)

Each partner evaluates the different HFPA on a pairwise visual judgement (see figure 4 for an example of measuring preferences). Applying the AHP methodology and the defined vote power, the ASSOC HFPA was out of the veto threshold and therefore was excluded from the final network performance model. Next, partners evaluate criteria of the two winning perspectives, once again applying visual pairwise judgements and AHP. The final result is depicted in the form of a concept map in figure 5 (network HFPA).

**5 Conclusions and further work**

*pmColNet* is an innovative approach to the collaborative design of network performance models in CNOs, combining information&knowledge management techniques with multi-criteria decision analysis ones. In this paper, a partial account of the later was presented, focusing on the steps necessary to achieve an agreed network performance model. From an epistemological perspective, the approach here presented can be considered as constructivist. Agreements regarding the network performance models are collectively built through concept maps explicitation and interpretation and through relational multi-criteria methods. Further work will involve the ability to store past network context, HFPA and results. This would enable for new networks to be advised for similar HFPA regarding its contextualisation: “similar networks have chosen this criteria: ...”

**References**

1. Sydow, Jorg and Milward, H. Brinton. "Reviewing the Evaluation Perspective: On Criteria, Occasions, Procedures, and Practices". (http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1605177 visited in 10/3/2010)

2. Camarinha-Matos, Luís M. and Abreu, A.. "Performance indicators for collaborative networks based on collaboration benefits". Production Planning and Control, Volume 18, No. 7, October 2007, 592-609. (2007)

3. Busi, Marco and Bititci, Umit S.. "Collaborative performance management: present gaps and future research". International Journal of Productivity and Performance Management, 2006, Volume 55, Issue 1, p. 7 – 25. (2006)

4. Zahir, Sajjad and Dobing, Brian. "Designing a web-based multi-criteria group support system (MCGSS)". Journal of Computer Information Systems 42(4), 50-60. (2002)

5. Todeva, Emanuela and Knoke, David. "Strategic alliances and models of collaboration"; Management Decision, Volume 43 Number 1 2005 pp. 123-148. (2005)

6. Carneiro, Luís et al.. "Redes Colaborativas de Elevado Desempenho no norte de Portugal". Public Report. INESC Porto. (2007).