# Using Key Alignment Indicators for Performance Evaluation in Collaborative Networks

Roberto Da Piedade Francisco, Américo Azevedo, João Bastos, António Almeida

► To cite this version:

Roberto Da Piedade Francisco, Américo Azevedo, João Bastos, António Almeida. Using Key Alignment Indicators for Performance Evaluation in Collaborative Networks. 12th Working Conference on Virtual Enterprises (PROVE), Oct 2011, São Paulo, Brazil. pp.159-166, 10.1007/978-3-642-23330-2_18. hal-01569982

HAL Id: hal-01569982

https://inria.hal.science/hal-01569982

Submitted on 28 Jul 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Using Key Alignment Indicators for Performance Evaluation in Collaborative Networks

Roberto da Piedade Francisco¹, Américo Azevedo¹, João Bastos¹, António Almeida¹,

¹ INESC Porto - Institute for Systems and Computer Engineering of Porto, Faculty of Engineering of the University of Porto, Porto, Portugal, {roberto.piedade, ala, joao.bastos, antonio.henrique}@fe.up.pt

**Abstract.** This paper aims to explore the performance alignment in collaborative network environments. It has been stated that performance management based on data collection and the evaluation of key performance indicators (KPIs) may not be effective due to the different indicators and measurement systems in place for the various participants in a collaborative network (CN). Therefore, measuring the strategic and inter-organizational alignment based on key alignment indicators (KAIs) can be an excellent alternative to improve performance evaluation systems. This approach has led to the exploration of the performance prediction paradigm and develops tools to estimate a performance and evaluate the degree of alignment by creating instances of a future performance in collaborative networks.

**Keywords:** Collaborative networks, performance management, alignment measurement.

## 1 Introduction

Constant changes in business environments force organizations to address the challenge of the growing demands of increasingly mature and saturated markets. This then obliges organizations to focus on achieving high levels of responsiveness and flexibility in the development, production and delivery of their products or services [1].

Performance evaluation in collaborative networks (CN) is a significant management function for supporting a successful business [2]. The different systems and performance indicators used by each partner can present difficulties when standardizing measurements. Therefore, an alignment measure is proposed to instantiate the overall performance of a CN.

This paper aims to highlight the relevant aspects of performance management and promote inter-organizational alignment in CNs, which is explained in the second section. The third section introduces theory, focusing on the performance prediction paradigm. This is followed by an outline of the alignment measurement in CNs, the related conceptual framework and the supporting mathematical tools are presented in section four and conclusions are outlined in section five.
## 2 Performance Management and Alignment in Collaborative Networks

Performance measuring and management are crucial in order to improve processes and implement solutions that will improve the efficiency and effectiveness of organizational processes [3]. Furthermore, Seifert [4] has argued that “performance is the degree of target achievement of a process regarding pre-determined and application-dependent criteria”. Complementing this definition, Seifert [4] also explains that performance measurement can be “understood as the measuring, analyzing and communication of the performance of business processes” and yet, if implemented efficiently it can have a significant and positive impact on organizations [5]. However, Taticchi [6] states that a performance measurement and management system is a broader system which is developed to collect, integrate and analyze performance measures to enhance decision-making processes while also evaluating strategies and promoting alignment.

Furthermore, Camarinha-Matos and Abreu [7] state that if the CN is able to simultaneously measure the individual performance of each member and the overall CN performance, then this will encourage participants to understand the benefits of this new paradigm.

The term alignment, although it has other connotations, is usually defined as an arrangement of groups or forces in relation to one another [8]. Thus, in order to contextualize this concept within the scope of collaborative networks it is possible to generalize that this term can be applied as a fit relationship between the participants of a CN. If an organization's strategy does not coincide with the targets, organizations should be aware of the need for adaptive systems in order to improve their level of effectiveness [9]. Kathuria, Joshi and Porth [10] state that alignment is important for formulating strategies, defining processes, supporting decision-making and, in particular, fitting key processes.

However, significant factors can have a negative or positive impact on alignment within a CN, for instance: trust, reliability, competence (skill level) and experience (know how). For example, according to Msanjila and Afsarmanesh [11], inter-organizational trust is not subjective like interpersonal trust and it is not always possible to know the values or past actions of potential partners in a collaborative relationship, particularly in the formation phase of a CN. Furthermore, the level of technological and commercial maturity of each participant could alter reliability which may lead to the exclusion of participants or the dissolution of the CN. Therefore, competence can be seen as the combination of knowledge, skills, technologies, physical systems, management and values and it gives organizations a competitive advantage in creating distinctive value that is recognized by customers [12].

It is therefore possible to determine that measuring and evaluating the behavior of the alignment factors is relevant in order to assess the overall network arrangement [13]. Furthermore, it is important to measure this alignment in the agreement moment and during the operation of the CN in order to verify whether a large gap in the alignment between partners appears.
# 3 A Predictive Approach in Performance Management

This new model of collaborative management can help organizations follow strategies that enable them to be more flexible and agile. The interval between analysis and evaluation and the time taken to react to process failures and develop solutions must be reduced in order to improve organizations' responsiveness.

Therefore, due to constant changes in organizational processes and their requirements, this work recommends a conceptual framework supported by a practical tool for performance estimation in order to support a proactive approach. Indeed, with a proactive performance prediction approach it is possible to foresee that the performance model will be based on the current status. Therefore, contrary to the reactive approach, that is not able predict what the module will become unless a trigger is detected, this approach will proactively react once a system concept/model change has been identified [14]. Consequently, this paper proposes a tool that learns the behavior of the system in order to anticipate the performance reaction to the changes made, and not only using data history analysis.

## 3.1 Feedforward Alignment

Having established that alignment can be measured using key alignment indicators (KAIs) estimation and performance classification tools were then developed and implemented in order to materialize this concept. In this innovative approach, a combination of feedforward and feedback analysis, supported by leading and lagging performance measures, respectively, can establish a predictive approach to measuring alignment in collaborative networks (Figure 1).

<figure><img src="image_2.png" /><figcaption>Fig. 1. Feedforward and Feedback Control System.</figcaption></figure>

Thus, according to Busi and Bititci [15],

> "Feedforward control involves the development and deployment of plans and objectives based on leading measures of real-time performance, while feedback control involves the measurement of performance against those objectives through historical lagging measures. Proactive performance management based on both feedforward and feedback control is based on the premise that a balanced set of leading and lagging performance measures should anticipate and not only correct bad performance."

The proposed approach will be crucial to supporting the subsequent development of the alignment and measuring toolset proposed in this research.
## 3.2 Decreasing Reaction Time

Since the success and effectiveness of performance measurement depends on the time taken by organizations to react and make improvements, it is critical not only to reduce the reaction time but, more importantly, anticipate it [4]. Therefore, modeling complex manufacturing systems, using a predictive tool through data fusion, which is not only concerned with statistical data, but also with factors that may influence the future of the CN, can present real benefits for industry nowadays.

With this in mind, it is important to extract knowledge on statistical analysis and data mining tools that are collecting information and knowledge from data in order to use this to predict future patterns of behavior. A predictive model consists of a series of predictive indicators and variables that are likely to influence future behavior.

According to Seifert [4], reaction time consists of two stages: the feedback time and the implementation time (Figure 2). The first stage is represented by the "time span between the evaluated period and the calculation of the KPIs" during the performance measurement process. The second stage is represented by the "time span between detection and elimination of the performance deficit" closing the cycle of performance management.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Performance prediction benefits (adapted from Seifert [4]).</figcaption></figure>

Another important aspect related to the performance prediction concept is the definition of appropriate and relevant performance indicators. This is critical for the success of the organization, both individually and in a network environment. Performance measurement and performance management systems should be designed to support proactive management based on both feedback and feedforward operation control.

## 4 Proposed Alignment Measurement Framework

As part of the research conducted, an explanatory and exploratory investigation was developed within a set of SMEs in collaborative networks in Brazil. During this research, a conceptual framework called CNPMS [16] was adopted for performance management in collaborative networks and an estimation tool was applied to provide predictive measures. Thus, developments were performed to verify if the fit between the CN's strategy and inter-organizational processes could be measured using an alignment measure.
## 4.1 Key Alignment Indicators

Important questions regarding the ‘fit’ concept were raised by Venkatraman [17] within the scope of strategic management. Indeed, this is a process geared to deducing not only whether the business environment and organizational structure are aligned (external fit) but also whether the structure and processes of the organization are aligned (internal fit). The ‘fit’ concept, as outlined in the literature, represents the alignment or configuration of the organizational strategy and takes contingencies faced by the organization within its business environment into account [17]. The internal fit is usually related to the performance improvements [18], to ensure a higher level of alignment, and once calculated it can represent the state of internal fit of a CN.

Since alignment can be measured using KAIs, this led to the exploration of the performance prediction paradigm and the development of tools to estimate and evaluate the performance of the future degree of alignment or fit. The KAIs were then chosen from the predictive KPIs that best represent the effectiveness of inter-organizational processes (Figure 3). Subsequently, the KAIs were used to evaluate the degree of compliance with the established goals. Thus the KAI values are compared with their target values and then each participant is classified. Using these individual values it calculates the Fit Degree where the overall alignment is classified by a fuzzy logic in order to outline future alignment for the period stipulated.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Using KAIs for performance evaluation in collaborative networks.</figcaption></figure>

The criteria for selecting the KAIs are derived from the CN decision makers’ expectations based on the strategy and the inter-organizational scope. Therefore, initially it is important to perform a survey on each of the participants with the main KPIs used as well as linking each of them following an independent axis: cost, time and quality. Depending on the complexity of the expected KAIs, the CN manager must not only select the independent axis that must be included in the KAI calculation, but also the corresponding KPIs, as each KAI can only be composed of one specific KPI or a combination of KPI’s. Subsequently, it is possible to define measurable alignment indicators that are capable of instantiating the overall CN performance state, according to the strategy defined for the entire network.
## 4.2 Fit Degree Supporting Tools

Since this research aims to develop tools that improve the reaction time in CNs, it is crucial to develop a tool to predict and estimate the performance of nonlinear systems. This approach is based on the leading and lagging factors that can influence the system's behavior. In order to fulfill these requirements, the so-called Performance Value Estimator (PVE) tool was used as a predictive control to monitor, learn and imitate the behavior of complex nonlinear systems that do not require profound knowledge of mathematics [19]. This tool, however, does require a thorough knowledge of the system that it will monitor and emulate. Indeed, this tool consists of Neural Networks and a Kalman Filter, two well-known mathematical tools that are currently used in the areas of automation and robotics.

The fact that Neural Networks (NN) is a non-linear tool with data-driven self-adaptive capabilities makes it a powerful tool for supporting the system's modeling tasks since it allows for the approximation of any continuous function with the desired accuracy. In reality, because the NN tool is able to model non-linear systems without prior knowledge of the relationship between the input and output variables, it can support companies with limited resources and emulate the system's behavior using non-complex past examples. Hence, it is easier to predict future performance using this modeling approach and considering the factors that can be anticipated and envisaged.

Indeed, with the NN modeling approach it is possible to emulate and anticipate the expected performance of a complex system. Nevertheless, one must be cautious with the use of the NN tool since there are a lot of causes and factors that can affect its performance. Therefore, the Kalman Filter was used in order to eliminate possibility of non-controllable errors (where possible) that can originate from the modeling system or from the leading and lagging factors measure. Due to the fact that this filter is capable of supporting estimations for past, present and future states even when the system modeling accuracy is unknown, this tool is normally applied to optimize the estimation of state models. In a higher mathematical layer of abstraction, the Kalman Filter can be seen as a tool that is used to estimate the instantaneous “state” of a dynamic system perturbed by white noise (random factors). Consequently, by incorporating these two significant approaches it was possible to develop a tool that is able to predict the evolution of the behavior of complex systems, minimizing the different errors and noises that can disturb the normal assessment of the performance of the system as presented by Azevedo and Almeida [19].

The PVE tool assumes an important role in this area in predicting the KPI values for the short and medium-term. Nevertheless, since alignment evaluation can be seen as a subjective exercise, in order to calculate and evaluate the inter-organizational alignment it was necessary to use a fuzzy approach. This approach informs the manager of the system's overall performance and takes the key alignment indicators into account and how these indicators will affect the overall performance of the collaborative network. As a result, a Fuzzy Logic System capable of evaluating the inter-organizational performance was included within the framework. The Fuzzy Logic was mainly developed to help decision makers solve classification problems when there is little knowledge and certainty about the system that will be controlled.
Therefore, in a Brazilian industrial supply chain collaborative network called G3, it was possible to define the 3D fuzzy decision surface based on the following KAIs: the delay in the delivery time (DDT), and the percentage of orders delivered with non-conformities (NON). It is then possible to visualize the 3D graph that represents the non-linearity desired for this system (Figure 4). For example, as the KAI values increase, the reliability of the partner decreases, as expected. The dark blue color in this figure represents “fits very well” and the yellow color represents “fits very badly”.

<figure><img src="image_5.png" /><figcaption>Fig. 4. 3D fuzzy decision surface.</figcaption></figure>

Therefore, based on the methodologies explored in the previous section it is possible to integrate the PVE tool and the Fuzzy expert system, in order to build a framework capable of grasping and learning the normal behavior of the non-linear system in study. This framework predicts the future CN performance, using the KAIs as global performance indicators and, finally, evaluates the alignment measure according to the priorities and requirements of the collaborative network in analysis.

# 5 Discussion and Conclusions

This paper presents a new approach to performance management in collaborative networks (CNs) by measuring the inter-organizational alignment. A collaborative performance forecasting framework was developed and implemented to verify this approach. This alignment evaluation toolset presents both merits and potential advantages for CNs. A major benefit comes from its simplicity and robustness in providing managers with relevant information with regard to network alignment assessment in present and future states to support decisions. Another benefit comes from the fact that prediction results are provided more quickly and in a more proactive way than in other approaches, providing the manager with relevant information to make decisions and manage situations effectively.

In the practical application of the G3 collaborative network, the expert manager reported that the gains obtained by using the tool to predict the future performance of the CN included: being able to specify the sales and production goals with higher accuracy and being able to implement improvement solutions using priority criteria. Therefore, it encourages the CNs to implement solutions to promote alignment in the inter-organizational processes.
**Acknowledgments.** We would like to thank INESC Porto for supporting this research.

References

1. Chituc, C.M., Azevedo, A.L.: Multi-Perspective Challenges on Collaborative Networks Business Environment. In: Camarinha-Matos, L.M., Afsarmanesh, H., Ortiz, A. (eds.). Collaborative Networks and their Breeding Environments. LNCS, pp. 25--32, Springer, New York (2005)

2. Gibbons, P.M., Burgess, S.C.: Introducing OEE as a Measure of Lean Six Sigma Capability. International Journal of Lean Six Sigma, 1, 134--156 (2010)

3. Sobotka, M., Platts, K.W.: Managing without Measuring: a Study of an Electricity Distribution Company. Measuring Business Excellence, 14, 28--42 (2010)

4. Seifert, M.: Collaboration Formation in Virtual Organisations by Applying Prospective Performance Measurement. Verlagsgruppe Mainz, Aachen (2009)

5. Spitzer, D.R.: Transforming Performance Measurement: Rethinking the Way We Measure and Drive Organizational Success. Amacom, New York (2007)

6. Taticchi, P.: Business Performance Measurement and Management: New Contexts, Themes and Challenges. Springer, Berlin (2010)

7. Camarinha-Matos, L.M., Abreu, A.: Performance Indicators for Collaborative Networks Based on Collaboration Benefits. Production Planning and Control, 18, 592--609 (2007)

8. Merriam-Webster Dictionary, http://www.merriam-webster.com/dictionary/alignment

9. Shimizu, T., Carvalho, M.M., Laurindo, F.J.B.: Strategic Alignment Process and Decision Support Systems: Theory and Case Studies. IRM, Hershey (2006)

10. Kathuria, R., Joshi, M.P., Porth, S.J.: Organizational Alignment and Performance: Past, Present and Future. Management Decision, 45, 504--517 (2007)

11. Msanjila, S.S., Afsarmanesh, H.: On Hard and Soft Models to Analyse Trust Life-cycle for Mediating Collaboration. In: Camarinha-Matos, L.M., Paraskakis, I., Afsarmanesh, H. (eds.). Leveraging Knowledge for Innovation in Collaborative Networks. LNCS, pp. 381--392, Springer, Heidelberg (2009)

12. Hamel, G.; Prahalad, C.K. Competing for the Future. Harvard Business School Press, Boston (1994)

13. Faria, L., Azevedo, A.: Strategic Production Networks: the Approach of Small Textile Industry. In: 7th IFIP Working Conference on Virtual Enterprises, pp. 609--616. Springer, New York (2006)

14. Yang, Y., Wu, X.: Mining in Anticipation for Concept Change: Proactive-Reactive Prediction in Data Streams. In: Data Mining and Knowledge Discovery, 13, 261-289 (2006)

15. Busi, M., Bititci, U.S.: Collaborative Performance Management: Present Gaps and Future Research. Int. J. of Productivity and Performance Management, 55, 7--25 (2006)

16. Francisco, R.D., Azevedo, A.: An SSM-Based Approach to Implement a Dynamic Performance Management System. In: Camarinha-Matos, L.M. Paraskakis, I., Afsarmanesh, H. (eds.). Leveraging Knowledge for Innovation in Collaborative Networks. LNCS, pp. 476--483, Springer, Heidelberg (2009)

17. Venkatraman, N., Camillus, J.C.: Exploring the Concept of ‘Fit’ in Strategic Management. Academy of Management Review, 9, pp. 1--23 (1984)

18. Silveira, G.J.C., Sousa, R.S.: Paradigms of Choice in Manufacturing Strategy: exploring performance relationships of fit, best practices, and capability-based approaches. International Journal of Operations & Production Management, 30. pp. 1219--1245 (2010)

19. Azevedo, A., Almeida, A.: Factory Templates for Digital Factories Framework. Robotics and Computer-Integrated Manufacturing, 27, 755--771 (2011)