Service Performance Assessment: A PI Toolset
Methodology for VEs

Alessandra Carosi, Mohammadreza Heydari, Christian Zanetti, Marco Taisch, Yves Ducq

► To cite this version:

Alessandra Carosi, Mohammadreza Heydari, Christian Zanetti, Marco Taisch, Yves Ducq. Service Per-
formance Assessment: A PI Toolset Methodology for VEs. IFIP International Conference on Advances
in Production Management Systems (APMS), Sep 2014, Ajaccio, France. pp.691-698, 10.1007/978-
3-662-44736-9_84 . hal-01387951

HAL Id: hal-01387951

https://inria.hal.science/hal-01387951

Submitted on 26 Oct 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Service performance assessment: a PI Toolset
methodology for VEs

Alessandra Carosi¹, Mohammadreza Heydari¹, Christian Zanetti¹, Marco Taisch¹,
Yves Ducq²

¹Politecnico di Milano, Piazza Leonardo Da Vinci 32, 20133, Milan, Italy

² Université Bordeaux, IMS, UMR 5218, F – 334400, Talence, France

{alessandra.carosi, mohammadreza.heydari, cristiano.zanetti,
marco.taisch}@polimi.it; yves.ducq@u-bordeaux.fr

**Abstract.**

Nowadays service sector is becoming more and more relevant in building successful collaborative economies. In this environment Virtual Enterprises (VEs) are forcing a change in the way traditional manufacturing systems are managed. Therefore measuring service performances plays an important role in turning company strategic goals into reality. Performance Indicators (PIs) consist in a supporting tool to assess service efficiency and effectiveness. Consequently, determining the most significant activities which need to be controlled and measured through proper PIs becomes essential. Within this paper, a PI Toolset is going to be presented and tested through industrial use case. The PI Toolset has been developed to support VEs in selecting significant activities, to manage governance processes and to support the design and implementation of specific PIs related to the precise use case objectives. Finally, a lesson learnt approach has been adopted so to stress strengths and weaknesses of both proposed methodology and tools.

**Keywords:** Virtual enterprises, Performance measurement systems and Performance Indicators, Service governance support toolset, PI Toolset.

# 1 Introduction

Participating in collaborative networks has become a key factor for service and manu-
facturing enterprises that try to gain differentiated advantages in today's market com-
petitiveness. Indeed enterprises cannot work in an autonomous way anymore as re-
sponsiveness, interoperability and collaboration become keywords of a successful
business management in service industries [1-3]. During recent years, a large number
of studies about collaborative networks have been developed to support enterprises
survive in market increasingly turbulences [1 & 4-5]. Therefore, for both manufactur-
ing and service enterprises collaboration became a key issue in addressing market
demands through sharing competences and resources [4]. Collaboration can be de-
fined as a common environment where companies are actively cooperating to reach
common goals and objectives [2]. A Virtual Enterprise (VE) is an aggregation of enterprises that represent a temporary alliance through sharing tangible and intangible assets such as information, knowledge, risk and profits [1] or [3 & 6]. A VE is created in order to perform a fast reaction and better respond to market demand and opportunities single enterprises would not be able to face individually. When the development of a new service is involved or if the temporary alliance of enterprises shifts from a product oriented approach to a service oriented one, literature refers to it as “Servitization<fnref n="1" />” process [7]. In order to be successful in the turbulent markets, VEs significantly need to deal with Performance Measurement Systems (PMS) and apply governance principles for managing the system. Hence PMS can support VE as a quantifying procedure in order to measure and monitor the effectiveness and efficiency of its past and current actions [8]. Nevertheless it can be stated that, although the advantages of VE are well known at the conceptual level, VE practical implementation is still far from the expectations [9]. At the same time several VE phases such as planning and creation as well as operation phase are still difficult to instantiate because they need to be properly adapted by advanced collaborative networks [1 & 10]. Some of the lacunas include the lack of common reference models and appropriate tools able to support these particular tasks. Therefore, in this paper a new methodology for VE governance and PI assessment has been developed so to support organizations and help decision makers to enhance VE services provided to support manufacturing products. In the next paragraphs, a literature review regarding PMS is proposed in order to present scientific open challenges and key issues evaluated while developing the PI Toolset. Several tools and methods have been developed by different authors and researchers in the last decades, each one with different advantages, disadvantages and measuring approaches such as ECOGRAI, PRISM, Balance Scorecard (BSC) and European Foundation for quality Management (EFQM) [11]. BSC model is based on the principle that a PMS should provide to decision makers at the strategic level relevant information to help managing several perspectives such as innovation and learning, customer relationship, internal processes and financial issues. On the other hand, EFQM model can be easily applied at any level of the enterprise and offers a comprehensive and coherent picture of an enterprise's health. With respect to their popularity several authors expressed some criticisms on using those models in collaborative strategies environment [1, 4 & 8]. Indeed there is an ample proof that both models are focused on single enterprise strategies and not in strategies based on collaboration [5]. ECOGRAI method has been developed to design and to implement performance indicator system to evaluate performances inside and among enterprises based on the strong principle that performance must support decisions. It can be used to evaluate the enterprise performance related to objectives in a global and detailed approach in a coherent way. Indeed ECOGRAI method clearly studies the decomposi-

<footnote n="1">¹ The transition from product to product+service or service is named Servitization process: “There is clear evidence that manufacturing firms are servitizing–either adding Services to or integrating services in their core products” (Davies et al 2006; Neely, 2009; Vandermerwe and Rada, 1988). Servitization levels vary from “tangible product” (lowest) to “product as a service” (highest level).</footnote>
tions of decision levels and focuses on the coherence of objectives. Furthermore, it is
able to analyse enterprise systems in a very detailed view linking governance issues
with the control of performances [1, 12]. Graph with Results and Activities Interrelat-
ed (GRAI) model consists in a governance modelling tool which can help the decision
makers harmonizing their enterprise systems and defining clearly the decisions, objec-
tives, and actions variables that need to be managed. GRAI model facilitates also the
integration among decisional levels (i.e. Strategic, Tactical & Operational) and func-
tions, but does not supply a reference PI list. Nevertheless, in order to define and to
implement a PMS based on decision model, a model driven approach would be re-
quired. The chosen one is MDSEA and is developed in the frame of MSEE project.
Model Driven Service Engineering Architecture (MDSEA) consists in a standard
framework for activity modelling that takes into consideration three different abstrac-
tion levels, in order to separate business and technical preoccupations and specifica-
tions but to ensure their alignment: Business Service Modelling (BSM), Technical
Independent Modelling (TIM) and Technical Specific Modelling (TSM). In particular
BSM level presents three levels of decision decomposition: strategic, tactical and
operational according to activity temporal period and horizon. In conclusion, similarly
to PMS, a proper governance modelling and assessing methodology will help the
decision makers harmonizing VE creation and measuring its progress towards the
desired goals. Therefore, the scientific result of this research activity will lead to the
creation of the PI Toolset methodology which focuses on managing service systems
governance issues and its performance assessment. In particular, the second chapter of
this paper presents PI Toolset features and components dividing its main composing
elements into progressive paragraphs. Finally chapter three describes the toolset adop-
tion results considering also further steps in scientific research.

## 2 PI Toolset

PI Toolset has been developed to support VEs managing and controlling their system activities. It is composed by a Service Governance Methodology, a PI method and a PI List that, when adopted together, create a coherent linkage between governance issues and PI assessment. A Performance Indicator (PI) is a quantified data which measures the efficiency of actions in the achievement of specific results. A methodology to drive PIs creation and selection has been developed so to turn it into a strategic tool. This methodology allows the development of a coherent list of service driven PIs able to monitor and regulate value exchange in enterprise networks. The PI Method has been created in order to generate specific PIs according to use case objectives which describe use case Governance processes mapped through Service Governance Framework methodology. Indeed, both PI Method and Service Governance Framework have been developed in order to create an integrated engineering approach on business management & assessment tools.
## 2.1 Service Governance Framework

Service Governance Framework methodology has been developed to support service modelling within a VE environment focusing on business goals definition. This conceptual framework has been developed on one hand in order to lay down the foundations for a governance model which can be then linked with coherent monitoring and controlling activities. On the other hand it is able to help the selection of highly exploitable PIs related to End User governance objectives. Therefore every Servitization process could be modelled through the proposed framework first of all defining clear functions and secondly defining the objectives at different decisional levels (i.e. Strategic, Tactical and Operational levels). Finally, the actions to achieve abovementioned objectives need to be specified. Service Governance Framework relies on a structure created merging, on one side, the GRAI method, and, on the other side, MDSEA model. In detail GRAI method has been selected for the linkage it creates among governance processes and the definition of precise objectives. While MDSEA model has been adopted as a standard reference in order to classify PIs into different level of decomposition; it means decomposition by level of abstraction (BSM, TIM and TSM) and decomposition by level of decision (Strategic, Tactical and Operational). At BSM we define the PIs, at TIM the formula to calculate the PI and at TSM the data and their origin required to execute the formula as well as the IT request and data base. In particular the framework focuses on BSM (Business Service Modelling) level used to elaborate high abstraction level model from users' point of view.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Service Governance Framework (conceptual schema)</figcaption></figure>

## 2.2 PI Method

PI method has been created to design, implement and classify, at BSM level, specific PIs related to specific use case objectives. After the Servitization process is modelled through Service Governance, PI Method can be adopted to generate proper PIs. Similarly to Service Governance Framework, PI Method has been created merging GRAI and MDSEA models. In particular, GRAI approach has been adopted because of its good integration between the consistency with decisional process and the focus on results. Hence, this method focuses on why we need PIs (i.e. to make which decisions), instead of sorting out the best indicators directly [5]. In addition, also Value Reference Model (VRM) has been used as a supporting tool to define and prioritize which PIs are needed to manage business processes. Indeed VRM provides a
wide description of standard processes, their inputs and outputs and also metrics and best practices. Therefore use cases could be supported by a wide sample of process categorization which can be used to select the business processes affecting use case strategy. MDSEA model has been used as a reference too, so to define at which level business processes affect Service Lifecycle Management (SLM). As described in Fig.2 PI Method provides a methodology to design and implement relevant PIs generated on the basis of the requirements identified within the Governance framework at the BSM level. Once the Service objectives have been defined, the identification of functions and affected business processes has been facilitated thanks to the support of VRM process classification. Finally PI can be defined: a PI List can be surfed in order to select proper PIs which are strictly linked with the already identified use case processes and objectives. A personalized PI List can be created so PIs can be exploited to monitor service activities.

<figure><img src="image_3.png" /><figcaption>Fig. 2. PI Method (PI Method and PI List)</figcaption></figure>

## 2.3 PI List

Coherently with Service Governance Framework and PI Method, the PI List structure has been created so to facilitate PIs selection and the linkage to defined objectives and functions. The list represents a supporting tool when assessing service performances and managing enterprise resources efficiency. In a similar way, also PI List has been created following MDSEA structure so to divide PIs into different levels of decomposition. On the other hand also VRM has been used as a reference so to adopt its categorization. Table 1 shows PI list structure: PIs are listed respecting VRM process categorization and are described reflecting also VRM processes features.

<table><caption>Table 1. PI List structure</caption><thead><tr><td>MDSEA</td><td>VRM process Classification</td><td>PI field</td><td>Dimension</td><td>PI metrics</td><td>PI formula</td></tr></thead></table>

## 3 Results

A first toolset adoption has been addressed on a real industrial pilot so to optimize the scientific results and maximize the efficiency of the proposed PI Toolset. This action has been conducted in order to test the method validity on the manufacturing world
and to simplify the structuring approach on industrial partners. All the Scientific results have been obtained through testing the activities together with Case A industrial pilot. The company is one of the largest European manufacturing industries in white goods sector aiming at providing its consumers with new advanced services. Therefore the toolset has been enhanced and refined through the implementation of a real Servitization process conducted by an industrial use case. The results still need to be validated but this contribution can be considered as a first step towards the development of PI Toolset scientific methodology. The following List of Actions synthesizes the passages use case has been asked to face when adopting the method:

1. Identify use case Virtual Enterprise;

2. Design service processes and map the phases with service modelling tool;

3. Identify Servitization process objectives (Governance Framework methodology);

4. Adopt PI Method to select PIs (PI Method and PI list);

5. Identify PIs related to Servitization objectives by surfing PI List;

6. Personalize and validate use case PI list.

<table><caption>Table 2. Service Governance Framework (Case A)</caption><thead><tr><th></th><th>E1</th><th>F1</th><th>F2</th><th>F3</th><th>F4</th><th>F5</th><th>F6</th><th>II</th></tr><tr><th></th><th>External Information</th><th>Customer decision</th><th>Customer ideation</th><th>Service-Product design</th><th>Service-Product Implementation</th><th>Service-Product planning</th><th>Service-Product delivery</th><th>Internal Information</th></tr></thead><tbody><tr><th>STRATEGIC<br>H= 2 Years<br>P= 6 Month</th><td>Existing<br>Services in competitive companies</td><td>Customer expectation in terms of services</td><td>Business plan for service proposition</td><td>Selection of design methodologies and partners</td><td>Selection of targeted goods and technologies</td><td>Annual service planning</td><td>Partner relationship organization</td><td>Business Strategy and Master planning</td></tr><tr><th>TACTICAL<br>H= 1 Year<br>P= 1 Month</th><td>Existing<br>HW & SW<br>Implementation technologies</td><td>Feedback on customer satisfaction</td><td>Assessment of existing services</td><td>Definition of PSS functions and design specifications</td><td>Action plan to modify production process</td><td>Planning of the specific service actions</td><td>General planning of service delivery</td><td>In-house Available technologies</td></tr><tr><th>OPERATIONAL<br>H= 1 Month<br>P= 1 Week</th><td>Advertising</td><td>Customers orders; Customers claims</td><td>Brainstorming meeting;</td><td>Detailed design planning</td><td>Implementation of modifications</td><td>Service scheduling; Feedback measurement</td><td>Short term delivery planning</td><td>Status of service production and service system</td></tr></tbody></table>

<table><caption>Table 3. Personalized PI List (Case A)</caption><thead><tr><th></th><th>F1</th><th>F2</th><th>F3</th><th>F4</th><th>F5</th><th>F6</th></tr><tr><th>ISEA</th><th>Customer relationship</th><th>Service ideation</th><th>Product-Service System design</th><th>Product-Service system development</th><th>Product-Service system planning</th><th>Product-Service system delivery</th></tr></thead><tbody><tr><th>STRATEGIC<br>H= 2 Years<br>P= 6 Month</th><td>ROI for each product-service (like the minimum ROI range); Net margin expected</td><td>Capability to implement cross-selling (through a CRM system)</td><td>Total cost of product-service system design</td><td>Global implementation costs</td><td>Amount of product-service sales for the next two years</td><td>Cost of delivery channels</td></tr><tr><th>TACTICAL<br>H= 1 year<br>P= 1 Month</th><td>Service Exploitation</td><td>Amount of sales per month</td><td>Time to design the PSS</td><td>Time to market</td><td>Turnover</td><td>Number of new customers/contracts</td></tr><tr><th>OPERATIONAL<br>H= 1 month<br>P= 1 week</th><td>Time to start up the service, Customer satisfaction rate</td><td>Increase of the new ideas</td><td>Delay/advance in design</td><td>Checking of timing and costing (deviation in%) of master plan</td><td>% of WMs with Carefree Washing Service produced in time</td><td>Product-service frequency</td></tr></tbody></table>

Table 2 shows the results related to the definition of Case A Service Governance Framework at the BSM level. In the horizontal axis (CC stands for Code-Category and NC for Name-Category) several functions have been identified following the SLM phases. Use case objectives have been defined at each stage: strategic, tactical and operational. While Table 3 presents the results obtained for Case A Personalized
PI List definition. Service Objectives have been linked with VRM Process categorization in order to facilitate involved business process selection. Secondly the PI List has been analysed so to define which are the proper PIs to be linked with Case A objectives, business processes and decision variables. Finally, a personalized PI List could have created in order to be used and exploited for performance assessment.

## 3.1 Lesson learnt

A lesson learned approach has been carried out in order to provide a roadmap for continuous improvement after validating the PI Toolset. PI Toolset implies a strong collaboration between scientific theoretical research and practical application. In order to maximize the methodology efficiency, pilots and researchers worked together both to develop and exploit methodologies and tools in a continuous “back and forward” process. Therefore scientific results optimization comes from both use case practical application and scientific-industrial collaboration. The final optimization of scientific results is presented synthesized in Table 4.

<table><caption>**Table 4. Scientific results optimization**</caption><thead><tr><td>LESSON LEARNT</td><td>STRENGTH</td><td>WEAKNESS</td><td>TREATS &amp; OPPORTUNITIES/ IMPROVED TASK</td></tr></thead><tbody><tr><td>-Need for an easier method;<br/>-Need for clarification on the tool;<br/>-Need for practical examples;<br/>-Need to make the user independent;<br/>-Need for optimization of PI selection.</td><td>-Reiterative methodology;<br/>-Validation of proposal<br/>-Exploitation of results;<br/>-Reduce the time to identify PIs;<br/>Personalization of PIs.</td><td>-It takes time for the user to understand the language of the methodology;<br/>-It takes time to make the user adopt the methodology;<br/>-Abundance of PIs.</td><td>-Easy to use methodology;<br/>-Methodology optimization of PI selection;<br/>-One tool for each component of the method;<br/>-Ready to use ICT tool;<br/>-Practical tools application explanation;<br/>-Independence of the user in the use of the tool;<br/>-Avoiding PIs proliferation;<br/>-Personalization of PIs.</td></tr></tbody></table>

## 3.2 Further steps

A further steps section investigates additional features which can be taken into account to improve the toolset in the future. Indeed, to generate additional positive results on Servitization managing tasks, some other scientific activities can be proposed to be integrated within the toolset for further development. PI toolset does not include them at the moment although they can be presented as scientific requirements to be considered for future integration. The list of additional features includes: PI calculation activity, visualization of performances, internal audit, trend analysis and feedback on performances. The implementation of above mentioned elements could allow VE partners managing better the whole Servitization process through the adoption of new strategic actions. PI Toolset could become a learning tool for enterprises offering an integrated vision on performance assessment and governance management able to increase organizations service maturity level. Finally, as additional further step, a new validation phase could be proposed through another industrial case adoption.
## 4 Conclusion

This paper defines a PI Toolset for VEs able to support organizations in managing their governance issues and identifying a proper performance assessment by selecting the activities to be measured through coherent PIs. PI Toolset can be adopted as a decisional supporting tool as it assists the selection of activities associated with use case objectives which need to be managed; secondly it helps linking those objectives and actions with PIs able to measure those tasks. Consequently it helps maximizing the evaluation of performances and optimizing the management of service governance. In conclusion PI Toolset is able to help VEs in identifying and understanding service system requirements, assess current and future service capabilities, identify significant PIs so to help taking correct decisions and finally avoid PIs proliferation.

### Acknowledgements.

This research has been partly funded by the European Union through the FoF-ICT Project named MSEE: Manufacturing Service Ecosystem (No. 284860).

## 5 References

1. Heydari, M., Taisch, C., Carosi, A., Service performance monitoring and control Toolset. The 6th CIRP Conference on Industrial Product-Service Systems, Windsor, Canada (2014).

2. Camarinha-Matos et alii: Collaborative networked organizations, Concepts and practice in manufacturing enterprises, Computers&Industrial Engineering 57, 2008 Elsevier (2009).

3. Afsarmanesh, H., Camarinha-Matos, L., A framework for management of VO breeding environment in Collaborative networks and their breeding environments, Springer (2005).

4. European Collaborative Networked Organizations Leadership IST IP506958 project (2007).

5. Ducq, Y., Vallespir B., Definition and aggregation of a performance measurement system in three Aeronautical workshops using the ECOGRAI Method, Production Planning and control, vol 16 n°2, pp. 163-177, (2005).

6. Martinez. M.T., Fouletier. P., Park. K.H., Favrel. J.: Virtual enterprise organisation, evolution and control, Int. J. Production Economics 74, pp. 225-238, (2001).

7. Thoben, K. D. Extended Products: Evolving Traditional Product concepts. *Engineering the knowledge Economy through Co-operation*, pp. 429-439, Bremen, (2001).

8. Neely, A., Adams, C., Kennerley, M.: The Performance Prism: The Scorecard for Measuring and Managing Business Success. Pearson Education, Harlow, (2002).

9. Dignum, V., Dignum, F.: Towards an agent-based infrastructure to support VOs, Collaborative Business Ecosystems and VEs, Kluwer Academic Publishers, Boston, (2002).

10. Camarinha-Matos, L., Afsarmanesh, H.: Elements of a base VE infrastructure. Computers in Industry 5, pp 139-163, (2003).

11. Sveiby, K.: Methods for Measuring Intangible Assets, Sveiby Knowledge Associates (2007).

12. Ducq, Y., Chen D., Doumeingts, G., A contribution of system theory to sustainable enterprise interoperability science base, Computers in Industry 63, pp.844-857, (2012).