# A Proposal of Performance Indicators for Collaborative Business Ecosystems

Paula Graça, Luis M. Camarinha-Matos

► To cite this version:

Paula Graça, Luis M. Camarinha-Matos. A Proposal of Performance Indicators for Collaborative Business Ecosystems. 17th Working Conference on Virtual Enterprises (PRO-VE), Oct 2016, Porto, Portugal. pp.253-264, 10.1007/978-3-319-45390-3_22. hal-01614592

HAL Id: hal-01614592

https://inria.hal.science/hal-01614592

Submitted on 11 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Proposal of Performance Indicators for Collaborative Business Ecosystems

Paula Graça<sup>a,b</sup>, Luís M. Camarinha-Matos<sup>a</sup>

**Abstract.** Business ecosystems enabled by the increasing use and improvement of communication networks, offer nowadays a powerful competitive advantage to business players and entrepreneurs. They form a collaborative new mean of economic and social value creation, addressing customers' needs, overcoming constraints of individual firms, increasing capabilities for new business opportunities, and accelerating learning and innovation. This paper proposes a set of performance indicators to measure some of these collaboration benefits, therefore motivating the sustainability and resilience of the business ecosystem. The presented results are based on simulation models, which intend to characterize the roles and interactions of a real life collaborative business ecosystem.

**Keywords:** Collaborative Network, Business Ecosystem, Digital Business Ecosystem, Performance Indicators.

## 1 Introduction

The increasing progress and use of information and communication technologies, namely computer networks, and collaboration platforms, have changed the ways of making business, which have moved from traditional industrial sectors to business ecosystems supported by collaborative platforms. The term Business Ecosystem was introduced by Moore [1], which used natural ecosystems as a metaphor to describe business environments. According to this author, a business ecosystem is "an economic community supported by a foundation of interacting organizations and individuals - the organisms of the business world. This economic community produces goods and services of value to customers, who themselves are members of the ecosystem". The players of the ecosystem "coevolve their capabilities and roles" [2] in a symbiotic business environment.

A recent business trends report [3] denotes the continued rise of business ecosystems as a highly relevant transition with substantial implications for the society, economy, and businesses. In fact, as argued by the author, a business

<footnote><sup>a</sup> Faculdade de Ciências e Tecnologia and Uninova CTS, Universidade Nova de Lisboa, Campus de Caparica, 2829-516 Caparica, Portugal</footnote>

<footnote><sup>b</sup> Instituto Superior de Engenharia de Lisboa, Instituto Politécnico de Lisboa, Rua Conselheiro Emídio Navarro 1, 1959-007 Lisboa, Portugal
mgraca@deetc.isel.pt, cam@uninova.pt</footnote>
ecosystem captures three main generic characteristics of any ecosystem: 1) Variety for a healthy ecosystem, due to the encouragement for participation of a diversity of organizations, that together can overcome the capabilities of any one of them; 2) Potential for ever more productive development of the ecosystem, due to the increasingly advanced ways of collaboration, boosted by high-tech tools of connectivity and interaction; and 3) Longevity and durability of the ecosystem, due to a collective awareness of its members to protect, foster, and sustain the ecosystem, motivated by common goals, benefits, and values. These characteristics of an ecosystem highlight and reinforce the purpose of this paper, which is driven by the following research question:

**"What is a reasonable set of key performance indicators to measure and assess collaboration benefits in a collaborative business ecosystem (CBE)?"** [4]

In order to address the above presented research question, the following hypothesis is considered: "*Collaboration benefits can be evaluated and made explicit if a set of indicators is established through a holistic combination of concepts of value and benefit, derived from a number of research areas such as value systems, social networks analysis, supply chain performance, and complex adaptive systems*". This hypothesis guides this research, which proposes a set of performance indicators and related metrics to assess benefits in a CBE, promoting its sustainability and resilience.

The remaining sections of this paper are organized as follows: section two presents a brief literature review, highlighting contributions from a number of related research fields, section three proposes a set of performance indicators to assess CBEs, and section four shows results of some metrics based on simulation data, to assess the coherence of the models. The last section discusses the contributions and future work.

## 2 Literature Review

The research area of collaborative networks [5], in particular their established reference models and taxonomies [6], provide a significant contribution to understand the structure and dynamics of a business ecosystem. A business ecosystem, as described in [6], is a sub-class of a Virtual Organizations Breeding Environment (VBE), i.e. a source network of organizations founded as a long-term strategic alliance. As such, a CBE is modeled as a community of organizations, whose connections are created during a given period (for instance one year), resulting in the formation of Virtual Organizations (VOs), according to collaborative business opportunities.

In order to propose performance indicators for a CBE, a literature research shows that there are already well-established indicators for individual organizations, of which the balanced scorecard (BSC) [7] is the most common example. However, to measure collaboration benefits in a network of organizations, or more specifically in a CBE, there is still a lack of solutions, despite some efforts made in a number of research areas. One example is a conceptual model for VBEs value systems using a BSC to monitor the VBE performance [8]. Another example is the value systems proposed in [9], [10], which identify collaborative social and economic core values, suggesting mechanisms to assess the alignment of the value systems of the network members. These mechanisms allow identifying conflicts that may affect the
performance of a CBE, although not suitable for performance measurement purposes. More concerned with collaboration benefits, an example presented in [11], identifies and characterizes benefits, such as innovation capacity, flexibility, agility, costs, and risks, among others, which constitute an important input for the establishment of the proposed metrics and associated performance indicators.

The research area of traditional supply chains, which have evolved to more collaborative platforms, known as supply chain collaboration (SCC), provides a wide number of performance models and metrics. For instance in [12], the authors made a comprehensive review of supply chain performance measurement methods comprising the period from 2000 to 2011, and found several techniques and models that were used, namely, AHP (Analytic Hierarchy Process), SCOR (Supply-chain Operations Reference Model), SEM (Structural Equation Model), BSC (Balanced Scorecard), Six Sigma, and combinations of two of them. In [13], more concerned with collaboration (SCC), the author conducted a survey in which he identified a number of mathematical and simulation models focused mainly on the performance evaluation. Some of these techniques and models can be adapted to CBEs; in particular, simulation models to characterize the business environment, and a subset of statistic models from SEM, widely used in the field of behavioral and social sciences.

Another related area is the social network analysis (SNA), which is currently a subject of much research work due to the growth of social networks. In SNA, the social structures are analyzed using network and graph theory, which provides measures of centrality (degree, betweenness, closeness, etc.), and cohesion (reciprocity, density, clustering, etc.). Inspired by SNA, several attempts have been made to apply these indicators to collaborative networks. For instance, a set of indicators to measure the value of social capital of a VBE [14], and a methodology for a network analysis of a business ecosystem [15], among others. However, the structural analysis of the network is limited when it comes to capture the value of the economic or social exchanges represented by the connections. To overcome these limitations, a value network analysis (VNA) model is proposed in [16], providing metrics to evaluate the complex dynamic exchanges of tangible and intangible values in the network. Even though these contributions, a solution to measure the performance of a CBE is not yet established.

The proposed performance indicators for CBEs described in the next section, are inspired mostly by these last presented approaches.

## 3 Metrics and Measures for Collaborative Business Ecosystems

The adopted research approach at this stage uses models with simulated data to assess the consistency and make a preliminary calibration of the measurement scale of the metrics and performance indicators proposed in this section. The approach is represented by the process depicted in Fig. 1. The analysis of the literature in a number of related research areas, such as enterprise performance indicators, value systems, collaboration benefits, social network analysis, and supply chain collaboration, inspired a set of metrics and measures with a potential applicability in
CBEs. Then, several simulation models of CBEs are used to assess the metrics and calibrate the scales using ratios and standard normalization methods.

A CBE model is represented by a weighted direct graph, where nodes stand for organizations, and arcs correspond to the collaborative transactions between them. The weights of the arcs stand for the number of times the collaborative transactions were performed in a given period (by the creation of VOs in the CBE). In the used models, random series of data were also considered, to simulate existing (e.g. portfolio of competencies) or new value created by the organizations (e.g. products, services, or patents), resulting or not from the collaboration, therefore not directly characterized by the graph. Finally, the last step resulted in a first proposal of a set of performance indicators to assess CBEs.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Process representing the used methodology to develop a set of PIs for CBEs.</figcaption></figure>

The performance indicators for CBEs introduced in this work and described in the following tables are: 1) Innovation Indicator (II), to measure the innovation potential of a CBE; 2) Contribution Indicator (CI), to evaluate the value creation of a CBE; 3) Prestige Indicator (PI), to assess the influence/prominence of each organization in the CBE; and 4) Resilience Indicator (RI), to assess the resilience of a CBE.

The choice of the above indicators is based on the performed literature analysis and mainly driven by societal, economic, and business concerns. The potential for innovation of a CBE reflects its health level, therefore contributing for social and economic value creation. On the other hand, in a business environment, the economic value is a strong argument for collaboration. As such, the contribution indicator is proposed to show the economic value of the collaboration, i.e. the value that an organization brought in the CBE, and the value that is gained by an organization from the CBE. The prestige indicator shows the distribution of collaboration among organizations in the CBE, revealing eventual prominence of one or more organizations, or even isolated organizations. A strong collaboration level enhances a common network identity, reinforcing collaboration trust, and increasing collaborative knowledge. At last, the resilience indicator in a business context assesses "*the capacity for an enterprise to survive, adapt, and grow in face of turbulent change*" [17]. Table 1 presents a brief description of these indicators, explaining their meaning for individual organizations $O_i$ in the CBE, and for the CBE as a whole.
<table><caption>Table 1. A short description of the proposed performance indicators.</caption><thead><tr><th colspan="3">Performance Indicators for CBEs</th></tr><tr><th>Indicator</th><th>Definition (for each O<sub>i</sub>)</th><th>Definition (for the CBE)</th></tr></thead><tbody><tr><td>Innovation Indicator (II)</td><td>II<sub>i</sub> - Measures the innovation potential of a member of the CBE</td><td>II<sub>CBE</sub> - Measures the innovation potential of the CBE as a whole</td></tr><tr><td>Contribution Indicator (CI)</td><td>CI<sub>i</sub> - Measures contribution for value creation of a member of the CBE</td><td>CI<sub>CBE</sub> = (CI<sub>CBEt</sub>, CI<sub>CBEd</sub>)<br/>Measures the total value creation of the collaboration in the CBE as a whole (CI<sub>CBEt</sub>), and the degree to which the most active member exceeds the contribution of the others (CI<sub>CBEt</sub>)</td></tr><tr><td>Prestige Indicator (PI)</td><td>PI<sub>i</sub> - Measures the influence/proeminence of a member of the CBE</td><td>PI<sub>CBE</sub> - Measures the average difference between the most influent member and that of all members of the CBE</td></tr><tr><td>Resilience Indicator (RI)</td><td>—</td><td>RI<sub>CBE</sub> = f (Di, Ef, Ad, Co)<br/>Assesses the resilience of the CBE, evaluating four main components: Diversity (Di), Efficiency (Ef), Adaptability (Ad), and Cohesion (Co)</td></tr></tbody></table>

Considering these performance indicators, a set of appropriate metrics had to be found in order to identify key measurable characteristics of each indicator. The suggested metrics are shown in Table 2, including mainly counting measurements (e.g. number of new products, services, or patents created by each organization or by the CBE as a whole), as well as measurements related to the network structure, such as centrality, which is a measure of the activity or level of influence in terms of relationships among the organizations of the CBE.

The metrics shown in Table 2 to assess the resilience of a CBE were inspired by the four fundamental characteristics of resilient living ecosystems considered in [18]: Diversity, Efficiency, Adaptability, and Cohesion, and were tailored to business ecosystems. In fact, the resilience of the CBE can be calculated as a function of these factors, if each of them can be translated in terms of one or more quantified metrics.
<table><caption>**Table 2.** A proposal of metrics to establish the proposed performance indicators for CBEs.</caption><thead><tr><th colspan="4">Metrics for Performance Indicators for CBEs</th></tr><tr><th>PIs</th><th>Org. O<sub>i</sub> / CBE</th><th>Metrics</th><th></th></tr></thead><tbody><tr><td rowspan="2">Innovation Indicator (II)</td><td rowspan="2">II<sub>i</sub></td><td>#PortPd<sub>i</sub></td><td>- Portfolio of products, services, or patents of the member O<sub>i</sub></td></tr><tr><td>#NewPd<sub>i</sub></td><td>- No. of new products, services, or patents generated by the member O<sub>i</sub></td></tr><tr><td rowspan="4">Contribution Indicator (CI)</td><td rowspan="2">II<sub>CBE</sub></td><td>{#VO<sub>1</sub>, ..., #VO<sub>n</sub>}</td><td>- No. of VOs in which the members O<sub>1</sub>, ..., O<sub>n</sub> participated</td></tr><tr><td>Σ<sub>i</sub> #PortPd<sub>i</sub></td><td>- Total portfolio of products, services, or patents of the CBE</td></tr><tr><td rowspan="2">CI<sub>CBE</sub> = (CI<sub>CBE</sub><sup>t</sup>, CI<sub>CBE</sub><sup>d</sup>)</td><td>{#NewPd<sub>1</sub>, ..., #NewPd<sub>n</sub>}</td><td>- No. of new products, services, or patents generated by the members O<sub>1</sub>, ..., O<sub>n</sub></td></tr><tr><td>Σ<sub>i</sub> #NewPd<sub>i</sub></td><td>- Total no. of new products generated in the CBE</td></tr><tr><td rowspan="2">Prestige Indicator (PI)</td><td rowspan="2">PI<sub>i</sub></td><td>#CoOp<sub>i</sub> in, #CoOp<sub>i</sub> out</td><td>- No. of collaborative opportunities the member O<sub>i</sub> gained from or brought in the CBE</td></tr><tr><td>Act in(O<sub>i</sub>), Act out(O<sub>i</sub>)</td><td>- Activity in/out of the member O<sub>i</sub> in the CBE</td></tr><tr><td rowspan="3">Resilience Indicator (RI)</td><td rowspan="3">PI<sub>CBE</sub></td><td>Σ<sub>i</sub> #CoOp<sub>i</sub></td><td>- Total no. of collaborative opportunities created in the CBE</td></tr><tr><td>{#CoOp<sub>1</sub>, ..., #CoOp<sub>n</sub>}</td><td>- No. of collaborative opportunities in which the members O<sub>1</sub>, ..., O<sub>n</sub> participated</td></tr><tr><td>Σ<sub>i</sub> [Act(O<sup>*</sup>) - Act(O<sub>i</sub>)]</td><td>- Sum of the differences between the activity of the most active member (O<sup>*</sup>) and that of all members of the CBE Act(O<sup>*</sup>) = #CoOp<sub>i</sub> of the most active member</td></tr><tr><td rowspan="2">PI<sub>CBE</sub></td><td rowspan="2">PI<sub>CBE</sub></td><td>#CCoOp<sub>i</sub></td><td>- No. of received contacts towards potential collaborative opportunities of the member O<sub>i</sub></td></tr><tr><td>Inf(O<sub>i</sub>)</td><td>- Influence of the member O<sub>i</sub>, i.e., the likelihood of O<sub>i</sub> to be contacted towards potential collaborative opportunities</td></tr><tr><td rowspan="2">PI<sub>CBE</sub></td><td rowspan="2">PI<sub>CBE</sub></td><td>{#CCoOp<sub>1</sub>, ..., #CCoOp<sub>n</sub>}</td><td>- No. of receiving contacts towards potential collaborative opportunities of the members O<sub>1</sub>, ..., O<sub>n</sub></td></tr><tr><td>Σ<sub>i</sub> [Inf(O<sup>*</sup>) - Inf(O<sub>i</sub>)]/(n-1)</td><td>- Average of the differences between the influence of the most influent member and that of all members of the CBE Inf(O<sup>*</sup>) = influence of the most influent member</td></tr><tr><td rowspan="7">Resilience Indicator (RI)</td><td rowspan="7">PI<sub>CBE</sub> = f (Di, Ef, Ad, Co)</td><td>Σ<sub>i</sub> #CoOp<sub>i</sub></td><td>- Total no. of collaborative opportunities created in the CBE</td></tr><tr><td>Σ<sub>i</sub> #PortPd<sub>i</sub></td><td>- Total portfolio of products, services, or patents of the CBE</td></tr><tr><td>Σ<sub>i</sub> #PortCp<sub>i</sub></td><td>- Total portfolio of competences of the CBE</td></tr><tr><td>Σ<sub>i</sub> #ShIntAs<sub>i</sub></td><td>- Total of shared intangible assets (documents, processes, knowledge, etc.) in the CBE</td></tr><tr><td>Σ<sub>i</sub> #CCoOp<sub>i</sub></td><td>- Total no. of contacts towards potential collaborative opportunities in the CBE</td></tr><tr><td>Σ<sub>i</sub> #RCCoOp<sub>i</sub></td><td>- Total no. of reciprocated contacts towards potential collaborative opportunities in the CBE</td></tr><tr><td>Σ<sub>i</sub> #PCCoOp<sub>i</sub></td><td>- Total no. of possible contacts between all pairs of members of the CBE</td></tr></tbody></table>

At this stage, the performance indicators can then be formulated as a function of the presented metrics. The suggested formulas described in Table 3, are calculated in
terms of ratios (case of II), or measures of degree centrality (case of CI), and betweenness centrality (case of PI).

<table><caption>Table 3. Proposed calculation formulas for the performance indicators to assess CBEs.</caption><thead><tr><th colspan="3">Performance Indicators for CBEs</th></tr><tr><th>PIs</th><th>Calculation</th><th>Remarks</th></tr></thead><tbody><tr><td rowspan="2">Innovation I. (II)</td><td>II<sub>i</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">#NewPd<sub>i</sub></div><div style="padding: 0 5px;">#PortPd<sub>i</sub></div></div></td><td>Is measured by the ratio between the two variables #NewPd<sub>i</sub> and PortPd<sub>i</sub></td></tr><tr><td>II<sub>CBE</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">Σ<sub>i</sub> #NewPd<sub>i</sub></div><div style="padding: 0 5px;">Σ<sub>i</sub> #PortPd<sub>i</sub></div></div> * r (#VO, #NewPd)</td><td>The ratio of the totals is weighted by the correlation r between the two variables: (#VO, #NewPd)</td></tr><tr><td rowspan="3">Contribution Indicator (CI)</td><td>CI<sub>i</sub> = (CI<sub>i</sub> in, CI<sub>i</sub> out)</td><td>Act in/out (O<sub>i</sub>) can be measured by the weighted indegree/outdegree centrality (C<sub>D</sub>) of the member O<sub>i</sub> in the CBE, which stands for the sum of direct connections in/out of the member O<sub>i</sub> to the n members O<sub>j</sub> with weight #CoOp<sub>ij</sub></td></tr><tr><td>CI<sub>i</sub> in = C<sub>D</sub>(O<sub>i</sub>) in = Σ<sub>j</sub> O<sub>ij</sub> #CoOp<sub>ij</sub> in</td><td>where C<sub>D</sub>(O<sub>i</sub>) in/out = Act in/out (O<sub>i</sub>)</td></tr><tr><td>CI<sub>i</sub> out = C<sub>D</sub>(O<sub>i</sub>) out = Σ<sub>j</sub> O<sub>ij</sub> #CoOp<sub>ij</sub> out</td><td>CI<sub>CBE</sub> can be measured by the weighted degree centrality (C<sub>D</sub>) of the CBE as a whole</td></tr><tr><td rowspan="3">Contribution Indicator (CI)</td><td>CI<sub>CBE</sub> = (CI<sub>CBE</sub> t, CI<sub>CBE</sub> d)</td><td>where C<sub>D</sub>(O<sub>i</sub>) = Act(O<sub>i</sub>) = Σ<sub>j</sub> O<sub>ij</sub> #CoOp<sub>ij</sub></td></tr><tr><td>CI<sub>CBE</sub> t = Σ<sub>i</sub> #CoOp<sub>i</sub></td><td>C<sub>D</sub>(O<sup>*</sup>) = Act(O<sup>*</sup>) = max[C<sub>D</sub>(O<sub>i</sub>)]</td></tr><tr><td>CI<sub>CBE</sub> d = C<sub>D</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">Σ<sub>i</sub> [C<sub>D</sub>(O<sup>*</sup>) - C<sub>D</sub>(O<sub>i</sub>)]</div><div style="padding: 0 5px;">(n - 2)(n - 1)</div></div></td><td>Can be measured by the weighted betweenness centrality (C<sub>B</sub>) of the member O<sub>i</sub> in the CBE, assuming that connections between any member O<sub>k</sub> and any other O<sub>j</sub> have weight of #CCoOp<sub>ij</sub></td></tr><tr><td rowspan="3">Prestige Indicator (PI)</td><td>PI<sub>i</sub> = C<sub>B</sub>(O<sub>i</sub>) = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">Σ<sub>k</sub> Σ<sub>j</sub> O<sub>k</sub> O<sub>j</sub>(O<sub>i</sub>)</div><div style="padding: 0 5px;">k &lt; j</div><div style="padding: 0 5px;">k &ne; j &ne; i</div></div></td><td>Can be measured by the weighted betweenness centrality (C<sub>B</sub>) of the member O<sub>i</sub> in the CBE, assuming that connections between any member O<sub>k</sub> and any other O<sub>j</sub> have weight of #CCoOp<sub>ij</sub></td></tr><tr><td>PI<sub>CBE</sub> = C<sub>B</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">Σ<sub>i</sub> [C<sub>B</sub>(O<sup>*</sup>) - C<sub>B</sub>(O<sub>i</sub>)]</div><div style="padding: 0 5px;">(n - 1)(n - 2)(n - 1)</div></div></td><td>Can be measured by the weighted betweenness centrality (C<sub>B</sub>) of the CBE as a whole, assuming that connections between any member O<sub>k</sub> and any other O<sub>j</sub> have weight of #CCoOp<sub>ij</sub></td></tr><tr><td>PI<sub>CBE</sub> = C<sub>B</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">C<sub>B</sub>(O<sub>i</sub>) = Inf(O<sub>i</sub>)</div><div style="padding: 0 5px;">C<sub>B</sub>(O<sup>*</sup>) = Inf(O<sup>*</sup>) = max[Inf(O<sub>i</sub>)]</div></div></td><td>Can be measured in function of the factors: Di, Ef, Ad, and Co:<br>(Di)versity = Port. Products + Port. Competences<br>(Ef)ficiency = ratio of C. Opportunities over the Contacts towards potential C. Opportunities<br>(Ad)aptability = Intangible Assets<br>(Co)hesion = Reciprocity + Density<br>where W<sub>i</sub> is the weight of each metric according to its relevance in the CBE (which can be evaluated by questionnaires, interviews, literature, etc.)</td></tr><tr><td rowspan="3">Resilience Indicator (RI)</td><td>RI<sub>CBE</sub> = f(Di, Ef, Ad, Co)</td><td></td></tr><tr><td>Di = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">Σ<sub>i</sub> #PortPd<sub>i</sub> * W<sub>i</sub></div><div style="padding: 0 5px;">Σ<sub>i</sub> W<sub>i</sub></div></div> + <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">Σ<sub>i</sub> #PortCp<sub>i</sub> * W<sub>i</sub></div><div style="padding: 0 5px;">Σ<sub>i</sub> W<sub>i</sub></div></div></td><td></td></tr><tr><td>Ef = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">Σ<sub>i</sub> #CoOp<sub>i</sub> * W<sub>i</sub></div><div style="padding: 0 5px;">Σ<sub>i</sub> #CCoOp<sub>i</sub> * W<sub>i</sub></div></div> Ad = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding: 0 5px;">Σ<sub>i</sub> #IntgAs<sub>i</sub> * W<sub>i</sub></div><div style="padding: 0 5px;">Σ<sub>i</sub> W<sub>i</sub></div></div></td><td></td></tr></tbody></table>

The measures of centrality are adopted from [19], but they will have to be reformulated in order to incorporate the weights of the connections, which represent collaborative opportunities and contacts towards potential collaborative opportunities. Then, the resulting indicators are normalized in relation to the size of the CBE
(number of organizations). Finally, RI is calculated as a function of the factors (Di, Ef, Ad, and Co), in which the evaluating metrics are weighted by a value according to their relevance in the context of the CBE.

# 4 Assessment and Calibration of the Performance Indicators

Among the performance indicators introduced in the previous section: Innovation Indicator (II), Contribution Indicator (CI), Prestige Indicator (PI), and Resilience Indicator (RI), II is chosen for the illustrative examples presented below. The II for a given organization $O_i$ ($\Pi_i$), measures the ratio between the new products, services, or patents created by that organization (#NewPd$_i$) during a given period, and its portfolio (#PortPd$_i$). The same indicator for the whole CBE, measures the ratio between the number of total products, services, or patents created in the CBE ($\Sigma_i$ #NewPd$_i$), and the total portfolio of the CBE ($\Sigma_i$ #PortPd$_i$). However, this ratio does not give any indication about collaboration, i.e., the indicator does not reflect a measure of the improvement of innovation due to collaboration. Thus, the ratio II for the whole CBE ($\Pi_{CBE}$) is weighted by a correlation coefficient (Spearman's or Pearson's) [20], denoted by $r$, between the number of VOs in which the organizations have participated (#VO), and the new products, services, or patents that they have created (#NewPd).

Pearson's correlation coefficient is a measure of the extent to which two quantitative variables (in this case #VO and #NewPd) are linearly related. It measures the strength of the association between the two variables, giving a value between +1 and -1 inclusive, where 1 is total positive correlation, 0 is no correlation, and -1 is total negative correlation. The Spearman correlation coefficient, is the usual Person $r$ applied to data in the form of ranks, measuring how well the relationship between two variables can be described using a monotonic function. The process to rank data in order to calculate the Spearman correlation, consists of replacing each element of a set, by its ranking position according to its relative value (the lowest is rated one, and the higher the number of elements of the set). For instance, considering the set of data $S_1=\{9, 2, 8, 5\}$, the resulting rank set is $RkS_1=\{4, 1, 3, 2\}$. When the set contains repeating elements, for instance the set $S_2=\{9, 2, 5, 5\}$, the repeated elements are rated with the mean of their ranking positions, e.g., the mean for the element 5 is $(2 + 3)/2 = 2.5)$, resulting in the rank set $RkS_2=\{4, 1, 2.5, 2.5\}$.

To assess the metrics and coherence of the performance indicator II, as stated above, three simulated scenarios were created ($CBE_1$, $CBE_2$, and $CBE_3$), and described in **Table 4**, Table 5, and Table 6. The three scenarios represent the same CBE composed of ten organizations, but showing different indexes of innovation, and collaboration performances. The goal of the first two scenarios is to show the influence of the correlation coefficient over the same value of the ratio II of the CBE as a whole, i.e., the smaller the correlation coefficient, the smaller the value of the $\Pi_{CBE}$. The third scenario shows how a high ratio II is pushed down caused by a very low correlation coefficient, meaning that the innovation capacity of the ecosystem was not due to collaboration.
**Table 4** represents the simulated scenario CBE₁ exhibiting hypothetical series of data for the metrics #PortPd, #VO, and #NewPd, and the calculated ratio IIᵢ, between 0 and 1, using linear normalization. The values of the Π<sub>CBE</sub> (0.64 and 0.58), are calculated by multiplying the ratio II (0.77) by the Pearson (0.84) and Spearman (0.75) correlation coefficients, respectively.

<table><caption>**Table 4.** Simulated scenario of the CBE₁ to assess the Innovation Indicator (II).</caption><thead><tr><th>#N</th><th>#PortPd</th><th>#VO</th><th>#NewPd</th><th>II</th><th>Rk(#VO,#NewPd)</th></tr></thead><tbody><tr><td>O<sub>1</sub></td><td>1</td><td>2</td><td>2</td><td><strong>0.22</strong></td><td>3</td><td>6</td></tr><tr><td>O<sub>2</sub></td><td>2</td><td>2</td><td>2</td><td><strong>0.11</strong></td><td>3</td><td>6</td></tr><tr><td>O<sub>3</sub></td><td>3</td><td>3</td><td>1</td><td><strong>0.04</strong></td><td>5.5</td><td>2</td></tr><tr><td>O<sub>4</sub></td><td>2</td><td>4</td><td>2</td><td><strong>0.11</strong></td><td>7.5</td><td>6</td></tr><tr><td>O<sub>5</sub></td><td>3</td><td>5</td><td>3</td><td><strong>0.11</strong></td><td>9</td><td>9</td></tr><tr><td>O<sub>6</sub></td><td>4</td><td>3</td><td>2</td><td><strong>0.06</strong></td><td>5.5</td><td>6</td></tr><tr><td>O<sub>7</sub></td><td>5</td><td>6</td><td>4</td><td><strong>0.09</strong></td><td>10</td><td>10</td></tr><tr><td>O<sub>8</sub></td><td>1</td><td>2</td><td>1</td><td><strong>0.11</strong></td><td>3</td><td>2</td></tr><tr><td>O<sub>9</sub></td><td>2</td><td>4</td><td>2</td><td><strong>0.11</strong></td><td>7.5</td><td>6</td></tr><tr><td>O<sub>10</sub></td><td>3</td><td>1</td><td>1</td><td><strong>0.04</strong></td><td>1</td><td>2</td></tr><tr><td>10</td><td>26</td><td></td><td>20</td><td><strong>0.77 r<sub>P</sub>: 0.84 r<sub>S</sub>: 0.75</strong></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td><strong>Π<sub>CBE</sub>: 0.64</strong></td><td></td><td><strong>0.58</strong></td></tr></tbody></table>

Notes: r<sub>P</sub> and r<sub>S</sub> denote Pearson and Spearman correlation coefficient, respectively

Rk(#VO, #NewPd) stands for rank data of #VO and #NewPd, used to find Spearman correlation coefficient

<figure><img src="image_3.png" /></figure>

Table 5 depicts another scenario (CBE₂) showing the same ratio II for the whole ecosystem (0.77). However, the collaboration coefficients are lower (0.51 and 0.50), resulting in a lower value (0.39) for the Π<sub>CBE</sub>.

<table><caption>**Table 5.** Simulated scenario of the CBE₂ to assess the Innovation Indicator (II).</caption><thead><tr><th>#N</th><th>#PortPd</th><th>#VO</th><th>#NewPd</th><th>II</th><th>Rk(#VO,#NewPd)</th></tr></thead><tbody><tr><td>O<sub>1</sub></td><td>1</td><td>2</td><td>1</td><td><strong>0.11</strong></td><td>3</td><td>1.5</td></tr><tr><td>O<sub>2</sub></td><td>2</td><td>2</td><td>2</td><td><strong>0.11</strong></td><td>3</td><td>5.5</td></tr><tr><td>O<sub>3</sub></td><td>3</td><td>3</td><td>2</td><td><strong>0.07</strong></td><td>5</td><td>5.5</td></tr><tr><td>O<sub>4</sub></td><td>2</td><td>5</td><td>2</td><td><strong>0.11</strong></td><td>8.5</td><td>5.5</td></tr><tr><td>O<sub>5</sub></td><td>3</td><td>5</td><td>2</td><td><strong>0.07</strong></td><td>8.5</td><td>5.5</td></tr><tr><td>O<sub>6</sub></td><td>4</td><td>6</td><td>2</td><td><strong>0.05</strong></td><td>10</td><td>5.5</td></tr><tr><td>O<sub>7</sub></td><td>5</td><td>4</td><td>3</td><td><strong>0.06</strong></td><td>6.5</td><td>9.5</td></tr><tr><td>O<sub>8</sub></td><td>1</td><td>2</td><td>2</td><td><strong>0.22</strong></td><td>3</td><td>5.5</td></tr><tr><td>O<sub>9</sub></td><td>2</td><td>4</td><td>3</td><td><strong>0.16</strong></td><td>6.5</td><td>9.5</td></tr><tr><td>O<sub>10</sub></td><td>3</td><td>1</td><td>1</td><td><strong>0.04</strong></td><td>1</td><td>1.5</td></tr><tr><td>10</td><td>26</td><td></td><td>20</td><td><strong>0.77 r<sub>P</sub>: 0.51 r<sub>S</sub>: 0.50</strong></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td><strong>Π<sub>CBE</sub>: 0.39</strong></td><td></td><td><strong>0.39</strong></td></tr></tbody></table>

<figure><img src="image_4.png" /></figure>

The last scenario (CBE₃), depicted in Table 6, exhibits a high ratio II (0.92), but a very low collaboration coefficients (0.13 and 0.10), resulting in equally low Π<sub>CBE</sub>.
values (0.12 and 0.09), meaning that the high improvement of innovation is not due to the collaboration in the CBE.

<table><caption>Table 6. Simulated scenario of the CBE₃ to assess the Innovation Indicator (II).</caption><thead><tr><th>#N</th><th>#PortPd</th><th>#VO</th><th>#NewPd</th><th>II</th><th>Rk(#VO,#NewPd)</th></tr></thead><tbody><tr><td>O<sub>1</sub></td><td>1</td><td>5</td><td>3</td><td><strong>0,25</strong></td><td>8,5</td><td>7</td></tr><tr><td>O<sub>2</sub></td><td>2</td><td>6</td><td>1</td><td><strong>0,04</strong></td><td>10</td><td>2,5</td></tr><tr><td>O<sub>3</sub></td><td>3</td><td>2</td><td>2</td><td><strong>0,06</strong></td><td>2,5</td><td>5</td></tr><tr><td>O<sub>4</sub></td><td>2</td><td>4</td><td>1</td><td><strong>0,04</strong></td><td>6</td><td>2,5</td></tr><tr><td>O<sub>5</sub></td><td>3</td><td>4</td><td>1</td><td><strong>0,03</strong></td><td>6</td><td>2,5</td></tr><tr><td>O<sub>6</sub></td><td>4</td><td>5</td><td>5</td><td><strong>0,11</strong></td><td>8,5</td><td>10</td></tr><tr><td>O<sub>7</sub></td><td>5</td><td>3</td><td>4</td><td><strong>0,07</strong></td><td>4</td><td>9</td></tr><tr><td>O<sub>8</sub></td><td>1</td><td>2</td><td>3</td><td><strong>0,25</strong></td><td>2,5</td><td>7</td></tr><tr><td>O<sub>9</sub></td><td>2</td><td>4</td><td>3</td><td><strong>0,13</strong></td><td>6</td><td>7</td></tr><tr><td>O<sub>10</sub></td><td>3</td><td>1</td><td>1</td><td><strong>0,03</strong></td><td>1</td><td>2,5</td></tr><tr><td>10</td><td>26</td><td></td><td></td><td>24</td><td><strong>0,92 r<sub>P</sub>: 0,13 r<sub>S</sub>: 0,10</strong></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td>Π<sub>CBE</sub>: 0,12</td><td>0,09</td><td></td></tr></tbody></table>

<figure><img src="image_5.png" /></figure>

Considering the boundary conditions of the correlation coefficient ($r = 1$ and $r = 0$), the value $r = 1$ maximizes the result of the $Π_{CBE}$, meaning that the innovation capacity is all up to collaboration. The value $r = 0$, pushes the resulting $Π_{CBE}$ down to 0, meaning that the innovation is not related to collaboration. A negative correlation coefficient is not considered, as it is not assumed that collaboration could influence negatively the innovation capability of the organizations.

The above analysis is based on simulated scenarios. For a further assessment of the proposed indicators, the ongoing work involves capturing real data from a concrete business ecosystem. This is being done for the ports area, a collaborative community offering integrated logistics and freight services. A port area is an interesting use case, as it offers a wide diversity of players and services, such as port authorities, customs, freight forwarders, multi-modal transport entities (rail, road, and maritime), suppliers, and customers, collaborating through increasingly integrated technological platforms, effectively contributing to the growth of the local economy and social value.

## 5 Conclusions and Further Work

The metrics and associated performance indicators presented along this paper, contribute as a first approach to accomplish the research question and related hypothesis stated in the first section, by proposing a set of indicators to assess collaboration benefits. A set of metrics were also suggested to quantify those benefits, and different scenarios of CBEs represented by models with hypothetical data were used in order to assess the validity of the measures associated to the indicators. The ongoing work, as mentioned above, encompasses the continuation of the assessment
of the proposed indicators, but capturing real data from concrete businesses ecosystems.

The future work is aimed at finding an answer to address the second research question, which consists of:

*"How to promote collaboration sustainability and resilience within a business ecosystem?"* [4]

In order to respond to this question, the following hypothesis is considered: *"Sustainability and resilience of collaboration in business ecosystems can be promoted if a system of incentives, combined with transparent assessment methods, is implemented at the ecosystem level"*. This hypothesis will guide the next steps of this research, which will be mostly supported by models using system dynamics (SD) combined with agent based (AB) simulation. The AnyLogic Multimethod Simulation Software [21], will provide the methods and simulation tools to create and explore these models. SD is a method to improve learning in complex systems, somewhat a method to develop computer models, to help learning about dynamic complexity, understand the sources of policy resistance, and design more effective policies [22].

The goal of applying SD at this stage of the research is first to capture and represent the organization of a CBE, considered as a complex system, which, along with stocks and flows structures, feedback processes, time delays, and other structures, determine its dynamic behavior. In a second phase, the performance indicators should be introduced to affect the dynamic behavior of the agents. Finally, a third phase will be dedicated to simulation to observe the agents' behavior, playing with the introduction of incentives/penalizations, resulting from the decisions of the managers.

The explained research approach, aims to specify, validate and make accepted by experts and the scientific community, a set of indicators for CBEs, which not only allow to evaluate the performance of the system, but also contribute to motivate its sustainability and resilience.

**Acknowledgment.** This work benefited from the ongoing research within the CoDIS (Collaborative Networks and Distributed Industrial Systems Group) which is part of both the New University of Lisbon (UNL) - Faculty of Sciences and Technology, and the UNINOVA - Institute for the Development of New Technologies. Partial support also comes from Fundação para a Ciência e Tecnologia through the PEST program UID/EEA/00066/2013.

**References**

1. Moore, J. F. "Predators and prey: a new ecology of competition." Harvard business review 71, no. 3: 75-83, 1993

2. Moore, J. F. The death of competition: leadership and strategy in the age of business ecosystems. HarperCollins Publishers, 1996

3. Kelly, E. "Introduction: Business ecosystems come of age.", 2015.

4. Graça, P., and L. M. Camarinha-Matos. "The Need of Performance Indicators for Collaborative Business Ecosystems." In Technological Innovation for Cloud-Based Engineering Systems, pp. 22-30. Springer International Publishing, 2015

5. Camarinha-Matos, L. M., and H. Afsarmanesh. Collaborative networks: A new scientific
discipline. J. Intelligent Manufacturing, vol. 16, N° 4-5, pp439-452, 2005

6. Camarinha-Matos, L. M., and H. Afsarmanesh. Collaborative networks: Reference Modeling. Springer Science & Business Media, 2008

7. Kaplan, R. S., and D. P. Norton. The balanced scorecard: translating strategy into action. Harvard Business Press, 1996

8. Romero, D., N. Galeano, and A. Molina. "A conceptual model for virtual breeding environments value systems." Establishing the foundation of collaborative networks: 43-52, 2007

9. Camarinha-Matos, L. M., and P. Macedo. "A conceptual model of value systems in collaborative networks." Journal of Intelligent Manufacturing 21, no. 3: 287-299, 2010

10. Macedo, P., and L. M. Camarinha-Matos. "A qualitative approach to assess the alignment of Value Systems in collaborative enterprises networks." Computers & Industrial Engineering 64, no. 1: 412-424, 2013

11. Abreu, A., and L. M. Camarinha-Matos. "A benefit analysis model for collaborative networks." In Collaborative networks: Reference modeling, pp. 253-276. Springer, 2008

12. Gopal, P. R. C., and J. Thakkar. "A review on supply chain performance measures and metrics: 2000-2011." International Journal of Productivity and Performance Management 61, no. 5: 518-547, 2012

13. Ramanathan, U. "Performance of supply chain collaboration-A simulation study." Expert Systems with Applications 41, no. 1: 210-220, 2014.

14. Abreu, A., and L. M. Camarinha-Matos. "An Approach to Measure Social Capital in Collaborative Networks." In Adaptation and Value Creating Collaborative Networks, pp. 29-40. Springer Berlin Heidelberg, 2011

15. Battistella, C., K. Colucci, A. F. De Toni, and F. Nonino. "Methodology of business ecosystems network analysis: A case study in Telecom Italia Future Centre." Technological Forecasting and Social Change 80, no. 6: 1194-1210, 2013

16. Allee, V., "Value Networks and the true nature of collaboration." Digital edition edn. ValueNet Works and Verna Allee Associates, 2011

17. Fiksel, J. "Sustainability and resilience: toward a systems approach." Sustainability: Science, Practice, & Policy 2, no. 2, 2006

18. Fiksel, J. "Designing resilient, sustainable systems." Environmental science & technology 37, no. 23: 5330-5339, 2003

19. Freeman, L. C. "Centrality in social networks conceptual clarification." Social networks 1, no. 3: 215-239, 1978

20. Myers, J. L., A. Well, and R. F. Lorch. Research design and statistical analysis. Routledge, 2010

21. "Multimethod Simulation Software and Solutions." http://www.anylogic.com/, Accessed May 12, 2016

22. Sterman, J. D. Business dynamics: systems thinking and modeling for a complex world. Vol. 19. Boston: Irwin/McGraw-Hill, 2000