# Using BSC and DEMATEL Method to Construct the Novel Product Concepts Evaluation System

Zhe Huang, Mickaël Gardoni

► To cite this version:

Zhe Huang, Mickaël Gardoni. Using BSC and DEMATEL Method to Construct the Novel Product Concepts Evaluation System. 17th IFIP International Conference on Product Lifecycle Management (PLM), Jul 2020, Rapperswil, Switzerland. pp.324-333, 10.1007/978-3-030-62807-9_26 hal-03753139

HAL Id: hal-03753139

https://inria.hal.science/hal-03753139v1

Submitted on 17 Aug 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Using BSC and DEMATEL Method to Construct the Novel Product Concepts Evaluation System

Zhe Huang¹* and Mickaël Gardoni ¹,²

¹ École de technologie supérieure, Montréal, Québec, Canada
hz8018@outlook.com*; mickael.gardoni@etsmtl.ca

² INSA de Strasbourg, Strasbourg, France
mickael.gardoni@insa-strasbourg.fr

**Abstract.** Concept assessment is a complex problem which needs to be evaluated comprehensively and multiple dimensions. This paper proposes a multidimensional concepts evaluation system based on Balanced Scorecard (BSC) and Decision-Making Trial and Evaluation Laboratory (DEMATEL) method. BSC provides a reference framework to select evaluation indicators. DEMATEL method can effectively analyze the mutual relationship and structure of components in the system. This methodology can help manager to evaluate from a holistic perspective. This paper identifies and constructs the causal relationships among the critical evaluation criteria. The cause and effect diagram analysis clearly indicate which aspect /criteria is more important to make continuous improvement, so as to enhance competitiveness.

**Keywords:** Concepts Evaluation, Balanced Scorecard (BSC), Decision-Making Trial and Evaluation Laboratory (DEMATEL).

## 1 Introduction

Innovation is a process of transforming concepts into useful products or services [1]. The concept evaluation has an important impact on the success and cost of innovation [2,3]. Rietzschel et al. [4] proposed that it is necessary to clearly determine the concept evaluation criteria and optimize the evaluation process, in order to choose the best concept. Hart [5] emphasized that technical feasibility is the most commonly used criterion for evaluating concepts. Shortening the development time and choosing the right strategy are important factors to survive in the market [6]. Schwarz and Bodendorf [7] proposed that good concepts must be feasible, new, profitable, useful, very mature, and so on.

The concept evaluation is a complex issue, multiple factors need to be considered comprehensively [8]. The identification of key influencing factors is therefore significant to the reasonable concept evaluation. Our study aims to propose a multidimensional approach to conduct a comprehensive concept assessment. In view of the complexity of the evaluation issue, this paper identifies first the key influencing factors of concept evaluation based on Balanced Score Card (BSC) method. BSC provides a reference framework for us to select concept evaluation indicators [9]. However, the
degree of mutual impact between key factors may be different [10]. This paper adopts Decision-Making Trial and Evaluation Laboratory (DEMATEL) method to identify the interdependence among criteria of evaluation system through a causal diagram. The proposed approach can aid decision makers to make decisions based on full consideration of the most relevant impact indicators for achieving goals. The next section presents the methodology used in this paper.

# 2 Methodology

## 2.1 Balanced Score Card (BSC)

Robert S. Kaplan and David P. Norton first proposed "balanced scorecard" (abbreviated to "BSC") in their article "Balanced Scorecard - measures to drive performance", which was published in Harvard Business Review in 1996 [9]. The BSC comprehensively takes into account a balance between short-term and long-term objectives financial and non-financial measures, and internal and external interests [11].

BSC provides a framework for thinking and acting sustainably [12]. This paper divides the concept evaluation system into four perspectives based on BSC method: Concept attractiveness perspective (P1), Financial perspective (P2), Market perspective (P3) and Innovative capabilities perspective (P4).

Concept Attractiveness Perspective (P1). Understanding of customer existing or potential needs is the starting point for innovation. The new concept shall be applicable to and effectively meet these needs [13,14]. In addition, the concept should be clear, complete and well communicated (specificity) [15].

Financial Perspective (P2). The success of the enterprise must finally translate into financial success. Concepts evaluation must also take into account financial factors. The development of new concepts is a long process. Cooper [16] emphasizes that the shortening of R&D cycle and the acceleration of market access can provide a powerful guarantee for enterprises to seize market opportunities.

Market Perspective (P3). In research of Frishammar et al. [17], successful firms generally analyze the prospective and current product offerings of competitors. It is very important to assess market segments in which the business will compete.

Innovative Capabilities Perspective (P4). Not all new product concepts can be developed into new products. Some concepts may be very good, but if the enterprise lacks the corresponding resource conditions, the concept lacks the possibility of development. The Innovative capabilities perspective mainly measures the ability to achieve transformation of concept, so as to better meet the changing needs of customers and achieve sustainable growth.

16 evaluation criteria in 4 dimensions of the concept evaluation index system based on BSC method are summarized in Table 1, which will be set as $P_{11}, P_{12},..., P_{ij}$.
<table><caption>**Table 1.** Evaluation index system.</caption><thead><tr><th>Dimension</th><th>Criteria</th><th>Explication</th></tr></thead><tbody><tr><td rowspan="5">Concept attractiveness perspective (P1)</td><td>Novelty (P<sub>11</sub>)</td><td>Original and uncommon.</td></tr><tr><td>Specificity (P<sub>12</sub>)</td><td>Clear, complete and explicit.</td></tr><tr><td>Workability (P<sub>13</sub>)</td><td>It does not violate known constraints or if it can be easily implemented.</td></tr><tr><td>Uniqueness (P<sub>14</sub>)</td><td>The degree to which the concept is unique.</td></tr><tr><td>Effectiveness (P<sub>15</sub>)</td><td>The degree to which the concept will solve the problem.</td></tr><tr><td rowspan="3">Financial perspective (P2)</td><td>Investment cost (P<sub>21</sub>)</td><td>Financial resources required.</td></tr><tr><td>R & D cycle (P<sub>22</sub>)</td><td>R & D cycle.</td></tr><tr><td>Anticipated profitability (P<sub>23</sub>)</td><td>Anticipated profitability of an investment in development.</td></tr><tr><td rowspan="4">Market perspective (P3)</td><td>Market need (potential) (P<sub>31</sub>)</td><td>Is it a growing market?</td></tr><tr><td>Forward-looking (P<sub>32</sub>)</td><td>Does this concept match the current development, is it reasonable in the market?</td></tr><tr><td>Competitive intensity (P<sub>33</sub>)</td><td>Market competition (number and intensity of competitors)</td></tr><tr><td>Entry barriers (P<sub>34</sub>)</td><td>Obstacles that make it difficult to enter a given market.</td></tr><tr><td rowspan="4">Innovative capabilities perspective (P4)</td><td>Access to technology (P<sub>41</sub>)</td><td>Measure complexity to master this technique</td></tr><tr><td>Relevant capabilities and experience (P<sub>42</sub>)</td><td>Do we have the necessary skills and experience?</td></tr><tr><td>Available physical resources (P<sub>43</sub>)</td><td>Difference between required and existing physical resources</td></tr><tr><td>Strategic alignment (P<sub>44</sub>)</td><td>Consistency with the company's strategy</td></tr></tbody></table>

## 2.2 Decision-Making Trial and Evaluation Laboratory (DEMATEL)

In view of the different degree of interaction between criteria, this paper made an in-depth analysis of the relationship among criteria based on the Decision-Making Trial and Evaluation Laboratory (DEMATEL) method. DEMATEL method developed by Battelle Memorial Institute between 1972 to 1979 [18]. DEMATEL method can not only visualize the causal relationship among variables, but also identify the key factors. DEMATEL method has been successfully applied to the operation safety evaluation of
urban rail stations [19], geographical environment evaluation and other fields. Here are the steps of DEMATEL [18, 20]:

**Step 1: Generating the Initial Direct Relation Matrix A.** An evaluation scale from 0 to 4 is used to measure the direct influence relationship among the indicators: 0 (no influence), 1(low influence), 2(medium influence), 3 (high influence), and 4 (very high influence), respectively. The initial direct relation matrix $A= (a_{ij})_{n \times n}$ is then obtained,

$$A= \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix} \quad (1)$$

where $a_{ij}$ denoted as the degree in which the factor $i$ affects the factor $j$ [20]. If factor $i$ has a very strong influence on factor $j$, then define $a_{ij} = 4$.

**Step 2: Normalizing the Direct-Relation Matrix D.** Normalize matrix A according to equation (2).

$$D = \lambda * A \quad (2)$$

where $\lambda = Min(\frac{1}{Max \sum_{j=1}^{n} a_{ij}} , \frac{1}{Max \sum_{i=1}^{n} a_{ij}})$; $1 \le i \le n, 1 \le j \le n$.

**Step 3: Construct the Total Relation Matrix T.** Use the following equation (3) to calculate the total relation matrix T:

$$T=D*(I-D)^{-1} \quad (3)$$

Where I denoted as the identity matrix.

**Step 4: Calculate (Equation 4 and 5) the Sum of the Elements of Each Row (R) and Each Column (C) of the Total Relation Matrix T.**

$$R_i = \sum_{j=1}^{n} t_{ij} \quad (i=1, 2...n) \quad (4)$$

$$C_j = \sum_{i=1}^{n} t_{ij} \quad (j=1, 2...n) \quad (5)$$

$R_i$ represents the effects given by the factor $i$ to other factors. $C_j$ indicate the effect received by the factor $j$ from other factors [21].

DEMATEL method can effectively show the complex causal structure. The relative importance degree among factors can be determined by calculating the value of (R + C) and (R - C). The causal diagram then can be drawn. The sum (R + C) is horizontal axis, called "Prominence", which shows the relative importance of each index. The difference (R - C) is vertical axis called "Relation", which divides the indicator into cause group and effect groups [21]. When (R - C) is positive, the factor $i$ belong to the
cause group; if (R - C) is negative, the factor *i* is a receiver which belongs to the effect group.

# 3 Case Study

## 3.1 24h of Innovation

"24h of innovation" is an international innovation competition organized by ETS (www.24h.com). Participants must develop innovative solutions, within 24 hours, to meet the challenges proposed by enterprises. The best solution for each site is ultimately evaluated by an international jury Committee from academia and industry to select the main winners of the year from local winners.

The goal of innovation competition is not only to encourage innovation, but also to help to find good concepts that can be transformed into efficient products. Varies evaluation factors coexist. This paper constructs the evaluation indicators system based on BSC and DEMATEL, in order to help select the promising concept which deserves further development. This system can effectively simplify the types and quantity of indicators and clarify the impact relationship between indicators, so as to help make decisions based on full consideration of various factors.

## 3.2 Proposed Methodology

On the basis of the above analysis, the procedure of concept evaluation index system with BSC and DEMATEL methods are presented as follows:

**Identify the Influential Criteria for Concept Evaluation and Generate the Direct-Relation Matrix (A).** The BSC method was used to identify the influential criteria for concept evaluation (Table 1), and DEMATEL was used to find out the influence degree among indicators. In this paper, the direct influence degree of criteria *i* on criteria *j* is divided into five grades (See Table 2). Three experts are invited to evaluate the interaction among criteria according to table 2. The initial average direct relation matrix A (Table 3, Table 4) is generated by summing up all the scores separately and averaging them.

<table><caption>**Table 2.** Direct impact degree of criteria.</caption><thead><tr><td>Direct influence degree of criteria <i>i</i> on criteria <i>j</i></td><td>Score</td></tr></thead><tbody><tr><td>no influence</td><td>0</td></tr><tr><td>low influence</td><td>1</td></tr><tr><td>medium influence</td><td>2</td></tr><tr><td>high influence</td><td>3</td></tr><tr><td>very high influence</td><td>4</td></tr></tbody></table>
<table><caption>Table 6. Total relation matrix (T) of criteria</caption><thead><tr><th>T2</th><th>P11</th><th>P12</th><th>P13</th><th>P14</th><th>P15</th><th>P21</th><th>P22</th><th>P23</th><th>P31</th><th>P32</th><th>P33</th><th>P34</th><th>P41</th><th>P42</th><th>P43</th><th>P44</th></tr></thead><tbody><tr><td>P11</td><td>0.07030</td><td>0.09999</td><td>0.11767</td><td>0.11780</td><td>0.10283</td><td>0.12272</td><td>0.10423</td><td>0.13033</td><td>0.09946</td><td>0.12761</td><td>0.13523</td><td>0.03461</td><td>0.10385</td><td>0.09613</td><td>0.08494</td><td>0.09226</td></tr><tr><td>P12</td><td>0.16098</td><td>0.15716</td><td>0.23228</td><td>0.21437</td><td>0.22937</td><td>0.24675</td><td>0.20356</td><td>0.27406</td><td>0.20808</td><td>0.19963</td><td>0.25419</td><td>0.07526</td><td>0.23624</td><td>0.23421</td><td>0.16831</td><td>0.19496</td></tr><tr><td>P13</td><td>0.16694</td><td>0.19018</td><td>0.14319</td><td>0.17234</td><td>0.20758</td><td>0.21414</td><td>0.18097</td><td>0.21051</td><td>0.16581</td><td>0.15521</td><td>0.21154</td><td>0.05235</td><td>0.22045</td><td>0.16760</td><td>0.14147</td><td>0.18576</td></tr><tr><td>P14</td><td>0.23374</td><td>0.23943</td><td>0.25229</td><td>0.17029</td><td>0.21564</td><td>0.29229</td><td>0.26548</td><td>0.27115</td><td>0.22866</td><td>0.24459</td><td>0.26586</td><td>0.09575</td><td>0.28349</td><td>0.25892</td><td>0.20465</td><td>0.20521</td></tr><tr><td>P15</td><td>0.18154</td><td>0.24287</td><td>0.25999</td><td>0.20291</td><td>0.16386</td><td>0.27586</td><td>0.25030</td><td>0.25000</td><td>0.22631</td><td>0.21748</td><td>0.23676</td><td>0.06283</td><td>0.24907</td><td>0.23334</td><td>0.18201</td><td>0.21470</td></tr><tr><td>P21</td><td>0.16101</td><td>0.17650</td><td>0.20589</td><td>0.18519</td><td>0.17409</td><td>0.17282</td><td>0.20406</td><td>0.19717</td><td>0.13746</td><td>0.17426</td><td>0.21917</td><td>0.04538</td><td>0.23583</td><td>0.18706</td><td>0.16935</td><td>0.16589</td></tr><tr><td>P22</td><td>0.10256</td><td>0.14574</td><td>0.14883</td><td>0.12804</td><td>0.17785</td><td>0.22125</td><td>0.12793</td><td>0.19688</td><td>0.13254</td><td>0.12463</td><td>0.19371</td><td>0.05106</td><td>0.16430</td><td>0.15898</td><td>0.12493</td><td>0.12449</td></tr><tr><td>P23</td><td>0.10157</td><td>0.12679</td><td>0.14114</td><td>0.13027</td><td>0.12306</td><td>0.21890</td><td>0.18827</td><td>0.13652</td><td>0.14451</td><td>0.17466</td><td>0.19764</td><td>0.07413</td><td>0.18571</td><td>0.16025</td><td>0.11917</td><td>0.14466</td></tr><tr><td>P31</td><td>0.21250</td><td>0.25248</td><td>0.25254</td><td>0.24034</td><td>0.24192</td><td>0.28946</td><td>0.24335</td><td>0.29053</td><td>0.15245</td><td>0.21907</td><td>0.28524</td><td>0.08626</td><td>0.21903</td><td>0.23538</td><td>0.18088</td><td>0.19242</td></tr><tr><td>P32</td><td>0.20060</td><td>0.19378</td><td>0.15675</td><td>0.17187</td><td>0.13737</td><td>0.24275</td><td>0.21498</td><td>0.21956</td><td>0.18564</td><td>0.13195</td><td>0.23045</td><td>0.07466</td><td>0.19324</td><td>0.17470</td><td>0.15661</td><td>0.14198</td></tr><tr><td>P33</td><td>0.14068</td><td>0.14732</td><td>0.16182</td><td>0.15645</td><td>0.13854</td><td>0.21060</td><td>0.19916</td><td>0.20665</td><td>0.15957</td><td>0.16697</td><td>0.14086</td><td>0.04886</td><td>0.19082</td><td>0.18503</td><td>0.10240</td><td>0.12201</td></tr><tr><td>P34</td><td>0.13715</td><td>0.12682</td><td>0.18505</td><td>0.13163</td><td>0.11816</td><td>0.22375</td><td>0.18603</td><td>0.20336</td><td>0.10750</td><td>0.17086</td><td>0.21305</td><td>0.04000</td><td>0.22502</td><td>0.19544</td><td>0.13850</td><td>0.17626</td></tr><tr><td>P41</td><td>0.19563</td><td>0.22594</td><td>0.23750</td><td>0.21385</td><td>0.22287</td><td>0.27956</td><td>0.25509</td><td>0.24623</td><td>0.15495</td><td>0.21940</td><td>0.24906</td><td>0.08871</td><td>0.18141</td><td>0.20028</td><td>0.13549</td><td>0.19987</td></tr><tr><td>P42</td><td>0.17566</td><td>0.19299</td><td>0.22972</td><td>0.18076</td><td>0.20615</td><td>0.25623</td><td>0.24123</td><td>0.23602</td><td>0.14030</td><td>0.19107</td><td>0.20418</td><td>0.04830</td><td>0.22755</td><td>0.14734</td><td>0.15161</td><td>0.19572</td></tr><tr><td>P43</td><td>0.06545</td><td>0.09581</td><td>0.10893</td><td>0.09423</td><td>0.09000</td><td>0.15670</td><td>0.15215</td><td>0.13433</td><td>0.06392</td><td>0.08214</td><td>0.11697</td><td>0.02469</td><td>0.11683</td><td>0.10810</td><td>0.06123</td><td>0.12665</td></tr><tr><td>P44</td><td>0.17756</td><td>0.17043</td><td>0.17935</td><td>0.18646</td><td>0.16764</td><td>0.23749</td><td>0.21180</td><td>0.21802</td><td>0.11389</td><td>0.19126</td><td>0.17345</td><td>0.04383</td><td>0.19081</td><td>0.19589</td><td>0.14819</td><td>0.11801</td></tr><tr><td>R</td><td>1.63997</td><td>3.28942</td><td>2.78603</td><td>3.72744</td><td>3.44982</td><td>2.81114</td><td>2.32373</td><td>2.36723</td><td>3.59386</td><td>2.82692</td><td>2.47772</td><td>2.57856</td><td>3.30584</td><td>3.02384</td><td>1.59813</td><td>2.72410</td></tr><tr><td>C</td><td>2.48388</td><td>2.78425</td><td>3.01294</td><td>2.69679</td><td>2.71592</td><td>3.66129</td><td>3.22858</td><td>3.42132</td><td>2.42106</td><td>2.79077</td><td>3.32735</td><td>0.94667</td><td>3.22366</td><td>2.93866</td><td>2.26976</td><td>2.60085</td></tr><tr><td>R+C</td><td>4.123849</td><td>6.073665</td><td>5.798971</td><td>6.424227</td><td>6.165741</td><td>6.472424</td><td>5.552312</td><td>5.788545</td><td>6.014918</td><td>5.617691</td><td>5.805078</td><td>3.525234</td><td>6.529498</td><td>5.962495</td><td>3.867886</td><td>5.324949</td></tr><tr><td>R-C</td><td>-0.84391</td><td>0.505173</td><td>-0.22691</td><td>1.030649</td><td>0.733895</td><td>-0.85015</td><td>-0.90485</td><td>-1.05409</td><td>1.172804</td><td>0.036145</td><td>-0.84963</td><td>1.631894</td><td>0.082179</td><td>0.08518</td><td>-0.67163</td><td>0.123243</td></tr></tbody></table>

# 4 Result Analysis

The analysis of causality is helpful to judge which evaluation dimensions are relatively important, which perspective needs continuous improvement [22]. DEMATEL presents the complex relationship among factors with cause and effect diagram.

## 4.1 Producing a Causal Diagram of Four Perspectives

According to Table 5, we can get the causal diagram of four perspectives (see Fig. 1).

<figure><img src="image_3.png" /><figcaption>Fig. 1. Cause and effect diagram (Dimension relationship)</figcaption></figure>

Perspectives was divided in cause group and effect group by the value of (R - C). Fig.1 shows that the perspective P1 and P2 belongs to the effect group which are influenced by the perspective P3 and P4. P3 and P4 constitute the cause group, which indicate that changes in the innovative capabilities dimension and the market dimension directly affect the concept attractiveness dimension and the financial dimension.
The horizon axis (R+C) indicates the importance degree of evaluation perspective. The greater the value of (R + C) is, the higher the importance is. As shown in Fig. 1, the (R+C) value of innovative capabilities perspective is the highest, which means that the impact of innovative capabilities perspective to other perspectives is the greatest. Innovative capabilities perspective affects the other three perspectives. Innovative capabilities perspective and concept attractiveness perspective affect also each other. Financial perspective is affected by concept attractiveness, market and innovative capabilities perspective. Compared with the concept attractiveness, financial or market perspective, improving innovative capabilities plays a crucial role to the success of innovation.

## 4.2 Producing a Causal Diagram of Indicators.

According to Table 6, the cause and effect diagram among evaluate indicators was constructed as Fig. 2.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Cause and effect diagram (Indicator relationship).</figcaption></figure>

From Fig. 2, Uniqueness has a greater correlation with other factors of concept attractiveness perspective. Investment cost is the most critical within financial perspective; market need and access to technology are the most critical within market perspective and innovative capabilities perspective, respectively. These factors have the greatest influence on other factors for each dimension. Enterprise managers should pay attention to these factors.

It can be seen from Fig. 2 that P12, P14, P15, P31, P32, P34, P41, P42 and P44 are cause factors according to their value of (R - C). P11, P13, P21, P22, P23, P33 and P43 are the result factors. The value of (R+C) shows the degree of importance of the factors. For example, in the concept attractiveness perspective, Uniqueness and Effectiveness have the greatest influence on other factors. Cause factors have great influence on other
factors. Relatively, the factor in effect group are on the level of being affected, and there
is no much room for improvement. If we want to improve the success rate of concept
development / innovation, the cause index should be first focused on.

**5 Conclusion**

Concept evaluation is a complex problem, which is influenced by a number of factors.
This paper establishes four dimensions and 16 specific evaluation criteria of the concept
evaluation index system. BSC provides a reference framework to select performance
indicators. DEMATEL method helps to analyze interrelationships of the criteria by di-
viding factors into cause and effect group. Cause factors are considered to be more
influent in the decision. Managers should adjust and improve factors in cause group.
The proposed integration approach can help to evaluate from a holistic perspective.
This methodology identifies the most important aspect to enhance competitivity, so as
to provide targeted basis for concept evaluation. In the future research, a priority of
concepts will be reasonably identified in combination with other methods (such as
ANP) to improve the accuracy of concept evaluation.

**References**

1. Robbins, S. P.: Organisational Behaviour: Global and Southern African Perspectives. Pearson South Africa (2001).

2. Cooper, R. G.: New products: What separates the winners from the losers and what drives success. PDMA handbook of new product development, 3-34(2013).

3. Cooper, R. G., Kleinschmidt, E. J.: Screening new products for potential winners. Long Range Planning, 26(6), 74-81(1993).

4. Rietzschel, E. F., Nijstad, B. A., Stroebe, W.: The selection of creative ideas after individual idea generation: Choosing between creativity and impact. British journal of psychology, 101(1), 47-68(2010).

5. Hart, S., Jan, H., E., Tzokas, N., & Commandeur, H. R. Industrial companies' evaluation criteria in new product development gates. Journal of product innovation management, 20(1), 22-36 (2003).

6. Bandinelli R., d'Avolio E., Rossi M., Terzi S., Rinaldi R. Assessing the Role of Knowledge Manage-ment in the New Product Development Process: An Empirical Study. In: Fukuda S., Bernard A., Gurumoor-thy B., Bouras A. (eds) Product Lifecycle Management for a Global Market. PLM 2014. IFIP Advances in Information and Communication Technology, vol 442. Springer, Berlin, Heidelberg (2014).

7. Schwarz, S., Bodendorf, F.: Attributive Idea Evaluation: A New Idea Evaluation Method for Corporate Open Innovation Communities. International Journal of Knowledge-Based Organizations (IJKBO), 2(1), 77-91 (2012).

8. Huang, Zhe., Ahmed, Cherifi & Gardoni, Mickaël. A model for supporting the ideas screen-
ing during front end of the innovation process based on combination of methods of EcaTRIZ,
AHP, and SWOT. Concurrent Engineering. 29(2020).

9. Kaplan, R. S., & Norton, D. P.: The balanced scorecard: translating strategy into action.
Harvard Business Press (1996).
10. Elham. F., Shamsuddin, A., Shahryar S.: Expanded DEMATEL for Determining Cause and Effect Group in Bidirectional Relations. Hindawi Publishing Corporation, The Scientific World Journal, Volume 2014, Article ID 103846, 7 pages (2014).

11. Yuan, F.C., Chiu, C.: A hierarchical design of casebased reasoning in the balanced scorecard application. Expert Systems with Applications 36; 333-342(2009).

12. Fabio, D. F., Antonella, P.: Key Success Factors for Organizational Innovation in the Fashion Industry, International Journal of Engineering Business Management, Aug. DOI: 10.5772/56882 (2013).

13. Dean, D. L., Hender, J. M., Rodgers, T. L., Santanen, E.: Identifying good ideas: constructs and scales for idea evaluation (2006).

14. Inoue M., Yamada S., Yamada T., Bracke S. A Design Method for Product Upgradability with Different Customer Demands. In: Fukuda S., Bernard A., Gurumoorthy B., Bouras A. (eds) Product Lifecycle Management for a Global Market. PLM 2014. IFIP Advances in Information and Communication Technology, vol 442. Springer, Berlin, Heidelberg (2014). https://doi.org/10.1007/978-3-662-45937-9_10.

15. Kudrowitz, B. M., Wallace, D.: Assessing the quality of ideas from pro-lific, early-stage product ideation. Journal of Engineering Design, 24(2), 120-139(2013).

16. Cooper, R. G.: Predevelopment activities determine new product success. Industrial Marketing Management, 17(3), 237-247 (1988).

17. Frishammar, J., Florén, H.: Where new product development begins: success factors, contingencies and balancing acts in the fuzzy front end. In 17th International Conference on Management of Technology, Dubai, April 5-8 (p. 47) (2008).

18. Du,Y.W., WANG,S.S.: Fuzzy comprehensive evaluation on science fund project performance evaluation based on DEMATEL method. Bulletin of National Natural Science Foundation of China, 18(02), 161-169 (2018).

19. LI Xinhua,LI Junhui.A study on an assessment system on safe operation of urban rail stations based on DEMATEL and ISM [J].Railway Transport and Economy, 40(7):116-121 (2018)

20. Maghsud, A., Jamshid S.S., Nafiseh, P., Mahdi S.: Developing a DEMATEL method to prioritize distribution centers in supply chain. Management Science Letters 1 279-288(2011).

21. Sheng, L.S., Xiao, Y.Y., Hu, C.L., Ping, Z.: DEMATEL Technique: A Systematic Review of the State-of-the-Art Literature on Methodologies and Applications. Hindawi Mathematical Problems in Engineering Volume 2018, 33 pages https://doi.org/10.1155/2018/3696457 (2018).

22. Zhang, S. X.: Fuzzy multi - criteria evaluation method and statistics: Taiwan wunan book publishing co. LTD. Taiwan (2012).