# Multicriteria Evaluation Method in PLM Environment:
A Pilot Study

Kévin Audoux, Floriane Laverne, Gianluca D'antonio, Frédéric Segonds,
Olivier Kerbrat, Paolo Chiabert, Améziane Joussat

► To cite this version:

Kévin Audoux, Floriane Laverne, Gianluca D'antonio, Frédéric Segonds, Olivier Kerbrat, et al. Multi-criteria Evaluation Method in PLM Environment: A Pilot Study. 15th IFIP International Conference on Product Lifecycle Management (PLM), Jul 2018, Turin, Italy. pp.534-542, 10.1007/978-3-030-01614-2_49. hal-02075617

HAL Id: hal-02075617

https://inria.hal.science/hal-02075617

Submitted on 21 Mar 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Multicriteria evaluation method in PLM environment: a
pilot study

Kevin Audoux¹, Floriane Laverne¹, Gianluca D'Antonio² Frédéric Segonds¹, Olivier Kerbrat³, Paolo Chiabert², Ameziane Aoussat¹

¹ LCPI / Arts et Métiers Paristech, 151 boulevard de l'Hopital 75015 Paris, France;

² Politecnico di Torino, C.so Duca degli Abruzzi 24, Torino 10129, Italy

³ Univ Rennes, ENS Rennes, CNRS, Gem - UMR 6183, F-35000 Rennes, France

e-mail :kevin.audoux@ensam.eu

**Abstract.** During the whole design process, evaluation stages are required and should consider more and more criteria but some of them are not mastered by designers. They also need support tools or methods to ease the product evaluation during the design process and avoid the subjectivity of this task, to limit the choice's risks and to allow the repeatability of the whole process. This article presents through an experiment the necessity of evaluation tools by comparing assisted and not-assisted evaluation of 9 products.

**Keywords:** Evaluation method, innovation, sustainable development

## 1 Introduction

To win the innovation race firms must set up innovation management evaluation during their design process to control the evolution of their products. This race reduces the time allowed for product development. This modification of the design requirements promotes the use of evaluation method during the design process. Each evaluation phase needs its own performance indicators and method to fit with the product's progress. As a performance criterion, the innovative characteristic of the product must be evaluated to validate the economic and functional viability.

However, the performance of the product depends on design and politic goals. More and more criteria of performance are added to suit the consumers' evolution and the policy evolution as the sustainable development.

Sustainable development is “development that meets the needs of the present without compromising the ability of future generations to meet their own needs” [1]. It is composed of three aspects: social, as the human being, environment, as the respect of the nature and economy as the industrial development. This approach is currently growing and is described as the next world problem is global warming as presented in the Climate change conference in 2015 [2] . Environmental impact reduction is sup-
ported by Policy [3] and is also promoted by consumers as a selection criteria [4]. By considering the sustainability impact of their product, firms can increase their profita-bility and improve the life cycle of their products by optimising the end of life of their product or reduce the consumption of the product manufacturing.

Innovation and sustainability are therefore promising opportunities for designing new products. The requirements relative to its themes must be integrated as soon as possible into the design process that means during the early design stages, to provide the market with innovative and sustainable products.

The motivation of this work is to expose the difficulty of multiple performances evaluation during the design process. The state of the art positioned the design meth-odologies in PLM approach and described the limits of evaluation tools for multiple performance assessment. An experiment is proposed to highlight the benefit of tools during the design process.

## 2 State of the art

Our study is focused on the evaluation stage of innovation and sustainability during the design process. The state of the art first aims to position the Product Lifecycle Management into the design process. In a second time, the integration of innovation and sustainability in the design process are investigated. In the third part of the state of the art, a description of evaluation tool typology is proposed.

### Early PLM and design methodologies

Design methodologies are widely used in our community. For engineering design practices in industry, engineers and managers are concerned with the results of design by following specific design process more rigorously to ensure that the final design result meets the objectives of the product design and development projects [5]. PLM is an approach to support this complexity, including the design methodology and the complete product life cycle [6], [7]. As PLM addresses the entire lifecycle of the product, it has a cross-functional nature and deals closely with the way a company runs [8]. With the development of Product Data Management (PDM), PLM and asso-ciated workflows, software firms have proposed solutions to the everyday problems of engineering design departments (versioning of documents, naming etc.). PLM aims to cover all the stages of product development, by integrating the processes and peo-ple taking part in the project [9]. Thus, PLM is an approach in which processes are just as important as data, or even more so. Thus, when dealing with PLM implementa-tion, the design process is naturally the starting backbone to define processes and associated workflows. Hence, if one wants to provide a successful PLM environment, it is mandatory to structure the methodologies that will be supported. Design process is also the backbone of an innovation process and early design stages, starting from the research of concepts to the delivery of a preliminary layout, are the key stages of the innovative design [10] as presented in Figure 1. If we focus on the early design
stages, current «PLM methodological backbones» are not evaluated by designers or

<figure><img src="image_2.png" /></figure>

end-users. This paper presents the benefit these kinds of tools could provide.

<caption>Figure 1: Design process and position of the Early Design Stages</caption>

**Innovation and sustainability in the design process**

The innovative aspect of a product is created during early design stages [11]. Two kinds of innovations are differentiated: incremental innovation which ameliorate a product and breakthrough innovation which corresponds to new products for new needs [12]. Saunders identifies characteristics of innovative products based on the study of popular products [13]. This study permits to identify innovation criteria for product development. The innovative aspect of a product can be managed by two approaches:

- An innovative design process: The whole process is modified and managed to increase the innovative characteristic of the product.

- An innovative evaluation of the project: each intermediate representation of the product is evaluated to control the innovative characteristic of the product.

Throughout the rest of the article, we proposed to study only the second approach, suitable for every type of design process.

As opposed to innovation, sustainability is not specific to early design stages. It can be evaluated during the whole process. The complexity of methods follows the evolution of the product during all design stages. First, method permits to identify the product scenario of use based on the environmental impact criteria [14], then select and improve the ideas [15], [16] also improve the product concept or architecture [17], [18] and then optimise the life cycle of the whole defined product [19].

Innovation and sustainability aspects of the product have to be evaluated and monitored during the design process. The different kinds of evaluation tool are described in the following part.

**Typology of evaluation tools in design process**

**2.1.1 Design evaluation**

Each successive stages of the design process is the alternation of generation and evaluation of intermediate representation of the product [20]. It means that firstly
designers must answer the specific problems of the product design and then select
with the appropriate criteria the best intermediate representations.

According to Blessing [21], criteria “are used to be able to focus the investigation of the existing situation; to assess the contribution of the findings of such investigations to the research goal; to focus the development of support on the most relevant factors; to plan the appropriate evaluation; to focus the realization of the support on this evaluation; and to assess the evaluation results.”

Two types are defined:

- Successful criteria: they represent the benefit generated by the placing on the market of the product

- Measurable success criteria: they represent the possible success of the product, innovative and sustainable indicators are used here for measuring the performance of the product.

### 2.1.2 Typology of evaluation tools

To quantify these measurable success criteria, a specific evaluation method is needed. Literature distinguishes two types of evaluation:

- Expert evaluation based on the knowledge of experts in the domain. This type of method is well used for innovation projects and relies on evaluation grids [22].

- Tools evaluation based on the quantification of product criteria. This kind of method needs tangible measurement and is used to determine environmental impact of intermediate representation with survey [15] or, matrix [23]. It is based on the measurement of objective criteria.

Evaluations are crucial during the design process and even more during the early design stage where the greater part of the engaged cost is defined. Innovation and sustainable design require experts. To reduce the resource dedicated to the evaluation stage and to draw on objective indicators, tools are the best compromise.

## 3 Evaluation needs during design process

In this section, an experiment made to investigate the impact of evaluation tools for designers is proposed. Twelve M.Sc. students attending courses on sustainable manufacturing have been involved: subjective and objective evaluation sessions are performed to understand whether the use of tools during evaluations is a benefit for designers or not. Secondly, a method for tool evaluation is proposed and tested.

### 3.1 Protocol

First, a set of products to be evaluated has been identified: nine heterogeneous products designed with particular concerns for the purpose of sustainability or innovation, or manufactured with innovative technologies (such as additive manufacturing) have been selected. The whole list of products is provided in Table 1.
The experiment is composed of two main steps and lasted 1h30 as described in Figure 2.

<figure><img src="image_3.png" /><figcaption>Figure 2: Planning for experiment</figcaption></figure>

In the first stage, each participant has to assess the sustainability and the innovation levels for each product without any support tool. Participants have to access a Google form and provide a subjective answer to the question “how much is the innovative aspect?” through a Likert scale (1-5): 1 is for bad and 5 is for good.

After this evaluation, the same set of products is evaluated by using two appropriate tools for sustainability [15] and innovation [13]. These tools were adapted in a Google form.

Then, the results have been collected and compared to identify whether differenced among the two evaluations can be found.
<table><caption>Table 1: Product Description</caption><thead><tr><td>Product</td><td>Description</td><td>Categories</td></tr></thead><tbody><tr><td>1</td><td>Single-blade electric razor used like classic one</td><td>Innovation</td></tr><tr><td>2</td><td>Optimised leg prosthesis for runners</td><td>Innovation</td></tr><tr><td>3</td><td>Electric car</td><td>Sustainability</td></tr><tr><td>4</td><td>Desk made with carton</td><td>Sustainability</td></tr><tr><td>5</td><td>Glasses manufactured by additive manufacturing</td><td>Innovation</td></tr><tr><td>6</td><td>Neckless manufactured by additive manufacturing</td><td>Innovation</td></tr><tr><td>7</td><td>Bicycle saddle/lock</td><td>Innovation</td></tr><tr><td>8</td><td>Edible water bubble</td><td>Sustainability</td></tr><tr><td>9</td><td>Plate made with leaf</td><td>Sustainability</td></tr></tbody></table>

## 3.2 Results

This pilot study needs the use of non-parametric methods due to the sample size. For each product, we analyse the distribution and the difference between the two kinds of evaluation by using the following method:

- Kolmogorov-Smirnov method is used for non-parametric sample. It compared each sample with a normal distribution. The results of each product are shown on Table 2. This method provide four criteria:

* the mean value of the evaluation

* the standard deviation

* the coefficient of deviation

* the signification that describe if the normal distribution is validated.

- Wilcoxon method permits to identify if the evolution between two samples of data are correlated. The results are shown on Table 3.
<table><caption>Table 2 Kolmogorov-Smirnov results</caption><thead><tr><th></th><th>Tools</th><th>Product</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th></tr></thead><tbody><tr><td rowspan="10">sustainability</td><td rowspan="4">with</td><td>mean</td><td>1,61</td><td>2,75</td><td>3,33</td><td>3,33</td><td>2,58</td><td>2,81</td><td>2,17</td><td>3,75</td><td>3,89</td></tr><tr><td>standard deviation</td><td>0,45</td><td>0,81</td><td>0,90</td><td>0,42</td><td>0,62</td><td>0,40</td><td>1,04</td><td>0,42</td><td>0,64</td></tr><tr><td>coefficient of variation</td><td>28,0%</td><td>29,5%</td><td>27,0%</td><td>12,6%</td><td>24,0%</td><td>14,2%</td><td>47,9%</td><td>11,2%</td><td>16,5%</td></tr><tr><td>signification</td><td>0,70</td><td>0,97</td><td>0,96</td><td>1,00</td><td>0,92</td><td>0,91</td><td>0,98</td><td>1,00</td><td>0,99</td></tr><tr><td rowspan="4">without</td><td>mean</td><td>2,58</td><td>3,00</td><td>4,18</td><td>3,58</td><td>3,07</td><td>2,57</td><td>3,08</td><td>4,73</td><td>4,75</td></tr><tr><td>standard deviation</td><td>1,24</td><td>0,60</td><td>0,81</td><td>1,16</td><td>1,00</td><td>1,28</td><td>0,96</td><td>0,59</td><td>0,58</td></tr><tr><td>coefficient of variation</td><td>48,1%</td><td>20,0%</td><td>19,4%</td><td>32,4%</td><td>32,6%</td><td>49,8%</td><td>31,2%</td><td>12,5%</td><td>12,2%</td></tr><tr><td>signification</td><td>0,69</td><td>0,14</td><td>0,21</td><td>0,32</td><td>0,33</td><td>0,62</td><td>0,46</td><td>0,00</td><td>0,00</td></tr><tr><td rowspan="10">innovation</td><td rowspan="4">with</td><td>mean</td><td>1,48</td><td>2,11</td><td>2,11</td><td>2,02</td><td>1,11</td><td>1,03</td><td>1,70</td><td>2,05</td><td>2,02</td></tr><tr><td>standard deviation</td><td>0,50</td><td>0,48</td><td>0,61</td><td>0,83</td><td>0,61</td><td>0,53</td><td>0,47</td><td>0,68</td><td>0,65</td></tr><tr><td>coefficient of variation</td><td>33,8%</td><td>22,7%</td><td>28,9%</td><td>41,1%</td><td>55,0%</td><td>51,5%</td><td>27,6%</td><td>33,2%</td><td>32,2%</td></tr><tr><td>signification</td><td>0,83</td><td>0,55</td><td>0,94</td><td>0,76</td><td>0,97</td><td>0,94</td><td>0,37</td><td>0,98</td><td>0,70</td></tr><tr><td rowspan="4">without</td><td>mean</td><td>2,75</td><td>4,25</td><td>3,88</td><td>3,50</td><td>2,29</td><td>2,36</td><td>4,08</td><td>4,73</td><td>3,56</td></tr><tr><td>standard deviation</td><td>1,29</td><td>1,06</td><td>0,86</td><td>1,45</td><td>0,99</td><td>1,01</td><td>0,76</td><td>0,46</td><td>1,26</td></tr><tr><td>coefficient of variation</td><td>46,9%</td><td>24,9%</td><td>22,2%</td><td>41,4%</td><td>43,2%</td><td>42,8%</td><td>18,6%</td><td>9,7%</td><td>35,4%</td></tr><tr><td>signification</td><td>0,41</td><td>0,12</td><td>0,20</td><td>0,22</td><td>0,32</td><td>0,57</td><td>0,48</td><td>0,04</td><td>0,65</td></tr></tbody></table>

Most of the results on Table 2 show that the hypothesis of a normal distribution is never rejected when the tool is used (signification value higher than 0.05) for both sustainability and innovation. When the tool is not used, the hypothesis of normal distribution is not accepted in three cases only (see the results highlighted in orange). In 14 out of the 15 remaining evaluation pairs, exhibit a reduced sustainability or innovation average score: therefore, if the participants are not guided during the evaluation process, they tend to overestimate products' performances. Further, in 12 out of 15 cases the overall standard deviation is smaller, although a clear pattern for the coefficient of variation (given by the ratio between the standard deviation and the mean value) cannot be identified: so, the overall evaluation range is reduced but, since also the average values are usually lower, the relative dispersion may increase.

For the case study, final products are evaluated, the use of tools permit to standardize and homogenize this step.

<table><caption>Table 3 Wilcoxon Test</caption><thead><tr><th rowspan="2">Product</th><th colspan="9">signification</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th></tr></thead><tbody><tr><td>sustainability</td><td>0,31</td><td>0,498</td><td>0,465</td><td>1</td><td>0,686</td><td>0,269</td><td>0,416</td><td>0,109</td><td>0,109</td></tr><tr><td>innovation</td><td>0,023</td><td>0,002</td><td>0,003</td><td>0,018</td><td>0,021</td><td>0,013</td><td>0,003</td><td>0,003</td><td>0,007</td></tr></tbody></table>
The results of Table 3 show the results of Wilcoxon test. It compared two linked-samples and test whether their evolutions are constant. If the signification is under 0.05, the hypothesis is invalidated. For innovation evaluation, the hypothesis is rejected that means that participants change significantly their point of view between the two evaluations. This result is in accordance with the Table 2, the range is clearly different with and without tools. This test shows that innovation is clearly a subjective characteristic and it is difficult to evaluate without tools.

## 3.3 Analysis and Conclusion of the experiment

This experiment shows that it is difficult to evaluate products regarding to different subjects. The Table 2 shows how much participants overestimate the performance of product. Moreover, it permits to identify that tools reduce the deviations. The Table 3 permits to understand how evolve the evaluation between the two steps. For sustainability aspect, the participants overestimate the performances but in average follow the same distribution with or without tools. For innovative aspect, the subjectivity of this characteristic can explain why the participants change drastically their evaluation.

This experiment was made with final products which are easier than during the design process. The less the product is developed the better the benefit of evaluation tool is.

These results show the need of assistance for evaluation stages. The next part of this article aims to validate a method of classification for evaluation method.

One limit of this experiment could be the reduce number of participants.

## Conclusion

This article proves through the experiment the necessity of evaluation tool during design by the comparison of two evaluations: an evaluation subjective and an evaluation with tools. This experiment was performed by engineering students and highlighted that tangible constraint as the sustainability is more valuable than subjective characteristic as innovation. Evaluation tools permit for both types of feature to reduce the uncertainty. Assistance makes repeatable and reliable evaluation stages.

The choice of evaluation tool depends of different condition link to the design process or designers' knowledge or product goals, and makes the selection more and more difficult. The second experiment is a validation of an evaluation method of tools. This method is using unlabelled criteria to fit with each disciplinary found in the design process. Engineering students had to make their own evaluation of scientific papers by following the proposed method.
References

[1] G. H. Brundtland, "Rapport Brundtland," Ministère Aff. Étrangères Dév. Int. L'Odyssée Dév. Durable, 1987.

[2] "The Paris Agreement - main page." [Online]. Available: http://unfccc.int/paris_agreement/items/9485.php. [Accessed: 25-Jan-2018].

[3] J. Rosiek, "THE IMPACT OF THE EU CLIMATE POLICY ON GREEN JOBS CREATION," *Econ. Manag.*, vol. 18, no. 4, pp. 697-714, 2013.

[4] W. Young, K. Hwang, S. McDonald, and C. J. Oates, "Sustainable consumption: green consumer behaviour when purchasing products," *Sustain. Dev.*, p. n/a-n/a, 2009.

[5] T. Tomiyama, P. Gu, Y. Jin, D. Lutters, C. Kind, and F. Kimura, "Design methodologies: Industrial and educational applications," *CIRP Ann. - Manuf. Technol.*, vol. 58, no. 2, pp. 543-565, 2009.

[6] D. H. Bergsjö, "Management of Mechatronic Product Data in PLM Systems," Chalmers University of Technology, Sweden, 2007.

[7] U. Sellgren, M. Törngren, D. Malvius, and M. Biehl, *PLM for Mechatronics integration*. 2018.

[8] M. Garetti, S. Terzi, N. Bertacci, and M. Brianza, "Organisational change and knowledge management in PLM implementation," *Int. J. Prod. Lifecycle Manag.*, vol. 1, no. 1, pp. 43-51, Jan. 2005.

[9] G. Schuh, H. Rozenfeld, D. Assmus, and E. Zancul, "Process oriented framework to support PLM implementation," *Comput. Ind.*, vol. 59, no. 2-3, pp. 210-218, Mar. 2008.

[10] F. Laverne, F. Segonds, G. D'Antonio, and M. Le Coq, "Enriching design with X through tailored additive manufacturing knowledge: a methodological proposal," *Int. J. Interact. Des. Manuf.*, vol. 11, no. 2, pp. 279-288, 2017.

[11] J. Perrin, *Concevoir l'innovation industrielle: Méthodologie de conception de l'innovation*. CNRS, 2001.

[12] D. Encaoua, D. Foray, A. Hatchuel, and J. Mairesse, "Les enjeux économiques de l'innovation, The economics of innovation: trends and issues," *Rev. Déconomie Polit.*, vol. 114, no. 2, pp. 133-168, 2004.

[13] M. N. Saunders, C. C. Seepersad, and K. Hölttä-Otto, "The characteristics of innovative, mechanical products," *J. Mech. Des.*, vol. 133, no. 2, pp. 1-9, 2011.

[14] F. Cluzel, B. Yannou, D. Millet, and Y. Leroy, "Eco-ideation and eco-selection of R&D projects portfolio in complex systems industries," *J. Clean. Prod.*, vol. 112, pp. 4329-4343, 2016.

[15] N. M. P. Bocken, J. M. Allwood, A. R. Willey, and J. M. H. King, "Development of an eco-ideation tool to identify stepwise greenhouse gas emissions reduction options for consumer goods," *J. Clean. Prod.*, vol. 19, no. 12, pp. 1279-1287, 2011.

[16] J. A. O'Hare, "Eco-innovation tools for the early stages: an industry-based investigation of tool customisation and introduction," University of Bath, 2010.
[17] M. F. Hassan, M. Z. M. Saman, S. Sharif, and B. Omar, "An Integrated MA-AHP Approach for Selecting the Highest Sustainability Index of a New Product," *Procedia - Soc. Behav. Sci.*, vol. 57, pp. 236-242, 2012.

[18] A. Halog, F. Schultmann, and O. Rentz, "Using quality function deployment for technique selection for optimum environmental performance improvement," *J. Clean. Prod.*, vol. 9, no. 5, pp. 387-394, 2001.

[19] A. M. Contreras, E. Rosa, M. Pérez, H. Van Langenhove, and J. Dewulf, "Comparative Life Cycle Assessment of four alternatives for using by-products of cane sugar production," *J. Clean. Prod.*, vol. 17, no. 8, pp. 772-779, 2009.

[20] T. J. Howard, S. J. Culley, and E. Dekoninck, "Describing the creative design process by the integration of engineering design and cognitive psychology literature," *Des. Stud.*, vol. 29, no. 2, pp. 160-180, 2008.

[21] L. T. M. Blessing and A. Chakrabarti, DRM, a Design Research Methodology. London: Springer London, 2009.

[22] B. Yannou, B. Zimmer, R. Farel, M. Jankovic, and J. Stal Le Cardinal, "Proofs of utility, innovation, profitability and concept for innovation selection," *ICED Seoul Korea*, 2013.

[23] M. Binz and H. Reichle, "Evaluation method to determine the success potential and the degree of innovation of technical product ideas and products," *35 Proc. ICED 05 15th Int. Conf. Eng. Des. Melb. Aust. 15-18082005*, 2005.