# Product Design Evaluation Method Using Consensus Measurement, Network Analysis, and AHP

Authors: Yan-pu Yang; Hong-liang Tian; Sheng-jie Jiao
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

Product evaluation practices spread throughout from initial creative stages to final products before communicating them to the market. They emphasize on assessing design alternatives against specified criteria, which can help promote design process, ensure design quality, and diminish design risk before making a decision. However, how to identify and improve the reliability of product evaluation opinions toward design alternatives has an important role in adding additional insurance and reducing uncertainty to successful product design. Aiming at this issue, this study employs a consensus model by integrating network analysis and analytic hierarchy process (AHP) method. The consensus model is constructed to measure the consistency of evaluators’ preferences and determine the rounds of evaluation. Complex network theory is integrated into the product evaluation process for network analysis of evaluators’ opinions, which can help determine evaluators’ weights in each evaluation round dynamically by changing the network topology. To obtain the weight of product evaluation indices, the AHP method is applied to avoid subjectivity given by evaluators. The process of the proposed method is presented, and the details are illustrated using a product evaluation example. The case study demonstrates that the proposed method is promising for improving the consensus level of evaluators’ opinions, reducing the influence of subjectivity, and finally improving the quality of design decision-making.


---

## Body

## 1. Introduction

As the international market globalizing at high speed, enterprises have been facing a fierce competition and are propelled to concentrate on product innovation to maintain a competitive advantage. Companies who can develop new products to meet market demand will probably succeed in the marketplace. A normative process of product design involves concept generation, evaluation, and selection [1]. As a common path followed by designers, it consists of several fundamental activities: identifying problem and requirements, finding design solutions, assessing and recording, and communicating results [2]. Among them, evaluative practices extend from initial creative stages to final products prior to communicating them to the market [3, 4]. Evaluation emphasizes on assessing design alternatives against specified criteria (e.g., aesthetics, functionality, and originality [5]) and can help promote design process and ensure design quality before making a decision [6]. Decisions made in the concept development phase are critical for the success of product development and largely determine the quality, cost, and desirability of the end product [7]. It is commonly believed that the majority cost of total product development is determined by design decisions [8]. Hence, invalid design evaluation and decision would incur unnecessary cost and time to redesign and compensate in the later phases of product development [9].An effective path for product evaluation that can help promote the success of final design concept is to form a design decision-making group to collect their opinions, which usually contains consumers, industrial designers, engineering designers, and marketers. Consumers may have unique knowledge in product usability and accessibility, while industrial designers mainly focus on aesthetics, structure, and ergonomic efficiency of a product. Engineer designers pay more attention to the realization and manufacturability of target product. As to marketers, the competitive advantage, possible sales, and prospect are their focus. Due to different concerns as well as knowledge, experience, and background gaps, it is common that an evaluator make a decision not only depending on his/her perception but also communicating with other members and referring to their opinions, which may make them change their assessment and not easy to reach an agreement. Inconsistency in evaluators’ opinions will not only affect the reliability of evaluation results but also improve the risk in product development. With the advantages of absorbing collective wisdom and beneficial opinions and promoting consensus, group decision-making (GDM) has been receiving increasing amount of research interest. For instance, Morente-Molinera et al. [10] utilized group decision-making and fuzzy ontologies to create a knowledge database for knowledge sharing and retrieving. Lourenzutti and Krohling [11] introduced the technique for order preference by similarity to ideal solution (TOPSIS) method to deal with heterogeneous information in a dynamic environment. Zhang et al. [12] applied incomplete reciprocal preference relations to measure the consistency of preferences provided by experts. Li et al. [13] proposed to use a weighted-power average operator to integrate heterogeneous information. To sum up, the normative resolution methods of GDM involve two different processes: the consensus process and selection process [14]. Moreover, the GDM problems can be defined in three heterogeneous frameworks [15]: different preference representation formats, diverse expression domains, and various backgrounds and levels of knowledge about the problem. Since heterogeneity is ubiquitous in many decision-making problems, GDM has been used satisfactorily and yielded valuable achievements in the fields of soft computing [16–18], social psychology [19, 20], artificial intelligence [21–23], and so on.Consensus is one of the significant issues widely considered in GDM problems which assures the unity of evaluators’ opinions and makes the result of decision-making acceptable to the majority [24]. When dealing with consensus issues in product evaluation process, it often involves limited sets of design solutions, evaluators, and indices. As evaluators give their judgment about design alternatives against specified criteria, it is essential to determine the weight of each evaluator and criteria. The two factors will determine the consensus and the final decision outcomes, which should be calculated objectively rather than subjective assignment. Furthermore, an appropriate consensus measurement model is essential to analyze the consensus level of evaluators’ judgment and determine how many rounds of product evaluation will be needed.Aiming to improve the opinion consistency among product design evaluators, ensure design quality, and reduce the risk of failure in product development, a consensus model is constructed by integrating network analysis and analytic hierarchy process (AHP) method. The network analysis method is introduced from the perspective of opinion dynamics [25], which applies complex network theory to treat each evaluator in the product evaluation process as a node and provides cohesion analysis according to the judgment given by evaluators about product alternatives. The interrelationship among nodes will change with the network evolving, which indicates that evaluators’ opinions will become aggregated to consistency or split. Each node can interact with other nodes and be affected to alter their status. Through removing a node and calculating network cohesion, evaluators’ weights can be obtained dynamically in each round. The AHP method is applied to compute the weight of indicators, which can help avoid the subjectivity given by evaluators. Through multistage network analysis, evaluators’ opinions can be aggregated considering their importance, can reduce the influence of subjectivity, and can finally improve the quality of design decision-making.The rest of the paper is organized as follows: Section2 interprets the requisite materials and methods. In Section 3, the method is verified using an example. Sections 4 and 5 provide final remarks.


## 2. Materials and Methods

### 2.1. Problem Statement

LetP=p1,p2,…,pmm≥2 be a set of evaluators who will take part in the product evaluation process. The weight vector is λ=λ1,λ2,…,λmT with λi≥0, i=1,2,…,m and ∑λi=1. The evaluators are invited to participate in product evaluation about a set of product design alternatives X=x1,x2,…,xtt≥2. Assume that the set of product evaluation indicators is C=c1,c2,…,cnn≥2 and the weight vector is W=w1,w2,…,wnT with wk≥0, k=1,2,…,m, and ∑wk=1. In evaluating process, product alternatives are judged using an 11-point (0–10) Likert scale [26] according to their performance on various indicators.The interactive evaluative process of product design can be described as follows: in each evaluation round, all evaluators give their opinions on each alternative and share their reviews. In roundt, evaluator ei adjusts opinion in round t-1 and gives the score of a product alternative based on his/her perception and other people’s evaluations. Evaluators involve consumers, designers, engineers, marketers, stakeholders, and so on. To avoid possible influence of subjectivity and authoritarianism, the weight vectors of evaluators and indicators will be determined by network analysis and AHP method, which will be discussed in the later section, respectively.


### 2.2. Methods

Aiming at dealing with evaluators’ dynamic opinions effectively, it is required to gauge the agreement achieved among the evaluators, analyze the interactive exchange of views, and clarify how the interactive network influence the design decision-making process. Later in this article, we will briefly describe three methods that are of interest to our study: consensus model, network analysis, and AHP.


#### 2.2.1. Consensus Model

Consensus means “most of” the individuals in a group agreeing to “almost all of” the options [27]. In the decision-making process of industrial product design, consensus reflects individuals’ cognitive consistency levels about product alternatives. A high consensus state implies low inconsistency and disagreement. The consensus methodology has been proved to be useful for solving conflicts and effective for knowledge integration [28]. One of the most significant discussions about the consensus method is consensus measurement and achievement. Measuring the consensus degree will help check the consistency of individuals’ opinions and avoid self-contradiction by employing a consensus model. Up to now, most studies concerning on the consensus model for consensus measurement have focused on the use of distance/similarity function based on measures and association measures [27]. Our proposal is based on distance computation to measure inconsistency. Through the consensus model, inconsistency improvement procedures can be implemented to aid decision-makers to elevate consensus level through intelligent optimization algorithms [29] and group discussion [30].Let vectorAij=a1ij,a2ij,…,anij represent the set of scoring of xj by pi according to C, and then we can get an evaluation matrix about xj, Aj=A1j,A2j,…,Amj. Normalization is computed as follows:(1)akij′=akij−min1≤k≤nmin1≤i≤makijmax1≤k≤nmax1≤i≤makij−min1≤k≤nmin1≤i≤makij.Euclidean distance is used to calculate the opinion difference between evaluatorh and evaluator l:(2)DAhj′,Alj′=∑k=1nwkakhj′−aklj′2.Then, the similarity of opinions by evaluatorh and evaluator l is(3)SAhj′,Alj′=1−DAhj′,Alj′.Consensus measurement of evaluators’ opinionsxj can be depicted by the following equation:(4)CONj=∑h=1m∑l=1mμh,l∑h=1m∑l=1mλhl,where μh,l=λhl,SAhj′,Alj′≥δ0,SAhj′,Alj′<δ and λhl is used for evaluating the joint weights of h and l with λhl=λh+λl/2. Specially, if h = l, then λhl=λh=λl=μl,l. δ represents the threshold of consensus degree, and exceeding this threshold implies reaching agreement, otherwise no agreement can be reached.Finally, the total score ofxj can be computed by(5)Pj=λAjWT,where λ, Aj, and WT represent evaluators’ weight vector, assessment matrices, and indicators’ weight vector, respectively.


#### 2.2.2. Network Analysis

Network analysis refers to employing graph theory and statistical physics to study the topological structure and properties of various networks [31, 32]. With graph theory, a network can be described as G=V,E with N=V nodes (vertices) and E connecting edges. The adjacency matrix is used to describe how the nodes are connected. In a connected undirected graph, the value of each element in the adjacency matrix equals to 1 if two nodes are connected and 0 otherwise. While in a directed network, all the edges are directed from one vertex to another. Furthermore, a weighted network can be created if the ties among nodes have weights assigned to them. Based on this topology structure, some special networks can be defined and analyzed effectively, especially in the case of complex networks such as social network, electric network, and biological network. The invention of small-world networks (SWN) [33] and scale-free networks (SFN) [34] has facilitated the development of complex network theory, which has promoted researchers to further study the application in industrial product design. Relevant researches mainly focus on product knowledge pushing [32], product family design [35, 36], information flows across product development stages [37], product competition research [38], and product lifecycle representation [39].Product evaluation process usually contains several interactive and dynamic subprocesses. Each subprocess reflects opinion interaction of evaluators. Using complex network theory to analyze is based on four hypotheses: (1) the information about the evaluation objects owned by the evaluators is symmetric; (2) evaluators have the independence to give preference; (3) evaluators will communicate with each other if the consensus level is not reached; and (4) the weight of each evaluator is calculated according to the similarity of their judgment. An example of an evaluation network with 5 evaluators and 2 subprocesses is shown in Figure1. It can be seen that the relationships among nodes change with evaluation proceeding, which implies that evaluators alter their assessment through interaction and discussion.(1)
Similarity weight: in network analysis, nodes are participants and the weights of edges reflect similarity weights of every two participants. Product evaluation network can be built as a complete graph where all evaluators will give their opinions. The similarity weights of every two evaluators are computed according to the Euclidean distance of their judgment with Equations (1)∼(3).
(2)
Network cohesion: from a structural standpoint, cohesion refers to the general level of connectedness of a network [40]. As design evaluation proceeds, the network cohesion will change in line with evaluators’ opinion convergence. It can be computed as follows:(6)
∂
WG
=
S
G
l
G
,Figure 1
An example of an evaluation network with 5 evaluators and 2 subprocesses.whereSG is the sum of the average weighted value of nodes and IG is the average distance of the weighted evaluation network. The weight of each node can be computed by the sum of similarity weights of other nodes:(7)λj′=∑i∈NjSAhj′,Alj′,where Nj is the neighbouring nodes of j; i,j∈1,m, and i≠j. Here we construct a complete graph network on hypothesis (3) in Section 2.2.2. Each node of the product evaluation network has the same adjacent nodes and node degree equal to m−1. The sum of the average weighted value of nodes can be computed as(8)SG=∑j=1mλj′Nj.The distance of two nodes can be described by their similarity weight. The larger the similarity weight is, the smaller the distance will be. The average distance of the weighted evaluation network can be depicted using Euclidean distance with the following equation:(9)lG=1mm−1∑h,l=1mDAhj′,Alj′,h≠l.To further analyze the importance of each node in product evaluation network, the consistency of group evaluation is computed by removing the evaluation information of an evaluator, which reflects the degree of node clustering in the network. Node importance can be used to represent individual influence. The more influential the node, the more likely it is to change others’ opinions in the communicating process, and the shorter the distance between the corresponding nodes. Therefore, the cohesion of the network will become smaller if the removed node has positive effect and large importance, and vice versa. The cohesion change can be described as(10)IMCpj=∂WG∂WG⋅pj+1m−2SG⋅pj,if∂WG⋅pj<∂WG,∂WG∂WG⋅pj,else,where ∂WG⋅pj and SG⋅pj represent recomputing network cohesion and the average weighted value, respectively, when node pj1≤j≤m in an evaluation round is removed.Finally, we can calculate the weights of the evaluators:(11)λj=IMCpj∑j=1mIMCpj.


#### 2.2.3. Analytic Hierarchy Process

The analytic hierarchy process (AHP) is a structured technique founded by Saaty [41, 42] for analyzing complex decisions, which has been used in many applications related to decision-making problems [43, 44]. In our work, we employ this method to obtain the weights of indicators. Matrix B=bijr×r is used to describe a set of pairwise judgments where bij>0 and bij×bji=1. Through Saaty’s 9-point scale, each bij can be assigned 1, 3, 5, 7, and 9 or their inverse, which qualitatively expresses preferences among options as equally, moderately, strong, very strongly, or absolutely important. Then the weights of indicators can be calculated as follows:(12)wi=∏j=1nbij1/n∑i=1n∏j=1nbij1/n.As a consistency check, a consistency index (CI) has also been proposed and defined by the following relation:(13)CI=λmax−nn−1,where λmax is the largest or principal eigenvalue of the matrix B and λmax=∑Ac⋅WT/n⋅W. By introducing the random index (RI) (seen in Table 1), the consistency ratio (CR) is computed as CR=CI/RI. Saaty proposed that if the value of CR is less than 0.1, then the matrix can be considered as having an acceptable consistency.Table 1
The random consistency indices.
n
1
2
3
4
5
6
7
8
9
10
RI
0
0
0.52
0.89
1.11
1.25
1.35
1.40
1.45
1.49


## 2.1. Problem Statement

LetP=p1,p2,…,pmm≥2 be a set of evaluators who will take part in the product evaluation process. The weight vector is λ=λ1,λ2,…,λmT with λi≥0, i=1,2,…,m and ∑λi=1. The evaluators are invited to participate in product evaluation about a set of product design alternatives X=x1,x2,…,xtt≥2. Assume that the set of product evaluation indicators is C=c1,c2,…,cnn≥2 and the weight vector is W=w1,w2,…,wnT with wk≥0, k=1,2,…,m, and ∑wk=1. In evaluating process, product alternatives are judged using an 11-point (0–10) Likert scale [26] according to their performance on various indicators.The interactive evaluative process of product design can be described as follows: in each evaluation round, all evaluators give their opinions on each alternative and share their reviews. In roundt, evaluator ei adjusts opinion in round t-1 and gives the score of a product alternative based on his/her perception and other people’s evaluations. Evaluators involve consumers, designers, engineers, marketers, stakeholders, and so on. To avoid possible influence of subjectivity and authoritarianism, the weight vectors of evaluators and indicators will be determined by network analysis and AHP method, which will be discussed in the later section, respectively.


## 2.2. Methods

Aiming at dealing with evaluators’ dynamic opinions effectively, it is required to gauge the agreement achieved among the evaluators, analyze the interactive exchange of views, and clarify how the interactive network influence the design decision-making process. Later in this article, we will briefly describe three methods that are of interest to our study: consensus model, network analysis, and AHP.


### 2.2.1. Consensus Model

Consensus means “most of” the individuals in a group agreeing to “almost all of” the options [27]. In the decision-making process of industrial product design, consensus reflects individuals’ cognitive consistency levels about product alternatives. A high consensus state implies low inconsistency and disagreement. The consensus methodology has been proved to be useful for solving conflicts and effective for knowledge integration [28]. One of the most significant discussions about the consensus method is consensus measurement and achievement. Measuring the consensus degree will help check the consistency of individuals’ opinions and avoid self-contradiction by employing a consensus model. Up to now, most studies concerning on the consensus model for consensus measurement have focused on the use of distance/similarity function based on measures and association measures [27]. Our proposal is based on distance computation to measure inconsistency. Through the consensus model, inconsistency improvement procedures can be implemented to aid decision-makers to elevate consensus level through intelligent optimization algorithms [29] and group discussion [30].Let vectorAij=a1ij,a2ij,…,anij represent the set of scoring of xj by pi according to C, and then we can get an evaluation matrix about xj, Aj=A1j,A2j,…,Amj. Normalization is computed as follows:(1)akij′=akij−min1≤k≤nmin1≤i≤makijmax1≤k≤nmax1≤i≤makij−min1≤k≤nmin1≤i≤makij.Euclidean distance is used to calculate the opinion difference between evaluatorh and evaluator l:(2)DAhj′,Alj′=∑k=1nwkakhj′−aklj′2.Then, the similarity of opinions by evaluatorh and evaluator l is(3)SAhj′,Alj′=1−DAhj′,Alj′.Consensus measurement of evaluators’ opinionsxj can be depicted by the following equation:(4)CONj=∑h=1m∑l=1mμh,l∑h=1m∑l=1mλhl,where μh,l=λhl,SAhj′,Alj′≥δ0,SAhj′,Alj′<δ and λhl is used for evaluating the joint weights of h and l with λhl=λh+λl/2. Specially, if h = l, then λhl=λh=λl=μl,l. δ represents the threshold of consensus degree, and exceeding this threshold implies reaching agreement, otherwise no agreement can be reached.Finally, the total score ofxj can be computed by(5)Pj=λAjWT,where λ, Aj, and WT represent evaluators’ weight vector, assessment matrices, and indicators’ weight vector, respectively.


### 2.2.2. Network Analysis

Network analysis refers to employing graph theory and statistical physics to study the topological structure and properties of various networks [31, 32]. With graph theory, a network can be described as G=V,E with N=V nodes (vertices) and E connecting edges. The adjacency matrix is used to describe how the nodes are connected. In a connected undirected graph, the value of each element in the adjacency matrix equals to 1 if two nodes are connected and 0 otherwise. While in a directed network, all the edges are directed from one vertex to another. Furthermore, a weighted network can be created if the ties among nodes have weights assigned to them. Based on this topology structure, some special networks can be defined and analyzed effectively, especially in the case of complex networks such as social network, electric network, and biological network. The invention of small-world networks (SWN) [33] and scale-free networks (SFN) [34] has facilitated the development of complex network theory, which has promoted researchers to further study the application in industrial product design. Relevant researches mainly focus on product knowledge pushing [32], product family design [35, 36], information flows across product development stages [37], product competition research [38], and product lifecycle representation [39].Product evaluation process usually contains several interactive and dynamic subprocesses. Each subprocess reflects opinion interaction of evaluators. Using complex network theory to analyze is based on four hypotheses: (1) the information about the evaluation objects owned by the evaluators is symmetric; (2) evaluators have the independence to give preference; (3) evaluators will communicate with each other if the consensus level is not reached; and (4) the weight of each evaluator is calculated according to the similarity of their judgment. An example of an evaluation network with 5 evaluators and 2 subprocesses is shown in Figure1. It can be seen that the relationships among nodes change with evaluation proceeding, which implies that evaluators alter their assessment through interaction and discussion.(1)
Similarity weight: in network analysis, nodes are participants and the weights of edges reflect similarity weights of every two participants. Product evaluation network can be built as a complete graph where all evaluators will give their opinions. The similarity weights of every two evaluators are computed according to the Euclidean distance of their judgment with Equations (1)∼(3).
(2)
Network cohesion: from a structural standpoint, cohesion refers to the general level of connectedness of a network [40]. As design evaluation proceeds, the network cohesion will change in line with evaluators’ opinion convergence. It can be computed as follows:(6)
∂
WG
=
S
G
l
G
,Figure 1
An example of an evaluation network with 5 evaluators and 2 subprocesses.whereSG is the sum of the average weighted value of nodes and IG is the average distance of the weighted evaluation network. The weight of each node can be computed by the sum of similarity weights of other nodes:(7)λj′=∑i∈NjSAhj′,Alj′,where Nj is the neighbouring nodes of j; i,j∈1,m, and i≠j. Here we construct a complete graph network on hypothesis (3) in Section 2.2.2. Each node of the product evaluation network has the same adjacent nodes and node degree equal to m−1. The sum of the average weighted value of nodes can be computed as(8)SG=∑j=1mλj′Nj.The distance of two nodes can be described by their similarity weight. The larger the similarity weight is, the smaller the distance will be. The average distance of the weighted evaluation network can be depicted using Euclidean distance with the following equation:(9)lG=1mm−1∑h,l=1mDAhj′,Alj′,h≠l.To further analyze the importance of each node in product evaluation network, the consistency of group evaluation is computed by removing the evaluation information of an evaluator, which reflects the degree of node clustering in the network. Node importance can be used to represent individual influence. The more influential the node, the more likely it is to change others’ opinions in the communicating process, and the shorter the distance between the corresponding nodes. Therefore, the cohesion of the network will become smaller if the removed node has positive effect and large importance, and vice versa. The cohesion change can be described as(10)IMCpj=∂WG∂WG⋅pj+1m−2SG⋅pj,if∂WG⋅pj<∂WG,∂WG∂WG⋅pj,else,where ∂WG⋅pj and SG⋅pj represent recomputing network cohesion and the average weighted value, respectively, when node pj1≤j≤m in an evaluation round is removed.Finally, we can calculate the weights of the evaluators:(11)λj=IMCpj∑j=1mIMCpj.


### 2.2.3. Analytic Hierarchy Process

The analytic hierarchy process (AHP) is a structured technique founded by Saaty [41, 42] for analyzing complex decisions, which has been used in many applications related to decision-making problems [43, 44]. In our work, we employ this method to obtain the weights of indicators. Matrix B=bijr×r is used to describe a set of pairwise judgments where bij>0 and bij×bji=1. Through Saaty’s 9-point scale, each bij can be assigned 1, 3, 5, 7, and 9 or their inverse, which qualitatively expresses preferences among options as equally, moderately, strong, very strongly, or absolutely important. Then the weights of indicators can be calculated as follows:(12)wi=∏j=1nbij1/n∑i=1n∏j=1nbij1/n.As a consistency check, a consistency index (CI) has also been proposed and defined by the following relation:(13)CI=λmax−nn−1,where λmax is the largest or principal eigenvalue of the matrix B and λmax=∑Ac⋅WT/n⋅W. By introducing the random index (RI) (seen in Table 1), the consistency ratio (CR) is computed as CR=CI/RI. Saaty proposed that if the value of CR is less than 0.1, then the matrix can be considered as having an acceptable consistency.Table 1
The random consistency indices.
n
1
2
3
4
5
6
7
8
9
10
RI
0
0
0.52
0.89
1.11
1.25
1.35
1.40
1.45
1.49


## 2.2.1. Consensus Model

Consensus means “most of” the individuals in a group agreeing to “almost all of” the options [27]. In the decision-making process of industrial product design, consensus reflects individuals’ cognitive consistency levels about product alternatives. A high consensus state implies low inconsistency and disagreement. The consensus methodology has been proved to be useful for solving conflicts and effective for knowledge integration [28]. One of the most significant discussions about the consensus method is consensus measurement and achievement. Measuring the consensus degree will help check the consistency of individuals’ opinions and avoid self-contradiction by employing a consensus model. Up to now, most studies concerning on the consensus model for consensus measurement have focused on the use of distance/similarity function based on measures and association measures [27]. Our proposal is based on distance computation to measure inconsistency. Through the consensus model, inconsistency improvement procedures can be implemented to aid decision-makers to elevate consensus level through intelligent optimization algorithms [29] and group discussion [30].Let vectorAij=a1ij,a2ij,…,anij represent the set of scoring of xj by pi according to C, and then we can get an evaluation matrix about xj, Aj=A1j,A2j,…,Amj. Normalization is computed as follows:(1)akij′=akij−min1≤k≤nmin1≤i≤makijmax1≤k≤nmax1≤i≤makij−min1≤k≤nmin1≤i≤makij.Euclidean distance is used to calculate the opinion difference between evaluatorh and evaluator l:(2)DAhj′,Alj′=∑k=1nwkakhj′−aklj′2.Then, the similarity of opinions by evaluatorh and evaluator l is(3)SAhj′,Alj′=1−DAhj′,Alj′.Consensus measurement of evaluators’ opinionsxj can be depicted by the following equation:(4)CONj=∑h=1m∑l=1mμh,l∑h=1m∑l=1mλhl,where μh,l=λhl,SAhj′,Alj′≥δ0,SAhj′,Alj′<δ and λhl is used for evaluating the joint weights of h and l with λhl=λh+λl/2. Specially, if h = l, then λhl=λh=λl=μl,l. δ represents the threshold of consensus degree, and exceeding this threshold implies reaching agreement, otherwise no agreement can be reached.Finally, the total score ofxj can be computed by(5)Pj=λAjWT,where λ, Aj, and WT represent evaluators’ weight vector, assessment matrices, and indicators’ weight vector, respectively.


## 2.2.2. Network Analysis

Network analysis refers to employing graph theory and statistical physics to study the topological structure and properties of various networks [31, 32]. With graph theory, a network can be described as G=V,E with N=V nodes (vertices) and E connecting edges. The adjacency matrix is used to describe how the nodes are connected. In a connected undirected graph, the value of each element in the adjacency matrix equals to 1 if two nodes are connected and 0 otherwise. While in a directed network, all the edges are directed from one vertex to another. Furthermore, a weighted network can be created if the ties among nodes have weights assigned to them. Based on this topology structure, some special networks can be defined and analyzed effectively, especially in the case of complex networks such as social network, electric network, and biological network. The invention of small-world networks (SWN) [33] and scale-free networks (SFN) [34] has facilitated the development of complex network theory, which has promoted researchers to further study the application in industrial product design. Relevant researches mainly focus on product knowledge pushing [32], product family design [35, 36], information flows across product development stages [37], product competition research [38], and product lifecycle representation [39].Product evaluation process usually contains several interactive and dynamic subprocesses. Each subprocess reflects opinion interaction of evaluators. Using complex network theory to analyze is based on four hypotheses: (1) the information about the evaluation objects owned by the evaluators is symmetric; (2) evaluators have the independence to give preference; (3) evaluators will communicate with each other if the consensus level is not reached; and (4) the weight of each evaluator is calculated according to the similarity of their judgment. An example of an evaluation network with 5 evaluators and 2 subprocesses is shown in Figure1. It can be seen that the relationships among nodes change with evaluation proceeding, which implies that evaluators alter their assessment through interaction and discussion.(1)
Similarity weight: in network analysis, nodes are participants and the weights of edges reflect similarity weights of every two participants. Product evaluation network can be built as a complete graph where all evaluators will give their opinions. The similarity weights of every two evaluators are computed according to the Euclidean distance of their judgment with Equations (1)∼(3).
(2)
Network cohesion: from a structural standpoint, cohesion refers to the general level of connectedness of a network [40]. As design evaluation proceeds, the network cohesion will change in line with evaluators’ opinion convergence. It can be computed as follows:(6)
∂
WG
=
S
G
l
G
,Figure 1
An example of an evaluation network with 5 evaluators and 2 subprocesses.whereSG is the sum of the average weighted value of nodes and IG is the average distance of the weighted evaluation network. The weight of each node can be computed by the sum of similarity weights of other nodes:(7)λj′=∑i∈NjSAhj′,Alj′,where Nj is the neighbouring nodes of j; i,j∈1,m, and i≠j. Here we construct a complete graph network on hypothesis (3) in Section 2.2.2. Each node of the product evaluation network has the same adjacent nodes and node degree equal to m−1. The sum of the average weighted value of nodes can be computed as(8)SG=∑j=1mλj′Nj.The distance of two nodes can be described by their similarity weight. The larger the similarity weight is, the smaller the distance will be. The average distance of the weighted evaluation network can be depicted using Euclidean distance with the following equation:(9)lG=1mm−1∑h,l=1mDAhj′,Alj′,h≠l.To further analyze the importance of each node in product evaluation network, the consistency of group evaluation is computed by removing the evaluation information of an evaluator, which reflects the degree of node clustering in the network. Node importance can be used to represent individual influence. The more influential the node, the more likely it is to change others’ opinions in the communicating process, and the shorter the distance between the corresponding nodes. Therefore, the cohesion of the network will become smaller if the removed node has positive effect and large importance, and vice versa. The cohesion change can be described as(10)IMCpj=∂WG∂WG⋅pj+1m−2SG⋅pj,if∂WG⋅pj<∂WG,∂WG∂WG⋅pj,else,where ∂WG⋅pj and SG⋅pj represent recomputing network cohesion and the average weighted value, respectively, when node pj1≤j≤m in an evaluation round is removed.Finally, we can calculate the weights of the evaluators:(11)λj=IMCpj∑j=1mIMCpj.


## 2.2.3. Analytic Hierarchy Process

The analytic hierarchy process (AHP) is a structured technique founded by Saaty [41, 42] for analyzing complex decisions, which has been used in many applications related to decision-making problems [43, 44]. In our work, we employ this method to obtain the weights of indicators. Matrix B=bijr×r is used to describe a set of pairwise judgments where bij>0 and bij×bji=1. Through Saaty’s 9-point scale, each bij can be assigned 1, 3, 5, 7, and 9 or their inverse, which qualitatively expresses preferences among options as equally, moderately, strong, very strongly, or absolutely important. Then the weights of indicators can be calculated as follows:(12)wi=∏j=1nbij1/n∑i=1n∏j=1nbij1/n.As a consistency check, a consistency index (CI) has also been proposed and defined by the following relation:(13)CI=λmax−nn−1,where λmax is the largest or principal eigenvalue of the matrix B and λmax=∑Ac⋅WT/n⋅W. By introducing the random index (RI) (seen in Table 1), the consistency ratio (CR) is computed as CR=CI/RI. Saaty proposed that if the value of CR is less than 0.1, then the matrix can be considered as having an acceptable consistency.Table 1
The random consistency indices.
n
1
2
3
4
5
6
7
8
9
10
RI
0
0
0.52
0.89
1.11
1.25
1.35
1.40
1.45
1.49


## 3. Results

The product evaluation problem of portable temperature tester design for asphalt pavement is taken as an example to show how the method works. 8 evaluators are recruited for the test, including 3 industrial designers with 2 years of work experience, 2 engineering designers with 3 years of work experience, and 3 users. To detail the product, we use a combination of product renderings, engineering models, and 3D printed mock-ups to present the solution, shown in Figure2.Figure 2
Product solution.A hierarchy of evaluation indicators about the above product is established including visual aesthetics, ergonomic comfort, and manufacturing feasibility. With the AHP method, evaluators are required to give their preference about indicators according to Saaty’s 9-point scale, shown in Table2.Table 2
Judgment matrixes and consistency ratios.
B
B
1
=
1
1
/
3
1
/
4
1
1
/
2
1
B
2
=
1
1
/
2
1
/
3
1
1
/
2
1
B
3
=
1
1
/
3
1
/
3
1
1
1
B
4
=
1
1
/
3
1
/
4
1
1
/
3
1
B
5
=
1
1
/
2
1
/
2
1
1
1
B
6
=
1
1
/
3
1
/
3
1
1
1
B
7
=
1
1
/
2
1
/
2
1
1
/
2
1
B
8
=
1
1
/
3
1
/
4
1
1
/
3
1
CR
0.016
0.009
0
0.064
0
0
0.009
0.064As shown in Table2, all judgments have passed the consistency check. By averaging, the comparison matrix, CR, and weight vector of indicators can be computed with Equations (12) and (13), shown as follows:(14)B=10.39580.322910.64581,CR=0.005,W=0.15,0.35,0.5T.Through 3 rounds of evaluation and discussion, evaluators reach a satisfactory consensus level which is set to 0.8. Judgments given by evaluators using the 11-point (0–10) Likert scale are listed in Table3. With Equations (4)∼(5) and Equations (6)∼(11), we can compute consensus degree, total score, weights of evaluators, and related parameters, shown in Table 4.Table 3
Evaluation in 3 rounds.
Evaluation round
Evaluation indicators
Evaluators
1
2
3
4
5
6
7
8
1
1
9
7
7
6
6
7
9
6
2
9
8
7
6
7
7
6
7
3
10
9
9
8
8
8
7
9
2
1
8
7
9
8
7
8
8
8
2
8
8
8
6
8
8
9
7
3
7
7
8
9
8
8
9
8
3
1
8
9
8
7
8
8
9
8
2
7
8
8
8
9
8
8
8
3
8
8
8
8
9
7
8
8Table 4
Consensus degree, weighting vectors of evaluators, and related parameters in 3 rounds.
Evaluation round
S
G
l
G
∂
WG
Weights of evaluators
Consensus degree
Total score
1
5.5670
0.2661
20.9201
0.0853, 0.0934, 0.1836 0.0919, 0.1830, 0.0934 0.0864, 0.1828
0.6185
7.7849
2
5.8828
0.2316
25.4038
0.0915, 0.0906, 0.1854 0.0830, 0.1860, 0.1856 0.0859, 0.0921
0.7334
7.9194
3
6.5882
0.1544
42.6641
0.0767, 0.1728, 0.1744 0.0800, 0.0719, 0.0768 0.1727, 0.1747
0.9628
8.0357The evaluators’ weight change in 3 rounds is depicted in Figure3. It can be seen that the weight changes of evaluators 1, 3, and 4 are minor, while the weights of other evaluators have been significantly altered. It implies that it may be not appropriate to assign fixed weights to evaluators. Their perception and cognition about target product will transform with the proceeding of design decision-making through group discussion and communication.Figure 3
The evaluators’ weight change in 3 rounds.


## 4. Discussion

(1)
The proposed product design evaluation model was constructed to gauge consensus level of evaluators’ opinions through consensus measurement, network analysis, and AHP. As the evaluation process proceeded, the cognition among evaluators about target product would transform and reach an agreement through discussion and communication, which could be verified by consensus degree changing in Table4 (the consensus degree was improved from 0.6185 to 0.9628). Therefore, the proposed consensus reaching process was proven to be effective and could help improve the opinion consistency among evaluators. Meanwhile, network cohesion change had the same trend which was elevated from 20.9201 to 42.6641. It indicated that the connectedness of the evaluation network was strengthened and evaluators’ weights were gained in a quantitative and automatic way. Similarly, the weight vector of indicators was computed by AHP to avoid the excessive subjectivity given by evaluators.
(2)
The number of evaluation rounds depends on the value ofδ. A high consensus threshold will increase the number of evaluation rounds and further will add fatigue to evaluators, while a low δ can make the evaluation process easy to conduct; however, it may lead to low reliability and credibility of the evaluation results. In the case study, δ was set to 0.8, and 3 rounds of evaluation were implemented. If δ was changed to 0.9, the number of evaluation rounds would have a notable increase to 5 or 6, which would be a burden for evaluators and not conducive to product development. Conversely, if δ was set to 0.6 then 1 round of evaluation would be workable. Hence, an appropriate consensus threshold is needed before product design evaluation. In our study, it was empirically determined.
(3)
Using network analysis to determine evaluators’ weight was suitable because their opinions would change if consensus was not reached, which indicated that the fixed weights of evaluators may not be appropriate for product design evaluation. The implement of network analysis was based on the hypothesis that the information about target product owned by evaluators was symmetric. Without this condition, the network would not be a complete graph and could also be analyzed with the proposed method. Evaluators’ opinion change shown in Table3 demonstrates that some evaluators altered their judgment while the others would stick to their original preference. In order to better conduct this process, a good knowledge sharing and integration mechanism is necessary.
(4)
Compared to traditional product design evaluation process, consensus measurement was introduced to guarantee that the outcome could satisfy the majority. The proposed consensus model involved two key parameters: evaluators’ weight and indicators’ weight, which were computed with network analysis and AHP method, respectively. This dynamic and quantitative analysis process can be extended to online platform, which would further avoid the negative impact of possible expert authoritarian.


## 5. Conclusions

The proposed approach for product design evaluation paves the way for decision-makers to integrate evaluators’ opinions into a reliable decision result for achieving a desired consensus. It is primarily composed of a consensus model for measuring the consistency of evaluators’ preference, a network analysis approach for depicting the variation of evaluators’ opinions and determining their weights in each evaluation round, and an AHP method for computing indicators’ weight. A case study demonstrates that the proposed method can help assess evaluators’ opinion consistency, identify and obtain evaluators’ weight changes dynamically, and determine the indicators’ weight in a quantitative way.This study makes the following contributions: (1) Using the consensus model assures the unity of evaluators’ opinions and makes the decision result reflect the preferences of the majority, which can help reduce the design decision-making risk in product development. (2) Identifying evaluators’ weight through network analysis and indicators’ weight with AHP will help promote consensus reaching and make the design decision-making process more quantitative and objective. (3) The proposed method can effectively assist decision-making in product design.As discussed in Section4, one issue that should be studied in the future is to determine the appropriate consensus threshold in a scientific way, which is crucial and essential before product design evaluation. Though significant progresses have been made on this issue, an effective and intelligent method will be more suitable. Another key issue is to construct an operative knowledge sharing and evolving mechanism with product evaluation proceeding, especially when the evaluation is carried out on an online platform.



---

*Source: 4042024-2019-12-19.xml*