# A New Uncertainty Evaluation Method and Its Application in Evaluating Software Quality

Authors: Jiqiang Chen; Litao Ma; Chao Wang; Hong Zhang; Jie Wan
Journal: Journal of Applied Mathematics
 (2014)

---

## Abstract

Uncertainty theory is a branch of axiomatic mathematics dealing with experts’ belief degree. Considering the uncertainty with experts’ belief degree in the evaluation system and the different roles which different indices play in evaluating the overall goal with a hierarchical structure, a new comprehensive evaluation method is constructed based on uncertainty theory. First, index scores and weights of indices are described by uncertain variables and evaluation grades are described by uncertain sets. Second, weights of indices with respect to the overall goal are introduced. Third, a new uncertainty comprehensive evaluation method is constructed and proved to be a generalization of the weighted average method. Finally, an application is developed in evaluating software quality, which shows the effectiveness of the new method.


---

## Body

## 1. Introduction

Due to human language and technology difficulties, it is difficult to provide an effect and objective evaluation for a system. Therefore, many scholars attempt to establish new mathematical methods to make evaluation results consistent with actual situations. Accordingly, Saaty [1] proposed the analytic hierarchy process (AHP), a method to address multicriteria decision analysis with quantitative and qualitative information. In AHP, human judgments are represented as exact numbers. However, some decision-makers may be reluctant or unable to assign exact numbers to comparison judgments because some evaluation criteria are subjective and qualitative. Therefore, Wu and Tsai [2] used both AHP and decision-making trial and evaluation laboratory methods to evaluate the criteria in autospare parts industry in Taiwan. Kumar et al. [3] presented a new general procedure to construct the membership and nonmembership functions of the fuzzy reliability using time-dependent intuitionistic fuzzy set. By using the finite Markov chain imbedding approach, Zhao and Cui [4] presented a unified formula with the product of matrices for evaluating the system state distribution for generalized multistate k-out-of-n: F systems. Chen et al. [5] provided an evaluation method for enterprisers making investment decisions under hybrid cloud environment using grey system theory. Lee et al. [6] proposed a systematic approach to evaluation of new service concepts by integrating the merit of group analytic hierarchy process in modeling multicriteria decision-making problems. Geng et al. [7] presented a new integrated design concept evaluation approach based on vague sets in order to provide a method for complicated multicriteria decision-making problem under uncertain environments. Aiming to evaluate the government and the monopolist about the consumer’s taste, literature [8] was devoted to the characterization and quantitative representation of imprecise and vague uncertainties and measures of information produced by sources of the considered type. Kramosil [9] introduced the possibilistic variants of both the minimax (the worst case) and the Bayesian optimization principles and applied them in decision-making under uncertainty processed. Using finite-time control and backstepping control approaches, Li et al. [10] proposed a new robust adaptive synchronization scheme to make the synchronization errors of the systems with parameter uncertainties zero in a finite time. Lan et al. [11] presented a bilevel fuzzy principal-agent model for optimal nonlinear taxation problems with asymmetric information and so on.The above methods address imprecise information, such as human language or experts’ degree of belief using fuzzy set theory (see Zadeh [12]), vague set theory, and grey system theory. However, for the evaluation system, the observed data are often not adequate and we have no choice but to invite some domain experts to evaluate the belief degree that an index belongs to an evaluation grade. In this situation, many surveys show that this imprecise information behaves like neither fuzziness nor randomness (see Liu [13] and Liu [14]). And it was showed by Kahneman and Tversky [15] that human beings usually overweight unlikely events. This fact makes the personal belief degree have much larger variance than the frequency.In this case, Liu [13] proposed uncertainty theory to deal with belief degree, and Liu [14] refined uncertainty theory. Nowadays, uncertainty theory has become a branch of axiomatic mathematics. The first fundamental concept in uncertainty theory is the uncertain measure, used to measure the degree of belief in an event. The second concept is the uncertain variable, used to represent quantities with imprecise information (e.g., the exact value of oil field reserve). The third concept is uncertainty distribution, which is used to describe uncertain variables. Uncertainty theory has been applied to many areas. Liu [16] established a theory and practice of uncertain programming, Liu [17] applied uncertainty theory to risk analysis and reliability analysis, Liu [18] studied hybrid logic and uncertain logic, Liu [19] proposed inference rule with applications to uncertain control, and Liu [20] studied uncertain process with applications to inference risk model. To explore the recent developments in uncertainty theory, readers may consult Liu [21].Subsequently, Liu [22] described the weights of indices and the score values of indices with uncertain variables and proposed a comprehensive evaluation method based on uncertainty theory. However, in some certain kinds of assessment domains, we find that different bottom indices play different roles in the evaluation of the overall goal. For example, suppose that S1 and S2 are two students whose four features are shown in Table 1.Table 1
Four features of studentsS1 and S2.
Students
Gender(male = 1, female = 0)
Age(years)
Body height(cm)
Body weight(kg)
S
1
1
20
180
80
S
2
1
20
170
75Therefore, the feature vectors ofS1 and S2 are V1=(1,20,180,80) and V2=(1,20,170,75). Because S1 and S2 are with the same “Gender” and “Age,” we cannot identify them from the two features. In other words, “Gender” and “Age” do not take effect at all in the identification. Furthermore, because the “Body Height” of S1 and S2 is 180 and 170, respectively, they can be identified by “Body Height.” Of course, they can also be identified by “Body Weight.” In a word, the four features play different roles in the identification of S1 and S2. For this reason, a weight for each bottom index with respect to the overall goal is introduced to show the different roles of different bottom indices. And without loss of generality, when different bottom indices play the same role in other assessment domains, weights of indices with respect to the overall goal are equal. Considering these reasons, a new evaluation method is proposed based on uncertainty theory.The structure of this paper is as follows. Section2 provides some relevant concepts about uncertainty theory. Section 3 establishes a new uncertainty evaluation method based on uncertain sets and uncertain variables. Section 4 gives an application of the proposed method in evaluating software quality, and the conclusions are presented in Section 5.


## 2. Preliminaries

In this section, we provide some useful definitions of uncertainty theory.LetΓ be a nonempty set, let L be a σ-algebra on Γ, and let R be a set of real numbers. Each element Λ∈L is called an event. The uncertain measure M, defined on L, was proposed by Liu [13] as follows.Definition 1 (see Liu [13]).
The set functionM is called an uncertain measure if it satisfies the following.
Axiom 1 (normality axiom). Consider M{Γ}=1.
Axiom 2 (duality axiom). Consider M{Λ}+M{Λc}=1 for any event Λ.
Axiom 3 (subadditivity axiom). For every countable sequence of events {Λi}, we have
(1)M{⋃i=1∞‍Λi}≤∑i=1∞‍M{Λi}.The triplet(Γ,L,M) is called an uncertainty space. Let (Γk,Lk,Mk) be uncertainty spaces for k=1,2,…. Write
(2)Γ=Γ1×Γ2,…,L=L1×L2×⋯.
Then the product uncertain measure M on the product σ-algebra L is defined by the following product axiom [23].Axiom 4 (product axiom). Let (Γk,Lk,Mk) be uncertainty spaces for k=1,2,…. The product uncertain measure M is an uncertain measure satisfying
(3)M{∏k=1∞‍Λk}=⋀k=1∞M{Λk},
where Λk are arbitrarily chosen events from Lk for k=1,2,…, respectively.Remark 2.
Uncertain measure is interpreted as the personal belief degree (not frequency) of an uncertain event that may occur. It depends on the personal knowledge concerning the event. The uncertain measure will change if the state of knowledge changes.Definition 3 (see Liu [13]).
An uncertain variable is a measurable functionξ from an uncertainty space (Γ,L,M) to the set of real numbers. That is, for any Borel set B of real numbers, the set
(4){ξ∈B}={γ∈Γ∣ξ(γ)∈B}
is an event.Definition 4 (see Liu [14]).
Letf be a real-valued measurable function, and let ξ1,ξ2,…,ξn be uncertain variables on (Γ,L,M). Then ξ=f(ξ1,ξ2,…,ξn) is an uncertain variable defined by
(5)ξ(γ)=f(ξ1(γ),ξ2(γ),…,ξn(γ)),∀γ∈Γ.Definition 5 (see Liu [14]).
The uncertainty distributionΦ of an uncertain variable ξ is defined by
(6)Φ(x)=M{ξ≤x}
for any real number x (see Figure 1).Figure 1
An uncertainty distribution.Definition 6 (see Liu [14]).
An uncertain variableξ is called zigzag if it has a zigzag uncertainty distribution
(7)Φ(x)={0,ifx≤a,(x-a)2(b-a),ifa≤x≤b,(x+c-2b)2(c-b),ifb≤x≤c,1,ifx≥c
denoted by Z(a,b,c), where a, b, c are real numbers with a<b<c.Definition 7 (see Liu [23]).
The uncertain variablesξ1,ξ2,…,ξn are said to be independent if
(8)M{⋂i=1n‍(ξi∈Bi)}=⋀i=1nM{ξi∈Bi}
for any Borel sets B1,B2,…,Bn of real numbers.Theorem 8 (see Liu [21]).
Assume thatξ1 and ξ2 are independent zigzag uncertain variables Z(a1,b1,c1) and Z(a2,b2,c2), respectively. Then the sum ξ1+ξ2 is also a zigzag uncertain variable Z(a1+a2,b1+b2,c1+c2); that is,
(9)Z(a1,b1,c1)+Z(a2,b2,c2)=Z(a1+a2,b1+b2,c1+c2).
The product of a zigzag uncertain variable Z(a,b,c) and a scalar number k>0 is also a zigzag uncertain variable Z(ka,kb,kc); that is,
(10)kZ(a,b,c)=Z(ka,kb,kc).Definition 9 (see Liu [13]).
Letξ be an uncertain variable. Then the expected value of ξ is defined by
(11)E[ξ]=∫0+∞‍M{ξ≥r}dr-∫-∞0‍M{ξ≤r}dr
provided that at least one of the two integrals is finite.Remark 10.
Expected value is the average value of uncertain variable in the sense of uncertain measure and represents the size of uncertain variable.Example 11 (see Liu [21]).
The zigzag uncertain variableξ~Z(a,b,c) has an expected value
(12)E[ξ]=a+2b+c4.Theorem 12 (see Liu [14]).
Letξ and η be independent uncertain variables with finite expected values. Then for any real numbers a and b, one has
(13)E[aξ+bη]=aE[ξ]+bE[η].An uncertain set is a set-valued function on an uncertainty space that attempts to model “unsharp concepts,” which are essentially sets but their boundaries are not sharply described (because of the ambiguity of human language), such as “young” and “tall.” A formal definition is given as follows.Definition 13 (see Liu [19]).
An uncertain set is a measurable functionξ from an uncertainty space (Γ,L,M) to a collection of sets of real numbers. For any Borel set B of real numbers, that is, for any Borel set B of real numbers, both of
(14){ξ⊂B}={γ∈Γ∣ξ(γ)⊂B},{B⊂ξ}={γ∈Γ∣B⊂ξ(γ)}
are events.Definition 14 (see Liu [24]).
An uncertain setξ is said to have a membership function μ if for any Borel set B of real numbers one has
(15)M{B⊂ξ}=inf⁡x∈Bμ(x),M{ξ⊂B}=1-sup⁡x∈Bcμ(x).The above equations will be called measure inversion formulas.Remark 15.
LetR be a set of real numbers. When an uncertain set has a membership function μ on R, we immediately have
(16)μ(x)=M{x∈ξ},∀x∈R.
Liu [24] proved that a real-valued function μ is a membership function if and only if 0≤μ(x)≤1 (see Figure 2).Figure 2
A membership function.Example 16 (see Liu [21]).
By a triangular uncertain set we mean the uncertain set fully determined by the triplet(a,b,c) of crisp numbers with a<b<c, whose membership function is
(17)μ(x)={(x-a)(b-a),ifa≤x≤b,(x-c)(b-c),ifb≤x≤c,0,otherwise.Definition 17 (see Liu [24]).
Letξ be an uncertain set with a membership function μ. Then the set-valued function
(18)μ-1(α)={x∈R∣μ(x)≥α},∀α∈[0,1]
is called the inverse membership function of ξ.Definition 18 (see Liu [25]).
The uncertain setsξ1,ξ2,…,ξn are said to be independent if for any Borel sets B1,B2,…,Bn one has
(19)M{⋂i=1n‍(ξi*⊂Bi)}=⋀i=1nM{ξi*⊂Bi},M{⋃i=1n‍(ξi*⊂Bi)}=⋁i=1nM{ξi*⊂Bi},
where ξi* are arbitrarily chosen from {ξi,ξic},  i=1,2,…,n, respectively.Theorem 19 (see Liu [24]).
Letξ and η be independent uncertain sets with membership functions μ and ν, respectively. Then their union ξ∪η has a membership function
(20)λ(x)=μ(x)∨ν(x).Theorem 20 (see Liu [24]).
Letξ and η be independent uncertain sets with membership functions μ and ν, respectively. Then their intersection ξ∩η has a membership function
(21)λ(x)=μ(x)∧ν(x).Theorem 21 (see Liu [24]).
Letξ be an uncertain set with membership function μ. Then its complement ξc has a membership function
(22)λ(x)=1-μ(x).Theorem 22 (see Liu [24]).
Letξ1,ξ2,…,ξn be independent uncertain sets with inverse membership functions μ1-1,μ2-1,…,μn-1, respectively. If the function f is a measurable function, then
(23)ξ=f(ξ1,ξ2,…,ξn)
is an uncertain set with inverse membership function
(24)λ-1(α)=f(μ1-1(α),μ2-1(α),…,μn-1(α)).Definition 23 (see Liu [19]).
Letξ be a nonempty uncertain set. Then the expected value of ξ is defined by
(25)E[ξ]=∫0+∞‍M{ξ⪰r}dr-∫-∞0‍M{ξ⪯r}dr
provided that at least one of the two integrals is finite.Please note thatξ⪰r represents “ξ is imaginarily included in [r,+∞)” and ξ⪯r represents “ξ is imaginarily included in (-∞,r].”Example 24 (see Liu [21]).
The triangular uncertain setξ=(a,b,c) has an expected value
(26)E[ξ]=a+2b+c4.


## 3. Uncertainty Evaluation Method

When making a comprehensive evaluation, factors influencing the grade of the overall goal should be considered. The index system is often represented by a three-layer hierarchical structure with the overall goal, the second layer, and the bottom layer (namely, the factors influencing the overall goal’s scaling). Experts always intend to show their own opinions and expectations of each evaluation index, so the evaluation results represent human uncertainty and belief degree. Therefore, the score value and weight of each evaluation index are represented by uncertain variables, and evaluation grades are represented by uncertain sets. Therefore, a new evaluation method based on uncertain variables and uncertain sets is proposed.


### 3.1. Establishment of Hierarchical Index Structure

Based on evaluation criteria, a hierarchical index structure is established by experts as in Figure3. There are l indices in the second level and mi indices immediately below index Bi (i=1,2,…,l), where m1+m2+⋯+ml=m. Let the index (in the bottom level) set be C={C11,C12,…,C1m1,C21,C22,…,C2m2,Cl1,Cl2,…,C1ml}.Figure 3
A simple hierarchical index structure.


### 3.2. Weights of Indices

Suppose that there arer experts in the evaluation group and the n degrees of importance are represented by uncertain variables η1,η2,…,ηn. Each expert gives a degree of importance for each index. For index Cij (i=1,2,…,t; j=1,2,…,mi), r1 experts select η1, r2 experts select η2,…,rn experts select ηn, where r1+r2+⋯+rn=r. Weight wij of index Cij is represented by
(27)wij=r1rη1+r2rη2+⋯+rnrηn,ii=1,2,…,t;j=1,2,…,mi.
According to Definition 3, weight wij is an uncertain variable and can be described by its uncertainty distribution Φij.


### 3.3. Grade Vectors of Bottom Indices

The score values from experts can be represented by uncertain variables. Let thet evaluation grades (e.g., Poor, Fair,…, Excellent) be represented by uncertain sets ξ1,ξ2,…,ξt, with membership functions μ1,μ2,…,μt, respectively. Let xij be the index score of index Cij. Then, the grade vector sij of index Cij is represented by
(28)sij=(sij1=μ1(xij),sij2=μ2(xij),…,sijt=μt(xij)),
where sijk (k=1,2,…,t) is the membership degree to which index Cij belongs to grade ξk.The next step is to construct a method to realize the transformation from the grade vectorsij of the bottom index Cij to the grade vector s(A) of the overall goal A.


### 3.4. Transformation Method

The transformation method can be obtained with the following three steps.Step 1.
To determine the importance of indexCij in the grading of the overall goal, the weight of index Cij with respect to the overall goal A is introduced beginning with the following formulas:
(29)Hij(A)=-∑k=1t‍sijk·lgsijk,Vij(A)=1-1lgtHij(A),λij(A)=Vij(A)∑i=1l∑j=1miVij(A),
where λij(A) is the weight of index Cij with respect to A. From the above formulas, we have
(30)0≤λij(A)≤1,∑i=1l‍∑j=1mi‍λij(A)=1.Step 2.
Becauseλij(A) shows the degree of the effective information offered by Cij in the evaluation of A, we calculate
(31)λij(A)·sijk.Step 3.
By weightwij(Bi) of index Cij with respect to Bi, we have
(32)Nk(Bi)=∑j=1mi‍wij(Bi)·λij(A)·sijk,sk(Bi)=Nk(Bi)∑k=1tNk(Bi).
From the above formulas, we have
(33)0≤sk(Bi)≤1,∑k=1t‍sk(Bi)=1.
Thus, we obtain the grade vector s(Bi)=(s1(Bi),s2(Bi),…,st(Bi)) of index Bi in the second level, where sk(Bi) indicates the degree to which Bi belongs to evaluation grade ξk.The grade vectors(A)=(s1(A),s2(A),…,st(A)) of the overall goal A can be obtained in the same way.Remark 25.
From the property ofHij(A), it is obtained that the larger the λij(A) is, the more important the role Cij plays in the evaluation of A is. If λij(A)=1, then the evaluation grade of A can be determined only by index Cij. If λij(A)=0, then index Cij does not play any role in determining the grade of A.Theorem 26.
The weighted-average method is a special case of the above method.Proof.
For each row vector(sj1,sj2,…,sj5), if there is one component sjk=1 for some k and the rest four components are 0, then Hij(A)=1, Vij(A)=1, and λij(A)=1/n. Thus, by formulas
(34)Nk(Bi)=∑j=1mi‍wij(Bi)·λij(A)·sijk,sk(Bi)=Nk(Bi)∑k=1tNk(Bi),
we have
(35)sk(A)=∑j=1n‍λj·sjk,k=1,2,…,t.
That is to say, the weighted-average method is a special case of the above method.


### 3.5. Identification

Sometimes, the evaluation scales are comparable (e.g., Poor, Fair,…, Excellent scale; “Fair” is better than “Poor”), and a partial order “≻” can be defined according to the actual situation. If evaluation scale C1 is better than C2, we denote C1≻C2; otherwise we denote C1≺C2. Of course, whether C1 is better than C2 depends on the actual situation. To address the ordered division (e.g., Poor, Fair,…, Excellent), Cheng [26] proposed the confidence degree principle.Confidence Degree Principle (Cheng [26]). Let “≻” be a partial order, let C1,C2,…,Ct be an ordered division, and let λ (0.5<λ<1) be the confidence level. If C1≻C2≻⋯≻Ct,
(36)k0=min⁡{k:∑i=1k‍μx(Ci)≥λ,1≤k≤t},
and if C1≺C2≺⋯≺Ct,
(37)k0=max⁡{k:∑i=1k‍μx(Ci)≥λ,1≤k≤t},
and then x belongs to Ck0 with at least the confidence level λ.


## 3.1. Establishment of Hierarchical Index Structure

Based on evaluation criteria, a hierarchical index structure is established by experts as in Figure3. There are l indices in the second level and mi indices immediately below index Bi (i=1,2,…,l), where m1+m2+⋯+ml=m. Let the index (in the bottom level) set be C={C11,C12,…,C1m1,C21,C22,…,C2m2,Cl1,Cl2,…,C1ml}.Figure 3
A simple hierarchical index structure.


## 3.2. Weights of Indices

Suppose that there arer experts in the evaluation group and the n degrees of importance are represented by uncertain variables η1,η2,…,ηn. Each expert gives a degree of importance for each index. For index Cij (i=1,2,…,t; j=1,2,…,mi), r1 experts select η1, r2 experts select η2,…,rn experts select ηn, where r1+r2+⋯+rn=r. Weight wij of index Cij is represented by
(27)wij=r1rη1+r2rη2+⋯+rnrηn,ii=1,2,…,t;j=1,2,…,mi.
According to Definition 3, weight wij is an uncertain variable and can be described by its uncertainty distribution Φij.


## 3.3. Grade Vectors of Bottom Indices

The score values from experts can be represented by uncertain variables. Let thet evaluation grades (e.g., Poor, Fair,…, Excellent) be represented by uncertain sets ξ1,ξ2,…,ξt, with membership functions μ1,μ2,…,μt, respectively. Let xij be the index score of index Cij. Then, the grade vector sij of index Cij is represented by
(28)sij=(sij1=μ1(xij),sij2=μ2(xij),…,sijt=μt(xij)),
where sijk (k=1,2,…,t) is the membership degree to which index Cij belongs to grade ξk.The next step is to construct a method to realize the transformation from the grade vectorsij of the bottom index Cij to the grade vector s(A) of the overall goal A.


## 3.4. Transformation Method

The transformation method can be obtained with the following three steps.Step 1.
To determine the importance of indexCij in the grading of the overall goal, the weight of index Cij with respect to the overall goal A is introduced beginning with the following formulas:
(29)Hij(A)=-∑k=1t‍sijk·lgsijk,Vij(A)=1-1lgtHij(A),λij(A)=Vij(A)∑i=1l∑j=1miVij(A),
where λij(A) is the weight of index Cij with respect to A. From the above formulas, we have
(30)0≤λij(A)≤1,∑i=1l‍∑j=1mi‍λij(A)=1.Step 2.
Becauseλij(A) shows the degree of the effective information offered by Cij in the evaluation of A, we calculate
(31)λij(A)·sijk.Step 3.
By weightwij(Bi) of index Cij with respect to Bi, we have
(32)Nk(Bi)=∑j=1mi‍wij(Bi)·λij(A)·sijk,sk(Bi)=Nk(Bi)∑k=1tNk(Bi).
From the above formulas, we have
(33)0≤sk(Bi)≤1,∑k=1t‍sk(Bi)=1.
Thus, we obtain the grade vector s(Bi)=(s1(Bi),s2(Bi),…,st(Bi)) of index Bi in the second level, where sk(Bi) indicates the degree to which Bi belongs to evaluation grade ξk.The grade vectors(A)=(s1(A),s2(A),…,st(A)) of the overall goal A can be obtained in the same way.Remark 25.
From the property ofHij(A), it is obtained that the larger the λij(A) is, the more important the role Cij plays in the evaluation of A is. If λij(A)=1, then the evaluation grade of A can be determined only by index Cij. If λij(A)=0, then index Cij does not play any role in determining the grade of A.Theorem 26.
The weighted-average method is a special case of the above method.Proof.
For each row vector(sj1,sj2,…,sj5), if there is one component sjk=1 for some k and the rest four components are 0, then Hij(A)=1, Vij(A)=1, and λij(A)=1/n. Thus, by formulas
(34)Nk(Bi)=∑j=1mi‍wij(Bi)·λij(A)·sijk,sk(Bi)=Nk(Bi)∑k=1tNk(Bi),
we have
(35)sk(A)=∑j=1n‍λj·sjk,k=1,2,…,t.
That is to say, the weighted-average method is a special case of the above method.


## 3.5. Identification

Sometimes, the evaluation scales are comparable (e.g., Poor, Fair,…, Excellent scale; “Fair” is better than “Poor”), and a partial order “≻” can be defined according to the actual situation. If evaluation scale C1 is better than C2, we denote C1≻C2; otherwise we denote C1≺C2. Of course, whether C1 is better than C2 depends on the actual situation. To address the ordered division (e.g., Poor, Fair,…, Excellent), Cheng [26] proposed the confidence degree principle.Confidence Degree Principle (Cheng [26]). Let “≻” be a partial order, let C1,C2,…,Ct be an ordered division, and let λ (0.5<λ<1) be the confidence level. If C1≻C2≻⋯≻Ct,
(36)k0=min⁡{k:∑i=1k‍μx(Ci)≥λ,1≤k≤t},
and if C1≺C2≺⋯≺Ct,
(37)k0=max⁡{k:∑i=1k‍μx(Ci)≥λ,1≤k≤t},
and then x belongs to Ck0 with at least the confidence level λ.


## 4. An Application in the Evaluation of Software Quality

In this section, an application of the proposed method in evaluation of software quality is given.


### 4.1. Evaluation System and Some Data

A problem of evaluating software quality was discussed in Li [27]. Next, we apply the proposed method to this evaluation problem, and the evaluation system is shown as in Table 2. Based on the evaluation criteria, the experts provided the scores of the bottom indices in Column 5 of Table 2. The evaluation grades are “Poor,” “Fair,” “Good,” “Very good,” and “Excellent.” They are represented by uncertain sets ξ1,ξ2,…,ξ5 with membership functions μ1,μ2,…,μ5, respectively, where membership functions
(38)μ1(x)={1,if0≤x≤0.2,(1-x)0.8,if0.2<x≤1,0,otherwise,μ2(x)={(x-0.2)0.8,if0.2≤x≤1,2-x,if1<x≤2,0,otherwise,μ3(x)={x-1,if1≤x≤2,3-x,if2<x≤3,0,otherwise,μ4(x)={x-2,if2≤x≤3,(3.8-x)0.8,if3<x≤3.8,0,otherwise,μ5(x)={(x-3)0.8,if3≤x≤3.8,1,if3.8<x≤4,0,otherwise
are given by experts according to their personal knowledge and actual situation of the evaluation. According to the scores given by experts (Column 5 in Table 2) and the above membership functions, the grade vectors (Column 4 in Table 2) can be obtained (see Section 4.3).Table 2
Evaluation system of software quality.
The overall goal
Second-level indices and weights
Bottom indices and weights
Grade vectors
Scores
Software qualityA
Install sectionB1(0.1)
Install difficultyC11(0.25)
(
0,0
,
0.50,0.50,0
)
2.50
Size after installationC12(0.25)
(
0,0
,
0,1.00,0
)
3.00
Memory occupationC13(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
Shortcuts creationC14(0.25)
(
0,0
,
0,0.50,0.50
)
3.40
General structureB2(0.15)
Complete moduleC21(0.2)
(
0,0
,
0.41,0.59,0
)
2.59
Appropriate gradeC22(0.2)
(
0,0
,
0,0.50,0.50
)
3.40
Reasonable module partitionC23(0.2)
(
0,0
,
1.00,0
,
0
)
2.00
Business integration levelC24(0.2)
(
0,0
,
0,0
,
1.00
)
4.00
Data import and extraction functionC25(0.2)
(
0,0
,
0.30,0.70,0
)
2.70
Security mechanismB3(0.15)
Permissions allocationC31(0.25)
(
0,1.00,0
,
0,0
)
1.00
Operating logC32(0.25)
(
0,0
,
0,1.00,0
)
3.00
Data backup and restorationC33(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
Data maintenanceC34(0.25)
(
0,0
,
0.50,0.50,0
)
2.50
Program and data processingB4(0.2)
ReliabilityC41(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
EfficiencyC42(0.25)
(
0,0
,
1.00,0
,
0
)
2.00
MaintainabilityC43(0.25)
(
0,0
,
1.00,0
,
0
)
2.00
PortabilityC44(0.25)
(
0,0
,
0,0
,
1.00
)
4.00
Code maintenanceB5(0.1)
Definable codeC51(0.33)
(
0,0
,
0,1.00,0
)
3.00
Gradable codeC52(0.33)
(
0,0
,
0.30,0.70,0
)
2.70
Proper code maintenanceC53(0.34)
(
0,0
,
0.50,0.50,0
)
2.50
Operating performanceB6(0.3)
Friendly interfaceC61(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
Easy-to-useC62(0.25)
(
0,0
,
0.50,0.50,0
)
2.50
Online helpC63(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
Multitasking switchingC64(0.25)
(
0,0
,
1.00,0
,
0
)
2.00


### 4.2. Weights of Indices

There are ten experts in the evaluation group, and there are five degrees of importance (namely, “most important,” “more important,” “important,” “less important,” and “unimportant”), which are represented by uncertain variablesη1,η2,…,η5 with the same uncertainty distribution Z(0.54,0.69,0.84). Taking indices C11, C12, C13, C14, for examples, the evaluation results from the experts are shown as in Table 3. Weight w1j of index C1j is an uncertain variable η1j. By ω1j=(r1/r)η1+⋯+(rn/r)ηn, j=1,2,3,4, and Theorem 8, we have η1j~Z(0.54,0.69,0.84). The vector w1=(w11,w12,w13,w14) is obtained by the 99-table, and the values of w1j are shown in Table 4.Table 3
Evaluation results given by experts.
Bottom indices
Most important
More important
Important
Less important
Unimportant
Install difficultyC11
1
2
3
2
2
Size after installationC12
2
1
3
2
2
Memory occupationC13
2
2
3
1
2
Shortcuts creationC14
2
3
2
2
1Table 4
The 99-table of weights for indicesC11,C12,C13,C14.
Weights
0.01
0.02
⋯
0.99
w
11
0.54
0.55
⋯
0.84
w
12
0.54
0.55
⋯
0.84
w
13
0.54
0.55
⋯
0.84
w
14
0.54
0.55
⋯
0.84By Example11 and Theorem 12, the expected value of w1=(w11,w12,w13,w14) is (0.78,0.78,0.78,0.78). Normalizing the expected value, the weights of indices C11, C12, C13, C14 are (0.25,0.25,0.25,0.25). The weights of the other indices can be obtained in the same way (the numbers in Column 3 of Table 2).


### 4.3. Grade Vectors of Bottom Indices

Taking the grade vector of indexC11 as an example, the score value x11 of index C11 is 2.50 and the membership functions of evaluation grades ξ1,ξ2,…,ξ5 are μ1,μ2,…,μ5, respectively. Therefore, the grade vector of index C11 is
(39)s(C11)=(μ1(x11),μ2(x11),…,μ5(x11))=(0,0,0.50,0.50,0).
Similarly, the other grade vectors of the bottom indices can be obtained by the scores of the bottom indices (Column 5 in Table 2).


### 4.4. Grade Vector of Software QualityA

In this subsection, the transformation algorithm established above is used to realize the transformation from the grade vectors of the bottom indices to the grade vector of software quality.


#### 4.4.1. Grade Vectors of Indices in the Second Level

We take the calculation of grade vectors(B1) of index B1 (install section) as an example.There are four indices,C11, C12, C13, C14, immediately below B1, and the grade-transition matrix of B1 is
(40)U(B1)=(000.500.5000001.000000.300.7000000.500.50).With the weights(w11,w12,w13,w14)=(0.25,0.25,0.25,0.25) of indices C11, C12, C13, C14, we can obtain
(41)s(B1)=(0,0,0.1706,0.7262,0.1032).
Similarly, we can obtain the grade vectors
(42)s(B2)=(0,0,0.3777,0.2815,0.3408),…,s(B6)=(0,0,0.5896,0.4104,0).


#### 4.4.2. Grade Vector of Software QualityA

With the grade vectorss(B1),…,s(B6) of indices B1,…,B6, we can obtain the grade-transition matrix of A as
(43)U(A)=(000.17060.72620.1032000.37770.28150.340800.31350.14760.53890000.60380.39620000.21720.78280000.58960.41040).
Similar to calculation (1), the grade vector s(A) of the overall goal is obtained as
(44)s(A)=(0,0.0356,0.4319,0.4903,0.0422).


### 4.5. Identification

In this example, the confidence level ofA belonging to ξ4 (Very good) is no less than 0.53 (0.4903+0.0422≥0.53) and the confidence level of A belonging to ξ3 (Good) is no less than 0.96 (0.4903+0.0422+0.4319≥0.96). Comparatively, it is more reasonable to define A in grade ξ4. Therefore, the quality of this software is “Very good.”


### 4.6. Comparative Analysis

In Li [27], the fuzzy max-min method is used in the software quality evaluation, and we calculate the results via the fuzzy weighted-average method with the same data in Table 2. The results are shown in Table 5. Table 5 shows that the proposed method and fuzzy max-min method produced the same result (i.e., the evaluation result is Very good) and the evaluation result from fuzzy weighted method is Good. And also there is some difference in the membership degree of each grade. Furthermore, the literatures [28, 29] discussed the algorithms for maximizing the soft margin, which show that the larger the difference between two adjacent grades is, the stronger the classification ability of the method is. Therefore, we analyzed the differences between two adjacent grades in Table 6. From Table 6, we can see that there are three differences (between ξ4 and ξ5, ξ3 and ξ4, and ξ2 and ξ3) and the proposed method is larger than the other two methods. Thus, the proposed method has a stronger classification ability in the evaluation of software quality.Table 5
Comparison of the results with the other two methods.
Grades
Poor
Fair
Good
Very good
Excellent
The proposed method
0
0.0356
0.4319
0.4903
0.0422
Fuzzy weighted-average method
0
0.1051
0.4507
0.4053
0.0389
Fuzzy max-min method
0
0.0822
0.4165
0.4697
0.0516Table 6
The differences between two adjacent grades.
Differences
ξ
1 and ξ2
ξ
2 and ξ3
ξ
3 and ξ4
ξ
4 and ξ5
The proposed method
0.0356
0.3963
0.0584
0.4481
Fuzzy weighted-average method
0.1051
0.3456
0.0454
0.3664
Fuzzy max-min method
0.0822
0.3343
0.0532
0.4181


## 4.1. Evaluation System and Some Data

A problem of evaluating software quality was discussed in Li [27]. Next, we apply the proposed method to this evaluation problem, and the evaluation system is shown as in Table 2. Based on the evaluation criteria, the experts provided the scores of the bottom indices in Column 5 of Table 2. The evaluation grades are “Poor,” “Fair,” “Good,” “Very good,” and “Excellent.” They are represented by uncertain sets ξ1,ξ2,…,ξ5 with membership functions μ1,μ2,…,μ5, respectively, where membership functions
(38)μ1(x)={1,if0≤x≤0.2,(1-x)0.8,if0.2<x≤1,0,otherwise,μ2(x)={(x-0.2)0.8,if0.2≤x≤1,2-x,if1<x≤2,0,otherwise,μ3(x)={x-1,if1≤x≤2,3-x,if2<x≤3,0,otherwise,μ4(x)={x-2,if2≤x≤3,(3.8-x)0.8,if3<x≤3.8,0,otherwise,μ5(x)={(x-3)0.8,if3≤x≤3.8,1,if3.8<x≤4,0,otherwise
are given by experts according to their personal knowledge and actual situation of the evaluation. According to the scores given by experts (Column 5 in Table 2) and the above membership functions, the grade vectors (Column 4 in Table 2) can be obtained (see Section 4.3).Table 2
Evaluation system of software quality.
The overall goal
Second-level indices and weights
Bottom indices and weights
Grade vectors
Scores
Software qualityA
Install sectionB1(0.1)
Install difficultyC11(0.25)
(
0,0
,
0.50,0.50,0
)
2.50
Size after installationC12(0.25)
(
0,0
,
0,1.00,0
)
3.00
Memory occupationC13(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
Shortcuts creationC14(0.25)
(
0,0
,
0,0.50,0.50
)
3.40
General structureB2(0.15)
Complete moduleC21(0.2)
(
0,0
,
0.41,0.59,0
)
2.59
Appropriate gradeC22(0.2)
(
0,0
,
0,0.50,0.50
)
3.40
Reasonable module partitionC23(0.2)
(
0,0
,
1.00,0
,
0
)
2.00
Business integration levelC24(0.2)
(
0,0
,
0,0
,
1.00
)
4.00
Data import and extraction functionC25(0.2)
(
0,0
,
0.30,0.70,0
)
2.70
Security mechanismB3(0.15)
Permissions allocationC31(0.25)
(
0,1.00,0
,
0,0
)
1.00
Operating logC32(0.25)
(
0,0
,
0,1.00,0
)
3.00
Data backup and restorationC33(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
Data maintenanceC34(0.25)
(
0,0
,
0.50,0.50,0
)
2.50
Program and data processingB4(0.2)
ReliabilityC41(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
EfficiencyC42(0.25)
(
0,0
,
1.00,0
,
0
)
2.00
MaintainabilityC43(0.25)
(
0,0
,
1.00,0
,
0
)
2.00
PortabilityC44(0.25)
(
0,0
,
0,0
,
1.00
)
4.00
Code maintenanceB5(0.1)
Definable codeC51(0.33)
(
0,0
,
0,1.00,0
)
3.00
Gradable codeC52(0.33)
(
0,0
,
0.30,0.70,0
)
2.70
Proper code maintenanceC53(0.34)
(
0,0
,
0.50,0.50,0
)
2.50
Operating performanceB6(0.3)
Friendly interfaceC61(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
Easy-to-useC62(0.25)
(
0,0
,
0.50,0.50,0
)
2.50
Online helpC63(0.25)
(
0,0
,
0.30,0.70,0
)
2.70
Multitasking switchingC64(0.25)
(
0,0
,
1.00,0
,
0
)
2.00


## 4.2. Weights of Indices

There are ten experts in the evaluation group, and there are five degrees of importance (namely, “most important,” “more important,” “important,” “less important,” and “unimportant”), which are represented by uncertain variablesη1,η2,…,η5 with the same uncertainty distribution Z(0.54,0.69,0.84). Taking indices C11, C12, C13, C14, for examples, the evaluation results from the experts are shown as in Table 3. Weight w1j of index C1j is an uncertain variable η1j. By ω1j=(r1/r)η1+⋯+(rn/r)ηn, j=1,2,3,4, and Theorem 8, we have η1j~Z(0.54,0.69,0.84). The vector w1=(w11,w12,w13,w14) is obtained by the 99-table, and the values of w1j are shown in Table 4.Table 3
Evaluation results given by experts.
Bottom indices
Most important
More important
Important
Less important
Unimportant
Install difficultyC11
1
2
3
2
2
Size after installationC12
2
1
3
2
2
Memory occupationC13
2
2
3
1
2
Shortcuts creationC14
2
3
2
2
1Table 4
The 99-table of weights for indicesC11,C12,C13,C14.
Weights
0.01
0.02
⋯
0.99
w
11
0.54
0.55
⋯
0.84
w
12
0.54
0.55
⋯
0.84
w
13
0.54
0.55
⋯
0.84
w
14
0.54
0.55
⋯
0.84By Example11 and Theorem 12, the expected value of w1=(w11,w12,w13,w14) is (0.78,0.78,0.78,0.78). Normalizing the expected value, the weights of indices C11, C12, C13, C14 are (0.25,0.25,0.25,0.25). The weights of the other indices can be obtained in the same way (the numbers in Column 3 of Table 2).


## 4.3. Grade Vectors of Bottom Indices

Taking the grade vector of indexC11 as an example, the score value x11 of index C11 is 2.50 and the membership functions of evaluation grades ξ1,ξ2,…,ξ5 are μ1,μ2,…,μ5, respectively. Therefore, the grade vector of index C11 is
(39)s(C11)=(μ1(x11),μ2(x11),…,μ5(x11))=(0,0,0.50,0.50,0).
Similarly, the other grade vectors of the bottom indices can be obtained by the scores of the bottom indices (Column 5 in Table 2).


## 4.4. Grade Vector of Software QualityA

In this subsection, the transformation algorithm established above is used to realize the transformation from the grade vectors of the bottom indices to the grade vector of software quality.


### 4.4.1. Grade Vectors of Indices in the Second Level

We take the calculation of grade vectors(B1) of index B1 (install section) as an example.There are four indices,C11, C12, C13, C14, immediately below B1, and the grade-transition matrix of B1 is
(40)U(B1)=(000.500.5000001.000000.300.7000000.500.50).With the weights(w11,w12,w13,w14)=(0.25,0.25,0.25,0.25) of indices C11, C12, C13, C14, we can obtain
(41)s(B1)=(0,0,0.1706,0.7262,0.1032).
Similarly, we can obtain the grade vectors
(42)s(B2)=(0,0,0.3777,0.2815,0.3408),…,s(B6)=(0,0,0.5896,0.4104,0).


### 4.4.2. Grade Vector of Software QualityA

With the grade vectorss(B1),…,s(B6) of indices B1,…,B6, we can obtain the grade-transition matrix of A as
(43)U(A)=(000.17060.72620.1032000.37770.28150.340800.31350.14760.53890000.60380.39620000.21720.78280000.58960.41040).
Similar to calculation (1), the grade vector s(A) of the overall goal is obtained as
(44)s(A)=(0,0.0356,0.4319,0.4903,0.0422).


## 4.4.1. Grade Vectors of Indices in the Second Level

We take the calculation of grade vectors(B1) of index B1 (install section) as an example.There are four indices,C11, C12, C13, C14, immediately below B1, and the grade-transition matrix of B1 is
(40)U(B1)=(000.500.5000001.000000.300.7000000.500.50).With the weights(w11,w12,w13,w14)=(0.25,0.25,0.25,0.25) of indices C11, C12, C13, C14, we can obtain
(41)s(B1)=(0,0,0.1706,0.7262,0.1032).
Similarly, we can obtain the grade vectors
(42)s(B2)=(0,0,0.3777,0.2815,0.3408),…,s(B6)=(0,0,0.5896,0.4104,0).


## 4.4.2. Grade Vector of Software QualityA

With the grade vectorss(B1),…,s(B6) of indices B1,…,B6, we can obtain the grade-transition matrix of A as
(43)U(A)=(000.17060.72620.1032000.37770.28150.340800.31350.14760.53890000.60380.39620000.21720.78280000.58960.41040).
Similar to calculation (1), the grade vector s(A) of the overall goal is obtained as
(44)s(A)=(0,0.0356,0.4319,0.4903,0.0422).


## 4.5. Identification

In this example, the confidence level ofA belonging to ξ4 (Very good) is no less than 0.53 (0.4903+0.0422≥0.53) and the confidence level of A belonging to ξ3 (Good) is no less than 0.96 (0.4903+0.0422+0.4319≥0.96). Comparatively, it is more reasonable to define A in grade ξ4. Therefore, the quality of this software is “Very good.”


## 4.6. Comparative Analysis

In Li [27], the fuzzy max-min method is used in the software quality evaluation, and we calculate the results via the fuzzy weighted-average method with the same data in Table 2. The results are shown in Table 5. Table 5 shows that the proposed method and fuzzy max-min method produced the same result (i.e., the evaluation result is Very good) and the evaluation result from fuzzy weighted method is Good. And also there is some difference in the membership degree of each grade. Furthermore, the literatures [28, 29] discussed the algorithms for maximizing the soft margin, which show that the larger the difference between two adjacent grades is, the stronger the classification ability of the method is. Therefore, we analyzed the differences between two adjacent grades in Table 6. From Table 6, we can see that there are three differences (between ξ4 and ξ5, ξ3 and ξ4, and ξ2 and ξ3) and the proposed method is larger than the other two methods. Thus, the proposed method has a stronger classification ability in the evaluation of software quality.Table 5
Comparison of the results with the other two methods.
Grades
Poor
Fair
Good
Very good
Excellent
The proposed method
0
0.0356
0.4319
0.4903
0.0422
Fuzzy weighted-average method
0
0.1051
0.4507
0.4053
0.0389
Fuzzy max-min method
0
0.0822
0.4165
0.4697
0.0516Table 6
The differences between two adjacent grades.
Differences
ξ
1 and ξ2
ξ
2 and ξ3
ξ
3 and ξ4
ξ
4 and ξ5
The proposed method
0.0356
0.3963
0.0584
0.4481
Fuzzy weighted-average method
0.1051
0.3456
0.0454
0.3664
Fuzzy max-min method
0.0822
0.3343
0.0532
0.4181


## 5. Conclusions

In this paper, weights for bottom indices with respect to the overall goal in an evaluation system are introduced and a new uncertainty evaluation method is proposed based on uncertain sets and uncertain variables. This method generalizes the weighted average method, and it is applied in the evaluation of software quality. Comparative analysis with other two methods shows that the proposed method has a stronger classification ability in the evaluation of software quality. More importantly, the proposed method in this paper can also be used in other evaluation systems with a hierarchical structure. Therefore, more applications of the proposed method can be underdeveloped further.



---

*Source: 145285-2014-07-02.xml*