# A Data Classifier Based on Maximum Likelihood Evidential Reasoning Rule

Authors: Hong He; Xuelin Zhang; Xiaobin Xu; Zhongrong Li; Yu Bai; Fang Liu; Felix Steyskal; Georg Brunauer
Journal: Mathematical Problems in Engineering
 (2023)

---

## Abstract

In Dempster–Shafer evidence theory (DST), some classical evidence combination rules can be used to fuse the multiple pieces of evidence, respectively abstracted from different attributes (features) so as to increase the accuracy of multiattribute classification decision making. However, most of them have not yet considered the interdependence among multiple pieces of evidence. The newly proposed maximum likelihood evidential reasoning (MAKER) rule measures such ubiquitous interdependence by introducing correlation factors into evidence combination. Hence, this paper designs a MAKER-based classifier to mine more correlation information for data classification. Finally, some numerical analysis (classification) experiments are carried out using five popular benchmark databases from the University of California, Irvine (UCI) to illustrate that the refined measure for evidence interdependence can aggregate the fused probability (belief degree) into real class label of a sample and further improve classification accuracy.


---

## Body

## 1. Introduction

Supervised learning is one typical class of method in machine learning. The linear or nonlinear relationship between the attributes or features of samples and their classes can be built by constructing a classifier. Hence, the classifier can infer the class labels of samples according to their input features [1]. The data classification is also an important part in the field of data mining and knowledge discovery. It is widely used in the biomedical field, security field, image processing field, and so on [2, 3]. The most common methods for classifier design are as follows: support vector machine (SVM) [4], artificial neural network (ANN) [5], naive Bayesian [6], k nearest neighbor (k-NN) [7], random forest [8], and so on.In the classification problem, different classification methods have their advantages and disadvantages. One can choose the feasible classification method to solve specific problems according to the actual application effect. However, almost all methods have to face a common uncertainty, that is, the sample classification errors are hardly avoided since the boundaries of features about the different classes are ambiguous. As a result, the feature value of a sample cannot correctly match its corresponding class [9–11]. DST evidence theory is one of the effective methods for modeling, reasoning, and fusion of uncertain information [12–17]. Moreover, to reduce the classification uncertainty, DST provides a type of classification method based on the basic probability assignment function (BPA) [18–20]. For example, in the frame of DST, the negation of BPA was utilized by Wu et al. [19] to model valuable information from multiple sources, and a new classification method based on the negation of BPA was proposed to achieve an effective classification. DST defines a “frame of discernment” (FoD) including all class labels and uses BPA called “evidence” to measure the belief degrees that a sample with multiple attributes (features) belongs to every class and the subsets of these classes. Then, some classical evidence combination mechanisms (Dempster’s rule [21], evidential reasoning rule [21], Yager’s rule [22], Smets’s rule [23], DSm rule [24], Dubois and Prade’s rule [25], etc.) can be used to fuse the multiple pieces of evidence so as to reducing the uncertainty of multifeature classification.In these classical rules, the evidential reasoning (ER) rule distinguish two conceptions, namely, the evidence reliability and the evidence importance for the first time. The former reflects the ability of single feature or its evidence to identify different classes; the latter points out the importance of evidence relative to other evidence. Such defined modeling for evidence characteristics can effectively increase classification accuracy [26–30]. For example, Xu et al. [31] proposed a classifier based on attribute vectorization and evidential reasoning, which effectively reduces the structural complexity and computational effort of ER classifiers with high-dimensional input attributes. For multiple classifiers with different pattern features, Liu et al. [32] proposed a new weighted combination method that effectively improves the classification accuracy.Along the idea of fine modeling, the features or attributes all depict different classes of samples from themselves angles with the form of respective evidence. If the feature “A” and the feature “B” are related to each other, then the corresponding evidence “A” and the evidence “B” must be related. Generally speaking, there are inevitable or potential connections among different features; hence, if we can dig such interdependence between the evidence “A” and the evidence “B” when they all describe the class of a certain sample, then evidence combination with the defined correlation information will produce more accurate classification results than those without considering evidence interdependence. Unfortunately, most of the classic rules mentioned above require that the evidence “A” and the evidence “B” be independent of each other or do not discuss such evidence interdependence problem.The newly proposed maximum likelihood evidential reasoning (MAKER) rule is the extended version of the ER rule and measures such ubiquitous interdependence by introducing correlation factors in evidence combination [32–34]. Hence, this paper designs a MAKER-based classifier to mine more correlation information for data classification. Firstly, some sample space models (SSMs) are constructed to describe the relationship between a single feature or multiple related features of training samples and their class labels, and then the corresponding evidence space models (ESMs) are generated by normalizing the likelihood functions about features and class labels in SSMs. Secondly, the correlation factors including the joint reliability and the interdependence degree, can be calculated by the generated ESMs. Thirdly, for the testing sample without a class label, the MAKER rule with correlation factors is used to fuse the corresponding pieces of evidence about the testing sample. Based on the fused results, a classification decision can be made to estimate the class label that the testing sample points to. Finally, some classification experiments are carried out using five popular benchmark databases from the UCI to illustrate that the refined measure for evidence interdependence can aggregate the fused belief degree into real class label of a sample and further improve classification accuracy.The rest of the article is organized as follows. Section2 introduces the MAKER rule. Section 3 focuses on the design of MAKER rule-based classifier. In Section 4, the effectiveness of the proposed classifier is verified by classification experiments. Section 5 summarizes the advantages and innovations of this work.


## 2. Preliminary

Suppose Θ = {F1, F2,…, FN} is a set in which all elements are mutually exclusive and independent with each other. If the set is a finite set and nonempty set, then Θ is called the frame of discernment (FoD). The power set of Θ is generally expressed in the form of P(Θ) or 2Θ.In MAKER rule, a piece of evidence generally represented by a basic probability assignment (BPA) function [35], as shown in the following equation:(1)ekj=θ,pθ,kj,θ⊆Θ,∑θ⊆Θpθ,kj=1,where ekj represents the k-th piece of evidence provided by the j-th feature, its element (θ, pθ,kj) indicates the probability mass that the evidence ek,j points to the proposition θ is pθ,kj.In addition, letrkj be the reliability of the evidence ekj, the inherent properties of the evidence itself, reflecting the ability that the j-th feature and its evidence to give correct classification results. Let wkj be the weight of the evidence ekj, reflecting the relative importance of ekj compared with other evidence. The weight of the evidence can be set subjectively and optimized by the training samples. The reliability of the evidence is obtained by the following equation:(2)rkj=∑θ⊆Θrθ,kjpθ,kj,here rθ,kj can be regarded as the conditional probability that proposition θ is true when the evidence ekj points to proposition θ.For two pieces of evidenceekj and eil, the probability mass of their joint support for θ is obtained by MAKER rule [32]:(3)pθ,e2=0,θ=∅,m˜θ,e2∑θ⊆Θm˜θ,e2,θ⊆Θ,θ≠∅,m˜θ,e2=1−rilmθ,kj+1−rkjmθ,il+∑B∩C=θγB,C,k,ij,lμB,C,k,ij,lmB,kjmC,il,∀θ⊆Θ,here mθ,kj=wθ,kjpθ,kj and mθ,il=wθ,ilpθ,il. wθ,kj represents the weight of pθ,kj. γB,C,k,ij,l is a non-negative correlation factor, the ratio of the joint reliability of ekj and eil to the product of their respective reliability, which can be trained by the optimization model(4)γB,C,k,ij,l=rk,ij,lrkjril,μB,C,k,ij,l is another correlation factor, the interdependence degree of ekj and eil defined by(5)μB,C,k,ij,l=0,ifpB,kj=0orpC,il=0,pB,C,k,ij,lpB,kjpC,ilotherwise,here pB,C,k,ij,l is the joint probability mass that ekj points to the class B and eil points to the class C. For the above rule, γB,C,k,ij,l=μB,C,k,ij,l=1 means that ekj and eil are independent of each other, MAKER rule is simplified to ER rule [36].ForJ pieces of evidence ej (j = 1.2,…, J) from J different information sources or features, MAKER rule can recursively fuse them [32].(6)pθ,eJ=0,θ=∅,m˜θ,eJ∑θ⊆Θm˜θ,eJ,θ⊆Θ,θ≠∅,m˜θ,ej=1−rjmθ,eJ−1+mPΘ,ej−1mθ,j+∑B∩C=θγB,C,e1,…,eJμB,C,e1,…,eJmB,ej−1mC,j,∀θ⊆Θ,m˜PΘ,eJ=1−rJmPΘ,eJ−1.


## 3. The Design of MAKER Rule-Based Classifier

### 3.1. Framework

To mine more useful information from the dataset and further improve the performance of the classifier, the following six-step framework is proposed. To reduce the data dimension, model complexity, and computation, Section3.2 first introduces the PCA-based feature selection method. To describe the subtle changes of class label with input variable, Section 3.3 exhibits the construction methods of SSMs and ESMs based on likelihood function normalization and estimation of correlation factors. To obtain accurate class label estimates for test samples, Section 3.4 describes the process in detail, including evidence activation and combination and classification decisions. To further improve the classification accuracy of the MAKER model, Section 3.5 proposes a genetic algorithm (GA)-based optimization method for MAKER model parameters. The whole process of the designed MAKER classifier is shown in Figure 1.Step 1 (Feature selection based on PCA): Based on the existing PCA technology, feature selection is performed on the sample data set containing features f1∼fL, and a simplified sample set consisting of principal component features x1∼xJ is obtained. More details are presented in Section 3.2.Step 2 (Constructions of SSMs & ESMs): Based on the sample data set obtained in step 1, the sample space model (SSM) and evidence space model (ESM) are constructed by calculating the similarity distribution of each sample pair in the sample data, which is used for the transformation of feature variables into reference classification evidence and realizes the accurate description of the coarse statistical characteristics of the sample data. More details can be found in Section 3.3.1.Step 3 (Calculation of correlation factor: interdependence degree): Before using the MAKER rule to fuse reference classification evidence, it is necessary to estimate the correlation factor between the reference evidence, that is, the interdependence degree. More details can be found in Section 3.3.2.Step 4 (Evidence activation and combination): The test sample of xt′ activates the ESM to obtain the activated reference classification evidence. And then MAKER rule is used to fuse the activated reference classification evidence to obtain local evidence. More details are presented in Section 3.4.1.Step 5 (Classification decision based on ER Rule): After obtaining multiple pieces of local evidence, classification decision evidence can be obtained based on ER rules. Based on the given classification decision criteria, an accurate classification decision is made according to the classification evidence O(xt′). More details can be found in Section 3.4.2.Step 6 (Parameter optimization of MAKER-based classifier via GA): The minimum means square error (MSE) between the estimated probability mass p^n,t and the real or ideal probability mass pn,t of the sample class label is taken as the objective function, and GA is used to globally optimize the parameters of the MAKER model to improve the classification accuracy of the MAKER model. More details are presented in Section 3.5.Figure 1
Implementation flowchart of MAKER classifier algorithm.


### 3.2. Feature Selection Based on PCA

Step 1.
For the sample withL features, each feature input activates two pieces of referential evidence in the ESM, and then the L input variables will activate 2L pieces of referential evidence. Subsequently, the MAKER rule will be used 2L times to combine them. Hence, the time required for evidence combination will increase exponentially with the number of input features. On the other hand, there will be more parameters optimized in the model optimization. It will significantly increase the calculation complexity. Therefore, this subsection reduces the dimension of the sample vector to less than four features via the PCA method.
Based on PCA, feature selection can be performed on the sample data set containing the featuresf1∼fL, and the principal component features x1∼xJ can be selected to form a new simplified sample data set, where J < L.Remark 1.
PCA is a commonly used multivariate statistical method, which can effectively extract features of the sample vector. The extracted feature (the principal component feature) obtained by PCA is a linear combination of the original features, and the principal components can be sorted according to the contribution rates from large to small, which can retain the important features of high-dimensional data and remove noise or unimportant features so as to improve the speed of data processing [37]. This extraction process is as follows: (1) standardize the value range of each feature in the original data sample set. (2) calculate the covariance matrix about the standardized features; (3) calculate the eigenvectors and eigenvalues of the covariance matrix; (4) arrange the eigenvalues in descending order and calculate the principal component matrix; (5) select principal components with eigenvalues greater than 1.


### 3.3. Constructions of SSMs and ESMs and Calculation of Correlation Factors

#### 3.3.1. Constructions of SSMs and ESMs

Step 2.
Let Θ = {F1,…,Fn,…,FN} is a class label set, including all possible categories of samples. The simplified sample set is X = {x1,…,xt,…,xT}, here the t-th training sample vector can be expressed as xt = [x1 (t),…,xj (t),…,xJ (t), F (t)], here xj(t) is the input of MAKER classifier, representing the sampling value of the j-th input feature xj, and F(t)∈Θ is the output of MAKER classifier, representing the class label that xt belongs to.Step 3.
For the training samples withJ different features, their j-th principal component feature has a set of reference values Aj = {Akj|k = 1.2, …, K}, K is the number of reference values, the initial value of Akj can be determined by expert experiences, and then adjusted by the following optimization model in Section 3.5. Based on such Aj, the relationship between input feature xj(t) of a sample vector xt and its reference values can be transformed into the form of belief distribution αkjt,αk+1jt [32], αkjt is the similarity between xj(t) and Akj, expressed by the piecewise linear function(7)αkjt=Ak+1j−xjtAk+1j−Akj,αk+1jt=1−αkjtAkj≤xjt≤Ak+1j,αk′jt=0,k′=1,K,k′≠k,k+1.Step 4.
By (7), the similarity distribution of all sample pairs (xj(t), F(t)) extracted from xt = [x1(t), …, xj (t),…,xJ (t), F(t)] can be projected into a sample space model (SSM) for describing statistical relationship between Akj and output Fn , as shown in Table 1.
Wherean,kj=∑t=1Tαkjt is the sum of the similarities in (7) that the sample pair (xj(t), F(t)) matches the reference value Akj and belongs to the class Fn. δn=∑k=1Kan,kj represents the sum of the similarities pointing to Fn. And ηk=∑n=1Nan,kj represents the sum of the similarities pointing to the k-th reference value Akj and ∑n=1Nδn=∑k=1Kηk=T.Table 1
The sample space model (SSM) for the input featurexj.
FxjA1j…Akj…AkjTotalF1a1,1j…a1,kj…a1,Kjδ1⋮⋮⋮⋮⋮⋮⋮Fnan,1j…an,kj…an,Kjδn⋮⋮⋮⋮⋮⋮⋮FNaN,1j…aN,kj…aN,KjδNTotalη1…ηk…ηKTStep 5.
The likelihood function can now be constructed as follows:(8)LAkj|Fn=an,kjδn.
It is the likelihood probability ofAkj under the occurrence of Fn.
Based on SSM in Table1 and the likelihood probability in (8), the corresponding evidence space model (ESM) about the feature xj can be generated, as shown in Table 2.
For each reference valueAkj, there is a reference evidence ekj to indicate the probability mass pn,kj pointing to the class label F ∈ Θ when xj = Akj, pn,kj can be obtained by the normalization operation of the likelihood functions as follows:(9)pn,kj=LAkj|Fn∑n=1NLAkj|Fn.
It is obvious that∑n=1Npn,kj=1. According to the definition of evidence in (1), the reference evidence ekj can be presented as follows:(10)ekj=F1,p1,kj,F2,p2,kj,FN,pN,kj.
For the sake of simplicity,ekj is abbreviated as ekj = [p1,kj, p2,kj,…, pN,kj]. As a result, the ESM can be regarded as a reference evidence matrix (REM) about xj.Table 2
The ESM for the featurexj.
Fxje1j…ekj…ekjA1j…Akj…AkjF1p1,1j…p1,kj…p1,Kj⋮⋮⋮⋮⋮⋮Fnpn,1j…pn,kj…pn,Kj⋮⋮⋮⋮⋮⋮FNpN,1j…pN,kj…pN,KjTotal1…1…1


#### 3.3.2. Calculation of Correlation Factor: Interdependence Degree

Step 6.
Considering the inevitable or potential connections among all features and their reference evidence, the interdependence degree can be calculated as follows:(11)μn,k1,…,kj,…,kJ1,…,j,…,J=0,ifpn,k11pn,k22…pn,kJJ=0,pn,k1,…,kj,…,kJ1,…,j,…,Jpn,k11pn,k22pn,kJJ,otherwise,where pn,k1,…,kj,…,kJ1,…,j,…,J represents the joint probability mass of all J input features pointing to the class Fn, which needs to be calculated by constructing joint SSM. In detail, the sample vector xt is transformed into the form of the joint similarity αk1,…,kj,…,kJ1,…,j,…J for the reference value combination Ak1,…,kj,…,kJ1,…,j,…,J=Ak11,…,Akjj,…,AkJJ(12)αk1,…,kj,…,kJ1,…,j,…Jt=αk11t…αkjjt…αkJJt.
It is the product of the similarity in (7) that each input feature xj′ matches its reference value Akj.Step 7.
Then, all training sample vector are transformed into the relationships between the input reference value combinationAk1,…,kj,…,kJ1,…,j,…J and the output class labels, and the SSM can be constructed by using the same procedure given in Section 3.3.1. Here, Table 3 takes the double features [xj, xl] as an example to illustrate the constructed SSM.
Wherean,kj,klj,l=∑t=1Tαkj,klj,lt is the sum of the joint similarities of all sample [xj, xl] belonging to Fn and matching the reference value combination Akj,klj,l=Akjj,Akll.Table 3
The joint SSM for the double features [xj, xl].
F[xj, xl]A1j…AKjjTotalA1l…Akll…AKllA1l…Akll…AKllF1a1,1,1j,l…a1,1,klj,l…a1,1,Klj,l…a1,KJ,1j,l…a1,KJ,klj,l…a1,KJ,Klj,lδ1⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮Fnan,1,1j,l…an,1,klj,l…an,1,Klj,l…an,KJ,1j,l…an,KJ,klj,l…an,KJ,Klj,lδn⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮FNaN,1,1j,l…aN,1,klj,l…aN,1,Klj,l…aN,KJ,1j,l…aN,KJ,klj,l…aN,KJ,Klj,lδNTotalη1,1…η1,kl…η1,Kl…ηKj,1…ηKj,kl…ηKj,KlTStep 8.
By using (8), the likelihood probability of Akj,klj,l under the occurrence of Fn can be obtained. By the normalization operation of the likelihood functions in (9), the ESM of the SSM in Table 3 can be constructed as listed in Table 4.Table 4
The joint ESM about the double features [xj, xl].
F[xj, xl]A1j…AKjjA1l…Akll…AKllA1l…Akll…AKlle1,1j,l…e1,klj,l…e1,Klj,leKj,1j,l…eKj,klj,l…eKj,Klj,lF1p1,1,1j,l…p1,1,klj,l…p1,1,Klj,l…p1,Kj,1j,l…p1,Kj,klj,l…p1,Kj,Klj,l⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮Fnpn,1,1j,l…pn,1,klj,l…pn,1,Klj,l…pn,Kj,1j,l…pn,Kj,klj,l…pn,Kj,Klj,l⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮FNpN,1,1j,l…pN,1,klj,l…pN,1,Klj,l…pN,Kj,1j,l…pN,Kj,klj,l…pN,Kj,Klj,l


### 3.4. Evidence Fusion Based on MAKER Rule

#### 3.4.1. Evidence Activation and Combination

Step 9.
For any set of test samplesx′t = [x1 (t),…,xj (t),…,xJ (t)], the input feature xj(t) must belong to the two reference values [Ajk, Ajk + 1] of the j-th feature, that is Akj ≤ xj (t) ≤ Ajk + 1. Thus, a pair of reference evidence corresponding to the two reference values is activated. For example, for any two input features [xj, xl] of the input vector, they will respectively activate two pairs of reference evidence ekll,ekl+1l and ekll,ekl+1l, thus MAKER rule needs to be used 4 times “ekjj⊕ekll, ekjj⊕ekl+1l, ekj+1j⊕ekll, ekj+1j⊕ekl+1l” to respectively obtain 4 pieces of local fused evidence, here “⊕” denotes MAKER fusion operation in (6). In this way, for the input vector with J features, 2J pieces of reference evidence will be activated, MAKER rule will be used 2J times to generate 2J pieces of local fused evidence, denoted as(13)es=Fn,pn,s,n=1,…,N,s=1,2,…,2J,∑n=1Npn,s=1.Remark 2.
Considering that the weight factorwn,kj, importance factor rn,kj and correlation factor γB,C,k,ij,l of the reference evidence are adjustable, the initial values of these parameters can be assumed to be 1, i.e. wn,kj=rn,kj=γB,C,k,ij,l=1.


#### 3.4.2. Classification Decision Based on ER Rule

Step 10.
In order to make classification decision, it is necessary to further fuse 2J pieces of local evidence es, s = 1.2, …,2J. In the above fusion operation of reference evidence, we have considered the interdependence among multiple pieces of reference evidence, hence the generated local fused evidence will be never dependent with each other. Thus, ER rule applicable to independent evidence will be used to fuse 2J pieces of independent evidence es and get the final fused result as follows:(14a)Oxt′=Fn,pn,e2J,n=1,2,…,N,where pn,e2J represents the probability mass or belief degree that the input x(t) points to the class Fn. O(•) means the output of the ER-based fusion operation, the detailed recursive fusion operation as follows:(14b)pn,e2J=m˜n,e2J∑n=1Nm˜n,e2J,m˜n,es=1−Rsmn,es−1+mPΘ,es−1mn,s+∑mn,es−1mn,s,m˜PΘ,es=1−RsmPΘ,es−1.
Obviously, ER rule is the special case of MAKER rule where the correlation factorsγ and μ are equal to 1 respectively, namely 2J pieces of local fused evidence are independent with each other. Here Θ = {F1,F2,…,Fn}, mn,s = Wspn,s. Rs and Ws are the reliability factor and the importance weight of the local evidence es respectively. Rs is the inherent characteristic of es, an important index to evaluate whether the evidence es can provide accurate classification information. It is generally believed that the higher the evidence reliability is, the higher the evidence importance weight is, so it can be considered that the importance weight of evidence is equal to its reliability Rs = Ws. In recursive fusion operation, Rs=1−mPΘ,s=∑n⊆Θmn,s. The proof of Rs can be seen in Appendix.Step 11.
After getting the final fused result in Step10, the current classification evidence O(xt′) is obtained, and the following classification-decision criteria for sample xt′ = [x1(t), …, xj(t), …, xJ(t)] is given as follows:(15)ifpn,e2J≥pm,e2J,m=1,2,n−1,n+1,…,N,thenoutputF=Fn.


### 3.5. Parameter Optimization of MAKER-Based Classifier via GA

Step 12.
It is necessary to optimize the parameters such as the reference values of the input features and the reliability and weight in MAKER rules to improve the performance of the classifier. Here, the minimum means square error (MSE) in (16a) between the estimated probability mass and the real or ideal probability mass of the sample class label is taken as the objective function to construct the optimization model. Equations (16b)–(16d) represent the constraints of the optimization parameters.(16a)minPξP=1T∗N∑t=1T∑n=1Npn,t−p^n,t2,(16b)s.t.Ak′−1j<Ak′j<Ak′+1j,j=1,…,J;k′=2,…,K−1,(16c)0≤wn,kj≤1,j=1,…,J;k=1,…,K,(16d)0≤rn,kj,rn,k1,…,kj,…,kJ1,…,j,…J≤1,where ξP represents the objective function, and p^n,t is the estimated probability mass equal to pn,e2J, pn,t is ideal probability mass, pn,t = 1, when the true class label of x(t) is Fn, pm,t = 0, m = 1, 2, n − 1, n + 1, …, N.(17)P=Akj,wn,kj,rn,kj,rn,k1,⋯,kj,⋯,kJ1,⋯,j,⋯,J|j=1,…,J;k=1,…,K;k′=2,…,K−1;n=1,…,N.
It represents the set of parameters to be optimized including the reference valueAkj of the input feature, the weight factor wn,kj of the activated evidence in Table 4, the numerator rn,kj, and denominator rn,k1,…,kj,…,kJ1,…,j,…J of the reliability factor in (4)Remark 3.
We introduce a GA as the optimization engine to offline and globally optimize the parameters of MAKER model. As an evolutionary algorithm, GA has strong robustness and good global search capabilities and GA is suitable for optimization of large-scale complex problems. According to the criterion of survival, it produces a better approximate solution to the initial population evolution from generation to generation. The initial population optimized by GA is mainly composed of two parts: some individuals are generated by expert knowledge and the others are randomly abstracted within the range of the observation input value [38]. Therefore, GA is utilized for parameter optimization of MAKER-based classifier in this paper to improve optimization efficiency and classification accuracy.Remark 4.
GA carries out combinatorial crossover and mutation operations with the help of genetic operators of natural genetics, and iteratively performs a series of GA operations. Through selection, recombination, mutation, reinsertion, and migration, a new population representing a new solution set is generated. With the iterative optimization of the population, the parameter setP is continuously optimized, and the probability mass pn,kj of ESM in Table 4 will be optimized gradually with the change of the reference value Akj.


## 3.1. Framework

To mine more useful information from the dataset and further improve the performance of the classifier, the following six-step framework is proposed. To reduce the data dimension, model complexity, and computation, Section3.2 first introduces the PCA-based feature selection method. To describe the subtle changes of class label with input variable, Section 3.3 exhibits the construction methods of SSMs and ESMs based on likelihood function normalization and estimation of correlation factors. To obtain accurate class label estimates for test samples, Section 3.4 describes the process in detail, including evidence activation and combination and classification decisions. To further improve the classification accuracy of the MAKER model, Section 3.5 proposes a genetic algorithm (GA)-based optimization method for MAKER model parameters. The whole process of the designed MAKER classifier is shown in Figure 1.Step 1 (Feature selection based on PCA): Based on the existing PCA technology, feature selection is performed on the sample data set containing features f1∼fL, and a simplified sample set consisting of principal component features x1∼xJ is obtained. More details are presented in Section 3.2.Step 2 (Constructions of SSMs & ESMs): Based on the sample data set obtained in step 1, the sample space model (SSM) and evidence space model (ESM) are constructed by calculating the similarity distribution of each sample pair in the sample data, which is used for the transformation of feature variables into reference classification evidence and realizes the accurate description of the coarse statistical characteristics of the sample data. More details can be found in Section 3.3.1.Step 3 (Calculation of correlation factor: interdependence degree): Before using the MAKER rule to fuse reference classification evidence, it is necessary to estimate the correlation factor between the reference evidence, that is, the interdependence degree. More details can be found in Section 3.3.2.Step 4 (Evidence activation and combination): The test sample of xt′ activates the ESM to obtain the activated reference classification evidence. And then MAKER rule is used to fuse the activated reference classification evidence to obtain local evidence. More details are presented in Section 3.4.1.Step 5 (Classification decision based on ER Rule): After obtaining multiple pieces of local evidence, classification decision evidence can be obtained based on ER rules. Based on the given classification decision criteria, an accurate classification decision is made according to the classification evidence O(xt′). More details can be found in Section 3.4.2.Step 6 (Parameter optimization of MAKER-based classifier via GA): The minimum means square error (MSE) between the estimated probability mass p^n,t and the real or ideal probability mass pn,t of the sample class label is taken as the objective function, and GA is used to globally optimize the parameters of the MAKER model to improve the classification accuracy of the MAKER model. More details are presented in Section 3.5.Figure 1
Implementation flowchart of MAKER classifier algorithm.


## 3.2. Feature Selection Based on PCA

Step 1.
For the sample withL features, each feature input activates two pieces of referential evidence in the ESM, and then the L input variables will activate 2L pieces of referential evidence. Subsequently, the MAKER rule will be used 2L times to combine them. Hence, the time required for evidence combination will increase exponentially with the number of input features. On the other hand, there will be more parameters optimized in the model optimization. It will significantly increase the calculation complexity. Therefore, this subsection reduces the dimension of the sample vector to less than four features via the PCA method.
Based on PCA, feature selection can be performed on the sample data set containing the featuresf1∼fL, and the principal component features x1∼xJ can be selected to form a new simplified sample data set, where J < L.Remark 1.
PCA is a commonly used multivariate statistical method, which can effectively extract features of the sample vector. The extracted feature (the principal component feature) obtained by PCA is a linear combination of the original features, and the principal components can be sorted according to the contribution rates from large to small, which can retain the important features of high-dimensional data and remove noise or unimportant features so as to improve the speed of data processing [37]. This extraction process is as follows: (1) standardize the value range of each feature in the original data sample set. (2) calculate the covariance matrix about the standardized features; (3) calculate the eigenvectors and eigenvalues of the covariance matrix; (4) arrange the eigenvalues in descending order and calculate the principal component matrix; (5) select principal components with eigenvalues greater than 1.


## 3.3. Constructions of SSMs and ESMs and Calculation of Correlation Factors

### 3.3.1. Constructions of SSMs and ESMs

Step 2.
Let Θ = {F1,…,Fn,…,FN} is a class label set, including all possible categories of samples. The simplified sample set is X = {x1,…,xt,…,xT}, here the t-th training sample vector can be expressed as xt = [x1 (t),…,xj (t),…,xJ (t), F (t)], here xj(t) is the input of MAKER classifier, representing the sampling value of the j-th input feature xj, and F(t)∈Θ is the output of MAKER classifier, representing the class label that xt belongs to.Step 3.
For the training samples withJ different features, their j-th principal component feature has a set of reference values Aj = {Akj|k = 1.2, …, K}, K is the number of reference values, the initial value of Akj can be determined by expert experiences, and then adjusted by the following optimization model in Section 3.5. Based on such Aj, the relationship between input feature xj(t) of a sample vector xt and its reference values can be transformed into the form of belief distribution αkjt,αk+1jt [32], αkjt is the similarity between xj(t) and Akj, expressed by the piecewise linear function(7)αkjt=Ak+1j−xjtAk+1j−Akj,αk+1jt=1−αkjtAkj≤xjt≤Ak+1j,αk′jt=0,k′=1,K,k′≠k,k+1.Step 4.
By (7), the similarity distribution of all sample pairs (xj(t), F(t)) extracted from xt = [x1(t), …, xj (t),…,xJ (t), F(t)] can be projected into a sample space model (SSM) for describing statistical relationship between Akj and output Fn , as shown in Table 1.
Wherean,kj=∑t=1Tαkjt is the sum of the similarities in (7) that the sample pair (xj(t), F(t)) matches the reference value Akj and belongs to the class Fn. δn=∑k=1Kan,kj represents the sum of the similarities pointing to Fn. And ηk=∑n=1Nan,kj represents the sum of the similarities pointing to the k-th reference value Akj and ∑n=1Nδn=∑k=1Kηk=T.Table 1
The sample space model (SSM) for the input featurexj.
FxjA1j…Akj…AkjTotalF1a1,1j…a1,kj…a1,Kjδ1⋮⋮⋮⋮⋮⋮⋮Fnan,1j…an,kj…an,Kjδn⋮⋮⋮⋮⋮⋮⋮FNaN,1j…aN,kj…aN,KjδNTotalη1…ηk…ηKTStep 5.
The likelihood function can now be constructed as follows:(8)LAkj|Fn=an,kjδn.
It is the likelihood probability ofAkj under the occurrence of Fn.
Based on SSM in Table1 and the likelihood probability in (8), the corresponding evidence space model (ESM) about the feature xj can be generated, as shown in Table 2.
For each reference valueAkj, there is a reference evidence ekj to indicate the probability mass pn,kj pointing to the class label F ∈ Θ when xj = Akj, pn,kj can be obtained by the normalization operation of the likelihood functions as follows:(9)pn,kj=LAkj|Fn∑n=1NLAkj|Fn.
It is obvious that∑n=1Npn,kj=1. According to the definition of evidence in (1), the reference evidence ekj can be presented as follows:(10)ekj=F1,p1,kj,F2,p2,kj,FN,pN,kj.
For the sake of simplicity,ekj is abbreviated as ekj = [p1,kj, p2,kj,…, pN,kj]. As a result, the ESM can be regarded as a reference evidence matrix (REM) about xj.Table 2
The ESM for the featurexj.
Fxje1j…ekj…ekjA1j…Akj…AkjF1p1,1j…p1,kj…p1,Kj⋮⋮⋮⋮⋮⋮Fnpn,1j…pn,kj…pn,Kj⋮⋮⋮⋮⋮⋮FNpN,1j…pN,kj…pN,KjTotal1…1…1


### 3.3.2. Calculation of Correlation Factor: Interdependence Degree

Step 6.
Considering the inevitable or potential connections among all features and their reference evidence, the interdependence degree can be calculated as follows:(11)μn,k1,…,kj,…,kJ1,…,j,…,J=0,ifpn,k11pn,k22…pn,kJJ=0,pn,k1,…,kj,…,kJ1,…,j,…,Jpn,k11pn,k22pn,kJJ,otherwise,where pn,k1,…,kj,…,kJ1,…,j,…,J represents the joint probability mass of all J input features pointing to the class Fn, which needs to be calculated by constructing joint SSM. In detail, the sample vector xt is transformed into the form of the joint similarity αk1,…,kj,…,kJ1,…,j,…J for the reference value combination Ak1,…,kj,…,kJ1,…,j,…,J=Ak11,…,Akjj,…,AkJJ(12)αk1,…,kj,…,kJ1,…,j,…Jt=αk11t…αkjjt…αkJJt.
It is the product of the similarity in (7) that each input feature xj′ matches its reference value Akj.Step 7.
Then, all training sample vector are transformed into the relationships between the input reference value combinationAk1,…,kj,…,kJ1,…,j,…J and the output class labels, and the SSM can be constructed by using the same procedure given in Section 3.3.1. Here, Table 3 takes the double features [xj, xl] as an example to illustrate the constructed SSM.
Wherean,kj,klj,l=∑t=1Tαkj,klj,lt is the sum of the joint similarities of all sample [xj, xl] belonging to Fn and matching the reference value combination Akj,klj,l=Akjj,Akll.Table 3
The joint SSM for the double features [xj, xl].
F[xj, xl]A1j…AKjjTotalA1l…Akll…AKllA1l…Akll…AKllF1a1,1,1j,l…a1,1,klj,l…a1,1,Klj,l…a1,KJ,1j,l…a1,KJ,klj,l…a1,KJ,Klj,lδ1⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮Fnan,1,1j,l…an,1,klj,l…an,1,Klj,l…an,KJ,1j,l…an,KJ,klj,l…an,KJ,Klj,lδn⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮FNaN,1,1j,l…aN,1,klj,l…aN,1,Klj,l…aN,KJ,1j,l…aN,KJ,klj,l…aN,KJ,Klj,lδNTotalη1,1…η1,kl…η1,Kl…ηKj,1…ηKj,kl…ηKj,KlTStep 8.
By using (8), the likelihood probability of Akj,klj,l under the occurrence of Fn can be obtained. By the normalization operation of the likelihood functions in (9), the ESM of the SSM in Table 3 can be constructed as listed in Table 4.Table 4
The joint ESM about the double features [xj, xl].
F[xj, xl]A1j…AKjjA1l…Akll…AKllA1l…Akll…AKlle1,1j,l…e1,klj,l…e1,Klj,leKj,1j,l…eKj,klj,l…eKj,Klj,lF1p1,1,1j,l…p1,1,klj,l…p1,1,Klj,l…p1,Kj,1j,l…p1,Kj,klj,l…p1,Kj,Klj,l⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮Fnpn,1,1j,l…pn,1,klj,l…pn,1,Klj,l…pn,Kj,1j,l…pn,Kj,klj,l…pn,Kj,Klj,l⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮FNpN,1,1j,l…pN,1,klj,l…pN,1,Klj,l…pN,Kj,1j,l…pN,Kj,klj,l…pN,Kj,Klj,l


## 3.3.1. Constructions of SSMs and ESMs

Step 2.
Let Θ = {F1,…,Fn,…,FN} is a class label set, including all possible categories of samples. The simplified sample set is X = {x1,…,xt,…,xT}, here the t-th training sample vector can be expressed as xt = [x1 (t),…,xj (t),…,xJ (t), F (t)], here xj(t) is the input of MAKER classifier, representing the sampling value of the j-th input feature xj, and F(t)∈Θ is the output of MAKER classifier, representing the class label that xt belongs to.Step 3.
For the training samples withJ different features, their j-th principal component feature has a set of reference values Aj = {Akj|k = 1.2, …, K}, K is the number of reference values, the initial value of Akj can be determined by expert experiences, and then adjusted by the following optimization model in Section 3.5. Based on such Aj, the relationship between input feature xj(t) of a sample vector xt and its reference values can be transformed into the form of belief distribution αkjt,αk+1jt [32], αkjt is the similarity between xj(t) and Akj, expressed by the piecewise linear function(7)αkjt=Ak+1j−xjtAk+1j−Akj,αk+1jt=1−αkjtAkj≤xjt≤Ak+1j,αk′jt=0,k′=1,K,k′≠k,k+1.Step 4.
By (7), the similarity distribution of all sample pairs (xj(t), F(t)) extracted from xt = [x1(t), …, xj (t),…,xJ (t), F(t)] can be projected into a sample space model (SSM) for describing statistical relationship between Akj and output Fn , as shown in Table 1.
Wherean,kj=∑t=1Tαkjt is the sum of the similarities in (7) that the sample pair (xj(t), F(t)) matches the reference value Akj and belongs to the class Fn. δn=∑k=1Kan,kj represents the sum of the similarities pointing to Fn. And ηk=∑n=1Nan,kj represents the sum of the similarities pointing to the k-th reference value Akj and ∑n=1Nδn=∑k=1Kηk=T.Table 1
The sample space model (SSM) for the input featurexj.
FxjA1j…Akj…AkjTotalF1a1,1j…a1,kj…a1,Kjδ1⋮⋮⋮⋮⋮⋮⋮Fnan,1j…an,kj…an,Kjδn⋮⋮⋮⋮⋮⋮⋮FNaN,1j…aN,kj…aN,KjδNTotalη1…ηk…ηKTStep 5.
The likelihood function can now be constructed as follows:(8)LAkj|Fn=an,kjδn.
It is the likelihood probability ofAkj under the occurrence of Fn.
Based on SSM in Table1 and the likelihood probability in (8), the corresponding evidence space model (ESM) about the feature xj can be generated, as shown in Table 2.
For each reference valueAkj, there is a reference evidence ekj to indicate the probability mass pn,kj pointing to the class label F ∈ Θ when xj = Akj, pn,kj can be obtained by the normalization operation of the likelihood functions as follows:(9)pn,kj=LAkj|Fn∑n=1NLAkj|Fn.
It is obvious that∑n=1Npn,kj=1. According to the definition of evidence in (1), the reference evidence ekj can be presented as follows:(10)ekj=F1,p1,kj,F2,p2,kj,FN,pN,kj.
For the sake of simplicity,ekj is abbreviated as ekj = [p1,kj, p2,kj,…, pN,kj]. As a result, the ESM can be regarded as a reference evidence matrix (REM) about xj.Table 2
The ESM for the featurexj.
Fxje1j…ekj…ekjA1j…Akj…AkjF1p1,1j…p1,kj…p1,Kj⋮⋮⋮⋮⋮⋮Fnpn,1j…pn,kj…pn,Kj⋮⋮⋮⋮⋮⋮FNpN,1j…pN,kj…pN,KjTotal1…1…1


## 3.3.2. Calculation of Correlation Factor: Interdependence Degree

Step 6.
Considering the inevitable or potential connections among all features and their reference evidence, the interdependence degree can be calculated as follows:(11)μn,k1,…,kj,…,kJ1,…,j,…,J=0,ifpn,k11pn,k22…pn,kJJ=0,pn,k1,…,kj,…,kJ1,…,j,…,Jpn,k11pn,k22pn,kJJ,otherwise,where pn,k1,…,kj,…,kJ1,…,j,…,J represents the joint probability mass of all J input features pointing to the class Fn, which needs to be calculated by constructing joint SSM. In detail, the sample vector xt is transformed into the form of the joint similarity αk1,…,kj,…,kJ1,…,j,…J for the reference value combination Ak1,…,kj,…,kJ1,…,j,…,J=Ak11,…,Akjj,…,AkJJ(12)αk1,…,kj,…,kJ1,…,j,…Jt=αk11t…αkjjt…αkJJt.
It is the product of the similarity in (7) that each input feature xj′ matches its reference value Akj.Step 7.
Then, all training sample vector are transformed into the relationships between the input reference value combinationAk1,…,kj,…,kJ1,…,j,…J and the output class labels, and the SSM can be constructed by using the same procedure given in Section 3.3.1. Here, Table 3 takes the double features [xj, xl] as an example to illustrate the constructed SSM.
Wherean,kj,klj,l=∑t=1Tαkj,klj,lt is the sum of the joint similarities of all sample [xj, xl] belonging to Fn and matching the reference value combination Akj,klj,l=Akjj,Akll.Table 3
The joint SSM for the double features [xj, xl].
F[xj, xl]A1j…AKjjTotalA1l…Akll…AKllA1l…Akll…AKllF1a1,1,1j,l…a1,1,klj,l…a1,1,Klj,l…a1,KJ,1j,l…a1,KJ,klj,l…a1,KJ,Klj,lδ1⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮Fnan,1,1j,l…an,1,klj,l…an,1,Klj,l…an,KJ,1j,l…an,KJ,klj,l…an,KJ,Klj,lδn⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮FNaN,1,1j,l…aN,1,klj,l…aN,1,Klj,l…aN,KJ,1j,l…aN,KJ,klj,l…aN,KJ,Klj,lδNTotalη1,1…η1,kl…η1,Kl…ηKj,1…ηKj,kl…ηKj,KlTStep 8.
By using (8), the likelihood probability of Akj,klj,l under the occurrence of Fn can be obtained. By the normalization operation of the likelihood functions in (9), the ESM of the SSM in Table 3 can be constructed as listed in Table 4.Table 4
The joint ESM about the double features [xj, xl].
F[xj, xl]A1j…AKjjA1l…Akll…AKllA1l…Akll…AKlle1,1j,l…e1,klj,l…e1,Klj,leKj,1j,l…eKj,klj,l…eKj,Klj,lF1p1,1,1j,l…p1,1,klj,l…p1,1,Klj,l…p1,Kj,1j,l…p1,Kj,klj,l…p1,Kj,Klj,l⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮Fnpn,1,1j,l…pn,1,klj,l…pn,1,Klj,l…pn,Kj,1j,l…pn,Kj,klj,l…pn,Kj,Klj,l⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮FNpN,1,1j,l…pN,1,klj,l…pN,1,Klj,l…pN,Kj,1j,l…pN,Kj,klj,l…pN,Kj,Klj,l


## 3.4. Evidence Fusion Based on MAKER Rule

### 3.4.1. Evidence Activation and Combination

Step 9.
For any set of test samplesx′t = [x1 (t),…,xj (t),…,xJ (t)], the input feature xj(t) must belong to the two reference values [Ajk, Ajk + 1] of the j-th feature, that is Akj ≤ xj (t) ≤ Ajk + 1. Thus, a pair of reference evidence corresponding to the two reference values is activated. For example, for any two input features [xj, xl] of the input vector, they will respectively activate two pairs of reference evidence ekll,ekl+1l and ekll,ekl+1l, thus MAKER rule needs to be used 4 times “ekjj⊕ekll, ekjj⊕ekl+1l, ekj+1j⊕ekll, ekj+1j⊕ekl+1l” to respectively obtain 4 pieces of local fused evidence, here “⊕” denotes MAKER fusion operation in (6). In this way, for the input vector with J features, 2J pieces of reference evidence will be activated, MAKER rule will be used 2J times to generate 2J pieces of local fused evidence, denoted as(13)es=Fn,pn,s,n=1,…,N,s=1,2,…,2J,∑n=1Npn,s=1.Remark 2.
Considering that the weight factorwn,kj, importance factor rn,kj and correlation factor γB,C,k,ij,l of the reference evidence are adjustable, the initial values of these parameters can be assumed to be 1, i.e. wn,kj=rn,kj=γB,C,k,ij,l=1.


### 3.4.2. Classification Decision Based on ER Rule

Step 10.
In order to make classification decision, it is necessary to further fuse 2J pieces of local evidence es, s = 1.2, …,2J. In the above fusion operation of reference evidence, we have considered the interdependence among multiple pieces of reference evidence, hence the generated local fused evidence will be never dependent with each other. Thus, ER rule applicable to independent evidence will be used to fuse 2J pieces of independent evidence es and get the final fused result as follows:(14a)Oxt′=Fn,pn,e2J,n=1,2,…,N,where pn,e2J represents the probability mass or belief degree that the input x(t) points to the class Fn. O(•) means the output of the ER-based fusion operation, the detailed recursive fusion operation as follows:(14b)pn,e2J=m˜n,e2J∑n=1Nm˜n,e2J,m˜n,es=1−Rsmn,es−1+mPΘ,es−1mn,s+∑mn,es−1mn,s,m˜PΘ,es=1−RsmPΘ,es−1.
Obviously, ER rule is the special case of MAKER rule where the correlation factorsγ and μ are equal to 1 respectively, namely 2J pieces of local fused evidence are independent with each other. Here Θ = {F1,F2,…,Fn}, mn,s = Wspn,s. Rs and Ws are the reliability factor and the importance weight of the local evidence es respectively. Rs is the inherent characteristic of es, an important index to evaluate whether the evidence es can provide accurate classification information. It is generally believed that the higher the evidence reliability is, the higher the evidence importance weight is, so it can be considered that the importance weight of evidence is equal to its reliability Rs = Ws. In recursive fusion operation, Rs=1−mPΘ,s=∑n⊆Θmn,s. The proof of Rs can be seen in Appendix.Step 11.
After getting the final fused result in Step10, the current classification evidence O(xt′) is obtained, and the following classification-decision criteria for sample xt′ = [x1(t), …, xj(t), …, xJ(t)] is given as follows:(15)ifpn,e2J≥pm,e2J,m=1,2,n−1,n+1,…,N,thenoutputF=Fn.


## 3.4.1. Evidence Activation and Combination

Step 9.
For any set of test samplesx′t = [x1 (t),…,xj (t),…,xJ (t)], the input feature xj(t) must belong to the two reference values [Ajk, Ajk + 1] of the j-th feature, that is Akj ≤ xj (t) ≤ Ajk + 1. Thus, a pair of reference evidence corresponding to the two reference values is activated. For example, for any two input features [xj, xl] of the input vector, they will respectively activate two pairs of reference evidence ekll,ekl+1l and ekll,ekl+1l, thus MAKER rule needs to be used 4 times “ekjj⊕ekll, ekjj⊕ekl+1l, ekj+1j⊕ekll, ekj+1j⊕ekl+1l” to respectively obtain 4 pieces of local fused evidence, here “⊕” denotes MAKER fusion operation in (6). In this way, for the input vector with J features, 2J pieces of reference evidence will be activated, MAKER rule will be used 2J times to generate 2J pieces of local fused evidence, denoted as(13)es=Fn,pn,s,n=1,…,N,s=1,2,…,2J,∑n=1Npn,s=1.Remark 2.
Considering that the weight factorwn,kj, importance factor rn,kj and correlation factor γB,C,k,ij,l of the reference evidence are adjustable, the initial values of these parameters can be assumed to be 1, i.e. wn,kj=rn,kj=γB,C,k,ij,l=1.


## 3.4.2. Classification Decision Based on ER Rule

Step 10.
In order to make classification decision, it is necessary to further fuse 2J pieces of local evidence es, s = 1.2, …,2J. In the above fusion operation of reference evidence, we have considered the interdependence among multiple pieces of reference evidence, hence the generated local fused evidence will be never dependent with each other. Thus, ER rule applicable to independent evidence will be used to fuse 2J pieces of independent evidence es and get the final fused result as follows:(14a)Oxt′=Fn,pn,e2J,n=1,2,…,N,where pn,e2J represents the probability mass or belief degree that the input x(t) points to the class Fn. O(•) means the output of the ER-based fusion operation, the detailed recursive fusion operation as follows:(14b)pn,e2J=m˜n,e2J∑n=1Nm˜n,e2J,m˜n,es=1−Rsmn,es−1+mPΘ,es−1mn,s+∑mn,es−1mn,s,m˜PΘ,es=1−RsmPΘ,es−1.
Obviously, ER rule is the special case of MAKER rule where the correlation factorsγ and μ are equal to 1 respectively, namely 2J pieces of local fused evidence are independent with each other. Here Θ = {F1,F2,…,Fn}, mn,s = Wspn,s. Rs and Ws are the reliability factor and the importance weight of the local evidence es respectively. Rs is the inherent characteristic of es, an important index to evaluate whether the evidence es can provide accurate classification information. It is generally believed that the higher the evidence reliability is, the higher the evidence importance weight is, so it can be considered that the importance weight of evidence is equal to its reliability Rs = Ws. In recursive fusion operation, Rs=1−mPΘ,s=∑n⊆Θmn,s. The proof of Rs can be seen in Appendix.Step 11.
After getting the final fused result in Step10, the current classification evidence O(xt′) is obtained, and the following classification-decision criteria for sample xt′ = [x1(t), …, xj(t), …, xJ(t)] is given as follows:(15)ifpn,e2J≥pm,e2J,m=1,2,n−1,n+1,…,N,thenoutputF=Fn.


## 3.5. Parameter Optimization of MAKER-Based Classifier via GA

Step 12.
It is necessary to optimize the parameters such as the reference values of the input features and the reliability and weight in MAKER rules to improve the performance of the classifier. Here, the minimum means square error (MSE) in (16a) between the estimated probability mass and the real or ideal probability mass of the sample class label is taken as the objective function to construct the optimization model. Equations (16b)–(16d) represent the constraints of the optimization parameters.(16a)minPξP=1T∗N∑t=1T∑n=1Npn,t−p^n,t2,(16b)s.t.Ak′−1j<Ak′j<Ak′+1j,j=1,…,J;k′=2,…,K−1,(16c)0≤wn,kj≤1,j=1,…,J;k=1,…,K,(16d)0≤rn,kj,rn,k1,…,kj,…,kJ1,…,j,…J≤1,where ξP represents the objective function, and p^n,t is the estimated probability mass equal to pn,e2J, pn,t is ideal probability mass, pn,t = 1, when the true class label of x(t) is Fn, pm,t = 0, m = 1, 2, n − 1, n + 1, …, N.(17)P=Akj,wn,kj,rn,kj,rn,k1,⋯,kj,⋯,kJ1,⋯,j,⋯,J|j=1,…,J;k=1,…,K;k′=2,…,K−1;n=1,…,N.
It represents the set of parameters to be optimized including the reference valueAkj of the input feature, the weight factor wn,kj of the activated evidence in Table 4, the numerator rn,kj, and denominator rn,k1,…,kj,…,kJ1,…,j,…J of the reliability factor in (4)Remark 3.
We introduce a GA as the optimization engine to offline and globally optimize the parameters of MAKER model. As an evolutionary algorithm, GA has strong robustness and good global search capabilities and GA is suitable for optimization of large-scale complex problems. According to the criterion of survival, it produces a better approximate solution to the initial population evolution from generation to generation. The initial population optimized by GA is mainly composed of two parts: some individuals are generated by expert knowledge and the others are randomly abstracted within the range of the observation input value [38]. Therefore, GA is utilized for parameter optimization of MAKER-based classifier in this paper to improve optimization efficiency and classification accuracy.Remark 4.
GA carries out combinatorial crossover and mutation operations with the help of genetic operators of natural genetics, and iteratively performs a series of GA operations. Through selection, recombination, mutation, reinsertion, and migration, a new population representing a new solution set is generated. With the iterative optimization of the population, the parameter setP is continuously optimized, and the probability mass pn,kj of ESM in Table 4 will be optimized gradually with the change of the reference value Akj.


## 4. Experiments

This section will use the classic “Seeds” database in the field of machine learning to illustrate the specific classification process based on MAKER rules. Furthermore, for “Seeds” database and other classic four databases of UCI benchmarks [39], a series of comparative experiments are carried out for the proposed classifier and other classical classifiers to demonstrate the effectiveness of MAKER-based classifier.


### 4.1. Seeds Data Set Classification Experiment

There are 210 samples in the Seeds database, each sample has 7 input features including area (f1), perimeter (f2), compactness (f3), length of kernel (f4), width of kernel (f5), asymmetry coefficient (f6), and length of kernel groove (f7). These 7 features can be used to jointly identify the seeds of three different wheats, then FoD can be determined as Θ = {F1, F2, F3}. The detailed experiment steps are as follows.


#### 4.1.1. Step 1: Feature Selection Based on PCA

We use the PCA method in Section3.2 to abstract the first 3 principal component features from 7 features of 210 samples, denoted as x1, x2, and x3 with their eigenvalues greater than 1. Thus, a condensed sample xt = [x1(t), x2(t), x3(t), F(t)] after PCA feature selection can be obtained, and F(t)∈Θ = {F1, F2, F3}. These sample vectors compose the sample set X = {x1,…, xt,…, xT}, T = 210. In this experiment, we randomly select 60% of the sample data of every class as the training dataset and the remaining 40% as the testing dataset.


#### 4.1.2.  Step 2: Construction of SSMs and ESMs

For the featuresx1, x2, and x3, their initial reference value sets need be given as A1 = {−3.3, −0.7.1.9.4.5} for x1, A2 = {−2.8, −1, 0.8, 2.55} for x2, A3 = {−1.61, −0.1, 1.4, 2.93} for x3 via experts’ knowledge. By using (7) in Section 3.3.1, the similarity between the training samples and its reference values can be calculated. By the casting operation, the initial SSMs of x1, x2, and x3 are constructed. Based on the constructed SSM and the likelihood probability in (8), the corresponding ESMs for x1, x2, x3 are generated, as shown in Tables 5–7, respectively.Table 5
The initial ESM for the input featurex1.
Fx1e11e21e31e41−3.3−0.71.94.5F10.09650.58230.30280F200.06880.69721F30.90350.348900Table 6
The initial ESM for the input featurex2.
Fx2e12e22e32e42−2.8−10.82.55F10.89320.50700.13330F20.02000.29590.43200.2616F30.08680.19710.43470.7384Table 7
The initial ESM for the input featurex3.
Fx3e13e23e33e43−1.61−0.11.42.93F10.23750.36270.33420.3219F20.37520.33810.29930.1788F30.38730.29920.36650.4993


#### 4.1.3.  Step 3: Calculation of the Correlation Factor: Interdependence Degree

According to Section 3.3.2, when the interdependence degreeμn,k1,…,kj,…,kJ1,…,j,…,J in (11) is calculated, the joint probability masses of the double feature [x1, x2] and the triple feature [x1, x2, x3] must be obtained by constructing joint SSMs and normalizing the corresponding likelihood functions. Table 8 lists the generated joint ESM for the double features [x1, x2] based on the reference value combination of x1′ and x2′. For the triple feature [x1, x2, x3], its EMS can be obtained by same way, here we will not illustrate this EMS matrix because it has too many (4 × 4 × 4 × 3) elements. As a result, all initial interdependence degree factors μn,k1,…,kj,…,kJ1,…,j,…,J (J = 3; n = 1, 2, 3; k1, k2, k3 ∈ {1, 2, 3, 4}) can be calculated by division operation about the elements in Tables 5–7, 9.Table 8
The optimized ESM for the input featurex1.
Fx1e11e21e31e41−3.3−0.97580.44104.5F10.07590.49380.50090.0612F2000.49910.9388F30.92410.506200Table 9
The initial joint ESM for the double features [x1, x2].
F[x1, x2]A11A12A13A14A21A22A32A42A12A22A32A42A12A22A32A42A12A22A32A42e1,21,1e1,21,2e1,21,3e1,21,4e1,22,1e1,22,2e1,22,3e1,22,4e1,23,1e1,23,2e1,23,3e1,23,4e1,24,1e1,24,2e1,24,3e1,24,4F10.58020.22180.044300.90540.74650.319200.95790.43510.086100000F200000.01420.05240.11230.06170.04210.56490.913911111F30.41980.77820.955710.08040.20110.56850.938300000000


#### 4.1.4.  Step 4: Evidence Activation and Combination

For any one test samplext′ = [x1(t), x2(t), x3(t)], each feature will activate two pieces of adjacent reference evidence, and then the MAKER rule will be used 23 times to generate 23 pieces of local fused evidence. In MAKER-based fusion operation in (3), the weight factor wn,kj of the activated evidence, the numerator rn,kj and denominator rn,k1,…,kj,…,kJ1,…,j,…J of the reliability factor are all initialized as 1.


#### 4.1.5.  Step 5: Classification Decision Based on the ER Rule

Next, 23 pieces of local fused evidence are fused by the ER rule, the classification decision can be made based the fused result.


#### 4.1.6.  Step 6: Parameter Optimization of MAKER-Based Classifier via GA

Based on the optimization model given in Section3.5, the parameter set is P=Ak′j,wn,kj,rn,kj,rn,k1,…,kj,…,kJ1,…,j,…J|j=1,2,3;k=1,2,3,4;k′=2,3;n=1,2,3 including total 318 parameters, namely Ak′j (3 × 2 = 6), wn,kj (12 × 3 = 36), rn,kj (12 × 3 = 36), rn,k1,…,kj,…,kJ1,…,j,…J (4 × 4 × 3 + 4 × 4 × 4 × 3 = 240). It can be seen that these adjustable parameters all correspond to the ESMs for single feature Tables 5–7 and the joint ESMs in Table 9 for the double and triple features. The following Tables 8, 10–12 , respectively, present the optimized ESMs. After optimization, the reference values of each input feature have changed significantly, and at the same time, the elements of these ESMs have also changed greatly with the optimized reference values.Table 10
The optimized ESM for the input featurex2.
Fx2e11e21e31e41−2.8−0.9818−0.27242.55F10.89320.67810.18340.0616F20.02040.21870.42350.3773F30.08640.10320.39310.5611Table 11
The optimized ESM for the input featurex3.
Fx3e11e21e31e41−1.61−0.80471.55032.93F10.14700.34620.35270.3077F20.37940.34560.31260.1328F30.47360.30820.33470.5595Table 12
The optimized joint ESM for the double features [x1, x2].
F[x1, x2]A11A12A13A14A21A22A32A42A12A22A32A42A12A22A32A42A12A22A32A42e1,21,1e1,21,2e1,21,3e1,21,4e1,22,1e1,22,2e1,22,3e1,22,4e1,23,1e1,23,2e1,23,3e1,23,4e1,24,1e1,24,2e1,24,3e1,24,4F10.55130.31910.06530.00780.85690.82050.34800.12650.96040.74120.25980.12640.96430.22980.02110F2000000000.03960.25880.74020.87360.03570.77020.97891F30.44870.68090.93470.99220.14310.17950.65200.873500000000Through the above five steps, the optimal MAKER-based classifier is designed. Here, a test sample vectorxt′ = [x1 (t), x2 (t), x3 (t)] = [−0.9292, −0.8718, 0.9465] with the class label F1 is randomly selected from the test sample set to illustrate the process of evidence fusion and classification decision in detail.The three input eigenvaluesx1(t), x2(t), and x3(t) of the test sample vector xt′ will activate three pairs of reference evidence including {e21, e31} in Table 8, {e22, e32} in Table 10, {e3 2, e3 3} in Table 11. According to MAKER fusion process shown in Figure 2, for the double feature [x1, x2], the corresponding interdependence degrees used in fusion operations “e21⊕ e22,” “e21⊕ e22” “e31⊕ e22,” “e31⊕ e32” can be obtained by the joint ESM in Table 13 and equation (11) as follows:(18)μ2,21,2=μ1,2,21,2,μ2.2,21,2,μ3,2,21,2=2.45,0,3.44,μ2,31,2=μ1,2,31,2,μ2,2,31,2,μ3,2,31,2=3.84,0,3.28,μ3,21,2=μ1,3,21,2,μ2,3,21,2,μ3,3,21,2=2.18,3.37,0,μ3,31,2=μ1,3,31,2,μ2,3,31,2,μ3,3,31,2=2.83,3.50,0.Figure 2
MAKER fusion process of reference evidence activated by the input vectorx.Table 13
The confusion matrix of the testing dataset.
True classClassification resultsF1F2F3F12422F20280F31027In the same way, based on the joint ESM for the triple feature [x1, x2, x3], the interdependence degrees of “e21⊕ e22⊕ e23,” “e21⊕ e32⊕ e23”,…, “e31⊕ e32⊕ e33” are as follows:(19)μ2,2,21,2,3=μ1,2,2,21,2,3,μ2,2,2,21,2,3,μ3,2,2,21,2,3=7.42,0,8.66,μ2,2,31,2,3=μ1,2,2,31,2,3,μ2,2,2,31,2,3,μ3,2,2,31,2,3=7.01,0,9.85,μ2,3,21,2,3=μ1,2,3,21,2,3,μ2,2,3,21,2,3,μ3,2,3,21,2,3=12.74,0,9.79,μ2,3,31,2,3=μ1,2,3,31,2,3,μ2,2,3,31,2,3,μ3,2,3,31,2,3=9.16,0,10.62,μ3,2,21,2,3=μ1,3,2,21,2,3,μ2,3,2,21,2,3,μ3,3,2,21,2,3=6.63,5.84,0,μ3,2,31,2,3=μ1,3,2,31,2,3,μ2,3,2,31,2,3,μ3,3,2,31,2,3=5.53,9.91,0,μ3,3,21,2,3=μ1,3,3,21,2,3,μ2,3,3,21,2,3,μ3,3,3,21,2,3=7.01,10.64,0,μ3,3,31,2,3=μ1,3,3,31,2,3,μ2,3,3,31,2,3,μ3,3,3,31,2,3=8.72,10.86,0.In addition, combining the reliability of the evidence elements obtained after optimization, using equations (2) and (4), the optimal reliability factors of the reference evidence {e21, e31}, {e22, e32}, and {e23, e33} can be obtained as r21 = 0.2189, r31 = 0.5608, r22 = 0.9711, r32 = 0.6646, r32 = 0.9112, r33 = 0.8060, and the optimal reliability ratio between the evidences is γ1,2 2.2 = 4.0704, γ1,2 2.3 = 5.1821, γ1,2 3.2 = 0.5234, γ3,31,2 = 2.0580, γ2,2,21,2,3 = 4.3341, γ2,2,31,2,3 = 4.6267, γ2,3,21,2,3= 6.6279, γ2,3,31,2,3 = 7.6688, γ3,2,21,2,3 = 1.7674, γ3,2,31,2,3 = 2.0569, γ3,3,21,2,3  = 2.6719, and γ3,3,31,2,3  = 2.5055.The optimal importance weights of reference evidence {e1 2, e1 3}, {e2 2, e2 3} and {e3 2, e3 3} as follows:(20)w21=0.8683,0.7830,0.8077,w31=0.9319,0.9196,0.7384,w22=0.9986,0.8867,0.9511,w32=0.6048,0.8616,0.9667,w23=0.7834,0.9391,0.9292,w33=0.9751,0.9161,0.7861.As a result, MAKER rule is used 23 times to obtain 8 pieces of local fused evidence {es | s = 1,2,…, 8} as follows:(21)e1=0.8109,0.0006,0.1885,e2=0.8305,0.0008,0.1687,e3=0.3204,0.0013,0.6783,e4=0.3017,0.0014,0.6970,e5=0.7506,0.2468,0.0026,e6=0.6841,0.3130,0.0029,e7=0.1569,0.8397,0.0034,e8=0.2440,0.7516,0.0044.Obviously, in every local fused evidence, the belief degree ofF1 are all much higher than that of any activated reference evidence. Hence, MAKER fusion operation effectively focuses belief degree on the true class F1, it is very conducive to subsequent classification decisions.Furthermore, {es | s = 1.2,…, −8} are fused by ER rule in (15) to obtain the final fused result(22)Oxt′=F1,0.9999,F2,0,F3,0.0001.In ER fusion operation, the reliability factorsRs of es are R1 = 0.9999, R2 = 0.9999, R3 = 0.9997, R4 = 0.9995, R5 = 0.9996, R6 = 0.9994, R7 = 0.9992, R8 = 0.9982, respectively, and the importance weight is equal to its reliability Rs = Ws.It can be seen that the final fused result almost completely points toF1, hence the estimated class label is consistent with the true one. The above test example shows that MAKER-based classifier can effectively identify data samples. Table 13 lists the confusion matrix of the testing dataset.To summarize, in this “Seeds” experiment, the classification accuracies for the training dataset are 88.10% (before optimization) and 93.65% (after optimization), so the optimization model presented in Section3.5 significantly improves the performance of the proposed classifier. From Table 13, it can be determined that the classification accuracy for the testing dataset is 94.05%.


### 4.2. Comparative Analyses between MAKER-Based Classifier and the Existing Mainstream Classifiers on Five UCI Benchmarks

In order to further verify the effectiveness of the newly proposed classifier, this section compares it with seven classic classifiers, namely, decision trees [40], naive Bayes [6], k nearest neighbors (k-NN) [7], Bayes net [41], random forest [8], DC rule-based classifier (DC-core sample) [42], and the original ER classifier [43].Besides the Seeds database used in the previous section, we selected other four classic database from UCI benchmark database listed in Table14 for comparative analyses. Iris database is often used in classification experiments, and it can predict which of the three species of Setosa, Versicolour, and Virginic the iris flower belongs to by using four petal features. Haberman’s Survival Database provides some statistical data on breast cancer surgery survivors, it can judge the survival time of patients based on three characteristics. Heart database describes whether the patient has heart disease through some information indicators related to heart disease. Wine database includes 13 features of Wine varieties from three different regions.Table 14
The basic information of the experimental database.
DatasetFeatureClassSamplesSeeds73210Iris43150Haberman32306Heart132270Wine133178Similarly, 60% of the samples in each database are randomly selected as the training dataset, while the rest 40% are used as the testing dataset. The experiments are repeated 100 times, and then the average classification accuracy (ACA) of each classifier is recorded, as shown in Table15. From the above experimental results, we can see that MAKER rule-based classifier considers the correlation between evidence, so its classification performance (0.8904) in general outperforms that (0.8771) of the ER-based classifier. Among the five databases, ACAs for the three databases (Seeds, Haberman, Heart) are obviously higher than ACAs of the ER-based classifier. All in all, MAKER rule-based classifier has the highest overall ACA 89.04% (0.8904) in all eight classifiers, with its performance being best for Seeds and Heart.Table 15
Classification results of eight classifiers.
Decision treesNaive bayesK-NNBayes netRandom forestDC-core samplesOriginal ERMAKERSeeds0.88100.88100.86900.90480.89260.90480.89560.9183Iris0.91670.93330.90000.91670.93330.96670.96190.9539Haberman0.73770.76230.65370.77870.66390.80000.70760.7787Heart0.77780.80560.75000.75930.80560.77780.84200.8497Wine0.85920.97180.94370.98590.97180.90690.97830.9620Average0.83450.87080.82330.86910.85430.87120.87710.8925The bold values indicate the optimal results corresponding to the certain classifier for a specific data set.In order to further verify the performance advantages of the maker classifier, different percentages of test samples (PT) are selected for 50 random experiments of seed data classification. In Table16, random forest classifier and the original ER classifier are selected to be compared with the MAKER classifier under the different PT cases. Two main statistical indexes (SI) of the three classifiers are calculated, including ACA and the standard deviation of classification accuracy (SDCA). SDCA describes the deviation between the average classification accuracy obtained in a random experiment and the single classification accuracy. The smaller the SDCA, the more stable the classification performance of the corresponding classifier, that is, there are rarely too good or poor classification results.Table 16
Statistical analysis of different percentages.
PT (%)SIRandom forestOriginal ERMAKER20ACA0.90210.90410.9243SDCA0.04440.03740.038730ACA0.89870.90320.9202SDCA0.03890.03060.028940ACA0.89260.89560.9183SDCA0.03170.02430.024150ACA0.88960.89430.9079SDCA0.02990.02390.0233The bold values indicate the optimal results corresponding to the certain classifier for specific SI and PT.As can be seen from Table16, as the PT increases from 20% to 50%, the number of training samples modeling the three classifiers also decreases, resulting in their aca gradually decreasing. However, compared with the other two methods, the SDCA of the MAKER classifier is relatively small, and its ACA is always the highest in all PT cases.


## 4.1. Seeds Data Set Classification Experiment

There are 210 samples in the Seeds database, each sample has 7 input features including area (f1), perimeter (f2), compactness (f3), length of kernel (f4), width of kernel (f5), asymmetry coefficient (f6), and length of kernel groove (f7). These 7 features can be used to jointly identify the seeds of three different wheats, then FoD can be determined as Θ = {F1, F2, F3}. The detailed experiment steps are as follows.


### 4.1.1. Step 1: Feature Selection Based on PCA

We use the PCA method in Section3.2 to abstract the first 3 principal component features from 7 features of 210 samples, denoted as x1, x2, and x3 with their eigenvalues greater than 1. Thus, a condensed sample xt = [x1(t), x2(t), x3(t), F(t)] after PCA feature selection can be obtained, and F(t)∈Θ = {F1, F2, F3}. These sample vectors compose the sample set X = {x1,…, xt,…, xT}, T = 210. In this experiment, we randomly select 60% of the sample data of every class as the training dataset and the remaining 40% as the testing dataset.


### 4.1.2.  Step 2: Construction of SSMs and ESMs

For the featuresx1, x2, and x3, their initial reference value sets need be given as A1 = {−3.3, −0.7.1.9.4.5} for x1, A2 = {−2.8, −1, 0.8, 2.55} for x2, A3 = {−1.61, −0.1, 1.4, 2.93} for x3 via experts’ knowledge. By using (7) in Section 3.3.1, the similarity between the training samples and its reference values can be calculated. By the casting operation, the initial SSMs of x1, x2, and x3 are constructed. Based on the constructed SSM and the likelihood probability in (8), the corresponding ESMs for x1, x2, x3 are generated, as shown in Tables 5–7, respectively.Table 5
The initial ESM for the input featurex1.
Fx1e11e21e31e41−3.3−0.71.94.5F10.09650.58230.30280F200.06880.69721F30.90350.348900Table 6
The initial ESM for the input featurex2.
Fx2e12e22e32e42−2.8−10.82.55F10.89320.50700.13330F20.02000.29590.43200.2616F30.08680.19710.43470.7384Table 7
The initial ESM for the input featurex3.
Fx3e13e23e33e43−1.61−0.11.42.93F10.23750.36270.33420.3219F20.37520.33810.29930.1788F30.38730.29920.36650.4993


### 4.1.3.  Step 3: Calculation of the Correlation Factor: Interdependence Degree

According to Section 3.3.2, when the interdependence degreeμn,k1,…,kj,…,kJ1,…,j,…,J in (11) is calculated, the joint probability masses of the double feature [x1, x2] and the triple feature [x1, x2, x3] must be obtained by constructing joint SSMs and normalizing the corresponding likelihood functions. Table 8 lists the generated joint ESM for the double features [x1, x2] based on the reference value combination of x1′ and x2′. For the triple feature [x1, x2, x3], its EMS can be obtained by same way, here we will not illustrate this EMS matrix because it has too many (4 × 4 × 4 × 3) elements. As a result, all initial interdependence degree factors μn,k1,…,kj,…,kJ1,…,j,…,J (J = 3; n = 1, 2, 3; k1, k2, k3 ∈ {1, 2, 3, 4}) can be calculated by division operation about the elements in Tables 5–7, 9.Table 8
The optimized ESM for the input featurex1.
Fx1e11e21e31e41−3.3−0.97580.44104.5F10.07590.49380.50090.0612F2000.49910.9388F30.92410.506200Table 9
The initial joint ESM for the double features [x1, x2].
F[x1, x2]A11A12A13A14A21A22A32A42A12A22A32A42A12A22A32A42A12A22A32A42e1,21,1e1,21,2e1,21,3e1,21,4e1,22,1e1,22,2e1,22,3e1,22,4e1,23,1e1,23,2e1,23,3e1,23,4e1,24,1e1,24,2e1,24,3e1,24,4F10.58020.22180.044300.90540.74650.319200.95790.43510.086100000F200000.01420.05240.11230.06170.04210.56490.913911111F30.41980.77820.955710.08040.20110.56850.938300000000


### 4.1.4.  Step 4: Evidence Activation and Combination

For any one test samplext′ = [x1(t), x2(t), x3(t)], each feature will activate two pieces of adjacent reference evidence, and then the MAKER rule will be used 23 times to generate 23 pieces of local fused evidence. In MAKER-based fusion operation in (3), the weight factor wn,kj of the activated evidence, the numerator rn,kj and denominator rn,k1,…,kj,…,kJ1,…,j,…J of the reliability factor are all initialized as 1.


### 4.1.5.  Step 5: Classification Decision Based on the ER Rule

Next, 23 pieces of local fused evidence are fused by the ER rule, the classification decision can be made based the fused result.


### 4.1.6.  Step 6: Parameter Optimization of MAKER-Based Classifier via GA

Based on the optimization model given in Section3.5, the parameter set is P=Ak′j,wn,kj,rn,kj,rn,k1,…,kj,…,kJ1,…,j,…J|j=1,2,3;k=1,2,3,4;k′=2,3;n=1,2,3 including total 318 parameters, namely Ak′j (3 × 2 = 6), wn,kj (12 × 3 = 36), rn,kj (12 × 3 = 36), rn,k1,…,kj,…,kJ1,…,j,…J (4 × 4 × 3 + 4 × 4 × 4 × 3 = 240). It can be seen that these adjustable parameters all correspond to the ESMs for single feature Tables 5–7 and the joint ESMs in Table 9 for the double and triple features. The following Tables 8, 10–12 , respectively, present the optimized ESMs. After optimization, the reference values of each input feature have changed significantly, and at the same time, the elements of these ESMs have also changed greatly with the optimized reference values.Table 10
The optimized ESM for the input featurex2.
Fx2e11e21e31e41−2.8−0.9818−0.27242.55F10.89320.67810.18340.0616F20.02040.21870.42350.3773F30.08640.10320.39310.5611Table 11
The optimized ESM for the input featurex3.
Fx3e11e21e31e41−1.61−0.80471.55032.93F10.14700.34620.35270.3077F20.37940.34560.31260.1328F30.47360.30820.33470.5595Table 12
The optimized joint ESM for the double features [x1, x2].
F[x1, x2]A11A12A13A14A21A22A32A42A12A22A32A42A12A22A32A42A12A22A32A42e1,21,1e1,21,2e1,21,3e1,21,4e1,22,1e1,22,2e1,22,3e1,22,4e1,23,1e1,23,2e1,23,3e1,23,4e1,24,1e1,24,2e1,24,3e1,24,4F10.55130.31910.06530.00780.85690.82050.34800.12650.96040.74120.25980.12640.96430.22980.02110F2000000000.03960.25880.74020.87360.03570.77020.97891F30.44870.68090.93470.99220.14310.17950.65200.873500000000Through the above five steps, the optimal MAKER-based classifier is designed. Here, a test sample vectorxt′ = [x1 (t), x2 (t), x3 (t)] = [−0.9292, −0.8718, 0.9465] with the class label F1 is randomly selected from the test sample set to illustrate the process of evidence fusion and classification decision in detail.The three input eigenvaluesx1(t), x2(t), and x3(t) of the test sample vector xt′ will activate three pairs of reference evidence including {e21, e31} in Table 8, {e22, e32} in Table 10, {e3 2, e3 3} in Table 11. According to MAKER fusion process shown in Figure 2, for the double feature [x1, x2], the corresponding interdependence degrees used in fusion operations “e21⊕ e22,” “e21⊕ e22” “e31⊕ e22,” “e31⊕ e32” can be obtained by the joint ESM in Table 13 and equation (11) as follows:(18)μ2,21,2=μ1,2,21,2,μ2.2,21,2,μ3,2,21,2=2.45,0,3.44,μ2,31,2=μ1,2,31,2,μ2,2,31,2,μ3,2,31,2=3.84,0,3.28,μ3,21,2=μ1,3,21,2,μ2,3,21,2,μ3,3,21,2=2.18,3.37,0,μ3,31,2=μ1,3,31,2,μ2,3,31,2,μ3,3,31,2=2.83,3.50,0.Figure 2
MAKER fusion process of reference evidence activated by the input vectorx.Table 13
The confusion matrix of the testing dataset.
True classClassification resultsF1F2F3F12422F20280F31027In the same way, based on the joint ESM for the triple feature [x1, x2, x3], the interdependence degrees of “e21⊕ e22⊕ e23,” “e21⊕ e32⊕ e23”,…, “e31⊕ e32⊕ e33” are as follows:(19)μ2,2,21,2,3=μ1,2,2,21,2,3,μ2,2,2,21,2,3,μ3,2,2,21,2,3=7.42,0,8.66,μ2,2,31,2,3=μ1,2,2,31,2,3,μ2,2,2,31,2,3,μ3,2,2,31,2,3=7.01,0,9.85,μ2,3,21,2,3=μ1,2,3,21,2,3,μ2,2,3,21,2,3,μ3,2,3,21,2,3=12.74,0,9.79,μ2,3,31,2,3=μ1,2,3,31,2,3,μ2,2,3,31,2,3,μ3,2,3,31,2,3=9.16,0,10.62,μ3,2,21,2,3=μ1,3,2,21,2,3,μ2,3,2,21,2,3,μ3,3,2,21,2,3=6.63,5.84,0,μ3,2,31,2,3=μ1,3,2,31,2,3,μ2,3,2,31,2,3,μ3,3,2,31,2,3=5.53,9.91,0,μ3,3,21,2,3=μ1,3,3,21,2,3,μ2,3,3,21,2,3,μ3,3,3,21,2,3=7.01,10.64,0,μ3,3,31,2,3=μ1,3,3,31,2,3,μ2,3,3,31,2,3,μ3,3,3,31,2,3=8.72,10.86,0.In addition, combining the reliability of the evidence elements obtained after optimization, using equations (2) and (4), the optimal reliability factors of the reference evidence {e21, e31}, {e22, e32}, and {e23, e33} can be obtained as r21 = 0.2189, r31 = 0.5608, r22 = 0.9711, r32 = 0.6646, r32 = 0.9112, r33 = 0.8060, and the optimal reliability ratio between the evidences is γ1,2 2.2 = 4.0704, γ1,2 2.3 = 5.1821, γ1,2 3.2 = 0.5234, γ3,31,2 = 2.0580, γ2,2,21,2,3 = 4.3341, γ2,2,31,2,3 = 4.6267, γ2,3,21,2,3= 6.6279, γ2,3,31,2,3 = 7.6688, γ3,2,21,2,3 = 1.7674, γ3,2,31,2,3 = 2.0569, γ3,3,21,2,3  = 2.6719, and γ3,3,31,2,3  = 2.5055.The optimal importance weights of reference evidence {e1 2, e1 3}, {e2 2, e2 3} and {e3 2, e3 3} as follows:(20)w21=0.8683,0.7830,0.8077,w31=0.9319,0.9196,0.7384,w22=0.9986,0.8867,0.9511,w32=0.6048,0.8616,0.9667,w23=0.7834,0.9391,0.9292,w33=0.9751,0.9161,0.7861.As a result, MAKER rule is used 23 times to obtain 8 pieces of local fused evidence {es | s = 1,2,…, 8} as follows:(21)e1=0.8109,0.0006,0.1885,e2=0.8305,0.0008,0.1687,e3=0.3204,0.0013,0.6783,e4=0.3017,0.0014,0.6970,e5=0.7506,0.2468,0.0026,e6=0.6841,0.3130,0.0029,e7=0.1569,0.8397,0.0034,e8=0.2440,0.7516,0.0044.Obviously, in every local fused evidence, the belief degree ofF1 are all much higher than that of any activated reference evidence. Hence, MAKER fusion operation effectively focuses belief degree on the true class F1, it is very conducive to subsequent classification decisions.Furthermore, {es | s = 1.2,…, −8} are fused by ER rule in (15) to obtain the final fused result(22)Oxt′=F1,0.9999,F2,0,F3,0.0001.In ER fusion operation, the reliability factorsRs of es are R1 = 0.9999, R2 = 0.9999, R3 = 0.9997, R4 = 0.9995, R5 = 0.9996, R6 = 0.9994, R7 = 0.9992, R8 = 0.9982, respectively, and the importance weight is equal to its reliability Rs = Ws.It can be seen that the final fused result almost completely points toF1, hence the estimated class label is consistent with the true one. The above test example shows that MAKER-based classifier can effectively identify data samples. Table 13 lists the confusion matrix of the testing dataset.To summarize, in this “Seeds” experiment, the classification accuracies for the training dataset are 88.10% (before optimization) and 93.65% (after optimization), so the optimization model presented in Section3.5 significantly improves the performance of the proposed classifier. From Table 13, it can be determined that the classification accuracy for the testing dataset is 94.05%.


## 4.1.1. Step 1: Feature Selection Based on PCA

We use the PCA method in Section3.2 to abstract the first 3 principal component features from 7 features of 210 samples, denoted as x1, x2, and x3 with their eigenvalues greater than 1. Thus, a condensed sample xt = [x1(t), x2(t), x3(t), F(t)] after PCA feature selection can be obtained, and F(t)∈Θ = {F1, F2, F3}. These sample vectors compose the sample set X = {x1,…, xt,…, xT}, T = 210. In this experiment, we randomly select 60% of the sample data of every class as the training dataset and the remaining 40% as the testing dataset.


## 4.1.2.  Step 2: Construction of SSMs and ESMs

For the featuresx1, x2, and x3, their initial reference value sets need be given as A1 = {−3.3, −0.7.1.9.4.5} for x1, A2 = {−2.8, −1, 0.8, 2.55} for x2, A3 = {−1.61, −0.1, 1.4, 2.93} for x3 via experts’ knowledge. By using (7) in Section 3.3.1, the similarity between the training samples and its reference values can be calculated. By the casting operation, the initial SSMs of x1, x2, and x3 are constructed. Based on the constructed SSM and the likelihood probability in (8), the corresponding ESMs for x1, x2, x3 are generated, as shown in Tables 5–7, respectively.Table 5
The initial ESM for the input featurex1.
Fx1e11e21e31e41−3.3−0.71.94.5F10.09650.58230.30280F200.06880.69721F30.90350.348900Table 6
The initial ESM for the input featurex2.
Fx2e12e22e32e42−2.8−10.82.55F10.89320.50700.13330F20.02000.29590.43200.2616F30.08680.19710.43470.7384Table 7
The initial ESM for the input featurex3.
Fx3e13e23e33e43−1.61−0.11.42.93F10.23750.36270.33420.3219F20.37520.33810.29930.1788F30.38730.29920.36650.4993


## 4.1.3.  Step 3: Calculation of the Correlation Factor: Interdependence Degree

According to Section 3.3.2, when the interdependence degreeμn,k1,…,kj,…,kJ1,…,j,…,J in (11) is calculated, the joint probability masses of the double feature [x1, x2] and the triple feature [x1, x2, x3] must be obtained by constructing joint SSMs and normalizing the corresponding likelihood functions. Table 8 lists the generated joint ESM for the double features [x1, x2] based on the reference value combination of x1′ and x2′. For the triple feature [x1, x2, x3], its EMS can be obtained by same way, here we will not illustrate this EMS matrix because it has too many (4 × 4 × 4 × 3) elements. As a result, all initial interdependence degree factors μn,k1,…,kj,…,kJ1,…,j,…,J (J = 3; n = 1, 2, 3; k1, k2, k3 ∈ {1, 2, 3, 4}) can be calculated by division operation about the elements in Tables 5–7, 9.Table 8
The optimized ESM for the input featurex1.
Fx1e11e21e31e41−3.3−0.97580.44104.5F10.07590.49380.50090.0612F2000.49910.9388F30.92410.506200Table 9
The initial joint ESM for the double features [x1, x2].
F[x1, x2]A11A12A13A14A21A22A32A42A12A22A32A42A12A22A32A42A12A22A32A42e1,21,1e1,21,2e1,21,3e1,21,4e1,22,1e1,22,2e1,22,3e1,22,4e1,23,1e1,23,2e1,23,3e1,23,4e1,24,1e1,24,2e1,24,3e1,24,4F10.58020.22180.044300.90540.74650.319200.95790.43510.086100000F200000.01420.05240.11230.06170.04210.56490.913911111F30.41980.77820.955710.08040.20110.56850.938300000000


## 4.1.4.  Step 4: Evidence Activation and Combination

For any one test samplext′ = [x1(t), x2(t), x3(t)], each feature will activate two pieces of adjacent reference evidence, and then the MAKER rule will be used 23 times to generate 23 pieces of local fused evidence. In MAKER-based fusion operation in (3), the weight factor wn,kj of the activated evidence, the numerator rn,kj and denominator rn,k1,…,kj,…,kJ1,…,j,…J of the reliability factor are all initialized as 1.


## 4.1.5.  Step 5: Classification Decision Based on the ER Rule

Next, 23 pieces of local fused evidence are fused by the ER rule, the classification decision can be made based the fused result.


## 4.1.6.  Step 6: Parameter Optimization of MAKER-Based Classifier via GA

Based on the optimization model given in Section3.5, the parameter set is P=Ak′j,wn,kj,rn,kj,rn,k1,…,kj,…,kJ1,…,j,…J|j=1,2,3;k=1,2,3,4;k′=2,3;n=1,2,3 including total 318 parameters, namely Ak′j (3 × 2 = 6), wn,kj (12 × 3 = 36), rn,kj (12 × 3 = 36), rn,k1,…,kj,…,kJ1,…,j,…J (4 × 4 × 3 + 4 × 4 × 4 × 3 = 240). It can be seen that these adjustable parameters all correspond to the ESMs for single feature Tables 5–7 and the joint ESMs in Table 9 for the double and triple features. The following Tables 8, 10–12 , respectively, present the optimized ESMs. After optimization, the reference values of each input feature have changed significantly, and at the same time, the elements of these ESMs have also changed greatly with the optimized reference values.Table 10
The optimized ESM for the input featurex2.
Fx2e11e21e31e41−2.8−0.9818−0.27242.55F10.89320.67810.18340.0616F20.02040.21870.42350.3773F30.08640.10320.39310.5611Table 11
The optimized ESM for the input featurex3.
Fx3e11e21e31e41−1.61−0.80471.55032.93F10.14700.34620.35270.3077F20.37940.34560.31260.1328F30.47360.30820.33470.5595Table 12
The optimized joint ESM for the double features [x1, x2].
F[x1, x2]A11A12A13A14A21A22A32A42A12A22A32A42A12A22A32A42A12A22A32A42e1,21,1e1,21,2e1,21,3e1,21,4e1,22,1e1,22,2e1,22,3e1,22,4e1,23,1e1,23,2e1,23,3e1,23,4e1,24,1e1,24,2e1,24,3e1,24,4F10.55130.31910.06530.00780.85690.82050.34800.12650.96040.74120.25980.12640.96430.22980.02110F2000000000.03960.25880.74020.87360.03570.77020.97891F30.44870.68090.93470.99220.14310.17950.65200.873500000000Through the above five steps, the optimal MAKER-based classifier is designed. Here, a test sample vectorxt′ = [x1 (t), x2 (t), x3 (t)] = [−0.9292, −0.8718, 0.9465] with the class label F1 is randomly selected from the test sample set to illustrate the process of evidence fusion and classification decision in detail.The three input eigenvaluesx1(t), x2(t), and x3(t) of the test sample vector xt′ will activate three pairs of reference evidence including {e21, e31} in Table 8, {e22, e32} in Table 10, {e3 2, e3 3} in Table 11. According to MAKER fusion process shown in Figure 2, for the double feature [x1, x2], the corresponding interdependence degrees used in fusion operations “e21⊕ e22,” “e21⊕ e22” “e31⊕ e22,” “e31⊕ e32” can be obtained by the joint ESM in Table 13 and equation (11) as follows:(18)μ2,21,2=μ1,2,21,2,μ2.2,21,2,μ3,2,21,2=2.45,0,3.44,μ2,31,2=μ1,2,31,2,μ2,2,31,2,μ3,2,31,2=3.84,0,3.28,μ3,21,2=μ1,3,21,2,μ2,3,21,2,μ3,3,21,2=2.18,3.37,0,μ3,31,2=μ1,3,31,2,μ2,3,31,2,μ3,3,31,2=2.83,3.50,0.Figure 2
MAKER fusion process of reference evidence activated by the input vectorx.Table 13
The confusion matrix of the testing dataset.
True classClassification resultsF1F2F3F12422F20280F31027In the same way, based on the joint ESM for the triple feature [x1, x2, x3], the interdependence degrees of “e21⊕ e22⊕ e23,” “e21⊕ e32⊕ e23”,…, “e31⊕ e32⊕ e33” are as follows:(19)μ2,2,21,2,3=μ1,2,2,21,2,3,μ2,2,2,21,2,3,μ3,2,2,21,2,3=7.42,0,8.66,μ2,2,31,2,3=μ1,2,2,31,2,3,μ2,2,2,31,2,3,μ3,2,2,31,2,3=7.01,0,9.85,μ2,3,21,2,3=μ1,2,3,21,2,3,μ2,2,3,21,2,3,μ3,2,3,21,2,3=12.74,0,9.79,μ2,3,31,2,3=μ1,2,3,31,2,3,μ2,2,3,31,2,3,μ3,2,3,31,2,3=9.16,0,10.62,μ3,2,21,2,3=μ1,3,2,21,2,3,μ2,3,2,21,2,3,μ3,3,2,21,2,3=6.63,5.84,0,μ3,2,31,2,3=μ1,3,2,31,2,3,μ2,3,2,31,2,3,μ3,3,2,31,2,3=5.53,9.91,0,μ3,3,21,2,3=μ1,3,3,21,2,3,μ2,3,3,21,2,3,μ3,3,3,21,2,3=7.01,10.64,0,μ3,3,31,2,3=μ1,3,3,31,2,3,μ2,3,3,31,2,3,μ3,3,3,31,2,3=8.72,10.86,0.In addition, combining the reliability of the evidence elements obtained after optimization, using equations (2) and (4), the optimal reliability factors of the reference evidence {e21, e31}, {e22, e32}, and {e23, e33} can be obtained as r21 = 0.2189, r31 = 0.5608, r22 = 0.9711, r32 = 0.6646, r32 = 0.9112, r33 = 0.8060, and the optimal reliability ratio between the evidences is γ1,2 2.2 = 4.0704, γ1,2 2.3 = 5.1821, γ1,2 3.2 = 0.5234, γ3,31,2 = 2.0580, γ2,2,21,2,3 = 4.3341, γ2,2,31,2,3 = 4.6267, γ2,3,21,2,3= 6.6279, γ2,3,31,2,3 = 7.6688, γ3,2,21,2,3 = 1.7674, γ3,2,31,2,3 = 2.0569, γ3,3,21,2,3  = 2.6719, and γ3,3,31,2,3  = 2.5055.The optimal importance weights of reference evidence {e1 2, e1 3}, {e2 2, e2 3} and {e3 2, e3 3} as follows:(20)w21=0.8683,0.7830,0.8077,w31=0.9319,0.9196,0.7384,w22=0.9986,0.8867,0.9511,w32=0.6048,0.8616,0.9667,w23=0.7834,0.9391,0.9292,w33=0.9751,0.9161,0.7861.As a result, MAKER rule is used 23 times to obtain 8 pieces of local fused evidence {es | s = 1,2,…, 8} as follows:(21)e1=0.8109,0.0006,0.1885,e2=0.8305,0.0008,0.1687,e3=0.3204,0.0013,0.6783,e4=0.3017,0.0014,0.6970,e5=0.7506,0.2468,0.0026,e6=0.6841,0.3130,0.0029,e7=0.1569,0.8397,0.0034,e8=0.2440,0.7516,0.0044.Obviously, in every local fused evidence, the belief degree ofF1 are all much higher than that of any activated reference evidence. Hence, MAKER fusion operation effectively focuses belief degree on the true class F1, it is very conducive to subsequent classification decisions.Furthermore, {es | s = 1.2,…, −8} are fused by ER rule in (15) to obtain the final fused result(22)Oxt′=F1,0.9999,F2,0,F3,0.0001.In ER fusion operation, the reliability factorsRs of es are R1 = 0.9999, R2 = 0.9999, R3 = 0.9997, R4 = 0.9995, R5 = 0.9996, R6 = 0.9994, R7 = 0.9992, R8 = 0.9982, respectively, and the importance weight is equal to its reliability Rs = Ws.It can be seen that the final fused result almost completely points toF1, hence the estimated class label is consistent with the true one. The above test example shows that MAKER-based classifier can effectively identify data samples. Table 13 lists the confusion matrix of the testing dataset.To summarize, in this “Seeds” experiment, the classification accuracies for the training dataset are 88.10% (before optimization) and 93.65% (after optimization), so the optimization model presented in Section3.5 significantly improves the performance of the proposed classifier. From Table 13, it can be determined that the classification accuracy for the testing dataset is 94.05%.


## 4.2. Comparative Analyses between MAKER-Based Classifier and the Existing Mainstream Classifiers on Five UCI Benchmarks

In order to further verify the effectiveness of the newly proposed classifier, this section compares it with seven classic classifiers, namely, decision trees [40], naive Bayes [6], k nearest neighbors (k-NN) [7], Bayes net [41], random forest [8], DC rule-based classifier (DC-core sample) [42], and the original ER classifier [43].Besides the Seeds database used in the previous section, we selected other four classic database from UCI benchmark database listed in Table14 for comparative analyses. Iris database is often used in classification experiments, and it can predict which of the three species of Setosa, Versicolour, and Virginic the iris flower belongs to by using four petal features. Haberman’s Survival Database provides some statistical data on breast cancer surgery survivors, it can judge the survival time of patients based on three characteristics. Heart database describes whether the patient has heart disease through some information indicators related to heart disease. Wine database includes 13 features of Wine varieties from three different regions.Table 14
The basic information of the experimental database.
DatasetFeatureClassSamplesSeeds73210Iris43150Haberman32306Heart132270Wine133178Similarly, 60% of the samples in each database are randomly selected as the training dataset, while the rest 40% are used as the testing dataset. The experiments are repeated 100 times, and then the average classification accuracy (ACA) of each classifier is recorded, as shown in Table15. From the above experimental results, we can see that MAKER rule-based classifier considers the correlation between evidence, so its classification performance (0.8904) in general outperforms that (0.8771) of the ER-based classifier. Among the five databases, ACAs for the three databases (Seeds, Haberman, Heart) are obviously higher than ACAs of the ER-based classifier. All in all, MAKER rule-based classifier has the highest overall ACA 89.04% (0.8904) in all eight classifiers, with its performance being best for Seeds and Heart.Table 15
Classification results of eight classifiers.
Decision treesNaive bayesK-NNBayes netRandom forestDC-core samplesOriginal ERMAKERSeeds0.88100.88100.86900.90480.89260.90480.89560.9183Iris0.91670.93330.90000.91670.93330.96670.96190.9539Haberman0.73770.76230.65370.77870.66390.80000.70760.7787Heart0.77780.80560.75000.75930.80560.77780.84200.8497Wine0.85920.97180.94370.98590.97180.90690.97830.9620Average0.83450.87080.82330.86910.85430.87120.87710.8925The bold values indicate the optimal results corresponding to the certain classifier for a specific data set.In order to further verify the performance advantages of the maker classifier, different percentages of test samples (PT) are selected for 50 random experiments of seed data classification. In Table16, random forest classifier and the original ER classifier are selected to be compared with the MAKER classifier under the different PT cases. Two main statistical indexes (SI) of the three classifiers are calculated, including ACA and the standard deviation of classification accuracy (SDCA). SDCA describes the deviation between the average classification accuracy obtained in a random experiment and the single classification accuracy. The smaller the SDCA, the more stable the classification performance of the corresponding classifier, that is, there are rarely too good or poor classification results.Table 16
Statistical analysis of different percentages.
PT (%)SIRandom forestOriginal ERMAKER20ACA0.90210.90410.9243SDCA0.04440.03740.038730ACA0.89870.90320.9202SDCA0.03890.03060.028940ACA0.89260.89560.9183SDCA0.03170.02430.024150ACA0.88960.89430.9079SDCA0.02990.02390.0233The bold values indicate the optimal results corresponding to the certain classifier for specific SI and PT.As can be seen from Table16, as the PT increases from 20% to 50%, the number of training samples modeling the three classifiers also decreases, resulting in their aca gradually decreasing. However, compared with the other two methods, the SDCA of the MAKER classifier is relatively small, and its ACA is always the highest in all PT cases.


## 5. Conclusion

This paper proposes a generalized classifier design method based on the MAKER rule. First, we use PCA to select main features for a high-dimensional data sample. Then ESMs are constructed through training samples, and the correlation factors between different reference evidence are obtained through joint ESMs. After that, we consider the reliability and weight of the evidence and use the MAKER rule to fuse the initial evidence. The class of the input vector is determined according to the fusing result. Finally, the parameters of the classifier are optimized by GA to improve the classification. Several classic classification methods are compared and analyzed on five typical classification databases to illustrate the effectiveness of the new classifier. The advantages and innovations of a MAKER-based classifier can be summarized as follows:(1)
MAKER-based classifier not only inherits the advantages of an ER-based classifier in dealing with uncertain information but also fully takes into account the correlation between evidence, making up for the deficiency that ER rules can only combine independent evidence.(2)
By considering correlation, MAKER-based classifier introduces more adjustable correlation factors to mine more correlation information for data classification. Through the parameter optimization process, classification performance can be greatly improved.(3)
Compared with other methods, the MAKER-based classifier uses SSM and ESM to objectively describe the statistical relationship between the input feature variables and output classes without relying on any prior probability model. The physical meanings of the adjustable parameters in MAKER model are very clear and understandable for users.Of course, the experimental data set in this article is limited. Only five popular UCI benchmark databases were selected. More datasets from real applications need to be further used to verify the effectiveness of the proposed method. Moreover, MAKER rule is the extension of the ER rule and has good application prospects. At present, MAKER rule has been successfully applied in the fields of fault diagnosis and data classifier. The application of MAKER rule has been rarely found in the field of project evaluation. However, the ER rule has been used for project management by Liu et al. [44–46] and yielded good results. In the future, the MAKER rule can be applied to the field of project management to solve the multiple-criteria R&D project selection problems.



---

*Source: 5933793-2023-01-14.xml*