# A Comprehensive Evaluation Approach that Highlights the 'Equilibrium' and Its Application in Evaluation on Regional Innovation Capability

Yanru Qi, Yun Liu, Jing An

► To cite this version:

Yanru Qi, Yun Liu, Jing An. A Comprehensive Evaluation Approach that Highlights the 'Equilibrium' and Its Application in Evaluation on Regional Innovation Capability. 15th International Conference on Informatics and Semiotics in Organisations (ICISO), May 2014, Shanghai, China. pp.292-301, 10.1007/978-3-642-55355-4_30. hal-01350935

HAL Id: hal-01350935

https://inria.hal.science/hal-01350935

Submitted on 2 Aug 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Comprehensive Evaluation Approach that Highlights the 'Equilibrium' and its Application in Evaluation on Regional Innovation Capability

Yanru Qi, Yun Liu, Jing An

School of Management and Economic, Beijing Institute of Technology, Beijing, China
yrqits@163.com
liuyun@bit.edu.cn
anjing@tobacco.gov.cn

**Abstract.** In this paper we proposed a novel method that aggregates the "Functionality" and the "Equilibrium" to calculate the weight of comprehensive evaluation problem. The method calculate the weight coefficient according to the principle of "variance drive" firstly, then considering the development of the system balance with relative equilibrium coefficient to arrive at the comprehensive evaluation results of various systems. The method reflects the scientificity, impartiality and rationality. So it can avoid the phenomenon "one swallow make a summer", which can promote the healthy development of the whole system. Finally, a numerical example was given to illustrate the effectiveness of the proposed method.

**Keywords.** Comprehensive Evaluation, Multiple Attribute Decision Making, Relatively Balanced Coefficient.

## 1 Introduction

Many comprehensive evaluation problems can be met frequently in realistic society, such as evaluation of the region innovation capability or urban competitiveness of a country or area. The method use now for this problems are classified into 9 classes, such as Comprehensive Evaluation methods [1-4], Operational Research methods [5-7], Statistical methods [8-9], Systematic Engineering methods [10-12], Fuzzy Mathematics methods [13-16].

Multi-index comprehensive evaluation is a kind of method to get a comprehensive index to make an overall evaluation and a vertical or horizontal comparison on evaluation object, by integrating the multiple index information which describes the different aspects of the evaluation object. This method is widely used in many areas since it could make accurate description for evaluation objects and process the dynamic objects which have many decision makers and indexes.

In the evaluation process, the comprehensive evaluation result is directly influenced by the determination of weight coefficient and evaluation index. There are several methods of determining the weight, which can be grossly divided into subjective and objective weighting method [17]. Subjective weighting method such as Delphi and AHP often depends on subjective experience, the result of which
instability with the influence of the expert experience or preference. Objective methods such as Maximizing Deviation Method[18] and Mean Square Difference Method[19] depends on practical data of every index, the result of these methods based on principle that “the difference is driven” may led to lopsided development of the evaluation system. To avoid the defects of subjective and objective weighting methods, a new weighting method is proposed[20], and the new mathematical programming model synthesized both subjective and objective characters is established. This research will set up a comprehensive evaluation model that introduce “relative equilibrium coefficient” into the objective weighting method to encouraging the advanced and urging on the backward in evaluation. With all these efforts, we can expect to realize the health development of system.

In the real evaluation process, fewer scholars consider whether the indexes of evaluation objects are in balanced development. In this paper we proposed a novel method that aggregates the “Functionality” and the “Equilibrium” to calculate the weight of comprehensive evaluation problem, the method utilize the deviations between each indicator and the average value to calculate the in the multi-index comprehensive evaluation, then considering the development of the system balance with relative equilibrium coefficient to arrive at the comprehensive evaluation results of various systems. The case analysis result is given to prove the effectiveness of this method in the end of this paper. This method gives consideration to both functionality and equilibrium of evaluation objects and realize the function of encourage advanced spur lagging behind.

# 2 Determination of index weight coefficients

## 2.1 Based on the weight determination method of “variance drive”

Setting the evaluation object set

$$O = \{o_1, o_2, \dots, o_n\} \qquad (1)$$

Index set

$$P = \{p_1, p_2, \dots, p_m\} \\ x_{ij} = x_j(o_i)(i = 1, 2, \dots, n; j = 1, 2, \dots, m) \qquad (2)$$
which is short-cut process of scheme $o_i$ about the indicator $p_j$. Then the index matrix about scheme set $O$ and index set $P$ can be expressed as

$$A = [x_{ij}]_{n \times m} = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1m} \\ x_{21} & x_{22} & \cdots & x_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \cdots & x_{nm} \end{bmatrix} \quad (3)$$

Without loss of generality, assume the index of index set $P$ are all extra-large, carry out the dimensionless treatment with the data in $A$ as below:

$$x_{ij}^* = \frac{x_{ij}}{\min_{i}\{x_{ij}\}}, \quad (i = 1, 2, \dots, n; j = 1, 2, \dots, m) \quad (4)$$

We mark $x_{ij}^*$ as $x_{ij}$ below for convenience

Gather the multi-target information with linear weighted model; assume that $y_i$ is the evaluating value of scheme $o_i$, then,

$$\begin{align} y_i &= \omega_1 x_{i1} + \omega_2 x_{i2} + \cdots + \omega_m x_{im} \nonumber \\ &= \sum_{j=1}^{m} \omega_j x_{ij} = \omega^T x_i \tag{5} \\ (i = 1, 2, 3, \dots, n) \nonumber \end{align}$$

In the formula $x_i = (x_{i1}, x_{i2}, \dots, x_{im})^T$, $\omega = (\omega_1, \omega_2, \dots, \omega_m)^T$ is $m$-dimensional vector, name $x_i$ as the index vector of $o_i$, $\omega$ as index weight vector.

If $y = [y_1 \ y_2 \ \dots \ y_n]^T$, then the formula (5) can be written as:

$$y = A\omega \quad (6)$$
We can regard n evaluation objects as n dots or vectors of m-dimensional space that consist of m- evaluation indexes. Then what we should do is select index weight coefficient to enlarge the difference of each evaluation objects and highlight the overall differences of each evaluation objects, based on that improved the method of maximizing deviations which was mentioned in literature [18], and build the function

$$H = \max \sum_{j=1}^{m} \omega_j \cdot \left( \sum_{i=1}^{n} |x_{ij} - x_j| \right) \quad (7)$$

If define

$$\sum_{j=1}^{m} \omega_j^2 = 1 \quad (8)$$

in advance, then we can transform the issue into the maximum problem of type(7) under this Limited-Term, that is select $\omega$ to make

$$H = \max \sum_{j=1}^{m} \omega_j \cdot \left( \sum_{i=1}^{n} |x_{ij} - x_j| \right) \\ \text{s.t.} \quad \sum_{j=1}^{m} \omega_j^2 = 1 \quad (9) \\ \omega > 0$$

Apply Lagrange conditional extreme to type (9), we can get

$$\sum_{j=1}^{n} |x_{ij} - x_j| = 2\lambda\omega_j \quad (10) \\ \sum_{j=1}^{m} \omega_j^2 = 1$$

And we can obtain the weight coefficient after normalization processing as follows,

$$\omega_j^* = \frac{\sum_{i=1}^{n} |x_{ij} - x_j|}{\sum_{j=1}^{m} \sum_{i=1}^{n} |x_{ij} - x_i|} \quad (21)$$

Utilize the weight coefficient obtained from formula (11) with formula (5), we can get the comprehensive index y based on 'differences drive'.
The analysis result of the system which is obtained from the evaluating value is independent on people's subjective judgment and strong objectivity. However, this evaluating value has the character of highlight indicators intrinsic difference, which can cause the system appear the "deformed" development phenomenon that focus on strengthening an index because of optimizing. In order to avoid the occurrence of this phenomenon, we introduce the system relative equilibrium coefficient concept. Meanwhile, we encourage these balanced development systems and punish these deformed development systems through the above coefficient to get the more close to the actual situation and acceptable evaluation result.

## 2.2 The introduction of relative equilibrium coefficient

In this paper, we will utilize the reciprocal of the coefficient of variation of decentralized characterization data to describe the balance of the system [21]. We define system $o_i$ (while the index value of $o_i$ is data after dimensionless process)

$$JH_i = \begin{cases} \frac{1}{s_i} \sum_{j=1}^{m} x_{ij} & (s_i = \sqrt{\frac{\sum_{j=1}^{m} (x_{ij} - x_i)^2}{m-1}} \neq 0; i = 1, 2, \dots, n) \\ \max\{JH_d\} + c, (s_i = 0; s_d \neq 0) \end{cases} \quad (32)$$

It is observed that the larger $JH_i$, the smaller volatility of numerical value between the indicators in the system is. Otherwise, the bigger volatility of the system indicators, the less balanced development of the system is. We use formula (14) to define the relative equilibrium coefficient of system.

$$\begin{aligned} JH_i^* &= n \cdot \frac{JH_i}{\sum_{i=1}^{n} JH_i} \\ \sum_{i=1}^{n} JH_i^* &= n \end{aligned} \quad (43)$$

There from we can deduce that, for these n systems, if $JH_i^*$ is less than 1, then there is a gap between the indicators of this system; if $JH_i^*$ is bigger than 1, then indicators in the system is relatively balanced; if $JH_i^*$ is equal to 1, then the development of indicators in the system is in a middle position among all the systems. For these systems that $JH_i^*$ is bigger than 1, we should increase its value to some extent as motivation. Meanwhile, for these "deformed" developed systems, we should
decrease its value as punishment so as to realize the disciplinary role of evaluation model.

## 2.3 A build-up model that introduce the relative equilibrium coefficient

On the base of relative equilibrium coefficient, we concentrate the system over again and obtain that

$$y_i^* = y_i + \frac{(JH_i^* - 1) * \sum_{i=1}^{n} y_i}{n * \sum_{i=1}^{n} |JH_i^* - 1|} \quad (54)$$

Via the objective evaluation of the difference process of formula (14), we can realize the function of promoting the smooth development and punish imbalanced development.

# 3 Example

Utilize the functionality and balanced aggregate model to calculate the region innovation capacity data of 31 provinces as tab 1 in literature [22],

<table><caption>Table 1. The region innovation capacity index of each province of 2011</caption><thead><tr><th>Regions</th><th>The Utility Value of Knowledge Creation</th><th>The Utility Value of Knowledge Acquisition</th><th>The Utility Value of Enterprise Innovation</th><th>The Utility Value of Innovation Environment</th><th>The Utility Value of Innovation Performance</th></tr></thead><tbody><tr><td>Jiangsu</td><td>40.94</td><td>50.59</td><td>62.85</td><td>50.85</td><td>50.58</td></tr><tr><td>Guangdong</td><td>48.92</td><td>44.54</td><td>50.81</td><td>55.10</td><td>56.95</td></tr><tr><td>Beijing</td><td>79.96</td><td>40.65</td><td>46.26</td><td>36.70</td><td>45.44</td></tr><tr><td>Shanghai</td><td>47.60</td><td>63.82</td><td>44.93</td><td>37.37</td><td>44.68</td></tr><tr><td>Zhejiang</td><td>31.89</td><td>30.91</td><td>56.53</td><td>41.22</td><td>36.87</td></tr><tr><td>Shandong</td><td>31.42</td><td>25.12</td><td>44.09</td><td>39.72</td><td>39.54</td></tr><tr><td>Tianjin</td><td>28.36</td><td>37.49</td><td>41.04</td><td>32.85</td><td>37.69</td></tr><tr><td>Hubei</td><td>24.89</td><td>21.33</td><td>36.79</td><td>30.27</td><td>34.57</td></tr><tr><td>Sichuang</td><td>26.00</td><td>26.18</td><td>29.17</td><td>34.41</td><td>31.12</td></tr><tr><td>Chongqing</td><td>20.6</td><td>25.43</td><td>35.66</td><td>32.01</td><td>30.12</td></tr><tr><td>Hunan</td><td>29.27</td><td>25.51</td><td>28.27</td><td>29.30</td><td>35.91</td></tr><tr><td>Liaoning</td><td>23.59</td><td>38.66</td><td>26.20</td><td>30.36</td><td>27.26</td></tr><tr><td>Anhui</td><td>19.65</td><td>15.88</td><td>33.29</td><td>30.54</td><td>36.35</td></tr><tr><td>Shaanxi</td><td>30.25</td><td>18.62</td><td>26.74</td><td>33.41</td><td>27.10</td></tr><tr><td>Henan</td><td>22.39</td><td>20.83</td><td>23.69</td><td>29.58</td><td>30.81</td></tr><tr><td>Fujian</td><td>17.38</td><td>21.50</td><td>21.88</td><td>26.71</td><td>30.91</td></tr><tr><td>Shanxi</td><td>20.61</td><td>15.69</td><td>30.85</td><td>21.80</td><td>26.12</td></tr></tbody></table>
<table><tr><td>Hebei</td><td>18.92</td><td>22.64</td><td>20.12</td><td>24.75</td><td>29.07</td></tr><tr><td>Heilongjiang</td><td>22.87</td><td>17.54</td><td>20.66</td><td>23.06</td><td>29.24</td></tr><tr><td>Guangxi</td><td>14.81</td><td>16.08</td><td>25.40</td><td>23.65</td><td>28.33</td></tr><tr><td>Jilin</td><td>17.58</td><td>13.37</td><td>19.04</td><td>24.43</td><td>33.47</td></tr><tr><td>Jiangxi</td><td>13.43</td><td>17.02</td><td>16.31</td><td>28.51</td><td>31.49</td></tr><tr><td>Hainan</td><td>18.74</td><td>26.06</td><td>12.35</td><td>23.27</td><td>31.65</td></tr><tr><td>Ningxia</td><td>13.06</td><td>16.82</td><td>24.44</td><td>23.87</td><td>19.41</td></tr><tr><td>Yunnan</td><td>18.03</td><td>17.09</td><td>21.86</td><td>20.22</td><td>24.75</td></tr><tr><td>Inner Mongolia</td><td>12.29</td><td>20.64</td><td>14.67</td><td>24.70</td><td>28.41</td></tr><tr><td>Xinjiang</td><td>12.95</td><td>17.49</td><td>16.83</td><td>23.05</td><td>29.22</td></tr><tr><td>Gansu</td><td>19.29</td><td>17.64</td><td>21.99</td><td>18.18</td><td>21.24</td></tr><tr><td>Guizhou</td><td>16.33</td><td>13.39</td><td>18.6</td><td>20.34</td><td>24.02</td></tr><tr><td>Xizang</td><td>5.79</td><td>5.82</td><td>14.6</td><td>24.95</td><td>34.00</td></tr><tr><td>Qinghai</td><td>7.30</td><td>17.43</td><td>14.16</td><td>20.67</td><td>19.41</td></tr></table>

from formula (11) that the weight of each index as table 2:

<table><caption>Table 2. The weight of each index</caption><thead><tr><th></th><th>The Utility Value of Knowledge Creation</th><th>The Utility Value of Knowledge Acquisition</th><th>The Utility Value of Enterprise Innovation</th><th>The Utility Value of Innovation Environment</th><th>The Utility Value of Innovation Performance</th></tr></thead><tbody><tr><th>weight</th><td>0.196032</td><td>0.181183</td><td>0.211934</td><td>0.130354</td><td>0.125122</td></tr></tbody></table>

$x_{ij}$ was separately substituted into formula (5), we can get the comprehensive evaluation value of innovation capacity of provinces, autonomous regions and municipalities. We sort the value and compared with literature [22] and the results as shown table 3.

<table><caption>Table 3. The relative equilibrium coefficient, evaluating value of assembled and sorting result of provinces, autonomous regions and municipalities</caption><thead><tr><th rowspan="2">Regions</th><th rowspan="2">Relative Equilibrium Coefficient</th><th colspan="4">Calculations</th></tr><tr><th>Evaluation Value of Literature<sup>[22]</sup></th><th>Evaluation Value of This Paper</th><th>Sorting Result of Literature<sup>[22]</sup></th><th>Sorting Result of This Paper</th></tr></thead><tbody><tr><td>Jiangsu</td><td>1.336999</td><td>52.27</td><td>51.76501</td><td>1</td><td>1</td></tr><tr><td>Guangdong</td><td>2.105035</td><td>51.89</td><td>51.58099</td><td>2</td><td>2</td></tr><tr><td>Beijing</td><td>0.585137</td><td>47.92</td><td>50.9122</td><td>3</td><td>3</td></tr><tr><td>Shanghai</td><td>0.989686</td><td>46.23</td><td>48.38886</td><td>4</td><td>4</td></tr><tr><td>Zhejiang</td><td>0.772414</td><td>41.23</td><td>39.83768</td><td>5</td><td>5</td></tr><tr><td>Shandong</td><td>0.961705</td><td>37.34</td><td>35.98531</td><td>6</td><td>7</td></tr><tr><td>Tianjin</td><td>1.460951</td><td>35.89</td><td>35.69753</td><td>7</td><td>6</td></tr><tr><td>Hubei</td><td>0.928789</td><td>30.61</td><td>29.71281</td><td>8</td><td>10</td></tr></tbody></table>
<table><tr><td>Sichuang</td><td>1.687775</td><td>29.95</td><td>29.50226</td><td>9</td><td>9</td></tr><tr><td>Chongqing</td><td>0.996219</td><td>29.85</td><td>29.31325</td><td>10</td><td>12</td></tr><tr><td>Hunan</td><td>1.57561</td><td>29.79</td><td>29.08532</td><td>11</td><td>8</td></tr><tr><td>Liaoning</td><td>1.021588</td><td>28.93</td><td>28.58283</td><td>12</td><td>11</td></tr><tr><td>Anhui</td><td>0.619462</td><td>28.56</td><td>26.89815</td><td>13</td><td>14</td></tr><tr><td>Shaanxi</td><td>1.00293</td><td>27.79</td><td>26.07996</td><td>14</td><td>13</td></tr><tr><td>Henan</td><td>1.159824</td><td>25.96</td><td>24.88072</td><td>15</td><td>15</td></tr><tr><td>Fujian</td><td>0.921067</td><td>24.16</td><td>22.95871</td><td>16</td><td>17</td></tr><tr><td>Shanxi</td><td>0.814262</td><td>23.83</td><td>22.76698</td><td>17</td><td>16</td></tr><tr><td>Hebei</td><td>1.165102</td><td>23.26</td><td>22.56933</td><td>18</td><td>18</td></tr><tr><td>Heilongjiang</td><td>1.073299</td><td>22.84</td><td>22.21028</td><td>19</td><td>19</td></tr><tr><td>Guangxi</td><td>0.742462</td><td>22.56</td><td>20.98269</td><td>20</td><td>21</td></tr><tr><td>Jilin</td><td>0.566841</td><td>22.2</td><td>20.87774</td><td>21</td><td>24</td></tr><tr><td>Jiangxi</td><td>0.537177</td><td>22.07</td><td>20.71971</td><td>22</td><td>25</td></tr><tr><td>Hainan</td><td>0.622538</td><td>21.95</td><td>20.43453</td><td>23</td><td>20</td></tr><tr><td>Ningxia</td><td>0.826361</td><td>20.89</td><td>20.06905</td><td>24</td><td>26</td></tr><tr><td>Yunnan</td><td>1.350361</td><td>20.74</td><td>19.51379</td><td>25</td><td>23</td></tr><tr><td>Inner Mongolia</td><td>0.608492</td><td>20.46</td><td>19.17662</td><td>26</td><td>27</td></tr><tr><td>Xinjiang</td><td>0.639</td><td>20.38</td><td>18.63339</td><td>27</td><td>28</td></tr><tr><td>Gansu</td><td>2.111367</td><td>19.83</td><td>18.54533</td><td>28</td><td>22</td></tr><tr><td>Guizhou</td><td>0.936691</td><td>19</td><td>17.97067</td><td>29</td><td>29</td></tr><tr><td>Xizang</td><td>0.280583</td><td>18.43</td><td>14.69616</td><td>30</td><td>30</td></tr><tr><td>Qinghai</td><td>0.600275</td><td>16.3</td><td>14.50313</td><td>31</td><td>31</td></tr></table>

The weight of each index provided by the literature [22] was $\omega = \{0.15,0.15,0.25,0.25,0.20\}$, compared with the weight indexes provided by the literature [22], this paper pays more attention on the role of enterprise innovation while studying the regional innovation capability. And accordingly, the weights of innovation environment and innovation performance are relatively reduced. After calculating the comprehensive evaluation value of each region's innovation capability by using the aggregation model which focuses more on equilibrium, we get an evaluation result which is close to the one from literature [22].

During the calculation process, we find that, the innovation capabilities of Jiangsu province, Beijing city, Guangdong province, Shanghai city and Zhejiang province are ranked from 1 to 5 based on the 2 calculation results. That is, in the development process, all of these 5 regions have got leading advantage in innovation even they have different equilibrium in knowledge creation, knowledge acquisition, enterprise innovation, innovation environment and innovation performance.

At the same time it shows that, all the index developments from Guangdong province are in a more balanced pace than the other 4 regions. There is still room for Zhejiang province to improve its capabilities on knowledge creation and knowledge
acquisition. And we also find Shanghai city has obvious shortcomings on innovation environment.

The regions ranked from 6-9 have no obvious differences from these 2 calculations. Most of these 14 provinces, cities and autonomous regions are located in eastern China. Considering the development equilibrium of each index, the innovation capability of Hunan province climbs up 3 places to the 8th comparing with the first time. While Hubei and Chongqing slips down 2 places.

The result shows that the utility value rankings of each index for Hunan province are the 8th, the 10th, the 13th, the 15th and the 9th, which means it has better equilibrium. By contrast of that, Hubei province should improve its capability of knowledge acquisition. And Chongqing should pay more attention on the development of knowledge creation capability.

The countries rank low in west of china, the innovation capabilities of most regions in western China are ranked backward. In the backward ranked regions, the regional innovation capability of Gansu province moves up 6 positions to the 22nd. Hainan and Yunnan move up 3 and 2 positions respectively to the 20th and 23rd. While both of Jilin and Jiangxi declines 3 positions to the 24th and 25th respectively.

The analysis shows that the enterprise innovation capability of Gansu province is ranked at the 2nd in the last 12 regions. In addition, the developments of its knowledge creation, knowledge acquisition, innovation environment and innovation performance are in balance and without obvious differences. Among the last 12 regions, Jilin and Jiangxi have more advantages in the innovation performance than the other regions. But the former should improve its capabilities of knowledge acquisition and enterprise innovation. And the latter should pay more attention on its capabilities of knowledge creation and enterprise innovation.

## 4 Conclusions

For the issue of less consideration on whether the system is in balanced development when evaluating the regional economic development and industry innovation capability, this paper proposes a comprehensive evaluation model which could highlight the ‘Equilibrium’ of each evaluation index development from the evaluation object. And also, this model is applied to the comprehensive evaluation on our country’s regional innovation capabilities. This model not only considers the influence on the evaluation result from each index value, and also, it considers the equilibrium of system development. During the evaluation process, it ‘promotes’ the system which has balanced development in each index. While it ‘punishes’ the system which only focuses on the development of some index and ignores the others’ development. Finally that will aggregate the ‘difference’ and ‘equilibrium’ to get a more reasonable and reliable evaluation result. And it will really work on ‘encourage the developed ones, and motivate the developing ones’.
References

1. Wang Zongjun. On the methods, problems and research trends of comprehensive evaluation. Journal of Management Sciences in China, vol.1, 75-79, July 1998. (in Chinese)

2. Hwang C L, Md Aasud A S. Multiple Objective Decision Making Methods and Applications. Berlin: Spring-Verlag Press, 1979, 2-325.

3. Lichtenberg Fank R. Issues in measuring industrial R&D. Research Policy, 157-163, 1990.

4. Charnes A, Cooper W W, Rhodes E. Measuring the efficiency of decision making units. European Journal of Operational Research, 429-444, Feb 1978.

5. Cooper W W, Tone K. Measures of inefficiency in data envelopment analysis and stochastic frontier estimation. European Journal of Operational Research, 72-78, Feb 1997.

6. Wang Junxia, Guan Jiancheng. The application of composition DEA method in measuring the performance of knowledge management of enterprise. Studies in Science of Science, 84-88, Jan 2002. (in Chinese)

7. He Xiaoqun. The Methods of Modern Statistics Analysis. Beijing: Press of Renming University of China, 1998. (in Chinese)

8. Jacques Savoy. Statistical inference in retrieval effectiveness evaluation. Information Processing and Management, vol.33, 495-512, Apr 1997.

9. Wallmark J Torkel. Quality of research measured by citation method and by peer review-a comparison. IEEE Trans on Engineering Management, vol.4, 218-222, Apr 1986.

10. Saaty T L. Fundamentals of Decision Making and Priority Theory with the Analytic Hierarchy Process. Princeton: RWS Publications, 35-127,1994.

11. Schen K S. Avoiding rank reversal in AHP decision-support models. European Journal of Operational, 36-38, Mar 1998.

12. Carrettoni F, Castano S, et al. RETISS: A real time security system for threat detection using fuzzy logic. In: Proceedings of 25th IEEE International Carnahan Conference on Security Technology. Taipei: 1991.247-269.

13. Chen S J, Hwang C L. Fuzzy Multiple Attribute Decision Making. Berlin: Springer Press, 163-287,1992.

14. Chen Guohong. Fuzzy mode identification in R&D project termination. Studies in Science of Science, vol.16, 68-74, Jan 1998. (in Chinese)

15. Dimitras A I, Slowinski R, Susmaga R, et al. Business failure prediction using rough sets. European Journal of Operational Research, vol.95, 24-37,1999.

16. Grabowski M R, Wallace W A. An expert system for maritime pilots: Its design and assessment using gaming. Management Science, vol.3, 1506-1520, Dec 1993.

17. Zhiping Fan, Quan Zhang, Jian Ma. An integration weight determine method of multiple attribute decision making. Journal of Management, vol.1, 50-53, Mar 1998. (in Chinese)

18. Yingmin Wang. Multiple attribute decision making and sorting with maximizing deviations method. China Soft Science, 36-38, Mar 1998. (in Chinese)

19. Mingtao Wang. Weight coefficient to determine the deviation and mean square deviation decision method in multi-index evaluation. China Soft Science, 100-103, Aug 1999. (in Chinese)

20. Zhiping Fan, Xuan Zhao. The subjective and objective method in multiple attribute decision making. Decision Making and Decision Support System. Vol.7, 87-91, Apr 1997. (in Chinese)

21. Changlin Mei, Jincheng Fan. Data Analysis Method. Beijing: Higher Education Press, 2006. (in Chinese)

22. Yajun Guo, Tianli Zhong. The report of Chinese regional innovation capacity of 2011-Study of regional innovation system in the Pearl River Delta. Beijing: Science Press, 2012.