# Weighting of Basic Indices for Value for Money Qualitative Evaluation of the Public-Private Partnership Aged Care Project through the Analytic Hierarchy Process

Authors: Song Ye; Ying Chen; Guangao Mao
Journal: Discrete Dynamics in Nature and Society
 (2021)

---

## Abstract

The evaluation of value for money (VFM) is an essential part of the pre-evaluation of public-private partnership (PPP) projects in China. It can be divided into qualitative evaluation and quantitative evaluation. However, there is no uniform weighting criterion for the indices of qualitative evaluation. To reasonably weigh the qualitative evaluation indices, this paper sorts and measures these indices through the analytic hierarchy process (AHP), using the public data on preapplied PPP aged care projects, and obtains the reference values. On this basis, several suggestions were put forward to standardize and develop VFM evaluation of PPP aged care projects, such as improve the scientific VFM evaluation system and realize whole-process performance appraisal. Based on actual project data, the qualitative evaluation is more objective than the analysis based on subjective experience. Therefore, our findings promote the application of VFM qualitative evaluation.


---

## Body

## 1. Introduction

The population of China is quickly ageing. However, nearly 90% of the elderly only receive home-based aged care, which is difficult given the small family structure in the fast-paced modern society. The soaring demand for aged care can hardly be met through social aged care projects because of their quasi-public products, long payback period, low rate of return, and complex risk factors. The ill-preparedness of social capital initiatives adds a heavy burden to the government, but neither could government investment satisfy the rising demand.Public-private partnership (PPP) refers to the cooperation between the government and social capital. The PPP Center of the Chinese Ministry of Finance has preapplied 105 PPP aged care projects and reserved 66 such projects. The adoption of PPP in aged care is lower than that in most other industries.Whether the PPP mode is suitable for a project is usually demonstrated through the evaluation of value for money (VFM). The Chinese Ministry of Finance once released theGuidelines of Value for Money Evaluation, which stipulate that VFM evaluation could be carried out qualitatively or quantitatively. The indices of qualitative evaluation include the degree of lifecycle integration, risk identification and allocation, performance orientation and innovation encouragement, potential competitiveness, government capacity, and financing ability. Together, these indices explain 80% of the VFM. The guidelines require project parties to form an expert group to assign weight to each of the six indices. As a result, the qualitative evaluation indices often have inconsistent weights across projects.Admittedly, some of the weight differences are necessary to reflect the unique features and needs of specific projects. However, the inconsistent weighting criteria make VFM qualitative evaluation formalistic and unable to support scientific decision-making. To solve the problem, this paper collects the data from many mature preapplied PPP aged care projects and discusses the reasonable weights of the basic indices in VFM qualitative evaluation through the analytic hierarchy process (AHP), thereby standardizing the evaluation system and improving the PPP aged care model.


## 2. Literature Review

### 2.1. Development of PPP Aged Care

In 2015, the Chinese Ministry of Finance issued theGuidelines of Value for Money Evaluation, kicking off a round of heated discussion on the PPP mode.Kang et al. [1] examined the feasibility of applying the PPP mode to the aged care industry from the perspective of economic applicability. Ng et al. [2] identified the operation defects of Hong Kong’s home care industry, proposed a conceptual framework to reveal the correlation governance system and the complementary system of performance evaluation, and relied on the framework to monitor the performance of social enterprises in fulfilling their social responsibilities. Zhang [3] suggested optimizing the construction and operation of aged care projects with the incentive mechanism and healthcare mechanism. In the context of the Internet Plus and PPP mode, Hao et al. [4] designed the development path that combines intelligent aged care with the PPP medical support project.Many scholars have noticed the low reporting and landing rates of PPP aged care projects and put forward measures to improve the quality evaluation system and credit constraint mechanism, trying to optimize project design and payment methods, and step up the supervision of service quality [5–7]. Liao and Liao and Xu proposed innovative ideas to break the development dilemma of PPP aged care projects from the perspective of cooperative governance and equity cooperation, respectively [8, 9].To sum up, the research on the PPP aged care is largely macroscopic, focusing on the development model and path. The VFM of PPP aged care projects has rarely been evaluated.


### 2.2. VFM Qualitative Evaluation of PPP Projects

The previous studies on the VFM evaluation of PPP projects mainly attempt to optimize the strategies of qualitative and quantitative evaluations.To overcome the limitation of traditional economic evaluation, Peterson and Skolits [10] suggested that the VFM framework may incorporate evidence collected both qualitatively and quantitatively, making the information more relevant to primary users. Based on actual PPP cases, Callens et al. [11] followed the logic of innovative procurement and collaborative innovation and compared fuzzy sets with in-depth qualitative data. Ren et al. [12] introduced building information modeling (BIM) into VFM evaluation and realized the automatic calculation, which partly or fully replaces subjective empirical evaluation.Through the AHP, Xiang et al. [13] optimized the influencing factors of VFM evaluation in PPP projects. Deng et al. carried out fuzzy comprehensive evaluation (FCE) to identify the key success factors of PPP projects [14]. Cui et al. [15] confirmed and verified the relationship between VFM drivers of PPP projects through questionnaire survey and scanning electron microscopy (SEM). Wu et al. [16] calculated index weights by the entropy method and constructed a VFM qualitative evaluation system for PPP renovation projects of old industrial buildings.The above studies primarily redefine and select qualitative indices or establish secondary indices under each primary index of qualitative evaluation.


### 2.3. Summary

The above review shows that empirical analysis is frequently used to deduce the development of PPP aged care projects from individual cases and optimize the path of development. In contrast, few scholars have designed index systems for VFM qualitative evaluation of PPP aged care projects. Moreover, the existing index systems are mostly expanded and redefined from old systems. There is virtually no research that weighs the indices of VFM qualitative evaluation for actual preapplied PPP aged care projects. To solve these problems, this paper introduces the AHP to analyze the PPP project database of the Chinese Ministry of Finance and assign reasonable weights to VFM qualitative evaluation indices, aiming to standardize the operation of PPP aged care projects.


## 2.1. Development of PPP Aged Care

In 2015, the Chinese Ministry of Finance issued theGuidelines of Value for Money Evaluation, kicking off a round of heated discussion on the PPP mode.Kang et al. [1] examined the feasibility of applying the PPP mode to the aged care industry from the perspective of economic applicability. Ng et al. [2] identified the operation defects of Hong Kong’s home care industry, proposed a conceptual framework to reveal the correlation governance system and the complementary system of performance evaluation, and relied on the framework to monitor the performance of social enterprises in fulfilling their social responsibilities. Zhang [3] suggested optimizing the construction and operation of aged care projects with the incentive mechanism and healthcare mechanism. In the context of the Internet Plus and PPP mode, Hao et al. [4] designed the development path that combines intelligent aged care with the PPP medical support project.Many scholars have noticed the low reporting and landing rates of PPP aged care projects and put forward measures to improve the quality evaluation system and credit constraint mechanism, trying to optimize project design and payment methods, and step up the supervision of service quality [5–7]. Liao and Liao and Xu proposed innovative ideas to break the development dilemma of PPP aged care projects from the perspective of cooperative governance and equity cooperation, respectively [8, 9].To sum up, the research on the PPP aged care is largely macroscopic, focusing on the development model and path. The VFM of PPP aged care projects has rarely been evaluated.


## 2.2. VFM Qualitative Evaluation of PPP Projects

The previous studies on the VFM evaluation of PPP projects mainly attempt to optimize the strategies of qualitative and quantitative evaluations.To overcome the limitation of traditional economic evaluation, Peterson and Skolits [10] suggested that the VFM framework may incorporate evidence collected both qualitatively and quantitatively, making the information more relevant to primary users. Based on actual PPP cases, Callens et al. [11] followed the logic of innovative procurement and collaborative innovation and compared fuzzy sets with in-depth qualitative data. Ren et al. [12] introduced building information modeling (BIM) into VFM evaluation and realized the automatic calculation, which partly or fully replaces subjective empirical evaluation.Through the AHP, Xiang et al. [13] optimized the influencing factors of VFM evaluation in PPP projects. Deng et al. carried out fuzzy comprehensive evaluation (FCE) to identify the key success factors of PPP projects [14]. Cui et al. [15] confirmed and verified the relationship between VFM drivers of PPP projects through questionnaire survey and scanning electron microscopy (SEM). Wu et al. [16] calculated index weights by the entropy method and constructed a VFM qualitative evaluation system for PPP renovation projects of old industrial buildings.The above studies primarily redefine and select qualitative indices or establish secondary indices under each primary index of qualitative evaluation.


## 2.3. Summary

The above review shows that empirical analysis is frequently used to deduce the development of PPP aged care projects from individual cases and optimize the path of development. In contrast, few scholars have designed index systems for VFM qualitative evaluation of PPP aged care projects. Moreover, the existing index systems are mostly expanded and redefined from old systems. There is virtually no research that weighs the indices of VFM qualitative evaluation for actual preapplied PPP aged care projects. To solve these problems, this paper introduces the AHP to analyze the PPP project database of the Chinese Ministry of Finance and assign reasonable weights to VFM qualitative evaluation indices, aiming to standardize the operation of PPP aged care projects.


## 3. Methodology

### 3.1. Technological Path

As of April 2021, there were 105 preapplied PPP aged care projects in the PPP project database of the PPP Center of the Chinese Ministry of Finance. These projects are distributed across 26 provinces and autonomous regions. According to theGuidelines for VFM Evaluation, VFM evaluation is a necessary procedure for PPP projects. Six basic indices are defined in the guidelines, namely, the degree of lifecycle integration, risk identification and allocation, performance orientation and innovation encouragement, potential competitiveness, government capacity, and financing ability. For simplicity, the six indices are hereinafter referred to as I1–I6, in turn. Since the cumulative weight of these indices is limited to 80%, the weight of any index should not surpass 20%. During the VFM evaluation, the number and definition of other indices are fuzzy, small, and project-specific. Therefore, this paper mainly focuses on the weights of the six basic indices.


### 3.2. Data Collection

The preapplied projects in China are located in different regions. Each of them is highly unique in terms of geography, economy, and policy. Here, the research data are collected from the preapplied projects in six provinces of Central China. These provinces are remarkably consistent and outstanding examples because they cover more than one-tenth of the Chinese territory, raise over one-quarter of the Chinese population, and contribute more than one-fifth of China’s economic output. These provinces are highly common in development as none of them lies on the border or coastlines.


### 3.3. Modeling Method

In the six provinces, there are a total of 30 preapplied PPP aged care projects. Ten of them lack public data on qualitative evaluation indices. For the remaining 20 projects, the weights of the 6 basic indices vary significantly from project to project. In each of the 20 projects, the weights of all indices were given by expert groups and could be regarded as expert scores. In theory, an index with a high score is deemed by the experts as more important than other indices. Therefore, the AHP was introduced to sort the indices of the 20 projects by importance, such as to rationalize the weight of each index.


## 3.1. Technological Path

As of April 2021, there were 105 preapplied PPP aged care projects in the PPP project database of the PPP Center of the Chinese Ministry of Finance. These projects are distributed across 26 provinces and autonomous regions. According to theGuidelines for VFM Evaluation, VFM evaluation is a necessary procedure for PPP projects. Six basic indices are defined in the guidelines, namely, the degree of lifecycle integration, risk identification and allocation, performance orientation and innovation encouragement, potential competitiveness, government capacity, and financing ability. For simplicity, the six indices are hereinafter referred to as I1–I6, in turn. Since the cumulative weight of these indices is limited to 80%, the weight of any index should not surpass 20%. During the VFM evaluation, the number and definition of other indices are fuzzy, small, and project-specific. Therefore, this paper mainly focuses on the weights of the six basic indices.


## 3.2. Data Collection

The preapplied projects in China are located in different regions. Each of them is highly unique in terms of geography, economy, and policy. Here, the research data are collected from the preapplied projects in six provinces of Central China. These provinces are remarkably consistent and outstanding examples because they cover more than one-tenth of the Chinese territory, raise over one-quarter of the Chinese population, and contribute more than one-fifth of China’s economic output. These provinces are highly common in development as none of them lies on the border or coastlines.


## 3.3. Modeling Method

In the six provinces, there are a total of 30 preapplied PPP aged care projects. Ten of them lack public data on qualitative evaluation indices. For the remaining 20 projects, the weights of the 6 basic indices vary significantly from project to project. In each of the 20 projects, the weights of all indices were given by expert groups and could be regarded as expert scores. In theory, an index with a high score is deemed by the experts as more important than other indices. Therefore, the AHP was introduced to sort the indices of the 20 projects by importance, such as to rationalize the weight of each index.


## 4. Empirical Analysis

### 4.1. Hierarchical Structure Model

Drawing on the AHP theory and our research purpose, this paper establishes a hierarchical structure model for the weighting of the basic indices in VFM qualitative evaluation of PPP aged care projects (Figure1).Figure 1
Hierarchical structure model for the weighting of the basic indices.Note. P1–P20 refer to the 20 projects.As shown in Figure1, this research only focuses on the weights of the six basic indicators, without comparing project alternatives. Therefore, some theories and processes of the AHP were applied in the following analysis.


### 4.2. Original Data

Table1 displays the weights of the basic indices for VFM qualitative evaluation of the 20 projects.Table 1
Weights of basic indices (unit: %).
I1I2I3I4I5I6P1151515151010P2161515101212P3151515101015P4181510101512P5151020151010P6181512101510P715152015510P815152015510P9151515151010P10151015101515P11101515101515P12101515101515P13151515101015P1415152015510P1515151520510P16151510102010P1715152015510P18151510151015P1915152015510P2015152015510Avg14.8514.5015.8513.0010.1011.70


### 4.3. Pairwise Comparison Matrix

Each index of the 20 projects has a weight, i.e., the degree of importance, determined by the project expert group. This weight can be regarded as an expert score, which is positively correlated with the importance of the corresponding index. Through pairwise comparison and matrix judgment on the 9-level scale (Table2), the mean difference of the six indices was not very big. Taking the mean difference of 1.5 as the first-level scale of pairwise comparison, the mean value of I1 was found to be 14.85. Hence, I2, with a mean of 14.50, was considered as important as I1, namely, a12 = 1; I3, with a mean of 15.85, was slightly more important than I4, that is, a34 = 2; I3 was little more important than I5, whose mean was 10.10, that is, a35 = 3. The rest could be deduced by analogy. In this way, a judgment matrix can be constructed as shown in Table 3.Table 2
Nine-level scale for the judgment of matrixaij.
ScalesMeanings1Indicating that factori is of equal importance to factor j in comparison between the two factors3Indicating that factori is little more important than factor j in comparison between the two factors5Indicating that factori is obviously a little more important than factor j in comparison between the two factors7Indicating that factori is strongly more important than factor j in comparison between the two factors9Indicating that factori is particularly more important than factor j in comparison between the two factors2, 4, 6, 8The middle assignment of the above two adjacent judgmentsReciprocalThe effect of factori compared to factor j is the inverse of aijTable 3
Pairwise comparison matrix.
I1I2I3I4I5I6I1111232I2111132I3111232I41/211/2121I51/31/31/31/211/2I61/21/21/2121


### 4.4. Single-Level Sorting

The weight vector was calculated by the product-sum method. Firstly, the geometric meanai of all factors in each row of the matrix was calculated by(1)ai=∏j=1naijn,i=1,2,…,n.Then, the weight vectorWi was derived by normalizing ai, namely, W1, W2, … W6 (Table 4):(2)Wi=ai∑i=1nai.Table 4
Consistency test of the AHP.
I1I2I3I4I5I6WiAWiλI11112320.231.406.026I21111320.211.266.103I31112320.231.406.026I41/211/21210.140.846.131I51/31/31/31/211/20.070.426.020I61/21/21/21210.120.736.033


### 4.5. Consistency Test

The maximum eigenvalueλmax of the judgment matrix is calculated and recorded in Table 4. For the incomplete yet uniform matrix A, λ > n (n = 6) meets the requirement of λ ≥ n of the n-order positive reciprocal matrix A:(3)λmax=∑AWinWi.Then, the consistency index CI was imported to the calculation. The smaller the CI, the stronger the consistency.(4)CI=λmax−nn−1.The calculation results show that CI = 0.0114, which is close to zero. In theory, the CI value indicates satisfactory consistency. Since the result is not equal to zero, it is necessary to introduce the random consistency index RI for further test. Hence, the consistency ratio CR was calculated:(5)CR=CIRI.The standard values of RI are directly related to the ordern of the matrix. When n = 6, RI is directly taken as 1.24 by looking up the table of standard values, and thus, CR = 0.01 < 0.1. Hence, the comparison matrix A was verified to be satisfactorily consistent.


### 4.6. Weighting of Basic Indices

The above test proves that the weightWi of each basic index (Table 4) is valid. Thus, the reference values for the weights of the basic indices were recommended under the condition that the cumulative weight is 80% (Table 5).Table 5
Reference weights of the basic indices (unit: %).
Whole lifecycle integration degreeRisk identification and distributionPerformance orientation and innovation encouragementPotential competitivenessGovernment agency capacityFinancing ability18171811610


## 4.1. Hierarchical Structure Model

Drawing on the AHP theory and our research purpose, this paper establishes a hierarchical structure model for the weighting of the basic indices in VFM qualitative evaluation of PPP aged care projects (Figure1).Figure 1
Hierarchical structure model for the weighting of the basic indices.Note. P1–P20 refer to the 20 projects.As shown in Figure1, this research only focuses on the weights of the six basic indicators, without comparing project alternatives. Therefore, some theories and processes of the AHP were applied in the following analysis.


## 4.2. Original Data

Table1 displays the weights of the basic indices for VFM qualitative evaluation of the 20 projects.Table 1
Weights of basic indices (unit: %).
I1I2I3I4I5I6P1151515151010P2161515101212P3151515101015P4181510101512P5151020151010P6181512101510P715152015510P815152015510P9151515151010P10151015101515P11101515101515P12101515101515P13151515101015P1415152015510P1515151520510P16151510102010P1715152015510P18151510151015P1915152015510P2015152015510Avg14.8514.5015.8513.0010.1011.70


## 4.3. Pairwise Comparison Matrix

Each index of the 20 projects has a weight, i.e., the degree of importance, determined by the project expert group. This weight can be regarded as an expert score, which is positively correlated with the importance of the corresponding index. Through pairwise comparison and matrix judgment on the 9-level scale (Table2), the mean difference of the six indices was not very big. Taking the mean difference of 1.5 as the first-level scale of pairwise comparison, the mean value of I1 was found to be 14.85. Hence, I2, with a mean of 14.50, was considered as important as I1, namely, a12 = 1; I3, with a mean of 15.85, was slightly more important than I4, that is, a34 = 2; I3 was little more important than I5, whose mean was 10.10, that is, a35 = 3. The rest could be deduced by analogy. In this way, a judgment matrix can be constructed as shown in Table 3.Table 2
Nine-level scale for the judgment of matrixaij.
ScalesMeanings1Indicating that factori is of equal importance to factor j in comparison between the two factors3Indicating that factori is little more important than factor j in comparison between the two factors5Indicating that factori is obviously a little more important than factor j in comparison between the two factors7Indicating that factori is strongly more important than factor j in comparison between the two factors9Indicating that factori is particularly more important than factor j in comparison between the two factors2, 4, 6, 8The middle assignment of the above two adjacent judgmentsReciprocalThe effect of factori compared to factor j is the inverse of aijTable 3
Pairwise comparison matrix.
I1I2I3I4I5I6I1111232I2111132I3111232I41/211/2121I51/31/31/31/211/2I61/21/21/2121


## 4.4. Single-Level Sorting

The weight vector was calculated by the product-sum method. Firstly, the geometric meanai of all factors in each row of the matrix was calculated by(1)ai=∏j=1naijn,i=1,2,…,n.Then, the weight vectorWi was derived by normalizing ai, namely, W1, W2, … W6 (Table 4):(2)Wi=ai∑i=1nai.Table 4
Consistency test of the AHP.
I1I2I3I4I5I6WiAWiλI11112320.231.406.026I21111320.211.266.103I31112320.231.406.026I41/211/21210.140.846.131I51/31/31/31/211/20.070.426.020I61/21/21/21210.120.736.033


## 4.5. Consistency Test

The maximum eigenvalueλmax of the judgment matrix is calculated and recorded in Table 4. For the incomplete yet uniform matrix A, λ > n (n = 6) meets the requirement of λ ≥ n of the n-order positive reciprocal matrix A:(3)λmax=∑AWinWi.Then, the consistency index CI was imported to the calculation. The smaller the CI, the stronger the consistency.(4)CI=λmax−nn−1.The calculation results show that CI = 0.0114, which is close to zero. In theory, the CI value indicates satisfactory consistency. Since the result is not equal to zero, it is necessary to introduce the random consistency index RI for further test. Hence, the consistency ratio CR was calculated:(5)CR=CIRI.The standard values of RI are directly related to the ordern of the matrix. When n = 6, RI is directly taken as 1.24 by looking up the table of standard values, and thus, CR = 0.01 < 0.1. Hence, the comparison matrix A was verified to be satisfactorily consistent.


## 4.6. Weighting of Basic Indices

The above test proves that the weightWi of each basic index (Table 4) is valid. Thus, the reference values for the weights of the basic indices were recommended under the condition that the cumulative weight is 80% (Table 5).Table 5
Reference weights of the basic indices (unit: %).
Whole lifecycle integration degreeRisk identification and distributionPerformance orientation and innovation encouragementPotential competitivenessGovernment agency capacityFinancing ability18171811610


## 5. Comparative Analysis

Before this study, Kang et al. measured and sorted the financing risks of the PPP mode through the AHP. The measured objects were the self-designed risk levels [1]. Their results are consistent with our study, but point to different directions. Using the analytic network process (ANP), Wu et al. weighed the influencing factors of VFM qualitative evaluation, including the basic and supplementary indices in the Guidelines for VFM Evaluation, as well as other influencing factors. Their results provide a limited guidance of subsequent projects, which extends the relevant studies on evaluation indicators and influencing factors. By contrast, the target of this study is the existing index system, and the results of this study directly guide the index weighting of VFM qualitative evaluation for PPP projects. Our results deviated slightly from the recommended values of 18%, 19%, 19%, 5%, 16%, and 9% for the six indices of VFM qualitative evaluation obtained by the entropy method [16]: the potentially competitive indices have higher weights, while the index of government capacity has a lower weight. From previous failed PPP cases, many projects fail to achieve the expected operating results after completion [17] due to inaccurate estimation of potential competitiveness. If the rights and obligations of both parties can be reasonably restricted during contract conclusion and realize standardized programmatic management, the government capacity does not need to be too high. After all, PPP project cooperation intends to introduce the efficient management and operational capacity of social capital.


## 6. Conclusions

This paper relies on the AHP to analyze the data on the basic indices of VFM qualitative evaluation for 20 PPP aged care projects in six provinces of Central China. The results show that the weight scores ofI1 and I3 were the highest among the six basic indices and close to the upper limit of 20%. They were followed by I2, which reflects the risk-sharing and mutually beneficial cooperation of PPP projects and the intention of improving operation efficiency and innovation ability. On this basis, two suggestions were presented to optimize the VFM qualitative evaluation of PPP aged care projects.First, we should improve the scientific evaluation system. First of all, it is necessary to ensure the integrity and accuracy of the information of the evaluation object and strengthen the internal and external communication of the expert group to obtain accurate and correct evaluation results. Secondly, qualitative evaluation subindexes should be determined, and a relatively complete multilevel evaluation index system should be established to facilitate expert evaluation and reference. Finally, it is hoped to refine the index scoring standard to avoid the difference between the basic index scoring of different projects, which reduces the credibility of qualitative evaluation.Second, the establishment of after-the-fact evaluation and performance appraisal mechanism: VFM evaluation is in the preliminary demonstration stage of the project, and it is crucial to determine whether the project is built or not. The VFM evaluation results should be combined with the project construction and operation performance during the whole lifecycle of the project, so as to strengthen the seriousness and accuracy of the VFM evaluation and ensure the objective and fair evaluation results.



---

*Source: 1227657-2021-09-22.xml*