Millennium Challenge Corporation: Independent Reviews and Consistent Approaches Will Strengthen Projections of Program Impact
Released: 2008-06-17T00:00:00

Letter

Background

MCC conducts its economic analyses during the compact development process, the first of three compact phases. These analyses include establishing the minimum ERR that the compact should achieve, projecting the compact’s ERR, and estimating the compact’s impact on income and growth. MCC did not publish ERR information for three of the four compacts we reviewed, but its public documents include statements of compact impact on income and poverty. Of the four compacts that we reviewed—Armenia, El Salvador, Lesotho, and Mozambique—three include funding for road projects, and three include funding for water projects, among other projects.

Compact Development Process

If MCC determines that a country is eligible for assistance, the country may submit a compact proposal, which generally comprises several projects. According to MCC guidelines, the proposal should include economic analyses of proposed projects to demonstrate their likely impact on growth and poverty in the country. MCC also may provide assistance and feedback in developing the proposal, including making grants to facilitate the development of the compact (see fig. 1).

After a country submits its proposal, MCC’s transaction team for the compact conducts a preliminary assessment of the proposal and reports its findings in an internal opportunity memo to the MCC investment committee. MCC assembles a different transaction team for each compact. If the opportunity memo is approved, the team launches a detailed due diligence review that includes economic analyses of the proposed projects. As members of MCC country transaction teams, MCC’s lead economists undertake these analyses while working with other members of the team and country officials. According to MCC, as part of this effort, MCC economists review the preliminary analyses performed by country counterparts and consultants. These due diligence reviews, including conducting economic analyses, have lasted, on average, slightly more than 10 months for the 16 countries with signed compacts as of March 31, 2008.

At the conclusion of due diligence, the transaction team sends an investment memo to the MCC investment committee, with recommendations based on its assessment of the proposal. MCC notifies Congress 15 days prior to beginning negotiations with the country, sending a formal congressional notification. For the four compacts we examined, the congressional notifications included some of the results of MCC’s economic analyses. If compact negotiations with the eligible country are successful, the investment committee submits the proposed compact to the MCC Board for approval. With the Board’s approval, MCC and the country sign the compact before completing additional agreements—such as a disbursement agreement and procurement agreement—and ultimately implementing the projects funded in the compact.

Economic Analyses

For each compact proposal, MCC and the eligible country conduct economic analyses, which MCC uses to inform its decisions to fund compacts and report to Congress and the public. MCC generally conducts these analyses at the compact and project levels (see fig. 2).

Aggregte economic rte of retrn (ERR)

The economic analyses include projections of compacts’ impact, including their ERR, the impact on income and poverty, and the projected number of beneficiaries.

ERR analysis. To provide a basis for assessing the compact and project ERRs, MCC sets the minimum acceptable ERR that compacts and projects should achieve to be eligible for funding. (See app. II for a discussion of MCC’s minimum ERRs for the four countries.) If the compact or project does not meet the minimum rate, MCC retains the discretion to fund it but requires justification based on the specific circumstances.

The ERR analysis compares costs and benefits, where the costs are the MCA grants and the country’s future recurrent costs—such as maintenance expenses—and the benefits are increases in incomes in recipient countries. (See fig. 3.) MCC guidelines state that normal practice is to calculate ERRs using 10-, 20-, and 30-year time horizons to determine project and compact ERRs’ sensitivity to varying time frames. If the ERR is sensitive to this time horizon, the guidelines require that this be noted explicitly.

Impact analysis. MCC’s guidelines state that economic analyses should quantify the proposed projects’ expected beneficiaries and expected impact on incomes and on poverty. Based on our analysis, MCC does not establish minimum thresholds for compact impact.

Public Reporting

While MCC’s transaction team reports both ERR and impact projections to the MCC investment committee in its investment memo, MCC’s public reporting for the four compacts in our review generally included information about only its projections of compact impact; for these four countries MCC published ERR projections for only Armenia. MCC’s public reporting included projections of compact impact on income and poverty for three countries and projections of gross domestic product (GDP) growth for one country. For each compact, MCC also estimated the number of beneficiaries. (See table 1.)

Compact Projects and Funding

The types of projects included in the four compacts vary, although El Salvador, Lesotho, and Mozambique include water projects and Armenia, El Salvador, and Mozambique include road projects. The compacts provide a total of approximately $1.56 billion in MCA assistance. (See fig. 4.)

MCC Used Different Time Frames and Methods to Calculate ERRs but Is Taking Steps to Increase Consistency

MCC used different time frames to calculate project-level ERRs; these differences did not affect the ERR significantly in 18 of the 20 compact projects we examined, but two ERRs would fall below the minimum ERR if MCC applied the 20-year time frame it used for other compacts and similar projects. At the compact level, the different time frames did not change the ERR significantly. In addition, we found that MCC used varying methods to account for the costs of same-sector projects, although its approaches to determining benefits were generally similar. MCC also used two different methods to calculate compact-level ERR; however, the choice of method did not change the ERRs significantly. In some cases, MCC did not fully retain its documentation of its economic analyses. MCC has recently taken steps to standardize elements of its economic analyses and improve its records management and plans to implement additional measures.

MCC Used Different Time Frames for Project and Compact ERRs

In its ERR calculations for 20 projects included in the four compacts we reviewed, MCC used a 20-year time frame for 9 projects and used different time frames for 11 projects. Nearly all project ERRs that MCC initially calculated met the minimum ERR set for each compact. Our analysis shows that for 9 of the 11 ERRs calculated with other time frames, recalculating the ERR with a 20-year time frame does not lower the ERR below the minimum ERR. However, for two projects, recalculating the ERR with a 20-year time frame produces a result below the minimum ERR.

The ERR for the El Salvador Community Infrastructure project, which MCC calculated as slightly below the minimum ERR of 10.8 percent over 25 years, drops to 9.0 percent when calculated over 20 years. Correcting a calculation error further reduces the El Salvador project’s ERR to 6.8 percent over 20 years—four percentage points below the El Salvador minimum ERR.

The ERR for the Mozambique roads project, which MCC calculated at 10.3 percent over 24 years, drops to 8.1 percent over 20 years, below the minimum ERR of 8.76 percent. Three of the four individual Mozambique roads MCC analyzed as part of this project also fall below the minimum ERR at 20 years.

If MCC had applied a 20-year time frame in calculating the ERR for these two projects, the projects might have been restructured to increase their ERR or MCC would have had to specifically justify the exception. MCC does not currently have a policy addressing what steps to take in cases where subsequent analysis results in a project ERR below the minimum. MCC stated that it would address changes such as this on a case-by-case basis depending on the timing of the change within the compact development process, and the magnitude of the change. MCC officials also noted that the ERR time frame is only one aspect of MCC’s analysis of the ERR’s sensitivity to various factors.

In addition, MCC used different time frames in calculating ERRs for comparable projects in different compacts. MCC calculated water project ERRs for El Salvador, Lesotho, and Mozambique over 25, 20, and 21 years, respectively, and calculated road project ERRs for Armenia, El Salvador, and Mozambique over 20, 25, and 24 years, respectively. MCC officials told us they explored the ERRs’ sensitivity to varying time frames, as MCC’s guidelines require, but they did not regard as prescriptive the guidelines’ statement that normal practice is to examine 10-, 20-, and 30-year time horizons. MCC officials also noted that MCC economists have recently committed to the use of a default 20-year time horizon for their analyses. MCC also will alter this time frame for specific circumstances or projects whose benefits have a longer duration—such as education projects and large-scale construction projects—or have a shorter or longer physical life expectancy.

At the compact level, our analysis shows that applying a 20-year time frame in place of varying time frames does not significantly affect the results of the ERR calculations. (See app. III for a summary of our comparison of the compact minimum ERR with the compact ERRs that MCC reported and our calculations of the 20-year ERRs.)

MCC Used Varying Methods in Calculating Project and Compact ERRs

In calculating project ERRs, we found that MCC used varying methods to account for the investment and recurrent cost components of water and road sector projects for the four compacts we reviewed.

Investment costs. For water projects, MCC counted the cost of construction in Lesotho as one lump sum in the first year of the analysis but phased in the cost in Mozambique and El Salvador over multiple years. Phasing the costs evenly over the 5-year compact in Lesotho would have increased the ERR for both the urban and rural water projects. For road projects, MCC phased the investment cost over time for Armenia, El Salvador, and Mozambique. However, in El Salvador, MCC accounted for the salvage value of the project in the last year of the analysis but did not include a similar projection in the analysis for Armenia and Mozambique.

Recurrent costs. For water projects, MCC counted Lesotho and Mozambique’s recurrent costs in the years after the original compact investment. For El Salvador, MCC counted recurrent costs only during the initial 5-year compact period. Including future recurrent costs would have reduced the ERR. For road projects, MCC spread recurrent costs over time for Armenia, El Salvador, and Mozambique.

According to our review, MCC’s approaches to determining the benefits of water and road projects were generally similar.

Water projects. MCC generally counted as water project benefits increased time available for work, resulting from less time spent fetching water or being ill, and lower spending on health care and water.

Road projects. MCC generally estimated traffic volumes on the roads and determined road project benefits based on the savings from reduced travel time and vehicle operating costs for existing and generated traffic.

In calculating the compact-level ERRs for the four compacts’ investment memos, MCC used two different methods.

For Armenia and Mozambique, MCC first determined the net benefits in each year for each project and then calculated the compact ERR based on the total net benefits.

For El Salvador and Lesotho, MCC first determined each project’s net benefits and ERR and then determined the compact ERR by averaging the project ERRs, weighting each ERR according to the project’s budgeted size relative to the overall compact.

According to MCC officials, the choice of method depended on the preference of the transaction team’s lead economist. Our analysis shows that using the second method to recalculate each compact ERR reduces it by less than 2 percentage points and does not reduce it below the minimum ERR. However, these results demonstrate that the choice of method influences the compact-level ERR and may affect the comparability of ERRs across compacts. (See app. III for details of our analysis of MCC compact ERRs.)

Insufficient Records Management Led to Discrepancies between Investment Memos and Underlying Analyses

In three of the four cases we reviewed, the projections of ERR reported to the MCC investment committee differ from those in MCC’s underlying spreadsheets.

For Armenia, MCC changed its estimate of project benefits after the investment memo.

For Lesotho, the MCC spreadsheets contain figures calculated using different methods or reflecting additional analysis after the investment memo.

For Mozambique, the calculation of the figures presented in the investment memo relied on spreadsheet formula links that were not properly updated at the time. According to MCC, when they provided the spreadsheets to us, they updated the formulas and overwrote the original calculations.

For these three compacts, MCC’s records management did not fully preserve the information and analysis used to support the investment decision. OMB guidance and GAO guidelines for internal controls both note the importance of controls over the information that U.S. agencies use to make decisions.

MCC Has Taken Several Steps to Improve the Consistency of ERR Analysis

After completing due diligence for the four compacts we studied, and during the course of our audit, MCC told us they took or began taking steps to increase the consistency of its analyses and improve its records management. For example, MCC committed to using 20 years as the default time frame for calculating ERRs and identified projects where this time frame may be altered; began revising its guidance to clarify that ERR sensitivity should be tested by varying the time frame of the analysis, rather than specifying 10-, 20-, and 30-year analyses; began developing standards for consistently analyzing certain types of projects across compacts; established that summing net benefits and costs across all projects is the preferred method for calculating a compact-level ERR; and planned to implement a data management system to centralize its records management.

MCC Made Analytic Errors in Compact Impact Projections and Used Varying Methods that Affected the Projections’ Results

MCC identified and corrected analytic errors in its projections of compact impact for Armenia, El Salvador, and Mozambique, generally reducing each compact’s estimated impact on income and poverty. According to MCC, the revised calculations would not have affected its approval of the compacts, but it will perform peer reviews of future impact projections. In addition, MCC used different methods for projecting compact impact on income and poverty, limiting the estimates’ comparability and replicability; its current guidance does not address the choice of method for these projections. MCC also used different methods to estimate numbers of compact beneficiaries, but stated that it is taking steps to provide more detailed guidance for estimating number of beneficiaries.

MCC Identified and Corrected Analytic Errors for Three Compacts

In responding to our questions about its impact analyses for Armenia, El Salvador, and Mozambique, MCC identified a number of analytic errors in its projections of impact on income and poverty. MCC subsequently corrected these errors, generally reducing the projected impacts on income and poverty for each compact.

Armenia. MCC determined that it had included road project benefits in estimating the income increase from agriculture for Armenia. MCC also used the wrong baseline in projecting poverty effects. Correcting these errors affected projections of the compact’s effect, reducing the estimated increase in rural areas’ real income from agriculture after 5 years from 5 percent to 3 percent and lowering the estimated decline in Armenia’s poverty rate from 6 percentage points to 3 percentage points. Figure 5 summarizes MCC’s original impact projections for Armenia and its revisions after correcting the errors it identified. rojecting per capita income increases and overestimated income El Salvador. MCC determined that it had made an error in i p increases in its projections of poverty rate reduction for El Salvador. In correcting these errors, MCC lowered its projections of income and poverty impact. Figure 6 summarizes MCC’s original impact projections for El Salvador and its revisions after correcting the errors it identifie d. MCC also presented alternative methods for identifying beneficiaries, which further reduces the results of its compact-level projections. (See app. IV.)

Mozambique. MCC determined that it had not updated formula links in its analytic spreadsheets for Mozambique. Correcting this error led MCC to revise the projected income increase upward by $4 million for 2015 and downward by $13 million for 2025. In addition, MCC corrected both the formula and the data used to calculate the effect of this income increase on poverty in Mozambique, presenting two alternative approaches for estimating poverty elasticity. In making these changes, MCC lowered the projected decline in the poverty rate as well as the number of persons likely to be lifted out of poverty because of the compact. For example, MCC originally projected a 7 percent reduction in Mozambique’s poverty rate in 2015; using the alternative approaches for estimating poverty elasticity, MCC projected a poverty rate reduction of either 0.6 percent or 2 percent in 2015. Likewise, MCC originally projected that 270,000 people would be lifted out of poverty in Mozambique in 2015; using the alternative approaches, MCC projected that either 27,000 persons or 56,000 persons would be lifted out of poverty in 2015. Figure 7 summarizes MCC’s original impact projections for Mozambique and its revisions after correcting errors and presenting alternative approaches for estimating poverty elasticity.

Revised Projections Would Not Have Changed Funding Decisions, but Future Compact Impact Projections Will Undergo Review

According to MCC officials, the revisions to its initial calculations of compact impact on income and poverty for Armenia, El Salvador, and Mozambique would not have changed MCC’s decision to recommend each compact to the MCC Board. The officials emphasized that MCC’s economic impact projections are one of many aspects of the due diligence process that inform its compact investment decisions. A senior official stated that MCC’s compact-level analysis is separate from the decision about whether to invest in specific projects; the two levels of analysis are connected but play different roles in MCC’s decision making. MCC officials also told us they used conservative data and assumptions to project compacts’ impact on income, growth, and poverty. However, future public statements would reflect MCC’s corrections and revised analyses.

In addition, MCC officials said that although lead economists independently review the economic projections performed by others on the transaction team for each country, calculations and assumptions performed by the lead economists for the compacts we reviewed did not undergo a final check for accuracy and validity. The officials stated that such a review might have caught the errors that MCC later identified and corrected. They further stated that, with fewer compacts undergoing due diligence in the future, MCC will have the staff capacity to ensure that such reviews are performed. MCC officials told us they have begun to implement this peer review for compacts currently under development. However, MCC has not documented its procedures for conducting these reviews, so it is unclear what criteria and level of detail the peer review includes.

MCC Used Varying Methods to Project Compact Impact on Income and Poverty

MCC used different methods to project the four compacts’ impact on income and poverty, limiting the estimates’ comparability and replicability. Although the method chosen can affect the results, MCC has not provided preferred methods, or guidelines for selecting a method, for these calculations.

MCC Used Varying Methods for Economic Growth Projections

MCC used a different method to estimate impact on economic growth for Armenia, El Salvador, and Mozambique than it used for Lesotho. As a result, the estimates are not comparable.

For Armenia, El Salvador, and Mozambique, MCC estimated the impact of compact projects on aggregate income or income growth. This method entailed summing the total benefits of individual compact projects and adding them to the income that would have prevailed without MCC.

For Lesotho, MCC estimated the impact of the compact on the country’s GDP growth rate using the published results from a World Bank simulation model. According to MCC, the World Bank’s model showed that a public and private investment of about $100 million would lead to an increase in GDP growth rate of 3.75 percentage points. Reasoning that its projected investment in the Lesotho compact is analogous to the investment in the World Bank model, MCC extrapolated from the World Bank’s estimate to project that the compact, if successfully implemented, would nearly double Lesotho’s GDP growth rate.

According to MCC, its use of GDP growth impacts based on the published World Bank model’s results was appropriate, in that the model was consistent with the scale and type of public infrastructure investments proposed under the compact. MCC also noted that the World Bank model provided more information than did MCC’s analyses for the other compacts. However, our analysis of the World Bank report shows some differences between MCC’s and the World Bank’s interventions. For example, whereas MCC’s compact comprises only increased public investment in infrastructure, the World Bank report’s model projects the combined effect of three elements: increase in garment exports, growth in commercial agriculture, and increase in public investment in infrastructure. Because the model’s three elements have different and interlinked effects, MCC’s extrapolation of the effect of one of these elements, public investment in infrastructure, to its compact requires a number of assumptions that cannot be validated.

Although MCC guidelines call for projecting compacts’ impact on economic growth starting from the project level, MCC has not provided preferred methods, or guidelines for selecting a method, for estimating compacts’ impact on GDP growth rate.

MCC Used Varying Methods to Project Poverty Reduction

MCC estimated compacts’ impact on the poverty rate for Armenia, El Salvador, and Mozambique based on the responsiveness, or elasticity, of the poverty rate to changes in income. MCC uses several income measures in this analysis. For Armenia, MCC estimated poverty reduction based on its elasticity with respect to agricultural value added to the local economy; for El Salvador, with respect to increased annual national GDP; and for Mozambique, with respect to increased annual regional GDP.

Each of these measures can be a valid measure of income and corresponding elasticity of the poverty rate to income changes. However, the types of data used to estimate poverty elasticity can significantly affect poverty impact projections. For example, in projecting the Mozambique compact’s impact on poverty, MCC initially used an ad hoc estimate of elasticity rather than an elasticity based on historical income and poverty data specific to Mozambique; according to MCC officials, the ad hoc elasticity was consistent with conservative cross-country estimates. In a response to our questions about its original analysis, MCC projected revised poverty impacts for Mozambique in two ways, in both cases correcting for analytic errors discussed above. For one revised projection, MCC used its initial elasticity based on cross-country data. For the second projection, MCC used an alternative elasticity calculated from Mozambique data, which resulted in lower poverty reduction estimates. The resulting estimates for the number of persons to be lifted out of poverty were 56,000 versus 27,000 persons by 2015 and 43,000 versus 32,000 persons by 2025, respectively.

MCC’s guidelines call for estimating compacts’ impact on reducing the poverty rate but do not discuss or prioritize various possible methods for estimating poverty reduction. Because the methods chosen for each compact can vary, the resulting estimates of poverty reduction may not be comparable or replicable across countries.

MCC Used Different Methods to Estimate Beneficiaries and Is Taking Steps to Strengthen Guidance

MCC used two different methods to estimate the number of beneficiaries it reported for the compacts we reviewed: (1) summing beneficiaries from the individual projects, with adjustments for double counting and timing of benefits, and (2) counting the intervention area’s entire population as beneficiaries.

For Armenia and Mozambique, MCC counted the population of the areas where the compact is being implemented as project-specific beneficiaries and summed these project totals to produce the reported number of beneficiaries. MCC corrected for double counting of beneficiaries in areas where more than one project was implemented. For Mozambique, MCC projected a gradual increase in the number of beneficiaries as benefits accrue over time, whereas for Armenia it did not.

For El Salvador, MCC reported that the entire population of the intervention area would benefit, although the project-specific beneficiary counts did not sum to the area’s population.

For Lesotho, assuming that the compact would have economywide impact, MCC reported the country’s entire population as beneficiaries although estimates of project beneficiaries were available.

MCC guidelines define program beneficiaries as individuals or groups that derive economic gains from MCC investments. However, the guidelines do not explicitly define economic gain or provide criteria for counting beneficiaries based on the amount of accrued benefits or the degree of exposure to compact-provided services. As a result, it is unclear whether MCC’s transaction teams would count as equal beneficiaries a person using a compact-related benefit once and a person using the same benefit regularly or whether MCC would apply an equivalence scale to adjust for varying use of the benefit. For example, for El Salvador education programs, MCC first counted as beneficiaries all individuals entering the programs—assuming they would experience some income increases, and later revised the method to include only individuals completing the programs, assuming they would experience measurable income increases. The transaction team’s choice of method for defining beneficiaries, in turn, affects the compact-level projections of impact on income and poverty. For example, MCC’s estimate of the number of persons to be lifted out of poverty by the compact as a whole ranges from 145,000 in the first case to 83,000 in the second, depending on the assumptions used. (See app. IV for more details.)

According to MCC officials, MCC is in the process of making its guidance regarding program beneficiaries more operational by establishing criteria for defining beneficiaries. The officials also noted that increases in income are considered a starting point for measuring economic gains and therefore determining the number of compact beneficiaries.

Conclusions

As MCC works in some of the world’s poorest countries, it has taken on the difficult challenge of projecting its compacts’ likely ERR, and economic growth and poverty effects. However, our analysis shows that analytic errors affected the results, and the choice of analytical method can change the results of economic projections. Without a consistent approach and preferred methods for its due diligence economic analyses, MCC’s transaction teams and individual team members have used different methods to identify compact results. This heterogeneity means that different teams could reach different conclusions about the worthiness of individual projects or compacts based on the method chosen. MCC has refined its guidance over time, and continues to do so. Further refinements that include formal procedures for reviewing the results of due diligence economic analyses and establishing a consistent approach with preferred methods—which could be modified if required by specific country conditions—would help MCC reduce the likelihood of errors and provide a common MCC lens for projecting compacts’ ERRs and impacts. This in turn would enhance the reliability and comparability of the information MCC uses internally for decision making as well as the information it provides to Congress and the public for oversight of MCC’s activities.

Recommendations for Executive Action

To improve the reliability and comparability of its projected ERR and economic impacts, we recommend that the CEO of MCC take the following actions: Adopt and implement written procedures for a secondary independent review of the methods and results of its economic analyses.

Improve MCC’s guidelines by identifying a consistent approach with preferred methods for projecting compacts’ impact on income and poverty.

Agency Comments and Our Evaluation

MCC provided comments regarding a draft of this report, which we have reprinted, with our response, in appendix V. MCC also provided technical clarifications, which we have incorporated as appropriate.

In commenting on a draft of this report, MCC concurred with our recommendations and outlined related steps that it is taking—including developing standard practices and templates, initiating an independent peer review process, and posting its economic analyses on the Internet. MCC commented that there is not a single cost-benefit standard practice and that in many cases the inconsistencies we identified were technically appropriate. We acknowledge that there is not a single standard practice; however, we maintain that consistent analytic approaches are needed to ensure the reliability of MCC’s analyses. According to MCC, the choice of analytic time frames was in some cases based on the judgment of the individual lead economist rather than on established criteria. The steps that MCC has stated it is taking—using a default time frame for analyses and subjecting the time frame decision to peer review—will help to enhance the comparability across compacts of the time frame of ERR calculations. MCC also stated that it disagreed with what it saw as our report’s implication that ERRs are disconnected from income and poverty effects. However, we assert that ERRs do not provide information about MCC’s impact on any specific population group, including the poor. Also, ERRs are a relative measure of benefits in relation to costs that can fluctuate owing to changes in costs alone—and therefore cannot be considered absolute measures of impact on income.

We are sending copies of this report to interested congressional committees as well as the Chief Executive Officer of MCC. We will also make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

At the request of the chairman of the House Committee on Foreign Affairs, we assessed the Millennium Challenge Corporation’s (MCC) economic analyses, including its projections of the compact’s economic rate of return (ERR), changes in income and poverty, and the number of beneficiaries. To carry out this review, we reviewed MCC compacts with four countries: Armenia, El Salvador, Lesotho, and Mozambique. We selected these countries based on their percentage of total compact funding, the recentness of the compact, and geographical representation. These four compacts make up a total of about $1.56 billion in compact assistance. When we began our work in August 2007, this represented about 41 percent of the $3.85 billion MCC had set aside for 13 compacts and included the most recent compacts signed in Eurasia, Latin America, and Africa. We chose to include two African countries in acknowledgement of MCC’s focus on Africa, which accounted for 7 of MCC’s 13 compact countries and more than half of the total compact value.

MCC’s compact-level projections are influenced by the specific data points and assumptions used in project- and activity-level analysis. We determined that the data we used were sufficiently reliable for the purposes of our analysis; however, within our scope of work, we did not independently evaluate the thousands of data points and assumptions at the project and activity levels or the sensitivity analyses used in MCC’s economic analyses and therefore are not assessing all aspects of the validity of MCC’s ERR or impact projections. We also did not assess MCC’s progress in implementing these compacts, and therefore its progress toward achieving projected compact results.

To assess MCC’s compact ERR projections, we reviewed MCC’s public and internal documents for statements of compact and project ERR. When we began our review, MCC had published the compact and project ERRs for only Armenia. However, for all four countries, MCC stated ERRs in the internal investment memo. Next, we consulted MCC’s guidance for ERR analyses as well as the spreadsheets provided by MCC to support its calculations of ERR and document its work. We reviewed MCC guidance on calculating ERR and minimum ERRs and compared MCC’s internal documentation and spreadsheets to elements of this guidance. We examined the spreadsheets to determine how MCC aggregated project- level ERRs into one compact-level ERR for the four countries in our review and identified the two approaches—-summing net benefits and using weighted averages—that MCC used. To determine the effects of MCC’s alternative approach on the compact ERR, we used MCC’s cost and benefit data to calculate the alternative ERRs. We also used these data to calculate compact- and project-level ERRs over different time horizons to explore the sensitivity of the ERRs to differing time horizons. To determine how MCC approached same-sector projects in different compacts, we studied road and water projects because these were each in three of the four compacts. We then compared the broad approaches MCC used in assessing the costs and benefits associated with these projects. We interviewed MCC economists and other officials regarding MCC’s ERR analysis to further discuss MCC’s approaches and clarify aspects of MCC’s analysis. We also consulted Office of Management and Budget (OMB) guidance and GAO guidance on establishing and implementing internal controls to inform our assessment of MCC’s processes for developing and maintaining information for the purpose of management decision making.

To assess MCC’s projections of the number of compact beneficiaries, and changes in income and poverty, we first compiled and analyzed MCC’s public documents for statements of compact impact. These documents included MCC’s compacts, compact summaries, congressional notifications and budget justifications, and MCC’s annual reports. MCC makes all of these documents available online. We identified statements that fell into four categories of economic projections: impact on national or regional income, impact on national or regional gross domestic product (GDP) growth rate, impact on national or regional poverty, and number of compact beneficiaries. MCC reviewed and concurred with our compilation and summary of these statements of compact impact. We also reviewed MCC’s guidance on projecting these economic impacts. We reviewed the spreadsheets for each compact that MCC used to conduct and document its economic analyses. After an initial examination of these spreadsheets, we met with MCC economists and other officials to discuss MCC’s methods and calculations for projecting compact impact. MCC officials also provided responses to our questions in written form. In these responses, MCC revised its initial calculations of its impacts in Armenia, El Salvador, and Mozambique and also presented alternative methods. We then reviewed the updated information that MCC submitted to us and calculated the magnitude of the difference in MCC’s original statements and those supported by its revised analyses. In the case of Lesotho, we reviewed the World Bank Country Economic Memo regarding the economic growth model that MCC used to estimate compact impact in Lesotho. We also reviewed World Bank country assistance strategies and International Monetary Fund Article IV consultation reports and Poverty Reduction Strategy Papers to improve our contextual understanding of each country’s compact program. Finally, we consulted OMB guidance and GAO guidance on establishing and implementing internal controls to inform our assessment of MCC’s internal processes.

We conducted this performance audit from August 2007 to June 2008, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: MCC Minimum Acceptable ERRs

MCC has issued two definitions of the minimum acceptable ERR, which MCC refers to as the hurdle rate, for compacts and projects. In its initial guidelines, issued in April 2005, MCC defined the hurdle rate as the average of the country’s real growth rates for the previous 3 years. The January 2006 guidelines did not set a hurdle rate. The November 2006 guidelines defined the hurdle rate as the greater of (1) two times the average real growth rate of GDP for the country for the most recent 3 years for which data are available or (2) two times the average real growth rate of GDP for all of the MCC eligible countries for each country for the most recent 3 years for which data are available. The November 2006 guidelines also state that the hurdle rate may not be higher than 15 percent.

In setting the hurdle rate for each compact that we reviewed, MCC applied the definition of the rate from the guidelines current at the time. For Armenia and Mozambique, MCC used its April 2005 guidance. For El Salvador, according to MCC officials, and for Lesotho, MCC used the definition in the November 2006 guidelines. (See table 2.)

In two cases, the ERRs that MCC initially calculated did not meet the compact hurdle rates. MCC originally calculated the El Salvador Community Infrastructure Project ERR as slightly below the hurdle rate at 10.4 percent over 25 years. MCC’s investment memo for Lesotho stated that the ERR for the Rural Water Project was under 6 percent. Subsequent revisions to the Lesotho analysis reduced the projected ERR to less than 1 percent. The Lesotho investment memo discusses the 6 percent ERR but notes a different view within the transaction team that other benefits are not captured by the economic analyses.

Appendix III: Compact ERRs

We assessed the sensitivity of MCC’s compact-level ERRs to the use of varying time frames and varying methods for calculating the compact-level ERR. We found in each case that the variance does affect the results, but does not change the ERRs to below the applicable hurdle rate.

Time Frames

MCC used different time frames to calculate and report compact ERRs.

For Armenia, MCC calculated and reported a 20-year time frame for the compact and both of its projects.

For El Salvador, MCC calculated a 25-year ERR for most projects and for some projects reported the ERR as a 25-year ERR in its investment memo. MCC officials explained that their El Salvador country counterparts originally performed the analysis over 25 years and that MCC’s lead economist determined this to be a reasonable approach.

For Lesotho, MCC used a 20-year ERR for all but one ERR calculation but did not state the time frame of the calculations in the investment memo.

For Mozambique, MCC used a 24-year ERR for most compact projects.

According to MCC officials, MCC intended to present a 25-year ERR, but a delay in the beginning of compact implementation pushed the compact’s time horizon 1 year into the future and reduced the time frame for the analysis to 24 years. The Mozambique investment memo does not state the number of years used for the time frame of the ERR calculation.

Table 3 summarizes our comparison of the compact hurdle rates with the ERRs that MCC reported and our calculations of the 20-year ERRs. Our analysis shows that applying a 20-year time frame in place of varying time frames does not significantly affect the results of the ERR calculations by lowering it below the hurdle rate.

Methods

Figure 8 illustrates the two different methods—summing net benefits and calculating a weighted average—that MCC used to determine a compact- level ERR.

We compared the results of the compact ERR calculation using different methods and determined that the choice of method affects the results, but does not reduce the ERR below the hurdle rate. See table 4 for details of our analysis.

Appendix IV: Impact of Alternative Beneficiary Counts on El Salvador Compact Impact Projections

During the course of our engagement, MCC changed its method for determining the beneficiaries and benefits of education projects in El Salvador. In its initial calculations for both projects, MCC assumed that all students entering the programs would experience an increase in income. In its revised calculations for the formal education project, MCC assumed that all nongraduates would have zero income increases—an underestimate of the impact—because of lack of data. For the informal education project, MCC estimated an income increase of more than 200 percent for those who obtain employment, based on a study conducted by the training institute in El Salvador. However, MCC assumed that many entering students would not complete the program or obtain employment and their income would therefore not increase. (See table 5.)

Appendix V: Comments from the Millennium Challenge Corporation

The following are GAO’s comments from the Millennium Challenge Corporation letter, dated May 23, 2008.

GAO Comments

1. We recognize MCC’s constructive responses to the issues identified in the course of our audit—including posting economic analyses on the Internet, developing standard practices and templates, and initiating an independent peer review process. 2. We acknowledge that there is not a single standard practice; however, we maintain that consistent analytic approaches are needed to ensure the reliability and comparability of MCC’s analyses. According to MCC, the choice of analytic time frames was in some cases based on the judgment of the individual lead economist rather than on established criteria. The steps that MCC has stated it is taking—using a default time frame for analyses and subjecting the time frame decision to peer review—will help to enhance the comparability of its of ERR calculations across compacts. 3. We disagree that MCC’s ERRs estimate the project’s effect on people’s lives. ERRs do not provide information about MCC’s impact on any specific population group, including the poor. ERRs also are a relative measure of benefits in relation to costs and can fluctuate owing to changes in costs alone. For example, if costs increase and the income benefits remain the same, the ERR would decrease. Therefore, ERRs cannot be considered an absolute measure of income benefits.

Appendix VI: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the person named above, Emil Friberg, Jr. (Assistant Director), Todd M. Anderson, Gergana Danailova-Trainor, Reid Lowe, Michael Simon, and Seyda Wentworth made key contributions to this report. Also, C. Etana Finkler, Ernie Jackson, and Tom McCool provided technical assistance.