Millennium Challenge Corporation: Independent Reviews and Consistent Approaches Will Strengthen Projections of Program Impact
Published: 2008-06-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In January 2004, Congress established the Millennium Challenge Corporation (MCC) for foreign assistance. Eligible countries submit compact proposals for MCC funding for projects aimed at reducing poverty through economic growth. To assess the proposed compacts' likely impact, MCC performs economic analyses estimating the compacts' economic rate of return (ERR) and effects on income and poverty as well as the number of compact beneficiaries. MCC uses these analyses to inform its decisions to fund proposed compacts and to inform Congress and the public about its progress in achieving its mission of poverty reduction through economic growth. GAO was asked to examine MCC's projections of (1) ERR and (2) compacts' impact on income and poverty as well as numbers of beneficiaries. GAO reviewed MCC's stated impacts and analyses for four MCC compacts that represented 41 percent of MCC's compact assistance and met with MCC officials.
section_title: What GAO Found
paragraphs: MCC used different time frames and methods to calculate ERRs for its compacts with Armenia, El Salvador, Lesotho, and Mozambique. In calculating ERR for 20 projects within the compacts, MCC used a 20-year time frame for 9 projects and used different time frames for the other 11. In 2 of the 11 projects, using a 20-year time frame, as MCC used for similar projects, would reduce the ERR below the level MCC set as the minimum acceptable ERR. At the compact level, MCC's use of varying time frames did not affect the ERRs significantly. MCC used varying methods to account for the costs of same-sector projects, although its approaches to determining project benefits were generally similar. MCC also used two different methods to calculate compact-level ERR; however, the choice of method did not reduce it below the minimum ERR. In three of the four compacts that we reviewed, MCC did not retain documentation of the economic analyses used to support the investment decision, but continued to modify the analyses. MCC has recently begun to standardize elements of its economic analyses and centralize its records management. MCC identified and corrected analytic errors in its projections of impact on income and poverty. MCC also used varying methods to project these impacts. In responding to GAO's questions about its published projections of impact on income and poverty, MCC identified analytic errors for three of the four compacts and, in correcting these errors, generally lowered the projected impacts. Correcting these errors raised one projection by 5 percent but reduced others by 3 percent to 96 percent. According to MCC officials, the revised projections would not have affected MCC's decision to recommend signing the compacts. The officials noted that, in the future, compact impact projections will undergo peer review. However, MCC has not documented procedures for these reviews. MCC used varying methods for its projections of impact on income and poverty, limiting comparability and replicability across compacts. To project impact on income for Armenia, El Salvador, and Mozambique, MCC estimated the compacts' impact by summing the total benefits of individual compact projects and adding them to the income that would have prevailed without MCC. However, for Lesotho, MCC estimated the impact on income based on the published results of a World Bank model based on elements different from those of the MCC compact. In response to our questions, MCC revised its initial estimate of the effect of income growth on poverty for Mozambique by presenting two estimates, based on Mozambique-specific and cross-country data, respectively. Although a number of methods for projecting poverty impact are valid, the method chosen can affect the results, and MCC's guidelines do not identify preferred methods for these calculations. MCC also used varying methods to estimate numbers of beneficiaries for the compacts and has not provided specific criteria for defining beneficiaries; however, MCC officials reported they are taking steps to provide more detailed guidance.

section_title: Letter
paragraphs: In January 2004, Congress established the Millennium Challenge  Corporation (MCC) to administer the Millennium Challenge Account  (MCA) for foreign assistance. MCC’s mission is to reduce poverty through  sustainable economic growth in some of the world’s poorest countries that  create and maintain sound policy environments. MCC has received  appropriations for fiscal years 2004 to 2008 totaling more than $7.5 billion  and, as of March 2008, has signed $5.5 billion in compacts with 16  countries. The President has requested $2.225 billion for fiscal year 2009.
MCC uses income and country performance criteria to annually select a  list of countries eligible for MCA assistance. Eligible countries may then  submit proposals for compacts containing multiple projects for MCC’s  review and approval. On receiving proposals for MCA assistance, MCC  undertakes a comprehensive review, or due diligence, which seeks to  ensure that the proposed compacts will advance MCC’s mission. As part of  due diligence, MCC assesses the potential economic impact of each  compact. During due diligence, MCC identifies a minimum acceptable  economic rate of return (ERR) for the compact and its projects; compares  estimated costs and benefits to determine the compact’s ERR; and  projects the compact’s impact, including its number of beneficiaries and  its impact on income, economic growth, and poverty.
MCC uses its projections, as well as other information gathered during due  diligence, to inform its internal decisions to fund proposed projects and  compacts. The results of the due diligence assessment are reported in an  investment memo—an internal document prepared by MCC’s transaction  team that analyzes the compact—submitted to MCC’s investment  committee. MCC also publishes its compact impact projections during  ongoing consultations with Congress. This information—found in MCC  documents such as compacts, compact summaries, annual reports, and  congressional notifications and budget justifications—sets expectations  for the compact and provides information to Congress and the public  about MCC’s progress in achieving its mission.
In July 2007, we reported that MCC’s portrayal of the projected impact of  its Vanuatu compact did not reflect MCC’s underlying analyses. Our  recommendations included that MCC revise its public reporting of the  Vanuatu compact’s projected impact and assess whether similar reporting  for other compacts accurately reflects underlying economic analyses. The  committee subsequently requested that we examine MCC’s economic  analyses for its compacts with other countries.
As agreed with your office, for this report, we assessed (1) MCC’s  projections of ERR and (2) MCC’s projections of compacts’ impact on  income and poverty as well as numbers of beneficiaries.
To carry out this review, we reviewed MCC compacts with four countries:  Armenia, El Salvador, Lesotho, and Mozambique. When we began our  work in August 2007, these four countries represented about 41 percent of  the $3.85 billion MCC had set aside for 13 compacts and included the most  recent compacts signed in Eurasia, Latin America, and Africa. To assess  MCC’s compact projections, we reviewed MCC public and internal  documents as well as relevant World Bank and International Monetary  Fund documents. We also reviewed MCC’s original and revised  spreadsheet calculations of its economic projections and interviewed MCC  economists and other officials in Washington, D.C., regarding these  analyses. Finally, we consulted Office of Management and Budget (OMB)  and GAO guidance on establishing and implementing internal controls.  We determined that the data we used were sufficiently reliable for the  purposes of our analysis; however, we did not independently assess the  reliability of all data and assumptions that affect the projections and that  MCC used in its underlying economic analyses of compact projects and  activities. We also did not assess MCC’s progress in implementing these  compacts or toward its projected results. We conducted this performance  audit from August 2007 to June 2008, in accordance with generally  accepted government auditing standards. Those standards require that we  plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objectives. (See app. I for additional details of our scope and  methodology.)
subsections: section_title: Background
paragraphs: MCC conducts its economic analyses during the compact development  process, the first of three compact phases. These analyses include  establishing the minimum ERR that the compact should achieve,  projecting the compact’s ERR, and estimating the compact’s impact on  income and growth. MCC did not publish ERR information for three of the  four compacts we reviewed, but its public documents include statements  of compact impact on income and poverty. Of the four compacts that we  reviewed—Armenia, El Salvador, Lesotho, and Mozambique—three  include funding for road projects, and three include funding for water  projects, among other projects.
subsections: section_title: Compact Development Process
paragraphs: If MCC determines that a country is eligible for assistance, the country  may submit a compact proposal, which generally comprises several  projects. According to MCC guidelines, the proposal should include  economic analyses of proposed projects to demonstrate their likely impact  on growth and poverty in the country. MCC also may provide assistance  and feedback in developing the proposal, including making grants to  facilitate the development of the compact (see fig. 1).
After a country submits its proposal, MCC’s transaction team for the  compact conducts a preliminary assessment of the proposal and reports  its findings in an internal opportunity memo to the MCC investment  committee. MCC assembles a different transaction team for each compact.  If the opportunity memo is approved, the team launches a detailed due  diligence review that includes economic analyses of the proposed  projects. As members of MCC country transaction teams, MCC’s lead  economists undertake these analyses while working with other members  of the team and country officials. According to MCC, as part of this effort,  MCC economists review the preliminary analyses performed by country  counterparts and consultants. These due diligence reviews, including  conducting economic analyses, have lasted, on average, slightly more than  10 months for the 16 countries with signed compacts as of March 31, 2008.
At the conclusion of due diligence, the transaction team sends an  investment memo to the MCC investment committee, with  recommendations based on its assessment of the proposal. MCC notifies  Congress 15 days prior to beginning negotiations with the country, sending  a formal congressional notification. For the four compacts we examined,  the congressional notifications included some of the results of MCC’s  economic analyses. If compact negotiations with the eligible country are  successful, the investment committee submits the proposed compact to  the MCC Board for approval. With the Board’s approval, MCC and the  country sign the compact before completing additional agreements—such  as a disbursement agreement and procurement agreement—and ultimately  implementing the projects funded in the compact.
section_title: Economic Analyses
paragraphs: For each compact proposal, MCC and the eligible country conduct  economic analyses, which MCC uses to inform its decisions to fund  compacts and report to Congress and the public. MCC generally conducts  these analyses at the compact and project levels (see fig. 2).
Aggregte economic rte of retrn (ERR)
The economic analyses include projections of compacts’ impact, including  their ERR, the impact on income and poverty, and the projected number of  beneficiaries.
ERR analysis. To provide a basis for assessing the compact and project  ERRs, MCC sets the minimum acceptable ERR that compacts and projects  should achieve to be eligible for funding. (See app. II for a discussion of  MCC’s minimum ERRs for the four countries.) If the compact or project  does not meet the minimum rate, MCC retains the discretion to fund it but  requires justification based on the specific circumstances.
The ERR analysis compares costs and benefits, where the costs are the MCA  grants and the country’s future recurrent costs—such as maintenance  expenses—and the benefits are increases in incomes in recipient countries.  (See fig. 3.) MCC guidelines state that normal practice is to calculate ERRs  using 10-, 20-, and 30-year time horizons to determine project and compact  ERRs’ sensitivity to varying time frames. If the ERR is sensitive to this time  horizon, the guidelines require that this be noted explicitly.
Impact analysis. MCC’s guidelines state that economic analyses should  quantify the proposed projects’ expected beneficiaries and expected  impact on incomes and on poverty. Based on our analysis, MCC does not  establish minimum thresholds for compact impact.
section_title: Public Reporting
paragraphs: While MCC’s transaction team reports both ERR and impact projections to  the MCC investment committee in its investment memo, MCC’s public  reporting for the four compacts in our review generally included information  about only its projections of compact impact; for these four countries MCC  published ERR projections for only Armenia. MCC’s public reporting included  projections of compact impact on income and poverty for three countries and  projections of gross domestic product (GDP) growth for one country. For  each compact, MCC also estimated the number of beneficiaries. (See table 1.)
section_title: Compact Projects and Funding
paragraphs: The types of projects included in the four compacts vary, although El  Salvador, Lesotho, and Mozambique include water projects and Armenia,  El Salvador, and Mozambique include road projects. The compacts provide  a total of approximately $1.56 billion in MCA assistance. (See fig. 4.)
section_title: MCC Used Different Time Frames and Methods to Calculate ERRs but Is Taking Steps to Increase Consistency
paragraphs: MCC used different time frames to calculate project-level ERRs; these  differences did not affect the ERR significantly in 18 of the 20 compact  projects we examined, but two ERRs would fall below the minimum ERR  if MCC applied the 20-year time frame it used for other compacts and  similar projects. At the compact level, the different time frames did not  change the ERR significantly. In addition, we found that MCC used varying  methods to account for the costs of same-sector projects, although its  approaches to determining benefits were generally similar. MCC also used  two different methods to calculate compact-level ERR; however, the  choice of method did not change the ERRs significantly. In some cases,  MCC did not fully retain its documentation of its economic analyses. MCC  has recently taken steps to standardize elements of its economic analyses  and improve its records management and plans to implement additional  measures.
subsections: section_title: MCC Used Different Time Frames for Project and Compact ERRs
paragraphs: In its ERR calculations for 20 projects included in the four compacts we  reviewed, MCC used a 20-year time frame for 9 projects and used different  time frames for 11 projects. Nearly all project ERRs that MCC initially  calculated met the minimum ERR set for each compact. Our analysis  shows that for 9 of the 11 ERRs calculated with other time frames,  recalculating the ERR with a 20-year time frame does not lower the ERR  below the minimum ERR. However, for two projects, recalculating the  ERR with a 20-year time frame produces a result below the minimum ERR.
The ERR for the El Salvador Community Infrastructure project, which  MCC calculated as slightly below the minimum ERR of 10.8 percent over  25 years, drops to 9.0 percent when calculated over 20 years. Correcting a  calculation error further reduces the El Salvador project’s ERR to 6.8  percent over 20 years—four percentage points below the El Salvador  minimum ERR.
The ERR for the Mozambique roads project, which MCC calculated at 10.3  percent over 24 years, drops to 8.1 percent over 20 years, below the  minimum ERR of 8.76 percent. Three of the four individual Mozambique  roads MCC analyzed as part of this project also fall below the minimum  ERR at 20 years.
If MCC had applied a 20-year time frame in calculating the ERR for these  two projects, the projects might have been restructured to increase their  ERR or MCC would have had to specifically justify the exception. MCC  does not currently have a policy addressing what steps to take in cases  where subsequent analysis results in a project ERR below the minimum.  MCC stated that it would address changes such as this on a case-by-case  basis depending on the timing of the change within the compact  development process, and the magnitude of the change. MCC officials also  noted that the ERR time frame is only one aspect of MCC’s analysis of the  ERR’s sensitivity to various factors.
In addition, MCC used different time frames in calculating ERRs for  comparable projects in different compacts. MCC calculated water project  ERRs for El Salvador, Lesotho, and Mozambique over 25, 20, and 21 years,  respectively, and calculated road project ERRs for Armenia, El Salvador,  and Mozambique over 20, 25, and 24 years, respectively. MCC officials told  us they explored the ERRs’ sensitivity to varying time frames, as MCC’s  guidelines require, but they did not regard as prescriptive the guidelines’  statement that normal practice is to examine 10-, 20-, and 30-year time  horizons. MCC officials also noted that MCC economists have recently  committed to the use of a default 20-year time horizon for their analyses.  MCC also will alter this time frame for specific circumstances or projects  whose benefits have a longer duration—such as education projects and  large-scale construction projects—or have a shorter or longer physical life  expectancy.
At the compact level, our analysis shows that applying a 20-year time  frame in place of varying time frames does not significantly affect the  results of the ERR calculations. (See app. III for a summary of our  comparison of the compact minimum ERR with the compact ERRs that  MCC reported and our calculations of the 20-year ERRs.)
section_title: MCC Used Varying Methods in Calculating Project and Compact ERRs
paragraphs: In calculating project ERRs, we found that MCC used varying methods to  account for the investment and recurrent cost components of water and  road sector projects for the four compacts we reviewed.
Investment costs. For water projects, MCC counted the cost of  construction in Lesotho as one lump sum in the first year of the analysis  but phased in the cost in Mozambique and El Salvador over multiple years.  Phasing the costs evenly over the 5-year compact in Lesotho would have  increased the ERR for both the urban and rural water projects. For road  projects, MCC phased the investment cost over time for Armenia, El  Salvador, and Mozambique. However, in El Salvador, MCC accounted for  the salvage value of the project in the last year of the analysis but did not  include a similar projection in the analysis for Armenia and Mozambique.
Recurrent costs. For water projects, MCC counted Lesotho and  Mozambique’s recurrent costs in the years after the original compact  investment. For El Salvador, MCC counted recurrent costs only during the  initial 5-year compact period. Including future recurrent costs would have  reduced the ERR. For road projects, MCC spread recurrent costs over time  for Armenia, El Salvador, and Mozambique.
According to our review, MCC’s approaches to determining the benefits of  water and road projects were generally similar.
Water projects. MCC generally counted as water project benefits  increased time available for work, resulting from less time spent fetching  water or being ill, and lower spending on health care and water.
Road projects. MCC generally estimated traffic volumes on the roads and  determined road project benefits based on the savings from reduced travel  time and vehicle operating costs for existing and generated traffic.
In calculating the compact-level ERRs for the four compacts’ investment  memos, MCC used two different methods.
For Armenia and Mozambique, MCC first determined the net benefits in  each year for each project and then calculated the compact ERR based on  the total net benefits.
For El Salvador and Lesotho, MCC first determined each project’s net  benefits and ERR and then determined the compact ERR by averaging the  project ERRs, weighting each ERR according to the project’s budgeted  size relative to the overall compact.
According to MCC officials, the choice of method depended on the  preference of the transaction team’s lead economist. Our analysis shows  that using the second method to recalculate each compact ERR reduces it  by less than 2 percentage points and does not reduce it below the  minimum ERR. However, these results demonstrate that the choice of  method influences the compact-level ERR and may affect the  comparability of ERRs across compacts. (See app. III for details of our  analysis of MCC compact ERRs.)
section_title: Insufficient Records Management Led to Discrepancies between Investment Memos and Underlying Analyses
paragraphs: In three of the four cases we reviewed, the projections of ERR reported to  the MCC investment committee differ from those in MCC’s underlying  spreadsheets.
For Armenia, MCC changed its estimate of project benefits after the  investment memo.
For Lesotho, the MCC spreadsheets contain figures calculated using  different methods or reflecting additional analysis after the investment  memo.
For Mozambique, the calculation of the figures presented in the  investment memo relied on spreadsheet formula links that were not  properly updated at the time. According to MCC, when they provided the  spreadsheets to us, they updated the formulas and overwrote the original  calculations.
For these three compacts, MCC’s records management did not fully  preserve the information and analysis used to support the investment  decision. OMB guidance and GAO guidelines for internal controls both  note the importance of controls over the information that U.S. agencies  use to make decisions.
section_title: MCC Has Taken Several Steps to Improve the Consistency of ERR Analysis
paragraphs: After completing due diligence for the four compacts we studied, and  during the course of our audit, MCC told us they took or began taking  steps to increase the consistency of its analyses and improve its records  management. For example, MCC    committed to using 20 years as the default time frame for calculating ERRs  and identified projects where this time frame may be altered;   began revising its guidance to clarify that ERR sensitivity should be tested  by varying the time frame of the analysis, rather than specifying 10-, 20-,  and 30-year analyses;    began developing standards for consistently analyzing certain types of  projects across compacts;    established that summing net benefits and costs across all projects is the  preferred method for calculating a compact-level ERR; and   planned to implement a data management system to centralize its records  management.
section_title: MCC Made Analytic Errors in Compact Impact Projections and Used Varying Methods that Affected the Projections’ Results
paragraphs: MCC identified and corrected analytic errors in its projections of compact  impact for Armenia, El Salvador, and Mozambique, generally reducing  each compact’s estimated impact on income and poverty. According to  MCC, the revised calculations would not have affected its approval of the  compacts, but it will perform peer reviews of future impact projections. In  addition, MCC used different methods for projecting compact impact on  income and poverty, limiting the estimates’ comparability and replicability;  its current guidance does not address the choice of method for these  projections. MCC also used different methods to estimate numbers of  compact beneficiaries, but stated that it is taking steps to provide more  detailed guidance for estimating number of beneficiaries.
subsections: section_title: MCC Identified and Corrected Analytic Errors for Three Compacts
paragraphs: In responding to our questions about its impact analyses for Armenia, El  Salvador, and Mozambique, MCC identified a number of analytic errors in  its projections of impact on income and poverty. MCC subsequently  corrected these errors, generally reducing the projected impacts on  income and poverty for each compact.
Armenia. MCC determined that it had included road project benefits in  estimating the income increase from agriculture for Armenia. MCC also  used the wrong baseline in projecting poverty effects. Correcting these  errors affected projections of the compact’s effect, reducing the estimated  increase in rural areas’ real income from agriculture after 5 years from 5  percent to 3 percent and lowering the estimated decline in Armenia’s  poverty rate from 6 percentage points to 3 percentage points. Figure 5  summarizes MCC’s original impact projections for Armenia and its  revisions after correcting the errors it identified. rojecting per capita income increases and overestimated income  El Salvador. MCC determined that it had made an error in i p increases in its projections of poverty rate reduction for El Salvador. In  correcting these errors, MCC lowered its projections of income and poverty impact. Figure 6 summarizes MCC’s original impact projections  for El Salvador and its revisions after correcting the errors it identifie d.  MCC also presented alternative methods for identifying beneficiaries,  which further reduces the results of its compact-level projections. (See  app. IV.)
Mozambique. MCC determined that it had not updated formula links in its  analytic spreadsheets for Mozambique. Correcting this error led MCC to  revise the projected income increase upward by $4 million for 2015 and  downward by $13 million for 2025. In addition, MCC corrected both the  formula and the data used to calculate the effect of this income increase on  poverty in Mozambique, presenting two alternative approaches for estimating  poverty elasticity. In making these changes, MCC lowered the projected  decline in the poverty rate as well as the number of persons likely to be lifted  out of poverty because of the compact. For example, MCC originally  projected a 7 percent reduction in Mozambique’s poverty rate in 2015; using  the alternative approaches for estimating poverty elasticity, MCC projected a  poverty rate reduction of either 0.6 percent or 2 percent in 2015. Likewise,  MCC originally projected that 270,000 people would be lifted out of poverty in  Mozambique in 2015; using the alternative approaches, MCC projected that  either 27,000 persons or 56,000 persons would be lifted out of poverty in 2015.  Figure 7 summarizes MCC’s original impact projections for Mozambique and  its revisions after correcting errors and presenting alternative approaches for  estimating poverty elasticity.
section_title: Revised Projections Would Not Have Changed Funding Decisions, but Future Compact Impact Projections Will Undergo Review
paragraphs: According to MCC officials, the revisions to its initial calculations of  compact impact on income and poverty for Armenia, El Salvador, and  Mozambique would not have changed MCC’s decision to recommend each  compact to the MCC Board. The officials emphasized that MCC’s  economic impact projections are one of many aspects of the due diligence  process that inform its compact investment decisions. A senior official  stated that MCC’s compact-level analysis is separate from the decision  about whether to invest in specific projects; the two levels of analysis are  connected but play different roles in MCC’s decision making. MCC  officials also told us they used conservative data and assumptions to  project compacts’ impact on income, growth, and poverty. However,  future public statements would reflect MCC’s corrections and revised  analyses.
In addition, MCC officials said that although lead economists  independently review the economic projections performed by others on  the transaction team for each country, calculations and assumptions  performed by the lead economists for the compacts we reviewed did not  undergo a final check for accuracy and validity. The officials stated that  such a review might have caught the errors that MCC later identified and  corrected. They further stated that, with fewer compacts undergoing due  diligence in the future, MCC will have the staff capacity to ensure that  such reviews are performed. MCC officials told us they have begun to  implement this peer review for compacts currently under development.  However, MCC has not documented its procedures for conducting these  reviews, so it is unclear what criteria and level of detail the peer review  includes.
section_title: MCC Used Varying Methods to Project Compact Impact on Income and Poverty
paragraphs: MCC used different methods to project the four compacts’ impact on  income and poverty, limiting the estimates’ comparability and replicability.  Although the method chosen can affect the results, MCC has not provided  preferred methods, or guidelines for selecting a method, for these  calculations.
subsections: section_title: MCC Used Varying Methods for Economic Growth Projections
paragraphs: MCC used a different method to estimate impact on economic growth for  Armenia, El Salvador, and Mozambique than it used for Lesotho. As a  result, the estimates are not comparable.
For Armenia, El Salvador, and Mozambique, MCC estimated the impact of  compact projects on aggregate income or income growth. This method  entailed summing the total benefits of individual compact projects and  adding them to the income that would have prevailed without MCC.
For Lesotho, MCC estimated the impact of the compact on the country’s  GDP growth rate using the published results from a World Bank  simulation model. According to MCC, the World Bank’s model showed  that a public and private investment of about $100 million would lead to an  increase in GDP growth rate of 3.75 percentage points. Reasoning that its  projected investment in the Lesotho compact is analogous to the  investment in the World Bank model, MCC extrapolated from the World  Bank’s estimate to project that the compact, if successfully implemented,  would nearly double Lesotho’s GDP growth rate.
According to MCC, its use of GDP growth impacts based on the published  World Bank model’s results was appropriate, in that the model was  consistent with the scale and type of public infrastructure investments  proposed under the compact. MCC also noted that the World Bank model  provided more information than did MCC’s analyses for the other  compacts. However, our analysis of the World Bank report shows some  differences between MCC’s and the World Bank’s interventions. For  example, whereas MCC’s compact comprises only increased public  investment in infrastructure, the World Bank report’s model projects the  combined effect of three elements: increase in garment exports, growth in  commercial agriculture, and increase in public investment in  infrastructure. Because the model’s three elements have different and  interlinked effects, MCC’s extrapolation of the effect of one of these  elements, public investment in infrastructure, to its compact requires a  number of assumptions that cannot be validated.
Although MCC guidelines call for projecting compacts’ impact on  economic growth starting from the project level, MCC has not provided  preferred methods, or guidelines for selecting a method, for estimating  compacts’ impact on GDP growth rate.
section_title: MCC Used Varying Methods to Project Poverty Reduction
paragraphs: MCC estimated compacts’ impact on the poverty rate for Armenia, El  Salvador, and Mozambique based on the responsiveness, or elasticity, of  the poverty rate to changes in income. MCC uses several income  measures in this analysis. For Armenia, MCC estimated poverty reduction  based on its elasticity with respect to agricultural value added to the local  economy; for El Salvador, with respect to increased annual national GDP;  and for Mozambique, with respect to increased annual regional GDP.
Each of these measures can be a valid measure of income and  corresponding elasticity of the poverty rate to income changes. However,  the types of data used to estimate poverty elasticity can significantly affect  poverty impact projections. For example, in projecting the Mozambique  compact’s impact on poverty, MCC initially used an ad hoc estimate of  elasticity rather than an elasticity based on historical income and poverty  data specific to Mozambique; according to MCC officials, the ad hoc  elasticity was consistent with conservative cross-country estimates. In a  response to our questions about its original analysis, MCC projected  revised poverty impacts for Mozambique in two ways, in both cases  correcting for analytic errors discussed above. For one revised projection,  MCC used its initial elasticity based on cross-country data. For the second  projection, MCC used an alternative elasticity calculated from  Mozambique data, which resulted in lower poverty reduction estimates.  The resulting estimates for the number of persons to be lifted out of  poverty were 56,000 versus 27,000 persons by 2015 and 43,000 versus  32,000 persons by 2025, respectively.
MCC’s guidelines call for estimating compacts’ impact on reducing the  poverty rate but do not discuss or prioritize various possible methods for  estimating poverty reduction. Because the methods chosen for each  compact can vary, the resulting estimates of poverty reduction may not be  comparable or replicable across countries.
section_title: MCC Used Different Methods to Estimate Beneficiaries and Is Taking Steps to Strengthen Guidance
paragraphs: MCC used two different methods to estimate the number of beneficiaries it  reported for the compacts we reviewed: (1) summing beneficiaries from  the individual projects, with adjustments for double counting and timing of  benefits, and (2) counting the intervention area’s entire population as  beneficiaries.
For Armenia and Mozambique, MCC counted the population of the areas  where the compact is being implemented as project-specific beneficiaries  and summed these project totals to produce the reported number of  beneficiaries. MCC corrected for double counting of beneficiaries in areas  where more than one project was implemented. For Mozambique, MCC  projected a gradual increase in the number of beneficiaries as benefits  accrue over time, whereas for Armenia it did not.
For El Salvador, MCC reported that the entire population of the  intervention area would benefit, although the project-specific beneficiary  counts did not sum to the area’s population.
For Lesotho, assuming that the compact would have economywide impact,  MCC reported the country’s entire population as beneficiaries although  estimates of project beneficiaries were available.
MCC guidelines define program beneficiaries as individuals or groups that  derive economic gains from MCC investments. However, the guidelines do  not explicitly define economic gain or provide criteria for counting  beneficiaries based on the amount of accrued benefits or the degree of  exposure to compact-provided services. As a result, it is unclear whether  MCC’s transaction teams would count as equal beneficiaries a person using a  compact-related benefit once and a person using the same benefit regularly or  whether MCC would apply an equivalence scale to adjust for varying use of  the benefit. For example, for El Salvador education programs, MCC first  counted as beneficiaries all individuals entering the programs—assuming  they would experience some income increases, and later revised the method  to include only individuals completing the programs, assuming they would  experience measurable income increases. The transaction team’s choice of  method for defining beneficiaries, in turn, affects the compact-level  projections of impact on income and poverty. For example, MCC’s estimate of  the number of persons to be lifted out of poverty by the compact as a whole  ranges from 145,000 in the first case to 83,000 in the second, depending on the  assumptions used. (See app. IV for more details.)
According to MCC officials, MCC is in the process of making its guidance  regarding program beneficiaries more operational by establishing criteria  for defining beneficiaries. The officials also noted that increases in income  are considered a starting point for measuring economic gains and  therefore determining the number of compact beneficiaries.
section_title: Conclusions
paragraphs: As MCC works in some of the world’s poorest countries, it has taken on  the difficult challenge of projecting its compacts’ likely ERR, and  economic growth and poverty effects. However, our analysis shows that  analytic errors affected the results, and the choice of analytical method  can change the results of economic projections. Without a consistent  approach and preferred methods for its due diligence economic analyses,  MCC’s transaction teams and individual team members have used different  methods to identify compact results. This heterogeneity means that  different teams could reach different conclusions about the worthiness of  individual projects or compacts based on the method chosen. MCC has  refined its guidance over time, and continues to do so. Further refinements  that include formal procedures for reviewing the results of due diligence  economic analyses and establishing a consistent approach with preferred  methods—which could be modified if required by specific country  conditions—would help MCC reduce the likelihood of errors and provide a  common MCC lens for projecting compacts’ ERRs and impacts. This in  turn would enhance the reliability and comparability of the information  MCC uses internally for decision making as well as the information it  provides to Congress and the public for oversight of MCC’s activities.
section_title: Recommendations for Executive Action
paragraphs: To improve the reliability and comparability of its projected ERR and  economic impacts, we recommend that the CEO of MCC take the  following actions:    Adopt and implement written procedures for a secondary independent  review of the methods and results of its economic analyses.
Improve MCC’s guidelines by identifying a consistent approach with  preferred methods for projecting compacts’ impact on income and  poverty.
section_title: Agency Comments and Our Evaluation
paragraphs: MCC provided comments regarding a draft of this report, which we have  reprinted, with our response, in appendix V. MCC also provided technical  clarifications, which we have incorporated as appropriate.
In commenting on a draft of this report, MCC concurred with our  recommendations and outlined related steps that it is taking—including  developing standard practices and templates, initiating an independent  peer review process, and posting its economic analyses on the Internet.  MCC commented that there is not a single cost-benefit standard practice  and that in many cases the inconsistencies we identified were technically  appropriate. We acknowledge that there is not a single standard practice;  however, we maintain that consistent analytic approaches are needed to  ensure the reliability of MCC’s analyses. According to MCC, the choice of  analytic time frames was in some cases based on the judgment of the  individual lead economist rather than on established criteria. The steps  that MCC has stated it is taking—using a default time frame for analyses  and subjecting the time frame decision to peer review—will help to  enhance the comparability across compacts of the time frame of ERR  calculations. MCC also stated that it disagreed with what it saw as our  report’s implication that ERRs are disconnected from income and poverty  effects. However, we assert that ERRs do not provide information about  MCC’s impact on any specific population group, including the poor. Also,  ERRs are a relative measure of benefits in relation to costs that can  fluctuate owing to changes in costs alone—and therefore cannot be  considered absolute measures of impact on income.
We are sending copies of this report to interested congressional  committees as well as the Chief Executive Officer of MCC. We will also  make copies available to others upon request. In addition, this report will  be available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: At the request of the chairman of the House Committee on Foreign Affairs,  we assessed the Millennium Challenge Corporation’s (MCC) economic  analyses, including its projections of the compact’s economic rate of  return (ERR), changes in income and poverty, and the number of  beneficiaries. To carry out this review, we reviewed MCC compacts with  four countries: Armenia, El Salvador, Lesotho, and Mozambique. We  selected these countries based on their percentage of total compact  funding, the recentness of the compact, and geographical representation.  These four compacts make up a total of about $1.56 billion in compact  assistance. When we began our work in August 2007, this represented  about 41 percent of the $3.85 billion MCC had set aside for 13 compacts  and included the most recent compacts signed in Eurasia, Latin America,  and Africa. We chose to include two African countries in  acknowledgement of MCC’s focus on Africa, which accounted for 7 of  MCC’s 13 compact countries and more than half of the total compact  value.
MCC’s compact-level projections are influenced by the specific data points  and assumptions used in project- and activity-level analysis. We  determined that the data we used were sufficiently reliable for the  purposes of our analysis; however, within our scope of work, we did not  independently evaluate the thousands of data points and assumptions at  the project and activity levels or the sensitivity analyses used in MCC’s  economic analyses and therefore are not assessing all aspects of the  validity of MCC’s ERR or impact projections. We also did not assess MCC’s  progress in implementing these compacts, and therefore its progress  toward achieving projected compact results.
To assess MCC’s compact ERR projections, we reviewed MCC’s public and  internal documents for statements of compact and project ERR. When we  began our review, MCC had published the compact and project ERRs for  only Armenia. However, for all four countries, MCC stated ERRs in the  internal investment memo. Next, we consulted MCC’s guidance for ERR  analyses as well as the spreadsheets provided by MCC to support its  calculations of ERR and document its work. We reviewed MCC guidance  on calculating ERR and minimum ERRs and compared MCC’s internal  documentation and spreadsheets to elements of this guidance. We  examined the spreadsheets to determine how MCC aggregated project- level ERRs into one compact-level ERR for the four countries in our  review and identified the two approaches—-summing net benefits and  using weighted averages—that MCC used. To determine the effects of  MCC’s alternative approach on the compact ERR, we used MCC’s cost and  benefit data to calculate the alternative ERRs. We also used these data to  calculate compact- and project-level ERRs over different time horizons to  explore the sensitivity of the ERRs to differing time horizons. To  determine how MCC approached same-sector projects in different  compacts, we studied road and water projects because these were each in  three of the four compacts. We then compared the broad approaches MCC  used in assessing the costs and benefits associated with these projects. We  interviewed MCC economists and other officials regarding MCC’s ERR  analysis to further discuss MCC’s approaches and clarify aspects of MCC’s  analysis. We also consulted Office of Management and Budget (OMB)  guidance and GAO guidance on establishing and implementing internal  controls to inform our assessment of MCC’s processes for developing and  maintaining information for the purpose of management decision making.
To assess MCC’s projections of the number of compact beneficiaries, and  changes in income and poverty, we first compiled and analyzed MCC’s  public documents for statements of compact impact. These documents  included MCC’s compacts, compact summaries, congressional  notifications and budget justifications, and MCC’s annual reports. MCC  makes all of these documents available online. We identified statements  that fell into four categories of economic projections: impact on national  or regional income, impact on national or regional gross domestic product  (GDP) growth rate, impact on national or regional poverty, and number of  compact beneficiaries. MCC reviewed and concurred with our compilation  and summary of these statements of compact impact. We also reviewed  MCC’s guidance on projecting these economic impacts. We reviewed the  spreadsheets for each compact that MCC used to conduct and document  its economic analyses. After an initial examination of these spreadsheets,  we met with MCC economists and other officials to discuss MCC’s  methods and calculations for projecting compact impact. MCC officials  also provided responses to our questions in written form. In these  responses, MCC revised its initial calculations of its impacts in Armenia, El  Salvador, and Mozambique and also presented alternative methods. We  then reviewed the updated information that MCC submitted to us and  calculated the magnitude of the difference in MCC’s original statements  and those supported by its revised analyses. In the case of Lesotho, we  reviewed the World Bank Country Economic Memo regarding the  economic growth model that MCC used to estimate compact impact in  Lesotho. We also reviewed World Bank country assistance strategies and  International Monetary Fund Article IV consultation reports and Poverty  Reduction Strategy Papers to improve our contextual understanding of  each country’s compact program. Finally, we consulted OMB guidance and  GAO guidance on establishing and implementing internal controls to  inform our assessment of MCC’s internal processes.
We conducted this performance audit from August 2007 to June 2008, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: MCC Minimum Acceptable ERRs
paragraphs: MCC has issued two definitions of the minimum acceptable ERR, which MCC  refers to as the hurdle rate, for compacts and projects. In its initial guidelines,  issued in April 2005, MCC defined the hurdle rate as the average of the  country’s real growth rates for the previous 3 years. The January 2006  guidelines did not set a hurdle rate. The November 2006 guidelines defined  the hurdle rate as the greater of (1) two times the average real growth rate of  GDP for the country for the most recent 3 years for which data are available  or (2) two times the average real growth rate of GDP for all of the MCC  eligible countries for each country for the most recent 3 years for which data  are available. The November 2006 guidelines also state that the hurdle rate  may not be higher than 15 percent.
In setting the hurdle rate for each compact that we reviewed, MCC applied  the definition of the rate from the guidelines current at the time. For  Armenia and Mozambique, MCC used its April 2005 guidance. For El  Salvador, according to MCC officials, and for Lesotho, MCC used the  definition in the November 2006 guidelines. (See table 2.)
In two cases, the ERRs that MCC initially calculated did not meet the  compact hurdle rates. MCC originally calculated the El Salvador  Community Infrastructure Project ERR as slightly below the hurdle rate at  10.4 percent over 25 years. MCC’s investment memo for Lesotho stated  that the ERR for the Rural Water Project was under 6 percent. Subsequent  revisions to the Lesotho analysis reduced the projected ERR to less than 1  percent. The Lesotho investment memo discusses the 6 percent ERR but  notes a different view within the transaction team that other benefits are  not captured by the economic analyses.
section_title: Appendix III: Compact ERRs
paragraphs: We assessed the sensitivity of MCC’s compact-level ERRs to the use of  varying time frames and varying methods for calculating the compact-level  ERR. We found in each case that the variance does affect the results, but  does not change the ERRs to below the applicable hurdle rate.
subsections: section_title: Time Frames
paragraphs: MCC used different time frames to calculate and report compact ERRs.
For Armenia, MCC calculated and reported a 20-year time frame for the  compact and both of its projects.
For El Salvador, MCC calculated a 25-year ERR for most projects and for  some projects reported the ERR as a 25-year ERR in its investment memo.  MCC officials explained that their El Salvador country counterparts  originally performed the analysis over 25 years and that MCC’s lead  economist determined this to be a reasonable approach.
For Lesotho, MCC used a 20-year ERR for all but one ERR calculation but  did not state the time frame of the calculations in the investment memo.
For Mozambique, MCC used a 24-year ERR for most compact projects.
According to MCC officials, MCC intended to present a 25-year ERR, but a  delay in the beginning of compact implementation pushed the compact’s  time horizon 1 year into the future and reduced the time frame for the  analysis to 24 years. The Mozambique investment memo does not state the  number of years used for the time frame of the ERR calculation.
Table 3 summarizes our comparison of the compact hurdle rates with the  ERRs that MCC reported and our calculations of the 20-year ERRs. Our  analysis shows that applying a 20-year time frame in place of varying time  frames does not significantly affect the results of the ERR calculations by  lowering it below the hurdle rate.
section_title: Methods
paragraphs: Figure 8 illustrates the two different methods—summing net benefits and  calculating a weighted average—that MCC used to determine a compact- level ERR.
We compared the results of the compact ERR calculation using different  methods and determined that the choice of method affects the results, but  does not reduce the ERR below the hurdle rate. See table 4 for details of  our analysis.
section_title: Appendix IV: Impact of Alternative Beneficiary Counts on El Salvador Compact Impact Projections
paragraphs: During the course of our engagement, MCC changed its method for  determining the beneficiaries and benefits of education projects in El  Salvador. In its initial calculations for both projects, MCC assumed that all  students entering the programs would experience an increase in income.  In its revised calculations for the formal education project, MCC assumed  that all nongraduates would have zero income increases—an  underestimate of the impact—because of lack of data. For the informal  education project, MCC estimated an income increase of more than 200  percent for those who obtain employment, based on a study conducted by  the training institute in El Salvador. However, MCC assumed that many  entering students would not complete the program or obtain employment  and their income would therefore not increase. (See table 5.)
section_title: Appendix V: Comments from the Millennium Challenge Corporation
paragraphs: The following are GAO’s comments from the Millennium Challenge  Corporation letter, dated May 23, 2008.
subsections: section_title: GAO Comments
paragraphs: 1.  We recognize MCC’s constructive responses to the issues identified in  the course of our audit—including posting economic analyses on the  Internet, developing standard practices and templates, and initiating an  independent peer review process.  2.  We acknowledge that there is not a single standard practice; however,  we maintain that consistent analytic approaches are needed to ensure  the reliability and comparability of MCC’s analyses. According to MCC,  the choice of analytic time frames was in some cases based on the  judgment of the individual lead economist rather than on established  criteria. The steps that MCC has stated it is taking—using a default  time frame for analyses and subjecting the time frame decision to peer  review—will help to enhance the comparability of its of ERR  calculations across compacts.  3.  We disagree that MCC’s ERRs estimate the project’s effect on people’s  lives. ERRs do not provide information about MCC’s impact on any  specific population group, including the poor. ERRs also are a relative  measure of benefits in relation to costs and can fluctuate owing to  changes in costs alone. For example, if costs increase and the income  benefits remain the same, the ERR would decrease. Therefore, ERRs  cannot be considered an absolute measure of income benefits.
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Emil Friberg, Jr. (Assistant  Director), Todd M. Anderson, Gergana Danailova-Trainor, Reid Lowe,  Michael Simon, and Seyda Wentworth made key contributions to this  report. Also, C. Etana Finkler, Ernie Jackson, and Tom McCool provided  technical assistance.