Millennium Challenge Corporation: Results of Transportation Infrastructure Projects in Seven Countries
Published: 2012-09-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To help developing countries reduce poverty and stimulate economic growth, MCC has approved 26 bilateral compact agreements totaling about $9.3 billion. In the seven compacts that ended in 2010 and 2011--Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, Armenia, and Benin--transportation infrastructure projects generally received about 50 percent of the compact's total funding. To measure the results of its compacts, MCC sets targets for various performance indicators--such as number of kilometers paved or volume of merchandise passing through a port--and estimates the number of beneficiaries. This report, responding to a congressional mandate, examines the extent to which MCC has, for transportation infrastructure projects, (1) achieved expected performance targets and (2) consistently estimated numbers of beneficiaries. GAO analyzed MCC documents, interviewed MCC officials, and drew on fieldwork completed for related work in four of the seven countries.
section_title: What GAO Found
paragraphs: The Millennium Challenge Corporation (MCC)--a U.S. government corporation-- recognizes the importance of a disciplined, transparent, and accountable approach to tracking compact results. However, it reduced the scopes of its early transportation infrastructure projects and reports mixed success in meeting key performance targets. In addition, problems with data quality call into question the reliability of those reported results. GAO found the following for the seven compacts ending in 2010 and 2011, each with a road project or a port project. Road Projects: (1) MCC reduced kilometers to be paved under six compacts--Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, and Armenia--by a combined 63 percent (from about 1,800 to 600 km) because of increased construction costs and political problems in partner governments. MCC reported meeting reduced targets for five compacts. However, for three compacts, MCC did not consistently account for kilometers completed with funding from third parties. (2) MCC reported meeting revised targets for road roughness--a measure of pavement quality--for five of the compacts. However, reported data have quality issues, including the inconsistent application of measurement methodologies and calculation errors that resulted in overstated results. (3) MCC reported meeting targets for annual average daily traffic--a measure of the volume of traffic using the road--for three of the compacts. However, weaknesses in traffic baseline estimates may have affected the establishment of targets and therefore MCC's ability to measure results. Port Projects: (1) In Cape Verde, MCC funding ($53.7 million) was insufficient to construct all planned port elements. As a result, MCC reduced the project's scope and deferred measuring the results of key indicators. (2) In Benin, MCC completed most of the envisioned scope of the port project. MCC's data show that the compact met the original target for one of three key performance indicators, volume of merchandise. (The other two indicators relate to the measurement of shipping costs.) However, GAO found that MCC's estimation of this indicator's baseline may lead to overstated results. In addition, data quality reviews identified problems with the data used, which MCA-Benin did not formally address. In 2009, MCC improved its methodology for estimating beneficiaries--people who realize income gains or expenditure savings as a result of its investment--by standardizing its approach. MCC subsequently revised its beneficiary numbers for all compacts. However, the new approach did not include a formal quality review process. As a result, implementation of the new approach suffered from varying degrees of quality problems. For example, (1) MCC did not implement its beneficiary estimation methodology consistently across early transportation infrastructure projects; (2) beneficiary calculations contained incorrect formulas and numbers, and differed from supporting documents; and (3) beneficiary figures in MCC's public documents were sometimes inaccurate.
section_title: What GAO Recommends
paragraphs: MCC should strengthen existing policies and practices regarding measuring and evaluating results data and formalize a quality review process to improve its beneficiary calculations. MCC agreed with all of our recommendations and outlined some steps the agency will take or has already taken to address them.

section_title: Letter
paragraphs: The Millennium Challenge Corporation (MCC), a U.S. government  corporation, was established in 2004 to provide aid to developing  countries that have demonstrated a commitment to ruling justly,  encouraging economic freedom, and investing in people. MCC provides  assistance to eligible countries through multiyear compact agreements to  fund specific projects aimed at reducing poverty and stimulating economic  growth. MCC’s core principles include assessing the results of its funding  to determine how its activities affect poverty and economic growth. To do  this, MCC has instituted a monitoring and evaluation approach that  includes identifying relevant indicators to measure results during and at  the end of project implementation and establishing performance targets  for each indicator. MCC’s efforts to determine how its activities will affect  poverty and economic growth also include estimating the number of  people who will benefit from its projects and analyzing the impact of its  projects on populations of specific interest.
As of June 2012, MCC had signed 26 compacts, committing a total of  approximately $9.3 billion. Seven of these compacts ended in 2010 or  2011: Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, Armenia,  and Benin. (See app. II for compact time frames.)
In the fiscal year 2008 Consolidated Appropriations Act, Congress  mandated that we review the results achieved by MCC compacts. We  have previously reported on the results of MCC’s transportation  infrastructure projects in four countries. In July 2011, we reported that  MCC had met reduced targets for its compacts with Cape Verde and  Honduras, the first two compacts to reach completion. In June 2012, we  reported that MCC transportation infrastructure projects in Georgia and  Benin varied in quality and may not be sustainable. Because much of  MCC’s funding for compacts that ended in 2010 and 2011 focused on  transportation infrastructure projects, this report examines the extent to  which MCC has, for transportation infrastructure projects, (1) achieved  expected performance targets and (2) used a consistent methodology to  estimate numbers of beneficiaries.
To assess the extent to which MCC has achieved its performance targets  for transportation infrastructure projects under compacts that ended in  2010 and 2011, we reviewed MCC guidance and policy documents and  analyzed compact agreements, monitoring and evaluation plans, and  results data. We compared actual results achieved at the end of the  compact for select performance indicators with MCC’s original targets  and, in some cases, with revised targets associated with each indicator.  Since MCC tracks many performance indicators for each compact project,  we selected a subset of indicators to examine for this report.
For the road projects, we examined kilometers of road completed,  roughness as a measure of road quality, and average annual daily  traffic. We selected those road project indicators because they  address the projects’ key objectives and because they are among the  “common indicators” MCC requires for all road projects, so that it can  aggregate results across countries.
For the port projects, we examined the volume of merchandise  through the port and two indicators related to the measurement of  shipping costs: container ship time at berth and container ship time at  anchor. Because MCC has not established common indicators for  ports, we selected indicators that addressed the projects’ key  objectives, corresponded to the common road indicators, and were  originally included in both port projects.
Additionally, we interviewed MCC officials about data quality and indicator  tracking tables. As part of our related engagements—focused on Cape  Verde and Honduras, and on Georgia and Benin—we visited compact  projects in-country and met with partner country government officials,  including those responsible for compact implementation, and with  contractors, project managers, construction supervisors, and relevant  private businesses.
To assess the extent to which MCC has consistently applied its  beneficiary estimation methodology, we reviewed MCC’s Guidelines for  Economic and Beneficiary Analysis as well as its beneficiary calculations  and accompanying supporting documents for the seven compacts that  ended in 2010 and 2011. We compared original and revised beneficiary  estimates for the transportation projects and assessed the justification  behind any changes. Data are considered reliable when they are  accurate, complete, consistent, and valid, given the uses for which they  are intended.
MCC enters into a legal relationship with partner country governments  that vest an accountable entity with responsibility for day-to-day  management of compact project implementation, including monitoring  and evaluation activities such as setting and revising performance  targets. Because such actions require MCC’s direct oversight and  approval, throughout this report, we attribute all decisions related to  project rescoping and compact targets to MCC. (See app. I for further  details of our objectives, scope, and methodology.)
We conducted this performance audit from August 2011 to September  2012, as part of a body of work on this subject, in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe the evidence obtained provides  a reasonable basis for our findings and conclusions based on our audit  objectives.
subsections: section_title: Background
subsections: section_title: MCC Organization
paragraphs: MCC is managed by a chief executive officer, appointed by the President  with the advice and consent of the Senate, and is overseen by a Board of  Directors. The Secretary of State serves as board chair and the Secretary  of the Treasury as vice-chair. MCC’s model of providing foreign aid is  based on a set of core principles deemed essential for effective  development assistance, including good governance, country ownership,  focus on results, and transparency. According to MCC, country ownership  of an MCC compact occurs when a country’s national government  controls the prioritization process during compact development, is  responsible for implementation, and is accountable to its domestic  stakeholders for decision making and results. To implement the compact,  the partner government establishes an accountable entity, referred to as  a Millennium Challenge Account (MCA). MCC provides the framework  and guidance for compact implementation, monitoring, and evaluation  that MCAs are to use in implementing compact projects.
section_title: Transportation Infrastructure Projects
paragraphs: The seven MCC compacts completed in 2010 and 2011 each included a  transportation infrastructure project that—except in the case of Armenia— received 50 percent or more of the compact’s total funding (see fig.1). For  the purposes of this report, we have defined transportation infrastructure  as public works that convey passengers or goods from one place to  another; infrastructure includes structures such as roads, seaports,  airports, and railways. We reviewed five compacts with road projects, one  with a port project, and one with both types of projects.
According to MCC’s Policy for Monitoring and Evaluation of Compacts  and Threshold Programs, performance monitoring helps track progress  toward compact goals and objectives and serves as a management tool.  Therefore, MCC requires partner countries to create a monitoring and  evaluation plan that they update throughout the course of the compact.  The plan’s monitoring component outlines the performance indicators by  which the compact results will be measured. It also establishes a  performance target for each indicator, including a baseline value (the  starting value of an indicator) and the expected year by which the target  will be achieved. For some indicators, baseline measurements are  necessary to establish interim and final targets. MCC and the MCAs  monitor the progress of compact activities using an indicator tracking  table, which is a reporting tool that displays targets and tracks progress  against them. (See fig. 2.)
MCC established common indicators in 2009 to aggregate results across  countries. For road projects, MCC has established several common  indicators, including the following:
Kilometers of road completed: A measure of the total length of road  completed under a compact. This indicator is a cumulative measure of  works completed.
Road roughness: A measure of pavement quality that affects ride  quality, vehicle delay costs, fuel consumption, and maintenance costs.  Roughness is used to determine the reduction in road-user costs and  indicates when resurfacing work is needed. New pavement roughness  ratings—as measured by the international roughness index (IRI)— should range from 1.5 to 3.5; the higher the IRI, the rougher the  pavement, with a roughness of 0 being perfectly smooth.
Average annual daily traffic: A measure of the volume and type of  traffic using a road.
MCC has not established common indicators for ports, because ports are  a small part of its portfolio (it has funded port projects in only two  countries). However, we selected certain key indicators to review  because they addressed the projects’ key objective of economic growth:
Volume of merchandise through the port: A measure of the total  volume of exports and imports passing through the port each year.
Container ship time at berth: A measure of the average number of  days a container ship spends at berth to load and/or unload its cargo.
Container ship time at anchor: A measure of the average number of  hours a container ship spends at anchor waiting for a berth.
MCC requires that MCAs contract an independent entity (such as a local  or international firm, research organization, or individual consultant) to  review the quality of compact performance data. These reviews  determine the utility, objectivity, and integrity of information gathered,  identify problems with data quality, and recommend actions to remedy  these issues. Reviews should ensure that data used to measure  indicators meet the following criteria: validity, reliability, timeliness,  precision, and integrity. MCC policy allows each country to determine the  frequency and timing of each review as well as the indicators to be  evaluated. This requirement was fulfilled for every compact except  Nicaragua. MCC policy requires the MCA to comment on the review,  including noting the recommendations it will implement. In addition, the  MCA should reflect, in a subsequent monitoring and evaluation plan, any  indicator changes made in response to data quality reviews. MCC policy  also requires the MCA to make publically available on its website a  summary of the final reviews and its comments. MCAs are responsible for  ensuring that MCC-approved recommendations are implemented.
section_title: Beneficiary Assessment
paragraphs: MCC defines beneficiaries as people who realize income gains or  expenditure savings as a result of its investment. MCC’s authorizing  legislation requires that compacts contain an identification of the intended  beneficiaries, disaggregated by income level, gender, and age, to the  maximum extent practicable. MCC notes that “the use of these   analyses and their availability to the general public are  trademarks of MCC’s commitment to transparency and results-based  aid.” In addition, MCC’s 2009 Guidelines for Economic and Beneficiary  Analysis maintain that identification of intended beneficiaries should not  be limited to counting the number of people who benefit from compacts  but should include a more detailed analysis of the effect of compacts on  different subgroups, including the poor, the elderly, women, children, and  ethnic subpopulations. MCC has not undertaken such detailed beneficiary  analysis for the seven compacts completed in 2010 or 2011.
section_title: MCC Data Show that Some Key Performance Targets Were Met, but Data Quality Problems Call into Question Reported Results
paragraphs: MCC reduced the scope of early road and port projects and reports mixed  success in meeting key performance targets. In addition, problems with  data quality call those reported results into question.
subsections: section_title: MCC Reduced Targets for Completed Roads by 63 Percent and Had Data Quality Problems
paragraphs: In each of the six compacts with road projects, MCC reduced the number  of kilometers to be completed. In addition, the roughness and annual  average daily traffic indicators had data quality problems. Most compacts  did not examine the three common road indicators in their data quality  reviews. In addition, MCC did not consistently account for third-party  funding.
MCC reduced by 63 percent the total number of road kilometers to be  completed for the six compacts with road projects, from an original  combined target of 1,822.8 kilometers to a final combined target of 623.9  kilometers. Five of six compacts met their reduced targets for kilometers  completed. Honduras was the only compact that did not meet its target  (see table 1).
In general, MCC decreased the scope of the road projects because of  increased construction costs, changes in exchange rates, or actions  taken by host country governments that were inconsistent with MCC’s  eligibility criteria. For example, MCC decreased the scope of the  compacts in Nicaragua and Armenia because of concerns about  democratic governance in the respective partner governments   (see table 2).
MCC data show that five of six compacts met original targets for road  roughness. In addition, MCC reports that the Georgia and Armenia  compacts surpassed their original roughness targets and built a smoother  road than originally planned. Honduras did not did not meet its  roughness target. (See table 3.)
We identified problems with the measurement of road roughness for  some of the six compacts with road projects, including the use of  inconsistent application of methodologies and calculation errors. These  problems call into question MCC’s reported roughness results.
Inconsistent application of methodologies. MCC determined  roughness using a visual scale for some roads and mechanical  measurements for others. While both methods are legitimate means of  measuring roughness, they have different degrees of accuracy.  Moreover, MCC did not apply either measurement method consistently  across compacts. First, for the two countries measuring roughness  visually, Cape Verde and Vanuatu, MCC inappropriately provided the  MCAs a World Bank scale for unpaved roads to assess the paved roads.  Second, for compacts where roughness was measured mechanically, the  MCAs used different machines that had varying levels of accuracy. Third,  for three compacts—Nicaragua, Honduras, and Armenia—the MCAs  measured roughness on both sides of the road and then averaged these  measurements to calculate a final measure. In contrast, MCA-Georgia  calculated roughness using measurements taken on one side of the road  only, potentially producing a less accurate measurement since roughness  can vary from one side of the road to the other.
Calculation errors. We found that errors calculating roughness led to an  overstatement of results. For example, MCA-Nicaragua recorded zeros— an IRI rating for a perfectly smooth road—for sections that could not be  measured, incorrectly lowering the overall roughness calculation from 4.8  to 3.4 for one road and indicating that the road was smoother than it  actually was. differing lengths of each segment of secondary road when aggregating  the roughness values and therefore overstated its reported results at 3.2  instead of 3.3. Although this difference is not large, consistent  methodologies should be used across compacts to allow a valid basis for  comparing and evaluating results.
For two of the road segments, the nonmeasurable sections of road were left blank and  the calculation of the final IRI used the value from the one measurable side of the road as  the overall measure for that kilometer.
subsections: section_title: MCC-Reported Results for Road Traffic Have Data Quality Issues
paragraphs: MCC data show that three of the six compacts with road projects met  original or revised targets for average annual daily traffic. The  Nicaragua and Armenia compacts met their original targets and the  Vanuatu compact met a reduced target. For the other three compacts,  Cape Verde did not meet its original target; Georgia did not meet its  reduced target; and MCC stated that a final traffic measure in Honduras is  not expected until late 2012. (See table 4.)
MCC did not consistently update the baseline estimates for the annual  average daily traffic indicator either by projecting growth or by using new  information when it became available. Because MCC establishes the  targets for some indicators by applying estimated growth rates to the  baselines, these identified weaknesses may have affected MCC’s ability  to create accurate traffic targets, thus affecting its reported results.
For the secondary roads project in Nicaragua, MCC stated that it  updated the baseline estimates established in 2007 by applying 1  year’s growth rate to the traffic measures, because construction was  expected to begin in fall 2008. However, MCC did not use this method  in Cape Verde, where there was a similar time lag.
For the section of highway to be rehabilitated in Nicaragua, MCC  retained a baseline calculated in a 2000 traffic study had 2007 data for the specific 18 kilometers of road it planned to  rehabilitate.
In Honduras, MCC used a baseline established before 2005 despite  the availability of data from a 2008 traffic survey. The 2008 traffic  survey results showed that traffic volumes already met or exceeded  the end-of-compact targets for most road segments. Because MCC  used the 2005 data, the final results for the project indicate a larger  change over the compact implementation timeframe than actually  occurred.
In addition, the 2000 traffic study was conducted for a 72-kilometer section of highway,  rather than the 18 kilometers designated for the project. quality reviews for Armenia included two of the three indicators, the  compact was put on hold before MCA-Armenia could fully correct  identified problems. In Vanuatu, data quality reviews conducted in 2008  and 2009 identified problems with the measurement of annual average  daily traffic. MCA-Vanuatu did not provide a response to either of the  recommendations in the review and made no changes to the definition of  the traffic indicator in the final monitoring and evaluation plan. The data  quality reviews for the Honduras and Cape Verde compacts did not  include any of the three road indicators. Table 5 summarizes the  treatment of the three common road indicators we examined in data  quality reviews for the six compacts.
The governments of three countries—Honduras, Cape Verde, and  Vanuatu—secured funding from a third party to rehabilitate some of the  road lengths that MCC removed from the scope of the compact.  However, we found that MCC did not consistently account for this third- party funding when reporting the final results of kilometers completed. As  a result, MCC understated its results in Honduras and overstated them in  Vanuatu.
The government of Honduras secured money from the Central  American Bank for Economic Integration to complete 57.6 kilometers  of the highway project, which MCC removed from the compact’s  original scope. However, MCA-Honduras did not revise its  kilometers-completed target to reflect this reduction in scope.  Therefore, MCC reports that it did not meet the compact’s final  kilometers-completed target, even though data suggest that the  kilometers MCC funded were complete at the end of the compact
The government of Cape Verde secured funding from the government  of Portugal to rehabilitate 22.9 kilometers of secondary roads that  MCC removed from the scope of its compact. MCA-Cape Verde  revised its kilometers-completed target to reflect this reduction in  scope and did not report these 22.9 kilometers as a part of its final  results. Therefore, MCC reports that it met the compact’s revised  kilometers-completed target.
The government of Vanuatu secured money from New Zealand to  assist in rehabilitating a 57.2 kilometer road, because the MCC  compact funds were not sufficient to complete all 57.2 kilometers.  However, MCC reported all 57.2 kilometers as a part of its final  results, even though it did not fund all of the work.
section_title: MCC Reduced the Cape Verde Port Project Scope and Did Not Meet Many Targets for the Benin Port Project
paragraphs: For the Cape Verde compact, MCC reduced the scope of the port project,  deferred the measurement of key indicators, and therefore did not  conduct any data quality reviews of those indicators. In Benin, MCC  completed most of the originally envisioned port project scope, and its  data show that the compact met the original target for one of the three  key indicators. However, we identified issues with the indicators’ baseline  estimates. In addition, data quality reviews identified problems with the  data behind the indicators, which MCA-Benin did not formally address.
MCC split the Cape Verde project into two phases in May 2008 because,  owing to increased construction costs, the $53.7 million allocated to the  port projects was insufficient to fund all of the planned improvements.  MCC funded phase 1 of the improvements, including construction of a  cargo storage area and an access road, and rehabilitation of Wharf 2.  The government of Cape Verde secured a loan from the government of  Portugal for $87 million to complete phase 2, that is, those elements  removed from the scope of the MCC project. The MCC-funded portion  of the construction (phase 1) represents about one-third of the total  expected cost of both phases. Nearly 100 percent of the works for phase  1 were completed by compact end. Phase 2 was awarded late in the  compact, and work was under way as of July 2012.
MCC deferred the measurement of key performance indicators when it  revised the project scope, because results for key indicators could be  measured only after full completion of the port (see table 6). MCC intends  to measure results for the key indicators when phase 2 is complete.
MCA-Cape Verde contracted for data quality reviews in 2007 and 2010.  The port project was reviewed in 2010; however, key indicators were not  evaluated, because MCA-Cape Verde deferred measurement of those  indicators until phase 2 was complete.
MCC data show that Benin maintained most of the original project scope  and met the original target for one of three key indicators. The project  scope included constructing or rehabilitating a jetty, a new wharf, and port  roads and installing lighting, security, and an electricity distribution  system. MCC reports that the Benin compact met the original target for  volume of merchandise through the port, but it did not meet targets for the  other two key indicators, container ship waiting time at berth and  container ship waiting time at anchor. See table 7 for a summary of key  performance results for the Benin port project.
We identified problems with MCC’s measurement of the volume-of- merchandise indicators, and MCA-Benin’s data quality reviews identified  additional problems with all three key indicators. For the volume-of- merchandise indicator in Benin, MCC used 2004 data from the Port  Authority to establish a baseline of 4.1 million metric tons in the compact’s  monitoring and evaluation plans and indicator tracking tables. MCA-Benin  did not subsequently update the baseline. However, the compact did not  enter into force until October 2006 and port improvement construction did  not begin until August 2009. Available annual data for Benin showed that  in 2005 the volume of merchandise had already increased to 5.2 million  metric tons. Without an updated baseline, the estimate may not  accurately reflect preconstruction conditions and may lead MCC to  overstate the degree to which port traffic increased because of the  compact.
MCA-Benin contracted data quality reviews in 2008, 2009, and 2011. The  2008 and 2009 data quality reviews identified issues with the three key  port indicators that we reviewed, but MCC did not address all of the  issues. The 2011 review did not examine these specific port indicators.
For the volume-of-merchandise indicator, MCA-contractors noted that  although data collection methods were consistent and the data were  valid, the Port Authority of Cotonou needed to improve the timeliness  of data related to this indicator because the data entry and control  procedures created delays in the availability of information. MCC  policy indicates that data should be sufficiently current to inform  management decisions because decisions depend on regular  collection of performance information. In 2008 and 2009, the  contractors recommended that MCA-Benin set up an automatic data  transfer system for information collected on ship stopover, to avoid  redundant data entry efforts by the port’s statistical services. Although  the 2009 review indicated that the port was resolving problems  through the creation of a port information system when the evaluation  was conducted, we found that MCA-Benin did not resolve the issues  identified for this indicator. No changes were made in the final  monitoring and evaluation plan to indicate that adjustments were  made in the data collection methods for this indicator. In addition,  MCA-Benin responses to the review did not address this issue.
For the container ship time at berth and container ship time at anchor  indicators, the 2008 and 2009 reviews noted that, while the data  collected were a direct measure of what the indicators were intended  to capture and the data collection procedures were consistent, there  were no historical data to support the baseline values for these  indicators. The reviews recommended using baseline values  established in a 2005 report by an international consulting firm. We  found that MCA-Benin did not resolve issues identified for these two  indicators. MCC and MCA officials stated that they decided not to  revise indicator baselines and targets as recommended, and no  revisions were made to baseline figures in subsequent monitoring and  evaluation plans. MCC and MCA officials provided no additional  explanation.
section_title: Data Resulting from Beneficiary Calculations May Not Be Valid
paragraphs: In 2009, MCC improved its beneficiary estimates by adopting a more  standardized approach, which it documented in its Guidelines for  Economic and Beneficiary Analysis; however, the updated guidelines do  not include a formal quality review process. MCC subsequently revised its  beneficiary estimates for ongoing compacts, but the implementation of the  new approach was not always consistent and suffered from varying  degrees of quality control problems. Although some of the individual  problems we identified were small in nature, taken as a whole they  reduce confidence in MCC’s estimates.
subsections: section_title: MCC’s Updated Beneficiary Analysis Guidelines Do Not Include a Formal Quality Review Process
paragraphs: In our previous work, we found that MCC did not have a consistent  methodology for estimating the number of beneficiaries across  compacts. As a result, MCC adopted a new, more standardized  approach, which it published in its 2009 Guidelines for Economic and  Beneficiary Analysis. MCC subsequently conducted a “beneficiary scrub,”  revising beneficiary numbers for ongoing compacts based on the 2009  guidelines. However, we found errors and inconsistencies in the  calculations. MCC stated that these scrubs are informally reviewed but  that they do not undergo a formal quality review process. Such a  process would ensure that the information and data provided have been  verified for consistency and accuracy.
section_title: Weaknesses Call into Question the Validity and Reliability of MCC’s Beneficiary Data
paragraphs: We identified three general weaknesses with MCC’s beneficiary  calculations. First, the beneficiary calculations in MCC’s internal  documents contain weaknesses such as incorrect formulas and numbers  as well as discrepancies with supporting documents. Second, MCC did  not apply a consistent methodology in estimating beneficiaries for its early  transportation infrastructure projects. Third, the beneficiary figures in  MCC’s public documents are sometimes inaccurate.
Some beneficiary calculations contained mistakes in the formulas used.  For example, the computations of the population growth rate for Georgia  and Cape Verde contained an error in the mathematical formula, causing  MCC to slightly overstate the growth rate in the first case and understate  it in the second case. In addition, erroneous numbers were used in four of  the seven compacts we reviewed.
For the Georgia road project, MCC stated in the beneficiary  calculation documents that “the original beneficiary estimate of 53,988  people counts all households living in the four rayons, or districts,  through which the road passes…The revised beneficiary estimate [of  58,079] is based on a 5 km catchment area of the Road.” MCC thus  originally claimed a lower number of beneficiaries for a larger  geographic area (the districts) than the more limited catchment area— the geographic area in which benefits may be expected to accrue—of  5 kilometers on either side of the road.
For the Benin port project, the population projection in the beneficiary  scrub began with a 2005 original baseline figure of 8,490,000, since  the port project is expected to raise the incomes of the entire  population of Benin. However, this number could not be derived from  its purported source, the 2002 population census.
For the Armenia road project, the document supporting the beneficiary  scrub calculation considered a catchment area for an erroneous  revised road length of 68 kilometers rather than the correct figure of  24.4 kilometers.
In the Cape Verde roads and bridges activity, the original beneficiary  numbers provided in the document supporting the beneficiary scrub  spreadsheet did not match the figures in the spreadsheet.
In the 2009 revision of its Guidelines for Economic and Beneficiary  Analysis, MCC standardized its methodology to estimate beneficiaries  and accordingly modified its beneficiary numbers in the spreadsheets for  its beneficiary scrubs. However, the amount of detail in the analysis  varied across compacts, and population growth rates were not always  calculated in a consistent manner.
Of the beneficiary scrubs we reviewed, four of seven contained  disaggregated data at the project level. Specifically, the Armenia,  Georgia, Nicaragua, and Vanuatu scrubs are made up of multiple,  relatively detailed spreadsheets. By contrast, one beneficiary scrub, for  Honduras, contained one page of macro-level summary data with no  explanation or sources for the figures. The remaining two of the seven  scrubs contained additional data but were not as detailed as the first four.  For example, the Cape Verde scrub included a second sheet of World  Bank development indicators on the country, and the Benin scrub had  some details on one of the compact projects. These varying levels of  detail suggest that the scrubs were executed in an ad hoc, rather than  consistent, manner.
MCC uses population growth rates to estimate the number of  beneficiaries in a 20-year projection period; for each compact, MCC  publicly presents the number of beneficiaries projected for the end of the  20-year period. However, the scrub sheets we reviewed showed that  MCC sometimes computed population growth rates differently across  compacts, even when the scrubs provided the same underlying  information.
For Honduras, MCC indicated that it took the population growth rates  for 2005, 2006, and 2007 from World Bank data and averaged the  three rates to get an average annual growth rate for the projection.
For Georgia and Cape Verde, MCC did not use the historical growth  rates provided but instead took total population figures in 2005 and  2007 and used them to compute an average annual growth rate for  the beneficiary projection.
For Vanuatu, MCC computed an average annual growth rate based  on four historical growth rates without specifying a source or year.
In three of seven compacts, we found that some numbers in MCC’s public  documents had not been updated or were inaccurate.
In the case of Benin and Nicaragua, the growth rate for the year 2000 was added to the  formula. beneficiaries for the compact as a whole (143,000) was smaller than  the beneficiaries of the infrastructure project alone (300,000).
For the Benin port project, the beneficiary estimate that appeared in a  2011 status report (8,791,832) was an original estimate from 2009  instead of the revised estimate of 16,119,058. In addition, the  supporting document accompanying the 2009 scrub spreadsheet  mistakenly reported 13,421,086 beneficiaries for the port project by  2026. However, the data in the scrub indicated that this figure referred  to the number of beneficiaries in 2020; the correct number of  projected beneficiaries for 2026 is 16,119,058.
For Armenia, the beneficiary estimate for the road project in a 2011  publication (6,216) did not reflect a revision of the estimate done in  2009 (6,356).
section_title: Conclusions
paragraphs: MCC recognizes the importance of a disciplined, transparent, and  accountable approach to tracking compact results to make well-informed  decisions about U.S. investments. It has established an extensive  monitoring and evaluation program that includes guidance and tools for  the partner countries and controls on data quality. In addition, MCC has  created and implemented a revised beneficiary calculation methodology  to ensure standardization across projects and compacts.
Even with these steps, problems exist with the data behind several of  MCC’s key performance indicators, limiting the reliability of MCC’s  reported results.
MCAs have not measured the results of common indicators in a  uniform manner—including how they account for additional third-party  funding—nor have they consistently reviewed the quality of data used  to report the indicators’ results. As such, MCC cannot confidently  compare or aggregate the results of these common indicators as  intended.
MCC requires the establishment of baselines to measure the impact  its assistance has on partner countries. However, because some  incorrect baselines were used, results may be over- or understated.
MCAs have not consistently fulfilled the MCC requirements that they  conduct independent data quality reviews and respond to issues the  reviews raised. As a result, MCC and its partners have missed  opportunities to improve data quality and facilitate data reporting and  aggregation.
In addition, MCC’s beneficiary calculations have problems that limit their  usefulness. Because MCC’s new approach to estimating beneficiary  numbers does not include a formal quality review process, the beneficiary  estimates contain incorrect numbers and formulas and inconsistent  calculations. While the impact of these problems is generally small, the  lack of a robust, formal quality review process nonetheless calls into  question the reliability of MCC’s estimates. Without a correct  representation of the compacts’ results and beneficiaries, MCC,  Congress, and other key stakeholders cannot accurately evaluate the  extent to which MCC is achieving its goals of poverty reduction and  economic growth.
section_title: Recommendations for Executive Action
paragraphs: We recommend that MCC’s Chief Executive Officer take the following  four actions.
To improve the reliability of results measurement, MCC should  improve guidance for common indicators by requiring that they are  included in data quality reviews, measured uniformly across  compacts, and reported in a standardized manner—including when  third-party funding is used to complete the original scope of a project;  improve guidance for baseline measurements by requiring MCAs to  document the date, source, and methodology for establishing the  baseline; and enforce current monitoring and evaluation policy requiring MCAs to  conduct data quality reviews and to provide written comments in  response to identified issues, discussing how recommendations will  be implemented or explaining why changes may not be made.
To ensure more accurate beneficiary numbers, MCC should  incorporate into the Guidelines for Economic and Beneficiary Analysis  a formal process for reviewing beneficiary calculations and analysis.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, MCC stated that it agrees  with our four recommendations and outlined steps it will take or has taken  to address them.
With respect to our first recommendation—to improve guidance for  common indicators—MCC stated that it has taken preliminary actions  to address it. We reviewed MCC’s May 2012 Guidance on Common  Indicators and found that it defines the indicators and specifies each  indicator’s unit of measurement and how each should be  disaggregated. In addition, when MCC activities are conducted jointly  with other organizations (e.g. when the activities include third-party  funding), MCC’s guidance instructs MCAs to report only MCC’s  contribution to a particular common indicator. In addition, the  guidance states that MCC will be explicit in reporting which changes  in outcome indicators are more likely the result of MCC investments  and which changes might also be influenced by other interventions.  However, the guidance lacks enough specificity to address other  weaknesses we identified. For example, when discussing road  roughness, the guidance does not instruct MCAs to measure both  sides of the road to account for variation in roughness from one side  of the road to the other. In addition, this guidance does not specify  that MCAs should include common indicators in their data quality  reviews.
With respect to our second recommendation—to improve guidance for  baseline measurements—MCC stated that its current monitoring and  evaluation policy requires that compacts’ monitoring and evaluation  plans document the date, source, and methodology for establishing  baseline measurements. MCC further stated that current MCA  monitoring and evaluation plans are compliant with this requirement.
With respect to our third recommendation—to enforce current  monitoring and evaluation policy requiring MCAs to conduct data  quality reviews and to provide written comments in response to  identified issues—MCC stated that it works with MCAs to ensure that  recommendations of data quality reviews are incorporated into  monitoring and evaluation plans and then implemented.
With respect to our fourth recommendation—to incorporate a formal  process for reviewing beneficiary calculations and analysis—MCC  noted that it will institute a formal process for reviewing beneficiary  calculations and address the weaknesses we identified in this report.
MCC asserted that the results of its compact investments are not  diminished by the data quality challenges highlighted in this report. MCC  noted, for example, that more than 623 kilometers of rehabilitated roads  now serve poor and rural households in the countries we reviewed,  linking farmers to markets and bolstering important regional trade routes.  Regarding our finding that the total target for kilometers of road  completed for the six compacts was reduced by 63 percent, MCC also  noted that the final target for Honduras, Georgia, Cape Verde, and  Vanuatu would be only 19 percent below the original target if we were to  exclude the road projects that MCC terminated or discontinued as a result  of the Armenian and Nicaraguan governments’ undemocratic behavior.
We have reprinted MCC’s comments in appendix V. We have also  incorporated technical comments from MCC in our report where  appropriate.
We are sending copies of this report to interested congressional  committees and the Millennium Challenge Corporation. In addition, this  report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff has any questions about this report, please contact  David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The fiscal year 2008 Consolidated Appropriations Act, Public Law 110- 161, mandated that GAO review the results of the Millennium Challenge  Corporation’s (MCC) compacts. This report examines the extent to which  MCC has, for transportation infrastructure projects in compacts ending in  2010 and 2011, (1) achieved expected performance targets and (2) used  a consistent methodology in estimating numbers of beneficiaries. The  seven countries with compacts relevant to our scope are Honduras, Cape  Verde, Nicaragua, Georgia, Vanuatu, Armenia, and Benin.
MCC enters into a legal relationship with partner country governments,  which vests responsibility for day-to-day management of compact project  implementation with an accountable entity—a Millennium Challenge  Account (MCA)—including monitoring and evaluation activities such as  setting and revising targets. Because such MCA actions require MCC’s  direct oversight and approval, throughout this report we attribute all  decisions related to project rescoping and compact targets to MCC.
To assess the extent to which MCC has achieved its performance targets  for transportation infrastructure projects in compacts ending in 2010 and  2011, we reviewed MCC guidance and policy documents and analyzed  compacts, monitoring and evaluation plans, and indicator tracking tables  for the compacts with Honduras, Cape Verde, Nicaragua, Georgia,  Vanuatu, Armenia, and Benin. In assessing and reporting MCC’s results,  we compared actual results achieved at the end of the compact for select  performance indicators with the original and, in some cases, revised  targets associated with each indicator. We considered the original target  to be the one first documented for each performance indicator and the  final target to be the one last documented in MCC monitoring documents.
Given that MCC tracks several performance indicators for each compact  project, we selected a subset of indicators to examine for this report. For  the roads projects, we examined kilometers of road completed, road  roughness, and average annual daily traffic. We selected these road  project indicators because (1) they address the projects’ key objectives  and (2) they are among the “common indicators” that MCC requires all  road projects to measure so that it can aggregate results across  countries. For the port projects, we examined volume of merchandise  through the port, container ship time at berth, and container ship time at  anchor. Although MCC has not established common indicators for the  port projects, we selected these port project indicators because they (1)  addressed the projects’ key objectives, (2) corresponded to the common  indicators established for the roads projects, and (3) were originally  included in both port projects.
To report the average value of targets and results for compact road  projects, we calculated a weighted average using the length of each road  segment. We performed this calculation for the average annual daily  traffic and roughness indicators for Cape Verde, Honduras, Vanuatu, and  Nicaragua because these MCAs report targets and results by road  segment. For Honduras, we did not weight the indicators using values  from sections 1 and 2 because the government of Honduras completed  these sections with funding from the Central American Bank for Economic  Integration. For Cape Verde, we did not include the average annual daily  traffic and roughness for bridge projects, because the kilometer lengths of  these sections were not available.
To determine the amount of funding used for transportation infrastructure  projects, we reviewed MCC financial data. We included compact  implementation funding—funds disbursed before entry into force to  facilitate the implementation of the compact—with other projects not  related to transportation infrastructure.
To determine which MCAs contracted for independent data quality  reviews, which transportation infrastructure indicators were evaluated,  and whether recommendations were made for common and key  indicators, we reviewed third-party data quality review reports. To  determine whether recommendations were implemented, we reviewed  MCA comments on the data quality reviews that MCC provided to us and  we reviewed final monitoring and evaluation plans.
To assess the extent to which MCC has consistently applied its  beneficiary estimation methodology, we reviewed MCC’s beneficiary  calculations and accompanying supporting documents for the seven  compacts that ended in 2010 and 2011. In particular, we examined for  consistency and accuracy the data and formulas on projected  beneficiaries for road and port projects in the Excel spreadsheets that  MCC used to update—or “scrub”—the beneficiary estimates. We  reviewed the documents explaining the calculations. We compared  original beneficiary figures for these transportation projects with the  revised ones and assessed the justification behind the changes. We also  examined quarterly status reports and the monitoring and evaluation  plans for each compact to check the accuracy of the publicly reported  beneficiary counts. We consulted with MCC officials when we found  discrepancies and errors in the data.
As part of related GAO engagements focusing on Cape Verde and  Honduras and on Georgia and Benin, we visited compact projects in- country and met with MCA officials, partner country government officials,  contractors, project managers, construction supervisors, and relevant  private businesses.
Data are considered reliable when they are accurate, complete,  consistent and valid, given the uses for which they are intended. To  assess the reliability of MCC indicator and beneficiary data, we (1)  reviewed MCC policy for the monitoring and evaluation of compacts and  for calculating beneficiary numbers, (2) interviewed MCC and MCA  officials regarding the reliability and validity of the data, (3) reviewed  independent data quality reviews required by MCC, (4) compared  performance indicators across original and final monitoring and evaluation  plans and indicator tracking tables to identify inconsistencies, and (5)  reviewed beneficiary calculations for accuracy. We identified some  weaknesses in the data that may affect the data’s reliability, as discussed  in this report.
Finally, some of the reports and documents referenced above were  written in French and Spanish. We translated these documents as  needed, creating English summaries to enable our analysis.
We conducted this performance audit from August 2011 to September  2012 in accordance with generally accepted government auditing  standards.to obtain sufficient, appropriate evidence to provide a reasonable basis  for our findings and conclusions based on our audit objectives. We  believe the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Compact Timeframes
paragraphs: The Millennium Challenge Corporation (MCC) completed seven compacts  in 2010 and 2011—Honduras, Cape Verde, Nicaragua, Georgia, Benin,  Vanuatu, and Armenia (see fig. 3). MCC commits funding when it signs a  compact, but MCC does not obligate funding until the compact enters into  force. MCC funding that the partner country does not spend for compact  activities by the end of the 5-year implementation timeframe must be  deobligated.
section_title: Appendix III: Roads Data
paragraphs: MCC provided documentation of kilometers completed; however, the documentation did not include  takeover certificates from contractors, the data source for final kilometers in other countries.
subsections: section_title: Country Honduras
section_title: Country Vanuatu
paragraphs: N/C = not complete.
section_title: Country Honduras
paragraphs: CA-5 Highway: Section 1  CA-5 Highway: Section 2  CA-5 Highway: Sections 3 & 4  Road 1: Orgãos-Pedra Badejo  Road 2: Cruz Grande-Calhetona  Road: Vila das Pombas-Eito  Bridge Ribeira da Torre-Ponta de Sol  Bridge A Paul (Vila das Pombas)
N/C = not complete.
section_title: Appendix IV: Port Data
paragraphs: Appendix IV: Port Data N/C = not complete.
N/C = not complete  MCC eliminated performance indicators after descoping the Cape Verde port project, because the  indicators could not be measured prior to completion of the Phase II works, including the wharf 1,  container yard, and breakwater. Phase II was funded by the government of Portugal and was  expected to be completed after the compact closed.
N/C = not complete.
section_title: Appendix V: Comments from the Millennium Challenge Corporation
section_title: Appendix VI: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Emil Friberg, Jr. (Assistant  Director), Miriam Carroll Fenton, Aryn Ehlow, Heather Hampton, and  Seyda Wentworth made key contributions to this report. In addition,  Michael Armes, Lynn Cothern, Reid Lowe, Ernie Jackson, Leslie Locke,  and Jena Sinkfield provided technical assistance.
section_title: Related GAO Products
paragraphs: Millennium Challenge Corporation: Georgian and Benin Transportation  Infrastructure Projects Varied in Quality and May Not Be Sustainable.  GAO-12-630. Washington, D.C.: June 27, 2012.
Millennium Challenge Corporation: Compacts in Cape Verde and  Honduras Achieved Reduced Targets. GAO-11-728. Washington, D.C.:  July 25, 2011.
Millennium Challenge Corporation: Summary Fact Sheet for 17  Compacts. GAO-10-797R. Washington, D.C.: July 14, 2010.
Millennium Challenge Corporation: MCC Has Addressed a Number of  Implementation Challenges, but Needs to Improve Financial Controls and  Infrastructure Planning. GAO-10-52. Washington, D.C.:   November 6, 2009.
Millennium Challenge Corporation: Independent Reviews and Consistent  Approaches Will Strengthen Projections of Program Impact. GAO-08-730.  Washington, D.C.: June 17, 2008.
Management Letter: Recommendations for Improvements to MCC’s  Internal Controls and Policies on Premium Class Air Travel.   GAO-08-468R. Washington, D.C.: February 29, 2008.
Millennium Challenge Corporation: Projected Impact of Vanuatu Compact  Is Overstated. GAO-07-1122T. Washington, D.C.: July 26, 2007.
Millennium Challenge Corporation: Vanuatu Compact Overstates  Projected Program Impact. GAO-07-909. Washington, D.C.:   July 11, 2007.
Millennium Challenge Corporation: Progress and Challenges with  Compacts in Africa. GAO-07-1049T. Washington, D.C.: June 28, 2007.
Millennium Challenge Corporation: Compact Implementation Structures  Are Being Established; Framework for Measuring Results Needs  Improvement. GAO-06-805. Washington, D.C.: July 28, 2006.
Analysis of Future Millennium Challenge Corporation Obligations.   GAO-06-466R. Washington, D.C.: February 21, 2006.
Millennium Challenge Corporation: Progress Made on Key Challenges in  First Year of Operations. GAO-05-625T. Washington, D.C.:   April 27, 2005.
Millennium Challenge Corporation: Progress Made on Key Challenges in  First Year of Operations. GAO-05-455T. Washington, D.C.:   April 26, 2005.