Millennium Challenge Corporation: Results of Transportation Infrastructure Projects in Seven Countries
Released: 2012-09-12T00:00:00

Letter

Background

MCC Organization

MCC is managed by a chief executive officer, appointed by the President with the advice and consent of the Senate, and is overseen by a Board of Directors. The Secretary of State serves as board chair and the Secretary of the Treasury as vice-chair. MCC’s model of providing foreign aid is based on a set of core principles deemed essential for effective development assistance, including good governance, country ownership, focus on results, and transparency. According to MCC, country ownership of an MCC compact occurs when a country’s national government controls the prioritization process during compact development, is responsible for implementation, and is accountable to its domestic stakeholders for decision making and results. To implement the compact, the partner government establishes an accountable entity, referred to as a Millennium Challenge Account (MCA). MCC provides the framework and guidance for compact implementation, monitoring, and evaluation that MCAs are to use in implementing compact projects.

Transportation Infrastructure Projects

The seven MCC compacts completed in 2010 and 2011 each included a transportation infrastructure project that—except in the case of Armenia— received 50 percent or more of the compact’s total funding (see fig.1). For the purposes of this report, we have defined transportation infrastructure as public works that convey passengers or goods from one place to another; infrastructure includes structures such as roads, seaports, airports, and railways. We reviewed five compacts with road projects, one with a port project, and one with both types of projects.

According to MCC’s Policy for Monitoring and Evaluation of Compacts and Threshold Programs, performance monitoring helps track progress toward compact goals and objectives and serves as a management tool. Therefore, MCC requires partner countries to create a monitoring and evaluation plan that they update throughout the course of the compact. The plan’s monitoring component outlines the performance indicators by which the compact results will be measured. It also establishes a performance target for each indicator, including a baseline value (the starting value of an indicator) and the expected year by which the target will be achieved. For some indicators, baseline measurements are necessary to establish interim and final targets. MCC and the MCAs monitor the progress of compact activities using an indicator tracking table, which is a reporting tool that displays targets and tracks progress against them. (See fig. 2.)

MCC established common indicators in 2009 to aggregate results across countries. For road projects, MCC has established several common indicators, including the following:

Kilometers of road completed: A measure of the total length of road completed under a compact. This indicator is a cumulative measure of works completed.

Road roughness: A measure of pavement quality that affects ride quality, vehicle delay costs, fuel consumption, and maintenance costs. Roughness is used to determine the reduction in road-user costs and indicates when resurfacing work is needed. New pavement roughness ratings—as measured by the international roughness index (IRI)— should range from 1.5 to 3.5; the higher the IRI, the rougher the pavement, with a roughness of 0 being perfectly smooth.

Average annual daily traffic: A measure of the volume and type of traffic using a road.

MCC has not established common indicators for ports, because ports are a small part of its portfolio (it has funded port projects in only two countries). However, we selected certain key indicators to review because they addressed the projects’ key objective of economic growth:

Volume of merchandise through the port: A measure of the total volume of exports and imports passing through the port each year.

Container ship time at berth: A measure of the average number of days a container ship spends at berth to load and/or unload its cargo.

Container ship time at anchor: A measure of the average number of hours a container ship spends at anchor waiting for a berth.

MCC requires that MCAs contract an independent entity (such as a local or international firm, research organization, or individual consultant) to review the quality of compact performance data. These reviews determine the utility, objectivity, and integrity of information gathered, identify problems with data quality, and recommend actions to remedy these issues. Reviews should ensure that data used to measure indicators meet the following criteria: validity, reliability, timeliness, precision, and integrity. MCC policy allows each country to determine the frequency and timing of each review as well as the indicators to be evaluated. This requirement was fulfilled for every compact except Nicaragua. MCC policy requires the MCA to comment on the review, including noting the recommendations it will implement. In addition, the MCA should reflect, in a subsequent monitoring and evaluation plan, any indicator changes made in response to data quality reviews. MCC policy also requires the MCA to make publically available on its website a summary of the final reviews and its comments. MCAs are responsible for ensuring that MCC-approved recommendations are implemented.

Beneficiary Assessment

MCC defines beneficiaries as people who realize income gains or expenditure savings as a result of its investment. MCC’s authorizing legislation requires that compacts contain an identification of the intended beneficiaries, disaggregated by income level, gender, and age, to the maximum extent practicable. MCC notes that “the use of these analyses and their availability to the general public are trademarks of MCC’s commitment to transparency and results-based aid.” In addition, MCC’s 2009 Guidelines for Economic and Beneficiary Analysis maintain that identification of intended beneficiaries should not be limited to counting the number of people who benefit from compacts but should include a more detailed analysis of the effect of compacts on different subgroups, including the poor, the elderly, women, children, and ethnic subpopulations. MCC has not undertaken such detailed beneficiary analysis for the seven compacts completed in 2010 or 2011.

MCC Data Show that Some Key Performance Targets Were Met, but Data Quality Problems Call into Question Reported Results

MCC reduced the scope of early road and port projects and reports mixed success in meeting key performance targets. In addition, problems with data quality call those reported results into question.

MCC Reduced Targets for Completed Roads by 63 Percent and Had Data Quality Problems

In each of the six compacts with road projects, MCC reduced the number of kilometers to be completed. In addition, the roughness and annual average daily traffic indicators had data quality problems. Most compacts did not examine the three common road indicators in their data quality reviews. In addition, MCC did not consistently account for third-party funding.

MCC reduced by 63 percent the total number of road kilometers to be completed for the six compacts with road projects, from an original combined target of 1,822.8 kilometers to a final combined target of 623.9 kilometers. Five of six compacts met their reduced targets for kilometers completed. Honduras was the only compact that did not meet its target (see table 1).

In general, MCC decreased the scope of the road projects because of increased construction costs, changes in exchange rates, or actions taken by host country governments that were inconsistent with MCC’s eligibility criteria. For example, MCC decreased the scope of the compacts in Nicaragua and Armenia because of concerns about democratic governance in the respective partner governments (see table 2).

MCC data show that five of six compacts met original targets for road roughness. In addition, MCC reports that the Georgia and Armenia compacts surpassed their original roughness targets and built a smoother road than originally planned. Honduras did not did not meet its roughness target. (See table 3.)

We identified problems with the measurement of road roughness for some of the six compacts with road projects, including the use of inconsistent application of methodologies and calculation errors. These problems call into question MCC’s reported roughness results.

Inconsistent application of methodologies. MCC determined roughness using a visual scale for some roads and mechanical measurements for others. While both methods are legitimate means of measuring roughness, they have different degrees of accuracy. Moreover, MCC did not apply either measurement method consistently across compacts. First, for the two countries measuring roughness visually, Cape Verde and Vanuatu, MCC inappropriately provided the MCAs a World Bank scale for unpaved roads to assess the paved roads. Second, for compacts where roughness was measured mechanically, the MCAs used different machines that had varying levels of accuracy. Third, for three compacts—Nicaragua, Honduras, and Armenia—the MCAs measured roughness on both sides of the road and then averaged these measurements to calculate a final measure. In contrast, MCA-Georgia calculated roughness using measurements taken on one side of the road only, potentially producing a less accurate measurement since roughness can vary from one side of the road to the other.

Calculation errors. We found that errors calculating roughness led to an overstatement of results. For example, MCA-Nicaragua recorded zeros— an IRI rating for a perfectly smooth road—for sections that could not be measured, incorrectly lowering the overall roughness calculation from 4.8 to 3.4 for one road and indicating that the road was smoother than it actually was. differing lengths of each segment of secondary road when aggregating the roughness values and therefore overstated its reported results at 3.2 instead of 3.3. Although this difference is not large, consistent methodologies should be used across compacts to allow a valid basis for comparing and evaluating results.

For two of the road segments, the nonmeasurable sections of road were left blank and the calculation of the final IRI used the value from the one measurable side of the road as the overall measure for that kilometer.

MCC-Reported Results for Road Traffic Have Data Quality Issues

MCC data show that three of the six compacts with road projects met original or revised targets for average annual daily traffic. The Nicaragua and Armenia compacts met their original targets and the Vanuatu compact met a reduced target. For the other three compacts, Cape Verde did not meet its original target; Georgia did not meet its reduced target; and MCC stated that a final traffic measure in Honduras is not expected until late 2012. (See table 4.)

MCC did not consistently update the baseline estimates for the annual average daily traffic indicator either by projecting growth or by using new information when it became available. Because MCC establishes the targets for some indicators by applying estimated growth rates to the baselines, these identified weaknesses may have affected MCC’s ability to create accurate traffic targets, thus affecting its reported results.

For the secondary roads project in Nicaragua, MCC stated that it updated the baseline estimates established in 2007 by applying 1 year’s growth rate to the traffic measures, because construction was expected to begin in fall 2008. However, MCC did not use this method in Cape Verde, where there was a similar time lag.

For the section of highway to be rehabilitated in Nicaragua, MCC retained a baseline calculated in a 2000 traffic study had 2007 data for the specific 18 kilometers of road it planned to rehabilitate.

In Honduras, MCC used a baseline established before 2005 despite the availability of data from a 2008 traffic survey. The 2008 traffic survey results showed that traffic volumes already met or exceeded the end-of-compact targets for most road segments. Because MCC used the 2005 data, the final results for the project indicate a larger change over the compact implementation timeframe than actually occurred.

In addition, the 2000 traffic study was conducted for a 72-kilometer section of highway, rather than the 18 kilometers designated for the project. quality reviews for Armenia included two of the three indicators, the compact was put on hold before MCA-Armenia could fully correct identified problems. In Vanuatu, data quality reviews conducted in 2008 and 2009 identified problems with the measurement of annual average daily traffic. MCA-Vanuatu did not provide a response to either of the recommendations in the review and made no changes to the definition of the traffic indicator in the final monitoring and evaluation plan. The data quality reviews for the Honduras and Cape Verde compacts did not include any of the three road indicators. Table 5 summarizes the treatment of the three common road indicators we examined in data quality reviews for the six compacts.

The governments of three countries—Honduras, Cape Verde, and Vanuatu—secured funding from a third party to rehabilitate some of the road lengths that MCC removed from the scope of the compact. However, we found that MCC did not consistently account for this third- party funding when reporting the final results of kilometers completed. As a result, MCC understated its results in Honduras and overstated them in Vanuatu.

The government of Honduras secured money from the Central American Bank for Economic Integration to complete 57.6 kilometers of the highway project, which MCC removed from the compact’s original scope. However, MCA-Honduras did not revise its kilometers-completed target to reflect this reduction in scope. Therefore, MCC reports that it did not meet the compact’s final kilometers-completed target, even though data suggest that the kilometers MCC funded were complete at the end of the compact

The government of Cape Verde secured funding from the government of Portugal to rehabilitate 22.9 kilometers of secondary roads that MCC removed from the scope of its compact. MCA-Cape Verde revised its kilometers-completed target to reflect this reduction in scope and did not report these 22.9 kilometers as a part of its final results. Therefore, MCC reports that it met the compact’s revised kilometers-completed target.

The government of Vanuatu secured money from New Zealand to assist in rehabilitating a 57.2 kilometer road, because the MCC compact funds were not sufficient to complete all 57.2 kilometers. However, MCC reported all 57.2 kilometers as a part of its final results, even though it did not fund all of the work.

MCC Reduced the Cape Verde Port Project Scope and Did Not Meet Many Targets for the Benin Port Project

For the Cape Verde compact, MCC reduced the scope of the port project, deferred the measurement of key indicators, and therefore did not conduct any data quality reviews of those indicators. In Benin, MCC completed most of the originally envisioned port project scope, and its data show that the compact met the original target for one of the three key indicators. However, we identified issues with the indicators’ baseline estimates. In addition, data quality reviews identified problems with the data behind the indicators, which MCA-Benin did not formally address.

MCC split the Cape Verde project into two phases in May 2008 because, owing to increased construction costs, the $53.7 million allocated to the port projects was insufficient to fund all of the planned improvements. MCC funded phase 1 of the improvements, including construction of a cargo storage area and an access road, and rehabilitation of Wharf 2. The government of Cape Verde secured a loan from the government of Portugal for $87 million to complete phase 2, that is, those elements removed from the scope of the MCC project. The MCC-funded portion of the construction (phase 1) represents about one-third of the total expected cost of both phases. Nearly 100 percent of the works for phase 1 were completed by compact end. Phase 2 was awarded late in the compact, and work was under way as of July 2012.

MCC deferred the measurement of key performance indicators when it revised the project scope, because results for key indicators could be measured only after full completion of the port (see table 6). MCC intends to measure results for the key indicators when phase 2 is complete.

MCA-Cape Verde contracted for data quality reviews in 2007 and 2010. The port project was reviewed in 2010; however, key indicators were not evaluated, because MCA-Cape Verde deferred measurement of those indicators until phase 2 was complete.

MCC data show that Benin maintained most of the original project scope and met the original target for one of three key indicators. The project scope included constructing or rehabilitating a jetty, a new wharf, and port roads and installing lighting, security, and an electricity distribution system. MCC reports that the Benin compact met the original target for volume of merchandise through the port, but it did not meet targets for the other two key indicators, container ship waiting time at berth and container ship waiting time at anchor. See table 7 for a summary of key performance results for the Benin port project.

We identified problems with MCC’s measurement of the volume-of- merchandise indicators, and MCA-Benin’s data quality reviews identified additional problems with all three key indicators. For the volume-of- merchandise indicator in Benin, MCC used 2004 data from the Port Authority to establish a baseline of 4.1 million metric tons in the compact’s monitoring and evaluation plans and indicator tracking tables. MCA-Benin did not subsequently update the baseline. However, the compact did not enter into force until October 2006 and port improvement construction did not begin until August 2009. Available annual data for Benin showed that in 2005 the volume of merchandise had already increased to 5.2 million metric tons. Without an updated baseline, the estimate may not accurately reflect preconstruction conditions and may lead MCC to overstate the degree to which port traffic increased because of the compact.

MCA-Benin contracted data quality reviews in 2008, 2009, and 2011. The 2008 and 2009 data quality reviews identified issues with the three key port indicators that we reviewed, but MCC did not address all of the issues. The 2011 review did not examine these specific port indicators.

For the volume-of-merchandise indicator, MCA-contractors noted that although data collection methods were consistent and the data were valid, the Port Authority of Cotonou needed to improve the timeliness of data related to this indicator because the data entry and control procedures created delays in the availability of information. MCC policy indicates that data should be sufficiently current to inform management decisions because decisions depend on regular collection of performance information. In 2008 and 2009, the contractors recommended that MCA-Benin set up an automatic data transfer system for information collected on ship stopover, to avoid redundant data entry efforts by the port’s statistical services. Although the 2009 review indicated that the port was resolving problems through the creation of a port information system when the evaluation was conducted, we found that MCA-Benin did not resolve the issues identified for this indicator. No changes were made in the final monitoring and evaluation plan to indicate that adjustments were made in the data collection methods for this indicator. In addition, MCA-Benin responses to the review did not address this issue.

For the container ship time at berth and container ship time at anchor indicators, the 2008 and 2009 reviews noted that, while the data collected were a direct measure of what the indicators were intended to capture and the data collection procedures were consistent, there were no historical data to support the baseline values for these indicators. The reviews recommended using baseline values established in a 2005 report by an international consulting firm. We found that MCA-Benin did not resolve issues identified for these two indicators. MCC and MCA officials stated that they decided not to revise indicator baselines and targets as recommended, and no revisions were made to baseline figures in subsequent monitoring and evaluation plans. MCC and MCA officials provided no additional explanation.

Data Resulting from Beneficiary Calculations May Not Be Valid

In 2009, MCC improved its beneficiary estimates by adopting a more standardized approach, which it documented in its Guidelines for Economic and Beneficiary Analysis; however, the updated guidelines do not include a formal quality review process. MCC subsequently revised its beneficiary estimates for ongoing compacts, but the implementation of the new approach was not always consistent and suffered from varying degrees of quality control problems. Although some of the individual problems we identified were small in nature, taken as a whole they reduce confidence in MCC’s estimates.

MCC’s Updated Beneficiary Analysis Guidelines Do Not Include a Formal Quality Review Process

In our previous work, we found that MCC did not have a consistent methodology for estimating the number of beneficiaries across compacts. As a result, MCC adopted a new, more standardized approach, which it published in its 2009 Guidelines for Economic and Beneficiary Analysis. MCC subsequently conducted a “beneficiary scrub,” revising beneficiary numbers for ongoing compacts based on the 2009 guidelines. However, we found errors and inconsistencies in the calculations. MCC stated that these scrubs are informally reviewed but that they do not undergo a formal quality review process. Such a process would ensure that the information and data provided have been verified for consistency and accuracy.

Weaknesses Call into Question the Validity and Reliability of MCC’s Beneficiary Data

We identified three general weaknesses with MCC’s beneficiary calculations. First, the beneficiary calculations in MCC’s internal documents contain weaknesses such as incorrect formulas and numbers as well as discrepancies with supporting documents. Second, MCC did not apply a consistent methodology in estimating beneficiaries for its early transportation infrastructure projects. Third, the beneficiary figures in MCC’s public documents are sometimes inaccurate.

Some beneficiary calculations contained mistakes in the formulas used. For example, the computations of the population growth rate for Georgia and Cape Verde contained an error in the mathematical formula, causing MCC to slightly overstate the growth rate in the first case and understate it in the second case. In addition, erroneous numbers were used in four of the seven compacts we reviewed.

For the Georgia road project, MCC stated in the beneficiary calculation documents that “the original beneficiary estimate of 53,988 people counts all households living in the four rayons, or districts, through which the road passes…The revised beneficiary estimate [of 58,079] is based on a 5 km catchment area of the Road.” MCC thus originally claimed a lower number of beneficiaries for a larger geographic area (the districts) than the more limited catchment area— the geographic area in which benefits may be expected to accrue—of 5 kilometers on either side of the road.

For the Benin port project, the population projection in the beneficiary scrub began with a 2005 original baseline figure of 8,490,000, since the port project is expected to raise the incomes of the entire population of Benin. However, this number could not be derived from its purported source, the 2002 population census.

For the Armenia road project, the document supporting the beneficiary scrub calculation considered a catchment area for an erroneous revised road length of 68 kilometers rather than the correct figure of 24.4 kilometers.

In the Cape Verde roads and bridges activity, the original beneficiary numbers provided in the document supporting the beneficiary scrub spreadsheet did not match the figures in the spreadsheet.

In the 2009 revision of its Guidelines for Economic and Beneficiary Analysis, MCC standardized its methodology to estimate beneficiaries and accordingly modified its beneficiary numbers in the spreadsheets for its beneficiary scrubs. However, the amount of detail in the analysis varied across compacts, and population growth rates were not always calculated in a consistent manner.

Of the beneficiary scrubs we reviewed, four of seven contained disaggregated data at the project level. Specifically, the Armenia, Georgia, Nicaragua, and Vanuatu scrubs are made up of multiple, relatively detailed spreadsheets. By contrast, one beneficiary scrub, for Honduras, contained one page of macro-level summary data with no explanation or sources for the figures. The remaining two of the seven scrubs contained additional data but were not as detailed as the first four. For example, the Cape Verde scrub included a second sheet of World Bank development indicators on the country, and the Benin scrub had some details on one of the compact projects. These varying levels of detail suggest that the scrubs were executed in an ad hoc, rather than consistent, manner.

MCC uses population growth rates to estimate the number of beneficiaries in a 20-year projection period; for each compact, MCC publicly presents the number of beneficiaries projected for the end of the 20-year period. However, the scrub sheets we reviewed showed that MCC sometimes computed population growth rates differently across compacts, even when the scrubs provided the same underlying information.

For Honduras, MCC indicated that it took the population growth rates for 2005, 2006, and 2007 from World Bank data and averaged the three rates to get an average annual growth rate for the projection.

For Georgia and Cape Verde, MCC did not use the historical growth rates provided but instead took total population figures in 2005 and 2007 and used them to compute an average annual growth rate for the beneficiary projection.

For Vanuatu, MCC computed an average annual growth rate based on four historical growth rates without specifying a source or year.

In three of seven compacts, we found that some numbers in MCC’s public documents had not been updated or were inaccurate.

In the case of Benin and Nicaragua, the growth rate for the year 2000 was added to the formula. beneficiaries for the compact as a whole (143,000) was smaller than the beneficiaries of the infrastructure project alone (300,000).

For the Benin port project, the beneficiary estimate that appeared in a 2011 status report (8,791,832) was an original estimate from 2009 instead of the revised estimate of 16,119,058. In addition, the supporting document accompanying the 2009 scrub spreadsheet mistakenly reported 13,421,086 beneficiaries for the port project by 2026. However, the data in the scrub indicated that this figure referred to the number of beneficiaries in 2020; the correct number of projected beneficiaries for 2026 is 16,119,058.

For Armenia, the beneficiary estimate for the road project in a 2011 publication (6,216) did not reflect a revision of the estimate done in 2009 (6,356).

Conclusions

MCC recognizes the importance of a disciplined, transparent, and accountable approach to tracking compact results to make well-informed decisions about U.S. investments. It has established an extensive monitoring and evaluation program that includes guidance and tools for the partner countries and controls on data quality. In addition, MCC has created and implemented a revised beneficiary calculation methodology to ensure standardization across projects and compacts.

Even with these steps, problems exist with the data behind several of MCC’s key performance indicators, limiting the reliability of MCC’s reported results.

MCAs have not measured the results of common indicators in a uniform manner—including how they account for additional third-party funding—nor have they consistently reviewed the quality of data used to report the indicators’ results. As such, MCC cannot confidently compare or aggregate the results of these common indicators as intended.

MCC requires the establishment of baselines to measure the impact its assistance has on partner countries. However, because some incorrect baselines were used, results may be over- or understated.

MCAs have not consistently fulfilled the MCC requirements that they conduct independent data quality reviews and respond to issues the reviews raised. As a result, MCC and its partners have missed opportunities to improve data quality and facilitate data reporting and aggregation.

In addition, MCC’s beneficiary calculations have problems that limit their usefulness. Because MCC’s new approach to estimating beneficiary numbers does not include a formal quality review process, the beneficiary estimates contain incorrect numbers and formulas and inconsistent calculations. While the impact of these problems is generally small, the lack of a robust, formal quality review process nonetheless calls into question the reliability of MCC’s estimates. Without a correct representation of the compacts’ results and beneficiaries, MCC, Congress, and other key stakeholders cannot accurately evaluate the extent to which MCC is achieving its goals of poverty reduction and economic growth.

Recommendations for Executive Action

We recommend that MCC’s Chief Executive Officer take the following four actions.

To improve the reliability of results measurement, MCC should improve guidance for common indicators by requiring that they are included in data quality reviews, measured uniformly across compacts, and reported in a standardized manner—including when third-party funding is used to complete the original scope of a project; improve guidance for baseline measurements by requiring MCAs to document the date, source, and methodology for establishing the baseline; and enforce current monitoring and evaluation policy requiring MCAs to conduct data quality reviews and to provide written comments in response to identified issues, discussing how recommendations will be implemented or explaining why changes may not be made.

To ensure more accurate beneficiary numbers, MCC should incorporate into the Guidelines for Economic and Beneficiary Analysis a formal process for reviewing beneficiary calculations and analysis.

Agency Comments and Our Evaluation

In written comments on a draft of this report, MCC stated that it agrees with our four recommendations and outlined steps it will take or has taken to address them.

With respect to our first recommendation—to improve guidance for common indicators—MCC stated that it has taken preliminary actions to address it. We reviewed MCC’s May 2012 Guidance on Common Indicators and found that it defines the indicators and specifies each indicator’s unit of measurement and how each should be disaggregated. In addition, when MCC activities are conducted jointly with other organizations (e.g. when the activities include third-party funding), MCC’s guidance instructs MCAs to report only MCC’s contribution to a particular common indicator. In addition, the guidance states that MCC will be explicit in reporting which changes in outcome indicators are more likely the result of MCC investments and which changes might also be influenced by other interventions. However, the guidance lacks enough specificity to address other weaknesses we identified. For example, when discussing road roughness, the guidance does not instruct MCAs to measure both sides of the road to account for variation in roughness from one side of the road to the other. In addition, this guidance does not specify that MCAs should include common indicators in their data quality reviews.

With respect to our second recommendation—to improve guidance for baseline measurements—MCC stated that its current monitoring and evaluation policy requires that compacts’ monitoring and evaluation plans document the date, source, and methodology for establishing baseline measurements. MCC further stated that current MCA monitoring and evaluation plans are compliant with this requirement.

With respect to our third recommendation—to enforce current monitoring and evaluation policy requiring MCAs to conduct data quality reviews and to provide written comments in response to identified issues—MCC stated that it works with MCAs to ensure that recommendations of data quality reviews are incorporated into monitoring and evaluation plans and then implemented.

With respect to our fourth recommendation—to incorporate a formal process for reviewing beneficiary calculations and analysis—MCC noted that it will institute a formal process for reviewing beneficiary calculations and address the weaknesses we identified in this report.

MCC asserted that the results of its compact investments are not diminished by the data quality challenges highlighted in this report. MCC noted, for example, that more than 623 kilometers of rehabilitated roads now serve poor and rural households in the countries we reviewed, linking farmers to markets and bolstering important regional trade routes. Regarding our finding that the total target for kilometers of road completed for the six compacts was reduced by 63 percent, MCC also noted that the final target for Honduras, Georgia, Cape Verde, and Vanuatu would be only 19 percent below the original target if we were to exclude the road projects that MCC terminated or discontinued as a result of the Armenian and Nicaraguan governments’ undemocratic behavior.

We have reprinted MCC’s comments in appendix V. We have also incorporated technical comments from MCC in our report where appropriate.

We are sending copies of this report to interested congressional committees and the Millennium Challenge Corporation. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff has any questions about this report, please contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

The fiscal year 2008 Consolidated Appropriations Act, Public Law 110- 161, mandated that GAO review the results of the Millennium Challenge Corporation’s (MCC) compacts. This report examines the extent to which MCC has, for transportation infrastructure projects in compacts ending in 2010 and 2011, (1) achieved expected performance targets and (2) used a consistent methodology in estimating numbers of beneficiaries. The seven countries with compacts relevant to our scope are Honduras, Cape Verde, Nicaragua, Georgia, Vanuatu, Armenia, and Benin.

MCC enters into a legal relationship with partner country governments, which vests responsibility for day-to-day management of compact project implementation with an accountable entity—a Millennium Challenge Account (MCA)—including monitoring and evaluation activities such as setting and revising targets. Because such MCA actions require MCC’s direct oversight and approval, throughout this report we attribute all decisions related to project rescoping and compact targets to MCC.

To assess the extent to which MCC has achieved its performance targets for transportation infrastructure projects in compacts ending in 2010 and 2011, we reviewed MCC guidance and policy documents and analyzed compacts, monitoring and evaluation plans, and indicator tracking tables for the compacts with Honduras, Cape Verde, Nicaragua, Georgia, Vanuatu, Armenia, and Benin. In assessing and reporting MCC’s results, we compared actual results achieved at the end of the compact for select performance indicators with the original and, in some cases, revised targets associated with each indicator. We considered the original target to be the one first documented for each performance indicator and the final target to be the one last documented in MCC monitoring documents.

Given that MCC tracks several performance indicators for each compact project, we selected a subset of indicators to examine for this report. For the roads projects, we examined kilometers of road completed, road roughness, and average annual daily traffic. We selected these road project indicators because (1) they address the projects’ key objectives and (2) they are among the “common indicators” that MCC requires all road projects to measure so that it can aggregate results across countries. For the port projects, we examined volume of merchandise through the port, container ship time at berth, and container ship time at anchor. Although MCC has not established common indicators for the port projects, we selected these port project indicators because they (1) addressed the projects’ key objectives, (2) corresponded to the common indicators established for the roads projects, and (3) were originally included in both port projects.

To report the average value of targets and results for compact road projects, we calculated a weighted average using the length of each road segment. We performed this calculation for the average annual daily traffic and roughness indicators for Cape Verde, Honduras, Vanuatu, and Nicaragua because these MCAs report targets and results by road segment. For Honduras, we did not weight the indicators using values from sections 1 and 2 because the government of Honduras completed these sections with funding from the Central American Bank for Economic Integration. For Cape Verde, we did not include the average annual daily traffic and roughness for bridge projects, because the kilometer lengths of these sections were not available.

To determine the amount of funding used for transportation infrastructure projects, we reviewed MCC financial data. We included compact implementation funding—funds disbursed before entry into force to facilitate the implementation of the compact—with other projects not related to transportation infrastructure.

To determine which MCAs contracted for independent data quality reviews, which transportation infrastructure indicators were evaluated, and whether recommendations were made for common and key indicators, we reviewed third-party data quality review reports. To determine whether recommendations were implemented, we reviewed MCA comments on the data quality reviews that MCC provided to us and we reviewed final monitoring and evaluation plans.

To assess the extent to which MCC has consistently applied its beneficiary estimation methodology, we reviewed MCC’s beneficiary calculations and accompanying supporting documents for the seven compacts that ended in 2010 and 2011. In particular, we examined for consistency and accuracy the data and formulas on projected beneficiaries for road and port projects in the Excel spreadsheets that MCC used to update—or “scrub”—the beneficiary estimates. We reviewed the documents explaining the calculations. We compared original beneficiary figures for these transportation projects with the revised ones and assessed the justification behind the changes. We also examined quarterly status reports and the monitoring and evaluation plans for each compact to check the accuracy of the publicly reported beneficiary counts. We consulted with MCC officials when we found discrepancies and errors in the data.

As part of related GAO engagements focusing on Cape Verde and Honduras and on Georgia and Benin, we visited compact projects in- country and met with MCA officials, partner country government officials, contractors, project managers, construction supervisors, and relevant private businesses.

Data are considered reliable when they are accurate, complete, consistent and valid, given the uses for which they are intended. To assess the reliability of MCC indicator and beneficiary data, we (1) reviewed MCC policy for the monitoring and evaluation of compacts and for calculating beneficiary numbers, (2) interviewed MCC and MCA officials regarding the reliability and validity of the data, (3) reviewed independent data quality reviews required by MCC, (4) compared performance indicators across original and final monitoring and evaluation plans and indicator tracking tables to identify inconsistencies, and (5) reviewed beneficiary calculations for accuracy. We identified some weaknesses in the data that may affect the data’s reliability, as discussed in this report.

Finally, some of the reports and documents referenced above were written in French and Spanish. We translated these documents as needed, creating English summaries to enable our analysis.

We conducted this performance audit from August 2011 to September 2012 in accordance with generally accepted government auditing standards.to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Compact Timeframes

The Millennium Challenge Corporation (MCC) completed seven compacts in 2010 and 2011—Honduras, Cape Verde, Nicaragua, Georgia, Benin, Vanuatu, and Armenia (see fig. 3). MCC commits funding when it signs a compact, but MCC does not obligate funding until the compact enters into force. MCC funding that the partner country does not spend for compact activities by the end of the 5-year implementation timeframe must be deobligated.

Appendix III: Roads Data

MCC provided documentation of kilometers completed; however, the documentation did not include takeover certificates from contractors, the data source for final kilometers in other countries.

Country Honduras

Country Vanuatu

N/C = not complete.

Country Honduras

CA-5 Highway: Section 1 CA-5 Highway: Section 2 CA-5 Highway: Sections 3 & 4 Road 1: Orgãos-Pedra Badejo Road 2: Cruz Grande-Calhetona Road: Vila das Pombas-Eito Bridge Ribeira da Torre-Ponta de Sol Bridge A Paul (Vila das Pombas)

N/C = not complete.

Appendix IV: Port Data

Appendix IV: Port Data N/C = not complete.

N/C = not complete MCC eliminated performance indicators after descoping the Cape Verde port project, because the indicators could not be measured prior to completion of the Phase II works, including the wharf 1, container yard, and breakwater. Phase II was funded by the government of Portugal and was expected to be completed after the compact closed.

N/C = not complete.

Appendix V: Comments from the Millennium Challenge Corporation

Appendix VI: GAO Contacts and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Emil Friberg, Jr. (Assistant Director), Miriam Carroll Fenton, Aryn Ehlow, Heather Hampton, and Seyda Wentworth made key contributions to this report. In addition, Michael Armes, Lynn Cothern, Reid Lowe, Ernie Jackson, Leslie Locke, and Jena Sinkfield provided technical assistance.

Related GAO Products

Millennium Challenge Corporation: Georgian and Benin Transportation Infrastructure Projects Varied in Quality and May Not Be Sustainable. GAO-12-630. Washington, D.C.: June 27, 2012.

Millennium Challenge Corporation: Compacts in Cape Verde and Honduras Achieved Reduced Targets. GAO-11-728. Washington, D.C.: July 25, 2011.

Millennium Challenge Corporation: Summary Fact Sheet for 17 Compacts. GAO-10-797R. Washington, D.C.: July 14, 2010.

Millennium Challenge Corporation: MCC Has Addressed a Number of Implementation Challenges, but Needs to Improve Financial Controls and Infrastructure Planning. GAO-10-52. Washington, D.C.: November 6, 2009.

Millennium Challenge Corporation: Independent Reviews and Consistent Approaches Will Strengthen Projections of Program Impact. GAO-08-730. Washington, D.C.: June 17, 2008.

Management Letter: Recommendations for Improvements to MCC’s Internal Controls and Policies on Premium Class Air Travel. GAO-08-468R. Washington, D.C.: February 29, 2008.

Millennium Challenge Corporation: Projected Impact of Vanuatu Compact Is Overstated. GAO-07-1122T. Washington, D.C.: July 26, 2007.

Millennium Challenge Corporation: Vanuatu Compact Overstates Projected Program Impact. GAO-07-909. Washington, D.C.: July 11, 2007.

Millennium Challenge Corporation: Progress and Challenges with Compacts in Africa. GAO-07-1049T. Washington, D.C.: June 28, 2007.

Millennium Challenge Corporation: Compact Implementation Structures Are Being Established; Framework for Measuring Results Needs Improvement. GAO-06-805. Washington, D.C.: July 28, 2006.

Analysis of Future Millennium Challenge Corporation Obligations. GAO-06-466R. Washington, D.C.: February 21, 2006.

Millennium Challenge Corporation: Progress Made on Key Challenges in First Year of Operations. GAO-05-625T. Washington, D.C.: April 27, 2005.

Millennium Challenge Corporation: Progress Made on Key Challenges in First Year of Operations. GAO-05-455T. Washington, D.C.: April 26, 2005.