Millennium Challenge Corporation: Compact Implementation Structures Are Being Established; Framework for Measuring Results Needs Improvement
Released: 2006-07-28T00:00:00

Letter

Background

MCC Principles

In pursuit of its mission to reduce poverty by supporting economic growth, MCC has identified and defined the following three key principles to guide its actions: Reward good policy. “Using objective indicators, countries are selected to receive assistance based on their performance in governing justly, investing in their citizens, and encouraging economic freedom.”

Operate in partnership. “Countries that receive MCA assistance are responsible for identifying the greatest barriers to their own development, ensuring civil society participation, and developing a multi-year MCC compact.”

Focus on results. “MCA assistance goes to those countries that have developed well-designed programs with clear objectives, benchmarks to measure expected results, procedures to ensure fiscal accountability for the use of MCA assistance, and a plan for effective monitoring and objective evaluation of results. Programs are designed so that recipient countries can sustain progress after the funding under the compact has ended.”

MCC Structure

MCC is a government corporation that is managed by a CEO appointed by the President with the advice and consent of the Senate and is overseen by a Board of Directors (MCC Board). The Secretary of State serves as board chair, and the Secretary of the Treasury serves as vice-chair. Other board members are the U.S. Trade Representative, the Administrator of the U.S. Agency for International Development (USAID), the CEO of MCC, and up to four Senate-confirmed public members who are appointed by the President from lists of individuals submitted by congressional leadership.

Eligibility for MCC Assistance

The Millennium Challenge Act of 2003 requires MCC to select countries as eligible for MCA assistance each fiscal year. Countries with per capita income at or below a set threshold may be selected as eligible for assistance if they pass MCC indicator criteria and are not statutorily barred from receiving U.S. assistance. MCC uses 16 indicators divided into three categories: Ruling Justly, Encouraging Economic Freedom, and Investing in People. To be eligible for MCA assistance, countries must score above the median relative to their peers on at least half of the indicators in each category and above the median on the indicator for combating corruption. MCC used these quantitative indicators, as well as the discretion implicit in the Millennium Challenge Act, to select 17 countries as eligible to apply for MCA compact assistance for fiscal years 2004 and 2005. For fiscal year 2006, MCC identified 23 countries as eligible for assistance—the 17 previously selected and 6 additional countries, which included lower- middle-income countries eligible for the first time in fiscal year 2006.

MCC’s Compact Development

After MCC selects eligible countries, they may begin a four-phase process that can lead to the entry into force of compacts (see fig. 1). Each phase of this process is discussed after figure 1. 1. Country proposal development. Eligible countries are invited to submit compact proposals, which are to be developed in consultation with members of civil society, including the private sector and nongovernmental organizations (NGO). The eligible country also identifies an “accountable entity” to manage the programs funded by MCC. Eligible countries submitting proposals are not guaranteed funding; instead, MCC assesses proposals through its due diligence review. As of May 2006, 14 of the 17 countries selected as eligible in fiscal year 2004 or 2005, and 1 of the 6 countries selected as eligible for the first time in fiscal year 2006, had submitted proposals accepted by MCC for due diligence review. 2. MCC’s due diligence review. MCC determines whether the proposal that an eligible country has submitted meets MCC criteria to ensure that proposed programs will be effective and funds will be well-used. Due diligence primarily occurs between MCC’s acceptances of an “opportunity memo” and an “investment memo.” MCC assembles a transaction team of MCC staff, personnel from other U.S. agencies, and consultants to conduct a preliminary assessment of a country’s proposal and reports the team’s findings in an opportunity memo to the MCC investment committee. The MCC investment committee consists of MCC’s CEO, vice presidents, and other senior officials. If the opportunity memo is approved, the transaction team launches a detailed due diligence review. The team assesses the country proposal, reports its findings, and makes recommendations based on its assessment in an investment memo to the MCC investment committee. As of May 2006, MCC was conducting due diligence analyses of seven eligible country proposals. 3. Compact negotiation and MCC Board approval. MCC may enter into compact negotiations with the eligible country before the investment memo is completed. If compact negotiations are successful, MCC staff formally submit the compact for MCC Board approval. Once the board approves the compact, MCC and the eligible country may sign it. As of March 2006, MCC had signed compacts with 8 of the 17 countries determined eligible in fiscal years 2004 and 2005. (See fig. 2.) MCC commits the full amount of the compact funding at signing but obligates and begins to disburse funds to implement projects only after the compact has entered into force. Under the Millennium Challenge Act, compacts may remain in force no longer than 5 years. The compacts stipulate that, with limited exceptions, all funds must be spent during that time. 4. MCC and compact country complete entry-into-force requirements.

MCC’s compact with each country identifies the following supplemental agreements that MCC and the country’s accountable entity must complete before the compact can enter into force.

The disbursement agreement sets out the “conditions precedent” and other requirements for disbursements from MCC and redisbursements to any person or entity. These conditions include performance targets for projects outlined in the compact.

The procurement agreement sets forth guidelines for all procurements of goods, works, and services financed with MCC funding.

Compact term sheets for supplemental agreements, which vary by country, and include documents such as a governance agreement, fiscal agent agreement, form of implementing entity agreement, and form of bank agreement.

After compacts enter into force, MCC may begin the disbursement of funds and countries may begin implementing projects. In the first eight compacts, approximately 53 percent of funding went to transportation and other infrastructure projects; 22 percent went to agriculture and rural development; 13 percent went to other project types; and 12 percent went to program management, monitoring, and evaluation. (See fig. 2.)

Compact Signature and Entry-into-Force Time Frames

The length of time from country eligibility selection to compact signature has varied, with proposal development and due diligence generally requiring the most time (see fig. 3). For the six countries whose compacts had entered into force as of the end of May 2006, completing the steps necessary for entry into force after compact signing took approximately 3 to 4 months for Madagascar, Cape Verde, and Honduras; approximately 7 months for Georgia; 2 months for Vanuatu; and about 10½ months for Nicaragua. For the two countries whose compacts had not entered into force as of the end of May 2006, 3 months had elapsed since compact signature for Benin, and 2 months had elapsed for Armenia (see fig. 3).

MCC Guidance

MCC has issued guidance and policies for its compact development process in several stages. Before publishing its initial guidance in May 2004, MCC provided countries with preliminary guidance addressing fiscal accountability and monitoring and evaluation. Figure 4 shows the evolution of MCC’s published guidance relative to the end of the due diligence process with the investment memo.

MCC Assessed Five Key Aspects of Proposals While Developing Guidance, but Limitations Affected the Accuracy of Economic Analyses

MCC undertook a wide range of activities in its due diligence of the Madagascar, Cape Verde, and Honduras proposals, while at the same time developing guidance on key aspects of the countries’ proposals. During due diligence, MCC primarily considered criteria related to the proposals’ consultative process, project coherence, environmental impact, institutional and financial sustainability, and economic analyses. MCC generally approved proposals that were based on a consultative process and returned proposals that lacked adequate consultations; however, MCC did not publish detailed criteria for the consultative process until 1 year after selecting the countries as eligible for MCC assistance. In assessing project coherence, MCC approved projects that were linked to the overall proposal objectives and rejected projects that were not, although its assessments used criteria that it had not yet published in its guidance. Additionally, MCC screened projects for likely environmental impacts and considered factors important for institutional and financial sustainability. Finally, MCC conducted economic analyses to assess the projects’ likely impact on economic growth. However, limitations in assumptions and data may have affected the analyses’ accuracy and led MCC to select projects that would not achieve its goals. Also, a lack of country involvement in the analyses does not reflect the MCC principle of working in partnership with countries and may have limited the countries’ understanding of the process.

MCC Assessed the Consultative Process as Guidance Evolved

MCC’s due diligence for Madagascar, Cape Verde, and Honduras assessed whether the countries had consulted with public and private sector and civil society stakeholders during proposal development. MCC officials told us that before beginning due diligence for the three countries, they assessed the consultative process in proposal drafts and, on the basis of this review, returned proposals that were not sufficiently founded on a consultative process. For example, according to the officials, MCC did not accept one country’s proposal because, although the country had consulted with stakeholders, its proposal did not reflect the priorities identified during the consultative process.

MCC documents indicate that during its due diligence for two of the three countries, MCC obtained the views of government, civil society, and private sector officials on how the governments conducted the consultative process. In addition, MCC assessed any previous experience the country had had with a consultative process For all three countries, MCC also reviewed factors such as the date, frequency, and locations of the process. MCC’s documentation of these assessments in Madagascar, Cape Verde, and Honduras indicated that key stakeholders had generally agreed on the proposed compact priorities. However, the MCC assessments noted some weaknesses in the Madagascar and Honduras governments’ management of the process: In Madagascar, some groups expressed concern to MCC that the government had provided short notice (less than 10 days) for consultative meetings, and that this might have limited rural groups’ participation. Additionally, the government did not communicate to participants its rationale for accepting or rejecting projects. However, despite these shortcomings, MCC noted “widespread agreement and enthusiasm for the…primary components of proposal” among the business community, local and international NGOs, civil society, and donors.

In Honduras, MCC found, on the basis of discussions with local civil society organizations and international NGOs, that weaknesses—such as the large size of meetings—had limited effective participation in the consultative process for the poverty reduction strategy, which formed the basis of Honduras’s MCC proposal. MCC’s assessment also indicated that the government had not directly asked the consulted groups to identify obstacles to growth. However, MCC found that these groups concurred with the priorities identified in the country proposal. According to MCC officials, they followed up with the Honduran government to address the weakness noted by MCC in Honduras’s consultative process. As a result, according to MCC documentation, the Honduran government conducted additional consultative sessions with civil society organizations and donors.

MCC’s due diligence of the countries also noted requirements for additional consultations during project implementation. For example, in some cases, countries were to undertake consultations with local stakeholders to identify project sites and conduct environmental assessments.

Our discussions with representatives of civil society groups and donors in Madagascar and Cape Verde indicated that they generally concurred with the compact proposals. In Madagascar, a representative from a key civil society organization noted weaknesses in the government’s conduct of the consultative process similar to those recorded by MCC. Nevertheless, stakeholders on the Madagascar advisory council, which includes various civil society and other organizations, said that the compact proposal generally accounted for their views, especially in comparison with other donor programs.

As a relatively new organization, MCC conducted due diligence reviews while it was developing consultative process guidance: Evolving guidance. While developing their proposals, the countries had access to general criteria in MCC’s 2004 guidance; however, in assessing the proposals, MCC applied the more specific criteria contained in its detailed 2005 guidance. As figure 5 shows, the 2005 guidance was issued 1 year after MCC announced the eligibility of Madagascar, Honduras, and Cape Verde. According to MCC’s Guidance for Developing Proposals for MCA Assistance in FY 2004, “…each proposal is expected to reflect the results of an open consultative process, integrating governmental interests with those of the private sector and civil society.” The 2004 guidance required that proposals include a description of the consultative process, such as how the proposal takes into account local-level perspectives of the country’s rural and urban poor, including women, and of private and voluntary organizations and the business community. Additionally, the guidance required the country to list all key participants, such as government and nongovernmental officials, who played a significant role in developing the proposal.

MCC’s 2005 Guidance on the Consultative Process more specifically requires eligible country governments to involve their citizens in identifying obstacles to economic growth and developing and prioritizing the development strategies and programs that will be included in the compact proposal. The 2005 guidance further states that an adequate consultative process should be timely, participatory, and meaningful. MCC’s guidance also took into account the country’s experience in using a consultative process to develop other national or poverty reduction strategies. If the compact was built on these consultations, MCC required some additional consultations to provide justification of country priorities in the MCA proposal.

Incomplete documentation. MCC’s documentation of its due diligence for the three countries presents summary findings, rather than an analysis of the extent to which the countries consulted with the rural and urban poor. For example, MCC’s due diligence documentation indicates that the countries’ governments included women’s groups and rural sector groups in their consultative process; however, the documentation does not indicate the extent to which these groups represented the poor. Additionally, in Madagascar and Honduras, MCC’s documentation does not indicate how MCC assessed the extent to which the consulted groups informed compact proposal priorities.

MCC Assessed Project Coherence While Developing Specific Guidance

In keeping with MCC’s emphasis on results, due diligence for the three countries also assessed whether proposed projects were linked to one or more of the country’s compact goals, and whether the projects addressed key impediments or constraints to achieving these goals. On the basis of its due diligence assessment, MCC rejected projects that were not linked to key constraints in two of the three countries we reviewed. Specifically, MCC rejected tourism and preschool education projects proposed by Honduras because they were not linked to the impediments to growth that emerged from the consultative process. MCC also rejected projects to construct feeder roads, which connect some watershed areas to the markets, and provide access to electricity in the rural areas of Cape Verde, because MCC’s due diligence did not indicate that these projects addressed key constraints in Cape Verde.

In reviewing MCC’s due diligence for the three countries, we found that MCC did not issue guidance stating that proposed projects should be linked to the compact goal until after it had concluded its due diligence assessments. MCC’s 2004 guidelines for proposal development broadly instruct eligible countries to identify priority areas, such as health or education, and their expected goals for each priority area over the term of the proposed compact. The guidance also asks the countries to show how these strategic goals are related to the economic growth and poverty reduction of the country. MCC’s November 2005 MCC Compact Assessment and Approval Guidelines more specifically indicates that MCC will assess how the project addresses compact goals. However, MCC issued the November 2005 guidance after completing its due diligence for the three countries. (See fig. 6.)

MCC Screened Projects for Environmental and Social Impact, Conditioning Funding on Detailed Assessment

MCC’s due diligence for the three countries included a review of the probable environmental and social impact of projects that met its economic analysis and other criteria. For projects that it deemed likely to cause adverse environmental and social impact, MCC required impact assessments or environmental analyses, including an impact management plan. required. For example, MCC assigned all projects in the Madagascar proposal to category C, because MCC determined that these projects were not likely to have adverse environmental and social impact. In contrast, MCC assigned infrastructure projects in Cape Verde’s and Honduras’s proposals to category A or B. For example, the highway expansion project in Honduras was assigned category A, because it involves the clearing of rights-of-way that will require compensation for more than 200 affected people. The port expansion project in the Cape Verde proposal was also assigned to category A, because it entails dredging and construction in and around an existing port.

Additionally, MCC assessed whether environmental impact assessments had been conducted for category A or B projects. For projects lacking environmental impact assessments, MCC conditioned project funding on the completion of such assessments as well as on the development of mitigation plans in consultation with affected groups. For projects that other organizations had assessed for environmental impact, MCC used a U.S. agency or a contractor to evaluate the assessment and determine its adequacy. For example, in Honduras, another donor had already conducted the environmental impact assessment for the highway segments proposed for MCC funding. As part of MCC due diligence, the U.S. Army Corps of Engineers reviewed these assessments and made recommendations that were incorporated in the existing assessments. However, in Cape Verde, the MCC contractor conducting due diligence found another organization’s assessment of the port project’s environmental impact to be inadequate. As a result, MCC required a new environmental assessment, along with plans to manage adverse impact, as a precondition for funding the project. MCC has allocated funds for this analysis in the compact budget for Cape Verde.

MCC’s 2004 proposal development guidelines do not address projects’ environmental and social impact. In assessing environmental impact, MCC applied criteria from the Millennium Challenge Act of 2003, which prohibits MCC from funding projects that are “likely to cause a significant environmental, health, or safety hazard.” MCC also used criteria laid out in its March 2005 interim Environmental Guidelines, which state that MCC will not fund projects that lack the appropriate screening or analysis for environmental impact.The guidance also states that the country has primary responsibility for conducting and monitoring environmental assessments. (See fig. 7.)

MCC Evaluated Country Commitment and Capacity and Estimated Project Impact to Assess Sustainability

In keeping with its emphasis on sustainable progress, MCC’s due diligence examined whether the three countries’ proposed projects could be sustained after their compacts expired. In assessing project sustainability, MCC reviewed each country’s policy and regulatory environment and commitment to reforms and financing of future maintenance costs; it also reviewed expected results from MCC-funded projects. In addition, MCC considered the countries’ institutional capacity to sustain proposed projects as well as other donors’ roles in strengthening countries’ capacity. institute such reforms. In the case of Madagascar, MCC reviewed policy reforms in land management and financial sectors that would benefit MCC-funded activities.

USAID- nd MCC-fnded business center re colocted in Antirabe, Masr. The USAID-fnded progrsupportrmers’ ccess to mrket nd provide linkge to lrge business. The MCC-fnded progrm will focus on technicassnce to frmer nd mke them creditworthy. Thee frmereek help from the USAID project lter. The USAID project i expected to re it exiting client lit with nd introdce thee client to the MCC progrm. Althogh USAID nd MCC project re conidered complementry, there i perception tht MCC i replcing USAID. Thi perception i trengthened y the fct tht USAID i loing abt one-third of it employee in Masr, ccording to U.S. offici with whom we poke. had limited institutional capacity to achieve the project objectives of increasing land security and promoting financial intermediation to increase rural savings and extension of credit. To build the government’s capacity, MCC budgeted funding for (1) staff recruitment and training at Madagascar’s land management department to support the land security project and (2) finance, management, and production training for rural producers and microfinance institutions to support the financial intermediation project. In addition, MCC considered the role of other donors in strengthening the countries’ institutional capacity. For example, while assessing road projects, MCC considered the World Bank’s road sector initiative, which includes institutional capacity- building in Cape Verde. In Honduras, MCC considered World Bank- funded and Inter-American Development Bank–funded programs intended to develop the transportation ministry’s management capacity and maintenance contracting capacity.

Project impact. MCC also relied on the assumptions used in its analysis of projects’ economic impact to determine the sustainability of agricultural sector projects for all three countries we reviewed. For example, MCC expects that as a result of MCC-funded technical assistance or credit to farmers and rural entrepreneurs, recipients will be able to generate enough income to afford these services by paying fees to providers. In cases where sustainability depends on achieving MCC’s projected impact, the soundness of MCC’s economic analysis, discussed in the next section of this report, will also be an important factor.

In its sustainability assessments, MCC generally adhered to guidance issued in 2005, rather than to guidance from 2004. MCC’s May 2004 proposal development guidance included a general requirement for a strategy to sustain progress after the compact’s expiration. MCC’s November 2005 MCC Compact Assessment and Approval Guidelines did not require such a strategy, but the guidelines required identification of factors contributing to institutional and financial sustainability for each project. (See fig. 8.)

Limitations in Assumptions and Data, as Well as Country Involvement, Affected the Accuracy of Economic Analyses

During its due diligence reviews for Madagascar, Cape Verde, and Honduras, MCC analyzed proposed projects’ probable impact on the country’s economic growth and poverty reduction. This analysis was intended both to assess whether these projects would achieve MCC’s goals and to provide a basis for monitoring their progress and evaluating their impact.

To predict each project’s impact on economic growth, MCC calculated an economic rate of return (ERR) —that is, the expected annual average return to the country’s firms, individuals, or sectors for each dollar that MCC spends on the project. In calculating projects’ ERRs, MCC used an economic model that includes the following elements (see fig. 9): MCC’s annual expenditures for the project, the project’s annual benefits to the country, predicted net benefits of the project, and the projected ERR.

ERRs for the proposed projects in Madagascar, Cape Verde, and Honduras varied considerably, ranging from 116 percent for Madagascar’s land titling project to 10 percent for a watershed management and agriculture support project in Cape Verde; the median ERR for projects in the three countries was 14 percent. However, because MCC analyzed the Madagascar and Honduras proposals before publishing its first set of economic analysis guidelines, the ERRs did not significantly affect those countries’ overall choice of projects. (See fig. 10.) MCC’s finding of a low ERR for part of Cape Verde’s watershed management and agriculture support project resulted in the country’s dropping an irrigation activity on one island.

We found limitations in the assumptions and data that MCC used in its analyses as well as in the countries’ involvement in the analyses. These limitations may negatively affect the accuracy of the analyses and the countries’ understanding of the analysis process, respectively.

Assumptions and data. Some of the assumptions and data that MCC used in its analyses do not fully reflect the countries’ socioeconomic environment. As a result, MCC cannot be assured that the projects it approved, partly on the basis of these analyses, will achieve the compacts’ goals. For example, in calculating the ERR for the Madagascar land titling project, MCC assumed that local small farmers would use newly titled land as collateral for loans, invest the borrowed funds in agricultural activities, and benefit from the increased income from those activities We discussed this assumption with focus group participants in Madagascar, including MCA and U.S. government officials, senior Madagascar government officials representing ministries affected by the compact, and bank representatives. Our discussions suggested that MCC’s analysis may have been overly optimistic in assuming that small farmers would mortgage 40 percent of their newly titled land in Madagascar’s uncertain market, which does not offer insurance. While MCC assumed that 40 percent of the newly titled land will be collateralized, focus group participants believed this was overly optimistic. As a result, MCC may have calculated an unrealistically high ERR for the land titling project. Furthermore, the project may be more likely to benefit farmers with large, secure landholdings and investors from outside the farming community, rather than the local small farmers it was intended to help.

Similarly, MCC’s economic model for a part of Madagascar’s finance project, the modernization of the National Savings Institution, may not accurately reflect the institution’s financial condition. MCC’s model for the institution’s modernization uses the institution’s April 2003 net profits (1.4 billion Malagasy francs) as a baseline for estimating the benefits of computerizing the bank. However, according to institution officials, 2003 is not a representative year for the bank, and it experienced unusually large net losses (13 billion Malagasy francs) after an economic crisis in 2002. As a result, MCC may have inaccurately estimated the project’s likely impact on the banking system.

Country involvement. In the two countries we visited, country representatives were not closely involved in MCC’s economic analyses of the proposed projects. This constrained the contribution of the analysis process to enhancing country partnership, and stakeholders’ understanding of MCC’s economic analysis—including the data analyzed, assumptions used in the analysis, and the expected outcomes. According to country officials in Madagascar and Cape Verde, MCC developed the economic models and selected data with little assistance from country representatives. According to the Cape Verde proposal team, they provided some data but did not participate in the actual analysis and did not have a clear understanding of the process and the criteria MCC used to assess proposed projects. We discussed these issues with MCC officials, who told us that the countries’ degree of involvement depended on the capability and willingness of the countries’ proposal development team to actively participate in the analyses.

MCC-Approved Compact Implementation Structures Are Not Yet Complete; Weaknesses in Monitoring and Evaluation Framework May Limit Measurement of Results

MCC gave Madagascar, Cape Verde, and Honduras authority to propose and develop the implementation structures that they will use to manage compacts, although MCC retains authority over a number of management decisions. To govern their programs, the countries have created management units under the direction of a steering committee, but they have had difficulty in filling key positions. The countries also have established structures for ensuring fiscal accountability and for managing procurements that appear to be effective; however, implementation is still at a very early stage, and some required elements of these structures are not yet in place. Finally, the countries have established frameworks for monitoring and evaluating the performance of MCC projects. However, the frameworks have weaknesses related to the inadequacy of baseline data, linkage of monitoring plans with economic models, methods of addressing uncertainty in achieving stated targets, and the timeliness of research designs for randomized controlled trials. These weaknesses may limit MCC’s ability to track and account for program results.

MCC-Approved Country Procurement Systems Have Effective Characteristics, but Are Early in Their Implementation

Consistent with MCC guidance, each of the three countries—Madagascar, Cape Verde, and Honduras—proposed its own procurement system, subject to MCC review and approval during due diligence. Although the approved systems have characteristics that we have found typical of effective procurement systems, their effectiveness has not yet been tested by many procurements. In addition, some of the staff and procedures needed to implement the systems are not yet in place at MCC headquarters or in compact countries.

MCC Allowed Countries to Propose Procurement Agents and Systems, but Retained Approval Rights

To capitalize on existing country knowledge and experience, MCC has given the countries flexibility to choose their procurement agents and standards, with the choice subject to MCC approval during due diligence. Madagascar and Cape Verde are using private and public sector procurement agents, respectively, and Honduras is using a combination of public and private sector agents. (See app. IV.) In contrast to donors such as the World Bank, MCC also allowed the countries to propose their own standards for managing procurements. (See app. IV for details of the countries’ procurement agents and standards.) MCC required the countries to adhere to “procurement principles” that include equal access to procurements, competition for awards, and transparency of the process. Madagascar, Cape Verde, and Honduras have used either existing World Bank standards or their own laws to govern MCC procurements. In its procurement agreement with each country, MCC included modifications to the country’s selected standards to reconcile them with U.S. law and MCC principles. For example, MCC required that countries not include preferences for domestic suppliers in solicitations paid for with MCC funds.

Although MCC’s agreements with the three countries give the countries a number of authorities over procurements, MCC retains certain approval rights. MCC’s fiscal accountability framework describes procurement as one of the highest-risk areas of fiscal accountability. MCC’s compact, disbursement agreement, and procurement agreement with each of the countries describe the relationship, roles, and authority of MCC, the procurement agent, and the compact country (see app. IV for details). Although these agreements have some common elements, each agreement is unique to the individual country. To determine when MCC approval of individual actions is appropriate, MCC included review thresholds in the procurement agreements keyed to procurement size and methods. Above cost thresholds, which vary among countries, MCC must approve items such as the procurement method, terms of reference, and selection. Below thresholds, the compact country may conduct procurements in keeping with the procurement plan without additional MCC oversight. These thresholds are a risk management tool that maximizes MCC control for larger transactions but leaves discretion to the compact country for smaller transactions.

Countries’ Procurement Systems Have Strengths, but MCC and Country Systems Are Not Fully Established

Although the three countries’ procurement systems vary, each has characteristics that we have previously identified as typical of effective international procurement systems. These characteristics are similar to the principles, such as equal access, competition, and transparency, that MCC applies in its review of the systems during due diligence. However, MCA officials in Madagascar and Cape Verde told us that they had completed few procurements that would test the systems in practice. Furthermore, MCC headquarters has not yet finished hiring its procurement staff, and procurement systems in Madagascar and Cape Verde are not yet fully established.

Incomplete staffing. As of May 2006, MCC headquarters had hired its senior director of procurement but had not yet hired five director-level procurement staff in its Accountability Department. According to MCC officials, the function of these vacant positions is currently being filled by six intermittent personal service contractors, and MCC has offered one of these contractors a full-time position. During our site visits in January and February 2006, staff in Madagascar and Cape Verde reported that the time frames for MCC review of procurements had been satisfactory. However, if MCC staffing does not increase as the countries submit more procurement decisions for approval, MCC may have difficulty in conducting timely reviews of these decisions.

Incomplete systems. During our site visits, we found that some elements of the procurement systems documented in MCC’s agreements with Madagascar and Cape Verde were not yet in place or had not functioned smoothly.

Neither country had yet established the procurement bid protest body required by MCC or put in place automated systems for procurement tracking and management, although both reported plans to do so in 2006. Madagascar also had not established a process for reviewing contractor change order requests.

During our visit to Madagascar, a senior MCC official discovered that the country’s procurement plan for months 2 through 4 did not reflect current project work plans. As a result, the procurement agent was preparing for procurements that were no longer needed. Members of the Madagascar management team told us that they would establish new procedures to coordinate future work plan changes with the procurement agent.

In Cape Verde, the procurement review commission lacked office space and was concerned about being able to handle the number of reviews required of it once procurements begin in earnest. The commission is comprised of members who are fully employed elsewhere and are not permitted to delegate their work. The members have worked nights and weekends or have negotiated with their supervisors to be released from their other jobs to perform the commission’s work. According to MCC, subsequent to our site visit, Cape Verde has taken actions to mitigate the risks to the efficient operation of the commission.

Both Madagascar and Cape Verde reported difficulties with an MCC requirement that documents be in both English and the local language. Cape Verdean procurement review commission members are not required to speak English but are expected to review documents prepared in English. Cape Verde was considering hiring a full-time translator to address this need. In Madagascar, the translation of a French document prepared by the implementer of the finance project and requested by MCC delayed the approval of needed project procurement.

MCC Has Established Country Monitoring and Evaluation Frameworks, but Weaknesses May Limit Measurement of Results

Each of the three countries’ (Madagascar, Cape Verde, and Honduras) programs includes a monitoring and evaluation framework that includes plans for data collection, data quality reviews, analysis, and interim and final reporting of results. We found several weaknesses in the monitoring and evaluation frameworks that could affect MCC’s ability to track and account for program results.

Countries’ Programs Include Monitoring and Evaluation Frameworks

Each of the three countries’ programs includes detailed plans for monitoring and evaluating program results. MCC approved Madagascar’s and Cape Verde’s plans in November 2005 and April 2006, respectively, while Honduras’s plan existed in April as a detailed draft but had not yet been approved. According to MCC officials, Honduras’s plan is not final pending the staffing of the monitoring and evaluation director position to ensure country understanding and buy-in. In accordance with MCC guidance, the countries’ plans include separate components for monitoring and evaluation: The monitoring component includes, among other things, key indicators that are linked as closely as possible to the variables identified in the economic analysis of the country’s proposed projects. These indicators are to be used throughout implementation to assess whether the program is likely to achieve the desired results. The monitoring component also identifies baseline and target values for each indicator and includes plans for periodic performance reports and data quality reviews. MCC guidelines state that monitoring can be on select indicators to minimize reporting requirements.

The evaluation component identifies, among other things, the methodology that will be used to assess the program’s impact, such as randomized controlled trials, and describes plans for collecting baseline, interim, and final data on program results.

Countries’ monitoring and evaluation plans use, to varying degrees, the economic analysis of the proposed projects to identify indicators for monitoring progress toward project objectives, calculate targets for each indicator, and evaluate the achievement of compact goals. The economic relationships specified in the models, such as the relationship between improved infrastructure and farm output, provide a basis for tracking project success. The monitoring framework also includes setting target values for indicators. Figure 13 illustrates indicators at various levels for a rural development project in Honduras.

Although the specifics of the countries’ plans vary, the monitoring component of each plan calls for the country’s MCA to submit periodic performance reports and data quality assessments to MCC. Performance reports will include quarterly assessments to alert the countries and MCC to any problems, periodic audits that analyze performance at all compact levels, and annual reports that consolidate the quarterly reports and recommend adjustments. Most performance data will be gathered by project implementers, either country government employees or contractors; the plans allow for contracting with other entities to prepare the reports. In addition, each plan calls for third-party data quality reviews over the course of the compact. For example, in Madagascar, data quality assessments are planned to occur every 6 months during the first year and annually thereafter.

Weaknesses in Monitoring and Evaluation Frameworks May Limit MCC’s Ability to Measure Results

In reviewing the frameworks for monitoring and evaluation in the three countries, we identified several challenges that MCC faces in ensuring accountability for results. These challenges include (1) ensuring the availability and quality of baseline data, (2) establishing clear links between the economic model and the monitoring and evaluation framework, (3) accounting for the degree of uncertainty in expected outcomes, and (4) using randomized controlled trials in compact countries.

Ensuring Baseline Data Availability and Quality

Baseline data are essential to measuring the results of MCC-funded compact projects. Although MCC has taken steps to collect baseline data for monitoring and evaluation, problems with data availability and quality may lead to challenges in measuring the progress and impact of the countries’ projects over time. MCC officials told us that they worked with their country counterparts to set up a Management Information System that can meet the requirements for collecting performance data. In addition, MCC evaluated the technical capabilities of the country staff and the information system the country proposes to use for data management purposes. Finally, MCC budgeted funding for surveys in Madagascar and Cape Verde to collect baseline data when it was not available. However, we found that some of the baseline data in the countries’ monitoring plans were not complete, and that some of the data MCC collected were not reliable.

Baseline data availability. In some instances, the countries’ monitoring and evaluation plans lack complete baseline data against which to measure progress. For example, two activity and final project indicators in Madagascar’s plan—“volume of production covered by warehouse receipts in zones” and “volume of microfinance institution lending in the zones”—currently lack baselines because the intervention zones have not yet been selected. Moreover, although the collection of performance data is closely linked to project implementation, Madagascar’s plan contains no intermediate outcome indicators and target values, thereby making it difficult to effectively track project progress. (See fig. 13 for an example of the role of intermediate outcome indicators in the monitoring structure of Honduras.)

Baseline data quality. MCC may face challenges in ensuring the quality of the baseline data that it uses to monitor and evaluate program impact and, as a result, may have difficulty in accurately measuring the impact of compact projects. MCC officials told us that it has been difficult to obtain accurate and reliable baseline data against which to measure program results. In some countries, MCC has funded surveys to obtain the needed baseline data. However, even with the additional resources provided by MCC, obtaining baseline data has been a challenge. For example, we found significant data quality problems associated with one of three surveys that MCC funded to collect baseline data in Madagascar. Our interviews with Madagascar and USAID officials who oversaw the survey revealed that the survey results, which were used to estimate average land values, are flawed in that they do not reflect recent significant changes in Madagascar’s currency. Madagascar’s compact goal is to increase household income, as measured by the percentage of increase in land values. Because of the survey error, the land value estimates may not be sufficiently reliable to evaluate the project impact and the compact as a whole.

Linking Monitoring Plans to Economic Analyses

Linking the indicators used to monitor and evaluate progress to the data and assumptions used in MCC’s due diligence economic analyses will also present a challenge. In reviewing the draft plan for Honduras, we found consistent linkages between the indicators for monitoring and evaluation and the variables and assumptions used in the economic model. However, in the plans for Madagascar and Cape Verde—the first two plans that MCC approved—we found instances where MCC did not sufficiently link the monitoring plans to the economic models, which may hamper its ability to effectively measure project results. For example: After signing its compact with Cape Verde, MCC changed the interim targets for seven indicators. In two cases, neither MCC nor Cape Verde was able to identify the methodology used to select the indicators in the monitoring and evaluation plan. According to MCC officials, they decided that the assumptions in the economic analysis were a poor basis for constructing the monitoring indicators and, therefore, chose other indicators and estimated the targets. According to MCC officials, the inability to identify the methodology, in conjunction with updated baselines and revised work plans in the country resulted in MCC’s and Cape Verde’s agreeing to reduce the interim targets that had been established.

MCC’s economic analysis for Madagascar’s Land Tenure Project, which is approximately one-third of the compact budget, did not identify the expected benefits from the separate project activities. Therefore, we could not track the linkage between the activities in the model to those in the monitoring and evaluation plan.

MCC’s approved monitoring and evaluation plan does not include tracking the results of two finance project activities—modernizing the National Savings Institution and opening bank branches—although they were included in economic analysis calculations during due diligence. According to an MCC official, these two finance project activities will be tracked at a higher level of aggregation—the finance project level—and monitored by tracking the number and value of new accounts. However, this approach may not adequately capture the outputs and benefits from the institution’s modernization and could potentially confound the effect of one activity (modernizing the institution) with that of other activities. For example, while an increase in the number and value of new accounts could result from the two finance project activities, it could also result from an overall increase in savings if customers invest in government savings bonds, issued recently and prior to the start of the MCC compact.

Accounting for Uncertainty in Achieving Target Values

Although the countries’ monitoring and evaluation plans acknowledge the uncertainty of achieving indicator target values, MCC project monitoring does not adequately address (1) the effect of potential variations in uncertainty on the range of acceptable target values or (2) the plausibility of target values. As a result, some targets specified in the countries’ monitoring plans may not be achieved.

MCC disbursement agreements include as a condition precedent that, if an indicator value observed during compact implementation does not fall within 10 percent of the agreed-on target values, MCC may withhold disbursements. MCC applies this 10 percent margin to all projects, regardless of type (e.g., agriculture or infrastructure). However, our analysis suggests that several factors could cause indicator values for many projects to fall outside the 10 percent range.

External factors. Uncertainty associated with external factors varies by country and project. For example, according to previous GAO work, external factors that could affect project implementation might include political instability, the lack of commitment of political leaders to necessary reforms, the magnitude of assistance from other bilateral and multilateral donors, weather conditions that affect crop yields, and the instability of international markets. These factors could cause indicator values to fall outside the 10 percent range used across all countries and projects.

Time factor. The 10 percent range also does not account for the increase in the uncertainty of targets over time—for example, target values for year 5 of a compact are likely to be less precise than those for year 1.

Therefore, to the extent that MCC bases its disbursement decisions on results falling within a common range, it may not fully account for variations in uncertainty across projects and over time.

According to MCC guidelines, the economic analyses and monitoring and evaluation plans should, as much as possible, be clearly linked. However, limitations in (1) the economic analyses due to problems with data quality, assumptions, and lack of country involvement and (2) the consistency between the economic analyses and the monitoring and evaluation plans constrain MCC’s ability to set plausible targets. If targets are overly optimistic, countries may fail to reach them and MCC may not be justified in halting disbursements because the countries failed due to unattainable targets. Conversely, setting too conservative a target may not prompt the country to fully utilize MCC resources. A lack of plausible targets may lead to MCC’s making ad hoc decisions regarding the consequences of missing targets and applying judgment subjectively and inconsistently in setting or modifying targets. MCC officials told us that they would use a missed target as a cue to discuss with the country’s MCA its reasons for missing the target, and, on the basis of those discussions, they would determine whether to use their authority to withhold funding. According to MCC officials, senior management approval would be needed to significantly modify targets; however, MCC currently has no policy or documentation that defines a “significant modification” of targets, especially for targets used as conditions precedent to disbursements.

Using Randomized Controlled Trials in Compact Countries

MCC has retained research organizations to help the countries evaluate program impact using, as appropriate, randomized controlled trials, but MCC’s involvement of these organizations after project implementation begins may limit their ability to evaluate impact accurately. These organizations’ scope of work may include training MCC and compact country staff; designing the trials and data collection; and proposing appropriate methodologies for, and analyzing results from, impact evaluations. According to MCC monitoring and evaluation officials, MCC has begun designing impact evaluations by identifying those program components that can and cannot be evaluated using randomized controlled trials, which MCC has indicated are its preferred method of impact evaluation. According to MCC officials, MCC considers evaluations using randomized controlled trials as “rigorous” and evaluations using other methodologies as “standard.”

MCC has hired an independent consultant experienced in impact evaluations to work with compact countries to assess the appropriateness of using randomized trials to evaluate MCC’s projects. When these assessments are completed, the five research organizations will be invited to compete to conduct randomized trials after compact implementation begins. However, at that point, the organizations will not have had an opportunity to assess the design of the countries’ evaluation strategy, including the adequacy and reliability of the baseline data. Without the involvement of these organizations before implementation of the relevant project(s) begins, MCC may not be able to ensure that they have the necessary data and have established appropriate research designs for their work. MCC officials told us that they thus far had not involved the five organizations because rigorous evaluations were turning out not to be feasible in some cases, or because the tasks were not large enough to warrant the use of the five research organizations.

Conclusions

MCC continues to mature and evolve as an institution, taking on the ambitious task of creating new, country-managed organizations while developing processes to oversee what are expected to be relatively large amounts of foreign assistance. Toward that end, MCC has taken positive steps with regard to establishing policies and procedures for MCA organizations. However, it has taken time to complete the numerous agreements necessary for compacts to enter into force. This could continue to present challenges, given that MCC is working simultaneously with a number of nations to develop and implement compacts.

As MCC moves forward, partnering with countries to develop well-founded economic assumptions will be crucial to establishing a foundation for the work of MCC and its partners. Furthermore, holding countries accountable for results requires, to the extent practical and cost-effective: collecting reliable and accurate baseline data, linking economic analyses to monitoring plans, addressing the uncertainty associated with program results, and ensuring the timely development of the research design for randomized controlled trials.

Recommendations

Because of the central role of reliable economic analyses and the importance of partnering with countries in achieving MCC goals and ensuring accountability for MCC programs, we recommend that the Chief Executive Officer of MCC take the following two steps: Ensure that MCC officials, in partnership with country representatives, perform economic analyses that more fully reflect the countries’ socioeconomic environment and are better understood by country public and private sector representatives.

To the extent practical and cost-effective, improve MCC’s monitoring obtaining more accurate and reliable baseline data needed to permit tracking progress during compact implementation; ensuring a clear linkage between MCC’s economic analyses and monitoring and evaluation frameworks; developing policies, procedures, and criteria for establishing targets and for adjusting those targets if unforeseen events affect outcomes; and taking steps to ensure the timely development of the needed research design for randomized controlled trials, if they are undertaken, prior to project implementation.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from MCC and the Department of State. In commenting on a draft of this report, MCC generally agreed with our findings, conclusions, and recommendations. MCC noted that our discussion of the evolving guidance provided to eligible countries in 2004 and 2005 was a result of the complex process of engaging with eligible countries while simultaneously developing policies and procedures. MCC stated that this criticism should not be valid beyond the initial instances covered in this report. We recognize that MCC was simultaneously addressing a number of issues during this period, but we felt it was important to discuss the evolving nature of guidance to provide a balanced perspective regarding the process eligible countries had to follow to sign compacts.

MCC also commented (1) that our characterization of data quality issues in Madagascar cited only a single survey and (2) that it differed with us on the appropriate level of aggregation in linking economic models and monitoring and evaluation plans. We note that there was one other instance of poor data quality in Madagascar, but that we focused on the Agricultural Productivity Survey because of the importance of accurately tracking land values to monitor results. We agree that disaggregation may not always be feasible, but note that aggregation poses some challenges that could limit the effectiveness of monitoring and evaluation. We reprinted MCC’s comments, with our responses, in appendix V. We also incorporated technical comments from MCC in our report where appropriate.

State commented that some of our findings reflect minor or transitory problems and provided specific observations regarding MCC’s evolving guidance, the assumptions used in MCC’s economic models, country participation, staffing delays, fiscal accountability structures, and the use of randomized controlled trials.

State noted that MCC’s guidance could be expected to evolve, given the newness of the organization, and that informal guidance from MCC was always available to eligible countries. We agree that guidance could be expected to evolve, but we sought to provide a balanced perspective by noting instances where MCC’s verbal guidance may not have been sufficient to assist countries in submitting proposals that met MCC’s criteria.

State questioned the findings from our Madagascar focus groups and our finding that MCC conducted economic analyses with limited country involvement. Our focus groups resulted in discussions of the assumptions used in the Madagascar economic analysis with a broad range of country stakeholders representing U.S. and Malagasy agencies and organizations involved in implementing the compact. Furthermore, MCC agreed in its comments that, in some cases, the level of country engagement on economic analysis could be improved. MCC outlined specific steps that it had taken to increase country involvement in economic analyses.

Regarding staffing delays and fiscal accountability structures, State commented that we offered no suggestions to MCC for things they could have done differently. We added additional material to the report to elaborate on the steps MCC has taken to reduce delays in staffing key positions. In regard to fiscal accountability, we agree that different countries will differ in their maturity of internal control. However, evaluating maturity is key to properly assessing risk and establishing effective oversight mechanisms.

Finally, State disagrees with what it terms our “reliance” on randomized controlled trials to measure success. This comment misconstrues our findings. We did not rely on or advocate this methodology, but rather are commenting on MCC’s use of randomized controlled trials as its preferred method of impact evaluation.

We have reprinted State’s comments, with our responses, in appendix VI.

We are sending copies of this report to interested congressional committees as well as the Secretary of State, the Secretary of the Treasury, the CEO of MCC, and the Administrator of USAID. We will also make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff has any questions about this report, please contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII.

MCC Actions in Response to April 2005 GAO Recommendations

The Millennium Challenge Corporation (MCC) has taken a number of steps to address our April 2005 recommendations regarding its strategic planning, internal controls, and human capital and governance policies, although some aspects of its organizational structure are not yet complete. MCC has prepared a strategic plan for fiscal years 2006 through 2011, and an annual performance plan for fiscal year 2006. MCC also has strengthened its internal controls and taken steps to implement some fiscal accountability mechanisms, established an audit committee within its governing board, and completed several required audits and reviews. However, MCC has not documented all financial control activities and continues to face risks from the poor interfacing of its systems with those of the Department of the Interior’s National Business Center (NBC). MCC has reassessed its staffing model, developed a plan for recruitment, and implemented an improved performance management system; but, it does not systematically track the use of staff resources to verify its human capital model. Finally, MCC has approved a corporate governance policy and taken steps to improve board involvement in planning, management, and communication; but it has not yet fully addressed risk management for the corporation.

MCC Has Developed a Strategic Plan to Enhance Accountability and Completed a 2006 Performance Plan

Consistent with our recommendation to enhance corporate accountability, MCC completed a strategic plan, approved by the Office of Management and Budget (OMB) in November 2005. In April 2006, MCC completed an annual performance plan that provides goals and benchmarks for assessing its performance in fiscal year 2006. This annual performance plan will enable MCC to report in the future on its progress in meeting its goals. As part of the annual performance plan process, MCC also has developed goals and benchmarks for its individual departments that support the attainment of the corporate goals identified in the strategic plan. (See table 4.)

MCC Has Made Significant Progress in Establishing Internal Controls

In response to our April 2005 recommendations, MCC has made significant progress in establishing internal controls over program and administrative operations at both the MCC headquarters and compact levels. MCC has made progress in each of the five components of internal controls discussed in our April 2005 testimony: (1) control environment, (2) risk assessment, (3) control activities, (4) monitoring, and (5) reporting.

Internal control environment. MCC has initiated several measures to establish a positive internal control environment, including documenting its organizational structure for the Administration and Finance and Accountability structures. The Fiscal Accountability area, which is a component of the Accountability structure, currently consists of a managing director and several directors; however, other positions to support the fiscal oversight of MCC compacts have yet to be filled. A formal ethics program has also been established for MCC headquarters.

Risk assessment. MCC is developing a process for assessing risks facing the corporation and its programs. To this end, MCC has hired a third- party consulting firm to support the implementation of processes, based on OMB A-123, Management’s Responsibility for Internal Control, December 2004, criteria, to address risks associated with domestic and foreign operations.

Control activities. MCC has instituted several control activities to reduce risk. MCC has submitted its Strategic Plan under the Government Performance and Results Act (GPRA) of 1993 and has completed required audits and reviews, such as those required under the Federal Managers Financial Integrity Act (FMFIA) of 1982 and the Federal Information Security Management Act (FISMA) of 2002. MCC is in the process of addressing the various material weaknesses, reportable conditions, findings, and recommendations identified in the audits. For example, MCC continues to address the formal documentation of control activities for the financial reporting process at MCC headquarters. MCC also contracted with a third-party service- provider, NBC, to maintain its accounting system and the recommended Statement on Auditing Standards No. 70 (SAS 70) review was performed; however, MCC is still addressing issues reported as a result of the review, such as the need to address manual processes that present inherent risks associated with accounting systems and related processes. For example, MCC forwards daily to NBC a package containing source documentation that is used by NBC to record transactions to the general ledger. Similarly, information for MCC’s travel expenses is prepared at MCC headquarters office and daily communicated to NBC through manual processes. MCC is aware of the need to address these issues and is working with NBC to mitigate or eliminate the manual processes.

Monitoring. MCC has taken steps to ensure ongoing monitoring and periodic testing of control activities. MCC’s investment committee embodies functions of monitoring and testing and operating as an integral part of MCC’s internal control program by overseeing the program units’ compliance with both the procedural and substantive elements required by its internal processes. Also, MCC conducted its first comprehensive survey of internal controls, performed by outside consultants, in conjunction with its annual audit. In addition, MCC has formed formal review panels to monitor the progress of addressing findings from internal and external reviews. For example, consistent with OMB guidance, MCC formed an FMFIA Management Review Panel to assess the results of the internal control survey along with the findings of MCC’s independent financial auditors. Similarly, MCC implemented a specific procedure to address recommendations from reviews and audits performed by its Inspector General (IG).

Reporting. MCC has made progress in establishing a process for assessing and reporting on the operating effectiveness of its internal controls. MCC established a formal board-level audit committee whose responsibilities include (1) financial controls; (2) the integrity of the reporting process; and (3) performance of the independent audit process. In addition, MCC’s FMFIA Management Review Panel assessed the results of the internal control survey, along with the findings of MCC’s independent financial auditors. The panel identified four material weaknesses and actions that MCC will be taking in future months to resolve them, which the acting Chief Executive Officer certified on November 7, 2005.

MCC officials told us that the development of new internal control policies and procedures and the revision of those already in place is a continuing process as MCC continues to mature as an organization. Table 5 summarizes MCC’s progress in addressing our April 2005 recommendations.

MCC Has Taken Steps to Develop Human Capital, but Does Not Track Allocation of Human Capital to Key Activities

MCC has taken steps and is continuing to further develop its human capital systems by (1) assessing its staffing needs; (2) improving its recruitment, development, and retention systems; and (3) implementing a performance management system linking compensation to individual contributions toward corporate goals. However, despite having plans to increase its staff by an additional 38 percent between May and September 2006, MCC does not systematically assess its staffing needs, has not developed a human capital plan, and has not yet fully implemented its improved performance management system, as follows: Staffing. Although MCC has completed an assessment of its human capital needs since our April 2005 recommendation, it does not systematically track the use of staff time on an ongoing basis. MCC’s updated assessment of its human capital needs shows that it plans to increase its staffing from 218 staff, as of May 2006, to approximately 300 staff, as of September 2006. MCC also has created an organization chart that includes the specific approved positions for each department under the new 300-person staffing model, and it is hiring for many of these positions using limited-term appointments to provide greater flexibility in filling future needs. According to MCC officials, MCC made its case to OMB for increasing staffing to 300 persons on the basis of an analysis of MCC staffs’ recollection of the amount of time they spent developing the Georgia compact, which was thought to have been the most complex compact development process to date. However, MCC officials felt that this analysis may not have fully captured the amount of time spent by some departments in developing the compact. Retrospective analysis was necessary because MCC does not track employees’ time on mission-related projects on an ongoing basis. Without such data, MCC management is not able to systematically assess the staffing requirements needed to carry out MCC’s mission and consistently align its human capital with its changing needs.

Recruitment and retention. MCC has identified priorities and committed resources for recruitment and is developing a human capital plan to address retention and training. To support its effort to hire approximately 82 additional staff between May and September 2006, MCC has retained an outside consultant firm to work on-site and help with recruitment and has identified positions as first- or second-tier hiring priorities. MCC officials also told us that they are developing an overall human capital plan that will include planned activities and a time frame for identifying critical skills and competencies for MCC’s key positions. The officials stated that the human capital plan also will include a strategy for staff retention and will address staff training. Currently, MCC has developed procedures for providing employees with outside training. MCC intends to develop a comprehensive training plan following the completion of the human capital plan, a draft of which it expects to circulate to MCC senior staff for their review and comment by September 30, 2006.

Performance management. In keeping with our recommendation, MCC has established a performance-based compensation framework. MCC provided us with documentation of its employee ratings process, showing that employee expectations and performance reviews were keyed to organizational goals. However, according to MCC officials, MCC did not incorporate the departmental performance plans for the year into the performance framework for 2006 until March 2006, as the annual performance plan neared completion. MCC is shifting from a calendar year to a fiscal year performance evaluation schedule to better align employee compensation with its annual corporate goals. MCC anticipates that its strategic plan, annual performance plan, department plans, and individual performance goals will be fully synchronized beginning in fiscal year 2007. (See table 6.)

MCC Has Taken Steps to Define Corporate Governance

The MCC Board of Directors (MCC Board) has taken steps to define the scope of its corporate governance and oversight. The MCC Board has approved a corporate governance policy developed by MCC with the involvement of staff from MCC Board agencies. According to MCC officials, the policy incorporates guidance on governance matters provided by board members at their previous meetings, and the board participated in formulating MCC’s strategic plan before approving it in December 2005. The board has established a board-level audit committee and a charter for that committee. According to MCC officials, to address risk, MCC is recruiting for the position of risk specialist and has used a contractor to support MCC risk analysis. Finally, to improve communication with stakeholders in eligible countries, MCC has published and distributed updated guidelines for compact development and eligibility. MCC also has developed a series of open forums where input is sought from groups with an interest in MCC. (See table 7.)

Scope and Methodology

At the request of the Senate Committee on Foreign Relations, we examined the structures and procedures MCC has developed in consultation with compact countries to manage compacts. Specifically, our work focused on (1) the key areas that MCC examined in its due diligence assessments of proposals for Madagascar, Cape Verde, and Honduras, and the criteria that MCC used in these assessments, and (2) the form and adequacy of the implementation structures that MCC and compact countries have put in place for governance, procurement, fiscal accountability, and monitoring and evaluation. In addition, we reviewed MCC’s progress in responding to our April 2005 recommendations on its corporate management and accountability structures (see app. I).

To accomplish our objectives, we reviewed MCC’s documentation of its processes and agreements, supplemented by interviews with MCC officials. We focused our review for objectives 1 and 2 primarily on the first three countries with signed compacts—Madagascar, Cape Verde, and Honduras. These countries’ compacts were the first to enter into force. To further our analysis for objectives 1 and 2, we also visited Cape Verde and Madagascar in January and February 2006. We selected Cape Verde and Madagascar for our site visits because they had advanced further than Honduras in filling key positions and beginning compact implementation. While in Cape Verde and Madagascar, we interviewed a number of MCC, Millennium Challenge Account (MCA), and government officials and visited project sites.

To identify MCC’s evaluation criteria and process for evaluating eligible country proposals in due diligence, we reviewed MCC guidance and the record of MCC analysis contained in (1) MCC’s “due diligence books,” which are its internal records of how it assessed proposals submitted by Madagascar, Cape Verde, and Honduras, and (2) investment memos, which are MCC’s analyses based on due diligence and internal recommendations to its investment committee. These documents are restricted from public dissemination due to their sensitive nature, but MCC made them available to us for analysis. We have coordinated with MCC on describing the information from these books in general terms without disclosing sensitive information. We used MCC’s definition of the due diligence process as beginning with MCC’s opportunity memo and ending with the acceptance of the investment memo by MCC’s investment committee. Our review, therefore, may not capture some changes and decisions made by MCC or eligible countries during proposal development and compact negotiations. To evaluate MCC’s assessments of proposals’ consultative process, project coherence, environmental and social impact, and institutional and financial sustainability, we relied primarily on MCC’s data and analysis contained in the due diligence books and, to some extent, in the investment memos. We compared MCC’s analysis in these documents with criteria outlined in MCC’s guidance. We were able to perform only limited independent verification of the use and adequacy of these criteria during our site visits. With regard to its economic analyses, MCC also made available to us the spreadsheet models it used to develop the economic rate of return calculations that formed the basis for its evaluations of the suitability of country-proposed projects for MCC funding. We independently analyzed these spreadsheets and validated their logic and conclusions on the basis of a review of economic literature and practices. In addition, in Madagascar, we conducted a series of focus groups with country officials to assess the data and logic used by MCC in developing their economic analysis.

To assess MCC’s compact implementation structures, we reviewed the compacts with Madagascar, Cape Verde, and Honduras and the supplemental agreements required for those compacts to enter into force. We supplemented this review with our site visits to Madagascar and Cape Verde. In all cases, our ability to analyze the adequacy of these structures was limited by their relative newness and limited use in actual implementation. We addressed the following four areas of MCC’s implementation structures: To determine the form of governance structures and key positions in these three countries, we reviewed the requirements of MCC compacts and supplemental agreements. We determined the progress of the country organizations in Madagascar, Cape Verde, and Honduras in filling these positions and establishing these structures by analyzing MCC’s reported staffing and status information. We independently assessed MCC’s progress in our site visits to Madagascar and Cape Verde. We discussed factors affecting the filling of these positions through discussions with MCC and compact country officials.

To assess the adequacy of the countries’ fiscal accountability structures, we reviewed MCC’s overall fiscal accountability framework and the operations in place in Madagascar and Cape Verde. We assessed the adequacy of these structures according to the criteria contained in GAO’s Standards for Internal Control in the Federal Government. We assessed MCC’s and the countries’ implementation of these structures by using criteria in the Internal Controls Capability Maturity Continuum developed by the independent risk consulting company, Protiviti, Inc. We independently verified the existence of the structures described in the plan and discussed its strengths and weaknesses during the site visits.

To assess MCC’s procurement structures, we reviewed the MCC fiscal accountability framework, and the implementing procurement documents for the first three compact countries. We then assessed the adequacy of MCC’s framework, using criteria identified in previous GAO reports on international procurement. To determine the status and factors affecting the implementation of this framework in Cape Verde and Madagascar, we interviewed compact country officials and obtained documentation of procurement procedures.

To determine the form of monitoring and evaluation structures in the three countries with entry into force, we reviewed the requirements of MCC compacts and supplemental agreements. We assessed the status of the country organizations in Madagascar, Cape Verde, and Honduras to establish these structures by analyzing the staffing and status information provided to us by MCC. We independently assessed MCC’s progress in our site visits to Madagascar and Cape Verde. Additionally, we reviewed the scope of work of the independent U.S. evaluation contractors retained by MCC, and closely reviewed the monitoring and evaluation plan for Madagascar—the only plan approved by MCC prior to April 2006. We also reviewed the plan for Cape Verde and the draft plan for Honduras. We assessed the adequacy of the Madagascar plan against the criteria of data quality, and consistency with the economic model and logic identified in MCC’s due diligence review of projects. We also applied general principles of economic logic, such as treatment of uncertainty in data, to assess how uncertainty was incorporated into MCC’s monitoring and evaluation framework.

To review MCC’s progress in responding to GAO’s April 2005 recommendations (see app. I), we examined MCC documents, such as its strategic plan, planning documents, policies, procedures, and human capital documents. In December 2005, MCC provided us with a letter outlining the steps that the corporation had taken in response to our recommendations. Using this as a basis for discussion, we held additional meetings with MCC officials and received additional documentation of MCC’s responses. We also reviewed the findings of the MCC IG analysis of the functions of the Corporation and met with the IG to determine the steps that MCC had taken in response to IG findings related to our recommendations.

We conducted our review from June 2005 through May 2006 in accordance with generally accepted government auditing standards.

Comments from the Millennium Challenge Corporation

The following are GAO’s comments on the Millennium Challenge Corporation letter dated July 7, 2006.

GAO Comments

1. MCC stated that “MCC's mandate to engage directly with eligible countries shortly after its creation did not allow for the possibility of developing and vetting policies and procedures in advance of such engagement.” We recognize that MCC was simultaneously addressing a number of issues during its initial years of operation—selecting eligible countries, setting up MCC as an organization, and developing guidance and policies while also working with countries to finalize compacts. In the context of MCC as an evolving organization, we felt it was important to discuss the evolving nature of guidance to provide a balanced perspective regarding the process that eligible countries had to follow to sign compacts. 2. MCC stated that our concern regarding baseline data quality stems from one survey in Madagascar. However, we found issues with data reliability in Madagascar with both the Agricultural Productivity Survey and Household Survey. In the second survey, the error was smaller; therefore, we focused on the Agricultural Productivity Survey to illustrate the more significant example.

The measurement of land values is very important for Madagascar's monitoring and evaluation plan. Both the compact-level goal of increasing household income and the program objective of increasing investment in rural Madagascar are measured in terms of land values. In each of the zones, the expected increase in household income is estimated at 5 percent of the average land value, and the expected increase in investment is estimated at 27 percent of the average land value. With a large error in baseline data, it will be difficult to accurately track progress toward the compact-level goal and program objective. 3. MCC noted that not every benefit can be isolated and measured during implementation, and that disaggregating may not be feasible. We recognize that not all outcomes of the economic analysis can be directly tracked with indicators at different levels of monitoring or for impact evaluation. There are trade-offs between cost and level of detail. However, as we note in this report, aggregation poses some challenges that could limit the effectiveness of monitoring and evaluation.

Comments from the Department of State

The following are GAO’s comments on the Department of State letter dated July 10, 2006.

GAO Comments

1.

In referring to our discussion about MCC’s guidance on project coherence, State Department commented that it would be erroneous to indicate that countries were wholly unaware of the need to design their compact programs to target compact goals, and that informal guidance from MCC could have been received at any time. We agree that countries could have requested informal guidance from MCC. According to MCC, they did provide verbal guidance to countries regarding the need to link projects to compact goals.

However, MCC rejected tourism and preschool education projects proposed by Honduras because they were not linked to the impediments to growth that emerged from the consultative process. MCC also rejected projects in Cape Verde because MCC’s due diligence did not indicate that these projects addressed key constraints in Cape Verde. Given that these two countries submitted projects that did not meet project coherence criteria, MCC’s verbal guidance may not have been sufficient to direct countries to submit proposals meeting these criteria.

We recognize that MCC was simultaneously addressing a number of issues during its initial years of operation—selecting eligible countries, setting up MCC as an organization, and developing guidance and policies while also working with countries to finalize compacts. In the context of MCC as an evolving organization, we felt it was important to discuss the evolving nature of guidance to provide a balanced perspective regarding the process that eligible countries had to follow to sign compacts. 2. State noted that we did not detail the methodology and composition of our Madagascar focus groups. We summarized the composition of the focus groups in the letter of this report. We met with these 29 individuals, including representatives of MCA-Madagascar and Madagascar ministries, during the course of our week in Madagascar and reviewed with them the assumptions used in MCC’s economic model for its projects. As previously discussed in this report, we found that these officials were not closely involved in MCC’s economic analysis. According to MCC and Madagascar officials, an MCC economist met with four Madagascar representatives in Paris, France, during compact negotiations and reviewed the model with them at that time. (MCC officials did travel to Madagascar during due diligence to assess proposed projects.) When we discussed this with MCC officials, they stated that not traveling to Madagascar to develop the model was a mistake, and one that has not been repeated. 3. State commented that we do not rebut MCC’s point that countries’ degree of involvement depended on their capability and willingness to participate. (See comment 2.)

In its letter, which is reprinted in appendix V, MCC agreed that in some cases, the level of country engagement on economic analysis could be improved. MCC also now requires that countries engage an economist on the core country team, provides examples of the economic rate of return analysis, and has MCC economists make early technical guidance trips to work with the core country team and the core country team economist. These procedures were not in place when MCC conducted the economic analysis for Madagascar, Cape Verde, or Honduras.

By not thoroughly involving the countries in this analysis, MCC risks not meeting its stated principle of focusing on results. In MCC’s results- based framework, the economic model is the basis for determining performance targets against which funding is conditioned. MCA- Madagascar may find that the targets are unrealistic if the targets have not been thoroughly discussed with, and understood by, Madagascar officials. 4. State noted that the time for the in-country MCA organizations to hire staff compares favorably with general government hiring. Management unit officials are hired by a compact country’s MCA program steering committee, not the U.S. government. As such, it is not “general government hiring” and should not be compared with it. 5. State correctly observed that we did not offer MCC a recommended alternative to its taking the time to find good candidates for positions. We did not make a recommendation for addressing this issue because MCC has adopted a policy implementing the authority given it by section 609(g) of the Millennium Challenge Act of 2003 to make grants to facilitate compact development and implementation. MCC’s current policy includes a provision where, if certain conditions are met, it may fund an eligible country’s request for “management support payments” for salaries, rent, and equipment for the country’s team prior to compact signature. We have added a discussion of this policy in the letter of this report to clarify this point.

6.

In referring to our discussion about the different maturity of internal control between Madagascar and Cape Verde, State pointed out that differences in the maturity of internal control between the countries are to be expected, given that some countries have historically more well-developed systems. We agree that different countries will differ in their maturity of internal control due to historical differences in the underlying systems and accountability infrastructures. The current maturity of internal control in each country, however, is a key consideration in assessing risk and establishing effective oversight mechanisms to deal with the unique risks in the key financial processes and activities of each country. 7. State referred to “GAO’s reliance on project evaluation criteria that make use of randomized controlled trials to measure success.” This comment misconstrues our findings on the use of randomized controlled trials. We do not rely on these criteria. We are commenting on MCC’s use of randomized controlled trials, which MCC has indicated are its preferred method of impact evaluation, and on the potential challenges to the use of this approach. State’s comments regarding cost-benefit analysis would be more appropriately addressed to MCC. 8. State correctly noted that we did not capture the change to one indicator for fiscal year 2006. We have updated the list of MCC indicators in footnote 8. 9. State suggested rewording MCC’s expected results to “MCC expects to permanently raise incomes.…” We did not use the word “permanently” because in the three compacts that we focused on for this report, it was not used to describe MCC’s expected results at the country level.

GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the person named above, Phil Herr (Assistant Director), Claude Adrien, Cara Bauer, Gergana Danailova-Trainor, Jeanette Franzel, Keith Kronin, Reid Lowe, Norma Samuel, Mona Sehgal, and Michael Simon made key contributions to this report. Also, Tim DiNapoli, David Dornisch, Etana Finkler, Ernie Jackson, Debra Johnson, Bruce Kutnick, Janice Latimer, Charlotte Moore, and Celia Thomas provided technical assistance.