Millennium Challenge Corporation: Compact Implementation Structures Are Being Established; Framework for Measuring Results Needs Improvement
Published: 2006-07-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In January 2004, Congress established the Millennium Challenge Corporation (MCC) to administer the Millennium Challenge Account. MCC's mission is to reduce poverty by supporting sustainable, transformative economic growth in developing countries that create and maintain sound policy environments. MCC has received more than $4.2 billion in appropriations, and, as of May 2006, it had disbursed $22.4 million to four countries whose signed MCC compacts have entered into force. For the first three countries with compact entry into force--Madagascar, Cape Verde, and Honduras--GAO was requested to examine (1) key aspects that MCC reviewed, and the criteria it used, in its due diligence assessments; and (2) the structures that have been established for implementing the compacts.
section_title: What GAO Found
paragraphs: MCC undertook a wide range of activities in its due diligence, including five key aspects of the Madagascar, Cape Verde, and Honduras proposals: (1) countries' consultation with local groups in developing compact proposals, (2) projects' coherence with compact goals, (3) environmental and social impacts, (4) institutional and financial sustainability, and (5) impact on economic growth and poverty reduction. MCC based its assessments on an evolving set of criteria: early, general guidance to the countries followed by later, more specific guidance. MCC's analyses of the projects' economic impact were limited in that some of the assumptions and data used may not reflect country conditions. As a result, the projects selected on the basis of the analyses may not achieve compact goals. In the two countries we visited, Madagascar and Cape Verde, MCC conducted the analyses with limited country participation, which resulted in countries' having little understanding of the process. MCC and the three countries have made progress in establishing compact country structures for oversight and management, procurement, fiscal accountability, and monitoring and evaluation, although some of these structures are not yet complete. The oversight structures allow for country management with MCC review, but some organizations were not fully staffed for months after the compacts entered into force. Madagascar and Cape Verde have implemented fiscal accountability structures for MCC-funded projects, and established procurement structures with effective characteristics; however, these structures are still largely untested and some are still under development. Finally, MCC and the countries have established monitoring and evaluation frameworks to track and account for program results. However, limitations in the baseline data collected, linkage to economic analyses, methods of addressing uncertainty associated with program results, and the timely design of randomized controlled trials may constrain MCC's ability to monitor and evaluate program results.

section_title: Letter
paragraphs: In January 2004, Congress established the Millennium Challenge  Corporation (MCC) to administer the Millennium Challenge Account  (MCA) for foreign assistance. MCC’s mission is to reduce poverty by  supporting sustainable, transformative economic growth in partnership  with developing countries that create and maintain sound policy  environments. MCC carries out its mission by funding projects or activities  in developing countries that demonstrate a commitment to MCA  objectives. MCC expects to raise incomes and lift thousands out of poverty  in countries receiving MCC assistance. MCC has received appropriations  for fiscal years 2004 to 2006 totaling more than $4.2 billion, about $3.8  billion of which has been set aside for compact assistance; as of May 2006,  it had committed $1.5 billion and disbursed $22.4 million in compact  assistance. The President has requested an additional $3 billion in MCC  funding for fiscal year 2007.
In April 2005, we reported on MCC’s first year of operations, focusing on its  process for determining country eligibility, progress in developing  compacts, coordination with stakeholders, and progress in establishing  management and accountability structures. We made several  recommendations to the Chief Executive Officer (CEO) of MCC and to the  Secretary of State, in her capacity as board chair, on ways to improve  MCC’s strategic planning, internal controls, and human capital and  corporate governance policies. At that time, MCC’s assessment—“due  diligence”—of eligible countries’ compact proposals had resulted in its  signing one compact for $110 million with Madagascar; by the end of May  2006, MCC had signed compacts with seven more countries. As of the end  of May 2006, six compact countries—Madagascar, Cape Verde, Honduras,  Georgia, Vanuatu, and Nicaragua—had signed the supplemental  agreements that MCC requires before compacts can “enter into force,”  when MCC begins to disburse funds for compact implementation. MCC has  disbursed funds to four of these six countries—Madagascar, Cape Verde,  Honduras, and Georgia—to begin implementing their compacts.
At your request, we examine in this report MCC’s procedures and  structures for developing and implementing compacts. We focused our  work on the first three countries to sign compacts: Madagascar, Cape  Verde, and Honduras. Specifically, we reviewed   the key areas that MCC examined in its due diligence assessments of the  three countries’ compact proposals and the criteria that MCC used in  these assessments and  the implementation structures that MCC and the three countries have  established for oversight and management, fiscal accountability,  procurement, and monitoring and evaluation of compacts.
We also report on MCC’s response to our April 2005 recommendations (see  app. I). In conducting our work, we analyzed MCC’s process for evaluating  eligible country proposals, including MCC’s guidance documents for  eligible countries, policies and procedures, economic analysis, and  documentation of its assessment of proposals submitted by Madagascar,  Cape Verde, and Honduras. In addition, we reviewed MCC’s compacts with  the three countries and the supplemental agreements required for those compacts to enter into force. We also reviewed the countries’  implementation plans, such as those for monitoring and evaluation. We  supplemented our evaluation of due diligence and implementation  documents with interviews with MCC officials and site visits to Madagascar  and Cape Verde, in January and February 2006. We selected these two  countries because they had advanced further than Honduras in filling key  positions and beginning compact implementation. These site visits  provided us with a firsthand perspective of the status of the program and  the challenges these countries faced in developing their compacts and in  beginning to implement their programs. Furthermore, we examined MCC’s  planning documents, policies, and procedures and interviewed senior MCC  officials. We conducted our review from June 2005 through May 2006 in  accordance with generally accepted government auditing standards. (See  app. II for additional details of our scope and methodology.)
subsections: section_title: Background
subsections: section_title: MCC Principles
paragraphs: In pursuit of its mission to reduce poverty by supporting economic growth,  MCC has identified and defined the following three key principles to guide  its actions: Reward good policy. “Using objective indicators, countries are selected  to receive assistance based on their performance in governing justly,  investing in their citizens, and encouraging economic freedom.”
Operate in partnership. “Countries that receive MCA assistance are  responsible for identifying the greatest barriers to their own  development, ensuring civil society participation, and developing a  multi-year MCC compact.”
Focus on results. “MCA assistance goes to those countries that have  developed well-designed programs with clear objectives, benchmarks to  measure expected results, procedures to ensure fiscal accountability for  the use of MCA assistance, and a plan for effective monitoring and  objective evaluation of results. Programs are designed so that recipient  countries can sustain progress after the funding under the compact has  ended.”
section_title: MCC Structure
paragraphs: MCC is a government corporation that is managed by a CEO appointed by  the President with the advice and consent of the Senate and is overseen by  a Board of Directors (MCC Board). The Secretary of State serves as board  chair, and the Secretary of the Treasury serves as vice-chair. Other board  members are the U.S. Trade Representative, the Administrator of the U.S.  Agency for International Development (USAID), the CEO of MCC, and up  to four Senate-confirmed public members who are appointed by the  President from lists of individuals submitted by congressional leadership.
section_title: Eligibility for MCC Assistance
paragraphs: The Millennium Challenge Act of 2003 requires MCC to select countries as  eligible for MCA assistance each fiscal year. Countries with per capita  income at or below a set threshold may be selected as eligible for  assistance if they pass MCC indicator criteria and are not statutorily barred  from receiving U.S. assistance. MCC uses 16 indicators divided into three  categories: Ruling Justly, Encouraging Economic Freedom, and Investing in  People. To be eligible for MCA assistance, countries must score above the  median relative to their peers on at least half of the indicators in each  category and above the median on the indicator for combating corruption.  MCC used these quantitative indicators, as well as the discretion implicit in  the Millennium Challenge Act, to select 17 countries as eligible to apply for  MCA compact assistance for fiscal years 2004 and 2005. For fiscal year  2006, MCC identified 23 countries as eligible for assistance—the 17  previously selected and 6 additional countries, which included lower- middle-income countries eligible for the first time in fiscal year 2006.
section_title: MCC’s Compact Development
paragraphs: After MCC selects eligible countries, they may begin a four-phase process  that can lead to the entry into force of compacts (see fig. 1). Each phase of  this process is discussed after figure 1. 1. Country proposal development. Eligible countries are invited to submit  compact proposals, which are to be developed in consultation with  members of civil society, including the private sector and  nongovernmental organizations (NGO). The eligible country also  identifies an “accountable entity” to manage the programs funded by  MCC. Eligible countries submitting proposals are not guaranteed  funding; instead, MCC assesses proposals through its due diligence  review. As of May 2006, 14 of the 17 countries selected as eligible in  fiscal year 2004 or 2005, and 1 of the 6 countries selected as eligible for  the first time in fiscal year 2006, had submitted proposals accepted by  MCC for due diligence review. 2. MCC’s due diligence review. MCC determines whether the proposal  that an eligible country has submitted meets MCC criteria to ensure  that proposed programs will be effective and funds will be well-used.  Due diligence primarily occurs between MCC’s acceptances of an  “opportunity memo” and an “investment memo.” MCC assembles a  transaction team of MCC staff, personnel from other U.S. agencies, and  consultants to conduct a preliminary assessment of a country’s  proposal and reports the team’s findings in an opportunity memo to the  MCC investment committee. The MCC investment committee consists  of MCC’s CEO, vice presidents, and other senior officials. If the  opportunity memo is approved, the transaction team launches a  detailed due diligence review. The team assesses the country proposal,  reports its findings, and makes recommendations based on its  assessment in an investment memo to the MCC investment committee.  As of May 2006, MCC was conducting due diligence analyses of seven  eligible country proposals. 3. Compact negotiation and MCC Board approval. MCC may enter into  compact negotiations with the eligible country before the investment  memo is completed. If compact negotiations are successful, MCC staff  formally submit the compact for MCC Board approval. Once the board  approves the compact, MCC and the eligible country may sign it. As of  March 2006, MCC had signed compacts with 8 of the 17 countries  determined eligible in fiscal years 2004 and 2005. (See fig. 2.) MCC  commits the full amount of the compact funding at signing but  obligates and begins to disburse funds to implement projects only after  the compact has entered into force. Under the Millennium Challenge  Act, compacts may remain in force no longer than 5 years. The  compacts stipulate that, with limited exceptions, all funds must be  spent during that time. 4. MCC and compact country complete entry-into-force requirements.
MCC’s compact with each country identifies the following  supplemental agreements that MCC and the country’s accountable  entity must complete before the compact can enter into force.
The disbursement agreement sets out the “conditions precedent” and  other requirements for disbursements from MCC and redisbursements  to any person or entity. These conditions include performance targets  for projects outlined in the compact.
The procurement agreement sets forth guidelines for all procurements  of goods, works, and services financed with MCC funding.
Compact term sheets for supplemental agreements, which vary by  country, and include documents such as a governance agreement, fiscal  agent agreement, form of implementing entity agreement, and form of  bank agreement.
After compacts enter into force, MCC may begin the disbursement of funds  and countries may begin implementing projects. In the first eight compacts,  approximately 53 percent of funding went to transportation and other  infrastructure projects; 22 percent went to agriculture and rural  development; 13 percent went to other project types; and 12 percent went  to program management, monitoring, and evaluation. (See fig. 2.)
section_title: Compact Signature and Entry-into-Force Time Frames
paragraphs: The length of time from country eligibility selection to compact signature  has varied, with proposal development and due diligence generally  requiring the most time (see fig. 3). For the six countries whose compacts  had entered into force as of the end of May 2006, completing the steps  necessary for entry into force after compact signing took approximately 3  to 4 months for Madagascar, Cape Verde, and Honduras; approximately 7  months for Georgia; 2 months for Vanuatu; and about 10½ months for  Nicaragua. For the two countries whose compacts had not entered into  force as of the end of May 2006, 3 months had elapsed since compact  signature for Benin, and 2 months had elapsed for Armenia (see fig. 3).
section_title: MCC Guidance
paragraphs: MCC has issued guidance and policies for its compact development  process in several stages. Before publishing its initial guidance in May 2004,  MCC provided countries with preliminary guidance addressing fiscal  accountability and monitoring and evaluation. Figure 4 shows the evolution  of MCC’s published guidance relative to the end of the due diligence  process with the investment memo.
section_title: MCC Assessed Five Key Aspects of Proposals While Developing Guidance, but Limitations Affected the Accuracy of Economic Analyses
paragraphs: MCC undertook a wide range of activities in its due diligence of the  Madagascar, Cape Verde, and Honduras proposals, while at the same time  developing guidance on key aspects of the countries’ proposals. During  due diligence, MCC primarily considered criteria related to the proposals’  consultative process, project coherence, environmental impact,  institutional and financial sustainability, and economic analyses. MCC  generally approved proposals that were based on a consultative process  and returned proposals that lacked adequate consultations; however, MCC  did not publish detailed criteria for the consultative process until 1 year  after selecting the countries as eligible for MCC assistance. In assessing  project coherence, MCC approved projects that were linked to the overall  proposal objectives and rejected projects that were not, although its  assessments used criteria that it had not yet published in its guidance.  Additionally, MCC screened projects for likely environmental impacts and  considered factors important for institutional and financial sustainability.  Finally, MCC conducted economic analyses to assess the projects’ likely  impact on economic growth. However, limitations in assumptions and data  may have affected the analyses’ accuracy and led MCC to select projects  that would not achieve its goals. Also, a lack of country involvement in the  analyses does not reflect the MCC principle of working in partnership with  countries and may have limited the countries’ understanding of the  process.
subsections: section_title: MCC Assessed the Consultative Process as Guidance Evolved
paragraphs: MCC’s due diligence for Madagascar, Cape Verde, and Honduras assessed  whether the countries had consulted with public and private sector and  civil society stakeholders during proposal development. MCC officials told  us that before beginning due diligence for the three countries, they  assessed the consultative process in proposal drafts and, on the basis of  this review, returned proposals that were not sufficiently founded on a  consultative process. For example, according to the officials, MCC did not  accept one country’s proposal because, although the country had consulted  with stakeholders, its proposal did not reflect the priorities identified  during the consultative process.
MCC documents indicate that during its due diligence for two of the three  countries, MCC obtained the views of government, civil society, and private  sector officials on how the governments conducted the consultative  process. In addition, MCC assessed any previous experience the country  had had with a consultative process For all three countries, MCC also  reviewed factors such as the date, frequency, and locations of the process.  MCC’s documentation of these assessments in Madagascar, Cape Verde,  and Honduras indicated that key stakeholders had generally agreed on the  proposed compact priorities. However, the MCC assessments noted some  weaknesses in the Madagascar and Honduras governments’ management of  the process:  In Madagascar, some groups expressed concern to MCC that the  government had provided short notice (less than 10 days) for  consultative meetings, and that this might have limited rural groups’  participation. Additionally, the government did not communicate to  participants its rationale for accepting or rejecting projects. However,  despite these shortcomings, MCC noted “widespread agreement and  enthusiasm for the…primary components of  proposal” among the  business community, local and international NGOs, civil society, and  donors.
In Honduras, MCC found, on the basis of discussions with local civil  society organizations and international NGOs, that weaknesses—such  as the large size of meetings—had limited effective participation in the  consultative process for the poverty reduction strategy, which formed  the basis of Honduras’s MCC proposal. MCC’s assessment also indicated  that the government had not directly asked the consulted groups to  identify obstacles to growth. However, MCC found that these groups  concurred with the priorities identified in the country proposal.  According to MCC officials, they followed up with the Honduran  government to address the weakness noted by MCC in Honduras’s  consultative process. As a result, according to MCC documentation, the  Honduran government conducted additional consultative sessions with  civil society organizations and donors.
MCC’s due diligence of the countries also noted requirements for additional  consultations during project implementation. For example, in some cases,  countries were to undertake consultations with local stakeholders to  identify project sites and conduct environmental assessments.
Our discussions with representatives of civil society groups and donors in  Madagascar and Cape Verde indicated that they generally concurred with  the compact proposals. In Madagascar, a representative from a key civil  society organization noted weaknesses in the government’s conduct of the  consultative process similar to those recorded by MCC. Nevertheless,  stakeholders on the Madagascar advisory council, which includes various  civil society and other organizations, said that the compact proposal  generally accounted for their views, especially in comparison with other  donor programs.
As a relatively new organization, MCC conducted due diligence reviews  while it was developing consultative process guidance:  Evolving guidance. While developing their proposals, the countries had  access to general criteria in MCC’s 2004 guidance; however, in assessing  the proposals, MCC applied the more specific criteria contained in its  detailed 2005 guidance. As figure 5 shows, the 2005 guidance was issued  1 year after MCC announced the eligibility of Madagascar, Honduras,  and Cape Verde. According to MCC’s Guidance for Developing  Proposals for MCA Assistance in FY 2004, “…each proposal is  expected to reflect the results of an open consultative process,  integrating governmental interests with those of the private sector and  civil society.” The 2004 guidance required that proposals include a  description of the consultative process, such as how the proposal takes  into account local-level perspectives of the country’s rural and urban  poor, including women, and of private and voluntary organizations and  the business community. Additionally, the guidance required the country  to list all key participants, such as government and nongovernmental  officials, who played a significant role in developing the proposal.
MCC’s 2005 Guidance on the Consultative Process more specifically  requires eligible country governments to involve their citizens in  identifying obstacles to economic growth and developing and  prioritizing the development strategies and programs that will be  included in the compact proposal. The 2005 guidance further states that  an adequate consultative process should be timely, participatory, and  meaningful. MCC’s guidance also took into account the country’s  experience in using a consultative process to develop other national or  poverty reduction strategies. If the compact was built on these  consultations, MCC required some additional consultations to provide  justification of country priorities in the MCA proposal.
Incomplete documentation. MCC’s documentation of its due diligence  for the three countries presents summary findings, rather than an  analysis of the extent to which the countries consulted with the rural  and urban poor. For example, MCC’s due diligence documentation  indicates that the countries’ governments included women’s groups and  rural sector groups in their consultative process; however, the  documentation does not indicate the extent to which these groups  represented the poor. Additionally, in Madagascar and Honduras, MCC’s  documentation does not indicate how MCC assessed the extent to  which the consulted groups informed compact proposal priorities.
section_title: MCC Assessed Project Coherence While Developing Specific Guidance
paragraphs: In keeping with MCC’s emphasis on results, due diligence for the three  countries also assessed whether proposed projects were linked to one or  more of the country’s compact goals, and whether the projects addressed  key impediments or constraints to achieving these goals. On the basis of its  due diligence assessment, MCC rejected projects that were not linked to  key constraints in two of the three countries we reviewed. Specifically,  MCC rejected tourism and preschool education projects proposed by  Honduras because they were not linked to the impediments to growth that  emerged from the consultative process. MCC also rejected projects to  construct feeder roads, which connect some watershed areas to the  markets, and provide access to electricity in the rural areas of Cape Verde,  because MCC’s due diligence did not indicate that these projects addressed  key constraints in Cape Verde.
In reviewing MCC’s due diligence for the three countries, we found that  MCC did not issue guidance stating that proposed projects should be linked  to the compact goal until after it had concluded its due diligence  assessments. MCC’s 2004 guidelines for proposal development broadly  instruct eligible countries to identify priority areas, such as health or  education, and their expected goals for each priority area over the term of  the proposed compact. The guidance also asks the countries to show how  these strategic goals are related to the economic growth and poverty  reduction of the country. MCC’s November 2005 MCC Compact Assessment  and Approval Guidelines more specifically indicates that MCC will assess  how the project addresses compact goals. However, MCC issued the November 2005 guidance after completing its due diligence for the three  countries. (See fig. 6.)
section_title: MCC Screened Projects for Environmental and Social Impact, Conditioning Funding on Detailed Assessment
paragraphs: MCC’s due diligence for the three countries included a review of the  probable environmental and social impact of projects that met its  economic analysis and other criteria. For projects that it deemed likely to  cause adverse environmental and social impact, MCC required impact  assessments or environmental analyses, including an impact management  plan. required. For example, MCC assigned all projects in the Madagascar  proposal to category C, because MCC determined that these projects were  not likely to have adverse environmental and social impact. In contrast,  MCC assigned infrastructure projects in Cape Verde’s and Honduras’s  proposals to category A or B. For example, the highway expansion project  in Honduras was assigned category A, because it involves the clearing of  rights-of-way that will require compensation for more than 200 affected  people. The port expansion project in the Cape Verde proposal was also  assigned to category A, because it entails dredging and construction in and  around an existing port.
Additionally, MCC assessed whether environmental impact assessments  had been conducted for category A or B projects. For projects lacking  environmental impact assessments, MCC conditioned project funding on  the completion of such assessments as well as on the development of  mitigation plans in consultation with affected groups. For projects that  other organizations had assessed for environmental impact, MCC used a  U.S. agency or a contractor to evaluate the assessment and determine its  adequacy. For example, in Honduras, another donor had already conducted  the environmental impact assessment for the highway segments proposed  for MCC funding. As part of MCC due diligence, the U.S. Army Corps of  Engineers reviewed these assessments and made recommendations that  were incorporated in the existing assessments. However, in Cape Verde,  the MCC contractor conducting due diligence found another organization’s  assessment of the port project’s environmental impact to be inadequate. As  a result, MCC required a new environmental assessment, along with plans  to manage adverse impact, as a precondition for funding the project. MCC  has allocated funds for this analysis in the compact budget for Cape Verde.
MCC’s 2004 proposal development guidelines do not address projects’  environmental and social impact. In assessing environmental impact, MCC  applied criteria from the Millennium Challenge Act of 2003, which prohibits  MCC from funding projects that are “likely to cause a significant  environmental, health, or safety hazard.” MCC also used criteria laid out in  its March 2005 interim Environmental Guidelines, which state that MCC  will not fund projects that lack the appropriate screening or analysis for environmental impact.The guidance also states that the country has  primary responsibility for conducting and monitoring environmental  assessments. (See fig. 7.)
section_title: MCC Evaluated Country Commitment and Capacity and Estimated Project Impact to Assess Sustainability
paragraphs: In keeping with its emphasis on sustainable progress, MCC’s due diligence  examined whether the three countries’ proposed projects could be  sustained after their compacts expired. In assessing project sustainability,  MCC reviewed each country’s policy and regulatory environment and  commitment to reforms and financing of future maintenance costs; it also  reviewed expected results from MCC-funded projects. In addition, MCC  considered the countries’ institutional capacity to sustain proposed  projects as well as other donors’ roles in strengthening countries’ capacity. institute such reforms. In the case of Madagascar, MCC reviewed policy  reforms in land management and financial sectors that would benefit  MCC-funded activities.
USAID- nd MCC-fnded business center re  colocted in Antirabe, Masr. The USAID-fnded progrsupportrmers’  ccess to mrket nd provide linkge to lrge  business. The MCC-fnded progrm will focus on  technicassnce to frmer nd mke them  creditworthy. Thee frmereek help from the  USAID project lter. The USAID project i expected  to re it exiting client lit with nd introdce  thee client to the MCC progrm. Althogh USAID  nd MCC project re conidered complementry,  there i  perception tht MCC i replcing USAID.  Thi perception i trengthened y the fct tht  USAID i loing abt one-third of it employee in  Masr, ccording to U.S. offici with whom  we poke. had limited institutional capacity to achieve the project objectives of  increasing land security and promoting financial intermediation to  increase rural savings and extension of credit. To build the government’s  capacity, MCC budgeted funding for (1) staff recruitment and training at  Madagascar’s land management department to support the land security  project and (2) finance, management, and production training for rural  producers and microfinance institutions to support the financial  intermediation project. In addition, MCC considered the role of other  donors in strengthening the countries’ institutional capacity. For  example, while assessing road projects, MCC considered the World  Bank’s road sector initiative, which includes institutional capacity- building in Cape Verde. In Honduras, MCC considered World Bank- funded and Inter-American Development Bank–funded programs  intended to develop the transportation ministry’s management capacity  and maintenance contracting capacity.
Project impact. MCC also relied on the assumptions used in its analysis  of projects’ economic impact to determine the sustainability of  agricultural sector projects for all three countries we reviewed. For  example, MCC expects that as a result of MCC-funded technical  assistance or credit to farmers and rural entrepreneurs, recipients will  be able to generate enough income to afford these services by paying  fees to providers. In cases where sustainability depends on achieving  MCC’s projected impact, the soundness of MCC’s economic analysis,  discussed in the next section of this report, will also be an important  factor.
In its sustainability assessments, MCC generally adhered to guidance  issued in 2005, rather than to guidance from 2004. MCC’s May 2004  proposal development guidance included a general requirement for a  strategy to sustain progress after the compact’s expiration. MCC’s  November 2005 MCC Compact Assessment and Approval Guidelines did  not require such a strategy, but the guidelines required identification of  factors contributing to institutional and financial sustainability for each  project. (See fig. 8.)
section_title: Limitations in Assumptions and Data, as Well as Country Involvement, Affected the Accuracy of Economic Analyses
paragraphs: During its due diligence reviews for Madagascar, Cape Verde, and  Honduras, MCC analyzed proposed projects’ probable impact on the  country’s economic growth and poverty reduction. This analysis was  intended both to assess whether these projects would achieve MCC’s goals  and to provide a basis for monitoring their progress and evaluating their  impact.
To predict each project’s impact on economic growth, MCC calculated an  economic rate of return (ERR) —that is, the expected annual average  return to the country’s firms, individuals, or sectors for each dollar that MCC spends on the project. In calculating projects’ ERRs, MCC used an  economic model that includes the following elements (see fig. 9):  MCC’s annual expenditures for the project,  the project’s annual benefits to the country,  predicted net benefits of the project, and  the projected ERR.
ERRs for the proposed projects in Madagascar, Cape Verde, and Honduras  varied considerably, ranging from 116 percent for Madagascar’s land titling  project to 10 percent for a watershed management and agriculture support  project in Cape Verde; the median ERR for projects in the three countries  was 14 percent. However, because MCC analyzed the Madagascar and  Honduras proposals before publishing its first set of economic analysis  guidelines, the ERRs did not significantly affect those countries’ overall  choice of projects. (See fig. 10.) MCC’s finding of a low ERR for part of  Cape Verde’s watershed management and agriculture support project  resulted in the country’s dropping an irrigation activity on one island.
We found limitations in the assumptions and data that MCC used in its  analyses as well as in the countries’ involvement in the analyses. These  limitations may negatively affect the accuracy of the analyses and the  countries’ understanding of the analysis process, respectively.
Assumptions and data. Some of the assumptions and data that MCC  used in its analyses do not fully reflect the countries’ socioeconomic  environment. As a result, MCC cannot be assured that the projects it  approved, partly on the basis of these analyses, will achieve the  compacts’ goals. For example, in calculating the ERR for the  Madagascar land titling project, MCC assumed that local small farmers  would use newly titled land as collateral for loans, invest the borrowed  funds in agricultural activities, and benefit from the increased income  from those activities We discussed this assumption with focus group  participants in Madagascar, including MCA and U.S. government  officials, senior Madagascar government officials representing  ministries affected by the compact, and bank representatives. Our  discussions suggested that MCC’s analysis may have been overly  optimistic in assuming that small farmers would mortgage 40 percent of  their newly titled land in Madagascar’s uncertain market, which does not  offer insurance. While MCC assumed that 40 percent of the newly titled  land will be collateralized, focus group participants believed this was  overly optimistic. As a result, MCC may have calculated an  unrealistically high ERR for the land titling project. Furthermore, the  project may be more likely to benefit farmers with large, secure landholdings and investors from outside the farming community, rather  than the local small farmers it was intended to help.
Similarly, MCC’s economic model for a part of Madagascar’s finance  project, the modernization of the National Savings Institution, may not  accurately reflect the institution’s financial condition. MCC’s model for  the institution’s modernization uses the institution’s April 2003 net  profits (1.4 billion Malagasy francs) as a baseline for estimating the  benefits of computerizing the bank. However, according to institution  officials, 2003 is not a representative year for the bank, and it  experienced unusually large net losses (13 billion Malagasy francs) after  an economic crisis in 2002. As a result, MCC may have inaccurately  estimated the project’s likely impact on the banking system.
Country involvement. In the two countries we visited, country  representatives were not closely involved in MCC’s economic analyses  of the proposed projects. This constrained the contribution of the  analysis process to enhancing country partnership, and stakeholders’  understanding of MCC’s economic analysis—including the data  analyzed, assumptions used in the analysis, and the expected outcomes.  According to country officials in Madagascar and Cape Verde, MCC  developed the economic models and selected data with little assistance  from country representatives. According to the Cape Verde proposal  team, they provided some data but did not participate in the actual  analysis and did not have a clear understanding of the process and the  criteria MCC used to assess proposed projects. We discussed these  issues with MCC officials, who told us that the countries’ degree of  involvement depended on the capability and willingness of the  countries’ proposal development team to actively participate in the  analyses.
section_title: MCC-Approved Compact Implementation Structures Are Not Yet Complete; Weaknesses in Monitoring and Evaluation Framework May Limit Measurement of Results
paragraphs: MCC gave Madagascar, Cape Verde, and Honduras authority to propose and  develop the implementation structures that they will use to manage  compacts, although MCC retains authority over a number of management  decisions. To govern their programs, the countries have created  management units under the direction of a steering committee, but they  have had difficulty in filling key positions. The countries also have  established structures for ensuring fiscal accountability and for managing  procurements that appear to be effective; however, implementation is still  at a very early stage, and some required elements of these structures are  not yet in place. Finally, the countries have established frameworks for  monitoring and evaluating the performance of MCC projects. However, the  frameworks have weaknesses related to the inadequacy of baseline data,  linkage of monitoring plans with economic models, methods of addressing  uncertainty in achieving stated targets, and the timeliness of research  designs for randomized controlled trials. These weaknesses may limit  MCC’s ability to track and account for program results.
subsections: section_title: MCC-Approved Country Procurement Systems Have Effective Characteristics, but Are Early in Their Implementation
paragraphs: Consistent with MCC guidance, each of the three countries—Madagascar,  Cape Verde, and Honduras—proposed its own procurement system,  subject to MCC review and approval during due diligence. Although the  approved systems have characteristics that we have found typical of  effective procurement systems, their effectiveness has not yet been tested  by many procurements. In addition, some of the staff and procedures  needed to implement the systems are not yet in place at MCC headquarters  or in compact countries.
subsections: section_title: MCC Allowed Countries to Propose Procurement Agents and Systems, but Retained Approval Rights
paragraphs: To capitalize on existing country knowledge and experience, MCC has  given the countries flexibility to choose their procurement agents and  standards, with the choice subject to MCC approval during due diligence.  Madagascar and Cape Verde are using private and public sector  procurement agents, respectively, and Honduras is using a combination of  public and private sector agents. (See app. IV.) In contrast to donors such  as the World Bank, MCC also allowed the countries to propose their own  standards for managing procurements. (See app. IV for details of the  countries’ procurement agents and standards.) MCC required the countries  to adhere to “procurement principles” that include equal access to  procurements, competition for awards, and transparency of the process.  Madagascar, Cape Verde, and Honduras have used either existing World  Bank standards or their own laws to govern MCC procurements. In its  procurement agreement with each country, MCC included modifications to  the country’s selected standards to reconcile them with U.S. law and MCC  principles. For example, MCC required that countries not include  preferences for domestic suppliers in solicitations paid for with MCC  funds.
Although MCC’s agreements with the three countries give the countries a  number of authorities over procurements, MCC retains certain approval  rights. MCC’s fiscal accountability framework describes procurement as  one of the highest-risk areas of fiscal accountability. MCC’s compact,  disbursement agreement, and procurement agreement with each of the  countries describe the relationship, roles, and authority of MCC, the  procurement agent, and the compact country (see app. IV for details).  Although these agreements have some common elements, each agreement  is unique to the individual country. To determine when MCC approval of  individual actions is appropriate, MCC included review thresholds in the  procurement agreements keyed to procurement size and methods. Above  cost thresholds, which vary among countries, MCC must approve items  such as the procurement method, terms of reference, and selection. Below  thresholds, the compact country may conduct procurements in keeping  with the procurement plan without additional MCC oversight. These  thresholds are a risk management tool that maximizes MCC control for  larger transactions but leaves discretion to the compact country for smaller  transactions.
section_title: Countries’ Procurement Systems Have Strengths, but MCC and Country Systems Are Not Fully Established
paragraphs: Although the three countries’ procurement systems vary, each has  characteristics that we have previously identified as typical of effective  international procurement systems. These characteristics are similar to  the principles, such as equal access, competition, and transparency, that  MCC applies in its review of the systems during due diligence. However,  MCA officials in Madagascar and Cape Verde told us that they had  completed few procurements that would test the systems in practice.  Furthermore, MCC headquarters has not yet finished hiring its  procurement staff, and procurement systems in Madagascar and Cape  Verde are not yet fully established.
Incomplete staffing. As of May 2006, MCC headquarters had hired its  senior director of procurement but had not yet hired five director-level  procurement staff in its Accountability Department. According to MCC  officials, the function of these vacant positions is currently being filled  by six intermittent personal service contractors, and MCC has offered  one of these contractors a full-time position. During our site visits in  January and February 2006, staff in Madagascar and Cape Verde  reported that the time frames for MCC review of procurements had been  satisfactory. However, if MCC staffing does not increase as the countries  submit more procurement decisions for approval, MCC may have  difficulty in conducting timely reviews of these decisions.
Incomplete systems. During our site visits, we found that some elements  of the procurement systems documented in MCC’s agreements with  Madagascar and Cape Verde were not yet in place or had not functioned  smoothly.
Neither country had yet established the procurement bid protest  body required by MCC or put in place automated systems for  procurement tracking and management, although both reported  plans to do so in 2006. Madagascar also had not established a process  for reviewing contractor change order requests.
During our visit to Madagascar, a senior MCC official discovered that  the country’s procurement plan for months 2 through 4 did not reflect  current project work plans. As a result, the procurement agent was  preparing for procurements that were no longer needed. Members of  the Madagascar management team told us that they would establish  new procedures to coordinate future work plan changes with the  procurement agent.
In Cape Verde, the procurement review commission lacked office  space and was concerned about being able to handle the number of  reviews required of it once procurements begin in earnest. The  commission is comprised of members who are fully employed  elsewhere and are not permitted to delegate their work. The  members have worked nights and weekends or have negotiated with  their supervisors to be released from their other jobs to perform the  commission’s work. According to MCC, subsequent to our site visit,  Cape Verde has taken actions to mitigate the risks to the efficient  operation of the commission.
Both Madagascar and Cape Verde reported difficulties with an MCC  requirement that documents be in both English and the local  language. Cape Verdean procurement review commission members  are not required to speak English but are expected to review  documents prepared in English. Cape Verde was considering hiring a  full-time translator to address this need. In Madagascar, the  translation of a French document prepared by the implementer of the  finance project and requested by MCC delayed the approval of  needed project procurement.
section_title: MCC Has Established Country Monitoring and Evaluation Frameworks, but Weaknesses May Limit Measurement of Results
paragraphs: Each of the three countries’ (Madagascar, Cape Verde, and Honduras)  programs includes a monitoring and evaluation framework that includes  plans for data collection, data quality reviews, analysis, and interim and  final reporting of results. We found several weaknesses in the monitoring  and evaluation frameworks that could affect MCC’s ability to track and  account for program results.
subsections: section_title: Countries’ Programs Include Monitoring and Evaluation Frameworks
paragraphs: Each of the three countries’ programs includes detailed plans for  monitoring and evaluating program results. MCC approved Madagascar’s  and Cape Verde’s plans in November 2005 and April 2006, respectively,  while Honduras’s plan existed in April as a detailed draft but had not yet  been approved. According to MCC officials, Honduras’s plan is not final  pending the staffing of the monitoring and evaluation director position to  ensure country understanding and buy-in. In accordance with MCC guidance, the countries’ plans include separate components for monitoring  and evaluation: The monitoring component includes, among other things, key  indicators that are linked as closely as possible to the variables  identified in the economic analysis of the country’s proposed projects.  These indicators are to be used throughout implementation to assess  whether the program is likely to achieve the desired results. The  monitoring component also identifies baseline and target values for  each indicator and includes plans for periodic performance reports and  data quality reviews. MCC guidelines state that monitoring can be on  select indicators to minimize reporting requirements.
The evaluation component identifies, among other things, the  methodology that will be used to assess the program’s impact, such as  randomized controlled trials, and describes plans for collecting  baseline, interim, and final data on program results.
Countries’ monitoring and evaluation plans use, to varying degrees, the  economic analysis of the proposed projects to identify indicators for  monitoring progress toward project objectives, calculate targets for each  indicator, and evaluate the achievement of compact goals. The economic  relationships specified in the models, such as the relationship between  improved infrastructure and farm output, provide a basis for tracking  project success. The monitoring framework also includes setting target  values for indicators. Figure 13 illustrates indicators at various levels for a  rural development project in Honduras.
Although the specifics of the countries’ plans vary, the monitoring  component of each plan calls for the country’s MCA to submit periodic  performance reports and data quality assessments to MCC. Performance  reports will include quarterly assessments to alert the countries and MCC  to any problems, periodic audits that analyze performance at all compact  levels, and annual reports that consolidate the quarterly reports and  recommend adjustments. Most performance data will be gathered by  project implementers, either country government employees or  contractors; the plans allow for contracting with other entities to prepare  the reports. In addition, each plan calls for third-party data quality reviews  over the course of the compact. For example, in Madagascar, data quality  assessments are planned to occur every 6 months during the first year and  annually thereafter.
section_title: Weaknesses in Monitoring and Evaluation Frameworks May Limit MCC’s Ability to Measure Results
paragraphs: In reviewing the frameworks for monitoring and evaluation in the three  countries, we identified several challenges that MCC faces in ensuring  accountability for results. These challenges include (1) ensuring the  availability and quality of baseline data, (2) establishing clear links between  the economic model and the monitoring and evaluation framework,   (3) accounting for the degree of uncertainty in expected outcomes, and   (4) using randomized controlled trials in compact countries.
section_title: Ensuring Baseline Data Availability and Quality
paragraphs: Baseline data are essential to measuring the results of MCC-funded  compact projects. Although MCC has taken steps to collect baseline data  for monitoring and evaluation, problems with data availability and quality  may lead to challenges in measuring the progress and impact of the  countries’ projects over time. MCC officials told us that they worked with  their country counterparts to set up a Management Information System  that can meet the requirements for collecting performance data. In  addition, MCC evaluated the technical capabilities of the country staff and  the information system the country proposes to use for data management  purposes. Finally, MCC budgeted funding for surveys in Madagascar and  Cape Verde to collect baseline data when it was not available. However, we  found that some of the baseline data in the countries’ monitoring plans  were not complete, and that some of the data MCC collected were not  reliable.
Baseline data availability. In some instances, the countries’ monitoring  and evaluation plans lack complete baseline data against which to  measure progress. For example, two activity and final project indicators  in Madagascar’s plan—“volume of production covered by warehouse  receipts in zones” and “volume of microfinance institution lending in the  zones”—currently lack baselines because the intervention zones have  not yet been selected. Moreover, although the collection of performance  data is closely linked to project implementation, Madagascar’s plan  contains no intermediate outcome indicators and target values, thereby  making it difficult to effectively track project progress. (See fig. 13 for  an example of the role of intermediate outcome indicators in the  monitoring structure of Honduras.)
Baseline data quality. MCC may face challenges in ensuring the quality  of the baseline data that it uses to monitor and evaluate program impact  and, as a result, may have difficulty in accurately measuring the impact  of compact projects. MCC officials told us that it has been difficult to  obtain accurate and reliable baseline data against which to measure  program results. In some countries, MCC has funded surveys to obtain  the needed baseline data. However, even with the additional resources  provided by MCC, obtaining baseline data has been a challenge. For  example, we found significant data quality problems associated with  one of three surveys that MCC funded to collect baseline data in  Madagascar. Our interviews with Madagascar and USAID officials who  oversaw the survey revealed that the survey results, which were used to  estimate average land values, are flawed in that they do not reflect  recent significant changes in Madagascar’s currency. Madagascar’s  compact goal is to increase household income, as measured by the  percentage of increase in land values. Because of the survey error, the  land value estimates may not be sufficiently reliable to evaluate the  project impact and the compact as a whole.
section_title: Linking Monitoring Plans to Economic Analyses
paragraphs: Linking the indicators used to monitor and evaluate progress to the data  and assumptions used in MCC’s due diligence economic analyses will also  present a challenge. In reviewing the draft plan for Honduras, we found  consistent linkages between the indicators for monitoring and evaluation  and the variables and assumptions used in the economic model. However,  in the plans for Madagascar and Cape Verde—the first two plans that MCC  approved—we found instances where MCC did not sufficiently link the  monitoring plans to the economic models, which may hamper its ability to  effectively measure project results. For example:  After signing its compact with Cape Verde, MCC changed the interim  targets for seven indicators. In two cases, neither MCC nor Cape Verde  was able to identify the methodology used to select the indicators in the  monitoring and evaluation plan. According to MCC officials, they  decided that the assumptions in the economic analysis were a poor  basis for constructing the monitoring indicators and, therefore, chose  other indicators and estimated the targets. According to MCC officials,  the inability to identify the methodology, in conjunction with updated  baselines and revised work plans in the country resulted in MCC’s and  Cape Verde’s agreeing to reduce the interim targets that had been  established.
MCC’s economic analysis for Madagascar’s Land Tenure Project, which  is approximately one-third of the compact budget, did not identify the  expected benefits from the separate project activities. Therefore, we  could not track the linkage between the activities in the model to those  in the monitoring and evaluation plan.
MCC’s approved monitoring and evaluation plan does not include  tracking the results of two finance project activities—modernizing the  National Savings Institution and opening bank branches—although they  were included in economic analysis calculations during due diligence.  According to an MCC official, these two finance project activities will be  tracked at a higher level of aggregation—the finance project level—and  monitored by tracking the number and value of new accounts. However,  this approach may not adequately capture the outputs and benefits from  the institution’s modernization and could potentially confound the effect  of one activity (modernizing the institution) with that of other activities.  For example, while an increase in the number and value of new  accounts could result from the two finance project activities, it could  also result from an overall increase in savings if customers invest in  government savings bonds, issued recently and prior to the start of the  MCC compact.
section_title: Accounting for Uncertainty in Achieving Target Values
paragraphs: Although the countries’ monitoring and evaluation plans acknowledge the  uncertainty of achieving indicator target values, MCC project monitoring  does not adequately address (1) the effect of potential variations in  uncertainty on the range of acceptable target values or (2) the plausibility  of target values. As a result, some targets specified in the countries’  monitoring plans may not be achieved.
MCC disbursement agreements include as a condition precedent that, if an  indicator value observed during compact implementation does not fall  within 10 percent of the agreed-on target values, MCC may withhold  disbursements. MCC applies this 10 percent margin to all projects,  regardless of type (e.g., agriculture or infrastructure). However, our  analysis suggests that several factors could cause indicator values for many  projects to fall outside the 10 percent range.
External factors. Uncertainty associated with external factors varies by  country and project. For example, according to previous GAO work,  external factors that could affect project implementation might include  political instability, the lack of commitment of political leaders to  necessary reforms, the magnitude of assistance from other bilateral and  multilateral donors, weather conditions that affect crop yields, and the  instability of international markets. These factors could cause  indicator values to fall outside the 10 percent range used across all  countries and projects.
Time factor. The 10 percent range also does not account for the increase  in the uncertainty of targets over time—for example, target values for  year 5 of a compact are likely to be less precise than those for year 1.
Therefore, to the extent that MCC bases its disbursement decisions on  results falling within a common range, it may not fully account for  variations in uncertainty across projects and over time.
According to MCC guidelines, the economic analyses and monitoring and  evaluation plans should, as much as possible, be clearly linked. However,  limitations in (1) the economic analyses due to problems with data quality,  assumptions, and lack of country involvement and (2) the consistency  between the economic analyses and the monitoring and evaluation plans  constrain MCC’s ability to set plausible targets. If targets are overly  optimistic, countries may fail to reach them and MCC may not be justified  in halting disbursements because the countries failed due to unattainable  targets. Conversely, setting too conservative a target may not prompt the  country to fully utilize MCC resources. A lack of plausible targets may lead  to MCC’s making ad hoc decisions regarding the consequences of missing  targets and applying judgment subjectively and inconsistently in setting or  modifying targets. MCC officials told us that they would use a missed target  as a cue to discuss with the country’s MCA its reasons for missing the  target, and, on the basis of those discussions, they would determine  whether to use their authority to withhold funding. According to MCC  officials, senior management approval would be needed to significantly  modify targets; however, MCC currently has no policy or documentation  that defines a “significant modification” of targets, especially for targets  used as conditions precedent to disbursements.
section_title: Using Randomized Controlled Trials in Compact Countries
paragraphs: MCC has retained research organizations to help the countries evaluate  program impact using, as appropriate, randomized controlled trials, but  MCC’s involvement of these organizations after project implementation  begins may limit their ability to evaluate impact accurately. These  organizations’ scope of work may include training MCC and compact  country staff; designing the trials and data collection; and proposing  appropriate methodologies for, and analyzing results from, impact  evaluations. According to MCC monitoring and evaluation officials, MCC  has begun designing impact evaluations by identifying those program  components that can and cannot be evaluated using randomized controlled  trials, which MCC has indicated are its preferred method of impact  evaluation. According to MCC officials, MCC considers evaluations using  randomized controlled trials as “rigorous” and evaluations using other  methodologies as “standard.”
MCC has hired an independent consultant experienced in impact  evaluations to work with compact countries to assess the appropriateness  of using randomized trials to evaluate MCC’s projects. When these  assessments are completed, the five research organizations will be invited  to compete to conduct randomized trials after compact implementation  begins. However, at that point, the organizations will not have had an  opportunity to assess the design of the countries’ evaluation strategy,  including the adequacy and reliability of the baseline data. Without the  involvement of these organizations before implementation of the relevant  project(s) begins, MCC may not be able to ensure that they have the  necessary data and have established appropriate research designs for their  work. MCC officials told us that they thus far had not involved the five  organizations because rigorous evaluations were turning out not to be  feasible in some cases, or because the tasks were not large enough to  warrant the use of the five research organizations.
section_title: Conclusions
paragraphs: MCC continues to mature and evolve as an institution, taking on the  ambitious task of creating new, country-managed organizations while  developing processes to oversee what are expected to be relatively large  amounts of foreign assistance. Toward that end, MCC has taken positive  steps with regard to establishing policies and procedures for MCA  organizations. However, it has taken time to complete the numerous  agreements necessary for compacts to enter into force. This could continue  to present challenges, given that MCC is working simultaneously with a  number of nations to develop and implement compacts.
As MCC moves forward, partnering with countries to develop well-founded  economic assumptions will be crucial to establishing a foundation for the  work of MCC and its partners. Furthermore, holding countries accountable  for results requires, to the extent practical and cost-effective: collecting  reliable and accurate baseline data, linking economic analyses to  monitoring plans, addressing the uncertainty associated with program  results, and ensuring the timely development of the research design for  randomized controlled trials.
section_title: Recommendations
paragraphs: Because of the central role of reliable economic analyses and the  importance of partnering with countries in achieving MCC goals and  ensuring accountability for MCC programs, we recommend that the Chief  Executive Officer of MCC take the following two steps:  Ensure that MCC officials, in partnership with country representatives,  perform economic analyses that more fully reflect the countries’  socioeconomic environment and are better understood by country  public and private sector representatives.
To the extent practical and cost-effective, improve MCC’s monitoring   obtaining more accurate and reliable baseline data needed to permit  tracking progress during compact implementation;  ensuring a clear linkage between MCC’s economic analyses and  monitoring and evaluation frameworks;  developing policies, procedures, and criteria for establishing targets  and for adjusting those targets if unforeseen events affect outcomes;  and  taking steps to ensure the timely development of the needed research  design for randomized controlled trials, if they are undertaken, prior  to project implementation.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from MCC and the  Department of State. In commenting on a draft of this report, MCC  generally agreed with our findings, conclusions, and recommendations.  MCC noted that our discussion of the evolving guidance provided to  eligible countries in 2004 and 2005 was a result of the complex process of  engaging with eligible countries while simultaneously developing policies  and procedures. MCC stated that this criticism should not be valid beyond  the initial instances covered in this report. We recognize that MCC was  simultaneously addressing a number of issues during this period, but we  felt it was important to discuss the evolving nature of guidance to provide a  balanced perspective regarding the process eligible countries had to follow  to sign compacts.
MCC also commented (1) that our characterization of data quality issues in  Madagascar cited only a single survey and (2) that it differed with us on the  appropriate level of aggregation in linking economic models and  monitoring and evaluation plans. We note that there was one other instance  of poor data quality in Madagascar, but that we focused on the Agricultural  Productivity Survey because of the importance of accurately tracking land  values to monitor results. We agree that disaggregation may not always be  feasible, but note that aggregation poses some challenges that could limit  the effectiveness of monitoring and evaluation. We reprinted MCC’s  comments, with our responses, in appendix V. We also incorporated  technical comments from MCC in our report where appropriate.
State commented that some of our findings reflect minor or transitory  problems and provided specific observations regarding MCC’s evolving  guidance, the assumptions used in MCC’s economic models, country  participation, staffing delays, fiscal accountability structures, and the use  of randomized controlled trials.
State noted that MCC’s guidance could be expected to evolve, given the  newness of the organization, and that informal guidance from MCC was  always available to eligible countries. We agree that guidance could be  expected to evolve, but we sought to provide a balanced perspective by  noting instances where MCC’s verbal guidance may not have been  sufficient to assist countries in submitting proposals that met MCC’s  criteria.
State questioned the findings from our Madagascar focus groups and our  finding that MCC conducted economic analyses with limited country  involvement. Our focus groups resulted in discussions of the assumptions  used in the Madagascar economic analysis with a broad range of country  stakeholders representing U.S. and Malagasy agencies and organizations  involved in implementing the compact. Furthermore, MCC agreed in its  comments that, in some cases, the level of country engagement on  economic analysis could be improved. MCC outlined specific steps that it  had taken to increase country involvement in economic analyses.
Regarding staffing delays and fiscal accountability structures, State  commented that we offered no suggestions to MCC for things they could  have done differently. We added additional material to the report to  elaborate on the steps MCC has taken to reduce delays in staffing key  positions. In regard to fiscal accountability, we agree that different  countries will differ in their maturity of internal control. However,  evaluating maturity is key to properly assessing risk and establishing  effective oversight mechanisms.
Finally, State disagrees with what it terms our “reliance” on randomized  controlled trials to measure success. This comment misconstrues our  findings. We did not rely on or advocate this methodology, but rather are  commenting on MCC’s use of randomized controlled trials as its preferred  method of impact evaluation.
We have reprinted State’s comments, with our responses, in appendix VI.
We are sending copies of this report to interested congressional  committees as well as the Secretary of State, the Secretary of the Treasury,  the CEO of MCC, and the Administrator of USAID. We will also make  copies available to others upon request. In addition, this report will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff has any questions about this report, please contact  David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to this  report are listed in appendix VII.
section_title: MCC Actions in Response to April 2005 GAO Recommendations
paragraphs: The Millennium Challenge Corporation (MCC) has taken a number of steps  to address our April 2005 recommendations regarding its strategic  planning, internal controls, and human capital and governance policies,  although some aspects of its organizational structure are not yet complete.  MCC has prepared a strategic plan for fiscal years 2006 through 2011, and  an annual performance plan for fiscal year 2006. MCC also has  strengthened its internal controls and taken steps to implement some fiscal  accountability mechanisms, established an audit committee within its  governing board, and completed several required audits and reviews.  However, MCC has not documented all financial control activities and  continues to face risks from the poor interfacing of its systems with those  of the Department of the Interior’s National Business Center (NBC). MCC  has reassessed its staffing model, developed a plan for recruitment, and  implemented an improved performance management system; but, it does  not systematically track the use of staff resources to verify its human  capital model. Finally, MCC has approved a corporate governance policy  and taken steps to improve board involvement in planning, management,  and communication; but it has not yet fully addressed risk management for  the corporation.
subsections: section_title: MCC Has Developed a Strategic Plan to Enhance Accountability and Completed a 2006 Performance Plan
paragraphs: Consistent with our recommendation to enhance corporate accountability,  MCC completed a strategic plan, approved by the Office of Management  and Budget (OMB) in November 2005. In April 2006, MCC completed an  annual performance plan that provides goals and benchmarks for assessing  its performance in fiscal year 2006. This annual performance plan will  enable MCC to report in the future on its progress in meeting its goals. As  part of the annual performance plan process, MCC also has developed  goals and benchmarks for its individual departments that support the  attainment of the corporate goals identified in the strategic plan. (See table  4.)
section_title: MCC Has Made Significant Progress in Establishing Internal Controls
paragraphs: In response to our April 2005 recommendations, MCC has made significant  progress in establishing internal controls over program and administrative  operations at both the MCC headquarters and compact levels. MCC has  made progress in each of the five components of internal controls  discussed in our April 2005 testimony: (1) control environment, (2) risk  assessment, (3) control activities, (4) monitoring, and (5) reporting.
Internal control environment. MCC has initiated several measures to  establish a positive internal control environment, including  documenting its organizational structure for the Administration and  Finance and Accountability structures. The Fiscal Accountability area,  which is a component of the Accountability structure, currently consists  of a managing director and several directors; however, other positions  to support the fiscal oversight of MCC compacts have yet to be filled. A  formal ethics program has also been established for MCC headquarters.
Risk assessment. MCC is developing a process for assessing risks facing  the corporation and its programs. To this end, MCC has hired a third- party consulting firm to support the implementation of processes, based  on OMB A-123, Management’s Responsibility for Internal Control,  December 2004, criteria, to address risks associated with domestic and  foreign operations.
Control activities. MCC has instituted several control activities to  reduce risk. MCC has submitted its Strategic Plan under the  Government Performance and Results Act (GPRA) of 1993 and has  completed required audits and reviews, such as those required under the  Federal Managers Financial Integrity Act (FMFIA) of 1982 and the  Federal Information Security Management Act (FISMA) of 2002. MCC is  in the process of addressing the various material weaknesses,  reportable conditions, findings, and recommendations identified in the  audits. For example, MCC continues to address the formal  documentation of control activities for the financial reporting process at  MCC headquarters. MCC also contracted with a third-party service- provider, NBC, to maintain its accounting system and the recommended  Statement on Auditing Standards No. 70 (SAS 70) review was  performed; however, MCC is still addressing issues reported as a result  of the review, such as the need to address manual processes that present  inherent risks associated with accounting systems and related  processes. For example, MCC forwards daily to NBC a package  containing source documentation that is used by NBC to record  transactions to the general ledger. Similarly, information for MCC’s  travel expenses is prepared at MCC headquarters office and daily  communicated to NBC through manual processes. MCC is aware of the  need to address these issues and is working with NBC to mitigate or  eliminate the manual processes.
Monitoring. MCC has taken steps to ensure ongoing monitoring and  periodic testing of control activities. MCC’s investment committee  embodies functions of monitoring and testing and operating as an  integral part of MCC’s internal control program by overseeing the  program units’ compliance with both the procedural and substantive  elements required by its internal processes. Also, MCC conducted its  first comprehensive survey of internal controls, performed by outside  consultants, in conjunction with its annual audit. In addition, MCC has  formed formal review panels to monitor the progress of addressing  findings from internal and external reviews. For example, consistent  with OMB guidance, MCC formed an FMFIA Management Review Panel  to assess the results of the internal control survey along with the  findings of MCC’s independent financial auditors. Similarly, MCC  implemented a specific procedure to address recommendations from  reviews and audits performed by its Inspector General (IG).
Reporting. MCC has made progress in establishing a process for  assessing and reporting on the operating effectiveness of its internal  controls. MCC established a formal board-level audit committee whose  responsibilities include (1) financial controls; (2) the integrity of the  reporting process; and (3) performance of the independent audit  process. In addition, MCC’s FMFIA Management Review Panel assessed  the results of the internal control survey, along with the findings of  MCC’s independent financial auditors. The panel identified four material  weaknesses and actions that MCC will be taking in future months to  resolve them, which the acting Chief Executive Officer certified on  November 7, 2005.
MCC officials told us that the development of new internal control policies  and procedures and the revision of those already in place is a continuing  process as MCC continues to mature as an organization. Table 5  summarizes MCC’s progress in addressing our April 2005  recommendations.
section_title: MCC Has Taken Steps to Develop Human Capital, but Does Not Track Allocation of Human Capital to Key Activities
paragraphs: MCC has taken steps and is continuing to further develop its human capital  systems by (1) assessing its staffing needs; (2) improving its recruitment,  development, and retention systems; and (3) implementing a performance  management system linking compensation to individual contributions  toward corporate goals. However, despite having plans to increase its staff  by an additional 38 percent between May and September 2006, MCC does  not systematically assess its staffing needs, has not developed a human  capital plan, and has not yet fully implemented its improved performance  management system, as follows:  Staffing. Although MCC has completed an assessment of its human  capital needs since our April 2005 recommendation, it does not  systematically track the use of staff time on an ongoing basis. MCC’s  updated assessment of its human capital needs shows that it plans to  increase its staffing from 218 staff, as of May 2006, to approximately 300  staff, as of September 2006. MCC also has created an organization chart  that includes the specific approved positions for each department under  the new 300-person staffing model, and it is hiring for many of these  positions using limited-term appointments to provide greater flexibility  in filling future needs. According to MCC officials, MCC made its case  to OMB for increasing staffing to 300 persons on the basis of an analysis  of MCC staffs’ recollection of the amount of time they spent developing  the Georgia compact, which was thought to have been the most  complex compact development process to date. However, MCC officials  felt that this analysis may not have fully captured the amount of time  spent by some departments in developing the compact. Retrospective  analysis was necessary because MCC does not track employees’ time on  mission-related projects on an ongoing basis. Without such data, MCC  management is not able to systematically assess the staffing  requirements needed to carry out MCC’s mission and consistently align  its human capital with its changing needs.
Recruitment and retention. MCC has identified priorities and  committed resources for recruitment and is developing a human capital  plan to address retention and training. To support its effort to hire  approximately 82 additional staff between May and September 2006,  MCC has retained an outside consultant firm to work on-site and help  with recruitment and has identified positions as first- or second-tier  hiring priorities. MCC officials also told us that they are developing an  overall human capital plan that will include planned activities and a time  frame for identifying critical skills and competencies for MCC’s key  positions. The officials stated that the human capital plan also will  include a strategy for staff retention and will address staff training.  Currently, MCC has developed procedures for providing employees with  outside training. MCC intends to develop a comprehensive training plan  following the completion of the human capital plan, a draft of which it  expects to circulate to MCC senior staff for their review and comment  by September 30, 2006.
Performance management. In keeping with our recommendation, MCC  has established a performance-based compensation framework. MCC  provided us with documentation of its employee ratings process,  showing that employee expectations and performance reviews were  keyed to organizational goals. However, according to MCC officials,  MCC did not incorporate the departmental performance plans for the  year into the performance framework for 2006 until March 2006, as the  annual performance plan neared completion. MCC is shifting from a  calendar year to a fiscal year performance evaluation schedule to better  align employee compensation with its annual corporate goals. MCC  anticipates that its strategic plan, annual performance plan, department  plans, and individual performance goals will be fully synchronized  beginning in fiscal year 2007. (See table 6.)
section_title: MCC Has Taken Steps to Define Corporate Governance
paragraphs: The MCC Board of Directors (MCC Board) has taken steps to define the  scope of its corporate governance and oversight. The MCC Board has  approved a corporate governance policy developed by MCC with the  involvement of staff from MCC Board agencies. According to MCC  officials, the policy incorporates guidance on governance matters provided  by board members at their previous meetings, and the board participated in  formulating MCC’s strategic plan before approving it in December 2005.  The board has established a board-level audit committee and a charter for  that committee. According to MCC officials, to address risk, MCC is  recruiting for the position of risk specialist and has used a contractor to  support MCC risk analysis. Finally, to improve communication with  stakeholders in eligible countries, MCC has published and distributed  updated guidelines for compact development and eligibility. MCC also has  developed a series of open forums where input is sought from groups with  an interest in MCC. (See table 7.)
section_title: Scope and Methodology
paragraphs: At the request of the Senate Committee on Foreign Relations, we examined  the structures and procedures MCC has developed in consultation with  compact countries to manage compacts. Specifically, our work focused on  (1) the key areas that MCC examined in its due diligence assessments of  proposals for Madagascar, Cape Verde, and Honduras, and the criteria that  MCC used in these assessments, and (2) the form and adequacy of the  implementation structures that MCC and compact countries have put in  place for governance, procurement, fiscal accountability, and monitoring  and evaluation. In addition, we reviewed MCC’s progress in responding to  our April 2005 recommendations on its corporate management and  accountability structures (see app. I).
To accomplish our objectives, we reviewed MCC’s documentation of its  processes and agreements, supplemented by interviews with MCC officials.  We focused our review for objectives 1 and 2 primarily on the first three  countries with signed compacts—Madagascar, Cape Verde, and Honduras.  These countries’ compacts were the first to enter into force. To further our  analysis for objectives 1 and 2, we also visited Cape Verde and Madagascar  in January and February 2006. We selected Cape Verde and Madagascar for  our site visits because they had advanced further than Honduras in filling  key positions and beginning compact implementation. While in Cape Verde  and Madagascar, we interviewed a number of MCC, Millennium Challenge  Account (MCA), and government officials and visited project sites.
To identify MCC’s evaluation criteria and process for evaluating eligible  country proposals in due diligence, we reviewed MCC guidance and the  record of MCC analysis contained in (1) MCC’s “due diligence books,”  which are its internal records of how it assessed proposals submitted by  Madagascar, Cape Verde, and Honduras, and (2) investment memos, which  are MCC’s analyses based on due diligence and internal recommendations  to its investment committee. These documents are restricted from public  dissemination due to their sensitive nature, but MCC made them available  to us for analysis. We have coordinated with MCC on describing the  information from these books in general terms without disclosing sensitive  information. We used MCC’s definition of the due diligence process as  beginning with MCC’s opportunity memo and ending with the acceptance  of the investment memo by MCC’s investment committee. Our review,  therefore, may not capture some changes and decisions made by MCC or  eligible countries during proposal development and compact negotiations.  To evaluate MCC’s assessments of proposals’ consultative process, project  coherence, environmental and social impact, and institutional and financial  sustainability, we relied primarily on MCC’s data and analysis contained in  the due diligence books and, to some extent, in the investment memos. We  compared MCC’s analysis in these documents with criteria outlined in  MCC’s guidance. We were able to perform only limited independent  verification of the use and adequacy of these criteria during our site visits.  With regard to its economic analyses, MCC also made available to us the  spreadsheet models it used to develop the economic rate of return  calculations that formed the basis for its evaluations of the suitability of  country-proposed projects for MCC funding. We independently analyzed  these spreadsheets and validated their logic and conclusions on the basis of  a review of economic literature and practices. In addition, in Madagascar,  we conducted a series of focus groups with country officials to assess the  data and logic used by MCC in developing their economic analysis.
To assess MCC’s compact implementation structures, we reviewed the  compacts with Madagascar, Cape Verde, and Honduras and the  supplemental agreements required for those compacts to enter into force.  We supplemented this review with our site visits to Madagascar and Cape  Verde. In all cases, our ability to analyze the adequacy of these structures  was limited by their relative newness and limited use in actual  implementation. We addressed the following four areas of MCC’s  implementation structures:  To determine the form of governance structures and key positions in  these three countries, we reviewed the requirements of MCC compacts  and supplemental agreements. We determined the progress of the  country organizations in Madagascar, Cape Verde, and Honduras in  filling these positions and establishing these structures by analyzing  MCC’s reported staffing and status information. We independently  assessed MCC’s progress in our site visits to Madagascar and Cape  Verde. We discussed factors affecting the filling of these positions  through discussions with MCC and compact country officials.
To assess the adequacy of the countries’ fiscal accountability structures,  we reviewed MCC’s overall fiscal accountability framework and the  operations in place in Madagascar and Cape Verde. We assessed the  adequacy of these structures according to the criteria contained in  GAO’s Standards for Internal Control in the Federal Government. We  assessed MCC’s and the countries’ implementation of these structures  by using criteria in the Internal Controls Capability Maturity  Continuum developed by the independent risk consulting company,  Protiviti, Inc. We independently verified the existence of the structures  described in the plan and discussed its strengths and weaknesses during  the site visits.
To assess MCC’s procurement structures, we reviewed the MCC fiscal  accountability framework, and the implementing procurement  documents for the first three compact countries. We then assessed the  adequacy of MCC’s framework, using criteria identified in previous GAO  reports on international procurement. To determine the status and  factors affecting the implementation of this framework in Cape Verde  and Madagascar, we interviewed compact country officials and obtained  documentation of procurement procedures.
To determine the form of monitoring and evaluation structures in the  three countries with entry into force, we reviewed the requirements of  MCC compacts and supplemental agreements. We assessed the status of  the country organizations in Madagascar, Cape Verde, and Honduras to  establish these structures by analyzing the staffing and status  information provided to us by MCC. We independently assessed MCC’s  progress in our site visits to Madagascar and Cape Verde. Additionally,  we reviewed the scope of work of the independent U.S. evaluation  contractors retained by MCC, and closely reviewed the monitoring and  evaluation plan for Madagascar—the only plan approved by MCC prior  to April 2006. We also reviewed the plan for Cape Verde and the draft  plan for Honduras. We assessed the adequacy of the Madagascar plan  against the criteria of data quality, and consistency with the economic  model and logic identified in MCC’s due diligence review of projects. We  also applied general principles of economic logic, such as treatment of  uncertainty in data, to assess how uncertainty was incorporated into  MCC’s monitoring and evaluation framework.
To review MCC’s progress in responding to GAO’s April 2005  recommendations (see app. I), we examined MCC documents, such as its  strategic plan, planning documents, policies, procedures, and human  capital documents. In December 2005, MCC provided us with a letter  outlining the steps that the corporation had taken in response to our  recommendations. Using this as a basis for discussion, we held additional  meetings with MCC officials and received additional documentation of  MCC’s responses. We also reviewed the findings of the MCC IG analysis of  the functions of the Corporation and met with the IG to determine the steps  that MCC had taken in response to IG findings related to our  recommendations.
We conducted our review from June 2005 through May 2006 in accordance  with generally accepted government auditing standards.
section_title: Comments from the Millennium Challenge Corporation
paragraphs: The following are GAO’s comments on the Millennium Challenge  Corporation letter dated July 7, 2006.
subsections: section_title: GAO Comments
paragraphs: 1. MCC stated that “MCC's mandate to engage directly with eligible  countries shortly after its creation did not allow for the possibility of  developing and vetting policies and procedures in advance of such  engagement.” We recognize that MCC was simultaneously addressing a  number of issues during its initial years of operation—selecting eligible  countries, setting up MCC as an organization, and developing guidance  and policies while also working with countries to finalize compacts. In  the context of MCC as an evolving organization, we felt it was  important to discuss the evolving nature of guidance to provide a  balanced perspective regarding the process that eligible countries had  to follow to sign compacts. 2. MCC stated that our concern regarding baseline data quality stems  from one survey in Madagascar. However, we found issues with data  reliability in Madagascar with both the Agricultural Productivity Survey  and Household Survey. In the second survey, the error was smaller;  therefore, we focused on the Agricultural Productivity Survey to  illustrate the more significant example.
The measurement of land values is very important for Madagascar's  monitoring and evaluation plan. Both the compact-level goal of  increasing household income and the program objective of increasing  investment in rural Madagascar are measured in terms of land values.  In each of the zones, the expected increase in household income is  estimated at 5 percent of the average land value, and the expected  increase in investment is estimated at 27 percent of the average land  value. With a large error in baseline data, it will be difficult to  accurately track progress toward the compact-level goal and program  objective. 3. MCC noted that not every benefit can be isolated and measured during  implementation, and that disaggregating may not be feasible. We  recognize that not all outcomes of the economic analysis can be  directly tracked with indicators at different levels of monitoring or for  impact evaluation. There are trade-offs between cost and level of detail.  However, as we note in this report, aggregation poses some challenges  that could limit the effectiveness of monitoring and evaluation.
section_title: Comments from the Department of State
paragraphs: The following are GAO’s comments on the Department of State letter dated  July 10, 2006.
subsections: section_title: GAO Comments
paragraphs: 1.
In referring to our discussion about MCC’s guidance on project  coherence, State Department commented that it would be erroneous to  indicate that countries were wholly unaware of the need to design their  compact programs to target compact goals, and that informal guidance  from MCC could have been received at any time. We agree that  countries could have requested informal guidance from MCC.  According to MCC, they did provide verbal guidance to countries  regarding the need to link projects to compact goals.
However, MCC rejected tourism and preschool education projects  proposed by Honduras because they were not linked to the  impediments to growth that emerged from the consultative process.  MCC also rejected projects in Cape Verde because MCC’s due diligence  did not indicate that these projects addressed key constraints in Cape  Verde. Given that these two countries submitted projects that did not  meet project coherence criteria, MCC’s verbal guidance may not have  been sufficient to direct countries to submit proposals meeting these  criteria.
We recognize that MCC was simultaneously addressing a number of  issues during its initial years of operation—selecting eligible countries,  setting up MCC as an organization, and developing guidance and  policies while also working with countries to finalize compacts. In the  context of MCC as an evolving organization, we felt it was important to  discuss the evolving nature of guidance to provide a balanced  perspective regarding the process that eligible countries had to follow  to sign compacts.  2. State noted that we did not detail the methodology and composition of  our Madagascar focus groups. We summarized the composition of the  focus groups in the letter of this report. We met with these 29  individuals, including representatives of MCA-Madagascar and  Madagascar ministries, during the course of our week in Madagascar  and reviewed with them the assumptions used in MCC’s economic  model for its projects. As previously discussed in this report, we found  that these officials were not closely involved in MCC’s economic  analysis. According to MCC and Madagascar officials, an MCC  economist met with four Madagascar representatives in Paris, France,  during compact negotiations and reviewed the model with them at that  time. (MCC officials did travel to Madagascar during due diligence to  assess proposed projects.) When we discussed this with MCC officials,  they stated that not traveling to Madagascar to develop the model was a  mistake, and one that has not been repeated. 3. State commented that we do not rebut MCC’s point that countries’  degree of involvement depended on their capability and willingness to  participate. (See comment 2.)
In its letter, which is reprinted in appendix V, MCC agreed that in some  cases, the level of country engagement on economic analysis could be  improved. MCC also now requires that countries engage an economist  on the core country team, provides examples of the economic rate of  return analysis, and has MCC economists make early technical  guidance trips to work with the core country team and the core country  team economist. These procedures were not in place when MCC  conducted the economic analysis for Madagascar, Cape Verde, or  Honduras.
By not thoroughly involving the countries in this analysis, MCC risks  not meeting its stated principle of focusing on results. In MCC’s results- based framework, the economic model is the basis for determining  performance targets against which funding is conditioned. MCA- Madagascar may find that the targets are unrealistic if the targets have  not been thoroughly discussed with, and understood by, Madagascar  officials. 4. State noted that the time for the in-country MCA organizations to hire  staff compares favorably with general government hiring. Management  unit officials are hired by a compact country’s MCA program steering  committee, not the U.S. government. As such, it is not “general  government hiring” and should not be compared with it.  5. State correctly observed that we did not offer MCC a recommended  alternative to its taking the time to find good candidates for positions.  We did not make a recommendation for addressing this issue because  MCC has adopted a policy implementing the authority given it by  section 609(g) of the Millennium Challenge Act of 2003 to make grants  to facilitate compact development and implementation. MCC’s current  policy includes a provision where, if certain conditions are met, it may  fund an eligible country’s request for “management support payments”  for salaries, rent, and equipment for the country’s team prior to  compact signature. We have added a discussion of this policy in the  letter of this report to clarify this point.
6.
In referring to our discussion about the different maturity of internal  control between Madagascar and Cape Verde, State pointed out that  differences in the maturity of internal control between the countries  are to be expected, given that some countries have historically more  well-developed systems. We agree that different countries will differ in  their maturity of internal control due to historical differences in the  underlying systems and accountability infrastructures. The current  maturity of internal control in each country, however, is a key  consideration in assessing risk and establishing effective oversight  mechanisms to deal with the unique risks in the key financial processes  and activities of each country. 7. State referred to “GAO’s reliance on project evaluation criteria that  make use of randomized controlled trials to measure success.” This  comment misconstrues our findings on the use of randomized  controlled trials. We do not rely on these criteria. We are commenting  on MCC’s use of randomized controlled trials, which MCC has indicated  are its preferred method of impact evaluation, and on the potential  challenges to the use of this approach. State’s comments regarding  cost-benefit analysis would be more appropriately addressed to MCC. 8. State correctly noted that we did not capture the change to one  indicator for fiscal year 2006. We have updated the list of MCC  indicators in footnote 8.  9. State suggested rewording MCC’s expected results to “MCC expects to  permanently raise incomes.…” We did not use the word “permanently”  because in the three compacts that we focused on for this report, it was  not used to describe MCC’s expected results at the country level.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Phil Herr (Assistant Director),  Claude Adrien, Cara Bauer, Gergana Danailova-Trainor, Jeanette Franzel,  Keith Kronin, Reid Lowe, Norma Samuel, Mona Sehgal, and Michael Simon  made key contributions to this report. Also, Tim DiNapoli, David Dornisch,  Etana Finkler, Ernie Jackson, Debra Johnson, Bruce Kutnick, Janice  Latimer, Charlotte Moore, and Celia Thomas provided technical assistance.