Federal Research: Opportunities Exist to Improve the Management and Oversight of Federally Funded Research and Development Centers
Released: 2008-11-07T00:00:00

Letter

Background

During World War II, the U.S. government partnered with academic scientists in ad-hoc laboratories and research groups to meet unique research and development (R&D) needs of the war effort. These efforts resulted in technologies such as the proximity fuse, advanced radar and sonar, and the atomic bomb. Those relationships were later re-structured into federal research centers to retain academic scientists in U.S. efforts to continue advancements in technology, and by the mid-1960’s the term “federally funded research and development centers” was applied to these entities. Since that time, the U.S. government has continued to rely on FFRDCs to develop technologies in areas such as combating terrorism and cancer, addressing energy challenges, and tackling evolving challenges in air travel. For example, one of DOE’s laboratories was used to invent and develop the cyclotron, which is a particle accelerator that produces high energy beams, critical to the field of nuclear physics for the past several decades.

Today, FFRDCs support their sponsoring federal agencies in diverse fields of study. For example, DOE sponsors the most FFRDCs—16 in total—all of which are research laboratories that conduct work in such areas as nuclear weapons, renewable energy sources, and environmental management. DHS recently established two FFRDCs: one to develop countermeasures for biological warfare agents and the other to provide decision makers with advice and assistance in such areas as analysis of the vulnerabilities of the nation’s critical infrastructures, standards for interoperability for field operators and first responders, and evaluating developing technologies for homeland security purposes.

FFRDCs are privately owned but government-funded entities that have long-term relationships with one or more federal agencies to perform research and development and related tasks. Even though they may be funded entirely, or nearly so, from the federal treasury, FFRDCs are regarded as contractors not federal agencies. In some cases, Congress has specifically authorized agencies to establish FFRDCs. For example, the 1991 appropriation for the Internal Revenue Service authorized the IRS to spend up to $15 million to establish an FFRDC as part of its tax systems modernization program.

According to the Federal Acquisition Regulation (FAR), FFRDCs are intended to meet special long-term research or development needs that cannot be met as effectively by existing in-house or contractor resources. In sponsoring an FFRDC, agencies draw on academic and private sector resources to accomplish tasks that are integral to the mission and operation of the sponsoring agency. In order to discharge responsibilities to their sponsoring agencies, the FAR notes that FFRDCs have special access, beyond that which is common for normal contractual relationships, to government and supplier data—including sensitive and proprietary data—and other government resources. Furthermore, the FAR requires FFRDCs to operate in the public interest with objectivity and independence, to be free of organizational conflicts of interest, and to fully disclose their affairs to the sponsoring agencies. FFRDCs may be operated by a university or consortium of universities; other nonprofit organizations; or a private industry contractor as an autonomous organization or a separate unit of a parent organization.

Agencies develop sponsoring agreements with FFRDCs to establish their research and development missions and prescribe how they will interact with the agency; the agencies then contract with organizations to operate the FFRDCs to accomplish those missions. At some agencies the sponsoring agreement is a separate document that is incorporated into the contract, and at other agencies the contract itself constitutes the sponsoring agreement. The sponsoring agreement and contract together identify the scope, purpose, and mission of the FFRDC and the responsibilities of the contractor in ensuring they are accomplished by the FFRDC.

Although the contract or sponsoring agreement may take various forms, the FAR requires FFRDC sponsoring agreements to contain certain key terms and conditions. For example, the agreement term may not exceed 5 years, but can be periodically renewed in increments not to exceed 5 years. Sponsoring agreements must also contain prohibitions against the FFRDCs competing with non-FFRDCs in response to a federal agency request for proposals for other than the operation of an FFRDC. The agreement also must delineate whether and under what circumstances the FFRDC may accept work from other agencies. In addition, these agreements may identify cost elements requiring advance agreement if cost-type contracts are used and include considerations affecting negotiation of fees where fees are determined appropriate by sponsors.

The National Science Foundation (NSF), which keeps general statistics on FFRDCs, identifies the following types of FFRDCs: Research and development (R&D) laboratories: fill voids where in-house and private sector R&D centers are unable to meet core agency needs. These FFRDCs are used to maintain long-term competency in sophisticated technology areas and develop and transfer important new technology to the private sector.

Study and analysis centers: used to provide independent analyses and advice in core areas important to their sponsors, including policy development, support for decision making, and identifying alternative approaches and new ideas on significant issues.

Systems engineering and integration centers: provide support for complex systems by assisting with the creation and choice of system concepts and architectures, the specification of technical system and subsystem requirements and interfaces, the development and acquisition of system hardware and software, the testing and verification of performance, the integration of new capabilities, and continuous improvement of system operations and logistics.

The NSF maintains a master list of the current FFRDCs and collects funding data from their agency sponsors on an annual basis. According to NSF data, R&D funding for FFRDCs has risen steadily across the federal government, increasing 40 percent from fiscal year 1996 to 2005, from $6.9 billion to $9.7 billion. (See fig. 1 below.) This does not represent the full amount of funding provided to FFRDCs by federal agencies, however, since it does not include non-R&D funding. Nevertheless, it is the only centrally reported information on federal funding for FFRDCs.

For a list of the 38 FFRDCs currently sponsored by the U.S. government, see appendix II.

Most Agencies Compete Cost- Reimbursement Contracts for Operating Their FFRDCs, but Some Do Not Have Specific Personal Conflict-of- Interest Requirements

The four agencies we reviewed use cost-reimbursement contracts with the organizations that operate their FFRDCs, and three of these agencies generally use full and open competition in awarding these contracts. While the agencies require that their FFRDCs be free from organizational conflicts of interest in accordance with federal regulations, only DOD and DOE have agencywide requirements that prescribe specific areas that FFRDC contractors must address to ensure their employees are free from personal conflicts of interest. DHS and HHS policies do not specifically prescribe areas that contractors must include to address these conflicts.

Three Agencies Generally Compete FFRDC Contracts, While DOD Does Not

Federal law and regulations require federal contracts to be competed unless they fall under specific exceptions to full and open competition. One such exception is awarding contracts to establish or maintain an essential engineering, research, or development capability to be provided by an FFRDC. While some agencies we reviewed awarded FFRDC contracts through other than full and open competition in the past, including sole-source contracts, three have generally used full and open competition in recent years.

Starting in the mid-1990’s, DOE took steps to improve FFRDC laboratory contractors’ performance with a series of contracting reforms, including increasing the use of competition in selecting contractors for its labs. Subsequent legislation required DOE to compete the award and extension of contracts used at its labs, singling out the Ames Laboratory, Argonne National Laboratory, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, and Los Alamos National Laboratory for mandatory competition because their contracts in effect at the time had been awarded more than 50 years ago. In addition, according to DOE officials, the Los Alamos contract was competed due to performance concerns with the contractor, and Argonne West’s contract was competed to combine its research mission with that of the Idaho National Engineering and Environmental Laboratory to form the Idaho National Laboratory. DOE now routinely uses competitive procedures on contracts for its FFRDC laboratories unless a justification for the use of other than competitive procedures is approved by the Secretary of Energy. Of DOE’s 16 FFRDCs, DOE has used full and open competition in the award of 13 contracts, is in the process of competing one contract, and plans to compete the remaining two contracts when their terms have been completed. For the 13 contracts that have been competed, in 2 cases the incumbent contractor received the new contract award, in 8 cases a new consortium or limited liability corporation was formed that included the incumbent contractor, and in 3 cases a different contractor was awarded the contract.

Other agencies also have used competitive procedures to award FFRDC contracts: HHS has conducted full and open competition on the contract for its cancer research lab since its establishment in 1972, resulting in some change in contractors over the years. Recently, however, HHS noncompetitively renewed the contract with the incumbent contractor. The last time it was competed, in 2001, HHS received no offers other than SAIC-Frederick, which has performed the contract satisfactorily since then. HHS publicly posted in FedBizOpps its intention to noncompetitively renew the operations and technical support contract with SAIC-Frederick for a potential 10-year period. Interested parties were allowed to submit capability statements, but despite some initial interest none were submitted.

DHS competed the initial contract awards for the start up of its two FFRDCs, with the award of the first contract in 2004. DHS plans to compete the award of the next studies and analyses FFRDC contract this year.

In contrast, DOD continues to award its FFRDC contracts on a sole-source basis under statutory exemptions to competition. In the early 1990s, a report by a Senate subcommittee and a Defense Science Board task force both criticized DOD’s management and use of its FFRDCs, including a lack of competition in contract award. This criticism mirrored an earlier GAO observation. GAO subsequently noted in a 1996 report, however, that DOD had begun to strengthen its process for justifying its use of FFRDCs under sole-source contracts for specific purposes. DOD plans to continue its sole-source contracting for the three FFRDC contracts that are due for renewal in 2008 and the six contracts to be renewed in 2010.

Agencies Use Cost- Reimbursement Contracts with Varying Types of Fee Structures, Primarily Funded through Program Offices

All of the FFRDC contracts we reviewed were cost-reimbursement contracts, most of which provided for payments of fixed, award, or incentive fees to the contractor in addition to reimbursement of incurred costs. Fixed fees often are used when, according to the agencies we reviewed, the FFRDC will need working capital or other miscellaneous expense requirements that cannot be covered through reimbursing direct and indirect costs. Fixed fees generally account for a small percentage of the overall contract costs; for fiscal year 2007 fixed fees paid to the FFRDCs we reviewed vary from a low of about 0.1 percent to a high of 3 percent. Award or incentive fees, on the other hand, are intended to motivate contractors toward such areas as excellent technical performance and cost effective management. These types of performance-based fees ranged from 1 to 7 percent at the agencies we reviewed.

Among agencies we reviewed, contract provisions on fees varied significantly: Most DOD contracts are cost-plus-fixed-fee, and DOD, as a general rule, does not provide award or incentive fees to its FFRDCs. DOD’s FFRDC management plan—its internal guidance document for DOD entities that sponsor FFRDCs—limits fees to amounts needed to fund ordinary and necessary business expenses that may not be otherwise recoverable under the reimbursement rules that apply to these types of contracts. For example, the FFRDC operator may incur a one-time expense to buy an expensive piece of needed equipment, but the government’s reimbursement rules require that this expense be recovered over several future years in accordance with an amortization schedule. DOD’s management plan indicates that fees are necessary in such instances to enable the contractor to service the debt incurred to buy the equipment and maintain the cash flow needed for the contractor’s business operations. DOD officials told us they scrutinize these fees carefully and do not always pay them. For example, the contract between DOD and the Massachusetts Institute of Technology (MIT), which operates the Lincoln Laboratory FFRDC, specifies that MIT will not receive such fees.

DOE and DHS use fixed fees, performance-based fees, and award terms, which can extend the length of the contract as a reward for good performance. For example, Sandia Corporation, a private company that operates Sandia National Laboratories, receives both a fixed fee and an incentive fee, which for fiscal year 2007 together amounted to about $23.2 million, an additional 1 percent beyond its estimated contract cost. In addition, Sandia Corporation has received award terms that have lengthened its contract by 10 years.

HHS provides only performance-based fees to the private company that operates its one FFRDC.

Rather than receiving direct appropriations, most FFRDCs are funded on a project-by-project basis by the customers, either within or outside of the sponsoring agency, that wish to use their services by using funds allocated to a program or office. FFRDC contracts generally specify a total estimated cost for work to be performed and provide for the issuance of modifications or orders for the performance of specific projects and tasks during the period of the contract.

Congressional appropriations conferees sometimes directed specific funding for some DHS and DOD FFRDCs in conference reports accompanying sponsoring agencies’ appropriations. For example, although according to DOD officials, 97 percent of its FFRDC funding comes from program or office allocations to fund specific projects, half of its FFRDCs receive some directed amounts specified in connection with DOD’s annual appropriations process. Specifically, for fiscal year 2008, the following DOD FFRDCs received conferee-directed funding in the DOD appropriations conference report: MIT Lincoln Laboratory Research Program, $30 million; the Software Engineering Institute, $26 million; the Center for Naval Analyses, $49 million; the RAND Project Air Force, $31 million; and the Arroyo Center, $20 million. In addition, DOD officials noted that the congressional defense committees sometimes direct DOD’s FFRDCs to perform specific studies for these committees through legislation or in committee reports. In fiscal year 2008, two DOD FFRDCs conducted 16 congressionally requested studies.

All Four Agencies Address Organizational Conflicts of Interest but Vary in Addressing Personal Conflicts of Interest of FFRDC Employees

As FFRDCs may have access to sensitive and proprietary information and because of the special relationship between sponsoring agencies and their FFRDCs, the FAR requires that FFRDC contractors be free from organizational conflicts of interest. In addition, we recently reported that, given the expanding roles that contractor employees play, government officials from the Office of Government Ethics and DOD believe that current requirements are inadequate to address potential personal conflicts of interest of contractor employees in positions to influence agency decisions. While each agency we reviewed requires FFRDC operators to be free of organizational conflicts of interest, DOD and DOE prescribe specific areas that FFRDC contractors must address to ensure their employees are free from personal conflicts of interest.

The FAR states that an organizational conflict of interest exists when because of other interests or relationships, an entity is unable or potentially unable to render impartial assistance or advice to the government or the entity might have an unfair competitive advantage. Because sponsors rely on FFRDCs to give impartial, technically sound, objective assistance or advice, FFRDCs are required to conduct their business in a manner befitting their special relationship with the government, to operate in the public interest with objectivity and independence, to be free from organizational conflicts of interest, and to fully disclose their affairs to the sponsoring agency. Each sponsoring agency we reviewed included conflict-of-interest clauses in its sponsoring agreements with contractors operating their FFRDCs. For example, a DHS FFRDC contract includes a clause that specifically prohibits contractors that have developed specifications or statements of work for solicitations from performing the work as either a prime or first-tier subcontractor.

In addition to organizational conflicts of interest requirements, DOD and DOE have specific requirements for their FFRDC contractors to guard against personal conflicts of interest of their employees. For purposes of this report, a personal conflict of interest may occur when an individual employed by an organization is in a position to materially influence an agency’s recommendations and/or decisions and who—because of his or her personal activities, relationships, or financial interests—may either lack or appear to lack objectivity or appear to be unduly influenced by personal financial interests. In January 2007, the Under Secretary of Defense (Acquisition, Technology, and Logistics) implemented an updated standard conflict-of-interest policy for all of DOD’s FFRDCs that requires FFRDC contractors to establish policies to address major areas of personal conflicts of interest such as gifts, outside activities, and financial interests. The updated policy and implementing procedures now are included in all DOD FFRDC sponsoring agreements and incorporated into the DOD FFRDC operating contracts. This action was prompted by public and congressional scrutiny of a perceived conflict of interest by the president of a DOD FFRDC who then voluntarily resigned. As a result, DOD’s Deputy General Counsel (Acquisition and Logistics) reviewed the conflict of interest policies and procedures in place at each of its FFRDCs and determined that although sponsoring agreements, contracts, and internal policies were adequate, they should be revised to better protect DOD from employee-related conflicts. DOD’s revised policy states that conflicts of interest could diminish an FFRDC’s objectivity and capacity to give impartial, technically sound, objective assistance or advice, which is essential to the research, particularly with regard to FFRDCs’ access to sensitive information. Therefore, the policy provides that FFRDC conflict of interest policies address such issues as gifts and outside activities and requires an annual submission of statements of financial interests from all FFRDC personnel in a position to make or materially influence research findings or recommendations that might affect outside interests.

DOE’s FFRDCs, which operate under management and operating (M&O) contracts—a special FAR designation for government-owned, contractor- operated facilities such as DOE’s—have additional provisions for addressing personal conflicts of interest. The provisions address such areas as reporting any outside employment that may constitute a personal conflict of interest. In addition, the National Nuclear Security Administration (NNSA), which sponsors three of DOE’s FFRDCs, is planning to implement additional requirements in its laboratory contracts later this year requiring contractors to disclose all employee personal conflict of interests, not just outside employment as is currently required. An NNSA procurement official noted that other personal conflict of interests may include any relationship of an employee, subcontractor employee, or consultant that may impair objectivity in performing contract work. NNSA officials stated that it plans to share the policy with the DOE policy office for potential application across the department.

Currently, DHS and HHS policies do not specifically prescribe areas that contractors must include to address employees personal conflicts. However, DHS officials stated that they provided guidance to the two contractors that operate DHS’s FFRDCs to implement requirements to address some of their employees’ personal conflicts with DHS’s interests. In addition, both DHS and HHS FFRDC contractors provide that their staff avoid or disclose financial interests or outside activities that may conflict with the interests of the company. For example, the contractor operating the FFRDC for HHS requires about 20 percent of its employees to report activities that may constitute a conflict with the company’s interests, but allows the bulk of its staff to self-determine when they need to report.

In May 2008, we reported that officials from the Office of Government Ethics expressed concerns that current federal requirements and policies are inadequate to prevent certain kinds of ethical violations on the part of contractor employees, particularly with regard to financial conflicts of interest, impaired impartiality, and misuse of information and authority. The acting director identified particular concerns with such conflicts of interest in the management and operations of large research facilities and laboratories. Our report noted that DOD ethics officials had generally the same concerns. Therefore, we recommended that DOD implement personal conflict-of-interest safeguards—similar to those for federal employees—for certain contractor employees.

Agencies Vary in FFRDC Oversight Approaches and Do Not Regularly Share Best Practices

Sponsoring agencies take various approaches in exercising oversight of their FFRDCs. The agencies determine appropriateness of work conducted by their FFRDCs; perform on-going and annual assessments of performance, costs and internal controls; and conduct comprehensive reviews prior to renewing sponsoring agreements. Each agency develops its own processes in these areas, and no formal interagency mechanisms exist to facilitate the sharing of FFRDC oversight best practices.

Agencies Approve Research Plans and Work Conducted at Their FFRDCs

To ensure work remains within each FFRDCs purpose, mission, scope of effort, and special competency, sponsoring agencies develop and approve annual research plans for the FFRDCs and review and approve FFRDC work assigned on a project-by-project basis. While the majority of each FFRDC’s work is done for its sponsoring agency, FFRDCs may perform work for other institutions, subject to sponsoring agency approval.

Officials at DOD, DOE, and DHS identified the processes they use to develop annual research plans that describe each FFRDC’s research agenda. For example, DHS designates an executive agent to ensure that its FFRDC is used for the agency’s intended purposes. Each year DHS develops a research plan that is reviewed and approved by the executive agent, including any subsequent changes. DHS also uses an Advisory Group to ensure that its FFRDCs produce work consistent with the sponsoring agreement. DOD has a similar mechanism for approving the annual research plan for its Lincoln Laboratory FFRDC. This FFRDC has a Joint Advisory Committee that annually reviews and approves the proposed research plan. Members of this committee include representatives from the various DOD services—e.g., Air Force, Army, and Navy—who are the users of the laboratory’s R&D capabilities. Of the four agencies included in our review, only HHS does not create a separate annual research plan for its FFRDC. Instead, the work at HHS’ FFRDC is guided by the National Cancer Institute’s overall mission, which is described in its annual budgetary and periodic strategic planning documents.

In determining the proposed research plan, DOD must abide by congressionally set workload caps. These caps were imposed in the 1990’s in response to concerns that DOD was inefficiently using its FFRDCs, and therefore, each fiscal year Congress sets an annual limitation on the Staffyears of Technical Effort (STE) that DOD FFRDCs can use to conduct work for the agency. The STE limitations aim to ensure that (1) work is appropriate and (2) limited resources are used for DOD’s highest priorities. Congress also sets an additional workload cap for DOD’s FFRDCs for certain intelligence programs. Once DOD receives from Congress the annual total for STEs, then DOD’s Office of the Undersecretary of Acquisition, Technology and Logistics allocates them across DOD’s FFRDCs based on priorities set forth in the annual research plan developed by each FFRDC. DOD officials observed that while the overall DOD budget has increased about 40 percent since the early 1990s, the STE caps have remained steady, and therefore, DOD must turn aside or defer some FFRDC-appropriate work to subsequent years. Although the majority of work that DOD’s FFRDCs conduct is subject to these limitations, the work that DOD FFRDCs conduct for non-DOD entities is not subject to these caps.

Each sponsoring agency also reviews and approves tasks for individual FFRDC projects to make sure that those tasks (1) are consistent with the core statement of the FFRDC and (2) would not constitute a “personal service” or inherently governmental function. Listed below are examples of procedures used by agencies included in our review to approve tasks for individual projects: DOD sponsors generally incorporate in their sponsoring agreement guidelines for performance of work by the FFRDC. The work is screened at various levels for appropriateness, beginning with FFRDC clients who request the work, then program and contract managers, and then it is reviewed and approved as well by the primary sponsor. In some cases, projects are entered into a computer-based tool, which the Air Force has developed to determine and develop its overall requirements for that year. The tool is intended to assist the Air Force in prioritizing requests for its FFRDC and in ensuring that work requested is in accordance with guidelines and that potential alternative sources have been considered.

DOE FFRDCs must document all DOE-funded projects using work authorizations to help ensure that the projects are consistent with DOE’s budget execution and program evaluation requirements. In addition, DOE uses an independent scientific peer-review approach—including faculty members and executives from other laboratories—at several of its FFRDC laboratories to ensure the work performed is appropriate for the FFRDC and scientifically sound. In some cases, DOE’s Office of Science holds scientific merit competitions between national laboratories (including FFRDCs), universities, and other research organizations for some R&D funding for specific projects.

HHS uses an automated “yellow task” system to determine if work is appropriate for its FFRDC, and several officials must approve requests for work, including the government contracting officer and overseeing project officer for the FFRDC, with reference to a set of criteria. This agency requires a concept review by advisory boards for the various HHS institutes to ensure the concept is appropriate for the FFRDC and meets its mission or special competency.

DHS requires certain officials at its sponsoring office to conduct a suitability review using established procedures for reviewing and approving DHS-sponsored tasks. This review is required under DHS’s Management Directive for FFRDCs.

FFRDCs are required to have their sponsors review and approve any work they conduct for others, and the four agencies included in our review have policies and procedures to do so. FFRDCs may conduct work for others when required capabilities are not otherwise available from the private sector. This work for others can be done for federal agencies, private sector companies, and local and state governments. The sponsoring agency of an FFRDC offers the work for others, with full costs charged to the requesting entity, to provide research and technical assistance to solve problems. At laboratory FFRDCs, work for others can include creating working models or prototypes. All work placed with the FFRDC must be within the purpose, mission, general scope of effort, or special competency of the FFRDC.

Work for others is considered a technology transfer mechanism, which helps in sharing knowledge and skills between the government and the private sector. Under work for others, according to DOD officials and federal regulation, the title to intellectual property generally belongs to the FFRDC conducting the work, and the government may obtain a nonexclusive, royalty-free license to such intellectual property or may choose to obtain the exclusive rights. As required by FAR, sponsoring agreements or sponsoring agencies we reviewed identified the extent to which their FFRDCs may perform work for other than the sponsors (other federal agencies, state or local government, nonprofit or profit organizations, etc.) and the procedures that must be followed by the sponsoring agency and the FFRDC. In addition, according to agency officials FFRDCs have a responsibility to steer inquiries about potential research for other entities to their primary sponsor’s attention for approval. Agency officials stated that they work with their FFRDCs when such situations arise.

DOE’s Office of Science established a “Work for Others Program” for all of its FFRDC laboratories. Under this program, the contractor of the FFRDC must draft, implement, and maintain formal policies, practices, and procedures, which must be submitted to the contracting officer for review and approval. In addition, DOE may conduct periodic appraisals of the contractor’s compliance with its Work for Others Program policies, practices, and procedures. For DOE’s National Nuclear Security Administration (NNSA), officials reported that the work for others process at the Sandia National Laboratories requires DOE approval before the Sandia Corporation develops the proposed statement of work, which is then sent to DOE’s site office for review and approval.

For DHS, each FFRDC includes the work for others policy in its management plan. For example, one management plan states that the FFRDC may perform work for others and that such work is subject to review by the sponsoring agency for compliance with criteria mutually agreed upon by the sponsor and the FFRDC contractor. The DHS FFRDC laboratory director said he routinely approves any work-for-others requests but gives first priority to the DHS-sponsored work. The sponsor for this FFRDC also periodically assesses whether its work for others impairs its ability to perform work for its sponsor.

HHS and DOD also have work-for-others programs for the FFRDCs they sponsor. For example, at HHS’s FFRDC the program is conducted under a bilateral contract between the entity that is requesting the work and the FFRDC to perform a defined scope of work for a defined cost. This agency developed a standard Work for Others Agreement for its FFRDC, the terms and conditions of which help ensure that the FFRDC complies with applicable laws, regulations, policies, and directives specified in its contract with the HHS.

Some agency sponsors report that work for others at their FFRDCs has grown in the past few years. For example, DOE officials said work for others at the Sandia National Laboratories related to nanotechnologies and cognitive sciences has grown in the last 3 years. As shown in table 1, the amount of work for others by FFRDCs since fiscal year 2001 has increased for many of the FFRDCs included in our review.

While funding for work for others has increased, some agencies in our review reported limiting the amount of work for others their FFRDCs conduct. For example, DOE’s Office of Science annually approves overall work-for-others funding levels at its laboratories based on a request from the laboratory and recommendation from the responsible site office. Any work-for-others program that is above 20 percent of the laboratory’s operating budget, or any request that represents a significant change from previous year’s work-for-others program will be reviewed in depth before the approval is provided. Similarly, DOE officials limit commitments to conduct work for others at the National Renewable Energy Laboratory’s to about 10 percent of the laboratory’s total workload.

Agencies Assess FFRDCs’ Performance, Costs, and Internal Controls

In addition to ensuring work is appropriate for their FFRDCs, the four sponsoring agencies in our case study regularly review the contractors’ performance in operating the FFRDCs, including reviewing and approving costs incurred in operations and internal control mechanisms. Agency performance evaluations for FFRDC contractors vary, particularly between those that incorporate performance elements into their contracts and those that do not. Furthermore, contracting officers at each agency regularly review costs to ensure that they are appropriate, in some cases relying on audits of costs and internal controls to highlight any potential issues.

Agencies Review Performance of FFRDC and Operating Contractor

All four agencies conduct at least annual reviews of the performance of their FFRDCs and contractors. At three agencies, the outcomes of these reviews provide the basis for contractors to earn performance-based incentives or awards. Specifically, DOE, HHS, and DHS provide for award fees to motivate contractors toward excellence in high performance, and contractors operating FFRDCs for DOE and DHS may earn additional contract extensions by exceeding performance expectations.

DOE uses a performance-based contracting approach with its FFRDCs, which includes several mechanisms to assess performance. First, DOE requires contractors to conduct annual self-assessments of their management and operational performance. Also, contracting officers conduct annual assessments of the performance of the FFRDC contractor, relying in part on user satisfaction surveys. All of this input contributes to each lab’s annual assessment rating. For example, Sandia National Laboratories, operated by Sandia Corporation (a subsidiary of Lockheed Martin) received an overall rating of “outstanding” for fiscal year 2007 and was awarded 91 percent of its available award fee ($7.6 million of a possible total fee of $8.4 million). DOE noted that Sandia National Laboratories’ scientific and engineering support of U.S. national security was an exceptional performance area. DOE publishes such “report cards” for its laboratories on the internet. DOE includes detailed performance requirements in each contract in a Performance Evaluation and Measurement Plan that is organized by goals, objectives, measures, and targets. The DOE Office of Science mandates that each of its ten FFRDC laboratories establish the same eight goals in each FFRDC’s contractual plan. For example, the Ernest Orlando Lawrence Berkeley National Laboratory, operated by the University of California, received high ratings in providing efficient and effective mission accomplishment and science and technology program management. These ratings resulted in an award of 94 percent or $4.2 million of the total available fee of $4.5 million.

HHS, which also uses performance-based contracting, has identified certain designated government personnel to be responsible for evaluation of the FFRDC contractor. This review process includes different levels of reviews, from coordinators who review performance evaluations to an FFRDC Performance Evaluation Board, which is responsible for assessing the contractor’s overall performance. The board rates each area of evaluation based on an established Performance Rating System to determine the amount of the contractor’s award fee. In fiscal year 2007, the National Cancer Institute at Frederick, operated by Science Applications International Corporation-Frederick (a subsidiary of Science Applications International Corporation), received 92 percent of its available award fee or $6.9 million of a possible $7.4 million.

Similar to the other agencies, DHS regularly conducts performance reviews throughout the life cycle of its FFRDC contract. This includes program reviews as described in the sponsoring agreement, midyear status reviews, technical progress reports, monthly and quarterly reports, and annual stakeholder surveys to ensure the FFRDC is meeting customer needs. DHS also drafts a multiyear improvement plan and collects performance metrics as evidence of the FFRDC’s performance. For fiscal year 2007, Battelle National Biodefense Institute, operating the National Biodefense Analysis and Countermeasures Center, received 82 percent of its performance-based award fee amounting to $1.4 million. According to DHS officials, Analytic Services, Inc., which operates the Homeland Security Institute, received a fixed fee of about 2 percent or approximately $.68 million for fiscal year 2007.

DOD conducts annual performance reviews and other internal reviews, such as conducting periodic program management reviews and annual customer surveys to monitor the performance of its FFRDCs in meeting their customers’ expectations. As part of this review process, major users are asked to provide their perspectives on such factors as the use and continuing need for the FFRDC, and how these users distinguish work to be performed by the FFRDC from work to be performed by others. According to DOD, these performance evaluations provide essential input to help it assess the effectiveness and efficiency of the FFRDC’s operations. Typically the performance reviews obtain ratings from FFRDC users and sponsors on a variety of factors including the quality and value of the work conducted by the FFRDCs, as well as its ability to meet technical needs, provide timely and responsive service, and manage costs.

Agencies Review Costs and Internal Controls

Federal regulations, policies, and contracts establish various cost, accounting, and auditing controls that agencies use to assess the adequacy of FFRDC management in ensuring cost-effective operations and ensure that costs of services being provided to the government are reasonable. Sponsors of the FFRDCs we reviewed employ a variety of financial and auditing oversight mechanisms to review contractors’ management controls, including incurred cost audits, general financial and operational audits, annual organizational audits, and audited financial statements. These mechanisms differ, depending on the agencies involved and the type of organization operating the FFRDCs.

Under cost-reimbursement contracts, the costs incurred are subject to cost principles applicable to the type of entity operating the FFRDC. Most FFRDC contracts we examined include a standard clause on allowable costs that limits contract costs to amounts that are reasonable and in compliance with applicable provisions of the FAR. Under the FAR, contracting officers are responsible for authorizing cost-reimbursement payments and may request audits at their discretion before a payment is made. In addition, when an allowable cost clause is included in a contract, the FAR requires that an indirect cost rate proposal be submitted annually for audit. At DOD, the Defense Contract Audit Agency (DCAA) generally performs both annual incurred cost audits and close-out audits for completed contracts and task orders at the end of an FFRDC’s 5-year contract term. The audit results are included in the comprehensive review of DOD’s continued need for its FFRDCs. DCAA also performs these types of audits for DHS’s FFRDCs. At DOE, the Office of the Inspector General is responsible for incurred cost audits for major facilities contractors. At HHS, officials stated that while the contracting officer for its FFRDC regularly reviews the incurred costs, no audits of these costs have been performed.

Agencies and FFRDC contractors also conduct financial and operational audits in addition to incurred cost audits. DOE relies primarily upon FFRDC contractors’ annual internal audits rather than on third-party monitoring through external audits. These internal audits are designed to implement DOE’s Cooperative Audit Strategy—a program that partners DOE’s Inspector General with contractors’ internal audit groups to maximize the overall audit coverage of M&O contractors’ operations and to fulfill the Inspector General’s responsibility for auditing the costs incurred by major facilities contractors. This cooperative audit strategy permits the Inspector General to make use of the work of contractors’ internal audit organizations to perform operational and financial audits, including incurred cost audits, and to assess the adequacy of contractors’ management control systems. DHS and DOD generally rely on audits performed by those agencies, a designated audit agency, or an accounting firm, though their FFRDC contractors usually perform some degree of internal audit or review function as part of their overall management activity.

In addition, all nonprofits and educational institutions that annually expend more than $500,000 in federal awards—including those that operate FFRDCs—are subject to the Single Audit Act which requires annual audits of: (1) financial statements, (2) internal controls, and (3) compliance with laws and regulations. We have previously reported these audits constitute a key accountability mechanism for federal awards and generally are performed by independent auditors. At DOD, for example, DCAA participates in single audits normally on a “coordinated basis”—at the election of the organization being audited—with the audited organization’s independent public accountant. The financial statements, schedules, corrective action plan, and audit reports make up the single audit package, which the audited organization is responsible for submitting to a federal clearing house designated by OMB to receive, distribute, and retain. DOD’s Office of Inspector General, for example, as a responsible federal agency, receives all single audit submissions for nonprofits and educational institutions that operate DOD’s FFRDCs. These audit results are employed by DOD as partial evidence of its FFRDCs’ cost- effectiveness and incorporated in the 5-year comprehensive reviews. These annual single audits for nonprofit and educational FFRDC contractors are a useful adjunct to other cost, accounting, and auditing controls discussed previously, designed to help determine contractor effectiveness, efficiency, and accountability in the management and operation of their FFRDCs.

Private contractors that publicly trade their securities on the exchanges— including those that operate FFRDCs—are registered with the Securities and Exchange Commission (SEC) and are required to file audited financial statements with the SEC. These audited statements must be prepared in conformity with generally accepted accounting principles (GAAP) and securities laws and regulations, including Sarbanes-Oxley, that address governance, auditing, and financial reporting. These financial statements are designed to disclose information for the benefit of the investing public, not to meet government agencies’ information needs. Accordingly, SAIC and Lockheed—private contractors that manage National Cancer Institute at Frederick and Sandia National Laboratories respectively—prepare audited financial statements for their corporate entities, but do not separately report information on their individual FFRDCs’ operations.

Finally, even though financial statements are not required by university and nonprofit sponsored FFRDCs, some of the FFRDCs in agencies we reviewed have audited financial statements prepared solely for their own operations. DOD’s Aerospace and DHS’s HSI and NBACC are examples. Most others’ financial operations, however, are included in the audited financial statements of their parent organizations or operating contractor. Some, like MITRE, which manages not only DOD’s C3I FFRDC but also two others (one for the Federal Aviation Administration and one for the Internal Revenue Service), provides supplemental schedules, with balance sheets, revenues and expenses, and sources and uses of funds for all three FFRDCs. Others, like the Institute for Defense Analyses, which also operates two other FFRDCs in addition to the Studies and Analyses Center for DOD, provide only a consolidated corporate statement with no information on specific FFRDCs.

Agencies Periodically Rejustify Their Sponsorship of FFRDCs

The FAR requires that a comprehensive review be undertaken prior to extending a sponsoring agreement for an FFRDC. We found that the four agencies in our case study were conducting and documenting these reviews, but noted that implementation of this requirement by each agency is based on its own distinct management policies, procedures, and practices.

During the reviews prior to agreement renewal, sponsoring agencies should include the following five areas identified by the FAR examination of the continued need for FFRDC to address its sponsor’s technical needs and mission requirements; consideration of alternative sources, if any, to meet those needs; assessment of the FFRDC’s efficiency and effectiveness in meeting the sponsor’s needs, including objectivity, independence, quick response capability, currency in its field(s) of expertise, and familiarity with the sponsor; assessment of the adequacy of FFRDC management in ensuring a cost- determination that the original reason for establishing the FFRDC still exists and that the sponsoring agreement is in compliance with FAR requirements for such agreements.

DOD sponsoring offices begin conducting detailed analyses for each of the five FAR review criteria approximately 1 to 2 years in advance of the renewal date. As DOD has received criticism in the past for its lack of competition in awarding FFRDC contracts, it now conducts detailed and lengthy comprehensive reviews prior to renewing FFRDC sponsoring agreements and contracts with incumbent providers. DOD’s FFRDC Management Plan lays out procedures to help provide consistency and thoroughness in meeting FAR provisions for the comprehensive review process. DOD procedures require, and the comprehensive reviews we examined generally provided, detailed examinations of the mission and technical requirements for each FFRDC user, and explanations of why capabilities cannot be provided as effectively by other alternative sources. For example, DOD convened a high level, independent Technical Review Panel to review whether Lincoln Laboratory’s research programs were within its mission as well as whether the research was effective, of high technical quality, and of critical importance to DOD. The panel— composed of a former Assistant Secretary of the Air Force, a former president of another FFRDC, former senior military officers, and a high level industry representative—found that no other organizations had the capacity to conduct a comparable research program. In addition, DOD sponsors use information from annual surveys of FFRDC users that address such performance areas as cost effectiveness and technical expertise. Determinations to continue or terminate the FFRDC agreement are made by the heads of sponsoring DOD components (e.g., the Secretary of the Army or Air Force) with review and concurrence by the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics.

DOE has a documented comprehensive review process that explicitly requires DOE sponsors to assess the use and continued need for the FFRDC before the term of the agreement has expired. DOE’s process requires that the review be conducted at the same time as the review regarding the decision to extend (by option) or compete its FFRDC operating contract. According to DOE’s regulation, the option period for these contracts may not exceed 5 years and the total term of the contract, including any options exercised, may not exceed 10 years. DOE relies on information developed as part of its annual performance review assessments as well as information developed through the contractor’s internal audit process to make this determination. The comprehensive review conducted prior to the most recent award of the contract to operate Sandia National Laboratories concluded that the FFRDC’s overall performance for the preceding 6 years had been outstanding. The Secretary of Energy determined that the criteria for establishing the FFRDC continued to be satisfied and that the sponsoring agreement was in compliance with FAR provisions.

At DHS, we found that its guidance and process for the comprehensive review mirror many aspects of the DOD process. DHS has undertaken only one such review to date, which was completed in May 2008. As of the time we completed our work, DHS officials told us that the documentation supporting the agency’s review had not yet been approved for release.

HHS—in contrast to the structured review processes of the other agencies—relies on the judgment of the sponsoring office’s senior management team, which reviews the need for the continued sponsorship of the FFRDC and determines whether it meets the FAR requirements.

Agency officials stated that this review relies on a discussion of the FFRDC’s ability to meet the agency’s needs within the FAR criteria, but noted there are no formal procedures laid out for this process. The final determination is approved by the director of the National Cancer Institute and then the director of the National Institutes of Health.

No Formal Interagency Mechanisms Exist for Sharing of Best Practices for Overseeing FFRDCs

Some agencies have used the experiences of other agencies as a model for their own oversight of their FFRDCs. There is no formal mechanism, however, for sharing of best practices and lessons learned among sponsoring agencies.

DHS officials have adopted several of DOD’s and DOE’s policies and procedures for managing FFRDCs to help their newly created FFRDCs gain efficiencies. DHS mirrored most of DOD’s FFRDC Management Plan, and officials have stated that the STE limitations for DOD could be a potentially useful tool for focusing FFRDCs on the most strategic and critical work for the agency. Also, DHS officials stated they have made use of DOE’s experience in contracting for and overseeing the operation of its laboratories, such as including a DOE official in the DHS process to select a contractor to operate its laboratory FFRDC. In addition, HHS officials said they are incorporating the DOE Blue Ribbon Report recommendation to set aside a portion of the incentive fee paid on their FFRDC contract to reward scientific innovations or research. The idea for the new contract is to base 80 percent of the available award fee in a performance period on operations and use the final 20 percent to reward innovation. HHS also may adopt the technique used by DOE of providing for contract extensions on the basis of demonstrated exceptional performance.

To take advantage of others’ experiences, some FFRDCs sponsored by particular agencies have formed informal groups to share information. For example, DOD’s FFRDCs have formed informal groups at the functional level—Chief Financial Officers, Chief Technology Officers, and General Counsels—which meet periodically to share information on issues of common concern. In addition, the security personnel from the DOD FFRDC contractors meet once a year to discuss security and export control related issues. The contractor officials at Sandia National Laboratories said they share best practices for operating DOE’s laboratory FFRDCs at forums such as the National Laboratory Improvement Council. This Council was also mentioned in a DOE review of management best practices for the national laboratories as one of the few groups that deliberate a broader and more integrated agenda among laboratories.

Despite these instances of information sharing within agencies and the acknowledgment by some officials of potential benefits in such knowledge sharing, no formal mechanisms exist for sharing information across agencies that sponsor and oversee FFRDCs. We reported in 2005 that federal agencies often carry out related programs in a fragmented, uncoordinated way, resulting in a patchwork of programs that can waste scarce funds, confuse and frustrate program customers, and limit the overall effectiveness of the federal effort. The report suggested frequent communication across agency boundaries can prevent misunderstandings, promote compatibility of standards, policies, and procedures, and enhance collaboration. For example, the Federal Laboratory Consortium for Technical Transfer was created to share information across national laboratories. This includes the FFRDC laboratories, but not the other types of FFRDCs. Some agency officials stated that there would be benefits to sharing such best practices.

Conclusions

All federal agencies that sponsor FFRDCs are subject to the same federal regulations, and each agency included in our review has developed its own processes and procedures to ensure compliance and conduct oversight of its FFRDCs. For the most part the differences in approaches are not of great consequence. In at least one key area, however, the different approaches have the potential to produce significantly different results. Specifically, while all FFRDCs are required to address organizational conflicts of interest, only DOD and DOE have requirements that their FFRDC contractors address specific areas of personal conflicts of interest of their employees. In light of the special relationship that FFRDCs have with their sponsoring agencies, which often involves access to sensitive or confidential information, it is critical not only that the FFRDC as an entity but also that employees of the entity in positions to make or influence research findings or agency decision making be free from conflicts. Lacking such safeguards, the FFRDC’s objectivity and ability to provide impartial, technically sound, objective assistance or advice may be diminished. The two agencies with the most experience sponsoring FFRDCs have recognized this gap and have taken steps to address personal conflicts of interest. These steps are consistent with our recent recommendation to DOD that highlighted the need for personal conflicts- of-interest safeguards for certain contractor employees. The other agencies included in our review of FFRDCs could benefit from additional protections in the area of personal conflicts of interest. Currently, although DHS and HHS have policies that generally require their FFRDC contractors to implement such safeguards, they lack the specificity needed to ensure their FFRDC contractors will consistently address employees’ personal conflicts of interest.

Conflict-of-interest requirements is only one of several areas where agencies that sponsor FFRDCs can learn from each other. Other areas include the use of effective and efficient oversight mechanisms such as incentive and award fees, obtaining competition, and conducting comprehensive reviews. In the absence of established knowledge-sharing mechanisms, however, agencies may be missing opportunities to enhance their management and oversight practices. Sharing knowledge among agencies that sponsor FFRDCs, as has been done informally in some instances, could help to ensure that agencies are aware of all the various tools available to enhance their ability to effectively oversee their FFRDCs.

Recommendations for Executive Action

To ensure that FFRDC employees operate in the government’s best interest, we recommend that the Secretary of Homeland Security revise agency policies to address specific areas for potential personal conflicts of interest for FFRDC personnel in a position to make or materially influence research findings or agency decision making; and that the Secretary of Health and Human Services review agency policy regarding personal conflicts of interest for its sponsored FFRDC and revise as appropriate to ensure that this policy addresses all personnel in a position to make or materially influence research findings or agency decision making.

To improve the sharing of oversight best practices among agencies that sponsor FFRDCs, we recommend that the Secretaries of Energy, Defense, Homeland Security, and Health and Human Services, which together sponsor the vast majority of the government’s FFRDCs, take the lead in establishing an ongoing forum for government personnel from these and other agencies that sponsor FFRDCs to discuss their agencies’ FFRDC policies and practices. Areas for knowledge sharing could include, for example, implementing personal conflicts of interest safeguards and processes for completing the justification reviews prior to renewing sponsoring agreements, among others.

Agency Comments and Our Evaluation

The Departments of Health and Human Services and Homeland Security concurred with our recommendation that they revise their conflict of interest policies. In addition, the departments of Defense, Energy, and Homeland Security all concurred with our recommendation to establish a forum to share best practices, while HHS is considering participation in such a forum. We received letters from Defense, Energy, and Health and Human Services, which are reprinted in appendixes III, IV, and V, respectively. In addition, the departments of Health and Human Services and Homeland Security provided technical comments, which we incorporated where appropriate.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this report. We then will provide copies of this report to the Secretaries of Defense, Energy, Health and Human Services and Homeland Security and other interested parties. In addition, this report will be made available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact us at (202) 512-4841 or woodsw@gao.gov or (202) 512-9846 or mittala@gao.gov. Key contributors to this report are acknowledged in appendix VI.

Appendix I: Objectives, Scope, and Methodology

To conduct this review, we chose a nongeneralizable sample of four of the nine federal agencies that sponsor FFRDCs: the departments of Energy (DOE) and Defense (DOD) have the longest histories in sponsoring federally funded research and development centers (FFRDCs) and sponsor the most—16 and 10, respectively; the Department of Homeland Security (DHS) has the 2 most recently established FFRDCs; the Department of Health and Human Services (HHS) has 1 FFRDC laboratory. From the collective 29 FFRDCs that those four agencies sponsor, we selected a nongeneralizable sample of 8 FFRDCs that represented variation among the type of operating contractor, including some operated by universities, some by nonprofits, and some by private industry. Within DOD and DHS, we chose FFRDCs that represent the variation among types these two agencies sponsor, while DOE and HHS only sponsor laboratory type FFRDCs. See appendix II for the FFRDCs included in our case study.

To identify sponsors’ contracting and oversight methods at the four agencies in our case study, we interviewed federal department officials at each office that sponsors FFRDCs as well as offices that have contractor management roles and audit roles: (1) DOE’s Office of Science, National Nuclear Security Administration, Office of Energy Efficiency and Renewable Energy, Office of Environmental Management, Office of Nuclear Energy, and Office of Inspector General; (2) DOD’s departments of the Navy, Air Force, and Army; Office of the Secretary of Defense; Office of Acquisition, Technology, and Logistics; Defense Contract Audit Agency; and the Defense Contract Management Agency; (3) HHS’s National Institutes of Health, National Cancer Institute, and National Institute of Allergy and Infectious Diseases; and (4) DHS’s Directorate for Science and Technology. In addition, we obtained and analyzed federal and agency policies and guidance, contracts for the FFRDCs in our case studies and other supporting documentation such as performance and award fee plans, sponsoring agreements (when separate from contracts), and a variety of audits and reviews. While we did not assess the effectiveness of or deficiencies in specific agencies’ controls, we reviewed agency documentation on incurred cost audits, general auditing controls, single audits, and audited financial statements. We also obtained and analyzed funding data from sponsoring agencies as well as from the National Science Foundation (NSF), which periodically collects and reports statistical information regarding FFRDCs, such as their sponsors, category types, contractors, and funding. While we did not independently verify the data for reliability, we reviewed the NSF's methodology and noted that it reports a 100 percent response rate, no item nonresponse, and no associated sampling errors.

For FFRDCs in our case study, we conducted on-site visits, interviewed key contractor administrative personnel, and obtained information and documentation on how they meet sponsoring agencies’ research needs and adhere to policy guidance. We observed examples of the types of research the FFRDCs conduct for their sponsors and obtained and analyzed documentation such as contractor ethics guidance and policies, performance plans, and annual reports.

To obtain the perspective of the government contracting community, we met with high-level representatives of the Professional Services Council, a membership association for companies that provide services to the U.S. federal government.

Appendix II: List of 38 Federally Funded Research and Development Centers

El Segundo, Calif.

RAND Corp.

Santa Monica, Calif.

MITRE Corp.

Bedford, Mass., and McLean, Va.

Alexandria, Va.

Institute for Defense Analyses Studies and Analyses Center Alexandria, Va.

Institute for Defense Analyses Communications and Computing Center Alexandria, Va.

Lexington, Mass.

RAND Corp.

Santa Monica, Calif.

RAND Corp.

Santa Monica, Calif.

Pittsburgh, Penn.

Argonne, Ill.

Upton, N.Y.

Brookhaven Science Associates, Inc.

Ernest Orlando Lawrence Berkeley National Laboratory Berkeley, Calif.

Universities Research Association, Inc.

Batavia, Ill.

Livermore, Calif.

Golden, Colo.

Midwest Research Institute; Battelle Memorial Institute; Bechtel National, Inc.

Oak Ridge, Tenn.

Richland, Wash.

Princeton, N.J. (subsidiary of Lockheed Martin Corp.)

Westinghouse Savannah River Co.

Aiken, S.C.

Stanford, Calif.

Newport News, Va.

National Cancer Institute at Frederick Frederick, Md. (wholly owned subsidiary of Science Applications International Corp)

Analytic Services, Inc.

Arlington, Va.

National Biodefense Analysis & Countermeasures Center Frederick, Md.

National Aeronautics and Space Administration Pasadena, Calif.

National Astronomy and Ionosphere Center Arecibo, P.R.

Boulder, Colo.

Association of Universities for Research in Astronomy, Inc.

Tucson, Ariz.

Associated Universities, Inc.

Charlottesville, Va.

Science and Technology Policy Institute Washington, D.C.

Center for Nuclear Waste Regulatory Analyses San Antonio, Tex.

MITRE Corp.

McLean, Va.

MITRE Corp.

McLean, Va.

Appendix III: Comments from the Department of Defense

Appendix IV: Comments from the Department of Energy

Appendix V: Comments from the Department of Health and Human Services

Appendix VI: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the individuals named above, key contributors to this report were John Neumann, Assistant Director; Cheryl Williams, Assistant Director; Sharron Candon; Suzanne Sterling; Jacqueline Wade; and Peter Zwanzig.