Federally Funded Research Centers: Agency Reviews of Employee Compensation and Center Performance
Released: 2014-09-10T00:00:00

Letter

Background

FFRDCs perform tasks such as research and development, systems engineering, and technical studies and analyses on behalf of sponsoring federal agencies. FFRDCs arose from partnerships between the federal government and academic researchers and scientists during World War II. Those partnerships were later restructured into federal research centers to retain scientists and became known as FFRDCs by the mid- 1960s. Since that time, FFRDCs have continued to support the research and development needs of various federal agencies. As of April 2014, there were 40 FFRDCs sponsored government-wide. A list of these FFRDCs and their federal sponsors, including those sponsored by DOE, DOD, and NSF that are within the scope of our review, is shown in figure 1.

The 16 FFRDCs that DOE sponsors are overseen by different program offices within the Department. Ten of these FFRDCs, including ORNL, are overseen by the Office of Science; three by the National Nuclear Security Administration (NNSA); one by the Office of Environmental Management; one by the Office of Nuclear Energy; and one by the Office of Energy Efficiency and Renewable Energy. DOE’s Office of Management manages department-wide policies relating to DOE’s FFRDCs, and DOE program offices control many of the decisions associated with the operations of the FFRDCs, including their budgets, structures, and missions. Federal officials located at or near each FFRDC site directly oversee the operations of the FFRDCs.

The 10 FFRDCs that DOD sponsors are managed by various military departments or divisions of DOD. The Under Secretary of Defense for Acquisition, Technology and Logistics is responsible for setting DOD-wide policies for FFRDCs, allocating staff years within an annual ceiling set by Congress at the FFRDCs, and approving renewals of FFRDC contracts, among other things. The Under Secretary carries out these duties through the Office of Acquisition Resources and Analysis. The DOD divisions that oversee FFRDCs are responsible for creating procedures to monitor the value, quality, and responsiveness of FFRDC work and ensuring appropriate contract support. The Deputy Assistant Secretary for Defense for Systems Engineering oversees NSEC, with assistance from Army and Air Force subsponsors. Two additional DOD divisions may provide administrative support for DOD contracts, including those for FFRDCs. The Defense Contract Management Agency (DCMA), a DOD component, is responsible for providing contract administration services for DOD buying activities, working directly with defense contractors to help ensure that goods and services are delivered on time, at projected cost, and that they meet performance requirements. The Defense Contract Audit Agency (DCAA), another DOD component, audits projected and actual costs associated with DOD contracts to ensure they are allowable, allocable, and reasonable in accordance with cost accounting standards and FAR rules.

The four FFRDCs that NSF sponsors are overseen by the Office of Budget, Finance, and Award Management and the divisions whose objectives the FFRDCs primarily support. For example, NSF’s Division of Atmospheric and Geospace Sciences oversees NCAR, with assistance from the other NSF offices. Of NSF-sponsored FFRDCs, three, including NCAR, are established through cooperative agreements rather than contracts as was done for the other 27 FFRDCs we reviewed. The remaining NSF-sponsored FFRDC, the Science and Technology Policy Institute, is sponsored by NSF on behalf of the White House Office of Science and Technology Policy (OSTP), and it provides analysis on significant domestic and international science and technology policies and developments for OSTP and other federal agencies.

NSF also maintains a master list of all FFRDCs and reports data on research and development expenditures at FFRDCs. Specifically, NSF collects data on funds expended for activities specifically organized to produce research outcomes. These data focus on research activities and may not include all expenditures, such as those associated with administering the FFRDC or construction or renovation of facilities.

FFRDCs are defined through the sponsoring agreement between the agency and the contractor retained to operate the FFRDC. At some agencies, the sponsoring agreement is a separate document that is incorporated into the contract or cooperative agreement and, at other agencies, the contract or cooperative agreement itself constitutes the sponsoring agreement. Historically, contractors have been retained for many years or decades as the operator of an FFRDC. Although FFRDCs are entirely federally funded, or nearly so, the organization operating the FFRDC is regarded as a contractor and not an agency of the United States. The contractor continues to exercise primary control over its business concerns, such as its personnel policies and compensation.

Contractors can charge a number of costs to federal government contracts, including certain types and levels of compensation costs. Some types of compensation costs are considered allowable under government contracts, while others are considered unallowable. For example, compensation in the form of bonuses are allowable under certain conditions, while compensation in the form of stock options are unallowable. In order to be allowable, costs must meet certain requirements, such as being reasonable and in compliance with the terms of the contract and generally accepted accounting principles and practices. In determining reasonableness, the FAR directs agencies to consider if the cost exceeds that which would be incurred by a prudent person in the conduct of competitive business, such as the extent to which a contractor employee’s compensation is consistent with other companies that are similar in size, industry, geographic area, and/or other factors. In addition, federal law caps the allowable total compensation for certain positions for which federal agencies may reimburse contractors. As shown in figure 2, the statutory cap on compensation reimbursements applies to pay (e.g., salaries, wages, bonuses, and other cash incentives) and some nonpay benefits (e.g., pensions and deferred compensation benefits) under contracts. Federal agencies may reimburse contractors for all or a portion of these costs, as long as the costs are allowable and the total reimbursement does not exceed the cap. The cap does not apply to other nonpay benefits, such as health insurance or other fringe benefits. FFRDC contractors, like other contractors, may provide compensation to their employees at levels they choose, but such compensation may be unallowable for reimbursement by federal sponsors.

From 1998 to 2012, the Office of Federal Procurement Policy (OFPP) calculated the cap amount annually based on the amount of compensation provided to the five most highly compensated employees in management positions at all publicly-owned U.S. companies with annual sales over $50 million. Over the 3-year period from fiscal year 2010 through fiscal year 2012, the cap rose from $693,951 in 2010 to $763,029 in 2011, and then to $952,308 in 2012. In 2013, Congress effectively fixed the cap for preexisting contracts to the fiscal year 2012 amount, and reduced the cap amount for new contracts to $487,000 (see fig. 3). Contracts awarded prior to June 24, 2014, are subject to the annual cap calculated by OFPP. Contracts awarded on or after June 24, 2014, are subject to a lower cap of $487,000. In addition, Congress amended the cap in 2011 to include all defense agency contractor employees and again in 2013 to include all civilian contractor employees instead of the top five most highly paid contractor employees—senior executives.

The agencies that sponsor FFRDCs are also subject to the federal standards for internal control that state that agencies should collect information needed to determine whether the agency is complying with laws and regulations—such as the cap on executive compensation—and clearly document transactions and other significant events and make such documentation readily available. The Standards for Internal Control in the Federal Government provide a framework for establishing and maintaining internal control and identifying and addressing major performance and management challenges, and internal controls comprise the plans, methods, and procedures used by agencies to meet missions, goals, and objectives.

Furthermore, federal agencies, including those that that sponsor FFRDCs, are subject the Government Performance and Results Act of 1993 (GPRA) and the GPRA Modernization Act of 2010. These acts require agencies to develop long-term and annual goals and measures and report on progress toward achieving those goals on an annual basis. We have reported that agencies face long-standing difficulties measuring the performance of various types of federal programs and activities, including contracts, grants, and research and development. Whereas federal agencies are subject to the performance reporting requirements of GPRA, as specified by the Office of Management and Budget (OMB), FFRDC contractors are not subject to any generally applicable, post- award reporting requirements. The reporting requirements for FFRDC contractors are specified in the applicable contracts or cooperative agreements between the sponsoring agency and FFRDC contractor.

The FAR establishes certain requirements that apply when agencies enter into contracts to sponsor FFRDCs. For example, while FFRDC contractors primarily perform work for the sponsoring agency, they may also enter into additional contracts or agreements to perform work for other entities. In such cases, the FAR requires that the contract include provisions allowing the sponsoring agency to approve that the additional work is within the purpose, mission, general scope of effort, or special competency of the FFRDC. In addition, prior to extending a contract or sponsoring agreement for an FFRDC, the FAR requires that the federal sponsor conduct a comprehensive review of the use and need for the FFRDC at least every 5 years. The FAR describes elements of that review to include the following: 1. an examination of the sponsor’s special technical needs and missions requirements that are performed by the FFRDC; 2. consideration of alternative sources; 3. an assessment of the efficiency and effectiveness of the FFRDC in meeting the sponsor’s needs; 4. an assessment of the adequacy of the FFRDC management in ensuring a cost effective operation; and 5. a determination that the sponsoring agreement is in compliance with other FAR requirements concerning FFRDCs (e.g., ensuring that the sponsoring agreement or contract contains a statement of purpose, provisions for termination, and prohibition on competition).

The FAR further requires that the head of the sponsoring agency approve or terminate sponsorship based on the results of the comprehensive review conducted in accordance with these elements. Because the elements are not required by the FAR, agencies have flexibility to determine the content of the comprehensive review and its documentation.

FFRDCs Sponsored by DOE Received More Total Funding Than Those Sponsored by DOD and NSF

The 30 FFRDCs sponsored by DOE, DOD, and NSF received nearly $84 billion in total funding for fiscal years 2008 through 2012. The 16 FFRDCs sponsored by DOE received more funding than the FFRDCs sponsored by the other two agencies and performed more work for entities other than the sponsoring agency. Specifically, DOE-sponsored FFRDCs received about 79 percent of the nearly $84 billion in total funding for FFRDCs sponsored by these agencies for fiscal years 2008 through 2012, according to our analysis of agency reports of contractor and agency data provided in response to our survey of FFRDC sponsors. As shown in figure 4, during this period, the 16 DOE-sponsored FFRDCs received about $66 billion in total funding, the 10 DOD-sponsored FFRDCs received about $15.7 billion in total funding, and the 4 NSF- sponsored FFRDCs received about $1.8 billion in total funding. Each year, DOE, DOD, and NSF collectively obligated between $12 and $15 billion to the 30 FFRDCs they sponsored over this period.

As shown in table 1, DOE obligated about 34 percent of its available budget authority toward the FFRDCs it sponsored, while DOD and NSF dedicated less than 1 percent and 4 percent, respectively, based on obligations as a percent of budget authority from fiscal year 2008 through fiscal year 2012. With regard to the significant share of DOE’s budget being obligated to its FFRDCs, DOE officials noted that for many decades the FFRDCs sponsored by the Department have been the primary means for carrying out its research and development mission. While FFRDCs represent a smaller proportion of the overall budget of DOD and NSF, in some cases they represent a larger proportion of individual office budgets at those agencies. For example, over one-third of the budget for NSF’s Division of Atmospheric and Geospace Sciences is used to support NCAR.

Most FFRDCs in our review received funding from entities other than their sponsoring agency, and DOE-sponsored FFRDCs received more funds from other entities than FFRDCs sponsored by DOD and NSF. For fiscal years 2008 through 2012, FFRDCs sponsored by DOE, DOD, and NSF received approximately $15 billion of the nearly $84 billion in total funding from sources other than their sponsoring agency—specifically from other federal agencies (about $13 billion) and nonfederal entities, such as state or local governments, and private entities (about $1.9 billion). Figure 5 shows the total funding reported at each of the 30 FFRDCs for fiscal years 2008 through 2012 and the amounts received from the sponsoring agency, other federal agencies, and nonfederal entities.

Many FFRDCs Sponsored by DOE, DOD, and NSF Spent Over Half of Their Total Funding on Compensation, and These Agencies Have Processes to Review FFRDC Contractor Compensation

Many FFRDCs sponsored by DOE, DOD, and NSF spent over half of their total funding on compensation for fiscal years 2010 through 2012, and sponsoring agencies have processes to review compensation annually to ensure that FFRDC contractors were not reimbursed in excess of the cap amount set in federal law. These agencies also have processes to review compensation more generally for all contractor employees.

Compensation Costs Generally Represented Half or More of Total FFRDC Funding

For fiscal years 2010 through 2012, 20 of 30 FFRDCs sponsored by DOE, DOD, and NSF spent over half of their annual funding on compensation, based on our analysis of agency and contractor data provided in response to our survey of FFRDC sponsors. According to these data, compensation costs ranged from 27 to 79 percent of annual funding at the FFRDCs sponsored by these agencies over this period. In total, these costs provided about 71,000 full-time equivalents (FTE) per year in fiscal years 2010 through 2012 at the 30 FFRDCs sponsored by these three agencies. In 2012, the average compensation cost for each FTE across the 30 FFRDCs was $135,694. In comparison, we previously reported that total government-wide compensation for federal employees for each FTE position averaged $116,828 in 2012. The average compensation cost for each FTE at a single FFRDC ranged from $88,000 to $177,000 per year. FFRDC sponsors also reported that the maximum salary received by executives at these FFRDCs ranged between $200,000 and $600,000, as shown in figure 6. (App. II, table 6, provides more information on compensation costs as a percentage of total FFRDC funding, the compensation cost per FTE, and average FTEs across fiscal years 2010 through 2012 at the 30 FFRDCs we reviewed.)

DOE, DOD, and NSF Have Processes to Review Compensation Reimbursements for Senior Executives and other FFRDC Contractor Employees

DOE, DOD, and NSF have processes to annually review compensation reimbursements for senior executives against the federal statutory cap and conduct general reviews of compensation for other employees.

DOE relies on its contracting officers to review senior executive compensation reimbursements against the federal statutory cap, and, until May of 2014, DOE generally did not document the total reimbursed compensation subject to the cap for these senior executives. DOE has additional processes in place to review compensation for all FFRDC contractor employees.

Reviews of Senior Executive Compensation Reimbursements Against the Statutory Cap DOE contracting officers evaluate the reimbursed compensation of senior executives to assess the applicability of the cap, and these evaluations differ for two groups of senior executives at its FFRDCs—(1) laboratory directors and (2) other executives—based on our discussions with DOE officials and review of DOE policy on executive compensation. Prior to May 2014, contracting officers evaluated the reimbursed portions of senior executives’ pay and nonpay benefits separately, and DOE officials told us that, in general, they did not document the total subject to the cap, except at the Savannah River National Laboratory (SRNL). For the laboratory directors, contracting officers evaluated reimbursable pay annually by approving a required form submitted by the contractor that documents the laboratory directors’ pay—salary and any bonuses or incentives—and any proposed changes. For other FFRDC senior executives, contracting officers evaluated reimbursable pay by approving the form documenting salary, bonuses, and incentives only at the start of the contract or when those individuals are hired. Thereafter, contracting officers evaluated pay for these senior executives on an annual basis through approval of a plan describing the contractor’s request for reimbursement for pay for all FFRDC employees, called the compensation increase plan. This plan describes pay for specific groups and does not describe the salaries, bonuses, or other incentives received by individual contractor senior executives. For example, ORNL’s compensation increase plan describes pay for its senior executives within a group of managers. Regarding nonpay benefits for senior executives, DOE officials told us that the only nonpay benefits subject to the cap that they receive are certain pension plans. Costs for these plans are based on a percentage of the executive’s salary and the plans are approved by the contracting officers. According to these officials, since the contracting officers approved reimbursable pay and the pension plans, they could calculate the laboratory directors’ total reimbursed compensation subject to the cap and estimate it for other executives.

In May 2014, DOE updated its policy on senior executive compensation to require documentation of all the laboratory directors’ pay and nonpay benefits subject to the cap in addition to the reviews described above. For other FFRDC senior executives under a compensation increase plan, officials told us that DOE limits contractor compensation to amounts less than the laboratory director by establishing the director’s salary as the maximum allowable at each FFRDC and requiring contractors to report the bonus and incentive pay for senior executives. Officials told us that DOE previously did not require all its contractors to submit documentation itemizing pay and nonpay benefits as was done at SRNL because the reimbursable compensation for senior executives at the FFRDCs the agency sponsors was well below the cap. These officials told us that, because the reduction in the cap increases the risk that FFRDC executives could receive pay subject to the cap, DOE decided to adopt a process similar to that previously used at SRNL. The FFRDCs currently sponsored by DOE are subject to the OFPP fiscal year 2012 calculated cap of $952,308, until such time as their current contracts end. New contracts awarded after June 2014 will be subject to the lower cap of $487,000. Additional information on the current contract award and end dates for these FFRDCs, and those sponsored by DOD and NSF, are shown in appendix II (table 7).

Other Reviews of Compensation for FFRDC Contractor Employees In addition to reviewing reimbursed compensation for senior executives, DOE officials told us that they also review various aspects of employee compensation for all FFRDC contractors. DOE officials annually approve the FFRDC contractors’ requests for pay increases through the compensation increase plan, described above, and review and approve the contractors’ overall compensation systems at least once during the term of the contract. The compensation increase plan includes supporting information for the contractor’s request, such as comparisons of average pay for groups of FFRDC employees against market averages for similar groups. For example, the compensation increase plan for ORNL describes the average pay for its administrative support employees and for its managers relative to average pay for those groups derived from market surveys. Compensation increase plan proposals, and the salary survey data used by the FFRDC contractor, are evaluated by DOE officials, who ensure that proposed funds are reasonable and will maintain appropriate comparability with market averages from the survey data. DOE officials also review an additional report provided each year by the FFRDC contractor that includes actual compensation reimbursed for groups of FFRDC employees to ensure compliance with the FFRDC compensation increase plans that DOE previously approved.

DOD relies on audits by DCAA to review documented compensation for senior executives and other employees against the cap on reimbursements, and DCAA and DCMA review the overall pay and nonpay benefits at DOD-sponsored FFRDCs.

Reviews of Senior Executive Compensation Reimbursements Against the Statutory Cap DOD relies on audits by the DCAA to review documented compensation for FFRDC employees against the cap on reimbursements. DCAA retrospectively evaluates the compensation of executives—including senior executives—separately from other groups of contractor employees. DCAA officials told us that they annually audit costs incurred on DOD contracts, and the review of reimbursements for executive compensation against the cap is a standard step in these audits. According to the document DCAA uses to assist auditors in planning and performing these audits, DCAA compares executive compensation claimed for reimbursement to prior years, determines if there are significant changes from prior years, or if claimed executive compensation is over the statutory cap on compensation. As part of this review, DCAA auditors told us that they obtain documented information from the contractor on the itemized components of pay and nonpay benefits for individual executives and amounts claimed for reimbursement and verify that contractors correctly calculated the allowable portion of compensation costs for each executive. DCAA officials may also request assistance from another DCAA team specializing in executive compensation audits. Auditors from this team told us that they evaluate compensation for employees or classes of employees against the FAR criteria on allowability, including compliance with the cap on compensation reimbursements, as well as their reasonableness. From 2009 to 2013, this team evaluated, or is in the process of evaluating, executive compensation at 5 of the 10 DOD-sponsored FFRDCs. In the completed evaluations, DCAA found no instances of reimbursements above the cap. For other contractor employees, DCAA’s incurred cost audits include evaluations of the contractors’ overall compensation system and internal controls, including the applicability of caps on compensation reimbursements. According to DOD and DCAA officials, few employees at DOD-sponsored FFRDCs receive compensation near the cap. For example, DOD officials responding to our survey stated that, of the 10 DOD-sponsored FFRDCs, 5 FFRDCs operated by three different contractors had a single senior executive employee with total compensation—including unreimbursed amounts—above the cap in place for fiscal years 2010 and 2011. Further, DCAA officials who audit 1 of the DOD-sponsored FFRDCs told us that the lower cap enacted in 2013 would affect only a few employees at that FFRDC contractor.

Other Reviews of Compensation for FFRDC Contractor Employees In addition to reviewing reimbursed compensation against the statutory cap, DCAA also evaluates compensation for individual employees or classes of employees, including executives, against the FAR criteria for reasonableness, such as conformity with compensation practices of other firms of the same size, industry, and geographic area. DOD and DCAA officials told us they analyzed the reasonableness of compensation costs as part of the annual incurred cost audits and during negotiation of the contract. The audit team that specializes in assessing executive compensation evaluates reasonableness by confirming executive compensation amounts and comparing them against surveys of compensation at similar firms. DCAA auditors also examine the reasonableness for other groups of employees if audits of the contractor’s overall compensation system and internal controls over compensation find that those systems cannot be relied upon to demonstrate reasonable levels of compensation and that the potential for unreasonable compensation exists. DOD also utilizes DCMA to evaluate some DOD FFRDC contractors’ indirect costs, which include pension and other benefit costs.

Most NSF-sponsored FFRDCs are not subject to the federal statutory cap on compensation reimbursements because they operate under cooperative agreements rather than contracts. Nonetheless, NSF reported that none of the four FFRDCs have senior executives with compensation above the cap and that NSF has procedures to review compensation at the FFRDCs it sponsors. NSF utilizes audits by DCAA to review documented compensation for executives at the one NSF- sponsored FFRDC that operated under a contract, and NSF conducts various reviews of compensation for all FFRDC employees.

Reviews of Senior Executive Compensation Reimbursements Against the Statutory Cap NSF has different procedures across the four FFRDCs it sponsors to review senior executive compensation. For the Science and Technology Policy Institute (STPI), the only NSF-sponsored FFRDC that operates under contract and is thus subject to the cap, DCAA reviews documented compensation because the FFRDC is operated under a contract by the Institute for Defense Analyses, which also operates FFRDCs sponsored by DOD that are subject to DCAA audit. For the three FFRDCs awarded through cooperative agreements, the cap does not apply. However, the agreements for these FFRDCs require that NSF approves the salary for the FFRDC directors. In addition, these agreements require that the FFRDC contractors notify NSF of all other forms of compensation for directors at two of these three FFRDCs and stipulates that no part of the management fee received by the third FFRDC may fund the director’s salary. NSF officials also noted that they had performed a salary comparison for the NCAR director based largely on survey data on salaries for college and university administrators. NSF responded to our survey that none of its FFRDCs had executives with compensation above the cap.

Other Reviews of Compensation for FFRDC Contractor Employees NSF officials told us that they conduct various reviews of compensation for all FFRDC employees, including reviews before awarding and during the period of performance of an FFRDC contract or agreement. For STPI, the one NSF-sponsored FFRDC operating under a contract, NSF, in response to our survey asking about steps they take to evaluate compensation, told us that they review the compensation rates charged by the FFRDC annually to ensure they are fair and reasonable. In addition, DCAA and DCMA review the compensation costs for STPI, as they do for DOD-sponsored FFRDCs. For the three FFRDCs under cooperative agreements, NSF officials told us that they reviewed a sample of compensation costs from the proposed budget before awarding new cooperative agreements. For example, before awarding the most recent cooperative agreement sponsoring NCAR, NSF officials reviewed NCAR’s budget and financial capability, including the total compensation plan, positions with corresponding salary grades, and the FFRDC operator’s procedures for determining compensation rates. NSF also reviewed contributions to retirement and medical benefits at NCAR and found that the FFRDC operator appeared to have made a thorough and detailed examination of salary rates and levels and that its policies and procedures for salary rates and total compensation were generally adequate. NSF has also incorporated terms regarding compensation into its cooperative agreements. For example, the cooperative agreements for NCAR and the National Optical Astronomy Observatory require the FFRDC operator to notify NSF of any substantive changes to its compensation policies and describe the rationale for any changes.

DOE, DOD, and NSF Use Multiple Reviews to Assess FFRDC Performance

DOE, DOD, and NSF assess FFRDC performance through three types of reviews—comprehensive reviews, annual performance reviews, and other review activities such as day-to-day oversight.

DOE, DOD, and NSF Conducted Comprehensive Reviews of the Use and Need for FFRDCs That Vary in Level of Detail

DOE, DOD, and NSF conducted comprehensive reviews of the use and need for the FFRDCs they sponsored and, in keeping with the flexibilities the FAR provides, these reviews varied in level of detail. The FAR requires sponsoring agencies to conduct a comprehensive review at least every 5 years on the use and need for an FFRDC prior to extending the contract or agreement, and it describes five elements it should include: (1) an examination of the special technical needs and mission requirements performed by the FFRDC, (2) a consideration of alternative sources, (3) an assessment of the efficiency and effectiveness of the FFRDC in meeting the sponsor’s needs, (4) an assessment of the adequacy of the FFRDC management in ensuring a cost-effective operation, and (5) a determination that the sponsoring agreement or contract complies with other FAR requirements concerning FFRDCs. Within those five elements, the FAR gives agencies the flexibility to determine how much information to include and how to document the review.

Based on our analysis of agency documents, DOE, DOD, and NSF met the FAR requirement to conduct a comprehensive review of the use and need for the FFRDCs they sponsored in most instances and, in all cases, they recommended the continuance of the FFRDC. DOE has conducted comprehensive reviews for all 16 of its FFRDCs; however, in two cases, DOE did not meet the FAR requirement to conduct a comprehensive review at least every 5 years. Los Alamos and Lawrence Livermore National Laboratories were the two exceptions. DOE last conducted comprehensive reviews for Los Alamos in 2005 and Lawrence Livermore in 2007. The department has not conducted additional reviews within the 5-year period required by the FAR. DOE officials told us that NNSA, which is responsible for conducting the reviews of the two laboratories, is updating its comprehensive reviews, and the officials expect them to be finalized in the next few months. DOD met the FAR requirement to conduct a comprehensive review on the use and need for all 10 of the FFRDCs we reviewed. NSF also met the FAR requirement to conduct a comprehensive review on the use and need for the 1 FFRDC it operates by contract. These comprehensive reviews stated that all of the FFRDCs were performing effectively or adequately and either recommended or supported continuing the FFRDC. Further, DOD officials told us that all of its 10 FFRDCs have retained the same contractor throughout the existence of their FFRDCs, and NSF officials said that 3 of its 4 FFRDCs have retained the same contractor. Since 2004, DOE has competed 6 of its 16 FFRDC contracts and replaced the incumbent contractor in four instances.

In keeping with the flexibilities that the FAR provides, DOE, DOD, and NSF documented the results of their most recent comprehensive reviews to varying degrees of detail and in how much information they included in addressing the FAR elements, as follows:

DOE. DOE’s comprehensive review documents generally followed a consistent format that provided general statements for each of the five FAR elements. The comprehensive reviews provided more information regarding the department’s technical needs and mission requirements for the FFRDC than they did for the other FAR elements, with almost all of the 16 FFRDC reviews documenting this element in detail. Most of DOE’s comprehensive reviews partially documented the other four FAR elements. For example, as part of the discussion on the efficiency and effectiveness of the FFRDC for ORNL, the comprehensive review described DOE’s satisfaction with the FFRDC contractor by providing the letter grades received by ORNL based on DOE’s annual appraisal process to evaluate the scientific, technological, managerial, and operational performance of the contractors who manage and operate each of its national laboratories. ORNL’s comprehensive review did not provide additional details on the outcome of the annual appraisals.

DOD. DOD’s comprehensive review documents generally documented each of the five elements in detail. Seven of the nine DOD comprehensive reviews contained supporting evidence on all five FAR elements, including a rationale for their determinations relating to each element. For example, regarding the FAR element to examine alternative sources, the NSEC comprehensive review described the relative merits of using DOD personnel, for-profit contractors, and other nonprofit contractors to complete the work currently done by NSEC. The other two reviews were partially documented, including limited supporting evidence regarding the efficiency and effectiveness of the FFRDC. One of these reviews also included limited supporting evidence related to the cost-effectiveness of the FFRDC and no documentation or discussion related to compliance of the sponsoring agreement with the FAR requirements.

NSF. The comprehensive review for NSF’s Science and Technology Policy Institute—according to NSF officials, the only one of NSF’s FFRDCs subject to the FAR requirement for a comprehensive review—included all five FAR elements and provided detailed information regarding the sponsor’s needs and the efficiency and effectiveness of the FFRDC and a limited discussion on alternative sources and the FFRDC’s cost-effectiveness.

DOE and DOD officials have developed guidance on how to conduct comprehensive reviews, and NSF officials stated that the FAR informs their reviews of NSF-sponsored FFRDCs. DOE’s guidance calls for a review of the use and need for continued operation as an FFRDC in accordance with the FAR, without additional elaboration on these FAR requirements. This guidance states that the comprehensive review should be completed as part of the agency’s process for deciding if it will extend or compete an FFRDC contract when the current contract’s term has ended. This guidance also calls for the preparation of an alternatives package 24 months before the FFRDC’s contract expires, consisting of an action memo for the Secretary to approve and other documentation, for example a discussion of the incumbent FFRDC contractor’s performance history, including technical, administrative, and cost performance. DOE officials stated that FFRDC reviews are required every 5 years in accordance with the FAR requirements, regardless of award provisions and competitions for the FFRDC contract.

DOD has issued guidance on how to conduct comprehensive reviews in the form of a memorandum to DOD offices and divisions on the management of FFRDCs. DOD’s guidance was issued to clarify the FAR requirements, assist DOD officials with the oversight and management of the FFRDCs they sponsor, and ensure consistency across the department. It includes references to certain FAR elements relating to the execution of comprehensive reviews and adds additional requirements for DOD sponsors. Among other things, the guidance assigns responsibilities to the Under Secretary of Defense for Acquisition, Technology and Logistics to establish and promulgate FFRDC procedures; allocate the number of staff years and funding within the annual ceiling set by Congress for all DOD-sponsored FFRDCs; and review and approve all comprehensive reviews for FFRDCs prior to a contract renewal decision. In addition, the guidance provides direction to DOD sponsors for reporting the results of the FFRDC comprehensive reviews and, according to DOD officials, helps DOD foster the optimal and appropriate management of FFRDCs in accordance with law, regulation, and policy. The guidance directs DOD sponsors to establish annual governance procedures to monitor the value, quality, and responsiveness of FFRDC work. DOD officials added that the guidance helps make sure that the sponsors are paying attention to the quality of the products they are getting from the FFRDC, and it allows the department to state with confidence that the services provided by the FFRDC are needed.

As stated earlier, according to NSF, the specific provisions in the FAR requiring a comprehensive review do not apply to three of its four FFRDCs because they are established under cooperative agreements. However, NSF officials said that the agency carries out a midpoint review for these three FFRDCs similar to the comprehensive review and its associated FAR requirements. For example, per the terms of the cooperative agreement with NCAR, NSF initiates a midpoint review once during the 5-year term of the agreement with the FFRDC operator. According to NSF officials, the midpoint review includes assessments of the management, science, and service functions by multiple teams of independent experts based on criteria given to them by NSF. The officials said that NSF has used the results of the midpoint review to inform changes to the cooperative agreement. For example, the most recent review was critical of the FFRDC operator’s efforts to coordinate and plan their educational programs, so NSF included specific requirements to increase coordination among these programs in the subsequent cooperative agreement.

DOE and DOD officials provided different explanations on how the comprehensive review process supports their assessments of FFRDC performance. DOE officials described the comprehensive review as a summation largely derived from the findings of its annual performance appraisals, and not a stand-alone evaluation or an exhaustive description of the FFRDC contractor’s performance. According to those officials, the DOE comprehensive reviews rely on annual performance reports to provide a multiyear assessment. In contrast, DOD officials and contractor representatives described the comprehensive review process as an opportunity to take a broad assessment of the FFRDC and its key competencies over and above DOD’s annual assessments of its FFRDCs. For example, NSEC’s comprehensive review served as an opportunity for the department to recognize that NSEC had developed expertise in countering improvised explosive devices, that DOD had an ongoing need for this work, and that DOD should elevate the importance of this work. In another example, the last comprehensive review for the Software Engineering Institute identified concerns about the execution and ability of the FFRDC to meet the sponsor’s need for more research engineers. The comprehensive review recommended a revised sponsoring agreement with substantial changes in governance and to the core statement. It also recommended removing most of the process improvement work from the FFRDC and emphasizing research. DOD implemented the recommendation to remove most process improvement work and the subsequent sponsoring agreement emphasized research.

DOE, DOD, and NSF Conduct Annual Performance Assessments, Many of Which Rely on User Surveys

DOE, DOD, and NSF annually review and document the performance of the FFRDCs they sponsor. Annual reviews are conducted to evaluate the performance of contractors who manage and operate the FFRDCs.

DOE. DOE develops annual performance plans and undertakes an annual appraisal process to evaluate its FFRDCs, including surveying federal officials responsible for the work at the DOE-sponsored centers. With respect to the 10 FFRDCs sponsored by DOE’s Office of Science, DOE officials conduct an annual appraisal process to evaluate the scientific, technological, managerial, and operational performance of the contractors who manage and operate each of its national laboratories. According to DOE officials, these evaluations provide the basis for determining performance fees and the possibility of earning additional years on the contract through an award term extension. They also serve to inform the decisions DOE makes regarding whether to extend or compete the contracts when they expire. For each of the FFRDCs sponsored by the Office of Science, DOE officials told us that they prepare a performance evaluation and measurement plan a few months prior to the beginning of the fiscal year that provides a common structure and scoring system using goals, objectives, and notable outcomes that are weighted in importance by the appropriate program office or other FFRDC users. These plans serve as the evaluation template for the FFRDCs and their contractors at the close of the fiscal year. At the conclusion of the fiscal year, a performance and evaluation report is developed for each DOE FFRDC that provides performance scores and the amount of performance-based fee earned by the contractor.

DOD. DOD officials are responsible for annual reviews of the FFRDCs sponsored by the department. Under DOD guidance, sponsors are to annually assess the technical quality, responsiveness, value, cost and timeliness of FFRDCs. According to DOD officials, the guidance does not specify how sponsors are to complete the annual FFRDC reviews; however, almost all DOD sponsors use surveys to obtain input about the FFRDC’s performance in support of these annual reviews, including the sponsors for NSEC. The timing of the annual review is determined by the DOD sponsors. For example, NSEC’s annual survey and associated review is typically completed by the end of March, at which time it is provided to the DOD sponsor for review. In addition, DOD officials compare FFRDC user survey responses with survey responses provided by the FFRDC contractor, which also provides input to the survey, to identify any inconsistencies in performance. Some of the information from the annual surveys may be used in support of the comprehensive review; for example, all of the last 5 years of annual survey responses provided by Army and Air Force were used to support the findings of NSEC’s latest comprehensive review.

NSF. On an annual basis, NSF’s FFRDC operators submit to NSF a retrospective review, called a program operating plan progress report, on FFRDC performance. This review describes progress made at an FFRDC based on milestones, objectives, targets, and deliverables described in a program operating plan, among other things. For example, according to NSF officials, for NCAR this review shall be submitted for review and approval by November 15 after the end of each fiscal year, and it is conducted by both NSF and the FFRDC operator and is reviewed by NSF officials. In addition, the FFRDC operator submits an annual management report to NSF that focuses on the FFRDC operator’s management activities for the year. This review describes the FFRDC operator’s performance of specific duties and its progress in meeting objectives. The format and content of this report is developed in consultation with NSF officials, and it is to be submitted within 90 days prior to the end of each fiscal year.

DOE, DOD, and NSF Engage in Other Ongoing Activities Critical to Assessing FFRDC Performance

Officials from DOE, DOD and NSF described the day-to-day interactions between agency and contractor officials, as well as other review activities as critical to assessing FFRDC performance.

DOE officials told us that agency site office officials co-located with the contractors are primarily responsible for assessing business and operational performance at their respective FFRDCs, and DOE’s program officials assess performance of the mission work.

According to DOE officials at ORNL, for example, the agency’s site office officials manage the contract with the FFRDC contractor and the resources associated with that effort; perform operational oversight and ensure regulatory compliance of contractor operations; perform regulatory compliance work and manage resources to ensure efficient operation; and provide stewardship of ORNL assets, such as research equipment and buildings. According to DOE’s ORNL site office officials and documents, review activities include routine, day-to-day monitoring through facility tours, walk-throughs, surveillances, work observations, document reviews, meeting attendance and participation, and ongoing interaction with contractor workers. These site office officials also conduct structured evaluations of contractor programs and performance to confirm that operations are being conducted in a safe and efficient manner and are in compliance with contract requirements. DOE’s ORNL site office and FFRDC contractor employees said that the standard set of documentation associated with performance and oversight, such as the annual performance plans and reports, is not fully sufficient to understand the full range of performance and associated oversight being performed by ORNL site office employees. DOE’s program office officials associated with ORNL assess the quality of the laboratory’s work through several types of evaluations, including reviews of project outcomes and peer reviews of the scientific work. For example, DOE’s program officials monitor performance through regular meetings with FFRDC technical staff to discuss specific projects. They also oversee peer-reviews that are performed on a triennial basis, working with experts external to the program from academia or other DOE laboratories to assess the quality of the work performed by the contractor for specific research projects.

According to DOE officials, FFRDC performance is also monitored as a part of its work authorization system, which DOE uses as a control process for work performed by contractors. Authorizations for most DOE-funded work must be documented in work authorizations, and each authorization must include a detailed statement of work to be performed and associated performance-based indicators or targets, including any additional guidance that will assist the contractor in the performance of the work. For example, the delivery of the work could be the publication of a paper in a scientific journal, and DOE would use peer review committees to determine if the FFRDC contractor met the objective as stated in the work authorization. FFRDC ORNL site office officials are responsible for authorizing the work put forth in work authorizations, while DOE program office officials set the technical specifications and evaluate the technical progress of the work.

DOD officials described a number of other activities performed as part of its ongoing oversight of its FFRDCs. For example, DOD officials said that they use a “tiered accountability” framework or structure when overseeing the contract to manage NSEC, which includes:

Day-to-day oversight. DOD program managers, contracting officers, and contracting officer representatives told us they perform some day- to-day oversight activities, including funding on the contract and monitoring the allocation of STE, or 1 year of fully burdened labor, charged against the contract throughout year. Sometimes, small adjustments to the STE levels are made, being driven by the overall budget or priorities of the program. As part of this oversight, DOD officials look at the scope of work on all projects and compare them against the core work statements as defined in the FFRDC sponsoring agreements to make sure that work is within the scope for the FFRDC.

Working with the DCAA. DOD officials told us the department’s administrative contracting officers work closely with DCAA on funding reports and on audits of funding.

Negotiating indirect rates. The DOD procuring contract officer negotiates FFRDC contractor’s indirect rates under the NSEC contract.

In addition, with respect to NSEC, DOD’s program managers told us that they request an annual and semiannual feedback and assessment for the work being done by the FFRDC on behalf of the Army and the Air Force. DOD officials also noted the importance of regular user forums between FFRDC staff and DOD users. In addition to the user surveys, DOD officials told us that the performance for NSEC’s FFRDC contractor is overseen through daily interactions between DOD and the FFRDC contractor staff at the project level. In addition, monthly or weekly meetings take place between DOD program and FFRDC contractor employees at the division level, and DOD and FFRDC contractor management officials also meet quarterly to discuss the overall performance of NSEC.

According to NSF officials, the agency’s oversight role with its FFRDCs is specified in the cooperative agreements NSF has with the FFRDCs. Generally, the terms and conditions laid out in the cooperative agreement provide NSF’s oversight guidelines. NSF officials said that they do not have employees collocated or on-site at the FFRDCs to perform oversight. Instead, NSF officials travel to the FFRDCs and FFRDC contractor employees travel to NSF headquarters to exchange information on the oversight of the FFRDCs.

Agency Comments

We provided a draft of this product to DOE, DOD, and NSF for their review and comment. Formal comments were not provided by DOE, DOD, and NSF. Technical comments provided by DOE, DOD, and NSF were incorporated, as appropriate.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Energy, Secretary of Defense, Director of the NSF, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

We selected a non-probability sample of Federally Funded Research and Development Centers (FFRDC) for review based on the master list of FFRDCs compiled by the National Science Foundation (NSF). We chose to review FFRDCs sponsored by the Department of Energy (DOE), Department of Defense (DOD), and NSF because these three agencies sponsor the largest numbers of FFRDCs—16, 10, and 4, respectively. Our findings related to these 30 FFRDCs are not generalizable to the complete set of 40 FFRDCs in operation in 2014 (see fig. 1). To select FFRDCs for site visits and additional review from among the 30 sponsored by DOE, DOD, and NSF, we identified the FFRDCs with the highest research and development expenditures reported by the NSF for fiscal year 2010 within each agency. The FFRDCs we chose for site visits at each agency were: (1) Oak Ridge National Laboratory, run by UT-Battelle, LLC, on behalf of DOE’s Office of Science; (2) the National Security Engineering Center, run by the MITRE Corporation on behalf of DOD; and (3) the National Center for Atmospheric Research, run by the University Corporation for Atmospheric Research on behalf of NSF. We evaluated additional documentation regarding the sponsorship of these FFRDCs and their performance.

To learn about FFRDC funding and compensation we surveyed the agency sponsors for the 30 FFRDCs in our scope, spoke to agency officials and contractor representatives, and reviewed agency policy and guidance documentation. We developed a standard questionnaire to obtain information on the obligations and funding received by FFRDCs, full-time equivalents (FTE), compensation costs at each FFRDC, and the controls agencies have in place to review or assess compensation. We asked for information on obligations and funding for the 5 most recent fiscal years available at the time of our review, fiscal years 2008 through 2012. We asked for information on employees and compensation for fiscal years 2010 through 2012—the 3 years for which data were available and the cap on compensation was set by regulation (i.e., before the statutory cap had been implemented)—and sponsoring agency efforts to review compensation.

Our survey of FFRDC sponsors asked for obligations provided to the FFRDC by the federal sponsoring agency, obligations provided to the FFRDC from all other federal government departments and agencies, and funding provided to the FFRDC from nonfederal, governmental entities (such as state or local governments) and nongovernmental entities for fiscal years 2008 through 2012. Our survey of FFRDC sponsors also asked for the total FFRDC employee compensation costs, including salary and nonpay benefits such as pensions, insurance, or fringe benefits, paid by the FFRDC contractor, for the total FTE employment during fiscal years 2010 through 2012, and for the range of salaries for senior executives whose compensation is charged or allocated to the FFRDC contract or agreement.

We pretested the questionnaire by phone with agency officials who had responsibilities for overseeing FFRDCs at DOE, DOD, and NSF in order to ensure that the survey questions and definitions were clear and that the information could feasibly be obtained. We made changes to the content of the questionnaire after each review based on the feedback we received. The questionnaire was distributed to agency liaisons in September 2013, and responses for all 30 FFRDC were received by December 2013. We reviewed the survey responses of agency and contractor data for consistency and reliability by examining information provided by the respondents on their data sources, the steps taken to ensure the accuracy and completeness of the data provided, and any known limitations. We also interviewed knowledgeable agency officials about the data, and obtained follow-up information by phone and e-mail from agency sponsors when necessary between January 2014 and April 2014. Agency officials submitted written clarifications to the original survey responses and follow-up questions, as necessary. Data obtained from the survey was also presented to the agencies for review prior to reporting. We found that the data were sufficiently reliable for our purposes.

To assess agency management and oversight of FFRDC operator performance, we conducted interviews with agency officials at DOE, DOD, and NSF and contractor officials at ORNL, NSEC, and NCAR to determine how performance was assessed and reviewed relevant documentation of those activities. These interviews included officials from DOE’s National Nuclear Security Administration, Office of Science, and Office of Management; DOD’s Office of Acquisition Resources and Analysis; and NSF’s Office of Budget, Finance, and Award Management who have oversight responsibilities for multiple FFRDCs. We also interviewed agency officials with direct oversight responsibilities for ORNL, NSEC, and NCAR, including the Oak Ridge Site Office, Deputy Assistant Secretary of Defense for Systems Engineering, and Division of Atmospheric and Geospace Sciences.

We obtained the most recent comprehensive reviews performed by the agency for 26 FFRDCs in our sample as of September 2013 and evaluated them against the elements described by the FAR. We received reviews completed between 2005 and 2013. Some FFRDCs included in our review were not subject to the requirement to perform a comprehensive review, according to their sponsoring agency. Because the FAR applies only to federal government acquisitions made by contract, cooperative agreements between sponsoring agencies and an FFRDC operator are generally not subject to requirements in the FAR. Three FFRDCs sponsored by NSF operate under cooperative agreements. According to NSF officials, the agency is not required to perform a comprehensive review for these FFRDCs and we excluded them from our analysis. In addition, we excluded the comprehensive review for the Center for Communications and Computing FFRDC due to security classification of material within the document. For the remaining 26 FFRDCs, we examined the comprehensive reviews for the extent that the review documented and supported the five elements described by the FAR. Two analysts independently categorized the extent of information provided for each element into three categories. If the documentation of the comprehensive review did not discuss the element at all, we categorized it as “Not Documented.” If the documentation in the comprehensive review provided a determination or finding related to the element, but it did not fully address the criteria or provide supporting evidence, we categorized it as “Partially Documented.” If the review provides a determination or finding for the element, and provides additional supporting evidence, we categorized it as “Documented.” Three analysts, including those that performed the initial review, then examined any differences between the two reviews and made a final determination for each comprehensive review and element.

We conducted this performance audit from December 2012 to July 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Funding and Contract Data for FFRDCs Sponsored by DOE, DOD, and NSF

To learn about funding and compensation at federally funded research and development centers (FFRDC) we surveyed the agency sponsors for the 30 FFRDCs sponsored by the Department of Energy (DOE), Department of Defense (DOD), and National Science Foundation (NSF) and collected information on their operators. Our survey asked for obligations to the FFRDC by the federal sponsoring agency, obligations provided to the FFRDC by all other federal government departments and agencies, and funding provided to the FFRDC from nonfederal, governmental entities (such as state or local governments) and nongovernmental entities for fiscal years 2008 through 2012. Table 2 provides the total obligations and funding to the 30 FFRDCs in our scope. Table 3 provides the obligations by the federal sponsor. Table 4 provides the obligations by all other federal government departments and agencies. Table 5 provides the funding from nonfederal entities. Table 6 provides data responding to our survey questions on the total FFRDC employee compensation costs paid by the FFRDC contractor and the total FTE employment during fiscal years 2010 through 2012. Table 7 describes the 30 FFRDCs we examined, their current operators, and start and end dates of their current contracts or cooperative agreements.

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Dan Haas (Assistant Director), John Barrett, Kevin Bray, Alysia D. Davis, R. Eli DeVan, Cindy Gilbert, Paul Kinney, Alison O’Neill, Kelly Rubin, Tind Shepper Ryen, Aaron Shiffrin, Kiki Theodoropoulos, Jack Wang, and Bill Woods made key contributions to this report.