Federally Funded Research Centers: Agency Reviews of Employee Compensation and Center Performance
Published: 2014-08-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Federal agencies sponsor 40 FFRDCs for research and development tasks that are integral to their missions. DOE, DOD, and NSF sponsor the largest number of FFRDCs—16, 10, and 4 centers, respectively. Federal agencies sponsor FFRDCs by contracting with nonprofit, university-affiliated, or private industry operators. Federal statute and regulations provide for reimbursements for compensation for FFRDC contractor employees and require that sponsoring agencies evaluate the use and need for their FFRDCs.
GAO was asked to review the management and oversight of FFRDCs. This report (1) describes funding for FFRDCs sponsored by DOE, DOD, and NSF for fiscal years 2008 through 2012; (2) examines compensation for FFRDC employees and these agencies' processes for review of compensation; and (3) determines how these agencies assess FFRDC performance. GAO surveyed the agency sponsors for the 30 FFRDCs, analyzed agency policies and reviews of these FFRDCs, and interviewed agency officials and contractor representatives.
section_title: What GAO Found
paragraphs: The 30 federally funded research and development centers (FFRDC) sponsored by the Department of Energy (DOE), Department of Defense (DOD), and National Science Foundation (NSF) received nearly $84 billion in total funding for fiscal years 2008 through 2012. Of these 30 centers, the 16 sponsored by DOE received about 79 percent of this funding according to GAO's analysis of sponsoring agencies' responses to a GAO survey on FFRDC funding and compensation. During this time, DOE obligated about 34 percent of its budget to the FFRDCs it sponsored, and DOD and NSF devoted less than 1 percent and 4 percent of their budgets, respectively. FFRDCs sponsored by these agencies received approximately $15 billion of their total funding from sources other than the sponsoring agency—specifically other federal agencies, nonfederal entities such as state or local governments, and private entities.
Many FFRDCs sponsored by DOE, DOD, and NSF spent over half of their total funding on employee compensation, and the three agencies had processes in place to review such compensation. For example, the agencies reviewed senior executive compensation to ensure that they do not reimburse FFRDC contractors in excess of the cap set in statute. All three agencies also have processes in place to document the total reimbursed compensation for senior executives against the cap, although DOE changed its policy during the course of GAO's work. In May 2014, DOE updated its policy on executive compensation to require documentation of compensation subject to the cap—a requirement that was not in place prior to this date. DOE officials noted that this change was due, in part, to the recent action by Congress in December 2013 to reduce the cap from $952,308 to $487,000.
DOE, DOD, and NSF assess performance of FFRDCs through three types of reviews: (1) comprehensive reviews—which the Federal Acquisition Regulation (FAR) requires at least every 5 years, (2) annual performance reviews, and (3) other review activities such as day-to-day oversight. DOE, DOD, and NSF conducted timely comprehensive reviews of the use and need for the FFRDCs they sponsored in most cases and, in all cases, the agencies recommended the continuance of the FFRDCs they sponsor. The FAR describes five elements the review should include, and DOE, DOD, and NSF generally included these elements, with varying levels of detail in keeping with the flexibilities the FAR provides. These agencies also have procedures to annually review and document the performance of the FFRDCs they sponsor, and many of these reviews use surveys of federal officials who interact with the centers. In addition, officials from DOE, DOD, and NSF told GAO that they engage in other day-to-day oversight activities to help them assess FFRDC performance, such as observing work and meeting with contractor employees.
section_title: What GAO Recommends
paragraphs: GAO is not making recommendations in this report. DOE, DOD, and NSF reviewed a draft of this report and did not provide formal comments. Technical comments provided by DOE, DOD, and NSF were incorporated, as appropriate.

section_title: Letter
paragraphs: The federal government supports research and development activities at  40 federally funded research and development centers (FFRDC).  FFRDCs are government-funded entities that have long-term  relationships with one or more federal agencies to perform research and  development and related tasks. FFRDCs are typically entirely federally  funded, or nearly so, but they are operated by contractors or other  nongovernmental organizations. The National Science Foundation (NSF)  has reported that federal agencies provide billions of dollars each year for  research and development activities at FFRDCs. Federal agencies  sponsor FFRDCs by establishing contracts or other agreements with  nonprofit, university-affiliated, or private industry organizations, which in  turn operate the FFRDCs. As described in the Federal Acquisition  Regulation (FAR), FFRDCs are intended to meet special, long-term  research or development needs of the sponsoring agencies that are  integral to their missions and cannot be met as effectively by existing  federal or non-FFRDC contractor resources.
The agencies that sponsor FFRDCs are subject to various laws and  regulations concerning the management and performance of their  activities. These laws and regulations guide the sponsors’ oversight  activities and shape the relationship between sponsors and the FFRDC  contractors. Federal law caps the total allowable compensation for certain  contractor positions for which federal agencies may reimburse  contractors. In addition, the FAR provides criteria for agencies and  contractors to use to determine what costs may comprise the  compensation charged to the government. The FAR also describes  characteristics of FFRDCs operated by contract and includes  requirements for their establishment, use, and review. Among other  things, the FAR encourages long-term relationships between the federal  government and FFRDCs to provide continuity and to attract high-quality  personnel to the FFRDCs. Because FFRDCs are contractor operated,  sponsoring agencies do not directly determine pay or nonpay benefits for  contractor employees.
As agencies provide a significant amount of funding for FFRDC activities,  you asked us to examine the management and oversight of FFRDCs. Our  objectives for this report were to: (1) describe the total funding for  FFRDCs associated with the Department of Energy (DOE), Department  of Defense (DOD), and NSF for the 5 most recent years available—fiscal  years 2008 through 2012; (2) examine compensation for FFRDC  employees and these agencies’ processes to review compensation; and  (3) determine how these agencies assess FFRDC performance.
To address our objectives, we selected a nonprobability sample of 30  FFRDCs for review based on a complete list of FFRDCs compiled by  NSF. We selected FFRDCs sponsored by DOE, DOD, and NSF because  these three agencies sponsor the largest numbers of FFRDCs—16, 10,  and 4, respectively. Of these 30 selected FFRDCs, we then identified for  site visits the FFRDC with the highest research and development  expenditures at each of the three agencies as reported by NSF for fiscal  year 2010, which were (1) Oak Ridge National Laboratory (ORNL),  operated by UT-Battelle, LLC, on behalf of DOE’s Office of Science; (2)  the National Security Engineering Center (NSEC), operated by the  MITRE Corporation on behalf of DOD; and (3) the National Center for  Atmospheric Research (NCAR), operated by the University Corporation  for Atmospheric Research on behalf of NSF. We visited these three  FFRDCs, spoke with contractor representatives and agency officials  overseeing them, and evaluated additional documentation regarding the  sponsorship of these FFRDCs and their performance. To examine  FFRDC funding and compensation, we surveyed, and received  responses from, the agency sponsors for all of the 30 FFRDCs in our  review. We also spoke with agency officials and contractor  representatives, and we reviewed agency policy and guidance  documentation they provided. Our survey asked about funding to the  FFRDCs for the 5 most recent fiscal years for which data were available,  or for fiscal years 2008 through 2012. Our survey also asked about  compensation from fiscal year 2010 through fiscal year 2012—the 3 years  for which data were available and the cap on compensation was set by  regulation (i.e., before the statutory cap had been implemented)—and  sponsoring agency efforts to review compensation. To assess the  reliability of the survey data, we reviewed the survey responses of agency  and contractor data for consistency and reliability by examining the  sources of the data, interviewing knowledgeable agency officials about  the data, and obtaining follow-up information from agency sponsors when  necessary. Agency officials submitted written clarifications to the original  survey responses and follow-up questions, as necessary. We found that  the data were sufficiently reliable for our purposes. To examine  sponsoring agencies’ assessments of FFRDC contractor performance,  we obtained the most recent comprehensive reviews developed by the  agencies that sponsored the FFRDCs and evaluated them against  elements described by the FAR. We also spoke with agency officials and  contractor representatives at the three selected FFRDCs to determine  how performance was assessed and reviewed relevant documentation of  that performance. Additional information on our scope and methodology  is in appendix I.
We conducted this performance audit from December 2012 to August  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: FFRDCs perform tasks such as research and development, systems  engineering, and technical studies and analyses on behalf of sponsoring  federal agencies. FFRDCs arose from partnerships between the federal  government and academic researchers and scientists during World War  II. Those partnerships were later restructured into federal research  centers to retain scientists and became known as FFRDCs by the mid- 1960s. Since that time, FFRDCs have continued to support the research  and development needs of various federal agencies. As of April 2014,  there were 40 FFRDCs sponsored government-wide. A list of these  FFRDCs and their federal sponsors, including those sponsored by DOE,  DOD, and NSF that are within the scope of our review, is shown in   figure 1.
The 16 FFRDCs that DOE sponsors are overseen by different program  offices within the Department. Ten of these FFRDCs, including ORNL,  are overseen by the Office of Science; three by the National Nuclear  Security Administration (NNSA); one by the Office of Environmental  Management; one by the Office of Nuclear Energy; and one by the Office  of Energy Efficiency and Renewable Energy. DOE’s Office of  Management manages department-wide policies relating to DOE’s  FFRDCs, and DOE program offices control many of the decisions  associated with the operations of the FFRDCs, including their budgets,  structures, and missions. Federal officials located at or near each FFRDC  site directly oversee the operations of the FFRDCs.
The 10 FFRDCs that DOD sponsors are managed by various military  departments or divisions of DOD. The Under Secretary of Defense for  Acquisition, Technology and Logistics is responsible for setting DOD-wide  policies for FFRDCs, allocating staff years within an annual ceiling set by  Congress at the FFRDCs, and approving renewals of FFRDC contracts,  among other things. The Under Secretary carries out these duties  through the Office of Acquisition Resources and Analysis. The DOD  divisions that oversee FFRDCs are responsible for creating procedures to  monitor the value, quality, and responsiveness of FFRDC work and  ensuring appropriate contract support. The Deputy Assistant Secretary for  Defense for Systems Engineering oversees NSEC, with assistance from  Army and Air Force subsponsors. Two additional DOD divisions may  provide administrative support for DOD contracts, including those for  FFRDCs. The Defense Contract Management Agency (DCMA), a DOD  component, is responsible for providing contract administration services  for DOD buying activities, working directly with defense contractors to  help ensure that goods and services are delivered on time, at projected  cost, and that they meet performance requirements. The Defense  Contract Audit Agency (DCAA), another DOD component, audits  projected and actual costs associated with DOD contracts to ensure they  are allowable, allocable, and reasonable in accordance with cost  accounting standards and FAR rules.
The four FFRDCs that NSF sponsors are overseen by the Office of  Budget, Finance, and Award Management and the divisions whose  objectives the FFRDCs primarily support. For example, NSF’s Division of  Atmospheric and Geospace Sciences oversees NCAR, with assistance  from the other NSF offices. Of NSF-sponsored FFRDCs, three, including  NCAR, are established through cooperative agreements rather than  contracts as was done for the other 27 FFRDCs we reviewed. The  remaining NSF-sponsored FFRDC, the Science and Technology Policy  Institute, is sponsored by NSF on behalf of the White House Office of  Science and Technology Policy (OSTP), and it provides analysis on  significant domestic and international science and technology policies and  developments for OSTP and other federal agencies.
NSF also maintains a master list of all FFRDCs and reports data on  research and development expenditures at FFRDCs. Specifically, NSF  collects data on funds expended for activities specifically organized to  produce research outcomes. These data focus on research activities and  may not include all expenditures, such as those associated with  administering the FFRDC or construction or renovation of facilities.
FFRDCs are defined through the sponsoring agreement between the  agency and the contractor retained to operate the FFRDC. At some  agencies, the sponsoring agreement is a separate document that is  incorporated into the contract or cooperative agreement and, at other  agencies, the contract or cooperative agreement itself constitutes the  sponsoring agreement. Historically, contractors have been retained for  many years or decades as the operator of an FFRDC. Although FFRDCs  are entirely federally funded, or nearly so, the organization operating the  FFRDC is regarded as a contractor and not an agency of the United  States. The contractor continues to exercise primary control over its  business concerns, such as its personnel policies and compensation.
Contractors can charge a number of costs to federal government  contracts, including certain types and levels of compensation costs. Some  types of compensation costs are considered allowable under government  contracts, while others are considered unallowable. For example,  compensation in the form of bonuses are allowable under certain  conditions, while compensation in the form of stock options are  unallowable. In order to be allowable, costs must meet certain  requirements, such as being reasonable and in compliance with the terms  of the contract and generally accepted accounting principles and  practices. In determining reasonableness, the FAR directs agencies to  consider if the cost exceeds that which would be incurred by a prudent  person in the conduct of competitive business, such as the extent to  which a contractor employee’s compensation is consistent with other  companies that are similar in size, industry, geographic area, and/or other  factors. In addition, federal law caps the allowable total compensation  for certain positions for which federal agencies may reimburse  contractors. As shown in figure 2, the statutory cap on compensation  reimbursements applies to pay (e.g., salaries, wages, bonuses, and other  cash incentives) and some nonpay benefits (e.g., pensions and deferred  compensation benefits) under contracts. Federal agencies may  reimburse contractors for all or a portion of these costs, as long as the  costs are allowable and the total reimbursement does not exceed the cap.  The cap does not apply to other nonpay benefits, such as health  insurance or other fringe benefits. FFRDC contractors, like other  contractors, may provide compensation to their employees at levels they  choose, but such compensation may be unallowable for reimbursement  by federal sponsors.
From 1998 to 2012, the Office of Federal Procurement Policy (OFPP)  calculated the cap amount annually based on the amount of  compensation provided to the five most highly compensated employees  in management positions at all publicly-owned U.S. companies with  annual sales over $50 million. Over the 3-year period from fiscal year  2010 through fiscal year 2012, the cap rose from $693,951 in 2010 to  $763,029 in 2011, and then to $952,308 in 2012. In 2013, Congress  effectively fixed the cap for preexisting contracts to the fiscal year 2012  amount, and reduced the cap amount for new contracts to $487,000 (see  fig. 3). Contracts awarded prior to June 24, 2014, are subject to the  annual cap calculated by OFPP. Contracts awarded on or after June 24,  2014, are subject to a lower cap of $487,000. In addition, Congress  amended the cap in 2011 to include all defense agency contractor  employees and again in 2013 to include all civilian contractor employees  instead of the top five most highly paid contractor employees—senior  executives.
The agencies that sponsor FFRDCs are also subject to the federal  standards for internal control that state that agencies should collect  information needed to determine whether the agency is complying with  laws and regulations—such as the cap on executive compensation—and  clearly document transactions and other significant events and make  such documentation readily available. The Standards for Internal Control  in the Federal Government provide a framework for establishing and  maintaining internal control and identifying and addressing major  performance and management challenges, and internal controls comprise  the plans, methods, and procedures used by agencies to meet missions,  goals, and objectives.
Furthermore, federal agencies, including those that that sponsor  FFRDCs, are subject the Government Performance and Results Act of  1993 (GPRA) and the GPRA Modernization Act of 2010. These acts  require agencies to develop long-term and annual goals and measures  and report on progress toward achieving those goals on an annual basis.  We have reported that agencies face long-standing difficulties measuring  the performance of various types of federal programs and activities,  including contracts, grants, and research and development. Whereas  federal agencies are subject to the performance reporting requirements of  GPRA, as specified by the Office of Management and Budget (OMB),  FFRDC contractors are not subject to any generally applicable, post- award reporting requirements. The reporting requirements for FFRDC  contractors are specified in the applicable contracts or cooperative  agreements between the sponsoring agency and FFRDC contractor.
The FAR establishes certain requirements that apply when agencies  enter into contracts to sponsor FFRDCs. For example, while FFRDC  contractors primarily perform work for the sponsoring agency, they may  also enter into additional contracts or agreements to perform work for  other entities. In such cases, the FAR requires that the contract include  provisions allowing the sponsoring agency to approve that the additional  work is within the purpose, mission, general scope of effort, or special  competency of the FFRDC. In addition, prior to extending a contract or  sponsoring agreement for an FFRDC, the FAR requires that the federal  sponsor conduct a comprehensive review of the use and need for the  FFRDC at least every 5 years. The FAR describes elements of that  review to include the following:  1.  an examination of the sponsor’s special technical needs and missions  requirements that are performed by the FFRDC;  2.  consideration of alternative sources;  3.  an assessment of the efficiency and effectiveness of the FFRDC in  meeting the sponsor’s needs;  4.  an assessment of the adequacy of the FFRDC management in  ensuring a cost effective operation; and  5.  a determination that the sponsoring agreement is in compliance with  other FAR requirements concerning FFRDCs (e.g., ensuring that the  sponsoring agreement or contract contains a statement of purpose,  provisions for termination, and prohibition on competition).
The FAR further requires that the head of the sponsoring agency approve  or terminate sponsorship based on the results of the comprehensive  review conducted in accordance with these elements. Because the  elements are not required by the FAR, agencies have flexibility to  determine the content of the comprehensive review and its  documentation.
section_title: FFRDCs Sponsored by DOE Received More Total Funding Than Those Sponsored by DOD and NSF
paragraphs: The 30 FFRDCs sponsored by DOE, DOD, and NSF received nearly $84  billion in total funding for fiscal years 2008 through 2012. The 16 FFRDCs  sponsored by DOE received more funding than the FFRDCs sponsored  by the other two agencies and performed more work for entities other  than the sponsoring agency. Specifically, DOE-sponsored FFRDCs  received about 79 percent of the nearly $84 billion in total funding for  FFRDCs sponsored by these agencies for fiscal years 2008 through  2012, according to our analysis of agency reports of contractor and  agency data provided in response to our survey of FFRDC sponsors. As  shown in figure 4, during this period, the 16 DOE-sponsored FFRDCs  received about $66 billion in total funding, the 10 DOD-sponsored  FFRDCs received about $15.7 billion in total funding, and the 4 NSF- sponsored FFRDCs received about $1.8 billion in total funding. Each  year, DOE, DOD, and NSF collectively obligated between $12 and $15  billion to the 30 FFRDCs they sponsored over this period.
As shown in table 1, DOE obligated about 34 percent of its available  budget authority toward the FFRDCs it sponsored, while DOD and NSF  dedicated less than 1 percent and 4 percent, respectively, based on  obligations as a percent of budget authority from fiscal year 2008 through  fiscal year 2012. With regard to the significant share of DOE’s budget  being obligated to its FFRDCs, DOE officials noted that for many decades  the FFRDCs sponsored by the Department have been the primary means  for carrying out its research and development mission. While FFRDCs  represent a smaller proportion of the overall budget of DOD and NSF, in  some cases they represent a larger proportion of individual office budgets  at those agencies. For example, over one-third of the budget for NSF’s  Division of Atmospheric and Geospace Sciences is used to support  NCAR.
Most FFRDCs in our review received funding from entities other than their  sponsoring agency, and DOE-sponsored FFRDCs received more funds  from other entities than FFRDCs sponsored by DOD and NSF. For fiscal  years 2008 through 2012, FFRDCs sponsored by DOE, DOD, and NSF  received approximately $15 billion of the nearly $84 billion in total funding  from sources other than their sponsoring agency—specifically from other  federal agencies (about $13 billion) and nonfederal entities, such as state  or local governments, and private entities (about $1.9 billion). Figure 5  shows the total funding reported at each of the 30 FFRDCs for fiscal  years 2008 through 2012 and the amounts received from the sponsoring  agency, other federal agencies, and nonfederal entities.
section_title: Many FFRDCs Sponsored by DOE, DOD, and NSF Spent Over Half of Their Total Funding on Compensation, and These Agencies Have Processes to Review FFRDC Contractor Compensation
paragraphs: Many FFRDCs sponsored by DOE, DOD, and NSF spent over half of  their total funding on compensation for fiscal years 2010 through 2012,  and sponsoring agencies have processes to review compensation  annually to ensure that FFRDC contractors were not reimbursed in  excess of the cap amount set in federal law. These agencies also have  processes to review compensation more generally for all contractor  employees.
subsections: section_title: Compensation Costs Generally Represented Half or More of Total FFRDC Funding
paragraphs: For fiscal years 2010 through 2012, 20 of 30 FFRDCs sponsored by  DOE, DOD, and NSF spent over half of their annual funding on  compensation, based on our analysis of agency and contractor data  provided in response to our survey of FFRDC sponsors. According to  these data, compensation costs ranged from 27 to 79 percent of annual  funding at the FFRDCs sponsored by these agencies over this period. In  total, these costs provided about 71,000 full-time equivalents (FTE) per  year in fiscal years 2010 through 2012 at the 30 FFRDCs sponsored by  these three agencies. In 2012, the average compensation cost for each  FTE across the 30 FFRDCs was $135,694. In comparison, we previously  reported that total government-wide compensation for federal employees  for each FTE position averaged $116,828 in 2012. The average  compensation cost for each FTE at a single FFRDC ranged from $88,000  to $177,000 per year. FFRDC sponsors also reported that the maximum  salary received by executives at these FFRDCs ranged between  $200,000 and $600,000, as shown in figure 6. (App. II, table 6, provides  more information on compensation costs as a percentage of total FFRDC  funding, the compensation cost per FTE, and average FTEs across fiscal  years 2010 through 2012 at the 30 FFRDCs we reviewed.)
section_title: DOE, DOD, and NSF Have Processes to Review Compensation Reimbursements for Senior Executives and other FFRDC Contractor Employees
paragraphs: DOE, DOD, and NSF have processes to annually review compensation  reimbursements for senior executives against the federal statutory cap  and conduct general reviews of compensation for other employees.
DOE relies on its contracting officers to review senior executive  compensation reimbursements against the federal statutory cap, and,  until May of 2014, DOE generally did not document the total reimbursed  compensation subject to the cap for these senior executives. DOE has  additional processes in place to review compensation for all FFRDC  contractor employees.
Reviews of Senior Executive Compensation Reimbursements  Against the Statutory Cap  DOE contracting officers evaluate the reimbursed compensation of senior  executives to assess the applicability of the cap, and these evaluations  differ for two groups of senior executives at its FFRDCs—(1) laboratory  directors and (2) other executives—based on our discussions with DOE  officials and review of DOE policy on executive compensation. Prior to  May 2014, contracting officers evaluated the reimbursed portions of  senior executives’ pay and nonpay benefits separately, and DOE  officials told us that, in general, they did not document the total subject to  the cap, except at the Savannah River National Laboratory (SRNL). For  the laboratory directors, contracting officers evaluated reimbursable pay  annually by approving a required form submitted by the contractor that  documents the laboratory directors’ pay—salary and any bonuses or  incentives—and any proposed changes. For other FFRDC senior  executives, contracting officers evaluated reimbursable pay by approving  the form documenting salary, bonuses, and incentives only at the start of  the contract or when those individuals are hired. Thereafter, contracting  officers evaluated pay for these senior executives on an annual basis  through approval of a plan describing the contractor’s request for  reimbursement for pay for all FFRDC employees, called the  compensation increase plan. This plan describes pay for specific groups  and does not describe the salaries, bonuses, or other incentives received  by individual contractor senior executives. For example, ORNL’s  compensation increase plan describes pay for its senior executives within  a group of managers. Regarding nonpay benefits for senior executives,  DOE officials told us that the only nonpay benefits subject to the cap that  they receive are certain pension plans. Costs for these plans are based  on a percentage of the executive’s salary and the plans are approved by  the contracting officers. According to these officials, since the contracting  officers approved reimbursable pay and the pension plans, they could  calculate the laboratory directors’ total reimbursed compensation subject  to the cap and estimate it for other executives.
In May 2014, DOE updated its policy on senior executive compensation  to require documentation of all the laboratory directors’ pay and nonpay  benefits subject to the cap in addition to the reviews described above. For  other FFRDC senior executives under a compensation increase plan,  officials told us that DOE limits contractor compensation to amounts less  than the laboratory director by establishing the director’s salary as the  maximum allowable at each FFRDC and requiring contractors to report  the bonus and incentive pay for senior executives. Officials told us that  DOE previously did not require all its contractors to submit documentation  itemizing pay and nonpay benefits as was done at SRNL because the  reimbursable compensation for senior executives at the FFRDCs the  agency sponsors was well below the cap. These officials told us that,  because the reduction in the cap increases the risk that FFRDC  executives could receive pay subject to the cap, DOE decided to adopt a  process similar to that previously used at SRNL. The FFRDCs currently  sponsored by DOE are subject to the OFPP fiscal year 2012 calculated  cap of $952,308, until such time as their current contracts end. New  contracts awarded after June 2014 will be subject to the lower cap of  $487,000. Additional information on the current contract award and end  dates for these FFRDCs, and those sponsored by DOD and NSF, are  shown in appendix II (table 7).
Other Reviews of Compensation for FFRDC Contractor Employees  In addition to reviewing reimbursed compensation for senior executives,  DOE officials told us that they also review various aspects of employee  compensation for all FFRDC contractors. DOE officials annually approve  the FFRDC contractors’ requests for pay increases through the  compensation increase plan, described above, and review and approve  the contractors’ overall compensation systems at least once during the  term of the contract. The compensation increase plan includes supporting  information for the contractor’s request, such as comparisons of average  pay for groups of FFRDC employees against market averages for similar  groups. For example, the compensation increase plan for ORNL  describes the average pay for its administrative support employees and  for its managers relative to average pay for those groups derived from  market surveys. Compensation increase plan proposals, and the salary  survey data used by the FFRDC contractor, are evaluated by DOE  officials, who ensure that proposed funds are reasonable and will  maintain appropriate comparability with market averages from the survey  data. DOE officials also review an additional report provided each year by  the FFRDC contractor that includes actual compensation reimbursed for  groups of FFRDC employees to ensure compliance with the FFRDC  compensation increase plans that DOE previously approved.
DOD relies on audits by DCAA to review documented compensation for  senior executives and other employees against the cap on  reimbursements, and DCAA and DCMA review the overall pay and  nonpay benefits at DOD-sponsored FFRDCs.
Reviews of Senior Executive Compensation Reimbursements  Against the Statutory Cap  DOD relies on audits by the DCAA to review documented compensation  for FFRDC employees against the cap on reimbursements. DCAA  retrospectively evaluates the compensation of executives—including  senior executives—separately from other groups of contractor  employees. DCAA officials told us that they annually audit costs  incurred on DOD contracts, and the review of reimbursements for  executive compensation against the cap is a standard step in these  audits. According to the document DCAA uses to assist auditors in  planning and performing these audits, DCAA compares executive  compensation claimed for reimbursement to prior years, determines if  there are significant changes from prior years, or if claimed executive  compensation is over the statutory cap on compensation. As part of this  review, DCAA auditors told us that they obtain documented information  from the contractor on the itemized components of pay and nonpay  benefits for individual executives and amounts claimed for reimbursement  and verify that contractors correctly calculated the allowable portion of  compensation costs for each executive. DCAA officials may also request  assistance from another DCAA team specializing in executive  compensation audits. Auditors from this team told us that they evaluate  compensation for employees or classes of employees against the FAR  criteria on allowability, including compliance with the cap on  compensation reimbursements, as well as their reasonableness. From  2009 to 2013, this team evaluated, or is in the process of evaluating,  executive compensation at 5 of the 10 DOD-sponsored FFRDCs. In the  completed evaluations, DCAA found no instances of reimbursements  above the cap. For other contractor employees, DCAA’s incurred cost  audits include evaluations of the contractors’ overall compensation  system and internal controls, including the applicability of caps on  compensation reimbursements. According to DOD and DCAA officials,  few employees at DOD-sponsored FFRDCs receive compensation near  the cap. For example, DOD officials responding to our survey stated that,  of the 10 DOD-sponsored FFRDCs, 5 FFRDCs operated by three  different contractors had a single senior executive employee with total  compensation—including unreimbursed amounts—above the cap in place  for fiscal years 2010 and 2011. Further, DCAA officials who audit 1 of  the DOD-sponsored FFRDCs told us that the lower cap enacted in 2013  would affect only a few employees at that FFRDC contractor.
Other Reviews of Compensation for FFRDC Contractor Employees  In addition to reviewing reimbursed compensation against the statutory  cap, DCAA also evaluates compensation for individual employees or  classes of employees, including executives, against the FAR criteria for  reasonableness, such as conformity with compensation practices of other  firms of the same size, industry, and geographic area. DOD and DCAA  officials told us they analyzed the reasonableness of compensation costs  as part of the annual incurred cost audits and during negotiation of the  contract. The audit team that specializes in assessing executive  compensation evaluates reasonableness by confirming executive  compensation amounts and comparing them against surveys of  compensation at similar firms. DCAA auditors also examine the  reasonableness for other groups of employees if audits of the contractor’s  overall compensation system and internal controls over compensation  find that those systems cannot be relied upon to demonstrate reasonable  levels of compensation and that the potential for unreasonable  compensation exists. DOD also utilizes DCMA to evaluate some DOD  FFRDC contractors’ indirect costs, which include pension and other  benefit costs.
Most NSF-sponsored FFRDCs are not subject to the federal statutory cap  on compensation reimbursements because they operate under  cooperative agreements rather than contracts. Nonetheless, NSF  reported that none of the four FFRDCs have senior executives with  compensation above the cap and that NSF has procedures to review  compensation at the FFRDCs it sponsors. NSF utilizes audits by DCAA to  review documented compensation for executives at the one NSF- sponsored FFRDC that operated under a contract, and NSF conducts  various reviews of compensation for all FFRDC employees.
Reviews of Senior Executive Compensation Reimbursements  Against the Statutory Cap  NSF has different procedures across the four FFRDCs it sponsors to  review senior executive compensation. For the Science and Technology  Policy Institute (STPI), the only NSF-sponsored FFRDC that operates  under contract and is thus subject to the cap, DCAA reviews documented  compensation because the FFRDC is operated under a contract by the  Institute for Defense Analyses, which also operates FFRDCs sponsored  by DOD that are subject to DCAA audit. For the three FFRDCs awarded  through cooperative agreements, the cap does not apply. However, the  agreements for these FFRDCs require that NSF approves the salary for  the FFRDC directors. In addition, these agreements require that the  FFRDC contractors notify NSF of all other forms of compensation for  directors at two of these three FFRDCs and stipulates that no part of the  management fee received by the third FFRDC may fund the director’s  salary. NSF officials also noted that they had performed a salary  comparison for the NCAR director based largely on survey data on  salaries for college and university administrators. NSF responded to our  survey that none of its FFRDCs had executives with compensation above  the cap.
Other Reviews of Compensation for FFRDC Contractor Employees  NSF officials told us that they conduct various reviews of compensation  for all FFRDC employees, including reviews before awarding and during  the period of performance of an FFRDC contract or agreement. For STPI,  the one NSF-sponsored FFRDC operating under a contract, NSF, in  response to our survey asking about steps they take to evaluate  compensation, told us that they review the compensation rates charged  by the FFRDC annually to ensure they are fair and reasonable. In  addition, DCAA and DCMA review the compensation costs for STPI, as  they do for DOD-sponsored FFRDCs. For the three FFRDCs under  cooperative agreements, NSF officials told us that they reviewed a  sample of compensation costs from the proposed budget before awarding  new cooperative agreements. For example, before awarding the most  recent cooperative agreement sponsoring NCAR, NSF officials reviewed  NCAR’s budget and financial capability, including the total compensation  plan, positions with corresponding salary grades, and the FFRDC  operator’s procedures for determining compensation rates. NSF also  reviewed contributions to retirement and medical benefits at NCAR and  found that the FFRDC operator appeared to have made a thorough and  detailed examination of salary rates and levels and that its policies and  procedures for salary rates and total compensation were generally  adequate. NSF has also incorporated terms regarding compensation into  its cooperative agreements. For example, the cooperative agreements for  NCAR and the National Optical Astronomy Observatory require the  FFRDC operator to notify NSF of any substantive changes to its  compensation policies and describe the rationale for any changes.
section_title: DOE, DOD, and NSF Use Multiple Reviews to Assess FFRDC Performance
paragraphs: DOE, DOD, and NSF assess FFRDC performance through three types of  reviews—comprehensive reviews, annual performance reviews, and  other review activities such as day-to-day oversight.
subsections: section_title: DOE, DOD, and NSF Conducted Comprehensive Reviews of the Use and Need for FFRDCs That Vary in Level of Detail
paragraphs: DOE, DOD, and NSF conducted comprehensive reviews of the use and  need for the FFRDCs they sponsored and, in keeping with the flexibilities  the FAR provides, these reviews varied in level of detail. The FAR  requires sponsoring agencies to conduct a comprehensive review at least  every 5 years on the use and need for an FFRDC prior to extending the  contract or agreement, and it describes five elements it should include:  (1) an examination of the special technical needs and mission  requirements performed by the FFRDC, (2) a consideration of alternative  sources, (3) an assessment of the efficiency and effectiveness of the  FFRDC in meeting the sponsor’s needs, (4) an assessment of the  adequacy of the FFRDC management in ensuring a cost-effective  operation, and (5) a determination that the sponsoring agreement or  contract complies with other FAR requirements concerning FFRDCs.  Within those five elements, the FAR gives agencies the flexibility to  determine how much information to include and how to document the  review.
Based on our analysis of agency documents, DOE, DOD, and NSF met  the FAR requirement to conduct a comprehensive review of the use and  need for the FFRDCs they sponsored in most instances and, in all cases,  they recommended the continuance of the FFRDC. DOE has conducted  comprehensive reviews for all 16 of its FFRDCs; however, in two cases,  DOE did not meet the FAR requirement to conduct a comprehensive  review at least every 5 years. Los Alamos and Lawrence Livermore  National Laboratories were the two exceptions. DOE last conducted  comprehensive reviews for Los Alamos in 2005 and Lawrence Livermore  in 2007. The department has not conducted additional reviews within the  5-year period required by the FAR. DOE officials told us that NNSA,  which is responsible for conducting the reviews of the two laboratories, is  updating its comprehensive reviews, and the officials expect them to be  finalized in the next few months. DOD met the FAR requirement to  conduct a comprehensive review on the use and need for all 10 of the  FFRDCs we reviewed. NSF also met the FAR requirement to conduct a  comprehensive review on the use and need for the 1 FFRDC it operates  by contract. These comprehensive reviews stated that all of the  FFRDCs were performing effectively or adequately and either  recommended or supported continuing the FFRDC. Further, DOD  officials told us that all of its 10 FFRDCs have retained the same  contractor throughout the existence of their FFRDCs, and NSF officials  said that 3 of its 4 FFRDCs have retained the same contractor. Since  2004, DOE has competed 6 of its 16 FFRDC contracts and replaced the  incumbent contractor in four instances.
In keeping with the flexibilities that the FAR provides, DOE, DOD, and  NSF documented the results of their most recent comprehensive reviews  to varying degrees of detail and in how much information they included in  addressing the FAR elements, as follows:
DOE. DOE’s comprehensive review documents generally followed a  consistent format that provided general statements for each of the five  FAR elements. The comprehensive reviews provided more  information regarding the department’s technical needs and mission  requirements for the FFRDC than they did for the other FAR  elements, with almost all of the 16 FFRDC reviews documenting this  element in detail. Most of DOE’s comprehensive reviews partially  documented the other four FAR elements. For example, as part of the  discussion on the efficiency and effectiveness of the FFRDC for  ORNL, the comprehensive review described DOE’s satisfaction with  the FFRDC contractor by providing the letter grades received by  ORNL based on DOE’s annual appraisal process to evaluate the  scientific, technological, managerial, and operational performance of  the contractors who manage and operate each of its national  laboratories. ORNL’s comprehensive review did not provide additional  details on the outcome of the annual appraisals.
DOD. DOD’s comprehensive review documents generally  documented each of the five elements in detail. Seven of the nine  DOD comprehensive reviews contained supporting evidence on all  five FAR elements, including a rationale for their determinations  relating to each element. For example, regarding the FAR element  to examine alternative sources, the NSEC comprehensive review  described the relative merits of using DOD personnel, for-profit  contractors, and other nonprofit contractors to complete the work  currently done by NSEC. The other two reviews were partially  documented, including limited supporting evidence regarding the  efficiency and effectiveness of the FFRDC. One of these reviews also  included limited supporting evidence related to the cost-effectiveness  of the FFRDC and no documentation or discussion related to  compliance of the sponsoring agreement with the FAR requirements.
NSF. The comprehensive review for NSF’s Science and Technology  Policy Institute—according to NSF officials, the only one of NSF’s  FFRDCs subject to the FAR requirement for a comprehensive  review—included all five FAR elements and provided detailed  information regarding the sponsor’s needs and the efficiency and  effectiveness of the FFRDC and a limited discussion on alternative  sources and the FFRDC’s cost-effectiveness.
DOE and DOD officials have developed guidance on how to conduct  comprehensive reviews, and NSF officials stated that the FAR informs  their reviews of NSF-sponsored FFRDCs. DOE’s guidance calls for a  review of the use and need for continued operation as an FFRDC in  accordance with the FAR, without additional elaboration on these FAR  requirements. This guidance states that the comprehensive review should  be completed as part of the agency’s process for deciding if it will extend  or compete an FFRDC contract when the current contract’s term has  ended. This guidance also calls for the preparation of an alternatives  package 24 months before the FFRDC’s contract expires, consisting of  an action memo for the Secretary to approve and other documentation,  for example a discussion of the incumbent FFRDC contractor’s  performance history, including technical, administrative, and cost  performance. DOE officials stated that FFRDC reviews are required  every 5 years in accordance with the FAR requirements, regardless of  award provisions and competitions for the FFRDC contract.
DOD has issued guidance on how to conduct comprehensive reviews in  the form of a memorandum to DOD offices and divisions on the  management of FFRDCs. DOD’s guidance was issued to clarify the FAR  requirements, assist DOD officials with the oversight and management of  the FFRDCs they sponsor, and ensure consistency across the  department. It includes references to certain FAR elements relating to  the execution of comprehensive reviews and adds additional  requirements for DOD sponsors. Among other things, the guidance  assigns responsibilities to the Under Secretary of Defense for Acquisition,  Technology and Logistics to establish and promulgate FFRDC  procedures; allocate the number of staff years and funding within the  annual ceiling set by Congress for all DOD-sponsored FFRDCs; and  review and approve all comprehensive reviews for FFRDCs prior to a  contract renewal decision. In addition, the guidance provides direction to  DOD sponsors for reporting the results of the FFRDC comprehensive  reviews and, according to DOD officials, helps DOD foster the optimal  and appropriate management of FFRDCs in accordance with law,  regulation, and policy. The guidance directs DOD sponsors to establish  annual governance procedures to monitor the value, quality, and  responsiveness of FFRDC work. DOD officials added that the guidance  helps make sure that the sponsors are paying attention to the quality of  the products they are getting from the FFRDC, and it allows the  department to state with confidence that the services provided by the  FFRDC are needed.
As stated earlier, according to NSF, the specific provisions in the FAR  requiring a comprehensive review do not apply to three of its four  FFRDCs because they are established under cooperative agreements.  However, NSF officials said that the agency carries out a midpoint review  for these three FFRDCs similar to the comprehensive review and its  associated FAR requirements. For example, per the terms of the  cooperative agreement with NCAR, NSF initiates a midpoint review once  during the 5-year term of the agreement with the FFRDC operator.  According to NSF officials, the midpoint review includes assessments of  the management, science, and service functions by multiple teams of  independent experts based on criteria given to them by NSF. The officials  said that NSF has used the results of the midpoint review to inform  changes to the cooperative agreement. For example, the most recent  review was critical of the FFRDC operator’s efforts to coordinate and plan  their educational programs, so NSF included specific requirements to  increase coordination among these programs in the subsequent  cooperative agreement.
DOE and DOD officials provided different explanations on how the  comprehensive review process supports their assessments of FFRDC  performance. DOE officials described the comprehensive review as a  summation largely derived from the findings of its annual performance  appraisals, and not a stand-alone evaluation or an exhaustive description  of the FFRDC contractor’s performance. According to those officials, the  DOE comprehensive reviews rely on annual performance reports to  provide a multiyear assessment. In contrast, DOD officials and contractor  representatives described the comprehensive review process as an  opportunity to take a broad assessment of the FFRDC and its key  competencies over and above DOD’s annual assessments of its  FFRDCs. For example, NSEC’s comprehensive review served as an  opportunity for the department to recognize that NSEC had developed  expertise in countering improvised explosive devices, that DOD had an  ongoing need for this work, and that DOD should elevate the importance  of this work. In another example, the last comprehensive review for the  Software Engineering Institute identified concerns about the execution  and ability of the FFRDC to meet the sponsor’s need for more research  engineers. The comprehensive review recommended a revised  sponsoring agreement with substantial changes in governance and to the  core statement. It also recommended removing most of the process  improvement work from the FFRDC and emphasizing research. DOD  implemented the recommendation to remove most process improvement  work and the subsequent sponsoring agreement emphasized research.
section_title: DOE, DOD, and NSF Conduct Annual Performance Assessments, Many of Which Rely on User Surveys
paragraphs: DOE, DOD, and NSF annually review and document the performance of  the FFRDCs they sponsor. Annual reviews are conducted to evaluate the  performance of contractors who manage and operate the FFRDCs.
DOE. DOE develops annual performance plans and undertakes an  annual appraisal process to evaluate its FFRDCs, including surveying  federal officials responsible for the work at the DOE-sponsored centers.  With respect to the 10 FFRDCs sponsored by DOE’s Office of Science,  DOE officials conduct an annual appraisal process to evaluate the  scientific, technological, managerial, and operational performance of the  contractors who manage and operate each of its national laboratories.  According to DOE officials, these evaluations provide the basis for  determining performance fees and the possibility of earning additional  years on the contract through an award term extension. They also serve  to inform the decisions DOE makes regarding whether to extend or  compete the contracts when they expire. For each of the FFRDCs  sponsored by the Office of Science, DOE officials told us that they  prepare a performance evaluation and measurement plan a few months  prior to the beginning of the fiscal year that provides a common structure  and scoring system using goals, objectives, and notable outcomes that  are weighted in importance by the appropriate program office or other  FFRDC users. These plans serve as the evaluation template for the  FFRDCs and their contractors at the close of the fiscal year. At the  conclusion of the fiscal year, a performance and evaluation report is  developed for each DOE FFRDC that provides performance scores and  the amount of performance-based fee earned by the contractor.
DOD. DOD officials are responsible for annual reviews of the FFRDCs  sponsored by the department. Under DOD guidance, sponsors are to  annually assess the technical quality, responsiveness, value, cost and  timeliness of FFRDCs. According to DOD officials, the guidance does not  specify how sponsors are to complete the annual FFRDC reviews;  however, almost all DOD sponsors use surveys to obtain input about the  FFRDC’s performance in support of these annual reviews, including the  sponsors for NSEC. The timing of the annual review is determined by the  DOD sponsors. For example, NSEC’s annual survey and associated  review is typically completed by the end of March, at which time it is  provided to the DOD sponsor for review. In addition, DOD officials  compare FFRDC user survey responses with survey responses provided  by the FFRDC contractor, which also provides input to the survey, to  identify any inconsistencies in performance. Some of the information from  the annual surveys may be used in support of the comprehensive review;  for example, all of the last 5 years of annual survey responses provided  by Army and Air Force were used to support the findings of NSEC’s latest  comprehensive review.
NSF. On an annual basis, NSF’s FFRDC operators submit to NSF a  retrospective review, called a program operating plan progress report, on  FFRDC performance. This review describes progress made at an FFRDC  based on milestones, objectives, targets, and deliverables described in a  program operating plan, among other things. For example, according to  NSF officials, for NCAR this review shall be submitted for review and  approval by November 15 after the end of each fiscal year, and it is  conducted by both NSF and the FFRDC operator and is reviewed by NSF  officials. In addition, the FFRDC operator submits an annual management  report to NSF that focuses on the FFRDC operator’s management  activities for the year. This review describes the FFRDC operator’s  performance of specific duties and its progress in meeting objectives. The  format and content of this report is developed in consultation with NSF  officials, and it is to be submitted within 90 days prior to the end of each  fiscal year.
section_title: DOE, DOD, and NSF Engage in Other Ongoing Activities Critical to Assessing FFRDC Performance
paragraphs: Officials from DOE, DOD and NSF described the day-to-day interactions  between agency and contractor officials, as well as other review activities  as critical to assessing FFRDC performance.
DOE officials told us that agency site office officials co-located with the  contractors are primarily responsible for assessing business and  operational performance at their respective FFRDCs, and DOE’s program  officials assess performance of the mission work.
According to DOE officials at ORNL, for example, the agency’s site office  officials manage the contract with the FFRDC contractor and the  resources associated with that effort; perform operational oversight and  ensure regulatory compliance of contractor operations; perform regulatory  compliance work and manage resources to ensure efficient operation;  and provide stewardship of ORNL assets, such as research equipment  and buildings. According to DOE’s ORNL site office officials and  documents, review activities include routine, day-to-day monitoring  through facility tours, walk-throughs, surveillances, work observations,  document reviews, meeting attendance and participation, and ongoing  interaction with contractor workers. These site office officials also conduct  structured evaluations of contractor programs and performance to confirm  that operations are being conducted in a safe and efficient manner and  are in compliance with contract requirements. DOE’s ORNL site office  and FFRDC contractor employees said that the standard set of  documentation associated with performance and oversight, such as the  annual performance plans and reports, is not fully sufficient to understand  the full range of performance and associated oversight being performed  by ORNL site office employees. DOE’s program office officials associated  with ORNL assess the quality of the laboratory’s work through several  types of evaluations, including reviews of project outcomes and peer  reviews of the scientific work. For example, DOE’s program officials  monitor performance through regular meetings with FFRDC technical  staff to discuss specific projects. They also oversee peer-reviews that are  performed on a triennial basis, working with experts external to the  program from academia or other DOE laboratories to assess the quality  of the work performed by the contractor for specific research projects.
According to DOE officials, FFRDC performance is also monitored as a  part of its work authorization system, which DOE uses as a control  process for work performed by contractors. Authorizations for most  DOE-funded work must be documented in work authorizations, and each  authorization must include a detailed statement of work to be performed  and associated performance-based indicators or targets, including any  additional guidance that will assist the contractor in the performance of  the work. For example, the delivery of the work could be the publication of  a paper in a scientific journal, and DOE would use peer review  committees to determine if the FFRDC contractor met the objective as  stated in the work authorization. FFRDC ORNL site office officials are  responsible for authorizing the work put forth in work authorizations, while  DOE program office officials set the technical specifications and evaluate  the technical progress of the work.
DOD officials described a number of other activities performed as part of  its ongoing oversight of its FFRDCs. For example, DOD officials said that  they use a “tiered accountability” framework or structure when overseeing  the contract to manage NSEC, which includes:
Day-to-day oversight. DOD program managers, contracting officers,  and contracting officer representatives told us they perform some day- to-day oversight activities, including funding on the contract and  monitoring the allocation of STE, or 1 year of fully burdened labor,  charged against the contract throughout year. Sometimes, small  adjustments to the STE levels are made, being driven by the overall  budget or priorities of the program. As part of this oversight, DOD  officials look at the scope of work on all projects and compare them  against the core work statements as defined in the FFRDC  sponsoring agreements to make sure that work is within the scope for  the FFRDC.
Working with the DCAA. DOD officials told us the department’s  administrative contracting officers work closely with DCAA on funding  reports and on audits of funding.
Negotiating indirect rates. The DOD procuring contract officer  negotiates FFRDC contractor’s indirect rates under the NSEC  contract.
In addition, with respect to NSEC, DOD’s program managers told us that  they request an annual and semiannual feedback and assessment for the  work being done by the FFRDC on behalf of the Army and the Air Force.  DOD officials also noted the importance of regular user forums between  FFRDC staff and DOD users. In addition to the user surveys, DOD  officials told us that the performance for NSEC’s FFRDC contractor is  overseen through daily interactions between DOD and the FFRDC  contractor staff at the project level. In addition, monthly or weekly  meetings take place between DOD program and FFRDC contractor  employees at the division level, and DOD and FFRDC contractor  management officials also meet quarterly to discuss the overall  performance of NSEC.
According to NSF officials, the agency’s oversight role with its FFRDCs is  specified in the cooperative agreements NSF has with the FFRDCs.  Generally, the terms and conditions laid out in the cooperative agreement  provide NSF’s oversight guidelines. NSF officials said that they do not  have employees collocated or on-site at the FFRDCs to perform  oversight. Instead, NSF officials travel to the FFRDCs and FFRDC  contractor employees travel to NSF headquarters to exchange  information on the oversight of the FFRDCs.
section_title: Agency Comments
paragraphs: We provided a draft of this product to DOE, DOD, and NSF for their  review and comment. Formal comments were not provided by DOE,  DOD, and NSF. Technical comments provided by DOE, DOD, and NSF  were incorporated, as appropriate.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of Energy,  Secretary of Defense, Director of the NSF, and other interested parties. In  addition, the report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or neumannj@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made key contributions to  this report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: We selected a non-probability sample of Federally Funded Research and  Development Centers (FFRDC) for review based on the master list of  FFRDCs compiled by the National Science Foundation (NSF). We chose  to review FFRDCs sponsored by the Department of Energy (DOE),  Department of Defense (DOD), and NSF because these three agencies  sponsor the largest numbers of FFRDCs—16, 10, and 4, respectively.  Our findings related to these 30 FFRDCs are not generalizable to the  complete set of 40 FFRDCs in operation in 2014 (see fig. 1). To select  FFRDCs for site visits and additional review from among the 30  sponsored by DOE, DOD, and NSF, we identified the FFRDCs with the  highest research and development expenditures reported by the NSF for  fiscal year 2010 within each agency. The FFRDCs we chose for site  visits at each agency were: (1) Oak Ridge National Laboratory, run by  UT-Battelle, LLC, on behalf of DOE’s Office of Science; (2) the National  Security Engineering Center, run by the MITRE Corporation on behalf of  DOD; and (3) the National Center for Atmospheric Research, run by the  University Corporation for Atmospheric Research on behalf of NSF. We  evaluated additional documentation regarding the sponsorship of these  FFRDCs and their performance.
To learn about FFRDC funding and compensation we surveyed the  agency sponsors for the 30 FFRDCs in our scope, spoke to agency  officials and contractor representatives, and reviewed agency policy and  guidance documentation. We developed a standard questionnaire to  obtain information on the obligations and funding received by FFRDCs,  full-time equivalents (FTE), compensation costs at each FFRDC, and the  controls agencies have in place to review or assess compensation. We  asked for information on obligations and funding for the 5 most recent  fiscal years available at the time of our review, fiscal years 2008 through  2012. We asked for information on employees and compensation for  fiscal years 2010 through 2012—the 3 years for which data were  available and the cap on compensation was set by regulation (i.e., before  the statutory cap had been implemented)—and sponsoring agency efforts  to review compensation.
Our survey of FFRDC sponsors asked for obligations provided to the  FFRDC by the federal sponsoring agency, obligations provided to the  FFRDC from all other federal government departments and agencies, and  funding provided to the FFRDC from nonfederal, governmental entities  (such as state or local governments) and nongovernmental entities for  fiscal years 2008 through 2012. Our survey of FFRDC sponsors also  asked for the total FFRDC employee compensation costs, including  salary and nonpay benefits such as pensions, insurance, or fringe  benefits, paid by the FFRDC contractor, for the total FTE employment  during fiscal years 2010 through 2012, and for the range of salaries for  senior executives whose compensation is charged or allocated to the  FFRDC contract or agreement.
We pretested the questionnaire by phone with agency officials who had  responsibilities for overseeing FFRDCs at DOE, DOD, and NSF in order  to ensure that the survey questions and definitions were clear and that  the information could feasibly be obtained. We made changes to the  content of the questionnaire after each review based on the feedback we  received. The questionnaire was distributed to agency liaisons in  September 2013, and responses for all 30 FFRDC were received by  December 2013. We reviewed the survey responses of agency and  contractor data for consistency and reliability by examining information  provided by the respondents on their data sources, the steps taken to  ensure the accuracy and completeness of the data provided, and any  known limitations. We also interviewed knowledgeable agency officials  about the data, and obtained follow-up information by phone and e-mail  from agency sponsors when necessary between January 2014 and April  2014. Agency officials submitted written clarifications to the original  survey responses and follow-up questions, as necessary. Data obtained  from the survey was also presented to the agencies for review prior to  reporting. We found that the data were sufficiently reliable for our  purposes.
To assess agency management and oversight of FFRDC operator  performance, we conducted interviews with agency officials at DOE,  DOD, and NSF and contractor officials at ORNL, NSEC, and NCAR to  determine how performance was assessed and reviewed relevant  documentation of those activities. These interviews included officials from  DOE’s National Nuclear Security Administration, Office of Science, and  Office of Management; DOD’s Office of Acquisition Resources and  Analysis; and NSF’s Office of Budget, Finance, and Award Management  who have oversight responsibilities for multiple FFRDCs. We also  interviewed agency officials with direct oversight responsibilities for  ORNL, NSEC, and NCAR, including the Oak Ridge Site Office, Deputy  Assistant Secretary of Defense for Systems Engineering, and Division of  Atmospheric and Geospace Sciences.
We obtained the most recent comprehensive reviews performed by the  agency for 26 FFRDCs in our sample as of September 2013 and  evaluated them against the elements described by the FAR. We received  reviews completed between 2005 and 2013. Some FFRDCs included in  our review were not subject to the requirement to perform a  comprehensive review, according to their sponsoring agency. Because  the FAR applies only to federal government acquisitions made by  contract, cooperative agreements between sponsoring agencies and an  FFRDC operator are generally not subject to requirements in the FAR.  Three FFRDCs sponsored by NSF operate under cooperative  agreements. According to NSF officials, the agency is not required to  perform a comprehensive review for these FFRDCs and we excluded  them from our analysis. In addition, we excluded the comprehensive  review for the Center for Communications and Computing FFRDC due to  security classification of material within the document. For the remaining  26 FFRDCs, we examined the comprehensive reviews for the extent that  the review documented and supported the five elements described by the  FAR. Two analysts independently categorized the extent of information  provided for each element into three categories. If the documentation of  the comprehensive review did not discuss the element at all, we  categorized it as “Not Documented.” If the documentation in the  comprehensive review provided a determination or finding related to the  element, but it did not fully address the criteria or provide supporting  evidence, we categorized it as “Partially Documented.” If the review  provides a determination or finding for the element, and provides  additional supporting evidence, we categorized it as “Documented.” Three  analysts, including those that performed the initial review, then examined  any differences between the two reviews and made a final determination  for each comprehensive review and element.
We conducted this performance audit from December 2012 to July 2014  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Funding and Contract Data for FFRDCs Sponsored by DOE, DOD, and NSF
paragraphs: To learn about funding and compensation at federally funded research  and development centers (FFRDC) we surveyed the agency sponsors for  the 30 FFRDCs sponsored by the Department of Energy (DOE),  Department of Defense (DOD), and National Science Foundation (NSF)  and collected information on their operators. Our survey asked for  obligations to the FFRDC by the federal sponsoring agency, obligations  provided to the FFRDC by all other federal government departments and  agencies, and funding provided to the FFRDC from nonfederal,  governmental entities (such as state or local governments) and  nongovernmental entities for fiscal years 2008 through 2012. Table 2  provides the total obligations and funding to the 30 FFRDCs in our scope.  Table 3 provides the obligations by the federal sponsor. Table 4 provides  the obligations by all other federal government departments and  agencies. Table 5 provides the funding from nonfederal entities. Table 6  provides data responding to our survey questions on the total FFRDC  employee compensation costs paid by the FFRDC contractor and the  total FTE employment during fiscal years 2010 through 2012. Table 7  describes the 30 FFRDCs we examined, their current operators, and start  and end dates of their current contracts or cooperative agreements.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Dan Haas (Assistant Director),  John Barrett, Kevin Bray, Alysia D. Davis, R. Eli DeVan, Cindy Gilbert,  Paul Kinney, Alison O’Neill, Kelly Rubin, Tind Shepper Ryen, Aaron  Shiffrin, Kiki Theodoropoulos, Jack Wang, and Bill Woods made key  contributions to this report.