Federal Research: DOD's Use of Study and Analysis Centers
Released: 2019-12-09T00:00:00

Letter

Background

FFRDCs arose from partnerships between the federal government and academic researchers and scientists during World War II. Those partnerships were later restructured into federal research centers to retain scientists, and they became known as FFRDCs by the mid-1960s. Since that time, FFRDCs have continued to perform tasks including technical studies and analyses, research and development, and systems engineering on behalf of federal agencies, such as DOD. In sponsoring an FFRDC, agencies draw on academic and private sector resources that can contribute to an agency’s ability to accomplish tasks that are integral to the mission and operation of the sponsoring agency. FFRDCs may be operated, managed, and/or administered by a university or consortium of universities, other nonprofit organizations, or a private industry firm as an autonomous organization or as a separate unit of a parent organization.

As of May 2019, federal agencies sponsored a total of 42 FFRDCs, 10 of which are sponsored by DOD. These 10 DOD-sponsored FFRDCs can be divided into three categories:

S&A Centers: These centers deliver independent and objective analyses and advise in core areas important to their sponsors in support of policy development and decision-making, among other things.

Research and Development Laboratories: These laboratories conduct research and development, focusing on the development and prototyping of new technologies and capabilities to meet DOD needs. For example, these laboratories engage in research programs that emphasize the evolution and demonstration of advanced concepts and technology, and transfer new technology to the private sector.

Systems Engineering and Integration Centers: These centers meet long-term technical and engineering needs to ensure complex systems meet operational requirements. Among other things, Systems Engineering and Integration Centers assist with testing system performance, development and acquisition of system hardware and software, integration of new capabilities, and the continuous improvement of system operations and logistics. Table 1 lists the 10 DOD-sponsored FFRDCs.

As shown in table 1, each of the 10 DOD-sponsored FFRDCs is managed by a specific military department or organization within DOD—referred to as the FFRDC primary sponsor. More broadly, the Office of the Under Secretary of Defense for Research and Engineering oversees and manages DOD’s FFRDC program.

Sponsoring Agreements

DOD’s relationships with FFRDCs are defined through sponsoring agreements between the primary sponsor (i.e., the DOD organization responsible for the overall use of the FFRDC) and the FFRDC parent organization. According to the FAR and DOD instruction, sponsoring agreements define the FFRDC’s purpose and mission and may not exceed 5 years in duration. DOD’s instruction also states that sponsoring agreements are to establish conditions under which DOD may award an FFRDC contract and describe the overarching requirements for operation of the FFRDC. For example, the DOD instruction states that sponsoring agreements are to describe constraints on the FFRDC parent organization that are necessary to preserve the integrity of the FFRDC, such as provisions to prevent the occurrence or appearance of organizational or personal conflicts of interest that may undermine the independence, objectivity, or credibility of the FFRDCs. The DOD instruction also states that sponsoring agreements will preclude FFRDCs from performing commercial work. In this regard, the FAR provides that sponsoring agreements are required to address whether or not the FFRDC may accept work from other entities and if so, the procedures to be followed and the limitations as to the work that can be accepted. Further, the DOD instruction and the FAR provide that sponsoring agreements will generally preclude FFRDCs from competing with any organization in response to a formal request for proposals other than the operation of the FFRDC. After the primary sponsor identifies the need for FFRDC work, and has defined FFRDC core competencies, roles, and responsibilities in the sponsoring agreement, the primary sponsor awards a noncompetitive contract to the FFRDC to support the sponsor’s research requirements, such as addressing national security issues and systems development.

Comprehensive Reviews

Prior to extending a contract or sponsoring agreement for an FFRDC, the FAR requires that the primary sponsor conduct a comprehensive review of the use and need for the FFRDC at least every 5 years. The FAR describes elements of what the comprehensive review should include, such as examination of the sponsor’s special technical needs and mission requirements performed by the FFRDC and assessment of the efficiency and effectiveness of the FFRDC in meeting the sponsor’s needs. The FAR further requires that the head of the sponsoring agency approve continuing or terminating sponsorship based on the results of the comprehensive review.

Initiating Work at FFRDCs

FFRDCs initiate work on specific projects at the request of “work sponsors,” or the entities that request the services of the FFRDC. Work sponsors can be the primary sponsor of the FFRDC or another entity. When initiating work at FFRDCs, the primary sponsor determines whether to approve research projects for the FFRDC before projects are placed on contract. Approval of research projects is based on the determination that work proposed is appropriate for the FFRDC and consistent with the FFRDC’s core competencies as documented in the sponsoring agreement. Additionally, the primary sponsor ensures FFRDC work efforts do not exceed available resources. Among other things, FFRDC work sponsors identify project requirements, propose an appropriate research design, confirm the work is appropriate and consistent with FFRDC core competencies, identify the source of project funding, and monitor the progress of the work to ensure FFRDC performance is satisfactory and meeting desired requirements.

In some instances, S&A Centers serve only a specific military department or office, while in other cases an FFRDC may serve a range of DOD entities. For example, RAND Arroyo Center broadly supports the analytic requirements of the Army in order to provide timely advice to help senior Army leadership make informed policy choices. Accordingly, the RAND Arroyo Center sponsoring agreement with the Department of the Army provides that the scope of RAND Arroyo Center work is to support Army sponsors throughout the Army requiring comprehensive analytical support. In contrast, the Institute for Defense Analyses (IDA) and RAND National Defense Research Institute serve DOD more broadly on national security issues. For example, according to IDA’s sponsoring agreement with DOD’s Office of the Under Secretary of Defense for Acquisition and Sustainment, the primary mission of IDA is to assist the Office of the Secretary of Defense and other Defense organizations in addressing important national security issues, particularly those requiring scientific and technical expertise.

Staff Years of Technical Effort

DOD manages the overall level of FFRDC work using a metric known as staff years of technical effort (STE), which is roughly equal to the work of one employee working for 1 year. Congress typically sets an annual limitation on the STE that may be funded for DOD FFRDCs to support non-intelligence programs on behalf of the agency (hereafter, Defense STE). Between fiscal years 2013 to 2017, Congress established an annual ceiling of 5,750 Defense STE available to DOD, of which 1,125 could be allocated to S&A Centers. In fiscal year 2018, Congress raised the ceiling on Defense STE to 6,030; however, the limit on S&A Centers remained unchanged. In managing Defense STE, DOD: consolidates annual Defense STE requirements for each fiscal year based on projected primary sponsor requirements and submits STE requirements to Congress; establishes Defense STE allocations for each DOD-sponsored FFRDC and provides associated funding limitations to each primary sponsor; monitors Defense STE usage and associated obligations; and provides an annual report to Congress at the end of each fiscal year outlining the Defense STE funded and associated DOD funds obligated for each FFRDC.

In addition to Defense STE, FFRDCs may support DOD intelligence activities under the Military Intelligence Program and the National Intelligence Program. Oversight for STE usage for these programs is provided by the Office of the Under Secretary of Defense for Intelligence and Office of the Director of National Intelligence, respectively. Military Intelligence Program and National Intelligence Program STE funding may not be used to support Defense STE requirements.

In October 2008, we reported that Congress implemented the Defense STE ceiling during the 1990s in response to concerns that DOD was inefficiently using its FFRDCs. In addition, we found that STE ceilings aimed to ensure that FFRDC work was appropriate and that resources, which were limited, were being used on DOD’s highest priorities. In December 2018, we reported that officials in the Office of the Secretary of Defense’s Studies and FFRDC Management Office stated that the ceiling significantly constrains the use of DOD’s FFRDCs and that DOD customer demand for FFRDC services is significantly greater than the annual ceiling set by Congress. Further, officials indicated at that time that FFRDC-related work must be deferred to later years when the limits are reached, since there are no other legally compliant alternatives capable of fulfilling these requirements. We did not make any recommendations related to this issue.

Reviewing FFRDC Performance

Following the completion of FFRDC work, the primary sponsor, with assistance from the work sponsor, reviews FFRDC performance in written assessments via questionnaires. In addition, the primary sponsor assesses FFRDC performance annually, addressing the technical quality, responsiveness, value, and timeliness of the work performed. Some of the information from the annual reviews may be used in support of the comprehensive review, such as to demonstrate the efficiency and effectiveness of the FFRDC in meeting the primary sponsor’s needs.

DOD Obligated about $3 Billion per Year to DOD-Sponsored FFRDCs from Fiscal Years 2013 through 2018

From fiscal years 2013 through 2018, total DOD obligations to the 10 DOD-sponsored FFRDCs generally increased annually from about $2.7 billion in fiscal year 2013 to approximately $3.2 billion in fiscal year 2018. Approximately 70 percent of total annual DOD obligations to DOD- sponsored FFRDCs between these fiscal years went to support non- intelligence programs and were comprised of DOD obligations associated with utilized Defense STE, or Defense STE obligations. Specifically, DOD Defense STE obligations ranged from about $1.9 billion in fiscal year 2013 to $2.2 billion in fiscal year 2018, with S&A Centers representing approximately 18 percent of these obligations. In addition to DOD Defense STE obligations, about 30 percent of total DOD obligations to DOD-sponsored FFRDCs between fiscal years 2013 through 2018 went towards other FFRDC-related activities and costs, such as intelligence program activities through the Military Intelligence Program and National Intelligence Program and capital equipment costs. Figure 1 shows DOD obligations by fiscal year to DOD-sponsored FFRDCs.

For fiscal years 2013 to 2018, the FFRDCs we reviewed in-depth—DOD’s S&A Centers—collectively accounted for about 18 percent of DOD Defense STE obligations annually, whereas Research and Development Laboratory FFRDCs and Systems Engineering and Integration Centers accounted for 27 and 55 percent, respectively (see figure 2).

DOD Defense STE obligations to S&A Centers rose from about $320 million in fiscal year 2013 to approximately $380 million in fiscal year 2018 totaling about $2.3 billion during this period. Within each S&A Center, obligations remained relatively constant over the 6 years, with obligations for some FFRDCs higher than obligations for others. For example, on average DOD obligated about $134 million annually to IDA between fiscal years 2013 through 2018, whereas DOD obligated approximately $39 million annually to RAND Arroyo Center during this timeframe.

DOD Defense STE obligations to S&A Centers were almost entirely awarded to support research projects requested by DOD. In some cases, work was done in response to congressional direction. For example, RAND Project Air Force (PAF) initiated a fiscal year 2017 independent review and assessment of the Ready Aircrew Program to respond to requirements outlined in the National Defense Authorization Act of Fiscal Year 2017. Overall, according to information provided by DOD sponsors and FFRDC representatives, between fiscal years 2013 through 2017, S&A Centers began work on about 600 research projects annually on behalf of DOD, with about 93 percent of these projects initiated at the request of DOD. The dollar value of these S&A Center projects ranged from about $2,000 to $11 million between fiscal years 2013 through 2017.

DOD Reported It Primarily Considered Strategic Relationships and FFRDC Core Competencies When Sponsoring S&A Centers and Initiating Projects

Sponsoring agreements note and primary sponsors reported in comprehensive reviews that S&A Centers are utilized because of DOD’s strategic relationships with FFRDCs. As described in the FAR, FFRDCs meet special, long-term research or development needs of the sponsoring agencies. Sponsoring agreements with S&A Centers outline the importance of strategic relationships that have helped these FFRDCs to develop and maintain in-depth knowledge of their sponsors’ and users’ programs and operations.

In our review of S&A Center sponsoring agreements and comprehensive reviews, we identified that strategic relationships between sponsors and S&A Centers are generally characterized by the stability of long-term capabilities in subject areas important to DOD, access to sensitive and proprietary data and information, and objectivity in the form of freedom from conflicts of interest. These documents also indicate that strategic relationships enable S&A Centers to maintain in-depth knowledge of work sponsor programs and operations. For example, in the 2015 sponsoring agreement between the Army and RAND Arroyo Center, the sponsoring agreement states that both the Army and RAND Arroyo Center share a strategic relationship, and that the RAND Arroyo Center is structured to maintain strong analytic expertise related to Army policy and operations. In addition, the sponsoring agreement outlines the importance of RAND Arroyo Center’s continuity of expertise to the Army, long-term research efforts, and high-quality staff.

Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD(A&S)) officials told us that S&A Centers are oftentimes chosen to perform work for DOD due to unique long-term strategic relationships with sponsors for independent and knowledgeable expertise within core competencies to address sponsors’ specific analytic requirements. In some cases, these strategic relationships date back to World War II. Regarding these strategic relationships, OUSD(A&S) officials also told us the primary sponsor has a degree of control over an FFRDC’s business affairs that can limit the risks of organizational conflicts of interest at FFRDCs.

DOD also cited strategic relationships between DOD and S&A Centers as a reason for using S&A Centers when initiating projects we reviewed. For example:

Prior to initiating a 2016 assessment of the impact of long-term fiscal trends on Army capabilities, the Army determined RAND Arroyo Center was uniquely qualified to conduct the research because the project required knowledge of defense planning scenarios that would have given an industry contractor a competitive advantage, potentially leading to a conflict of interest. The Army also identified RAND Arroyo Center’s long-standing expertise on security cooperation when requesting a fiscal year 2013 study on assessing value in Army security cooperation as a reason RAND Arroyo Center was uniquely suited to complete the study.

Navy primary sponsor officials identified the long-term relationship between CNA, the FFRDC, and the Navy, which has led to broad subject-matter expertise in naval matters, as a reason they used CNA for the fiscal year 2016 study on the assessment of the effects of possible policy changes to a career track program for military officers trained to work with other military services. CNA leadership chose two researchers to lead the effort, one of which had prior experience in this area.

An OUSD(A&S) official cited RAND National Defense Research Institute’s (NDRI) longstanding portfolio on military workforce issues as a reason for using RAND NDRI for a fiscal year 2017 study on the military’s 40-year pay table. An official told us that RAND NDRI’s prior work in this area would allow for a quicker response and more in- depth analysis to respond to the work request.

In addition to the strategic relationships, sponsoring agreements and comprehensive reviews cited FFRDC core competencies as key factors in establishing and continuing relationships with S&A Centers, which is consistent with provisions outlined in DOD Instruction 5000.77. The DOD instruction states that FFRDCs maintain long-term competencies and capabilities to meet DOD needs that cannot be met by government or other private sector resources as effectively, and these competencies derive from the sponsor’s analytical requirements. In general, core competencies include expertise in engineering, research and development, and analysis, and are further described in FFRDC sponsoring agreements and comprehensive reviews. For example:

The Navy 2015 comprehensive review of CNA states that CNA satisfies the Navy’s need for highly specialized skills and competencies in Navy warfighting and warfighting support— particularly research staff from CNA’s studies and analyses division— to accomplish their operational missions.

The 2019 sponsoring agreement between DOD’s OUSD(A&S) and IDA outlined the need for technical and analytical support, citing IDA’s four core competencies as the scope of work of the FFRDC: systems and capabilities evaluations, technology assessments, force and strategy assessments, and resource and support analyses.

The Army 2010 and 2014 comprehensive reviews of RAND Arroyo Center stated that RAND Arroyo Center has currency in all requisite Army proficiencies, provides a multidisciplinary research process that integrates and applies competencies with an assurance of consistently high quality, and also has the ability to apply competencies with expedience when an Army request for analytic support requires a quick response.

OUSD(A&S)’s sponsoring agreement with RAND NDRI defines RAND NDRI’s research capability and core competencies such as, but not limited to, global and national security, defense acquisition, intelligence, and system risk management as means to satisfy essential needs of the FFRDC’s work sponsors for policy research and analysis.

Primary sponsor officials we spoke with also told us that FFRDC staff skills and knowledge related to FFRDC core competencies are important to DOD. For example:

Navy officials said CNA is uniquely suited to perform work for the Navy due to CNA’s core competencies relating to maritime defense analysis and how those competencies align with Navy goals and requirements.

Army officials told us that RAND Arroyo Center staff has extensive background knowledge and analytical skills relating to reserve affairs, manpower policy, and war game analysis, among other things, in providing work for the Army.

Air Force officials told us that RAND PAF has robust knowledge of Air Force processes and maintains top staff and researchers in each core competency.

OUSD(A&S) officials also told us that sponsors and FFRDCs have a relationship in which sponsors rely on FFRDCs for independent and knowledgeable expertise within their core competencies to address sponsors’ analytic requirements.

As shown in figure 4, DOD’s S&A Center primary sponsors identified 3 to 15 core competencies in their sponsoring agreements with each S&A Center.

DOD cited FFRDCs’ core competencies as factors that contributed to using S&A Centers when initiating projects we reviewed, as provided by DOD instruction. For example:

When initiating a fiscal year 2016 CNA assessment on the effects of possible policy changes to a career track program for military officers who are trained to work with other military services, DOD’s Office of the Under Secretary of Defense for Personnel and Readiness cited CNA’s core competencies of analysis of maritime resources; maritime program planning; and maritime policies, strategies, and doctrines as justification for using CNA to perform the work, among other things.

In initiating a fiscal year 2014 IDA analysis on satellite ground control, DOD’s Office of the Deputy Assistant Secretary of Defense for Space and Intelligence cited IDA’s core competencies related to technology, such as systems and capabilities evaluations, as justification for using IDA for the research.

DOD Uses S&A Center Research in a Variety of Ways and Takes Some Steps to Assess the Value of Research and Centers

DOD Uses S&A Center Research to Inform Decisions, Shape Guidance, and Identify Potential Efficiencies

DOD reports that it uses studies and analyses to inform decision-making; shape guidance, policies, and training; and identify opportunities to save time and money.

Inform decision-making. For example, a 2016 study conducted by the RAND Arroyo Center on linking Army cost and performance found that the Army needed an updated tool to inform more strategic allocation of its resources. Among other things, the study contributed to updated strategies to measure the Army’s performance regarding force structure and readiness as well as the cost implications for these activities. According to an Army official, the study contributed to the development of updated Army metrics for cost and other performance indicators. In another example, a 2013 research project conducted by RAND NDRI on effectiveness measures of a DOD program to reduce the threat from infectious diseases and biological weapons developed and recommended two sets of metrics to improve program evaluation efforts. According to OUSD(A&S) officials, DOD used the recommended metrics to develop program performance measures.

Shape guidance, policies, and training. For example, a 2013 study conducted by RAND NDRI on the root causes related to DOD weapons programs cost overruns found, among other things, that DOD needed to re-examine its assumptions when estimating a program’s cost, schedule, and technical performance. OUSD(A&S) officials told us the study contributed to DOD’s decision to update its policy, processes, management practices, and training curriculum so as to improve estimates. In another example, a 2013 study conducted by RAND Arroyo Center on the value of security missions conducted by the Army’s geographically aligned forces found that the use of these forces improved the efficiency of security planning and preparation and recommended a range of process and planning improvements for the Army. According to an Army official, the Army used several of the recommendations to update guidance for preparation and planning for future missions involving regionally aligned forces.

Identify opportunities to improve efficiency. For example, a 2013 study conducted by RAND Arroyo Center on marketing and resources needed for Army recruiting efforts identified strategies aimed at optimizing the Army’s annual spending, estimated at nearly $1 billion for recruiters, enlistment bonuses, and television advertising. An Army official said that the Army has used the recruiting tool developed by RAND Arroyo Center for this study to make decisions and the Army estimates the tool can reduce costs by potentially hundreds of millions of dollars annually. In another example, DOD reported in its 2015 comprehensive review of RAND PAF that a 2010 study conducted by RAND PAF on aircraft maintenance at centralized repair facilities found that these facilities should be consolidated. According to the Air Force primary sponsor, this study helped the Air Force make decisions that led to saving up to $300 million annually as well as saving time on aircraft inspections.

DOD Has Taken Steps to Assess the Value of S&A Center Research and the Centers

In terms of assessing the outcomes of research, we found that DOD primary sponsors took steps to assess the value of S&A Center research and the centers. The DOD instruction requires that primary sponsors assess the efficiency and effectiveness of the FFRDC in meeting DOD needs in comprehensive reviews, including a review and summary of FFRDC accomplishments and their effectiveness utilizing factors such as quality and timeliness of the work produced and value of projects assessed. Additionally, the DOD instruction provides that the factors of technical quality, responsiveness, value, and timeliness be addressed in annual performance reviews. DOD’s FFRDC Management Plan—which preceded the DOD instruction and was in effect until the DOD instruction became effective in January 2018—also required primary sponsors to annually assess the value of FFRDC performance, among other factors, and include summaries of these annual assessments in comprehensive reviews. Primary sponsors generally assess the value of S&A Center research through annual performance reviews (through performance evaluation questionnaires to solicit feedback from work sponsors) and comprehensive reviews.

To monitor the execution of research projects, primary sponsors regularly solicit work sponsor input regarding S&A Centers’ performance, including the value, technical quality, responsiveness, and timeliness of the work performed. Time frames for soliciting this input vary by primary sponsor but most do this annually. These questionnaires include one or more sections for work sponsors to add comments about S&A Center work and allow work sponsors to rate S&A Center performance. Some of these questionnaires use a numerical scale. For example, the Air Force questionnaire sent to RAND PAF work sponsors asks respondents to rate project value using a scale from 1 through 10, with 1 indicating “very poor” and 10 “very good.” The OUSD(A&S) questionnaire sent to IDA work sponsors asks respondents to rate the value of IDA’s work and results using a scale from 1 through 5, where 1 symbolizes either “strongly agree” or “outstanding performance” and 5 symbolizes “strongly disagree” or “poor performance.”

FFRDC primary sponsors conduct comprehensive reviews at least every 5 years to, among other things, identify the accomplishments made by each FFRDC. In August 2014, we reported that DOD officials described the comprehensive review process as an opportunity to take a broad assessment of the FFRDC and its key competencies beyond the annual assessments of FFRDCs. Included in these comprehensive reviews is a summary of FFRDC accomplishments and effectiveness in meeting work sponsors’ needs since the last comprehensive review.

In our examination of the most recent comprehensive reviews for each of the five S&A Centers, we found that the comprehensive reviews summarize the results from the performance evaluation questionnaires and assessed the value of the research in varying ways. For example, the Army questionnaire to RAND Arroyo Center work sponsors assessed value in terms of whether a project was worth the investment monetarily. OUSD(A&S) questionnaires sent to work sponsors assessed the value of IDA work in relation to whether the results were useful, consistent with the level of effort, and if IDA brought competence, expertise, and helpful perspectives to the issues. The Army reported in the 2014 comprehensive review of RAND Arroyo Center that between fiscal years 2010 through 2013, work sponsors provided overwhelmingly positive results that RAND Arroyo Center performance was “worth the level of effort.” DOD’s Office of the Under Secretary of Defense for Acquisitions, Technology, and Logistics—RAND NDRI’s primary sponsor prior to the DOD reorganization in 2018—reported in the 2014 comprehensive review of RAND NDRI that in fiscal year 2013, work sponsors provided overwhelmingly positive results that RAND NDRI performance provided long-term value.

We also found that comprehensive reviews included anecdotal examples of how DOD used S&A Center research. For example, the Army 2014 comprehensive review of RAND Arroyo Center highlighted 53 of 114 research projects completed between fiscal years 2010 through 2013 to demonstrate how RAND Arroyo Center work met Army research requirements. Likewise, the Air Force primary sponsor’s 2015 comprehensive review of RAND PAF highlighted 28 of 207 research projects completed between fiscal years 2010 and 2014 to demonstrate how the Air Force leveraged RAND PAF work to improve efficiency in the department. An Air Force official told us that RAND PAF, and not the Air Force, collected 28 project examples for the purposes of the comprehensive review.

DOD Does Not Track Whether S&A Center Recommendations Have Been Implemented, but Recently Took Steps Intended to Improve Insights

Another potential way to assess the outcomes of research is to track to what extent a research project’s recommendations were implemented, and how. Neither DOD nor primary sponsors currently track the implementation of S&A Center research project recommendations. While primary sponsors are not tracking recommendations, in 2015 one of the S&A Centers—RAND PAF—began tracking recommendations made to the Air Force. According to a RAND PAF representative, the tracking system captures the issue, approach, conclusions, opportunities, and outcomes for each completed project. A RAND PAF representative told us that tracking recommendations is useful for demonstrating the value that RAND PAF provides the Air Force. In April 2019, a Navy official told us that the Navy is working on a database to track CNA reports, including recommendations, report topic, work sponsor, and project funding, among other things, to prevent duplication of requests. The Navy official said this effort is expected to be completed in 2019. Both OUSD(A&S) and Army officials told us that while they do not currently track recommendations, they are considering doing so as part of their oversight efforts. Further, Army officials told us that it is important for the sponsor that implements the recommendations to track how and whether that information was used.

While tracking recommendations is useful according to some primary sponsors, some DOD officials cautioned that tracking recommendations would not provide insights into the overall value across all S&A Center research. DOD officials told us that recommendations are only one potential outcome of S&A Center research and that the value of a study may not be specifically linked to a recommendation. For example, Navy officials said that CNA’s projects may present the Navy with options and associated courses of action rather than formal recommendations, and DOD officials also told us that S&A Center work can provide value to DOD that is not always represented by recommendations, such as presentations or research aimed at contributing to the understanding of a particular issue, but without specific recommendations.

In February 2019, DOD’s Office of the Under Secretary of Defense for Personnel and Readiness issued a memorandum related to the oversight of the Personnel and Readiness Studies and Analysis program. The memorandum tasked the program director with developing a studies and analysis program framework that improves accountability for project results and the implementation of study recommendations. Personnel and Readiness also issued a template “action memo” providing for an executive summary of completed projects as well as implementation plans delineating recommendations made, implementation approach, and plan of action for each recommendation. According to a senior Personnel and Readiness official, work sponsors with reports that were completed or published since September 2018 are subject to these actions. This official noted that the purpose is to increase accountability of the Personnel and Readiness staff regarding the use of FFRDCs and to develop an overall picture of the value proposition of FFRDC research. It is too soon to tell to what extent these memorandums will affect DOD’s insights on its implementation of S&A Center recommendations.

DOD and the S&A Centers We Reviewed Have Conflict of Interest Policies and Practices

Regulation Requires FFRDCs to Operate Free from Conflicts of Interest

What are Conflicts of Interest? A Personal Conflict of Interest exists when an individual employed by an organization is in a position that could materially influence research findings or recommendations and may lack objectivity due to their financial interests, personal activity, or relationships. An Organizational Conflict of Interest exists when, because of other interests or relationships, an entity is unable or potentially unable to render impartial assistance or advice to the government or the entity might have an unfair competitive advantage.

The Federal Acquisition Regulation (FAR) requires an FFRDC to conduct its business in a manner befitting its special relationship with the government and to be free from conflicts of interest. To perform its responsibilities to the sponsoring agency, an FFRDC and its employees have access beyond that which is common in a normal contractual relationship, including access to sensitive and proprietary data and information, equipment, and property. To accomplish this, the FAR and DOD instruction state that an FFRDC must be free from conflicts of interest and fully disclose financial and outside interests to the sponsoring agency. Conflicts of interest can be personal or organizational. Personal conflicts of interest can be, but are not limited to, financial interests of the employee or close family members, other employment, gifts, consulting relationships, other forms of research funding or support, investment in the form of stock or bonds ownership, real estate, or business ownership. Additionally, the DOD instruction outlines steps FFRDC parent organizations should take to prevent and mitigate conflicts of interest. These steps include, but are not limited to, having procedures in place to screen employees for potential conflicts of interest; requiring disclosure of financial and other interests that might affect the employee’s objectivity; establishing policies and procedures to protect proprietary, privileged, and sensitive information from disclosure; and reporting any conflicts of interest to the applicable contracting officer or contracting officer’s representative and the primary sponsor as soon as it is identified. See figure 5 for DOD’s conflict of interest elements outlined in the DOD instruction that primary sponsors are to require from FFRDC parent organizations.

All Five S&A Centers Have Conflict of Interest Policies and Practices

Each of the five S&A Centers we reviewed has corporate-wide conflict of interest policies and practices which incorporate various key elements of the DOD instruction. For example, all S&A Center policies we reviewed have measures that require personnel to protect proprietary, privileged, and sensitive information. S&A Center representatives told us they undertake various approaches in practice that meet key elements in the DOD instruction in order to ensure they operate in the public interest with objectivity and independence. For example:

Reviewing all personnel annually or on a task-by-task basis for conflicts of interest. Generally, representatives we spoke with from the five S&A Centers address conflicts of interest annually or task-by- task, which is an option outlined in the DOD instruction. For instance, RAND representatives said they perform task-by-task, instead of annual, conflict of interest reviews because staff do not know which projects they will be working on during the year. In addition, IDA and RAND representatives told us they screen employees upon hire as well as when an employee initiates a new project, and both IDA and RAND have automated their screening processes. IDA representatives explained that their automated tool screens personnel at the initiation of each project, by including, for example, a process to determine if staff assigned to a project have any affiliations with industry or companies and competitors in the particular field of study. If staff or members of their households do have affiliations, IDA may issue a waiver if the financial interest (such as but not limited to stocks, stock options, and bonds) in a single company is below $15,000, the threshold for disclosure outlined in the DOD instruction. IDA representatives also told us that IDA staff are required to self- report any changes to previous financial interest disclosures during the year. In another example, RAND representatives said their automated conflict of interest tool screens for conflicts of interest by comparing areas of work RAND performs to similar areas in the private sector. Additionally, the system will identify any staff that have not submitted a conflict of interest statement within a year.

Providing initial and annual conflict of interest training for all personnel. S&A Center representatives told us that they perform training related to or specifically covering conflicts of interest in varying ways. IDA’s corporate-wide conflict of interest policy includes initial and annual conflict of interest training elements, as outlined in the DOD instruction. For example, IDA’s policy states that all employees are to participate in conflict of interest training upon initial hire, and in annual refresher training thereafter. The other four S&A Centers did not explicitly include annual conflict of interest training in corporate-wide policies, but representatives told us they provide annual ethics training, which includes training on conflicts of interest, to their employees. For example, CNA representatives told us they provide ethics and conflicts of interest training to staff, which is required by their contract with the Navy. CNA representatives told us that if CNA staff do not complete the required training, the staff will be blocked from accessing CNA’s time card system and will not receive pay until the training is complete. In another example, RAND representatives told us they have annual training that covers ethics, conflicts of interest, and culture and discrimination issues for newly hired staff.

Representatives from each of the S&A Centers told us they attempt to mitigate potential conflicts of interest as soon as the potential conflicts become known and before they become a reportable conflict. For example, CNA representatives told us that in one instance, a CNA employee’s spouse worked for the Navy and CNA mitigated this potential conflict by transferring the employee to another project where the relationship did not pose a potential conflict. In another example, when a RAND employee inherited stock in the middle of a project, a potential conflict of interest was mitigated by the employee selling the inherited stock. In another instance, a RAND employee was initially staffed to a project related to an area of work a spouse worked on commercially, and RAND mitigated the potential conflict by recusing the employee from the project.

Agency Comments

We provided a draft of this report to DOD for review and comment. In its comments, DOD concurred with our findings. DOD also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or LudwigsonJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II.

Appendix I: Selected Study and Analysis Center Research Projects

Table 2 provides detailed information on the 22 projects we selected for review.

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact:

Staff Acknowledgments

In addition to the contact named above, Janet McKelvey (Assistant Director), Andrew Burton (Analyst-in-Charge), Mallory Bryan, Lisa Fisher, and Jordan Kudrna made key contributions to this report. Additional assistance was provided by Marie Ahearn, Pete Anderson, Jenny Chanley, Joseph Cook, Julia Kennon, Tind Shepper Ryen, and Roxanna Sun.