prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.



Report 2: The Secret Service plays a critical role in protecting the President, Vice President, their immediate families, and national leaders, among others  In addition, the component is responsible for safeguarding the nation’s currency and financial payment systems  To accomplish its mission, Secret Service officials reported that, as of June 2018, the component had approximately 7,100 employees (including the Uniformed Division, special agents, and administrative, professional, and technical staff)  These employees were assigned to the component’s headquarters in Washington, D C , and 133 field offices located throughout the world (including 115 domestic offices and 18 international offices)  The Secret Service’s employees are heavily dependent on the component’s IT infrastructure and communications systems to perform their daily duties  According to data reported on the Office of Management and Budget’s IT Dashboard, the component planned to spend approximately $104 8 million in fiscal year 2018 to modernize and maintain its IT environment  To manage this IT environment, the Secret Service hired a full-time CIO in November 2015  In addition, in an effort to improve its management structure, the component consolidated all IT staff and assets under this new CIO in March 2017  OCIO officials stated that these staff include the government employees who provide direct and indirect support of the day-to-day operations of the Secret Service’s enterprise systems and services  According to Secret Service officials, the component’s IT workforce included 190 staff, as of July 2018  These officials stated that 166 of these employees were located in the component’s headquarters in Washington, D C , and 24 were located in domestic field offices  The officials also reported that these July 2018 staffing levels were below their current approved staffing level of 220 staff (which included 44 positions in domestic field offices)  Secret Service IT staff also deploy to other locations, as necessary, to provide support for certain security activities  For example, the Secret Service reported that, in 2017, OCIO deployed over 79 staff to New York, N Y , to provide communications support during the United Nations General Assembly  As a component of DHS, the Secret Service must follow the department’s policies and processes for managing acquisitions, including IT acquisitions  DHS categorizes its acquisition programs according to three levels that are determined by the life cycle costs of the programs  These levels then determine the extent of required program and project management and the acquisition decision authority (the individual responsible for management and oversight of the acquisition)  The department also categorizes its acquisition programs as major or non- major based on expected cost  Table 1 describes the levels of DHS’s acquisition programs and their associated acquisition decision authorities  DHS’s policies and processes for managing major acquisition programs are primarily set forth in its Acquisition Management Directive 102-01 and Acquisition Management Instruction 102-01-001  In particular, these policies establish that a major acquisition program’s decision authority is to review the program at a series of predetermined acquisition decision events to assess whether the program is ready to proceed through the acquisition life cycle phases  Figure 1 depicts the acquisition life cycle established in DHS acquisition management policy  DHS’s Acquisition Management Directive and Instruction do not establish an acquisition life cycle framework for the department’s non-major acquisition programs  Instead, according to the Instruction, Component Acquisition Executives (i e , the senior acquisition official within a component that is responsible for implementation, management, and oversight of the component’s acquisition process) are required to establish component-specific non-major acquisition policies and guidance that support the “spirit and intent” of the department’s acquisition policies  To that end, the Secret Service developed a policy that establishes an acquisition life cycle framework for its non-major acquisition programs  This acquisition framework for the component’s non-major acquisition programs is consistent with the acquisition framework that DHS established for its major acquisition programs  In particular, the Secret Service’s framework includes the same phases and decision events as DHS’s framework (e g , acquisition decision event 2A, the point at which the acquisition decision authority determines whether a program may proceed into the obtain phase)  In addition, DHS’s Systems Engineering Life Cycle Instruction and Guidebook outline a framework of major systems engineering activities and technical reviews that are to be conducted by all DHS programs and projects, both major and non-major  This framework is intended to ensure that appropriate systems engineering activities are planned and implemented, and that a program’s development effort is meeting the business need  In particular, the systems engineering life cycle framework consists of nine major activities (e g , requirements definition, integration, and testing) and a set of related technical reviews (e g , preliminary design review) and artifacts (e g , requirements documents)  DHS policy allows programs to tailor these activities, technical reviews, and artifacts based on the unique characteristics of the program (e g , scope, complexity, and risk)  For example, a program may combine systems engineering technical reviews and artifacts, or add additional reviews  This tailored approach must be documented in a program’s systems engineering life cycle tailoring plan  The systems engineering technical reviews are intended to provide DHS the opportunity to determine how well a program has completed the necessary systems engineering activities  Each technical review includes a minimum set of exit criteria that must be satisfied before a program may move on to the next systems engineering activity  At the end of the technical review, the program manager must develop a technical review completion letter that documents the outcome of the review, including stakeholder concurrence that the exit criteria were satisfied  Moreover, DHS’s agile instruction, which was first issued in April 2016 and updated in April 2018, identifies agile as the preferred development approach for the department’s IT programs and projects  Agile is a type of incremental (i e , modular) development, which calls for the rapid delivery of software in small, short increments rather than in the typically long, sequential phases of a traditional waterfall approach  DHS’s agile instruction also states that component CIOs are to set modular (i e , incremental) outcomes and target measures to monitor progress in achieving agile implementation for IT programs and projects  To that end, the department identified core metrics that its agile IT programs are to use to monitor progress, including the number of story points completed per release and the number of releases per quarter  Further, DHS policy and guidance have established an acquisition (i e , contract) review process that is intended to enable the DHS CIO to review and effectively guide the department’s IT expenditures  According to the department’s IT acquisition review guidance, DHS components with a CIO (which includes the Secret Service) are to submit to DHS OCIO for review, IT acquisitions that (1) have total estimated procurement values of $2 5 million or more; and (2) are funded by a level 1, 2, or 3 program with a life cycle cost estimate of at least $50 million (i e , a major investment, as defined by DHS’s capital planning and investment control guidance)  DHS policies and guidance also establish numerous responsibilities for the department’s component-level CIOs that are aimed at ensuring proper oversight and management of the components’ IT investments  Among other things, these component-level CIO responsibilities relate to topics such as IT budgeting, portfolio management, and oversight of programs’ systems engineering life cycles  Table 2 identifies 14 selected IT oversight responsibilities for DHS’s component CIOs  The Secret Service acquires IT infrastructure and services that are intended to improve its ability to execute its investigation and protection missions  According to data reported on the Office of Management and Budget’s IT Dashboard, the Secret Service planned to spend about $104 8 million on IT in fiscal year 2018, which included approximately $34 6 million for the development and modernization of its IT infrastructure and services, and about $70 2 million for the operations and maintenance of this infrastructure (including 21 existing IT systems)  Also according to data reported on the IT Dashboard, as of April 2018, the Secret Service had one major IT investment (called the Information Integration and Technology Transformation and discussed in more detail later in this report), seven non-major IT investments, and one non- standard infrastructure investment  Figure 2 depicts the Secret Service’s planned IT spending for fiscal year 2018  The Secret Service has faced long-standing challenges in managing its IT infrastructure  For example, A National Security Agency audit of the Secret Service’s IT environment in 2008 identified network and system vulnerabilities that needed immediate remediation to protect the component’s systems and electronic information  The Secret Service determined in 2010 that it had IT capability gaps associated with three key areas: network security, information sharing and situational awareness, and operational communications  The component reported that it required a significant IT modernization effort with sustained investment of resources to replace dated and restrictive network and communications capabilities  The Secret Service also reported in 2010 that it had 42 mission- support applications that were operating on a 1980’s mainframe that lacked multi-level security (i e , the ability to view classified information from two security levels, such as secret and top secret, at the same time), was beyond its equipment life cycle, and was at risk of failing  Further, in 2011, DHS’s Office of Inspector General reported that the Secret Service’s existing infrastructure did not meet current operational requirements  According to the Secret Service, this dated infrastructure was unable to support newer technologies (e g , Internet protocol), share common DHS enterprise services, or migrate to the department’s consolidated data centers  To address challenges with its IT environment, in 2009, the Secret Service initiated the IITT investment, which is intended to modernize and enhance the component’s infrastructure, communications systems, applications, and processes  In particular, IITT is a portfolio of programs and projects that are meant to, among other things, improve systems availability in support of the Secret Service’s business operations, increase interoperability with other government systems and networks, enhance the component’s system and network security, and enable scalability to support growth  From 2010 to July 2018, according to OCIO officials, the Secret Service spent approximately $392 million on IITT  In fiscal year 2018, the component had planned to spend approximately $42 7 million on IITT (i e , about 40 percent of its total planned IT spending for the fiscal year), according to data reported on the Office of Management and Budget’s IT Dashboard  In total, the planned life cycle cost estimate for IITT is at least $811 million  As of June 2018, IITT was a major investment comprised of two programs (one of which included three projects) and one standalone project (i e , it was not part of another program) that had capabilities that were in planning or development and modernization  These programs and project were the Enabling Capabilities program, Enterprise Resource Management System program (which included three projects that were each being implemented using an agile methodology: Uniformed Division Resource Management System, Events Management, and Enterprise-wide Scheduling), and the Multi-Level Security project  Table 3 describes the IITT programs and projects that had capabilities that were in planning or development and modernization, as of June 2018  The table also includes the associated level, acquisition decision authority, estimated life cycle costs, and planned or actual dates of operational capability for each of the programs and projects  (Appendix II also provides additional information on these programs and projects ) The Enabling Capabilities program within IITT is designated as a major acquisition program  As such, its acquisition decision authority is the DHS Under Secretary for Management, and both DHS and the Secret Service provide oversight to this program  IITT’s other program and project—the Enterprise Resource Management System program (which includes three projects, as discussed earlier) and Multi-Level Security project—are designated non-major acquisition programs  In June 2011, DHS’s Under Secretary for Management delegated acquisition decision authority for this non-major program and project to the Secret Service Component Acquisition Executive  As such, oversight of the Enterprise Resource Management System program (including its three projects) and the Multi- Level Security project is conducted primarily at the component level  The Secret Service also implemented other capabilities that are now in operations and maintenance (i e , the capabilities have been fielded and are operational) as part of the IITT investment, such as a capability to move data between systems in separate classification levels (e g , top secret and secret) and communications interoperability  Table 4 describes IITT capabilities that are in operations and maintenance  DHS, including the Secret Service, has faced long-standing challenges in effectively managing its workforce  In January 2003, we designated the implementation and transformation of DHS as high risk, including its management of human capital, because it had to transform 22 agencies—several with major management challenges—into one department  This represented an enormous and complex undertaking that would require time to achieve in an effective and efficient manner  Since that time, the department has made important progress in strengthening and integrating its management functions  Nevertheless, we have continued to report that significant work remains for DHS to improve these management functions  Among other things, we previously reported that the department had lower average employee morale than the average for the rest of the federal government  We also reported that, in 2011, based on employee responses to the Office of Personnel Management’s Federal Employee Viewpoint Survey—a tool that measures employees’ perceptions of whether and to what extent conditions characterizing successful organizations are present in their agency—DHS was ranked 31st out of 33 large agencies on the Partnership for Public Service’s Best Places to Work in the Federal Government rankings  The most recent results of these surveys in 2017 showed that DHS continues to maintain its low rankings  DHS’s Office of Inspector General has reported on challenges that the Secret Service has faced in managing its IT workforce  Specifically, in October 2016, the Inspector General reported that the Secret Service CIO did not have oversight of, or authority over, all IT resources, including the workforce; in particular, almost all of the component’s IT employees were located in a division outside of OCIO; and the Secret Service had vacancies in key positions responsible for managing IT, including not having a full-time CIO from December 2014 through November 2015  As previously discussed, the Secret Service has taken actions to address these two issues with the management of its IT workforce  These actions included hiring its full-time CIO in November 2015 and consolidating the workforce and all IT assets under this CIO in March 2017  Of the 14 selected responsibilities established for component-level CIOs in DHS’s IT management policies, the Secret Service CIO had fully implemented 11 responsibilities and had partially implemented 3 responsibilities  Table 5 summarizes the extent to which the Secret Service CIO had implemented each of the 14 responsibilities  The Secret Service CIO fully implemented 11 of the 14 selected component-level CIO responsibilities  Examples of the responsibilities that the CIO fully implemented are as follows: Develop, implement, and maintain a detailed IT strategic plan  Consistent with DHS’s IT Integration and Management directive, in January 2017, the Secret Service CIO developed an IT strategic plan that outlined the CIO’s strategic IT goals and objectives, as well as tasks intended to meet the goals and objectives  The CIO maintained this strategic plan, to include updating it in January 2018  The CIO also took steps to implement the tasks identified within the strategic plan, such as working to develop an IT training program  In particular, as part of this effort to develop an IT training program, OCIO identified recommended training for the office’s various IT workforce groups (discussed in more detail later in this report)  Concur with each program’s and/or project’s systems engineering life cycle tailoring plan  In accordance with DHS’s Systems Engineering Life Cycle instruction, the Secret Service CIO concurred with the systems engineering life cycle tailoring plan for one program and three projects included in the Secret Service’s IITT investment  Specifically, the CIO documented his approval via his signature on the tailoring plans for IITT’s Enabling Capabilities program, and Multi-Level Security, Uniformed Division Resource Management System, and Events Management projects  Participate on DHS’s CIO Council, Enterprise Architecture Board, or other councils/boards as appropriate, and appoint employees to serve when necessary  As required by DHS’s IT Integration and Management directive, the Secret Service CIO participated on two required DHS-level councils/boards, and appointed a delegate to serve in his place, when necessary  Specifically, the Secret Service CIO or the CIO’s delegate—the Deputy CIO—attended bi-monthly meetings of the DHS CIO Council  In addition, another Secret Service CIO appointee—the component’s Chief Architect—attended an ad hoc meeting of the Enterprise Architecture Board in June 2017  In addition, the Secret Service CIO had partially implemented three component-level CIO responsibilities, as follows  Manage the component IT investment portfolio, including establishing a component-level IT acquisition review process that enables component and DHS review of component acquisitions (i e , contracts) that contain IT  As directed in DHS’s Capital Planning and Investment Control directive and guidebook, the Secret Service CIO took steps to manage the component’s IT investment portfolio, including reviewing certain contracts containing IT  For example, among our random sample of 33 IT contracts that the Secret Service awarded between October 1, 2016, and June 30, 2017, we found that the CIO or the CIO’s delegate had reviewed 31 of these contracts  However, the CIO had not established and documented a defined process for reviewing contracts containing IT, which may have contributed to why the CIO or the CIO’s delegate did not review 2 of the 33 contracts in our sample  OCIO officials were unable to explain why neither of these officials reviewed the 2 contracts, which had a combined planned total procurement value of approximately $1 75 million  In particular, one of the contracts, with a planned total procurement value of about $1,122,934, was to provide credentialing services for the 2017 Presidential Inauguration  The other contract, with a planned total procurement value of about $629,337, was to provide maintenance support for a logistics system  The OCIO officials acknowledged that both contracts should have been approved by one of these officials  Without establishing and documenting an IT acquisition review process that ensures that the CIO or the CIO’s delegate reviews all contracts containing IT, as appropriate, the CIO’s ability to analyze the contracts to ensure that they are a cost-effective use of resources and are aligned with the component’s missions and goals is limited  Ensure all component IT policies are in compliance and alignment with DHS IT directives and instructions  As required by DHS’s IT Integration and Management directive, the Secret Service CIO had ensured that certain component IT policies were in compliance and alignment with DHS IT directives and instructions  For example, in alignment with the department’s IT Integration and Management directive, the Secret Service’s Investment Governance for IT policy specifies that the component CIO (in conjunction with each Secret Service Office) is responsible for developing the component IT spend plan, as well as developing and maintaining an IT strategic plan  However, the Secret Service’s enterprise governance policy was not in compliance with DHS’s IT Integration and Management directive  Specifically, while the department’s policy states that the Secret Service CIO is responsible for developing and reviewing the component’s IT budget formulation and execution, the Secret Service’s enterprise governance policy does not specify this as the CIO’s responsibility  According to OCIO officials, the Secret Service CIO participates in the development and review of the IT budget formulation and execution as a member of the Executive Resources Board (the Secret Service’s highest-level governing body, which has the final decision authority and responsibility for enterprise governance), and the Secret Service Deputy CIO is a voting member of the Enterprise Governance Council (the Secret Service’s second-level governance body and advisory council to the Executive Resources Board)  However, the Secret Service’s enterprise governance policy has not been updated to reflect these roles  The Secret Service did not update its enterprise governance policy to properly reflect the CIO’s and Deputy CIO’s roles on the Executive Resources Board or Enterprise Governance Council because OCIO officials were not aware that these roles were not properly documented in the component’s policy until we identified this issue during our review  Further compounding the issue of the Secret Service’s enterprise governance policy not properly reflecting the CIO’s and Deputy CIO’s roles and responsibilities on the component’s governance boards is that the Secret Service has not developed a charter for its Executive Resources Board  We have previously reported that a best practice for effective investment management is to define and document the board’s membership, roles, and responsibilities  One such way to do so is via a charter  According to Secret Service officials, the component does not have a charter for the board because, while the Secret Service has established the board pursuant to law, there is little statutory guidance on how the board must be formalized, including whether a charter is required  The officials acknowledged that development of a board charter is a best practice  They stated that, in response to our review, the component has begun efforts to develop a charter for the Executive Resources Board, but they did not know when it would be completed  Until the Secret Service updates its enterprise governance policy to specify (1) the CIO’s current role and responsibilities on the Executive Resources Board, to include developing and reviewing the IT budget formulation and execution, and (2) the Deputy CIO’s role and responsibilities on the Enterprise Governance Council, the CIO’s ability to develop and review the component’s IT budget may be limited  Further, until the Secret Service develops a charter for its Executive Resources Board that specifies the roles and responsibilities of all board members, including the CIO, the Secret Service will not be effectively positioned to ensure that all members understand their roles and responsibilities on the board and will perform them as expected  Set modular outcomes and target measures to monitor the progress in achieving agile implementation for IT programs and/or projects within their component  Consistent with DHS policy, the Secret Service CIO has set modular outcomes and target measures to monitor the progress of two IITT projects that the component is implementing using an agile methodology—Uniformed Division Resource Management System and Events Management  For example, the modular outcomes set for these projects included measuring planned and actual burndown (i e , the number of user stories completed)  In addition, the projects were to measure their velocity (i e , the rate of work completed) for each sprint (i e , a set period of time during which the development team is expected to complete tasks related to developing a piece of working software)  However, the modular outcomes and target measures did not include product quality or post-deployment user satisfaction, although such measures are leading practices for managing agile projects  According to Secret Service OCIO officials, the component does not mandate the specific metrics that its agile projects are to use; instead, each project is to determine the metrics based on stakeholder requirements and unique project characteristics  The officials further stated that these metrics are to be documented in an acquisition program baseline and program management plan; this baseline and program management plan are then to be approved by the CIO  To its credit, the component’s one agile project that, as of May 2018, had deployed its system to users—the Uniformed Division Resource Management System—did measure product quality  OCIO officials also stated that they regularly receive verbal, undocumented feedback from users on the system and they plan to conduct a documented user satisfaction survey on this system by September 2018  Nevertheless, without ensuring that product quality and post- deployment user satisfaction metrics are included in the modular outcomes and target measures that the CIO sets for monitoring agile projects, the Secret Service lacks assurance that the Events Management project or other future agile projects will measure product quality or post-deployment user satisfaction  Without guidance specifying that agile projects track these metrics, the projects may not do so and the CIO may be limited in his knowledge of the progress being made on these projects  Workforce planning and management is essential for ensuring that federal agencies have the talent, skill, and experience mix they need to execute their missions and program goals  To help agencies effectively conduct workforce planning and management, the Office of Personnel Management, the Chief Human Capital Officers Council, DHS, the Secret Service, and we have identified numerous leading practices related to five workforce areas: strategic planning, recruitment and hiring, training and development, employee morale, and performance management  Table 6 identifies the five workforce areas and 15 selected leading practices associated with these areas (3 practices within each area)  Of the five selected workforce planning and management areas, the Secret Service had substantially implemented two of the areas and minimally implemented three of the areas for its IT workforce  In addition, of the 15 selected leading practices associated with these workforce planning and management areas, the Secret Service had fully implemented 3 practices, partly implemented 8 practices, and did not implement any aspects of 4 practices  Table 7 summarizes the extent to which the Secret Service had implemented for its IT workforce the five selected workforce planning and management areas and 15 selected leading practices associated with those areas, as of June 2018  Strategic workforce planning is an essential activity that an agency needs to conduct to ensure that its human capital program aligns with its current and emerging mission and programmatic goals, and that the agency is able to meet its future needs  We previously identified numerous leading practices related to IT strategic workforce planning, including that an organization should (1) establish and maintain a strategic workforce planning process, including developing all competency and staffing needs; (2) regularly assess competency and staffing needs, and analyze the IT workforce to identify gaps in those areas; and (3) develop strategies and plans to address gaps in competencies and staffing  The Secret Service minimally implemented the three selected leading practices associated with the IT strategic workforce planning area  Specifically, the component partly implemented two of the practices and did not implement one practice  Table 8 lists these selected leading practices and provides our assessment of the Secret Service’s implementation of the practices  Establish and maintain a strategic workforce planning process, including developing all competency and staffing needs—partly implemented  The Secret Service took steps to establish a strategic workforce planning process for its IT workforce  For example, the Secret Service CIO developed and maintained a plan that identified strategic workforce planning tasks, to include analyzing the staffing requirements of the IT workforce  In addition, the Secret Service defined general core competencies (e g , communication and customer service) for its workforce, including IT staff  However, OCIO did not identify all required knowledge and skills needed to support this office’s functions  In particular, while OCIO identified certain technical competencies that its IT workforce needs, such as cybersecurity, the office did not identify and document all of the technical competencies that it needs  OCIO officials stated that they did not identify and document the technical competencies that the office needs because the Secret Service was focused on reorganizing the IT workforce under a single, centralized reporting chain within the CIO’s office  Consequently, the officials stated that they had not completed the work to identify all required IT knowledge and skills necessary to support the office  Yet, the Secret Service completed the IT workforce reorganization effort over a year ago, in March 2017 and, since then, OCIO has not identified all of the required IT knowledge and skills that the office needs  OCIO officials told us that they plan to identify all of the technical competency needs for the IT workforce, but they were unable to specify a time frame for when these needs would be fully identified  Until OCIO identifies all of the required knowledge and skills for the IT workforce, the office will be limited in its ability to identify and address any competency gaps associated with this workforce  In addition, the Secret Service did not reliably determine the number of IT staff that it needs in order to support OCIO’s functions  Specifically, in January 2017, an independent review of the staffing model that the component used to identify its IT workforce staffing needs found that the model was not based on any verifiable underlying data  In late August 2018, Office of Human Resources officials reported that they had hired a contractor in early August 2018 to update the staffing model to improve the quality of the data  These officials expected the contractor to finish updating the model by August 2019  The officials plan to use the updated model to identify the Secret Service’s IT workforce staffing needs for fiscal year 2021  Updating the staffing model to incorporate verifiable workload data should increase the likelihood that the Secret Service is able to appropriately identify its staffing needs for its IT workforce  Regularly assess competency and staffing needs, and analyze the IT workforce to identify gaps in those areas—not implemented  The Secret Service regularly assessed the competency and staffing needs for 1 of the occupational series within its IT workforce (i e , the 2210 IT Specialist series)  However, it did not regularly assess the competency and staffing needs for the remaining 11 occupational series that are associated with the component’s IT workforce, nor identify any gaps that it had in those areas  OCIO officials stated that they had not assessed these needs or identified competency or staffing gaps because, among other things, the Secret Service was focused on reorganizing the IT workforce under a single, centralized reporting chain within the CIO’s office  However, as previously mentioned, the component completed this effort in March 2017, but OCIO did not subsequently assess its competency and staffing needs, nor identify gaps in those areas  OCIO officials reported that they plan to assess the competencies of the IT workforce to identify any gaps that may exist; however, they were unable to identify a specific date by which they expect to have the capacity to complete this assessment  Until OCIO regularly analyzes the IT workforce to identify its competency needs and any gaps it may have, OCIO will be limited in its ability to determine whether its IT workforce has the necessary knowledge and skills to meet its mission and goals  Further, Office of Human Resources officials reported that they plan to update the staffing model that they use to identify their IT staffing needs to include more reliable workload data  However, as discussed earlier, the Secret Service had not yet developed that updated model to determine its IT staffing needs  Office of Human Resources officials reported that once they update the staffing model they plan to re- evaluate the Secret Service’s IT staffing needs  The officials also stated that, going forward, they plan to reassess these needs each year as part of the annual budget cycle  Regular assessments of the IT workforce’s staffing needs should increase the likelihood that the Secret Service is able to appropriately identify the number of IT staff it needs to meet its mission and programmatic goals  Develop strategies and plans to address gaps in competencies and staffing—partly implemented  The Secret Service developed recruiting and hiring strategies to address certain competency and staffing needs (e g , cybersecurity) for its IT workforce  These strategies included, among other things, participating in DHS-wide recruiting events and using special hiring authorities  However, because OCIO did not identify all of its IT competency and staffing needs, and lacked a current analysis of its entire IT workforce, the Secret Service could not provide assurance that the recruiting and hiring strategies it developed were specifically targeted towards addressing current OCIO competency and staffing gaps  For example, without an analysis of the IT workforce’s skills, OCIO did not know the extent to which it had gaps in areas such as device management and cloud computing  As a result, the Secret Service’s recruiting strategies may not have been targeted to address any gaps in those areas  Until the Secret Service updates its recruiting and hiring strategies and plans to address all IT competency and staffing gaps identified (after OCIO completes its analysis of the entire IT workforce, as discussed earlier), the Secret Service will be limited in its ability to effectively recruit and hire staff to fill those gaps  According to the Office of Personnel Management, the Chief Human Capital Officers Council, and our prior work, once an agency has determined the critical skills and competencies that it needs to achieve programmatic goals, and identifies any competency or staffing gaps in its current workforce, the agency should be positioned to build effective recruiting and hiring programs  It is important that an agency has these programs in place to ensure that it can effectively recruit and hire employees with the appropriate skills to meet its various mission requirements  The Office of Personnel Management, the Chief Human Capital Officers Council, and we have also identified numerous leading practices associated with effective recruitment and hiring programs  Among these practices, an agency should (1) implement recruiting and hiring activities to address skill and staffing gaps by using the strategies and plans developed during the strategic workforce planning process; (2) establish and track metrics to monitor the effectiveness of the recruitment program and hiring process, including their effectiveness at addressing skill and staffing gaps, and report to agency leadership on progress addressing those gaps; and (3) adjust recruitment plans and hiring activities based on recruitment and hiring effectiveness metrics  The Secret Service minimally implemented the selected three leading practices associated with the recruitment and hiring workforce area  Specifically, the component partly implemented one of the three practices and did not implement the other two practices  Table 9 lists these selected practices and provides our assessment of the Secret Service’s implementation of the practices  Implement recruiting and hiring activities to address skill and staffing gaps by using the strategies and plans developed during the strategic workforce planning process—partly implemented  OCIO officials implemented the activities identified in the Secret Service’s recruiting and hiring plans  For example, as identified in its recruiting plan, OCIO participated in a February 2017 career fair to recruit job applicants at a technology conference  In addition, in August 2017, OCIO participated in a DHS-wide recruiting event  Secret Service officials reported that, during this event, they conducted four interviews for positions in OCIO  However, as previously discussed, OCIO did not identify all of its IT competency and staffing needs, and lacked a current analysis of its entire IT workforce  Without complete knowledge of its current IT competency and staffing gaps, the Secret Service could not provide assurance that the recruiting and hiring strategies that it had implemented fully addressed these gaps  Establish and track metrics to monitor the effectiveness of the recruitment program and hiring process, including their effectiveness at addressing skill and staffing gaps, and report to agency leadership on progress addressing those gaps—not implemented  The Secret Service had not established and tracked metrics for monitoring the effectiveness of its recruitment and hiring activities for the IT workforce  Officials in the Office of Human Resources attributed this to staffing constraints and said their priority was to address existing staffing gaps associated with the Secret Service’s law enforcement groups  In June 2018, Office of Human Resources officials stated that they plan to implement metrics to monitor the effectiveness of the hiring process for the IT workforce by October 2018  The officials also stated that they were in the process of determining (1) the metrics that are to be used to monitor the effectiveness of their workforce recruiting efforts and (2) whether they need to acquire new technology to support this effort  However, the officials did not know when they would implement the metrics for assessing the effectiveness of the recruitment activities and whether they would report the results to leadership  Until the Office of Human Resources (1) develops and tracks metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce, including their effectiveness at addressing skill and staffing gaps; and (2) reports to component leadership on those metrics, the Secret Service and the Office of Human Resources will be limited in their ability to analyze the recruitment program to determine whether the program is effectively addressing IT skill and staffing gaps  Further, Secret Service leadership will lack the information necessary to make effective recruitment decisions  Adjust recruitment plans and hiring activities based on recruitment and hiring effectiveness metrics—not implemented  While the Secret Service CIO stated in June 2018 that he planned to adjust the office’s recruiting and hiring strategies to focus on entry- level staff rather than mid-career employees, this planned adjustment was not based on metrics that the Secret Service was tracking  Instead, the CIO stated that he planned to make this change because his office determined that previous mid-career applicants were often unwilling or unable to wait for the Secret Service’s lengthy, required background investigation process to be completed  However, as previously mentioned, the Secret Service did not develop and implement any metrics for assessing the effectiveness of the recruitment and hiring activities for the IT workforce  As a result, the Office of Human Resources and OCIO were not able to use such metrics to inform adjustments to their recruiting and hiring plan and activities, thus, reducing their ability to target potential candidates for hiring  Until the Office of Human Resources and OCIO adjust their recruitment and hiring plans and activities as necessary, after establishing and tracking metrics for assessing the effectiveness of these activities for the IT workforce, the Secret Service will be limited in its ability to ensure that its recruiting plans and activities are appropriately targeted to potential candidates  In addition, the component will lack assurance that these plans and activities will effectively address skill and staffing gaps within its IT workforce  An organization should invest in training and developing its employees to help ensure that its workforce has the information, skills, and competencies that it needs to work effectively  In addition, training and development programs are an integral part of a learning environment that can enhance an organization’s ability to attract and retain employees with the skills and competencies needed to achieve cost-effective and timely results  DHS, the Secret Service, and we have previously identified numerous leading training and development-related practices  Among those practices, an organization should (1) establish a training and development program to assist the agency in achieving its mission and goals; (2) use tracking and other control mechanisms to ensure that employees receive appropriate training and meet certification requirements, when applicable; and (3) collect and assess performance data (including qualitative or quantitative measures, as appropriate) to determine how the training program contributes to improved performance and results  The Secret Service minimally implemented the selected three leading practices associated with the training and development workforce area  Specifically, the component partly implemented two of the three practices and did not implement one practice  Table 10 lists these selected leading practices and provides our assessment of the Secret Service’s implementation of the practices  Establish a training and development program to assist the agency in achieving its mission and goals—partly implemented  OCIO was in the process of developing a training program for its IT workforce  For example, OCIO developed a draft training plan that identified recommended training for the office’s various IT workforce groups (e g , voice communications employees)  However, the office had not defined the required training for each IT workforce group  In addition, OCIO officials had not yet determined which activities they would implement as part of the training program (e g , soliciting employee feedback after training is completed and evaluating the effectiveness of specific training courses), nor did they implement those activities  OCIO officials stated that they had not yet fully implemented a training program because their annual training budget for fiscal year 2018 was not sufficient to implement such a program  However, resource constrained programs especially benefit from identifying and prioritizing training activities to inform training budget decisions  Until OCIO (1) defines the required training for each IT workforce group, (2) determines the activities that it will include in its IT workforce training and development program based on its available training budget, and (3) implements those activities, the office may be limited in its ability to ensure that the IT workforce has the necessary knowledge and skills for their respective positions  Use tracking and other control mechanisms to ensure that employees receive appropriate training and meet certification requirements, when applicable—partly implemented  OCIO used a training system to track that the managers for IITT’s programs had met certain certification requirements for their respective positions  In addition, OCIO manually tracked the technical training that certain IT staff took  However, as discussed earlier, OCIO did not define the required training for each IT workforce group  As such, the office was unable to ensure that IT staff received the appropriate training relevant to their respective positions  Until it ensures that IT staff complete training specific to their positions (after defining the training required for each workforce group), OCIO will have limited assurance that the workforce has the necessary knowledge and skills  Collect and assess performance data (including qualitative or quantitative measures, as appropriate) to determine how the training program contributes to improved performance and results—not implemented  As previously discussed, OCIO did not fully implement a training program for the IT workforce; as such, the office was unable to collect and assess performance data related to such a program  OCIO officials stated that, once they fully implement a training program, they intend to collect and assess data on how this program contributes to improved performance  However, the officials were unable to specify a time frame for when they would do so  Until OCIO collects and assesses performance data (including qualitative or quantitative measures, as appropriate) to determine how the IT training program contributes to improved performance and results (once the training program is implemented), the office may be limited in its knowledge of whether the training program is contributing to improved performance and results  Employee morale is important to organizational performance and an organization’s ability to retain talent to perform its mission  We have previously identified numerous leading practices for improving employee morale  Among other things, we have found that an organization should (1) determine root causes of employee morale problems by analyzing employee survey results using techniques such as comparing demographic groups, benchmarking against similar organizations, and linking root cause findings to action plans; and develop and implement action plans to improve employee morale; (2) establish and track metrics of success for improving employee morale, and report to agency leadership on progress improving morale; and (3) maintain leadership support and commitment to ensure continued progress in improving employee morale, and demonstrate sustained improvement in morale  With regard to its IT workforce, the Secret Service substantially implemented the selected three practices associated with the employee morale workforce area  Specifically, the component fully implemented two of the selected practices and partly implemented one practice  Table 11 lists these selected practices and provides our assessment of the Secret Service’s implementation of the practices  Determine root causes of employee morale problems by analyzing employee survey results using techniques such as comparing demographic groups, benchmarking against similar organizations, and linking root cause findings to action plans  Develop and implement action plans to improve employee morale—fully implemented  The Secret Service used survey analysis techniques to determine the root causes of its low employee morale, on which we have previously reported  For example, the component conducted a benchmarking exercise where it compared the morale of the Secret Service’s employees, including IT staff, to data on the morale of employees at other agencies, including the U S  Capitol Police, U S  Coast Guard, and the Drug Enforcement Administration  As part of this exercise, the Secret Service also compared its employee work-life offerings (e g , on-site childcare and telework program) to those available at other agencies  In addition, the Secret Service developed and implemented action plans for improving employee morale  Among these action plans, for example, the component implemented a student loan repayment program and expanded its tuition assistance program’s eligibility requirements  Establish and track metrics of success for improving employee morale, and report to agency leadership on progress improving morale—fully implemented  The Secret Service tracked metrics for improving employee morale and reported the results to leadership  For example, the component tracked metrics on the percentage of the workforce, including IT staff, that participated in the student loan repayment and tuition assistance programs  In addition, the Chief Strategy Officer reported to the Chief Operating Officer the results related to meeting those metrics  Maintain leadership support and commitment to ensure continued progress in improving employee morale, and demonstrate sustained improvement in morale—partly implemented  Secret Service leadership developed and implemented initiatives that demonstrated their commitment to improving the morale of the Secret Service’s workforce  For example, since 2014, the Secret Service had worked with a contractor to identify ways to improve the morale of its entire workforce, including IT staff  However, as of June 2018, the Secret Service was unable to demonstrate that it had sustained improvement in the morale of the component’s IT staff  In particular, the component was only able to provide IT workforce-specific results from one employee morale assessment that was conducted subsequent to the consolidation of this workforce into OCIO in March 2017  These results were from an assessment conducted by the component’s Inspection Division in December 2017 (the assessment found that the majority of the Secret Service’s IT employees rated their morale as “very good” or “excellent ”) While the component also provided certain employee morale results from the Office of Personnel Management’s Federal Employee Viewpoint Survey in 2017, these results were not specific to the IT workforce  Instead, this workforce’s results were combined with those from staff in another Secret Service division  According to OCIO officials, the results were combined because, at the time of the survey, the IT workforce was administratively identified as being part of that other division  OCIO officials stated that, going forward, they plan to continue to assess the morale of the IT workforce on an annual basis as part of the Federal Employee Viewpoint Survey  In addition, the officials stated that OCIO-specific results may be available as part of the 2018 survey results, which the officials expect to receive by September 2018  By measuring employee satisfaction on an annual basis, the Secret Service should have increased knowledge of whether its initiatives that are aimed at improving employee morale are in fact increasing employee satisfaction  Agencies can use performance management systems as a tool to foster a results-oriented organizational culture that links individual performance to organizational goals  We have previously identified numerous leading practices related to performance management that are intended to enhance performance and ensure individual accountability  Among the performance management practices, agencies should (1) establish a performance management system that differentiates levels of staff performance and defines competencies in order to provide a fuller assessment of performance, (2) explicitly align individual performance expectations with organizational goals to help individuals see the connection between their daily activities and organizational goals, and (3) periodically provide individuals with regular performance feedback  The Secret Service substantially implemented the selected three leading practices associated with the performance management workforce area  Specifically, the component fully implemented one of the three practices and partly implemented the other two practices  Table 12 lists these selected leading practices and provides our assessment of the Secret Service’s implementation of the practices  Establish a performance management system that differentiates levels of staff performance and defines competencies in order to provide a fuller assessment of performance—partly implemented  The Secret Service’s performance management process requires leadership to make meaningful distinctions between levels of staff performance  In particular, the component’s performance plans for IT staff, which are developed by the Office of Human Resources and tailored by OCIO, as necessary, specify the criteria that leadership use to determine if an individual has met or exceeded the expectations associated with each competency identified in their respective performance plan  The performance plans include pre-established, department-wide competencies that are set by DHS, as well as occupational series-specific goals that may be updated by the Secret Service  However, because OCIO did not fully define and document all of its technical competency needs for the IT workforce, as discussed earlier, the Secret Service’s performance plans for IT staff did not include performance expectations related to the full set of technical competencies required for their respective positions  In addition, because OCIO officials were unable to specify a time frame for when they will identify all of the technical competency needs for the IT workforce (as previously discussed), the officials were also unable to specify a time frame for when they would update the IT workforce’s performance plans to include those relevant technical competencies  Until OCIO updates the performance plans for each occupational series within the IT workforce to include the relevant technical competencies, once identified, against which IT staff performance should be assessed, the office will be limited in its ability to provide IT staff with a complete assessment of their performance  In addition, Secret Service management will have limited knowledge of the extent to which IT staff are meeting all relevant technical competencies  Explicitly align individual performance expectations with organizational goals to help individuals see the connection between their daily activities and organizational goals—partly implemented  The Secret Service’s performance plans for IT staff identified certain goals that appeared to be related to organizational goals and objectives  For example, the performance plan for the Telecommunications Specialist occupational series (which is one of the series included in OCIO’s IT workforce) identified a goal for staff to support the voice, wireless, radio, satellite, and video systems serving the Secret Service’s protective and investigative mission  This performance plan goal appeared to be related to the component’s strategic goal on Advanced Technology, which included an objective to create the infrastructure needed to fulfill mission responsibilities  However, the Secret Service was unable to provide documentation that explicitly showed how individual employee performance links to organizational goals, such as a mapping of the goals identified in employee performance plans to organizational goals  Specifically, while Office of Human Resources officials stated that each Secret Service directorate is responsible for ensuring that employee goals map to high-level organizational goals, OCIO officials stated that they did not complete this mapping  The officials were unable to explain why they did not align the goals in their employees’ performance plans to the component’s high-level goals  According to the officials, the Secret Service is in the process of implementing a new automated tool that will require each office to explicitly align individual performance expectations to organizational goals  The officials stated that OCIO plans to use this tool to create employees’ fiscal year 2019 performance plans  By explicitly demonstrating how individual performance expectations align with organizational goals, the Secret Service’s IT staff should have a better understanding of how their daily activities contribute towards achieving the Secret Service’s goals  Periodically provide individuals with regular performance feedback—fully implemented  Secret Service leadership periodically provided their IT staff with performance feedback  Specifically, on an annual basis, OCIO staff received feedback during a mid-year and end-of-year performance feedback assessment  In our prior work, we have stressed that candid and constructive feedback can help individuals maximize their contribution and potential for understanding and realizing the goals and objectives of an organization  Further, this feedback is one of the strongest drivers of employee engagement  According to leading practices of the Software Engineering Institute, effective program oversight includes monitoring program performance and conducting reviews at predetermined checkpoints or milestones  This is done by, among other things, comparing actual cost, schedule, and performance data with estimates in the program plan and identifying significant deviations from established targets or thresholds for acceptable performance levels  In addition, the Software Engineering Institute previously identified leading practices for effectively monitoring the performance of agile projects  According to the Institute, agile development methods focus on delivering usable, working software frequently; as such, it is important to measure the value delivered during each iteration of these projects  To that end, the Institute reported that agile projects should be measured on velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  DHS and the Secret Service had fully implemented the selected leading practice for monitoring the performance of one program and three projects within the IITT investment, and conducting reviews of this program and these projects at predetermined checkpoints  In addition, with regard to the selected leading practice for monitoring agile projects, the Secret Service had fully implemented this practice for one of its two projects being implemented using agile and had partially implemented this practice for the other project  Table 13 provides a summary of DHS’s and the Secret Service’s implementation of these leading practices, as relevant for one program and three projects within IITT  Monitor program performance and conduct reviews at predetermined checkpoints or milestones  Consistent with leading practices, DHS and the Secret Service monitored the performance of IITT’s program and projects by comparing actual cost, schedule, and performance information against planned targets and conducting reviews at predetermined checkpoints  For example, within the Secret Service: The Enabling Capabilities program and Multi-Level Security project monitored their contractors’ costs spent to-date on a monthly basis and compared them to the total contract amounts  OCIO used integrated master schedules to monitor the schedule performance of the Enabling Capabilities program and Multi-Level Security project  OCIO also monitored the cost, schedule, and performance of the Uniformed Division Resource Management System and Events Management projects during monthly status reviews  In addition, DHS and the Secret Service conducted acquisition decision event reviews and systems engineering life cycle technical reviews of IITT’s program and projects at predetermined checkpoints and, when applicable, identified deviations from established cost, schedule, and performance targets  For example: Secret Service OCIO met with DHS’s Office of Program Accountability and Risk Management in February 2017, and with DHS’s Acting Under Secretary for Management in June 2017, to discuss a schedule breach for the Enabling Capabilities program  In particular, the Enabling Capabilities program informed DHS that the program needed to change the planned date for acquisition decision event 3 (the point at which a decision is made to fully deploy the system) in order to conduct tests in an operational environment prior to that decision event  This delay was due to the Secret Service misunderstanding the tests that it was required to conduct prior to that decision event  Specifically, the Enabling Capabilities program had conducted tests on “production representative” systems, but these tests were not sufficient to meet the requirements for acquisition decision event 3  The project team for Multi-Level Security identified that certain technical issues they had experienced would delay system deployment and full operational capability (the point at which an investment becomes fully operational)  As such, in October 2017, the project notified the Secret Service Component Acquisition Executive of these expected delays  In particular, the web browser that was intended to provide users on “Sensitive But Unclassified” workstations the ability to view information from different security levels, experienced technical delays in meeting personal identity verification requirements  The project team also described for the executive how the schedule delay would affect the project’s performance metrics and funding, and subsequently updated the project plan accordingly  Measure and monitor agile projects on, among other things, velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of features and user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  Secret Service OCIO measured its two agile projects—Uniformed Division Resource Management System and Events Management— using certain agile metrics  In particular, OCIO officials measured the Uniformed Division Resource Management System and Events Management projects using key metrics related to velocity and development progression  For example, the officials measured development progression for both projects on a daily basis  In addition, OCIO officials monitored each project’s progress against these metrics during bi-weekly reviews that they conducted with each project team  The OCIO officials also tracked product quality metrics for the Uniformed Division Resource Management System  For example, on a monthly basis, the officials tracked the number of helpdesk tickets that had been resolved related to the system  In addition, on a quarterly basis, they tracked the number of Uniformed Division Resource Management System defects that (1) had been fixed and (2) were in the backlog  However, while OCIO officials received certain post-deployment user satisfaction information from end-users of the Uniformed Division Resource Management System by, among other things, tracking the number of helpdesk tickets related to the system and via daily verbal, undocumented feedback from certain Uniformed Division officers, OCIO officials had not fully measured and documented post- deployment user satisfaction with the system, such as via a survey of employees who use the system  The officials stated that they had not conducted and documented a survey because they were focused on (1) addressing software performance issues that occurred after they deployed the system to a limited number of users, and (2) continuing system deployment to the remaining users after they addressed the performance issues  OCIO officials stated that they plan to conduct such a documented survey by the end of September 2018  The results of the user satisfaction survey should provide OCIO with important information on whether the Uniformed Division Resource Management System is meeting users’ needs  The Secret Service’s full implementation of 11 of 14 component-level CIO responsibilities constitutes a significant effort to establish CIO oversight for the component’s IT portfolio  Additional efforts to fully implement the remaining 3 responsibilities, including ensuring that all IT contracts are reviewed, as appropriate; ensuring that the Secret Service’s enterprise governance policy appropriately specifies the CIO’s role in developing and reviewing the component’s IT budget formulation and execution; and ensuring agile projects measure product quality and post-deployment user satisfaction, will further position the CIO to effectively manage the Secret Service’s IT portfolio  When effectively implemented, IT workforce planning and management activities can facilitate the successful accomplishment of an agency’s mission  However, the Secret Service had not fully implemented all of the 15 selected practices for its IT workforce for any of the five areas— strategic planning, recruitment and hiring, training and development, employee morale, and performance management  The Secret Service’s lack of (1) a strategic workforce planning process, including the identification of all required knowledge and skills, assessment of competency gaps, and targeted strategies to address specific gaps in competencies and staffing; (2) targeted recruiting activities, including metrics to monitor the effectiveness of the recruitment program and adjustment of the recruitment program and hiring efforts based on metrics; (3) a training program, including the identification of required training for IT staff, ensuring that staff take required training, and assessment of performance data regarding the training program; and (4) a performance management system that includes all relevant technical competencies, greatly limits its ability to ensure the timely and effective acquisition and maintenance of the Secret Service’s IT infrastructure and services  On the other hand, by monitoring program performance and conducting reviews at predetermined checkpoints for one program and three projects associated with the IITT investment, in accordance with leading practices, the Secret Service and DHS provided important oversight needed to guide that program and those projects  Measuring projects on leading agile metrics also provided the Secret Service CIO with important information on project performance  We are making the following 13 recommendations to the Director of the Secret Service: The Director should ensure that the CIO establishes and documents an IT acquisition review process that ensures the CIO or the CIO’s delegate reviews all contracts containing IT, as appropriate  (Recommendation 1) The Director should update the enterprise governance policy to specify (1) the CIO’s current role and responsibilities on the Executive Resources Board, to include developing and reviewing the IT budget formulation and execution; and (2) the Deputy CIO’s role and responsibilities on the Enterprise Governance Council  (Recommendation 2) The Director should ensure that the Secret Service develops a charter for its Executive Resources Board that specifies the roles and responsibilities of all board members, including the CIO  (Recommendation 3) The Director should ensure that the CIO includes product quality and post-deployment user satisfaction metrics in the modular outcomes and target measures that the CIO sets for monitoring agile projects  (Recommendation 4) The Director should ensure that the CIO identifies all of the required knowledge and skills for the IT workforce  (Recommendation 5) The Director should ensure that the CIO regularly analyzes the IT workforce to identify its competency needs and any gaps it may have  (Recommendation 6) The Director should ensure that, after OCIO completes an analysis of the IT workforce to identify any competency and staffing gaps it may have, the Secret Service updates its recruiting and hiring strategies and plans to address those gaps, as necessary  (Recommendation 7) The Director should ensure that the Office of Human Resources (1) develops and tracks metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce, including their effectiveness at addressing skill and staffing gaps; and (2) reports to component leadership on those metrics  (Recommendation 8) The Director should ensure that the Office of Human Resources and OCIO adjust their recruitment and hiring plans and activities, as necessary, after establishing and tracking metrics for assessing the effectiveness of these activities for the IT workforce  (Recommendation 9) The Director should ensure that the CIO (1) defines the required training for each IT workforce group, (2) determines the activities that OCIO will include in its IT workforce training and development program based on its available training budget, and (3) implements those activities  (Recommendation 10) The Director should ensure that the CIO ensures that the IT workforce completes training specific to their positions (after defining the training required for each workforce group)  (Recommendation 11) The Director should ensure that the CIO collects and assesses performance data (including qualitative or quantitative measures, as appropriate) to determine how the IT training program contributes to improved performance and results (once the training program is implemented)  (Recommendation 12) The Director should ensure that the CIO updates the performance plans for each occupational series within the IT workforce to include the relevant technical competencies, once identified, against which IT staff performance should be assessed  (Recommendation 13) DHS provided written comments on a draft of this report, which are reprinted in appendix III  In its comments, the department concurred with all 13 of our recommendations and provided estimated completion dates for implementing each of them  For example, with regard to recommendation 2, the department stated that the Secret Service would update its enterprise governance policy and related policies to outline the roles and responsibilities of the CIO and Deputy CIO, among others, by March 31, 2019  In addition, for recommendation 13, the department stated that the Secret Service OCIO will include relevant technical competencies in performance plans, as appropriate, in the next performance cycle that starts in July 2019  If implemented effectively, these actions should address the weaknesses we identified  The department also identified a number of other actions that it said had been taken to address our recommendations  For example, in response to recommendation 8, which calls for the Office of Human Resources to (1) develop and track metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce and (2) report to component leadership on those metrics, DHS stated that the Secret Service’s Office of Human Resources’ Outreach Branch provides to the department metrics on recruitment efforts toward designated priority mission-critical occupations  However, for fiscal year 2017, only 1 of the 12 occupational series associated with the Secret Service’s IT workforce was designated as a mission-critical occupation for the component (i e , the 2210 IT Specialist series)  The 11 other occupational series were not designated as mission- critical occupations  In addition, for fiscal year 2018, none of these 12 occupational series were designated as mission-critical occupations  As such, metrics on recruiting for these IT series may not have been reported to DHS leadership  Moreover, while we requested documentation of the recruiting metrics for the Secret Service’s IT workforce and, during the course of our review, had multiple subsequent discussions with the Secret Service regarding such metrics, the component did not provide documentation that demonstrated it had established recruiting metrics for its IT workforce  Tracking such metrics and reporting the results to Secret Service leadership, as we recommended, would provide management with important information necessary to make effective recruitment decisions  Further, in response to recommendation 10, which among other things, calls for the CIO to define the required training for each IT workforce group, the department stated that the Secret Service OCIO recently developed training requirements for each workforce group, which were issued during our audit  However, while during our audit OCIO provided a list of recommended training courses, the office did not identify them as being required courses  Defining training that is required for each IT workforce group, as we recommended, would inform OCIO of the necessary training for each position and enable the office to prioritize this training, to ensure that its staff have the needed knowledge and skills  In addition to the aforementioned comments, we received technical comments from DHS and Secret Service officials, which we incorporated, as appropriate  We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Director of the Secret Service, and other interested parties  In addition, this report is available at no charge on the GAO website at http://www gao gov  Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or HarrisCC@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made major contributions to this report are listed in appendix IV  Our objectives were to evaluate the extent to which: (1) the U S  Secret Service (Secret Service) Chief Information Officer (CIO) has implemented selected information technology (IT) oversight responsibilities, (2) the Secret Service has implemented leading workforce planning and management practices for its IT workforce, and (3) the Secret Service and the Department of Homeland Security (DHS) have implemented selected performance and progress monitoring practices for the Information Integration and Technology Transformation (IITT) investment  To address the first objective, we analyzed DHS’s policies and guidance on IT management to identify the responsibilities that were to be implemented by the component-level CIO related to overseeing the Secret Service’s IT portfolio, including existing systems, acquisitions, and investments  From the list of 33 responsibilities that we identified, we then excluded the responsibility that was associated with information security, which is expected to be addressed as part of a separate, subsequent GAO review  We also excluded those responsibilities that were significantly large in scope (e g , implement an enterprise architecture) or that, in our professional judgment, lacked specificity (e g , provide timely delivery of mission IT services)  As a result, we excluded from consideration for this review a total of 10 CIO responsibilities  For the 23 that remained, we then combined certain responsibilities that overlapped with other related responsibilities  For example, we combined related responsibilities on the component CIO’s review of IT contracts  As a result, we identified 14 responsibilities for review  We validated with the acting DHS CIO that these responsibilities were key responsibilities for the department’s component-level CIOs  We then included all 14 of the responsibilities in our review  The 14 selected component-level CIO responsibilities were: 1  Develop and review the component IT budget formulation and execution  2  Manage the component IT investment portfolio, including establishing an IT acquisition review process that enables component and DHS review of component acquisitions (i e , contracts) that contain IT  3  Develop, implement, and maintain a detailed IT strategic plan  4  Ensure all component IT policies are in compliance and alignment with DHS IT directives and instructions  5  Concur with each program’s and/or project’s systems engineering life cycle tailoring plan  6  Support the Component Acquisition Executive to ensure processes are established that enable systems engineering life cycle technical reviews and that they are adhered to by programs and/or projects  7  Ensure that all systems engineering life cycle technical review exit criteria are satisfied for each of the component’s IT programs and/or projects  8  Ensure the necessary systems engineering life cycle activities have been satisfactorily completed as planned for each of the component’s IT programs and/or projects  9  Concur with the systems engineering life cycle technical review completion letter for each of the component’s IT programs and/or projects  10  Maintain oversight of their component’s agile development approach for IT by appointing the responsible personnel, identifying investments for adoption, and reviewing artifacts  11  With Component Acquisition Executives, evaluate and approve the application of agile development for IT programs consistent with the component’s agile development approach  12  Set modular outcomes and target measures to monitor the progress in achieving agile implementation for IT programs and/or projects within their component  13  Participate on DHS’s CIO Council, Enterprise Architecture Board, or other councils/boards as appropriate, and appoint employees to serve when necessary  14  Meet the IT competency requirements established by the DHS CIO, as required in the component CIO’s performance plan  To determine the extent to which the Secret Service CIO has implemented these responsibilities, we obtained and assessed relevant component documentation and compared it to the responsibilities  Specifically, we obtained and analyzed documentation including evidence of the CIO’s participation on the Secret Service governance board that has final decision authority and responsibility for enterprise governance, including the IT budget; monthly program management reports showing the CIO’s oversight of IT programs, projects, and systems; monthly status reports on program spending; the Secret Service’s IT strategic plan; the Secret Service’s enterprise governance policy; meeting minutes from the DHS board and council on which the CIO participated (i e , the CIO Council and Enterprise Architecture Board); and documentation demonstrating whether the CIO met the IT competency requirements  In addition, we obtained and analyzed relevant documentation related to the CIO’s oversight of the major IT investments on which the Secret Service was spending development, modernization, and enhancement funds during fiscal year 2017  As of July 2017, the component had one investment—IITT—that met this criterion  IITT is a portfolio investment that, as of July 2017, included two programs (one of which included three projects) and one standalone project (i e , it was not part of another program) that had capabilities that were in planning or development and modernization: the Enabling Capabilities program, Enterprise Resource Management System program (which included three projects, called Uniformed Division Resource Management System, Events Management, and Enterprise-wide Scheduling), and Multi-Level Security project  In particular, we obtained and analyzed documentation related to the CIO’s oversight of the systems engineering life cycles for IITT’s Enabling Capabilities program and the Uniformed Division Resource Management System, Events Management, and Multi-Level Security projects  This documentation included acquisition program baselines, systems engineering life cycle tailoring plans, and systems engineering life cycle technical review briefings and completion letters  We then compared the documentation against the five selected systems engineering life cycle oversight responsibilities (responsibilities 5, 6, 7, 8, and 9)  We also obtained and analyzed documentation related to the CIO’s oversight of two projects that the Secret Service was implementing using an agile methodology—Uniformed Division Resource Management System and Events Management  Specifically, we obtained and assessed documentation of (1) the CIO’s approval for these projects to be implemented using an agile methodology and (2) the agile development metrics that the CIO established for each of these projects  We then compared this documentation to the three agile development-related component-level CIO responsibilities (responsibilities 10, 11, and 12)  Further, to determine the extent to which the Secret Service CIO had established an IT acquisition (i e , contract) review process that enabled component and DHS review of component contracts that contain IT (which is part of responsibility 2), we first asked Secret Service officials to provide us with a list of all new, unclassified IT contracts that the component awarded between October 1, 2016, and June 30, 2017  The Secret Service officials provided a list of 54 contracts  We validated that these were contracts for IT or IT services by: (1) searching for them in the Federal Procurement Data System – Next Generation; (2) identifying their associated product or service codes, as reported in that system; and (3) determining whether those codes were included in the universe of 79 IT product or service codes identified by the Category Management Leadership Council  In validating the list of 54 contracts provided by the Secret Service, we determined that 5 of the contracts were not associated with an IT product or service code  As such, we removed those contracts from the list  In addition, we found that three other items identified by the component were not in the Federal Procurement Data System – Next Generation  Secret Service officials subsequently confirmed that these three items were not contracts  We therefore removed these three items from the list  As such, the final list of validated contracts identified by the Secret Service included 46 IT contracts  In addition, to identify any IT contracts that were not included in the list provided by the Secret Service, we conducted a search of the Federal Procurement Data System – Next Generation to identify all unclassified contracts that (1) the component awarded between October 1, 2016, and June 30, 2017; (2) were not a modification of a contract; and (3) were associated with 1 of the 79 IT product or service codes identified by the Category Management Leadership Council  Based on these criteria, we identified 144 Secret Service IT contracts in the Federal Procurement Data System – Next Generation (these 144 contracts included the 46 contracts previously identified by Secret Service officials)  We then asked Secret Service officials to validate the accuracy, completeness, and reliability of these data, which they did  From each of these two lists of IT contracts (i e , the list of 46 IT contracts identified by the Secret Service and the list of 144 IT contracts that we identified from the Federal Procurement Data System – Next Generation), we then selected random, non-generalizable samples of contracts, as described below  First, from the list of 46 IT contracts identified by Secret Service officials, we removed 4 contracts that had total values of less than $10,000  To ensure that we selected across all contract sizes, we randomly selected 12 contracts from the remaining list of 42 contracts, using the following cost ranges: $10,000 to $50,000 (4 contracts), more than $50,000 to less than $250,000 (4 contracts), and more than $250,000 (4 contracts)  Second, from our list of 144 IT contracts that we identified from the Federal Procurement Data System – Next Generation, we removed the 46 contracts identified by Secret Service officials  We also removed 12 contracts that had total values of less than $10,000  To ensure that we selected across all contract sizes, we randomly selected 21 contracts from the remaining list of 86 contracts, using the following cost ranges: $10,000 to $50,000 (7 contracts), more than $50,000 to less than $250,000 (7 contracts), and more than $250,000 (7 contracts)  In total, we selected 33 IT contracts for review  We separated the contracts into the three cost ranges identified above in order to ensure that contracts of different value levels had been selected  This enabled us to determine the extent to which the CIO appropriately reviewed contracts of all values  To determine the extent to which the CIO had established an IT contract approval process that enabled the Secret Service and DHS, as appropriate, to review IT contracts, we first asked Secret Service Office of the CIO (OCIO) officials for documentation of their IT contract approval process  These officials were unable to provide such documentation  Instead, the officials stated that the Secret Service CIO or the CIO’s delegate approves all IT contracts prior to award  The officials also provided documentation that identified four staff to whom the CIO had delegated his approval authority  Further, the officials stated that, in accordance with DHS’s October 2016 IT acquisition review guidance, they submitted to DHS OCIO for approval any IT contracts that met DHS’s thresholds for review, including those that (1) had total estimated procurement values of $2 5 million or more, and (2) were associated with a major investment  Based on the IT acquisition review process that Secret Service OCIO officials described, we then obtained and analyzed each of the 33 selected IT contracts and associated approval documentation to determine whether or not the Secret Service CIO or the CIO’s delegate had approved each of the contracts  In particular, we (1) reviewed the name of the contract approver on the approval documentation, and (2) compared the signature dates that were on the contracts to the signature dates that were identified on the associated approval documentation  In addition, to determine whether or not the Secret Service CIO submitted to DHS OCIO for approval the IT contracts that (1) had total estimated procurement values of $2 5 million or more, and (2) were associated with major investments, we first analyzed the 144 Secret Service IT contracts that we had previously pulled from the Federal Procurement Data System – Next Generation to determine which contracts met the $2 5 million threshold  We identified 4 contracts that met this threshold  We then requested that OCIO identify the levels (i e , major or non-major) of the investments associated with these contracts  According to OCIO officials, 3 of the 4 contracts were associated with non-major investments and 1 was not associated with an investment  As such, based on DHS’s October 2016 IT acquisition review guidance, none of these contracts needed to be submitted to DHS OCIO for review  We also interviewed Secret Service officials, including the CIO and Deputy CIO, regarding the CIO’s implementation of the 14 selected component-level responsibilities  We assessed the evidence against the selected responsibilities to determine the extent to which the CIO had implemented them  To address the second objective—determining the extent to which the Secret Service had implemented leading workforce planning and management practices for its IT workforce—we first identified seven topic areas associated with human capital management based on the following sources: The Office of Personnel Management’s Human Capital Framework  Office of Personnel Management and the Chief Human Capital Officers Council Subcommittee for Hiring and Succession Planning, End-to-End Hiring Initiative  GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others  GAO, IT Workforce: Key Practices Help Ensure Strong Integrated Program Teams; Selected Departments Need to Assess Skill Gaps  GAO, Department of Homeland Security: Taking Further Action to Better Determine Causes of Morale Problems Would Assist in Targeting Action Plans  GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government  GAO, Results-Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success  DHS acquisition guidance  Secret Service acquisition guidance  Among these topic areas, we then selected five areas that, in our professional judgment, were of particular importance to successful workforce planning and management  They were also previously identified as part of our high-risk and key issues work on human capital management  These areas include: (1) strategic planning, (2) recruitment and hiring, (3) training and development, (4) employee morale, and (5) performance management  We also reviewed these same sources and identified numerous leading practices associated with the five topic areas  Among these leading practices, we then selected three leading practices within each of the five areas (for a total of 15 selected practices)  The selected practices were foundational practices that, in our professional judgment, were of particular importance to successful workforce planning and management  Table 14 identifies the five selected workforce areas and 15 selected associated practices  To determine the extent to which the Secret Service had implemented the selected leading workforce planning and management practices for its IT workforce, we obtained and assessed documentation and compared it against the 15 selected practices  In particular, we analyzed the Secret Service’s human capital strategic plan, human capital staffing plan, IT strategic plan, documentation of the component’s staffing model that it used to determine the number of IT staff needed, an independent verification and validation report on the component’s staffing models, documentation of the current number of IT staff, the Secret Service’s recruitment and outreach plans, documentation of DHS’s hiring authorities (which are applicable to the Secret Service), the Secret Service’s training strategic plan, IT workforce training plan, action plans for improving employee morale, and templates used for measuring and reporting employee performance  We also interviewed Secret Service officials—including the CIO, Deputy CIO, and workforce planning staff—about the component’s workforce- related policies and documentation  Further, we discussed with the officials the Secret Service’s efforts to implement the selected workforce practices for its IT workforce  Regarding our assessments of the Secret Service’s implementation of the 15 selected workforce planning and management practices, we assessed a practice as being fully implemented if component officials provided supporting documentation that demonstrated all aspects of the practice  We assessed a practice as not implemented if the officials did not provide any supporting documentation for that practice, or if the documentation provided did not demonstrate any aspect of the practice  We assessed a practice as being partly implemented if the officials provided supporting documentation that demonstrated some, but not all, aspects of the selected practice  In addition, related to our assessments of the Secret Service’s implementation of the five selected overall workforce areas, we assessed each area as follows, based on the implementation of the three selected practices within each area: Fully implemented: The Secret Service provided evidence that it had fully implemented all three of the selected practices within the workforce area; Substantially implemented: The Secret Service provided evidence that it had either fully implemented two selected practices and partly implemented the remaining one selected practice within the workforce area, or fully implemented one selected practice and partly implemented the remaining two selected practices within the workforce area; Partially implemented: The Secret Service provided evidence that it had partly implemented each of the three selected practices within the workforce area; Minimally implemented: The Secret Service provided evidence that it partly implemented two selected practices and not implemented the remaining one selected practice within the workforce area, or partly implemented one selected practice and not implemented the remaining two selected practices within the workforce area; or Not implemented: The Secret Service did not provide evidence that it had implemented any of the three selected practices within the workforce area  To address the third objective—determining the extent to which the Secret Service and DHS have implemented selected performance and progress monitoring practices for IITT—we reviewed leading project monitoring practices and guidance from the Software Engineering Institute  First, we reviewed the practices within the Project Monitoring and Control process area of the Institute’s Capability Maturity Model Integration® for Acquisition  Based on our review, we identified four practices associated with monitoring program performance and progress  In our professional judgment, all four of these practices were of significance to managing the IITT investment given the phase of the life cycle that the investment was in  As such, we elected to include all four of these practices in our review, and combined them into one practice, as follows: Monitor program performance and conduct reviews at predetermined checkpoints or milestones by, among other things, comparing actual cost, schedule, and performance data with estimates in the program plan and identifying significant deviations from established targets or thresholds for acceptable performance levels  Next, given the agile development methodology that the Secret Service was using for certain projects within IITT, we reviewed the Software Engineering Institute’s technical note on the progress monitoring of agile contractors  Based on our review, and in consultation with an internal expert, we selected four agile metrics that the Institute identified as important for successful agile implementations and that, in our professional judgment, were of most significance to monitoring the performance of IITT’s agile projects  We then combined these four metrics into one practice, as follows: Measure and monitor agile projects on velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of features and user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  To determine the extent to which DHS and the Secret Service had implemented the first selected practice, we analyzed relevant program management and governance documentation for IITT’s Enabling Capabilities program, and Multi-Level Security, Uniformed Division Resource Management System, and Events Management projects  In particular, we analyzed acquisition program baselines, DHS acquisition decision event memorandums, artifacts from DHS and Secret Service program oversight reviews, cost monitoring reports, program integrated master schedules, and program status briefings, and compared this documentation to the selected practice  We also interviewed Secret Service OCIO officials regarding the Secret Service’s and DHS’s efforts to monitor the IITT investment’s performance and progress  To determine the extent to which the Secret Service had implemented the second selected practice related to measuring and monitoring agile projects on agile metrics (i e , velocity, development progression, product quality, and post-deployment user satisfaction), we obtained and analyzed agile-related documentation for the two projects that the Secret Service was implementing using an agile methodology—Uniformed Division Resource Management System and Events Management  Specifically, to determine the extent to which the Secret Service was measuring and monitoring these two projects on metrics for velocity and development progression, we obtained and analyzed documentation, such as sprint burndown charts and monthly program status reports, and compared it to the selected practice  In addition, the agile metrics for product quality and post-deployment user satisfaction were only applicable to projects that had been deployed to users  As such, these metrics were applicable to the Uniformed Division Resource Management System (which the Secret Service had deployed to users) and were not applicable to Events Management (which the Secret Service had not yet deployed to users, as of early May 2018)  We therefore obtained and analyzed documentation demonstrating that Secret Service OCIO measured product defects for the Uniformed Division Resource Management System  We also requested documentation demonstrating that OCIO had measured and monitored post-deployment user satisfaction for this project, including via a survey  OCIO officials stated that they had not conducted such a survey and were unable to provide documentation demonstrating they had measured post- deployment user satisfaction for the Uniformed Division Resource Management System  To assess the reliability of the cost, schedule, and agile-related data that were in DHS and the Secret Service’s program management and governance documentation for the IITT investment, we (1) analyzed related documentation and assessed the data against existing agency records to identify consistency in the information, and (2) examined the data for obvious outliers, incomplete, or unusual entries  We determined that the data in these documents were sufficiently reliable for our purpose, which was to evaluate the extent to which DHS and the Secret Service had implemented processes for monitoring the IITT investment’s performance and progress  We conducted this performance audit from May 2017 to November 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  As of June 2018, the Secret Service’s Information Integration and Technology Transformation (IITT) investment included two programs (one of which included three projects) and one project that had capabilities that were in planning or development and modernization, as described below: Enabling Capabilities  This program is intended to, among other things, (1) modernize and enhance the Secret Service’s information technology (IT) network infrastructure, including increasing bandwidth and improving the speed and reliability of the Secret Service’s IT system performance; (2) enhance cybersecurity to protect against potential intrusions and viruses; and (3) provide counterintelligence and data mining capabilities to improve officials’ ability to perform the Secret Service’s investigative mission  Enterprise Resource Management System  This program comprises three projects that are intended to provide: a system that will enable the Secret Service’s Uniformed Division to efficiently and effectively plan, provision, and schedule missions (this project is referred to as Uniformed Division Resource Management System), a system that will unify the logistical actions (e g , assigning personnel) surrounding special events that Secret Service agents need to protect, such as the United Nations General Assembly (this project is referred to as Events Management), and a capability for creating schedules for Secret Service agents and administrative, professional, and technical staff, as well as the ability to generate reports on information such as monthly hours worked (this project is referred to as Enterprise-wide Scheduling)  Multi-Level Security  This project is intended to enable authorized Secret Service users to view two levels of classified information on a single workstation  Previously, data at various security levels were contained and used in multiple disparate systems  Multi-Level Security is intended to streamline users’ access to information at different security levels in order to enable them to more quickly and effectively perform their duties  Table 15 provides the planned life cycle cost and schedule estimates (threshold values) for each IITT program and project that had capabilities in planning or development and modernization, as of June 2018  In addition, the table describes any changes in those cost and schedule estimates, as well as the key reasons for any changes, as identified by officials from the Secret Service’s Office of the Chief Information Officer  In addition to the contact named above, the following staff made key contributions to this report: Shannin O’Neill (Assistant Director), Emily Kuhn (Analyst-in-Charge), Quintin Dorsey, Rebecca Eyler, Javier Irizarry, and Paige Teigen 
Report 2:  the United States tends to underinvest in dual-use technologies  National security requirements alone often do not justify major DOD support, and market prospects alone often appear to be too long-term or high risk to justify US industry carrying the entire development burden     The committee encourages use of cooperative agreements and other transactions in lieu of grants or contracts     The provision would require that at least 50 percent of funding over the life of a partnership derive from non-federal sources but would allow for a smaller industry share at the start  The conference report stated that the partnerships should focus on programs that fit into the security needs within DARPA  The conference report also stated that OTs are appropriate for those cases where the "regulations applicable to the allocation of patent and data rights under the procurement statutes may not be appropriate to partnership arrangements in certain cases " National Defense Authorization Act for FY1993 ( P L  102-484 ) Cost Sharing Section 4221 established 10 U S C  2511, which required DOD to establish cooperative arrangements with industry, educational institutions, federal labs, and other entities, to pursue research, development, and application of dual-use technologies  The section authorized DOD to use grants, contracts, cooperative agreements, or OTs to create these partnerships, and that the Federal government should not contribute more than 50% of the costs related to projects under this authority  National Defense Authorization Act for FY1994 ( P L  103-160 ) Expanded Authority Section 827 established 10 U S C  2358, which gave the Secretary of Defense and the Secretaries of the military departments the authority to conduct basic, advanced, and applied research through the use of contracts, cooperative agreements, grants, and OTs  Previously, OTs were only authorized for advanced research  Prototype Authorities Section 845 granted DARPA the authority to use OTs for prototype projects directly related to weapons or weapon systems proposed to be acquired by DOD  Section 845 required that "to the maximum extent practicable," prototypes be competitively awarded  This authority was set to terminate after three years  Section 845 remained as a note to 10 U S C  2371 until separately codified as 10 U S C  2371b in the FY2016 NDAA ( P L  114-92 )  Federal Acquisition Streamlining Act of 1994 ( P L  103-355 ) Section 1301 redesignated the language in 10 U S C  2358 (granting the authority to use OTs) to 10 U S C  2371  Reporting Requirements Section 1301 also required DOD to submit an annual report to the armed services committees, to include a general description of the other transactions, including the technologies involved in the research, the potential military and, if any, commercial utility of such technologies, the reasons for not using a contract or grant to provide support for such research, and the amount of payments, if any, received during the fiscal year pursuant to a clause in the other transactions and to what accounts such payments were credited  National Defense Authorization Act for FY1997 ( P L  104-201 ) Section 203 required that a senior DOD official be designated in OSD, and that the officials' sole responsibility be developing policy related to, and ensuring implementation of, DOD's dual-use technology program  This section authorized DOD to use OTs (as well as contracts, cooperative agreements, and grants) for dual-use projects only if the project "is entered into through the use of competitive procedures " Section 743 granted DOD the authority to use OTs to conduct research on Gulf War Syndrome, to determine its relationship to possible exposures of members of the Armed Forces to chemical warfare agents and hazardous materials, and the use of inoculations and new drugs  Expanded Prototype Authorities Section 804 amended Section 845 of the FY1994 NDAA by extending to the military departments and officials designated by the Secretary of Defense, the authority to use OTs for certain prototype projects  This authority, originally granted solely to DARPA and set to expire after three years, was given a new termination date of September 30, 1999  Reporting Requirements Section 267 modified elements of the annual report to the armed services committees  National Defense Authorization Act for FY1998 ( P L  105-85 ) Section 832 amended 10 U S C  2371 by clarifying that certain information submitted to DOD (i e  a proposal, business plan, technical information) be protected from disclosure pursuant to 5 U S C  552 for a period of five years  Strom Thurmond National Defense Authorization Act for FY1999 ( P L  105-261 ) Section 241 extended the sunset day for the authority to use OTs for prototypes from September 30, 1999, to September 30, 2001  Section 817 amended Section 2371 of Title 10, United States Code, clarifying that information submitted by outside parties in cooperative agreements for basic, applied, and advanced research is protected from disclosure under Section 552 of Title 5, United States Code  Department of Defense Appropriations Act, 1999 ( P L  105-262 ) While the enacted FY1999 defense appropriations bill ( P L  105-262 ) did not include legislative language addressing OTs, H Rept  105-591 , which accompanied the House-reported version of H R  4103 , included language expressing the House Appropriations Committee's "serious reservations" regarding the Air Force's then-proposed use of an OT agreement—instead of a contract— to develop the Evolved Expendable Launch Vehicle (EELV) program  The committee noted that "under [OTs] traditional safeguards which protect the government's interest in large acquisition programs are largely absent," and required the Under Secretary of Defense for Acquisition, Technology, and Logistics (now the Under Secretary of Defense for Acquisition and Sustainment) and the DOD Inspector General to certify to the congressional defense committees that the use of an OT was appropriate for the EELV program, and that "adequate safeguards exist[ed] to protect the government's interest and monitor program performance " National Defense Authorization Act for FY2000 ( P L  106-65 ) Section 801 required that for prototypes using OT authorities, DOD ensure that GAO, under its audit authority, have access to records relating to other transaction prototype agreements exceeding $5 million  Section 801 allowed for a waiver to GAO access and exempted entities that over the last year have not entered into an agreement with DOD that provided for audit access by a government entity  According to the Senate report: Senior DOD officials have sought legislation to extend other transaction authority to production contracts  Under current authority, there is some debate about whether GAO has audit access to other transactions  As the size, costs, and complexity of programs being funded using other transactions increases, the committee wants to ensure that the GAO has audit access in relation to the higher levels of spending and added risks  Reporting Requirements The Senate report also addressed reporting requirements and congressional intent to review the use of OTs  The report stated the following: The committee is assessing the utility of other transaction prototype authority  The statement of managers accompanying the Strom Thurmond National Defense Authorization Act of 1999 directed the Secretary of Defense to report on the use of this authority to the congressional defense committees, no later than March 1, 1999  In addition, both the Department of Defense Inspector General and the General Accounting Office are reviewing the use of other transaction prototype authority and will report to Congress in the coming year  The committee is interested in the extent that new commercial firms are entering the DOD marketplace through the use of other transaction authority, as well as the degree of cost sharing between the government and non-federal government parties  The committee is also interested in any lessons learned from the broad exemptions to federal law provided by other transaction authority  For example, other transactions are exempt from the Competition in Contracting Act, Truth in Negotiations Act, Contract Disputes Act, Antikickback Act of 1986, Procurement Integrity Act, Service Contract Act, Buy American Act, and chapter 137 of title 10, United States Code  Questions have been raised about whether the government's interest is adequately protected in the absence of the applicability of these statutes  Conversely, advocates of the view that the government should take advantage of the flexibility of other transactions have championed proposals to extend other transaction authority to production  The committee directs the Secretary of Defense to provide a new report that updates information in the March 1999 report on the use of other transaction prototype authority to the congressional defense committees by February 1, 2000  National Defense Authorization Act for FY2001 ( P L  106-398 ) Limitations on the Use of OTs Section 803 limited the use of OTs for prototype projects to only those circumstances when at least one nontraditional defense contractor significantly participates in the project, one-third of the total cost of the project is paid out of funds provided by parties to the transaction other than the federal government, or the senior procurement executive determines in writing that exceptional circumstances justify use of an OT  Nontraditional defense contractor was defined as an entity that for a period of one year has not entered into or performed "any contract that is subject to full coverage under the cost accounting standards" or "any other contract in excess of $500,000 to carry out prototype projects or to perform basic, applied, or advanced research projects for a Federal agency, that is subject to the Federal Acquisition Regulation '' Section 803 also extended the authority to use OTs for prototypes from September 30, 2001, to September 30, 2004  According to the Senate report, the intent of using OTs for prototypes is to attract companies that typically do not do business with the Department of Defense and encourage cost sharing and experimentation in potentially more efficient ways of doing business with traditional defense contractors  Other transaction authority is an important acquisition tool that can facilitate the incorporation of commercial technology into military weapon systems  In an environment where, in many areas, commercial technology is now more advanced than defense technology, it is imperative that the Department continue to have the flexibility to use innovative contractual instruments that provide access to this technology  There are, however, improvements that can be made in managing and overseeing these contractual arrangements  Section 804 clarified the extent of GAO's access to records in instances where the party in question has only done business with the government in the preceding year through an OT or cooperative agreement  National Defense Authorization Act for FY2002 ( P L  107-107 ) Expanded Authority—Follow-on Production Section 822 of the FY2002 NDAA granted DOD the authority to award a follow-on production contract for prototype projects when at least one-third of the total cost of the prototype project is to be paid out of funds provided by non-federal government sources  Under this authority, such a follow-on contract could be awarded without competition if the prototype project was successfully completed, the number of units in the production contract does not exceed the number of units specified in the underlying prototype agreement, and the price for each unit does not exceed the price specified in the underlying transaction  Department of Defense and Emergency Supplemental Appropriations for Recovery from and Response to Terrorist Attacks on the United States Act, 2002 ( P L  107-117 ) Establishment of Army Venture Capital Initiative (AVCI) Section 8150 designated $25 million of the FY2002 funds made available for Army Research, Development, Test, and Evaluation (RDT&E) to be made available to the Secretary of the Army for the purpose of funding a venture capital investment corporation established pursuant to 10 U S C  2371  A 2014 RAND report stated that OT authorities were used only to form the AVCI, and were not used to acquire products or services: "While OT authorities were used to form [AVCI] itself, any volume of the Army's purchase of products and services from [AVCI] companies is conducted under the FAR " National Defense Authorization Act for FY2004 ( P L  108-136 ) Expanded Authority Section 847 of the FY2004 NDAA extended the authority to use an OT for developing prototypes to improve weapons or weapon systems currently in use by the Armed Forces  Previously, such authority was restricted to prototypes "directly relevant to weapons or weapon systems proposed to be acquired or developed" by DOD  Section 847 also established a pilot program for transitioning prototypes to follow-on contracts for production for nontraditional defense contractors  Under the pilot program, such a follow-on contract could be treated as a commercial item or an item developed with both federal and private sector funds (for purposes of negotiating intellectual property rights)  The pilot program was restricted to contracts with nontraditional defense contractors, where the value of the contract does not exceed $50 million (approximately $70 million in FY2018 dollars), and that are firm-fixed price or fixed price with economic adjustment  The pilot program was set to sunset September 30, 2008  Section 1441 authorized any agency that engages in basic, applied, or advanced research and development projects that facilitated defense against or recovery from terrorism or nuclear, biological, chemical, or radiological attack to exercise the same general authority given to DOD as found in 10 U S C  2371 (including for prototype projects)  Reporting Requirements Section 1031 sunset the annual reporting requirement for OT after the report covering FY2006 was submitted  National Defense Authorization Act for FY2006 ( P L  109-163 ) Restricted Authority and Notification Requirements Section 212 of the FY2006 NDAA directed the Army to procure the Future Combat System using contract procedures set forth in part 15 of the Federal Acquisition Regulation, in lieu of an OT  Section 823 extended ethics requirements to OT prototype authority and required the congressional defense committees be notified in writing at least 30 days before such authority is exercised  Certification Requirements Section 823 also amended Section 845 of the FY1994 NDAA, requiring a written determination by a senior procurement executive for other transaction prototype projects estimated between $20 million and $100 million, and for a written determination by the Under Secretary of Defense for Acquisition, Technology, and Logistics for prototype projects that exceed $100 million  According to the Senate report: Section 845 was intended to be used for limited prototype projects, particularly those in which the Department seeks to engage nontraditional defense contractors that may be averse to the requirements imposed by a standard Department procurement contract  For this reason, the statement of managers accompanying the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999 (Public Law 105–261) states: The conferees continue to believe that the section 845 authority should only be used in the exceptional cases where it can be clearly demonstrated that a normal contract or grant will not allow sufficient access to affordable technologies  The conferees are especially concerned that such authority not be used to circumvent the appropriate management controls in the standard acquisition and budgeting process  …  The committee does not believe that the $20 9 billion agreement entered between the Army and the Lead Systems Integrator for the FCS program is consistent with the language and intent of section 845 authority  Section 845 authority is intended to be used for limited prototype projects, particularly those in which the Department of Defense seeks to engage nontraditional defense contractors that may be averse to the requirements imposed by a standard Department contract  Department of Defense Appropriations Act, 2007 ( H R  5631 ) Reporting Requirement The FY2007 defense appropriations bill ( P L  109-289 ) did not include language addressing OTs  The conference report ( H Rept  109-676 ) included language expressing the conferees' "[concern] with the continued use of OTA contracts by the Missile Defense Agency," as such contracts "lack the customary safeguards found under FAR-based contracts for organizational conflict of interest, truth in negotiations and submission of cost and pricing data " The conferees "strongly encourage[d]" the Missile Defense Agency to convert "large development and procurement contracts using OTA to FAR-based contracts," and directed the Missile Defense Agency to submit a report to the congressional defense committees on the use of OTs, to include the number, value, and justification for the use of such agreements  National Defense Authorization Act for FY2008 ( P L  110-181 ) Section 823 extended the authority for prototype projects for five more years, from September 30, 2008, to September 30, 2013  National Defense Authorization Act for FY2009 ( P L  110-417 ) Section 822 required DOD to issue guidance on rights in technical data under non-FAR agreements, including OTs  Section 822 also required that appropriate provisions relating to rights in technical data be included in non-FAR agreements, consistent with policy guidance  This requirements is in statute at 10 U S C  2320 note  Section 824 expanded the scope of the pilot program for transition to follow-on contracts for certain prototype projects to include research projects carried out under 10 U S C  2371  Authority to use the pilot program, set to expire September 30, 2008, was extended to September 30, 2010  Section 874 required OT data be included in the Federal Procurement Data System  National Defense Authorization Act for FY2011 ( P L  111-383 ) Section 866 changed the definition of nontraditional defense contractor, conforming the definition to that found in 10 U S C  2302(9)  National Defense Authorization Act for FY2013 ( P L  112-239 ) Section 863 extended the authority for using OTs for prototype projects from September 30, 2013, to September 30, 2018  Carl Levin and Howard P  "Buck" McKeon National Defense Authorization Act for FY2015 ( P L  113-291 ) Expanded Authority Section 812 expanded the authority to use OT for prototypes, to include those "directly related to enhancing the mission effectiveness of military personnel and the supporting platforms, systems, components, or materials proposed to be acquired or developed by the Department of Defense, or to improvement of platforms, systems, components, or materials in use by the Armed Forces " Prior to the FY2015 NDAA, OTs could only be used for prototypes relating to weapons or weapon systems proposed to be developed, or for the improvement of weapons or weapon systems currently in use  Reporting Requirements Section 1071 repealed the reporting requirement language found in 10 U S C  2371, relating to OTs for research projects  National Defense Authorization Act for FY2016 ( P L  114-92 ) Until the FY2016 NDAA, the prototyping and follow-on production authority established in Section 845 of the FY1994 NDAA (as amended) was found in 10 U S C  2371 note  Section 815 of the FY2016 NDAA simultaneously repealed Section 845 of the FY1994 NDAA and put the repealed language into the newly created 10 U S C  2371b  The FY2016 NDAA also modified 2371b by making the authority permanent  Expanded Authority and Small Business The authorities in Section 2371b were expanded to allow their use when "all significant participants in the transaction other than the Federal government are small businesses or nontraditional contractors" and when the agency determines that using an OT would expand the defense supply base in a manner that could not be accomplished through a contract  Section 815 eased the restriction on follow-on production contracts or transactions  Section 815 amended the definition of a nontraditional defense contractor found in 10 U S C  2302 to be an entity that is not currently performing, and for one year prior to an OT has not performed on any contract or subcontract that is subject to full coverage under the cost accounting standards pursuant to Section 1502 of Title 41, U S C  Section 815 also required DOD to update its guidance to reflect changes in the statute  The conference report stated that Congress believed OTs are an attractive option for firms and organizations that do not usually participate in government contracting due to typical overhead burdens and the "one size fits all" rules governing defense acquisition  The report also stated that OTs could support DOD's effort to access new sources of technological innovation, specifically with Silicon Valley startup firms and small commercial firms  National Defense Authorization Act for FY2018 ( P L  115-91 ) Expanded Authority Section 216 of the FY2018 NDAA authorized nonprofit research institutions to enter into OTs with DOD for prototype projects  Section 862 amended 10 U S C  2358, granting the Secretary of Defense and the military departments the authority to pursue basic research, applied research, advanced research, and development projects under the OT authorities granted in Sections 2371 and 2371b of Title 10  Workforce Section 802 required DOD to establish a cadre of intellectual property experts to advise, assist, and provide resources to program offices who are developing intellectual property strategies for contracts and agreements  Section 863 required training and education for personnel involved in OTs and other innovative contracting methods  Certification Requirements and Small Business Section 864 adjusted the language of the statute to state that the dollar threshold relates to the specific transaction for a prototype project and not for the value of the entire project  Section 864 defined a transaction for follow-on production to include "all individual prototype sub-projects awarded under the transaction to a consortium of United States industry and academic institutions " Section 864 also increased the dollar thresholds for required approvals and defined the term small business to include small businesses under Section 9 of the Small Business Act to ensure that companies participating in the Small Business Innovation Research and Small Business Technology Transfer programs were considered small businesses for the purposes of the cost-sharing requirements  Miscellaneous Section 867 required the Secretary of Defense to establish a preference for OTs in the "execution of science and technology and prototyping programs " Section 1711 required DOD to carry out a pilot program to "assess the feasibility and advisability of increasing the manufacturing capability of the defense industrial base " Pursuant to the pilot, Section 1711 authorized DOD to use OTs to support production capabilities in small and medium-sized manufacturers  John S  McCain National Defense Authorization Act for FY2019 ( P L  115-232 ) Section 211 of the FY2019 NDAA clarified that follow-on production of a prototype or subproject within a consortium may occur if the individual prototype or subproject is complete; all projects associated with the consortium do not need to be completed before follow-on production of a specific prototype  Expanded Authority The FY2019 NDAA authorized the use of OTs to develop enhanced personal protective equipment (Section 226) and to carry out research under the Explosive Ordnance Disposal Defense Program (Section 311)  Reporting Requirements Section 244 required the Defense Innovation Unit to submit a report to Congress, to include the number of traditional and nontraditional defense contractors with DOD contracts or other transactions resulting directly from the unit's initiatives  Section 873 required DOD to submit an annual report through 2021, summarizing DOD's use of OTs, including organizations involved; number of transactions; amounts of payments; and purpose, description, and status of projects  Department of Defense Appropriations Act, 2019 ( P L  115-245 ) Reporting Requirements While the enacted FY2019 defense appropriations bill did not include legislative language addressing OTs, the conference report included language expressing the conferees' "[concern] with the lack of transparency surrounding the employment of OTA, particularly for follow-on production " The conferees directed DOD to provide quarterly reports to the House and Senate appropriations committees listing each active OT, and to include the following information on each agreement: funding military service or DOD component; major command (if applicable); contracting activity; appropriation title; budget line item; minimum and maximum award value; vendor; obligations and expenditures to date; product service code; period of performance; and indication if the OT agreement included an option for follow-on production (with a description of the scope of anticipated follow-on production)  The conferees also directed GAO to review DOD's use of OTs to determine whether the "employment of this authority conforms to applicable statutes and guidelines, to include the identification of any potential conflicts " GAO was also required to report on the extent to which OTs have been used since FY2016  Notification Requirements The House report to accompany H R  6157 acknowledged OTs as an "important tool to provide flexibility and agility for cutting-edge research and development projects and prototypes " However, the report stated its concern "with the lack of transparency on the use of OTA authority for follow-on production procurements," and directed that no funds could be obligated or expended for a follow-on production contract or a transaction carried out under 10 U S C  2371b, until 30 days after the Secretary of Defense provides the congressional defense committees with a notification of the proposed contract or transaction, to include a justification of why an OT is being used for production  Appendix B  Non-DOD Federal Agencies with Agency-Wide OT or Related Authorities A number of agencies have varying other transaction or similar authorities, as reflected in Table B-1   The table below is not a comprehensive or definitive listing of every federal government entity with OT or related authorities  In some instances, offices, agencies, commissions, and other federal government entities have OT or related authorities that are only associated with certain programs or projects, such as the National Institutes of Health (which has OT authority for such specific activities such as the National Heart, Blood Vessel, Lung, and Blood Diseases and Blood Resources Program [42 U S C  §285b-3] and the Cures Acceleration Network [42 U S C  §287a])  Appendix C  Reliability of Data on Other Transactions All data have imperfections and limitations  FPDS-NG data can be used to identify broad trends and produce rough estimates, or to gather information about specific contracts  Some observers say that despite their shortcomings, FPDS-NG data are substantially more comprehensive than what is available in most other countries in the world  Understanding the limitations of government procurement data—including knowing when, how, and to what extent to rely on data—can help policymakers incorporate FPDS-NG data more effectively into their decisionmaking process  FPDS-NG OT Data Quality and Accuracy Issues Decisionmakers should be cautious when using data from FPDS-NG to develop policy or otherwise draw conclusions, especially with respect to OTs  In some cases, the data themselves may not be reliable  In other instances, a query for particular data may return differing results, depending on the parameters and timing of the analysis  In particular, all DOD data entered into FPDS-NG are subject to a 90-day delay, and updates to "data, including new actions, modifications, and corrections are made on a regular basis," which could result in changes to "data     for current and/or prior fiscal years " Inconsistencies in FPDS-NG Data Within FPDS-NG, two primary collections of obligation data exist: one associated with standard government procurement contracts or modifications to such contracts, and the other associated with prototype OT agreements or modifications to such agreements  FPDS-NG's collection of prototype OT data allows for the input of additional data elements—such as nongovernment dollars associated with cost-share prototype OT agreements—not included in FPDS-NG's collection of standard government procurement contract data  More consequentially, FPDS-NG's prototype OT data include two similar data elements that allow users to identify the fiscal year a prototype OT agreement was signed or modified  One, labeled in the database as "Fiscal Year," appears to allow users entering data into the system to manually assign a fiscal year to a transaction  FPDS-NG users entering data into the system appear to have interpreted this data element in various, conflicting ways  For example, a Department of the Air Force OT agreement was signed in February 2016 for the development of rocket propulsion system prototypes under the Evolved Expendable Launch Vehicle (EELV) program  FPDS-NG records an obligation of $115 million in FY2020 for this agreement  Fiscal law bars DOD from obligating money now for future fiscal years that have not yet occurred  The second, labeled "Contract Fiscal Year," appears to be based on the date the prototype OT agreement was signed or modified  See Table C-1 for a comparison of the "Fiscal Year" and "Contract Fiscal Year" elements for selected new prototype OT agreements signed between FY2013 and FY2017  If a user selects the Fiscal Year data element when attempting to review high-level data on recent trends in the use of prototype OT agreements within DOD—such as the total amount obligated for prototype OT agreements on an annual basis—that user will obtain a substantially different result than if he or she selects the Contract Fiscal Year data element for a similar analysis  See Table C-2 for a comparison of action obligations and nongovernment contributions using the Fiscal Year and Contract Fiscal Year data elements for new prototype OT agreements signed between FY2013 and FY2017  DOD OT Data Analysis Methodological Issues in Congressional Reports A March 2017 report to Congress entitled "An Assessment of Cost-Sharing in Other Transaction Agreements for Prototype Projects," completed by the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics indicated that In FY2016, DOD obligated (Note: Two outliers in 2016 excluded) $1 billion in section 2371b awards and received $68 million in cost-share contributions after excluding two significant trend outliers  DOD cited FPDS-NG as the source for its analysis, and defined the "significant trend outliers" excluded as "two $40 million OTAs with total cost-share of $270 million," likely referring to two DARPA prototype OT agreements conducted on a cost-share basis initiated in FY2016  However, a CRS analysis of the same FPDS-NG data identified numerous inconsistencies in DOD's methodological approach  Recreation of DOD Methodology Specifically, DOD appears to have conducted its analysis using the "Fiscal Year" data element referenced in this report's discussion of FPDS-NG OT data quality and accuracy issues, which would attribute some prototype OT activities to the wrong fiscal year for the purposes of comparative trend analysis  DOD also compared two disparate transaction types: the reported "$1 billion in Section 2371b awards" includes all action obligations associated with ongoing prototype OT activities in FY2016, including transactions associated with prototype OT indefinite delivery contracts, while the "$68 million in cost-share contributions" includes only cost-share contributions associated with new prototype OT agreements  If DOD used the "Contract Fiscal Year" data element, excluded the identified "trend outliers," and focused on all action obligations and cost-share contributions associated with ongoing prototype OT activities, it would have found instead that the department obligated $1 4 billion for prototype OT agreements in FY2016, with an additional $313 5 million in cost-share contributions from the private sector  On the other hand, if DOD used the "Contract Fiscal Year" data element, excluded the identified "trend outliers," and focused on only action obligations and cost-share contributions associated with new prototype OT agreements, it would have found instead that the department obligated $400 million for prototype OT agreements in FY2016, with an additional $272 1 million in cost-share contributions from the private sector  Appendix D  Other Transaction Authority Statutes 10 U S C  §2371  Research projects: transactions other than contracts and grants (a) ADDITIONAL FORMS OF TRANSACTIONS AUTHORIZED — The Secretary of Defense and the Secretary of each military department may enter into transactions (other than contracts, cooperative agreements, and grants) under the authority of this subsection in carrying out basic, applied, and advanced research projects  The authority under this subsection is in addition to the authority provided in Section 2358 of this title to use contracts, cooperative agreements, and grants in carrying out such projects  (b) EXERCISE OF AUTHORITY BY SECRETARY OF DEFENSE — In any exercise of the authority in subsection (a), the Secretary of Defense shall act through the Defense Advanced Research Projects Agency or any other element of the Department of Defense that the Secretary may designate  (c) ADVANCE PAYMENTS — The authority provided under subsection (a) may be exercised without regard to Section 3324 of Title 31  (d) RECOVERY OF FUNDS — (1) A cooperative agreement for performance of basic, applied, or advanced research authorized by Section 2358 of this title and a transaction authorized by subsection (a) may include a clause that requires a person or other entity to make payments to the Department of Defense or any other department or agency of the Federal Government as a condition for receiving support under the agreement or other transaction  (2) The amount of any payment received by the Federal Government pursuant to a requirement imposed under paragraph (1) may be credited, to the extent authorized by the Secretary of Defense, to the appropriate account established under subsection (f)  Amounts so credited shall be merged with other funds in the account and shall be available for the same purposes and the same period for which other funds in such account are available  (e) CONDITIONS — (1) The Secretary of Defense shall ensure that- (A) to the maximum extent practicable, no cooperative agreement containing a clause under subsection (d) and no transaction entered into under subsection (a) provides for research that duplicates research being conducted under existing programs carried out by the Department of Defense; and (B) to the extent that the Secretary determines practicable, the funds provided by the Government under a cooperative agreement containing a clause under subsection (d) or a transaction authorized by subsection (a) do not exceed the total amount provided by other parties to the cooperative agreement or other transaction  (2) A cooperative agreement containing a clause under subsection (d) or a transaction authorized by subsection (a) may be used for a research project when the use of a standard contract, grant, or cooperative agreement for such project is not feasible or appropriate  (f) SUPPORT ACCOUNTS — There is hereby established on the books of the Treasury separate accounts for each of the military departments and the Defense Advanced Research Projects Agency for support of research projects and development projects provided for in cooperative agreements containing a clause under subsection (d) and research projects provided for in transactions entered into under subsection (a)  Funds in those accounts shall be available for the payment of such support  (g) EDUCATION AND TRAINING —The Secretary of Defense shall— (1) ensure that management, technical, and contracting personnel of the Department of Defense involved in the award or administration of transactions under this section or other innovative forms of contracting are afforded opportunities for adequate education and training; and (2) establish minimum levels and requirements for continuous and experiential learning for such personnel, including levels and requirements for acquisition certification programs  (h) REGULATIONS — The Secretary of Defense shall prescribe regulations to carry out this section  (i) Protection of Certain Information From Disclosure -(1) Disclosure of information described in paragraph (2) is not required, and may not be compelled, under Section 552 of Title 5 for five years after the date on which the information is received by the Department of Defense  (2)(A) Paragraph (1) applies to information described in subparagraph (B) that is in the records of the Department of Defense if the information was submitted to the Department in a competitive or noncompetitive process having the potential for resulting in an award, to the party submitting the information, of a cooperative agreement for performance of basic, applied, or advanced research authorized by Section 2358 of this title or another transaction authorized by subsection (a)  (B) The information referred to in subparagraph (A) is the following: (i) A proposal, proposal abstract, and supporting documents  (ii) A business plan submitted on a confidential basis  (iii) Technical information submitted on a confidential basis  10 U S C  §2371b  Authority of the Department of Defense to carry out certain prototype projects (a) AUTHORITY — (1) Subject to paragraph (2), the Director of the Defense Advanced Research Projects Agency, the Secretary of a military department, or any other official designated by the Secretary of Defense may, under the authority of Section 2371 of this title, carry out prototype projects that are directly relevant to enhancing the mission effectiveness of military personnel and the supporting platforms, systems, components, or materials proposed to be acquired or developed by the Department of Defense, or to improvement of platforms, systems, components, or materials in use by the armed forces  (2) The authority of this section- (A) may be exercised for a transaction (for a prototype project) that is expected to cost the Department of Defense in excess of $100,000,000 but not in excess of $500,000,000 (including all options) only upon a written determination by the senior procurement executive for the agency as designated for the purpose of Section 1702(c) of Title 41, or, for the Defense Advanced Research Projects Agency or the Missile Defense Agency, the director of the agency that- (i) the requirements of subsection (d) will be met; an d (ii) the use of the authority of this section is essential to promoting the success of the prototype project; and (B) may be exercised for a transaction (for a prototype project) that is expected to cost the Department of Defense in excess of $500,000,000 (including all options) only if- (i) the Under Secretary of Defense for Acquisition, Technology, and Logistics determines in writing that- (I) the requirements of subsection (d) will be met; an d (II) the use of the authority of this section is essential to meet critical national security objectives; and (ii) the congressional defense committees are notified in writing at least 30 days before such authority is exercised  (3) The authority of a senior procurement executive or director of the Defense Advanced Research Projects Agency or Missile Defense Agency under paragraph (2)(A), and the authority of the Under Secretary of Defense for Acquisition, Technology, and Logistics under paragraph (2)(B), may not be delegated  (b) EXERCISE OF AUTHORITY — (1) Subsections (e)(1)(B) and (e)(2) of such Section 2371 shall not apply to projects carried out under subsection (a)  (2) To the maximum extent practicable, competitive procedures shall be used when entering into agreements to carry out projects under subsection (a)  (c) COMPTROLLER GENERAL ACCESS TO INFORMATION — (1) Each agreement entered into by an official referred to in subsection (a) to carry out a project under that subsection that provides for payments in a total amount in excess of $5,000,000 shall include a clause that provides for the Comptroller General, in the discretion of the Comptroller General, to examine the records of any party to the agreement or any entity that participates in the performance of the agreement  (2) The requirement in paragraph (1) shall not apply with respect to a party or entity, or a subordinate element of a party or entity, that has not entered into any other agreement that provides for audit access by a Government entity in the year prior to the date of the agreement  (3) (A) The right provided to the Comptroller General in a clause of an agreement under paragraph (1) is limited as provided in subparagraph (B) in the case of a party to the agreement, an entity that participates in the performance of the agreement, or a subordinate element of that party or entity if the only agreements or other transactions that the party, entity, or subordinate element entered into with Government entities in the year prior to the date of that agreement are cooperative agreements or transactions that were entered into under this section or Section 2371 of this title  (B) The only records of a party, other entity, or subordinate element referred to in subparagraph (A) that the Comptroller General may examine in the exercise of the right referred to in that subparagraph are records of the same type as the records that the Government has had the right to examine under the audit access clauses of the previous agreements or transactions referred to in such subparagraph that were entered into by that particular party, entity, or subordinate element  (4) The head of the contracting activity that is carrying out the agreement may waive the applicability of the requirement in paragraph (1) to the agreement if the head of the contracting activity determines that it would not be in the public interest to apply the requirement to the agreement  The waiver shall be effective with respect to the agreement only if the head of the contracting activity transmits a notification of the waiver to Congress and the Comptroller General before entering into the agreement  The notification shall include the rationale for the determination  (5) The Comptroller General may not examine records pursuant to a clause included in an agreement under paragraph (1) more than three years after the final payment is made by the United States under the agreement  (d) APPROPRIATE USE OF AUTHORITY — (1) The Secretary of Defense shall ensure that no official of an agency enters into a transaction (other than a contract, grant, or cooperative agreement) for a prototype project under the authority of this section unless one of the following conditions is met: (A) There is at least one nontraditional defense contractor or nonprofit research institution participating to a significant extent in the prototype project  (B) All significant participants in the transaction other than the Federal Government are small businesses (including small businesses participating in a program described under Section 9 of the Small Business Act (15 U S C  638)) or nontraditional defense contractors  (C) At least one third of the total cost of the prototype project is to be paid out of funds provided by sources other than other than the Federal Government  (D) The senior procurement executive for the agency determines in writing that exceptional circumstances justify the use of a transaction that provides for innovative business arrangements or structures that would not be feasible or appropriate under a contract, or would provide an opportunity to expand the defense supply base in a manner that would not be practical or feasible under a contract  (2) (A) Except as provided in subparagraph (B), the amounts counted for the purposes of this subsection as being provided, or to be provided, by a party to a transaction with respect to a prototype project that is entered into under this section other than the Federal Government do not include costs that were incurred before the date on which the transaction becomes effective  (B) Costs that were incurred for a prototype project by a party after the beginning of negotiations resulting in a transaction (other than a contract, grant, or cooperative agreement) with respect to the project before the date on which the transaction becomes effective may be counted for purposes of this subsection as being provided, or to be provided, by the party to the transaction if and to the extent that the official responsible for entering into the transaction determines in writing that- (i) the party incurred the costs in anticipation of entering into the transaction; and (ii) it was appropriate for the party to incur the costs before the transaction became effective in order to ensure the successful implementation of the transaction  (e) Definitions —In this section: (1) The term "nontraditional defense contractor" has the meaning given the term under Section 2302(9) of this title  (2) The term "small business" means a small business concern as defined under Section 3 of the Small Business Act (15 U S C  632)  (f) FOLLOW-ON PRODUCTION CONTRACTS OR TRANSACTIONS — (1) A transaction entered into under this section for a prototype project may provide for the award of a follow-on production contract or transaction to the participants in the transaction  A transaction includes all individual prototype subprojects awarded under the transaction to a consortium of United States industry and academic institutions  (2) A follow-on production contract or transaction provided for in a transaction under paragraph (1) may be awarded to the participants in the transaction without the use of competitive procedures, notwithstanding the requirements of Section 2304 of this title, if - (A) competitive procedures were used for the selection of parties for participation in the transaction; and (B) the participants in the transaction successfully completed the prototype project provided for in the transaction  (3) Contracts and transactions entered into pursuant to this subsection may be awarded using the authority in subsection (a), under the authority of Chapter 137 of this title, or under such procedures, terms, and conditions as the Secretary of Defense may establish by regulation  (g) AUTHORITY TO PROVIDE PROTOTYPES AND FOLLOW-ON PRODUCTION ITEMS AS GOVERNMENT-FURNISHED EQUIPMENT — An agreement entered into pursuant to the authority of subsection (a) or a follow-on contract or transaction entered into pursuant to the authority of subsection (f) may provide for prototypes or follow-on production items to be provided to another contractor as Government-furnished equipment  (h) APPLICABILITY OF PROCUREMENT ETHICS REQUIREMENTS — An agreement entered into under the authority of this section shall be treated as a Federal agency procurement for the purposes of Chapter 21 of Title 41  10 U S C  §2373  Procurement for experimental purposes (a) AUTHORITY — The Secretary of Defense and the Secretaries of the military departments may each buy ordnance, signal, chemical activity, transportation, energy, medical, space-flight, and aeronautical supplies, including parts and accessories, and designs thereof, that the Secretary of Defense or the Secretary concerned considers necessary for experimental or test purposes in the development of the best supplies that are needed for the national defense  (b) PROCEDURES — Purchases under this section may be made inside or outside the United States and by contract or otherwise  Chapter 137 of this title applies only when such purchases are made in quantities greater than necessary for experimentation, technical evaluation, assessment of operational utility, or safety or to provide a residual operational capability 

ground_truth: The improper payment rate is a sentinel measure of program integrity risks for the Medicaid program. CMS and the states oversee Medicaid, whose size, structure, and diversity make it vulnerable to improper payments. CMS estimates the Medicaid improper payment rate annually through its PERM, which includes an estimate for Medicaid managed care, in which states contract with MCOs to provide services to Medicaid enrollees. GAO was asked to study the PERM methodology for managed care. In this report, GAO examined the extent to which the PERM accounts for program integrity risks in Medicaid managed care, including CMS's and states' oversight. GAO identified program integrity risks reported in 27 federal and state audits and investigations issued between January 2012 and September 2017; reviewed federal regulations and guidance on the PERM and CMS's Focused Program Integrity Reviews; and contacted program integrity officials in the 16 states with a majority of 2016 Medicaid spending for managed care, as well as CMS officials and program integrity experts. The Centers for Medicare & Medicaid Services' (CMS) estimate of improper payments for Medicaid managed care has limitations that are not mitigated by the agency's and states' current oversight efforts. One component of the Payment Error Rate Measurement (PERM) measures the accuracy of capitated payments, which are periodic payments that state Medicaid agencies make to managed care organizations (MCO) to provide services to enrollees and to cover other allowable costs, such as administrative expenses. However, the managed care component of the PERM neither includes a medical review of services delivered to enrollees, nor reviews of MCO records or data. Further, GAO's review of the 27 federal and state audits and investigations identified key program risks. Ten of the 27 federal and state audits and investigations identified about $68 million in overpayments and unallowable MCO costs that were not accounted for by PERM estimates; another of these investigations resulted in a $137.5 million settlement. These audits and investigations were conducted over more than 5 years and involved a small fraction of the more than 270 MCOs operating nationwide as of September 2017. To the extent that overpayments and unallowable costs are unidentified and not removed from the cost data used to set capitation rates, they may allow inflated MCO payments and minimize the appearance of program risks in Medicaid managed care. CMS and states have taken steps to improve oversight of Medicaid managed care through updated regulations, focused reviews of states' managed care programs, and federal program integrity contractors' audits of managed care services. However, some of these efforts went into effect only recently, and others are unlikely to address the risks in managed care across all states. Furthermore, these efforts do not ensure the identification and reporting of overpayments to providers and unallowable costs by MCOs. Federal internal control standards call for agency management to identify and respond to risks. Without addressing key risks, such as the extent of overpayments and unallowable costs, CMS cannot be certain that its estimated improper payment rate for managed care (0.3 percent compared with 12.9 percent in Medicaid fee-for-service) accurately reflects program risks. The Administrator of CMS should consider and take steps to mitigate the program risks that are not measured in the PERM, such as overpayments and unallowable costs; such an effort could include actions such as revising the PERM methodology or focusing additional audit resources on managed care. HHS concurred with this recommendation. HHS also provided technical comments, which were incorporated as appropriate.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: In accordance with the Improper Payments Information Act of 2002 (IPIA), as amended, and OMB guidance, CMS developed the PERM to estimate the national Medicaid improper payment rate  CMS has other mechanisms to review and assess program integrity risks in state Medicaid managed care programs, and it uses information from the PERM to target its program integrity activities and oversight of states’ Medicaid programs  IPIA requires federal executive branch agencies to, among other things, (1) identify programs and activities that may be susceptible to significant improper payments; and (2), on an annual basis, estimate the amount of improper payments for susceptible programs and activities  Agency heads must produce a statistically valid estimate or an estimate that is otherwise appropriate, using an OMB-approved alternate methodology  Those agencies with programs identified by OMB as being high priority for additional oversight and review are required to submit annual reports to their Inspectors General detailing the actions the agency plans to take to recover improper payments and prevent future improper payments  The Inspector General of each agency submitting such a report is required to review the quality of the improper payment estimates and methodology, among other things  OMB designated Medicaid as a high priority program  In addition, the Improper Payments Elimination and Recovery Act of 2010 requires the Inspector General of each agency to conduct a compliance review to report on the agency’s compliance with several criteria, one of which is that an agency has reported an improper payment rate of less than 10 percent for each program and activity  IPIA also directed OMB to issue guidance for agencies in implementing the IPIA improper payments requirements  Among other things, the OMB guidance requires that agencies review payments made at the point that federal funds are transferred to nonfederal entities and report on the root causes of identified improper payments  To calculate the Medicaid improper payment rate through the PERM, CMS computes an annual rolling average of improper payment rates across all states based on a 17-state, 3-year rotation cycle  In accordance with IPIA, as amended, OMB approved CMS’s PERM methodology, and the HHS-OIG conducts annual compliance reviews  Beginning with its annual improper payment compliance review for fiscal year 2014, the HHS-OIG established a rotating approach to reviewing the estimation methodology for high-priority programs, including Medicaid, that OMB deemed susceptible to improper payments  Due to the number and complexity of the programs, the HHS-OIG methodology reviews are scheduled to be performed over a 4-year period; the PERM estimation methodology will be reviewed as a part of its fiscal year 2017 compliance review  Each of the three components of the Medicaid PERM—FFS, managed care, and eligibility—is estimated differently: The FFS component of the PERM measures errors in a sample of FFS claims, which are records of services provided and the amount the Medicaid program paid for these services  For the majority of sampled FFS claims, the PERM review contractor performs a medical review, which includes a review of the medical documentation to determine errors that do not meet federal and state policies, such as medically unnecessary services, diagnosis coding errors, and policy violations  Any FFS claims that were paid for services that should have been covered under a managed care plan’s capitated payment are also considered errors  The managed care component of the PERM measures errors that occur in the capitated payments that state Medicaid agencies make to managed care organizations (MCO) on behalf of enrollees  Capitated payments are periodic payments approved by CMS that state Medicaid agencies make to contracted MCOs to cover the provision of medical services to enrollees, as well as the MCOs’ administrative expenses and their profits or earnings  The PERM assesses whether any payments made to the MCOs were in amounts different than those the state agency is contractually required to pay, which are approved by CMS  In contrast to the FFS component, the managed care component of the PERM neither includes a medical review of services delivered to enrollees, nor reviews of MCO records or data  The eligibility component of the PERM measures errors in state determinations of whether enrollees meet categorical and financial criteria for receipt of benefits under the Medicaid program  The eligibility component assesses determinations for both FFS and managed care enrollees  This component has not been calculated since 2014; instead, CMS piloted different approaches to update the methodologies used to assess enrollee eligibility, as the Patient Protection and Affordable Care Act changed income eligibility requirements for nonelderly, nonpregnant individuals who qualify for Medicaid  Beginning in the 2019 reporting year, eligibility reviews under the PERM will resume and will be conducted by a federal contractor  Medicaid program integrity consists of efforts to ensure that federal and state expenditures are used to deliver quality, necessary care to eligible enrollees, and efforts to prevent fraud, waste, and abuse  We have found in prior work that CMS’s and states’ program integrity efforts focused primarily on payments and services delivered under FFS and did not closely examine program integrity in Medicaid managed care  For Medicaid managed care, CMS has largely delegated program integrity oversight of MCOs to the states  States, in turn, generally oversee MCOs and the providers under contract to MCOs through their contracts with the MCOs and reporting requirements  Some program integrity risks for managed care are similar to those in FFS, including payments made for nonenrolled, ineligible, or deceased individuals; payments to ineligible, excluded, or deceased providers; and payments to providers for improper or false claims, such as payments for services that are not medically necessary  Other program integrity risks are more unique to managed care  For example, capitated payments generally reflect the average cost to provide covered services to enrollees, rather than a specific service  Federal law requires capitation rates to be actuarially sound, meaning that, among other things, they must be reasonably calculated for the populations expected to be covered and for the services expected to be furnished under contract  In order to receive federal funds for its managed care program, a state is required to submit the rates it pays MCOs and the methodology it uses to set those rates to CMS for review and approval  Additionally, federal and state oversight of Medicaid managed care can include ensuring that MCOs fulfill contractual provisions within their managed care contracts  In some cases, these provisions relate directly to program integrity activities, including plans and procedures for identifying, recovering, and reporting on overpayments made to providers  The managed care component of the PERM measures the accuracy of the capitated payments state Medicaid agencies make to MCOs  Specifically, a CMS contractor examines whether the state agency made capitated payments only for eligible enrollees, made capitated payments for the correct amount based on the contract and coverage requirements (time period and geographic location), made capitated payments based on the correct rate for enrollees, and did not make any duplicate payments for enrollees  CMS officials noted that the agency established capitated payments as the level of review, because the capitation rate is the transaction used to determine the federal match in managed care  In general, the federal government matches most state expenditures for Medicaid services on the basis of a statutory formula  In FFS, the federal match is provided for the amount the state pays a health care provider for delivering services to enrollees  With managed care, the federal match is provided for the amount of the capitation rate the state pays the MCO  Capitated payments do not directly relate to the provision of a specific service, but reflect the average cost to provide covered services to enrollees  As a result, CMS officials maintain that the capitated payment is the lowest transaction level at which the agency can clearly identify federal funds without making significant assumptions  Because the managed care component of the PERM review is limited to measuring capitated payments, it does not account for other program integrity risks—such as overpayments to providers and unallowable MCO costs  In addition to errors in capitated payments included in PERM reviews, CMS regulations state that overpayments in managed care include any payment made to an MCO or provider under contract to an MCO to which the MCO or provider is not entitled under Medicaid  Such overpayments included payments for services that were not provided or medically necessary; or to ineligible, excluded, or deceased providers, which are not measured by the PERM  Unallowable MCO costs refers to operating costs that MCOs cannot claim under their managed care contracts, such as certain marketing costs, or that the MCO reported incorrectly  Among the 27 audits and investigations of Medicaid managed care programs we reviewed, 10 identified about $68 million in MCO overpayments to providers and unallowable MCO costs that were not accounted for in PERM estimates  In addition, one investigation of an MCO operating in nine states resulted in a $137 5 million settlement to resolve allegations of false claims  (See app  I for a complete list of the audits and investigations we identified ) However, the full extent of these overpayments and unallowable costs is unknown, because these audits and investigations were conducted over more than 5 years and involved a small fraction of the more than 270 MCOs operating nationwide as of September 2017  Specifically, 24 of the audits and investigations represented reviews in 10 states and, in many cases, focused on individual providers or MCOs; there were about 90 MCOs operating in the 10 states as of September 2017, according to the Kaiser Family Foundation  Some examples of the audits and investigations that identified overpayments and unallowable costs include the following: The Washington State Auditor’s Office found that two MCOs made $17 5 million in overpayments to providers in 2010, which may have increased the state’s 2013 capitation rates  The New York State Comptroller found that two MCOs paid over $6 6 million to excluded and deceased providers from 2011 through 2014  The Massachusetts State Auditor found that one MCO paid $420,000 for health care services and unauthorized prescriptions from excluded providers in 2013 and 2014  The Department of Justice alleged that an MCO operating in several states submitted inflated expenditure information to the state Medicaid agencies, falsified encounter data, and manipulated claims costs and service provision costs in nine states  The MCO agreed to pay over $137 5 million to resolve these claims  The Texas State Auditor’s Office found that an MCO reported $3 8 million in unallowable costs for advertising, company events, gifts, and stock options, along with $34 million in other questionable costs in 2015  The New York State Comptroller also found that an MCO claimed over $260,000 in unallowable administrative expenses, which contributed to an increase in capitation rates across the state  To the extent that the state does not identify or know of MCO overpayments to providers or unallowable MCO costs, the overpayments and unallowable costs could inflate future capitation rates, as the Washington State Auditor and New York State Comptroller noted in their findings  The PERM assesses the accuracy of capitated payments that states make to MCOs  States set capitation rates based on cost data— historical utilization and spending—that MCOs submit to the state Medicaid agencies, but the PERM does not consider these data  Unless removed from these cost data, unidentified overpayments and unallowable costs would likely inflate the MCO cost data that states use to set capitation rates  (See fig  1 ) As a result, future capitation rates would also be inflated, resulting in higher state and federal spending  In fiscal year 2017, the Medicaid managed care improper payment rate was 0 3 percent, while the FFS improper payment rate was 12 9 percent, leading to an assumption that the estimated risks in managed care are less significant than those estimated in FFS  However, the managed care component of the PERM does not determine whether MCO payments to providers were for services that were medically necessary, actually provided, accurately billed and delivered by eligible providers, or whether the MCO costs were allowable and appropriate  As a result, the PERM improper payment estimate potentially understates the extent of program integrity risks in Medicaid managed care  Moreover, this potential understatement in the PERM’s improper payment rate estimate may curtail investigations into the appropriateness of MCO spending  We previously reported that CMS and state program integrity efforts did not closely examine program integrity in Medicaid managed care, focusing primarily on payments and services delivered under FFS  Our current review of the 27 audits we identified encompassed a 5-year period, suggesting that reviews of managed care continue to be limited  An official from a state auditor’s office we spoke with suggested that some states may not audit services delivered under managed care, because of a low improper payment rate  In addition, he noted that his state Medicaid agency used the relatively low payment error rate in managed care as an indicator of few program integrity problems  As noted, CMS has increased its focus on and worked with states to improve oversight of Medicaid managed care; however, these efforts and the oversight efforts of states do not ensure the identification and reporting of overpayments and unallowable costs  In recent years, the agency has sought to strengthen oversight of managed care programs through updated regulations; reviews of states’ managed care programs (Focused Program Integrity Reviews) and collaborative audits, which are conducted jointly by federal program integrity contractors and states; and state monitoring of overpayments  Regulations  In May 2016, CMS updated its regulations for managed care programs in order to strengthen oversight  The updated regulations require a number of additional program integrity activities, such as those listed below  If fully implemented, these updated regulations may help with the identification and removal of overpayments and unallowable costs from data used to set future capitation rates  Under these regulations States must arrange for an independent audit of the accuracy, truthfulness, and completeness of the encounter and financial data submitted by MCOs, at least once every 3 years  Through contracts with MCOs, states must require MCOs to have a mechanism through which providers report and return overpayments to the MCOs  States must also require MCOs to promptly report any identified or recovered overpayments—specifying those that are potentially fraudulent—and submit an annual report on recovered overpayments to their state  States must use this information when setting actuarially sound capitation rates  Through contracts with MCOs, states must also require MCOs to report specific data, information, and documentation  In addition, the MCO’s chief executive officer or authorized representative must certify the accuracy and completeness of the reported data, information, and documentation  States must enroll MCO providers that are not otherwise enrolled with the state to provide services to enrollees in Medicaid FFS, and revalidate the enrollment at least once every 5 years  Initially this requirement was to start for MCO contracts beginning on July 1, 2018  Subsequently enacted legislation codified this requirement in statute and moved the implementation to January 1, 2018  It is too early to know if these regulations will assure better oversight of MCO payments to providers and the data used to set future capitation rates  The above program integrity requirements only went into effect recently—for contracts starting on or after July 1, 2017, and January 1, 2018  In addition, CMS issued a notice in June 2017 stating that the agency will use its enforcement discretion to assist states that are unable to implement new requirements by the required compliance date  Also, CMS has delayed issuance of implementing guidance for certain provisions until the agency completes its review, a step that may further delay states’ implementation  The agency has designated Medicaid managed care for “deregulatory action” and plans to propose a new rule, but has not indicated which of these provisions, if any, would be revised  Focused Program Integrity Reviews  In fiscal year 2014, CMS implemented its Focused Program Integrity Reviews in order to target high-risk program integrity areas in each state, including managed care  As we previously reported, these focused reviews are narrower in scope than the prior reviews conducted by CMS, but they still involve on-site visits to states  In its focused reviews of managed care, CMS found that several states had incomplete oversight of MCO payments to providers, even though the agency relies on states to verify reported MCO overpayments and to ensure the overpayments are excluded from the data used to set capitation rates  In the 27 focused reviews of managed care from 2014 to 2017, CMS found that MCOs in 17 states reported fewer overpayments to their state Medicaid agencies than CMS would expect  For example, MCOs in at least 5 states reported that overpayments were less than 0 1 percent of their total managed care expenditures; while CMS noted in 1 focused review that overpayments typically equal 1 to 10 percent of total expenditures in managed care  CMS also found that 5 of the 27 states did not verify that MCOs excluded overpayments from these data, and 1 state did not exclude overpayments from the capitation rate setting  This is consistent with our March 2017 report in which we noted that CMS commonly found that MCOs reported low amounts of recovered overpayments and conducted few reviews to identify overpayments  Also, officials from three of the five states we interviewed for that report said the focused reviews gave them leverage in dealing with MCOs or led MCOs to focus more on program integrity  We also reported that CMS officials recommended states take steps to improve their oversight of MCOs, based on the focused review findings  The findings from CMS’s focused reviews of managed care also highlight the need for greater federal oversight of states  Without these reviews, it is unclear if states would independently identify MCOs’ reporting of overpayments or work to strengthen MCO reporting  Yet, CMS has not yet published the focused reviews of managed care in 13 states, and it may only conduct a focused review in a state once every three or more years  Given CMS’s timeline for the focused reviews, it may take years to determine if corrective actions result in improved program integrity in services delivered through managed care  Collaborative audits  CMS has expanded the federal-state collaborative audits beyond FFS, and has begun to engage states to participate in collaborative audits of MCOs and providers under contract to MCOs  As a part of the collaborative audit process, the state volunteers to jointly develop the audit processes the federal contractors follow  CMS officials told us that federal contractors have completed 14 collaborative audits of providers under contract to MCOs in three states—Arizona, the District of Columbia, and Tennessee  Only the audit of Trusted Healthcare, an MCO in the District of Columbia, has been published  That audit identified $129,000 in overpayments in a sample of MCO payments to providers, which, if generalized to all of the MCO’s payments over 6 months, would equate to over $4 million in overpayments  According to CMS, three additional states—Louisiana, Nebraska, and New Hampshire—have shown interest in collaborative audits of their MCOs, although such audits require states to prepare data files for the federal contractor and commit staff time  In our March 2017 report, we found that states’ participation in FFS collaborative audits varied and some states reported barriers to their participation  Expanding collaborative audits in managed care will require commitment from and coordination with states  State monitoring of overpayments in managed care  States are required to report overpayments they have identified and recouped along with state expenditures on a quarterly basis  However, based on the responses of the program integrity officials in 13 of the 16 states we contacted, most officials were unable to define the magnitude of overpayments in their managed care programs, which may signify a need for greater federal oversight or coordination  Specifically, officials in 7 of the 13 states could not or did not identify the share of total reported Medicaid overpayments that occurred in managed care  In 11 of the 13 states, officials responded that they did not directly monitor MCO payments to providers  Of those 11 states, officials in 4 said they depend on MCOs to report overpayments and exclude the overpayments from the data used to set capitation rates  As long as states are not taking action to identify overpayments in managed care, they cannot be assured that they are accurately paying MCOs for medically necessary services provided to enrollees  Federal internal control standards call for agency management to identify, analyze, and respond to risks  CMS has taken some steps to identify, analyze, and respond to risks through its regulations, Focused Program Integrity Reviews, and collaborative audits  However, key CMS and state oversight efforts fall short of mitigating the limitations of the PERM estimates of improper payments for managed care, because they do not ensure the identification and reporting of overpayments to providers and unallowable MCO costs  Without addressing these key risks, CMS and states cannot ensure the integrity of Medicaid managed care programs  The 0 3 percent improper payment rate for Medicaid managed care, as measured by the PERM, is significantly lower than the improper payment rate of 12 9 percent for Medicaid FFS  However, this difference does not signal better oversight; rather, it represents differences in the review criteria between FFS and managed care, which result in a less complete accounting for the program integrity risks in managed care  The PERM does not account for key program integrity risks in Medicaid managed care: specifically, unidentified overpayments and unallowable costs  One federal investigation of an MCO operating in nine states resulted in a settlement of $137 5 million to resolve allegations of false claims that were not captured in the national Medicaid improper payment rate estimate  Further, CMS found that MCOs and states do not provide sufficient oversight in Medicaid managed care to address the risks that are not accounted for in the PERM, findings that are reinforced by our reports on Medicaid managed care program integrity  CMS has taken steps to improve its oversight of Medicaid managed care, yet these efforts fall short of ensuring that the agency and states will be able to identify and address overpayments to providers and unallowable MCO costs  Without better measurement of program risks—particularly as expenditures for Medicaid managed care continue to grow—CMS cannot be certain that the low improper payment rate for managed care, as measured by the PERM, accurately reflects lower risks in managed care  The Administrator of CMS should consider and take steps to mitigate the program risks that are not measured in the PERM, such as overpayments and unallowable costs; such an effort could include actions such as revising the PERM methodology or focusing additional audit resources on managed care  (Recommendation 1) We provided a draft of this report to the Department of Health and Human Services (HHS) for comment  In its written comments, HHS concurred with our recommendation, and indicated that it will review regulatory authority and audit resources to determine the best way to account for Medicaid program risks that are not accounted for in the PERM  However, HHS stated that the PERM is not intended to measure all Medicaid program integrity risks, and utilizing the PERM measurement in that way would be a misunderstanding and misuse of the reported rate  HHS also commented that a review of payments from MCOs to providers is outside the scope of IPIA  In addition, HHS asserted that including such a review would diminish the value of PERM reporting—because it would require significant assumptions about the amount of federal share in MCO payments to providers  Further, HHS maintained that such a review also would result in a measurement that was not comparable to other programs or agencies, which would diminish the value of government- wide improper payment rate reporting  We acknowledge that the current PERM methodology has been approved by OMB  However, we maintain that the PERM likely underestimates program integrity risks in Medicaid managed care  To ensure the appropriate targeting of program integrity activities, CMS needs better information about these risks  Given the size of the Medicaid program, its vulnerability to improper payments, and the growth in managed care, it is critical to have a full accounting of program integrity risks in managed care in order to best ensure the integrity of the whole Medicaid program  In its written comments, HHS also summarized several activities it uses to oversee and support states’ Medicaid program integrity efforts, including state program integrity reviews; collaborative audits conducted by federal contractors; Medicaid Integrity Institute training for state employees; and the Medicaid Provider Enrollment Compendium  HHS also provided technical comments, which we incorporated as appropriate  HHS’s comments are reprinted in appendix II  As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date  At that time, we will send copies to the Secretary of Health and Human Services, the Administrator of CMS, appropriate congressional committees, and other interested parties  In addition, the report will be available at no charge on the GAO website at http://www gao gov  If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or at yocomc@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff that made key contributions to this report are listed in appendix III  We reviewed 16 federal and state audits and 11 notices of investigations of Medicaid managed care organizations (MCO) and providers issued from January 2012 to September 2017  As the findings below show, the audits and investigations represent a limited number of reviews that, in many cases, focused on individual states and individual providers or MCOs within that state  Given the limited scope and number of states reviewed, the amount of the overpayments and unallowable costs occurring nationwide is unknown  These audits and investigations show cases of MCO overpayments to providers or unallowable costs, which are not accounted for by the Centers for Medicare & Medicaid Services’ Payment Error Rate Measurement (PERM) ; errors in capitated payments (e g , capitated payments made for deceased individuals), which are accounted for in the PERM; and gaps in managed care oversight  When reporting overpayments and unallowable costs identified in the audits and investigations, we only include amounts specifically attributed to MCOs in our total  This total does not include the following: overpayments and unallowable costs identified in those audits and investigations that did not distinguish between the amounts attributable to MCOs, Medicaid fee-for-service, or Medicare; overpayments and unallowable costs identified in criminal proceedings that are not yet resolved; and errors in capitated payments, as those payments would be reviewed by the PERM  As a result, the total amount of overpayments and unallowable costs and capitated payment errors in this appendix exceed what we report  In addition to the contact named above, Leslie V  Gordon (Assistant Director), Pauline Adams (Analyst-in-Charge), Erika Huber, and Drew Long made key contributions to this report  Also contributing were Muriel Brown and Jennifer Whitworth 


Report 2:  the United States tends to underinvest in dual-use technologies  National security requirements alone often do not justify major DOD support, and market prospects alone often appear to be too long-term or high risk to justify US industry carrying the entire development burden     The committee encourages use of cooperative agreements and other transactions in lieu of grants or contracts     The provision would require that at least 50 percent of funding over the life of a partnership derive from non-federal sources but would allow for a smaller industry share at the start  The conference report stated that the partnerships should focus on programs that fit into the security needs within DARPA  The conference report also stated that OTs are appropriate for those cases where the "regulations applicable to the allocation of patent and data rights under the procurement statutes may not be appropriate to partnership arrangements in certain cases " National Defense Authorization Act for FY1993 ( P L  102-484 ) Cost Sharing Section 4221 established 10 U S C  2511, which required DOD to establish cooperative arrangements with industry, educational institutions, federal labs, and other entities, to pursue research, development, and application of dual-use technologies  The section authorized DOD to use grants, contracts, cooperative agreements, or OTs to create these partnerships, and that the Federal government should not contribute more than 50% of the costs related to projects under this authority  National Defense Authorization Act for FY1994 ( P L  103-160 ) Expanded Authority Section 827 established 10 U S C  2358, which gave the Secretary of Defense and the Secretaries of the military departments the authority to conduct basic, advanced, and applied research through the use of contracts, cooperative agreements, grants, and OTs  Previously, OTs were only authorized for advanced research  Prototype Authorities Section 845 granted DARPA the authority to use OTs for prototype projects directly related to weapons or weapon systems proposed to be acquired by DOD  Section 845 required that "to the maximum extent practicable," prototypes be competitively awarded  This authority was set to terminate after three years  Section 845 remained as a note to 10 U S C  2371 until separately codified as 10 U S C  2371b in the FY2016 NDAA ( P L  114-92 )  Federal Acquisition Streamlining Act of 1994 ( P L  103-355 ) Section 1301 redesignated the language in 10 U S C  2358 (granting the authority to use OTs) to 10 U S C  2371  Reporting Requirements Section 1301 also required DOD to submit an annual report to the armed services committees, to include a general description of the other transactions, including the technologies involved in the research, the potential military and, if any, commercial utility of such technologies, the reasons for not using a contract or grant to provide support for such research, and the amount of payments, if any, received during the fiscal year pursuant to a clause in the other transactions and to what accounts such payments were credited  National Defense Authorization Act for FY1997 ( P L  104-201 ) Section 203 required that a senior DOD official be designated in OSD, and that the officials' sole responsibility be developing policy related to, and ensuring implementation of, DOD's dual-use technology program  This section authorized DOD to use OTs (as well as contracts, cooperative agreements, and grants) for dual-use projects only if the project "is entered into through the use of competitive procedures " Section 743 granted DOD the authority to use OTs to conduct research on Gulf War Syndrome, to determine its relationship to possible exposures of members of the Armed Forces to chemical warfare agents and hazardous materials, and the use of inoculations and new drugs  Expanded Prototype Authorities Section 804 amended Section 845 of the FY1994 NDAA by extending to the military departments and officials designated by the Secretary of Defense, the authority to use OTs for certain prototype projects  This authority, originally granted solely to DARPA and set to expire after three years, was given a new termination date of September 30, 1999  Reporting Requirements Section 267 modified elements of the annual report to the armed services committees  National Defense Authorization Act for FY1998 ( P L  105-85 ) Section 832 amended 10 U S C  2371 by clarifying that certain information submitted to DOD (i e  a proposal, business plan, technical information) be protected from disclosure pursuant to 5 U S C  552 for a period of five years  Strom Thurmond National Defense Authorization Act for FY1999 ( P L  105-261 ) Section 241 extended the sunset day for the authority to use OTs for prototypes from September 30, 1999, to September 30, 2001  Section 817 amended Section 2371 of Title 10, United States Code, clarifying that information submitted by outside parties in cooperative agreements for basic, applied, and advanced research is protected from disclosure under Section 552 of Title 5, United States Code  Department of Defense Appropriations Act, 1999 ( P L  105-262 ) While the enacted FY1999 defense appropriations bill ( P L  105-262 ) did not include legislative language addressing OTs, H Rept  105-591 , which accompanied the House-reported version of H R  4103 , included language expressing the House Appropriations Committee's "serious reservations" regarding the Air Force's then-proposed use of an OT agreement—instead of a contract— to develop the Evolved Expendable Launch Vehicle (EELV) program  The committee noted that "under [OTs] traditional safeguards which protect the government's interest in large acquisition programs are largely absent," and required the Under Secretary of Defense for Acquisition, Technology, and Logistics (now the Under Secretary of Defense for Acquisition and Sustainment) and the DOD Inspector General to certify to the congressional defense committees that the use of an OT was appropriate for the EELV program, and that "adequate safeguards exist[ed] to protect the government's interest and monitor program performance " National Defense Authorization Act for FY2000 ( P L  106-65 ) Section 801 required that for prototypes using OT authorities, DOD ensure that GAO, under its audit authority, have access to records relating to other transaction prototype agreements exceeding $5 million  Section 801 allowed for a waiver to GAO access and exempted entities that over the last year have not entered into an agreement with DOD that provided for audit access by a government entity  According to the Senate report: Senior DOD officials have sought legislation to extend other transaction authority to production contracts  Under current authority, there is some debate about whether GAO has audit access to other transactions  As the size, costs, and complexity of programs being funded using other transactions increases, the committee wants to ensure that the GAO has audit access in relation to the higher levels of spending and added risks  Reporting Requirements The Senate report also addressed reporting requirements and congressional intent to review the use of OTs  The report stated the following: The committee is assessing the utility of other transaction prototype authority  The statement of managers accompanying the Strom Thurmond National Defense Authorization Act of 1999 directed the Secretary of Defense to report on the use of this authority to the congressional defense committees, no later than March 1, 1999  In addition, both the Department of Defense Inspector General and the General Accounting Office are reviewing the use of other transaction prototype authority and will report to Congress in the coming year  The committee is interested in the extent that new commercial firms are entering the DOD marketplace through the use of other transaction authority, as well as the degree of cost sharing between the government and non-federal government parties  The committee is also interested in any lessons learned from the broad exemptions to federal law provided by other transaction authority  For example, other transactions are exempt from the Competition in Contracting Act, Truth in Negotiations Act, Contract Disputes Act, Antikickback Act of 1986, Procurement Integrity Act, Service Contract Act, Buy American Act, and chapter 137 of title 10, United States Code  Questions have been raised about whether the government's interest is adequately protected in the absence of the applicability of these statutes  Conversely, advocates of the view that the government should take advantage of the flexibility of other transactions have championed proposals to extend other transaction authority to production  The committee directs the Secretary of Defense to provide a new report that updates information in the March 1999 report on the use of other transaction prototype authority to the congressional defense committees by February 1, 2000  National Defense Authorization Act for FY2001 ( P L  106-398 ) Limitations on the Use of OTs Section 803 limited the use of OTs for prototype projects to only those circumstances when at least one nontraditional defense contractor significantly participates in the project, one-third of the total cost of the project is paid out of funds provided by parties to the transaction other than the federal government, or the senior procurement executive determines in writing that exceptional circumstances justify use of an OT  Nontraditional defense contractor was defined as an entity that for a period of one year has not entered into or performed "any contract that is subject to full coverage under the cost accounting standards" or "any other contract in excess of $500,000 to carry out prototype projects or to perform basic, applied, or advanced research projects for a Federal agency, that is subject to the Federal Acquisition Regulation '' Section 803 also extended the authority to use OTs for prototypes from September 30, 2001, to September 30, 2004  According to the Senate report, the intent of using OTs for prototypes is to attract companies that typically do not do business with the Department of Defense and encourage cost sharing and experimentation in potentially more efficient ways of doing business with traditional defense contractors  Other transaction authority is an important acquisition tool that can facilitate the incorporation of commercial technology into military weapon systems  In an environment where, in many areas, commercial technology is now more advanced than defense technology, it is imperative that the Department continue to have the flexibility to use innovative contractual instruments that provide access to this technology  There are, however, improvements that can be made in managing and overseeing these contractual arrangements  Section 804 clarified the extent of GAO's access to records in instances where the party in question has only done business with the government in the preceding year through an OT or cooperative agreement  National Defense Authorization Act for FY2002 ( P L  107-107 ) Expanded Authority—Follow-on Production Section 822 of the FY2002 NDAA granted DOD the authority to award a follow-on production contract for prototype projects when at least one-third of the total cost of the prototype project is to be paid out of funds provided by non-federal government sources  Under this authority, such a follow-on contract could be awarded without competition if the prototype project was successfully completed, the number of units in the production contract does not exceed the number of units specified in the underlying prototype agreement, and the price for each unit does not exceed the price specified in the underlying transaction  Department of Defense and Emergency Supplemental Appropriations for Recovery from and Response to Terrorist Attacks on the United States Act, 2002 ( P L  107-117 ) Establishment of Army Venture Capital Initiative (AVCI) Section 8150 designated $25 million of the FY2002 funds made available for Army Research, Development, Test, and Evaluation (RDT&E) to be made available to the Secretary of the Army for the purpose of funding a venture capital investment corporation established pursuant to 10 U S C  2371  A 2014 RAND report stated that OT authorities were used only to form the AVCI, and were not used to acquire products or services: "While OT authorities were used to form [AVCI] itself, any volume of the Army's purchase of products and services from [AVCI] companies is conducted under the FAR " National Defense Authorization Act for FY2004 ( P L  108-136 ) Expanded Authority Section 847 of the FY2004 NDAA extended the authority to use an OT for developing prototypes to improve weapons or weapon systems currently in use by the Armed Forces  Previously, such authority was restricted to prototypes "directly relevant to weapons or weapon systems proposed to be acquired or developed" by DOD  Section 847 also established a pilot program for transitioning prototypes to follow-on contracts for production for nontraditional defense contractors  Under the pilot program, such a follow-on contract could be treated as a commercial item or an item developed with both federal and private sector funds (for purposes of negotiating intellectual property rights)  The pilot program was restricted to contracts with nontraditional defense contractors, where the value of the contract does not exceed $50 million (approximately $70 million in FY2018 dollars), and that are firm-fixed price or fixed price with economic adjustment  The pilot program was set to sunset September 30, 2008  Section 1441 authorized any agency that engages in basic, applied, or advanced research and development projects that facilitated defense against or recovery from terrorism or nuclear, biological, chemical, or radiological attack to exercise the same general authority given to DOD as found in 10 U S C  2371 (including for prototype projects)  Reporting Requirements Section 1031 sunset the annual reporting requirement for OT after the report covering FY2006 was submitted  National Defense Authorization Act for FY2006 ( P L  109-163 ) Restricted Authority and Notification Requirements Section 212 of the FY2006 NDAA directed the Army to procure the Future Combat System using contract procedures set forth in part 15 of the Federal Acquisition Regulation, in lieu of an OT  Section 823 extended ethics requirements to OT prototype authority and required the congressional defense committees be notified in writing at least 30 days before such authority is exercised  Certification Requirements Section 823 also amended Section 845 of the FY1994 NDAA, requiring a written determination by a senior procurement executive for other transaction prototype projects estimated between $20 million and $100 million, and for a written determination by the Under Secretary of Defense for Acquisition, Technology, and Logistics for prototype projects that exceed $100 million  According to the Senate report: Section 845 was intended to be used for limited prototype projects, particularly those in which the Department seeks to engage nontraditional defense contractors that may be averse to the requirements imposed by a standard Department procurement contract  For this reason, the statement of managers accompanying the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999 (Public Law 105–261) states: The conferees continue to believe that the section 845 authority should only be used in the exceptional cases where it can be clearly demonstrated that a normal contract or grant will not allow sufficient access to affordable technologies  The conferees are especially concerned that such authority not be used to circumvent the appropriate management controls in the standard acquisition and budgeting process  …  The committee does not believe that the $20 9 billion agreement entered between the Army and the Lead Systems Integrator for the FCS program is consistent with the language and intent of section 845 authority  Section 845 authority is intended to be used for limited prototype projects, particularly those in which the Department of Defense seeks to engage nontraditional defense contractors that may be averse to the requirements imposed by a standard Department contract  Department of Defense Appropriations Act, 2007 ( H R  5631 ) Reporting Requirement The FY2007 defense appropriations bill ( P L  109-289 ) did not include language addressing OTs  The conference report ( H Rept  109-676 ) included language expressing the conferees' "[concern] with the continued use of OTA contracts by the Missile Defense Agency," as such contracts "lack the customary safeguards found under FAR-based contracts for organizational conflict of interest, truth in negotiations and submission of cost and pricing data " The conferees "strongly encourage[d]" the Missile Defense Agency to convert "large development and procurement contracts using OTA to FAR-based contracts," and directed the Missile Defense Agency to submit a report to the congressional defense committees on the use of OTs, to include the number, value, and justification for the use of such agreements  National Defense Authorization Act for FY2008 ( P L  110-181 ) Section 823 extended the authority for prototype projects for five more years, from September 30, 2008, to September 30, 2013  National Defense Authorization Act for FY2009 ( P L  110-417 ) Section 822 required DOD to issue guidance on rights in technical data under non-FAR agreements, including OTs  Section 822 also required that appropriate provisions relating to rights in technical data be included in non-FAR agreements, consistent with policy guidance  This requirements is in statute at 10 U S C  2320 note  Section 824 expanded the scope of the pilot program for transition to follow-on contracts for certain prototype projects to include research projects carried out under 10 U S C  2371  Authority to use the pilot program, set to expire September 30, 2008, was extended to September 30, 2010  Section 874 required OT data be included in the Federal Procurement Data System  National Defense Authorization Act for FY2011 ( P L  111-383 ) Section 866 changed the definition of nontraditional defense contractor, conforming the definition to that found in 10 U S C  2302(9)  National Defense Authorization Act for FY2013 ( P L  112-239 ) Section 863 extended the authority for using OTs for prototype projects from September 30, 2013, to September 30, 2018  Carl Levin and Howard P  "Buck" McKeon National Defense Authorization Act for FY2015 ( P L  113-291 ) Expanded Authority Section 812 expanded the authority to use OT for prototypes, to include those "directly related to enhancing the mission effectiveness of military personnel and the supporting platforms, systems, components, or materials proposed to be acquired or developed by the Department of Defense, or to improvement of platforms, systems, components, or materials in use by the Armed Forces " Prior to the FY2015 NDAA, OTs could only be used for prototypes relating to weapons or weapon systems proposed to be developed, or for the improvement of weapons or weapon systems currently in use  Reporting Requirements Section 1071 repealed the reporting requirement language found in 10 U S C  2371, relating to OTs for research projects  National Defense Authorization Act for FY2016 ( P L  114-92 ) Until the FY2016 NDAA, the prototyping and follow-on production authority established in Section 845 of the FY1994 NDAA (as amended) was found in 10 U S C  2371 note  Section 815 of the FY2016 NDAA simultaneously repealed Section 845 of the FY1994 NDAA and put the repealed language into the newly created 10 U S C  2371b  The FY2016 NDAA also modified 2371b by making the authority permanent  Expanded Authority and Small Business The authorities in Section 2371b were expanded to allow their use when "all significant participants in the transaction other than the Federal government are small businesses or nontraditional contractors" and when the agency determines that using an OT would expand the defense supply base in a manner that could not be accomplished through a contract  Section 815 eased the restriction on follow-on production contracts or transactions  Section 815 amended the definition of a nontraditional defense contractor found in 10 U S C  2302 to be an entity that is not currently performing, and for one year prior to an OT has not performed on any contract or subcontract that is subject to full coverage under the cost accounting standards pursuant to Section 1502 of Title 41, U S C  Section 815 also required DOD to update its guidance to reflect changes in the statute  The conference report stated that Congress believed OTs are an attractive option for firms and organizations that do not usually participate in government contracting due to typical overhead burdens and the "one size fits all" rules governing defense acquisition  The report also stated that OTs could support DOD's effort to access new sources of technological innovation, specifically with Silicon Valley startup firms and small commercial firms  National Defense Authorization Act for FY2018 ( P L  115-91 ) Expanded Authority Section 216 of the FY2018 NDAA authorized nonprofit research institutions to enter into OTs with DOD for prototype projects  Section 862 amended 10 U S C  2358, granting the Secretary of Defense and the military departments the authority to pursue basic research, applied research, advanced research, and development projects under the OT authorities granted in Sections 2371 and 2371b of Title 10  Workforce Section 802 required DOD to establish a cadre of intellectual property experts to advise, assist, and provide resources to program offices who are developing intellectual property strategies for contracts and agreements  Section 863 required training and education for personnel involved in OTs and other innovative contracting methods  Certification Requirements and Small Business Section 864 adjusted the language of the statute to state that the dollar threshold relates to the specific transaction for a prototype project and not for the value of the entire project  Section 864 defined a transaction for follow-on production to include "all individual prototype sub-projects awarded under the transaction to a consortium of United States industry and academic institutions " Section 864 also increased the dollar thresholds for required approvals and defined the term small business to include small businesses under Section 9 of the Small Business Act to ensure that companies participating in the Small Business Innovation Research and Small Business Technology Transfer programs were considered small businesses for the purposes of the cost-sharing requirements  Miscellaneous Section 867 required the Secretary of Defense to establish a preference for OTs in the "execution of science and technology and prototyping programs " Section 1711 required DOD to carry out a pilot program to "assess the feasibility and advisability of increasing the manufacturing capability of the defense industrial base " Pursuant to the pilot, Section 1711 authorized DOD to use OTs to support production capabilities in small and medium-sized manufacturers  John S  McCain National Defense Authorization Act for FY2019 ( P L  115-232 ) Section 211 of the FY2019 NDAA clarified that follow-on production of a prototype or subproject within a consortium may occur if the individual prototype or subproject is complete; all projects associated with the consortium do not need to be completed before follow-on production of a specific prototype  Expanded Authority The FY2019 NDAA authorized the use of OTs to develop enhanced personal protective equipment (Section 226) and to carry out research under the Explosive Ordnance Disposal Defense Program (Section 311)  Reporting Requirements Section 244 required the Defense Innovation Unit to submit a report to Congress, to include the number of traditional and nontraditional defense contractors with DOD contracts or other transactions resulting directly from the unit's initiatives  Section 873 required DOD to submit an annual report through 2021, summarizing DOD's use of OTs, including organizations involved; number of transactions; amounts of payments; and purpose, description, and status of projects  Department of Defense Appropriations Act, 2019 ( P L  115-245 ) Reporting Requirements While the enacted FY2019 defense appropriations bill did not include legislative language addressing OTs, the conference report included language expressing the conferees' "[concern] with the lack of transparency surrounding the employment of OTA, particularly for follow-on production " The conferees directed DOD to provide quarterly reports to the House and Senate appropriations committees listing each active OT, and to include the following information on each agreement: funding military service or DOD component; major command (if applicable); contracting activity; appropriation title; budget line item; minimum and maximum award value; vendor; obligations and expenditures to date; product service code; period of performance; and indication if the OT agreement included an option for follow-on production (with a description of the scope of anticipated follow-on production)  The conferees also directed GAO to review DOD's use of OTs to determine whether the "employment of this authority conforms to applicable statutes and guidelines, to include the identification of any potential conflicts " GAO was also required to report on the extent to which OTs have been used since FY2016  Notification Requirements The House report to accompany H R  6157 acknowledged OTs as an "important tool to provide flexibility and agility for cutting-edge research and development projects and prototypes " However, the report stated its concern "with the lack of transparency on the use of OTA authority for follow-on production procurements," and directed that no funds could be obligated or expended for a follow-on production contract or a transaction carried out under 10 U S C  2371b, until 30 days after the Secretary of Defense provides the congressional defense committees with a notification of the proposed contract or transaction, to include a justification of why an OT is being used for production  Appendix B  Non-DOD Federal Agencies with Agency-Wide OT or Related Authorities A number of agencies have varying other transaction or similar authorities, as reflected in Table B-1   The table below is not a comprehensive or definitive listing of every federal government entity with OT or related authorities  In some instances, offices, agencies, commissions, and other federal government entities have OT or related authorities that are only associated with certain programs or projects, such as the National Institutes of Health (which has OT authority for such specific activities such as the National Heart, Blood Vessel, Lung, and Blood Diseases and Blood Resources Program [42 U S C  §285b-3] and the Cures Acceleration Network [42 U S C  §287a])  Appendix C  Reliability of Data on Other Transactions All data have imperfections and limitations  FPDS-NG data can be used to identify broad trends and produce rough estimates, or to gather information about specific contracts  Some observers say that despite their shortcomings, FPDS-NG data are substantially more comprehensive than what is available in most other countries in the world  Understanding the limitations of government procurement data—including knowing when, how, and to what extent to rely on data—can help policymakers incorporate FPDS-NG data more effectively into their decisionmaking process  FPDS-NG OT Data Quality and Accuracy Issues Decisionmakers should be cautious when using data from FPDS-NG to develop policy or otherwise draw conclusions, especially with respect to OTs  In some cases, the data themselves may not be reliable  In other instances, a query for particular data may return differing results, depending on the parameters and timing of the analysis  In particular, all DOD data entered into FPDS-NG are subject to a 90-day delay, and updates to "data, including new actions, modifications, and corrections are made on a regular basis," which could result in changes to "data     for current and/or prior fiscal years " Inconsistencies in FPDS-NG Data Within FPDS-NG, two primary collections of obligation data exist: one associated with standard government procurement contracts or modifications to such contracts, and the other associated with prototype OT agreements or modifications to such agreements  FPDS-NG's collection of prototype OT data allows for the input of additional data elements—such as nongovernment dollars associated with cost-share prototype OT agreements—not included in FPDS-NG's collection of standard government procurement contract data  More consequentially, FPDS-NG's prototype OT data include two similar data elements that allow users to identify the fiscal year a prototype OT agreement was signed or modified  One, labeled in the database as "Fiscal Year," appears to allow users entering data into the system to manually assign a fiscal year to a transaction  FPDS-NG users entering data into the system appear to have interpreted this data element in various, conflicting ways  For example, a Department of the Air Force OT agreement was signed in February 2016 for the development of rocket propulsion system prototypes under the Evolved Expendable Launch Vehicle (EELV) program  FPDS-NG records an obligation of $115 million in FY2020 for this agreement  Fiscal law bars DOD from obligating money now for future fiscal years that have not yet occurred  The second, labeled "Contract Fiscal Year," appears to be based on the date the prototype OT agreement was signed or modified  See Table C-1 for a comparison of the "Fiscal Year" and "Contract Fiscal Year" elements for selected new prototype OT agreements signed between FY2013 and FY2017  If a user selects the Fiscal Year data element when attempting to review high-level data on recent trends in the use of prototype OT agreements within DOD—such as the total amount obligated for prototype OT agreements on an annual basis—that user will obtain a substantially different result than if he or she selects the Contract Fiscal Year data element for a similar analysis  See Table C-2 for a comparison of action obligations and nongovernment contributions using the Fiscal Year and Contract Fiscal Year data elements for new prototype OT agreements signed between FY2013 and FY2017  DOD OT Data Analysis Methodological Issues in Congressional Reports A March 2017 report to Congress entitled "An Assessment of Cost-Sharing in Other Transaction Agreements for Prototype Projects," completed by the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics indicated that In FY2016, DOD obligated (Note: Two outliers in 2016 excluded) $1 billion in section 2371b awards and received $68 million in cost-share contributions after excluding two significant trend outliers  DOD cited FPDS-NG as the source for its analysis, and defined the "significant trend outliers" excluded as "two $40 million OTAs with total cost-share of $270 million," likely referring to two DARPA prototype OT agreements conducted on a cost-share basis initiated in FY2016  However, a CRS analysis of the same FPDS-NG data identified numerous inconsistencies in DOD's methodological approach  Recreation of DOD Methodology Specifically, DOD appears to have conducted its analysis using the "Fiscal Year" data element referenced in this report's discussion of FPDS-NG OT data quality and accuracy issues, which would attribute some prototype OT activities to the wrong fiscal year for the purposes of comparative trend analysis  DOD also compared two disparate transaction types: the reported "$1 billion in Section 2371b awards" includes all action obligations associated with ongoing prototype OT activities in FY2016, including transactions associated with prototype OT indefinite delivery contracts, while the "$68 million in cost-share contributions" includes only cost-share contributions associated with new prototype OT agreements  If DOD used the "Contract Fiscal Year" data element, excluded the identified "trend outliers," and focused on all action obligations and cost-share contributions associated with ongoing prototype OT activities, it would have found instead that the department obligated $1 4 billion for prototype OT agreements in FY2016, with an additional $313 5 million in cost-share contributions from the private sector  On the other hand, if DOD used the "Contract Fiscal Year" data element, excluded the identified "trend outliers," and focused on only action obligations and cost-share contributions associated with new prototype OT agreements, it would have found instead that the department obligated $400 million for prototype OT agreements in FY2016, with an additional $272 1 million in cost-share contributions from the private sector  Appendix D  Other Transaction Authority Statutes 10 U S C  §2371  Research projects: transactions other than contracts and grants (a) ADDITIONAL FORMS OF TRANSACTIONS AUTHORIZED — The Secretary of Defense and the Secretary of each military department may enter into transactions (other than contracts, cooperative agreements, and grants) under the authority of this subsection in carrying out basic, applied, and advanced research projects  The authority under this subsection is in addition to the authority provided in Section 2358 of this title to use contracts, cooperative agreements, and grants in carrying out such projects  (b) EXERCISE OF AUTHORITY BY SECRETARY OF DEFENSE — In any exercise of the authority in subsection (a), the Secretary of Defense shall act through the Defense Advanced Research Projects Agency or any other element of the Department of Defense that the Secretary may designate  (c) ADVANCE PAYMENTS — The authority provided under subsection (a) may be exercised without regard to Section 3324 of Title 31  (d) RECOVERY OF FUNDS — (1) A cooperative agreement for performance of basic, applied, or advanced research authorized by Section 2358 of this title and a transaction authorized by subsection (a) may include a clause that requires a person or other entity to make payments to the Department of Defense or any other department or agency of the Federal Government as a condition for receiving support under the agreement or other transaction  (2) The amount of any payment received by the Federal Government pursuant to a requirement imposed under paragraph (1) may be credited, to the extent authorized by the Secretary of Defense, to the appropriate account established under subsection (f)  Amounts so credited shall be merged with other funds in the account and shall be available for the same purposes and the same period for which other funds in such account are available  (e) CONDITIONS — (1) The Secretary of Defense shall ensure that- (A) to the maximum extent practicable, no cooperative agreement containing a clause under subsection (d) and no transaction entered into under subsection (a) provides for research that duplicates research being conducted under existing programs carried out by the Department of Defense; and (B) to the extent that the Secretary determines practicable, the funds provided by the Government under a cooperative agreement containing a clause under subsection (d) or a transaction authorized by subsection (a) do not exceed the total amount provided by other parties to the cooperative agreement or other transaction  (2) A cooperative agreement containing a clause under subsection (d) or a transaction authorized by subsection (a) may be used for a research project when the use of a standard contract, grant, or cooperative agreement for such project is not feasible or appropriate  (f) SUPPORT ACCOUNTS — There is hereby established on the books of the Treasury separate accounts for each of the military departments and the Defense Advanced Research Projects Agency for support of research projects and development projects provided for in cooperative agreements containing a clause under subsection (d) and research projects provided for in transactions entered into under subsection (a)  Funds in those accounts shall be available for the payment of such support  (g) EDUCATION AND TRAINING —The Secretary of Defense shall— (1) ensure that management, technical, and contracting personnel of the Department of Defense involved in the award or administration of transactions under this section or other innovative forms of contracting are afforded opportunities for adequate education and training; and (2) establish minimum levels and requirements for continuous and experiential learning for such personnel, including levels and requirements for acquisition certification programs  (h) REGULATIONS — The Secretary of Defense shall prescribe regulations to carry out this section  (i) Protection of Certain Information From Disclosure -(1) Disclosure of information described in paragraph (2) is not required, and may not be compelled, under Section 552 of Title 5 for five years after the date on which the information is received by the Department of Defense  (2)(A) Paragraph (1) applies to information described in subparagraph (B) that is in the records of the Department of Defense if the information was submitted to the Department in a competitive or noncompetitive process having the potential for resulting in an award, to the party submitting the information, of a cooperative agreement for performance of basic, applied, or advanced research authorized by Section 2358 of this title or another transaction authorized by subsection (a)  (B) The information referred to in subparagraph (A) is the following: (i) A proposal, proposal abstract, and supporting documents  (ii) A business plan submitted on a confidential basis  (iii) Technical information submitted on a confidential basis  10 U S C  §2371b  Authority of the Department of Defense to carry out certain prototype projects (a) AUTHORITY — (1) Subject to paragraph (2), the Director of the Defense Advanced Research Projects Agency, the Secretary of a military department, or any other official designated by the Secretary of Defense may, under the authority of Section 2371 of this title, carry out prototype projects that are directly relevant to enhancing the mission effectiveness of military personnel and the supporting platforms, systems, components, or materials proposed to be acquired or developed by the Department of Defense, or to improvement of platforms, systems, components, or materials in use by the armed forces  (2) The authority of this section- (A) may be exercised for a transaction (for a prototype project) that is expected to cost the Department of Defense in excess of $100,000,000 but not in excess of $500,000,000 (including all options) only upon a written determination by the senior procurement executive for the agency as designated for the purpose of Section 1702(c) of Title 41, or, for the Defense Advanced Research Projects Agency or the Missile Defense Agency, the director of the agency that- (i) the requirements of subsection (d) will be met; an d (ii) the use of the authority of this section is essential to promoting the success of the prototype project; and (B) may be exercised for a transaction (for a prototype project) that is expected to cost the Department of Defense in excess of $500,000,000 (including all options) only if- (i) the Under Secretary of Defense for Acquisition, Technology, and Logistics determines in writing that- (I) the requirements of subsection (d) will be met; an d (II) the use of the authority of this section is essential to meet critical national security objectives; and (ii) the congressional defense committees are notified in writing at least 30 days before such authority is exercised  (3) The authority of a senior procurement executive or director of the Defense Advanced Research Projects Agency or Missile Defense Agency under paragraph (2)(A), and the authority of the Under Secretary of Defense for Acquisition, Technology, and Logistics under paragraph (2)(B), may not be delegated  (b) EXERCISE OF AUTHORITY — (1) Subsections (e)(1)(B) and (e)(2) of such Section 2371 shall not apply to projects carried out under subsection (a)  (2) To the maximum extent practicable, competitive procedures shall be used when entering into agreements to carry out projects under subsection (a)  (c) COMPTROLLER GENERAL ACCESS TO INFORMATION — (1) Each agreement entered into by an official referred to in subsection (a) to carry out a project under that subsection that provides for payments in a total amount in excess of $5,000,000 shall include a clause that provides for the Comptroller General, in the discretion of the Comptroller General, to examine the records of any party to the agreement or any entity that participates in the performance of the agreement  (2) The requirement in paragraph (1) shall not apply with respect to a party or entity, or a subordinate element of a party or entity, that has not entered into any other agreement that provides for audit access by a Government entity in the year prior to the date of the agreement  (3) (A) The right provided to the Comptroller General in a clause of an agreement under paragraph (1) is limited as provided in subparagraph (B) in the case of a party to the agreement, an entity that participates in the performance of the agreement, or a subordinate element of that party or entity if the only agreements or other transactions that the party, entity, or subordinate element entered into with Government entities in the year prior to the date of that agreement are cooperative agreements or transactions that were entered into under this section or Section 2371 of this title  (B) The only records of a party, other entity, or subordinate element referred to in subparagraph (A) that the Comptroller General may examine in the exercise of the right referred to in that subparagraph are records of the same type as the records that the Government has had the right to examine under the audit access clauses of the previous agreements or transactions referred to in such subparagraph that were entered into by that particular party, entity, or subordinate element  (4) The head of the contracting activity that is carrying out the agreement may waive the applicability of the requirement in paragraph (1) to the agreement if the head of the contracting activity determines that it would not be in the public interest to apply the requirement to the agreement  The waiver shall be effective with respect to the agreement only if the head of the contracting activity transmits a notification of the waiver to Congress and the Comptroller General before entering into the agreement  The notification shall include the rationale for the determination  (5) The Comptroller General may not examine records pursuant to a clause included in an agreement under paragraph (1) more than three years after the final payment is made by the United States under the agreement  (d) APPROPRIATE USE OF AUTHORITY — (1) The Secretary of Defense shall ensure that no official of an agency enters into a transaction (other than a contract, grant, or cooperative agreement) for a prototype project under the authority of this section unless one of the following conditions is met: (A) There is at least one nontraditional defense contractor or nonprofit research institution participating to a significant extent in the prototype project  (B) All significant participants in the transaction other than the Federal Government are small businesses (including small businesses participating in a program described under Section 9 of the Small Business Act (15 U S C  638)) or nontraditional defense contractors  (C) At least one third of the total cost of the prototype project is to be paid out of funds provided by sources other than other than the Federal Government  (D) The senior procurement executive for the agency determines in writing that exceptional circumstances justify the use of a transaction that provides for innovative business arrangements or structures that would not be feasible or appropriate under a contract, or would provide an opportunity to expand the defense supply base in a manner that would not be practical or feasible under a contract  (2) (A) Except as provided in subparagraph (B), the amounts counted for the purposes of this subsection as being provided, or to be provided, by a party to a transaction with respect to a prototype project that is entered into under this section other than the Federal Government do not include costs that were incurred before the date on which the transaction becomes effective  (B) Costs that were incurred for a prototype project by a party after the beginning of negotiations resulting in a transaction (other than a contract, grant, or cooperative agreement) with respect to the project before the date on which the transaction becomes effective may be counted for purposes of this subsection as being provided, or to be provided, by the party to the transaction if and to the extent that the official responsible for entering into the transaction determines in writing that- (i) the party incurred the costs in anticipation of entering into the transaction; and (ii) it was appropriate for the party to incur the costs before the transaction became effective in order to ensure the successful implementation of the transaction  (e) Definitions —In this section: (1) The term "nontraditional defense contractor" has the meaning given the term under Section 2302(9) of this title  (2) The term "small business" means a small business concern as defined under Section 3 of the Small Business Act (15 U S C  632)  (f) FOLLOW-ON PRODUCTION CONTRACTS OR TRANSACTIONS — (1) A transaction entered into under this section for a prototype project may provide for the award of a follow-on production contract or transaction to the participants in the transaction  A transaction includes all individual prototype subprojects awarded under the transaction to a consortium of United States industry and academic institutions  (2) A follow-on production contract or transaction provided for in a transaction under paragraph (1) may be awarded to the participants in the transaction without the use of competitive procedures, notwithstanding the requirements of Section 2304 of this title, if - (A) competitive procedures were used for the selection of parties for participation in the transaction; and (B) the participants in the transaction successfully completed the prototype project provided for in the transaction  (3) Contracts and transactions entered into pursuant to this subsection may be awarded using the authority in subsection (a), under the authority of Chapter 137 of this title, or under such procedures, terms, and conditions as the Secretary of Defense may establish by regulation  (g) AUTHORITY TO PROVIDE PROTOTYPES AND FOLLOW-ON PRODUCTION ITEMS AS GOVERNMENT-FURNISHED EQUIPMENT — An agreement entered into pursuant to the authority of subsection (a) or a follow-on contract or transaction entered into pursuant to the authority of subsection (f) may provide for prototypes or follow-on production items to be provided to another contractor as Government-furnished equipment  (h) APPLICABILITY OF PROCUREMENT ETHICS REQUIREMENTS — An agreement entered into under the authority of this section shall be treated as a Federal agency procurement for the purposes of Chapter 21 of Title 41  10 U S C  §2373  Procurement for experimental purposes (a) AUTHORITY — The Secretary of Defense and the Secretaries of the military departments may each buy ordnance, signal, chemical activity, transportation, energy, medical, space-flight, and aeronautical supplies, including parts and accessories, and designs thereof, that the Secretary of Defense or the Secretary concerned considers necessary for experimental or test purposes in the development of the best supplies that are needed for the national defense  (b) PROCEDURES — Purchases under this section may be made inside or outside the United States and by contract or otherwise  Chapter 137 of this title applies only when such purchases are made in quantities greater than necessary for experimentation, technical evaluation, assessment of operational utility, or safety or to provide a residual operational capability 

ground_truth: Commonly known for protecting the President, the Secret Service also plays a leading role in investigating and preventing financial and electronic crimes. To accomplish its mission, the Secret Service relies heavily on the use of IT infrastructure and systems. In 2009, the component initiated the IITT investment—a portfolio of programs and projects that are intended to, among other things, improve systems availability and security in support of the component's business operations. GAO was asked to review the Secret Service's oversight of its IT portfolio and workforce. This report discusses the extent to which the (1) CIO implemented selected IT oversight responsibilities, (2) Secret Service implemented leading IT workforce planning and management practices, and (3) Secret Service and DHS implemented selected performance monitoring practices for IITT. GAO assessed agency documentation against 14 selected component CIO responsibilities established in DHS policy; 15 selected leading workforce planning and management practices within 5 topic areas; and two selected leading industry project monitoring practices that, among other things, were, in GAO's professional judgment, of most significance to managing IITT. The U.S. Secret Service (Secret Service) Chief Information Officer (CIO) fully implemented 11 of 14 selected information technology (IT) oversight responsibilities, and partially implemented the remaining 3. The CIO partially implemented the responsibilities to establish a process that ensures the Secret Service reviews IT contracts; ensure that the component's IT policies align with the Department of Homeland Security's (DHS) policies; and set incremental targets to monitor program progress. Additional efforts to fully implement these 3 responsibilities will further position the CIO to effectively manage the IT portfolio. Of the 15 selected practices within the 5 workforce planning and management areas, the Secret Service fully implemented 3 practices, partly implemented 8, and did not implement 4 (see table). Within the strategic planning area, the component partly implemented the practice to, among other things, develop IT competency needs. While the Secret Service had defined general core competencies for its workforce, the Office of the CIO (OCIO) did not identify all of the technical competencies needed to support its functions. As a result, the office was limited in its ability to address any IT competency gaps that may exist. Also, while work remains to improve morale across the component, the Secret Service substantially implemented the employee morale practices for its IT staff. Secret Service officials said the gaps in implementing the workforce practices were due to, among other things, their focus on reorganizing the IT workforce within OCIO. Until the Secret Service fully implements these practices for its IT workforce, it may be limited in its ability to ensure the timely and effective acquisition and maintenance of the component's IT infrastructure and services. Of the two selected IT project monitoring practices, DHS and the Secret Service fully implemented the first practice to monitor the performance of the Information Integration and Technology Transformation (IITT) investment. In addition, for the second practice—to monitor projects on incremental development metrics—the Secret Service fully implemented the practice on one of IITT's projects and partially implemented it on another. In particular, OCIO did not fully measure post-deployment user satisfaction with the system on one project. OCIO plans to conduct a user satisfaction survey of the system by September 2018, which should inform the office on whether the system is meeting users' needs. GAO is making 13 recommendations, including that the Secret Service establish a process that ensures the CIO reviews all IT contracts, as appropriate; and identify the skills needed for its IT workforce. DHS concurred with all recommendations and provided estimated dates for implementing each of them.