U.S. Secret Service: Action Needed to Address Gaps in IT Workforce Planning and Management Practices
Released: 2018-11-15T00:00:00

Letter

Background

The Secret Service plays a critical role in protecting the President, Vice President, their immediate families, and national leaders, among others. In addition, the component is responsible for safeguarding the nation’s currency and financial payment systems. To accomplish its mission, Secret Service officials reported that, as of June 2018, the component had approximately 7,100 employees (including the Uniformed Division, special agents, and administrative, professional, and technical staff). These employees were assigned to the component’s headquarters in Washington, D.C., and 133 field offices located throughout the world (including 115 domestic offices and 18 international offices).

The Secret Service’s employees are heavily dependent on the component’s IT infrastructure and communications systems to perform their daily duties. According to data reported on the Office of Management and Budget’s IT Dashboard, the component planned to spend approximately $104.8 million in fiscal year 2018 to modernize and maintain its IT environment.

To manage this IT environment, the Secret Service hired a full-time CIO in November 2015. In addition, in an effort to improve its management structure, the component consolidated all IT staff and assets under this new CIO in March 2017. OCIO officials stated that these staff include the government employees who provide direct and indirect support of the day-to-day operations of the Secret Service’s enterprise systems and services.

According to Secret Service officials, the component’s IT workforce included 190 staff, as of July 2018. These officials stated that 166 of these employees were located in the component’s headquarters in Washington, D.C., and 24 were located in domestic field offices. The officials also reported that these July 2018 staffing levels were below their current approved staffing level of 220 staff (which included 44 positions in domestic field offices).

Secret Service IT staff also deploy to other locations, as necessary, to provide support for certain security activities. For example, the Secret Service reported that, in 2017, OCIO deployed over 79 staff to New York, N.Y., to provide communications support during the United Nations General Assembly.

DHS IT Acquisition Policies and Guidance

As a component of DHS, the Secret Service must follow the department’s policies and processes for managing acquisitions, including IT acquisitions. DHS categorizes its acquisition programs according to three levels that are determined by the life cycle costs of the programs. These levels then determine the extent of required program and project management and the acquisition decision authority (the individual responsible for management and oversight of the acquisition). The department also categorizes its acquisition programs as major or non- major based on expected cost. Table 1 describes the levels of DHS’s acquisition programs and their associated acquisition decision authorities.

DHS’s policies and processes for managing major acquisition programs are primarily set forth in its Acquisition Management Directive 102-01 and Acquisition Management Instruction 102-01-001. In particular, these policies establish that a major acquisition program’s decision authority is to review the program at a series of predetermined acquisition decision events to assess whether the program is ready to proceed through the acquisition life cycle phases. Figure 1 depicts the acquisition life cycle established in DHS acquisition management policy.

DHS’s Acquisition Management Directive and Instruction do not establish an acquisition life cycle framework for the department’s non-major acquisition programs. Instead, according to the Instruction, Component Acquisition Executives (i.e., the senior acquisition official within a component that is responsible for implementation, management, and oversight of the component’s acquisition process) are required to establish component-specific non-major acquisition policies and guidance that support the “spirit and intent” of the department’s acquisition policies.

To that end, the Secret Service developed a policy that establishes an acquisition life cycle framework for its non-major acquisition programs. This acquisition framework for the component’s non-major acquisition programs is consistent with the acquisition framework that DHS established for its major acquisition programs. In particular, the Secret Service’s framework includes the same phases and decision events as DHS’s framework (e.g., acquisition decision event 2A, the point at which the acquisition decision authority determines whether a program may proceed into the obtain phase).

In addition, DHS’s Systems Engineering Life Cycle Instruction and Guidebook outline a framework of major systems engineering activities and technical reviews that are to be conducted by all DHS programs and projects, both major and non-major. This framework is intended to ensure that appropriate systems engineering activities are planned and implemented, and that a program’s development effort is meeting the business need.

In particular, the systems engineering life cycle framework consists of nine major activities (e.g., requirements definition, integration, and testing) and a set of related technical reviews (e.g., preliminary design review) and artifacts (e.g., requirements documents). DHS policy allows programs to tailor these activities, technical reviews, and artifacts based on the unique characteristics of the program (e.g., scope, complexity, and risk). For example, a program may combine systems engineering technical reviews and artifacts, or add additional reviews. This tailored approach must be documented in a program’s systems engineering life cycle tailoring plan.

The systems engineering technical reviews are intended to provide DHS the opportunity to determine how well a program has completed the necessary systems engineering activities. Each technical review includes a minimum set of exit criteria that must be satisfied before a program may move on to the next systems engineering activity. At the end of the technical review, the program manager must develop a technical review completion letter that documents the outcome of the review, including stakeholder concurrence that the exit criteria were satisfied.

Moreover, DHS’s agile instruction, which was first issued in April 2016 and updated in April 2018, identifies agile as the preferred development approach for the department’s IT programs and projects. Agile is a type of incremental (i.e., modular) development, which calls for the rapid delivery of software in small, short increments rather than in the typically long, sequential phases of a traditional waterfall approach. DHS’s agile instruction also states that component CIOs are to set modular (i.e., incremental) outcomes and target measures to monitor progress in achieving agile implementation for IT programs and projects. To that end, the department identified core metrics that its agile IT programs are to use to monitor progress, including the number of story points completed per release and the number of releases per quarter.

Further, DHS policy and guidance have established an acquisition (i.e., contract) review process that is intended to enable the DHS CIO to review and effectively guide the department’s IT expenditures. According to the department’s IT acquisition review guidance, DHS components with a CIO (which includes the Secret Service) are to submit to DHS OCIO for review, IT acquisitions that (1) have total estimated procurement values of $2.5 million or more; and (2) are funded by a level 1, 2, or 3 program with a life cycle cost estimate of at least $50 million (i.e., a major investment, as defined by DHS’s capital planning and investment control guidance).

DHS Policies Outline Component-Level CIO Responsibilities

DHS policies and guidance also establish numerous responsibilities for the department’s component-level CIOs that are aimed at ensuring proper oversight and management of the components’ IT investments. Among other things, these component-level CIO responsibilities relate to topics such as IT budgeting, portfolio management, and oversight of programs’ systems engineering life cycles. Table 2 identifies 14 selected IT oversight responsibilities for DHS’s component CIOs.

Overview of the Secret Service’s IT Portfolio

The Secret Service acquires IT infrastructure and services that are intended to improve its ability to execute its investigation and protection missions. According to data reported on the Office of Management and Budget’s IT Dashboard, the Secret Service planned to spend about $104.8 million on IT in fiscal year 2018, which included approximately $34.6 million for the development and modernization of its IT infrastructure and services, and about $70.2 million for the operations and maintenance of this infrastructure (including 21 existing IT systems). Also according to data reported on the IT Dashboard, as of April 2018, the Secret Service had one major IT investment (called the Information Integration and Technology Transformation and discussed in more detail later in this report), seven non-major IT investments, and one non- standard infrastructure investment. Figure 2 depicts the Secret Service’s planned IT spending for fiscal year 2018.

The Secret Service Initiated the Information Integration and Technology Transformation Investment to Address IT Challenges

The Secret Service has faced long-standing challenges in managing its IT infrastructure. For example,

A National Security Agency audit of the Secret Service’s IT environment in 2008 identified network and system vulnerabilities that needed immediate remediation to protect the component’s systems and electronic information.

The Secret Service determined in 2010 that it had IT capability gaps associated with three key areas: network security, information sharing and situational awareness, and operational communications. The component reported that it required a significant IT modernization effort with sustained investment of resources to replace dated and restrictive network and communications capabilities.

The Secret Service also reported in 2010 that it had 42 mission- support applications that were operating on a 1980’s mainframe that lacked multi-level security (i.e., the ability to view classified information from two security levels, such as secret and top secret, at the same time), was beyond its equipment life cycle, and was at risk of failing.

Further, in 2011, DHS’s Office of Inspector General reported that the Secret Service’s existing infrastructure did not meet current operational requirements. According to the Secret Service, this dated infrastructure was unable to support newer technologies (e.g., Internet protocol), share common DHS enterprise services, or migrate to the department’s consolidated data centers.

To address challenges with its IT environment, in 2009, the Secret Service initiated the IITT investment, which is intended to modernize and enhance the component’s infrastructure, communications systems, applications, and processes. In particular, IITT is a portfolio of programs and projects that are meant to, among other things, improve systems availability in support of the Secret Service’s business operations, increase interoperability with other government systems and networks, enhance the component’s system and network security, and enable scalability to support growth.

From 2010 to July 2018, according to OCIO officials, the Secret Service spent approximately $392 million on IITT. In fiscal year 2018, the component had planned to spend approximately $42.7 million on IITT (i.e., about 40 percent of its total planned IT spending for the fiscal year), according to data reported on the Office of Management and Budget’s IT Dashboard. In total, the planned life cycle cost estimate for IITT is at least $811 million.

As of June 2018, IITT was a major investment comprised of two programs (one of which included three projects) and one standalone project (i.e., it was not part of another program) that had capabilities that were in planning or development and modernization. These programs and project were the Enabling Capabilities program, Enterprise Resource Management System program (which included three projects that were each being implemented using an agile methodology: Uniformed Division Resource Management System, Events Management, and Enterprise-wide Scheduling), and the Multi-Level Security project.

Table 3 describes the IITT programs and projects that had capabilities that were in planning or development and modernization, as of June 2018. The table also includes the associated level, acquisition decision authority, estimated life cycle costs, and planned or actual dates of operational capability for each of the programs and projects. (Appendix II also provides additional information on these programs and projects.)

The Enabling Capabilities program within IITT is designated as a major acquisition program. As such, its acquisition decision authority is the DHS Under Secretary for Management, and both DHS and the Secret Service provide oversight to this program. IITT’s other program and project—the Enterprise Resource Management System program (which includes three projects, as discussed earlier) and Multi-Level Security project—are designated non-major acquisition programs. In June 2011, DHS’s Under Secretary for Management delegated acquisition decision authority for this non-major program and project to the Secret Service Component Acquisition Executive. As such, oversight of the Enterprise Resource Management System program (including its three projects) and the Multi- Level Security project is conducted primarily at the component level.

The Secret Service also implemented other capabilities that are now in operations and maintenance (i.e., the capabilities have been fielded and are operational) as part of the IITT investment, such as a capability to move data between systems in separate classification levels (e.g., top secret and secret) and communications interoperability. Table 4 describes IITT capabilities that are in operations and maintenance.

DHS’s Management of Human Capital Is a High- Risk Effort

DHS, including the Secret Service, has faced long-standing challenges in effectively managing its workforce. In January 2003, we designated the implementation and transformation of DHS as high risk, including its management of human capital, because it had to transform 22 agencies—several with major management challenges—into one department. This represented an enormous and complex undertaking that would require time to achieve in an effective and efficient manner. Since that time, the department has made important progress in strengthening and integrating its management functions.

Nevertheless, we have continued to report that significant work remains for DHS to improve these management functions. Among other things, we previously reported that the department had lower average employee morale than the average for the rest of the federal government. We also reported that, in 2011, based on employee responses to the Office of Personnel Management’s Federal Employee Viewpoint Survey—a tool that measures employees’ perceptions of whether and to what extent conditions characterizing successful organizations are present in their agency—DHS was ranked 31st out of 33 large agencies on the Partnership for Public Service’s Best Places to Work in the Federal Government rankings. The most recent results of these surveys in 2017 showed that DHS continues to maintain its low rankings.

DHS’s Office of Inspector General has reported on challenges that the Secret Service has faced in managing its IT workforce. Specifically, in October 2016, the Inspector General reported that the Secret Service CIO did not have oversight of, or authority over, all IT resources, including the workforce; in particular, almost all of the component’s IT employees were located in a division outside of OCIO; and the Secret Service had vacancies in key positions responsible for managing IT, including not having a full-time CIO from December 2014 through November 2015.

As previously discussed, the Secret Service has taken actions to address these two issues with the management of its IT workforce. These actions included hiring its full-time CIO in November 2015 and consolidating the workforce and all IT assets under this CIO in March 2017.

The Secret Service CIO Fully Implemented Most of the Required Responsibilities

Of the 14 selected responsibilities established for component-level CIOs in DHS’s IT management policies, the Secret Service CIO had fully implemented 11 responsibilities and had partially implemented 3 responsibilities. Table 5 summarizes the extent to which the Secret Service CIO had implemented each of the 14 responsibilities.

The Secret Service CIO fully implemented 11 of the 14 selected component-level CIO responsibilities. Examples of the responsibilities that the CIO fully implemented are as follows:

Develop, implement, and maintain a detailed IT strategic plan. Consistent with DHS’s IT Integration and Management directive, in January 2017, the Secret Service CIO developed an IT strategic plan that outlined the CIO’s strategic IT goals and objectives, as well as tasks intended to meet the goals and objectives. The CIO maintained this strategic plan, to include updating it in January 2018. The CIO also took steps to implement the tasks identified within the strategic plan, such as working to develop an IT training program. In particular, as part of this effort to develop an IT training program, OCIO identified recommended training for the office’s various IT workforce groups (discussed in more detail later in this report).

Concur with each program’s and/or project’s systems engineering life cycle tailoring plan. In accordance with DHS’s Systems Engineering Life Cycle instruction, the Secret Service CIO concurred with the systems engineering life cycle tailoring plan for one program and three projects included in the Secret Service’s IITT investment. Specifically, the CIO documented his approval via his signature on the tailoring plans for IITT’s Enabling Capabilities program, and Multi-Level Security, Uniformed Division Resource Management System, and Events Management projects.

Participate on DHS’s CIO Council, Enterprise Architecture Board, or other councils/boards as appropriate, and appoint employees to serve when necessary. As required by DHS’s IT Integration and Management directive, the Secret Service CIO participated on two required DHS-level councils/boards, and appointed a delegate to serve in his place, when necessary. Specifically, the Secret Service CIO or the CIO’s delegate—the Deputy CIO—attended bi-monthly meetings of the DHS CIO Council. In addition, another Secret Service CIO appointee—the component’s Chief Architect—attended an ad hoc meeting of the Enterprise Architecture Board in June 2017.

In addition, the Secret Service CIO had partially implemented three component-level CIO responsibilities, as follows.

Manage the component IT investment portfolio, including establishing a component-level IT acquisition review process that enables component and DHS review of component acquisitions (i.e., contracts) that contain IT. As directed in DHS’s Capital Planning and Investment Control directive and guidebook, the Secret Service CIO took steps to manage the component’s IT investment portfolio, including reviewing certain contracts containing IT. For example, among our random sample of 33 IT contracts that the Secret Service awarded between October 1, 2016, and June 30, 2017, we found that the CIO or the CIO’s delegate had reviewed 31 of these contracts.

However, the CIO had not established and documented a defined process for reviewing contracts containing IT, which may have contributed to why the CIO or the CIO’s delegate did not review 2 of the 33 contracts in our sample. OCIO officials were unable to explain why neither of these officials reviewed the 2 contracts, which had a combined planned total procurement value of approximately $1.75 million. In particular, one of the contracts, with a planned total procurement value of about $1,122,934, was to provide credentialing services for the 2017 Presidential Inauguration. The other contract, with a planned total procurement value of about $629,337, was to provide maintenance support for a logistics system. The OCIO officials acknowledged that both contracts should have been approved by one of these officials. Without establishing and documenting an IT acquisition review process that ensures that the CIO or the CIO’s delegate reviews all contracts containing IT, as appropriate, the CIO’s ability to analyze the contracts to ensure that they are a cost-effective use of resources and are aligned with the component’s missions and goals is limited.

Ensure all component IT policies are in compliance and alignment with DHS IT directives and instructions. As required by DHS’s IT Integration and Management directive, the Secret Service CIO had ensured that certain component IT policies were in compliance and alignment with DHS IT directives and instructions. For example, in alignment with the department’s IT Integration and Management directive, the Secret Service’s Investment Governance for IT policy specifies that the component CIO (in conjunction with each Secret Service Office) is responsible for developing the component IT spend plan, as well as developing and maintaining an IT strategic plan.

However, the Secret Service’s enterprise governance policy was not in compliance with DHS’s IT Integration and Management directive. Specifically, while the department’s policy states that the Secret Service CIO is responsible for developing and reviewing the component’s IT budget formulation and execution, the Secret Service’s enterprise governance policy does not specify this as the CIO’s responsibility.

According to OCIO officials, the Secret Service CIO participates in the development and review of the IT budget formulation and execution as a member of the Executive Resources Board (the Secret Service’s highest-level governing body, which has the final decision authority and responsibility for enterprise governance), and the Secret Service Deputy CIO is a voting member of the Enterprise Governance Council (the Secret Service’s second-level governance body and advisory council to the Executive Resources Board). However, the Secret Service’s enterprise governance policy has not been updated to reflect these roles. The Secret Service did not update its enterprise governance policy to properly reflect the CIO’s and Deputy CIO’s roles on the Executive Resources Board or Enterprise Governance Council because OCIO officials were not aware that these roles were not properly documented in the component’s policy until we identified this issue during our review.

Further compounding the issue of the Secret Service’s enterprise governance policy not properly reflecting the CIO’s and Deputy CIO’s roles and responsibilities on the component’s governance boards is that the Secret Service has not developed a charter for its Executive Resources Board. We have previously reported that a best practice for effective investment management is to define and document the board’s membership, roles, and responsibilities. One such way to do so is via a charter.

According to Secret Service officials, the component does not have a charter for the board because, while the Secret Service has established the board pursuant to law, there is little statutory guidance on how the board must be formalized, including whether a charter is required. The officials acknowledged that development of a board charter is a best practice. They stated that, in response to our review, the component has begun efforts to develop a charter for the Executive Resources Board, but they did not know when it would be completed.

Until the Secret Service updates its enterprise governance policy to specify (1) the CIO’s current role and responsibilities on the Executive Resources Board, to include developing and reviewing the IT budget formulation and execution, and (2) the Deputy CIO’s role and responsibilities on the Enterprise Governance Council, the CIO’s ability to develop and review the component’s IT budget may be limited. Further, until the Secret Service develops a charter for its Executive Resources Board that specifies the roles and responsibilities of all board members, including the CIO, the Secret Service will not be effectively positioned to ensure that all members understand their roles and responsibilities on the board and will perform them as expected.

Set modular outcomes and target measures to monitor the progress in achieving agile implementation for IT programs and/or projects within their component. Consistent with DHS policy, the Secret Service CIO has set modular outcomes and target measures to monitor the progress of two IITT projects that the component is implementing using an agile methodology—Uniformed Division Resource Management System and Events Management. For example, the modular outcomes set for these projects included measuring planned and actual burndown (i.e., the number of user stories completed). In addition, the projects were to measure their velocity (i.e., the rate of work completed) for each sprint (i.e., a set period of time during which the development team is expected to complete tasks related to developing a piece of working software).

However, the modular outcomes and target measures did not include product quality or post-deployment user satisfaction, although such measures are leading practices for managing agile projects. According to Secret Service OCIO officials, the component does not mandate the specific metrics that its agile projects are to use; instead, each project is to determine the metrics based on stakeholder requirements and unique project characteristics. The officials further stated that these metrics are to be documented in an acquisition program baseline and program management plan; this baseline and program management plan are then to be approved by the CIO. To its credit, the component’s one agile project that, as of May 2018, had deployed its system to users—the Uniformed Division Resource Management System—did measure product quality. OCIO officials also stated that they regularly receive verbal, undocumented feedback from users on the system and they plan to conduct a documented user satisfaction survey on this system by September 2018.

Nevertheless, without ensuring that product quality and post- deployment user satisfaction metrics are included in the modular outcomes and target measures that the CIO sets for monitoring agile projects, the Secret Service lacks assurance that the Events Management project or other future agile projects will measure product quality or post-deployment user satisfaction. Without guidance specifying that agile projects track these metrics, the projects may not do so and the CIO may be limited in his knowledge of the progress being made on these projects.

The Secret Service Did Not Fully Implement the Majority of the Selected Leading Planning and Management Practices for Its IT Workforce

Workforce planning and management is essential for ensuring that federal agencies have the talent, skill, and experience mix they need to execute their missions and program goals. To help agencies effectively conduct workforce planning and management, the Office of Personnel Management, the Chief Human Capital Officers Council, DHS, the Secret Service, and we have identified numerous leading practices related to five workforce areas: strategic planning, recruitment and hiring, training and development, employee morale, and performance management. Table 6 identifies the five workforce areas and 15 selected leading practices associated with these areas (3 practices within each area).

Of the five selected workforce planning and management areas, the Secret Service had substantially implemented two of the areas and minimally implemented three of the areas for its IT workforce. In addition, of the 15 selected leading practices associated with these workforce planning and management areas, the Secret Service had fully implemented 3 practices, partly implemented 8 practices, and did not implement any aspects of 4 practices. Table 7 summarizes the extent to which the Secret Service had implemented for its IT workforce the five selected workforce planning and management areas and 15 selected leading practices associated with those areas, as of June 2018.

The Secret Service Minimally Implemented Selected Leading IT Strategic Workforce Planning Practices

Strategic workforce planning is an essential activity that an agency needs to conduct to ensure that its human capital program aligns with its current and emerging mission and programmatic goals, and that the agency is able to meet its future needs. We previously identified numerous leading practices related to IT strategic workforce planning, including that an organization should (1) establish and maintain a strategic workforce planning process, including developing all competency and staffing needs; (2) regularly assess competency and staffing needs, and analyze the IT workforce to identify gaps in those areas; and (3) develop strategies and plans to address gaps in competencies and staffing.

The Secret Service minimally implemented the three selected leading practices associated with the IT strategic workforce planning area. Specifically, the component partly implemented two of the practices and did not implement one practice. Table 8 lists these selected leading practices and provides our assessment of the Secret Service’s implementation of the practices.

Establish and maintain a strategic workforce planning process, including developing all competency and staffing needs—partly implemented. The Secret Service took steps to establish a strategic workforce planning process for its IT workforce. For example, the Secret Service CIO developed and maintained a plan that identified strategic workforce planning tasks, to include analyzing the staffing requirements of the IT workforce. In addition, the Secret Service defined general core competencies (e.g., communication and customer service) for its workforce, including IT staff.

However, OCIO did not identify all required knowledge and skills needed to support this office’s functions. In particular, while OCIO identified certain technical competencies that its IT workforce needs, such as cybersecurity, the office did not identify and document all of the technical competencies that it needs. OCIO officials stated that they did not identify and document the technical competencies that the office needs because the Secret Service was focused on reorganizing the IT workforce under a single, centralized reporting chain within the CIO’s office. Consequently, the officials stated that they had not completed the work to identify all required IT knowledge and skills necessary to support the office.

Yet, the Secret Service completed the IT workforce reorganization effort over a year ago, in March 2017 and, since then, OCIO has not identified all of the required IT knowledge and skills that the office needs. OCIO officials told us that they plan to identify all of the technical competency needs for the IT workforce, but they were unable to specify a time frame for when these needs would be fully identified. Until OCIO identifies all of the required knowledge and skills for the IT workforce, the office will be limited in its ability to identify and address any competency gaps associated with this workforce.

In addition, the Secret Service did not reliably determine the number of IT staff that it needs in order to support OCIO’s functions. Specifically, in January 2017, an independent review of the staffing model that the component used to identify its IT workforce staffing needs found that the model was not based on any verifiable underlying data. In late August 2018, Office of Human Resources officials reported that they had hired a contractor in early August 2018 to update the staffing model to improve the quality of the data. These officials expected the contractor to finish updating the model by August 2019. The officials plan to use the updated model to identify the Secret Service’s IT workforce staffing needs for fiscal year 2021. Updating the staffing model to incorporate verifiable workload data should increase the likelihood that the Secret Service is able to appropriately identify its staffing needs for its IT workforce.

Regularly assess competency and staffing needs, and analyze the IT workforce to identify gaps in those areas—not implemented. The Secret Service regularly assessed the competency and staffing needs for 1 of the occupational series within its IT workforce (i.e., the 2210 IT Specialist series). However, it did not regularly assess the competency and staffing needs for the remaining 11 occupational series that are associated with the component’s IT workforce, nor identify any gaps that it had in those areas.

OCIO officials stated that they had not assessed these needs or identified competency or staffing gaps because, among other things, the Secret Service was focused on reorganizing the IT workforce under a single, centralized reporting chain within the CIO’s office. However, as previously mentioned, the component completed this effort in March 2017, but OCIO did not subsequently assess its competency and staffing needs, nor identify gaps in those areas.

OCIO officials reported that they plan to assess the competencies of the IT workforce to identify any gaps that may exist; however, they were unable to identify a specific date by which they expect to have the capacity to complete this assessment. Until OCIO regularly analyzes the IT workforce to identify its competency needs and any gaps it may have, OCIO will be limited in its ability to determine whether its IT workforce has the necessary knowledge and skills to meet its mission and goals.

Further, Office of Human Resources officials reported that they plan to update the staffing model that they use to identify their IT staffing needs to include more reliable workload data. However, as discussed earlier, the Secret Service had not yet developed that updated model to determine its IT staffing needs. Office of Human Resources officials reported that once they update the staffing model they plan to re- evaluate the Secret Service’s IT staffing needs. The officials also stated that, going forward, they plan to reassess these needs each year as part of the annual budget cycle. Regular assessments of the IT workforce’s staffing needs should increase the likelihood that the Secret Service is able to appropriately identify the number of IT staff it needs to meet its mission and programmatic goals.

Develop strategies and plans to address gaps in competencies and staffing—partly implemented. The Secret Service developed recruiting and hiring strategies to address certain competency and staffing needs (e.g., cybersecurity) for its IT workforce. These strategies included, among other things, participating in DHS-wide recruiting events and using special hiring authorities.

However, because OCIO did not identify all of its IT competency and staffing needs, and lacked a current analysis of its entire IT workforce, the Secret Service could not provide assurance that the recruiting and hiring strategies it developed were specifically targeted towards addressing current OCIO competency and staffing gaps. For example, without an analysis of the IT workforce’s skills, OCIO did not know the extent to which it had gaps in areas such as device management and cloud computing. As a result, the Secret Service’s recruiting strategies may not have been targeted to address any gaps in those areas. Until the Secret Service updates its recruiting and hiring strategies and plans to address all IT competency and staffing gaps identified (after OCIO completes its analysis of the entire IT workforce, as discussed earlier), the Secret Service will be limited in its ability to effectively recruit and hire staff to fill those gaps.

The Secret Service Minimally Implemented Selected Leading Recruitment and Hiring Practices

According to the Office of Personnel Management, the Chief Human Capital Officers Council, and our prior work, once an agency has determined the critical skills and competencies that it needs to achieve programmatic goals, and identifies any competency or staffing gaps in its current workforce, the agency should be positioned to build effective recruiting and hiring programs. It is important that an agency has these programs in place to ensure that it can effectively recruit and hire employees with the appropriate skills to meet its various mission requirements.

The Office of Personnel Management, the Chief Human Capital Officers Council, and we have also identified numerous leading practices associated with effective recruitment and hiring programs. Among these practices, an agency should (1) implement recruiting and hiring activities to address skill and staffing gaps by using the strategies and plans developed during the strategic workforce planning process; (2) establish and track metrics to monitor the effectiveness of the recruitment program and hiring process, including their effectiveness at addressing skill and staffing gaps, and report to agency leadership on progress addressing those gaps; and (3) adjust recruitment plans and hiring activities based on recruitment and hiring effectiveness metrics.

The Secret Service minimally implemented the selected three leading practices associated with the recruitment and hiring workforce area. Specifically, the component partly implemented one of the three practices and did not implement the other two practices. Table 9 lists these selected practices and provides our assessment of the Secret Service’s implementation of the practices.

Implement recruiting and hiring activities to address skill and staffing gaps by using the strategies and plans developed during the strategic workforce planning process—partly implemented. OCIO officials implemented the activities identified in the Secret Service’s recruiting and hiring plans. For example, as identified in its recruiting plan, OCIO participated in a February 2017 career fair to recruit job applicants at a technology conference. In addition, in August 2017, OCIO participated in a DHS-wide recruiting event. Secret Service officials reported that, during this event, they conducted four interviews for positions in OCIO.

However, as previously discussed, OCIO did not identify all of its IT competency and staffing needs, and lacked a current analysis of its entire IT workforce. Without complete knowledge of its current IT competency and staffing gaps, the Secret Service could not provide assurance that the recruiting and hiring strategies that it had implemented fully addressed these gaps.

Establish and track metrics to monitor the effectiveness of the recruitment program and hiring process, including their effectiveness at addressing skill and staffing gaps, and report to agency leadership on progress addressing those gaps—not implemented. The Secret Service had not established and tracked metrics for monitoring the effectiveness of its recruitment and hiring activities for the IT workforce. Officials in the Office of Human Resources attributed this to staffing constraints and said their priority was to address existing staffing gaps associated with the Secret Service’s law enforcement groups.

In June 2018, Office of Human Resources officials stated that they plan to implement metrics to monitor the effectiveness of the hiring process for the IT workforce by October 2018. The officials also stated that they were in the process of determining (1) the metrics that are to be used to monitor the effectiveness of their workforce recruiting efforts and (2) whether they need to acquire new technology to support this effort. However, the officials did not know when they would implement the metrics for assessing the effectiveness of the recruitment activities and whether they would report the results to leadership.

Until the Office of Human Resources (1) develops and tracks metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce, including their effectiveness at addressing skill and staffing gaps; and (2) reports to component leadership on those metrics, the Secret Service and the Office of Human Resources will be limited in their ability to analyze the recruitment program to determine whether the program is effectively addressing IT skill and staffing gaps. Further, Secret Service leadership will lack the information necessary to make effective recruitment decisions.

Adjust recruitment plans and hiring activities based on recruitment and hiring effectiveness metrics—not implemented. While the Secret Service CIO stated in June 2018 that he planned to adjust the office’s recruiting and hiring strategies to focus on entry- level staff rather than mid-career employees, this planned adjustment was not based on metrics that the Secret Service was tracking. Instead, the CIO stated that he planned to make this change because his office determined that previous mid-career applicants were often unwilling or unable to wait for the Secret Service’s lengthy, required background investigation process to be completed.

However, as previously mentioned, the Secret Service did not develop and implement any metrics for assessing the effectiveness of the recruitment and hiring activities for the IT workforce. As a result, the Office of Human Resources and OCIO were not able to use such metrics to inform adjustments to their recruiting and hiring plan and activities, thus, reducing their ability to target potential candidates for hiring.

Until the Office of Human Resources and OCIO adjust their recruitment and hiring plans and activities as necessary, after establishing and tracking metrics for assessing the effectiveness of these activities for the IT workforce, the Secret Service will be limited in its ability to ensure that its recruiting plans and activities are appropriately targeted to potential candidates. In addition, the component will lack assurance that these plans and activities will effectively address skill and staffing gaps within its IT workforce.

The Secret Service Minimally Implemented Selected Leading Training and Development Practices

An organization should invest in training and developing its employees to help ensure that its workforce has the information, skills, and competencies that it needs to work effectively. In addition, training and development programs are an integral part of a learning environment that can enhance an organization’s ability to attract and retain employees with the skills and competencies needed to achieve cost-effective and timely results.

DHS, the Secret Service, and we have previously identified numerous leading training and development-related practices. Among those practices, an organization should (1) establish a training and development program to assist the agency in achieving its mission and goals; (2) use tracking and other control mechanisms to ensure that employees receive appropriate training and meet certification requirements, when applicable; and (3) collect and assess performance data (including qualitative or quantitative measures, as appropriate) to determine how the training program contributes to improved performance and results.

The Secret Service minimally implemented the selected three leading practices associated with the training and development workforce area. Specifically, the component partly implemented two of the three practices and did not implement one practice. Table 10 lists these selected leading practices and provides our assessment of the Secret Service’s implementation of the practices.

Establish a training and development program to assist the agency in achieving its mission and goals—partly implemented. OCIO was in the process of developing a training program for its IT workforce. For example, OCIO developed a draft training plan that identified recommended training for the office’s various IT workforce groups (e.g., voice communications employees).

However, the office had not defined the required training for each IT workforce group. In addition, OCIO officials had not yet determined which activities they would implement as part of the training program (e.g., soliciting employee feedback after training is completed and evaluating the effectiveness of specific training courses), nor did they implement those activities.

OCIO officials stated that they had not yet fully implemented a training program because their annual training budget for fiscal year 2018 was not sufficient to implement such a program. However, resource constrained programs especially benefit from identifying and prioritizing training activities to inform training budget decisions. Until OCIO (1) defines the required training for each IT workforce group, (2) determines the activities that it will include in its IT workforce training and development program based on its available training budget, and (3) implements those activities, the office may be limited in its ability to ensure that the IT workforce has the necessary knowledge and skills for their respective positions.

Use tracking and other control mechanisms to ensure that employees receive appropriate training and meet certification requirements, when applicable—partly implemented. OCIO used a training system to track that the managers for IITT’s programs had met certain certification requirements for their respective positions. In addition, OCIO manually tracked the technical training that certain IT staff took.

However, as discussed earlier, OCIO did not define the required training for each IT workforce group. As such, the office was unable to ensure that IT staff received the appropriate training relevant to their respective positions. Until it ensures that IT staff complete training specific to their positions (after defining the training required for each workforce group), OCIO will have limited assurance that the workforce has the necessary knowledge and skills.

Collect and assess performance data (including qualitative or quantitative measures, as appropriate) to determine how the training program contributes to improved performance and results—not implemented. As previously discussed, OCIO did not fully implement a training program for the IT workforce; as such, the office was unable to collect and assess performance data related to such a program. OCIO officials stated that, once they fully implement a training program, they intend to collect and assess data on how this program contributes to improved performance. However, the officials were unable to specify a time frame for when they would do so.

Until OCIO collects and assesses performance data (including qualitative or quantitative measures, as appropriate) to determine how the IT training program contributes to improved performance and results (once the training program is implemented), the office may be limited in its knowledge of whether the training program is contributing to improved performance and results.

The Secret Service Substantially Implemented Selected Leading Practices for Improving the Morale of Its IT Workforce, but Did Not Demonstrate Sustained Improvement

Employee morale is important to organizational performance and an organization’s ability to retain talent to perform its mission. We have previously identified numerous leading practices for improving employee morale. Among other things, we have found that an organization should (1) determine root causes of employee morale problems by analyzing employee survey results using techniques such as comparing demographic groups, benchmarking against similar organizations, and linking root cause findings to action plans; and develop and implement action plans to improve employee morale; (2) establish and track metrics of success for improving employee morale, and report to agency leadership on progress improving morale; and (3) maintain leadership support and commitment to ensure continued progress in improving employee morale, and demonstrate sustained improvement in morale.

With regard to its IT workforce, the Secret Service substantially implemented the selected three practices associated with the employee morale workforce area. Specifically, the component fully implemented two of the selected practices and partly implemented one practice. Table 11 lists these selected practices and provides our assessment of the Secret Service’s implementation of the practices.

Determine root causes of employee morale problems by analyzing employee survey results using techniques such as comparing demographic groups, benchmarking against similar organizations, and linking root cause findings to action plans. Develop and implement action plans to improve employee morale—fully implemented. The Secret Service used survey analysis techniques to determine the root causes of its low employee morale, on which we have previously reported. For example, the component conducted a benchmarking exercise where it compared the morale of the Secret Service’s employees, including IT staff, to data on the morale of employees at other agencies, including the U.S. Capitol Police, U.S. Coast Guard, and the Drug Enforcement Administration. As part of this exercise, the Secret Service also compared its employee work-life offerings (e.g., on-site childcare and telework program) to those available at other agencies.

In addition, the Secret Service developed and implemented action plans for improving employee morale. Among these action plans, for example, the component implemented a student loan repayment program and expanded its tuition assistance program’s eligibility requirements.

Establish and track metrics of success for improving employee morale, and report to agency leadership on progress improving morale—fully implemented. The Secret Service tracked metrics for improving employee morale and reported the results to leadership. For example, the component tracked metrics on the percentage of the workforce, including IT staff, that participated in the student loan repayment and tuition assistance programs. In addition, the Chief Strategy Officer reported to the Chief Operating Officer the results related to meeting those metrics.

Maintain leadership support and commitment to ensure continued progress in improving employee morale, and demonstrate sustained improvement in morale—partly implemented. Secret Service leadership developed and implemented initiatives that demonstrated their commitment to improving the morale of the Secret Service’s workforce. For example, since 2014, the Secret Service had worked with a contractor to identify ways to improve the morale of its entire workforce, including IT staff.

However, as of June 2018, the Secret Service was unable to demonstrate that it had sustained improvement in the morale of the component’s IT staff. In particular, the component was only able to provide IT workforce-specific results from one employee morale assessment that was conducted subsequent to the consolidation of this workforce into OCIO in March 2017. These results were from an assessment conducted by the component’s Inspection Division in December 2017 (the assessment found that the majority of the Secret Service’s IT employees rated their morale as “very good” or “excellent.”)

While the component also provided certain employee morale results from the Office of Personnel Management’s Federal Employee Viewpoint Survey in 2017, these results were not specific to the IT workforce. Instead, this workforce’s results were combined with those from staff in another Secret Service division. According to OCIO officials, the results were combined because, at the time of the survey, the IT workforce was administratively identified as being part of that other division.

OCIO officials stated that, going forward, they plan to continue to assess the morale of the IT workforce on an annual basis as part of the Federal Employee Viewpoint Survey. In addition, the officials stated that OCIO-specific results may be available as part of the 2018 survey results, which the officials expect to receive by September 2018. By measuring employee satisfaction on an annual basis, the Secret Service should have increased knowledge of whether its initiatives that are aimed at improving employee morale are in fact increasing employee satisfaction.

The Secret Service Substantially Implemented Selected Performance Management Leading Practices, but Did Not Explicitly Align Expectations with Organizational Goals

Agencies can use performance management systems as a tool to foster a results-oriented organizational culture that links individual performance to organizational goals. We have previously identified numerous leading practices related to performance management that are intended to enhance performance and ensure individual accountability. Among the performance management practices, agencies should (1) establish a performance management system that differentiates levels of staff performance and defines competencies in order to provide a fuller assessment of performance, (2) explicitly align individual performance expectations with organizational goals to help individuals see the connection between their daily activities and organizational goals, and (3) periodically provide individuals with regular performance feedback.

The Secret Service substantially implemented the selected three leading practices associated with the performance management workforce area. Specifically, the component fully implemented one of the three practices and partly implemented the other two practices. Table 12 lists these selected leading practices and provides our assessment of the Secret Service’s implementation of the practices.

Establish a performance management system that differentiates levels of staff performance and defines competencies in order to provide a fuller assessment of performance—partly implemented. The Secret Service’s performance management process requires leadership to make meaningful distinctions between levels of staff performance. In particular, the component’s performance plans for IT staff, which are developed by the Office of Human Resources and tailored by OCIO, as necessary, specify the criteria that leadership use to determine if an individual has met or exceeded the expectations associated with each competency identified in their respective performance plan. The performance plans include pre-established, department-wide competencies that are set by DHS, as well as occupational series-specific goals that may be updated by the Secret Service.

However, because OCIO did not fully define and document all of its technical competency needs for the IT workforce, as discussed earlier, the Secret Service’s performance plans for IT staff did not include performance expectations related to the full set of technical competencies required for their respective positions. In addition, because OCIO officials were unable to specify a time frame for when they will identify all of the technical competency needs for the IT workforce (as previously discussed), the officials were also unable to specify a time frame for when they would update the IT workforce’s performance plans to include those relevant technical competencies.

Until OCIO updates the performance plans for each occupational series within the IT workforce to include the relevant technical competencies, once identified, against which IT staff performance should be assessed, the office will be limited in its ability to provide IT staff with a complete assessment of their performance. In addition, Secret Service management will have limited knowledge of the extent to which IT staff are meeting all relevant technical competencies.

Explicitly align individual performance expectations with organizational goals to help individuals see the connection between their daily activities and organizational goals—partly implemented. The Secret Service’s performance plans for IT staff identified certain goals that appeared to be related to organizational goals and objectives. For example, the performance plan for the Telecommunications Specialist occupational series (which is one of the series included in OCIO’s IT workforce) identified a goal for staff to support the voice, wireless, radio, satellite, and video systems serving the Secret Service’s protective and investigative mission. This performance plan goal appeared to be related to the component’s strategic goal on Advanced Technology, which included an objective to create the infrastructure needed to fulfill mission responsibilities.

However, the Secret Service was unable to provide documentation that explicitly showed how individual employee performance links to organizational goals, such as a mapping of the goals identified in employee performance plans to organizational goals. Specifically, while Office of Human Resources officials stated that each Secret Service directorate is responsible for ensuring that employee goals map to high-level organizational goals, OCIO officials stated that they did not complete this mapping. The officials were unable to explain why they did not align the goals in their employees’ performance plans to the component’s high-level goals.

According to the officials, the Secret Service is in the process of implementing a new automated tool that will require each office to explicitly align individual performance expectations to organizational goals. The officials stated that OCIO plans to use this tool to create employees’ fiscal year 2019 performance plans. By explicitly demonstrating how individual performance expectations align with organizational goals, the Secret Service’s IT staff should have a better understanding of how their daily activities contribute towards achieving the Secret Service’s goals.

Periodically provide individuals with regular performance feedback—fully implemented. Secret Service leadership periodically provided their IT staff with performance feedback. Specifically, on an annual basis, OCIO staff received feedback during a mid-year and end-of-year performance feedback assessment. In our prior work, we have stressed that candid and constructive feedback can help individuals maximize their contribution and potential for understanding and realizing the goals and objectives of an organization. Further, this feedback is one of the strongest drivers of employee engagement.

The Secret Service and DHS Implemented Selected Leading Monitoring Practices for the IITT Investment

According to leading practices of the Software Engineering Institute, effective program oversight includes monitoring program performance and conducting reviews at predetermined checkpoints or milestones. This is done by, among other things, comparing actual cost, schedule, and performance data with estimates in the program plan and identifying significant deviations from established targets or thresholds for acceptable performance levels.

In addition, the Software Engineering Institute previously identified leading practices for effectively monitoring the performance of agile projects. According to the Institute, agile development methods focus on delivering usable, working software frequently; as such, it is important to measure the value delivered during each iteration of these projects. To that end, the Institute reported that agile projects should be measured on velocity (i.e., number of story points completed per sprint or release), development progression (e.g., the number of user stories planned and accepted), product quality (e.g., number of defects), and post-deployment user satisfaction.

DHS and the Secret Service had fully implemented the selected leading practice for monitoring the performance of one program and three projects within the IITT investment, and conducting reviews of this program and these projects at predetermined checkpoints. In addition, with regard to the selected leading practice for monitoring agile projects, the Secret Service had fully implemented this practice for one of its two projects being implemented using agile and had partially implemented this practice for the other project. Table 13 provides a summary of DHS’s and the Secret Service’s implementation of these leading practices, as relevant for one program and three projects within IITT.

Monitor program performance and conduct reviews at predetermined checkpoints or milestones. Consistent with leading practices, DHS and the Secret Service monitored the performance of IITT’s program and projects by comparing actual cost, schedule, and performance information against planned targets and conducting reviews at predetermined checkpoints. For example, within the Secret Service:

The Enabling Capabilities program and Multi-Level Security project monitored their contractors’ costs spent to-date on a monthly basis and compared them to the total contract amounts.

OCIO used integrated master schedules to monitor the schedule performance of the Enabling Capabilities program and Multi-Level Security project.

OCIO also monitored the cost, schedule, and performance of the Uniformed Division Resource Management System and Events Management projects during monthly status reviews.

In addition, DHS and the Secret Service conducted acquisition decision event reviews and systems engineering life cycle technical reviews of IITT’s program and projects at predetermined checkpoints and, when applicable, identified deviations from established cost, schedule, and performance targets. For example:

Secret Service OCIO met with DHS’s Office of Program Accountability and Risk Management in February 2017, and with DHS’s Acting Under Secretary for Management in June 2017, to discuss a schedule breach for the Enabling Capabilities program. In particular, the Enabling Capabilities program informed DHS that the program needed to change the planned date for acquisition decision event 3 (the point at which a decision is made to fully deploy the system) in order to conduct tests in an operational environment prior to that decision event. This delay was due to the Secret Service misunderstanding the tests that it was required to conduct prior to that decision event. Specifically, the Enabling Capabilities program had conducted tests on “production representative” systems, but these tests were not sufficient to meet the requirements for acquisition decision event 3.

The project team for Multi-Level Security identified that certain technical issues they had experienced would delay system deployment and full operational capability (the point at which an investment becomes fully operational). As such, in October 2017, the project notified the Secret Service Component Acquisition Executive of these expected delays. In particular, the web browser that was intended to provide users on “Sensitive But Unclassified” workstations the ability to view information from different security levels, experienced technical delays in meeting personal identity verification requirements. The project team also described for the executive how the schedule delay would affect the project’s performance metrics and funding, and subsequently updated the project plan accordingly.

Measure and monitor agile projects on, among other things, velocity (i.e., number of story points completed per sprint or release), development progression (e.g., the number of features and user stories planned and accepted), product quality (e.g., number of defects), and post-deployment user satisfaction. Secret Service OCIO measured its two agile projects—Uniformed Division Resource Management System and Events Management— using certain agile metrics. In particular, OCIO officials measured the Uniformed Division Resource Management System and Events Management projects using key metrics related to velocity and development progression. For example, the officials measured development progression for both projects on a daily basis. In addition, OCIO officials monitored each project’s progress against these metrics during bi-weekly reviews that they conducted with each project team.

The OCIO officials also tracked product quality metrics for the Uniformed Division Resource Management System. For example, on a monthly basis, the officials tracked the number of helpdesk tickets that had been resolved related to the system. In addition, on a quarterly basis, they tracked the number of Uniformed Division Resource Management System defects that (1) had been fixed and (2) were in the backlog.

However, while OCIO officials received certain post-deployment user satisfaction information from end-users of the Uniformed Division Resource Management System by, among other things, tracking the number of helpdesk tickets related to the system and via daily verbal, undocumented feedback from certain Uniformed Division officers, OCIO officials had not fully measured and documented post- deployment user satisfaction with the system, such as via a survey of employees who use the system. The officials stated that they had not conducted and documented a survey because they were focused on (1) addressing software performance issues that occurred after they deployed the system to a limited number of users, and (2) continuing system deployment to the remaining users after they addressed the performance issues.

OCIO officials stated that they plan to conduct such a documented survey by the end of September 2018. The results of the user satisfaction survey should provide OCIO with important information on whether the Uniformed Division Resource Management System is meeting users’ needs.

Conclusions

The Secret Service’s full implementation of 11 of 14 component-level CIO responsibilities constitutes a significant effort to establish CIO oversight for the component’s IT portfolio. Additional efforts to fully implement the remaining 3 responsibilities, including ensuring that all IT contracts are reviewed, as appropriate; ensuring that the Secret Service’s enterprise governance policy appropriately specifies the CIO’s role in developing and reviewing the component’s IT budget formulation and execution; and ensuring agile projects measure product quality and post-deployment user satisfaction, will further position the CIO to effectively manage the Secret Service’s IT portfolio.

When effectively implemented, IT workforce planning and management activities can facilitate the successful accomplishment of an agency’s mission. However, the Secret Service had not fully implemented all of the 15 selected practices for its IT workforce for any of the five areas— strategic planning, recruitment and hiring, training and development, employee morale, and performance management. The Secret Service’s lack of (1) a strategic workforce planning process, including the identification of all required knowledge and skills, assessment of competency gaps, and targeted strategies to address specific gaps in competencies and staffing; (2) targeted recruiting activities, including metrics to monitor the effectiveness of the recruitment program and adjustment of the recruitment program and hiring efforts based on metrics; (3) a training program, including the identification of required training for IT staff, ensuring that staff take required training, and assessment of performance data regarding the training program; and (4) a performance management system that includes all relevant technical competencies, greatly limits its ability to ensure the timely and effective acquisition and maintenance of the Secret Service’s IT infrastructure and services.

On the other hand, by monitoring program performance and conducting reviews at predetermined checkpoints for one program and three projects associated with the IITT investment, in accordance with leading practices, the Secret Service and DHS provided important oversight needed to guide that program and those projects. Measuring projects on leading agile metrics also provided the Secret Service CIO with important information on project performance.

Recommendations for Executive Action

We are making the following 13 recommendations to the Director of the Secret Service: The Director should ensure that the CIO establishes and documents an IT acquisition review process that ensures the CIO or the CIO’s delegate reviews all contracts containing IT, as appropriate. (Recommendation 1)

The Director should update the enterprise governance policy to specify (1) the CIO’s current role and responsibilities on the Executive Resources Board, to include developing and reviewing the IT budget formulation and execution; and (2) the Deputy CIO’s role and responsibilities on the Enterprise Governance Council. (Recommendation 2)

The Director should ensure that the Secret Service develops a charter for its Executive Resources Board that specifies the roles and responsibilities of all board members, including the CIO. (Recommendation 3)

The Director should ensure that the CIO includes product quality and post-deployment user satisfaction metrics in the modular outcomes and target measures that the CIO sets for monitoring agile projects. (Recommendation 4)

The Director should ensure that the CIO identifies all of the required knowledge and skills for the IT workforce. (Recommendation 5)

The Director should ensure that the CIO regularly analyzes the IT workforce to identify its competency needs and any gaps it may have. (Recommendation 6)

The Director should ensure that, after OCIO completes an analysis of the IT workforce to identify any competency and staffing gaps it may have, the Secret Service updates its recruiting and hiring strategies and plans to address those gaps, as necessary. (Recommendation 7)

The Director should ensure that the Office of Human Resources (1) develops and tracks metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce, including their effectiveness at addressing skill and staffing gaps; and (2) reports to component leadership on those metrics. (Recommendation 8)

The Director should ensure that the Office of Human Resources and OCIO adjust their recruitment and hiring plans and activities, as necessary, after establishing and tracking metrics for assessing the effectiveness of these activities for the IT workforce. (Recommendation 9)

The Director should ensure that the CIO (1) defines the required training for each IT workforce group, (2) determines the activities that OCIO will include in its IT workforce training and development program based on its available training budget, and (3) implements those activities. (Recommendation 10)

The Director should ensure that the CIO ensures that the IT workforce completes training specific to their positions (after defining the training required for each workforce group). (Recommendation 11)

The Director should ensure that the CIO collects and assesses performance data (including qualitative or quantitative measures, as appropriate) to determine how the IT training program contributes to improved performance and results (once the training program is implemented). (Recommendation 12)

The Director should ensure that the CIO updates the performance plans for each occupational series within the IT workforce to include the relevant technical competencies, once identified, against which IT staff performance should be assessed. (Recommendation 13)

Agency Comments and Our Evaluation

DHS provided written comments on a draft of this report, which are reprinted in appendix III. In its comments, the department concurred with all 13 of our recommendations and provided estimated completion dates for implementing each of them.

For example, with regard to recommendation 2, the department stated that the Secret Service would update its enterprise governance policy and related policies to outline the roles and responsibilities of the CIO and Deputy CIO, among others, by March 31, 2019. In addition, for recommendation 13, the department stated that the Secret Service OCIO will include relevant technical competencies in performance plans, as appropriate, in the next performance cycle that starts in July 2019. If implemented effectively, these actions should address the weaknesses we identified.

The department also identified a number of other actions that it said had been taken to address our recommendations. For example, in response to recommendation 8, which calls for the Office of Human Resources to (1) develop and track metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce and (2) report to component leadership on those metrics, DHS stated that the Secret Service’s Office of Human Resources’ Outreach Branch provides to the department metrics on recruitment efforts toward designated priority mission-critical occupations.

However, for fiscal year 2017, only 1 of the 12 occupational series associated with the Secret Service’s IT workforce was designated as a mission-critical occupation for the component (i.e., the 2210 IT Specialist series). The 11 other occupational series were not designated as mission- critical occupations. In addition, for fiscal year 2018, none of these 12 occupational series were designated as mission-critical occupations. As such, metrics on recruiting for these IT series may not have been reported to DHS leadership.

Moreover, while we requested documentation of the recruiting metrics for the Secret Service’s IT workforce and, during the course of our review, had multiple subsequent discussions with the Secret Service regarding such metrics, the component did not provide documentation that demonstrated it had established recruiting metrics for its IT workforce. Tracking such metrics and reporting the results to Secret Service leadership, as we recommended, would provide management with important information necessary to make effective recruitment decisions.

Further, in response to recommendation 10, which among other things, calls for the CIO to define the required training for each IT workforce group, the department stated that the Secret Service OCIO recently developed training requirements for each workforce group, which were issued during our audit. However, while during our audit OCIO provided a list of recommended training courses, the office did not identify them as being required courses. Defining training that is required for each IT workforce group, as we recommended, would inform OCIO of the necessary training for each position and enable the office to prioritize this training, to ensure that its staff have the needed knowledge and skills.

In addition to the aforementioned comments, we received technical comments from DHS and Secret Service officials, which we incorporated, as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Director of the Secret Service, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov.

Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or HarrisCC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to evaluate the extent to which: (1) the U.S. Secret Service (Secret Service) Chief Information Officer (CIO) has implemented selected information technology (IT) oversight responsibilities, (2) the Secret Service has implemented leading workforce planning and management practices for its IT workforce, and (3) the Secret Service and the Department of Homeland Security (DHS) have implemented selected performance and progress monitoring practices for the Information Integration and Technology Transformation (IITT) investment.

To address the first objective, we analyzed DHS’s policies and guidance on IT management to identify the responsibilities that were to be implemented by the component-level CIO related to overseeing the Secret Service’s IT portfolio, including existing systems, acquisitions, and investments. From the list of 33 responsibilities that we identified, we then excluded the responsibility that was associated with information security, which is expected to be addressed as part of a separate, subsequent GAO review. We also excluded those responsibilities that were significantly large in scope (e.g., implement an enterprise architecture) or that, in our professional judgment, lacked specificity (e.g., provide timely delivery of mission IT services). As a result, we excluded from consideration for this review a total of 10 CIO responsibilities.

For the 23 that remained, we then combined certain responsibilities that overlapped with other related responsibilities. For example, we combined related responsibilities on the component CIO’s review of IT contracts. As a result, we identified 14 responsibilities for review. We validated with the acting DHS CIO that these responsibilities were key responsibilities for the department’s component-level CIOs. We then included all 14 of the responsibilities in our review.

The 14 selected component-level CIO responsibilities were: 1. Develop and review the component IT budget formulation and execution. 2. Manage the component IT investment portfolio, including establishing an IT acquisition review process that enables component and DHS review of component acquisitions (i.e., contracts) that contain IT. 3. Develop, implement, and maintain a detailed IT strategic plan. 4. Ensure all component IT policies are in compliance and alignment with DHS IT directives and instructions. 5. Concur with each program’s and/or project’s systems engineering life cycle tailoring plan. 6. Support the Component Acquisition Executive to ensure processes are established that enable systems engineering life cycle technical reviews and that they are adhered to by programs and/or projects. 7. Ensure that all systems engineering life cycle technical review exit criteria are satisfied for each of the component’s IT programs and/or projects. 8. Ensure the necessary systems engineering life cycle activities have been satisfactorily completed as planned for each of the component’s IT programs and/or projects. 9. Concur with the systems engineering life cycle technical review completion letter for each of the component’s IT programs and/or projects. 10. Maintain oversight of their component’s agile development approach for IT by appointing the responsible personnel, identifying investments for adoption, and reviewing artifacts. 11. With Component Acquisition Executives, evaluate and approve the application of agile development for IT programs consistent with the component’s agile development approach. 12. Set modular outcomes and target measures to monitor the progress in achieving agile implementation for IT programs and/or projects within their component. 13. Participate on DHS’s CIO Council, Enterprise Architecture Board, or other councils/boards as appropriate, and appoint employees to serve when necessary. 14. Meet the IT competency requirements established by the DHS CIO, as required in the component CIO’s performance plan.

To determine the extent to which the Secret Service CIO has implemented these responsibilities, we obtained and assessed relevant component documentation and compared it to the responsibilities. Specifically, we obtained and analyzed documentation including evidence of the CIO’s participation on the Secret Service governance board that has final decision authority and responsibility for enterprise governance, including the IT budget; monthly program management reports showing the CIO’s oversight of IT programs, projects, and systems; monthly status reports on program spending; the Secret Service’s IT strategic plan; the Secret Service’s enterprise governance policy; meeting minutes from the DHS board and council on which the CIO participated (i.e., the CIO Council and Enterprise Architecture Board); and documentation demonstrating whether the CIO met the IT competency requirements.

In addition, we obtained and analyzed relevant documentation related to the CIO’s oversight of the major IT investments on which the Secret Service was spending development, modernization, and enhancement funds during fiscal year 2017. As of July 2017, the component had one investment—IITT—that met this criterion. IITT is a portfolio investment that, as of July 2017, included two programs (one of which included three projects) and one standalone project (i.e., it was not part of another program) that had capabilities that were in planning or development and modernization: the Enabling Capabilities program, Enterprise Resource Management System program (which included three projects, called Uniformed Division Resource Management System, Events Management, and Enterprise-wide Scheduling), and Multi-Level Security project.

In particular, we obtained and analyzed documentation related to the CIO’s oversight of the systems engineering life cycles for IITT’s Enabling Capabilities program and the Uniformed Division Resource Management System, Events Management, and Multi-Level Security projects. This documentation included acquisition program baselines, systems engineering life cycle tailoring plans, and systems engineering life cycle technical review briefings and completion letters. We then compared the documentation against the five selected systems engineering life cycle oversight responsibilities (responsibilities 5, 6, 7, 8, and 9).

We also obtained and analyzed documentation related to the CIO’s oversight of two projects that the Secret Service was implementing using an agile methodology—Uniformed Division Resource Management System and Events Management. Specifically, we obtained and assessed documentation of (1) the CIO’s approval for these projects to be implemented using an agile methodology and (2) the agile development metrics that the CIO established for each of these projects. We then compared this documentation to the three agile development-related component-level CIO responsibilities (responsibilities 10, 11, and 12).

Further, to determine the extent to which the Secret Service CIO had established an IT acquisition (i.e., contract) review process that enabled component and DHS review of component contracts that contain IT (which is part of responsibility 2), we first asked Secret Service officials to provide us with a list of all new, unclassified IT contracts that the component awarded between October 1, 2016, and June 30, 2017. The Secret Service officials provided a list of 54 contracts. We validated that these were contracts for IT or IT services by: (1) searching for them in the Federal Procurement Data System – Next Generation; (2) identifying their associated product or service codes, as reported in that system; and (3) determining whether those codes were included in the universe of 79 IT product or service codes identified by the Category Management Leadership Council.

In validating the list of 54 contracts provided by the Secret Service, we determined that 5 of the contracts were not associated with an IT product or service code. As such, we removed those contracts from the list. In addition, we found that three other items identified by the component were not in the Federal Procurement Data System – Next Generation. Secret Service officials subsequently confirmed that these three items were not contracts. We therefore removed these three items from the list. As such, the final list of validated contracts identified by the Secret Service included 46 IT contracts.

In addition, to identify any IT contracts that were not included in the list provided by the Secret Service, we conducted a search of the Federal Procurement Data System – Next Generation to identify all unclassified contracts that (1) the component awarded between October 1, 2016, and June 30, 2017; (2) were not a modification of a contract; and (3) were associated with 1 of the 79 IT product or service codes identified by the Category Management Leadership Council. Based on these criteria, we identified 144 Secret Service IT contracts in the Federal Procurement Data System – Next Generation (these 144 contracts included the 46 contracts previously identified by Secret Service officials). We then asked Secret Service officials to validate the accuracy, completeness, and reliability of these data, which they did.

From each of these two lists of IT contracts (i.e., the list of 46 IT contracts identified by the Secret Service and the list of 144 IT contracts that we identified from the Federal Procurement Data System – Next Generation), we then selected random, non-generalizable samples of contracts, as described below.

First, from the list of 46 IT contracts identified by Secret Service officials, we removed 4 contracts that had total values of less than $10,000. To ensure that we selected across all contract sizes, we randomly selected 12 contracts from the remaining list of 42 contracts, using the following cost ranges: $10,000 to $50,000 (4 contracts), more than $50,000 to less than $250,000 (4 contracts), and more than $250,000 (4 contracts).

Second, from our list of 144 IT contracts that we identified from the Federal Procurement Data System – Next Generation, we removed the 46 contracts identified by Secret Service officials. We also removed 12 contracts that had total values of less than $10,000. To ensure that we selected across all contract sizes, we randomly selected 21 contracts from the remaining list of 86 contracts, using the following cost ranges: $10,000 to $50,000 (7 contracts), more than $50,000 to less than $250,000 (7 contracts), and more than $250,000 (7 contracts).

In total, we selected 33 IT contracts for review. We separated the contracts into the three cost ranges identified above in order to ensure that contracts of different value levels had been selected. This enabled us to determine the extent to which the CIO appropriately reviewed contracts of all values.

To determine the extent to which the CIO had established an IT contract approval process that enabled the Secret Service and DHS, as appropriate, to review IT contracts, we first asked Secret Service Office of the CIO (OCIO) officials for documentation of their IT contract approval process. These officials were unable to provide such documentation. Instead, the officials stated that the Secret Service CIO or the CIO’s delegate approves all IT contracts prior to award. The officials also provided documentation that identified four staff to whom the CIO had delegated his approval authority. Further, the officials stated that, in accordance with DHS’s October 2016 IT acquisition review guidance, they submitted to DHS OCIO for approval any IT contracts that met DHS’s thresholds for review, including those that (1) had total estimated procurement values of $2.5 million or more, and (2) were associated with a major investment.

Based on the IT acquisition review process that Secret Service OCIO officials described, we then obtained and analyzed each of the 33 selected IT contracts and associated approval documentation to determine whether or not the Secret Service CIO or the CIO’s delegate had approved each of the contracts. In particular, we (1) reviewed the name of the contract approver on the approval documentation, and (2) compared the signature dates that were on the contracts to the signature dates that were identified on the associated approval documentation.

In addition, to determine whether or not the Secret Service CIO submitted to DHS OCIO for approval the IT contracts that (1) had total estimated procurement values of $2.5 million or more, and (2) were associated with major investments, we first analyzed the 144 Secret Service IT contracts that we had previously pulled from the Federal Procurement Data System – Next Generation to determine which contracts met the $2.5 million threshold. We identified 4 contracts that met this threshold. We then requested that OCIO identify the levels (i.e., major or non-major) of the investments associated with these contracts. According to OCIO officials, 3 of the 4 contracts were associated with non-major investments and 1 was not associated with an investment. As such, based on DHS’s October 2016 IT acquisition review guidance, none of these contracts needed to be submitted to DHS OCIO for review.

We also interviewed Secret Service officials, including the CIO and Deputy CIO, regarding the CIO’s implementation of the 14 selected component-level responsibilities. We assessed the evidence against the selected responsibilities to determine the extent to which the CIO had implemented them.

To address the second objective—determining the extent to which the Secret Service had implemented leading workforce planning and management practices for its IT workforce—we first identified seven topic areas associated with human capital management based on the following sources:

The Office of Personnel Management’s Human Capital Framework.

Office of Personnel Management and the Chief Human Capital Officers Council Subcommittee for Hiring and Succession Planning, End-to-End Hiring Initiative.

GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others.

GAO, IT Workforce: Key Practices Help Ensure Strong Integrated Program Teams; Selected Departments Need to Assess Skill Gaps.

GAO, Department of Homeland Security: Taking Further Action to Better Determine Causes of Morale Problems Would Assist in Targeting Action Plans.

GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government.

GAO, Results-Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success.

DHS acquisition guidance.

Secret Service acquisition guidance.

Among these topic areas, we then selected five areas that, in our professional judgment, were of particular importance to successful workforce planning and management. They were also previously identified as part of our high-risk and key issues work on human capital management. These areas include: (1) strategic planning, (2) recruitment and hiring, (3) training and development, (4) employee morale, and (5) performance management.

We also reviewed these same sources and identified numerous leading practices associated with the five topic areas. Among these leading practices, we then selected three leading practices within each of the five areas (for a total of 15 selected practices). The selected practices were foundational practices that, in our professional judgment, were of particular importance to successful workforce planning and management.

Table 14 identifies the five selected workforce areas and 15 selected associated practices.

To determine the extent to which the Secret Service had implemented the selected leading workforce planning and management practices for its IT workforce, we obtained and assessed documentation and compared it against the 15 selected practices. In particular, we analyzed the Secret Service’s human capital strategic plan, human capital staffing plan, IT strategic plan, documentation of the component’s staffing model that it used to determine the number of IT staff needed, an independent verification and validation report on the component’s staffing models, documentation of the current number of IT staff, the Secret Service’s recruitment and outreach plans, documentation of DHS’s hiring authorities (which are applicable to the Secret Service), the Secret Service’s training strategic plan, IT workforce training plan, action plans for improving employee morale, and templates used for measuring and reporting employee performance.

We also interviewed Secret Service officials—including the CIO, Deputy CIO, and workforce planning staff—about the component’s workforce- related policies and documentation. Further, we discussed with the officials the Secret Service’s efforts to implement the selected workforce practices for its IT workforce.

Regarding our assessments of the Secret Service’s implementation of the 15 selected workforce planning and management practices, we assessed a practice as being fully implemented if component officials provided supporting documentation that demonstrated all aspects of the practice. We assessed a practice as not implemented if the officials did not provide any supporting documentation for that practice, or if the documentation provided did not demonstrate any aspect of the practice. We assessed a practice as being partly implemented if the officials provided supporting documentation that demonstrated some, but not all, aspects of the selected practice.

In addition, related to our assessments of the Secret Service’s implementation of the five selected overall workforce areas, we assessed each area as follows, based on the implementation of the three selected practices within each area:

Fully implemented: The Secret Service provided evidence that it had fully implemented all three of the selected practices within the workforce area;

Substantially implemented: The Secret Service provided evidence that it had either fully implemented two selected practices and partly implemented the remaining one selected practice within the workforce area, or fully implemented one selected practice and partly implemented the remaining two selected practices within the workforce area;

Partially implemented: The Secret Service provided evidence that it had partly implemented each of the three selected practices within the workforce area;

Minimally implemented: The Secret Service provided evidence that it partly implemented two selected practices and not implemented the remaining one selected practice within the workforce area, or partly implemented one selected practice and not implemented the remaining two selected practices within the workforce area; or

Not implemented: The Secret Service did not provide evidence that it had implemented any of the three selected practices within the workforce area.

To address the third objective—determining the extent to which the Secret Service and DHS have implemented selected performance and progress monitoring practices for IITT—we reviewed leading project monitoring practices and guidance from the Software Engineering Institute. First, we reviewed the practices within the Project Monitoring and Control process area of the Institute’s Capability Maturity Model Integration® for Acquisition. Based on our review, we identified four practices associated with monitoring program performance and progress. In our professional judgment, all four of these practices were of significance to managing the IITT investment given the phase of the life cycle that the investment was in. As such, we elected to include all four of these practices in our review, and combined them into one practice, as follows:

Monitor program performance and conduct reviews at predetermined checkpoints or milestones by, among other things, comparing actual cost, schedule, and performance data with estimates in the program plan and identifying significant deviations from established targets or thresholds for acceptable performance levels.

Next, given the agile development methodology that the Secret Service was using for certain projects within IITT, we reviewed the Software Engineering Institute’s technical note on the progress monitoring of agile contractors. Based on our review, and in consultation with an internal expert, we selected four agile metrics that the Institute identified as important for successful agile implementations and that, in our professional judgment, were of most significance to monitoring the performance of IITT’s agile projects. We then combined these four metrics into one practice, as follows:

Measure and monitor agile projects on velocity (i.e., number of story points completed per sprint or release), development progression (e.g., the number of features and user stories planned and accepted), product quality (e.g., number of defects), and post-deployment user satisfaction.

To determine the extent to which DHS and the Secret Service had implemented the first selected practice, we analyzed relevant program management and governance documentation for IITT’s Enabling Capabilities program, and Multi-Level Security, Uniformed Division Resource Management System, and Events Management projects. In particular, we analyzed acquisition program baselines, DHS acquisition decision event memorandums, artifacts from DHS and Secret Service program oversight reviews, cost monitoring reports, program integrated master schedules, and program status briefings, and compared this documentation to the selected practice. We also interviewed Secret Service OCIO officials regarding the Secret Service’s and DHS’s efforts to monitor the IITT investment’s performance and progress.

To determine the extent to which the Secret Service had implemented the second selected practice related to measuring and monitoring agile projects on agile metrics (i.e., velocity, development progression, product quality, and post-deployment user satisfaction), we obtained and analyzed agile-related documentation for the two projects that the Secret Service was implementing using an agile methodology—Uniformed Division Resource Management System and Events Management. Specifically, to determine the extent to which the Secret Service was measuring and monitoring these two projects on metrics for velocity and development progression, we obtained and analyzed documentation, such as sprint burndown charts and monthly program status reports, and compared it to the selected practice.

In addition, the agile metrics for product quality and post-deployment user satisfaction were only applicable to projects that had been deployed to users. As such, these metrics were applicable to the Uniformed Division Resource Management System (which the Secret Service had deployed to users) and were not applicable to Events Management (which the Secret Service had not yet deployed to users, as of early May 2018).

We therefore obtained and analyzed documentation demonstrating that Secret Service OCIO measured product defects for the Uniformed Division Resource Management System. We also requested documentation demonstrating that OCIO had measured and monitored post-deployment user satisfaction for this project, including via a survey. OCIO officials stated that they had not conducted such a survey and were unable to provide documentation demonstrating they had measured post- deployment user satisfaction for the Uniformed Division Resource Management System.

To assess the reliability of the cost, schedule, and agile-related data that were in DHS and the Secret Service’s program management and governance documentation for the IITT investment, we (1) analyzed related documentation and assessed the data against existing agency records to identify consistency in the information, and (2) examined the data for obvious outliers, incomplete, or unusual entries. We determined that the data in these documents were sufficiently reliable for our purpose, which was to evaluate the extent to which DHS and the Secret Service had implemented processes for monitoring the IITT investment’s performance and progress.

We conducted this performance audit from May 2017 to November 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Description of the U.S. Secret Service’s Information Integration and Technology Transformation Investment’s Programs and Projects

As of June 2018, the Secret Service’s Information Integration and Technology Transformation (IITT) investment included two programs (one of which included three projects) and one project that had capabilities that were in planning or development and modernization, as described below:

Enabling Capabilities. This program is intended to, among other things, (1) modernize and enhance the Secret Service’s information technology (IT) network infrastructure, including increasing bandwidth and improving the speed and reliability of the Secret Service’s IT system performance; (2) enhance cybersecurity to protect against potential intrusions and viruses; and (3) provide counterintelligence and data mining capabilities to improve officials’ ability to perform the Secret Service’s investigative mission.

Enterprise Resource Management System. This program comprises three projects that are intended to provide: a system that will enable the Secret Service’s Uniformed Division to efficiently and effectively plan, provision, and schedule missions (this project is referred to as Uniformed Division Resource Management System), a system that will unify the logistical actions (e.g., assigning personnel) surrounding special events that Secret Service agents need to protect, such as the United Nations General Assembly (this project is referred to as Events Management), and a capability for creating schedules for Secret Service agents and administrative, professional, and technical staff, as well as the ability to generate reports on information such as monthly hours worked (this project is referred to as Enterprise-wide Scheduling).

Multi-Level Security. This project is intended to enable authorized Secret Service users to view two levels of classified information on a single workstation. Previously, data at various security levels were contained and used in multiple disparate systems. Multi-Level Security is intended to streamline users’ access to information at different security levels in order to enable them to more quickly and effectively perform their duties.

Table 15 provides the planned life cycle cost and schedule estimates (threshold values) for each IITT program and project that had capabilities in planning or development and modernization, as of June 2018. In addition, the table describes any changes in those cost and schedule estimates, as well as the key reasons for any changes, as identified by officials from the Secret Service’s Office of the Chief Information Officer.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff made key contributions to this report: Shannin O’Neill (Assistant Director), Emily Kuhn (Analyst-in-Charge), Quintin Dorsey, Rebecca Eyler, Javier Irizarry, and Paige Teigen.