Digital Service Programs: Assessing Results and Coordinating with Chief Information Officers Can Improve Delivery of Federal Projects
Published: 2016-08-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In an effort to improve IT across the federal government, in March 2014 GSA established 18F, which provides IT services (e.g., develop websites) to agencies. In addition, in August 2014 the Administration established USDS, which aims to improve public-facing federal IT services. The President's Budget for fiscal year 2016 also proposed funding for agencies to establish their own digital service teams.
GAO was asked to review 18F and USDS. GAO's objectives were to (1) describe 18F and USDS efforts to address problems with IT projects and agencies' views of services provided, (2) assess these programs' efforts against practices for performance measurement and project prioritization, and (3) assess agency plans to establish their own digital service teams. To do so, GAO reviewed 32 18F projects and 13 USDS projects that were underway or completed as of August 2015 and surveyed agencies about these projects; reviewed 18F and USDS in key performance measurement and project prioritization practices; reviewed 25 agencies' efforts to establish digital service teams; and reviewed documentation from four agencies, which were chosen based on their progress made in establishing digital service teams.
section_title: What GAO Found
paragraphs: The General Service Administration's (GSA) 18F and Office of Management and Budget's (OMB) U.S. Digital Service (USDS) have provided a variety of services to agencies supporting their information technology (IT) efforts. Specifically, 18F staff helped 18 agencies with 32 projects and generally provided development and consulting services, including software development solutions and acquisition consulting. In addition, USDS provided assistance on 13 projects across 11 agencies and generally provided consulting services, including quality assurance, problem identification and recommendations, and software engineering. Further, according to GAO's survey, managers were generally satisfied with the services they received from 18F and USDS on these projects (see table).
Source: GAO survey of agency project managers that engaged with 18F and U.S. Digital Service. | GAO-16-602
Both 18F and USDS have partially implemented practices to identify and help agencies address problems with IT projects. Specifically, 18F has developed several outcome-oriented goals and related performance measures, as well as procedures for prioritizing projects; however, not all of its goals are outcome-oriented and it has not yet fully measured program performance. Similarly, USDS has developed goals, but they are not all outcome-oriented and it has established performance measures for only one of its goals. USDS has also measured progress for just one goal. Until 18F and USDS fully implement these practices, it will be difficult to hold the programs accountable for results.
Agencies are beginning to establish digital service teams. Of the 25 agencies included in the President's proposed funding for agency digital service teams, OMB has established charters with 6 agencies for their digital service teams. In addition, according to the Deputy USDS Administrator, USDS expects to establish charters with an additional 2 agencies by the end of the fiscal year—the Department of Education and the Small Business Administration. For the remaining 16 agencies, as of April 2016, 8 agencies reported that they plan to establish digital service teams but have yet to establish charters with USDS. The other 9 agencies reported that they do not plan to establish digital service teams by September 2016 and most noted that it was because they did not receive requested funding to do so. Further, of the 4 agencies GAO selected to review, only 1 has defined the relationship between its digital service team and the agency Chief Information Officer (CIO). This is due, in part, to the fact that USDS policy does not describe the expected relationship between CIOs and these teams. Until OMB updates its policy and ensures that the responsibilities between the CIOs and digital services teams are clearly defined, it is unclear whether CIOs will be able to fulfill their statutory responsibilities with respect to IT management of the projects undertaken by the digital service teams.
section_title: What GAO Recommends
paragraphs: GAO is making two recommendations to GSA and two recommendations to OMB to improve goals and performance measurement. GAO is also recommending that OMB update policy regarding CIOs and digital services teams. GSA and OMB concurred with the recommendations.

section_title: Letter
paragraphs: Information systems are critical to the health, economy, and security of  the nation. To support these systems, the federal government plans to  invest more than $89 billion on information technology (IT) in fiscal year  2017. However, prior IT expenditures too often have produced failed  projects—that is, projects with multimillion dollar cost overruns and  schedule delays measured in years, with questionable mission-related  achievements. In light of these ongoing challenges, in February 2015 we  added improving the management of IT acquisitions and operations to our  list of high-risk areas for the federal government.
In an effort to improve federal IT management, in March 2014 the  General Services Administration (GSA) established 18F, a team that  provides IT services (e.g., develop websites and provide software  development training) to federal agencies on a reimbursable basis.  Similar to 18F, in August 2014 the President established the U.S. Digital  Service (USDS) within the Office of Management and Budget (OMB),  which aims to improve the most important public-facing federal digital  services. In addition, the President’s Budget for fiscal year 2016 proposed  funding for agencies to establish their own agency digital service teams.
You asked us to review 18F and USDS, as well as agency digital service  teams. Our objectives were to (1) describe 18F and USDS efforts to  address problems with IT projects and agencies’ views of services  provided, (2) assess these programs’ efforts against practices for  performance measurement and project prioritization, and (3) assess  agency plans to establish their own digital service teams.
In addressing our first objective, we reviewed 32 projects across 18  agencies for which 18F provided services to agencies, and 13 projects at  11 agencies for which USDS provided services. To identify the projects,  we obtained the list of completed and ongoing projects at agencies for  which 18F and USDS provided services, as of August 2015 and removed  projects without agency customers (e.g., internal projects and  development of guides for other agencies). The selected projects and  associated agencies are identified in appendix II. We then analyzed  information obtained from the projects describing the services each of the  selected projects received from 18F and USDS. We also conducted a  customer satisfaction survey of the managers of all selected projects to  determine their level of satisfaction with the services provided by 18F and  USDS. Although the survey responses cannot be used to generalize the  opinions and satisfaction of all customers that received services from 18F  and USDS programs, the responses provide data for our defined  population.
To address the second objective, we compared 18F and USDS policies  procedures, plans, and practices to leading practices identified by federal  law and GAO on performance measurement and project prioritization.
To address our third objective, we administered a data collection  instrument on plans to establish digital service teams to the 25 agencies  with funding proposed in the President’s Budget for fiscal year 2016.  Additionally, we reviewed USDS’s plans—to include interviews with  USDS officials—for providing assistance to agencies that planned to  establish a digital service team in fiscal year 2016.
In addition, we selected four agencies as case studies to review the  relationships between agency Chief Information Officers (CIO) and  agency digital service teams. To choose these agencies, we identified the  three agencies that had established a charter with USDS as of January  2016—the Departments of Defense, Homeland Security, and State. We  also selected the Department of Veterans Affairs because, as of January  2016, it had the most staff of any agency digital service team. For these  agencies, we evaluated agency policies and procedures to determine the  extent to which agencies had documented the relationships between  digital service teams and agency CIOs. We also conducted interviews  with the CIOs of the Departments of Defense, Homeland Security, and  State, as well as the Veterans Affairs Principal Deputy Assistant  Secretary for the Office of Information and Technology.
We conducted this performance audit from July 2015 to August 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I contains further  details about our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: Investments in IT can enrich people’s lives and improve organizational  performance. During the last two decades the Internet has matured from  being a means for academics and scientists to communicate with each  other to a national resource where citizens can interact with their  government in many ways, such as by receiving services, supplying and  obtaining information, asking questions, and providing comments on  proposed rules.
However, while these investments have the potential to improve lives and  organizations, some federally funded IT projects can—and have— become risky, costly, unproductive mistakes. We have previously testified  that the federal government has spent billions of dollars on failed or  troubled IT investments, such as  the Office of Personnel Management’s Retirement Systems  Modernization program, which was canceled in February 2011, after  spending approximately $231 million on the agency’s third attempt to  automate the processing of federal employee retirement claims;  the tri-agency National Polar-orbiting Operational Environmental  Satellite System, which was stopped in February 2010 by the  Administration after the program spent 16 years and almost $5  billion;  the Department of Veterans Affairs’ Scheduling Replacement Project,  which was terminated in September 2009 after spending an estimated  $127 million over 9 years; and  the Department of Health and Human Services’ (HHS) Healthcare.gov  website and its supporting systems, which were to facilitate the  establishment of a health insurance marketplace by January 2014,  encountered significant cost increases, schedule slips, and delayed  functionality. In a series of reports we identified numerous planning,  oversight, security, and system development challenges faced by this  program and made recommendations to address them.
In light of these failures and other challenges, last year we introduced a  new government-wide high-risk area, Improving the Management of IT  Acquisitions and Operations.
subsections: section_title: Digital Service Teams Are Intended to Improve the Federal Government’s IT Efforts
paragraphs: 18F and USDS were formed in 2014 to help address the federal  government’s troubled IT efforts. Both programs have similar missions of  improving public-facing federal digital services.
subsections: section_title: 18F’s Mission and Organization
paragraphs: 18F was created in March 2014 by GSA with the mission of transforming  the way the federal government builds and buys digital services.  Agencies across the federal government have access to 18F services.  Work is largely initiated by agencies seeking assistance from 18F and  then the program decides how and if it will provide assistance. According  to GSA, 18F seeks to accomplish its mission by providing a team of  expert designers, developers, technologists, researchers, and product  specialists to help rapidly deploy tools and online services that are  reusable, less costly, and are easier for people and businesses to use. In  addition, 18F has several guiding principles, to include the use of open  source development, user-centered design, and Agile software  development.  18F is an office within the Technology Transformation Service within GSA  that was recently formed in May 2016. 18F is led by the Deputy  Commissioner for the Technology Transformation Service, who reports to  the service’s Commissioner. Prior to May 2016, 18F was located within  the Office of Citizen Services and Innovative Technologies and reported  to the Associate Administrator for Citizen Services and Innovative  Technologies. In January 2016 GSA began piloting a new organizational  structure for 18F that centers around five business units.
Custom Partner Solutions. Provides agencies with custom  application solutions. This unit also provides consulting services to  assist agencies in deciding whether to build, what to build, how to  build it, and who will build it.
Products and Platforms. Provides agencies with access to tools that  address common government-wide needs.
Transformation Services. Aims to improve how agencies acquire  and manage IT by providing them with consulting services, to include  new management models, modern software development practices,  and hiring processes.
Acquisition Services. Provides acquisition services and solutions to  support digital service delivery, including access to vendors  specializing in Agile software development, and request for proposal  development consultation.
Learn. Provides agencies with education, workshops, outreach, and  communication tools on developing and managing digital services.
To provide the products and services offered by each business unit, 18F  relied on 173 staff to carry out its mission, as of March 2016. The staff are  assigned to different projects that are managed by the business units.  According to18F, the program used special hiring authorities for the vast  majority of its staff: Schedule A excepted service authorities were used to  hire 162 staff. These authorities permit the appointment of qualified  personnel without the use of a competitive examination process. GSA has  appointed its staff to terms that are not to exceed 2 years. According to  the Director of the 18F Talent division, after the initial appointment has  ended, GSA has the option of appointing staff to an additional term not to  exceed 2 years.
GSA funds 18F through the Acquisition Services Fund—a revolving fund,  which operates on the revenue generated from its business units rather  than an appropriation received from Congress. The Federal Acquisition  Service, with the concurrence of the Administrator of General Services,  has used the fund to invest in the development of 18F products and  services that will be resold by GSA and used by other organizations.  18F is to recover costs through the Acquisition Services Fund  reimbursement authority for work related to acquisitions and the Economy  Act reimbursement authority for all other projects. According to the  memorandum of agreement between 18F and the Federal Acquisition  Service, 18F is required to have a plan to achieve full cost recovery. In  order to recover its costs, 18F is to establish interagency agreements with  partner agencies and will charge them for actual time and material costs,  as well as a fixed overhead amount. Table 1 describes 18F’s revenue,  expenses, and net operating results for fiscal years 2014 and 2015. Table  2 describes 18F’s projected revenue, expenses, and net operating results  for fiscal years 2016 through 2019.
As shown in table 2, according to its projections, 18F plans to generate  revenue that meets or exceeds operating expenses and cost of goods  sold beginning in fiscal year 2019.
In May 2016 the GSA Inspector General reported on an information  security weakness pertaining to 18F. Specifically, according to the  report, 18F misconfigured a messaging and collaboration application,  which resulted in the potential exposure of personally identifiable  information (PII). 18F officials told us that, based on the preliminary  results of their ongoing review, information such as individual’s first  names, last names, e-mail addresses, and phone numbers were made  available on the messaging and collaboration platform’s databases, and  could have been accessible by authorized users of the application.  Those officials also stated that, based on the preliminary results of their  ongoing review, more sensitive PII, such as Social Security numbers and  protected health information, were not exposed. They added that they are  continuing a detailed review, in coordination with the GSA IT organization,  to confirm that more sensitive PII were not made available.
section_title: USDS’s Mission and Organization
paragraphs: According to the Administration, in 2013 it initiated an effort that brought  together a group of digital and technology experts from the private sector  that helped fix Healthcare.gov. In an effort to apply similar resources to  additional projects, in August 2014 the Administration announced the  launch of USDS, to be led by an Administrator and Deputy Federal CIO  who reports to the Federal CIO. According to OMB, USDS’s mission is  to transform the most important public-facing digital services. USDS  selects which projects it will apply resources to and generally initiates the  effort with agencies.
To accomplish its mission, USDS aims to recruit private sector experts  (e.g., IT engineers and designers) and leading civil servants, and then  deploy small teams to partner them with government agencies. With the  help of these experts, OMB states that USDS applies best practices in  product design and engineering to improve the usefulness, user  experience, and reliability of the most important public-facing federal  digital services. As of November 2015, USDS staff totaled about 98  individuals. Similar to 18F, USDS assigns individuals directly to projects  aimed at achieving its mission.
USDS has used special hiring authorities for the vast majority of its staff.  Specifically:
Schedule A excepted service. According to USDS, as of November  2015, 52 USDS staff members were hired using the Schedule A  excepted service hiring authority. According to the USDS  Administrator, appointments made using this authority are not to  exceed 2 years. At the end of that period, staff can be appointed for  an additional term of no more than 2 years.
Intermittent consultants. According to USDS, as of November 2015,  39 USDS staff members were intermittent consultants—that is,  individuals hired through a noncompetitive process to serve as  consultants on an intermittent basis or without a regular tour of duty.  The USDS Administrator explained that some of these staff are  eventually converted to temporary appointments under the Schedule  A authority.
According to its Administrator, USDS does not generally make permanent  appointments for its staff because it allows the program to continuously  bring in new staff and ensure that its ideas are continually evolving.
USDS reported spending $318,778 during fiscal year 2014 and  approximately $4.7 million during fiscal year 2015. For fiscal year 2016,  USDS plans to spend approximately $14 million, and the President’s  fiscal year 2017 budget estimated obligations of $18 million for USDS.
section_title: Agency Digital Service Teams
paragraphs: In an effort to make improvements to critical IT services throughout the  federal government, the Presidents’ Budget for fiscal year 2016 proposed  funding for the 24 Chief Financial Officers Act agencies, as well as the  National Archives and Records Administration, to establish digital service  teams. USDS policy calls for these agencies to, among other things, hire  or designate an executive for managing their digital service teams.
Additionally, USDS has established a hiring pipeline for digital service  experts—that is, a unified process managed by USDS for accepting and  reviewing applications, performing initial interviews, and providing  agencies with candidates for their digital service teams. According to  OMB, before using this service, agencies must agree to a charter with the  USDS Administrator.
section_title: Roles and Responsibilities for Overseeing IT Investments
paragraphs: Over the last three decades, several laws have been enacted to assist  federal agencies in managing IT investments. For example, the  Paperwork Reduction Act of 1995 requires that OMB develop and  oversee policies, principles, standards, and guidelines for federal agency  IT functions, including periodic evaluations of major information  systems. In addition, the Clinger-Cohen Act of 1996, among other  things, requires agency heads to appoint CIOs and specifies many of  their responsibilities. With regard to IT management, CIOs are  responsible for implementing and enforcing applicable government-wide  and agency IT management principles, standards, and guidelines;  assuming responsibility and accountability for IT investments; and  monitoring the performance of IT programs and advising the agency head  whether to continue, modify, or terminate such programs.
Most recently, in December 2014, IT reform legislation (commonly  referred to as the Federal Information Technology Acquisition Reform Act  or FITARA) was enacted, which required most major executive branch  agencies to ensure that the CIO had a significant role in the decision  process for IT budgeting, as well as the management, governance, and  oversight processes related to IT. The law also required that CIOs  review and approve (1) all contracts for IT services associated with major  IT investments prior to executing them and (2) the appointment of any  other employee with the title of CIO, or who functions in the capacity of a  CIO, for any component organization within the agency. OMB also  released guidance in June 2015 that reinforces the importance of agency  CIOs and describes how agencies are to implement the law.
OMB plays a key role in helping federal agencies address these laws and  manage their investments by working with them to better plan, justify, and  determine how much they need to spend on projects and how to manage  approved projects. Within OMB, the Office of E-Government and  Information Technology, headed by the Federal CIO, directs the policy  and strategic planning of federal IT investments and is responsible for  oversight of federal technology spending.
section_title: 18F and USDS Provided a Variety of Development and Consulting Services Supporting Agency Technology Efforts and Agencies Were Generally Satisfied with the Programs
paragraphs: 18F and USDS have provided a variety of development and consulting  services to agencies to support their technology efforts. Specifically,  between March 2014 and August 2015, 18F staff helped 18 agencies  with 32 projects and generally provided six types of services to the  agencies, the majority of which related to development work. In addition,  between August 2014 and August 2015, USDS provided assistance on  13 projects at 11 agencies and provided seven types of consulting  services.
Further, agencies were generally satisfied with the services they received  from 18F and USDS. Specifically, of the 26 18F survey respondents, 23  were very satisfied or moderately satisfied and 3 were moderately  dissatisfied. For USDS, all 9 survey respondents were very satisfied or  moderately satisfied.
subsections: section_title: 18F Has Provided a Variety of Products and Services; the Majority of Projects Were Development Work
paragraphs: Between March 2014 and August 2015, GSA’s 18F staff helped 18  agencies with 32 projects, and generally provided services relating to its  five business units: Custom Partner Solutions, Products and Platforms,  Transformation Services, Acquisition Services, and Learn. In addition,  18F also provided agency digital service team candidate qualification  reviews in support of USDS.
Custom Partner Solutions. 18F helped 11 agencies with a total of  19 projects relating to developing custom software solutions. Out of  the 19 projects, 12 were related to website design and development.  For example, regarding GSA’s Pulse project—a website that displays  data about the extent to which federal websites are adopting best  practices, such as hypertext transfer protocol over secure sockets  layer (SSL)/ transport layer security (TLS) (HTTPS)—18F designed,  developed, and delivered the first iteration of Pulse within 6 weeks of  the project kick-off. According to the GSA office responsible for  managing the project, the first iteration has led to positive outcomes  for government-wide adoption of best practices; for example, between  June 2015 and January 2016, the percentage of federal websites  using HTTPS increased from 27 percent to 38 percent.
As another example, officials from the Department of Education’s  college choice project stated that 18F helped develop the project’s  website, which the public can use to search among colleges to find  schools that meet their needs (e.g., degrees offered, location, size,  graduation rate, average salary after attendance). 18F also helped  two agencies, HHS and the Department of Defense, on two projects  to develop application programming interfaces—sets of routines,  protocols, and tools for building software applications that specify how  software components should interact.
Acquisition Services. 18F helped seven agencies on seven projects  relating to acquisition services consulting. For example, 18F  provided the Department of State’s Bureau of International  Information Programs with cloud computing services offered under  a GSA blanket purchase agreement (BPA)—specifically, cloud  management services (e.g., developers, testing and quality  assurance, cloud architect) and infrastructure-as-a-service.  According to the Department of State, the department was able to  deploy its instance of the infrastructure service only 1 month after it  executed an interagency agreement with 18F. In addition, according  to Social Security Administration officials, 18F helped the agency to  incorporate Agile software development practices into their requests  for proposals for their Disability Case Processing System.
Learn. 18F provided services to four agencies on four projects  regarding training, such as educating agency officials on Agile  software development. For example, 18F conducted training  workshops on Agile software development techniques with the Social  Security Administration and Small Business Administration. In  addition, according to the Department of Labor’s Wage and Hour  Division officials, 18F conducted a 3-day workshop on IT  modernization.
Transformation Services. 18F assisted two agencies on two  projects to help acquire the people, processes, and technology  needed to successfully deliver digital services. For example, 18F  assisted the Environmental Protection Agency on an agency-wide  technology transformation. According to an official within the office of  the CIO, 18F assisted the agency with e-Manifest—a system used to  track toxic waste shipments. The official noted that 18F provided  user-centered design, Agile coaching, prototype development  services, and Agile and modular acquisition services. Further, the  official stated that 18F helped turn around the project and significantly  decreased the time of delivery for e-Manifest.
Products and Platforms. 18F helped two agencies on two projects  related to developing software solutions that can potentially be  reused at other federal agencies. For example, according to GSA  officials responsible for managing GSA’s Communicart project, 18F  provided the agency with an e-mail-based tool for approving office  supply purchases.
Agency digital service team candidate qualification review. 18F  worked with USDS to recruit and hire team members for agency  digital service teams. According to 18F officials, it provided USDS with  subject matter experts to review qualifications of candidates for  agency digital service teams.
Of the 32 18F projects, 6 are associated with major IT investments.  Cumulatively, the federal government plans to spend $853 million on  these investments in fiscal year 2016. Additionally, risk evaluations  performed by CIOs that were obtained from the IT Dashboard showed  that three of these investments were rated as low or moderately low risk  and three investments were rated medium risk. Table 3 describes the  associated investments, including their primary functional areas,  planned fiscal year 2016 spending, and CIO rating as of May 2016.  18F is also developing products and services—including an Agile delivery  service blanket purchase agreement (BPA), cloud.gov, and a shared  authentication platform:
Agile delivery service BPA. 18F established this project in order to  support its need for Agile delivery services, including Agile software  development. In August and September 2015, GSA awarded BPAs to  17 vendors. The BPAs are for 5 years and allow GSA to place orders  against them for up to 13 specific labor categories relating to Agile  software development (e.g., product manager, backend web  developer, Agile coach) at fixed unit prices.
The BPAs do not obligate any funds; rather, they enable participating  vendors to compete for follow-on task orders from GSA. In cases  where 18F determines that it should use the Agile BPA to provide  services to partner agencies, GSA anticipates that 18F will work with  that agency to develop a request for quotations and the other  documents needed for a competition with Agile BPA vendors.
In June 2016 GSA issued its first task order under the Agile BPA for  building a web-based dashboard that would describe the status of  vendors in the certification process for the Federal Risk and  Authorization Management Program (FedRAMP)—a government- wide program, managed by GSA, to provide joint authorizations and  continuous security monitoring services for cloud computing services  for all federal agencies.
The initial BPAs were established under the first of three anticipated  award pools—all of which are part of the “alpha” component of the  Agile BPA project. 18F officials stated that they planned to establish  BPAs for the other two pools in June 2016. They also anticipate a  future beta version of the project that could potentially allow federal  agencies beyond 18F to issue task orders directly to vendors. Officials  stated that they expect to have a plan for the next steps of the beta  version of this project by December of 2016.  18F officials have also expressed interest in creating additional  marketplaces, such as those relating to data management, developer  productivity tools, cybersecurity, and health IT. As of March 2016, 18F  did not have time frames for when it planned to develop these  additional marketplaces.
Cloud.gov. 18F also developed the cloud.gov service, which is an  open source platform-as-a-service that agencies can use to manage  and deploy applications. 18F initially built cloud.gov in order to enable  the group to use applications it developed for partner agencies. In  creating the service, 18F decided to offer it to other agencies  because, according to 18F officials, cloud.gov offers a developer- friendly, secure platform, with tools that agencies can use to  accelerate the process of assessing information security controls and  authorizing systems to operate. According to 18F, the goal of  cloud.gov is to provide government developers and their contractor  partners the ability to easily deploy systems to a cloud infrastructure  with better efficiency, effectiveness, and security than current  alternatives.
According to a roadmap for cloud.gov, 18F plans to receive full  FedRAMP Joint Authorization Board approval for this service by  November 2016. Once available, the group anticipates requiring  agencies to pay for this service through an interagency agreement  with 18F.
Shared authentication platform. In May 2016 18F announced that it  was initiating an effort to create a platform for users who need to log  into federal websites for government services. According to 18F, this  system is designed to be each citizen’s “one account” with the  government and allow the public to verify an identity, log into  government websites, and if necessary, recover an account. As of  May 2016, 18F plans to conduct prototyping activities through  September 2016 and did not have plans beyond that time frame.
In addition to developing future products and services, 18F created a  variety of guides and standards for use internally as well as by agency  digital service teams. These guides address topics such as accessibility,  application programming interfaces, and Agile software development.
section_title: USDS Provided Seven Types of Consulting Services Aimed at Helping Agencies Improve IT
paragraphs: From August 2014 through August 2015, USDS provided assistance on  13 projects across 11 agencies. The group generally provided seven  types of consulting services: quality assurance, problem identification and  recommendations, website consultation, system stabilization, information  security assessment, software engineering, and data management.
Quality assurance. Three of the 13 projects related to providing  quality assurance services. For example, regarding the Social  Security Administration’s Disability Case Processing System, USDS  reviewed the quality of the software and made recommendations that,  according to the agency, resulted in costs savings. Additionally, for  the Departments of Veterans Affairs and Defense Service Treatment  Record project, USDS provided engineers who identified and resolved  errors in the process of exchanging records between the two  departments, according to the Department of Veterans Affairs.  Further, for the HHS Healthcare.gov system, the group performed  services aimed at optimizing the reliability of the system, according to  HHS.
Problem identification and recommendations. USDS identified  problems and made recommendations for three projects. For all three  projects, it performed a discovery sprint—a quick (typically 2 week)  review of an agency’s challenges, which is to culminate in a clear  understanding of the problems and recommendations for how to  address the issues. For example, according to USDS, the group  performed a discovery sprint for the Department of the Treasury  Internal Revenue Service that focused on three areas: authentication  of taxpayers, modernizing systems through event-driven  architecture, and redesigning the agency’s website. USDS delivered  recommendations to the Internal Revenue Service with  recommendations and also suggested that work initially focus on  taxpayer authentication. Consistent with these recommendations,  according to USDS, the group and the agency focused on  authentication, to include re-opening of the online application Get  Transcript.
For the Department of Justice Federal Bureau of Investigation’s  National Incident Based Reporting System, according to USDS, the  group performed a discovery sprint and made several  recommendations for accelerating deployment of the system.
Website consultation. USDS provided consultation services for  three agency website projects. For example, for the Office of the U.S.  Trade Representative’s Trans-Pacific Partnership Trade Agreements  website, USDS provided website design advice and confirmed that  the agency had the necessary scalability to support the number of  anticipated visitors. Additionally, it consulted with the Office of  Personnel Management (OPM) on the design, implementation, and  development of a website for providing information on reported data  breaches.
System stabilization. For the Department of State’s Consular  Consolidated Database, according to USDS, it helped stabilize the  system and return it to operational service after a multi-week outage  in June 2015.
Information security assessment. USDS helped with an  information security assessment regarding Electronic Questionnaires  for Investigations Processing, which encompasses the electronic  applications used to process federal background check  investigations.
Software engineering. For the Department of Homeland Security  U.S. Citizenship and Immigration Services Transformation project,  USDS’s software engineering advisors provided guidance on private  sector best practices in delivering modern digital services. According  to the department, the group’s work has supported accomplishments  such as increasing the frequency of software releases and improving  adoption of Agile development best practices.
Data management. For the Department of Homeland Security Office  of Immigration Statistics, USDS helped to develop monthly reports on  immigration enforcement priority statistics. According to the  department, USDS supported the development of processes for  obtaining data from other offices within the department and  generating the monthly reports. According to the department, after 7  weeks of working with USDS, it was able to develop a proof of  concept that reduced the report generating process from a month to 1  day.
Seven of the 13 projects are associated with major IT investments.  Cumulatively, the federal government plans to spend over $1.24 billion on  these investments in fiscal year 2016. Three investments were rated by  their CIOs as low or moderately low risk and four investments were rated  as being medium risk. Table 4 describes the associated investments,  including their primary functional areas, planned fiscal year 2016  spending, and CIO ratings as of May 2016.
In addition to providing services to agencies, USDS has developed  products to help agencies improve federal IT services. For example, it  developed the Digital Services Playbook to provide government-wide  recommendations on practices for building digital services. The group  also created the TechFAR Handbook to explain how agencies can use  the Digital Services Playbook in ways that are consistent with the Federal  Acquisition Regulation. Further, USDS, in collaboration with 18F,  developed the draft version of U.S. Web Design Standards, which  includes a visual style guide and a collection of common user interface  components. With this guide, USDS aims to improve government  website consistency and accessibility.
In addition to developing guidance, USDS, in collaboration with OMB’s  Office of Federal Procurement Policy, used challenge.gov to incentivize  the public to create a digital service training program for federal contract  professionals. The challenge winner received $250,000 to develop and  pilot a training program. Additionally, the Deputy Administrator for USDS  stated that 30 federal contract professionals from more than 10 agencies  completed this pilot program in March 2016. According to OMB, the  program is being revised and transitioned to the Federal Acquisition  Institute, where it will be included as part of a certification for digital  service contracting officers.
section_title: A Majority of Surveyed Agency Project Managers Were Satisfied with Services Provided by 18F and USDS
paragraphs: In response to a satisfaction survey we administered to agency managers  of selected 18F and USDS projects, a majority of managers were  satisfied with the services they received from the groups. Specifically,  the average score for services provided by 18F was 4.38 (on a 5-point  satisfaction scale, where 1 is very dissatisfied and 5 is very satisfied) and  the average score for the services provided by USDS was 4.67. Table 5  describes the survey results for 18F and USDS.
In addition to providing scores, the survey respondents also provided  written comments. Regarding 18F, five factors were cited by two or more  respondents as contributing to their satisfaction with the services the  program provided: delivering quality products and services, providing  good customer service, completing tasks in a timely manner, employing  staff with valuable knowledge and skills, and providing valuable education  to agencies. For example, one respondent stated that 18F has an expert  staff that helped the team understand Agile software development and  incorporate user-centered design into the agency’s development process.
With respect to USDS, four factors were cited by two or more  respondents as contributing to their satisfaction with its services:  delivering quality services, providing good customer service, completing  tasks in a timely manner, and employing staff with valuable knowledge  and skills. For instance, one respondent stated that USDS responded to  the agency’s request in a matter of hours, quickly developed an  understanding of the agency’s IT system, and pushed to improve the  system, even in areas beyond the scope of USDS’s responsibility.
Although the majority of agencies were satisfied, a minority of  respondents provided written comments describing their dissatisfaction  with services provided by 18F. For example, six respondents cited poor  customer service, four respondents cited higher than expected costs, and  one respondent stated that 18F’s use of open source code may not meet  the agency’s information security requirements.
In a written response to these comments, 18F stated that it has received  a variety of feedback from its partners and has modified and updated its  processes continuously over the past 2 years. For example, with respect  to higher than expected costs, 18F stated that project costs sometimes  needed to be adjusted mid-project to address, among other things, higher  than expected infrastructure usage or unexpected delays. To address this  issue, 18F stated that it uses the assistance of subject matter experts to  estimate project costs, and wrote a guide to assist with, among other  things, better managing the budgets of ongoing projects. Regarding 18F’s  use of open source code, it stated that it has worked with its partners to  discuss the use of open source software and information security  practices.
section_title: USDS and 18F Did Not Fully Define Goals and Measure Performance
paragraphs: To assess actual results, prioritize limited resources, and ensure that the  most critical projects receive attention, USDS and 18F should establish  and implement the following key practices:
Define outcome-oriented goals and measure performance. Our  previous work and federal law stress the importance of focusing on  outcome-oriented goals and performance measures to assess the  actual results, effects, or impact of a program or activity compared to  its intended purpose. Goals should be used to elaborate on a  program’s mission statement and should be aligned with performance  measures. In turn, performance measures should be tied to program  goals and demonstrate the degree to which the desired results were  achieved. To do so, performance measures should have targets to  help assess whether goals were achieved by comparing projected  performance and actual results. Finally, goals and performance  measures should be outcome-oriented—that is, they should address  the results of products and services.
Establish and implement procedures for prioritizing IT projects.  We have reported that establishing and implementing procedures, to  include criteria, for prioritizing projects can help organizations  consistently select projects based on their contributions to the  strategic goals of the organization. Doing so will better position  agencies to effectively prioritize projects and use the best mix of  limited resources to move toward its goals.
subsections: section_title: 18F Has Goals and Procedures for Prioritizing Projects, but Needs to Fully Define Outcome- Oriented Goals and Measure Performance
paragraphs: 18F has developed several outcome-orientated goals, performance  measures, and procedures for prioritizing projects, which it has largely  implemented. However, not all of its goals are outcome-oriented and it  has not yet measured program performance.
Define Outcome-Oriented Goals and Measure Performance  At the conclusion of our review in May 2016, 18F provided 5 goals and 17  associated performance measures that the organization aims to achieve  by September 2016 (see table 6).
To 18F’s credit, several of its goals and performance measures appear to  be outcome-oriented. For example, the goal of delivering two  government-wide platform services and the associated performance  measures are outcome-oriented in that they address results—that is,  delivering services to partner agencies.
However, not all of the goals and performance measures appear to be  outcome-oriented. For example, the goal of growing 18F to 215 staff while  sustaining a healthy culture and its associated measure of hiring 47 staff  do not focus on results of products or services. Further, not all of the  performance measures have targets. For example, seven of the  performance measures state that 18F will establish performance  indicators, but 18F has yet to do so. Moreover, 18F does not have goals  and associated measures that describe how it plans to achieve its  mission after September 2016.
In addition, although 18F is required to have a plan to achieve full cost  recovery, it has yet to recover costs and its projections for when this will  occur have slipped over time. Specifically, in June 2015, 18F projected  that it would fully recover its costs for an entire fiscal year beginning in  2016; however, in May 2016, 18F provided revised projections indicating  that it would recover costs beginning in fiscal year 2019. Those  projections also indicated that, in the worst case, it would not do so  through 2022, the final year of its projections. Establishing performance  measures and targets that are tied to achieving full cost recovery would  help management gauge whether the program is on track to meet its  projections. However, 18F has not established such performance  measures and targets.
Finally, 18F has yet to fully assess the actual results of its activities.  Specifically, the group has not assessed its performance in accordance  with the 17 performance measures it developed. 18F’s then-parent  organization assessed its own performance quarterly beginning in the 4th  quarter of fiscal year 2015, including for measures that 18F was  responsible for. However, this review process did not include or make  reference to the 17 measures developed to gauge 18F’s performance,  and thus do not provide insight into how well it is achieving its own  mission.
In a written response, GSA stated that 18F performance is measured as  part of the Technology Transformation Service’s goals and measures and  that these goals and measures should form the basis for our review.  However, the Technology Transformation Service’s goals and measures  do not describe how GSA aims to achieve the specific mission of 18F.
Until it establishes goals and performance measures beyond September  2016, ensures that all of its goals and performance measures are  outcome-oriented, and that its performance measures have targets, 18F  will not have a clear definition of what it wants to accomplish. Additionally,  without developing performance measures and targets tied to achieving  full cost recovery, GSA will lack a fully defined approach to begin  recovering all costs in fiscal year 2019. Further, until 18F fully measures  actual results, it will not be positioned to assess the status of its activities  and determine the areas that need improvement.
Establish and Implement Procedures for Prioritizing IT Projects  18F has developed procedures, including criteria, for prioritizing projects  and largely implemented its procedures. Specifically, according to the  Director of Business Strategy, potential projects are discussed during  weekly intake meetings. As part of these meetings, 18F discusses project  decision documents, which outline the business, technical, and design  elements, as well as the schedule, scope, and resources needed to fulfill  the client’s needs. Using these documents, 18F determines whether  proposed projects meet, among other things, the following criteria: (1) the  project is aligned with the products and services offered by 18F, (2) it can  be completed in a time frame that meets the agency’s needs and at a  cost that fits the agency’s budget, and (3) the project’s government  transformation potential (e.g., impact on the public, cost savings). These  documents are used by the business unit leads to make a final decision  about whether to accept the projects.  18F has largely implemented its procedures. To its credit, with respect to  the 14 projects that 18F selected since establishing its prioritization and  selection process, 18F developed a decision document for 12 of the 14  projects. However, 18F did not develop a decision document for the 2  remaining projects—the Nuclear Regulatory Commission’s Master Data  Management project and GSA’s labs.usa.gov project.
With respect to the Nuclear Regulatory Commission’s Master Data  Management project, 18F officials explained that this project only required  staff from one division; as such, that division was able to independently  prioritize and select this project. Additionally, regarding the GSA  labs.usa.gov project, 18F officials said the Associate Administrator for the  Office of Citizen Services and Innovative Technologies directed 18F to  provide assistance.
If 18F consistently follows its process for prioritizing projects, it will be  better positioned to apply resources to IT projects with the greatest need  of improvement.
section_title: USDS Has Goals and a Process for Prioritizing Projects, but More Work Remains to Define Outcome-Oriented Goals and Measure Performance
paragraphs: While USDS has developed program goals and a process for prioritizing  projects, it has not fully implemented important program management  practices.
Define Outcome-Oriented Goals and Measure Performance  In November 2015 USDS developed four goals to be achieved by  December 2017: (1) recruit and place over 200 digital service experts in  strategic roles at agencies and cultivate a continually growing pipeline of  quality technical talent through USDS, (2) measurably improve five to  eight of the government’s most important services, (3) begin the  implementation of at least one outstanding common platform, and (4)  increase the quality and quantity of technical vendors working with  government and cultivate better buyers within government. Additionally,  USDS established a performance measure with a target for one of its  goals. Specifically, it has a measure for its first goal as it plans to  measure the extent to which it will hire 200 digital service experts by  December 2017.
To its credit, several of the goals appear to be outcome-oriented. For  example, improving five to eight services is outcome-oriented in that it  addresses results. However, USDS has not established performance  measures or targets for its other goals. In addition, the program’s first  goal—recruit and place over 200 digital service experts in strategic roles  at agencies and cultivate a continually growing pipeline of quality  technical talent through USDS—does not appear to be outcome-oriented.  Further, USDS has only measured actual results for one of its goals.  Specifically, for the goal of placing digital service experts at agencies, as  of May 2016, USDS officials stated that they had 152 digital service  experts. However, USDS has not measured actual results for the other  three goals.
USDS officials provided examples of how they informally measure  performance for the other three goals. For example, for the goal of  measurably improving five to eight of the government’s most important  services, the USDS Administrator stated that approximately 1 million  visitors viewed the Department of Education’s College Scorecard website  in the initial days after it was deployed.
However, USDS has not documented these measures or the associated  results to date. Until USDS ensures that all of its goals are outcome- oriented and establishes performance measures and targets for each  goal, it will be difficult to hold the program accountable for results.  Additionally, without an assessment of actual results, it is unclear what  impact USDS’s actions are having relative to its mission and whether  investments in agency digital service teams are justified.
Establish and Implement Procedures for Prioritizing Projects  USDS has developed procedures and criteria for prioritizing projects. To  identify projects to be considered, USDS is to use, among other sources,  June 2015 and June 2016 OMB reports to Congress that identify the 10  highest-priority federal IT projects in development. To prioritize projects,  USDS has the following three criteria, which are listed in their order of  importance (1) What will do the greatest good for the greatest number of  people in the greatest need? (2) How cost-efficient will the USDS  investment be? and (3) What potential exists to use or reuse a  technological solution across the government? Using these criteria,  USDS intends to create a list of all potential projects, to include their  descriptions and information on resources needs. This list is to be used  by USDS leadership to make decisions about which projects to pursue.
To its credit, USDS created a list of all potential, ongoing, and completed  projects, which included project descriptions and resource needs.  Additionally, USDS has engaged with 6 of the 10 priority IT projects  identified in the June 2015 and June 2016 reports, including HHS’s  Healthcare.gov project and the Department of Homeland Security’s U.S.  Citizenship and Immigration Services Transformation. Additionally,  according to a USDS staff member, USDS considered the remaining 4  projects and decided not to engage with them to date.
Although USDS has yet to develop a quarterly report on the 10 high  priority programs, which it was directed by Congress to develop, it  expects to issue its first report by September 2016. Specifically, in  December 2015, Congress modified its direction for the Executive Office  of the President to develop the reports regarding the top 10 high priority  programs and specifically called for USDS to do so on a quarterly basis.
If USDS develops its report on the 10 high priority programs within the  established time frame and on a quarterly basis thereafter, and considers  the programs identified in these reports as part of its prioritization  process, it will have greater assurance that it will apply resources to the IT  projects with the greatest need of improvement.
section_title: Agencies Have Begun to Establish Digital Service Teams, but OMB Has Not Taken Steps to Ensure CIO Coordination
paragraphs: To help agencies effectively deliver digital services, the President’s  Budget for fiscal year 2016 proposed funding for digital service teams at  25 agencies—the 24 Chief Financial Officers Act agencies, as well as the  National Archives and Records Administration. According to USDS policy,  agencies are to, among other things, hire or designate an executive for  managing their digital service teams. In addition, USDS has called for the  deputy head of these agencies (or equivalent) to, among other things,  agree to a charter with the USDS Administrator. After agreeing to a  charter, according to USDS, agencies can use USDS’s hiring pipeline for  digital service experts.
Of the 25 agencies included in the President’s budget proposal to  establish teams, OMB has established charters with 6 agencies for their  digital service teams—the Departments of Defense, Health and Human  Services, Homeland Security, the Treasury, State, and Veterans Affairs.  The charters establish the executives for managing digital service teams  and describe the reporting relationships between the team leaders and  agency leadership.
In addition, according to the Deputy USDS Administrator, USDS expects  to establish charters with an additional 2 agencies by the end of the fiscal  year—the Department of Education and the Small Business  Administration. For the remaining 16 agencies, as of April 2016, 8  agencies reported that they plan to establish digital service teams but  have yet to establish charters with USDS—the Department of Housing  and Urban Development, Environmental Protection Agency, General  Services Administration, National Aeronautics and Space Administration,  National Archives and Records Administration, National Science  Foundation, Nuclear Regulatory Commission, and Office of Personnel  Management. Of the other 9 agencies, 8 reported that they do not plan to  establish digital service teams by September 2016 because they did not  receive requested funding—the Departments of Agriculture, Commerce,  Energy, the Interior, Justice, Labor, and Transportation; and the U.S.  Agency for International Development. The remaining agency, the Social  Security Administration, does not plan to establish a team because,  according to officials, it does not have large, public-facing IT projects that  are troubled. Table 7 summarizes agency and OMB efforts to establish  digital service teams.
subsections: section_title: OMB Did Not Ensure That Agencies Defined the Relationship between the Digital Service Team and CIOs
paragraphs: Congress has recognized the importance of having a strong agency CIO.  In 1996, the Clinger-Cohen Act established the position of agency CIO  and, among other things, gave these officials responsibility for IT  investments, including IT acquisitions, monitoring the performance of IT  programs, and advising the agency head whether to continue, modify, or  terminate such programs. More recently, in December 2014, FITARA was  enacted into law. It required most major executive branch agencies to  ensure that the CIO has a significant role in the decision process for IT  budgeting, as well as the management, governance, and oversight  processes related to IT. The law also required that CIOs review and  approve (1) all contracts for IT services associated with major IT  investments prior to executing them and (2) the appointment of CIOs for  any component within the agency. OMB also released guidance in June  2015 that reinforces the importance of agency CIOs and describes how  agencies are to implement FITARA. Further, according to our prior  work, leading organizations clearly define responsibilities and authorities  governing the relationships between the CIO and other agency  components that use IT.
Only one of the four agencies we selected for review—the Department of  Homeland Security—defined the relationship between the executive for  managing the digital service team and the agency CIO. Specifically, the  Department of Homeland Security established a charter for its digital  service team, signed by both the Administrator of USDS and the Deputy  Secretary, which outlines the reporting structure and authorities for the  digital services executive, including the relationship with the CIO. For  example, according to the charter, the digital services executive will report  on a day-to-day basis to the CIO, but will also report directly to the Deputy  Secretary.
However, the other three agencies we reviewed—the Departments of  Defense, State, and Veterans Affairs—have not defined the role of  agency CIOs with regard to these teams. Although they have established  charters for these teams, which describe the reporting structure between  the digital services executive and senior agency leadership, the charters  do not describe the role of the agencies’ CIOs and they have not  documented this information elsewhere.
The Department of Defense CIO and the Department of Veterans Affairs  Principal Deputy Assistant Secretary for the Office of Information and  Technology told us that they work closely with their agency digital service  teams. However, while these officials have coordinated with the agency  digital service teams, the roles and responsibilities governing these  relationships should be described to ensure that CIOs can carry out their  statutory responsibilities.
In contrast to the Departments of Defense and Veterans Affairs, the State  CIO told us that he has had limited involvement in the department’s digital  service team. He added that he believes it will be important for CIOs to  be involved in agency digital service teams in order to sustain their  efforts.
In written comments, OMB acknowledged that the Department of State’s  charter does not describe the role of the CIO, but stated that the  Departments of Defense and Veterans Affairs digital service team  charters at least partially address the relationship between digital service  teams and agency CIOs. Specifically, with respect to the Department of  Defense, OMB stated that the charter calls for senior leadership, including  the department’s CIO, to ensure that digital service team projects proceed  without delay. Additionally, according to OMB, the charter for the  Veterans Affairs digital service team calls for the team to be located in  and supported by the department’s CIO organization. However, these  requirements do not address the specific responsibilities or authorities of  the Departments Defense and Veterans Affairs’ CIOs with regard to their  digital service teams.
The lack of defined relationships is due, in large part, to the fact that  USDS policy on digital service teams does not describe the expected  relationship between agency CIOs and these teams. As previously  mentioned, USDS policy calls for the digital service team leader to report  directly to the head of the agency or its deputy; however, it does not  describe the expected responsibilities and authorities governing the  relationship of the CIO.
Until OMB updates the USDS policy to clearly define the responsibilities  and authorities governing the relationships between CIOs and digital  service teams and ensures that existing agency digital service team  charters or other documentation reflect this policy, agency CIOs may not  be effectively involved in the digital service teams. This is inconsistent  with long-standing law, as well as the recently enacted FITARA, and  OMB’s guidance on CIO responsibilities, and may hinder the ability for  CIOs to carry out their responsibilities for IT management of the projects  undertaken by the digital service teams.
section_title: Conclusions
paragraphs: By hiring technology and software development experts and using leading  software development practices, both 18F and USDS have provided a  variety of useful services to federal agencies. Most surveyed agency  project managers that partnered with 18F and USDS were satisfied with  the services provided.
It is important for USDS and 18F to establish outcome-oriented goals,  measure performance, and prioritize projects, particularly since these are  valuable management tools that could aid in the transfer of knowledge  when critical temporary staff leave these organizations and are replaced.  To their credit, both 18F and USDS have developed several outcome- orientated goals and procedures for prioritizing projects. However, the  goals and associated performance measures and targets were not always  outcome-oriented. Additionally, they have not fully measured program  performance. As a result, it will be difficult to hold the programs  accountable for results. Moreover, without documented measures and  results for USDS, it is unclear whether investments in agency digital  service teams are justified. Further, by delaying the date for when it  projects to fully recover its costs and not having associated performance  measures, 18F is at risk of not having the information necessary for GSA  leadership to determine whether to continue using the Acquisition  Services Fund for 18F operations.
Although OMB has called for agencies to establish digital service teams,  USDS policy does not require agencies to define the expected  responsibilities and authorities governing the relationships between CIOs  and digital service teams. To fulfill their statutory responsibilities, including  as most recently enacted in FITARA and reinforced in OMB guidance,  and ensure that CIOs have a significant role in the decision making  process for projects undertaken by the digital service teams, such defined  relationships are essential.
section_title: Recommendations for Executive Action
paragraphs: To effectively measure 18F’s performance, we recommend that the  Administrator of GSA direct the Commissioner for the Technology  Transformation Service to take the following two actions: ensure that goals and associated performance measures are  outcome-oriented and that performance measures have targets,  including  performance measures and targets tied to fully recovering  program costs; and  goals, performance measures, and targets for how the program  will achieve its mission after September 2016; and assess actual results for each performance measure.
To effectively measure performance, prioritize USDS’s resources, and  ensure that CIOs play an integral role in agency digital service teams, we  recommend that the Director of the Office of Management and Budget  direct the Federal Chief Information Officer to take the following three  actions: ensure that all goals and associated performance measures are  outcome-oriented and that performance measures have targets; assess actual results for each performance measure; and update USDS policy to clearly define the responsibilities and  authorities governing the relationships between CIOs and the digital  service teams and require existing agency digital service teams to  address this policy. In doing so, the Federal Chief Information Officer  should ensure that this policy is aligned with relevant federal law and  OMB guidance on CIO responsibilities and authorities.
section_title: Agency Comments and our Evaluation
paragraphs: We provided a copy of a draft of this report to GSA, OMB, and 27  agencies to which we did not make recommendations. We received  comments from GSA and OMB, stating that they agreed with our  recommendations, and from 3 agencies—the Department of Housing and  Urban Development, National Science Foundation, and National Archives  and Records Administration—describing their plans to establish digital  service teams. The remaining 24 agencies stated that they had no  comments. The following is a discussion of each agency’s comments.
In its written comments, GSA concurred with the two  recommendations and described planned actions to address them.  The agency also provided technical comments, which we have  incorporated in the report as appropriate. GSA’s comments are  printed in appendix III.
In its written comments, OMB generally concurred with the three  recommendations and described planned actions to address them. In  a draft of this report, we had included a recommendation to OMB that  it establish a time frame for developing the report identifying the  highest priority projects, develop the report within that established  time frame and on a quarterly basis thereafter, and consider the  highest priority IT projects as part of the established process for  prioritizing projects. Subsequently, in June 2016 OMB provided a  second report identifying the highest priority projects and stated that  the next report would be issued by September 2016. Given these  actions, we have removed this recommendation from our report. The  agency also provided technical comments, which we have  incorporated in the report as appropriate. OMB’s comments are  reprinted in appendix IV.
In written comments, the Department of Housing and Urban  Development described activities underway for establishing a digital  service team. The department’s comments are reprinted in appendix  V.
In written comments, the National Archives and Records  Administration stated that it plans to establish a digital service team  and is currently working with USDS to develop a charter. The  agency’s comments are reprinted in appendix VI.
In comments provided via e-mail on June 29, 2016, a senior advisor  from the National Science Foundation stated that the agency plans to  fund a digital service team from its fiscal year 2016 appropriation to  focus on transforming its digital services with the greatest impact to  citizens and businesses so they are easier to use and more cost- effective to build and maintain.
Multiple agencies also provided technical comments, which we have  incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees, the Director of the Office of Management and Budget, the  Administrator of GSA, the secretaries and agency heads of the  departments and agencies addressed in this report, and other interested  parties. In addition, the report will be available at no charge on GAO’s  website at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-9286 or pownerd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) describe 18F and U.S. Digital Service (USDS)  efforts to identify and address problems with information technology (IT)  projects and agencies’ views of services provided, (2) assess these  programs’ efforts against practices for performance measurement and  project prioritization, and (3) assess agency plans to establish their own  digital service teams.
In addressing our first objective, we reviewed 32 projects across 18  agencies for which 18F provided services to agencies, and 13 projects at  11 agencies for which USDS provided services. To identify these  projects, we obtained the list of 52 completed and ongoing projects for  18F, as of August 2015; and the 17 completed or ongoing projects for  USDS, as of August 2015. For the 18F program, we added a project  identified by the Nuclear Regulatory Commission that it initiated with 18F  in July 2015 but that was not included in General Services  Administration’s (GSA) list of 18F projects. We removed 18 projects that  did not have agency customers. In addition, we removed 1 project that  was terminated without substantial work performed by 18F and 2  projects that, as of March 2016, had not yet been initiated.
Regarding USDS, we removed 2 projects that did not use USDS staff  (e.g., projects that used staff from 18F or an agency digital service  team), and 1 project that did not have an agency customer. We also  consolidated 2 projects into 1 project because the customer agency  considered them to be a single project. The final 32 18F projects and  associated 18 agencies, as well as the final 13 USDS projects and  associated 11 agencies are identified in appendix II.
We administered a data collection instrument to each of the selected  projects about the services they received from 18F and USDS, and the  extent to which the projects were associated with major IT investments.  We then analyzed information obtained from the completed data  collection instruments describing the services they received from 18F and  USDS. We also reviewed information obtained from 18F and USDS  regarding key projects that did not have agency customers.
Additionally, we conducted a web-based survey of the agency managers  of selected 18F and USDS projects. We designed a draft questionnaire in  close collaboration with our survey specialist. We also conducted pretests  with officials at the Environmental Protection Agency, the Office of  Management and Budget (OMB), and GSA. From these pretests, we  made revisions as necessary to reduce the likelihood of overall and item  non-response as well as reporting errors on our questions.
We sent the survey via e-mail to the managers of the selected 32 18F  and 13 USDS projects from January 12, 2016, through February 18,  2016. Log-in information was e-mailed to all contacts. We contacted  project managers by telephone and e-mailed those who had not  completed the questionnaire at multiple points during the data collection  period. We closed the survey on March 31, 2016. We received a  completed questionnaire from the managers of 35 of the 43 selected  projects (81 percent)—27 of the 32 selected 18F projects (84 percent)  and 10 of the 13 selected USDS projects (77 percent).
Because we surveyed all of the project managers and therefore did not  conduct any sampling for our survey, our data are not subject to sampling  errors. However, the practical difficulties of conducting any survey may  introduce non-sampling errors. For example, differences in how a  particular question is interpreted, the sources of information available to  respondents, or the types of people who do not respond to a question can  introduce errors into the survey results. We included steps in both the  data collection and data analysis stages to minimize such non-sampling  errors. Our analysts resolved difficulties that respondents had in  completing our survey. Although the survey responses cannot be used to  generalize the opinions and satisfaction of all customers that receive  services from 18F and USDS programs, the responses provide data for  our defined population.
In our questionnaire we asked the managers of all projects to identify the  extent to which they are satisfied or dissatisfied with the services provided  by 18F and USDS programs. To determine the extent to which both  programs are providing satisfactory services to its customers, we  described the results on a 5-point satisfaction scale, where 5 is “very  satisfied” and 1 is “very dissatisfied.”
To obtain additional narrative and supporting context, survey respondents  were given multiple opportunities to provide additional open-ended  comments throughout our survey. Using these open-ended responses,  we conducted a content analysis in order to identify common factors.
To address the second objective, we reviewed federal laws and guidance  on performance measurement, and GAO’s guidance on investment  management. We then identified the following practices relevant to  entities that provide IT services:
Define outcome-oriented goals and measure performance.
According to federal law and our previous work, outcome-oriented  goals and performance measures are vital to assess the actual  results, effects, or impact of a program or activity compared to its  mission.
Establish and implement procedures for prioritizing IT projects.
According to GAO’s guidance on investment management,  establishing procedures, to include criteria, for prioritizing projects can  help organizations consistently select projects based on their  contributions to the strategic goals of the organization.
We analyzed 18F and USDS policies, procedures, plans, and practices  and compared them to the identified areas.
To address our third objective, we administered a data collection  instrument on plans to establish digital service teams to the 25 agencies  with funding proposed in the President’s Budget for fiscal year 2016.  Additionally, we reviewed USDS’s plans—to include interviews with  USDS officials—for providing assistance to agencies that planned to  establish a digital service team in fiscal year 2016.
In addition, we selected four agencies as case studies to determine the  extent to which agencies had documented the relationships between  digital service teams and agency Chief Information Officers (CIO). To  choose these agencies, we identified the three agencies that had  established a charter with USDS as of January 2016—the Departments of  Defense, Homeland Security, and State. We also selected the  Department of Veterans Affairs because, as of January 2016, it had the  most staff of any agency digital service team.
For these agencies, we evaluated the extent to which agency policies and  procedures, including digital service team charters, clearly defined  responsibilities and authorities governing the relationships between the  CIO and other agency organizations that use IT (in the case of this report,  the other agency organizations that use IT were the agency digital service  teams). Further, we conducted interviews with the CIOs of the  Departments of Defense, Homeland Security, and State, as well as the  Veterans Affairs Principal Deputy Assistant Secretary for the Office of  Information and Technology. We also shared our analysis with OMB  officials to review, comment, and provide additional information.
We conducted this performance audit from July 2015 to August 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Projects for which 18F and U.S. Digital Service Provided Assistance
paragraphs: Between March 2014 and August 2015, the General Services  Administration’s (GSA) 18F staff helped 18 agencies with 32 projects, and  generally provided services relating to its five business units: Custom  Partner Solutions, Products and Platforms, Transformation Services,  Acquisition Services, and Learn. In addition, 18F also provided Agency  Digital Service Team Candidate Qualification Reviews. Table 8 describes  each project, to include the associated agency, project name, project  description, and service provided.
Between August 2014 and August 2015, USDS provided assistance on  13 projects across 11 agencies. USDS generally provided seven types of  consulting services: quality assurance, research, website development,  system stabilization, information security assessment, software  engineering, and data management. Table 9 describes each project, to  include the associated agency, project name, project description, and  service provided.
section_title: Appendix III: Comments from the General Services Administration
section_title: Appendix IV: Comments from the Office of Management and Budget
section_title: Appendix V: Comments from the Department of Housing and Urban Development
section_title: Appendix VI: Comments from the National Archives and Records Administration
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Nick Marinos (Assistant Director), Kavita  Daitnarayan, Rebecca Eyler, Kaelin Kuhn, Jamelyn Payan, and Tina  Torabi.