Performance.gov: Long-Term Strategy Needed to Improve Website Usability
Published: 2016-08-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Congress took steps to improve federal performance reporting through GPRAMA by requiring that OMB provide performance information via a publicly available central website, Performance.gov. In June 2013, GAO reported on the initial development of Performance.gov. GPRAMA includes a provision for GAO to periodically review its implementation. This report assesses OMB's (1) effort to ensure Performance.gov's usefulness, and (2) strategic plan for the website. GAO compared elements of Performance.gov to GSA's Digitalgov.gov requirements for federal websites. GAO summarized prior work on OMB's implementation of selected GPRAMA requirements. GAO also interviewed OMB, PIC, and GSA staff about recommendations GAO made on developing the website and Performance.gov's strategic plan.
section_title: What GAO Found
paragraphs: The Office of Management and Budget (OMB), General Services Administration (GSA), and the Performance Improvement Council (PIC) took several steps to improve the usefulness of Performance.gov, a website intended to serve as the public window to the federal government's goals and performance. However, their actions do not fully meet selected Digitalgov.gov requirements for federal websites (which are based on relevant statutes, regulations, and executive orders) and do not fully meet provisions of the GPRA Modernization Act of 2010 (GPRAMA):
In accordance with Digitalgov.gov, GSA, on behalf of OMB, issued a usability test in September 2013. The test identified issues with the website's accessibility, purpose, data visualizations, and search function. However, OMB and GSA have not addressed all of the test's findings.
OMB and the PIC are tracking 18 of 24 website performance measures required by Digitalgov.gov, but have not set goals for those measures. In June 2013, GAO recommended they track measures and set goals for those measures. However, those recommendations remain open.
OMB has not met all of the GPRAMA public reporting requirements for Performance.gov. In particular, GAO identified several areas where OMB is not fully meeting agency priority and cross-agency priority goal public reporting requirements. OMB and PIC staff told GAO they are aware that Performance.gov is not fully GPRAMA compliant, but in moving forward, are focused on ensuring its compliance.
According to OMB and PIC staff, limited resources have prevented them from taking actions to address the 2013 usability test and setting goals for measures. By not fully implementing Digitalgov.gov requirements and GAO's recommendations on GPRAMA requirements, Performance.gov will continue to have difficulty serving its intended purpose as a central website where users can easily locate government-wide performance information.
OMB does not have a strategic plan for the website that will help guide staff in the future. Specifically, OMB does not have a customer outreach strategy that incorporates, as appropriate, information about how OMB intends to (1) inform users of changes on Performance.gov, (2) use social media as a method of communication, and (3) use mobile devices and applications. OMB also lacks an archiving plan to retain data and content on Performance.gov. Agency-wide strategic planning practices required under law can serve as leading practices for planning at lower levels within federal agencies, such as individual programs or initiatives. Consistent with these practices, strategic plans should contain goals and objectives, approaches, and resources. OMB staff said they had not developed a strategic plan for Performance.gov because they wanted to allow transition time for the operations and website maintenance contractor hired in August 2015. OMB staff also said that, in February 2016, they hired a Digital Services Director to develop a strategic plan and manage the website's long-term development. Without a strategic plan, OMB will not know the resources it needs or the steps to take to meet requirements, and to ensure the site provides useful information to the public.
section_title: What GAO Recommends
paragraphs: GAO is making three recommendations that OMB work with GSA and the PIC to 1) ensure the information presented on Performance.gov consistently complies with GPRAMA public reporting requirements for the website's content; 2) analyze and, where appropriate, implement usability test results to improve Performance.gov; and 3) develop a strategic plan for the future of Performance.gov that includes goals, objectives, and resources needed to meet website requirements; a customer outreach plan; and a strategy to manage and archive data. OMB staff agreed with GAO's recommendations and provided technical clarifications, which GAO incorporated as appropriate.

section_title: Letter
paragraphs: Performance.gov is intended to serve as the public window to the federal  government’s goals and performance in key areas. The Office of  Management and Budget (OMB) created it in response to a requirement  of the GPRA Modernization Act of 2010 (GPRAMA). GPRAMA requires  important performance information, including agency performance plans  and reports, to be available via a central, government-wide website.  GPRAMA further requires OMB to post a list of all federal programs  identified by agencies with related budget and performance information,  and quarterly updates on agency priority goals (APG) and cross-agency  priority (CAP) goals on the website.
In June 2013, we reported on Performance.gov’s development and  identified issues related to unclear performance measures, inconsistent  user experience, and accessibility, navigation, and search capability  challenges. We recommended OMB systematically collect customer  feedback, enhance the usability of the website design, and collect  performance information to develop goals. OMB agreed with our  recommendations and indicated it would take steps to implement them.
This report is part of our response to a mandate to periodically assess the  implementation of the GPRA Modernization Act of 2010. This report  assesses OMB’s (1) efforts to ensure Performance.gov’s usefulness, and  (2) strategic plan for Performance.gov.
To address our objectives, we reviewed the 22 Digitalgov.gov  requirements for federal websites and digital services, and selected 9 for  our assessment of Performance.gov. The selected requirements are  those most associated with customer feedback and outreach, usability,  performance measures, and records management. We assessed OMB’s  efforts (in collaboration with the Performance Improvement Council (PIC)  and the General Services Administration (GSA)) to ensure the usefulness  of Performance.gov and OMB’s strategic plan by comparing the steps  taken and documentation for each to the selected requirements.
To further address our first objective, we examined documentation on  how OMB was (1) seeking information from various audiences about their  needs concerning Performance.gov; (2) ensuring the website was  clarifying ways audiences can use it; and (3) tracking a broader range of  performance and customer satisfaction measures, and setting goals for  those measures. We also used the information to follow up on the status  of the recommendations from our 2013 report. In addition, we reviewed  requirements outlined in GPRAMA for Performance.gov, including public  reporting requirements for APGs, CAP goals, and the federal program  inventory. We also interviewed staff from OMB’s Office of Performance  and Personnel Management, the PIC, and GSA to determine the actions  taken thus far to address our prior recommendations. For reports with  open recommendations as they relate to Performance.gov, see    appendix I.
To further address our second objective, we requested documentation on  OMB’s strategic plan for Performance.gov, its social media and customer  outreach strategy, and its web records plan. However, the agency has not  developed these documents. We interviewed staff from OMB, the PIC,  and GSA on Performance.gov’s strategic plan, social media and  customer outreach strategy, web records plan, and accessibility on  mobile devices. We compared any steps taken on those actions to  Digitalgov.gov requirements. See appendix II for additional information  about the objectives, scope, and methodology of this report.
We conducted this performance audit from July 2015 to August 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: GPRAMA required OMB to establish a single, performance-related  website by October 1, 2012. The site would provide program and  performance information readily accessible and easily found on the  Internet by the public and members of Congress. OMB made  Performance.gov available to the public in August 2011. OMB’s stated  goals for Performance.gov include providing (1) a public view into  government performance to support transparency, and (2) executive  branch management capabilities to enhance senior leadership decision  making. Performance.gov is a repository of federal government  performance information. One of OMB’s main goals of the website is to  link agency programs, their relationships, and contributions to strategic  objectives. This is intended to increase the utility of this information  through enhanced agency comparisons, supporting both benchmarking  and best practice identification.
GPRAMA lists the data elements that are required to be reported on  Performance.gov, including quarterly updates for APGs and CAP goals.  OMB guidance provides more detailed direction to facilitate the  submission of that information. APGs are target areas where agency  leaders want to achieve near-term performance increases. APGs are  often referred to as the agencies’ highest priority performance goals. CAP  goals are outcome-oriented goals covering a limited number of  crosscutting policy areas, as well as goals to improve management  across the federal government. These goals are intended to cover areas  where increased cross-agency collaboration is needed to improve  progress towards shared, complex policy, or management objectives.  This includes attracting foreign investment to spur job growth and  enabling agencies to recruit and hire the best talent. In March 2013,  federal agencies added the first performance updates for CAP goals and  APGs to Performance.gov.
subsections: section_title: Roles and Responsibilities
paragraphs: Several entities maintain and operate Performance.gov. Figure 1 provides  an overview of the Performance.gov governance structure.
Office of Management and Budget. OMB is responsible for setting  the direction, vision, policy, and guidance of Performance.gov and  ensuring its effective operation. OMB has partnered with GSA and the  PIC, and has contract support from eKuber Ventures Inc. to provide  key services for the site. OMB also collaborated with GSA to establish  a Performance Management Line of Business (PMLOB) to further  guide the administration of Performance.gov.
Agencies. GPRAMA requires that agencies make their respective  strategic plans, performance plans and reports, and information about  their APGs (as applicable), including quarterly updates, available for  publication on Performance.gov. Twenty-two agencies have web  pages on Performance.gov that provide links to their strategic plans,  annual performance plans, and annual performance reports; report  agency progress on government-wide management initiatives; and  show agency contributions to the CAP goals. There are 31 additional  agencies that do not have a dedicated web page on Performance.gov.  Instead, they provide links to their strategic plans or to their agency  plans and reports pages. Agencies submit their annual and, as  applicable, quarterly performance information for publication on the  website through the Performance Reporting Entry Portal (PREP)  System. While OMB and PIC give feedback on updates to the  information or suggest changes, ultimately, the agency decides what  information is published on its Performance.gov page.
General Services Administration. GSA builds the technical platform,  provides project management of Performance.gov, and determines  the business requirements and priorities.
Performance Improvement Council. The PIC is chaired by OMB’s  Deputy Director for Management and is composed of Performance  Improvement Officers (PIO) from each of the 24 Chief Financial  Officers (CFO) Act agencies as well as other PIOs and individuals  designated by the Chair. The PIC facilitates the exchange of useful  practices to strengthen agency performance management, such as  through cross-agency working groups. The PIC is supported by an  Executive Director and a team of eight full-time staff who conduct  implementation planning and coordination on crosscutting  performance areas. In coordination with OMB, the PIC provides  several types of guidance to agencies. It also trains agency officials  responsible for updating the quarterly information on Performance.gov  and provides liaisons to answer those officials’ questions.
Contractor (eKuber Ventures Inc.). In August 2015, GSA awarded  software services company eKuber Ventures a contract to provide  operations support and maintenance of Performance.gov and the  PREP system. According to the contract, eKuber will provide training  and information technology help desk support. The current contract  runs through August 2020.
Performance Management Line of Business (PMLOB). PMLOB is  an interagency effort to develop government-wide performance  management capabilities to help meet GPRAMA transparency  requirements. It is also designed to support government-wide  performance management efforts. PMLOB’s key objectives, according  to its 2013 charter, include, among others, developing  Performance.gov into a GPRAMA-compliant data tool.
section_title: Funding
paragraphs: When first established, Performance.gov was funded by GSA’s E- Government Fund. Beginning in 2013, this funding was supplemented by  agency fees. According to the PMLOB Program Charter and PIC staff,  E-Government funding (now funding from the Federal Citizen Services  Fund) was to be primarily used for enhancing the site, developing new  functionality, and focusing on specific areas identified by management.  However, according to agency staff, due to limited funding, they are  focusing on system maintenance rather than new development. Staff also  said Performance.gov received $1,029,922 from the Federal Citizen  Services Fund for fiscal year 2016. However, OMB and PIC staff told us  GSA used $700,000 for other purposes leaving $329,922 for the website.
Additional funding for Performance.gov is collected from 15 agencies  through interagency agreements with GSA. These agreements document  the services GSA will provide and the fees the agencies will pay.  Through the interagency agreements, GSA collects approximately  $795,000 annually—$53,000 from each of the 15 agencies. According  to the PMLOB charter, agency fees are to cover the data collection  capabilities and some operational and maintenance costs. PIC staff told  us they plan to request a 4 percent increase in agency fees in fiscal year  2017 to meet the increasing costs for operating the website.
section_title: Digitalgov.gov Requirements for Federal Websites
paragraphs: Digitalgov.gov contains a checklist of requirements for federal websites  and digital services that are based on relevant statutes, regulations,  executive orders, or policy documents. Digitalgov.gov is managed by  GSA and is designed to help agencies provide digital services and  information to the public. Our review focused on requirements related to  customer feedback and outreach, usability, performance measures, and  records management. See appendix II for additional information on our  selected requirements.
section_title: While OMB Has Enhanced Performance.gov, Further Actions Could Make It More Useful
paragraphs: While OMB, GSA, and the PIC took several steps to improve  Performance.gov, their actions neither meet Digitalgov.gov or GPRAMA  requirements, nor completely address our prior recommendations. For  example, OMB, GSA, and the PIC collect some customer feedback, but  have not engaged broader audiences. While GSA, on behalf on OMB,  also conducted a usability test, OMB has not addressed all of the findings  from that test. Further, OMB and the PIC track 18 of 24 recommended  performance measures, but have not set goals for those measures. In  addition, our prior work identified several areas where OMB is not fully  meeting GPRAMA requirements for Performance.gov. This includes  making all the required information for APGs, CAP goals, and the federal  program inventory available on the website. OMB has not yet  implemented the recommendations we made related to these findings.
subsections: section_title: OMB, GSA, and the PIC Collect Some Customer Feedback but Have Not Engaged Broader Audiences
paragraphs: OMB, GSA, and PIC staff took some steps to prioritize, store, and  address user feedback for the website. According to Digitalgov.gov  requirements, agencies need to understand the needs of their customers  by collecting and addressing customer feedback, and use those data and  feedback to continuously improve programs. Moreover, a focus of  GPRAMA is to make federal performance information more accessible to  the public.
The following methods are used to collect customer feedback and  improve the website:  Website survey. PIC staff told us that a website survey is configured to  appear on Performance.gov for about 20 percent of visits. The website  survey asks users to rate their overall experience on Performance.gov,  how likely they are to return, and how likely they are to recommend it to  someone else, among other things. Survey results from October 2014 to  December 2015 showed that 48 percent of survey respondents rated their  overall experience on Performance.gov an 8 out of 10 or above. Of the  568 respondents during this period, 69 percent were likely or very likely to  return to the website, and 61 percent of respondents were likely or very  likely to recommend Performance.gov to someone else.
Feedback button. OMB and the PIC also collect feedback through the  “Feedback” button on Performance.gov’s home page. The “Feedback”  button leads the user to a web form that sends feedback directly to GSA,  as shown in figure 2. Examples of feedback submitted through the form  include: notification of broken links, comments about outdated agency  information, and questions about where specific information can be found  on the website.
PREP working group. OMB staff stated that they have been focused on  enhancing the internal PREP system that is used by agency officials to  submit the annual and quarterly performance information required to be  on the website by GPRAMA. To help in this effort, the PREP working  group—a voluntary group of agency officials responsible for updating their  respective agency’s performance reporting information—provides  feedback to the contractor on the PREP users’ needs. For example, the  working group members were given the opportunity to test updates to the  PREP system and suggest improvements.
According to PIC staff, feedback submitted through the website survey,  “Feedback” button, PREP working group, and e-mails to the  Performance.gov help desk are prioritized and logged into the product  backlog, a system used by GSA to store feedback. According to PIC staff,  feedback is prioritized based on several factors, including the value it  would bring to a larger audience, the cost and estimated time to  implement, and the risk and opportunity cost of addressing the feedback.  The highest priority items on the product backlog system are placed on  the monthly prioritized list and are handled by the contractor. Examples of  addressed feedback from the product backlog include modifying graphs  to accurately display APG data and updating Performance.gov’s  Frequently Asked Questions page to reflect current information.
section_title: OMB and GSA Conducted a Usability Test, but Key Actions Remain Unaddressed
paragraphs: GSA, on behalf of OMB, conducted a usability test on Performance.gov  and issued the findings in September 2013. Digitalgov.gov recommends  conducting usability testing to collect general feedback from users about  the design and functionality of a website, offering invaluable insights into  what needs improvement. Further, Digitalgov.gov states that simple  usability tests are a quick way to identify major problems and give  agencies the tools to take immediate action to improve the website.
The September 2013 usability test report found several problems with  Performance.gov and made recommendations to improve the usability of  the website, detailed in table 1.
The specific findings of the usability test and status of actions taken to  address those findings are as follows:  Accessibility. The 2013 GSA usability test found portions of  Performance.gov that were inaccessible to people with visual disabilities.  According to OMB staff, changes were made to the website to address  the identified accessibility issues consistent with Section 508  requirements. PIC staff also told us that when the PIC adds new content  to the website, it is tested for accessibility with tools such as screen  readers. Agencies are responsible for ensuring the content they submit to  the website is accessible, such as the quarterly updates. Digitalgov.gov’s  guidance on Section 508 states that accessibility testing should be  conducted before a web page launches or when making significant  changes to digital products and services.
Purpose. Some usability testers who participated in the 2013 GSA  usability test also reported that they were confused about the purpose of  the website because the main page did not clearly explain either  Performance.gov’s purpose or the target audience. Digitalgov.gov plain  writing requirements state that content should be written in a clear,  concise, and well-organized manner. We reviewed related federal  websites and found examples of clearly stated purposes on the home  page. Figure 3 compares the DATA.gov home page to that of  Performance.gov. Data.gov has a simple explanation of the site and uses  icons to depict the available topics. Performance.gov has two large  paragraphs of text explaining the benefits of establishing APGs and CAP  goals, but it does not clarify its purpose as a central, government-wide  website where users can find these APGs and CAP goals for federal  agencies.
Performance.gov’s lack of a simple and clear explanation of its purpose  could potentially confuse users. Without guidance about tasks that can be  accomplished on a website, along with explanations of the different areas  of the website and navigation assistance, website users may be unable to  successfully achieve their objectives.
Data visualizations. The GSA usability test also reported that some  users had trouble locating graphs or data visualizations, and  understanding whether agencies had met target goals. For example, the  usability test noted that users wanted a “goal line” showing target values.  In 2013, we also reported on the importance of making the information  and data on a performance-reporting website engaging and easily  understandable. Figure 4 shows how “Measures of Australia’s Progress,”  a website designed to show users how Australia is progressing, provides  color-coded indicators of how a performance metric is performing relative  to the goal. In contrast, Performance.gov does not have color-coded  indicators or other data visualizations that help users understand if  agencies are meeting their goals.
Search. The 2013 GSA usability test also examined how well the home  page’s search function performed. The test revealed that search terms  were not highlighted in search results and results of the search did not  necessarily match the search terms. For example, when users searched  for the “National Institutes of Health,” search results returned the home  page for the National Science Foundation instead of pages directly  related to the National Institutes of Health. PIC staff told us that the  search algorithm was modified to highlight the search terms in the search  results. However, the algorithm has not been updated to return search  results more reflective of the search terms. The contract with eKuber  discusses an option of improving the search function. However, OMB  staff told us they have not decided whether they will invest in enhancing  the website’s search capability.
OMB staff told us that they have not implemented all the usability test  recommendations because of limited resources. Additionally, while the  eKuber contract, awarded August 19, 2015, has options to address some  usability issues, OMB has not prioritized which usability issues need to be  addressed first or a timeline for addressing these issues. Specifically, the  contract contains an option to enhance the usability of the site, which  GSA can exercise at a later date during the contract period. These  contract options include improving the website search function and  enhancing data visualizations on Performance.gov, among other things.
In 2013, we also found that some Performance.gov users reported issues  with accessibility, navigation, and search capabilities. Specifically, we  found that OMB had not articulated ways that intended audiences, such  as members of the public, Congress, and agency staff, could use the  website or the information available through it to accomplish other specific  tasks. For example, the website gave no indication or examples of the  ways that various audiences could use Performance.gov to facilitate  coordination or communication about goals, activities, and performance  between agencies. At that time, we recommended OMB clarify ways that  intended audiences could use the information on Performance.gov to  accomplish specific tasks and identify the design changes that would be  required to facilitate that change. OMB agreed with our recommendation  and subsequently conducted the usability test. Although the actions OMB  has taken are a step in the right direction, they do not fully address our  prior findings with regard to how people could use the website to  complete specific tasks. As a result, our prior recommendation remains  open. Thus, if usability issues are not addressed, Performance.gov users  could continue to have difficulties using and understanding the content  posted on the site.
Addressing usability issues could also help resolve another open  recommendation from our 2013 report concerning engaging wider  audiences—such as congressional staff and interested members of the  public—to ensure the site is meeting a broad set of user needs.  Specifically, in 2013, we recommended that OMB seek to more  systematically collect information on the needs of a more varied  audience, including through the use of customer satisfaction surveys and  other approaches recommended by leading practices. OMB also agreed  with this recommendation. While the efforts OMB and the PIC have taken  on usability testing and collecting and using customer feedback are steps  in the right direction, these actions do not completely address our prior  recommendation on engaging a potentially broader audience about the  website’s usefulness. As a result, our previous recommendation remains  open.
section_title: OMB and the PIC Have Made Progress in Tracking Additional Performance Measures, but Have Yet to Establish Goals
paragraphs: Digitalgov.gov requirements state managers should track, analyze, and  report on a minimum baseline set of performance, search, customer  satisfaction, and other measures. This allows staff to get a holistic view of  how well online information and services are delivered. As of May 2016,  OMB and the PIC were tracking 18 of the 24 recommended  Digitalgov.gov measures. PIC staff told us that they began using the  Digital Analytics Program (DAP) to track performance measures for  Performance.gov in March 2014. A DAP staff member further explained  that the web analytics program is not currently customized to track the six  remaining measures for Performance.gov.
This represents some improvement from 2013, when we found OMB and  GSA monitored visitors’ use of Performance.gov by tracking 15 of the 24  website performance measures. At that time, we recommended that  OMB seek to ensure that all performance, search, and customer  satisfaction measures, consistent with leading practices, are tracked for  the website, and, where appropriate, OMB should create goals for those  measures to help identify and prioritize potential improvements to  Performance.gov. While OMB agreed with the recommendation, it still  does not track all the measures we recommended and that are required  by Digitalgov.gov.
OMB and the PIC are now tracking four customer satisfaction measures  that were not tracked in 2013. However, two other measures related to  searches—”top referring search terms” and “percentage of visitors using  site search”—that OMB and the PIC tracked in 2013 are not being  tracked in 2016. Table 2 shows the performance measures tracked for  Performance.gov in 2013 as compared to the performance measures  tracked in 2016.
Additionally, OMB and the PIC have not set goals or targets for any of the  measures they are tracking. Under the strategic planning requirements  established under GPRA and enhanced by GPRAMA—which can also  serve as leading practices for planning for individual initiatives—agencies  are to establish performance goals to define the level of performance to  be achieved. In addition, agencies are required to establish performance  indicators to assess the progress towards the goal, and later evaluate  whether the goal has been met. Furthermore, our 2013 report found that  goals or targets had not been established for the performance measures  the agency was tracking. We recommended that OMB develop goals or  targets for those measures. Our prior recommendation remains open.  PIC staff told us that goals were not set for Performance.gov performance  measures, and improvements have not been made because of limited  staffing resources.
In February 2016, the PIC hired a new Digital Services Director who will  be responsible for reviewing performance measures tracked by DAP and  making recommendations for changes to the website accordingly, among  other things. Without tracking all recommended search and customer  satisfaction measures, and establishing goals or targets for these  measures, it will be difficult for OMB and the PIC to know if they are  meeting customer needs and if they are delivering information to identify  and prioritize potential improvements to the website.
section_title: Our Prior Reports Found Performance.gov Does Not Consistently Meet GPRAMA Requirements
paragraphs: Our prior work has found that OMB has not met all of the GPRAMA public  reporting requirements for Performance.gov. In particular, our work  identified several areas where OMB was not fully meeting APG and CAP  goal public reporting requirements. Additionally, the inventory of federal  programs had not been updated on Performance.gov since the  inventory’s initial release in May 2013. Based on recent communications  with OMB staff, these issues have not been fully resolved.
APGs. In September 2015, we reported that Performance.gov provided  limited information on the quality of performance information used to  measure progress on selected APGs. GPRAMA requires agencies to  publicly report on how they are ensuring the accuracy and reliability of the  performance information they use to measure progress towards these  APGs. The six agencies we reviewed for the 2015 report used various  sections of Performance.gov to discuss some of the performance  information quality requirements for APGs. But none of the agencies met  all five GPRAMA requirements for their individual APGs. Moreover,  while there is no place on the website that is set aside to discuss the  quality of performance information for each APG, we found hyperlinks on  Performance.gov to the selected agencies’ performance plans and  reports. However, there was no explanation of where to find discussions  on the quality of performance information in these plans and reports.
We concluded that while OMB had directed agencies to discuss the  quality of APG performance information in their annual performance plans  and reports for several years, the selected agencies’ plans and reports  often did not. We made two recommendations to OMB aimed at  improving public reporting. We recommended that OMB, working with  the PIC, identify practices participating agencies can use to improve their  public reporting in their performance plans and reports of how they are  ensuring the quality of performance information used to measure  progress towards APGs. PIC staff took steps to address this  recommendation. In May 2016, PIC staff reported that they had  summarized the self-assessments completed by the performance  improvement officers (PIOs) and their deputies on their agency’s data  quality policies and procedures. The PIC staff summary we reviewed  identified aspects of data quality in which agencies had generally rated  their performance highest, and other aspects of data quality in which  agencies had rated their performance lowest. We believe the PIC’s efforts  should help PIOs examine their agency’s data quality policies and  procedures, and ultimately improve data quality and the information  provided to external audiences. We also recommended that OMB,  working with the PIC, identify additional changes that are needed for its  guidance to agencies on ensuring the quality of performance information  for APGs on Performance.gov. As of June 2016, OMB has not updated its  guidance.
CAP goals. In May 2016, we reported that OMB and the PIC had  incorporated lessons learned from the prior CAP goal period (2012-2014)  and provided ongoing assistance to CAP goal teams. Based in part on  our June 2014 recommendation, OMB and the PIC updated guidance and  developed a new reporting template to help improve public reporting on  the implementation of CAP goals. OMB and the PIC also implemented  strategies to build agency capacity to work across agency lines, such as  assigning agency goal leaders, providing ongoing guidance and  assistance to CAP goal teams, and holding senior-level reviews. We also  found that the selected CAP goal’s quarterly progress updates— published on Performance.gov—met a number of GPRAMA reporting  requirements, including identifying contributors and reporting strategies  for performance improvement and quarterly results.
Furthermore, in our May 2016 report, while we found that most of the  selected CAP goal teams were consistently reporting the status of  quarterly milestones to track goal progress, they had not established  quarterly targets as required by GPRAMA. Also, all of the selected CAP  goal teams reported that they were working to develop performance  measures. While they were at various stages of the process, they were  not reporting on these efforts consistently. In that report, we  recommended that OMB and the PIC report on Performance.gov the  actions that CAP goal teams are taking, or plan to take, to develop  performance measures and quarterly targets. OMB staff generally agreed  with the recommendation. With improved reporting of performance  information, the CAP goal teams will be better positioned to demonstrate  goal progress at the end of 4-year goal period. OMB has not yet  confirmed the specific actions it plans to take in response to this  recommendation.
Federal program inventory. In October 2014, we found the approach  used by OMB and agencies to develop an inventory of all federal  programs along with related budget and performance information had not  fully met GPRAMA requirements. GPRAMA requires OMB to make this  information publicly available on Performance.gov. PIC staff reported  that the federal program inventory requirement was initially met in fiscal  year 2013 by presenting data from agency plans and reports as PDF  attachments on Performance.gov. However, we found that OMB had not  published an inventory of federal programs on Performance.gov since  2013. Moreover, OMB did not expect an update of the program  inventories to happen before May 2017 because the staff who would work  on the program inventories were heavily involved in Digital Accountability  and Transparency Act of 2014 (DATA Act) implementation. Further, in  October 2014, we reported that implementation of the DATA Act could be  tied to the program inventories because the act requires federal agencies  to publicly report information about any funding made available to, or  expended by, an agency or a component of the agency at least quarterly.  We also found that agency reporting for both sets of requirements was  web based. This could more easily enable linkages between the two or  facilitate incorporating information from each other.
In July 2015, we recommended that OMB should accelerate efforts to  determine how best to merge DATA Act purposes and requirements with  the GPRAMA requirement to produce a federal program inventory. In  April 2016, PIC staff told us that they will work with the DATA Act  implementation team to determine how to best integrate the GPRAMA  and DATA Act requirements. However, they did not provide specific  details. Until OMB determines a strategy to integrate GPRAMA and DATA  Act requirements, OMB will not know what resources or steps it needs to  take to ensure the requirements are met and incorporated on  Performance.gov.
OMB and PIC officials have told us that they are focused on ensuring  Performance.gov is GPRAMA compliant and are aware that the website  is not fully consistent with GPRAMA requirements. Without all the  required GPRAMA information, Performance.gov will not be transparent  and may fall short of meeting user needs.
section_title: OMB Does Not Have a Strategic Plan for the Future of Performance.gov
paragraphs: OMB does not have a strategic plan for the website that will help guide  officials in the future. Specifically, we found that OMB does not have a  customer outreach strategy that incorporates, as appropriate, information  about how OMB intends to (1) inform users of changes on  Performance.gov, (2) use social media as a method of communication,  and (3) use mobile devices and applications. Finally, OMB lacks an  archiving plan that details how OMB plans to manage the data and  content on Performance.gov.
subsections: section_title: OMB Does Not Yet Have a Strategic Plan to Guide the Future Development of Performance.gov
paragraphs: OMB has not developed a strategic plan to guide the future of  Performance.gov. Agency-wide strategic planning practices required  under GPRA, and enhanced by GPRAMA, can serve as leading practices  for planning at lower levels within federal agencies, such as individual  programs or initiatives. Under this law, strategic plans are the starting  point and basic underpinning for results-oriented management. Among  other things, strategic plans should contain goals and objectives,  approaches, and resources needed to achieve those long-term goals and  objectives. When we began our review, OMB staff said that they had not  developed a strategic plan for Performance.gov because a new  contractor, eKuber, had just started a few months prior, and they wanted  to allow time for the contractor to transition into its new role.  eKuber took over the Performance.gov contract in the late summer of  2015. Since that time, OMB and the PIC have taken an important first  step towards developing a strategic plan by hiring a Digital Services  Director in February 2016. The Digital Services Director’s responsibilities  include developing a strategic plan and managing the long-term  development of Performance.gov. Hiring a new Director who has  responsibility for outlining needed improvements is a move in the right  direction. However, without a plan for the future, OMB will not know what  resources it will need or steps it needs to take to ensure all requirements  are met and incorporated on Performance.gov. Such a plan could prove  especially valuable in maintaining continuity during the upcoming  presidential transition.
section_title: OMB and the PIC Do Not Have a Customer Outreach Strategy
paragraphs: OMB and the PIC have not developed a customer outreach strategy. The  Digital Government: Building a 21st Century Platform to Better Serve the  American People strategy calls for federal websites to become more  customer centric by responding to the needs of users and making it  easier to find and share information, and accomplish important tasks  “anytime, anywhere, any device.” It also calls on agencies to embrace  new technologies to drive participation and to develop innovative,  transparent, user-facing products and services efficiently and effectively.  Without developing a user outreach plan, OMB risks being unable to  provide services to its users where they need it most. Specifically, we  identified three areas where Performance.gov did not have a customer  outreach strategy that incorporates, as appropriate, information about  how OMB intends to (1) inform users of changes on Performance.gov, (2)  use social media as a method of communication, and (3) use mobile  devices and applications.
subsections: section_title: Performance.gov Does Not Inform Users of Latest Updates
paragraphs: Performance.gov does not inform users when a new quarterly update has  been published. Website usability guidelines created by GSA in  conjunction with Health and Human Services (HHS) call for websites to  inform users when changes are made. Most of the time, OMB publishes  a blog post alerting users that new quarterly data have been updated;  however, this blog post is made available through www.WhiteHouse.gov,  not Performance.gov. While a blog post on WhiteHouse.gov will inform  some users that the data have been updated, more Performance.gov  users would be informed of the updates if they were cross posted to the  Performance.gov home page.
The lack of alerts is particularly problematic because OMB does not  always meet the deadlines published in OMB Circular No. A-11 (which  provides guidance to agencies) for the APG and CAP goal quarterly  updates. For example, in the 2015 version of OMB Circular No. A-11,  OMB estimated the update for the first quarter of 2016 would be  published around March 17, 2016. However, these data were not  published until March 30, 2016. PIC staff told us that, in some cases,  OMB held the publication date until it could be coordinated with a press  release or a blog posting, which was the case for this quarterly update.  The delay in publishing the quarterly updates highlights the need to  inform users of when the website has been updated.
Other federal agencies have updated users about new information on  their website by including a “Latest News” or “Updates” section on their  home page or by implementing date and time stamps. For example, the  U.S. Census Bureau home page provides users with a “Latest News”  section, along with a calendar of events for the upcoming week, to allow  users to easily access the most up-to-date information (see figure 5).
OMB and PIC staff said that most of the site’s users are other federal  agencies and employees who already generally know when information  will be updated. The public and members of Congress, however, would  not know when information has been updated. In May 2016, PIC staff told  us they are considering a number of options to inform users of changes  on Performance.gov. This includes a banner on the home screen  identifying new information or using time stamps and a timeline to  highlight quarterly updates, among other options. While PIC staff are in  the early stages of planning, we continue to believe communicating  changes on the website will help enhance the usability for all  Performance.gov users. Without a systematic method for informing users  of when the website has been updated, OMB and the PIC are missing an  opportunity to disseminate information to a broader base of users and  meet users where they are.
On a related issue, in some instances, we were unable to determine  when some information on the website was last updated. For example,  we reviewed the content on seven web pages–Overview, Strategies,  Progress Update, Next Steps, Indicators, Continuing Programs and Other  Factors, and Related Goals–associated with 22 APGs. We found that  none of them had a time or date stamp of when the information was last  updated. Date or time stamps can provide users with a clear idea of  when a web page was last reviewed or updated. This increases a  website’s transparency. We also reviewed the Progress Update page of  each of the 22 APGs. On three of those pages, we were unable to identify  any text that would tell the reader when the page was last updated, such  as the quarter or year of the update. According to PIC staff, the  information on the APG pages always reflects the most current  information. As previously noted, GPRAMA requires the APGs to be  updated quarterly. Without a date or time stamp or some indication of  when the data were last updated, system users, including decision  makers, are unable to determine how current the website’s data are.
section_title: OMB Has Not Incorporated Social Media into Its Customer Outreach
paragraphs: Performance.gov provides users with social media links to share  information on the website. However, OMB and the PIC are not using  social media services, such as Twitter or Facebook, to interact with the  site’s users. Digitalgov.gov’s social media requirement states that staff  should use social outreach tools to interact with customers and improve  the customer experience. Furthermore, we found that other federal  websites have social media pages and updates linked on the home page.  For example, the U.S. Department of Housing and Urban Development  (HUD) website has a feed on its home page showing tweets from its  Twitter account. These tweets provide users with the latest information  from HUD, including policy updates and recent news and events related  to HUD’s mission (see figure 6).
According to OMB and PIC staff, they have not used social media for  customer outreach because they have limited staffing resources to  manage a social media strategy. However, in May 2016, OMB and PIC  staff told us they are planning to hire a Communications Advisor who will  be responsible for creating a social media strategy, among other tasks.  This is an important first step. Moving forward with a social media  strategy should help publicize Performance.gov and improve user  experience. Without such a strategy, OMB and the PIC are missing an  opportunity to interact with customers and to improve the customer  experience.
section_title: Performance.gov Is Accessible on Mobile Devices, but OMB Has Not Yet Determined Whether to Pursue Mobile Applications
paragraphs: Performance.gov is accessible on mobile devices, such as mobile phones  and tablets, as Digitalgov.gov recommends. The Performance.gov home  page and the eight main tabs are accessible and readable on a mobile  device. We were able to access agency information and read about  strategic goals and APGs on a mobile device. However, OMB does not  have a mobile application (also known as an app) for Performance.gov.  The U.S. Digital Services Playbook outlines key successful practices from  the private sector that would help the government build effective digital  resources, including mobile applications. According to the playbook, it is  important for staff to understand how users access the website, whether it  is through a computer or mobile device. Digital products may better cater  to how the users interact with the website. OMB and PIC staff told us they  have not determined whether it will be beneficial to develop a mobile  application for its users. Considering the need for a mobile application  when developing a customer outreach plan may help OMB determine  future resource requirements.
section_title: OMB and the PIC Do Not Have a Plan to Maintain Archived Data
paragraphs: As of the end of February 2016, OMB and the PIC no longer maintain a  full archive of Performance.gov. This means that the data that were  previously published as a part of the 2012-2013 APGs are no longer  available to the public. Additionally, on an ongoing basis, OMB and the  PIC do not publish the progress updates from a prior quarter’s update for  any APG, once the newest quarter’s data are released. However,  previous iterations of CAP goal progress have been maintained for the  public to access via the CAP goal update pages. PIC staff told us that the  platform used to previously archive the site was no longer supported. PIC  staff told us they would like to migrate the archived data back onto  Performance.gov so that it can be accessed publicly again, if funding for  the project becomes available.
The Digitalgov.gov requirement for records management cites National  Archives and Records Administration (NARA) guidance on managing web  records, which is based on statutory requirements. This guidance states  federal websites are part of an agency’s approach to serving the public  and agencies should conduct risk assessments to determine what parts  of their websites should be documented and have records kept. Once  an agency determines which content to keep or archive, staff should then  develop a web records schedule to document and store that content. The  PIC does not have a web records schedule to determine how to manage  Performance.gov content and data. Instead, the PIC told us GSA is  instituting a new method of managing and tracking the progress of its web  projects. A web records schedule would provide stakeholders with  important information about plans to archive the data and content  published on Performance.gov. Further, without an archive of  Performance.gov, users can no longer compare long-term agency priority  goals and progress made toward those goals. This affects the website’s  transparency.
section_title: Conclusions
paragraphs: Since Performance.gov was launched in 2011, OMB has worked with the  PIC and GSA to develop a single website that will meet federal  requirements for the public reporting of agency performance information.  While OMB, GSA, and the PIC have taken several steps to improve  Performance.gov, their actions do not fully meet Digitalgov.gov  requirements or completely address our prior recommendations. For  example, while OMB and GSA conducted a usability test of the website,  they have not addressed all of the test’s findings. Further, OMB has  experienced several challenges implementing APG and CAP goal  reporting, and the federal program inventory requirements outlined in  GPRAMA. Without improving usability and fully implementing GPRAMA  requirements, Performance.gov will have difficulty serving its intended  purpose as a central website where users can find government-wide  performance information easily.
OMB has not developed a strategic plan for Performance.gov. OMB and  the PIC took an important first step by hiring a Digital Services Director for  the PIC. OMB now needs to outline the goals and objectives, approaches,  and resources needed for the future development of Performance.gov.  Further, OMB does not have a customer outreach strategy that explores  additional ways to display updated information on the Performance.gov  home page and to use social media and mobile applications for outreach.  Additionally, outlining a plan to manage and archive the content and data  on Performance.gov in a systematic way will increase the transparency of  the website. Without a strategic plan that incorporates all of these areas,  it will be difficult for decision makers to prioritize and plan for future  website improvements. Such a plan could prove especially valuable in  maintaining continuity during the upcoming presidential transition.
section_title: Recommendations for Executive Action
paragraphs: We recommend the Director of the Office of Management and Budget, in  consultation with the Performance Improvement Council and General  Services Administration, take the following three actions:  1.  Ensure the information presented on Performance.gov consistently  complies with GPRAMA public reporting requirements for the  website’s content.  2.  Analyze and, where appropriate, implement usability test results to  improve Performance.gov.  3.  Develop a strategic plan for the future of Performance.gov. Among  other things, this plan should include:  the goals, objectives, and resources needed to consistently meet  Digitalgov.gov and GPRAMA requirements; a customer outreach plan that considers how (1) OMB informs  users of changes in Performance.gov, (2) OMB uses social media  as a method of communication, and (3) users access  Performance.gov so that OMB could, as appropriate, deploy  mobile applications to communicate effectively; and a strategy to manage and archive the content and data on  Performance.gov in accordance with NARA guidance.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Director of OMB and the  Administrator of GSA for review and comment. On August 5, 2016, we  met with OMB, PIC and GSA staff. OMB and PIC staff provided us with  oral comments on the report and we made technical changes as  appropriate. OMB staff agreed with the recommendations in the report.  GSA did not have comments on the report.
We are sending copies of this report to the Director of OMB and the  Administrator of GSA as well as appropriate congressional committees  and other interested parties. In addition, this report is available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.
section_title: Appendix I: Selected Open GAO Recommendations That May Affect Performance.gov
paragraphs: Status update  1.  The agency has not yet provided  reporting on Cross Agency Priority (CAP)  goal progress, we recommend that the  Director of the Office of Management and  Budget (OMB), working with the  Performance Improvement Council (PIC),  take the following action: report on  Performance.gov the actions that CAP  goal teams are taking, or plan to take, to  develop performance measure and  quarterly targets. information on what actions it has taken  in response to this recommendation.
1.  To help participating agencies improve  1.  We reviewed updates OMB published to  their public reporting, the Director of OMB,  working with the PIC Executive Director,  should identify additional changes that  need to be made in OMB’s guidance to  agencies related to ensuring the quality of  performance information for agency  performance goals (APG) on  Performance.gov.
Circular No. A-11 in July 2016. Circular  No. A-11 continues to direct agencies to  provide data quality information for  publication on Performance.gov or to  provide a hyperlink from  Performance.gov to relevant explanation  in agencies' performance reports, which  was a requirement OMB added in June  2015 in response to our preliminary  findings. However, our review found that  OMB also needed to update the  template agencies complete for  Performance.gov updates to make it  easier for agencies to publish this  information on the website. The July  2016 update of Circular No.  A-11 does  not indicate whether this template has  been updated or whether additional  changes to A-11 are needed. We will  continue to monitor OMB and the PIC’s  efforts to address our recommendation.
1.  To ensure that federal program spending  1. data are provided to the public in a  transparent, useful, and timely manner,  the Director of OMB should accelerate  efforts to determine how best to merge  Digital Accountability and Transparency  Act (DATA Act) purposes and  requirements with the GPRA  Modernization Act of 2010 (GPRAMA)  requirement to produce a federal program  inventory.
In April 2016, OMB staff told us that  identifying programs for the purposes of  DATA Act reporting would not be  completed until after May 2017.  However, they said they have convened  a working group to develop and vet a  set of options to establish a  government-wide definition for program  that is meaningful across multiple  communities and contexts (such as  budget, contracting, and grants).
Status update  1. federal program inventory requirements  and to make the inventories more useful,  the Director of OMB should better present  a more coherent picture of all federal  programs:  revise relevant guidance to direct  agencies to collaborate with each  other in defining and identifying  programs that contribute to common  outcomes;  revise relevant guidance to provide a  time frame for what constitutes  “persistent over time” that agencies  can use as a decision rule for  whether to include short-term efforts  as programs; and  define plans for when additional  agencies will be required to develop  program inventories.
In June 2016, OMB staff stated that they  have not taken any actions in response  to our recommendations related to the  federal program inventory, as they  continue to determine how best to  implement inventory requirements in  coordination with those of the DATA Act.  In our July 2015 testimony on DATA Act  implementation, we recommended that  OMB accelerate efforts to determine  how to best merge DATA Act purposes  and requirements with the GPRAMA  requirement to produce a federal  program inventory. However, at the  same hearing, the Acting Deputy  Director for Management and Controller  at OMB stated that he did not expect an  update of the program inventories to  happen before May 2017.
2.  To ensure the effective implementation of  2.  As of June 2016, OMB had not taken  federal program inventory requirements  and to make the inventories more useful,  the Director of OMB should, to better  present a more coherent picture of all  federal programs, include tax  expenditures in the federal program  inventory effort by  designating tax expenditure as a  program type in relevant guidance;  and  developing, in coordination with the  Secretary of the Treasury, a tax  expenditure inventory that identifies  each tax expenditure and provides a  description of how the tax  expenditure is defined, its purpose,  and related performance and budget  information. action to include tax expenditures in the  federal program inventory. GPRAMA  requires OMB to publish a list of all  federal programs on a central,  government-wide website. The federal  program inventory is the primary tool for  agencies to identify programs that  contribute to their goals, according to  OMB’s guidance. By including tax  expenditures in the inventory, OMB  could help ensure that agencies are  properly identifying the contributions of  tax expenditures to the achievement of  their goals. Although OMB published an  initial inventory covering the programs of  24 federal agencies in May 2013, OMB  decided to postpone further  development of the inventory in order to  coordinate with the implementation of  the DATA Act. In our July 2015  testimony, we recommended that OMB  accelerate efforts to merge DATA Act  purposes with the production of a  federal program inventory.
subsections: section_title: Related report
paragraphs: Status update  3.
In June 2016, OMB staff stated that they  have not taken any actions in response  to our recommendations related to the  federal program inventory, as they  continue to determine how best to  implement inventory requirements in  coordination with those of the DATA Act.  In our July 2015 testimony on DATA Act  implementation, we recommended that  OMB accelerate efforts to determine  how best to merge DATA Act purposes  and requirements with the GPRAMA  requirement to produce a federal  program inventory. However, at the  same hearing, the Acting Deputy  Director for Management and Controller  at OMB stated that he did not expect an  update of the program inventories to  happen before May 2017 because the  staff that would work on the program  inventories were heavily involved in  DATA Act implementation. revise relevant guidance to direct  agencies to consult with relevant  congressional committees and  stakeholders on their program  definition approach and identified  programs when developing or  updating their inventories;  revise relevant guidance to direct  agencies to identify in their  inventories the performance goal(s)  to which each program contributes;  and  ensure, during OMB reviews of  inventories, that agencies  consistently identify, as applicable,  the strategic goals, strategic  objectives, APGs, and CAP goals  each program supports.
Status update  1.  OMB has taken some steps to address  could use the information on the  Performance.gov website to accomplish  specific tasks and specify the design  changes that would be required to  facilitate that use. this recommendation, but additional  actions are needed. The General  Services Administration, on behalf of  OMB, issued a usability test report on  Performance.gov in September 2013.  The test found that (1) sections of the  website were not accessible; (2) users  were unclear about the purpose of  Performance.gov, its intended  audiences, and what users can do on  the website; (3) users had difficultly  locating graphics and understanding if  agencies had met their goals; and (4)  the search functionality produced poor  results and the search terms were not  highlighted. This usability test produced  several recommendations based on  these findings. According to OMB and  PIC staff in August 2015 and May 2016,  they have taken some actions to  address the recommendations. For  example, staff addressed the  accessibility issue and partly addressed  the search issue. However, OMB has  not yet addressed the other two  recommendations. Further, OMB and  the PIC have not clarified the ways that  intended audiences can use the  information on the website to  accomplish specific tasks, or specified  design changes that would be required  to facilitate that use, as we described in  our report. We will continue to monitor  progress.
section_title: Related report
paragraphs: Status update  2.  OMB, GSA and the PIC systematically  information on the needs of a broader  audience, including through the use of  customer satisfaction surveys and other  approaches recommended by federal  guidance. collect information on the needs of its  users by implementing a website  survey, a feedback form on  Performance.gov, and a working group  focused on improving the PREP system.  Staff have set up a backlog system to  prioritize, store, and address user  feedback. According to PIC staff,  feedback is prioritized based on several  factors. These factors include the value  it would bring to a larger audience, the  cost and estimated time to implement,  and the risk and opportunity cost of  addressing the feedback. The highest  priority items on the product backlog  system are placed on the monthly  prioritized list for the contractor to begin  work on. However, OMB and the PIC  have not identified, engaged, or  collected information on the needs of a  broader audience, such as interested  members of the public, and how those  needs might be addressed through the  website.
3.  seek to ensure that all performance,  3.  As of May 2016, OMB and the PIC were  search, and customer satisfaction metrics,  consistent with leading practices outlined  in federal guidance, are tracked for the  website, and, where appropriate, create  goals for those metrics to help identify and  prioritize potential improvements to  Performance.gov. monitoring 18 of the 24 recommended  performance measures. PIC staff said  that they now track the performance  measures through the Digital Analytics  Program, which does not track the  remaining six measures. OMB and PIC  staff have not created goals for the  measures they track to help identify and  prioritize potential improvements to  Performance.gov. We will continue to  monitor progress.
1.  To ensure that the PIC has a clear plan  1.  The PIC developed a strategic plan for  for accomplishing its goals and  evaluating its progress, the Director of  OMB should work with the PIC to update  its strategic plan and review the PIC’s  goals, measures, and strategies for  achieving performance, and revise them  if appropriate.
2015, which identified its mission, goals  and strategies, and core responsibilities  for achieving them. PIC staff reported  that they plan to update the document for  2016 with a more robust plan.
Status update  1.  According to information provided by  guidance are made, the Director of OMB  should work with the PIC to test and  implement these provisions.
OMB and PIC staff in June 2015,  although OMB revised its guidance as  we recommended, it did not work with  the PIC to test implementation of these  provisions. Instead, they told us that  both PIC and OMB staff ensure  agencies are implementing these  provisions of their guidance when  reviewing agencies’ quarterly update  submissions for APGs. However, our  analysis of agencies’ updates in July  2014 found implementation of these  provisions continues to be mixed. We  will continue to monitor progress.
Performance.gov to include additional  information about APGs, the Director of  OMB should ensure that agencies adhere  to OMB’s guidance for website updates by  providing complete information about the  organizations, program activities,  regulations, tax expenditures, policies,  and other activities—both within and  external to the agency—that contribute to  each APG. ensure that agencies provide complete  information about tax expenditures  contributing to their APGs. According to  information provided by OMB staff in  April 2015, agencies were asked to  identify organizations, program  activities, regulations, policies, tax  expenditures, and other activities  contributing to their 2014-2015 APGs.  This process began as part of the  September 2014 update to  Performance.gov, with opportunities for  revisions in subsequent quarterly  updates. We found that while agencies  had made progress in identifying  external organizations and programs for  their APGs, they did not present this  information consistently on  Performance.gov. Although each APG  web page has a location where  agencies are to identify contributing  programs, agencies did not always  identify external organizations and  programs there. Instead, they identified  these external contributors elsewhere,  such as APG overview or strategy  sections. In June 2015, OMB staff said  they would work with agency officials to  help ensure information is presented in  the appropriate area of  Performance.gov in future updates.
section_title: Related report
paragraphs: Performance.gov to include additional  information about APGs, the Director of  OMB should ensure that agencies adhere  to OMB’s guidance for website updates by  providing a description of how input from  congressional consultations was  incorporated into each APG.
Performance.gov for the 2014-2015  APGs generally found that either  agencies did not include this information  or they had not updated it to reflect the  most recent round of stakeholder  engagement. In June 2015, OMB staff  reported that they will focus agency  attention on this issue during the  development of the 2016-2017 APGs, to  be published in October 2015. We will  continue to monitor progress.
section_title: Appendix II: Objectives, Scope, and Methodology
paragraphs: This report is part of our response to a mandate to assess the  implementation of the GPRA Modernization Act of 2010 (GPRAMA). This  report assesses the Office of Management and Budget’s (OMB) (1)  efforts to ensure the usefulness of Performance.gov, and (2) strategic  plan for Performance.gov.
To address our objectives, we reviewed the 22 Digitalgov.gov  requirements for federal websites and digital services, and selected 9 for  our assessment of Performance.gov, as shown in table 4. The selected  requirements are those most associated with customer feedback and  outreach, usability, performance measures, and records management.
We assessed OMB’s efforts (in collaboration with the Performance  Improvement Council (PIC) and the General Services Administration  (GSA)) to ensure the usefulness of Performance.gov and OMB’s strategic  plan by comparing steps taken and documentation for each to the  selected requirements.
To further address our first objective, we examined documentation on  how OMB was (1) seeking information from various audiences about their  needs concerning Performance.gov, (2) ensuring the website was  clarifying ways audiences can use Performance.gov, and (3) tracking a  broader range of performance and customer satisfaction measures and  setting goals for those measures. We used the information to follow up on  the status of the recommendations in our 2013 report. Further, we  collected documentation on customer service feedback, including survey  data collected from the website survey and customer service feedback  logs, and analyzed the results. We reviewed the findings of the  September 2013 usability test GSA conducted. We collected  documentation on the performance measures tracked for  Performance.gov and compared it with the recommended measures on  Digitialgov.gov. We reviewed requirements for agency performance plans  established under the Government Performance and Results Act of 1993  and enhanced by GPRAMA, and used them as a source for leading  practices on setting goals for collected performance measures and  customer service feedback.
We reviewed requirements outlined in GPRAMA for Performance.gov,  including public reporting requirements for agency priority goals (APGs),  cross-agency priority (CAP) goals, and the federal program inventory.  We reviewed other related guidance, such as OMB Circular No. A-11,  Preparation, Submission, and Execution of the Budget. We reviewed  several of our prior related reports and summarized the findings and  recommendations related to OMB’s implementation of the selected  GPRAMA requirements. We selected these reports because they  represent our most current reports on the implementation of these  selected GPRAMA requirements.
We also interviewed staff from OMB’s Office of Performance and  Personnel Management, the PIC, and GSA to determine the actions  taken in response to our recommendations about clarifying ways intended  audiences can use Performance.gov, systematically collecting feedback  from a broader audience, and tracking recommended performance  measures. Further, we communicated with OMB and PIC staff to  determine the actions taken thus far to address our prior  recommendations in relation to APGs, CAP goals, and the federal  program inventory.
To further address our second objective, we requested documentation on  OMB’s strategic plan for Performance.gov, social media and customer  outreach strategy, and web records plan. We found the agency has not  developed these documents. We also interviewed staff from OMB, the  PIC, and GSA on the Performance.gov website’s strategic plan, social  media and customer outreach strategy, web records plan, and  accessibility on mobile devices. We compared any steps taken on those  actions to Digitalgov.gov requirements.
We reviewed the Digital Services Playbook, which outlines key successful  practices from the private sector that would help the government build  effective digital resources. The playbook provides guidance on  interacting with users through different channels. It was used to analyze  the extent to which Performance.gov is accessible on mobile devices. We  reviewed Performance.gov to determine if it was accessible on mobile  devices by visiting each main tab of the website on multiple devices. We  also identified NARA guidance on web records management. This  guidance assists agencies in how to properly manage and schedule web  records. We reviewed the Research-Based Web Design and Usability  Guidelines for guidelines about informing website users of changes to the  website. The Research-Based Web Design and Usability Guidelines  provide guidance on a broad range of web design and communication  issues.
We also reviewed a selection of APG pages on Performance.gov to  document whether the page had an indication of the last time it was  updated. We concentrated on evaluating the 22 agencies listed on  Performance.gov, which are mostly department level agencies. The  additional 31 agencies listed on Performance.gov do not have dedicated  pages on the website, and therefore were not evaluated as part of this  analysis. For the 22 selected agencies, we focused on the APG’s for  fiscal years 2014 to 2015 because the number of APGs for fiscal years  2016 to 2017 was not yet finalized at the time of the analysis. We  randomly selected one APG from each of the 22 agency pages for  review. For each selected APG, we reviewed all of the tabs to determine  if there were any date or time stamps to indicate the last time the page  was updated. We also reviewed in more detail the Progress Update and  Indicators pages to evaluate whether users could determine through page  context when the information or data on a given page was last updated.
We conducted this performance audit from July 2015 to August 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Lisa Pearson, Assistant Director;  Sonya Phillips, Analyst-in-Charge, supervised the development of this  report. Caroline Prado, Robyn Trotter, and Edith Yuh made significant  contributions to all aspects of this report. Karin Fangman, Robert  Gebhart, Kirsten Lauber, and Donna Miller provided additional  assistance.