Performance.gov: Long-Term Strategy Needed to Improve Website Usability
Released: 2016-08-30T00:00:00

Letter

Background

GPRAMA required OMB to establish a single, performance-related website by October 1, 2012. The site would provide program and performance information readily accessible and easily found on the Internet by the public and members of Congress. OMB made Performance.gov available to the public in August 2011. OMB’s stated goals for Performance.gov include providing (1) a public view into government performance to support transparency, and (2) executive branch management capabilities to enhance senior leadership decision making. Performance.gov is a repository of federal government performance information. One of OMB’s main goals of the website is to link agency programs, their relationships, and contributions to strategic objectives. This is intended to increase the utility of this information through enhanced agency comparisons, supporting both benchmarking and best practice identification.

GPRAMA lists the data elements that are required to be reported on Performance.gov, including quarterly updates for APGs and CAP goals. OMB guidance provides more detailed direction to facilitate the submission of that information. APGs are target areas where agency leaders want to achieve near-term performance increases. APGs are often referred to as the agencies’ highest priority performance goals. CAP goals are outcome-oriented goals covering a limited number of crosscutting policy areas, as well as goals to improve management across the federal government. These goals are intended to cover areas where increased cross-agency collaboration is needed to improve progress towards shared, complex policy, or management objectives. This includes attracting foreign investment to spur job growth and enabling agencies to recruit and hire the best talent. In March 2013, federal agencies added the first performance updates for CAP goals and APGs to Performance.gov.

Roles and Responsibilities

Several entities maintain and operate Performance.gov. Figure 1 provides an overview of the Performance.gov governance structure.

Office of Management and Budget. OMB is responsible for setting the direction, vision, policy, and guidance of Performance.gov and ensuring its effective operation. OMB has partnered with GSA and the PIC, and has contract support from eKuber Ventures Inc. to provide key services for the site. OMB also collaborated with GSA to establish a Performance Management Line of Business (PMLOB) to further guide the administration of Performance.gov.

Agencies. GPRAMA requires that agencies make their respective strategic plans, performance plans and reports, and information about their APGs (as applicable), including quarterly updates, available for publication on Performance.gov. Twenty-two agencies have web pages on Performance.gov that provide links to their strategic plans, annual performance plans, and annual performance reports; report agency progress on government-wide management initiatives; and show agency contributions to the CAP goals. There are 31 additional agencies that do not have a dedicated web page on Performance.gov. Instead, they provide links to their strategic plans or to their agency plans and reports pages. Agencies submit their annual and, as applicable, quarterly performance information for publication on the website through the Performance Reporting Entry Portal (PREP) System. While OMB and PIC give feedback on updates to the information or suggest changes, ultimately, the agency decides what information is published on its Performance.gov page.

General Services Administration. GSA builds the technical platform, provides project management of Performance.gov, and determines the business requirements and priorities.

Performance Improvement Council. The PIC is chaired by OMB’s Deputy Director for Management and is composed of Performance Improvement Officers (PIO) from each of the 24 Chief Financial Officers (CFO) Act agencies as well as other PIOs and individuals designated by the Chair. The PIC facilitates the exchange of useful practices to strengthen agency performance management, such as through cross-agency working groups. The PIC is supported by an Executive Director and a team of eight full-time staff who conduct implementation planning and coordination on crosscutting performance areas. In coordination with OMB, the PIC provides several types of guidance to agencies. It also trains agency officials responsible for updating the quarterly information on Performance.gov and provides liaisons to answer those officials’ questions.

Contractor (eKuber Ventures Inc.). In August 2015, GSA awarded software services company eKuber Ventures a contract to provide operations support and maintenance of Performance.gov and the PREP system. According to the contract, eKuber will provide training and information technology help desk support. The current contract runs through August 2020.

Performance Management Line of Business (PMLOB). PMLOB is an interagency effort to develop government-wide performance management capabilities to help meet GPRAMA transparency requirements. It is also designed to support government-wide performance management efforts. PMLOB’s key objectives, according to its 2013 charter, include, among others, developing Performance.gov into a GPRAMA-compliant data tool.

Funding

When first established, Performance.gov was funded by GSA’s E- Government Fund. Beginning in 2013, this funding was supplemented by agency fees. According to the PMLOB Program Charter and PIC staff, E-Government funding (now funding from the Federal Citizen Services Fund) was to be primarily used for enhancing the site, developing new functionality, and focusing on specific areas identified by management. However, according to agency staff, due to limited funding, they are focusing on system maintenance rather than new development. Staff also said Performance.gov received $1,029,922 from the Federal Citizen Services Fund for fiscal year 2016. However, OMB and PIC staff told us GSA used $700,000 for other purposes leaving $329,922 for the website.

Additional funding for Performance.gov is collected from 15 agencies through interagency agreements with GSA. These agreements document the services GSA will provide and the fees the agencies will pay. Through the interagency agreements, GSA collects approximately $795,000 annually—$53,000 from each of the 15 agencies. According to the PMLOB charter, agency fees are to cover the data collection capabilities and some operational and maintenance costs. PIC staff told us they plan to request a 4 percent increase in agency fees in fiscal year 2017 to meet the increasing costs for operating the website.

Digitalgov.gov Requirements for Federal Websites

Digitalgov.gov contains a checklist of requirements for federal websites and digital services that are based on relevant statutes, regulations, executive orders, or policy documents. Digitalgov.gov is managed by GSA and is designed to help agencies provide digital services and information to the public. Our review focused on requirements related to customer feedback and outreach, usability, performance measures, and records management. See appendix II for additional information on our selected requirements.

While OMB Has Enhanced Performance.gov, Further Actions Could Make It More Useful

While OMB, GSA, and the PIC took several steps to improve Performance.gov, their actions neither meet Digitalgov.gov or GPRAMA requirements, nor completely address our prior recommendations. For example, OMB, GSA, and the PIC collect some customer feedback, but have not engaged broader audiences. While GSA, on behalf on OMB, also conducted a usability test, OMB has not addressed all of the findings from that test. Further, OMB and the PIC track 18 of 24 recommended performance measures, but have not set goals for those measures. In addition, our prior work identified several areas where OMB is not fully meeting GPRAMA requirements for Performance.gov. This includes making all the required information for APGs, CAP goals, and the federal program inventory available on the website. OMB has not yet implemented the recommendations we made related to these findings.

OMB, GSA, and the PIC Collect Some Customer Feedback but Have Not Engaged Broader Audiences

OMB, GSA, and PIC staff took some steps to prioritize, store, and address user feedback for the website. According to Digitalgov.gov requirements, agencies need to understand the needs of their customers by collecting and addressing customer feedback, and use those data and feedback to continuously improve programs. Moreover, a focus of GPRAMA is to make federal performance information more accessible to the public.

The following methods are used to collect customer feedback and improve the website: Website survey. PIC staff told us that a website survey is configured to appear on Performance.gov for about 20 percent of visits. The website survey asks users to rate their overall experience on Performance.gov, how likely they are to return, and how likely they are to recommend it to someone else, among other things. Survey results from October 2014 to December 2015 showed that 48 percent of survey respondents rated their overall experience on Performance.gov an 8 out of 10 or above. Of the 568 respondents during this period, 69 percent were likely or very likely to return to the website, and 61 percent of respondents were likely or very likely to recommend Performance.gov to someone else.

Feedback button. OMB and the PIC also collect feedback through the “Feedback” button on Performance.gov’s home page. The “Feedback” button leads the user to a web form that sends feedback directly to GSA, as shown in figure 2. Examples of feedback submitted through the form include: notification of broken links, comments about outdated agency information, and questions about where specific information can be found on the website.

PREP working group. OMB staff stated that they have been focused on enhancing the internal PREP system that is used by agency officials to submit the annual and quarterly performance information required to be on the website by GPRAMA. To help in this effort, the PREP working group—a voluntary group of agency officials responsible for updating their respective agency’s performance reporting information—provides feedback to the contractor on the PREP users’ needs. For example, the working group members were given the opportunity to test updates to the PREP system and suggest improvements.

According to PIC staff, feedback submitted through the website survey, “Feedback” button, PREP working group, and e-mails to the Performance.gov help desk are prioritized and logged into the product backlog, a system used by GSA to store feedback. According to PIC staff, feedback is prioritized based on several factors, including the value it would bring to a larger audience, the cost and estimated time to implement, and the risk and opportunity cost of addressing the feedback. The highest priority items on the product backlog system are placed on the monthly prioritized list and are handled by the contractor. Examples of addressed feedback from the product backlog include modifying graphs to accurately display APG data and updating Performance.gov’s Frequently Asked Questions page to reflect current information.

OMB and GSA Conducted a Usability Test, but Key Actions Remain Unaddressed

GSA, on behalf of OMB, conducted a usability test on Performance.gov and issued the findings in September 2013. Digitalgov.gov recommends conducting usability testing to collect general feedback from users about the design and functionality of a website, offering invaluable insights into what needs improvement. Further, Digitalgov.gov states that simple usability tests are a quick way to identify major problems and give agencies the tools to take immediate action to improve the website.

The September 2013 usability test report found several problems with Performance.gov and made recommendations to improve the usability of the website, detailed in table 1.

The specific findings of the usability test and status of actions taken to address those findings are as follows: Accessibility. The 2013 GSA usability test found portions of Performance.gov that were inaccessible to people with visual disabilities. According to OMB staff, changes were made to the website to address the identified accessibility issues consistent with Section 508 requirements. PIC staff also told us that when the PIC adds new content to the website, it is tested for accessibility with tools such as screen readers. Agencies are responsible for ensuring the content they submit to the website is accessible, such as the quarterly updates. Digitalgov.gov’s guidance on Section 508 states that accessibility testing should be conducted before a web page launches or when making significant changes to digital products and services.

Purpose. Some usability testers who participated in the 2013 GSA usability test also reported that they were confused about the purpose of the website because the main page did not clearly explain either Performance.gov’s purpose or the target audience. Digitalgov.gov plain writing requirements state that content should be written in a clear, concise, and well-organized manner. We reviewed related federal websites and found examples of clearly stated purposes on the home page. Figure 3 compares the DATA.gov home page to that of Performance.gov. Data.gov has a simple explanation of the site and uses icons to depict the available topics. Performance.gov has two large paragraphs of text explaining the benefits of establishing APGs and CAP goals, but it does not clarify its purpose as a central, government-wide website where users can find these APGs and CAP goals for federal agencies.

Performance.gov’s lack of a simple and clear explanation of its purpose could potentially confuse users. Without guidance about tasks that can be accomplished on a website, along with explanations of the different areas of the website and navigation assistance, website users may be unable to successfully achieve their objectives.

Data visualizations. The GSA usability test also reported that some users had trouble locating graphs or data visualizations, and understanding whether agencies had met target goals. For example, the usability test noted that users wanted a “goal line” showing target values. In 2013, we also reported on the importance of making the information and data on a performance-reporting website engaging and easily understandable. Figure 4 shows how “Measures of Australia’s Progress,” a website designed to show users how Australia is progressing, provides color-coded indicators of how a performance metric is performing relative to the goal. In contrast, Performance.gov does not have color-coded indicators or other data visualizations that help users understand if agencies are meeting their goals.

Search. The 2013 GSA usability test also examined how well the home page’s search function performed. The test revealed that search terms were not highlighted in search results and results of the search did not necessarily match the search terms. For example, when users searched for the “National Institutes of Health,” search results returned the home page for the National Science Foundation instead of pages directly related to the National Institutes of Health. PIC staff told us that the search algorithm was modified to highlight the search terms in the search results. However, the algorithm has not been updated to return search results more reflective of the search terms. The contract with eKuber discusses an option of improving the search function. However, OMB staff told us they have not decided whether they will invest in enhancing the website’s search capability.

OMB staff told us that they have not implemented all the usability test recommendations because of limited resources. Additionally, while the eKuber contract, awarded August 19, 2015, has options to address some usability issues, OMB has not prioritized which usability issues need to be addressed first or a timeline for addressing these issues. Specifically, the contract contains an option to enhance the usability of the site, which GSA can exercise at a later date during the contract period. These contract options include improving the website search function and enhancing data visualizations on Performance.gov, among other things.

In 2013, we also found that some Performance.gov users reported issues with accessibility, navigation, and search capabilities. Specifically, we found that OMB had not articulated ways that intended audiences, such as members of the public, Congress, and agency staff, could use the website or the information available through it to accomplish other specific tasks. For example, the website gave no indication or examples of the ways that various audiences could use Performance.gov to facilitate coordination or communication about goals, activities, and performance between agencies. At that time, we recommended OMB clarify ways that intended audiences could use the information on Performance.gov to accomplish specific tasks and identify the design changes that would be required to facilitate that change. OMB agreed with our recommendation and subsequently conducted the usability test. Although the actions OMB has taken are a step in the right direction, they do not fully address our prior findings with regard to how people could use the website to complete specific tasks. As a result, our prior recommendation remains open. Thus, if usability issues are not addressed, Performance.gov users could continue to have difficulties using and understanding the content posted on the site.

Addressing usability issues could also help resolve another open recommendation from our 2013 report concerning engaging wider audiences—such as congressional staff and interested members of the public—to ensure the site is meeting a broad set of user needs. Specifically, in 2013, we recommended that OMB seek to more systematically collect information on the needs of a more varied audience, including through the use of customer satisfaction surveys and other approaches recommended by leading practices. OMB also agreed with this recommendation. While the efforts OMB and the PIC have taken on usability testing and collecting and using customer feedback are steps in the right direction, these actions do not completely address our prior recommendation on engaging a potentially broader audience about the website’s usefulness. As a result, our previous recommendation remains open.

OMB and the PIC Have Made Progress in Tracking Additional Performance Measures, but Have Yet to Establish Goals

Digitalgov.gov requirements state managers should track, analyze, and report on a minimum baseline set of performance, search, customer satisfaction, and other measures. This allows staff to get a holistic view of how well online information and services are delivered. As of May 2016, OMB and the PIC were tracking 18 of the 24 recommended Digitalgov.gov measures. PIC staff told us that they began using the Digital Analytics Program (DAP) to track performance measures for Performance.gov in March 2014. A DAP staff member further explained that the web analytics program is not currently customized to track the six remaining measures for Performance.gov.

This represents some improvement from 2013, when we found OMB and GSA monitored visitors’ use of Performance.gov by tracking 15 of the 24 website performance measures. At that time, we recommended that OMB seek to ensure that all performance, search, and customer satisfaction measures, consistent with leading practices, are tracked for the website, and, where appropriate, OMB should create goals for those measures to help identify and prioritize potential improvements to Performance.gov. While OMB agreed with the recommendation, it still does not track all the measures we recommended and that are required by Digitalgov.gov.

OMB and the PIC are now tracking four customer satisfaction measures that were not tracked in 2013. However, two other measures related to searches—”top referring search terms” and “percentage of visitors using site search”—that OMB and the PIC tracked in 2013 are not being tracked in 2016. Table 2 shows the performance measures tracked for Performance.gov in 2013 as compared to the performance measures tracked in 2016.

Additionally, OMB and the PIC have not set goals or targets for any of the measures they are tracking. Under the strategic planning requirements established under GPRA and enhanced by GPRAMA—which can also serve as leading practices for planning for individual initiatives—agencies are to establish performance goals to define the level of performance to be achieved. In addition, agencies are required to establish performance indicators to assess the progress towards the goal, and later evaluate whether the goal has been met. Furthermore, our 2013 report found that goals or targets had not been established for the performance measures the agency was tracking. We recommended that OMB develop goals or targets for those measures. Our prior recommendation remains open. PIC staff told us that goals were not set for Performance.gov performance measures, and improvements have not been made because of limited staffing resources.

In February 2016, the PIC hired a new Digital Services Director who will be responsible for reviewing performance measures tracked by DAP and making recommendations for changes to the website accordingly, among other things. Without tracking all recommended search and customer satisfaction measures, and establishing goals or targets for these measures, it will be difficult for OMB and the PIC to know if they are meeting customer needs and if they are delivering information to identify and prioritize potential improvements to the website.

Our Prior Reports Found Performance.gov Does Not Consistently Meet GPRAMA Requirements

Our prior work has found that OMB has not met all of the GPRAMA public reporting requirements for Performance.gov. In particular, our work identified several areas where OMB was not fully meeting APG and CAP goal public reporting requirements. Additionally, the inventory of federal programs had not been updated on Performance.gov since the inventory’s initial release in May 2013. Based on recent communications with OMB staff, these issues have not been fully resolved.

APGs. In September 2015, we reported that Performance.gov provided limited information on the quality of performance information used to measure progress on selected APGs. GPRAMA requires agencies to publicly report on how they are ensuring the accuracy and reliability of the performance information they use to measure progress towards these APGs. The six agencies we reviewed for the 2015 report used various sections of Performance.gov to discuss some of the performance information quality requirements for APGs. But none of the agencies met all five GPRAMA requirements for their individual APGs. Moreover, while there is no place on the website that is set aside to discuss the quality of performance information for each APG, we found hyperlinks on Performance.gov to the selected agencies’ performance plans and reports. However, there was no explanation of where to find discussions on the quality of performance information in these plans and reports.

We concluded that while OMB had directed agencies to discuss the quality of APG performance information in their annual performance plans and reports for several years, the selected agencies’ plans and reports often did not. We made two recommendations to OMB aimed at improving public reporting. We recommended that OMB, working with the PIC, identify practices participating agencies can use to improve their public reporting in their performance plans and reports of how they are ensuring the quality of performance information used to measure progress towards APGs. PIC staff took steps to address this recommendation. In May 2016, PIC staff reported that they had summarized the self-assessments completed by the performance improvement officers (PIOs) and their deputies on their agency’s data quality policies and procedures. The PIC staff summary we reviewed identified aspects of data quality in which agencies had generally rated their performance highest, and other aspects of data quality in which agencies had rated their performance lowest. We believe the PIC’s efforts should help PIOs examine their agency’s data quality policies and procedures, and ultimately improve data quality and the information provided to external audiences. We also recommended that OMB, working with the PIC, identify additional changes that are needed for its guidance to agencies on ensuring the quality of performance information for APGs on Performance.gov. As of June 2016, OMB has not updated its guidance.

CAP goals. In May 2016, we reported that OMB and the PIC had incorporated lessons learned from the prior CAP goal period (2012-2014) and provided ongoing assistance to CAP goal teams. Based in part on our June 2014 recommendation, OMB and the PIC updated guidance and developed a new reporting template to help improve public reporting on the implementation of CAP goals. OMB and the PIC also implemented strategies to build agency capacity to work across agency lines, such as assigning agency goal leaders, providing ongoing guidance and assistance to CAP goal teams, and holding senior-level reviews. We also found that the selected CAP goal’s quarterly progress updates— published on Performance.gov—met a number of GPRAMA reporting requirements, including identifying contributors and reporting strategies for performance improvement and quarterly results.

Furthermore, in our May 2016 report, while we found that most of the selected CAP goal teams were consistently reporting the status of quarterly milestones to track goal progress, they had not established quarterly targets as required by GPRAMA. Also, all of the selected CAP goal teams reported that they were working to develop performance measures. While they were at various stages of the process, they were not reporting on these efforts consistently. In that report, we recommended that OMB and the PIC report on Performance.gov the actions that CAP goal teams are taking, or plan to take, to develop performance measures and quarterly targets. OMB staff generally agreed with the recommendation. With improved reporting of performance information, the CAP goal teams will be better positioned to demonstrate goal progress at the end of 4-year goal period. OMB has not yet confirmed the specific actions it plans to take in response to this recommendation.

Federal program inventory. In October 2014, we found the approach used by OMB and agencies to develop an inventory of all federal programs along with related budget and performance information had not fully met GPRAMA requirements. GPRAMA requires OMB to make this information publicly available on Performance.gov. PIC staff reported that the federal program inventory requirement was initially met in fiscal year 2013 by presenting data from agency plans and reports as PDF attachments on Performance.gov. However, we found that OMB had not published an inventory of federal programs on Performance.gov since 2013. Moreover, OMB did not expect an update of the program inventories to happen before May 2017 because the staff who would work on the program inventories were heavily involved in Digital Accountability and Transparency Act of 2014 (DATA Act) implementation. Further, in October 2014, we reported that implementation of the DATA Act could be tied to the program inventories because the act requires federal agencies to publicly report information about any funding made available to, or expended by, an agency or a component of the agency at least quarterly. We also found that agency reporting for both sets of requirements was web based. This could more easily enable linkages between the two or facilitate incorporating information from each other.

In July 2015, we recommended that OMB should accelerate efforts to determine how best to merge DATA Act purposes and requirements with the GPRAMA requirement to produce a federal program inventory. In April 2016, PIC staff told us that they will work with the DATA Act implementation team to determine how to best integrate the GPRAMA and DATA Act requirements. However, they did not provide specific details. Until OMB determines a strategy to integrate GPRAMA and DATA Act requirements, OMB will not know what resources or steps it needs to take to ensure the requirements are met and incorporated on Performance.gov.

OMB and PIC officials have told us that they are focused on ensuring Performance.gov is GPRAMA compliant and are aware that the website is not fully consistent with GPRAMA requirements. Without all the required GPRAMA information, Performance.gov will not be transparent and may fall short of meeting user needs.

OMB Does Not Have a Strategic Plan for the Future of Performance.gov

OMB does not have a strategic plan for the website that will help guide officials in the future. Specifically, we found that OMB does not have a customer outreach strategy that incorporates, as appropriate, information about how OMB intends to (1) inform users of changes on Performance.gov, (2) use social media as a method of communication, and (3) use mobile devices and applications. Finally, OMB lacks an archiving plan that details how OMB plans to manage the data and content on Performance.gov.

OMB Does Not Yet Have a Strategic Plan to Guide the Future Development of Performance.gov

OMB has not developed a strategic plan to guide the future of Performance.gov. Agency-wide strategic planning practices required under GPRA, and enhanced by GPRAMA, can serve as leading practices for planning at lower levels within federal agencies, such as individual programs or initiatives. Under this law, strategic plans are the starting point and basic underpinning for results-oriented management. Among other things, strategic plans should contain goals and objectives, approaches, and resources needed to achieve those long-term goals and objectives. When we began our review, OMB staff said that they had not developed a strategic plan for Performance.gov because a new contractor, eKuber, had just started a few months prior, and they wanted to allow time for the contractor to transition into its new role. eKuber took over the Performance.gov contract in the late summer of 2015. Since that time, OMB and the PIC have taken an important first step towards developing a strategic plan by hiring a Digital Services Director in February 2016. The Digital Services Director’s responsibilities include developing a strategic plan and managing the long-term development of Performance.gov. Hiring a new Director who has responsibility for outlining needed improvements is a move in the right direction. However, without a plan for the future, OMB will not know what resources it will need or steps it needs to take to ensure all requirements are met and incorporated on Performance.gov. Such a plan could prove especially valuable in maintaining continuity during the upcoming presidential transition.

OMB and the PIC Do Not Have a Customer Outreach Strategy

OMB and the PIC have not developed a customer outreach strategy. The Digital Government: Building a 21st Century Platform to Better Serve the American People strategy calls for federal websites to become more customer centric by responding to the needs of users and making it easier to find and share information, and accomplish important tasks “anytime, anywhere, any device.” It also calls on agencies to embrace new technologies to drive participation and to develop innovative, transparent, user-facing products and services efficiently and effectively. Without developing a user outreach plan, OMB risks being unable to provide services to its users where they need it most. Specifically, we identified three areas where Performance.gov did not have a customer outreach strategy that incorporates, as appropriate, information about how OMB intends to (1) inform users of changes on Performance.gov, (2) use social media as a method of communication, and (3) use mobile devices and applications.

Performance.gov Does Not Inform Users of Latest Updates

Performance.gov does not inform users when a new quarterly update has been published. Website usability guidelines created by GSA in conjunction with Health and Human Services (HHS) call for websites to inform users when changes are made. Most of the time, OMB publishes a blog post alerting users that new quarterly data have been updated; however, this blog post is made available through www.WhiteHouse.gov, not Performance.gov. While a blog post on WhiteHouse.gov will inform some users that the data have been updated, more Performance.gov users would be informed of the updates if they were cross posted to the Performance.gov home page.

The lack of alerts is particularly problematic because OMB does not always meet the deadlines published in OMB Circular No. A-11 (which provides guidance to agencies) for the APG and CAP goal quarterly updates. For example, in the 2015 version of OMB Circular No. A-11, OMB estimated the update for the first quarter of 2016 would be published around March 17, 2016. However, these data were not published until March 30, 2016. PIC staff told us that, in some cases, OMB held the publication date until it could be coordinated with a press release or a blog posting, which was the case for this quarterly update. The delay in publishing the quarterly updates highlights the need to inform users of when the website has been updated.

Other federal agencies have updated users about new information on their website by including a “Latest News” or “Updates” section on their home page or by implementing date and time stamps. For example, the U.S. Census Bureau home page provides users with a “Latest News” section, along with a calendar of events for the upcoming week, to allow users to easily access the most up-to-date information (see figure 5).

OMB and PIC staff said that most of the site’s users are other federal agencies and employees who already generally know when information will be updated. The public and members of Congress, however, would not know when information has been updated. In May 2016, PIC staff told us they are considering a number of options to inform users of changes on Performance.gov. This includes a banner on the home screen identifying new information or using time stamps and a timeline to highlight quarterly updates, among other options. While PIC staff are in the early stages of planning, we continue to believe communicating changes on the website will help enhance the usability for all Performance.gov users. Without a systematic method for informing users of when the website has been updated, OMB and the PIC are missing an opportunity to disseminate information to a broader base of users and meet users where they are.

On a related issue, in some instances, we were unable to determine when some information on the website was last updated. For example, we reviewed the content on seven web pages–Overview, Strategies, Progress Update, Next Steps, Indicators, Continuing Programs and Other Factors, and Related Goals–associated with 22 APGs. We found that none of them had a time or date stamp of when the information was last updated. Date or time stamps can provide users with a clear idea of when a web page was last reviewed or updated. This increases a website’s transparency. We also reviewed the Progress Update page of each of the 22 APGs. On three of those pages, we were unable to identify any text that would tell the reader when the page was last updated, such as the quarter or year of the update. According to PIC staff, the information on the APG pages always reflects the most current information. As previously noted, GPRAMA requires the APGs to be updated quarterly. Without a date or time stamp or some indication of when the data were last updated, system users, including decision makers, are unable to determine how current the website’s data are.

OMB Has Not Incorporated Social Media into Its Customer Outreach

Performance.gov provides users with social media links to share information on the website. However, OMB and the PIC are not using social media services, such as Twitter or Facebook, to interact with the site’s users. Digitalgov.gov’s social media requirement states that staff should use social outreach tools to interact with customers and improve the customer experience. Furthermore, we found that other federal websites have social media pages and updates linked on the home page. For example, the U.S. Department of Housing and Urban Development (HUD) website has a feed on its home page showing tweets from its Twitter account. These tweets provide users with the latest information from HUD, including policy updates and recent news and events related to HUD’s mission (see figure 6).

According to OMB and PIC staff, they have not used social media for customer outreach because they have limited staffing resources to manage a social media strategy. However, in May 2016, OMB and PIC staff told us they are planning to hire a Communications Advisor who will be responsible for creating a social media strategy, among other tasks. This is an important first step. Moving forward with a social media strategy should help publicize Performance.gov and improve user experience. Without such a strategy, OMB and the PIC are missing an opportunity to interact with customers and to improve the customer experience.

Performance.gov Is Accessible on Mobile Devices, but OMB Has Not Yet Determined Whether to Pursue Mobile Applications

Performance.gov is accessible on mobile devices, such as mobile phones and tablets, as Digitalgov.gov recommends. The Performance.gov home page and the eight main tabs are accessible and readable on a mobile device. We were able to access agency information and read about strategic goals and APGs on a mobile device. However, OMB does not have a mobile application (also known as an app) for Performance.gov. The U.S. Digital Services Playbook outlines key successful practices from the private sector that would help the government build effective digital resources, including mobile applications. According to the playbook, it is important for staff to understand how users access the website, whether it is through a computer or mobile device. Digital products may better cater to how the users interact with the website. OMB and PIC staff told us they have not determined whether it will be beneficial to develop a mobile application for its users. Considering the need for a mobile application when developing a customer outreach plan may help OMB determine future resource requirements.

OMB and the PIC Do Not Have a Plan to Maintain Archived Data

As of the end of February 2016, OMB and the PIC no longer maintain a full archive of Performance.gov. This means that the data that were previously published as a part of the 2012-2013 APGs are no longer available to the public. Additionally, on an ongoing basis, OMB and the PIC do not publish the progress updates from a prior quarter’s update for any APG, once the newest quarter’s data are released. However, previous iterations of CAP goal progress have been maintained for the public to access via the CAP goal update pages. PIC staff told us that the platform used to previously archive the site was no longer supported. PIC staff told us they would like to migrate the archived data back onto Performance.gov so that it can be accessed publicly again, if funding for the project becomes available.

The Digitalgov.gov requirement for records management cites National Archives and Records Administration (NARA) guidance on managing web records, which is based on statutory requirements. This guidance states federal websites are part of an agency’s approach to serving the public and agencies should conduct risk assessments to determine what parts of their websites should be documented and have records kept. Once an agency determines which content to keep or archive, staff should then develop a web records schedule to document and store that content. The PIC does not have a web records schedule to determine how to manage Performance.gov content and data. Instead, the PIC told us GSA is instituting a new method of managing and tracking the progress of its web projects. A web records schedule would provide stakeholders with important information about plans to archive the data and content published on Performance.gov. Further, without an archive of Performance.gov, users can no longer compare long-term agency priority goals and progress made toward those goals. This affects the website’s transparency.

Conclusions

Since Performance.gov was launched in 2011, OMB has worked with the PIC and GSA to develop a single website that will meet federal requirements for the public reporting of agency performance information. While OMB, GSA, and the PIC have taken several steps to improve Performance.gov, their actions do not fully meet Digitalgov.gov requirements or completely address our prior recommendations. For example, while OMB and GSA conducted a usability test of the website, they have not addressed all of the test’s findings. Further, OMB has experienced several challenges implementing APG and CAP goal reporting, and the federal program inventory requirements outlined in GPRAMA. Without improving usability and fully implementing GPRAMA requirements, Performance.gov will have difficulty serving its intended purpose as a central website where users can find government-wide performance information easily.

OMB has not developed a strategic plan for Performance.gov. OMB and the PIC took an important first step by hiring a Digital Services Director for the PIC. OMB now needs to outline the goals and objectives, approaches, and resources needed for the future development of Performance.gov. Further, OMB does not have a customer outreach strategy that explores additional ways to display updated information on the Performance.gov home page and to use social media and mobile applications for outreach. Additionally, outlining a plan to manage and archive the content and data on Performance.gov in a systematic way will increase the transparency of the website. Without a strategic plan that incorporates all of these areas, it will be difficult for decision makers to prioritize and plan for future website improvements. Such a plan could prove especially valuable in maintaining continuity during the upcoming presidential transition.

Recommendations for Executive Action

We recommend the Director of the Office of Management and Budget, in consultation with the Performance Improvement Council and General Services Administration, take the following three actions: 1. Ensure the information presented on Performance.gov consistently complies with GPRAMA public reporting requirements for the website’s content. 2. Analyze and, where appropriate, implement usability test results to improve Performance.gov. 3. Develop a strategic plan for the future of Performance.gov. Among other things, this plan should include: the goals, objectives, and resources needed to consistently meet Digitalgov.gov and GPRAMA requirements; a customer outreach plan that considers how (1) OMB informs users of changes in Performance.gov, (2) OMB uses social media as a method of communication, and (3) users access Performance.gov so that OMB could, as appropriate, deploy mobile applications to communicate effectively; and a strategy to manage and archive the content and data on Performance.gov in accordance with NARA guidance.

Agency Comments and Our Evaluation

We provided a draft of this report to the Director of OMB and the Administrator of GSA for review and comment. On August 5, 2016, we met with OMB, PIC and GSA staff. OMB and PIC staff provided us with oral comments on the report and we made technical changes as appropriate. OMB staff agreed with the recommendations in the report. GSA did not have comments on the report.

We are sending copies of this report to the Director of OMB and the Administrator of GSA as well as appropriate congressional committees and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Selected Open GAO Recommendations That May Affect Performance.gov

Status update 1. The agency has not yet provided reporting on Cross Agency Priority (CAP) goal progress, we recommend that the Director of the Office of Management and Budget (OMB), working with the Performance Improvement Council (PIC), take the following action: report on Performance.gov the actions that CAP goal teams are taking, or plan to take, to develop performance measure and quarterly targets. information on what actions it has taken in response to this recommendation.

1. To help participating agencies improve 1. We reviewed updates OMB published to their public reporting, the Director of OMB, working with the PIC Executive Director, should identify additional changes that need to be made in OMB’s guidance to agencies related to ensuring the quality of performance information for agency performance goals (APG) on Performance.gov.

Circular No. A-11 in July 2016. Circular No. A-11 continues to direct agencies to provide data quality information for publication on Performance.gov or to provide a hyperlink from Performance.gov to relevant explanation in agencies' performance reports, which was a requirement OMB added in June 2015 in response to our preliminary findings. However, our review found that OMB also needed to update the template agencies complete for Performance.gov updates to make it easier for agencies to publish this information on the website. The July 2016 update of Circular No. A-11 does not indicate whether this template has been updated or whether additional changes to A-11 are needed. We will continue to monitor OMB and the PIC’s efforts to address our recommendation.

1. To ensure that federal program spending 1. data are provided to the public in a transparent, useful, and timely manner, the Director of OMB should accelerate efforts to determine how best to merge Digital Accountability and Transparency Act (DATA Act) purposes and requirements with the GPRA Modernization Act of 2010 (GPRAMA) requirement to produce a federal program inventory.

In April 2016, OMB staff told us that identifying programs for the purposes of DATA Act reporting would not be completed until after May 2017. However, they said they have convened a working group to develop and vet a set of options to establish a government-wide definition for program that is meaningful across multiple communities and contexts (such as budget, contracting, and grants).

Status update 1. federal program inventory requirements and to make the inventories more useful, the Director of OMB should better present a more coherent picture of all federal programs: revise relevant guidance to direct agencies to collaborate with each other in defining and identifying programs that contribute to common outcomes; revise relevant guidance to provide a time frame for what constitutes “persistent over time” that agencies can use as a decision rule for whether to include short-term efforts as programs; and define plans for when additional agencies will be required to develop program inventories.

In June 2016, OMB staff stated that they have not taken any actions in response to our recommendations related to the federal program inventory, as they continue to determine how best to implement inventory requirements in coordination with those of the DATA Act. In our July 2015 testimony on DATA Act implementation, we recommended that OMB accelerate efforts to determine how to best merge DATA Act purposes and requirements with the GPRAMA requirement to produce a federal program inventory. However, at the same hearing, the Acting Deputy Director for Management and Controller at OMB stated that he did not expect an update of the program inventories to happen before May 2017.

2. To ensure the effective implementation of 2. As of June 2016, OMB had not taken federal program inventory requirements and to make the inventories more useful, the Director of OMB should, to better present a more coherent picture of all federal programs, include tax expenditures in the federal program inventory effort by designating tax expenditure as a program type in relevant guidance; and developing, in coordination with the Secretary of the Treasury, a tax expenditure inventory that identifies each tax expenditure and provides a description of how the tax expenditure is defined, its purpose, and related performance and budget information. action to include tax expenditures in the federal program inventory. GPRAMA requires OMB to publish a list of all federal programs on a central, government-wide website. The federal program inventory is the primary tool for agencies to identify programs that contribute to their goals, according to OMB’s guidance. By including tax expenditures in the inventory, OMB could help ensure that agencies are properly identifying the contributions of tax expenditures to the achievement of their goals. Although OMB published an initial inventory covering the programs of 24 federal agencies in May 2013, OMB decided to postpone further development of the inventory in order to coordinate with the implementation of the DATA Act. In our July 2015 testimony, we recommended that OMB accelerate efforts to merge DATA Act purposes with the production of a federal program inventory.

Related report

Status update 3.

In June 2016, OMB staff stated that they have not taken any actions in response to our recommendations related to the federal program inventory, as they continue to determine how best to implement inventory requirements in coordination with those of the DATA Act. In our July 2015 testimony on DATA Act implementation, we recommended that OMB accelerate efforts to determine how best to merge DATA Act purposes and requirements with the GPRAMA requirement to produce a federal program inventory. However, at the same hearing, the Acting Deputy Director for Management and Controller at OMB stated that he did not expect an update of the program inventories to happen before May 2017 because the staff that would work on the program inventories were heavily involved in DATA Act implementation. revise relevant guidance to direct agencies to consult with relevant congressional committees and stakeholders on their program definition approach and identified programs when developing or updating their inventories; revise relevant guidance to direct agencies to identify in their inventories the performance goal(s) to which each program contributes; and ensure, during OMB reviews of inventories, that agencies consistently identify, as applicable, the strategic goals, strategic objectives, APGs, and CAP goals each program supports.

Status update 1. OMB has taken some steps to address could use the information on the Performance.gov website to accomplish specific tasks and specify the design changes that would be required to facilitate that use. this recommendation, but additional actions are needed. The General Services Administration, on behalf of OMB, issued a usability test report on Performance.gov in September 2013. The test found that (1) sections of the website were not accessible; (2) users were unclear about the purpose of Performance.gov, its intended audiences, and what users can do on the website; (3) users had difficultly locating graphics and understanding if agencies had met their goals; and (4) the search functionality produced poor results and the search terms were not highlighted. This usability test produced several recommendations based on these findings. According to OMB and PIC staff in August 2015 and May 2016, they have taken some actions to address the recommendations. For example, staff addressed the accessibility issue and partly addressed the search issue. However, OMB has not yet addressed the other two recommendations. Further, OMB and the PIC have not clarified the ways that intended audiences can use the information on the website to accomplish specific tasks, or specified design changes that would be required to facilitate that use, as we described in our report. We will continue to monitor progress.

Related report

Status update 2. OMB, GSA and the PIC systematically information on the needs of a broader audience, including through the use of customer satisfaction surveys and other approaches recommended by federal guidance. collect information on the needs of its users by implementing a website survey, a feedback form on Performance.gov, and a working group focused on improving the PREP system. Staff have set up a backlog system to prioritize, store, and address user feedback. According to PIC staff, feedback is prioritized based on several factors. These factors include the value it would bring to a larger audience, the cost and estimated time to implement, and the risk and opportunity cost of addressing the feedback. The highest priority items on the product backlog system are placed on the monthly prioritized list for the contractor to begin work on. However, OMB and the PIC have not identified, engaged, or collected information on the needs of a broader audience, such as interested members of the public, and how those needs might be addressed through the website.

3. seek to ensure that all performance, 3. As of May 2016, OMB and the PIC were search, and customer satisfaction metrics, consistent with leading practices outlined in federal guidance, are tracked for the website, and, where appropriate, create goals for those metrics to help identify and prioritize potential improvements to Performance.gov. monitoring 18 of the 24 recommended performance measures. PIC staff said that they now track the performance measures through the Digital Analytics Program, which does not track the remaining six measures. OMB and PIC staff have not created goals for the measures they track to help identify and prioritize potential improvements to Performance.gov. We will continue to monitor progress.

1. To ensure that the PIC has a clear plan 1. The PIC developed a strategic plan for for accomplishing its goals and evaluating its progress, the Director of OMB should work with the PIC to update its strategic plan and review the PIC’s goals, measures, and strategies for achieving performance, and revise them if appropriate.

2015, which identified its mission, goals and strategies, and core responsibilities for achieving them. PIC staff reported that they plan to update the document for 2016 with a more robust plan.

Status update 1. According to information provided by guidance are made, the Director of OMB should work with the PIC to test and implement these provisions.

OMB and PIC staff in June 2015, although OMB revised its guidance as we recommended, it did not work with the PIC to test implementation of these provisions. Instead, they told us that both PIC and OMB staff ensure agencies are implementing these provisions of their guidance when reviewing agencies’ quarterly update submissions for APGs. However, our analysis of agencies’ updates in July 2014 found implementation of these provisions continues to be mixed. We will continue to monitor progress.

Performance.gov to include additional information about APGs, the Director of OMB should ensure that agencies adhere to OMB’s guidance for website updates by providing complete information about the organizations, program activities, regulations, tax expenditures, policies, and other activities—both within and external to the agency—that contribute to each APG. ensure that agencies provide complete information about tax expenditures contributing to their APGs. According to information provided by OMB staff in April 2015, agencies were asked to identify organizations, program activities, regulations, policies, tax expenditures, and other activities contributing to their 2014-2015 APGs. This process began as part of the September 2014 update to Performance.gov, with opportunities for revisions in subsequent quarterly updates. We found that while agencies had made progress in identifying external organizations and programs for their APGs, they did not present this information consistently on Performance.gov. Although each APG web page has a location where agencies are to identify contributing programs, agencies did not always identify external organizations and programs there. Instead, they identified these external contributors elsewhere, such as APG overview or strategy sections. In June 2015, OMB staff said they would work with agency officials to help ensure information is presented in the appropriate area of Performance.gov in future updates.

Related report

Performance.gov to include additional information about APGs, the Director of OMB should ensure that agencies adhere to OMB’s guidance for website updates by providing a description of how input from congressional consultations was incorporated into each APG.

Performance.gov for the 2014-2015 APGs generally found that either agencies did not include this information or they had not updated it to reflect the most recent round of stakeholder engagement. In June 2015, OMB staff reported that they will focus agency attention on this issue during the development of the 2016-2017 APGs, to be published in October 2015. We will continue to monitor progress.

Appendix II: Objectives, Scope, and Methodology

This report is part of our response to a mandate to assess the implementation of the GPRA Modernization Act of 2010 (GPRAMA). This report assesses the Office of Management and Budget’s (OMB) (1) efforts to ensure the usefulness of Performance.gov, and (2) strategic plan for Performance.gov.

To address our objectives, we reviewed the 22 Digitalgov.gov requirements for federal websites and digital services, and selected 9 for our assessment of Performance.gov, as shown in table 4. The selected requirements are those most associated with customer feedback and outreach, usability, performance measures, and records management.

We assessed OMB’s efforts (in collaboration with the Performance Improvement Council (PIC) and the General Services Administration (GSA)) to ensure the usefulness of Performance.gov and OMB’s strategic plan by comparing steps taken and documentation for each to the selected requirements.

To further address our first objective, we examined documentation on how OMB was (1) seeking information from various audiences about their needs concerning Performance.gov, (2) ensuring the website was clarifying ways audiences can use Performance.gov, and (3) tracking a broader range of performance and customer satisfaction measures and setting goals for those measures. We used the information to follow up on the status of the recommendations in our 2013 report. Further, we collected documentation on customer service feedback, including survey data collected from the website survey and customer service feedback logs, and analyzed the results. We reviewed the findings of the September 2013 usability test GSA conducted. We collected documentation on the performance measures tracked for Performance.gov and compared it with the recommended measures on Digitialgov.gov. We reviewed requirements for agency performance plans established under the Government Performance and Results Act of 1993 and enhanced by GPRAMA, and used them as a source for leading practices on setting goals for collected performance measures and customer service feedback.

We reviewed requirements outlined in GPRAMA for Performance.gov, including public reporting requirements for agency priority goals (APGs), cross-agency priority (CAP) goals, and the federal program inventory. We reviewed other related guidance, such as OMB Circular No. A-11, Preparation, Submission, and Execution of the Budget. We reviewed several of our prior related reports and summarized the findings and recommendations related to OMB’s implementation of the selected GPRAMA requirements. We selected these reports because they represent our most current reports on the implementation of these selected GPRAMA requirements.

We also interviewed staff from OMB’s Office of Performance and Personnel Management, the PIC, and GSA to determine the actions taken in response to our recommendations about clarifying ways intended audiences can use Performance.gov, systematically collecting feedback from a broader audience, and tracking recommended performance measures. Further, we communicated with OMB and PIC staff to determine the actions taken thus far to address our prior recommendations in relation to APGs, CAP goals, and the federal program inventory.

To further address our second objective, we requested documentation on OMB’s strategic plan for Performance.gov, social media and customer outreach strategy, and web records plan. We found the agency has not developed these documents. We also interviewed staff from OMB, the PIC, and GSA on the Performance.gov website’s strategic plan, social media and customer outreach strategy, web records plan, and accessibility on mobile devices. We compared any steps taken on those actions to Digitalgov.gov requirements.

We reviewed the Digital Services Playbook, which outlines key successful practices from the private sector that would help the government build effective digital resources. The playbook provides guidance on interacting with users through different channels. It was used to analyze the extent to which Performance.gov is accessible on mobile devices. We reviewed Performance.gov to determine if it was accessible on mobile devices by visiting each main tab of the website on multiple devices. We also identified NARA guidance on web records management. This guidance assists agencies in how to properly manage and schedule web records. We reviewed the Research-Based Web Design and Usability Guidelines for guidelines about informing website users of changes to the website. The Research-Based Web Design and Usability Guidelines provide guidance on a broad range of web design and communication issues.

We also reviewed a selection of APG pages on Performance.gov to document whether the page had an indication of the last time it was updated. We concentrated on evaluating the 22 agencies listed on Performance.gov, which are mostly department level agencies. The additional 31 agencies listed on Performance.gov do not have dedicated pages on the website, and therefore were not evaluated as part of this analysis. For the 22 selected agencies, we focused on the APG’s for fiscal years 2014 to 2015 because the number of APGs for fiscal years 2016 to 2017 was not yet finalized at the time of the analysis. We randomly selected one APG from each of the 22 agency pages for review. For each selected APG, we reviewed all of the tabs to determine if there were any date or time stamps to indicate the last time the page was updated. We also reviewed in more detail the Progress Update and Indicators pages to evaluate whether users could determine through page context when the information or data on a given page was last updated.

We conducted this performance audit from July 2015 to August 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Lisa Pearson, Assistant Director; Sonya Phillips, Analyst-in-Charge, supervised the development of this report. Caroline Prado, Robyn Trotter, and Edith Yuh made significant contributions to all aspects of this report. Karin Fangman, Robert Gebhart, Kirsten Lauber, and Donna Miller provided additional assistance.