Managing for Results: Greater Transparency Needed in Public Reporting on the Quality of Performance Information for Selected Agencies' Priority Goals
Published: 2015-09-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Federal agencies have not always clearly and transparently explained to Congress and the public how they ensure the quality of their performance information. GPRAMA requires agencies to publicly explain how they ensure the accuracy and reliability of their performance information used to assess progress for their APGs.
This is one of a series of GAO reports examining the implementation of GPRAMA, as required by the act. This report assesses how well selected agencies publicly reported on the quality of performance information used to measure progress on APGs. GAO selected six agencies–the Departments of Agriculture, Defense, Interior, and Labor, and NASA and DHS– based on GAO's 2013 federal managers survey on their agency's use of performance information. GAO reviewed information concerning these agencies' APGs published on Performance.gov and in their annual performance plans and reports.
section_title: What GAO Found
paragraphs: The six agencies GAO reviewed generally did not publicly report on how they ensured the accuracy and reliability of performance information used to measure progress on their highest priority performance goals, referred to as agency priority goals (APGs). The GPRA Modernization Act of 2010 (GPRAMA) requires agencies to identify the following when publicly reporting on their APGs: 1) how performance information was verified and validated; 2) data sources; 3) level of accuracy required for intended use; 4) any limitations at the required level of accuracy; and 5) how the agency will compensate for such limitations (if needed) to reach the required level of accuracy. GPRAMA requires agencies to provide this information to the Office of Management and Budget (OMB) for publication on Performance.gov. GPRAMA also directs agencies to provide this information for performance goals, which include APGs, in their annual performance plans and reports. While all six agencies described how they ensured the quality of their performance information overall, GAO found discussions about performance information quality addressing all five GPRAMA requirements in only the Department of Homeland Security's (DHS) performance plans and reports.
Source: GAO analysis of selected agencies' performance plans and reports. | GAO-15-788
OMB and the Performance Improvement Council (PIC)–a cross-agency council of agency performance improvement officers–established the Data Quality Cross-Agency Working Group in February 2015. The group has identified several goals, such as improving the reliability and quality of performance information, and could serve as a vehicle for disseminating good practices in public reporting on data quality.
section_title: What GAO Recommends
paragraphs: GAO recommends that all six of the agencies work with OMB to describe on Performance.gov how they are ensuring the quality of their APGs's performance information, and that all agencies, except for DHS, also describe this information in their annual performance plans and reports. GAO also recommends that OMB, working with the PIC, focus on ways the PIC's data quality working group can improve public reporting for APGs. OMB did not comment on the recommendations, but the six agencies generally concurred or identified actions they planned to take to implement them.

section_title: Letter
paragraphs: The federal government is one of the world’s largest and most complex  entities. To ensure that federal agencies are effectively meeting their  missions, both top leaders and agency managers must have accurate  and reliable performance information to monitor and track the progress  they are making on achieving their goals. For Congress and the public to  have confidence in the quality of the performance information that federal  agencies are using to assess and achieve results, that information must  be publicly reported in a clear and readily accessible way. Further, our  work examining fragmentation, overlap, and duplication in federal  government programs has demonstrated the need for more reliable and  consistent federal data.
Our previous work has shown that federal agencies do not always explain  how they ensure the quality of their publicly-reported performance  information. credibility of performance information–such as agencies discussing data  limitations and the actions they are taking to address these limitations.   To improve federal transparency and Congress’s and the public’s ability  to understand how federal agencies ensure the quality of performance  information, the GPRA Modernization Act of 2010 (GPRAMA)  requires  agencies to provide this information on a public website– Performance.gov–and in their annual performance plans and reports.
GAO, Performance Reporting: Few Agencies Reported on the Completeness and  Reliability of Performance Data, GAO-02-372 (Washington, D.C.: Apr. 26, 2002).
These GPRAMA requirements expanded upon earlier requirements in the  Government Performance and Results Act of 1993 (GPRA).
We are required by GPRAMA to periodically report on the implementation  of the act, and this report is part of that series. report is to assess how well selected agencies publicly reported on the  quality of performance information being used to measure progress for  their highest priority performance goals–which GPRAMA refers to as  agency priority goals (APGs). To conduct our assessment, we selected  six agencies’ APGs for review based on responses to the following  question from our 2013 federal managers survey: “I have sufficient  information on the validity of the performance data I use to make  decisions.” question was the strongest predictor of how a manager ranked his or her  agency on its use of performance information. We used responses to this  question to select six agencies in the following manner:   Our analysis indicated that a manager’s response on this
Ranked on their responses to this question, we selected two agencies  from the top third of agencies: The National Aeronautics and Space  Administration and the Department of Labor.
GPRAMA, § 15(b). See, for example, GAO, Managing for Results: Executive Branch  Should More Fully Implement the GPRA Modernization Act to Address Pressing  Governance Challenges, GAO-13-518 (Washington, D.C.: June 26, 2013), which  summarizes our work on the initial implementation of GPRAMA. We will issue an updated  assessment of GPRAMA’s implementation in September 2015.
GAO, Managing for Results: Agencies’ Trends in the Use of Performance Information to  Make Decisions, GAO-14-747 (Washington, D.C.: Sept. 26, 2014).
Ranked on their responses to this question, we selected two agencies  from the middle third of agencies: The Departments of the Interior and  Defense.
Ranked on their responses to this question, we selected two agencies  from the bottom third of agencies: The Departments of Agriculture and  Homeland Security.
We reviewed all the APGs these six agencies had identified for fiscal  years 2014 and 2015. This made for a sample of 23 APGs, which is  approximately one-quarter of all federal APGs identified for fiscal years  2014 and 2015.  To assess how well the selected agencies publicly  reported on the quality of performance information for each of their APGs,  we reviewed the information published on Performance.gov concerning  the APGs in our sample. We also reviewed the information the selected  agencies published in their annual performance plans and reports  concerning these APGs. We reviewed, if available, the agencies’ most  recent performance plans for fiscal year 2015 and agencies’ updated  performance plans for fiscal year 2014. We also reviewed, if available,  agencies’ performance plans for fiscal year 2016 and updated  performance plans for fiscal year 2015. We also reviewed the agencies’  most recent performance reports which covered fiscal years 2013 and  2014.
We reviewed this information to see if it addressed the relevant  requirements of GPRAMA and Office of Management and Budget (OMB)  guidance to agencies. Because both the act and guidance state that  agencies should report on performance information quality for  performance goals, including for each APG, we first looked for an  explanation of how the agency ensured the quality of performance  information for each of its APGs on Performance.gov and in agency  performance plans and reports. For each APG, we determined whether  the agency addressed all GPRAMA requirements for its performance  information quality discussion. If we did not find the required performance  information quality discussions for an agency’s APG, we next looked for  statements in performance plans and reports of how agencies said they  were ensuring the quality of their agencies’ performance information  overall. We focused on assessing agencies’ public reporting. Therefore,  our report makes no assessment of internal processes the selected  agencies may have for ensuring the quality of their performance  information.
We interviewed officials at each of the selected agencies and staff from  the Office of Management and Budget (OMB), who are responsible for  working with agencies to implement GPRAMA requirements, and staff  from the Performance Improvement Council (PIC), an interagency council  comprised of agency performance improvement officers (PIO). We  shared the preliminary results of our review with agency officials and  asked them to direct us to additional information, if they believed we had  missed anything in our review.
We conducted this performance audit from August 2014 to September  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: GPRAMA requires agencies to publicly report on how they are ensuring  the accuracy and reliability of the performance information they use to  measure progress towards APGs and performance goals.
Priority Goals and Performance Measures: GPRAMA requires  agencies to identify their highest priority performance goals as APGs and  have ambitious targets for these APGs that can be achieved within 2  years. measures to track the progress they are making on achieving their APGs  or identify alternative ways of measuring progress, such as milestones for  completing major deliverables for the APG (for more information on the  measures and milestones the selected agencies identified, see appendix  I).
31 U.S.C. § 1120(b).
GPRAMA uses the term “measured values” instead of performance data. We define  verification as the assessment of completeness, accuracy, consistency, timeliness, and  related quality control practices for performance information. We define validation as the  assessment of whether performance information is appropriate for the performance  measure. For more information, see, GAO, Performance Plans: Selected Approaches for  Verification and Validation of Agency Performance Information, GAO/GGD-99-139  (Washington, D.C.: July 30, 1999). the level of accuracy required for the intended use of the data; any limitations to the data at the required level of accuracy; and how the agency will compensate for such limitations (if needed) to  reach the required level of accuracy.
GPRAMA requires agencies to provide information to OMB that  addresses all five requirements for each of their APGs for publication on a  website (Performance.gov).  Agencies also must address all five  requirements for performance goals in their performance plans and  reports.
GPRAMA states that Performance.gov shall consolidate information  about each APG, thereby making this information readily accessible to  the public, members of Congress, and congressional committees.  GPRAMA makes OMB responsible for Performance.gov and requires  agencies to provide OMB with quarterly updates on their APGs, including  how they are ensuring the quality of performance information, for  publication on Performance.gov.
Further, GPRAMA continues transparency requirements set in GPRA that  require agencies to publish annual performance plans and reports (see  text box). While GPRAMA requires certain information to be reported in  performance plans and certain other information to be reported in  performance reports, the Reports Consolidation Act of 2000 authorizes  agencies–with the concurrence of OMB–to consolidate performance  plans and reports into a single publication that covers past actual and  future planned performance.
GPRAMA’s Requirements for Agencies’ Annual Performance Plans and Reports  Performance plans should identify the planned level of performance for the current fiscal year and the next fiscal year, explain how the agency will ensure the accuracy and reliability of its performance information, identify the agency’s priority goals (APGs), and be published every February, concurrent with the President’s Budget.  summarize the actual level of performance agencies achieved during the previous five fiscal  years, explain how the agency ensures the accuracy and reliability of its performance information, and be published every February.
Guidance and Information Sharing on Implementing GPRAMA: OMB  provides guidance to agencies in Circular A-11 on how to implement  GPRAMA. OMB updates A-11 annually, and the most recent update was  published in June 2015.  GPRAMA also established in law an  interagency council–the PIC–chaired by OMB and composed of agency  PIOs to facilitate the exchange of useful practices to strengthen agency  performance management, such as through cross-agency working  groups.
section_title: Public Reporting of Performance Information Quality for Selected Agencies’ Priority Goals Was Not Transparent
paragraphs: The selected 23 APGs we reviewed are intended to drive progress in  important and complex areas, such as assisting veterans, addressing  climate change, and protecting workers. Given the significance and  complexity of many APGs, congressional and public understanding  regarding how federal agencies are measuring and assessing progress  toward these goals is important. GPRAMA requires agencies to publicly  report on how they are ensuring the accuracy and reliability of the  performance information they use to measure progress towards these  APGs. However, our review found that overall, it would be challenging  for Congress and the public to understand how the selected agencies are  ensuring that the performance information they report for their 23 APGs is  accurate and reliable–that is, suitable for making judgments about agency  progress or decisions for different courses of action.
subsections: section_title: Performance.gov Provided Limited Information on the Quality of Performance Information Used to Measure Progress on Selected Agency Priority Goals
paragraphs: We found limited information on Performance.gov on the quality of  performance information used to assess progress on the six selected  agencies’ 23 APGs. While each agency has a section dedicated to its  priority goals on Performance.gov, there is no place on the website that is  set aside to discuss the quality of performance information for each APG.  The six agencies we reviewed used various sections of Performance.gov  to discuss some of the performance information quality requirements for  APGs. But, none of the agencies addressed all five GPRAMA  requirements for their individual APGs. Moreover, while we found  hyperlinks from Performance.gov to the selected agencies’ performance  plans and reports, there was no explanation on Performance.gov of  where to find performance information quality discussions in these plans  and reports.
We discussed our preliminary findings with OMB staff in January 2015. In  response, OMB updated its A-11 guidance in June 2015 to direct  agencies to either provide information for publication on Performance.gov  of how they are ensuring the quality of performance information for their  APGs, or provide a hyperlink from Performance.gov to an appendix in  their performance report that discusses the quality of their performance  information. OMB staff stated that this information will likely not be  available until agencies start reporting on the next set of APGs (for fiscal  years 2016 and 2017).  This is because OMB will need to update a  template that agencies complete for their Performance.gov updates. OMB  staff confirmed in July 2015 that they are still using a version of this  template that they provided to us in January 2015 that has not yet been  updated to reflect this change.
section_title: Selected Agencies’ Annual Performance Plans and Reports Provided Incomplete Information on the Quality of Performance Information Used to Measure Progress on Priority Goals
paragraphs: Five of the six agencies’ performance plans and reports we reviewed did  not describe how they ensured the quality of performance information for  their individual APGs. On the other hand, all six agencies did describe  how they ensured the quality of their performance information overall. Of  the 23 APGs in our sample, we could only find performance information  quality discussions that addressed all five of the GPRAMA requirements  for 3 APGs, which belonged to the Department of Homeland Security.
The U.S. Department of Agriculture (USDA) Provided Some Information  for How Performance Information Quality Is Ensured For One of Three  APGs, but Did Not Address All Requirements  USDA identified its APGs and briefly summarized the results achieved for  each APG in its performance report for fiscal year 2014. Further, USDA’s  performance report provided data quality discussion for each of the  performance measures presented in the 2014 report, which included the  two performance measures USDA used to measure progress on its  Reduce the Number of Foodborne Salmonella Illnesses APG.  However,  USDA did not address all of the GPRAMA requirements for this APG or  its two other APGs, as shown by table 1. For the Reduce the Number of  Foodborne Salmonella Illnesses APG, USDA addressed two of the five  GPRAMA requirements. For example, USDA identified the Centers for  Disease Control and Prevention (CDC) as the source of the data  measuring the number of illnesses from products USDA’s Food Safety  and Inspection Service regulates. USDA also noted that CDC receives  information from state and local health agencies concerning outbreaks of  illnesses.  USDA acknowledges that the quality of the data can vary by  reporting agency, which is an example of identifying a potential limitation.  While the fiscal year 2014 performance report did not explain how USDA  and CDC are compensating for this limitation, USDA did provide a  hyperlink in its performance report to a CDC web page that provided  more detailed information about tracking and reporting of foodborne  illnesses.  Further, while USDA described a number of steps it is taking  to reduce illnesses and detect contamination in food products, USDA did  not explain to external audiences what level of accuracy it requires to  make decisions related to this APG. Our prior work has identified  improving oversight of food safety as a high-risk area, emphasizing the  need to improve planning and collaboration among USDA and other  federal food safety agencies. This makes it important for USDA to expand  its performance information quality discussion and address all  requirements for this APG.
USDA’s performance plans and report did not address any of the five  GPRAMA requirements regarding the quality of the performance  information for its two other APGs: Create New Economic Opportunities  and Improve the Health of Our Nation’s Soils. The lack of information  related to the two other APGs makes it more challenging for Congress  and the public to understand how USDA is ensuring the quality of  performance information, including potential limitations. For example,  USDA’s Inspector General identified the need to develop effective  performance measures as a management challenge facing the agency  and raised concerns about the accuracy of some performance  information. But, USDA’s performance plans and report do not address  this issue with regard to two of its APGs.officials in USDA’s Office of Budget and Program Analysis, and they  acknowledged their agency could improve its public reporting on the  quality of its performance information for APGs.
We shared our analysis with  USDA’s performance report for fiscal year 2014 contained an explanation  on how the agency ensures the quality of its performance information  overall, which is reproduced below (see text box). This helps external  audiences understand that USDA’s methodology for collecting  performance information has been vetted by scientists and  policymakers.
U.S. Department of Agriculture’s Statement on Performance Information Quality–2014 Annual  Performance Report  The data used by the Department to measure performance are collected using a standardized  methodology. This methodology has been vetted by federally employed scientists and policymakers,  and, ultimately, the Under Secretaries of the respective mission areas. All attest to the completeness,  reliability, and quality of the data.
The Department of Defense (DOD) Highlighted Progress for All APGs,  but Did Not Explain How Performance Information Quality Is Ensured  DOD included performance discussions for all of its APGs and stated  whether the agency had met its interim or final APG targets in its  performance reports for fiscal years 2013 and 2014; however, it did not  address the performance information quality requirements for each APG,  as shown in table 2.  While DOD’s fiscal year 2013 and 2014  performance reports do not describe agency-wide guidance for ensuring  performance information quality, we found an explanation of DOD’s  guidance regarding performance information quality in DOD’s “2014  Performance Plan Update.” DOD officials said this document was  intended to serve as the agency’s performance plan for fiscal year 2015.  As reflected in table 2, the document provided a description of DOD’s  APGs for fiscal years 2014 and 2015 and contained a brief statement  addressing agency-wide data verification and validation practices. It  states that, “at the beginning of the fiscal year, goal leaders provide action  plans and verification and validation forms on each performance goal  listed in the .”
Officials in the Office of the Deputy Chief Management Officer said that  DOD’s fiscal year 2014 performance plan update constituted their  agency’s fiscal year 2015 performance plan, although this purpose is not  clearly explained in the document.  DOD officials told us their agency  would publish its 2016 performance plan as part of its agency strategic  plan, which was intended for publication by the end of summer 2015. The  Secretary of Defense signed this plan covering fiscal years 2015-2018,  which is dated July 31, 2015.  Our review of the plan indicated that it  described DOD’s performance management process. However, it did not  explain how DOD will address all of the performance information quality  requirements for each of the agency’s APGs. In addition, the plan did not  make clear which sections of the plan were intended to address the  requirement to publish a performance plan establishing performance  goals for 2016.
Department of Defense, Annual Energy Management Report: Fiscal Year 2014 (May  2015). related to the GPRAMA requirements, such as on their verification and  validation processes.
Improving the public reporting of performance information quality is  important because DOD’s Reform the DOD Acquisition Process and DOD  Financial Statement Audit Readiness APGs address areas that we have  identified as high risk. For the acquisition APG, we reported in 2014 that  DOD expects to invest $1.5 trillion in its portfolio of major defense  acquisition programs, making it particularly important that DOD explains  how it is ensuring the quality of performance information for this APG. Similarly, given that DOD is responsible for more than half of the federal  government’s discretionary spending, we reported that it is particularly  important DOD has accurate, timely, and useful financial information. The  reliability of DOD’s financial information and ability to maintain effective  accountability for its resources will be increasingly important to the federal  government’s ability to make sound resource-allocation decisions.
The Department of Homeland Security (DHS) Addressed GPRAMA  Requirements in Explaining How Performance Information Quality Is  Ensured for All APGs  DHS presented information about performance information quality for all  three of its APGs in its performance plans and reports, which included  detailed discussion for 10 of the 14 performance measures used to  measure progress on these APGs.  Specifically, DHS published an  appendix to its performance plans and reports with detailed performance  information quality discussion for these measures.
For each measure,  DHS’s appendix describes the related program, the scope of the data, the  source and collection methodology for the data, and an assessment of  data reliability. For example, for the number of convicted criminal aliens  that Immigration and Customs Enforcement removes from the country for  the Enforce and Administer Our Immigration Laws APG, DHS addresses  the requirements for explaining verification and validation. It explains how  headquarters staff looks for unusual patterns in data field offices have  entered into a database tracking removals. DHS also states it conducts  additional checks by cross-referencing data on removals reported by  detention facilities and field offices, and says a statistical tracking unit  does further checks. Related to the intended use of performance  information, DHS explains for each performance measure discussed in  this appendix how it uses the measure for decision making. For example,  DHS explains that its measure of the number of convicted criminal aliens  removed from the country reflects the “full impact” of its program activities  in this area. This helps a reader understand that DHS will need a high  level of accuracy to ensure that its programs are achieving its goals for  this area.
We found examples of DHS acknowledging potential limitations for some  APG performance measures. For example, DHS acknowledged that the  average number of days to process inquiries from individuals  experiencing difficulties with travel screening for its Aviation Security APG  does not include the time DHS is waiting for the traveler to submit all  required documents. This helps a reader understand that the  performance measure may not reflect the total number of days it takes to  resolve issues impeding an individual from traveling. DHS addresses the  final requirement–how the agency will compensate for limitations to reach  the required level of accuracy, if needed–by stating that each APG  performance measure discussed in the appendix is reliable. Further, DHS  also provided more specific explanation of corrective actions for some  performance measures. For the Ensure Resilience to Disasters APG,  DHS acknowledges that there is some variation in how states and  territories assess their capabilities for dealing with disasters. But, it also  explains that federal officials provide technical assistance and review the  submissions from state and territorial officials to ensure that they align  with guidance the Federal Emergency Management Administration  provides to states and territories for how to assess their capabilities.
In addition, DHS’s performance plans and reports explained how the  agency ensures the quality of its performance information overall. DHS  states it has an agency-wide performance management framework, which  includes a process for verifying and validating its performance  information. For example, DHS explains that one of the steps it takes is to  have an independent review team assess the completeness and reliability  of its performance measurement data. By addressing the five GPRAMA  requirements for its APGs, DHS’s performance plans and reports helped  external audiences better understand how it ensures the accuracy and  reliability of performance information for the agency’s highest priority  performance goals.
The Department of the Interior (Interior) Described How Performance  Information Quality is Ensured Overall, but Did Not Address GPRAMA  Requirements for its APGs  Interior’s performance plan and report covering fiscal years 2014 through  2016 provided an overall explanation of how it verified and validated its  performance information, which is reproduced in figure 1.  Interior’s plan  and report states that it requires component agencies to have verification  and validation processes, and referred to a more detailed document on  the website of Interior’s Office of Policy, Management, and Budget “Data  Verification and Validation Standards.” These standards provide direction  to component agencies on how to verify and validate performance  information and other aspects of performance information quality.  For  example, the standards explain that component agencies should  document data sources, describe the accuracy limits of data, and identify  data limitations. Interior shared these standards in June 2015 with other  agencies participating in a Performance Improvement Council cross- agency working group on data quality.
Interior’s performance plans and reports did not explain how performance  information quality was ensured for its individual APGs. As shown above,  Interior’s statement on verification and validation is written at a high level  and does not explain the specific steps component agencies took to  ensure that performance information for each APG was accurate and  reliable. Interior’s Deputy Performance Improvement Officer noted that  the performance plans and reports discussed the agency’s performance  in mission areas that relate to the APGs, and thereby provided contextual  information that would allow the public to understand the quality of  performance information for these APGs. Thus, they provided information  on performance targets and past performance for some of the  performance measures related to APGs. The available contextual  information in the performance plans and reports did not address all of  the GPRAMA performance information quality requirements for each  APG. For example, related to its Water Conservation APG, which aims to  increase the available water supply in the western states, Interior  identifies the number of people and farmers the Bureau of Reclamation  delivers water to, and reports on the acre feet of water conservation  capability enabled through Reclamation’s programs. While this helps  external audiences understand the importance of this APG and the  related mission area, Interior does not explain how it is ensuring that it is  accurately measuring the water conservation capability enabled through  Reclamation’s programs.
Interior explains that operating regulations require inspection of leases which produce  high volumes of oil or natural gas and those leases that have a history of noncompliance  at least once a year. These inspections help ensure that hydrocarbon production on  federally managed lands are properly accounted for and results in accurate royalty  payments to the public and Indian owners of such minerals. federal leases. Also, this revenue is one of the federal government’s  largest nontax source of revenue.
The Department of Labor (Labor) Described the Overall Quality of its  Performance Information, but Not on How Quality Is Ensured for its  Individual APGs  Labor’s performance reports contained limited discussion of performance  information quality for its APGs, as table 5 shows.  The reports referred  readers to its Summary of Performance and Financial Information, which  includes an attestation statement from the Secretary of Labor as to the  reliability and completeness of the agency’s performance information (we  reproduce these statements in the text boxes below). These statements  provide the reader with the Secretary’s assurance of the quality of Labor’s  performance information. However, these statements do not describe  what practices are in place to ensure that the agency is using accurate  and reliable performance information to measure and report on progress  for its individual APGs.
Department of Labor’s Data Completeness and Reliability Statement–Fiscal Year 2014 Annual  Performance Report  The Fiscal Year 2014 Summary of Performance and Financial Information includes an assessment by  the Secretary of the reliability and completeness of DOL (Labor’s) performance data reported under  the GPRAMA. The Department satisfies this requirement with Agency Head-level attestations that the  data do not contain significant limitations that can lead to inaccurate assessments of results.
The Secretary of Labor’s Data Quality Attestation Statement–Fiscal Year 2014 Summary of  Performance and Financial Information  Secretary’s Attestation Statement  I attest that the summarized financial and performance data included in this document as well as the  data in the Agency Financial Report and the Annual Performance Report are complete and reliable in  accordance with Federal requirements.
When we shared our analysis with Labor officials, they said that they  were not fully aware of the performance information quality requirements  for APGs. However, they emphasized that their agency places  considerable emphasis on ensuring the quality of its performance  information, and on conducting program evaluations to assess the  effectiveness of its programs. Labor officials referred us to other agency  publications–The Department of Labor FY 2014-2018 Strategic Plan and  The Department of Labor FY 2016 Congressional Budget Justification–for  explanation to the public of the activities they said their agency was taking  to ensure the quality of its performance information. We confirmed that  these publications do describe the agency’s research and evaluation  agenda. For example, Labor states in its strategic plan that it is committed  to improving the quality of performance information by conducting future  evaluations to ensure the outcome data it reports are accurate. While  Labor’s strategic plan does not explain the extent to which these data  quality studies will focus on issues related to its APGs, Labor does  provide valuable information and important context on how it plans to  ensure the quality of its performance information. Nevertheless,  GPRAMA’s performance information quality requirements are important  because our previous work on fragmentation, overlap, and duplication  shows that Labor needs to report more transparently on program  performance for its veterans’ employment and training programs. This  relates to one of Labor’s APGs on improving employment outcomes for  veterans. Specifically, we reported in 2012 that Labor provided Congress  with an annual veterans’ program report that provided certain  performance information, such as the number of disabled and recently  separated veterans who received intensive services. But, we found that  Labor was not reporting these results relative to the performance goals it  had set.  We recommended in our prior work that Labor report both  performance goals and associated performance outcomes for its  veterans’ employment and training programs. Labor agreed with our  recommendation and has made some progress in addressing it. For  example, Labor has reported on how the results achieved for the  performance measure it uses to measure progress on one of its APGs  (percent of veterans receiving intensive services served by Disabled  Veterans Outreach Program specialists) compare to the targets it set.  Labor provided this information for this performance measure in both its  performance report and on Performance.gov.
The National Aeronautics and Space Administration (NASA) Described Its  Overall Approach for Ensuring Performance Information Quality, but Does  Not Explain How Performance Information Quality for its APGs Is  Ensured  NASA’s performance plans and reports characterize the agency as a  performance-based organization committed to managing toward specific,  measurable goals, and to using performance information to continually  improve operations. As shown in table 6, these performance plans and  reports explain how the agency ensures the quality of its performance  information overall. For example, NASA stated in its performance plans  and reports that it held internal reviews for its projects, determined  technology readiness levels, and required mission directorates and  mission support offices to submit evidence supporting all performance  measure ratings.such as scientific review committees and aeronautics technical evaluation  bodies, to help it validate program performance. These statements  provide valuable insight into how NASA measures its performance and  uses evidence.
Further, NASA stated that it used external entities,  However, NASA’s performance plans and reports did not explain how the  agency ensured the quality of performance information for individual  APGs. NASA presented concise summaries of each APG, progress  updates, and next steps. However, there was little explanation provided  for external audiences which described how NASA took the approach it  has outlined for ensuring the overall quality of its performance  information, and applied this approach to individual APGs.
In our discussions with NASA officials, they emphasized that they do  collect information related to all of the GPRAMA requirements. To  illustrate this, NASA officials demonstrated their internal Performance  Measure Manager system to us in April 2015.  NASA officials told us that  the system functions as a warehouse for agency performance information  and they upload information from the system to Performance.gov for  quarterly APG updates and to help develop their performance plans and  reports. They showed us that this system collects information on the  quality of performance information for a range of performance measures,  including for APGs. For example, the internal database has a field for  verification and validation materials for the James Webb Space  Telescope APG, and identifies a data limitation for it.  However, NASA  does not publicly report all of the information the system collects on how it  ensures the quality of performance information for its APGs.
NASA officials expressed concern about how well the GPRAMA  performance information quality requirements can be applied to their  agency’s performance reporting. NASA officials said that they use what  GPRAMA and Office of Management and Budget Circular A-11 refer to as  an alternative form of performance measurement that, among other  things, allows an agency to use milestones for completing major  deliverables for the APG instead of performance measures.  They  further explained that they use numerous milestones to measure progress  on their APGs. They added that NASA only reports on key quarterly  milestones in its performance plans and reports and on  Performance.gov.  Unlike other agencies, they noted they often do not  have quantitative data sets for their performance information. However,  the GPRAMA performance information quality requirements apply to all  APGs, even if the agency is using milestones.
As we noted in our 2015 High Risk update, NASA plans to invest billions  of dollars in the coming years to explore space, understand Earth’s  environment, and conduct aeronautics research. We designated NASA’s  acquisition management as high risk in 1990 in view of NASA’s history of  persistent cost growth and schedule slippage in the majority of its major  projects. Going forward, we noted in our February 2015 high risk update  that it will be critical for NASA to ensure adequate and ongoing  assessments of risks related to two of its APGs for developing new  systems for exploring deep space and the James Webb Space  Telescope. However, as our review shows, NASA has not explained to  external audiences how it is ensuring the quality of performance  information for these APGs related to these high-risk areas. Without such  information, it will be more difficult for Congress and the public to  understand whether NASA is effectively measuring progress toward these  APGs, and whether the billions of dollars being spent to accomplish these  important efforts are being used effectively.
section_title: Cross-Agency Working Group Could Help Agencies and OMB Improve Public Reporting on the Quality of Performance Information Used to Measure Progress on Priority Goals
paragraphs: In 2015, OMB and the Performance Improvement Council (PIC)  established the Data Quality Cross-Agency Working Group. This group  met for the first time in February 2015; and four other meetings were held  in April, May, June, and July 2015. As of June 2015, PIC staff reported  that a total of 12 agencies were participating, which is more than half of  the agencies with APGs. Three of the six agencies we selected for  review–the Departments of Defense and Homeland Security (DHS) and  NASA–are participating. DHS and NASA officials told us that they have  made or plan to make presentations at these meetings on their agencies’  performance information quality processes. In addition, Interior’s Deputy  Performance Improvement Officer shared his agency’s verification and  validation standards with the group. An additional nine agencies–the  Departments of Commerce, Education, Health and Human Services,  Justice, Treasury, and Veterans Affairs, and the Environmental Protection  Agency, Small Business Administration, and Social Security  Administration–are also participating in the group.  According to meeting  notes for the May 2015 meeting provided by OMB and PIC staff, the  group had identified several goals:  improve the reliability and quality of performance information and of  the reporting process; set standards and develop consistency across agencies; and highlight good performance measures and accurate and appropriate  performance information.
The May 2015 meeting notes state the group’s end product will be to  identify solutions agencies have used to solve a data quality problem. In  June 2015, the PIC’s Executive Director explained that while the group is  still working on defining this end product, it wants to develop a collection  of useful and leading practices that can be shared with agency officials.  The PIC’s Executive Director and her staff also noted that this end  product could include providing recommendations to OMB on changes  that could be made to the A-11 guidance on how to address GPRAMA’s  performance information quality requirements. OMB staff also indicated to  us that they would like to get input from the group on additional changes  that could be made to A-11. PIC staff told us that the participating  agencies met with OMB staff in July 2015 to discuss additional changes  that should be made to A-11.
section_title: Conclusions
paragraphs: For more than two decades, agencies have been required to publicly  report on the quality of their performance information in annual  performance plans. More recently, agencies have also been required to  report on Performance.gov on how they will ensure the accuracy and  reliability of the performance information used to measure progress on  each of their highest priority performance goals, the APGs. However,  insufficient progress has been made. While OMB for several years has  directed agencies to discuss the quality of APG performance information  in their annual performance plans and reports, the selected agencies’  plans and reports often did not. OMB recently changed its guidance to  require agencies to provide this information for publication on  Performance.gov. The next key step is to build upon this recent guidance  to implement the change and make this important information readily  accessible to the public and Congress.
This overall lack of transparency means that members of Congress,  citizens, journalists, and researchers seeking information about agency  performance related to priority goals have to search in multiple places,  and often end up finding no explanation of the quality of performance  information for APGs. For agencies to maintain the confidence of  Congress and the public that they are indeed achieving their priority goals  for the challenging and complex results they seek to achieve, agencies  will need to provide more transparent explanations of how they are  ensuring the accuracy and reliability of performance information for their  APGs.
More broadly, our review shows that five agencies continue to provide  limited information in their annual performance plans and reports  concerning the quality of performance information for their APGs. The  same is true for all six agencies on Performance.gov. In some cases, the  needed context and information may be available within the agency for  the agency’s use. However, this information is not consistently provided  to external audiences. The Performance Improvement Council’s (PIC)  Data Quality Cross-Agency Working Group provides a potential forum for  agencies to collaborate and share information on this topic, and the group  is defining its intended end product. Given the shortcomings our review  identified at the majority of the six agencies reviewed, the working group  could help agencies identify practices that will help them more clearly  explain to Congress and the public how they are ensuring that the  performance information for their highest priority performance goals is  accurate and reliable. OMB could also work with this PIC working group  to continue updating its guidance to agencies to ensure that this  information is readily accessible on Performance.gov.
section_title: Recommendations for Executive Action
paragraphs: To improve the public reporting about how agencies are ensuring the  quality of performance information used to measure progress towards  their priority goals, we recommend the following actions:
The Secretaries of Agriculture, Defense, Homeland Security, Interior,  and Labor, and the Administrator of NASA should more fully address  GPRAMA requirements and OMB guidance by working with OMB to  describe on Performance.gov how they are ensuring the quality of  performance information used to measure progress towards their  APGs.
The Secretaries of Agriculture, Defense, Interior, and Labor, and the  Administrator of NASA should more fully address GPRAMA  requirements and OMB guidance by describing in their agencies’  annual performance plans and reports how they are ensuring the  quality of performance information used to measure progress towards  their APGs.
To help participating agencies improve their public reporting, we  recommend that the Director of OMB, working with the PIC Executive  Director, should:  Identify additional changes that need to be made in OMB’s guidance  to agencies related to ensuring the quality of performance information  for APGs on Performance.gov.
Identify practices participating agencies can use to improve their  public reporting in their performance plans and reports of how they  are ensuring the quality of performance information used to measure  progress towards APGs.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Director of OMB and the  Secretaries of Agriculture, Defense, Homeland Security, Interior, and  Labor, and the Administrator of NASA.
The Department of the Interior and NASA concurred with the  recommendations directed to them, and discussed specific actions they  plan to take to address these recommendations. Interior and NASA’s  written responses are reproduced in appendixes II and III.
In its response, NASA also shared a concern about how we portrayed its  high-risk reporting in the draft report. It stated that our draft report  suggested that the select milestones identified as part of its performance  reporting are the sole mechanisms NASA uses to assess risks and  measure progress towards launching the James Webb Space Telescope,  and developing new systems for human exploration of deep space. NASA  further stated that to comply with reporting requirements related to APGs,  it has opted to provide information on key quarterly milestones that the  public can easily understand. To address NASA’s concern, we revised  the report to recognize that NASA officials told us they use numerous  milestones to measure progress on their APGs. NASA also said that it  only reports on key quarterly milestones in its performance plans and  reports, and on Performance.gov. NASA also provided technical  clarifications, which we incorporated as appropriate.
The Department of Homeland Security (DHS) also concurred with the  recommendation directed to it. However, DHS stated that it has already  taken action to implement our recommendation to work with OMB to  describe on Performance.gov how DHS is ensuring the quality of  performance information used to measure progress towards its APGs.  Thus, DHS regards the recommendation as resolved and closed. DHS  stated that on July 1, 2015, agency officials in its Office of Program  Analysis and Evaluation provided OMB with several specific suggestions  to consider as possible enhancements to the internal system that OMB  uses to gather agency data for public posting on Performance.gov. This  will allow agencies to include more comprehensive data quality  information on this public website. DHS’s efforts are an important step  toward addressing our recommendation. However, as our review found,  and DHS recognizes in its response letter, more will need to be done to  make DHS’s explanations of performance information quality for its APGs  accessible to external audiences on Performance.gov. For example, in  our report we noted that OMB’s updated A-11 guidance in June 2015  gives agencies the option of providing a hyperlink from Performance.gov  to an appendix in their performance reports containing their performance  information quality discussion, which DHS could do. We will continue to  monitor DHS’s efforts to work with OMB to fully implement the  recommendation. DHS’s written comments are reproduced in appendix  IV.
The Department of Defense (DOD) partially concurred with the  recommendations directed to it. DOD stated that it has ongoing actions to  improve the quality of performance information, and to make better use of  that information in management. However, DOD stated that it did not  agree that making discussion of the process of managing the quality of  performance information a part of either the agency strategic plan or  annual reporting has any major management value. We disagree. First,  GPRAMA does not require, nor did we recommend, that DOD provide  information on the quality of performance information for its agency  priority goals in its agency strategic plan. Second, GPRAMA does require  that this information be provided in agency performance plans and reports  and on Performance.gov. We continue to believe it is important for DOD  to fully address these GPRAMA requirements because, as described in  our report, two of DOD’s APGs address areas we have identified as high  risk. Also, DOD is responsible for more than half of the federal  government’s discretionary spending. DOD’s written comments are  reproduced in appendix V.
The Departments of Agriculture (USDA) and Labor did not comment on  the recommendations directed to them. However, they both discussed  specific actions they plan to take to improve the quality of their publicly- reported performance information for their agency priority goals. In  comments relayed to us in an August 14, 2015, e-mail from the Associate  Director of USDA’s Office of Budget and Program Analysis, who is also  the agency’s Performance Improvement Officer (PIO), he stated the  agency would ensure that a description of the quality of performance  information be added for each performance measure included in its APGs  for fiscal years 2016 and 2017. He also stated that USDA will work with  OMB to put this information on Performance.gov or in its annual  performance plan and report with a reference to that information on  Performance.gov. He also stated a reference to this information would be  provided in USDA’s annual performance plan. He also provided us with a  technical clarification, which we incorporated.
In its response, Labor raised a concern about statements in our draft  report regarding information on several of its programs that serve  veterans, which were drawn from our prior work. Labor stated that the  report incorrectly asserts that it does not report on the number of veterans  receiving intensive services relative to performance goals. Rather, Labor  stated that it has used the Veterans’ Employment and Training Services  measure for the percent of veterans being served by the Disabled  Veterans’ Outreach Program as an APG for 4 years, and has included  how results relate to performance goals. Additionally, Labor asserted that  outcomes for the Veterans Workforce Investment Program have been  included in the annual report to Congress in fiscal years 2013 and 2014.  We revised the report to reflect Labor’s updated actions as appropriate.  Labor’s written comments are reproduced in appendix VI.
In an August 26, 2015, e-mail from OMB’s liaison to GAO, OMB did not  comment on the recommendations, but provided technical clarifications,  which we incorporated as appropriate.
We are sending copies of this report to the Director of OMB and the  heads of the agencies we reviewed as well as appropriate congressional  committees and other interested parties. In addition, this report is  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VII.
section_title: Appendix I: Selected Agencies’ Priority Goals and Performance Measures and Milestones
subsections: section_title: Total for DOD: 4 goals
section_title: Total for Interior: 6 goals
section_title: Appendix II: Comments from the Department of the Interior
section_title: Appendix III: Comments from the National Aeronautics and Space Administration
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: Comments from the Department of Defense
section_title: Appendix VI: Comments from the Department of Labor
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: J. Christopher Mihm, (202) 512-6806 or mihmj@gao.gov.
section_title: Staff Acknowledgements
paragraphs: In addition to the contact named above, Sarah E. Veale, Assistant  Director, and Michael O’Neill, Analyst-in-Charge, supervised the  development of this report. Virginia Chanley, Emily Christoff, Erik  Kjeldgaard, and A.J. Stephens made significant contributions to all  aspects of this report. Deirdre Duffy and Robert Robinson provided  additional assistance.