Data Act: Section 5 Pilot Design Issues Need to Be Addressed to Meet Goal of Reducing Recipient Reporting Burden
Published: 2016-04-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The DATA Act directs OMB or a designated federal agency to establish a pilot program to develop recommendations for simplifying federal award reporting for grants and contracts. The grants portion will test six ways to reduce recipient reporting burden while the procurement portion will initially focus on centralizing contractor reporting of certified payroll. The act requires GAO to review DATA Act implementation as it proceeds.
This report (1) describes OMB's approach to the DATA Act pilot requirements, (2) assesses whether current plans and activities will likely allow OMB and its partners to meet the requirements under the act, and (3) evaluates the extent to which designs for the grants and procurement portions of the pilot are consistent with leading practices. GAO reviewed available pilot documentation; assessed them against leading practices for pilot design; and interviewed staff at OMB, HHS, and GSA, as well as groups representing recipients of federal grants and contracts. GAO will conduct a follow-on review focused on OMB's implementation of its pilot designs.
section_title: What GAO Found
paragraphs: As required by the Digital Accountability and Transparency Act of 2014 (DATA Act), the Office of Management and Budget (OMB) is conducting a pilot program, known as the Section 5 Pilot, aimed at developing recommendations for reducing recipient reporting burden for grantees and contractors. OMB partnered with the Department of Health and Human Services (HHS) to design and implement the grants portion of the pilot, and with the General Services Administration (GSA) to implement the procurement portion. OMB launched the Section 5 Pilot in May 2015 and expects to continue pilot-related activities until at least May 2017.
If implemented according to HHS's proposed plan, the grants portion of the pilot will likely meet the requirements established under the act. In contrast, GAO has concerns with how the procurement portion of the pilot will contribute to the Section 5 Pilot's design requirements. For example, OMB has not fully described how it will select pilot participants that will result in a diverse group of contractors as required by the act. OMB staff stated that they intend to select participants for testing the procurement pilot by using a nongeneralizable sample of contractor data, but they have not provided a detailed, documented sampling plan.
The design of the grants portion of the pilot partially adhered to leading practices. Although five out of the six grants test models had clear and measurable objectives, only one had specific details about how potential findings could be scalable to be generalizable beyond the context of the pilot. HHS officials said they have updated their plan to address these concerns but that plan was not provided in time to allow GAO to analyze it for this review.
The design of the procurement portion of the pilot did not reflect leading practices. For example, the plan did not include specific information on the methodology, strategy, or types of data to be collected. Further, the plan we reviewed did not address the extent to which the proposed pilot approach would be scalable to produce recommendations that could be applied government-wide. The design also did not indicate how data will be evaluated to draw conclusions. Finally, while OMB has solicited general comments related to contractor reporting pain points, it has not released specific details on the design to stakeholders despite their repeated requests for that information.
section_title: What GAO Recommends
paragraphs: GAO recommends that OMB (1) clearly document how the procurement portion of the pilot will contribute to the design requirements under the DATA Act and (2) ensure that the design of the procurement portion of the pilot reflects leading practices. OMB, HHS, and GSA did not comment on our recommendations. GAO incorporated technical comments from OMB and HHS where appropriate.

section_title: Letter
paragraphs: The federal government awards hundreds of billions of dollars in grants  and contracts annually. To help ensure compliance with applicable laws  and regulations, the recipients of these funds are required to report  spending and other information. However, grant recipients and federal  contractors often face challenges related to duplicative and burdensome  reporting.
Effective implementation of the Digital Accountability and Transparency  Act of 2014 (DATA Act) offers the promise of addressing some of the  challenges of duplicative and burdensome reporting as it requires the  Office of Management and Budget (OMB) and Department of the  Treasury (Treasury) to establish standardized government-wide financial  data standards. A key lesson learned from our prior reports on the  American Recovery and Reinvestment Act of 2009 is that standardized  data could decrease the burden on federal fund recipients and increase  the accuracy of the data reported. In addition, section 5 of the Federal  Funding Accountability and Transparency Act of 2006 (FFATA), as  amended by the DATA Act, provides another opportunity for simplifying  reporting for federal contracts, awards, and subawards. It directs OMB, or  a federal agency designated by OMB, to establish a pilot program to  develop recommendations for eliminating unnecessary duplication in  financial reporting and reducing compliance costs for federal award  recipients.
This report is our latest work in response to a statutory provision to review  DATA Act implementation. This report (1) describes the administration’s  approach to the Section 5 Pilot requirements, (2) assesses whether  current activities and plans that were available during the review period  will likely allow OMB and its partners to meet requirements and time  frames established under the Section 5 Pilot, and (3) evaluates the extent  to which the pilot design are consistent with leading practices.
To address these objectives, we assessed pilot activities by reviewing  design documentation from the Department of Health and Human  Services (HHS) and OMB’s Office of Federal Procurement Policy (OFPP).  Our reviews were based on the latest design plans available at the time— specifically, HHS’s draft plan dated November 16, 2015, and OFPP’s  working draft dated November 28, 2015. We interviewed OMB, HHS, and  General Services Administration (GSA) officials responsible for  implementing section 5 of the act. We also interviewed officials from  organizations representing key nonfederal stakeholders, including state  governments, private-sector contractors, and other federal fund  recipients. To identify leading practices for pilot design, we reviewed past  work evaluating and assessing pilots as well as relevant studies from  academia and other entities. We plan to conduct a review on the  implementation of the pilot and the extent to which its design will allow  OMB to develop recommendations for reducing recipient reporting burden  later this year. Additional details regarding our objectives, scope, and  methodology are provided in appendix I.
We conducted this performance audit from May 2015 to April 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The DATA Act became law in May 2014 and holds considerable promise  for shedding more light on how federal funds are spent. To improve the  transparency and quality of the federal spending data made available to  the public, the DATA Act directed OMB and Treasury to establish  government-wide data standards that include common data elements for  reporting financial and payment information by May 2015. Under the act,  federal agencies must begin reporting financial spending data using these  standards by May 2017 and publicly post spending data in a machine- readable format by May 2018.
subsections: section_title: Section 5 Pilot Requirements for Pilot to Simplify Federal Award Reporting
paragraphs: The DATA Act also requires that OMB, or an agency it designates,  establish a pilot program to facilitate the development of  recommendations to (1) standardize reporting elements across the  federal government, (2) eliminate unnecessary duplication in financial  reporting, and (3) reduce compliance costs for recipients of federal  awards. The act established reporting requirements and timeframes for  implementation of the pilot. See figure 1 for a timeline of these deadlines.
The DATA Act also sets specific requirements related to the pilot’s  design. First, the pilot must collect data during a 12-month reporting  cycle. The pilot must also include a diverse group of recipients such as  awardees receiving a range of awards as long as the total value of the  awards falls within the statutory range. To the extent practicable, the pilot  is to include recipients who receive federal awards from multiple  programs across multiple agencies. Finally, the pilot must include a  combination of federal contracts, grants, and subawards with an  aggregate value between $1 billion and $2 billion.
In addition, OMB must review the information recipients are required to  report to identify common reporting elements across the federal  government, unnecessary duplication in financial reporting, and  unnecessarily burdensome reporting requirements for recipients of  federal awards. This review is to be done in consultation with relevant  federal agencies and recipients of federal awards, including state and  local governments and institutions of higher education.
section_title: Leading Practices for Effective Pilot Design
paragraphs: A well-developed and documented pilot program can help ensure that  agency assessments produce information needed to make effective  program and policy decisions. Such a process enhances the quality,  credibility, and usefulness of evaluations in addition to helping to ensure  that time and resources are used effectively. We have identified five  leading practices that, taken together, form a framework for effective pilot  design. To identify these practices, we reviewed our prior work as well as  academic literature related to the design of pilot and evaluation  programs. By following these leading practices, agencies can promote a  consistent and effective pilot design process. We shared these practices  with OMB, HHS, and GSA staff, who found them to be reasonable and  appropriate, and applicable to the Section 5 Pilot.  1.  Establish well-defined, appropriate, clear, and measurable objectives.
Such objectives should have specific statements of the  accomplishments necessary to meet the objectives. Clear and  measurable objectives can help ensure that appropriate evaluation  data are collected from the outset of pilot implementation so that data  will subsequently be available to measure performance against the  objectives. Broad study objectives should be translated into specific,  researchable questions that articulate what will be assessed.  2.  Clearly articulate assessment methodology and data gathering  strategy that addresses all components of the pilot program and  includes key features of a sound plan.
Key features of a clearly articulated methodology include a strategy  for comparing the pilot implementation and results with other efforts, a  clear plan that details the type and source of the data necessary to  evaluate the pilot, and methods for data collection including the timing  and frequency.  3.  Identify criteria or standards for identifying lessons about the pilot to  inform decisions about scalability and whether, how, and when to  integrate pilot activities into overall efforts.
The purpose of a pilot is generally to inform a decision on whether  and how to implement a new approach in a broader context.  Therefore, it is critically important to consider how well the lessons  learned from the pilot can be applied in other, broader settings. To  assess scalability, criteria should relate to the similarity or  comparability of the pilot to the range of circumstances and population  expected in full implementation. The criteria or standards can be  based on lessons from past experiences or other related efforts  known to influence implementation and performance as well as on  literature reviews and stakeholder input, among other sources. The  criteria and standards should be observable and measureable events,  actions, or characteristics that provide evidence that the pilot  objectives have been met. Choosing well-regarded criteria against  which to make comparisons can lead to strong, defensible  conclusions.  4.  Develop a detailed data-analysis plan to track the pilot program’s  implementation and performance and evaluate the final results of the  project and draw conclusions on whether, how, and when to integrate  pilot activities into overall efforts.
A detailed data-analysis plan identifies who will do the analysis as well  as when and how data will be analyzed to measure the pilot  program’s implementation and performance. The results will show the  successes and challenges of the pilot, and in turn, how the pilot can  be incorporated into broader efforts. Some elements of a detailed  data-analysis plan include talking to users, managers, and  developers; evaluating the lessons learned to improve procedures  moving forward; and other appropriate measures.  5.  Ensure appropriate two-way stakeholder communication and input at  all stages of the pilot project, including design, implementation, data  gathering, and assessment  Appropriate two-way stakeholder communication and input should  occur at all stages of the pilot, including design, implementation, data  gathering, and assessment. Failure to effectively engage with  stakeholders, and understand and address their views can undermine  or derail an initiative. To that end, it is critical that agencies identify  who the relevant stakeholders are, and communicate early and often  to address their concerns and convey the initiative’s overarching  benefits.
section_title: OMB Is Conducting a Pilot Focused on Reducing Grantee and Contractor Reporting Burden to Meet Its Requirements under Section 5
paragraphs: OMB has established a Section 5 Pilot with two primary focus areas—one  on federal grants and another on federal contracts (procurement). OMB’s  Office of Federal Financial Management is responsible for the grants  portion of the pilot and has designated the Department of Health and  Human Services (HHS) to serve as its executing agent. On the  contracting side, OMB’s OFPP is responsible for leading the procurement  portion and is working with various entities including 18F and the Chief  Acquisitions Officers’ Council (CAOC). Specifically, 18F is designing the  system to be tested as part of the pilot. GSA’s Office of Government-wide  Policy is responsible for providing federal register notices; and its  Integrated Award Environment provides guidance and technical  considerations. OMB launched a number of pilot-related initiatives in May  2015 and expects to continue activities until at least May 2017.
subsections: section_title: Grants Portion of the Pilot Will Test Six Ways to Reduce Grantee Reporting Burden
paragraphs: As the executing agent for the grants portion of the pilot, HHS has  developed six “test models” that will evaluate different approaches to  potentially reducing grantee reporting burden. These six models are the  specific grants tools, forms, or processes that will be tested and analyzed  under the pilot to determine if adopting these changes will actually  contribute to the program’s objectives of reducing reporting burden,  duplication, and compliance costs.
Taken as a whole, the six test models examine a variety of grant reporting  issues that HHS has identified as presenting challenges. HHS officials  told us that they have received comments through the National Dialogue,  a website for grant recipients and contractors to discuss issues including  compliance costs, reporting burden, eliminating duplication, and  standardizing processes. In addition, the officials obtained feedback on  areas of concern from grantees involved in earlier HHS efforts to  streamline grants reporting. They used that information to inform the  development of the six test models.
Officials from advocacy groups representing grant recipients and federal  contractors told us that they initially expected the grants portion of the  pilot to be an extension of the Grants Reporting Information Project  (GRIP) proof of concept that was launched following the enactment of the  American Recovery and Reinvestment Act of 2009 rather than the six test  models. HHS officials told us they would have liked to more fully  replicate the GRIP, however, that would have required broader  participation from agencies than was available for the Section 5 Pilot.
The following provides high-level summaries of each of the six test  models. For additional details, see appendix II.
HHS intends to assess whether an online and searchable repository  for data standards will facilitate grant reporting. To do this, HHS  developed the Common Data Element Repository (CDER) Library,  which is intended to be an authorized source for data elements and  definitions for use by the federal government and recipients reporting  grant information.
The CDER Library is also intended to encourage the use of common  definitions for grants-related terms by nonfederal stakeholders and  federal agencies. As of March 2016, the publicly-available version of  the CDER Library contained 112 data elements from a variety of  sources, including the Federal Acquisition Regulation (FAR), OMB  Circular A-11, and the Uniform Grant Guidance. It also included  several data elements standardized in accordance with DATA Act  requirements.
HHS has developed a version of the CDER Library, accessible only to  federal agencies, that contains a much more detailed database of  more than 9,000 elements. This federal-agency-only version of the  CDER Library also identifies which grant reporting forms these data  elements come from so that users can see how many forms require  the same data element and which agencies request that information  from grantees. HHS officials told us that they believe the CDER  Library has the potential to be a powerful tool for streamlining  definitions and forms.
HHS intends to test whether it will be possible to use a consolidated  Federal Financial Report (FFR) to allow grantees to submit multiple  reporting forms into one system. The FFR, reported on the Standard  Form 425, is used for reporting grants expenditures for the recipients  of federal assistance. HHS believes that a consolidated FFR will allow  participants to submit complete information once instead of through  multiple entry points. A consolidated FFR could provide a single point  of data entry, earlier validation of FFR data, and potential future  streamlining of the grants close-out process.
According to HHS officials, this test model is intended to be a  continuation of the GRIP launched during the American Recovery and  Reinvestment Act of 2009. The aim of that effort was to determine the  feasibility of developing a centralized government-wide collection  system for federal agencies and recipients of federal awards.
HHS is examining ways to reduce duplicate and redundant  information contained in Single Audit forms. The Single Audit Act  requires states, local governments, and nonprofit organizations  expending $750,000 or more in federal awards in a year to obtain an  audit in accordance with the requirements set forth in the act. HHS  intends to test whether some grant forms related to the single audit  could be combined.
HHS plans to examine whether a consolidated Notice of Award  coversheet might reduce reporting burden by allowing grant recipients  to locate required reporting data in one place, rather than attempting  to find information on coversheets that differ by agency.
HHS added a new section to the Grants.gov website, called Learn  Grants, intended to make it easier for stakeholders to find, learn  about, and apply for federal grants. The Learn Grants website  provides links to grant policies, processes, funding, and other grant  lifecycle information. HHS officials said they want to use this test  model to determine whether the Learn Grants site could effectively  engage stakeholders and provide training early in the grants lifecycle  process that, in turn, would have a positive effect on recipient  compliance during post-award activities.
section_title: Procurement Portion of the Pilot Will Focus on Centralizing Certified Payroll Reporting
paragraphs: The procurement portion of the pilot will be focused on examining the  feasibility of centralizing the reporting of certain required information.  Depending on the contract, there may be many types of information  contractors must report. OFPP staff told us the pilot will initially focus on  the reporting of certified payroll. This is one specific FAR requirement  only applicable to contracts for construction within the United States.
Specifically, OFPP has identified opportunities to improve upon the  current unstandardized reporting format under which some employers  report data electronically while others use manual paper processes.  Further, OFPP intends to identify which data elements would be included  in reporting, the method of data transmission, and other related details.
This narrow approach stands in contrast to the grants portion of the pilot  where HHS has a broader, more comprehensive plan to explore several  areas where grantee reporting burden might be reduced. OFPP staff  explained that its decision to focus on certified payroll reporting arose out  of feedback from the procurement community. They also noted that the  Section 5 Pilot is one of a number of government-wide initiatives to  reduce contractor burden and streamline procurement processes, such  as GSA’s Integrated Award Environment initiative to integrate acquisition  systems into one streamlined environment.
To better understand the issue of certified payroll reporting and its  potential suitability as a subject for the procurement portion of the Section  5 Pilot, the CAOC engaged GSA’s 18F through an interagency  agreement to interview contractors, contracting officers, business owners,  government employees, and subject-matter experts (SME). As a result of  that effort, 18F identified major categories of burdens and constraints  related to certified payroll reporting and potential recommendations on  how to address them.
OFPP staff said they once again worked with 18F in winter 2016 to gather  requirements for building a prototype system to centralize the reporting of  certified payroll data. The 18F staff we spoke with noted that they will  build a prototype to explore potential solutions for reducing contractor  burden through user research and testing. OFPP staff will develop and  evaluate metrics for the pilot. OFPP intends to test the system in summer  2016.
section_title: National Dialogue Sought Feedback from Grantees and Contractors on Ways to Simplify Federal Award Reporting
paragraphs: In May 2015, OMB, CAOC, GSA, and HHS launched the National  Dialogue, a website for grant recipients and federal contractors to discuss  issues including compliance costs, reporting burden, eliminating  duplication, and standardizing processes. OMB staff told us that they  used the National Dialogue as a feedback mechanism for the grants and  procurement portions of the pilot. This was one of the first publicly  announced pilot-related activities. The website will accept comments  through May 2017. OMB and GSA staff told us that they plan to actively  review and address the input they receive.
The website is intended to be a useful tool for obtaining information about  issues of concern to their respective communities. Discussions related to  grantee reporting have been significantly more active than those focused  on procurement. Although the comments vary widely in topic, there are a  number of substantive suggestions for how grantee reporting burdens can  be reduced. While HHS officials told us that the dialogue was intentionally  designed so that feedback would be submitted anonymously, some  commenters have self-identified the institution they represent, including  the Council on Governmental Relations, Association on American  Universities, Association of Public and Land-grant Universities, and  Coalition for Government Procurement.
section_title: The Grants Portion of Section 5 Pilot Is Generally On Track to Meet DATA Act Requirements but the Procurement Portion Is Not
subsections: section_title: The Design of the Grants Portion of the Pilot Is Generally On Track to Meet DATA Act Requirements
paragraphs: If HHS effectively implements its stated plans for the grants portion of  the Section 5 Pilot, it is likely that the grants portion of the pilot will  comply with the act. These requirements call for the grants portion’s  design to include the following:  DATA Act Requirement 1: Collect data during a 12-month reporting  cycle. HHS’s November 2015 design documentation shows that it will  begin collecting data for these six test models by May 2016. This would  allow for data to be collected on these test models during a 12-month  reporting cycle before May 2017, when the pilot is required to terminate.  We believe these timeframes should provide sufficient time for HHS to  incorporate public comments by May 2016 and allow for a full 12-month  data collection cycle.
DATA Act Requirement 2: Include a diverse group of federal award  recipients and, to the extent practicable, recipients who receive federal  awards from multiple programs across multiple agencies. HHS officials  told us that they have developed a detailed plan to select participants,  which will include state and local governments, universities, and other  types of grant recipients. HHS officials explained that the grants portion of  the pilot will include recipients who received a range of federal funding  amounts and will not be limited to one agency or grant program.
HHS officials initially told us that they could not provide us with the  revised plan because it was still under review by OMB. We did receive a  copy of the revised plan at the end of March 2016, but because of the  timing we were unable to fully review it in time for the release of this  report. We will provide our assessment of the plan as part of future work  as we continue to monitor the design and implementation of the Section 5  Pilot.
DATA Act Requirement 3: Include a combination of federal contracts,  grants, and subawards, with an aggregate value of not less than $1 billion  but not more than $2 billion. HHS officials told us that they are still  determining how to meet the requirement for total award value because  they want to ensure the pool of pilot participants is as diverse and large  as possible while still being legally compliant. Specifically, one of their  selection considerations is the award value of grants received by  awardees. Further, HHS officials have explored strategies to ensure that  they do not exceed the maximum dollar amount threshold. HHS officials  told us that they expect to make decisions related to how to meet this  requirement in early 2016.
section_title: The Design of the Procurement Portion May Not Contribute to Pilot’s Ability to Meet DATA Act Requirements
paragraphs: We have concerns about the extent that the design of the procurement  portion of the pilot reflects the requirements specified in the DATA Act.  OFPP’s plans to address those statutory design requirements discussed  below reflect the status of the procurement portion of the pilot described  by OFPP staff and related documents we reviewed.
DATA Act Requirement 1: Collect data during a 12-month reporting cycle.  The design of the procurement portion of the pilot is at risk of not  including data collected during a 12-month reporting cycle in a meaningful  way. To meet this requirement, OFPP and GSA would need to begin  collecting data no later than May 9, 2016.
When we spoke with OFPP staff, they stated that by launching the  National Dialogue in May 2015, they believe they will have met the act’s  requirements that data collection take place during a 12-month reporting  cycle. Further, staff also considered comments received from other efforts  including the Open Dialogue on Improving How to Do Business with the  Federal Government conducted in 2014 to meet this requirement.  However, neither of these dialogues included comments that specifically  mentioned the issue of certified payroll. As a result, we do not believe  those comments provide meaningful and relevant data on the  effectiveness of a centralized portal for certified payroll reporting.
As a result of design and development delays, OFPP will not be able to  collect meaningful and useful data for the procurement portion of the pilot  until summer 2016, when it expects to complete the development of a  centralized portal through which participants will submit certified payroll  data. OFPP started exploring ways to streamline certified payroll reporting  in spring 2015.
OFPP said that due to staffing challenges, work on designing a prototype  for a system to be tested under the pilot did not begin until late February  2016. At that time, the CAOC signed an agreement with GSA’s 18F to  begin what it expected to be a 10-week design period. Cognizant staff  expect this design work will take place between March and May 2016.  However, a contractor cannot begin building an actual “production”  version of the system to be tested under the pilot until 18F designs the  prototype, which is expected to be completed by the beginning of May  2016. Therefore, this leaves at most a few weeks to develop the  centralized reporting portal before May 9, 2016—the date which the pilot  must begin for meaningful and useful data to be collected in a full 12- month period. OFPP staff told us that they do not intend to begin testing a  centralized reporting portal until late summer 2016.
According to OFPP and GSA staff, they were faced with delays due to bid  protests related to the contracting mechanism GSA intends to use to  select a contractor to build the portal to be tested under the pilot.  However, as of March 2016, these bid protests have been resolved and  no longer present a barrier in awarding the contract. While we agree that  these protests could pose a barrier to awarding the contract to develop  the testing portal, we do not believe that OFPP needed to wait until they  were resolved before moving forward with 18F’s development of a  prototype for the portal.
Given the resolution of these bid protests, OFPP staff said that they are  working with 18F to assess the feasibility of expediting project timelines to  launch the prototype sooner than expected so that they could potentially  collect 10 months of data through the certified payroll reporting portal.  Given the weekly or bi-weekly reporting of certified payroll, this approach  may result in a sufficient amount of meaningful and useful data on which  OFPP can base conclusions related to its hypothesis. However, it is  important that OFPP clearly conveys and documents its rationale for how  its approach will contribute to the collection of meaningful and useful data  consistent with the timeframes established under the act.
DATA Act Requirement 2: Include a diverse group of federal award  recipients and, to the extent practicable, recipients who receive federal  awards from multiple programs across multiple agencies. OFPP and GSA  do not yet have a detailed plan for selecting participants that will result in  a diverse group of recipients with awards from multiple programs and  agencies. However, there is some documentation related to OFPP’s  approach for selecting participants in the project plan and in a Federal  Register notice issued on November 24, 2015. For example, the draft  plan identifies the Federal Procurement Data System-Next Generation as  the mechanism that will be used for identifying which contracts and  contractors to include in the pilot. OFPP staff also told us that they  intend to cover both large and small industries. While valuable  information, these documents do not clearly convey how the procurement  portion of the pilot would specifically contribute to meeting the act’s  requirement regarding diversity of participants.
OFPP staff told us that for the purposes of meeting the pilot requirements  they consider any individual or group that provided information to the  National Dialogue to be a participant in the pilot. However, as previously  mentioned, individuals and groups that have commented on the National  Dialogue did not provide any comments related to certified payroll.  Therefore, it is unclear how they could be considered pilot participants.
Additionally, OFPP staff were unable to tell us how they plan to count  commenters that are not contract awardees, but instead are organizations  representing groups of federal contractors. It is unclear how OFPP can  ensure the universe of commenters is diverse because it does not control  who comments on the dialogue.
OFPP staff stated that they also intend to select participants for testing  their prototype system using a nongeneralizable sample of contractor  data reported through the Federal Procurement Data System-Next  Generation. However, they did not provide us with specific information on  how they would ensure that the sample met all requirements under the  act, nor did they provide a detailed, documented sampling plan equivalent  to the grants portion of the pilot. As a result, it will be important for OFPP  to clearly document its rationale for how its approach will allow for the  inclusion of a diverse group of federal contractors, as required by the act.
DATA Act Requirement 3: Include a combination of federal contracts,  grants, and subawards, with an aggregate value of not less than $1 billion  but not more than $2 billion. OFPP staff told us OMB could meet this  dollar range requirement through the grants and procurement portions of  the pilot collectively. Under such an approach, it would be important for  each portion of the pilot to know how much it is contributing to meet the  required award range. Our understanding of the grants portion of the pilot  suggests that that it has a plan for doing this. Less apparent are the  specifics of how the procurement portion of the pilot would do so.
section_title: Design of Grants Portion of the Pilot Partially Adheres to Leading Practices While Design of the Procurement Portion Does Not
paragraphs: We assessed the designs of the grants and procurement portions of the  pilot against leading practices that we identified from our prior work and  other sources. In continuation of our constructive engagement approach  for working with agencies implementing the DATA Act, we shared the  results of our analysis with HHS and OFPP staff who told us that they will  consider our input as they continue to update and revise their plans.
subsections: section_title: Design of the Grants Portion of the Pilot Partially Reflects Leading Practices
paragraphs: HHS’s November 2015 design for the grants portion of the pilot generally  applied leading practices. As noted above, while we have received a  revised plan for the design of the grants portion, we were unable to fully  review it in time for the release of this report. We will provide our   assessment of that plan in a forthcoming review that will focus on the  pilot’s implementation.
DATA Act Grants Test Models  Under the Office of Management and  Budget’s (OMB) direction, the Department of  Health and Human Services (HHS) intends to  develop recommendations for reducing  grantee reporting burden by testing different  areas. HHS will develop and test:
An online repository for data elements  and definitions that is intended to be an  authoritative source for data elements and  definitions, called the Common Data  Element Repository (CDER) Library.
A federal agency-only version of the  CDER Library containing more than 9,000  grants data elements that identify which  specific grant forms these data elements  come from, so that users can see how  many forms require the same data  element and which agencies request that  information.
Leading Practice 1: Establish Well-Defined, Appropriate, Clear, and  Measurable Objectives. Each of the six grants test models at least  partially met the leading practice that pilots have well-defined,  appropriate, clear, and measurable objectives. For example, one of the  Single Audit test models has the clearly defined objective of testing  whether two forms containing duplicative information can be combined to  reduce recipient reporting burden. This objective is measurable and  appropriately linked to the purposes of the Section 5 Pilot overall, which  include eliminating unnecessary duplication in financial reporting and  reducing compliance costs for recipients of federal awards. In another  example, one of the CDER Library test models has a clearly established  objective of determining whether access to an authoritative source for  common data element definitions would help grant recipients complete  necessary forms accurately and in a timely manner. The CDER Library  test model also identifies specific metrics that would allow them to  measure whether they are able to achieve its stated objectives.
In our initial review of these test models, we provided feedback to HHS  that the other CDER Library test did not have a clear, fully established  objective. In response, HHS officials explained that the objective of that  test model is to compare data elements and forms used across the  federal government with the goal of consolidating these forms and  ultimately passing on reporting efficiencies to grant recipients.
Leading Practice 2: Clearly Articulate an Assessment Methodology. Five  of the six test models did not clearly articulate an assessment  methodology. In contrast, for the Learn Grants test model, HHS described  how it planned to use webinars, conference presentations, and other  events to increase awareness inside and outside of government about the  grants-related resources available on Grants.gov. The plan also includes  a detailed timeline for executing the test model, as well as HHS’s  methodology for conducting pre- and post-tests of pilot participants. HHS  officials told us that they worked with a federal SME with previous  experience working on Grants.gov to help develop and refine the  assessment methodology.
The remaining five test models have less clearly articulated assessment  methodologies. For example, for the consolidated FFR test model, HHS  said it will survey grant recipients on their experiences when submitting  their reports into one system rather than multiple entry points; but we  found that the plans lacked detail about how surveys will be designed and  administered. In addition, the plan did not provide specific information  about the participants HHS intends to survey, nor did it provide details  regarding how HHS will compare survey results for recipients in the pilot  versus those not participating in the pilot.
In meetings with senior HHS officials, we raised these and similar  concerns about the Notice of Award test model and one of the CDER  Library models. For the other CDER Library test model, we found that  HHS’s plans did not identify the data sources or metrics that would be  used in the assessment methodology. In those feedback meetings, HHS  officials said many of the concerns have been addressed in the revised  plan.
Leading Practice 3: Ensure Scalability of Pilot Design. HHS documented  an overall structure for how each test model is integrated into the overall  grants portion of the pilot. However, the documented design lacks specific  details about how HHS intends to evaluate the performance of each test  model to inform decisions about scalability. Specifically, five of the six test  models include either no or few specifics about how any observed  reduction in burden could be generalizable beyond the context of the  pilot. For example, HHS’s plan for the consolidated FFR test model  indicates that it will be tested using grantees who receive awards from the  Administration for Children and Families (ACF), a subunit of HHS.
However, the plan does not specify how ACF will select participants or  how results from ACF grant recipients can be applied government-wide.  HHS officials told us that ACF has a list of potential participants. Given  the size and complexity of ACF’s grant recipients, the officials believed  that these participants would provide a good basis for scalability should  the FFR test model prove to be successful. According to HHS officials,  they have developed a comprehensive sampling plan for selecting  participants for each of the six test models. They will reach out to selected  participants to begin data collection in May 2016. We have recently been  provided with the draft sampling plan and will provide our assessment of it  in our forthcoming review on the implementation of the Section 5 Pilot.
Leading Practice 4: Develop a Plan to Evaluate Pilot Results. The design  for five of HHS’s six test models provides some level of detail on how it  plans to evaluate pilot results. For instance, HHS’s Learn Grants test  model provides a description of a methodology to measure knowledge  about the grants lifecycle. It will compare a group of recipients that has  access to certain grant resources contained in a public on-line portal to  another group of recipients that does not. HHS’s plans indicate that the  results from both tests will be analyzed to evaluate knowledge gained by  participants to draw conclusions about the effectiveness of the Learn  Grants tab on the Grants.gov website.
However, the documented pilot design lacks specific detail on how HHS  plans to analyze the data it gathered and how it will draw conclusions  about integrating the pilot activities into overall grant reporting efforts. For  example, both CDER Library test models reference an analysis plan for  evaluating to see if burden has been reduced. The plans do not indicate  how HHS would determine if a particular time threshold represents a true  reduction in burden and whether that burden is measured in minutes,  hours, or some other unit of analysis.
Similarly, the Single Audit and Notice of Award test models indicate that  HHS will use results from surveys and focus groups, including  documenting benefits and challenges raised by participants; yet HHS’s  plans for these two test models do not specify how HHS will compile  these results and distill them into actionable recommendations. HHS  officials told us that their revised planning documents are to include this  additional level of detail to address our concerns.
Leading Practice 5: Ensure Appropriate Two-Way Stakeholder  Communication. HHS has engaged in two-way stakeholder  communications for all six of its test models. It also has taken a number of  actions to obtain input from grant recipients including posting questions  on the National Dialogue to solicit feedback on how to ease grantee  reporting burden. Further, HHS has been involved in a number of  outreach activities including presentations at conferences, town hall  events, and webinars to identify areas of reporting burden and  duplication, and to collect ideas to streamline reporting. HHS also used  these forums to provide updates on the progress of the design and  specific information on the six test models. HHS supplemented input  received through the National Dialogue with feedback from SME to help  design the test models. An HHS official told us they identified SMEs  based on their experience working with federal grants, grant recipients,  and systems being tested.
HHS officials provided several examples of how they engaged in two-way  communication with stakeholders when developing their test models. For  example, HHS consulted with a federal official who used to work for  Grants.gov to help develop the Learn Grants test model and the pre- and  post-test evaluations associated with it. For the FFR test model, HHS  consulted with officials who work in ACF and the Payment Management  System. HHS also worked with other SMEs from across the federal  government to develop other test models. According to a HHS official,  SMEs were asked to critically assess the methodology for each of the  models with the intent of making each model more effective.
More recently, in January 2016, HHS pre-tested proposed Section 5 Pilot  test models and obtained feedback on ways to improve them with  advocacy groups representing those in the grants recipient communities  including state and local governments as well as research universities.  Also included were representatives from the auditing and software  development industries. HHS officials told us that they have made  significant revisions in response to the pre-tests and feedback to their  documented design.
However, HHS has additional opportunities to foster two-way dialogue  with recipients of federal funds. Officials from advocacy groups  representing federal funding recipients told us that they are still waiting for  information about how their membership can be more engaged in the pilot  process. For example, an official from the National Association of State  Auditors, Comptrollers, and Treasurers told us that following a webinar for  their membership hosted by the Association of Government Accountants  in November 2015 on the Section 5 Pilot, they collected the names of  more than 20 state and local government representatives who were  interested in participating in the grants portion of the pilot. This official  said the names were given to HHS, but the association has not received  any information on how these volunteers can participate in the pilot. HHS  officials said that once they receive OMB approval on their sampling  methodology for selecting participants, they will be able to reach out to  those who expressed interest in being a part of the pilot.
We provided our assessment of the design of the grants portion of the  pilot to HHS officials, who told us that they generally concurred with our  analysis and had updated their plan to address many of these concerns.  As noted above, we did not have time to review this update in this report  because we did not receive the plan in time. For details of our  assessment of the design of the six grants test models, see appendix II.
section_title: Design of the Procurement Portion of the Pilot Is Generally Not Meeting Leading Practices
paragraphs: Based on our review of the working draft plan for the procurement portion  of the pilot dated November 2015, related documents, and interviews with  cognizant staff, we found that the design did not reflect leading practices  for pilot design. Further, while the plan included some information  regarding responsibilities of stakeholders involved in the procurement  portion of the pilot, specific roles and deliverables were not clearly  described for all phases of the pilot. For example, the written draft plan  listed broad areas of responsibilities—such as “manage funding” or  “Federal Register Notice”—but did not detail what stakeholders would be  working on related to those activities.
OFPP staff described additional actions to supplement the information  contained in the draft plan. This information included their decision to  initially focus the design of the procurement pilot on testing the feasibility  of centralizing certified payroll reporting by contractors subject to the  Davis-Bacon and related acts because of public feedback on the need to  reduce duplicate reporting. However, even after taking this additional  information into account, we found that the design was neither well- developed nor documented in accordance with leading practices to allow  for the development of effective recommendations to simplify reporting for  contractors, as described below.
Leading Practice 1: Establish Well-Defined, Appropriate, Clear, and  Measurable Objectives. The working draft plan provided by OFPP does  not include specifics pertaining to the proposed focus of certified payroll  reporting. OFPP staff told us that they believe submitting certified payroll  information through a centralized portal would reduce contractor reporting  burden. They explained that this topic was selected because they learned  that it was a particular pain point for contractors as a result of various  outreach efforts including 18F’s discovery process. The draft plan also  does not provide specifics regarding the particular objectives and  hypothesis that will be tested by the pilot. OFPP staff stated that,  consistent with their view of agile practices, they intend to further refine  their approach as 18F develops its prototype and additional work  proceeds with the pilot.
Leading Practice 2: Clearly Articulate an Assessment Methodology. The  draft plan we reviewed did not include detailed information on the  methodology, strategy, or types of data planned to be collected. The draft  plan referenced an information-gathering effort conducted by GSA’s 18F  to discover challenges and develop recommendations for burden  reduction. However, OFPP staff could not provide any evidence that this  effort resulted in specific methodologies or data-collection strategies  related to centralizing certified payroll reporting.
According to 18F staff, a second phase of the procurement portion of the  pilot will begin in March 2016. OFPP staff said that during this phase, 18F  will research, design, and test a prototype that will become a basis for the  centralized portal that will be tested under the pilot. This prototype will be  vetted in workshops with stakeholders who will test, among other things,  the metrics, functionality, and accessibility of the prototype and any  needed changes. 18F expects the second phase to be completed by May  2016, after which OFPP will begin the third phase of the pilot later this  summer. In that phase, a contractor will develop a centralized portal  based on 18F’s design that could be used to test the submission and  review of certified payroll data.
Additionally, OFPP staff told us that they intended to collect data in  accordance with FAR requirements and would compare the information  collected in the portal with that being submitted through other methods.  However, OFPP was not able to provide specific details on its pilot  methodology, such as how it intends to compare results of contractors  that use the prototype and those that do not, identify the type and source  of data necessary to evaluate the pilot, and establish the timing and  frequency of the data to be collected. Without these details, the  procurement methodology design does not address all components of a  pilot program nor does it include key design features that would meet  leading practices.
Leading Practice 3: Ensure Scalability of Pilot Design. The draft design of  the procurement portion of the pilot that we reviewed did not address the  issue of scalability or efforts to ensure that conclusions and  recommendations resulting from the pilot could be applied government- wide. However, OFPP staff indicated that they plan to develop a sampling  approach that will allow them to collect data from a population that is  representative of federal contractors. Specifically, they said that they will  select a diverse group of participants by potentially pulling data from the  Federal Procurement Data System-Next Generation. Using that  database, they expect to be able to select a range of small and large  contractors that are required to report certified payroll under Davis-Bacon  and related acts. However, without documentation providing details of a  sampling methodology, measures, and a data analysis plan, the design  cannot ensure the scalability of the results or findings from the pilot.
Leading Practice 4: Develop Plan to Evaluate Pilot Results. The draft  procurement plan does not indicate how data will be evaluated to track  program performance, how final results will be evaluated, or conclusions  drawn. OFPP staff told us that although they believe it is early in the  process to have finalized evaluation plans, they are considering a number  of options for evaluating whether a centralized certified payroll portal  would cost more or less than current reporting approaches. Specifically,  they said that they expect to have some quantifiable data to allow for  straightforward analysis and will evaluate the qualitative data from the  certified payroll portal as well as the National Dialogue. However, the  absence of a detailed data analysis plan suggests that OFPP lacks a  sound approach to evaluate pilot results.
Leading Practice 5: Ensure Appropriate Two-Way Stakeholder  Communication. OFPP’s plans for obtaining stakeholder input and  fostering two-way dialogue have not yet been developed to engage public  participation and feedback on its approach for designing and  implementing the procurement portion of the pilot. Similar to the approach  taken by HHS, OFPP staff told us that they used comments posted on the  National Dialogue to inform the design of the procurement portion.  However, as previously mentioned, we have concerns about the  usefulness of that approach because none of the three comments they  received on the dialogue were related to certified payroll. OFPP staff said  they also used comments posted on the 2014 open dialogue on  improving procurement processes to inform their pilot design.
From commentary posted on both sites, OFPP identified certified payroll  reporting as a pain point that could be further explored through the pilot  project. OFPP staff told us that they engaged GSA’s 18F to conduct the  discovery phase of the pilot design to better understand areas of  significant reporting burden related to certified payroll with a select group  of stakeholders that included contractors, federal agency officials, and  contracting officers. A Federal Register notice was also issued on  November 24, 2015 to solicit public comments on the reporting burden of  the procurement portion of the pilot under the Paperwork Reduction Act.
Although OFPP obtained stakeholder input to identify areas of focus for  the design of the procurement portion of the pilot, it has not engaged  them to solicit input on other stages of the pilot, including design,  implementation, data gathering, and assessment. Further, OFPP has not  released specific information about the design of the pilot, nor has it made  information about pilot participation available to stakeholders despite  repeated requests for information from those participating in monthly calls  hosted by the Association for Government Accountants and Treasury.
In addition to being a leading practice for pilot design, our previous work  examining grants management streamlining initiatives found that  stakeholder communication is not just “pushing the message out,” but  should also facilitate a two-way, honest exchange and allow for feedback  from relevant stakeholders. We found that a lack of opportunities to  provide timely feedback resulted in poor implementation and prioritization  of streamlining initiatives and limited recipients’ use and understanding of  new systems. As such, it will be important for OFPP to engage with the  procurement community on its pilot design so that it can be improved  based on public input. In addition, more effective two-way  communications could also be a strategy for recruiting participants for the  procurement portion of the pilot.
section_title: Conclusions
paragraphs: In crafting the DATA Act, Congress sought to reduce the burden and cost  of reporting for the recipients of federal funds. Toward that end, OMB,  partnering with other federal agencies, has taken steps to design the  Section 5 Pilot that will explore potential ways to reduce the burden and  cost of reporting on federal funds for both the federal grantee and  procurement communities. However, we found uneven progress in the  grants and procurement portions of the pilot. OMB and HHS have made  considerable progress designing an overall approach that will examine a  variety of potential ways to simplify reporting for grant recipients. In  addition to generally being on track to meet the specific requirements set  out in the act, we found that the proposed design of the grants portion of  the pilot partially adheres to leading practices.
In contrast, our review of the design of the procurement portion of the  pilot raises several concerns. In the absence of a detailed design and risk  management plans for executing the pilot moving forward, it is unclear  how the design of the procurement portion will reflect the requirements  set forth by section 5 of the act. Because of project delays to date, it will  be especially important for OMB to communicate to Congress and  interested stakeholders how it plans to address key aspects of these  requirements, such as the collection of meaningful and useful data over a  12-month reporting cycle and including a diverse group of participants  with federal contracts totaling from $1 billion to $2 billion.
Moreover, the design we reviewed for the procurement portion of the pilot  did not reflect leading practices to allow for the development of effective  recommendations to simplify reporting for contractors. Moving forward,  given the tight timelines set out in the act, it will be important for OMB to  redouble its focus on the design and implementation of the procurement  portion. Without a sound design that applies leading practices, the  recommendations to Congress for reducing reporting burden for  contractors coming out of this effort may be late, of limited use, or  incomplete.
section_title: Recommendations for Executive Action
paragraphs: 1.  To help ensure and more clearly convey how the procurement  portion of the pilot will contribute to meeting the Section 5 Pilot  design requirements, we recommend that the Director of OMB  determine and clearly document (1) how it will collect certified  payroll data over a 12-month reporting cycle, (2) ensure the  diversity of pilot participants, and (3) how the inclusion of federal  contracts will contribute to an aggregate amount of $1 billion to $2  billion.  2.  To enable the development of effective recommendations for  reducing reporting burden for contractors, the Director of OMB  should ensure that the procurement portion of the pilot reflects  leading practices for pilot design.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to OMB, HHS, and GSA for review and  comment. OMB and HHS provided technical comments that we have  incorporated throughout the report, as appropriate. OMB and HHS did not  offer a view on our recommendations. GSA did not have any comments.
We are sending copies of this report to the Director of OMB, Secretary of  HHS, Administrator of GSA, and appropriate congressional addressees.  In addition, the report is available at no charge on the GAO website at  https://www.gao.gov.
If you or your staff have any questions about this report, please contact  me on (202) 512-6806 or by email at sagerm@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This review (1) describes the administration’s approach to the Section 5  Pilot; (2) assesses whether current activities and plans will likely allow the  Office of Management and Budget (OMB) and its partners to meet  requirements and time frames established under the Section 5 Pilot; and  (3) evaluates the extent to which the design for the pilot is consistent with  leading practices.
To describe the administration’s approach to the pilot, we assessed  documents related to pilot activities and interviewed OMB, Department of  Health and Human Services (HHS), and General Services Administration  (GSA) officials and staff responsible for implementing the Section 5 Pilot.  Specifically, we reviewed documentation from HHS and OMB’s Office of  Federal Procurement Policy (OFPP). Our reviews were based on the  latest design plans available at the time. We also interviewed officials  from organizations representing key non-federal stakeholders including  state and local governments, private-sector contractors, and other federal  fund recipients.
To assess whether the Section 5 Pilot design would be likely to meet the  statutory design requirements, we reviewed section 5 of the Federal  Funding Accountability and Transparency Act of 2006, as added by the  Digital Accountability and Transparency Act of 2014 (DATA Act) to  understand the deadlines and design requirements. We reviewed the  draft design documents to assess OMB and its partners’ plans for  meeting these requirements. To supplement our review of those plans,  we also spoke with cognizant staff implementing these pilots at OMB,  HHS, and GSA.
To identify and analyze leading practices for pilot design, we reviewed our  past work evaluating and assessing pilots. Additionally, we also relied on  our technical guidance on designing evaluations. Further, we reviewed  relevant studies from academia as well as other entities, such as the  Brookings Institution and the Federal Demonstration Partnership. We  reviewed reports from organizations that have expertise on conducting  pilot programs and experience in scaling pilot results that could be  applied government-wide. We also shared these leading practices with  the agencies in this review during our audit work.
To assess the extent to which the Section 5 Pilot design adhered to these  leading practices, we reviewed documented designs and plans for both  the grants and procurement portions of the pilot. To evaluate the grants  portion of the pilot, we focused on a draft design document from  November 2015. HHS officials told us that they have updated that plan.  Because we did not receive this update until the end of March 2016, we  did not have time to include its content for this report. As such, our  assessment is based on the November 2015 plan. We intend to review  the updated plan as we continue our work on DATA Act implementation.  We have supplemented our assessment with information HHS officials  provided to us during subsequent interviews, as appropriate. For the  procurement portion, we reviewed a working draft plan from November  2015. While it is unclear whether there has been an updated version, we  have also provided additional details from discussions with OFPP  officials, as appropriate.
To evaluate the grants and procurement portions of the pilot, we applied  the five leading practices we identified to OMB and HHS’s design  documents. Each of those assessments were subsequently verified by  another individual. We determined that the design met the criteria when  we saw evidence that all aspects of a leading practice were met. When  we were unable to assess whether all aspects of a leading practice were  met without additional information, we determined that the design partially  met the criteria. Finally, when we saw no evidence of a leading practice,  we determined that the criteria was not met. In continuation of our  constructive engagement approach on the DATA Act for working with  agencies implementing the act, we provided HHS and OMB with  feedback on the design of the grants and procurement portions of the  pilot during our review. These officials generally accepted our feedback  and, in some instances, noted that they have or would make changes to  their design as a result of our input.
We conducted this performance audit from May 2015 to April 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Detailed Results of the Assessment of Design of the Grants Portion of the Section 5 Pilot Compared to Leading Practices
paragraphs: This appendix provides detailed information regarding our assessment of  the pilot design for the grants portion of the Section 5 Pilot. We assessed  each of the Department of Health and Human Services’s (HHS) six test  models against the five leading practices for pilot design described in the  report. Using HHS’s November 2015 design plans and relevant  supporting information available during the preparation of this report, we  determined whether each test model met, partially met, or did not meet  those leading practices.
section_title: Appendix III: Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, J. Christopher Mihm (Managing  Director), Peter Del Toro (Assistant Director), Shirley Hwang (analyst-in- charge), Aaron Colsher, Kathleen Drennan, Jason Lyuke, Kiran  Sreepada, and David Watsula made major contributions to this report.  Other key contributors include Lisette Baylor, Brandon Booth, Jenny  Chanley, Robert Gebhart, Donna Miller, Carl Ramirez, Andrew J.  Stephens, and Tatiana Winger. Additional members of GAO’s DATA Act  Internal Working Group also contributed to the development of this report.