Data Act: Section 5 Pilot Design Issues Need to Be Addressed to Meet Goal of Reducing Recipient Reporting Burden
Released: 2016-04-19T00:00:00

Letter

Background

The DATA Act became law in May 2014 and holds considerable promise for shedding more light on how federal funds are spent. To improve the transparency and quality of the federal spending data made available to the public, the DATA Act directed OMB and Treasury to establish government-wide data standards that include common data elements for reporting financial and payment information by May 2015. Under the act, federal agencies must begin reporting financial spending data using these standards by May 2017 and publicly post spending data in a machine- readable format by May 2018.

Section 5 Pilot Requirements for Pilot to Simplify Federal Award Reporting

The DATA Act also requires that OMB, or an agency it designates, establish a pilot program to facilitate the development of recommendations to (1) standardize reporting elements across the federal government, (2) eliminate unnecessary duplication in financial reporting, and (3) reduce compliance costs for recipients of federal awards. The act established reporting requirements and timeframes for implementation of the pilot. See figure 1 for a timeline of these deadlines.

The DATA Act also sets specific requirements related to the pilot’s design. First, the pilot must collect data during a 12-month reporting cycle. The pilot must also include a diverse group of recipients such as awardees receiving a range of awards as long as the total value of the awards falls within the statutory range. To the extent practicable, the pilot is to include recipients who receive federal awards from multiple programs across multiple agencies. Finally, the pilot must include a combination of federal contracts, grants, and subawards with an aggregate value between $1 billion and $2 billion.

In addition, OMB must review the information recipients are required to report to identify common reporting elements across the federal government, unnecessary duplication in financial reporting, and unnecessarily burdensome reporting requirements for recipients of federal awards. This review is to be done in consultation with relevant federal agencies and recipients of federal awards, including state and local governments and institutions of higher education.

Leading Practices for Effective Pilot Design

A well-developed and documented pilot program can help ensure that agency assessments produce information needed to make effective program and policy decisions. Such a process enhances the quality, credibility, and usefulness of evaluations in addition to helping to ensure that time and resources are used effectively. We have identified five leading practices that, taken together, form a framework for effective pilot design. To identify these practices, we reviewed our prior work as well as academic literature related to the design of pilot and evaluation programs. By following these leading practices, agencies can promote a consistent and effective pilot design process. We shared these practices with OMB, HHS, and GSA staff, who found them to be reasonable and appropriate, and applicable to the Section 5 Pilot. 1. Establish well-defined, appropriate, clear, and measurable objectives.

Such objectives should have specific statements of the accomplishments necessary to meet the objectives. Clear and measurable objectives can help ensure that appropriate evaluation data are collected from the outset of pilot implementation so that data will subsequently be available to measure performance against the objectives. Broad study objectives should be translated into specific, researchable questions that articulate what will be assessed. 2. Clearly articulate assessment methodology and data gathering strategy that addresses all components of the pilot program and includes key features of a sound plan.

Key features of a clearly articulated methodology include a strategy for comparing the pilot implementation and results with other efforts, a clear plan that details the type and source of the data necessary to evaluate the pilot, and methods for data collection including the timing and frequency. 3. Identify criteria or standards for identifying lessons about the pilot to inform decisions about scalability and whether, how, and when to integrate pilot activities into overall efforts.

The purpose of a pilot is generally to inform a decision on whether and how to implement a new approach in a broader context. Therefore, it is critically important to consider how well the lessons learned from the pilot can be applied in other, broader settings. To assess scalability, criteria should relate to the similarity or comparability of the pilot to the range of circumstances and population expected in full implementation. The criteria or standards can be based on lessons from past experiences or other related efforts known to influence implementation and performance as well as on literature reviews and stakeholder input, among other sources. The criteria and standards should be observable and measureable events, actions, or characteristics that provide evidence that the pilot objectives have been met. Choosing well-regarded criteria against which to make comparisons can lead to strong, defensible conclusions. 4. Develop a detailed data-analysis plan to track the pilot program’s implementation and performance and evaluate the final results of the project and draw conclusions on whether, how, and when to integrate pilot activities into overall efforts.

A detailed data-analysis plan identifies who will do the analysis as well as when and how data will be analyzed to measure the pilot program’s implementation and performance. The results will show the successes and challenges of the pilot, and in turn, how the pilot can be incorporated into broader efforts. Some elements of a detailed data-analysis plan include talking to users, managers, and developers; evaluating the lessons learned to improve procedures moving forward; and other appropriate measures. 5. Ensure appropriate two-way stakeholder communication and input at all stages of the pilot project, including design, implementation, data gathering, and assessment Appropriate two-way stakeholder communication and input should occur at all stages of the pilot, including design, implementation, data gathering, and assessment. Failure to effectively engage with stakeholders, and understand and address their views can undermine or derail an initiative. To that end, it is critical that agencies identify who the relevant stakeholders are, and communicate early and often to address their concerns and convey the initiative’s overarching benefits.

OMB Is Conducting a Pilot Focused on Reducing Grantee and Contractor Reporting Burden to Meet Its Requirements under Section 5

OMB has established a Section 5 Pilot with two primary focus areas—one on federal grants and another on federal contracts (procurement). OMB’s Office of Federal Financial Management is responsible for the grants portion of the pilot and has designated the Department of Health and Human Services (HHS) to serve as its executing agent. On the contracting side, OMB’s OFPP is responsible for leading the procurement portion and is working with various entities including 18F and the Chief Acquisitions Officers’ Council (CAOC). Specifically, 18F is designing the system to be tested as part of the pilot. GSA’s Office of Government-wide Policy is responsible for providing federal register notices; and its Integrated Award Environment provides guidance and technical considerations. OMB launched a number of pilot-related initiatives in May 2015 and expects to continue activities until at least May 2017.

Grants Portion of the Pilot Will Test Six Ways to Reduce Grantee Reporting Burden

As the executing agent for the grants portion of the pilot, HHS has developed six “test models” that will evaluate different approaches to potentially reducing grantee reporting burden. These six models are the specific grants tools, forms, or processes that will be tested and analyzed under the pilot to determine if adopting these changes will actually contribute to the program’s objectives of reducing reporting burden, duplication, and compliance costs.

Taken as a whole, the six test models examine a variety of grant reporting issues that HHS has identified as presenting challenges. HHS officials told us that they have received comments through the National Dialogue, a website for grant recipients and contractors to discuss issues including compliance costs, reporting burden, eliminating duplication, and standardizing processes. In addition, the officials obtained feedback on areas of concern from grantees involved in earlier HHS efforts to streamline grants reporting. They used that information to inform the development of the six test models.

Officials from advocacy groups representing grant recipients and federal contractors told us that they initially expected the grants portion of the pilot to be an extension of the Grants Reporting Information Project (GRIP) proof of concept that was launched following the enactment of the American Recovery and Reinvestment Act of 2009 rather than the six test models. HHS officials told us they would have liked to more fully replicate the GRIP, however, that would have required broader participation from agencies than was available for the Section 5 Pilot.

The following provides high-level summaries of each of the six test models. For additional details, see appendix II.

HHS intends to assess whether an online and searchable repository for data standards will facilitate grant reporting. To do this, HHS developed the Common Data Element Repository (CDER) Library, which is intended to be an authorized source for data elements and definitions for use by the federal government and recipients reporting grant information.

The CDER Library is also intended to encourage the use of common definitions for grants-related terms by nonfederal stakeholders and federal agencies. As of March 2016, the publicly-available version of the CDER Library contained 112 data elements from a variety of sources, including the Federal Acquisition Regulation (FAR), OMB Circular A-11, and the Uniform Grant Guidance. It also included several data elements standardized in accordance with DATA Act requirements.

HHS has developed a version of the CDER Library, accessible only to federal agencies, that contains a much more detailed database of more than 9,000 elements. This federal-agency-only version of the CDER Library also identifies which grant reporting forms these data elements come from so that users can see how many forms require the same data element and which agencies request that information from grantees. HHS officials told us that they believe the CDER Library has the potential to be a powerful tool for streamlining definitions and forms.

HHS intends to test whether it will be possible to use a consolidated Federal Financial Report (FFR) to allow grantees to submit multiple reporting forms into one system. The FFR, reported on the Standard Form 425, is used for reporting grants expenditures for the recipients of federal assistance. HHS believes that a consolidated FFR will allow participants to submit complete information once instead of through multiple entry points. A consolidated FFR could provide a single point of data entry, earlier validation of FFR data, and potential future streamlining of the grants close-out process.

According to HHS officials, this test model is intended to be a continuation of the GRIP launched during the American Recovery and Reinvestment Act of 2009. The aim of that effort was to determine the feasibility of developing a centralized government-wide collection system for federal agencies and recipients of federal awards.

HHS is examining ways to reduce duplicate and redundant information contained in Single Audit forms. The Single Audit Act requires states, local governments, and nonprofit organizations expending $750,000 or more in federal awards in a year to obtain an audit in accordance with the requirements set forth in the act. HHS intends to test whether some grant forms related to the single audit could be combined.

HHS plans to examine whether a consolidated Notice of Award coversheet might reduce reporting burden by allowing grant recipients to locate required reporting data in one place, rather than attempting to find information on coversheets that differ by agency.

HHS added a new section to the Grants.gov website, called Learn Grants, intended to make it easier for stakeholders to find, learn about, and apply for federal grants. The Learn Grants website provides links to grant policies, processes, funding, and other grant lifecycle information. HHS officials said they want to use this test model to determine whether the Learn Grants site could effectively engage stakeholders and provide training early in the grants lifecycle process that, in turn, would have a positive effect on recipient compliance during post-award activities.

Procurement Portion of the Pilot Will Focus on Centralizing Certified Payroll Reporting

The procurement portion of the pilot will be focused on examining the feasibility of centralizing the reporting of certain required information. Depending on the contract, there may be many types of information contractors must report. OFPP staff told us the pilot will initially focus on the reporting of certified payroll. This is one specific FAR requirement only applicable to contracts for construction within the United States.

Specifically, OFPP has identified opportunities to improve upon the current unstandardized reporting format under which some employers report data electronically while others use manual paper processes. Further, OFPP intends to identify which data elements would be included in reporting, the method of data transmission, and other related details.

This narrow approach stands in contrast to the grants portion of the pilot where HHS has a broader, more comprehensive plan to explore several areas where grantee reporting burden might be reduced. OFPP staff explained that its decision to focus on certified payroll reporting arose out of feedback from the procurement community. They also noted that the Section 5 Pilot is one of a number of government-wide initiatives to reduce contractor burden and streamline procurement processes, such as GSA’s Integrated Award Environment initiative to integrate acquisition systems into one streamlined environment.

To better understand the issue of certified payroll reporting and its potential suitability as a subject for the procurement portion of the Section 5 Pilot, the CAOC engaged GSA’s 18F through an interagency agreement to interview contractors, contracting officers, business owners, government employees, and subject-matter experts (SME). As a result of that effort, 18F identified major categories of burdens and constraints related to certified payroll reporting and potential recommendations on how to address them.

OFPP staff said they once again worked with 18F in winter 2016 to gather requirements for building a prototype system to centralize the reporting of certified payroll data. The 18F staff we spoke with noted that they will build a prototype to explore potential solutions for reducing contractor burden through user research and testing. OFPP staff will develop and evaluate metrics for the pilot. OFPP intends to test the system in summer 2016.

National Dialogue Sought Feedback from Grantees and Contractors on Ways to Simplify Federal Award Reporting

In May 2015, OMB, CAOC, GSA, and HHS launched the National Dialogue, a website for grant recipients and federal contractors to discuss issues including compliance costs, reporting burden, eliminating duplication, and standardizing processes. OMB staff told us that they used the National Dialogue as a feedback mechanism for the grants and procurement portions of the pilot. This was one of the first publicly announced pilot-related activities. The website will accept comments through May 2017. OMB and GSA staff told us that they plan to actively review and address the input they receive.

The website is intended to be a useful tool for obtaining information about issues of concern to their respective communities. Discussions related to grantee reporting have been significantly more active than those focused on procurement. Although the comments vary widely in topic, there are a number of substantive suggestions for how grantee reporting burdens can be reduced. While HHS officials told us that the dialogue was intentionally designed so that feedback would be submitted anonymously, some commenters have self-identified the institution they represent, including the Council on Governmental Relations, Association on American Universities, Association of Public and Land-grant Universities, and Coalition for Government Procurement.

The Grants Portion of Section 5 Pilot Is Generally On Track to Meet DATA Act Requirements but the Procurement Portion Is Not

The Design of the Grants Portion of the Pilot Is Generally On Track to Meet DATA Act Requirements

If HHS effectively implements its stated plans for the grants portion of the Section 5 Pilot, it is likely that the grants portion of the pilot will comply with the act. These requirements call for the grants portion’s design to include the following: DATA Act Requirement 1: Collect data during a 12-month reporting cycle. HHS’s November 2015 design documentation shows that it will begin collecting data for these six test models by May 2016. This would allow for data to be collected on these test models during a 12-month reporting cycle before May 2017, when the pilot is required to terminate. We believe these timeframes should provide sufficient time for HHS to incorporate public comments by May 2016 and allow for a full 12-month data collection cycle.

DATA Act Requirement 2: Include a diverse group of federal award recipients and, to the extent practicable, recipients who receive federal awards from multiple programs across multiple agencies. HHS officials told us that they have developed a detailed plan to select participants, which will include state and local governments, universities, and other types of grant recipients. HHS officials explained that the grants portion of the pilot will include recipients who received a range of federal funding amounts and will not be limited to one agency or grant program.

HHS officials initially told us that they could not provide us with the revised plan because it was still under review by OMB. We did receive a copy of the revised plan at the end of March 2016, but because of the timing we were unable to fully review it in time for the release of this report. We will provide our assessment of the plan as part of future work as we continue to monitor the design and implementation of the Section 5 Pilot.

DATA Act Requirement 3: Include a combination of federal contracts, grants, and subawards, with an aggregate value of not less than $1 billion but not more than $2 billion. HHS officials told us that they are still determining how to meet the requirement for total award value because they want to ensure the pool of pilot participants is as diverse and large as possible while still being legally compliant. Specifically, one of their selection considerations is the award value of grants received by awardees. Further, HHS officials have explored strategies to ensure that they do not exceed the maximum dollar amount threshold. HHS officials told us that they expect to make decisions related to how to meet this requirement in early 2016.

The Design of the Procurement Portion May Not Contribute to Pilot’s Ability to Meet DATA Act Requirements

We have concerns about the extent that the design of the procurement portion of the pilot reflects the requirements specified in the DATA Act. OFPP’s plans to address those statutory design requirements discussed below reflect the status of the procurement portion of the pilot described by OFPP staff and related documents we reviewed.

DATA Act Requirement 1: Collect data during a 12-month reporting cycle. The design of the procurement portion of the pilot is at risk of not including data collected during a 12-month reporting cycle in a meaningful way. To meet this requirement, OFPP and GSA would need to begin collecting data no later than May 9, 2016.

When we spoke with OFPP staff, they stated that by launching the National Dialogue in May 2015, they believe they will have met the act’s requirements that data collection take place during a 12-month reporting cycle. Further, staff also considered comments received from other efforts including the Open Dialogue on Improving How to Do Business with the Federal Government conducted in 2014 to meet this requirement. However, neither of these dialogues included comments that specifically mentioned the issue of certified payroll. As a result, we do not believe those comments provide meaningful and relevant data on the effectiveness of a centralized portal for certified payroll reporting.

As a result of design and development delays, OFPP will not be able to collect meaningful and useful data for the procurement portion of the pilot until summer 2016, when it expects to complete the development of a centralized portal through which participants will submit certified payroll data. OFPP started exploring ways to streamline certified payroll reporting in spring 2015.

OFPP said that due to staffing challenges, work on designing a prototype for a system to be tested under the pilot did not begin until late February 2016. At that time, the CAOC signed an agreement with GSA’s 18F to begin what it expected to be a 10-week design period. Cognizant staff expect this design work will take place between March and May 2016. However, a contractor cannot begin building an actual “production” version of the system to be tested under the pilot until 18F designs the prototype, which is expected to be completed by the beginning of May 2016. Therefore, this leaves at most a few weeks to develop the centralized reporting portal before May 9, 2016—the date which the pilot must begin for meaningful and useful data to be collected in a full 12- month period. OFPP staff told us that they do not intend to begin testing a centralized reporting portal until late summer 2016.

According to OFPP and GSA staff, they were faced with delays due to bid protests related to the contracting mechanism GSA intends to use to select a contractor to build the portal to be tested under the pilot. However, as of March 2016, these bid protests have been resolved and no longer present a barrier in awarding the contract. While we agree that these protests could pose a barrier to awarding the contract to develop the testing portal, we do not believe that OFPP needed to wait until they were resolved before moving forward with 18F’s development of a prototype for the portal.

Given the resolution of these bid protests, OFPP staff said that they are working with 18F to assess the feasibility of expediting project timelines to launch the prototype sooner than expected so that they could potentially collect 10 months of data through the certified payroll reporting portal. Given the weekly or bi-weekly reporting of certified payroll, this approach may result in a sufficient amount of meaningful and useful data on which OFPP can base conclusions related to its hypothesis. However, it is important that OFPP clearly conveys and documents its rationale for how its approach will contribute to the collection of meaningful and useful data consistent with the timeframes established under the act.

DATA Act Requirement 2: Include a diverse group of federal award recipients and, to the extent practicable, recipients who receive federal awards from multiple programs across multiple agencies. OFPP and GSA do not yet have a detailed plan for selecting participants that will result in a diverse group of recipients with awards from multiple programs and agencies. However, there is some documentation related to OFPP’s approach for selecting participants in the project plan and in a Federal Register notice issued on November 24, 2015. For example, the draft plan identifies the Federal Procurement Data System-Next Generation as the mechanism that will be used for identifying which contracts and contractors to include in the pilot. OFPP staff also told us that they intend to cover both large and small industries. While valuable information, these documents do not clearly convey how the procurement portion of the pilot would specifically contribute to meeting the act’s requirement regarding diversity of participants.

OFPP staff told us that for the purposes of meeting the pilot requirements they consider any individual or group that provided information to the National Dialogue to be a participant in the pilot. However, as previously mentioned, individuals and groups that have commented on the National Dialogue did not provide any comments related to certified payroll. Therefore, it is unclear how they could be considered pilot participants.

Additionally, OFPP staff were unable to tell us how they plan to count commenters that are not contract awardees, but instead are organizations representing groups of federal contractors. It is unclear how OFPP can ensure the universe of commenters is diverse because it does not control who comments on the dialogue.

OFPP staff stated that they also intend to select participants for testing their prototype system using a nongeneralizable sample of contractor data reported through the Federal Procurement Data System-Next Generation. However, they did not provide us with specific information on how they would ensure that the sample met all requirements under the act, nor did they provide a detailed, documented sampling plan equivalent to the grants portion of the pilot. As a result, it will be important for OFPP to clearly document its rationale for how its approach will allow for the inclusion of a diverse group of federal contractors, as required by the act.

DATA Act Requirement 3: Include a combination of federal contracts, grants, and subawards, with an aggregate value of not less than $1 billion but not more than $2 billion. OFPP staff told us OMB could meet this dollar range requirement through the grants and procurement portions of the pilot collectively. Under such an approach, it would be important for each portion of the pilot to know how much it is contributing to meet the required award range. Our understanding of the grants portion of the pilot suggests that that it has a plan for doing this. Less apparent are the specifics of how the procurement portion of the pilot would do so.

Design of Grants Portion of the Pilot Partially Adheres to Leading Practices While Design of the Procurement Portion Does Not

We assessed the designs of the grants and procurement portions of the pilot against leading practices that we identified from our prior work and other sources. In continuation of our constructive engagement approach for working with agencies implementing the DATA Act, we shared the results of our analysis with HHS and OFPP staff who told us that they will consider our input as they continue to update and revise their plans.

Design of the Grants Portion of the Pilot Partially Reflects Leading Practices

HHS’s November 2015 design for the grants portion of the pilot generally applied leading practices. As noted above, while we have received a revised plan for the design of the grants portion, we were unable to fully review it in time for the release of this report. We will provide our assessment of that plan in a forthcoming review that will focus on the pilot’s implementation.

DATA Act Grants Test Models Under the Office of Management and Budget’s (OMB) direction, the Department of Health and Human Services (HHS) intends to develop recommendations for reducing grantee reporting burden by testing different areas. HHS will develop and test:

An online repository for data elements and definitions that is intended to be an authoritative source for data elements and definitions, called the Common Data Element Repository (CDER) Library.

A federal agency-only version of the CDER Library containing more than 9,000 grants data elements that identify which specific grant forms these data elements come from, so that users can see how many forms require the same data element and which agencies request that information.

Leading Practice 1: Establish Well-Defined, Appropriate, Clear, and Measurable Objectives. Each of the six grants test models at least partially met the leading practice that pilots have well-defined, appropriate, clear, and measurable objectives. For example, one of the Single Audit test models has the clearly defined objective of testing whether two forms containing duplicative information can be combined to reduce recipient reporting burden. This objective is measurable and appropriately linked to the purposes of the Section 5 Pilot overall, which include eliminating unnecessary duplication in financial reporting and reducing compliance costs for recipients of federal awards. In another example, one of the CDER Library test models has a clearly established objective of determining whether access to an authoritative source for common data element definitions would help grant recipients complete necessary forms accurately and in a timely manner. The CDER Library test model also identifies specific metrics that would allow them to measure whether they are able to achieve its stated objectives.

In our initial review of these test models, we provided feedback to HHS that the other CDER Library test did not have a clear, fully established objective. In response, HHS officials explained that the objective of that test model is to compare data elements and forms used across the federal government with the goal of consolidating these forms and ultimately passing on reporting efficiencies to grant recipients.

Leading Practice 2: Clearly Articulate an Assessment Methodology. Five of the six test models did not clearly articulate an assessment methodology. In contrast, for the Learn Grants test model, HHS described how it planned to use webinars, conference presentations, and other events to increase awareness inside and outside of government about the grants-related resources available on Grants.gov. The plan also includes a detailed timeline for executing the test model, as well as HHS’s methodology for conducting pre- and post-tests of pilot participants. HHS officials told us that they worked with a federal SME with previous experience working on Grants.gov to help develop and refine the assessment methodology.

The remaining five test models have less clearly articulated assessment methodologies. For example, for the consolidated FFR test model, HHS said it will survey grant recipients on their experiences when submitting their reports into one system rather than multiple entry points; but we found that the plans lacked detail about how surveys will be designed and administered. In addition, the plan did not provide specific information about the participants HHS intends to survey, nor did it provide details regarding how HHS will compare survey results for recipients in the pilot versus those not participating in the pilot.

In meetings with senior HHS officials, we raised these and similar concerns about the Notice of Award test model and one of the CDER Library models. For the other CDER Library test model, we found that HHS’s plans did not identify the data sources or metrics that would be used in the assessment methodology. In those feedback meetings, HHS officials said many of the concerns have been addressed in the revised plan.

Leading Practice 3: Ensure Scalability of Pilot Design. HHS documented an overall structure for how each test model is integrated into the overall grants portion of the pilot. However, the documented design lacks specific details about how HHS intends to evaluate the performance of each test model to inform decisions about scalability. Specifically, five of the six test models include either no or few specifics about how any observed reduction in burden could be generalizable beyond the context of the pilot. For example, HHS’s plan for the consolidated FFR test model indicates that it will be tested using grantees who receive awards from the Administration for Children and Families (ACF), a subunit of HHS.

However, the plan does not specify how ACF will select participants or how results from ACF grant recipients can be applied government-wide. HHS officials told us that ACF has a list of potential participants. Given the size and complexity of ACF’s grant recipients, the officials believed that these participants would provide a good basis for scalability should the FFR test model prove to be successful. According to HHS officials, they have developed a comprehensive sampling plan for selecting participants for each of the six test models. They will reach out to selected participants to begin data collection in May 2016. We have recently been provided with the draft sampling plan and will provide our assessment of it in our forthcoming review on the implementation of the Section 5 Pilot.

Leading Practice 4: Develop a Plan to Evaluate Pilot Results. The design for five of HHS’s six test models provides some level of detail on how it plans to evaluate pilot results. For instance, HHS’s Learn Grants test model provides a description of a methodology to measure knowledge about the grants lifecycle. It will compare a group of recipients that has access to certain grant resources contained in a public on-line portal to another group of recipients that does not. HHS’s plans indicate that the results from both tests will be analyzed to evaluate knowledge gained by participants to draw conclusions about the effectiveness of the Learn Grants tab on the Grants.gov website.

However, the documented pilot design lacks specific detail on how HHS plans to analyze the data it gathered and how it will draw conclusions about integrating the pilot activities into overall grant reporting efforts. For example, both CDER Library test models reference an analysis plan for evaluating to see if burden has been reduced. The plans do not indicate how HHS would determine if a particular time threshold represents a true reduction in burden and whether that burden is measured in minutes, hours, or some other unit of analysis.

Similarly, the Single Audit and Notice of Award test models indicate that HHS will use results from surveys and focus groups, including documenting benefits and challenges raised by participants; yet HHS’s plans for these two test models do not specify how HHS will compile these results and distill them into actionable recommendations. HHS officials told us that their revised planning documents are to include this additional level of detail to address our concerns.

Leading Practice 5: Ensure Appropriate Two-Way Stakeholder Communication. HHS has engaged in two-way stakeholder communications for all six of its test models. It also has taken a number of actions to obtain input from grant recipients including posting questions on the National Dialogue to solicit feedback on how to ease grantee reporting burden. Further, HHS has been involved in a number of outreach activities including presentations at conferences, town hall events, and webinars to identify areas of reporting burden and duplication, and to collect ideas to streamline reporting. HHS also used these forums to provide updates on the progress of the design and specific information on the six test models. HHS supplemented input received through the National Dialogue with feedback from SME to help design the test models. An HHS official told us they identified SMEs based on their experience working with federal grants, grant recipients, and systems being tested.

HHS officials provided several examples of how they engaged in two-way communication with stakeholders when developing their test models. For example, HHS consulted with a federal official who used to work for Grants.gov to help develop the Learn Grants test model and the pre- and post-test evaluations associated with it. For the FFR test model, HHS consulted with officials who work in ACF and the Payment Management System. HHS also worked with other SMEs from across the federal government to develop other test models. According to a HHS official, SMEs were asked to critically assess the methodology for each of the models with the intent of making each model more effective.

More recently, in January 2016, HHS pre-tested proposed Section 5 Pilot test models and obtained feedback on ways to improve them with advocacy groups representing those in the grants recipient communities including state and local governments as well as research universities. Also included were representatives from the auditing and software development industries. HHS officials told us that they have made significant revisions in response to the pre-tests and feedback to their documented design.

However, HHS has additional opportunities to foster two-way dialogue with recipients of federal funds. Officials from advocacy groups representing federal funding recipients told us that they are still waiting for information about how their membership can be more engaged in the pilot process. For example, an official from the National Association of State Auditors, Comptrollers, and Treasurers told us that following a webinar for their membership hosted by the Association of Government Accountants in November 2015 on the Section 5 Pilot, they collected the names of more than 20 state and local government representatives who were interested in participating in the grants portion of the pilot. This official said the names were given to HHS, but the association has not received any information on how these volunteers can participate in the pilot. HHS officials said that once they receive OMB approval on their sampling methodology for selecting participants, they will be able to reach out to those who expressed interest in being a part of the pilot.

We provided our assessment of the design of the grants portion of the pilot to HHS officials, who told us that they generally concurred with our analysis and had updated their plan to address many of these concerns. As noted above, we did not have time to review this update in this report because we did not receive the plan in time. For details of our assessment of the design of the six grants test models, see appendix II.

Design of the Procurement Portion of the Pilot Is Generally Not Meeting Leading Practices

Based on our review of the working draft plan for the procurement portion of the pilot dated November 2015, related documents, and interviews with cognizant staff, we found that the design did not reflect leading practices for pilot design. Further, while the plan included some information regarding responsibilities of stakeholders involved in the procurement portion of the pilot, specific roles and deliverables were not clearly described for all phases of the pilot. For example, the written draft plan listed broad areas of responsibilities—such as “manage funding” or “Federal Register Notice”—but did not detail what stakeholders would be working on related to those activities.

OFPP staff described additional actions to supplement the information contained in the draft plan. This information included their decision to initially focus the design of the procurement pilot on testing the feasibility of centralizing certified payroll reporting by contractors subject to the Davis-Bacon and related acts because of public feedback on the need to reduce duplicate reporting. However, even after taking this additional information into account, we found that the design was neither well- developed nor documented in accordance with leading practices to allow for the development of effective recommendations to simplify reporting for contractors, as described below.

Leading Practice 1: Establish Well-Defined, Appropriate, Clear, and Measurable Objectives. The working draft plan provided by OFPP does not include specifics pertaining to the proposed focus of certified payroll reporting. OFPP staff told us that they believe submitting certified payroll information through a centralized portal would reduce contractor reporting burden. They explained that this topic was selected because they learned that it was a particular pain point for contractors as a result of various outreach efforts including 18F’s discovery process. The draft plan also does not provide specifics regarding the particular objectives and hypothesis that will be tested by the pilot. OFPP staff stated that, consistent with their view of agile practices, they intend to further refine their approach as 18F develops its prototype and additional work proceeds with the pilot.

Leading Practice 2: Clearly Articulate an Assessment Methodology. The draft plan we reviewed did not include detailed information on the methodology, strategy, or types of data planned to be collected. The draft plan referenced an information-gathering effort conducted by GSA’s 18F to discover challenges and develop recommendations for burden reduction. However, OFPP staff could not provide any evidence that this effort resulted in specific methodologies or data-collection strategies related to centralizing certified payroll reporting.

According to 18F staff, a second phase of the procurement portion of the pilot will begin in March 2016. OFPP staff said that during this phase, 18F will research, design, and test a prototype that will become a basis for the centralized portal that will be tested under the pilot. This prototype will be vetted in workshops with stakeholders who will test, among other things, the metrics, functionality, and accessibility of the prototype and any needed changes. 18F expects the second phase to be completed by May 2016, after which OFPP will begin the third phase of the pilot later this summer. In that phase, a contractor will develop a centralized portal based on 18F’s design that could be used to test the submission and review of certified payroll data.

Additionally, OFPP staff told us that they intended to collect data in accordance with FAR requirements and would compare the information collected in the portal with that being submitted through other methods. However, OFPP was not able to provide specific details on its pilot methodology, such as how it intends to compare results of contractors that use the prototype and those that do not, identify the type and source of data necessary to evaluate the pilot, and establish the timing and frequency of the data to be collected. Without these details, the procurement methodology design does not address all components of a pilot program nor does it include key design features that would meet leading practices.

Leading Practice 3: Ensure Scalability of Pilot Design. The draft design of the procurement portion of the pilot that we reviewed did not address the issue of scalability or efforts to ensure that conclusions and recommendations resulting from the pilot could be applied government- wide. However, OFPP staff indicated that they plan to develop a sampling approach that will allow them to collect data from a population that is representative of federal contractors. Specifically, they said that they will select a diverse group of participants by potentially pulling data from the Federal Procurement Data System-Next Generation. Using that database, they expect to be able to select a range of small and large contractors that are required to report certified payroll under Davis-Bacon and related acts. However, without documentation providing details of a sampling methodology, measures, and a data analysis plan, the design cannot ensure the scalability of the results or findings from the pilot.

Leading Practice 4: Develop Plan to Evaluate Pilot Results. The draft procurement plan does not indicate how data will be evaluated to track program performance, how final results will be evaluated, or conclusions drawn. OFPP staff told us that although they believe it is early in the process to have finalized evaluation plans, they are considering a number of options for evaluating whether a centralized certified payroll portal would cost more or less than current reporting approaches. Specifically, they said that they expect to have some quantifiable data to allow for straightforward analysis and will evaluate the qualitative data from the certified payroll portal as well as the National Dialogue. However, the absence of a detailed data analysis plan suggests that OFPP lacks a sound approach to evaluate pilot results.

Leading Practice 5: Ensure Appropriate Two-Way Stakeholder Communication. OFPP’s plans for obtaining stakeholder input and fostering two-way dialogue have not yet been developed to engage public participation and feedback on its approach for designing and implementing the procurement portion of the pilot. Similar to the approach taken by HHS, OFPP staff told us that they used comments posted on the National Dialogue to inform the design of the procurement portion. However, as previously mentioned, we have concerns about the usefulness of that approach because none of the three comments they received on the dialogue were related to certified payroll. OFPP staff said they also used comments posted on the 2014 open dialogue on improving procurement processes to inform their pilot design.

From commentary posted on both sites, OFPP identified certified payroll reporting as a pain point that could be further explored through the pilot project. OFPP staff told us that they engaged GSA’s 18F to conduct the discovery phase of the pilot design to better understand areas of significant reporting burden related to certified payroll with a select group of stakeholders that included contractors, federal agency officials, and contracting officers. A Federal Register notice was also issued on November 24, 2015 to solicit public comments on the reporting burden of the procurement portion of the pilot under the Paperwork Reduction Act.

Although OFPP obtained stakeholder input to identify areas of focus for the design of the procurement portion of the pilot, it has not engaged them to solicit input on other stages of the pilot, including design, implementation, data gathering, and assessment. Further, OFPP has not released specific information about the design of the pilot, nor has it made information about pilot participation available to stakeholders despite repeated requests for information from those participating in monthly calls hosted by the Association for Government Accountants and Treasury.

In addition to being a leading practice for pilot design, our previous work examining grants management streamlining initiatives found that stakeholder communication is not just “pushing the message out,” but should also facilitate a two-way, honest exchange and allow for feedback from relevant stakeholders. We found that a lack of opportunities to provide timely feedback resulted in poor implementation and prioritization of streamlining initiatives and limited recipients’ use and understanding of new systems. As such, it will be important for OFPP to engage with the procurement community on its pilot design so that it can be improved based on public input. In addition, more effective two-way communications could also be a strategy for recruiting participants for the procurement portion of the pilot.

Conclusions

In crafting the DATA Act, Congress sought to reduce the burden and cost of reporting for the recipients of federal funds. Toward that end, OMB, partnering with other federal agencies, has taken steps to design the Section 5 Pilot that will explore potential ways to reduce the burden and cost of reporting on federal funds for both the federal grantee and procurement communities. However, we found uneven progress in the grants and procurement portions of the pilot. OMB and HHS have made considerable progress designing an overall approach that will examine a variety of potential ways to simplify reporting for grant recipients. In addition to generally being on track to meet the specific requirements set out in the act, we found that the proposed design of the grants portion of the pilot partially adheres to leading practices.

In contrast, our review of the design of the procurement portion of the pilot raises several concerns. In the absence of a detailed design and risk management plans for executing the pilot moving forward, it is unclear how the design of the procurement portion will reflect the requirements set forth by section 5 of the act. Because of project delays to date, it will be especially important for OMB to communicate to Congress and interested stakeholders how it plans to address key aspects of these requirements, such as the collection of meaningful and useful data over a 12-month reporting cycle and including a diverse group of participants with federal contracts totaling from $1 billion to $2 billion.

Moreover, the design we reviewed for the procurement portion of the pilot did not reflect leading practices to allow for the development of effective recommendations to simplify reporting for contractors. Moving forward, given the tight timelines set out in the act, it will be important for OMB to redouble its focus on the design and implementation of the procurement portion. Without a sound design that applies leading practices, the recommendations to Congress for reducing reporting burden for contractors coming out of this effort may be late, of limited use, or incomplete.

Recommendations for Executive Action

1. To help ensure and more clearly convey how the procurement portion of the pilot will contribute to meeting the Section 5 Pilot design requirements, we recommend that the Director of OMB determine and clearly document (1) how it will collect certified payroll data over a 12-month reporting cycle, (2) ensure the diversity of pilot participants, and (3) how the inclusion of federal contracts will contribute to an aggregate amount of $1 billion to $2 billion. 2. To enable the development of effective recommendations for reducing reporting burden for contractors, the Director of OMB should ensure that the procurement portion of the pilot reflects leading practices for pilot design.

Agency Comments and Our Evaluation

We provided a draft of this report to OMB, HHS, and GSA for review and comment. OMB and HHS provided technical comments that we have incorporated throughout the report, as appropriate. OMB and HHS did not offer a view on our recommendations. GSA did not have any comments.

We are sending copies of this report to the Director of OMB, Secretary of HHS, Administrator of GSA, and appropriate congressional addressees. In addition, the report is available at no charge on the GAO website at https://www.gao.gov.

If you or your staff have any questions about this report, please contact me on (202) 512-6806 or by email at sagerm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This review (1) describes the administration’s approach to the Section 5 Pilot; (2) assesses whether current activities and plans will likely allow the Office of Management and Budget (OMB) and its partners to meet requirements and time frames established under the Section 5 Pilot; and (3) evaluates the extent to which the design for the pilot is consistent with leading practices.

To describe the administration’s approach to the pilot, we assessed documents related to pilot activities and interviewed OMB, Department of Health and Human Services (HHS), and General Services Administration (GSA) officials and staff responsible for implementing the Section 5 Pilot. Specifically, we reviewed documentation from HHS and OMB’s Office of Federal Procurement Policy (OFPP). Our reviews were based on the latest design plans available at the time. We also interviewed officials from organizations representing key non-federal stakeholders including state and local governments, private-sector contractors, and other federal fund recipients.

To assess whether the Section 5 Pilot design would be likely to meet the statutory design requirements, we reviewed section 5 of the Federal Funding Accountability and Transparency Act of 2006, as added by the Digital Accountability and Transparency Act of 2014 (DATA Act) to understand the deadlines and design requirements. We reviewed the draft design documents to assess OMB and its partners’ plans for meeting these requirements. To supplement our review of those plans, we also spoke with cognizant staff implementing these pilots at OMB, HHS, and GSA.

To identify and analyze leading practices for pilot design, we reviewed our past work evaluating and assessing pilots. Additionally, we also relied on our technical guidance on designing evaluations. Further, we reviewed relevant studies from academia as well as other entities, such as the Brookings Institution and the Federal Demonstration Partnership. We reviewed reports from organizations that have expertise on conducting pilot programs and experience in scaling pilot results that could be applied government-wide. We also shared these leading practices with the agencies in this review during our audit work.

To assess the extent to which the Section 5 Pilot design adhered to these leading practices, we reviewed documented designs and plans for both the grants and procurement portions of the pilot. To evaluate the grants portion of the pilot, we focused on a draft design document from November 2015. HHS officials told us that they have updated that plan. Because we did not receive this update until the end of March 2016, we did not have time to include its content for this report. As such, our assessment is based on the November 2015 plan. We intend to review the updated plan as we continue our work on DATA Act implementation. We have supplemented our assessment with information HHS officials provided to us during subsequent interviews, as appropriate. For the procurement portion, we reviewed a working draft plan from November 2015. While it is unclear whether there has been an updated version, we have also provided additional details from discussions with OFPP officials, as appropriate.

To evaluate the grants and procurement portions of the pilot, we applied the five leading practices we identified to OMB and HHS’s design documents. Each of those assessments were subsequently verified by another individual. We determined that the design met the criteria when we saw evidence that all aspects of a leading practice were met. When we were unable to assess whether all aspects of a leading practice were met without additional information, we determined that the design partially met the criteria. Finally, when we saw no evidence of a leading practice, we determined that the criteria was not met. In continuation of our constructive engagement approach on the DATA Act for working with agencies implementing the act, we provided HHS and OMB with feedback on the design of the grants and procurement portions of the pilot during our review. These officials generally accepted our feedback and, in some instances, noted that they have or would make changes to their design as a result of our input.

We conducted this performance audit from May 2015 to April 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Detailed Results of the Assessment of Design of the Grants Portion of the Section 5 Pilot Compared to Leading Practices

This appendix provides detailed information regarding our assessment of the pilot design for the grants portion of the Section 5 Pilot. We assessed each of the Department of Health and Human Services’s (HHS) six test models against the five leading practices for pilot design described in the report. Using HHS’s November 2015 design plans and relevant supporting information available during the preparation of this report, we determined whether each test model met, partially met, or did not meet those leading practices.

Appendix III: Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, J. Christopher Mihm (Managing Director), Peter Del Toro (Assistant Director), Shirley Hwang (analyst-in- charge), Aaron Colsher, Kathleen Drennan, Jason Lyuke, Kiran Sreepada, and David Watsula made major contributions to this report. Other key contributors include Lisette Baylor, Brandon Booth, Jenny Chanley, Robert Gebhart, Donna Miller, Carl Ramirez, Andrew J. Stephens, and Tatiana Winger. Additional members of GAO’s DATA Act Internal Working Group also contributed to the development of this report.