Data Act: OMB and Treasury Have Issued Additional Guidance and Have Improved Pilot Design but Implementation Challenges Remain
Released: 2016-12-08T00:00:00

Letter

Background

Signed into law on May 9, 2014, the DATA Act expanded on previous federal transparency legislation to link federal agency spending to federal program activities so that taxpayers and policymakers can more effectively track federal spending. The DATA Act requires government- wide reporting on a greater variety of federal funds as well as tracking of these funds at multiple points in the federal spending lifecycle. The act also calls for the federal government to set government-wide data standards, identify ways to reduce reporting burdens for grantees and contractors (Section 5 Pilot), and regularly review data quality to help improve the transparency and accountability of federal spending data.

OMB and Treasury have taken significant steps toward implementing the act’s various requirements including standardizing data element definitions, issuing guidance to help agencies develop their implementation plans, and designing a pilot for developing recommendations to reduce recipient reporting burden. We have previously reported on these efforts and others and have identified a number of ongoing challenges that will need to be addressed in order to successfully meet the act’s requirements. Throughout our ongoing oversight of OMB’s and Treasury’s efforts to implement the act, we have coordinated closely with OMB and Treasury to provide timely feedback and have made a number of recommendations that, if addressed, could help ensure the full and effective implementation of the act. OMB and Treasury have made progress implementing 5 of our recommendations related to DATA Act implementation. However, additional effort is needed to address 11 previous GAO recommendations that remain open. See appendix II for a list of our previous recommendations relating to the DATA Act and their implementation status.

Initial Steps Taken to Develop a Governance Structure but Further Steps Needed to Maintain the Integrity of Data Standards

OMB and Treasury are developing a governance structure, but more work will be needed to ensure that this structure is consistent with key practices for developing and maintaining the integrity of data standards over time. In July 2015, we reported that OMB and Treasury took initial steps to develop organizational structures for project governance but had not yet established a formal framework for providing data governance throughout the lifecycle of developing and implementing standards. Such a framework is key for ensuring that the integrity of data standards is maintained over time. Accordingly, we recommended that OMB and Treasury establish a clear set of policies and procedures for developing and maintaining data standards that are consistent with leading practices.

OMB and Treasury generally agreed with our recommendation and, in response, engaged a contractor to interview key stakeholders and develop a set of potential next steps. The first of these steps was to establish a new Data Standards Committee that will be responsible for maintaining established standards and developing new data elements or data definitions that could affect more than one functional community (e.g., financial management, financial assistance, and procurement). According to OMB staff, the Data Standards Committee held its inaugural meeting on September 15, 2016, and will meet on a monthly basis. The committee has also drafted a charter that will delineate the scope of the committee’s work, as well as the composition and responsibilities of its members. According to OMB staff, members include representatives from a range of federal communities including the grants, procurement, financial management and human resources communities, as well as representatives of several interagency councils including the Chief Information Officers Council and the Performance Improvement Council. OMB staff told us that the committee will focus on clarifying existing data standard definitions, including the definition of predominant place of performance, and identifying new standards that may be needed going forward. In October 2016, according to OMB staff, the charter was under review by the DATA Act Executive Steering Committee.

Several data governance models exist that could inform OMB’s and Treasury’s efforts to ensure the integrity of the data standards over time.

These models define data governance as an institutionalized system of decision rights and accountabilities for planning, overseeing, and controlling data management. Many of these models promote a common set of key practices that include establishing clear policies and procedures for developing, managing, and enforcing data standards. A common set of key practices endorsed by standards setting organizations recommend that data governance structures should include the key practices shown in the text box below. We have shared these key practices with OMB and Treasury.

Key Practices for Data Governance Structures i.

Developing and approving data standards. ii.

Managing, controlling, monitoring, and enforcing consistent application of data standards. iii.

Making decisions about changes to existing data standards and resolving conflicts related to the application of data standards. iv.

Obtaining input from stakeholders and involving them in key decisions, as appropriate. v.

Delineating roles and responsibilities for decision-making and accountability, including roles and responsibilities for stakeholder input on key decisions.

OMB and Treasury have not yet institutionalized and clearly documented policies and procedures that are consistent with these key practices. For example, processes have not been developed to both approve new standards and ensure that already established standards are consistently applied and enforced across the federal government.

One reason why having a robust, institutionalized data governance structure is important is to provide consistent data management during times of change and transition. The transition to a new administration presents one such situation. We have previously reported that, given the importance of continuity when implementing complex, government-wide initiatives, the potential for gaps in leadership that can occur as administrations change can impact the effectiveness and efficiency of such efforts, potentially resulting in delays and missed deadlines. Such transitions may disrupt the momentum for meeting implementation timeframes or cause the government to fail to continue to build on previous accomplishments. The absence of a robust and institutionalized data governance structure presents additional potential risks regarding the integrity of data standards over time and the ability of agencies to meet their statutory timelines in the event that priorities shift with the incoming administration or momentum is lost.

Agencies’ Implementation Plan Updates Indicate Continuing Challenges to Meeting the DATA Act Reporting Deadline

In June 2016, OMB directed the 24 CFO Act agencies to update their initial DATA Act implementation plans that they submitted in response to OMB’s May 2015 request. Each agency was to (1) update its timeline and milestones and explain the agency’s progress to date and the remaining actions it would take to implement the act in accordance with the suggested steps in Treasury’s DATA Act Implementation Playbook (Version 2.0) (Playbook 2.0), (2) report costs to date and estimated total future costs, and (3) explain any new challenges and mitigation strategies.

In reviewing the 24 CFO Act agencies’ implementation plan updates that we obtained from the agencies, we found the following:

Each of the 24 CFO Act agencies’ updates included timelines and milestones and most of the updates included most of the OMB required information. For example, most of the 24 CFO Act agencies included remaining actions the agencies would take to implement the suggested steps in Playbook 2.0.

Some of the CFO Act agencies did not include information about some of the remaining actions to implement the suggested steps in Playbook 2.0. For example, 5 of the 24 CFO Act agencies did not include information about testing for completeness and accuracy of data elements submitted to Treasury, 11 CFO Act agencies did not include information about workflows for addressing validation errors and revisions needed to agency data submissions, and 13 CFO Act agencies did not include information about testing linkages of program and financial data or possible interim solutions to link such data, if needed. Without such information in agencies’ updates, it may be more difficult for OMB and Treasury to determine where to target their monitoring and assistance efforts to help ensure the DATA Act is successfully implemented.

Our review of the CFO Act agencies’ August 2016 implementation plan updates found that 21 of the 24 CFO Act agencies reported costs to date and future estimated costs to implement the DATA Act reporting requirements. One agency reported future estimated costs, but did not report costs to date. Two agencies did not provide any cost estimates. Total cumulative and future estimated costs for full DATA Act implementation that was reported by 22 CFO Act agencies in their implementation plan updates ranged from approximately $1.0 million to $59.1 million, for a total of about $202.4 million. This total estimated cost reported by CFO Act agencies to implement the DATA Act includes costs for systems integration and modernization. It is important to note that the estimated total costs reported by CFO Act agencies to implement the DATA Act requirements is relatively small when compared to the almost $81 billion spent on information technology by the CFO Act agencies in fiscal year 2016 alone.

See appendix III for more details about the information that OMB required CFO Act agencies to include in their implementation plan updates, remaining actions to implement the suggested steps in Playbook 2.0, and the number of CFO Act agencies that included the information.

In our July 2016 report, we reported on challenges agencies included in their initial implementation plans. The implementation plan updates indicate that 19 of the 24 CFO Act agencies continue to face challenges in their efforts to implement the DATA Act. Based on our review of the 24 CFO Act agency implementation plan updates, we identified four overarching categories of challenges reported by agencies that may impede their ability to effectively and efficiently implement the DATA Act: systems integration issues, lack of resources, evolving and complex reporting requirements, and inadequate guidance. See table 4 in appendix III, which describes the categories of challenges and the number of CFO Act agencies reporting challenges in each category. Some of the challenges reported by the CFO Act agencies in their updates include the following: Systems integration. Nineteen of the 24 CFO Act agencies reported challenges related to systems integration, which include concerns with systems limitations, modernization efforts, and timing. For example, one agency reported that validation presents challenges because its financial systems are not properly integrated with procurement and grant systems. Similarly, the agency reported that several of its components are undergoing grant, procurement, or financial system improvements that coincide with implementing the DATA Act, which could pose a risk to timely DATA Act implementation if the improvements are delayed. Another agency reported that, for one of its legacy systems, obtaining the unique identifier to generate award financial data will likely be a manual process. The lack of properly integrated systems increases the risk that agencies may have difficulty compiling and validating the information they are required to report under the DATA Act by the May 2017 reporting deadline for agencies to submit their financial and payment information.

Resources. Fourteen of the 24 CFO Act agencies reported challenges related to staffing issues or funding constraints. For example, one agency reported that expertise related to feeder systems and data will be needed to successfully implement the DATA Act, but such subject matter experts may not be available. Another agency reported that meeting the reporting deadline is highly dependent on receiving requisite funding and resources. The lack of sufficient resources, including staff expertise and proper funding, increases the risk that agencies may have difficulty taking all the actions needed in a timely manner to fully implement the requirements of the DATA Act.

Reporting. Thirteen of the 24 CFO Act agencies reported challenges related to mandatory DATA Act reporting requirements, including concerns with data quality and their ability to report all the required data elements in their initial DATA Act submissions, as well as senior accountable officials (SAO) certification and reporting non-financial data. For example, one agency reported that its SAO may be unable to certify the quality of data if OMB’s guidance for the SAO certification cannot be supported by existing processes. Another agency reported concerns with the burden of reconciling account data with financial and award data. In addition, two agencies reported challenges with reporting beginning balances at the level of detail required by the DATA Act. Two agencies reported concerns with protecting sensitive and classified data. One agency also reported ongoing issues with inconsistent quality of data submitted from their financial systems. Another agency reported that certain data elements are not currently available for all document types, and is considering pulling these data elements from other source systems to the extent possible. A lack of complete and accurate agency data increases the risk that agencies may not be able to meet the DATA Act reporting requirements within the mandated timeframes.

Guidance. Eleven of the 24 CFO Act agencies reported ongoing challenges related to the timely issuance of, and ongoing changes to, OMB policy and Treasury guidance. Eight agencies reported that if policy or technical guidance continues to evolve or be delayed, the agencies’ ability to comply with the May 2017 reporting deadline could be affected. Some agencies also reported concerns about the requirement for SAOs to certify the data reported quarterly. For example, one agency reported that if guidance clarifying certification procedures is delayed, it may not have time to implement appropriate validation steps needed to give assurance over the data. Because of the lack of timely and consistent guidance, agencies may need to continuously update or change their processes, which could adversely affect their ability to meet the DATA Act requirements.

As noted above, the information reported by the CFO Act agencies in their implementation plan updates indicates that some agencies are at increased risk of not meeting the May 2017 reporting deadline because of these challenges. In addition, inspectors general for some agencies, such as the Departments of Labor and Housing and Urban Development, have issued readiness review reports that also indicate their respective agencies are at risk of not meeting the reporting deadline. As discussed further below, the technical software requirements for agency reporting are still evolving, so any changes to the technical requirements over the next few months could also affect agencies’ abilities to meet the reporting deadline.

In August 2016, in response to our prior recommendation, OMB established procedures for reviewing and using agency implementation plan updates that include procedures for identifying ongoing challenges. In its procedures document, OMB states that it has received input from a significant number of agency staff via office hours, emails, regular meetings, agency visits, and other methods regarding the challenges agencies are experiencing as they work toward implementation since the submission of their original plans. OMB’s document also states that it has worked to address these challenges and provide both policy and technical guidance as needed. Further, the document stated that requiring agencies to update plans will allow OMB to address challenges that agencies are not directly reaching out to OMB about or that numerous agencies are experiencing. According to the procedures document, OMB will also be monitoring progress toward the statutory deadline and setting up meetings with any of the 24 CFO Act agencies that OMB identifies as being at risk of not meeting the implementation deadline. OMB will schedule these visits by reviewing the implementation plan updates and discerning which agencies appear to be experiencing the most challenges to implementation.

To help address their challenges, 16 of the 24 CFO Act agencies reported that they use certain mitigating strategies in their implementation plan updates. Based on our review, we identified seven overarching categories of mitigating strategies reported by these agencies to address DATA Act implementation challenges: making changes to internal policies and procedures, leveraging existing resources, using external resources, continuing communications, employing manual and temporary workarounds, monitoring and developing guidance, and enhancing existing systems. These strategies, as a whole, were similar to the mitigating strategies reported by agencies in their initial implementation plans. The most commonly reported categories of mitigating strategies were changing internal policies and procedures and leveraging existing resources. See table 5 in appendix III for descriptions of the categories of mitigating strategies and the number of CFO Act agencies that report using strategies from each category.

Progress Made in Data Standards and Technical Framework, but Uncertainties Persist as Implementation Proceeds

OMB Released Additional Policy Guidance to Clarify Agency Reporting Requirements

In May 2016, in response to our prior recommendation, OMB released additional guidance on reporting financial and award information required under the act to address potential clarity, consistency, and quality issues with the definitions of standardized data elements. In January 2016 we reported that ensuring that data definitions are generally consistent with leading practices is important because limitations with the definitions could lead to inconsistent or inaccurate reporting, among other issues. We also reported that although the standardized data element definitions issued by OMB largely adhered to leading practices for establishing data definitions, several definitions had limitations that could lead to inconsistent reporting. While OMB’s additional guidance addresses some of the limitations we identified, it does not address all the clarity issues we identified.

Specifically, OMB’s additional guidance addresses (1) reporting financial and award level data, (2) establishing linkage between agency award and financial systems using a unique award identifier, and (3) providing assurances that data submitted to Treasury for publication on USASpending.gov is sufficiently valid and reliable. For example, OMB’s Management Procedures Memorandum No. 2016-03 directs agencies to leverage existing procedures for providing assurances of the quality of their DATA Act data submissions and directs agency SAOs to provide reasonable assurance that their internal controls support the reliability and validity of the data submitted to Treasury for publication on USASpending.gov. OMB’s memorandum notes that assurance means that, at a minimum, the data reported are based on appropriate internal control and risk management strategies identified in OMB Circular A- 123. OMB expects SAO assurance of the data through this process would mean that data submitted to Treasury by May 2017 complies with existing controls for ensuring the data quality.

However, our prior work has shown that relying on these quality assurance processes is not sufficient to address the accuracy and completeness challenges that we have previously identified. Additionally, as we reported in August 2016, Offices of Inspector General, which are required to assess the completeness, timeliness, quality, and accuracy of data submitted under the act, have expressed concerns about agencies’ abilities to provide assurances of the quality of their data. The inspectors general are particularly concerned about their agencies’ ability to provide quality assurances for data that are not directly provided by the agency, such as data submitted by non-federal entities who receive federal awards.

To address these concerns, OMB released draft guidance in August 2016 that specifies DATA Act reporting responsibilities when an intragovernmental transfer (both allocation transfers and buy/sell transfers) is involved, explains how to report financial assistance awards with personally identifiable information (PII), and clarifies the SAO assurance process over the data submitted to the broker. OMB staff told us that this most recent policy guidance was drafted in response to questions and concerns reported by agencies in their implementation plan updates, as well in meetings with senior OMB and Treasury officials intended to assess agency implementation status. Among other challenges, agencies indicated the need for additional guidance on reporting intergovernmental transfers, providing assurances over their data, and reporting insurance information. For example, officials from USDA, one of our case example agencies, told us that they are waiting for guidance on insurance and indemnity reporting, but no guidance has been issued. Absent any new guidance, they plan to report insurance as they have under the Federal Funding and Accountability and Transparency Act of 2006 (FFATA). OMB staff told us that they received feedback from 30 different agencies and reviewed over 200 comments on the draft guidance. The final guidance, OMB M-17-04, was issued on November 4, 2016.

Although OMB has made some progress with these efforts, other data definitions lack clarity—including primary place of performance and award description—which still needs to be addressed to ensure agencies report consistent and comparable data. These challenges, as well as the challenges identified by agencies, underscore the need for OMB and Treasury to fully address our prior recommendation to provide agencies with additional guidance to address potential clarity issues. OMB staff told us that the newly established Data Standards Committee will be responsible for developing guidance to provide additional operational clarity regarding these data definitions; however, they were unable to provide a specific timeframe for when this would be done.

Late Release of the Schema Version 1.0 May Pose Risks for Implementation Delays at Some Agencies

Treasury released the schema version 1.0 on April 29, 2016—4 months later than planned and approximately a year before reporting is required to begin under the act. The schema version 1.0 is intended to standardize the way financial assistance awards, contracts, and other financial data will be collected and reported under the DATA Act. Treasury expects the guidance provided in the schema version 1.0 will provide a stable base for agencies to develop the necessary data submission procedures. We have previously reported that a significant delay in releasing version 1.0 of the schema would likely have consequences for timely implementation of the act. Agencies are using schema version 1.0 to plan what changes are needed to systems and business processes to be able to capture and submit the required data. Under the act, agencies must report data in compliance with established standards by May 2017. Toward that end, OMB and Treasury have directed agencies to begin submitting data by the beginning of the second quarter of fiscal year 2017 (January 2017) with the intention of publically reporting that data by May 2017.

OMB’s summary of agencies’ implementation plan updates acknowledged that the delay in the release of schema version 1.0 delayed agency timelines for implementation. This document also recognized that the iterative approach being used to develop and release guidance has posed challenges for some agencies as changes in the guidance may require them to re-work some of their implementation project plans. Our analysis of the implementation plan updates submitted by the agencies to OMB confirms this. We found that 11 of the 24 CFO Act agencies highlighted challenges related to the guidance provided by OMB and Treasury in their implementation plan updates. One of the commonly cited challenges concerned complications arising due to the iterative nature or late release of the guidance. For example, one agency reported that developing its implementation plan was highly dependent upon the concurrent development of the schema version 1.0 and technical guidance being developed by Treasury. This agency stated that any delays or changes to these components will significantly affect its solution design, development and testing schedule, and cost estimate.

Treasury Continues to Test and Revise the DATA Act Broker

A key component of the reporting framework laid out in the schema version 1.0 is the DATA Act Broker, a system to standardize data formatting and assist reporting agencies in validating their data prior to submitting it to Treasury. See figure 1 for a depiction of how Treasury expects the broker to operate.

Treasury’s software development team has been iteratively testing and developing the broker using what Treasury describes as an agile development process. Treasury released the first version of the broker in spring 2016 and it continues to develop the system’s capabilities through 2-week software development cycles, called sprints. On September 30, 2016, Treasury released a version of the broker, which it stated was fully capable of performing the key functions of extracting and validating agency data. Treasury officials told us that although they plan to continue to refine the broker to improve its functionality and overall user experience, they have no plans to alter these key functions.

According to Treasury guidance documents, agencies are expected to use the broker to upload three files containing data pulled from the agencies’ internal financial and award management systems. These files will undergo two types of validation checks in the broker before being submitted to Treasury: data element validations and complex validations. Data element validations check whether data elements comply with specific format requirements such as field type and character length. Complex validations perform tasks such as checking data against other sources or using calculation rules to verify whether certain data elements sum up to each other. Treasury has configured these complex validation rules so that if a rule is not met, the broker can either produce a warning message while still accepting the data for submission or produce a fatal error, which prevents submission of the data altogether. As of September 30, 2016, data uploaded to the broker needs to successfully meet less than a quarter of these complex validation checks in order to be accepted for submission to Treasury. Treasury officials said that this choice was made in order to allow agencies more flexibility to test the broker, and that the data submissions will be required to pass more of these validation rules at a later date. According to Treasury documents, in a future release of the broker, data uploaded to the broker will need to successfully meet about half of the complex validation checks in order to be accepted for submission to Treasury. Treasury officials said that having about half of the validation rules produce warnings rather than fatal errors would provide agency officials with the flexibility to correct issues flagged by the broker or not to do so, depending on their knowledge of the context and situation of specific data elements. For example, for some of the programs, grants award-level information may not be reported for security reasons.

In addition to assisting agencies in collecting and validating agency- generated data, the broker also extracts award and sub-award information from existing government-wide award reporting systems and helps ensure these files are in the standard format. This function was added during software development efforts in late September and early October 2016. Unlike the files submitted by agencies, these extracted files with award and sub-award information are not subject to any validations in the broker. However, Treasury implemented additional validation checks on the file containing agency financial assistance award information through its source system, the Award Submission Portal. These checks include verifying that required information is present and formatted correctly.

Treasury officials told us that the responsibility for ensuring the accuracy of these files lies with the DATA Act SAO at each agency. For example, OMB Management Procedures Memorandum 2016-03 specifies that SAOs must provide reasonable assurance that their internal controls support the reliability and validity of the agency account-level and award- level data submitted to Treasury for publication. Before final submission of the data files in the broker, the SAO is responsible for assuring that, at a minimum, the data reported are based on appropriate internal control and risk management strategies identified in OMB Circular A-123. Treasury officials said that if SAOs are not able to provide this assurance, their agency will be prevented from submitting the files and their data will not be included in the data reporting based on the current broker design. Currently, the broker does not allow agencies to submit their data with qualifications, such as known quality limitations, so data that does not completely meet the criteria for SAO assurance will not be reported, even with qualifications. OMB staff and Treasury officials said that they are reconsidering this position and are exploring ways that agencies can submit data with qualifications and how these qualifications can be conveyed to the public.

Agency Data Submissions are Under Development

Agencies have made progress toward creating their data submissions and testing them in the broker, but work remains to be done before actual reporting can begin. Treasury has made empty sample files available to agencies so they can begin testing their data files in the broker without having completed building all of them. As of October 2016, 21 of 24 CFO Act agencies reported that they had begun testing their data files in the broker, but only the National Aeronautics and Space Administration had completed testing the broker and revised its data files accordingly. Treasury also collects data from the four shared service providers that are helping to manage data submissions for their agency clients. As of October 2016, two of these shared service providers reported to Treasury that they had finished building the data files for submission to the broker.

In August 2016, we reported that agencies we reviewed are relying on a series of software patches from their enterprise resource planning (ERP) vendors to facilitate their data submissions. ERP vendors are developing patches that will extract data to help their clients develop files that comply with DATA Act requirements. According to vendors, these patches will help link an agency’s financial and award systems, create additional fields in existing systems to report new data elements, and extract data files formatted for submission to Treasury. Patches that will facilitate the generation of agency file submissions were planned to be completed between August 2016 and February 2017. As of September 2016, the release of one of these patches has been delayed. Oracle, one of the ERP vendors developing these patches, had planned to release a patch that would allow award attributes to be captured in their clients’ core purchasing systems and general ledger journals in August 2016, but the release was delayed until September 13, 2016. Representatives from SAP, another such ERP vendor, said that they were able to deliver one of the needed patches to their clients in August 2016 and an additional patch in October. But, they also said that changes and adjustments to the broker had delayed their progress towards creating a patch that can format their clients’ data files for submission.

It will be more difficult for agencies that are relying on ERP vendor patches to test their data files in the broker until the patches have been implemented since the patches will enable them to construct and format their files for submission to the broker. Two agencies reported in their implementation plan updates that a delay in the release of the patches could jeopardize complete and timely data submission for May 2017. Treasury officials told us that agencies should still be able to create and submit the required files to the broker without these patches. These officials said that when designing the schema and broker, they chose to use a simple file format for data submissions so that agencies would be able to create these files without a specialized software solution. Treasury officials acknowledged that patches will make the submission process easier, but also pointed out that not every agency is able to take advantage of software patches. Some agencies reported in their implementation plan updates that they developed plans for interim solutions to construct these files until the patches can be developed, tested, and configured. However, some of these interim solutions rely on manual processing, which can be burdensome and increase the risk for errors. For example, USDA officials said that the effort to create an interim solution has been very resource intensive. This process involved surveying USDA’s bureaus to identify how their systems are configured and using that information to modify the financial system. HHS has also developed an interim reporting solution that can generate the required files without depending on a patch. However, HHS officials said this interim solution is complex and their processes cannot be fully automated until the Oracle patch is released. Furthermore, since these processes are not fully automated, they carry a risk of errors being introduced though human error. Agencies that are developing interim solutions will only have until May 2017 to test the data before the reporting deadline. An OMB document commended these agencies for developing robust contingency plans since this will better position them for timely implementation, but acknowledged that that long-term reporting solutions are still needed.

Updated Design of Section 5 Pilot Meets DATA Act Requirements and Mostly Reflects Leading Practices, but the Procurement Portion Does Not Clearly Document How its Focus on Certified Payroll Reporting Will Apply to Other Reporting Requirements

Revised Pilot Design Meets DATA Act Requirements

As required by the DATA Act, OMB is conducting a pilot program, known as the Section 5 Pilot, aimed at developing recommendations for reducing reporting burden for grantees and contractors. The Section 5 Pilot has two primary focus areas—federal grants and federal contracts (procurements). OMB partnered with HHS to design and implement the grants portion of the pilot and with the General Services Administration (GSA) to implement the procurement portion. As the executing agent for the grants portion of the pilot, HHS developed six “test models” to evaluate different approaches to potentially reducing grantee reporting burden. On the procurement portion of the pilot, OMB’s Office of Federal Procurement Policy (OFPP) worked with GSA’s 18F to develop and test a proof of concept reporting portal for reports required by the Federal Acquisition Regulation (FAR) and is piloting it with the centralized reporting of certified payroll by contractors working construction projects in the United States. In March 2016, a revised plan describing the design of the grants portion of the pilot was released, which updated the November 2015 version we previously reviewed. This was followed, in July 2016, by a revised version of the design for the procurement portion. See table 1 for a summary of the test models components that comprise the grants and procurement portions of the Section 5 Pilot.

We determined that the updated design for both portions of the Section 5 Pilot meets the statutory requirements for the pilot established under the DATA Act. Specifically, the DATA Act requires that the pilot program should include the following design features: (1) collect data during a 12- month reporting cycle; (2) include a diverse group of federal award recipients and, to the extent practicable, recipients that receive federal awards from multiple programs across multiple agencies; and (3) include a combination of federal contracts, grants, and subawards with an aggregate value between $1 billion and $2 billion. Based on our review of design documents as well as interviews with cognizant agency staff, there has been substantial improvement in this area since our last review, when the design lacked specifics in the procurement portion of the pilot, which made it difficult to determine whether the design of the overall pilot would meet these requirements.

Grants and Procurement Portions of the Pilot Partially Reflect All 5 Leading Practices for Pilot Design

Both the grants and procurement portions of the pilot showed substantial improvements in the extent to which they reflect leading practices for pilot design (shown in the textbox below). We found that HHS’s March 2016 revised design for the grants portion of the pilot partly reflects all five of the leading practices for effective pilot design—an improvement from our prior assessment. For example, in our April 2016 review we found that the grants design lacked specific details regarding how the data will be analyzed and how conclusions will be reached about integrating the pilot activities into overall grant reporting efforts. Based on our feedback, OMB and HHS developed a plan to analyze survey and other data prior to the start of data analysis. This plan specifies the types of quantitative and qualitative data analysis HHS intends to conduct for each test model and how that assessment links back to the stated hypotheses. HHS also added a sampling plan and information on participant outreach efforts to the design of the grants portion of the pilot which helped it to meet the leading practices for effective pilot design.

Leading Practices for Effective Pilot Design 1. Establish well-defined, appropriate, clear, and measurable objectives. 2. Clearly articulate an assessment methodology and data gathering strategy that addresses all components of the pilot program and includes key features of a sound plan.

3.

Identify criteria or standards for identifying lessons about the pilot to inform decisions about scalability and whether, how, and when to integrate pilot activities into overall efforts. 4. Develop a detailed data-analysis plan to track the pilot program’s implementation and performance and evaluate the final results of the project and draw conclusions on whether, how, and when to integrate pilot activities into overall efforts. 5. Ensure appropriate two-way stakeholder communication and input at all stages of the pilot project, including design, implementation, data gathering, and assessment.

OMB’s July 2016 revision of the design of the procurement portion of the Section 5 Pilot also showed substantial improvements in reflecting the leading practices for effective pilot design. Compared to the previous version, dated November 2015, we identified progress in several areas. For example, the revised procurement design identified hypotheses for each objective and contained objectives that were linked to metrics that should facilitate OMB’s ability to collect appropriate evaluation data. The revised design also provides additional details regarding the procurement portion’s intended assessment methodology. It specifies that participants will submit payroll information to the centralized test portal on a weekly basis and that OMB will use focus groups to collect qualitative data from agency staff that use these data for contract management and oversight purposes. Furthermore, the revised design includes a data-analysis plan that describes how OMB will collect, track, and analyze data produced by the pilot. Finally, the revised procurement design provides additional detail about how potential findings could be scalable from the experiences of the individual pilot participants to the larger population of contractors required to submit certified payroll reports in compliance with Davis Bacon requirements. Toward that end, the revised procurement pilot design contains a sampling plan that provides criteria for selecting a diverse group of participants.

Design of the Procurement Portion of the Pilot Does Not Clearly Document How its Focus on Certified Payroll Reporting Will Apply to Other Procurement Reporting Requirements

We found some areas where the revised procurement design does not fully reflect leading practices for effective pilot design. These largely relate to how OMB intends to broaden the pilot’s initial focus on centralizing certified payroll reporting to other types of FAR-required reporting. The procurement design presents a reasonable set of factors for why OMB decided to initially select certified payroll reporting for testing the potential usefulness of a centralized reporting portal to reduce reporting burden. However, the plan does not take the next step of clearly describing and documenting how findings related to centralized certified payroll reporting will be more broadly applicable to the many other types of required reporting under the FAR beyond citing general concepts such as data pre-population and system integration. More specifically, the current design lacks a plan for testing the assumption that the experiences contractors have with centralized certified payroll reporting will be similar when they use the system to meet different reporting requirements and other databases. This is of particular concern given the diversity of reporting requirements contained in the FAR. In fact, OMB staff have identified over 100 different types of FAR reporting requirements with different reporting frequencies, mechanisms, and required information.

OMB staff told us that they expect to test the centralized portal on other types of FAR-required reporting and the revised design briefly mentions other FAR requirements such as those for service contracts and affirmative action plans. However, the revised design does not provide any details on how this will be done. The absence of an assessment methodology and an approach to test the scalability of the design when applied to procurement reporting requirements beyond certified payroll reporting is inconsistent with leading practices for pilot design and raises questions about whether the pilot design will meet its stated objective of reducing procurement reporting burden more broadly.

Implementation of Grants Portion of the Pilot is Underway, but the Procurement Portion is Not Scheduled to Begin Until Early 2017

HHS has taken a number of steps to begin implementing the design of the grants portion of the pilot. For example, they are recruiting participants for all of the test models and have begun administering data collection instruments for all of the test models. HHS has engaged in a number of outreach efforts to recruit participants for its test models. These officials told us that they have attended an estimated 70 events since 2015 to discuss the grants pilot, during which they provided information to interested attendees on how to get involved. Additionally, as of August 2016, HHS officials reported e-mailing almost 8,000 potential participants and plan on emailing additional prospects, if needed, in order to reach an established minimum number of participants for each test model.

GSA’s 18F completed a prototype for the procurement portion of the Section 5 Pilot at the end of May 2016 and presented it to OMB in June 2016. 18F’s role was to explore how an electronic certified payroll reporting portal could reduce contractor burden for federal Davis-Bacon contracts. In August 2016, GSA’s Federal Acquisition Service, the implementation lead for the pilot, awarded NuAxis the contract to build the prototype from information obtained as a result of the 18F prototype process. GSA officials told us that starting in September 2016, NuAxis began developing a web-based reporting interface that will allow users to centrally enter and submit certified payroll data. They plan to make this interface compatible with other existing systems, such as the System for Award Management (SAM) and Wage Determination Online (WDOL) to access relevant data sources.

In late November 2016, OMB staff and GSA officials informed us that they decided to delay launching the portal to conduct the procurement portion of the pilot in order to ensure that security procedures designed to protect personally identifiable information (PII) were in place. GSA officials told us that the centralized reporting portal that would be used to collect data on certified payroll did not receive the required Authority to Operate because it did not include necessary security measures to protect the PII that would be submitted by contractors participating in the pilot. Before the portal can be used to collect PII, GSA officials said they needed to issue a System of Records Notice and redesign the certified payroll reporting platform so that it conforms to agency security procedures. As a result of these additional steps, GSA officials expect to be able to begin collecting data through the centralized reporting portal sometime between late January 2017 and late February 2017. OMB staff said that despite the security-related delay, they still plan on collecting 12 months of data through the procurement pilot as required by the act.

In order to meet the act’s requirement that OMB deliver a report to Congress on ways to reduce recipient reporting burden by August 2017, OMB staff told us that they plan to only include data collected up to June or July 2017 in order to allow for sufficient time to analyze the results and incorporate them into the report’s findings. However, these staff said that that they plan to continue to collect data through the procurement portion of the pilot until they obtain a full 12 months of contractors’ experiences with centralized payroll reporting. Afterwards, OMB plans to analyze this data, compare it to the smaller data set produced for the August 2017 report to Congress and, if necessary, make any needed revisions to the findings and recommendations contained in the report previously submitted to Congress.

Conclusions

Across the federal government, agencies have efforts under way to implement the DATA Act by the May 2017 deadline and the success of these efforts will depend on, among other things, OMB and Treasury’s efforts to address agency-reported challenges and build an infrastructure to effectively support government-wide implementation. OMB and Treasury have made progress but still need to fully address the recommendations we have made in our previous reports. For example, OMB and Treasury can build upon the initial step of establishing a data standards committee responsible for maintaining already established standards and identifying new standards towards the goal of establishing an institutionalized system of data management that follows key practices and ensures the integrity of the data standards over time. In this context, implementing our prior recommendations will be critical to OMB’s and Treasury’s progress.

Among the areas where progress has been made in setting a foundation for successfully implementing the act is the Section 5 Pilot to reduce reporting burden. In particular, the design of the procurement portion of the pilot has improved substantially, including the extent to which it reflects leading practices of pilot design. However, despite advances in several areas, the current design remains limited by its lack of specifics regarding how a pilot focused on assessing contractors’ experiences with a centralized portal designed for certified payroll reporting will be applicable to many other federal procurement reporting requirements. By addressing issues such as this and continuing to focus on implementing the act, the administration greatly increases the likelihood of creating a system that will achieve the goals of the act—to increase the transparency of financial information and improve the usefulness of that data to Congress, federal managers, and the American people.

Recommendation for Executive Action

In order to ensure that the procurement portion of the Section 5 Pilot better reflects leading practices for effective pilot design, we recommend that the Director of OMB clearly document in the pilot’s design how data collected through the centralized certified payroll reporting portal will be used to test hypotheses related to reducing reporting burden involving other procurement reporting requirements. This should include documenting the extent to which recommendations based on data collected for certified payroll reporting would be scalable to other FAR- required reporting and providing additional details about the methodology that would be used to assess this expanded capability in the future.

Agency Comments and Our Evaluation

We provided a draft of this report to the Secretaries of Agriculture, Health and Human Services, and the Treasury; the Director of OMB; the Chief Executive Officer of the Corporation of National and Community Service; and the Administrator of the General Services Administration for review and comment. OMB, Treasury, CNCS, HHS, and GSA provided us with technical comments which we incorporated as appropriate. USDA had no comments. OMB and Treasury also provided written comments which are summarized below and reproduced in appendices IV and V, respectively.

In written comments submitted to us, OMB provided an overview of their implementation efforts since the passage of the DATA Act. These efforts include issuing three memoranda providing implementation guidance to federal agencies; finalizing 57 data standards for use on USASpending.gov; establishing the Data Standards Committee to develop and maintain standards for federal spending; and developing and executing the Section 5 pilot. The OMB response also noted that OMB and Treasury met with each of the 24 CFO Act agencies to discuss their implementation timelines, risks, and mitigation strategies, and took steps to address issues that could affect successful implementation. Through these meetings, OMB staff learned that 19 of the 24 CFO Act agencies expect that they will fully meet the May 2017 deadline for DATA Act implementation. OMB neither agreed nor disagreed with GAO’s recommendation.

In their written response, Treasury provided an overview of the steps they have taken to implement the DATA Act’s requirements and assist agencies in meeting their requirements under the act including OMB’s and Treasury’s issuance of uniform data standards, Treasury’s DATA Act Implementation Playbook, version 2.0, and the DATA Act Information Model Schema version 1.0. The Treasury response also noted that as a result of the aggressive implementation timelines specified in the act and the complexity associated with linking hundreds of disconnected data elements across the federal government, they made the decision to use an iterative approach to provide incremental technical guidance to agencies. According to Treasury, among other things, this iterative approach enabled agencies and other key stakeholders to provide feedback and contribute to improving the technical guidance and the public website.

We are sending copies of this report to the Secretaries of Agriculture, Health and Human Services, and the Treasury; the Director of OMB; the Chief Executive Officer of the Corporation of National and Community Service; the Administrator of the General Services Administration; as well as interested congressional committees and other interested parties.

This report will be available at no charge on our website at http://www.gao.gov.If you or your staff has any questions about this report, please contact J. Christopher Mihm at (202) 512-6806 or Mihmj@gao.gov or Paula M. Rascona at (202) 512-9816 or Rasconap@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

This review is part of an ongoing effort to provide interim reports on the progress being made in implementing the Digital Accountability and Transparency Act of 2014 (DATA Act), while also meeting the reporting requirements for us mandated by the act. This report examines: (1) steps taken to establish a clear data governance structure which is particularly important during the upcoming transition to a new administration; (2) challenges reported by Chief Financial Officers Act of 1990 (CFO Act) agencies in their implementation plan updates; (3) the operationalization of government-wide data standards and the technical specifications for data reporting; and (4) updated designs for the Section 5 Pilot for reducing recipient reporting burden and progress made in its implementation.

To describe the Office of Management and Budget’s (OMB) and the Department of the Treasury’s (Treasury) efforts to implement a data governance structure for the DATA Act, we identified common key practices for establishing effective data governance structures. To identify key practices for data governance we reviewed our past reports to identify applicable laws, regulations, and guidance, as well as reports from other entities that could inform our work. To select the sources we used to identify key practices for establishing an effective data governance program, we identified organizations that had data governance expertise, had previously published work on data governance, were frequently cited as a primary source, or some combination of these qualifications. In addition, because the DATA Act requires that established data standards incorporate widely accepted common data elements such as those developed by international voluntary consensus standards bodies, federal agencies with authority over contracting and financial assistance, and accounting standards organizations, we selected a range of organizations, including domestic and international standards-setting organizations, industry groups or associations, and federal agencies, to ensure we had a comprehensive understanding of data governance key practices across several domains.

All of the organizations we identified endorse establishing and using a governance structure to oversee how data standards, digital content, and other data assets are developed, managed and implemented. Based on these selection factors we drew on work from the following organizations to help us identify data governance key practices: American Institute of Certified Public Accountants, American National Standards Institute, Carnegie-Mellon University-Software Engineering Institute, Data Governance Institute, Data Management Association International, Oracle, National Association of State Chief Information Officers, National Institute of Standards and Technology, Digital Services Advisory Group and the Department of Education-Privacy Technical Assistance Center. We also met with OMB and Treasury to obtain information on the status of their efforts to address our previous recommendation that they establish a data governance structure.

To determine the implementation challenges reported by CFO Act agencies in their DATA Act implementation plan updates, we requested and received the updates from the 24 CFO Act agencies. We reviewed these implementation plan updates and assessed the information against OMB’s requirements and the revised guidance in Treasury’s DATA Act Implementation Playbook (Version 2.0) (Playbook 2.0) to determine whether the updates contained the information required by OMB—(1) an updated timeline and milestones with an explanation of the agency’s progress to date and the remaining actions it would take to implement the act in accordance with the suggested steps in Playbook 2.0, (2) costs to date and estimated total future costs, and (3) an explanation of any new challenges and mitigation strategies. We analyzed the agency-reported challenges and mitigating strategies and categorized them. We compared the categories of challenges reported by the CFO Act agencies in their implementation plan updates to the challenges that had been reported in their initial implementation plans in 2015 to identify any new categories of challenges. We interviewed cognizant OMB and Treasury officials and obtained any supporting documentation to further understand the implementation challenges reported by agencies in their implementation plan updates and OMB and Treasury’s processes and controls for reviewing the updated implementation information and monitoring agencies’ progress. We also met with OMB and Treasury to obtain information on the status of efforts to address our previous recommendations related to agency implementation plans.

To assess efforts to date to operationalize government-wide standards we reviewed OMB policy guidance intended to facilitate agency reporting as well as guidance intended to respond to agency requests that OMB clarify how to report specific transactions. We also interviewed OMB staff and Treasury officials to obtain information about plans for additional guidance as well as to assess the extent to which issued guidance is responsive to agency questions, requests for additional clarity on their reporting requirements, or both. We met with OMB and Treasury to obtain information on the status of efforts to address our previous recommendation related to the provision of policy guidance.

To examine the technical structure and specifications for reporting, we assessed Treasury’s processes for developing technical guidance and reviewed applicable technical documentation related to the schema version, 1.0 and the broker. We reviewed the broker made available by Treasury through open source code posted on a public website (GitHub repositories associated with the DATA Act) in order to understand its functionality and validations. In addition, we observed several demonstrations of how agencies submit their data to a prototype of the broker and the feedback produced by the system regarding data verification. We also interviewed knowledgeable officials from OMB, Treasury, and selected federal agencies and inspectors general, as well as enterprise resource planning (ERP) vendors assisting federal agencies with technical implementation.

To obtain specific information on how agencies use the technical guidance, we selected three agencies based on whether they were in compliance with existing federal requirements for federal financial management systems, the type of federal funding provided (such as grants, loans, or procurements), and their status as a federal shared service provider for financial management. Based on these selection factors, we chose the Department of Health and Human Services (HHS), the Department of Agriculture (USDA), and the Corporation for National and Community Service (CNCS). Although the information obtained from these three agencies is not generalizable to all agencies, they illustrate a range of conditions under which agencies are implementing the act. These are the same three agencies we selected for our January 2016 and August 2016 reports. This allowed us to assess progress in DATA Act implementation at these agencies since our last review. At each agency, we reviewed DATA Act implementation plan updates and interviewed officials responsible for implementation and DATA Act implementation team members. We met with OMB and Treasury to obtain information on the status of efforts to address our recommendation related to providing technical guidance.

To assess whether the Section 5 pilot designs meet statutory design requirements, we reviewed Section 5 of the Federal Funding Accountability and Transparency Act of 2006, as amended by the DATA Act, to understand the deadlines and design requirements. We reviewed the draft design documents to assess OMB and its partners’ plans for meeting these requirements. To supplement our review of those plans, we also spoke with cognizant staff implementing these pilots at OMB, HHS, and General Services Administration.

To assess the extent to which the Section 5 pilot designs adhered to leading practices for effective pilot design, we reviewed the documented designs for both the grants and procurement portions of the pilot. To evaluate the grants portion of the pilot, we reviewed the draft design document from March 2016 as well as data collection instruments such as surveys and quizzes. We supplemented our assessment with information HHS officials provided to us during subsequent interviews, as appropriate. For the procurement portion, we reviewed the draft design document from July 2016. Additionally, we supplemented our assessment with information officials from OMB’s Office of Federal Procurement Policy (OFPP) provided to us during subsequent interviews, as appropriate. To assess the grants and procurement portions of the pilot, we applied the five leading practices for effective pilot design we identified to both portion’s design documents. Each of these analyst assessments were subsequently verified by a second analyst. We determined that the design met the criteria when we saw evidence that all aspects of a leading practice were met. When we were unable to assess whether all aspects of a leading practice were met, we determined that the design partially met the criteria. Finally, when we saw no evidence of a leading practice, or if we identified a critical gap or shortcoming related to the practice, we determined that the criteria were not met. In continuation of our constructive engagement approach for working with agencies implementing the DATA Act, we provided HHS and OMB with feedback on the design of the grants and procurement portions of the pilot during our review. These officials generally accepted our feedback as useful and, in some instances, noted that they have or planned to make changes to their design as a result of our input. We also met with OMB to obtain information on the status of efforts to address our recommendation related to the design of the pilot for reducing recipient reporting burden.

We conducted the work upon which this report is based from May 2016 to December 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Status of GAO’s Recommendations Related to the DATA Act

Appendix III: Information Included in Chief Financial Officers Act Agencies’ Implementation Plan Updates

In May 2015, the Office of Management and Budget (OMB) directed federal agencies to submit Digital Accountability and Transparency Act of 2014 (DATA Act) implementation plans to OMB concurrent with the agencies’ fiscal year 2017 budget requests. In June 2015, the Department of the Treasury (Treasury) issued guidance—DATA Act Implementation Playbook (Version 1.0)—to help agencies prepare their implementation plans. We reviewed these implementation plans and on July 29, 2016, we issued a report on the results of our review.

In June 2016, OMB directed Chief Financial Officers Act of 1990 (CFO Act) agencies to submit updates to their initial DATA Act implementation plans by August 12, 2016. The updates were to (1) update timeline and milestones and explain the agency’s progress to date and the remaining actions it would take to implement the act in accordance with the suggested steps in the DATA Act Implementation Playbook (Version 2.0) (Playbook 2.0), (2) report costs to date and estimated total future costs, and (3) explain any new challenges and mitigation strategies.

Treasury’s DATA Act Implementation Playbook (Version 1.0) contained eight suggested steps and a timeline for agencies to use as they began to develop their DATA Act implementation plans. Steps 1 through 4 were to be completed by September 2015. However, as of October 2016, only 16 of the 24 CFO Act agencies reported that they had completed all steps 1 through 4. For example, there were four agencies reporting that they had not completed their inventory of data and identified the gaps in systems and processes for data elements (step 3).

DATA Act Implementation Playbook (Version 1.0) indicated that agencies would be working on steps 5 through 8 throughout fiscal years 2016 and 2017. Playbook 2.0—issued June 24, 2016—includes, among other things, expanded guidance on actions to be included in steps 5 through 8. Playbook 2.0 did not include expected timeframes for agencies to complete each step, rather, it referred agencies to Treasury’s implementation roadmap, which includes high level milestones for Treasury’s technical deliverables. Playbook 2.0 states that agencies can use the milestones in Treasury’s implementation roadmap to help determine their own implementation milestones. Descriptions of steps 5 through 8 from Playbook 2.0 follow:

Step 5: Prepare Data for Submission to the Broker. This step involves reviewing the schema version 1.0, extracting data from source systems, mapping agency data to the schema version 1.0, and implementing system changes as needed to collect and link data.

Step 6: Test Broker Outputs and Ensure Data are Valid. Agencies may use the broker to verify the data files the agency plans to submit to Treasury. The broker uses validation rules to test the completeness and accuracy of the data elements and linkages between financial and award data. The broker also tests whether the data passes basic validations within the schema version 1.0.

Step 7: Update Data. This step involves updating information and systems. If data does not pass validation (see Step 6), the broker will provide error details to the agency. The agency should then reference the authoritative data sources and address the discrepancies.

Step 8: Submit Data. Once the data is linked, validated, and standardized, agencies are to submit the data to Treasury for posting on USASpending.gov or a successor system. Agency senior accountable officials (SAO) are to provide reasonable assurance that their internal controls support the reliability and validity of the agency account-level and award-level data they submit to Treasury. This assurance is to be provided quarterly with data submissions beginning with fiscal year 2017 second quarter data. The SAO assurance means, at a minimum, that data reported are based on appropriate internal controls and risk management strategies identified in OMB Circular A-123.

Table 3 shows the information that OMB required CFO Act agencies to include in their implementation plan updates, information on remaining actions the agencies should take to implement suggested steps 5 through 8 in Playbook 2.0, and the number of CFO Act agencies that included the information.

Table 4 describes the categories of challenges reported by 19 of the 24 CFO Act agencies in their implementation plan updates and the number of agencies reporting challenges in each category. Five CFO Act agencies did not identify any challenges in their implementation plan updates.

Table 5 describes the mitigating strategies reported by 16 of the 24 CFO Act agencies in their implementation plan updates and the number of agencies reporting mitigating strategies in each category.

Appendix IV: Comments from Office of Management and Budget

Appendix V: Comments from the Department of the Treasury

Appendix VI: Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the above contacts, Peter Del Toro (Assistant Director), Michael LaForge (Assistant Director), Kathleen Drennan (analyst-in- charge), Diane Morris (analyst-in-charge), Michelle Sager, Shirley Hwang, Aaron Colsher, Katherine Morris, Sophia Tan, Thomas Hackney, Charles Jones, Laura Pacheco, Maria Belaval, Carrol Warfield, Jr., Mark Canter, James Sweetman, Jr., Andrew J. Stephens, Carl Ramirez and Jenny Chanley made major contributions to this report. Additional members of GAO’s DATA Act Internal Working Group also contributed to the development of this report.