Federal Human Resources Data: OPM Should Improve the Availability and Reliability of Payroll Data to Support Accountability and Workforce Analytics
Released: 2016-10-07T00:00:00

Letter

Background

The Office of Personnel Management (OPM) is tasked with providing human resources, leadership, and support to federal agencies to manage their human capital functions. For OPM to effectively perform this role, executive branch agencies are required to report information on their civilian employees to OPM and ensure that workforce data meet certain standards developed by OPM. OPM has developed these data to carry out its strategic goal of serving as a thought leader in data-driven human resource management and policy decision-making. The Enterprise Human Resources Integration (EHRI) system is OPM’s primary repository for human capital data to support these efforts.

OPM developed EHRI to (1) provide for comprehensive knowledge management and workforce analysis, forecasting, and reporting to further strategic management of human capital across the executive branch; (2) facilitate the electronic exchange of standardized human resources data within and across agencies and systems and the associated benefits and cost savings; and (3) provide unification and consistency in human capital data across the executive branch. In addition, OPM’s updated system and integrated data were expected to accrue savings to the federal government, reduce redundancy among agency systems, streamline the various processes involved in tracking and managing federal employment, and facilitate human capital management activities by providing storage, access, and exchange of standard electronic information through a data repository of standardized core human capital data for most executive branch employees.

While the personnel database predated the EHRI Data Warehouse, the payroll database was newly developed for OPM’s e-payroll initiative to consolidate agency payroll processes. The payroll database contains individual payroll records for approximately 2.0 million federal employees and is the primary governmentwide source for payroll information on federal employees. The records consist of data elements such as an EHRI ID for linking files, agency time charge categories, and pay rates.

The consolidation of agency payroll processes—known as the e-payroll project—provided the opportunity for OPM to begin collecting standardized governmentwide payroll data. As part of the e-payroll initiative, OPM consolidated the operations of 22 federal payroll system providers for the 116 executive branch agencies into four primary providers—General Services Administration’s (GSA) National Payroll Center (NPC), the Department of Defense’s Defense Finance and Accounting Service (DFAS), Department of Interior’s Interior Business Center (IBC), and Department of Agriculture’s National Finance Center (NFC). Consolidation was undertaken to simplify and standardize federal payroll policies and procedures, and better integrate payroll with other human capital and finance functions.

Processing Payroll from Time and Attendance Systems

Most federal agencies rely on one of the four payroll service centers, DFAS, IBC, NPC, or NFC, to process employee pay. Payroll service centers receive employees’ bi-weekly time sheets which come from a variety of time and attendance (TA) systems from the agencies they service. Generally, TA systems allow employees to specify time spent on different work and leave categories, such as the number of regular or overtime hours worked or the number of annual leave or sick leave hours taken in a given pay period (PP). However, the level of detail regarding the exact nature of the work or leave time varies depending on agency policies and systems for recording employee work time. While the service centers have consolidated payroll reporting, there are still variations among the centers and the TA systems agencies use to submit employee time sheets. Some systems are maintained by the service center and employees from various agencies access those systems to record their hours. For example, GSA has only one TA system that agencies use to record work and leave hours. Other systems are maintained by the agency and may reflect specific TA accounting needs of the agency. For example, NFC processes time sheets from several different TA systems, some of which are agency specific, and DFAS processes payroll for the Department of Defense, Veterans Affairs, and others through systems including the Business Management Redesign (e-Biz) TA system and the Automated Time, Attendance, and Production System (ATAAPS) (see figure 1).

In light of the significant variation in TA systems and service centers involved in processing TA information into payroll records, OPM established core standards for consistency in reporting and recording certain types of work and leave hours. These standards apply at the agency level as well as the service center level. Some of these standards are based on official leave authorized in statute. For example, federal employees are authorized to be absent from duty without a loss in pay or charge to leave for legal holidays and for activities such as jury duty, attendance at a military funeral, bone-marrow or organ donation, and certain union activities. Other standards are based on OPM guidance for excused absences due to inclement weather or blood donation, among others, charged as administrative leave. Agencies follow common recording practices for annual leave and sick leave. These core standards at the agency level are an important part of the process for reporting to OPM because they allow the service centers to collapse certain fields in a consistent way. While agencies may have specific time codes and time keeping practices to meet their needs, core standards for service center reporting dictate how these codes should be collapsed for reporting to OPM. For example, agencies may have detailed categories for various types of administrative leave, but segments of those charge codes apply generally to the category of administrative leave, enabling service centers to aggregate these data from TA systems.

Internal Controls and Data Reliability

OPM relies on agencies and service centers to ensure that the data they submit are timely, accurate, complete, and compiled in accordance with OPM standards. However, federal internal control standards specify that even when external parties, such as service centers in this case, perform operational processes for an agency, management—in this case OPM— retains responsibility for the performance of responsibilities assigned to those organizations. Consequently, OPM management is responsible for understanding the controls each service center has designed, implemented, and operates for payroll processing and how the service centers’ internal control systems impact OPM’s internal controls for the payroll data. Underlying requirements for data standards and data quality efforts are the standards for internal control which apply to all executive branch government functions. According to Standards for Internal Control in the Federal Government , effective internal control systems have certain attributes, including reliable internal and external sources that provide data that are reasonably free from error and bias and faithfully represent what they purport to represent. Another attribute involves management evaluating both internal and external sources of data for reliability, and obtaining data on a timely basis so that they can be used for effective monitoring.

OPM Has Not Made EHRI Payroll Data Widely Available Even Though They Have Potential to Support OPM’s Strategic and Open Data Goals

Use of the EHRI payroll database has been limited because OPM has not made it widely available. This is in contrast to other, related OPM datasets, such as the EHRI personnel database, which OPM has prepared for use and made publicly available through multiple mechanisms including FedScope, an online tool for data analytics. Because the EHRI payroll database has potential to be used for accountability, research, and data-driven human resource management and policy decision making, making it available would support OPM’s strategic and open data goals.

Unlike Other, Related Databases, OPM Has Not Made the EHRI Payroll Data Widely Available

The EHRI payroll data have rarely been used since the database became operational in 2009. We identified four instances where the data have been used, primarily by OPM or by GAO to respond to Congressional requests for information. Specifically, (1) OPM used EHRI payroll data to calculate rough estimates of official time—paid time that employees spend on union-related activities—for its 2009 to 2012 reports to Congress; (2) we made similar use of EHRI payroll data to estimate use of official time in selected federal agencies in a 2014 report, which revealed limitations in OPM’s method of estimating the governmentwide costs of official time; (3) we used EHRI payroll data in a 2014 review that found inconsistencies in how agencies recorded and reported the use of administrative leave; and (4) we used EHRI payroll data in 2016 to report on the use of administrative leave at the Department of Homeland Security (DHS). In this final case, our use of EHRI data enabled a more detailed examination of DHS’s use of administrative leave and helped verify the reliability of the information obtained from DHS.

Aside from these four instances, the data remain largely unused because OPM does not make the data available to the larger research community or to federal agencies. This is in contrast to other OPM data, such as the EHRI personnel data, which has been widely used since OPM made it available. OPM has taken specific steps to make these other human resources-related data available, but has not taken any of these steps for the EHRI payroll database. For example, OPM has made the EHRI personnel database available by (1) cleaning it and preparing it for statistical analysis; (2) integrating it with other data in a repository known as the EHRI Statistical Data Mart; (3) making deidentified data—that is, data without personally identifiable information—accessible through FedScope, an online data analytics tool that draws on data in the Statistical Data Mart; (4) listing data that are available (either by download or by request) on the OPM website and on Data.gov; and (5) sharing requested data with other parties, such as think tanks and academic researchers.

Data in the EHRI Statistical Data Mart are also processed and repackaged to make them more available and usable. Specifically, data received by OPM from agencies and stored in the EHRI Data Warehouse are further processed and cleaned and placed into a format better suited for analysis. This process involves additional corrections and generates additional data elements likely to be useful for statistical analysis. These processed and prepared data are then submitted to the EHRI Statistical Data Mart, which forms the basis for FedScope. Both the FedScope analytics tool and the downloadable datasets are accompanied by documentation that clarifies the meanings of the data elements and limitations associated with the data.

OPM also makes other EHRI data available through its website and Data.gov. Established in 2009, Data.gov is administered by the General Services Administration (GSA) as a public portal for government data in accordance with the government-wide open data policy. It includes information about and links to datasets from executive branch agencies. As of September 2016, users are able to find references to the EHRI personnel and retirement data on Data.gov. From Data.gov, users can follow links to the personnel data in FedScope and request access to the retirement data. OPM also offers a suite of analytic tools for agencies to perform workforce analyses and forecasting on the data in the EHRI Data Warehouse.

Unlike the EHRI personnel and retirement data, the EHRI payroll data have not been made available in any of these ways. The documentation that accompanies the EHRI Statistical Data Mart specifically notes the absence of payroll data as a limitation, warning users that the data elements related to pay reflect only annualized rates of pay, and that employees’ actual pay may be lower or higher due to such factors as overtime or leave without pay, which would be addressed if the payroll data were integrated into the Statistical Data Mart. The four databases within the EHRI Data Warehouse were designed with linking identifiers to enable such integration. Even though this capability exists, for the past seven years, the payroll database has not been linked with other EHRI databases, is not incorporated into the Statistical Data Mart, and has been left largely unchecked and unused. Until the payroll data are made available, such as by incorporating them into the Statistical Data Mart and linking them to the other EHRI databases as designed, OPM will not be able to crosscheck data across the databases for accuracy, and the data will not benefit from the processing and preparation for statistical analysis that is performed for data in the Statistical Data Mart.

Because reliability issues are often identified during use of the data, greater use of the EHRI payroll data by other parties would also have the benefit of helping to improve and establish the data’s reliability. As noted in GAO’s Assessing the Reliability of Computer-Processed Data, past users can be valuable sources of information about the completeness, accuracy, limitations, and usability of a dataset. For example, our prior reports that utilized the EHRI payroll data uncovered reliability issues, and OPM itself discovered a reliability issue when it attempted to use the data to analyze the use of sick leave (an issue we describe later in this report).

EHRI Payroll Data Have Potential to Support Research and Analysis in Areas Related to OPM’s Strategic and Open Data Goals

The EHRI payroll data have potential to be used for research and analysis on topics related to OPM’s strategic and open data goals. In particular, EHRI payroll data include detailed information on pay, incentives, leave, work activities, telework, and other aspects of the federal workforce that could support OPM’s strategic and open data goals for data-driven research in areas such as audits and human resource analytics and decision making, according to our review of literature and interviews with OPM officials. OPM’s strategic goals call for the agency to develop and provide access to data systems that support human resources–related research and analytics both within and outside of OPM. As part of its strategic goal to serve as the thought leader in research and data-driven human resource management and policy decision making, OPM’s strategies include (1) developing data systems to support such analysis and (2) fostering partnerships with work groups, agencies, universities, and industry to access and analyze data. As part of its strategic goal to manage information technology systems efficiently and effectively, OPM’s strategies include providing greater access to human resources data and enabling data analytics to inform policy and decisions. In addition, OPM has open data goals that involve making data available and usable, in part to help ensure governmentwide accountability. In particular, OPM’s flagship enterprise information management initiative, a part of its most recent Open Government Plan, includes (1) ensuring that data are easily retrievable and highly usable for analytics and decision making, (2) promoting a culture of collaboration and partnerships with external stakeholders, and (3) releasing data to foster a broader conversation with the public by allowing third parties to conduct their own analyses and even create their own applications using OPM data.

EHRI Payroll Data Have the Potential to Support Governmentwide Accountability

The utility of the EHRI payroll data for governmentwide accountability is demonstrated in audits that have been conducted using agency-specific data. For example, the Department of Defense Inspector General (DOD IG) used agency payroll data to conduct an audit of the Defense Finance and Accounting Service (DFAS), DOD’s payroll provider. Specifically, by using agency payroll data, the DOD IG was able to identify improper payments to federal civilian employees. Improper payments occur when funds go to the wrong recipient, the recipient receives an incorrect amount of funds, or the recipient uses the funds in an improper manner. DFAS determined that payments were being made to accounts with invalid social security numbers, to employees under the legal employment age, and to multiple employees into the same bank account. These improper payments amounted to more than $15 million over a six-year period. We identified similar audits for improper payments conducted by the Small Business Administration (SBA) and the Social Security Administration Office of Inspector General (SSA IG).

Agency-specific payroll data have also supported audits of the use of retention incentives. For example, in 2011, the Department of Veterans Affairs (VA) Office of Inspector General examined retention incentives paid to VA employees in fiscal year 2010. Using agency payroll data, it found that officials responsible for reviewing and approving retention incentives did not adequately justify and document awards in accordance with VA policy. Also, VA officials did not always terminate retention incentives at the end of set payment periods. As a result of their review, the VA Inspector General questioned the appropriateness of nearly 80 percent of incentives it reviewed. These incentives totaled about $1.06 million in FY 2010.

Agency-specific payroll data have also been used to audit agencies’ reports on the amounts they have withheld or deducted from employees’ pay for retirement, health benefits, and life insurance. For example, the DOD IG, in collaboration with OPM, has checked payroll data against Official Personnel Files to assess whether withholdings from pay appear reasonable for employees in multiple departments whose pay is processed through DFAS. The Department of Agriculture IG has done a similar audit for employees in multiple departments whose pay is processed through the National Finance Center (NFC). Identifying payroll fraud, monitoring retention incentives, and assuring accuracy of withholdings are issues that can affect all agencies. The EHRI payroll database contains data elements necessary for conducting such audits, with the advantage that it includes data for all executive branch agencies, thus enabling governmentwide reviews.

EHRI Payroll Data Have the Potential to Support Human Resource Analytics and Decision Making

Our review of the literature and interviews with OPM officials suggest that key elements in the EHRI payroll data have the potential to be used to understand a variety of human capital outcomes in the federal government. We identified studies that could have benefitted from the availability of the EHRI payroll data to examine (1) the relationship between demographic characteristics and pay disparities and costs of compensation; (2) the use of flexibilities, such as telework, on employee retention and motivation; and (3) the use of various types of leave. Because the EHRI payroll data were unavailable, the studies we identified tended to make use of data that were less precise, less directly relevant, or less comprehensive than the EHRI payroll data. These studies illustrate some of types of research that could be done more precisely or more comprehensively if EHRI payroll data were made available.

According to our review of the literature, EHRI payroll data could also inform studies of disparities in pay among demographic groups in the federal workforce and enable more precise analysis of the costs of federal compensation. For example, a 2015 article in the Journal of Public Administration Research and Theory compared long-term professional mobility between federal employees who received veterans’ preferences and those who did not. To measure mobility, the study relied on employees’ General Schedule (GS) grade levels from OPM’s Personnel data system. A 2012 study in the Internal Review on Public and Non- Profit Marketing examined the relationship between performance-based pay initiatives and discrimination complaints in selected federal agencies using agency data related to equal employment opportunity records. A 2009 study in the American Review of Public Administration examined potential drivers of the narrowing pay gap between men and women in the federal government, including changes in seniority, differences in fields of study, and women’s migration into traditionally male fields. To measure the pay gap, these researchers relied on data on employees’ self-reported annual salary.

Although these studies of disparities provide insight into the impact of human capital decisions concerning hiring and pay, they had to rely upon GS grade levels, administrative data, or average annual salary levels to assess outcomes. Each of these measures has limitations that could have been addressed if EHRI payroll data had been available. This is because the EHRI payroll data were designed to provide standardized, governmentwide information by pay period regarding actual pay, incentive pay, telework and leave hours, and numerous other data elements related to federal work activities and compensation. The data used in the three studies, however, did not provide precise measures of compensation because pay ranges overlap grade levels in the GS system and individuals with different grades can receive similar pay rates. In addition, approximately 30 percent of federal employees are not covered by the GS system and would be excluded from such assessments by default. Available measures of annualized salary used in these studies are also imprecise. An employee’s actual earnings may include other forms of pay (for example, overtime or shift differentials) not included in adjusted basic pay, or may be less than the annualized rate because of the employee’s work schedule (for example, less than full time non-seasonal) or individual circumstances (for example, leave without pay). Also, incorporating administrative records of pay into analyses can be challenging because methods of collecting and reporting otherwise similar payroll information varies significantly across federal agencies. In contrast, if EHRI payroll data had been available for these studies, they could have addressed some of these limitations because the data are designed to reflect actual pay, including any special pay, overtime pay, or other incentives and awards an individual might receive each pay period. They are also centralized and designed to be consistent across agencies.

Studies of the impact of workforce flexibilities, such as telework, on employee retention and motivation also demonstrate potential uses of the EHRI payroll database. A 2010 article in Public Manager described how agencies, such as the U.S. Nuclear Regulatory Commission, have used telework to improve retention of employees with critical skills. Similarly, a 2013 study published in the American Review of Public Administration found that federal employees who engaged in frequent or infrequent telework were no more likely than their counterparts who do not telework to express an intention to leave, while a 2012 study in the same journal concluded that employees at the Department of Health and Human Services who telework were not significantly more motivated than those who choose not to telework. Instead of using time spent teleworking drawn from time and attendance records, both of these studies relied on data from the Federal Employee Viewpoint Survey, in which federal employees self-report whether they engage in telework “frequently” or “infrequently.” These studies would have benefitted from EHRI payroll data, which include telework fields that, if reliable, could yield more precise findings by allowing researchers to use the actual number of hours or days employees teleworked per pay period, rather than employees’ generalized descriptions of their telework frequency.

The telework fields in the EHRI payroll data could also help OPM to meet statutory requirements to monitor and report on governmentwide use of telework, and OPM has recently issued a memo to agencies indicating that it will start using the payroll data to do so. In our 2012 report on OPM’s ability to meet this requirement, we found that estimates of telework among federal employees were limited to data calls to agencies because some agencies did not track telework in their time and attendance systems. In that report, we concluded that the accuracy of telework participation and frequency data for some agencies was questionable. The EHRI payroll reporting requirements now include data elements for continuous and episodic telework. The availability and use of these data elements for analysis would allow for more accurate and efficient assessments to meet statutory reporting requirements, and would be of use to policymakers.

Our review of literature also indicates that the EHRI payroll data are potentially useful for analyzing the use of leave. OPM officials told us that they would use the data to analyze use of sick leave and annual leave across the federal government if they had sufficient resources. In the past, officials conducted such analyses by requesting data on an ad hoc basis from agencies, but, according to OPM officials, that process was too resource-intensive to continue. Our review of recent studies suggests that researchers share OPM’s interest in these topics. For example, a 2015 study in the Journal of Occupational and Environmental Medicine used time and attendance records from an unidentified federal agency to examine sick leave use among different demographic groups. Using survey data, another study examined the impact of different office designs on sick leave use among Swedish workers, finding that open office plans were associated with significantly higher reported rates of sick leave use. However, research on trends in leave use and impacts of certain policies on leave use across the federal government has been limited in the past by a lack of comprehensive and standardized leave use data, which could be addressed if EHRI payroll data were made available to agencies and researchers.

Collectively, these studies demonstrate the value of fields within the EHRI payroll database—such as pay, telework, leave, and other compensation data—in assessing human capital outcomes. However, in all of the studies we identified, researchers relied on proxy, annualized, or self- reported measures, as opposed to actual measures of pay, telework, leave, and other key variables. Further, researchers typically relied upon data covering a limited number of federal agencies or employees. Compared to the EHRI payroll data, these sources do not provide governmentwide data or the same level of precision or detail for assessing policy outcomes among federal employees, which can affect the results of analysis. Studies relying on annualized salary may over or understate actual compensation given the timing of personnel actions, such as hires, separations, promotions, and leave, which can affect the actual amount of pay employees receive in a year. Studies relying on information about a small subset of the federal workforce may not provide reliable insights about overall federal human capital trends or policy effects. We and others have noted the importance of appropriate methods and data in comparing benefits and wages among federal employees and their private-sector counterparts.

Other data sources that have been used instead of the EHRI payroll data—such as OPM’s EHRI personnel database and the Federal Employee Viewpoint Survey (FEVS)—also have limitations that could be addressed if the payroll data were made available. For example, the EHRI personnel database does not contain information on the amounts of time spent on sick leave, annual leave, administrative leave, official time activities, and telework, among other variables relevant to compensation and time use studies. OPM’s FEVS—a governmentwide database of federal employee perceptions on their agency’s policies and practices— contains data elements related to pay, but limitations on the reliability of these data elements have been identified.

In addition to the specific studies we reviewed in detail, we identified hundreds of articles on topics that correspond to EHRI payroll data fields. For example, we identified 276 articles with the phrases “administrative leave,” “annual leave,” “court leave,” “family leave,” “medical leave,” “military leave,” or “unpaid leave” in their titles that have been published in peer-reviewed journals since 2009, when OPM launched EHRI. In addition, we identified 37 peer-reviewed studies with the term “telework” in their titles and six with the phrase “performance- based pay” in their titles. Although an in-depth assessment would be necessary to determine the reliability of individual fields for any of the specific purposes noted above, our basic reliability testing suggests that several key fields in the EHRI payroll data are reasonably complete and contain data within expected ranges—and therefore would have potential to support research on these topics if the EHRI payroll data were made available. (See appendix I for more detailed results of our electronic tests of EHRI payroll data reliability.) As long as the EHRI payroll data remain unavailable, federal pay and work-related research will be limited and OPM will continue to miss opportunities to support its strategic and open data goals.

Weaknesses in Internal Controls Complicate OPM’s Ability to Leverage EHRI Payroll Data in Support of Its Mission

OPM has designed and implemented some control activities to ensure the reliability of EHRI payroll data, but weaknesses in these controls limit OPM’s ability to fully leverage these data in support of its mission. As described earlier, we assessed OPM’s internal controls on the payroll data against two of the five elements of the Standards for Internal Control in the Federal Government: control activities and monitoring. Control activities are the actions management establishes through policies and procedures to achieve its objectives, including appropriate documentation of internal controls. Control activities help agencies ensure the reliability of data within information systems, such as the EHRI payroll system. Monitoring is necessary to promptly resolve the findings of audits and other reviews so that corrective actions necessary to achieve objectives are taken in a timely manner. A deficiency exists when the design, implementation, or operation of a control does not allow management or personnel to achieve control objectives or address related risks.

Internal Control Weaknesses Increase the Risk of EHRI Payroll Data Reliability Issues

While OPM internal controls provide some assurance of the reliability of EHRI payroll data, weaknesses in the design or implementation of certain control activities and monitoring controls for the EHRI payroll database increase the risk of reliability issues that may limit OPM’s ability to fully leverage the data in support of its mission. Specifically, (1) weaknesses in control activities have resulted in limited quality checks and acceptance of unreliable data into the EHRI payroll database; and (2) weaknesses in monitoring activities have resulted in failure to address these reliability issues and increased risk that these issues may compound over time. Table 1 lists these control components and related activities, along with an assessment of whether they provide reasonable assurance of OPM’s ability to achieve its objectives in these areas.

OPM guidance includes requirements for automated controls in support of data quality, such as defining data parameters and tolerances, identifying data errors, checking for completeness, and taking corrective actions when necessary. According to EHRI documentation and OPM officials, automated edit checks are performed by the data system software to check the validity of individual data elements, the proper relationship of values among associated data elements, and data format specifications. The rules check the value and format of fields, including record identifying fields, such as birthdate and agency, as well as non-record identifying fields, such as hours of leave. Specifically, they check to make sure that fields are formatted as numbers, dates, or text, depending on the designed content of the field, and that all values in a field fall within a defined range of possible values. Further, OPM applies three relational edits to ensure (1) that actions taken to add an employee to the system are not associated with an employee already in the system, (2) that actions taken to correct a record are associated with an existing record, and (3) that actions taken to delete a record are associated with an existing record. See table 2 for a description of the fields that are checked, the rule that is applied, and the action taken if the rule identifies an error.

Data that fail OPM’s automated checks are considered errors, and the edit rules for the EHRI payroll system specify that data with error rates greater than 3 percent will not be accepted. However, according to OPM officials, the payroll data enters the EHRI Data Warehouse with very few other edits. OPM officials noted that they intend to define additional edits that could be applied to the payroll data during the data loading process. Federal standards for internal control state that management should design control activities to achieve objectives and respond to risks. However, due to the limited nature of these edits EHRI payroll data have a higher risk of data reliability issues that may limit OPM’s ability to fully leverage the data in support of its mission. For example, the results of our electronic testing of data from 2010-2015 found fields with missing data, logical errors, and out-of-range values. (For selected results of electronic testing of the data, see appendix I.)

OPM’s automated rules also require the system to check the number of records for each agency every pay period, and are designed to reject submissions and generate an automated report for the service centers when the number of records has changed by more than 5 percent. While the reports are generated, OPM officials told us that resource constraints preclude them from having the same level of controls in place for the payroll data as they do for other EHRI data, including lack of capacity to follow up on missing payroll submissions. Federal standards for internal control state that management should evaluate information processing objectives to meet the defined information requirements, such as for completeness and accuracy. However, the EHRI payroll system accepted multiple submissions of data that should have been rejected by this rule. Specifically, our testing of the data found that, for nine separate pay periods in fiscal year 2014, payroll data records for agencies contained less than 1 percent of the affected agency’s total civilian workforce. In all, 17 of the 24 CFO Act agencies were affected by this problem at least once in fiscal year 2014 (see table 3). Without these data, government- wide analytics that cover any of these impacted dates will be similarly limited and incomplete.

Although these submissions should have been flagged and rejected by OPM’s edit check for having a greater than 5 percent change in the number of records from one pay period to the next, OPM was unaware of the missing data until we identified the problem. This was due, in part, to inadequate monitoring controls, which are described in more detail below. In addition, while OPM designed these control activities to meet requirements for completeness and accuracy, the controls have not always met their objective and therefore do not provide sufficient assurance that completeness requirements will be achieved. Failing to evaluate information processing objectives to see if they meet the defined information requirements for completeness increases the risk that some EHRI payroll data will be unreliable.

Control Activities: Controls on User Access

OPM has established roles and responsibilities for users and other controls safeguarding accountability for data quality and security, and maintains information on data access and use activity. As designed, these user controls are intended to provide reasonable assurance that control objectives will be achieved if OPM monitors them. According to OPM officials, EHRI payroll database users must complete an application to gain access and service provider points of contact are given access credentials once access forms are submitted and approved. Applications of users requiring administrative privileges on information system accounts receive additional scrutiny. In addition, OPM documentation establishes processes for updating the list of authorized users when accounts are created, deactivated, or deleted—for example, specifying that account passwords are to expire after 60 days and that accounts that are inactive for 60 days are to be deactivated.

OPM has also designed controls to capture and save some metadata tied to data loading and data provider submissions and user access. Automated processes capture metadata on user access and those logs are stored in system audit tables which are archived on a monthly basis and retained indefinitely, according to OPM officials. Logs detailing access for the three most recent months are available online. Within the data warehouse program where payroll data reside, applications have auditing functionality for user activity which captures what the user did as well as what was accessed. Web application activity is also tracked, and logs are retained indefinitely. Reports issued to providers can be reconstituted in real time and this information can be used for investigation. However, OPM officials told us they have not used this information for such investigations or reviews. As a result of this incomplete implementation of access controls, OPM does not know whether these controls are working appropriately.

Control Activities: Appropriate Documentation of Transactions and Internal Controls

The two primary sources of documentation that guide submissions of EHRI payroll data into the system—the Guide to Human Resources Reporting (GHRR), and the Guide to Data Standards Part B (GDS)—are not up-to-date and do not provide sufficient assurance that control objectives will be achieved. OPM relies on the service centers and agencies to assure the accuracy of payroll data submissions, as outlined in these documents. For example, the GHRR outlines each data element, its required format, and whether it must be included in the database. The GDS outlines the required format for submissions to EHRI, including the file content, notification of transmission to OPM, file naming conventions, and transmission frequency. The GDS also acknowledges that the edits outlined for service centers and agencies in that document constitute the minimum required level of quality control and encourages agencies to supplement them based on the specifics of their internal programs and operations. The GHRR was last updated July 2013 to reflect the inclusion of telework variables, but the GDS has not been updated since March 2012. Given the changes to the system and the control weaknesses noted above, OPM officials noted that all payroll data standards will have to be reworked to ensure they are robust for data collection and programming by the payroll providers. OPM officials did note their intention to update these guides to align with system, regulatory, and other changes, but did not have a detailed plan or timeframe for doing so. Federal standards for internal control state that management should document internal controls to ensure that all transactions, documentation, and records are properly managed and maintained. OPM cannot ensure that the data quality control changes made to the system are fully implemented without updating its guidance documents. Out-of-date documentation does not provide sufficient assurance that control objectives will be achieved. Until OPM updates this documentation, it faces increased risk that data submissions will not be consistent with current requirements and recent changes to the system, which could affect the reliability of data submissions.

Monitoring: Ongoing Monitoring During Normal Operations

As described above, the EHRI payroll system is designed to reject data and produce data quality reports when data error rates exceed 3 percent or when the number of records at an agency changes by more than 5 percent from one pay period to the next. According to OPM officials, biweekly EHRI data quality control reports and error files are made available to payroll providers on the EHRI portal. This quality control reporting is kept in the EHRI Data Warehouse indefinitely and the quality control reports issued to providers can be reconstructed. The GHRR directs payroll providers to monitor these reports for deviations from previous norms and analyze them to identify potential issues in systems that gather and send EHRI data from the agency to OPM. OPM officials told us that they do not monitor these reports to identify and resolve problems, and resource constraints prevent the agency from following up with the payroll providers. While inconsistent implementation of control activities allowed incomplete data to be accepted by the EHRI system, this limitation in monitoring controls led these incomplete submissions to remain undetected and unaddressed by OPM. In addition, without timely review and correction of problems identified in these reports, OPM risks errors compounding with each biweekly data submission, as the error tolerance checks involve comparison of each new submission to the most recent submission, which itself may have been incomplete. Further, without timely identification and correction of such problems, missing data may not be recoverable. For example, in response to the missing data issue noted above, OPM contacted the relevant service centers to locate the missing files. However, service centers only retain data for 18 months from the original date of submission. If the controls were working as designed, the service center would have been required to provide a corrected submission before the end of this retention period, and OPM would have reasonable assurance that the data for these pay periods were complete. Because of the delay in identifying this error, when OPM finally did request the data from the service centers, corrected data submissions were expected to require a significant amount of work because the retention period had passed. OPM was unable to provide the data within the time frames of this engagement and it is unclear whether OPM will be able to retrieve the missing data from the relevant service centers. Federal standards for internal control state that management should establish monitoring activities for the internal control system and evaluate the results, and should remediate identified internal control deficiencies on a timely basis. Without appropriate efforts to review and respond to system generated reports, OPM does not have sufficient assurance that the control objective will be achieved and the risk of submissions of inaccurate or incomplete data is increased.

Data Reliability Issues Limit OPM’s Ability to Fully Leverage EHRI Payroll Data in Support of Its Mission

While OPM’s internal controls provide some assurance of the reliability of some of the EHRI payroll data, the weaknesses in control activities (controls for completeness, accuracy, and validity of information processing and appropriate documentation) and monitoring controls (ongoing during normal operations) may increase the risk for data reliability issues to arise and persist in the EHRI payroll data. We have also identified several data reliability issues through electronic testing of EHRI payroll data, in past GAO work, and through interviews with OPM officials. Collectively, these issues present challenges for fully leveraging the EHRI data, and may limit OPM’s ability to utilize the data for some analyses in support of its mission and strategic goals.

We found a variety of potential data reliability issues from our electronic testing of EHRI payroll data, as illustrated in appendix I. In some cases, these issues indicate the potential for reliability issues that may limit OPM’s ability to fully leverage the data in support of its mission. For example, we found that the EHRI payroll data includes records for six entities that should not be in the system due to exemptions from OPM reporting requirements, as shown in table 4. When using EHRI payroll data, the unintentional inclusion of these entities could impact some analyses and limit OPM’s ability to draw valid conclusions from the data.

We also found a small number of instances of social security numbers being assigned to multiple EHRI records, as shown in table 5 below.

When using EHRI Payroll data, this could indicate that some individuals appear in the data more than once, potentially impacting some analyses and limiting OPM’s ability to draw valid conclusions from the data.

In a 2014 report, we found that weakness in OPM’s documentation for transactions and internal controls led to inconsistent reporting of administrative leave data and inclusion of some excepted agencies’ data in payroll feeds. Specifically, in our report of agencies’ use of administrative leave we found differences between agencies’ leave recording practices and what OPM officials consider paid administrative leave. In response, OPM issued guidance to agencies to review how they record administrative leave and clarify that administrative leave should not be routinely used for an extended time. This guidance can help agencies and payroll providers to provide more consistent data on administrative leave, and improve the usefulness of EHRI payroll data for related analyses.

OPM officials also told us about data reliability issues beyond those identified in this review. For example, OPM officials told us that, in 2015, they discovered a problem related to data on sick leave. Specifically, due to a programming error, the data received from payroll providers that sum the number of sick leave hours an employee used in a year was populating an unrelated field in the EHRI payroll database. As a result, according to OPM officials, the amount of sick leave an employee used in any given year was not accurate. One of these officials told us that this problem may also apply to other variables. This suggests that OPM’s edit checks, which were designed to maintain a minimum level of quality control, may not sufficiently reduce the risk of these types of errors.

OPM officials told us they had plans to update EHRI security protocols, payroll documentation, testing for reliability issues, and data standards, but OPM has not documented these plans or created a schedule to implement them. For example, OPM officials told us that they planned to update their documentation beginning in FY 2016, working collaboratively with OPM’s program policy office, federal agencies, and shared service centers, and other stakeholders. Although the EHRI payroll data was not a part of the 2015 OPM data breach, agency officials told us that the agency is evaluating its current security posture and making necessary changes to protect the privacy and integrity of all the data they manage. According to OPM officials, these plans include preparing to deploy a new secure portal to applications and tools; improving use of encryption; masking and redaction when appropriate and prudent; consolidating data from multiple data sets into more secure databases; utilizing better and more secure user management tools and audit trail logging; and providing new forms of user authentication, among other potential security and access measures.

OPM officials also said they are planning to correct the issue they had identified with the sick leave variable and that they were in the process of testing other variables to see if they had the same problem. However, OPM officials told us that these actions would require resources and reprioritization of the existing workload, and that a project plan and timeframes had not yet been developed. Further, OPM officials noted that the agency has a critical leadership role in addressing the complete data life cycle, and that agencies and service centers also play a critical role in assuring data quality. Accordingly, OPM officials said they were seeking a comprehensive solution that includes agency and service center actions to ensure accurate data are submitted to EHRI.

As yet, OPM has not linked EHRI payroll correction activities back to specific agency objectives or created a schedule for implementing these changes. GAO’s schedule assessment guide notes that a well-planned schedule is a fundamental management tool that can help government agencies gauge progress, identify and resolve potential problems, and determine the amount and timing of resource needs. Without a well- planned schedule—developed in consideration of how it will contribute to OPM’s objectives and risks—OPM may not be able to appropriately prioritize and execute the necessary changes.

Conclusions

Although use of EHRI payroll data has been limited to date, it carries significant potential to support governmentwide accountability and human resource analytics and decision making. In our review of peer-reviewed journals we identified hundreds of articles on topics that correspond to EHRI payroll data fields. Unfortunately, however, EHRI payroll data will continue to be underutilized until—consistent with its own strategic and open data goals—OPM makes the data available to potential users, as it does other databases within the EHRI system. While data collection and storage is not without cost, EHRI’s centralized, standardized, and comprehensive features offer the promise of efficient, cost effective, and more precise analytics. In preparing the data to make them available, OPM will need to take steps to process and clean them as it does for the EHRI personnel data. This is the first step toward improved reliability. Our basic reliability testing suggests that several key fields in the EHRI payroll data are reasonably complete and contain data within expected ranges— and therefore could have potential to support research on these topics.

However, while some fields in the current EHRI payroll data may be sufficiently reliable for certain types of audits and workforce analytics, other fields suffer from reliability issues that limit the range of purposes for which the data can be used. This is because OPM has not designed sufficient control activities to assure data quality, has not evaluated or consistently implemented the control activities it has designed, and has not updated key documentation to support quality submissions of data. Compounding these problems is OPM’s failure to monitor ongoing operations, for example, by reviewing system generated reports. Without timely identification and correction, data quality problems will continue undetected and remain uncorrected. While OPM officials noted their intention to address these shortcomings, they do not have plans with specific actions and time frames for doing so. Without a schedule specifying when these planned changes will be made, OPM officials will be unable to gauge progress, identify and resolve potential problems, or determine the amount and timing of resource needs related to the desired changes. As a result, OPM faces an increased risk of implementing ineffective or contradictory changes, and of facing delays in completing these activities. Until relevant changes are made, existing problems can continue to compound as data for 2 million federal civilian employees are received biweekly. Without available and reliable payroll data, OPM and others must continue to rely on data that are more costly, imprecise, or limited in scope—missing opportunities to leverage centralized, standardized data that is essential for accountability and well-informed management and policy decisions.

Recommendations for Executive Action

GAO is making five recommendations to the Director of OPM.

GAO recommends that the Director of OPM take the following action to support its strategic and open data goals: Improve the availability of the EHRI payroll data—for example, by preparing the data for analytics, making them available through online tools such as FedScope, and including them among the EHRI data sources on the OPM website and Data.gov.

GAO recommends that the Director of OPM take the following two actions to improve internal controls for data quality:

Update EHRI payroll database documentation to be consistent with current field definitions and requirements, including the Guide to Human Resources Reporting and the Guide to Data Standards, Part B; and

Consistently monitor system-generated error and edit check reports and ensure that timely action is taken to address identified issues.

GAO recommends that the Director of OPM take the following two actions to integrate the payroll data into the larger suite of EHRI databases:

Develop a schedule for executing these plans; and

Evaluate existing internal control activities and develop new control activities for EHRI payroll data, such as implementing transactional edit checks that leverage the information in the other EHRI datasets.

Agency Comments and Our Evaluation

We provided a draft of this report for review and comment to the Director of OPM. OPM agreed with our recommendations. In its comments (reproduced in appendix III), OPM noted that a lasting and effective solution for enhancing the quality of payroll data requires consistent data quality not just in the “last mile” after delivery to the EHRI system, but also at the origination of the data. OPM also noted that implementation of these recommendations will require collaboration between various stakeholders and appropriate resources. We agree. As we note in the report, while payroll processing is more consolidated than in the past, agencies still use a variety of time and attendance (TA) systems, which can vary in the level of detail with which work or leave time is recorded depending on agency policies and systems. In addition, there are variations in the systems and processes of the payroll providers. These variations across agencies and across payroll providers underscore the importance of updated documentation for reporting and consistent monitoring of error reports. In addition, through its leadership role in the OPM-managed Human Resources Line of Business, OPM can consider action for ensuring data quality—for example, by including data quality indicators among its performance measures for the payroll providers. OPM also provided technical comments which we incorporated, as appropriate.

We will send copies to the appropriate congressional addressees and the Director of the U.S. Office of Personnel Management, as well as to other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2700 or KingsburyN@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. The names of GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Selected Data Reliability Testing Results

This Appendix presents selected results from our electronic testing of the EHRI payroll data. The results are grouped into three categories: (1) tests for missing data, (2) tests for logical errors, and (3) tests for potentially invalid values. 1. Missing Data Tests Incomplete data can limit both the ability to conduct desired analyses and the usefulness of any analysis conducted. For example, if a large amount of data is missing, as was the case with data missing for entire agencies for some pay periods, it may not be possible to complete analysis of that agency for the missing periods. If a smaller proportion of data is missing, analysis may still be possible. However, any analysis completed using these data will be limited in its accuracy and validity, which may increase the risk of drawing inappropriate or invalid conclusions.

Because the EHRI payroll data have been identified as potentially useful for government-wide studies of telework behavior, we tested for missing data in telework-related fields. As shown in table 6, we found data in these fields to be missing entirely in 2011 and largely incomplete in 2012, years for which reporting on this variable was not required by OPM. The percentage missing is based on the number of records without values out of all records within a fiscal year. As a result, estimates of governmentwide telework participation, as shown in table 7, are likely to be inaccurate for these years. 2. Logical Error Tests Logical testing can reveal data reliability issues among and within individual records. For example, logical testing can assess whether there are duplicates among records in the data system. Our electronic testing assessed whether there were duplicate records in the EHRI payroll data. We also looked for records with multiple payments, either from the same agency or from different agencies, in a single pay period—a form of duplication that could also indicate problems with data reliability. As shown in tables 8, 9, 10, and 11, we found no instances of a complete duplicate records, few instances of EHRI IDs associated with more than one social security number, and that generally less than 1 percent of records were associated with multiple payments in a single pay period.

Logical testing can also uncover data reliability issues within individual records. As shown in table 12, we found a number of variables with questionable values, given the values of other variables for the same record. For example, we found instances where the amount of annual leave used was greater than the amount available, and other instances where the data indicate an agency contribution to Federal Employee Group Life Insurance (FEGLI) for an employee who has not made a contribution. In both cases, this should not be possible under typical circumstances, and may indicate a data reliability issue. 3. Outlier and Out of Range Tests Tests for invalid formats and values can reveal obvious errors about data. For example, as shown in table 13, we tested the format of the social security numbers (SSN) in EHRI, which should all be nine digit numbers, and found some cases where these numbers were not properly formatted, indicating a potential data reliability issue that could prevent analysis of individuals when attempting to match SSNs.

We also tested the values of fields in the EHRI payroll data to determine whether any records were outside of the expected ranges. As shown in table 14, we found several variables where the minimum value was below the expected possible floor or the maximum value was above the expected ceiling. For example we found a maximum value for Student Loan Repayments of $100,000, which is well above the expected ceiling for such payments. We also found that the lowest minimum value for total salary in a pay period was negative $99,140, while all salary values should generally be positive for normal records (non “correction” records).

Appendix II: Standards and Methods for Data Reliability Assessment

Data reliability assessments—a process consistent with internal control standards—gather and evaluate the information needed to determine whether data can be used to answer specific research questions. In this context, reliability means that data are reasonably complete and accurate to answer the intended questions that OPM, agencies, policy organizations, and academics might have about the federal workforce. Reliability assessments are specific to the context of the particular characteristics of the research project and the risk associated with the possibility of using insufficiently reliable data. Errors are considered acceptable if the associated risk has been assessed and a conclusion reached that the errors are not substantial enough to cause a reasonable person, aware of the errors, to doubt a finding, conclusion, or recommendation based on the data. To determine whether data are sufficiently reliable for a specific research purpose, one must consider the expected importance of the data in the final report; corroborating evidence; level of risk of using the data; and the results of assessment work conducted to date.

Completeness, accuracy, and validity are all components of reliability.

Completeness refers to the extent to which relevant records are present and the fields in each record are populated appropriately. For example, are the payroll records for all on-board employees at an agency recorded for every pay period within the calendar year?

Accuracy refers to the extent to which recorded data reflect the actual underlying information. For example, do the recorded hours of annual leave in an employee’s payroll record accurately reflect the number of annual leave hours they reported in their time and attendance form?

Validity, for the purposes of this report, refers to whether the data actually represent what is being measured. For example, if we are measuring the extent of overtime in the federal government and we use a field that records a certain type of administratively uncontrollable overtime, does that represent the extent of overtime use or might there be other ways overtime is recorded?

Data reliability assessments as a process include (1) reviewing existing information about the data and conducting interviews with officials from the entity or entities that collect the data, (2) reviewing selected system controls, and (3) performing tests on the data, such as advanced electronic analysis and tracing to and from source documents.

Appendix III: Comments from the Office of Personnel Management

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgements

In addition to the contact named above, the following individuals made important contributions to this report: Sidney Schwartz, Director; Rebecca Shea, Assistant Director; Russ Burnett; Steven Putansu; David Blanding, Hiwotte Amare; Joanna Berry; Amy Bowser; Tim Carr; Melinda Cordero; Sara Daleski; Lorraine Ettaro; Dani Greene; Donna Miller; Laura Pacheco; and Jeffrey Schmerling.