Federal Telework: Program Measurement Continues to Confront Data Reliability Issues
Released: 2012-04-19T00:00:00

Letter

Background

The Telework Enhancement Act of 2010, enacted in December 2010, requires each executive agency to designate a telework managing officer, develop training programs, establish a telework policy, and submit an annual report to the Chair and Vice Chair of the Chief Human Capital Officers Council on the agency’s efforts to promote telework. Under the act, OPM is to play a leading role in helping executive agencies implement the new telework provisions, which include setting telework goals and establishing qualitative and quantitative measures. The law requires OPM to provide policy and guidance for telework in several areas, including pay and leave, agency closure, performance management, official worksite information, recruitment and retention, and accommodations for employees with disabilities. In an annual report to Congress, OPM is to assess each agency’s progress toward goals for participation and other goals relating to telework, such as emergency readiness. The first of these reports under the act is due to Congress in June 2012.

Since 2002, OPM has used a telework survey—the data call—to annually collect information from the executive agencies in order to provide Congress with a report on the status of telework across these agencies. OPM conducts this data call to determine the extent to which agency employees are teleworking and to gauge agency progress in various aspects of their telework programs, such as participation, policy, eligibility, cost savings, and technology, as well as to provide examples of barriers agencies face to implementing telework programs.

However, throughout the past decade, OPM has been concerned about the reliability of the telework data it receives from executive agencies because, although data reported from agencies have improved, OPM continues to consider it an estimate of telework participation and frequency. In its 2003 and 2007 telework reports to Congress, OPM raised concerns about the ability of agencies to track employee participation in their telework programs. In its 2008 report, OPM identified weaknesses in the methodology most agencies used to collect and report telework participation data and OPM stated that inconsistencies within data systems and inaccuracies triggered by hand- counting telework agreements could affect data reliability. OPM cautions that existing measures of telework participation are a barrier to measuring any increase in telework as the measures vary widely in validity and reliability and limit the capability of any federal body to track the actual level and frequency of telework participation.

At the request of Congress, we have previously reported on telework programs across the federal government and have made recommendations related to the reliability of agency-reported data. In a 2005 report, we reviewed the telework data for five federal agencies and found they had reported the total number of employees who were eligible to telework, but had included individuals who were, in fact, excluded from participation based on various criteria such as employee performance, thereby raising concerns about the reliability of the telework data reported by these agencies. In addition, none of the agencies could report the actual number of employees who teleworked or how often they did so because none had fully implemented the capability to track this through their time and attendance systems. Our 2007 testimony reiterated our concern that agencies were measuring employee participation in telework based on their potential to telework rather than their actual usage.

More recently, we reported that since the 2004 data call, OPM asked agencies if they had integrated telework into agency emergency and continuity of operations plans, but agencies had no guidance as to what constitutes incorporating telework into continuity and emergency planning. This lack of a definition or description raised concerns about the reliability of reports on this matter.

OPM Revised the 2011 Telework Data Call to Prepare for Reporting under the Telework Enhancement Act of 2010

In response to the Telework Enhancement Act of 2010, OPM revised the 2011 data call and provided instructions to executive agency respondents that incorporated common definitions and standards to use in providing OPM with their agency data. The revisions and additions to the 2011 data call were developed in consultation with an Interagency Telework Measurement Group (ITMG), which OPM formed in January 2011. (See app. II for a comparison of definitional and instructional changes from the 2010 and 2011 telework data calls.) The OPM official designated with leading the ITMG said 10 agency officials from 7 agencies were selected for the group because of their knowledge of federal telework programs. According to the OPM official, the ITMG provided expertise in telework program implementation, policy, and methodology development, work/life balance programs, and expertise in research methods, such as surveys. The ITMG met on biweekly from January 2011 until July 2011 and resumed biweekly meetings in September 2011 with the goal of addressing three primary topics:

Definitions of key terms, such as telework, eligibility, and employee, to use in the 2011 data call. ITMG interpreted some requirements of the act and developed additional instructions to encourage common reporting methodology across the agencies. For example, according to OPM officials, ITMG clarified the definition of eligibility in light of agencies’ concerns that the act did not specifically define the categories of employees that should be eligible to participate in their agencies’ telework programs, and therefore notified about their eligibility to telework. The group instructed respondents to ensure they excluded military and contract personnel as employees when reporting their telework data. The group also clarified that respondents should include full-time, part-time and intermittent employees when responding to questions about telework participation and frequency.

Revision and/or addition of data call questions. For example, OPM officials stated that in collaboration with the ITMG, they clarified the definition of telework to specifically state that telework includes what is generally referred to as remote work, but excludes mobile work and work done on official travel. OPM officials added a new question to capture the number of mobile workers. This addressed a reliability problem from the previous data call when some agencies included mobile work in reporting telework.

Revision and development of data collection instruments, in addition to the data call, to collect telework information. For example, ITMG worked to revise telework-related items in the Federal Employee Viewpoint Survey, an OPM data collection instrument that gauges employees’ perceptions of their agency. In the 2011 survey three out of 84 questions focused on telework. In the past, OPM has found this survey to provide complementary employee views on telework. The ITMG also assisted in developing focus groups of telework managing officers and telework coordinators to identify issues, challenges and strategies associated with implementing telework programs at the agency level, such as successful telework implementation strategies, as well as barriers to telework.

According to an OPM official, these revisions also included questions that may enable them to better understand the differences between telework programs across executive agencies, including differences in training on telework, use of technology, and how agencies responded to the requirements of the act. This official said such information will help inform the development of telework programs.

OPM also made changes to the 2011 data call time period for which employee telework participation and frequency was to be reported. This change was made to allow agencies time to develop telework policies in accordance with the act and to allow OPM time to meet its reporting obligation under the act (see fig. 1). In previous data calls, OPM asked agencies for telework data during the calendar year (12 months), if available. OPM reduced the time period for the 2011 data call to 4 weeks, as it decided this was the best methodology to meet its reporting requirements under the act. Agencies were to select a 4 week period during September and October on which to report. In addition to the change in time period for requested telework data, the 2011 data call asked for more detail on employee telework participation, frequency of employee telework participation, and additional information on telework policy and program implementation, and telework goals as required by the act.

OPM officials believe the information they obtained from the 2011 data call will enable the agency to satisfy some of the act’s reporting requirements, but for this report to Congress, OPM cannot fulfill other requirements. OPM officials stated that it is not feasible for OPM to measure agencies’ progress against their Telework Enhancement Act of 2010 goals, since these goals were established in June 2011, and the time period for agencies’ actual participation data was September/October 2011—just 3 to 4 months after agencies established their goals. However according to OPM officials, based on the information collected, OPM will be able to report the percentage and frequency of telework at individual agencies.

To communicate changes to the 2011 data call, OPM officials increased their training efforts to aid executive agency officials in developing a common understanding of terms, key concepts, and the objectives of the data call. According to OPM officials, in July 2011, OPM officials responsible for the data call met with agency respondents to provide an introduction and overview to the 2011 data call. The meeting covered the new requirements under the act, and the planned timeframe for agency reporting and OPM processing and analysis of the data collected through the data call. OPM could not require agency officials to attend. Nonetheless, OPM reminded agency officials responsible for the data call, that it was important that they attend both September and October training sessions being offered by OPM. The September training session covered the data call questions and incorporated specific content of the near final data call. The October training session reviewed specific instructions on how to enter information into the online data call form, in addition to reviewing instructions and questions from respondents. While some of the information provided at the two training sessions was similar, each session contained some new information usually in response to questions raised at a previous session. OPM staff also maintained and disseminated via email a list of the most frequently asked questions posed by data call respondents.

Various Methods of Data Collection and Ongoing Changes to the Data Call Make It Difficult to Assure Data Reliability and to Make Year-to-Year Comparisons

The act requires OPM to report, among other things, year-to-year executive agency progress on the number of employees who telework. To accomplish this, OPM needs reliable baseline data on telework participation to be able to make year-to-year comparisons. However, OPM officials expressed continuing concerns over the reliability of quantitative participation and frequency data submitted by the agencies through the 2011 telework data call. OPM officials explained that 2011 is a transition year for telework programs across executive agencies, and some agencies made changes to their policies to bring their telework programs into compliance with the requirements of the act. For example, some agencies were implementing new data collection systems while collecting data on telework. These agencies first needed to create and implement new policies, and then consider and establish processes related to the collection of telework data to report to OPM. OPM officials responsible for preparing the report to Congress stated that because of these changes, it would not be appropriate to compare the 2011 data to data collected in prior years.

OPM will not be able to make comparisons to prior years due to the previously discussed modifications to the data call, such as changes to definitions. Definitional changes, including the change to specifically exclude mobile workers, could result in agencies interpreting and reporting telework participation and frequency of telework differently than they did in the past. While these changes should improve the consistency of data going forward, according to OPM officials there is no way that OPM can assure that all agencies would provide comparable responses to the same data call questions. Moreover, OPM officials have said that key terms and definitions for the 2012 data call may continue to evolve. If OPM reports agency telework progress based on data collected using definitions revised year to year, OPM may reach erroneous conclusions, although OPM officials have said they are taking steps to try and prevent this possibility.

Another reason why comparisons to prior data calls would be invalid is because of the changes OPM made to the time period when agencies reported telework data. Maintaining consistent data series over time necessitates using consistent data collection procedures for ongoing data collections. OPM said that changing the reporting period from 1 calendar year to 4 weeks may result in greater data consistency because in previous data calls, agencies may have reported data from different time periods within a calendar year. Now all agencies will report data from a narrower and more similar time frame. However, there are no available studies to support that the new time period is representative or “typical” of other months in total or of the experience of particular agencies. OPM officials said that they will need to indicate this in their report to Congress.

While OPM made changes to the 2011 data call to allow it to meet some of its reporting requirements under the act and better assist agencies in responding to the data call, our analysis found the 2011 data call did not fully meet some generally accepted survey standards. According to these standards, to be valid, survey questions must adequately represent the concept or behavior in question and consistently predict outcomes. Moreover, questions must be designed and asked so that each recipient will understand and answer the same question in the same way. But for the 2011 data call, there can be no assurance that all respondents were aware of associated definitions and instructions provided by the training sessions and the frequently asked questions (FAQ). Although OPM invested more in providing training in preparation for the 2011 data call than in previous data calls and it disseminated training slides to all invitees of its final training session, attendance lists were not recoverable for all training sessions so there can be no assurance that all respondents received training or reviewed the slides. Consequently, some data call respondents may not have been aware of the definitions and instructions provided in the training sessions or in the FAQ.

Additionally, some of the information provided in the training sessions was inconsistent. For example, OPM officials said that during the last training session they instructed respondents to report an employee “telework day” if the employee teleworked for any portion of a work day. However, this clarification of “telework day” was not given in either the July overview meeting or in the other training session, and not clearly included in the instructions in the survey instrument or distributed in the FAQ. This information may have been important to agencies reporting on situational telework through automated systems intended to capture more precise data. OPM officials also explained that responses to certain questions should reference the same time period, and this information was not available in the data call instructions. During one training session, an OPM official said she instructed those participants who determined employee participation by counting telework agreements to limit the agreements counted to those in effect after the agency implemented its telework program under the act; however, this information was not available in the online data call instructions. Uncertainty about whether data call respondents attended both the September and October training sessions, and variations in training sessions, could cause agencies to have different understandings of data call concepts and terms. OPM recognizes that the existing measures of federal telework participation vary in validity and reliability, which affects agencies’ ability to report accurate data, and it is taking steps to verify data submitted by respondents to provide a more accurate picture of telework in the federal government based on current definitions and collection methodologies. However, as a result of issues raised above, respondents may have provided inconsistent or inaccurate data on topics required by the act.

OPM officials anticipate that telework data will be more reliable next year because of the expected governmentwide implementation of automated data collection based on time and attendance records. As we have reported, OPM has concluded from research that the most reliable telework data are collected through time and attendance tracking systems. Data collected through automated systems eliminate the need to track telework data by counting telework agreements or relying on estimates. Since 2003, OPM has consistently expressed concerns about the methods agencies use to collect telework data. In its previous telework reports to Congress, OPM has advocated for the development of an automated data collection system. OPM officials noted that OPM does not control what telework data responding agencies maintain, or their methods of data collection. Executive agencies provide telework participation and frequency data using a variety of methods, such as relying on estimates, counting telework agreements, and using automated time and attendance records to track telework participation.

In an effort to collect more uniform data across agencies, OPM officials are standardizing definitions and data elements for use in automated time and attendance systems. For example, OPM has identified routine telework hours in a pay period as a data element for automated data collection and provided a standardized definition that will be used by all agencies using the Enterprise Human Resources Integration (EHRI) system. OPM has introduced a timeline for modifying the existing EHRI system to allow OPM to collect telework data from executive agencies. According OPM, agencies will begin piloting these automated data collection systems for the 2012 telework data call. OPM began to discuss automated governmentwide data collection with the ITMG in July 2011. According to OPM’s timeline, OPM began to communicate internal requirements for automated telework data collection to telework managing officers in March 2012, but an OPM official has stated they do not expect full automation of telework data until 2013. This official also noted that different agencies have varying abilities to implement this new type of data collection and reporting mechanism, and considering different levels of comfort with new systems, it will take time to adjust to this method. However, continuous improvement efforts sometimes result in a trade-off between the desire for data consistency and a need to improve data collection and maintenance of a consistent data series over time.

Because of the eventual planned move to automation, OPM may not be able to use 2011 data as a baseline. With the planned change to the method of data collection, it may not be possible to compare the 2011 data to future data. The 2011 data call requested data for a 2-month period, and some data call respondents relied upon estimates. Planned automation can provide a more uniform and accurate method for collecting telework data, however it may make comparisons using the 2011 telework data as a baseline difficult. However, OPM officials believe the 2011 data will provide an improved report of telework status because of standardized definitions and the more uniform time period of data collection. According to OPM officials, some executive agencies will need time to become comfortable with automated reporting systems and, during a period of transition to a new system, there could be initial reliability issues. However, these officials said that the 2011 data, notwithstanding its limitations, will be useful in identifying and understanding any major agency changes in reported participation that could occur during a transition to automated data collection. Such changes could alert OPM to possible transition related issues in agencies’ conversion to automated governmentwide data call collection efforts in 2012 and 2013. In addition, for those agencies already responding on the basis of time and attendance reporting, major changes in agency responses provide OPM the opportunity to confirm with agencies that the uniform definitions are being consistently applied.

Conclusions

The Telework Enhancement Act of 2010 requires OPM to collect telework data and report annually to Congress, which emphasizes the need for telework data to be valid and reliable. OPM revised the 2011 telework data call in order to allow it to meet some of the act’s reporting requirements and assist agencies in responding to the data call. This revision resulted in changes to terminology, as well as changes in the collection time period of requested telework data. OPM provided greater assistance to agencies through training on changes to the 2011 data call to improve accuracy of agency reporting of telework participation and frequency for the data call reporting period of September and October 2011. However, agencies use various methods, which OPM does not control, to report, collect, and maintain telework data, and this could affect the reliability of the telework data submitted. In addition, variation in training sessions and OPM’s uncertainty as to whether all respondents attended training, could lead to respondents’ potential misunderstanding of important terms and instructions. The validity and reliability of the reported 2011 telework data for some of the responding agencies may be questionable, and therefore agency telework participation and frequency data will not likely be comparable with previous data calls because of differences in definitions used, time periods of reporting, and individual agency tracking methods.

With the revised 2011 data call, OPM establishes a baseline it could use to conduct a limited crosscheck of data collected through a governmentwide automated telework data collection system, which OPM plans to implement over the course of 2012 and 2013. OPM expects that automated data collection will provide it increasingly more reliable data on which to report progress. However, these efforts to improve future automated data collection may result in changes to agencies’ methods of data collection and a trade-off between the desire for consistency with previous data calls for comparison purposes and a need to improve overall data collection.

Recommendations for Executive Action

To improve OPM’s annual reporting of telework to Congress, we recommend that the OPM Director take the following two actions:

Ensure that the reliability limitations related to the 2011 telework data call are clearly reported in its June 2012 report to Congress by fully describing how existing measures of telework participation vary widely in validity and reliability and limit the capability of OPM to reliably report the actual level and frequency of telework participation.

Continue efforts to improve data collection and gather information that allows for the appropriate qualification of year-to-year comparisons and informs users about the effects of data collection changes going forward.

Agency Comments and Our Evaluation

We provided a draft of this report to the Director of OPM for review and comment. The Associate Director of OPM provided written comments, which we have reprinted in appendix III. In summary, OPM partially concurred with our first recommendation and fully concurred with the second. OPM highlighted a number of actions the agency has under way or plans to undertake in response. For the first recommendation, OPM noted that inadequate methods of data collection exist at the agency level and OPM continues to address this data reliability issue through training on evaluation and measurement. While this is an important step in addressing data reliability issues, OPM should ensure that telework data reliability limitations are clearly reported in their annual reports to Congress. For the second recommendation, OPM noted its continued plans to automate collection of telework data, and to regularly meet with telework managing officers and telework coordinators to keep them updated on changes to telework policy and data collection. OPM also provided a number of technical comments, which we incorporated as appropriate.

We are sending copies of this report to the Chairman and Ranking Member of the Subcommittee on the Federal Workforce, U.S. Postal Service and Labor Policy, Committee on Oversight and Government Reform, House of Representatives; and the Director of OPM. In addition, this report will be available at no charge on the GAO website at www.gao.gov. If you have any questions about this report, please contact me at 202-512-6806 or jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

This report (1) describes the Office of Personnel Management’s (OPM) actions to respond to the requirements of the Telework Enhancement Act of 2010 and (2) assesses how OPM is handling and addressing identified data reliability issues in the 2011 telework data call.

To address these two objectives, we reviewed relevant reports and guidance published by OPM that describe the status of telework programs across executive agencies and previous telework data calls and their instructions. We also reviewed previous GAO reports on telework and the reliability of OPM’s telework data. Lastly, we interviewed the OPM officials responsible for the planning, design, implementation, and analysis of the 2011 telework data call. This included discussions on the role of the Interagency Telework Measurement Group (ITMG), the process for developing the definitions and key terms used in the data call, the training and assistance provided to executive agency officials responsible for completing the data call, and agency plans to address outstanding data reliability issues associated with the data call. We conducted additional analysis to answer selected objectives as described below.

To assess the extent to which the 2011 telework data call met generally accepted survey methodology, GAO internal experts in survey research identified principles from the Office of Management and Budget’s (OMB) Standards and Guidelines for Statistical Surveys relevant to assessing the 2011 telework data call. We also used relevant aspects of GAO’s guide to Developing and Using Questionnaires. Using the OMB principles, two analysts independently reviewed the data call, supporting documentation, and clarifying information provided in interviews to assess the extent to which the data call methodology met research practices. The initial rate of agreement across 15 rated practices, including those rated as having insufficient information to judge, was 12 of 15, or 80 percent agreement. The rating for the three practices on which there was initial disagreement were reconciled by the two analysts conducting the review and the reconciled rating was then reviewed by a third analyst with survey expertise. The third analyst did not recommend any changes to the reconciled ratings.

We also simulated completion of the Web-based data call, accessing and responding to the data call in the same manner as executive agency respondents.

Table 1 outlines the generally accepted survey research principles, derived from OMB’s guidelines, which we used in our assessment. This is not an exhaustive list of all OMB guidelines. When we completed this review in April of 2012, OPM had not yet completed analyzing and reporting on the results of telework data call. Based on this, we could not yet assess whether the data call met all of the OMB principles related to data analysis and reporting. Some OMB principles, such as those related to sample design, were not appropriate to apply to the telework data call. These principles were therefore excluded as not relevant to our review.

We conducted this performance audit from June 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Revisions to OPM’s Telework Data Call to Collect Information to Prepare for Reporting under the Telework Enhancement Act of 2010

Instructions

2010 Telework data call Time period: Based on the agency’s calendar year (Jan. 1 to Dec. 31, 2009).

2011 Telework data call Time period: Based on the months of Sept. and Oct. 2011.

Participation: agencies invited to participate in data call.

Participation: agencies required to submit telework data to OPM.

Definitions

Telework: “Telework refers to any arrangement in which an employee regularly performs officially assigned duties at home or other work sites geographically convenient to the residence of the employee.”

Telework: “Telework is a work arrangement that allows an employee to perform work, during any part of regular, paid hours, at an approved alternative worksite (e.g., home, telework center). This definition of telework includes what is generally referred to as remote work but does not include any part of work done while on official travel or mobile work. See the following clarifications on remote and mobile work as telework.”

Employee: Employee refers to federal civilian employees excluding military personnel and contractors. Employee may also include full-time, part-time, and/or intermittent employees.

Eligibility: An employee is eligible to participate in telework if all of the following parameters are true: positions require, on a daily basis (every work day), direct handling of secure materials, or on- site activity that cannot possibly be handled remotely or at an alternate worksite; most recent federal government performance rating of record (or its equivalent) is below fully successful or conduct has resulted in disciplinary action within the last year.

The employee has not been officially disciplined for being absent without permission for more than 5 days in any calendar year. The employee has not been officially disciplined for violations of subpart G of the Standards of Ethical Conduct for Employees of the Executive Branch. Teleworking does not diminish the employee’s performance or agency operations. Participation and performance complies with the requirements and expectations of his/her telework agreement. The employee’s official duties do not require on a FULL daily basis (ALL DAY, every work day): direct handling of secure materials determined to be inappropriate for telework by the agency head; or on-site activity that cannot be handled remotely or at an alternate worksite.

The employee and/or the employee’s position are not disqualified based on additional criteria established by the organization.

2010 Telework data call Types of telework: None provided 2011 Telework data call Types of telework:

Routine: telework that occurs as part of an ongoing, regular schedule, and Situational: telework that is approved on a case-by-case basis, where the hours worked were not part of a previously approved, ongoing and regular telework schedule (e.g., telework as a result of special work assignments or doctor appointment.)

Situational telework is sometimes also referred to as episodic, intermittent, unscheduled or ad-hoc telework.

Appendix III: Comments from the Office of Personnel Management

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, William Doherty, Assistant Director, and Keith O’Brien, analyst-in-charge, led the development of this report. Virginia Chanley, Patricia Donahue, Robert Gebhart, Jill Lacey, and Joseph Santiago made significant contributions to this report. Karin Fangman provided legal counsel. Shirley Hwang, Jessica Nierenberg, and Kathleen Padulchick verified the information in the report.

Related GAO Products

Emergency Preparedness: Agencies Need Coordinated Guidance on Incorporating Telework into Emergency and Continuity Planning. GAO-11-628. Washington, D.C.: July 22, 2011.

Human Capital: Telework Programs Need Clear Goals and Reliable Data. GAO-08-261T. Washington, D.C.: November 6, 2007.

Human Capital: Greater Focus on Results in Telework Programs Needed. GAO-07-1002T. Washington, D.C.: June 12, 2007.

Agency Telework Methodologies: Departments of Commerce, Justice, State, the Small Business Administration, and the Securities and Exchange Commission. GAO-05-1055R. Washington, D.C.: September 27, 2005.

Human Capital: Key Practices to Increasing Federal Telework. GAO-04-950T. Washington, D.C.: July 8, 2004.

Human Capital: Further Guidance, Assistance, and Coordination Can Improve Federal Telework Efforts. GAO-03-679. Washington, D.C.: July 18, 2003.