Federal Telework: Program Measurement Continues to Confront Data Reliability Issues
Published: 2012-04-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Telework Enhancement Act of 2010 requires OPM to report to Congress on the degree of telework participation at executive agencies. To meet this requirement, OPM collects information on agency telework programs through an annual survey, which it refers to as the data call. However, concerns exist about the reliability of these data.
GAO was asked to assess OPMs: (1) actions in response to the requirements of the act and (2) handling of identified data reliability issues in the 2011 telework data call. To address these objectives GAO reviewed its previous reports addressing telework data reliability, and used the Office of Management and Budgets guidance for federal surveys to review OPMs (1) plans to collect telework participation data from agencies and (2) development of a data collection instrument. GAO interviewed key OPM officials about its implementation of the 2011 data call.
section_title: What GAO Found
paragraphs: To prepare for its reporting obligations under the Telework Enhancement Act of 2010, the Office of Personnel Management (OPM) assembled the Interagency Telework Measurement Group, consisting of officials from several federal agencies, to assist in revising the telework data callthe survey OPM has used since 2002 to collect telework data from executive agencies. This group standardized key terms such as telework, employee, and eligibility to promote a common reporting methodology among the agencies. The revised telework data call also included changes to the time period for which OPM requested agencies report telework data, and included more extensive training for respondents.
Because of changes made to the data call to allow OPM to meet requirements of the act and assist agencies in responding to the data call, OPM officials believe they will be able to provide to Congress an improved report on telework in June 2012. However, these changes also mean that OPM officials will not be able to use participation and frequency data from the 2011 data call to compare to data from previous years and across agencies. OPM officials have noted that this could limit OPMs ability to report agency progress in its first report to Congress. The ability to compare with previous years is affected by:
agencies use of methods of varying reliability to collect telework data, and some agencies made changes to their data collection systems for the 2011 data call. Executive agencies provide telework participation and frequency data by relying on estimates, counting telework agreements, or using automated time and attendance records.
modifications to the data call instrument, including changes to terminology and the time period during which telework data was requested. OPM officials said they expect these changes will improve the consistency of data. But if OPM reports progress based on data collected using changing terminology and from different time periods, the agency may reach erroneous conclusions.
Participants at the two data call training sessions may not have received the same reporting instructions, and uncertainty about whether all agency respondents attended training, created a risk that some respondents may be unaware of important terms and instructions. While some of the information provided at the two training sessions was similar, each session contained some new information, usually in response to questions raised at a previous session.
Future data call improvement efforts could result in a trade-off between the desire for maintenance of a consistent data series over time for comparison with previous data calls and a need to improve data collection. According to OPM, agencies will begin piloting automated telework data collection during 2012 and 2013. OPM expects that this method of data collection will provide it more reliable data than other methods. However, these efforts to standardize methods for tracking telework data may result in changes to agencies methods of data collection. The 2011 data call, notwithstanding its limitations, will be useful to help OPM identify and understand major changes in reported participation data that could occur during a transition to automated data collection.
section_title: What GAO Recommends
paragraphs: GAO recommends that OPM (1) clearly report reliability limitations with the 2011 telework data call in its June 2012 report to Congress and (2) continue efforts to improve data collection and gather information to allow for the appropriate qualification of year-to-year comparisons and inform users about the effects of data collection changes going forward. OPM partially concurred with the first recommendation. However, GAO believes it should report limitations in its annual report. OPM fully concurred with the second. OPM provided a number of technical comments which GAO incorporated as appropriate.

section_title: Letter
paragraphs: Congress has encouraged federal agencies to expand their use of  telework by requiring that executive agencies establish telework policies,  incorporate those policies into continuity of operations plans, and provide  interactive telework training to employees and managers. The Office of  Personnel Management (OPM) has reported telework can enhance the  productivity and flexibility of federal employees; help recruit and retain a  qualified workforce; and reduce real estate, overhead, energy, and  environmental costs. The Telework Enhancement Act of 2010 established  a statutory framework for implementing a comprehensive federal telework  program.implement a policy under which eligible employees may be authorized to  telework. The act also requires OPM to report annually to Congress on  the telework programs of each agency, including the level of participation  in telework in each agency, each agency’s goal for increasing employee  participation in telework, agency progress during the reporting period  toward meeting telework goals, and reasons for significant increases or  decreases from previous year telework participation.
The act requires executive agencies to establish and to  In 2010, OPM set a high-priority performance goal of increasing by 50  percent the number of eligible federal workers who telework by fiscal year  2011. To measure agency progress toward this and agency telework  goals, OPM will use data it collects through an annual survey referred to  as the data call. However, concerns have existed about the reliability of  these data. According to OPM, in order to help agencies overcome  barriers related to telework and raise federal telework participation rates,  the agency needs reliable, systematically collected agency data.
Because of the new telework reporting requirements in the act and the  need for reliable data to assess the agencies’ progress in implementing  their individual telework programs, you asked us to assess OPM’s: actions to respond to the reporting requirements of the Telework  Enhancement Act of 2010, and handling of identified data reliability issues in the 2011 telework data  call.
To answer these questions, we reviewed relevant reports and guidance  published by OPM that describe the status of telework programs across  executive agencies, and previous telework data calls and their  instructions. We also reviewed our previous reports on telework and the  reliability of OPM’s telework data. We identified the relevant generally  accepted survey standards as promulgated by the Office of Management  and Budget and compared OPM’s design and implementation of the 2011  telework data call against these standards.
We interviewed the OPM officials responsible for the planning, design,  implementation, and analysis of the 2011 telework data call. This included  discussions on the role of the Interagency Telework Measurement Group  (ITMG), the process for developing the definitions and key terms used in  the data call, the training and assistance provided to executive agency  officials responsible for completing the data call, and agency plans to  address outstanding data reliability issues associated with the data call.  We were also able to simulate completion of the web-based data call that  allowed us to respond to the data call in the same manner as the other  agency respondents. This simulation gave us the opportunity to  determine the ease of use of the online data call instrument, and whether  there were any potential design or functionality issues that could affect  the validity and reliability of the data.
We conducted this performance audit from June 2011 through April 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Telework Enhancement Act of 2010, enacted in December 2010,  requires each executive agency to designate a telework managing officer,  develop training programs, establish a telework policy, and submit an  annual report to the Chair and Vice Chair of the Chief Human Capital  Officers Council on the agency’s efforts to promote telework. Under the  act, OPM is to play a leading role in helping executive agencies  implement the new telework provisions, which include setting telework  goals and establishing qualitative and quantitative measures. The law  requires OPM to provide policy and guidance for telework in several  areas, including pay and leave, agency closure, performance  management, official worksite information, recruitment and retention, and  accommodations for employees with disabilities. In an annual report to  Congress, OPM is to assess each agency’s progress toward goals for  participation and other goals relating to telework, such as emergency  readiness. The first of these reports under the act is due to Congress in  June 2012.
Since 2002, OPM has used a telework survey—the data call—to annually  collect information from the executive agencies in order to provide  Congress with a report on the status of telework across these agencies.  OPM conducts this data call to determine the extent to which agency  employees are teleworking and to gauge agency progress in various  aspects of their telework programs, such as participation, policy, eligibility,  cost savings, and technology, as well as to provide examples of barriers  agencies face to implementing telework programs.
However, throughout the past decade, OPM has been concerned about  the reliability of the telework data it receives from executive agencies  because, although data reported from agencies have improved, OPM  continues to consider it an estimate of telework participation and  frequency. In its 2003 and 2007 telework reports to Congress, OPM raised concerns about the ability of agencies to track employee  participation in their telework programs. In its 2008 report, OPM  identified weaknesses in the methodology most agencies used to collect  and report telework participation data and OPM stated that  inconsistencies within data systems and inaccuracies triggered by hand- counting telework agreements could affect data reliability. OPM cautions  that existing measures of telework participation are a barrier to measuring  any increase in telework as the measures vary widely in validity and  reliability and limit the capability of any federal body to track the actual  level and frequency of telework participation.
At the request of Congress, we have previously reported on telework  programs across the federal government and have made  recommendations related to the reliability of agency-reported data. In a  2005 report, we reviewed the telework data for five federal agencies and  found they had reported the total number of employees who were eligible  to telework, but had included individuals who were, in fact, excluded from  participation based on various criteria such as employee performance,  thereby raising concerns about the reliability of the telework data reported  by these agencies. In addition, none of the agencies could report the  actual number of employees who teleworked or how often they did so  because none had fully implemented the capability to track this through  their time and attendance systems. Our 2007 testimony reiterated our  concern that agencies were measuring employee participation in telework  based on their potential to telework rather than their actual usage.
More recently, we reported that since the 2004 data call, OPM asked  agencies if they had integrated telework into agency emergency and  continuity of operations plans, but agencies had no guidance as to what  constitutes incorporating telework into continuity and emergency  planning. This lack of a definition or description raised concerns about the  reliability of reports on this matter.
section_title: OPM Revised the 2011 Telework Data Call to Prepare for Reporting under the Telework Enhancement Act of 2010
paragraphs: In response to the Telework Enhancement Act of 2010, OPM revised the  2011 data call and provided instructions to executive agency respondents  that incorporated common definitions and standards to use in providing  OPM with their agency data. The revisions and additions to the 2011 data  call were developed in consultation with an Interagency Telework  Measurement Group (ITMG), which OPM formed in January 2011. (See  app. II for a comparison of definitional and instructional changes from the  2010 and 2011 telework data calls.) The OPM official designated with  leading the ITMG said 10 agency officials from 7 agencies were selected  for the group because of their knowledge of federal telework programs.  According to the OPM official, the ITMG provided expertise in telework  program implementation, policy, and methodology development, work/life  balance programs, and expertise in research methods, such as surveys.  The ITMG met on biweekly from January 2011 until July 2011 and  resumed biweekly meetings in September 2011 with the goal of  addressing three primary topics:
Definitions of key terms, such as telework, eligibility, and employee, to  use in the 2011 data call. ITMG interpreted some requirements of the  act and developed additional instructions to encourage common  reporting methodology across the agencies. For example, according  to OPM officials, ITMG clarified the definition of eligibility in light of  agencies’ concerns that the act did not specifically define the  categories of employees that should be eligible to participate in their  agencies’ telework programs, and therefore notified about their  eligibility to telework. The group instructed respondents to ensure they  excluded military and contract personnel as employees when  reporting their telework data. The group also clarified that respondents  should include full-time, part-time and intermittent employees when  responding to questions about telework participation and frequency.
Revision and/or addition of data call questions. For example, OPM  officials stated that in collaboration with the ITMG, they clarified the  definition of telework to specifically state that telework includes what is  generally referred to as remote work, but excludes mobile work and  work done on official travel. OPM officials added a new question to  capture the number of mobile workers. This addressed a reliability  problem from the previous data call when some agencies included  mobile work in reporting telework.
Revision and development of data collection instruments, in addition  to the data call, to collect telework information. For example, ITMG  worked to revise telework-related items in the Federal Employee  Viewpoint Survey, an OPM data collection instrument that gauges  employees’ perceptions of their agency. In the 2011 survey three out  of 84 questions focused on telework. In the past, OPM has found this  survey to provide complementary employee views on telework. The  ITMG also assisted in developing focus groups of telework managing  officers and telework coordinators to identify issues, challenges and  strategies associated with implementing telework programs at the  agency level, such as successful telework implementation strategies,  as well as barriers to telework.
According to an OPM official, these revisions also included questions that  may enable them to better understand the differences between telework  programs across executive agencies, including differences in training on  telework, use of technology, and how agencies responded to the  requirements of the act. This official said such information will help inform  the development of telework programs.
OPM also made changes to the 2011 data call time period for which  employee telework participation and frequency was to be reported. This  change was made to allow agencies time to develop telework policies in  accordance with the act and to allow OPM time to meet its reporting  obligation under the act (see fig. 1). In previous data calls, OPM asked  agencies for telework data during the calendar year (12 months), if  available. OPM reduced the time period for the 2011 data call to 4 weeks,  as it decided this was the best methodology to meet its reporting  requirements under the act. Agencies were to select a 4 week period  during September and October on which to report. In addition to the  change in time period for requested telework data, the 2011 data call  asked for more detail on employee telework participation, frequency of  employee telework participation, and additional information on telework  policy and program implementation, and telework goals as required by  the act.
OPM officials believe the information they obtained from the 2011 data  call will enable the agency to satisfy some of the act’s reporting  requirements, but for this report to Congress, OPM cannot fulfill other  requirements. OPM officials stated that it is not feasible for OPM to  measure agencies’ progress against their Telework Enhancement Act of  2010 goals, since these goals were established in June 2011, and the  time period for agencies’ actual participation data was  September/October 2011—just 3 to 4 months after agencies established  their goals. However according to OPM officials, based on the information  collected, OPM will be able to report the percentage and frequency of  telework at individual agencies.
To communicate changes to the 2011 data call, OPM officials increased  their training efforts to aid executive agency officials in developing a  common understanding of terms, key concepts, and the objectives of the  data call. According to OPM officials, in July 2011, OPM officials  responsible for the data call met with agency respondents to provide an  introduction and overview to the 2011 data call. The meeting covered the  new requirements under the act, and the planned timeframe for agency  reporting and OPM processing and analysis of the data collected through  the data call. OPM could not require agency officials to attend.  Nonetheless, OPM reminded agency officials responsible for the data  call, that it was important that they attend both September and October  training sessions being offered by OPM. The September training session  covered the data call questions and incorporated specific content of the  near final data call. The October training session reviewed specific  instructions on how to enter information into the online data call form, in  addition to reviewing instructions and questions from respondents. While  some of the information provided at the two training sessions was similar,  each session contained some new information usually in response to  questions raised at a previous session. OPM staff also maintained and  disseminated via email a list of the most frequently asked questions  posed by data call respondents.
section_title: Various Methods of Data Collection and Ongoing Changes to the Data Call Make It Difficult to Assure Data Reliability and to Make Year-to-Year Comparisons
paragraphs: The act requires OPM to report, among other things, year-to-year  executive agency progress on the number of employees who telework. To  accomplish this, OPM needs reliable baseline data on telework  participation to be able to make year-to-year comparisons. However,  OPM officials expressed continuing concerns over the reliability of  quantitative participation and frequency data submitted by the agencies  through the 2011 telework data call. OPM officials explained that 2011 is  a transition year for telework programs across executive agencies, and  some agencies made changes to their policies to bring their telework  programs into compliance with the requirements of the act. For example,  some agencies were implementing new data collection systems while  collecting data on telework. These agencies first needed to create and  implement new policies, and then consider and establish processes  related to the collection of telework data to report to OPM. OPM officials  responsible for preparing the report to Congress stated that because of  these changes, it would not be appropriate to compare the 2011 data to  data collected in prior years.
OPM will not be able to make comparisons to prior years due to the  previously discussed modifications to the data call, such as changes to  definitions. Definitional changes, including the change to specifically  exclude mobile workers, could result in agencies interpreting and  reporting telework participation and frequency of telework differently than  they did in the past. While these changes should improve the consistency  of data going forward, according to OPM officials there is no way that  OPM can assure that all agencies would provide comparable responses  to the same data call questions. Moreover, OPM officials have said that  key terms and definitions for the 2012 data call may continue to evolve. If  OPM reports agency telework progress based on data collected using  definitions revised year to year, OPM may reach erroneous conclusions,  although OPM officials have said they are taking steps to try and prevent  this possibility.
Another reason why comparisons to prior data calls would be invalid is  because of the changes OPM made to the time period when agencies  reported telework data. Maintaining consistent data series over time  necessitates using consistent data collection procedures for ongoing data  collections. OPM said that changing the reporting period from 1 calendar  year to 4 weeks may result in greater data consistency because in  previous data calls, agencies may have reported data from different time  periods within a calendar year. Now all agencies will report data from a  narrower and more similar time frame. However, there are no available  studies to support that the new time period is representative or “typical” of  other months in total or of the experience of particular agencies. OPM  officials said that they will need to indicate this in their report to Congress.
While OPM made changes to the 2011 data call to allow it to meet some  of its reporting requirements under the act and better assist agencies in  responding to the data call, our analysis found the 2011 data call did not  fully meet some generally accepted survey standards. According to these  standards, to be valid, survey questions must adequately represent the  concept or behavior in question and consistently predict outcomes.  Moreover, questions must be designed and asked so that each recipient  will understand and answer the same question in the same way. But for  the 2011 data call, there can be no assurance that all respondents were  aware of associated definitions and instructions provided by the training  sessions and the frequently asked questions (FAQ). Although OPM  invested more in providing training in preparation for the 2011 data call  than in previous data calls and it disseminated training slides to all  invitees of its final training session, attendance lists were not recoverable  for all training sessions so there can be no assurance that all respondents  received training or reviewed the slides. Consequently, some data call  respondents may not have been aware of the definitions and instructions  provided in the training sessions or in the FAQ.
Additionally, some of the information provided in the training sessions  was inconsistent. For example, OPM officials said that during the last  training session they instructed respondents to report an employee  “telework day” if the employee teleworked for any portion of a work day.  However, this clarification of “telework day” was not given in either the  July overview meeting or in the other training session, and not clearly  included in the instructions in the survey instrument or distributed in the  FAQ. This information may have been important to agencies reporting on  situational telework through automated systems intended to capture more  precise data. OPM officials also explained that responses to certain  questions should reference the same time period, and this information  was not available in the data call instructions. During one training  session, an OPM official said she instructed those participants who  determined employee participation by counting telework agreements to  limit the agreements counted to those in effect after the agency  implemented its telework program under the act; however, this  information was not available in the online data call instructions.  Uncertainty about whether data call respondents attended both the  September and October training sessions, and variations in training  sessions, could cause agencies to have different understandings of data  call concepts and terms. OPM recognizes that the existing measures of  federal telework participation vary in validity and reliability, which affects  agencies’ ability to report accurate data, and it is taking steps to verify  data submitted by respondents to provide a more accurate picture of  telework in the federal government based on current definitions and  collection methodologies. However, as a result of issues raised above,  respondents may have provided inconsistent or inaccurate data on topics  required by the act.
OPM officials anticipate that telework data will be more reliable next year  because of the expected governmentwide implementation of automated  data collection based on time and attendance records. As we have  reported, OPM has concluded from research that the most reliable  telework data are collected through time and attendance tracking  systems. Data collected through automated systems eliminate the need  to track telework data by counting telework agreements or relying on  estimates. Since 2003, OPM has consistently expressed concerns about  the methods agencies use to collect telework data. In its previous  telework reports to Congress, OPM has advocated for the development of  an automated data collection system. OPM officials noted that OPM does  not control what telework data responding agencies maintain, or their  methods of data collection. Executive agencies provide telework  participation and frequency data using a variety of methods, such as  relying on estimates, counting telework agreements, and using automated  time and attendance records to track telework participation.
In an effort to collect more uniform data across agencies, OPM officials  are standardizing definitions and data elements for use in automated time  and attendance systems. For example, OPM has identified routine  telework hours in a pay period as a data element for automated data  collection and provided a standardized definition that will be used by all  agencies using the Enterprise Human Resources Integration (EHRI)  system. OPM has introduced a timeline for modifying the existing EHRI  system to allow OPM to collect telework data from executive agencies.  According OPM, agencies will begin piloting these automated data  collection systems for the 2012 telework data call. OPM began to discuss  automated governmentwide data collection with the ITMG in July 2011.  According to OPM’s timeline, OPM began to communicate internal  requirements for automated telework data collection to telework  managing officers in March 2012, but an OPM official has stated they do  not expect full automation of telework data until 2013. This official also  noted that different agencies have varying abilities to implement this new  type of data collection and reporting mechanism, and considering  different levels of comfort with new systems, it will take time to adjust to  this method. However, continuous improvement efforts sometimes result  in a trade-off between the desire for data consistency and a need to  improve data collection and maintenance of a consistent data series over  time.
Because of the eventual planned move to automation, OPM may not be  able to use 2011 data as a baseline. With the planned change to the  method of data collection, it may not be possible to compare the 2011  data to future data. The 2011 data call requested data for a 2-month  period, and some data call respondents relied upon estimates. Planned  automation can provide a more uniform and accurate method for  collecting telework data, however it may make comparisons using the  2011 telework data as a baseline difficult. However, OPM officials believe  the 2011 data will provide an improved report of telework status because  of standardized definitions and the more uniform time period of data  collection. According to OPM officials, some executive agencies will need  time to become comfortable with automated reporting systems and,  during a period of transition to a new system, there could be initial  reliability issues. However, these officials said that the 2011 data,  notwithstanding its limitations, will be useful in identifying and  understanding any major agency changes in reported participation that  could occur during a transition to automated data collection. Such  changes could alert OPM to possible transition related issues in agencies’  conversion to automated governmentwide data call collection efforts in  2012 and 2013. In addition, for those agencies already responding on the  basis of time and attendance reporting, major changes in agency  responses provide OPM the opportunity to confirm with agencies that the  uniform definitions are being consistently applied.
section_title: Conclusions
paragraphs: The Telework Enhancement Act of 2010 requires OPM to collect telework  data and report annually to Congress, which emphasizes the need for  telework data to be valid and reliable. OPM revised the 2011 telework  data call in order to allow it to meet some of the act’s reporting  requirements and assist agencies in responding to the data call. This  revision resulted in changes to terminology, as well as changes in the  collection time period of requested telework data. OPM provided greater  assistance to agencies through training on changes to the 2011 data call  to improve accuracy of agency reporting of telework participation and  frequency for the data call reporting period of September and October  2011. However, agencies use various methods, which OPM does not  control, to report, collect, and maintain telework data, and this could affect  the reliability of the telework data submitted. In addition, variation in  training sessions and OPM’s uncertainty as to whether all respondents  attended training, could lead to respondents’ potential misunderstanding  of important terms and instructions. The validity and reliability of the  reported 2011 telework data for some of the responding agencies may be  questionable, and therefore agency telework participation and frequency  data will not likely be comparable with previous data calls because of  differences in definitions used, time periods of reporting, and individual  agency tracking methods.
With the revised 2011 data call, OPM establishes a baseline it could use  to conduct a limited crosscheck of data collected through a  governmentwide automated telework data collection system, which OPM  plans to implement over the course of 2012 and 2013. OPM expects that  automated data collection will provide it increasingly more reliable data on  which to report progress. However, these efforts to improve future  automated data collection may result in changes to agencies’ methods of  data collection and a trade-off between the desire for consistency with  previous data calls for comparison purposes and a need to improve  overall data collection.
section_title: Recommendations for Executive Action
paragraphs: To improve OPM’s annual reporting of telework to Congress, we  recommend that the OPM Director take the following two actions:
Ensure that the reliability limitations related to the 2011 telework data  call are clearly reported in its June 2012 report to Congress by fully  describing how existing measures of telework participation vary widely  in validity and reliability and limit the capability of OPM to reliably  report the actual level and frequency of telework participation.
Continue efforts to improve data collection and gather information that  allows for the appropriate qualification of year-to-year comparisons  and informs users about the effects of data collection changes going  forward.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Director of OPM for review and  comment. The Associate Director of OPM provided written comments,  which we have reprinted in appendix III. In summary, OPM partially  concurred with our first recommendation and fully concurred with the  second. OPM highlighted a number of actions the agency has under way  or plans to undertake in response. For the first recommendation, OPM  noted that inadequate methods of data collection exist at the agency level  and OPM continues to address this data reliability issue through training  on evaluation and measurement. While this is an important step in  addressing data reliability issues, OPM should ensure that telework data  reliability limitations are clearly reported in their annual reports to  Congress. For the second recommendation, OPM noted its continued  plans to automate collection of telework data, and to regularly meet with  telework managing officers and telework coordinators to keep them  updated on changes to telework policy and data collection. OPM also  provided a number of technical comments, which we incorporated as  appropriate.
We are sending copies of this report to the Chairman and Ranking  Member of the Subcommittee on the Federal Workforce, U.S. Postal  Service and Labor Policy, Committee on Oversight and Government  Reform, House of Representatives; and the Director of OPM. In addition,  this report will be available at no charge on the GAO website at  www.gao.gov. If you have any questions about this report, please contact  me at 202-512-6806 or jonesy@gao.gov. Contact points for our Offices of   Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report (1) describes the Office of Personnel Management’s (OPM)  actions to respond to the requirements of the Telework Enhancement Act  of 2010 and (2) assesses how OPM is handling and addressing identified  data reliability issues in the 2011 telework data call.
To address these two objectives, we reviewed relevant reports and  guidance published by OPM that describe the status of telework  programs across executive agencies and previous telework data calls and  their instructions. We also reviewed previous GAO reports on telework  and the reliability of OPM’s telework data. Lastly, we interviewed the OPM  officials responsible for the planning, design, implementation, and  analysis of the 2011 telework data call. This included discussions on the  role of the Interagency Telework Measurement Group (ITMG), the  process for developing the definitions and key terms used in the data call,  the training and assistance provided to executive agency officials  responsible for completing the data call, and agency plans to address  outstanding data reliability issues associated with the data call. We  conducted additional analysis to answer selected objectives as described  below.
To assess the extent to which the 2011 telework data call met generally  accepted survey methodology, GAO internal experts in survey research  identified principles from the Office of Management and Budget’s (OMB)  Standards and Guidelines for Statistical Surveys relevant to assessing  the 2011 telework data call. We also used relevant aspects of GAO’s  guide to Developing and Using Questionnaires. Using the OMB  principles, two analysts independently reviewed the data call, supporting  documentation, and clarifying information provided in interviews to assess  the extent to which the data call methodology met research practices. The  initial rate of agreement across 15 rated practices, including those rated  as having insufficient information to judge, was 12 of 15, or 80 percent  agreement. The rating for the three practices on which there was initial  disagreement were reconciled by the two analysts conducting the review  and the reconciled rating was then reviewed by a third analyst with survey  expertise. The third analyst did not recommend any changes to the  reconciled ratings.
We also simulated completion of the Web-based data call, accessing and  responding to the data call in the same manner as executive agency  respondents.
Table 1 outlines the generally accepted survey research principles,  derived from OMB’s guidelines, which we used in our assessment. This is  not an exhaustive list of all OMB guidelines. When we completed this  review in April of 2012, OPM had not yet completed analyzing and  reporting on the results of telework data call. Based on this, we could not  yet assess whether the data call met all of the OMB principles related to  data analysis and reporting. Some OMB principles, such as those related  to sample design, were not appropriate to apply to the telework data call.  These principles were therefore excluded as not relevant to our review.
We conducted this performance audit from June 2011 through April 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Revisions to OPM’s Telework Data Call to Collect Information to Prepare for Reporting under the Telework Enhancement Act of 2010
subsections: section_title: Instructions
paragraphs: 2010 Telework data call  Time period: Based on the agency’s calendar year  (Jan. 1 to Dec. 31, 2009).
2011 Telework data call  Time period: Based on the months of Sept. and Oct. 2011.
Participation: agencies invited to participate in data  call.
Participation: agencies required to submit telework data to  OPM.
section_title: Definitions
paragraphs: Telework:  “Telework refers to any arrangement in which an  employee regularly performs officially assigned duties  at home or other work sites geographically  convenient to the residence of the employee.”
Telework:  “Telework is a work arrangement that allows an employee to  perform work, during any part of regular, paid hours, at an  approved alternative worksite (e.g., home, telework center).  This definition of telework includes what is generally referred  to as remote work but does not include any part of work done  while on official travel or mobile work. See the following  clarifications on remote and mobile work as telework.”
Employee:  Employee refers to federal civilian employees excluding  military personnel and contractors. Employee may also  include full-time, part-time, and/or intermittent employees.
Eligibility:  An employee is eligible to participate in telework if all of the  following parameters are true:  positions require, on a daily basis (every work  day), direct handling of secure materials, or on- site activity that cannot possibly be handled  remotely or at an alternate worksite; most recent federal government performance  rating of record (or its equivalent) is below fully  successful or conduct has resulted in disciplinary  action within the last year.
The employee has not been officially disciplined for being  absent without permission for more than 5 days in any  calendar year.  The employee has not been officially disciplined for  violations of subpart G of the Standards of Ethical  Conduct for Employees of the Executive Branch.  Teleworking does not diminish the employee’s  performance or agency operations.  Participation and performance complies with the  requirements and expectations of his/her telework  agreement.  The employee’s official duties do not require on a FULL  daily basis (ALL DAY, every work day):  direct handling of secure materials  determined to be inappropriate for telework  by the agency head; or  on-site activity that cannot be handled  remotely or at an alternate worksite.
The employee and/or the employee’s position are not  disqualified based on additional criteria established by  the organization.
2010 Telework data call  Types of telework:  None provided  2011 Telework data call  Types of telework:
Routine: telework that occurs as part of an ongoing,  regular schedule, and  Situational: telework that is approved on a case-by-case  basis, where the hours worked were not part of a  previously approved, ongoing and regular telework  schedule (e.g., telework as a result of special work  assignments or doctor appointment.)
Situational telework is sometimes also referred to as episodic, intermittent, unscheduled or ad-hoc  telework.
section_title: Appendix III: Comments from the Office of Personnel Management
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, William Doherty, Assistant  Director, and Keith O’Brien, analyst-in-charge, led the development of this  report. Virginia Chanley, Patricia Donahue, Robert Gebhart, Jill Lacey,  and Joseph Santiago made significant contributions to this report. Karin  Fangman provided legal counsel. Shirley Hwang, Jessica Nierenberg,  and Kathleen Padulchick verified the information in the report.
section_title: Related GAO Products
paragraphs: Emergency Preparedness: Agencies Need Coordinated Guidance on  Incorporating Telework into Emergency and Continuity Planning.   GAO-11-628. Washington, D.C.: July 22, 2011.
Human Capital: Telework Programs Need Clear Goals and Reliable Data.  GAO-08-261T. Washington, D.C.: November 6, 2007.
Human Capital: Greater Focus on Results in Telework Programs Needed.  GAO-07-1002T. Washington, D.C.: June 12, 2007.
Agency Telework Methodologies: Departments of Commerce, Justice,  State, the Small Business Administration, and the Securities and  Exchange Commission. GAO-05-1055R. Washington, D.C.: September  27, 2005.
Human Capital: Key Practices to Increasing Federal Telework.   GAO-04-950T. Washington, D.C.: July 8, 2004.
Human Capital: Further Guidance, Assistance, and Coordination Can  Improve Federal Telework Efforts. GAO-03-679. Washington, D.C.:   July 18, 2003.