Federal Workforce: Additional Analysis and Sharing of Promising Practices Could Improve Employee Engagement and Performance
Published: 2015-07-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Research on both private- and public-sector organizations has found that increased levels of engagement—generally defined as the sense of purpose and commitment employees feel toward their employer and its mission—can lead to better organizational performance.
GAO was asked to review recent trends in federal employee engagement and steps OPM and agencies are taking to improve it. Among other things, this report: (1) describes trends in employee engagement from 2006 through 2014, (2) identifies practices in improving employee engagement, and (3) evaluates OPM's tools and resources to support employee engagement.
To meet these objectives, GAO analyzed responses to FEVS questions from 2006 through 2014, conducted a regression analysis, and reviewed OPM documents and interviewed OPM and other agency officials.
section_title: What GAO Found
paragraphs: From 2006 through 2014, government-wide engagement levels—as measured by the Office of Personnel Management's (OPM) Employee Engagement Index (EEI)—increased to an estimated high of 67 percent in 2011 and then declined to an estimated 63 percent in 2014. This decline is attributable to several large agencies—including the Department of Defense—bringing down the government-wide average. The government-wide decline masks the fact that the majority of federal agencies either sustained or increased EEI levels during the period. Of the three components that comprise the EEI—employees' perceptions of agency leaders, supervisors, and their intrinsic work experience—perceptions of leaders consistently received the lowest score.
GAO's regression analysis of selected Federal Employee Viewpoint Survey (FEVS) questions identified six practices as key drivers of the EEI (see table), with constructive performance conversations being the strongest.
For example, at one agency, supervisors and employees developed a set of topics for quarterly performance conversations to ensure that employees receive consistent and regular constructive feedback and coaching.
OPM developed resources to help agencies use EEI data to strengthen employee engagement but fell short of supporting a holistic approach to improving engagement and linking to performance. For example, OPM does not report whether annual EEI changes are statistically significant—that is, whether the changes were meaningful or due to random chance. Likewise, OPM does not analyze which FEVS questions are associated with higher EEI scores. This information would help agencies better focus their efforts to improve engagement and target resources. Further, OPM has provided limited examples or lessons learned on linking engagement to agency performance, which agencies will need to inform their next survey cycle.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Director of OPM take the following three actions: (1) report annually on drivers of the EEI, (2) provide information on statistically significant changes in EEI scores, and (3) share examples and lessons learned to improve engagement and link engagement to performance in time to inform results of the next survey cycle. OPM concurred with the first recommendation and partially concurred with the second and third recommendations. GAO continues to believe that additional action on these recommendations is needed as discussed in the report.

section_title: Letter
paragraphs: A growing body of research on both private- and public-sector  organizations has found that increased levels of engagement—generally  defined as the sense of purpose and commitment employees feel toward  their employer and its mission—can lead to better organizational  performance. Employee engagement is particularly important within  federal agencies, where employees influence the well-being and safety of  the public in myriad ways, such as by conducting advanced scientific  research, verifying and administering benefits, or ensuring the safety of  our workplaces, airports, and national borders. However, government- wide levels of employee engagement have recently declined 4  percentage points, from an estimated 67 percent in 2011, to an estimated  63 percent in 2014, as measured by the Office of Personnel  Management’s (OPM) Federal Employee Viewpoint Survey (FEVS), and  a score OPM derived from the FEVS beginning in 2010—the Employee  Engagement Index (EEI).
In advance of the 2015 FEVS cycle, which began this spring, the  administration elevated the importance of strengthening employee  engagement across government. For example, strengthening employee  engagement is one of three subgoals of the People and Culture Cross  Agency Priority (CAP) goal, established under the GPRA Modernization  Act of 2010 (GPRAMA). Moreover, agency leaders are to be held  accountable for making employee engagement a priority, as well as an  integral part of their agency’s performance management system. The  administration also set a goal for these efforts: by the issuance of the  2016 FEVS results, the federal government is expected to increase  employee engagement—as measured by the EEI—from 63 percent to 67  percent. In addition, as part of their annual performance plans and  appraisals, members of the Senior Executive Service (SES) will be  responsible for improving employee engagement within their  organizations and for creating inclusive work environments.
Given the decline in employee engagement government-wide, you asked  us to examine the federal government’s efforts to strengthen employee  engagement. This report (1) describes trends in employee engagement  as measured by the FEVS, (2) identifies key practices and lessons  learned in developing and implementing strategies to improve employee  engagement, and (3) evaluates OPM’s tools and resources to support  agency efforts to improve engagement as it relates to organizational  performance. In April 2015, we testified on our review’s preliminary  results.
To describe trends in employee engagement as measured by the EEI, we  analyzed responses to questions from the FEVS (from which the EEI is  derived) for the years 2006 through 2014. We started with 2006 to include  trends across two different presidential administrations. Because OPM  calculates the EEI and its component scores at the group level, we used  data from OPM to recalculate the EEI for each individual, which enabled  us to conduct regression analysis and assess the statistical significance  of changes in the EEI. The individual level calculation is scaled between 0  and 100 and is based on the proportion of each individual’s positive  responses to the 15 constituent EEI questions. For 2006 and 2008, we  calculated the EEI using only 11 of the 15 EEI questions because the  2006 and 2008 surveys did not include four EEI questions that were  added to the survey in 2010. We did not make comparisons from 2008 to  2010 because of the change in the index composition.
We analyzed this information government-wide, by agency, and for  selected employee population groups. For each analysis, we determined  statistically significant changes in the EEI. To assess the reliability of the  FEVS data, we examined descriptive statistics and data distribution, and  reviewed missing data. We also reviewed FEVS technical documentation  as well as the statistical code OPM uses to generate the index and  variance estimates. Based on this analysis, we found the data sufficiently  reliable for our purposes. For additional details on our analysis of FEVS  data, see appendixes I and II.
To identify key practices and lessons learned in developing and  implementing strategies to improve employee engagement, we (1)  reviewed relevant literature and interviewed knowledgeable researchers,  government officials from the United Kingdom, Canada, and Australia  responsible for their comparable public-sector employee survey, and  consultants on employee engagement; (2) used linear multiple regression  analysis to assess the relationship between specific FEVS questions and  the 2014 EEI, after controlling for other factors; and (3) reviewed  documents and interviewed officials from three case study agencies. We  considered the agencies that had the highest average sustained EEI  scores, most improved overall EEI scores; and most improved leadership  component scores in the EEI from 2010 to 2014. We selected the  National Credit Union Administration (NCUA), the Federal Trade  Commission (FTC), and the Department of Education (Education) to  ensure that we included agencies that had high scores in one or more of  the three metrics we identified and that we included at least one Chief  Financial Officers Act of 1990 (CFO Act) agency.
To evaluate the support OPM provides to agencies to improve employee  engagement, we reviewed OPM guidance, tools, and resources regarding  use of FEVS data and related engagement action planning; we also  interviewed OPM and case study agency officials and members of the  Chief Human Capital Officers (CHCO) Council and National Council on  Federal Labor-Management Relations joint working group on employee  engagement.
We conducted this performance audit from July 2014 to July 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Engaged employees are more than simply satisfied with their jobs.  Instead, according to employee engagement literature, engaged  employees are passionate about and energized by what they do, are committed to the organization, the mission, and their job, and are more likely to put forth extra effort to get the job done.  take pride in their work,  A number of studies of private-sector entities have found that increased  levels of engagement result in better individual and organizational  performance including increased employee performance, productivity,  and profit margins; higher customer service ratings; fewer safety  incidents; and less absenteeism and turnover. Studies of the public  sector, while more limited, have shown similar benefits. For example, the  Merit Systems Protection Board (MSPB) found that higher levels of  employee engagement in federal agencies led to improved agency  performance, less absenteeism, and fewer equal employment opportunity  complaints.
The FEVS measures employees’ perceptions of whether, and to what  extent, conditions characterizing successful organizations are present in  their agencies. OPM has conducted this survey every year since 2010.  The EEI is composed of 15 FEVS questions covering the following areas:
Leaders lead, which surveys employees’ perceptions of the integrity of  leadership, as well as employees’ perception of leadership behaviors  such as communication and workforce motivation.
Supervisors, which surveys employees’ perceptions of the  interpersonal relationship between worker and supervisor, including  trust, respect, and support.  Intrinsic work experience, which surveys employees’ feelings of  motivation and competency relating to their role in the workplace.
According to OPM, the EEI does not directly measure employee  engagement. Instead, it covers the conditions that lead to employee  engagement. Specifically, OPM noted that organizational conditions lead  to feelings of engagement, which in turn lead to engagement behaviors,  such as discretionary effort, and then to optimum organizational  performance.
Sometimes the EEI is discussed in the same context as another  workforce metric, the Partnership for Public Service’s Best Places to  Work in the Federal Government rankings. Although these scores are  also derived from the FEVS, they were created as a way of rating  employee satisfaction and commitment across federal agencies. The  rankings are calculated using a weighted formula of three different  questions from OPM’s FEVS: (1) I recommend my organization as a good  place to work, (2) considering everything, how satisfied are you with your  job, and (3) considering everything, how satisfied are you with your  organization.
section_title: Most Agencies Did Not Follow the Government-wide Downward Trend and Maintained or Improved Engagement Levels
paragraphs: The recent government-wide average decline in the EEI masks the fact  that the majority of federal agencies sustained and a few increased EEI  levels during the same period. From 2006 through 2014, government- wide EEI levels increased to an estimated high of 67 percent in 2011 and  then declined to an estimated 63 percent in 2014, as shown in figure 1.
The decline in the EEI that began after 2011 is the result of several large  agencies bringing down the government-wide average. For example, we  found that 13 out of 47 agencies saw a statistically significant decline in  their EEI score from 2013 to 2014. While this is 28 percent of agencies,  they employ nearly 69 percent of federal employees and include the  Department of Defense, Department of Homeland Security, and  Department of Veterans Affairs. Meanwhile, the majority of agencies  sustained their EEI levels and a few improved them, as shown in figure 2.  Between 2013 and 2014, of 47 agencies included in our analysis of the  EEI, 3 increased their scores, 31 held steady, and 13 declined.
The recent government-wide downward trend in employee engagement  levels coincided with external events—such as sequestration, furloughs,  and a 3-year freeze on statutory annual pay adjustments from 2011 to  2013—that OPM and others contend negatively affected federal  employee morale. For example, in March 2014, we reported that  officials from agencies—including those that furloughed employees— raised concerns about how sequestration affected the morale of current  employees.
Even one agency with a downward trending engagement score is not to  be taken lightly and there is opportunity for improvement at all federal  agencies. However, the large number of agencies that sustained or  increased their levels of employee engagement during this time suggests  that agencies can positively influence employee engagement levels even  as they weather difficult external circumstances. For example, the FTC  maintained a consistent estimated 76 percent EEI score—well above the  government-wide average—throughout the period of general decline.
subsections: section_title: Leadership Component of the EEI Consistently Scores the Lowest
paragraphs: Of the three components that comprise the EEI, employees’ perceptions  of leaders consistently received the lowest score, and at times was about  20 percentage points lower than the other components. Moreover, from a  high point in 2011, leadership scores saw the greatest decrease and  accounted for much of the government-wide average decline in the EEI,  as figure 3 shows.
The questions comprising the EEI leadership component focus on  integrity of leadership and on leadership behaviors such as  communication and workforce motivation (see table 1). Three of the five  questions are specific to senior leaders—department or agency heads  and their immediate leadership team responsible for directing policies and  priorities and typically members of the Senior Executive Service or  equivalent (career or political). Two are specific to managers—those in  management positions who typically supervise one or more supervisors.
In our 2003 work on transformations, we found that leaders are the key to  organizational change—they must set the direction, pace, and tone, and  provide a clear, consistent rationale that brings everyone together behind  a single mission.
The relative strength of the supervisors component of the EEI suggests  that the employee-supervisor relationship is an important aspect of  employee engagement. These questions focus on the interpersonal  relationship between worker and supervisor and concern supervisors’  support for employee development, employees’ respect, trust, and  confidence in their supervisor, and employee perceptions of an immediate  supervisor’s performance. This is consistent with MSPB research, which  suggests that first-line supervisors are key to employee engagement and  organizational performance.
Intrinsic work experience was the strongest EEI component prior to 2011,  but fell during the period of government-wide decline in engagement  levels. These questions reflect employees’ feelings of motivation and  competency related to their role in the workplace, such as their sense of  accomplishment and their perception of utilization of their skills.
section_title: Pay Category and Supervisory Status Had the Widest Variation in EEI Levels
paragraphs: Understanding how employee engagement varies within differing  populations of employees can enable agency leaders to consider how  different cohorts experience their environment and thus can help  leadership determine how to focus engagement efforts. For example,  knowing that employees with fewer supervisory responsibilities could be  less engaged—and could be having a negative effect on organizational  performance—could spur agency leaders to direct additional resources to  understanding the needs of this subset of the workforce and improving  their sense of engagement.
We found that government-wide, the greatest variation in EEI levels was  related to pay category and supervisory status. For example, respondents  in progressively lower General Schedule (GS) pay categories had  progressively lower levels of engagement government-wide. In contrast,  employees in the SES pay category reported consistently higher  engagement levels—at least 10 percent more than any lower pay  category. While there was less difference between the EEI levels of other  pay categories, employees in the GS 13 through 15 categories  consistently had higher EEI levels than employees in all other lower GS  pay categories. Employees in the Prevailing Rate System, commonly  known as the wage grade system, consistently had the lowest EEI  levels. For example, in 2014, EEI levels for respondents in the SES pay  category were an estimated 84.2 percent compared to an estimated 54.7  percent for respondents in the wage grade pay category.
Similarly, respondents with fewer supervisory responsibilities had  progressively lower EEI levels government-wide. Because employees in  higher pay categories are likely to have more supervisory responsibilities,  responses by pay category and supervisory status represent similar  populations. Variations in EEI levels by supervisory status are shown in  figure 4.
With respect to other populations of employees, agency tenure, federal  tenure, age, and race consistently resulted in some variation but less than  pay category and supervisory status. For example, in 2014, American  Indian or Alaska Native respondents reported the lowest engagement for  any race category with an estimated EEI of 57.6. Asian respondents  reported the highest levels of engagement with an estimated EEI score of  68.4.
Gender, ethnicity (Hispanic/non-Hispanic), and work location  (headquarters/field) consistently had the least variation. For example, in  2014, males had an estimated EEI of 63.2 percent and females an  estimated EEI of 63.3 percent. For results of our analysis of employee  population groups, see appendix II.
section_title: Key Drivers and Lessons Learned for Strengthening Employee Engagement
paragraphs: Overall we found that what matters most in improving engagement levels  is valuing employees—that is, an authentic focus on their performance,  career development, and inclusion and involvement in decisions affecting  their work. The key is identifying what practices to implement and how to  implement them, which can and should come from multiple sources— FEVS and other data sources, other agencies, and OPM. The lessons  learned from our three case study agencies were that the goal should not  be to just increase a number—that is, have a high EEI—but should also  include a focus on improving the organization.
subsections: section_title: Performance Conversations Are the Strongest Drivers of EEI Levels
paragraphs: Of the various topics covered by the FEVS that we analyzed, we  identified six that had the strongest association with higher EEI levels  compared to others, as described in figure 5. We used regression  analysis to test which selected FEVS questions best predicted levels of  employee engagement as measured by the GAO-calculated EEI, after  controlling for other factors such as employee characteristics and agency.
Constructive performance conversations. We found that having  constructive performance conversations was the strongest driver of the  EEI. For the question “My supervisor provides me with constructive  suggestions to improve my job performance,” we found that, controlling  for other factors, someone who answered “strongly agree” on that FEVS  question would have, on average, an engagement score that was more  than 20 percentage points higher than someone who answered “strongly  disagree” on the 5-point response scale.
As we found in our March 2003 report on performance management,  candid and constructive feedback helps individuals maximize their  contribution and potential for understanding and realizing the goals and  objectives of an organization. At Education, one of our case study  agencies, the Office of the Chief Information Officer (OCIO) implemented  a process to help ensure that constructive performance conversations  regularly occur. In addition to department-wide requirements for  supervisors to hold two performance conversations a year, OCIO officials  said that they require all supervisors to offer OCIO employees optional  quarterly conversations. These quarterly performance conversations are  guided by a set of specific topics that supervisors and employees  developed together to ensure that employees receive consistent and  regular constructive feedback and coaching.
Career development and training. Our analysis found that career  development and training was the second strongest driver. For the  question, “I am given a real opportunity to improve my skills in my  organization,” we found that, controlling for other factors, someone who  answered “strongly agree” to that question would have, on average, an  engagement score that was 16 percentage points higher than someone  who answered “strongly disagree.”
As we found in 2004, the essential aim of training and development  programs is to assist an agency in achieving its mission and goals by  improving individual and, ultimately, organizational performance. At  NCUA, another of our case study agencies, officials said the agency  focused on providing training for employees throughout their careers. For  example, NCUA requires each employee to develop an individual  development plan. For employees new to credit union examining—a  majority of employees—NCUA has a standardized 18-month training  program that combines classroom and practical work. New examiners  must complete a core set of courses and may also choose additional  elective courses. NCUA officials said that they are constantly assessing  formal and informal training for entry-level employees to identify areas to  improve the curriculum and instruction. For more experienced examiners,  NCUA provides continuing training and development, according to these  officials.
Remaining drivers. For the remaining four drivers, we found that,  controlling for other factors, someone who answered “strongly agree” or  “very satisfied” to those questions would have, on average, an  engagement score that was 12 percentage points higher than someone  who answered “strongly disagree” or “very dissatisfied.” Those four  drivers are work-life balance (“My supervisor supports my need to  balance work and other life issues”), inclusive work environment  (“Supervisors work well with employees of different backgrounds”),  employee involvement (“How satisfied are you with your involvement in  decisions that affect your work”), and communication from management  (“How satisfied are you with the information you receive from  management on what’s going on in your organization”). Examples of how  our three case study agencies implemented practices consistent with  these drivers include the following:
Work-life balance. FTC officials implemented an outreach  strategy to inform staff about child and elder care resources after  learning that employees were not aware of the services or did not  know that they qualified for these services. Officials said employee  knowledge of and agency commitment to these kinds of programs  enhances supervisor support for work-life balance. Similarly, to  support work-life balance, as part of its engagED initiative,  Education revised telework policies, provided training for  managers and employees on the new polices and on working in a  telework environment, and improved infrastructure to make  telework as effective as time spent in the office, according to  Education officials.
Inclusive work environment. The FTC established an agency- wide Diversity Council to develop comprehensive strategies to  promote understanding and opportunity throughout FTC. FTC  officials said that employees of all levels were interested in  forming such a council. This included employees who experienced  firsthand the diversity issues as well as managers who could  address those issues. The goal of FTC’s Diversity Council— composed of representatives from each bureau and office—is to  engage employees and supervisors across the agency, make  recommendations for improving diversity, and foster the  professional development of all agency employees, according to  these officials.
Employee involvement. Education’s Office of General Counsel  (OGC) has a permanent employee-driven Workforce Improvement  Team (WIT) that grew out of an office-wide meeting with  employees at all levels to involve employees in the discussions  about the FEVS results. As a result of this group’s work,  Education’s OGC management introduced additional training and  professional development opportunities and improved employee  on-boarding through a new handbook and mentoring program.  Education’s OGC officials said that the staff-driven WIT has  created feelings of stronger ownership, engagement, and  influence in office decision making. Education’s OGC officials said  that OGC’s management seeks feedback from staff, including  from the WIT, to evaluate the effectiveness of improvement  efforts. These officials said this strengthens two-way  communication, which improves employee engagement and  organizational performance.
Communication from management. NCUA officials told us that  the head of the agency and its senior leaders communicate with  line employees (who are mostly in the field) through quarterly  webinar meetings. The meetings are scheduled to accommodate  the field employees’ frequent travel schedule and generally start  with any “hot topics” and continue with discussion of agency  efforts to meet mission goals. The agency head takes questions in  advance and during the webinar and, when needed, participants  research and share responses with agency employees. According  to NCUA officials, these regular, substantive conversations  demonstrate top leadership’s commitment and respect for all  employees as valued business partners.
These key drivers can help agencies develop a culture of engagement as  agencies embed them into the fabric of everyday management practices,  rather than simply reacting to the results of the most recent FEVS.  Importantly, these six practices were generally the consistent drivers of  higher EEI levels when we analyzed them government-wide, by CFO Act  agency, and by selected employee populations (such as agency tenure  and supervisory status). Because these six practices are the strongest  predictors of the EEI, this suggests they could be the starting points for all  agencies embarking on efforts to improve engagement.
section_title: Lessons Learned to Improve Employee Engagement Efforts
paragraphs: Our case study agencies also identified three key lessons for improving  employee engagement.
Any change must be implemented using effective management  practices.
The EEI alone is not enough; agencies must look to other sources of  data for a complete picture of employee engagement levels in their  organization and its components.  Improving engagement and organizational performance takes time  and does not neatly follow the survey cycle; change may involve  several efforts and effects are seen at different points in time.
subsections: section_title: Use Effective Management Practices to Build a Culture of Engagement
paragraphs: Our three case study agencies attributed their high or increasing levels of  engagement to overall effective management practices more so than to  efforts specifically aimed at improving engagement levels. Officials at  these agencies said they pay attention to employee engagement scores,  but also focus on overall positive organizational health and culture and on  how their agency implements change efforts. Some of the practices  agencies cited parallel those we identified in 2003 as key to successful  organizational transformation, including top leadership involvement,  consistency, creating a line of sight linking individual results to  organizational performance, and employee outreach.
Top leadership involvement. Officials from all three of our case study  agencies said that top agency leaders were directly involved in  organizational improvement efforts. We have previously reported top  leadership that is clearly and personally leading the change presents  stability and provides an identifiable source for employees to rally around  and helps the process/efforts stay the course. For example, Education  officials said Education’s Chief Information Officer is directly involved in  efforts to address FEVS scores—he is directly involved in the data  analysis, reviewing Education’s OCIO action plans developed by each of  his subordinate directors, overseeing implementation of strategies, and  assessing their effectiveness.
Consistency. Officials at Education’s OCIO said it is important to ensure  that policies are applied consistently, which is the goal of that office’s  Speaking with One Voice initiative. The biweekly management meetings  to discuss and clarify the implementation of department policies (e.g.,  telework, resources, and employee bonuses) were instituted after  conversations with employees revealed that policies were inconsistently  applied. As a result of the initiative, Education’s OCIO officials said  employees know that senior leaders are paying attention to how policies  affect employees and are accountable for ensuring appropriate  implementation.
Line of sight. FTC officials emphasized the importance of creating a line  of sight between the agency’s mission and the work of each employee.  As we have previously reported, successful organizations create a “line of  sight” showing how team, unit, and individual performance can contribute  to overall organizational results. FTC officials said that the agency lists  every employee that contributed to a case in the pleadings, from the  attorneys and paralegals to the information technology specialists who  provided computer support. Officials said that legal actions are the  culmination of the efforts of many employees, both mission and mission- support staff, and including their names on pleadings helps create a line  of sight from each employee’s contribution to the organization’s success.  Further, FTC officials said they recognize how mission support functions,  such as excellent human resources customer service contribute to the  agency mission. For example, FTC officials said that they emphasize to  the human resources staff that their prompt handling of payroll and  benefits issues contributes to the overall efficiency and mission  accomplishment by minimizing the time other FTC employees expend on  these concerns. As a result, mission employees can focus on  accomplishing their mission-related responsibilities.
Employee outreach. According to officials at all three of our case study  agencies, they all reach out to employees and their labor union  representatives, if applicable, to obtain insight into their FEVS scores or  to inform other improvement efforts. Our 2003 report found that employee  involvement strengthens the improvement process by including frontline  perspectives and experiences. By participating in improvement task  teams, employees have additional opportunities to share their  experiences and shape policies and procedures as they are being  developed and implemented. For example, in 2012, while NCUA’s EEI  score was above the government-wide level, FEVS questions about  awards, performance appraisals, and merit-based promotions were its  lowest scoring categories. NCUA officials said they contracted with an  external facilitator to conduct workshops and webinar-based feedback  sessions with employees to gain insight into their FEVS results and  identify root causes influencing the survey scores. These officials said  that using external facilitators offered employees confidentiality and  created an environment that encouraged open conversations. Based on  these feedback sessions, NCUA created an internal employee-driven  committee to inform revisions to the awards, performance appraisals, and  merit-based promotion process, and developed recommendations for  NCUA’s management to implement these changes. Most of the  committee’s recommendations were implemented.
section_title: Supplemental Data Sources Enable Agencies to Pinpoint Efforts
paragraphs: According to officials at our case study agencies, while the EEI provides a  useful barometer for engagement, other indicators can provide officials  with a deeper insight into reasons for engagement levels and areas for  improvement. Other data such as turnover rates and equal employment  opportunity (EEO) complaints—which are likely already collected by  federal agencies—can provide additional insight and strategies for  improving employee engagement. Notably, MSPB found that there is a  statistically significant correlation between higher levels of employee  engagement and fewer EEO complaints. Officials in the three case  study agencies said that they pay attention to their FEVS scores, but  other sources of data can provide explanatory or agency-specific  information valuable to developing improvement strategies.
As one example, NCUA officials said they identified a slight increase in  turnover in recent years from the human capital data and are reviewing  their exit survey process in calendar year 2015 to determine the best way  to gather this information as well as to identify trends in reasons for  employees leaving. NCUA officials said providing applicants with a clear  understanding of the work will help to ensure a good position fit initially,  which leads employees to stay engaged with the organization. Credit  union examiners often travel between credit unions and do not regularly  report to an office. Therefore, NCUA is revising its vacancy  announcements to better communicate the nontraditional work  environment of field staff, which officials said comprises nearly 80 percent  of the agency’s workforce. For example, NCUA officials said that  applicants are asked to also indicate preferences on working  independently.
section_title: Effects of Engagement Efforts Occur Over Multiple Years
paragraphs: Case study agency officials told us that they take a multi-year, multi-prong  approach to improving engagement and do not base engagement efforts  solely on the survey cycle or focus their attention on year to year changes  in the EEI. Some case study agency officials said a single survey cycle  does not provide enough time to implement changes and see results  because real change usually takes more than 1 year. The FEVS cycle  begins around May and agencies receive results in September or  October. It may be late-winter or early-spring before an agency will have  designed an action plan. By the time the next survey cycle begins,  agencies may still be interpreting results and developing and  implementing their action plans.
Moreover, according to case study agency and other officials we  interviewed, the annual survey cycle does not allow enough time for  employees’ perceptions to change before the next cycle begins. For  example, an Education official said that it took a few years to see the  effects of engagement-related actions. Members of the Chief Human  Capital Officers Council and National Council on Federal Labor- Management Relations joint working group on employee engagement  said that the effects of initiatives implemented to improve engagement will  not be reflected in the EEI scores for at least a couple of years, which  makes evaluating their effectiveness challenging.
OPM officials agreed that efforts to improve engagement should not be  based on the survey cycle, but noted the benefits of an annual survey.  Specifically, OPM stated that agencies are increasingly using the FEVS  as a management tool to help them understand issues at all levels of an  organization and to take specific action to improve employee engagement  and performance. An annual survey such as FEVS can help ensure that  newly appointed agency officials (or a new administration) can maintain  momentum for change, as the surveys suggest employees are expecting  their voices to be heard. Further, OPM officials noted if agencies,  managers, and supervisors know that their employees will have the  opportunity to provide feedback each year, they are more likely to take  responsibility for influencing positive change.
Instead of focusing exclusively on FEVS and EEI scores, case study  agencies took a longer term approach to their engagement efforts. For  example, according to officials, Education established engagED, a long- term cultural change initiative, to build a more innovative, collaborative  and results-oriented agency and create a more engaged workforce. This  initiative focused on three areas—increasing multi-way communications,  performance accountability, and professional growth opportunities—which  officials said were consistently identified as challenges through the FEVS  analysis and facilitated feedback discussions. Instead of focusing on one  specific FEVS question, Education identified these broad themes to bring  about a systemic change, according to officials. EngagED addressed  these challenges through several actions intended to prompt thoughtful  discussion among employees, accountability for results, and  developmental opportunities. Among others, these actions included the  following:
Quarterly all-staff meetings with the Secretary to discuss various  topics.
A “lunches with leaders” program allowed agency employees more  access, input, and participation in key topics discussed by senior  agency leaders.
A redesigned performance appraisal system to simplify and  standardize performance rating levels to more clearly reflect  performance expectations, and consistently recognize and reward  successful performers within principal offices and across the agency.
Periodic leadership summits to provide agency leaders with  developmental activities identified by staff that are focused on teams,  individual leadership, and problem resolution.
An Education Policy Briefing Series to provide agency employees with  an opportunity to learn about cutting-edge education issues that relate  to the goals and work of the agency and to provide a forum for staff to  interact and share expertise.
According to Education officials, this longer-term, broad-based approach  helped Education improve its scores even while government-wide scores  were decreasing. Officials said they monitor each year’s results, but their  success demonstrates the value of taking a long-term perspective.
section_title: OPM Has Created Tools and Resources to Improve Engagement, but They Fall Short of Supporting a Holistic Approach to Improving Engagement and Linking to Performance
subsections: section_title: Limitations in Data Analysis and Reporting Hinder Agencies’ Ability to Target Resources and Assess Progress
paragraphs: OPM provides a number of tools and resources to support agencies’  efforts to use EEI data to identify areas that need improvement, as shown  in table 2. However, these tools do not provide agencies with the drivers  of the EEI or enable agencies to determine if changes in EEI levels are  meaningful.
One of OPM’s key strategic goals is to help agencies create inclusive  work environments where a diverse federal workforce is fully engaged  and energized to put forth its best effort, achieve their agency’s mission,  and remain committed to public service. OPM’s strategic plan for fiscal  years 2014-2018 states that it will ensure agencies target, address, and  measure key drivers of employee engagement. However, OPM does not  analyze the drivers of the EEI or provide agencies the tools to do so. A  driver analysis based on FEVS questions can help agencies more  effectively target limited resources and can provide a roadmap to design  strategies to improve EEI levels.
OPM officials told us that they do not conduct a driver analysis to  determine which FEVS questions are associated with higher EEI scores  and report them via OPM’s online tools because they would have to use a  more complicated and less transparent method of calculating the EEI.  Specifically, to conduct a driver analysis OPM would have to calculate the  EEI for each individual respondent by determining the proportion of  positive responses to the 15 EEI questions for each respondent. OPM  would also have to account for unanswered questions. OPM currently  calculates the EEI by averaging all the positive responses to the EEI  questions for the group of respondents. While OPM officials  acknowledged the importance and value of the individual level calculation  for determining drivers, they said that the benefit of the current method of  calculating the EEI is that it is simpler and officials can see how the  scores are calculated—as an average. However, OPM officials noted that  they could separately conduct a driver analysis outside of the online tools.  In 2006, OPM conducted a regression analysis to identify which questions  best predicted overall job satisfaction, overall satisfaction with the  organization, and intent to stay or leave. This suggests that OPM has  the capability to conduct such an analysis.
Research on employee engagement emphasizes the importance of  identifying the drivers of an engagement or related metric as an initial  step in improving employee engagement. For example, the Partnership  for Public Service’s Best Places to Work in the Federal Government  guidance lists a driver analysis as a key element in determining where  agencies should focus their action planning efforts. If managers  understand the drivers of engagement, then they can better target their  engagement efforts, particularly in times of limited resources. The results  of our driver analysis demonstrate consistency of results government- wide and by CFO Act agency as well as selected employee populations.  Similar or even more limited investment of OPM resources could yield  information that would benefit all agencies. However, by not determining  which FEVS questions are associated with higher EEI levels, OPM is  missing an opportunity to assist agencies in targeting their engagement  resources.
subsections: section_title: Agencies Lack Key Data to Assess Progress in Strengthening Engagement
paragraphs: As noted earlier, the administration has said agency leaders will be held  accountable for making employee engagement a priority and making it an  integral part of their agency’s performance management system. For  example, the December 2014 memorandum on engagement and  performance calls on agencies to incorporate engagement measures into  SES and agency performance plans. According to the memorandum,  SES performance plans are to include a measurable component related  to improving employee engagement by 2016. GPRAMA annual  performance plans are to include baselines and organizational targets for  strengthening employee engagement with a focus on a percent change.  As agencies begin to use engagement measures to inform other  performance measurement decisions, understanding whether EEI  changes are statistically significant will become especially important.
However, OPM does not report whether changes in agency EEI scores  are statistically significant—that is, whether the change is meaningful and  not due to random chance. As a result, agency officials do not have the  information they need to appropriately interpret changes in the EEI. OPM  does provide agencies with absolute changes in the EEI—increases and  decreases that may or may not be statistically significant due to sampling  variability. The method we used to determine statistical significance  showed that only 34 percent (16 of 47) of the absolute changes in agency  EEI scores from 2013 to 2014 were actually statistically significant.
Without understanding whether changes are statistically significant,  managers may take action based on data that has limited meaning. For  example, a manager might assume an annual increase in the EEI meant  specific engagement efforts were successful when they were not or  assume an annual decline in the EEI meant specific engagement efforts  were not successful and abandon an effort too soon. Statistical  significance is a function of two things: (1) the size of the change—the  increase or decrease in the EEI, and (2) the size of the population  sampled. In general, the smaller the sample, the larger the change needs  to be before it is statistically significant, and the larger the sample, the  smaller a change needs to be to be significant. In FEVS, sample sizes  tend to be substantially smaller at smaller agencies. For example, from  2013 to 2014 the Federal Mediation and Conciliation Service, with a  workforce of over 200 employees, had a 2 percentage point change in the  EEI, which was not statistically significant, based on the method we used  to determine statistical significance. During that same period, the  Department of Defense—the largest federal agency with over 700,000  employees—had about a 0.7 percentage point decrease in the EEI; this  was a statistically significant change. Without knowing whether changes  in the EEI are statistically significant, agency officials do not have the  context to determine whether a change is meaningful.
section_title: OPM Has Efforts to Identify and Share Promising Practices Underway But Needs More Focus on Linking Engagement and Performance
paragraphs: As agencies move from analyzing data to developing strategies to  improving engagement and linking it to organizational performance, the  specific strategies and lessons learned from the experiences of other  agencies can be beneficial to those who may be seeking information  related to improving employee engagement and performance. Table 3  describes OPM efforts to help agencies develop and implement strategies  to improve employee engagement and link engagement to performance.
OPM has several efforts under way to support agencies’ efforts to  develop and implement strategies to improve engagement and link it to  performance. Most of these efforts focus on the identification and sharing  of promising practices, where OPM works in partnership with federal  agencies. During the course of our audit, OPM officials described the  following avenues for identifying these practices—the Engagement  Outreach Team, in-person events, the CHCO Council–National Council  on Federal Labor-Management Relations Employee Engagement  Working Group, and through the designation of agency-appointed Senior  Accountable Officials. OPM officials said their goal is to use the  Community of Practice page on UnlockTalent.gov as the platform for  sharing the identified promising practices. However, as of May 2015,  OPM has only posted limited content on the Community of Practice  page—video clips from three events.
Engagement Outreach Team. OPM’s Engagement Outreach Team is a  seven-person intra-agency team formed in August 2014, representing  four OPM offices. Each outreach team member provides individualized  support to three to four President’s Management Council agencies and  identifies potential promising practices. OPM officials said the Outreach  Team speaks with each of their assigned agencies, and based on these  conversations, the Outreach Team meets weekly to share successes and  challenges, address action items on their respective agencies’ efforts,  and identify potential promising practices.
However, as of May 2015, none of the promising practices identified by  the Outreach Team had been added to the Community of Practice page  on UnlockTalent.gov and OPM officials said they did not have a plan for  how they will use the information gathered from President’s Management  Council agencies to inform the Community of Practice page. In April  2015, OPM officials said the Outreach Team was evaluating all of the  qualitative data received from agencies, including feedback on previous  events and resources offered by the team to determine the next best  steps forward, including developing content for the Community of Practice  page of UnlockTalent.gov. However, officials did not provide a time frame  for completing the evaluation or developing the content.
In-person events. During the fall of 2014, OPM hosted three in-person,  Washington, D.C.-based events for agency officials aimed at identifying  promising practices in improving employee engagement. Specifically, in  November 2014, OPM partnered with OMB and the Presidential  Personnel Office to hold the Federal Employee Engagement Forum event  at the White House. Panels of public- and private-sector representatives  discussed (1) approaches to improving employee engagement and  performance through use of data, (2) best practices in collaborative  engagement strategies, and (3) the best use of tools and strategies to  improve employee engagement and performance. OPM officials said  approximately 150 people attended including senior agency officials or  their staff and representatives from national labor unions and private- sector organizations. OPM also held two sessions for agency officials in  OPM’s Innovation Lab to identify strategies for improving employee  engagement. OPM officials said about 20 people attended each session  from 11 different agencies. Participants included CHCOs and other senior  leaders as well as other agency officials involved with employee  engagement efforts.
On May 18, 2015, following a briefing on the preliminary findings of our  audit, OPM posted video clips on the Community of Practice page of  UnlockTalent.gov from the Federal Employee Engagement Forum held at  the White House and another in-person event, the SES Leadership  Event. Regarding the Innovation Lab events, according to OPM officials,  summaries of the sessions were shared with participants and by request  with other officials. OPM officials said they have no further plans for when  and how to share the Innovation Lab summaries with a broader  government-wide audience.
CHCO Council–National Council on Federal Labor-Management  Relations Employee Engagement Working Group. Coordinated by the  CHCO Council Executive Director, the working group was launched in  February 2014 with representatives from 15 agencies and 8  nongovernment organizations including federal employee unions and  organizations representing managers and executives. The group meets  on approximately a quarterly basis and is organized into three  subcommittees—(1) promising practices, (2) key enablers and barriers,  and (3) measures and incentives. Two representatives, one each from an  agency and a labor union, co-chair the full working group and each of the  subcommittees. The group’s 2014 work culminated in presentations to the  CHCO Council and the National Council on Federal Labor-Management  Relations.
According to the CHCO Council Executive Director, a key goal of the  working group is to identify practices that will be shared on the  Community of Practice page of UnlockTalent.gov. However, despite the  preliminary work as reflected in the November 2014 presentations, no  practices had been posted on the Community of Practice page, as of May  2015. Further, according to the CHCO Council Executive Director,  agencies have been reluctant to hold their engagement practices up as a  model.
Agency-Appointed Senior Accountable Officials.  According to OPM  officials, from March through May 2015, OPM met once with each  agency’s Senior Accountable Official to discuss their agency’s  engagement baseline, plans for engagement activities, assistance  needed, and best practices. In addition, OPM officials said in May 2015  they hosted a workshop for the Senior Accountable Officials to share  leading practices and strategies to address common challenges. OPM  officials said materials from this event were provided to participants, and  included examples from participating agencies. OPM officials said after  this workshop, they designed five workgroups for Senior Accountable  Officials or their designee to work with officials from the People and  Culture CAP goal to address specific engagement challenges. OPM  officials said the workgroup’s outcomes and next steps will be shared in  the fall of 2015, when the 2015 survey results are available to agencies.
subsections: section_title: Agencies Lack Examples or Guidance to Link Engagement to Agency Performance
paragraphs: Linking increased employee engagement to improved organizational  performance is important because it recognizes that improved  engagement is not an end in itself. Instead, the ultimate aim is to enhance  the ability of agencies to cost-effectively carry out their missions. The  administration’s December 2014 memorandum calls for agencies to  establish the linkage between employee engagement and mission  performance. Specifically, agencies are to collect return on investment  information—that is, whether a change in engagement levels resulted in  improvements to performance metrics related to agency mission, such as  a reduction in error rates.
OPM, as the lead agency on the CAP employee engagement subgoal,  has provided limited examples or guidance for how agencies could  establish a linkage between engagement and performance. According to  OMB officials, establishing such a link is the key step in agency efforts to  improve employee engagement and performance. OPM officials said they  began a study on how to link engagement to outcomes at three agencies  whose outputs can be quantitatively measured. These agencies are the  U.S. Mint’s Bureau of Engraving and Printing, the Patent and Trademark  Office, and OPM’s Retirement Services Division. Specifically, OPM  planned to analyze the relationship between EEI scores and production or  performance measures at a team level. However, this effort has been  delayed with no estimated completion date, and officials said that linking  the EEI to team level performance or production outputs has been more  difficult than anticipated. Following a briefing on the preliminary findings of  our audit, OPM posted a video clip from the Federal Employee  Engagement Forum panel on linking metrics to business outcomes. While  this is an important step, the video segment featured only one example  from a federal agency.
Because experience with linking employee engagement in the public  sector to organizational performance is limited (as indicated by our  interviews with OPM and OMB officials and the literature that we  reviewed), agencies do not have a clear model on how to make this link  and demonstrate that it resulted in improved mission accomplishment.  Even at our three case study agencies—each of which have high or  improving EEI scores—officials said that they lacked sufficient in-house  expertise to develop and conduct such an analysis.
Pursuant to the People and Culture Employee Engagement CAP subgoal,  the administration has established a target of improving employee  engagement government-wide to 67 percent by the issuance of 2016  FEVS results, a 4 percentage point increase from the 2014 government- wide score. Without the ability to link increased engagement to  improved performance, the extent to which an increase of 4 percentage  points will translate into improved performance is unclear. Further, given  the time horizon necessary to see real improvements, it is unclear if  efforts to improve engagement can be achieved by 2016.
section_title: Conclusions
paragraphs: Higher levels of employee engagement can translate into higher levels of  organizational performance. However, for agencies to attain the ultimate  goal of improving organizational performance, agencies must take a  holistic approach—analyzing data, developing and implementing  strategies to improve engagement, and linking their efforts to improved  performance.
The tools and resources OPM has developed represent an important first  step toward helping agencies improve employee engagement. However,  OPM cannot ensure officials are correctly understanding and using EEI  data, because it does not report on whether EEI changes are statistically  significant—that is, whether the changes are due to something other than  chance. OPM also does not determine the FEVS questions that lead to  increased EEI levels, which would enable agencies to focus on areas that  drive engagement.
At the same time, informed decisions require more than just EEI data.  When faced with multiple options, agencies need to target their resources  to the practices found to drive employee engagement—notably  constructive performance conversations. Moreover, as indicated by our  case study agencies, how changes are implemented—including the  involvement of top leadership, consistency, employee outreach, and  creating a line-of-sight between individual and organizational  performance—ultimately affects whether those changes merely produce  temporary compliance or result in sustainable cultural transformation.  Further, other data can be used to provide more specific information on  what needs attention. However, efforts take time and should not be seen  solely as an annual effort measured by the results of the next survey, but  as a continuous process.
Although OPM has developed a process for identifying and collecting  promising practices, OPM’s sharing of promising practices has been  limited, and whether and when its efforts will come to fruition is unclear.  Promising practices around analyzing EEI data and implementing  improvement strategies are important; however, given the public sector’s  limited experience linking engagement to performance, promising  practices on creating this linkage are critical.
section_title: Recommendations for Executive Action
paragraphs: In furtherance of its role to support agencies’ efforts to improve employee  engagement and performance, we recommend that the Director of OPM  take the following three actions:  1.  To enable agencies to better target resources for engagement efforts,  OPM should annually analyze and report on drivers of the EEI  government-wide and by selected subsets of the federal workforce,  such as agencies or employee population groups.  2.  To enable agencies to identify meaningful changes in EEI levels,  OPM should provide agencies with information on whether annual  changes to EEI scores, both government-wide and by selected  subsets of the federal workforce, are statistically significant.  3.  To ensure agencies are leveraging promising practices and lessons  learned from other agencies in developing effective strategies to  improve engagement and performance, OPM should, in partnership  with federal agencies, expand its efforts to share promising practices to include  information on linking engagement to mission accomplishment  and monitoring how engagement investments improve  performance through data-driven reviews, like HRstat; and  implement its strategy to share these practices in time to inform  agency efforts stemming from their 2015 FEVS results.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this product to the Director of OPM for comment.  In written comments, which are reproduced in appendix V, OPM  concurred with our first recommendation and partially concurred with our  second and third recommendations.
OPM stated that it concurred with our recommendation to analyze and  report on the drivers of employee engagement government-wide and by  subsets of the federal workforce.  OPM said that starting with the release  of the 2015 FEVS, it is committed to conducting, analyzing, and reporting  on key drivers of the EEI at the government-wide level and has formed a  working group to make recommendations on the scope of the analysis  and the reporting in subsets of the federal workforce.
OPM stated that it partially concurred with our recommendation to provide  information on whether changes are statistically significant, and noted  that among other actions, the working group referenced above is to  provide guidance on how best to disseminate this information to  agencies. At the same time, OPM maintained that in addition to the EEI,  agencies should also rely on multiple indicators to assess organizational  performance. This was a key point that we made in our report.  Specifically, we said that in addition to the EEI, other indicators—such as  turnover data—can provide officials with additional insights into reasons  for engagement levels and areas for organizational improvement.   However, in order for agencies to meaningfully use EEI data in their  analyses, we continue to recommend that OPM provide agencies with  information on whether changes in the EEI are statistically significant.
With respect to our third recommendation, OPM noted that while it agreed  with our recommendation to share promising practices in time to inform  agency efforts based on the FEVS 2015 survey results, OPM disagreed  with our assessment of OPM’s efforts.  In its written comments, OPM  outlined the steps taken to identify and share promising practices after  being briefed on the findings of our audit. For example, OPM noted that in  May 2015, it held a workshop with the Senior Accountable Officials  responsible for agency engagement efforts in an effort to share promising  practices and generate solutions to common challenges; OPM also noted  it has formed workgroups among the Senior Accountable Officials around  specific topic areas, with a goal of sharing those practices with the larger  federal community this fall.  We have modified our report to include more  information on OPM’s efforts and plans to share promising practices with  agencies in time to inform the analysis of the FEVS 2015 survey results.   However, until such information is shared with the larger federal  community and includes models for linking engagement to mission  accomplishment, we continue to believe that OPM should take additional  actions to assist agencies in leveraging their lessons learned, as we  recommended.
We also provided a draft of this product to the Secretary of Education,  Chairwoman of the FTC, and the Chairman of NCUA for technical review  and comment. We received technical comments from them that we  incorporated, as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Director of the Office of Personnel  Management, and other interested parties. In addition, the report will be  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report please contact  me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  product are listed in appendix VI.
section_title: Appendix I: Data Preparation and Calculation of the Employee Engagement Index
paragraphs: To determine the trends in employee engagement as measured by the  Office of Personnel Management (OPM) Federal Employee Viewpoint  Survey (FEVS) and identify key practices to improve employee  engagement we analyzed (1) employee engagement index scores (EEI)  from 2006 to 2014 and (2) the extent to which selected 2014 FEVS  questions predicted EEI scores. This appendix describes our  methodology for preparing the dataset and for calculating the EEI to  conduct these analyses. For the methodology and results of our analysis  of EEI trends, see appendix II; for our methodology for identifying the  drivers of the 2014 EEI, see appendix III; and for the results of our driver  analysis, see appendix IV.
subsections: section_title: Federal Employee Viewpoint Survey
paragraphs: The FEVS provides a snapshot of employees’ perceptions about how  effectively agencies manage their workforce. Topic areas are employees’  (1) work experience, (2) work unit, (3) agency, (4) supervisor, (5)  leadership, (6) satisfaction, (7) work-life, and (8) demographics. OPM has  administered the FEVS annually since 2010; from 2002 to 2010, OPM  administered the survey biennially. The FEVS includes a core set of  questions, and agencies have the option of adding questions to the  surveys sent to their employees.
The FEVS is based on a sample of full- and part-time, permanent, non- seasonal employees of departments and large, small, and independent  agencies, which in 2014 represented about 97 percent of the federal  executive branch workforce. According to OPM, the sample is designed  to ensure representative survey results would be reported by agency,  subagency, and senior leader status as well as for the overall federal  workforce. Once the necessary sample size is determined for an agency,  if more than 75 percent of the workforce would be sampled, OPM  conducts a full census of all permanent, non-seasonal employees. For  2014, the total sample size was 872,495. According to OPM, this size  was more than sufficient to ensure a 95 percent chance that the true  population value would be between plus or minus 1 percent of any  estimated percentage for the total federal workforce.
Government-wide, in 2014, 392,752 employees completed surveys for a  response rate of 46.8 percent. Among departments and large agencies,  the Department of Veterans Affairs had the lowest response rate—32.6  percent—and the National Science Foundation had the highest—77.3  percent.
section_title: Data Set Provided by OPM
paragraphs: We analyzed the 2006 through 2014 FEVS data file containing the 84  core questions and demographic variables provided to us by OPM. The  datasets contain the full demographic and work unit location information  on the respondents. OPM forward coded older FEVS datasets to match  the coding for the 2014 survey so that question numbering and response  categories were consistent across time. If a question from the 2014  survey instrument was not present in a previous year, OPM coded the  questions as missing in the older data set.
To assess the reliability of the FEVS data, we examined descriptive  statistics, data distribution, and reviewed missing data. We also reviewed  FEVS technical documentation as well as the statistical code OPM uses  to generate the index and variance estimates, and we interviewed officials  responsible for collecting, processing, and analyzing the data. On the  basis of these procedures, we believe the data were sufficiently reliable  for use in the analysis presented in this report.
section_title: Employee Engagement Index
paragraphs: OPM first introduced the EEI in 2010, when it contained 8 questions.  OPM revised it in 2011 to contain the 15 questions that currently  comprise the EEI. According to OPM, the EEI is a measure of the  conditions conducive to engagement. The EEI consists of three  components—leaders lead, supervisors, and intrinsic work experience.  OPM calculates the EEI by averaging the component scores, which are  an average of the percent of positive responses to each question in the  respective component. For a full list of EEI questions, see table 1 earlier  in this report.
section_title: GAO’s Calculation of the EEI
paragraphs: To determine trends in employee engagement as measured by the FEVS  for the years 2006 to 2014, we calculated the EEI using the data provided  by OPM. For 2010 to 2014, we calculated the index using responses to  the 15 questions contained in the current index. For 2006 and 2008, we  calculated the EEI using only 11 of the 15 EEI questions because the  2006 and 2008 surveys did not include four questions that were added to  the survey in 2010. Because OPM calculates the EEI and its component  scores at the group level, we used data from OPM to recalculate the EEI  for each individual, which enabled us to conduct regression analysis and  assess the statistical significance of changes in EEI. The individual level  calculation is scaled between 0 and 100 and is based on the proportion of  each individual’s positive responses to the 15 constituent EEI questions.
To test the comparability of the 2010 to 2014 estimates, which we  calculated based on the 15 questions in the current EEI, and the 2006  and 2008 estimates, which we calculated based on the 11 EEI questions  in the survey at that time, we recalculated the EEI for 2010 to 2014 using  just the 11 EEI questions present in the 2006 and 2008 survey data. The  revised EEI estimates for 2010 to 2014 based upon just the 11 questions  averaged about 2.9 percentage points less than the EEI estimates based  on the full 15 questions. While the trend line with a peak in 2011 was  similar when the estimates were plotted by year, we did not make  comparisons from 2008 to 2010 because of the change in the index  composition.
To generate estimates for agencies and employee population groups, we  aggregated the index across individuals using the appropriate sample  weights. We followed the replicate weight variance estimation  methodology recommended by OPM to generate sample variance  estimates for the index scores. This enabled us to analyze the drivers of  engagement and assess the statistical significance of differences.
To ensure that our calculation of the EEI would yield sufficiently similar  results as OPM’s methodology for 2014, we assessed the correlation  between the two versions. When aggregated to the agency level, our  index is nearly perfectly correlated with the OPM measure in 2014  (ρ=0.99971). To confirm the cohesiveness of the individual index, we  calculated Cronbach’s alpha, a measure of internal consistency that  ranges from zero to 1 for 2014. The alpha value of .94 suggests that the  scale of the items captures the majority of the variation in the underlying  items, indicating high internal consistency.
section_title: Appendix II: Trends in Employee Engagement Index Levels
paragraphs: Using the Employee Engagement Index (EEI) calculated at the individual  level, for 2006 through 2014 we analyzed the EEI government-wide, including the EEI components scores; by agency for those with a minimum of 5 years of Federal Employee  Viewpoint Survey (FEVS) data for the years analyzed and at least 100  respondents, and by employee population measured by the FEVS.
For each analysis, we determined statistically significant year-to-year  changes in the EEI from 2006 to 2014, with the exception of 2008 to 2010  because the questions in the EEI for those years were not comparable.  We identified statistical differences by assessing whether the 95 percent  confidence intervals of two estimates overlapped or not rather than  conducting multiple t-tests; confidence intervals that do not overlap  represent differences that are statistically significant. If the change was  statistically significant, there is less than a 5 percent probability that the  difference occurred by chance. This method of assessing difference is  conservative, in that it may underestimate the amount of statistically  significant differences in cases of minor overlap of confidence intervals,  but does not require us to use a testing methodology modification such as  a Bonferroni adjustment to account for multiple comparisons. Table 4  below shows the downward trend in the government-wide EEI and for two  of the three components.
Tables 5 and 6 show the EEI trends for the 47 agencies with a minimum  of 5 years of FEVS data for the years analyzed and at least 100  respondents and whether year to year changes were statistically  significant. These tables are the basis for figure 2.
subsections: section_title: Employee Population Group Results
paragraphs: Table 7 shows the estimated EEI by employee population category. To  determine the extent of variation in responses within an employee  population, we measured the greatest possible amount of variation in  each of the years 2006 through 2014. Specifically, we measured the  percentage point difference between the confidence interval upper bound  of the category with the highest EEI and the lower bound confidence  interval of the category with lowest EEI in each of the years. Pay category  and supervisory status consistently had the widest variation in EEI  scores.
section_title: Appendix III: GAO’s Analysis of the Drivers of the Employee Engagement Index
paragraphs: Our analysis of the drivers of engagement measures the extent to which  selected FEVS questions predict the EEI. To conduct this analysis, we  reviewed relevant literature and interviewed knowledgeable individuals to  identify and refine a list of potential drivers of engagement, and then  identified corresponding FEVS questions not included in the EEI. Using  FEVS 2014 data, we then used multiple linear regression analysis to  assess the correlation between the driver questions and the EEI,  controlling for other factors such as agency and employee characteristics.  We used both statistical significance and the magnitude of regression  coefficients to define drivers of the EEI. We conducted sensitivity tests to  ensure that our results were robust to differences in model specification,  functional form, and to the exclusion of cases with high levels of missing  data. In addition to our government-wide analysis, we analyzed the  drivers among employees at each of the Chief Financial Officers Act  agencies and selected employee populations. Results of these analyses  were generally consistent with our government-wide analysis.
subsections: section_title: Identification of FEVS questions
paragraphs: To determine the FEVS questions to include in our statistical model we  reviewed relevant literature and interviewed knowledgeable researchers,  government officials from the United Kingdom, Canada, and Australia  responsible for their comparable public-sector employee survey, and  consultants on employee engagement. We then categorized all the  potential drivers identified by sector—such as academia, consultants, and  the federal government. We selected the drivers identified by two or more  sectors, as well as those for public policy reasons we considered  important to include in our model. We subsequently identified the  corresponding FEVS questions not included in the EEI that reflected the  concepts for each of the drivers. We selected at least one FEVS question  as a proxy for each of the potential drivers that we identified, as shown in  table 8. The questions that we selected were those we determined to be  the most actionable by managers and representative of the potential  driver. We also selected three drivers and questions for other public  policy considerations.
section_title: Government-wide Statistical Analysis
paragraphs: To assess the relationship between potential drivers and employee  engagement as measured by our index, controlling for other factors, we  used linear multiple regression analysis using FEVS 2014 data. Our  ordinary least squares regression analysis assesses the unique  correlation between the potential drivers and engagement, controlling for  respondent characteristics. For most models, we controlled for  supervisory status, agency tenure, location, veteran’s status, and age. In  other models we also controlled for respondent’s sex, education, reported  likelihood of leaving their job in the near future, race, Hispanic ethnicity,  disability status, and sexual orientation. With the exception of age group  and agency, we used the modal category of the sample as a referent  category. We also included a variable to control for how many of the 15  index questions the respondent answered. In general, our agency  variable included dummies for 37 individual agencies and an intercept for  the remaining agencies, which tended to be substantially smaller in size;  results were similar when we tested a model with intercepts for all  individual agencies available in the sample. We used statistical software  to account for the sample design in our variance estimates.
The 37 agencies include the 24 CFO Act agencies as well as other agencies that  participated in early FEVS data collection efforts.  is a sample specific measure of how well the variation in the model’s independent  variables (such as agency or demographics) predicts the variation in the dependent or  outcome variable (here, engagement). It runs from 0 to 1, with a score of 0 suggesting that  the model has no explanatory power and a score of 1 suggesting that the independent  variables predict 100 percent of the variation in the dependent variable. some questions offering a “do not know” response category. For  respondents missing or answering “do not know” to specific driver  questions, we imputed data using the agency-level average for that  individual to avoid losing cases in estimation through listwise deletion.
Our model treats drivers of engagement as linear predictors of the  engagement index. This is a strong assumption in light of the fact that the  response categories to the driver questions are ordinal, rather than  interval data. In other words, while the responses are ordered, the  difference between two adjacent categories (such as very negative and  somewhat negative) is not necessarily equal to the difference between  two other adjacent categories (such as neutral and somewhat positive),  and therefore the assumption of linearity may not be appropriate. We  conducted sensitivity tests to ensure that our results were similar when  we treated the drivers as categorical variables including intercepts for  item nonresponse and no basis to judge responses. Given that our results  were similar under either specification, we decided to use the linear  covariates in our models to ease the interpretation of results and to  reduce the degrees of freedom required to estimate the model.
Given the large number of cases in our government-wide analysis, nearly  all of the coefficients on the drivers in the model were statistically  significant. Accordingly, we incorporated a substantive threshold in our  determination of whether an independent variable acted as a driver. We  considered variables to be drivers of engagement if they had a coefficient  that rounded to 3 or above, indicating that on average, each increase in  positivity of responses was associated with a 3 percentage point increase  in the 0 to 100 measure of engagement. In other words, a coefficient of 3  implies that, compared to a respondent who answered neutrally to a given  driver question, a respondent who answered very positively (which is two  units above neutral) would have a predicted engagement score 6  percentage points higher.
The results for our government-wide model appear in table 9 in appendix  IV. They demonstrate that while almost all of the questions we tested  attained statistical significance, a subset of questions could be considered  drivers in that they had statistically significant coefficients that rounded to  3 or above. As shown in table 9, we identified six driver questions that  strongly and significantly predicted the employee engagement score after  controlling for agency and employee characteristics—these were  questions 1, 42, 46, 55, 63 and 64.
The strongest driver from our model was an employee’s response to  question 46, a question related to performance management, which asks  whether “my supervisor provides me with constructive suggestions to  improve my job performance.” Compared to employees who gave a  neutral response to this question, employees who strongly agreed had an  average employee engagement score approximately 10.5 percentage  points higher on a 0 to 100 scale after controlling for other factors such as  agency, employee characteristics and other drivers. Similarly, compared  to employees who responded strongly disagree, employees who  answered strongly agree had an EEI score, on average, more than 20  points higher.
The second strongest driver we identified was question 1, “I am given a  real opportunity to improve my skills in my organization.” Compared to  those who answered neutral to this measure of career development and  training, those who answered strongly agree had predicted engagement  scores approximately 8 percentage points higher. Compared to a strongly  disagree response, the EEI score was, on average, approximately16  points higher.
Four other questions had slightly smaller coefficients that rounded to 3 or  above, suggesting that a respondent who answered strongly agree (or  very satisfied) would have a predicted engagement level approximately 5  to 7 points higher than one who answered neutral, controlling for other  factors. Compared to a strongly disagree (or very dissatisfied) response,  the EEI score was, on average, approximately 12 points higher. These  four questions were question 42 (“My supervisor supports my need to  balance work and life issues”); question 55 (“Supervisors work well with  employees of different backgrounds”); question 63 (“How satisfied are  you with your involvement in decisions that affect your work?”); and  question 64 (“How satisfied are you with the information you receive from  management on what is going on in your organization?”)
The coefficients in our analysis indicate the unique association between a  given independent variable, such as a driver or employee population  control, accounting for the potential effects of other variables. The R In light of these results, as well as  reasons cited above, we determined that the linear specification is  sufficient for identifying which independent variables appear to best  predict variation in the engagement index. However, we also recognize  that it may be appropriate to relax the assumption of linearity depending  on the research question.
Finally, we tested our model on subsets of the population with response  patterns that could reflect data quality issues, such as missing more than  a third of the index questions or more than half of the driver variables. Our  results from these analyses were consistent with the overall government- wide estimates.
subsections: section_title: Limitations
paragraphs: The potential drivers we considered in our models were selected based  on an extensive review of academic, government, and policy-related  literature and a logical assessment of the particular concepts with which  they related. However, researchers may disagree over which FEVS  questions provide the best and most actionable proxies for the drivers we  identified. Had we selected different questions as proxies for drivers  found in the literature, our results may have been different.
FEVS was not initially designed with the express purpose of measuring  engagement or of identifying factors related to engagement. To the extent  policymakers seek to use data to assess drivers of engagement, best  practices suggest designing a survey or questions to align expressly with  the concepts of interest. Although we believe that FEVS as designed and  currently implemented is sufficient for an analysis such as that presented  above, our sensitivity tests suggest that alternative measures of  engagement or drivers might provide different insights as to which factors  most strongly predict engagement.
Our analysis does not provide insight into the validity of the EEI for  measuring conditions conducive to engagement or employee  engagement directly. Although we found that the 15 questions comprising  the EEI had strong internal cohesion, we did not conduct factor analysis  or additional research to determine whether an alternative scale or  questions better captured the concept of engagement.
Our model is not a causal assessment of the relationship between the  specific FEVS questions included in our model and increased  engagement. While our results identify some areas that might relate to  increased engagement, we cannot be certain that an investment in a  specific driver will result in increases in employee engagement. However,  our results do confirm a general consistency of which drivers of the EEI,  as measured by questions currently available in FEVS data, appear to be  statistically and substantively significant across a wide range of agencies  and subgroups. In other words, across agencies and selected employee  population groups, positive responses to the six FEVS questions in our  government-wide model were associated with increases in the EEI.
section_title: Agency Driver Analysis
paragraphs: To assess potential drivers for agencies, we replicated our government- wide regression model among employees at the 24 CFO Act agencies.  We limited our analysis to the CFO Act agencies because they were of  sufficient size so as to produce reliable results. As shown in tables 10  through 15 in appendix IV, the drivers for the CFO Act agencies were  generally consistent with the results of our government-wide analysis,  with some exceptions. For example, for several agencies, question 63  would not meet our definition of a driver in that the coefficient does not  always round to 3 or above.
section_title: Employee Population Group Analysis
paragraphs: We analyzed the drivers by employee population groups. We selected  employee population groups (1) with different amounts of variation in EEI  levels within the employee population group; (2) with distinct subsets of  the employee population from the others selected (for example, we did  not select both pay category and supervisory status because the  categories would represent similar populations); and (3) for which, in our  opinion, agencies could identify actionable steps for a subset of the  employee population group. The employee population groups we  analyzed were supervisory status, age, veterans status, work location  (headquarters or field), and agency tenure. We then estimated versions of  our regression model that included a subset of employee characteristic  control variables to assess which potential drivers most strongly  correlated with EEI, after controlling for agency and other factors. As  shown in tables 16 to 20 in appendix IV, the drivers for the selected  employee population groups were generally consistent with the results of  our government-wide analysis, with minor exceptions. For example, when  analyzing potential drivers by age group, question 63 does not reach our  threshold for a driver among respondents younger than 40.
section_title: Appendix IV: Results of GAO’s Analysis of the Drivers of the Employee Engagement Index
paragraphs: Appendix IV: Results of GAO’s Analysis of the  Drivers of the Employee Engagement Index Question/Variable  Federal Employee Viewpoint Survey question included in GAO  model  Question No. 1: I am given a real opportunity to improve my  skills in my organization.
Question No. 42: My supervisor supports my need to balance  work and other life issues.
Question No. 46: My supervisor provides me with constructive  suggestions to improve my job performance.
Question No. 55: Supervisors work well with employees of  different backgrounds.
Question No. 63: How satisfied are you with your involvement  in decisions that affect your work?
Question No. 64: How satisfied are you with the information  you receive from management on what’s going on in your  organization?
Question No. 9: I have sufficient resources to get my job done.
Question No. 10: My workload is reasonable.
Question No. 14: Physical conditions allow employees to  perform their jobs well.
Question No. 17: I can disclose a suspected violation of any  law, rule or regulation without fear of reprisal.
Question No. 20: The people I work with cooperate to get the  job done.
Question No. 24: In my work unit, differences in performance  are recognized in a meaningful way.
Question No. 29: The workforce has the job-relevant  knowledge and skills necessary to accomplish organizational  goals.
Question No. 32: Creativity and innovation are rewarded.
Question No. 37: Arbitrary action, personal favoritism and  coercion for partisan political purposes are not tolerated.
Question No. 38: Prohibited Personnel Practices are not  tolerated.
Question No. 41: I believe the results of this survey will be  used to make my agency a better place to work.
Question No. 70: Considering everything, how satisfied are  you with your pay?  Imputation flags by question  Question 1  0.62  ref. . ref. .
Education level  Less than high school  High school diploma/GED or equivalent  Some college (no degree)
Associate’s degree (e.g. AA, AS)
Bachelor’s degree (e.g. BA, BS) ref. .
Master’s degree (e.g. MA, MS, MBA)
Doctoral/professional degree (e.g. Ph.D., MD, JD)
Agency tenure  Less than 1 year  ref. . ref. .
Military service status  No prior military service  ref. .
Currently in National Guard or Reserves  ref. .
Missing  Age group  25 and under  ref. .
Yes, to take another job within the federal government  ref. .
Yes, to take another job outside the federal government  0.2  ref. .
Race  American Indian or Alaska Native  Native Hawaiian or other Pacific Islander  ref. .
Sexual orientation  Heterosexual or straight  ref. .
Gay, lesbian, bisexual, or transgender  I prefer not to say  Flag for number of the 15 index questions for which  respondent answered do not know or gave no response  No index questions missing/do not know  ref. .
1 to 5 index questions missing/do not know  6 to 10 index questions missing/do not know  More than 10 index questions missing/do not know  Office of Management and Budget  ref.
Court Services and Offender Supervision Agency  National Aeronautics and Space Administration  National Archives and Records Administration  Legend: * p<0.05, ** p<0.01, *** p<0.001  Constant: -46.42*** R-squared: 0.74 Number of cases: 392,749  Design degrees of freedom: 392,667  imputation flags for potential driver variables, subgroup models control for 37 distinct agencies, the  number of index questions missing, as well as the following employee population group variables:  supervisory status, location, age group and military status. We incorporated a substantive threshold in  our determination of whether an independent variable acted as a driver. We considered variables to  be drivers of engagement if they had a coefficient that rounded to 3 or above, indicating that on  average, each increase in positivity of responses was associated with a 3 percentage point increase  in the 0 to 100 scale.  imputation flags for potential driver variables, subgroup models control for 37 distinct agencies, the  number of index questions missing, as well as the following employee population group variables:  agency tenure, supervisory status, age group, and military status. We incorporated a substantive  threshold in our determination of whether an independent variable acted as a driver. We considered  variables to be drivers of engagement if they had a coefficient that rounded to 3 or above, indicating  that on average, each increase in positivity of responses was associated with a 3 percentage point  increase in the 0 to 100 scale.
section_title: Appendix V: Comments from the Office of Personnel Management
section_title: Appendix VI: GAO Contact and Acknowledgments
paragraphs: For further information regarding this statement, please contact Robert  Goldenkoff, Director, Strategic Issues, at (202) 512-2757 or  goldenkoffr@gao.gov.
Individuals making key contributions to this statement include Chelsa  Gurkin, Assistant Director; Tamara Stenzel, Analyst-in-Charge; Carl  Barden, Alyssia Borsella, Martin De Alteriis, Deirdre Duffy, Robert  Gebhart, Donna Miller, Anna Maria Ortiz, Ulyana Panchishin, Jerry  Sandau, and Karissa Schafer.