Federal Workforce: Additional Analysis and Sharing of Promising Practices Could Improve Employee Engagement and Performance
Released: 2015-07-15T00:00:00

Letter

Background

Engaged employees are more than simply satisfied with their jobs. Instead, according to employee engagement literature, engaged employees are passionate about and energized by what they do, are committed to the organization, the mission, and their job, and are more likely to put forth extra effort to get the job done. take pride in their work, A number of studies of private-sector entities have found that increased levels of engagement result in better individual and organizational performance including increased employee performance, productivity, and profit margins; higher customer service ratings; fewer safety incidents; and less absenteeism and turnover. Studies of the public sector, while more limited, have shown similar benefits. For example, the Merit Systems Protection Board (MSPB) found that higher levels of employee engagement in federal agencies led to improved agency performance, less absenteeism, and fewer equal employment opportunity complaints.

The FEVS measures employees’ perceptions of whether, and to what extent, conditions characterizing successful organizations are present in their agencies. OPM has conducted this survey every year since 2010. The EEI is composed of 15 FEVS questions covering the following areas:

Leaders lead, which surveys employees’ perceptions of the integrity of leadership, as well as employees’ perception of leadership behaviors such as communication and workforce motivation.

Supervisors, which surveys employees’ perceptions of the interpersonal relationship between worker and supervisor, including trust, respect, and support. Intrinsic work experience, which surveys employees’ feelings of motivation and competency relating to their role in the workplace.

According to OPM, the EEI does not directly measure employee engagement. Instead, it covers the conditions that lead to employee engagement. Specifically, OPM noted that organizational conditions lead to feelings of engagement, which in turn lead to engagement behaviors, such as discretionary effort, and then to optimum organizational performance.

Sometimes the EEI is discussed in the same context as another workforce metric, the Partnership for Public Service’s Best Places to Work in the Federal Government rankings. Although these scores are also derived from the FEVS, they were created as a way of rating employee satisfaction and commitment across federal agencies. The rankings are calculated using a weighted formula of three different questions from OPM’s FEVS: (1) I recommend my organization as a good place to work, (2) considering everything, how satisfied are you with your job, and (3) considering everything, how satisfied are you with your organization.

Most Agencies Did Not Follow the Government-wide Downward Trend and Maintained or Improved Engagement Levels

The recent government-wide average decline in the EEI masks the fact that the majority of federal agencies sustained and a few increased EEI levels during the same period. From 2006 through 2014, government- wide EEI levels increased to an estimated high of 67 percent in 2011 and then declined to an estimated 63 percent in 2014, as shown in figure 1.

The decline in the EEI that began after 2011 is the result of several large agencies bringing down the government-wide average. For example, we found that 13 out of 47 agencies saw a statistically significant decline in their EEI score from 2013 to 2014. While this is 28 percent of agencies, they employ nearly 69 percent of federal employees and include the Department of Defense, Department of Homeland Security, and Department of Veterans Affairs. Meanwhile, the majority of agencies sustained their EEI levels and a few improved them, as shown in figure 2. Between 2013 and 2014, of 47 agencies included in our analysis of the EEI, 3 increased their scores, 31 held steady, and 13 declined.

The recent government-wide downward trend in employee engagement levels coincided with external events—such as sequestration, furloughs, and a 3-year freeze on statutory annual pay adjustments from 2011 to 2013—that OPM and others contend negatively affected federal employee morale. For example, in March 2014, we reported that officials from agencies—including those that furloughed employees— raised concerns about how sequestration affected the morale of current employees.

Even one agency with a downward trending engagement score is not to be taken lightly and there is opportunity for improvement at all federal agencies. However, the large number of agencies that sustained or increased their levels of employee engagement during this time suggests that agencies can positively influence employee engagement levels even as they weather difficult external circumstances. For example, the FTC maintained a consistent estimated 76 percent EEI score—well above the government-wide average—throughout the period of general decline.

Leadership Component of the EEI Consistently Scores the Lowest

Of the three components that comprise the EEI, employees’ perceptions of leaders consistently received the lowest score, and at times was about 20 percentage points lower than the other components. Moreover, from a high point in 2011, leadership scores saw the greatest decrease and accounted for much of the government-wide average decline in the EEI, as figure 3 shows.

The questions comprising the EEI leadership component focus on integrity of leadership and on leadership behaviors such as communication and workforce motivation (see table 1). Three of the five questions are specific to senior leaders—department or agency heads and their immediate leadership team responsible for directing policies and priorities and typically members of the Senior Executive Service or equivalent (career or political). Two are specific to managers—those in management positions who typically supervise one or more supervisors.

In our 2003 work on transformations, we found that leaders are the key to organizational change—they must set the direction, pace, and tone, and provide a clear, consistent rationale that brings everyone together behind a single mission.

The relative strength of the supervisors component of the EEI suggests that the employee-supervisor relationship is an important aspect of employee engagement. These questions focus on the interpersonal relationship between worker and supervisor and concern supervisors’ support for employee development, employees’ respect, trust, and confidence in their supervisor, and employee perceptions of an immediate supervisor’s performance. This is consistent with MSPB research, which suggests that first-line supervisors are key to employee engagement and organizational performance.

Intrinsic work experience was the strongest EEI component prior to 2011, but fell during the period of government-wide decline in engagement levels. These questions reflect employees’ feelings of motivation and competency related to their role in the workplace, such as their sense of accomplishment and their perception of utilization of their skills.

Pay Category and Supervisory Status Had the Widest Variation in EEI Levels

Understanding how employee engagement varies within differing populations of employees can enable agency leaders to consider how different cohorts experience their environment and thus can help leadership determine how to focus engagement efforts. For example, knowing that employees with fewer supervisory responsibilities could be less engaged—and could be having a negative effect on organizational performance—could spur agency leaders to direct additional resources to understanding the needs of this subset of the workforce and improving their sense of engagement.

We found that government-wide, the greatest variation in EEI levels was related to pay category and supervisory status. For example, respondents in progressively lower General Schedule (GS) pay categories had progressively lower levels of engagement government-wide. In contrast, employees in the SES pay category reported consistently higher engagement levels—at least 10 percent more than any lower pay category. While there was less difference between the EEI levels of other pay categories, employees in the GS 13 through 15 categories consistently had higher EEI levels than employees in all other lower GS pay categories. Employees in the Prevailing Rate System, commonly known as the wage grade system, consistently had the lowest EEI levels. For example, in 2014, EEI levels for respondents in the SES pay category were an estimated 84.2 percent compared to an estimated 54.7 percent for respondents in the wage grade pay category.

Similarly, respondents with fewer supervisory responsibilities had progressively lower EEI levels government-wide. Because employees in higher pay categories are likely to have more supervisory responsibilities, responses by pay category and supervisory status represent similar populations. Variations in EEI levels by supervisory status are shown in figure 4.

With respect to other populations of employees, agency tenure, federal tenure, age, and race consistently resulted in some variation but less than pay category and supervisory status. For example, in 2014, American Indian or Alaska Native respondents reported the lowest engagement for any race category with an estimated EEI of 57.6. Asian respondents reported the highest levels of engagement with an estimated EEI score of 68.4.

Gender, ethnicity (Hispanic/non-Hispanic), and work location (headquarters/field) consistently had the least variation. For example, in 2014, males had an estimated EEI of 63.2 percent and females an estimated EEI of 63.3 percent. For results of our analysis of employee population groups, see appendix II.

Key Drivers and Lessons Learned for Strengthening Employee Engagement

Overall we found that what matters most in improving engagement levels is valuing employees—that is, an authentic focus on their performance, career development, and inclusion and involvement in decisions affecting their work. The key is identifying what practices to implement and how to implement them, which can and should come from multiple sources— FEVS and other data sources, other agencies, and OPM. The lessons learned from our three case study agencies were that the goal should not be to just increase a number—that is, have a high EEI—but should also include a focus on improving the organization.

Performance Conversations Are the Strongest Drivers of EEI Levels

Of the various topics covered by the FEVS that we analyzed, we identified six that had the strongest association with higher EEI levels compared to others, as described in figure 5. We used regression analysis to test which selected FEVS questions best predicted levels of employee engagement as measured by the GAO-calculated EEI, after controlling for other factors such as employee characteristics and agency.

Constructive performance conversations. We found that having constructive performance conversations was the strongest driver of the EEI. For the question “My supervisor provides me with constructive suggestions to improve my job performance,” we found that, controlling for other factors, someone who answered “strongly agree” on that FEVS question would have, on average, an engagement score that was more than 20 percentage points higher than someone who answered “strongly disagree” on the 5-point response scale.

As we found in our March 2003 report on performance management, candid and constructive feedback helps individuals maximize their contribution and potential for understanding and realizing the goals and objectives of an organization. At Education, one of our case study agencies, the Office of the Chief Information Officer (OCIO) implemented a process to help ensure that constructive performance conversations regularly occur. In addition to department-wide requirements for supervisors to hold two performance conversations a year, OCIO officials said that they require all supervisors to offer OCIO employees optional quarterly conversations. These quarterly performance conversations are guided by a set of specific topics that supervisors and employees developed together to ensure that employees receive consistent and regular constructive feedback and coaching.

Career development and training. Our analysis found that career development and training was the second strongest driver. For the question, “I am given a real opportunity to improve my skills in my organization,” we found that, controlling for other factors, someone who answered “strongly agree” to that question would have, on average, an engagement score that was 16 percentage points higher than someone who answered “strongly disagree.”

As we found in 2004, the essential aim of training and development programs is to assist an agency in achieving its mission and goals by improving individual and, ultimately, organizational performance. At NCUA, another of our case study agencies, officials said the agency focused on providing training for employees throughout their careers. For example, NCUA requires each employee to develop an individual development plan. For employees new to credit union examining—a majority of employees—NCUA has a standardized 18-month training program that combines classroom and practical work. New examiners must complete a core set of courses and may also choose additional elective courses. NCUA officials said that they are constantly assessing formal and informal training for entry-level employees to identify areas to improve the curriculum and instruction. For more experienced examiners, NCUA provides continuing training and development, according to these officials.

Remaining drivers. For the remaining four drivers, we found that, controlling for other factors, someone who answered “strongly agree” or “very satisfied” to those questions would have, on average, an engagement score that was 12 percentage points higher than someone who answered “strongly disagree” or “very dissatisfied.” Those four drivers are work-life balance (“My supervisor supports my need to balance work and other life issues”), inclusive work environment (“Supervisors work well with employees of different backgrounds”), employee involvement (“How satisfied are you with your involvement in decisions that affect your work”), and communication from management (“How satisfied are you with the information you receive from management on what’s going on in your organization”). Examples of how our three case study agencies implemented practices consistent with these drivers include the following:

Work-life balance. FTC officials implemented an outreach strategy to inform staff about child and elder care resources after learning that employees were not aware of the services or did not know that they qualified for these services. Officials said employee knowledge of and agency commitment to these kinds of programs enhances supervisor support for work-life balance. Similarly, to support work-life balance, as part of its engagED initiative, Education revised telework policies, provided training for managers and employees on the new polices and on working in a telework environment, and improved infrastructure to make telework as effective as time spent in the office, according to Education officials.

Inclusive work environment. The FTC established an agency- wide Diversity Council to develop comprehensive strategies to promote understanding and opportunity throughout FTC. FTC officials said that employees of all levels were interested in forming such a council. This included employees who experienced firsthand the diversity issues as well as managers who could address those issues. The goal of FTC’s Diversity Council— composed of representatives from each bureau and office—is to engage employees and supervisors across the agency, make recommendations for improving diversity, and foster the professional development of all agency employees, according to these officials.

Employee involvement. Education’s Office of General Counsel (OGC) has a permanent employee-driven Workforce Improvement Team (WIT) that grew out of an office-wide meeting with employees at all levels to involve employees in the discussions about the FEVS results. As a result of this group’s work, Education’s OGC management introduced additional training and professional development opportunities and improved employee on-boarding through a new handbook and mentoring program. Education’s OGC officials said that the staff-driven WIT has created feelings of stronger ownership, engagement, and influence in office decision making. Education’s OGC officials said that OGC’s management seeks feedback from staff, including from the WIT, to evaluate the effectiveness of improvement efforts. These officials said this strengthens two-way communication, which improves employee engagement and organizational performance.

Communication from management. NCUA officials told us that the head of the agency and its senior leaders communicate with line employees (who are mostly in the field) through quarterly webinar meetings. The meetings are scheduled to accommodate the field employees’ frequent travel schedule and generally start with any “hot topics” and continue with discussion of agency efforts to meet mission goals. The agency head takes questions in advance and during the webinar and, when needed, participants research and share responses with agency employees. According to NCUA officials, these regular, substantive conversations demonstrate top leadership’s commitment and respect for all employees as valued business partners.

These key drivers can help agencies develop a culture of engagement as agencies embed them into the fabric of everyday management practices, rather than simply reacting to the results of the most recent FEVS. Importantly, these six practices were generally the consistent drivers of higher EEI levels when we analyzed them government-wide, by CFO Act agency, and by selected employee populations (such as agency tenure and supervisory status). Because these six practices are the strongest predictors of the EEI, this suggests they could be the starting points for all agencies embarking on efforts to improve engagement.

Lessons Learned to Improve Employee Engagement Efforts

Our case study agencies also identified three key lessons for improving employee engagement.

Any change must be implemented using effective management practices.

The EEI alone is not enough; agencies must look to other sources of data for a complete picture of employee engagement levels in their organization and its components. Improving engagement and organizational performance takes time and does not neatly follow the survey cycle; change may involve several efforts and effects are seen at different points in time.

Use Effective Management Practices to Build a Culture of Engagement

Our three case study agencies attributed their high or increasing levels of engagement to overall effective management practices more so than to efforts specifically aimed at improving engagement levels. Officials at these agencies said they pay attention to employee engagement scores, but also focus on overall positive organizational health and culture and on how their agency implements change efforts. Some of the practices agencies cited parallel those we identified in 2003 as key to successful organizational transformation, including top leadership involvement, consistency, creating a line of sight linking individual results to organizational performance, and employee outreach.

Top leadership involvement. Officials from all three of our case study agencies said that top agency leaders were directly involved in organizational improvement efforts. We have previously reported top leadership that is clearly and personally leading the change presents stability and provides an identifiable source for employees to rally around and helps the process/efforts stay the course. For example, Education officials said Education’s Chief Information Officer is directly involved in efforts to address FEVS scores—he is directly involved in the data analysis, reviewing Education’s OCIO action plans developed by each of his subordinate directors, overseeing implementation of strategies, and assessing their effectiveness.

Consistency. Officials at Education’s OCIO said it is important to ensure that policies are applied consistently, which is the goal of that office’s Speaking with One Voice initiative. The biweekly management meetings to discuss and clarify the implementation of department policies (e.g., telework, resources, and employee bonuses) were instituted after conversations with employees revealed that policies were inconsistently applied. As a result of the initiative, Education’s OCIO officials said employees know that senior leaders are paying attention to how policies affect employees and are accountable for ensuring appropriate implementation.

Line of sight. FTC officials emphasized the importance of creating a line of sight between the agency’s mission and the work of each employee. As we have previously reported, successful organizations create a “line of sight” showing how team, unit, and individual performance can contribute to overall organizational results. FTC officials said that the agency lists every employee that contributed to a case in the pleadings, from the attorneys and paralegals to the information technology specialists who provided computer support. Officials said that legal actions are the culmination of the efforts of many employees, both mission and mission- support staff, and including their names on pleadings helps create a line of sight from each employee’s contribution to the organization’s success. Further, FTC officials said they recognize how mission support functions, such as excellent human resources customer service contribute to the agency mission. For example, FTC officials said that they emphasize to the human resources staff that their prompt handling of payroll and benefits issues contributes to the overall efficiency and mission accomplishment by minimizing the time other FTC employees expend on these concerns. As a result, mission employees can focus on accomplishing their mission-related responsibilities.

Employee outreach. According to officials at all three of our case study agencies, they all reach out to employees and their labor union representatives, if applicable, to obtain insight into their FEVS scores or to inform other improvement efforts. Our 2003 report found that employee involvement strengthens the improvement process by including frontline perspectives and experiences. By participating in improvement task teams, employees have additional opportunities to share their experiences and shape policies and procedures as they are being developed and implemented. For example, in 2012, while NCUA’s EEI score was above the government-wide level, FEVS questions about awards, performance appraisals, and merit-based promotions were its lowest scoring categories. NCUA officials said they contracted with an external facilitator to conduct workshops and webinar-based feedback sessions with employees to gain insight into their FEVS results and identify root causes influencing the survey scores. These officials said that using external facilitators offered employees confidentiality and created an environment that encouraged open conversations. Based on these feedback sessions, NCUA created an internal employee-driven committee to inform revisions to the awards, performance appraisals, and merit-based promotion process, and developed recommendations for NCUA’s management to implement these changes. Most of the committee’s recommendations were implemented.

Supplemental Data Sources Enable Agencies to Pinpoint Efforts

According to officials at our case study agencies, while the EEI provides a useful barometer for engagement, other indicators can provide officials with a deeper insight into reasons for engagement levels and areas for improvement. Other data such as turnover rates and equal employment opportunity (EEO) complaints—which are likely already collected by federal agencies—can provide additional insight and strategies for improving employee engagement. Notably, MSPB found that there is a statistically significant correlation between higher levels of employee engagement and fewer EEO complaints. Officials in the three case study agencies said that they pay attention to their FEVS scores, but other sources of data can provide explanatory or agency-specific information valuable to developing improvement strategies.

As one example, NCUA officials said they identified a slight increase in turnover in recent years from the human capital data and are reviewing their exit survey process in calendar year 2015 to determine the best way to gather this information as well as to identify trends in reasons for employees leaving. NCUA officials said providing applicants with a clear understanding of the work will help to ensure a good position fit initially, which leads employees to stay engaged with the organization. Credit union examiners often travel between credit unions and do not regularly report to an office. Therefore, NCUA is revising its vacancy announcements to better communicate the nontraditional work environment of field staff, which officials said comprises nearly 80 percent of the agency’s workforce. For example, NCUA officials said that applicants are asked to also indicate preferences on working independently.

Effects of Engagement Efforts Occur Over Multiple Years

Case study agency officials told us that they take a multi-year, multi-prong approach to improving engagement and do not base engagement efforts solely on the survey cycle or focus their attention on year to year changes in the EEI. Some case study agency officials said a single survey cycle does not provide enough time to implement changes and see results because real change usually takes more than 1 year. The FEVS cycle begins around May and agencies receive results in September or October. It may be late-winter or early-spring before an agency will have designed an action plan. By the time the next survey cycle begins, agencies may still be interpreting results and developing and implementing their action plans.

Moreover, according to case study agency and other officials we interviewed, the annual survey cycle does not allow enough time for employees’ perceptions to change before the next cycle begins. For example, an Education official said that it took a few years to see the effects of engagement-related actions. Members of the Chief Human Capital Officers Council and National Council on Federal Labor- Management Relations joint working group on employee engagement said that the effects of initiatives implemented to improve engagement will not be reflected in the EEI scores for at least a couple of years, which makes evaluating their effectiveness challenging.

OPM officials agreed that efforts to improve engagement should not be based on the survey cycle, but noted the benefits of an annual survey. Specifically, OPM stated that agencies are increasingly using the FEVS as a management tool to help them understand issues at all levels of an organization and to take specific action to improve employee engagement and performance. An annual survey such as FEVS can help ensure that newly appointed agency officials (or a new administration) can maintain momentum for change, as the surveys suggest employees are expecting their voices to be heard. Further, OPM officials noted if agencies, managers, and supervisors know that their employees will have the opportunity to provide feedback each year, they are more likely to take responsibility for influencing positive change.

Instead of focusing exclusively on FEVS and EEI scores, case study agencies took a longer term approach to their engagement efforts. For example, according to officials, Education established engagED, a long- term cultural change initiative, to build a more innovative, collaborative and results-oriented agency and create a more engaged workforce. This initiative focused on three areas—increasing multi-way communications, performance accountability, and professional growth opportunities—which officials said were consistently identified as challenges through the FEVS analysis and facilitated feedback discussions. Instead of focusing on one specific FEVS question, Education identified these broad themes to bring about a systemic change, according to officials. EngagED addressed these challenges through several actions intended to prompt thoughtful discussion among employees, accountability for results, and developmental opportunities. Among others, these actions included the following:

Quarterly all-staff meetings with the Secretary to discuss various topics.

A “lunches with leaders” program allowed agency employees more access, input, and participation in key topics discussed by senior agency leaders.

A redesigned performance appraisal system to simplify and standardize performance rating levels to more clearly reflect performance expectations, and consistently recognize and reward successful performers within principal offices and across the agency.

Periodic leadership summits to provide agency leaders with developmental activities identified by staff that are focused on teams, individual leadership, and problem resolution.

An Education Policy Briefing Series to provide agency employees with an opportunity to learn about cutting-edge education issues that relate to the goals and work of the agency and to provide a forum for staff to interact and share expertise.

According to Education officials, this longer-term, broad-based approach helped Education improve its scores even while government-wide scores were decreasing. Officials said they monitor each year’s results, but their success demonstrates the value of taking a long-term perspective.

OPM Has Created Tools and Resources to Improve Engagement, but They Fall Short of Supporting a Holistic Approach to Improving Engagement and Linking to Performance

Limitations in Data Analysis and Reporting Hinder Agencies’ Ability to Target Resources and Assess Progress

OPM provides a number of tools and resources to support agencies’ efforts to use EEI data to identify areas that need improvement, as shown in table 2. However, these tools do not provide agencies with the drivers of the EEI or enable agencies to determine if changes in EEI levels are meaningful.

One of OPM’s key strategic goals is to help agencies create inclusive work environments where a diverse federal workforce is fully engaged and energized to put forth its best effort, achieve their agency’s mission, and remain committed to public service. OPM’s strategic plan for fiscal years 2014-2018 states that it will ensure agencies target, address, and measure key drivers of employee engagement. However, OPM does not analyze the drivers of the EEI or provide agencies the tools to do so. A driver analysis based on FEVS questions can help agencies more effectively target limited resources and can provide a roadmap to design strategies to improve EEI levels.

OPM officials told us that they do not conduct a driver analysis to determine which FEVS questions are associated with higher EEI scores and report them via OPM’s online tools because they would have to use a more complicated and less transparent method of calculating the EEI. Specifically, to conduct a driver analysis OPM would have to calculate the EEI for each individual respondent by determining the proportion of positive responses to the 15 EEI questions for each respondent. OPM would also have to account for unanswered questions. OPM currently calculates the EEI by averaging all the positive responses to the EEI questions for the group of respondents. While OPM officials acknowledged the importance and value of the individual level calculation for determining drivers, they said that the benefit of the current method of calculating the EEI is that it is simpler and officials can see how the scores are calculated—as an average. However, OPM officials noted that they could separately conduct a driver analysis outside of the online tools. In 2006, OPM conducted a regression analysis to identify which questions best predicted overall job satisfaction, overall satisfaction with the organization, and intent to stay or leave. This suggests that OPM has the capability to conduct such an analysis.

Research on employee engagement emphasizes the importance of identifying the drivers of an engagement or related metric as an initial step in improving employee engagement. For example, the Partnership for Public Service’s Best Places to Work in the Federal Government guidance lists a driver analysis as a key element in determining where agencies should focus their action planning efforts. If managers understand the drivers of engagement, then they can better target their engagement efforts, particularly in times of limited resources. The results of our driver analysis demonstrate consistency of results government- wide and by CFO Act agency as well as selected employee populations. Similar or even more limited investment of OPM resources could yield information that would benefit all agencies. However, by not determining which FEVS questions are associated with higher EEI levels, OPM is missing an opportunity to assist agencies in targeting their engagement resources.

Agencies Lack Key Data to Assess Progress in Strengthening Engagement

As noted earlier, the administration has said agency leaders will be held accountable for making employee engagement a priority and making it an integral part of their agency’s performance management system. For example, the December 2014 memorandum on engagement and performance calls on agencies to incorporate engagement measures into SES and agency performance plans. According to the memorandum, SES performance plans are to include a measurable component related to improving employee engagement by 2016. GPRAMA annual performance plans are to include baselines and organizational targets for strengthening employee engagement with a focus on a percent change. As agencies begin to use engagement measures to inform other performance measurement decisions, understanding whether EEI changes are statistically significant will become especially important.

However, OPM does not report whether changes in agency EEI scores are statistically significant—that is, whether the change is meaningful and not due to random chance. As a result, agency officials do not have the information they need to appropriately interpret changes in the EEI. OPM does provide agencies with absolute changes in the EEI—increases and decreases that may or may not be statistically significant due to sampling variability. The method we used to determine statistical significance showed that only 34 percent (16 of 47) of the absolute changes in agency EEI scores from 2013 to 2014 were actually statistically significant.

Without understanding whether changes are statistically significant, managers may take action based on data that has limited meaning. For example, a manager might assume an annual increase in the EEI meant specific engagement efforts were successful when they were not or assume an annual decline in the EEI meant specific engagement efforts were not successful and abandon an effort too soon. Statistical significance is a function of two things: (1) the size of the change—the increase or decrease in the EEI, and (2) the size of the population sampled. In general, the smaller the sample, the larger the change needs to be before it is statistically significant, and the larger the sample, the smaller a change needs to be to be significant. In FEVS, sample sizes tend to be substantially smaller at smaller agencies. For example, from 2013 to 2014 the Federal Mediation and Conciliation Service, with a workforce of over 200 employees, had a 2 percentage point change in the EEI, which was not statistically significant, based on the method we used to determine statistical significance. During that same period, the Department of Defense—the largest federal agency with over 700,000 employees—had about a 0.7 percentage point decrease in the EEI; this was a statistically significant change. Without knowing whether changes in the EEI are statistically significant, agency officials do not have the context to determine whether a change is meaningful.

OPM Has Efforts to Identify and Share Promising Practices Underway But Needs More Focus on Linking Engagement and Performance

As agencies move from analyzing data to developing strategies to improving engagement and linking it to organizational performance, the specific strategies and lessons learned from the experiences of other agencies can be beneficial to those who may be seeking information related to improving employee engagement and performance. Table 3 describes OPM efforts to help agencies develop and implement strategies to improve employee engagement and link engagement to performance.

OPM has several efforts under way to support agencies’ efforts to develop and implement strategies to improve engagement and link it to performance. Most of these efforts focus on the identification and sharing of promising practices, where OPM works in partnership with federal agencies. During the course of our audit, OPM officials described the following avenues for identifying these practices—the Engagement Outreach Team, in-person events, the CHCO Council–National Council on Federal Labor-Management Relations Employee Engagement Working Group, and through the designation of agency-appointed Senior Accountable Officials. OPM officials said their goal is to use the Community of Practice page on UnlockTalent.gov as the platform for sharing the identified promising practices. However, as of May 2015, OPM has only posted limited content on the Community of Practice page—video clips from three events.

Engagement Outreach Team. OPM’s Engagement Outreach Team is a seven-person intra-agency team formed in August 2014, representing four OPM offices. Each outreach team member provides individualized support to three to four President’s Management Council agencies and identifies potential promising practices. OPM officials said the Outreach Team speaks with each of their assigned agencies, and based on these conversations, the Outreach Team meets weekly to share successes and challenges, address action items on their respective agencies’ efforts, and identify potential promising practices.

However, as of May 2015, none of the promising practices identified by the Outreach Team had been added to the Community of Practice page on UnlockTalent.gov and OPM officials said they did not have a plan for how they will use the information gathered from President’s Management Council agencies to inform the Community of Practice page. In April 2015, OPM officials said the Outreach Team was evaluating all of the qualitative data received from agencies, including feedback on previous events and resources offered by the team to determine the next best steps forward, including developing content for the Community of Practice page of UnlockTalent.gov. However, officials did not provide a time frame for completing the evaluation or developing the content.

In-person events. During the fall of 2014, OPM hosted three in-person, Washington, D.C.-based events for agency officials aimed at identifying promising practices in improving employee engagement. Specifically, in November 2014, OPM partnered with OMB and the Presidential Personnel Office to hold the Federal Employee Engagement Forum event at the White House. Panels of public- and private-sector representatives discussed (1) approaches to improving employee engagement and performance through use of data, (2) best practices in collaborative engagement strategies, and (3) the best use of tools and strategies to improve employee engagement and performance. OPM officials said approximately 150 people attended including senior agency officials or their staff and representatives from national labor unions and private- sector organizations. OPM also held two sessions for agency officials in OPM’s Innovation Lab to identify strategies for improving employee engagement. OPM officials said about 20 people attended each session from 11 different agencies. Participants included CHCOs and other senior leaders as well as other agency officials involved with employee engagement efforts.

On May 18, 2015, following a briefing on the preliminary findings of our audit, OPM posted video clips on the Community of Practice page of UnlockTalent.gov from the Federal Employee Engagement Forum held at the White House and another in-person event, the SES Leadership Event. Regarding the Innovation Lab events, according to OPM officials, summaries of the sessions were shared with participants and by request with other officials. OPM officials said they have no further plans for when and how to share the Innovation Lab summaries with a broader government-wide audience.

CHCO Council–National Council on Federal Labor-Management Relations Employee Engagement Working Group. Coordinated by the CHCO Council Executive Director, the working group was launched in February 2014 with representatives from 15 agencies and 8 nongovernment organizations including federal employee unions and organizations representing managers and executives. The group meets on approximately a quarterly basis and is organized into three subcommittees—(1) promising practices, (2) key enablers and barriers, and (3) measures and incentives. Two representatives, one each from an agency and a labor union, co-chair the full working group and each of the subcommittees. The group’s 2014 work culminated in presentations to the CHCO Council and the National Council on Federal Labor-Management Relations.

According to the CHCO Council Executive Director, a key goal of the working group is to identify practices that will be shared on the Community of Practice page of UnlockTalent.gov. However, despite the preliminary work as reflected in the November 2014 presentations, no practices had been posted on the Community of Practice page, as of May 2015. Further, according to the CHCO Council Executive Director, agencies have been reluctant to hold their engagement practices up as a model.

Agency-Appointed Senior Accountable Officials. According to OPM officials, from March through May 2015, OPM met once with each agency’s Senior Accountable Official to discuss their agency’s engagement baseline, plans for engagement activities, assistance needed, and best practices. In addition, OPM officials said in May 2015 they hosted a workshop for the Senior Accountable Officials to share leading practices and strategies to address common challenges. OPM officials said materials from this event were provided to participants, and included examples from participating agencies. OPM officials said after this workshop, they designed five workgroups for Senior Accountable Officials or their designee to work with officials from the People and Culture CAP goal to address specific engagement challenges. OPM officials said the workgroup’s outcomes and next steps will be shared in the fall of 2015, when the 2015 survey results are available to agencies.

Agencies Lack Examples or Guidance to Link Engagement to Agency Performance

Linking increased employee engagement to improved organizational performance is important because it recognizes that improved engagement is not an end in itself. Instead, the ultimate aim is to enhance the ability of agencies to cost-effectively carry out their missions. The administration’s December 2014 memorandum calls for agencies to establish the linkage between employee engagement and mission performance. Specifically, agencies are to collect return on investment information—that is, whether a change in engagement levels resulted in improvements to performance metrics related to agency mission, such as a reduction in error rates.

OPM, as the lead agency on the CAP employee engagement subgoal, has provided limited examples or guidance for how agencies could establish a linkage between engagement and performance. According to OMB officials, establishing such a link is the key step in agency efforts to improve employee engagement and performance. OPM officials said they began a study on how to link engagement to outcomes at three agencies whose outputs can be quantitatively measured. These agencies are the U.S. Mint’s Bureau of Engraving and Printing, the Patent and Trademark Office, and OPM’s Retirement Services Division. Specifically, OPM planned to analyze the relationship between EEI scores and production or performance measures at a team level. However, this effort has been delayed with no estimated completion date, and officials said that linking the EEI to team level performance or production outputs has been more difficult than anticipated. Following a briefing on the preliminary findings of our audit, OPM posted a video clip from the Federal Employee Engagement Forum panel on linking metrics to business outcomes. While this is an important step, the video segment featured only one example from a federal agency.

Because experience with linking employee engagement in the public sector to organizational performance is limited (as indicated by our interviews with OPM and OMB officials and the literature that we reviewed), agencies do not have a clear model on how to make this link and demonstrate that it resulted in improved mission accomplishment. Even at our three case study agencies—each of which have high or improving EEI scores—officials said that they lacked sufficient in-house expertise to develop and conduct such an analysis.

Pursuant to the People and Culture Employee Engagement CAP subgoal, the administration has established a target of improving employee engagement government-wide to 67 percent by the issuance of 2016 FEVS results, a 4 percentage point increase from the 2014 government- wide score. Without the ability to link increased engagement to improved performance, the extent to which an increase of 4 percentage points will translate into improved performance is unclear. Further, given the time horizon necessary to see real improvements, it is unclear if efforts to improve engagement can be achieved by 2016.

Conclusions

Higher levels of employee engagement can translate into higher levels of organizational performance. However, for agencies to attain the ultimate goal of improving organizational performance, agencies must take a holistic approach—analyzing data, developing and implementing strategies to improve engagement, and linking their efforts to improved performance.

The tools and resources OPM has developed represent an important first step toward helping agencies improve employee engagement. However, OPM cannot ensure officials are correctly understanding and using EEI data, because it does not report on whether EEI changes are statistically significant—that is, whether the changes are due to something other than chance. OPM also does not determine the FEVS questions that lead to increased EEI levels, which would enable agencies to focus on areas that drive engagement.

At the same time, informed decisions require more than just EEI data. When faced with multiple options, agencies need to target their resources to the practices found to drive employee engagement—notably constructive performance conversations. Moreover, as indicated by our case study agencies, how changes are implemented—including the involvement of top leadership, consistency, employee outreach, and creating a line-of-sight between individual and organizational performance—ultimately affects whether those changes merely produce temporary compliance or result in sustainable cultural transformation. Further, other data can be used to provide more specific information on what needs attention. However, efforts take time and should not be seen solely as an annual effort measured by the results of the next survey, but as a continuous process.

Although OPM has developed a process for identifying and collecting promising practices, OPM’s sharing of promising practices has been limited, and whether and when its efforts will come to fruition is unclear. Promising practices around analyzing EEI data and implementing improvement strategies are important; however, given the public sector’s limited experience linking engagement to performance, promising practices on creating this linkage are critical.

Recommendations for Executive Action

In furtherance of its role to support agencies’ efforts to improve employee engagement and performance, we recommend that the Director of OPM take the following three actions: 1. To enable agencies to better target resources for engagement efforts, OPM should annually analyze and report on drivers of the EEI government-wide and by selected subsets of the federal workforce, such as agencies or employee population groups. 2. To enable agencies to identify meaningful changes in EEI levels, OPM should provide agencies with information on whether annual changes to EEI scores, both government-wide and by selected subsets of the federal workforce, are statistically significant. 3. To ensure agencies are leveraging promising practices and lessons learned from other agencies in developing effective strategies to improve engagement and performance, OPM should, in partnership with federal agencies, expand its efforts to share promising practices to include information on linking engagement to mission accomplishment and monitoring how engagement investments improve performance through data-driven reviews, like HRstat; and implement its strategy to share these practices in time to inform agency efforts stemming from their 2015 FEVS results.

Agency Comments and Our Evaluation

We provided a draft of this product to the Director of OPM for comment. In written comments, which are reproduced in appendix V, OPM concurred with our first recommendation and partially concurred with our second and third recommendations.

OPM stated that it concurred with our recommendation to analyze and report on the drivers of employee engagement government-wide and by subsets of the federal workforce. OPM said that starting with the release of the 2015 FEVS, it is committed to conducting, analyzing, and reporting on key drivers of the EEI at the government-wide level and has formed a working group to make recommendations on the scope of the analysis and the reporting in subsets of the federal workforce.

OPM stated that it partially concurred with our recommendation to provide information on whether changes are statistically significant, and noted that among other actions, the working group referenced above is to provide guidance on how best to disseminate this information to agencies. At the same time, OPM maintained that in addition to the EEI, agencies should also rely on multiple indicators to assess organizational performance. This was a key point that we made in our report. Specifically, we said that in addition to the EEI, other indicators—such as turnover data—can provide officials with additional insights into reasons for engagement levels and areas for organizational improvement. However, in order for agencies to meaningfully use EEI data in their analyses, we continue to recommend that OPM provide agencies with information on whether changes in the EEI are statistically significant.

With respect to our third recommendation, OPM noted that while it agreed with our recommendation to share promising practices in time to inform agency efforts based on the FEVS 2015 survey results, OPM disagreed with our assessment of OPM’s efforts. In its written comments, OPM outlined the steps taken to identify and share promising practices after being briefed on the findings of our audit. For example, OPM noted that in May 2015, it held a workshop with the Senior Accountable Officials responsible for agency engagement efforts in an effort to share promising practices and generate solutions to common challenges; OPM also noted it has formed workgroups among the Senior Accountable Officials around specific topic areas, with a goal of sharing those practices with the larger federal community this fall. We have modified our report to include more information on OPM’s efforts and plans to share promising practices with agencies in time to inform the analysis of the FEVS 2015 survey results. However, until such information is shared with the larger federal community and includes models for linking engagement to mission accomplishment, we continue to believe that OPM should take additional actions to assist agencies in leveraging their lessons learned, as we recommended.

We also provided a draft of this product to the Secretary of Education, Chairwoman of the FTC, and the Chairman of NCUA for technical review and comment. We received technical comments from them that we incorporated, as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Director of the Office of Personnel Management, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this product are listed in appendix VI.

Appendix I: Data Preparation and Calculation of the Employee Engagement Index

To determine the trends in employee engagement as measured by the Office of Personnel Management (OPM) Federal Employee Viewpoint Survey (FEVS) and identify key practices to improve employee engagement we analyzed (1) employee engagement index scores (EEI) from 2006 to 2014 and (2) the extent to which selected 2014 FEVS questions predicted EEI scores. This appendix describes our methodology for preparing the dataset and for calculating the EEI to conduct these analyses. For the methodology and results of our analysis of EEI trends, see appendix II; for our methodology for identifying the drivers of the 2014 EEI, see appendix III; and for the results of our driver analysis, see appendix IV.

Federal Employee Viewpoint Survey

The FEVS provides a snapshot of employees’ perceptions about how effectively agencies manage their workforce. Topic areas are employees’ (1) work experience, (2) work unit, (3) agency, (4) supervisor, (5) leadership, (6) satisfaction, (7) work-life, and (8) demographics. OPM has administered the FEVS annually since 2010; from 2002 to 2010, OPM administered the survey biennially. The FEVS includes a core set of questions, and agencies have the option of adding questions to the surveys sent to their employees.

The FEVS is based on a sample of full- and part-time, permanent, non- seasonal employees of departments and large, small, and independent agencies, which in 2014 represented about 97 percent of the federal executive branch workforce. According to OPM, the sample is designed to ensure representative survey results would be reported by agency, subagency, and senior leader status as well as for the overall federal workforce. Once the necessary sample size is determined for an agency, if more than 75 percent of the workforce would be sampled, OPM conducts a full census of all permanent, non-seasonal employees. For 2014, the total sample size was 872,495. According to OPM, this size was more than sufficient to ensure a 95 percent chance that the true population value would be between plus or minus 1 percent of any estimated percentage for the total federal workforce.

Government-wide, in 2014, 392,752 employees completed surveys for a response rate of 46.8 percent. Among departments and large agencies, the Department of Veterans Affairs had the lowest response rate—32.6 percent—and the National Science Foundation had the highest—77.3 percent.

Data Set Provided by OPM

We analyzed the 2006 through 2014 FEVS data file containing the 84 core questions and demographic variables provided to us by OPM. The datasets contain the full demographic and work unit location information on the respondents. OPM forward coded older FEVS datasets to match the coding for the 2014 survey so that question numbering and response categories were consistent across time. If a question from the 2014 survey instrument was not present in a previous year, OPM coded the questions as missing in the older data set.

To assess the reliability of the FEVS data, we examined descriptive statistics, data distribution, and reviewed missing data. We also reviewed FEVS technical documentation as well as the statistical code OPM uses to generate the index and variance estimates, and we interviewed officials responsible for collecting, processing, and analyzing the data. On the basis of these procedures, we believe the data were sufficiently reliable for use in the analysis presented in this report.

Employee Engagement Index

OPM first introduced the EEI in 2010, when it contained 8 questions. OPM revised it in 2011 to contain the 15 questions that currently comprise the EEI. According to OPM, the EEI is a measure of the conditions conducive to engagement. The EEI consists of three components—leaders lead, supervisors, and intrinsic work experience. OPM calculates the EEI by averaging the component scores, which are an average of the percent of positive responses to each question in the respective component. For a full list of EEI questions, see table 1 earlier in this report.

GAO’s Calculation of the EEI

To determine trends in employee engagement as measured by the FEVS for the years 2006 to 2014, we calculated the EEI using the data provided by OPM. For 2010 to 2014, we calculated the index using responses to the 15 questions contained in the current index. For 2006 and 2008, we calculated the EEI using only 11 of the 15 EEI questions because the 2006 and 2008 surveys did not include four questions that were added to the survey in 2010. Because OPM calculates the EEI and its component scores at the group level, we used data from OPM to recalculate the EEI for each individual, which enabled us to conduct regression analysis and assess the statistical significance of changes in EEI. The individual level calculation is scaled between 0 and 100 and is based on the proportion of each individual’s positive responses to the 15 constituent EEI questions.

To test the comparability of the 2010 to 2014 estimates, which we calculated based on the 15 questions in the current EEI, and the 2006 and 2008 estimates, which we calculated based on the 11 EEI questions in the survey at that time, we recalculated the EEI for 2010 to 2014 using just the 11 EEI questions present in the 2006 and 2008 survey data. The revised EEI estimates for 2010 to 2014 based upon just the 11 questions averaged about 2.9 percentage points less than the EEI estimates based on the full 15 questions. While the trend line with a peak in 2011 was similar when the estimates were plotted by year, we did not make comparisons from 2008 to 2010 because of the change in the index composition.

To generate estimates for agencies and employee population groups, we aggregated the index across individuals using the appropriate sample weights. We followed the replicate weight variance estimation methodology recommended by OPM to generate sample variance estimates for the index scores. This enabled us to analyze the drivers of engagement and assess the statistical significance of differences.

To ensure that our calculation of the EEI would yield sufficiently similar results as OPM’s methodology for 2014, we assessed the correlation between the two versions. When aggregated to the agency level, our index is nearly perfectly correlated with the OPM measure in 2014 (ρ=0.99971). To confirm the cohesiveness of the individual index, we calculated Cronbach’s alpha, a measure of internal consistency that ranges from zero to 1 for 2014. The alpha value of .94 suggests that the scale of the items captures the majority of the variation in the underlying items, indicating high internal consistency.

Appendix II: Trends in Employee Engagement Index Levels

Using the Employee Engagement Index (EEI) calculated at the individual level, for 2006 through 2014 we analyzed the EEI government-wide, including the EEI components scores; by agency for those with a minimum of 5 years of Federal Employee Viewpoint Survey (FEVS) data for the years analyzed and at least 100 respondents, and by employee population measured by the FEVS.

For each analysis, we determined statistically significant year-to-year changes in the EEI from 2006 to 2014, with the exception of 2008 to 2010 because the questions in the EEI for those years were not comparable. We identified statistical differences by assessing whether the 95 percent confidence intervals of two estimates overlapped or not rather than conducting multiple t-tests; confidence intervals that do not overlap represent differences that are statistically significant. If the change was statistically significant, there is less than a 5 percent probability that the difference occurred by chance. This method of assessing difference is conservative, in that it may underestimate the amount of statistically significant differences in cases of minor overlap of confidence intervals, but does not require us to use a testing methodology modification such as a Bonferroni adjustment to account for multiple comparisons. Table 4 below shows the downward trend in the government-wide EEI and for two of the three components.

Tables 5 and 6 show the EEI trends for the 47 agencies with a minimum of 5 years of FEVS data for the years analyzed and at least 100 respondents and whether year to year changes were statistically significant. These tables are the basis for figure 2.

Employee Population Group Results

Table 7 shows the estimated EEI by employee population category. To determine the extent of variation in responses within an employee population, we measured the greatest possible amount of variation in each of the years 2006 through 2014. Specifically, we measured the percentage point difference between the confidence interval upper bound of the category with the highest EEI and the lower bound confidence interval of the category with lowest EEI in each of the years. Pay category and supervisory status consistently had the widest variation in EEI scores.

Appendix III: GAO’s Analysis of the Drivers of the Employee Engagement Index

Our analysis of the drivers of engagement measures the extent to which selected FEVS questions predict the EEI. To conduct this analysis, we reviewed relevant literature and interviewed knowledgeable individuals to identify and refine a list of potential drivers of engagement, and then identified corresponding FEVS questions not included in the EEI. Using FEVS 2014 data, we then used multiple linear regression analysis to assess the correlation between the driver questions and the EEI, controlling for other factors such as agency and employee characteristics. We used both statistical significance and the magnitude of regression coefficients to define drivers of the EEI. We conducted sensitivity tests to ensure that our results were robust to differences in model specification, functional form, and to the exclusion of cases with high levels of missing data. In addition to our government-wide analysis, we analyzed the drivers among employees at each of the Chief Financial Officers Act agencies and selected employee populations. Results of these analyses were generally consistent with our government-wide analysis.

Identification of FEVS questions

To determine the FEVS questions to include in our statistical model we reviewed relevant literature and interviewed knowledgeable researchers, government officials from the United Kingdom, Canada, and Australia responsible for their comparable public-sector employee survey, and consultants on employee engagement. We then categorized all the potential drivers identified by sector—such as academia, consultants, and the federal government. We selected the drivers identified by two or more sectors, as well as those for public policy reasons we considered important to include in our model. We subsequently identified the corresponding FEVS questions not included in the EEI that reflected the concepts for each of the drivers. We selected at least one FEVS question as a proxy for each of the potential drivers that we identified, as shown in table 8. The questions that we selected were those we determined to be the most actionable by managers and representative of the potential driver. We also selected three drivers and questions for other public policy considerations.

Government-wide Statistical Analysis

To assess the relationship between potential drivers and employee engagement as measured by our index, controlling for other factors, we used linear multiple regression analysis using FEVS 2014 data. Our ordinary least squares regression analysis assesses the unique correlation between the potential drivers and engagement, controlling for respondent characteristics. For most models, we controlled for supervisory status, agency tenure, location, veteran’s status, and age. In other models we also controlled for respondent’s sex, education, reported likelihood of leaving their job in the near future, race, Hispanic ethnicity, disability status, and sexual orientation. With the exception of age group and agency, we used the modal category of the sample as a referent category. We also included a variable to control for how many of the 15 index questions the respondent answered. In general, our agency variable included dummies for 37 individual agencies and an intercept for the remaining agencies, which tended to be substantially smaller in size; results were similar when we tested a model with intercepts for all individual agencies available in the sample. We used statistical software to account for the sample design in our variance estimates.

The 37 agencies include the 24 CFO Act agencies as well as other agencies that participated in early FEVS data collection efforts. is a sample specific measure of how well the variation in the model’s independent variables (such as agency or demographics) predicts the variation in the dependent or outcome variable (here, engagement). It runs from 0 to 1, with a score of 0 suggesting that the model has no explanatory power and a score of 1 suggesting that the independent variables predict 100 percent of the variation in the dependent variable. some questions offering a “do not know” response category. For respondents missing or answering “do not know” to specific driver questions, we imputed data using the agency-level average for that individual to avoid losing cases in estimation through listwise deletion.

Our model treats drivers of engagement as linear predictors of the engagement index. This is a strong assumption in light of the fact that the response categories to the driver questions are ordinal, rather than interval data. In other words, while the responses are ordered, the difference between two adjacent categories (such as very negative and somewhat negative) is not necessarily equal to the difference between two other adjacent categories (such as neutral and somewhat positive), and therefore the assumption of linearity may not be appropriate. We conducted sensitivity tests to ensure that our results were similar when we treated the drivers as categorical variables including intercepts for item nonresponse and no basis to judge responses. Given that our results were similar under either specification, we decided to use the linear covariates in our models to ease the interpretation of results and to reduce the degrees of freedom required to estimate the model.

Given the large number of cases in our government-wide analysis, nearly all of the coefficients on the drivers in the model were statistically significant. Accordingly, we incorporated a substantive threshold in our determination of whether an independent variable acted as a driver. We considered variables to be drivers of engagement if they had a coefficient that rounded to 3 or above, indicating that on average, each increase in positivity of responses was associated with a 3 percentage point increase in the 0 to 100 measure of engagement. In other words, a coefficient of 3 implies that, compared to a respondent who answered neutrally to a given driver question, a respondent who answered very positively (which is two units above neutral) would have a predicted engagement score 6 percentage points higher.

The results for our government-wide model appear in table 9 in appendix IV. They demonstrate that while almost all of the questions we tested attained statistical significance, a subset of questions could be considered drivers in that they had statistically significant coefficients that rounded to 3 or above. As shown in table 9, we identified six driver questions that strongly and significantly predicted the employee engagement score after controlling for agency and employee characteristics—these were questions 1, 42, 46, 55, 63 and 64.

The strongest driver from our model was an employee’s response to question 46, a question related to performance management, which asks whether “my supervisor provides me with constructive suggestions to improve my job performance.” Compared to employees who gave a neutral response to this question, employees who strongly agreed had an average employee engagement score approximately 10.5 percentage points higher on a 0 to 100 scale after controlling for other factors such as agency, employee characteristics and other drivers. Similarly, compared to employees who responded strongly disagree, employees who answered strongly agree had an EEI score, on average, more than 20 points higher.

The second strongest driver we identified was question 1, “I am given a real opportunity to improve my skills in my organization.” Compared to those who answered neutral to this measure of career development and training, those who answered strongly agree had predicted engagement scores approximately 8 percentage points higher. Compared to a strongly disagree response, the EEI score was, on average, approximately16 points higher.

Four other questions had slightly smaller coefficients that rounded to 3 or above, suggesting that a respondent who answered strongly agree (or very satisfied) would have a predicted engagement level approximately 5 to 7 points higher than one who answered neutral, controlling for other factors. Compared to a strongly disagree (or very dissatisfied) response, the EEI score was, on average, approximately 12 points higher. These four questions were question 42 (“My supervisor supports my need to balance work and life issues”); question 55 (“Supervisors work well with employees of different backgrounds”); question 63 (“How satisfied are you with your involvement in decisions that affect your work?”); and question 64 (“How satisfied are you with the information you receive from management on what is going on in your organization?”)

The coefficients in our analysis indicate the unique association between a given independent variable, such as a driver or employee population control, accounting for the potential effects of other variables. The R In light of these results, as well as reasons cited above, we determined that the linear specification is sufficient for identifying which independent variables appear to best predict variation in the engagement index. However, we also recognize that it may be appropriate to relax the assumption of linearity depending on the research question.

Finally, we tested our model on subsets of the population with response patterns that could reflect data quality issues, such as missing more than a third of the index questions or more than half of the driver variables. Our results from these analyses were consistent with the overall government- wide estimates.

Limitations

The potential drivers we considered in our models were selected based on an extensive review of academic, government, and policy-related literature and a logical assessment of the particular concepts with which they related. However, researchers may disagree over which FEVS questions provide the best and most actionable proxies for the drivers we identified. Had we selected different questions as proxies for drivers found in the literature, our results may have been different.

FEVS was not initially designed with the express purpose of measuring engagement or of identifying factors related to engagement. To the extent policymakers seek to use data to assess drivers of engagement, best practices suggest designing a survey or questions to align expressly with the concepts of interest. Although we believe that FEVS as designed and currently implemented is sufficient for an analysis such as that presented above, our sensitivity tests suggest that alternative measures of engagement or drivers might provide different insights as to which factors most strongly predict engagement.

Our analysis does not provide insight into the validity of the EEI for measuring conditions conducive to engagement or employee engagement directly. Although we found that the 15 questions comprising the EEI had strong internal cohesion, we did not conduct factor analysis or additional research to determine whether an alternative scale or questions better captured the concept of engagement.

Our model is not a causal assessment of the relationship between the specific FEVS questions included in our model and increased engagement. While our results identify some areas that might relate to increased engagement, we cannot be certain that an investment in a specific driver will result in increases in employee engagement. However, our results do confirm a general consistency of which drivers of the EEI, as measured by questions currently available in FEVS data, appear to be statistically and substantively significant across a wide range of agencies and subgroups. In other words, across agencies and selected employee population groups, positive responses to the six FEVS questions in our government-wide model were associated with increases in the EEI.

Agency Driver Analysis

To assess potential drivers for agencies, we replicated our government- wide regression model among employees at the 24 CFO Act agencies. We limited our analysis to the CFO Act agencies because they were of sufficient size so as to produce reliable results. As shown in tables 10 through 15 in appendix IV, the drivers for the CFO Act agencies were generally consistent with the results of our government-wide analysis, with some exceptions. For example, for several agencies, question 63 would not meet our definition of a driver in that the coefficient does not always round to 3 or above.

Employee Population Group Analysis

We analyzed the drivers by employee population groups. We selected employee population groups (1) with different amounts of variation in EEI levels within the employee population group; (2) with distinct subsets of the employee population from the others selected (for example, we did not select both pay category and supervisory status because the categories would represent similar populations); and (3) for which, in our opinion, agencies could identify actionable steps for a subset of the employee population group. The employee population groups we analyzed were supervisory status, age, veterans status, work location (headquarters or field), and agency tenure. We then estimated versions of our regression model that included a subset of employee characteristic control variables to assess which potential drivers most strongly correlated with EEI, after controlling for agency and other factors. As shown in tables 16 to 20 in appendix IV, the drivers for the selected employee population groups were generally consistent with the results of our government-wide analysis, with minor exceptions. For example, when analyzing potential drivers by age group, question 63 does not reach our threshold for a driver among respondents younger than 40.

Appendix IV: Results of GAO’s Analysis of the Drivers of the Employee Engagement Index

Appendix IV: Results of GAO’s Analysis of the Drivers of the Employee Engagement Index Question/Variable Federal Employee Viewpoint Survey question included in GAO model Question No. 1: I am given a real opportunity to improve my skills in my organization.

Question No. 42: My supervisor supports my need to balance work and other life issues.

Question No. 46: My supervisor provides me with constructive suggestions to improve my job performance.

Question No. 55: Supervisors work well with employees of different backgrounds.

Question No. 63: How satisfied are you with your involvement in decisions that affect your work?

Question No. 64: How satisfied are you with the information you receive from management on what’s going on in your organization?

Question No. 9: I have sufficient resources to get my job done.

Question No. 10: My workload is reasonable.

Question No. 14: Physical conditions allow employees to perform their jobs well.

Question No. 17: I can disclose a suspected violation of any law, rule or regulation without fear of reprisal.

Question No. 20: The people I work with cooperate to get the job done.

Question No. 24: In my work unit, differences in performance are recognized in a meaningful way.

Question No. 29: The workforce has the job-relevant knowledge and skills necessary to accomplish organizational goals.

Question No. 32: Creativity and innovation are rewarded.

Question No. 37: Arbitrary action, personal favoritism and coercion for partisan political purposes are not tolerated.

Question No. 38: Prohibited Personnel Practices are not tolerated.

Question No. 41: I believe the results of this survey will be used to make my agency a better place to work.

Question No. 70: Considering everything, how satisfied are you with your pay? Imputation flags by question Question 1 0.62 ref. . ref. .

Education level Less than high school High school diploma/GED or equivalent Some college (no degree)

Associate’s degree (e.g. AA, AS)

Bachelor’s degree (e.g. BA, BS) ref. .

Master’s degree (e.g. MA, MS, MBA)

Doctoral/professional degree (e.g. Ph.D., MD, JD)

Agency tenure Less than 1 year ref. . ref. .

Military service status No prior military service ref. .

Currently in National Guard or Reserves ref. .

Missing Age group 25 and under ref. .

Yes, to take another job within the federal government ref. .

Yes, to take another job outside the federal government 0.2 ref. .

Race American Indian or Alaska Native Native Hawaiian or other Pacific Islander ref. .

Sexual orientation Heterosexual or straight ref. .

Gay, lesbian, bisexual, or transgender I prefer not to say Flag for number of the 15 index questions for which respondent answered do not know or gave no response No index questions missing/do not know ref. .

1 to 5 index questions missing/do not know 6 to 10 index questions missing/do not know More than 10 index questions missing/do not know Office of Management and Budget ref.

Court Services and Offender Supervision Agency National Aeronautics and Space Administration National Archives and Records Administration Legend: * p<0.05, ** p<0.01, *** p<0.001 Constant: -46.42*** R-squared: 0.74 Number of cases: 392,749 Design degrees of freedom: 392,667 imputation flags for potential driver variables, subgroup models control for 37 distinct agencies, the number of index questions missing, as well as the following employee population group variables: supervisory status, location, age group and military status. We incorporated a substantive threshold in our determination of whether an independent variable acted as a driver. We considered variables to be drivers of engagement if they had a coefficient that rounded to 3 or above, indicating that on average, each increase in positivity of responses was associated with a 3 percentage point increase in the 0 to 100 scale. imputation flags for potential driver variables, subgroup models control for 37 distinct agencies, the number of index questions missing, as well as the following employee population group variables: agency tenure, supervisory status, age group, and military status. We incorporated a substantive threshold in our determination of whether an independent variable acted as a driver. We considered variables to be drivers of engagement if they had a coefficient that rounded to 3 or above, indicating that on average, each increase in positivity of responses was associated with a 3 percentage point increase in the 0 to 100 scale.

Appendix V: Comments from the Office of Personnel Management

Appendix VI: GAO Contact and Acknowledgments

For further information regarding this statement, please contact Robert Goldenkoff, Director, Strategic Issues, at (202) 512-2757 or goldenkoffr@gao.gov.

Individuals making key contributions to this statement include Chelsa Gurkin, Assistant Director; Tamara Stenzel, Analyst-in-Charge; Carl Barden, Alyssia Borsella, Martin De Alteriis, Deirdre Duffy, Robert Gebhart, Donna Miller, Anna Maria Ortiz, Ulyana Panchishin, Jerry Sandau, and Karissa Schafer.