A few example measurements of

agency health

Jonathan Auerbach

George Mason University

jauerba@gmu.edu

@jauerbach_stats

Measuring the Health of the Federal Statistical System

FCSM Research and Policy Conference

October 26, 2023
Goal: suggest some ways to measure health

▶ Three examples:

1. Measure the timeliness of data produced by BJS. (Source: BJS

website)

2. Measure the retention of employees working at ERS. (Source: OPM

FedScope)

3. Measure whether employees at various agencies are able to do their

jobs. (Source: OPM FEVS)

▶ Main challenges:

▷ Not always clear what a healthy agency should look like, and whether

an agency appears healthy is sensitive to how health is measured.

▷ Not all data are available publicly for all agencies, and data are not

always consistent across agencies or over time.
In 2020, data released by BJS took >3 years on

average to process—nearly 2x the 2015-2019 average 2015

2016

2017

2018 2019

2020

2021

2022

0

1

2

3

4

2016 2018 2020 2022

year

average delay between first release of data and reference period

 at the Bureau of Justice Statistics by year of release (delay in years)
But the spike in lateness was due to a 2018-2019

backlog—which is not easily monitored in real time

2015

2016

2017

2018

2019

2020

2021

0

5

10

15

20

25

2016 2018 2020

year

total number of surveys outstanding at the Bureau of Justice Statistics 

(survey "outstanding" if reference year < year < release year)
112 employees left ERS (i.e. “separated”) in

2019—more than 4x the 2009-2018 average

Advanced Degree Other

2005 2010 2015 2020 2005 2010 2015 2020

0

20

40

60

year

number of separations

from the Economic Research Service
But ERS tenure was already declining—low tenure

years correspond with low budget support years

2005

2006

2007

2008

2009

2010

2011 2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

0

10

20

30

2005 2010 2015 2020

fiscal year

"period life expectancy" of employees

at the Economic Research Service
Majority of employees believe they can do their job

40%

60%

80%

100%

2012 2014 2016 2018

year

percent of federal employees who agree with

Q29. My unit has job−relevant knowledge/skills to accomplish organizational goals.

Agriculture REE

Census Bureau

NOAA

BLS

NASA Ames Research Center

Langley Research Center

Naval Research
But only half believe innovation is rewarded

Agriculture REE

Census Bureau

NOAA

BLS

NASAAmes Research Center

Langley Research Center

Naval Research

40%

60%

80%

100%

2012 2014 2016 2018

year

percent of federal employees who agree with

 Q32. Creativity and innovation are rewarded.
And fewer believe new recruits have the right skills

Agriculture REE

Census Bureau

NOAA

BLS

NASAAmes Research Center

Langley Research Center

Naval Research

40%

60%

80%

100%

2012 2014 2016 2018

year

percent of federal employees who agree with

Q21. My work unit is able to recruit people with the right skills.
FEVS provides unique insight into data production

▶ The main benefit of FEVS is that employees provide their viewpoint

on abstract topics that cannot be measured directly.

▷ One drawback is that the data are only available publicly at the

“subagency” level.

▶ Another drawback is that FEVS only represents the viewpoint of the

federal employees who choose to respond.

▷ We propose the viewpoint of federal data users also be solicited.

▶ Main challenges:

▷ Not always clear what a healthy agency should look like, and whether

an agency appears healthy is sensitive to how health is measured.

▷ Not all data are available publicly for all agencies, and data are not

always consistent across agencies or over time.
Appendix: Can FEVS be used to measure autonomy?

Agriculture REE

Ce

N

n

O

su

A

s

A

Bureau

BLS

NASAAmes Research Center

Langley Research Center

Naval Research

60%

70%

80%

90%

100%

2012 2014 2016 2018

year

percent of federal employees who agree with

Q17. I can disclose a suspected violation of any law, rule or reg. w/o fear of reprisal.