How Accurate Are Census Population Estimates?
By: Lyman Stone
Tags: Census, Economics, Puerto Rico, Demographics, Migration

How Accurate Are Census Population Estimates?

Fairly Good, But Far From Perfect

Next week, we will get new Census estimates of state and national population. The headlines will focus on estimated population change from July 1, 2016 to July 1, 2017. That’s an interesting story.

But these are just estimates. It’s not actually as if they went out and knocked on doors or something. That only happens in Decennial Census years. So these estimates are a best guess, but do nonetheless have error rates.

As a result, there are often revisions to back-year estimates. These revisions can be very large, especially in years where Census changes their estimation methodology in a meaningful way, as they did last year. As such, while the main press will focus on the year-over-year estimated change, I make a point of analyzing the revisions to back-year data.

Those revisions can be meaningful. Here’s the average state-level population revision size, for each estimate-year and vintage-year, expressed as the average absolute value change in estimated population:

Apologies this chart isn’t interactive today. Having technical difficulties with Datawrapper. Get interactive version here.

As you can see, the final intercensal revisions were way bigger than any of the annual revision series.

But what may be a bit less obvious is that the size of revisions has declined in the current decade. Here’s the revision size for each year-vintage, but instead of calendar year, it’s years-behind-vintage. “0” in V2016 means the V2016 estimate of 2016, while 0 in V2009 means the V2009 estimate of 2009. 3 in V2015 means the V2015 estimate of 2012, while 3 in V2008 means the V2008 estimate for 2005. Basically, it’s a way of lining up years for which vintages had similar amounts of available information, to see if Census’ information-controlled-estimate-volatility has changed over time. I’ll ignore the intercensal numbers because they are so much bigger and have a form of information (Census results) not available to the other vintages.

As you can see, Census’ revision size is a lot lower than it was. Comparing 2014 to 2004 yields lower revision scale for every year sequence. The same is true for 2015 vs. 2005, except for year 4 where V2015 made a historic revision in a few states, and comparing 2016 to 2006 really shows a big gap.

Some of this is Hurricane Katrina. But then again, the mid-2000s are going to get hurricane volatility in estimates due to Hurricanes Irma, Harvey, and Maria as well!

But on the whole, it seems like Census has reduced the average size of changes. This may be due to method changes. The American Community Survey has improved a lot since the mid 2000s, most components of change data are available earlier than they used to be, and a deeper field of commentary and data work outside of Census may be helping Census improve their estimates.

Or, maybe that’s all wrong, and Census is artificially under-estimating population volatility. One way we can estimate that is to look at the standard deviation in population growth rates for each year shown in each vintage, and compare them to the final intercensal estimate: did Census over- or under-estimate population volatility in the last decade?

It looks as if most estimates did a pretty good job of estimating the amount of variation in growth rates, though on the whole it seems like a bias towards overstatement of growth variance in the estimates, except in 2006 where estimates understated the shock of Hurricane Katrina.

So it doesn’t seem wildly likely that Census is systematically under-estimating the degree of variation in state growth rates. On the whole, Census may allocate population change in the wrong states sometimes, but they basically tend to be right about major changes in the degree of inequality in population growth across states. If they say growth is becoming more uneven, then it’s probably becoming more uneven.

Okay, so, the question then has to be, “How well do population estimates predict Census population?”

To do this, we will annualize growth rates to get decadal growth. For each vintage, we will their estimates up to their last year, then take the annualized growth rate for the decade to that point and extrapolate it forward. Comparing to the 2010 number, we’ll see how different each vintage’s error is for each state.

As you can see, different year-vintages can have a big impact on your expectation of 2010 population. Your expectation of 2010 population for Wyoming would have shifted by about 7% over the course of the forecast period, rising each time, and still wasn’t high enough! A similar story is true for many states: Census was continually pushing up estimates for some states, and those states, kinda sorta tended to also get similar-direction revisions in the last year. Broadly speaking, the more Census had to scramble to follow population estimates upwards via continual upward revision forecasts for 2010, the bigger the absolute value of total-decadal-revision, and last-year-revision, and the bigger the change in during-period-forecasts, the bigger the corresponding change in net revisions as well.

Perhaps an example will help. Let’s look at Wyoming. The chart below shows the various estimate-and-linear-forecast projections of Wyoming state population.

As you can see, each subsequent Wyoming population estimate implied higher and higher 2010 population… and the final intercensal tally came in higher still, including in back-years!

We can see this on the negative side too. Let’s look at Michigan.

Michigan’s population forecast went down, down, down, down, and down, and the final came in… even more down.

Now, there are exceptions. Some states get revisions down then up, some up then down, some just sort of churn… but the point is that, to the extent revisions were biased from 2000–2010, they tended to understate the extremity of change, particularly in back-years.

What does this tell us about current population estimates? Well, it suggests that the pattern of past revision is at least slightly indicative of future revision. Revisions have been much smaller this time around, which suggests future revisions may also be smaller, as data quality has improved markedly.

So…. is this useful at all?