FEMA’s National Risk Index has some weird results
By: Kevin Zerbe
Tags: Fema, Risk, Climate Change, Natural Hazards, Vulnerability

FEMA’s National Risk Index has some weird results

Natural hazard analysis is tough. Doing it for an entire country is even tougher. So why does FEMA stand by some of the NRI’s inaccuracies? Kevin Zerbe Dec 9, 2020·6 min read

It’s great to consider social vulnerability when assessing natural hazard risks. This is a capability many of us have spent years trying to incorporate meaningfully into our hazard risk analyses. Stronger vulnerability assessments will tend to look at social vulnerability as a qualitative supplement to the quantitative natural hazard risk analysis detailing things like physical exposure and likelihoods of future hazard occurrences. Too often, though, social vulnerability is overlooked or treated as an afterthought because it’s extremely difficult to quantify it and fit it into traditional risk assessment forumlas. For this, I applaud FEMA for giving it a real go. But I’m surprised to see FEMA standing by some of the results from the NRI.

For example, Cowlitz County (a non-coastal county in Washington), according to the NRI, is the most at-risk county to coastal flooding in the whole country. The Bronx, according to the NRI, is the most at-risk place in the country for tornadoes.

This seem weird to anybody else?

They say it’s because Cowlitz is a socially vulnerable county and sea level rise is going to make flooding on the Columbia River worse in the future. All true, but what about socially vulnerable places already dealing with sea level rise on top of increasingly strong hurricanes and other storms? Surely they have more risk, right now today, than Cowlitz County does.

And tornadoes in NYC? Again, they cite the Bronx’s high social vulnerability as the reason — even though they admit tornadoes are rare there. But what about places where tornadoes are not rare that also have high social vulnerability? Surely they have more risk than the Bronx does in spite of the Bronx’s social vulnerability score.

In all honesty, I’m not surprised the NRI wound up with some eyebrow-raising results. A large nationwide model like this one is bound to have some inaccuracies when applied to smaller, more granular scales. There’s just no way a large-scale model like this one will fit perfectly to every Census tract in the country. Anyone who does any kind of physical or statistical modeling grapples with this and works to minimize error as much as possible, but acknowledges errors are going to happen. The hope is your results will mostly work, but there’s always a caveat that they may not apply perfectly all the time. Or that results may be different with different variables or parameterization.

What I am surprised by is FEMA’s lack of acknowledging that the NRI is bound to have some issues here and there and should be validated against other means of assessing risk at local and regional scales. There should be a disclaimer saying these results may or may not apply to every jurisdiction, so users should be sure to validate them before taking them at face value. Instead, according to this article anyway, they seem to truly believe Cowlitz is more at risk to coastal flooding than anywhere else in the entire country. More than Norfolk, New Orleans, or Miami — all legitimately coastal places with large populations of socially vulnerable residents as well as various military, commerce, and other real estate exposed to sea level rise and other coastal hazards right now.

A bit of a disclaimer for NRI users

FEMA doesn’t view these results as minor inaccuracies of a large-scale model, and that’s a little worrisome. I hope any users of the NRI know better.

But just in case, here’s the disclaimer I wish FEMA would’ve included alongside the NRI’s release:

Firstly, it’s an index. Indices are great, but they almost never tell the whole story. By their very nature, indices are generalized and compare statistics over large geographic areas. That’s definitely the case with the NRI, which is comparing risk in your jurisdiction relative to the highest and lowest risk scores in the entire country. This might be a helpful way to look at risk if you’re FEMA and your jurisdiction is the entire country, but for most of us we are concerned with a much smaller geographic area. The risk scores in the NRI, as a result, may or may not make sense for everyone. Even FEMA says the following on their website:

“The NRI alone does not generate the risk assessment portion of the plan, but it supports its development with high quality, comprehensive data…”

And also:

“The NRI leverages available source data for 18 natural hazards, social vulnerability, and community resilience to develop a baseline relative risk measurement for each United States county and Census tract.” (emphasis added)

The keyword there is baseline. The NRI is a good starting point for your risk assessment, but starting points are not usually where you end up. All baselines in statistics have variation on either side, with scores above and below the baseline. If you want to use the NRI, you will need to test the results for your jurisdiction against what you know about natural hazard risk from your local perspective. In other words, you need to test whether the baseline score applies to you. Chances are that a national index like this one won’t capture all the nuance of risk at jurisdiction levels.

Secondly, the NRI score itself uses monetized estimated annual losses to buildings, populations, and agricultural assets in its calculations. What this means is that risk is defined by the estimated yearly damage (in dollars) to your buildings, people, and agriculture. This may mean that scores are skewed toward where there are a lot of buildings and people (i.e. big cities, like NYC for example) and/or lots of agricultural assets (i.e. the Midwest). If this sounds like your community, then excellent! The NRI’s scores may be closer to reality for you than others. Either way, you need to test their baseline score with your local knowledge. Also, reliable probabilistic estimates of yearly damage from natural hazards are not available in every county or Census tract in the US, so there may be some assumptions that FEMA is making that might not make sense for your jurisdiction.

And remember that the likelihood of damage to real estate and people is only one way to go about assessing risk. You are required to determine the risks to your community assets if you want FEMA to approve your hazard mitigation plan, but you get to decide what your community assets are. They can (and should) include buildings, people, and agricultural assets, but don’t forget about environmental assets, cultural and historical assets, or assets in other jurisdictions you may rely on (e.g., economic assets like ports or transportation systems). These are more difficult to monetize and more difficult to determine an annual estimate of probable damage. That doesn’t mean they should be left out of your risk assessment!

Finally, risk is one component of the plan. Vulnerability is something different. The NRI does not capture vulnerability at jurisdiction scales. Vulnerability is defined by the characteristics of your community assets that make them more or less susceptible to damage from natural hazards. Since you are the one defining your community assets, and you are the one determining their susceptibility to harm, there is no way for that information to be captured by the NRI.

So, feel free to incorporate the NRI in your planning efforts. It’s a good place to start. But in almost every case, it will need to be refined by you and zoomed in from FEMA’s country-wide 100,000 foot view to a scale that makes more sense for you. If you see some weird results for your town or county, take a closer look before assuming FEMA’s right and your gut is wrong.