Spatial Autocorrelation: Neighbors Affecting Neighbors | by Mahbubul Alam | Towards Data Science

The purpose of this article is to introduce an important concept in geospatial analysis — “Spatial Autocorrelation” with a focus on the following things: (i) key concepts of autocorrelation; (ii) industry applications and (iii) an implementation in R
“Everything is related to everything else. But near things are more related than distant things”. Waldo Tobler’s (1969) First Law of Geography
To understand Tobler’s Law above, let’s do a thought experiment.
Let’s randomly pick a house from a listing website such as Zillow and let’s say the listing price of the house is $400K. Next, say, the house next to it is also listed for sale. Now, if you are to predict the price of the second house and given two options, $450K and $1.2M, which one would you pick?
If you picked $450K then you already sub-consciously know what a spatial autocorrelation is. It is indeed a correlation between two neighbors in some common features (e.g. house price). “Auto” means self, “correlation” means association. So in a more formal way of saying — autocorrelation is a measure of similarity between nearby observations that describes the degree to which observations of a variable (e.g. population density) are similar to each other at a given spatial scale. Or simply speaking, “the Moran’s I statistic is the correlation coefficient for the relationship between a variable (like income) and its surrounding values” (source).
Note that there is a similar concept called temporal autocorrelation, where observations (e.g. height, weight etc.) in one time point is correlated to observation in the previous time point. Such temporal autocorrelation is a useful concept in time series forecasting.
We can always have a hunch about spatial autocorrelation of some features just by looking on the map. However, there is a formal way to measure it quantitatively, and Moran’s I is one such statistic used for this purpose. It takes values between -1 and +1 (like normal correlation coefficient, r, would) while providing an associated p-value as a test for significance (discussed below). A positive Moran’s I indicates that similar observations are closer to each other, whereas negative values imply dissimilar values are clustered. Values around 0 would indicate that there is no autocorrelation, and instead, observed values are randomly distributed.
Once Moran’s I is measured for a particular dataset, how can you know if it is significant or not under the null hypothesis that observations are randomly distributed?
One way to measure the significance is to perform a Monte Carlo test. It is done in three steps: (i) first, observations (e.g. house prices) are randomly distributed in different spatial units (e.g. county polygons, raster cells etc), (ii) then, a Moran’s I is calculated for this random dataset, (iii) and finally, the simulated Moran’s I is compared to the observed Moran’s I. This 3-step process is repeated many many times. After running all the simulations the probability p is calculated as follows:
(# of simulated Moran’s I greater than observed Moran’s I +1)/(# of simulations ran + 1)
For example, say,
number of simulations = 200
number of simulated Moran's I that are higher than observed value = 2
then according to the equation above the probability of null hypothesis being true:
p = (1+1)/(200+1) = 0.009
where p<0.01, therefore, the null hypothesis that the observations (e.g. population, house prices) are randomly distributed in space can be rejected. In other words, the observed autocorrelation of house prices is pretty significant.
Autocorrelation is a key concept in spatial analysis that has wide ranging industry applications. Some of them are as follows:
A measure of spatial inequality/diversity: whether income, population, race etc. are clustered or uniformly distributed in certain areas — can be computed using Moran’s I.
It’s been used to identify contamination hotspots of rare earth elements in urban soils of London
Has been used to demonstrate different distance-decay functions of variables of interest (e.g. how far away house prices drop as a function of distance from urban center?).
Autocorrelation is frequently used in many other machine learning algorithms (e.g. spatial regression, classification, cluster analysis) as well as in spatial data visualization as part of EDA (e.g. heatmap, hotspot analysis)
Step 1: Install dependencies
library(sf)library(spdep)library(tigris)library(acs)library(tidyverse)library(tmap)
Step 2: Get the data
The data I am using for this demo is a median household income dataset at the county/census tract level, retrieved from American Community Survey. To access the data users have to put a request for API key. In my case, it was processed fairly quickly once requested.
api="MY API KEY"api.key.install(key=api)acs.tables.install()set.seed(1234) # because we are randomizing part of the process# Access the shapefiles <- get_acs(geography = "tract", variables = "B19013_001",                 state = c("VA"), geometry = TRUE)# remove NA values is anys <- na.omit(s)# select column to work withs <- subset(s, select=c("estimate"))
Step 3: Exploratory data analysis
# check data skewnesshist(s$estimate, main=NULL)# check for outliersboxplot(s$estimate, horizontal = TRUE)# plot variabletm_shape(s) + tm_fill(col="estimate", style="quantile", n=5, palette="Greens") +tm_legend(outside=TRUE)
Step 4: Modeling
# define neighbornb <- poly2nb(s, queen=TRUE) # here nb list all ID numbers of neighbors;# assign weights to neighborslw <- nb2listw(nb, style="W", zero.policy=TRUE) # equal weights# compute neighbor averageinc.lag <- lag.listw(lw, s$estimate)# plot polygons vs lagsplot(inc.lag ~ s$estimate, pch=16, asp=1)M1 <- lm(inc.lag ~ s$estimate)abline(M1, col="blue")# access Moran's coeffcoef(M1)[2]# calculating Moran coeff with one lineI <- moran(s$estimate, lw, length(nb), Szero(lw))[1]
Step 5: Test of hypothesis
# hypothesis test with moran.test functionmoran.test(s$estimate,lw, alternative="greater")# using Monte-Carlo simulationMC<- moran.mc(s$estimate, lw, nsim=999, alternative="greater")# View results (including p-value)MC# plot Null distributionplot(MC)
Understanding spatial autocorrelation is an important concept in spatial data analytics — not only for understanding spatial pattern and variation of data, but also for use in business decisions. It is also a key statistic providing inputs to advanced spatially explicit statistical and machine learning algorithms. In terms of implementation, once you have the data it’s fairly easy to implement. I implemented in R environment but there are some Python libraries (e.g. geopandas) too, however, in my experience R has been more user friendly because of easy access to county/census tract scale datasets using existing packages.