SC DOT and Zillow Home Sales
By: James Williams
Tags: Data Visualization, Heroku, Traffic, Dash, Data Science

How has traffic in South Carolina changed?

Introduction.

One of my least favorite parts of the day is my daily commute.

Anecdotally, I feel as if traffic has worsened in South Carolina since I moved here in 2007, and it’s worsening at an increasing rate every year. In my quest to verify this claim, I stumbled across some data on the SC-DOT GIS page that seemed promising. Here’s where the data can be downloaded.

The data in question that I’m examining are Annual Average Daily Traffic (AADT) counts from the years 2009–2018. These data are recorded at the level of StationID and RouteIdentifier with additional information as well, including Latitude and Longitude of the recording Station. I downloaded the .zip files for each separately, renamed them, and consolidated them into a single folder. The data come in both .shp and .dbf (Xbase) format.

SC DOT has some interactive ArcGIS maps with these data points plotted already here. However, these plots don’t fully convey (to me) the true amount of traffic in SC, as all station points are plotted as little cars with no information about AADT, nor the change in traffic patterns year over year. Neither do they address any underlying causes of what may be driving potential traffic pattern changes.

Tangentially, SC DOT does provide a wealth of other data for citizens to browse, some of which look quite interesting. I am particularly interested in how the “South Carolina Roads by Pavement Status” dataset folds into the questions at hand here, but that goes beyond the scope of this post. Perhaps to be addressed later.

Getting back on topic, one of the most intuitive drivers of change in traffic patterns could be population growth in the areas nearby. I chose to use a data set from Zillow — https://www.zillow.com/research/data/ — that details Monthly Home Sales by ZipCode as a proxy to population growth per Zip Code.

Finally, I also downloaded a data set from OpenDataSoft that cross references ZipCode to latitude and longitude. I only downloaded data for South Carolina.

Questions to Answer.

While conducting this exploratory analysis, there are a few key questions I am looking to answer.

What do the data look like and what does it take to get them in a unified format? Has overall traffic increased in SC over the past 10 years? Are there are any counties or routes in SC that show more aggressive traffic growth? Does number of home sales in Zip Codes within a certain radius of a station (e.g. 50 km) have any bearing on traffic numbers the next or same year?

AADT Data Wrangling.

I’ll skip most of the technical details, as they are quite dull, but speak to the process. Technical details can be found in my GitHub repo for this project. The main steps are as follows:

Read data for each year from .dbf files into separate pandas dataframes Map column names to new names, drop duplicated and/or extraneous fields —little consistency in presence or naming of fields across years, which made this quite tedious. Ensure records per year are unique for station_id, route_identifier, and route_number — these are the only fields present across all dataframes. These fields represent a unique point on a road in SC where AADT is being measured. Their combination can be considered as a unique key for each dataframe. Only a small percentage of the data had more than one row per year across these fields and I needed to ensure they were unique to allow merging the dataframes for each year. I averaged the AADT values across the group and dropped duplicates. More detail as to the reason behind the existence of the duplicated rows can be found on my GitHub. Using the most recent data, from 2018, update data points (e.g. Lat/Long, County Name, etc.) across all dataframes by station_id, route_identifier, and route_number. This is to ensure consistency on these points between frames. Merge all data into one traffic_df.

Here’s a glimpse at the schema our final dataframe:

And here’s a little test plot of our data:

test_plot = traffic_df.loc[traffic_df.year == 2018].sample(10000)

x_plot = test_plot.longitude.values

y_plot = test_plot.latitude.values

cmap = plt.get_cmap(‘jet’)

c = np.log(test_plot.average_daily_traffic)

s = np.log(test_plot.average_daily_traffic) fig, ax = plt.subplots(1, 1, figsize=(10, 10))

plt.scatter(x_plot, y_plot, s=s, c=c)

South Carolina!

I decided to look at a AADT values by county and route separately.

The above is a plot of Avg AADT values for the top 15 counties in SC by year, excluding Interstate traffic. For full disclosure and clarity, the plotted values are an average of all the individual AADT values (themselves an annual average) for each station in the county. Clearly Charleston county tops the charts in AvgAADT, followed closely by Berkeley, Beaufort, and Horry county. These top 15 counties only show a modest increase in AvgAADT values year over year.

Now, here’s the MaxAADT values for the top 15 counties:

The MaxAADT per county is much higher than the AvgAADT, as expected. There are obviously stations within these counties that experience much lower average daily traffic than the most high-traffic areas, dragging down the average. Charleston county is by far the most heavily trafficked area in SC. Interestingly, Horry county MaxAADT values drop significantly from 2013 to 2014 — it might be interesting to dig in and find out why. Additionally, it would be interesting to investigate which stations are generating the highest AADT numbers per county.

I also decided to examine AvgAADT values by Route:

Unsurprisingly, Interstate routes are at the top of the list, with the top 5 experiencing Average Annual Driver growth of almost 10k drivers from 2009 to 2018.

Looking at the MaxAADT values:

Again, Interstates are on top. Apparently some portions of I-26 saw nearly 200k daily drivers on average for 2016 and 2017.

I wanted to get a sense of whether or not AADT values have been increasing year over year, so I calculated the percent change from 2009 AADT values to 2018 AADT values for each station. I found that around 50% of stations experienced a percent change of 4% or more from 2009 to 2018, with 25% of stations experiencing a percentage change of 18% or more!

As an important note, I did notice some stations reporting percent changes in the 1000’s. Upon closer examination of the data, I discovered many of these stations were reporting low AADT values for a few years before values spiked. It would be worth evaluating the veracity of some of the numbers coming from reporting stations before making widespread claims about overall traffic increases in SC.

This is really only a cursory look at the data, as there a many different ways we could segment and analyze. To facilitate exploration of the data I deployed a simple Dash app to Heroku, which can be found here. Here’s a screenshot of the app:

Zillow Sales data by Zip Code.

As noted above, I downloaded a dataset from Zillow that details total home sales by Zip Code and Year-Month. The data came in very wide format:

Head of the Zillow dataset

Wrangling the Zillow sales data involved:

Filtering to only South Carolina Zip Codes Dropping columns I don’t care about Stack from wide to tall format Get the sum of sales per zipcode by year

The tail of the resultant frame looks like this:

And here’s what home sales in the past 10 years look for SC:

Wow! It wouldn’t surprise me to see a corresponding increase in traffic.

Zip to Lat/Long Xref.

Now that we have yearly home sales by Zip Code, we want to be able to link these home sales to a latitude/longitude point. As previously mentioned, I downloaded a dataset from OpenDataSoft that cross references ZipCode to latitude and longitude. I was able to merge this dataset into the home sales dataset to add a lat/long point to each ZipCode.

Home Sales to AADT.

At this point I’d like to calculate how far each Zip Code is from each station. Then, we can choose a cutoff radius (say 15km) and calculate the sum of all home sales within the radius per year for each station.

To accomplish this I:

Cross-joined all Zip lat/long data to all AADT data Used the haversine formula to calculate how far each station is from each Zip Code, credit to this link and this link. Calculate sum of all home sales within a 15 km radius of each station per year. Merge back into the traffic data

Then, I grouped the data by station and calculated the Pearson’s Correlation Coefficient between each station’s AADT numbers year over year and the number home sales within a 15km radius year over year.

Here is the histogram of all correlation coefficients calculated;

Avg Daily Traffic at 55% of stations have at least a modest correlation (0.2) with home sales in a 15km radius of station within the same year. 40% of stations have a strong correlation (>= 0.5). We also have stations that have strong negative correlations between AADT and home sales!

I also looked at the correlation between AADT values and home sales two years prior for each station:

Avg Daily Traffic at 60% of stations have at least a modest correlation (0.2) with home sales in a 15km radius of station within the same year. 47% of stations have a strong correlation (>= 0.5). Further analysis and examination of segments within the dataset (e.g. route type, etc) would be needed before anything conclusive can be said about causation, although intuitively you would surmise that increasing home sales means there more people on the road.

Additionally, you might play around with the lag in years for home sales and the radius at which home sales are calculated.

Conclusion.

Overall I managed to answer, at least cursorily, the questions I laid out at the beginning of this article:

The data are not in an easily digestible format and require significant munging before they are ready for analysis

Overall traffic has certainly increased in SC over the past 10 years, with 50% of stations experiencing a percent change of 4% or more

The Interstates experienced the most heavy AADT growth year over year. AvgAADT values increased modestly for counties across SC, with Charleston county clearly bringing home the prize for most drivers on the road.

There does seem to be a relationship between home sales within a certain radius of a station and AADT values, though nothing definitive can be said.

Thanks for reading! Please visit my Github repo or the Heroku/Dash app if you’re so inclined.