A Look at Spatial Happiness in Cities, using Tweepy, Text Blob, and ArcGIS.
By: Benny Friedman
Tags: Urban Planning, Machine Learning, Sentiment, GIS, Twitter

How can we measure the built environment with social media?

Understanding the effects of the built environment has always been a fascination for me. How do we judge great development? How can we know urban planning is succeeding in making life better? Can we be data-driven in our analysis?

To find out how people feel about a place, architects often spend months convening community meetings and conducting surveys to gain a detailed understanding. But if we are looking for a much quicker, less exact view, Twitter sentiment analysis can perhaps provide a standardized, high volume way to understand how people feel at any given time in any given place. Well, at least how their social media selves feel.

The kind of Twitter sentiment analysis I am thinking of marries knowledge about the tone of tweets with placing those tweets geographically. To conduct the investigation I sampled random points in two different cities: Los Angeles and Chicago. I used the Tweepy API to pull tweets, Text Blob to analyze sentiment, and ArcGIS to interpolate the sentiments between the sample points.

The Analysis.

The first hill to climb was to build a simple script to understand the sentiment of tweets near any given point on earth. My input would be a lat/long coordinate, and I wanted the output to be the average sentiment of tweets within 1 mile of that point. The code is below:

Methods utilizing TextBlob and Tweepy to derive subjectivity and polarity of tweets

For each point, the final function makes use of Tweepy to search for tweets within a specified radius. The API returns up to 100 tweets from the last day or so from within a given radius. Each tweet is evaluated for “Polarity” and “Subjectivity” using Text Blob. Polarity is returned as a float between -1 and 1, negative to positive. Subjectivity is returned as a float between 0 and 1. Using pandas, the polarities are averaged to get a polarity score for the point.

Tweet content and their polarity scores

Once we know how to capture and understand sentiment around a point, the next step is to generate sample points for each city. I wanted to evaluate spatial twitter sentiment at around 95% confidence and a 5% margin of error. That means, for example, for my LA County AOI, I needed 294 random samples. In order to make sure not to sample underpopulated areas, I first erased all areas specified as “Parks and Openspace” in the “LA County Parks and Openspace” dataset provided by the county. Then using the parkless dataset I created 294 random points with the ArcGIS “Create Random Points” tool. These points were fed into the script outlined earlier to get sample average sentiments.

In order to interpolate the spatial Twitter sentiment for the full urban areas, I made use of the “Ordinary Kriging” tool. Kriging looks at the relationship between sample point distance and sentiment values, to build linear equations to predict average Twitter sentiment between known samples. Kriging enables us to get good predictions for full AOIs.

Examining Spatial Twitter Sentiment

After running this analysis on LA and Chicago, it is possible to see how Twitter sentiment varies throughout each city.

Los Angeles, 11/28/2020

Los Angeles spatial sentiment on 11/28/2020

I ran my Los Angeles analysis on 11/28/2020 and the results can be examined in the above map. The map is symbolized by z-score, so areas that show up as having positive and negative sentiment are particularly that way. Most areas are expected to fall close to the mean and be symbolized as neither particularly positive nor negative. Looking at the map, there are some interesting patterns. The Malibu area is particularly happy compared to the rest of the LA area. Through this analysis, it is hard to know what that is, although I would guess it has something to do with living close to beautiful beaches, natural parks, and being particularly affluent. Although, based on this analysis, it's hard to say that the wealthier you are, the happier you are. Many other affluent areas of the city are not particularly happy. See the Beverly and Hollywood Hills areas.

Chicago, 11/28/2020

Chicago spatial sentiment on 11/28/2020

I ran the Chicago analysis on 11/28/2020 as well. The maps for both LA and Chicago are symbolized in the same way and on the same data distribution so they can be directly compared. Unlike LA there are three particularly happy areas of the Cook County/Chicago area. In the north, there are the broad Hoffman Estates and Glenview/Skokie areas, and in the south, an area near Calumet City on the Illinois-Indiana border. It also seems clear that, at least on this day, the southern part of Cook County, represents higher negative sentiment.

Conclusions

I will be the first to say that there are many things wrong with this analysis. First, I only take into account a snapshot of information. If we wanted to more accurately understand how Twitter sentiment varies over space, it would be helpful to collect data for a longer period of time. Maybe a month's worth of data or even several years. Second, I think that broad-based Twitter sentiment is not particularly correlated with a person’s happiness, and a more elegant classification model is in order. Finally, there is a lot of self-selection that comes along with using Twitter data. Only some people use Twitter, and people who use it a lot will be over considered. For example. if you sampled tweets around 1600 Pennsylvania Ave. in Washington, DC, between 2016 and 2020, you might get too many tweets from one individual, and not enough from the rest of the local population.

Though, flawed, I think this analysis shows the potential power of Twitter sentiment data. Using tweets, it becomes possible to compare and contrast sentiment across time and space, quickly and cheaply, in a standardized way. And this kind of analysis is not restricted to looking at positive vs negative sentiment. More targeted models can be made to look for more interesting trends in tweets. Where are people talking about a specific thing? How far away does knowledge of some change travel? It is really an open box for analysis.

Next Steps

My main goal in publishing this is to start a conversation about evaluating the places we live in powerful and scalable ways. Please feel free to comment on this article, send me an email, or connect with me on LinkedIn. I’d love to hear your comments and suggestions!

Email: benny.friedman2@gmail.com

LinkedIn: https://www.linkedin.com/in/benjaminfriedman2/