New to Data Visualization? Start with New York City | by Thomas Hikaru Clark | Towards Data Science

Like many others, I have been working from home for several weeks due to the coronavirus. Inspired in part by Terence Shin’s great article on data science projects to try during quarantine, I decided to use some of my free time on improving my data science skills. I am a high school computer science teacher, and adding data science skills to the portfolio of content that I can teach my students is important to me, given the rising prominence of data science in today’s world. One thing I’ve learned is that if you are trying to teach yourself data visualization, New York City is a great place to start, and I will explain why in this article. Along the way, I will provide some examples of bite-size NYC-themed projects you can try to become more familiar with how to process and visualize data in Python.
New York City provides a wealth of public data at a high level of granularity (in some cases down to the individual building). You can pretty easily find information about assessed property value, income brackets by zip code, locations of public facilities, and more. Additionally, there are excellent custom downloaders available to get just the data you need.
The NYC Open Data website is a great starting point to search publicly available datasets and download files.
NYC Open Data Portal: https://opendata.cityofnewyork.us/
The New York City government has released a dataset called PLUTO which contains tax-related information about every lot in the city. Using this massive dataset, you can all manner of information about any given plot in the city, such as the number of square feet of building space, the number of floors, the assessed property value for tax purposes, the name of the owner, and more. Additionally, the MapPLUTO dataset merges this data with geographic data, providing polygons for each lot in the city. A great tool for accessing this data is Chris Whong’s custom PLUTO downloader, which lets you select exactly which areas of the city you would like to download, along with which columns of the PLUTO data you want.
Chris Whong’s NYC PLUTO custom data downloader: https://chriswhong.github.io/plutoplus/
The New York City Capital Planning Platform is another excellent custom downloaded where you view all public facilities, such as parks, schools, hospitals, youth centers, parking lots, and more. Once you have refined your selection, you can download the data as a CSV file.
New York City Capital Planning Platform: https://capitalplanning.nyc.gov/map/facilities#10/40.7128/-74.0807
As an iconic global city that is simultaneously a business Mecca, a tourist hot-spot, and a backdrop for countless films and shows, New York has instant brand-name recognition. Most people probably have some basic familiarity with aspects of New York’s geography, such as Central Park, the five boroughs, some of New York’s many neighborhoods (like the Upper East Side, Harlem, etc.).
This recognizability may help you make sense of your data as you are pre-processing it, or give you a sanity check on your results once you hit run. But even if you have no familiarity with the city, there is a good chance that others viewing your work will be familiar. This makes projects involving New York City data great for a personal portfolio, blog, or other medium that you wish to share with others as a demonstration of your skills.
New York is both very dense and very diverse. These factors help make visualizations relating to New York City more interesting; very meaningful and data-rich visualizations can be made at the level of a single neighborhood. Its diversity gives many dimensions along which to analyze and visualize data, especially in visual or map form, including the following examples:
The relationship between socioeconomic status and geography
The relationship between race/ethnicity and geography
The relationship between language and geography
This project’s objective is to visualize each lot’s distance to the nearest library. Using a very similar process, however, you could also visualize distance to many different features of interest, such as subway stations, parks, schools, or hospitals. A choropleth map provides an excellent way to visualize this large amount of geographic data at a glance. Using this approach, it becomes easy to identify potential “reading deserts” or “green space deserts” in the city.
I used the MapPLUTO dataset to load information about all lots in Manhattan, including their latitude and longitude. I also used the New York City Capital Planning Platform to download data on all libraries in Manhattan, including their names and latitude/longitude. I then loaded this data into my Python program as a Pandas DataFrame.
My first attempt to calculate the nearest library for each lot was a naive brute-force approach, as seen in the snippet below. This turned out to be much too slow, owing to the large amount of redundant computation.
df['dist'] = df['coord'].apply(lambda x: min([geopy.distance.distance(x, y).km for y in libs['coord']]))
My big break was in remembering a lecture from college in which the professor mentioned something called a K-D tree. This is a data structure that is optimized for quickly finding the nearest neighbor of a given point. An implementation of this is contained in the sklearn.neighbors library. Since we are dealing with (latitude, longitude) coordinate pairs, we should use a BallTree with a “haversine” distance metric. Using this library to calculate the nearest library sped up the runtime by several orders of magnitude.
Use choropleth maps wisely. They can be great for certain situations, but misleading in others. For example, a county-level choropleth map of 2016 presidential election results in the United States would give the impression that the overwhelming majority of the country voted red, yet this is misleading because more densely populated areas tended to vote blue.Algorithmic efficiency matters. Especially when working with large datasets, pay attention to whether you are performing many redundant computations. Do some research to see if tools exist to complete parts of your task more optimally so that you are not re-inventing the wheel.
Use choropleth maps wisely. They can be great for certain situations, but misleading in others. For example, a county-level choropleth map of 2016 presidential election results in the United States would give the impression that the overwhelming majority of the country voted red, yet this is misleading because more densely populated areas tended to vote blue.
Algorithmic efficiency matters. Especially when working with large datasets, pay attention to whether you are performing many redundant computations. Do some research to see if tools exist to complete parts of your task more optimally so that you are not re-inventing the wheel.
Access data files and code at https://github.com/thomashikaru/nycvisualization.
This project’s objective is to visualize the population density of different neighborhoods in New York City. With slight variations, the same approach can be used to visualize distribution of ethnicity, income, or other characteristics.
In addition to dividing the city up into individual lots or zip codes, it is also possible to break the city up into Neighborhood Tabulation Areas. These correspond to the familiar neighborhoods of New York, such as the Upper West Side, Chinatown, or Park Slope. It was relatively easy to find a data file with each neighborhood’s population, as well as using the NYC Open Data tool to download a GeoJSON file with the polygons for neighborhood boundaries.
Coloring the map by total population might be misleading because the neighborhoods vary quite widely in size. Instead, we can color by the map by population density. This requires calculating the area of each neighborhood, which is not directly contained in the data. Using the area library, we can compute each GeoJSON polygon’s area, then use these values to calculate density (population divided by area).
Use choropleth maps wisely (again). The sizes of each neighborhood vary widely and are somewhat arbitrary. It is more meaningful to visualize the density of each neighborhood than the absolute population.Combining data from different sources. The population data for each neighborhood was stored in one CSV file (which I read into a Pandas DataFrame), while the polygons for each neighborhood were stored in the GeoJSON file. I went through the GeoJSON file and made a dictionary mapping each neighborhood code to its area, then used the Pandas.Series.map() function to match up neighborhoods in the DataFrame with the corresponding area in order to calculate density values.
Use choropleth maps wisely (again). The sizes of each neighborhood vary widely and are somewhat arbitrary. It is more meaningful to visualize the density of each neighborhood than the absolute population.
Combining data from different sources. The population data for each neighborhood was stored in one CSV file (which I read into a Pandas DataFrame), while the polygons for each neighborhood were stored in the GeoJSON file. I went through the GeoJSON file and made a dictionary mapping each neighborhood code to its area, then used the Pandas.Series.map() function to match up neighborhoods in the DataFrame with the corresponding area in order to calculate density values.
Access data files and code at https://github.com/thomashikaru/nycvisualization.
If you are new to data science or data visualization, consider choosing New York City as a starting point to spend less time hunting for data and more time acquiring new skills. Take advantage of the wealth of available data and the excellent custom downloaders that exist.
Try one of the projects mentioned above to get familiar with useful data science tools such as Pandas and Plotly, as well as common data formats such as GeoJSON and CSV.
I hope you learned something new. Best of luck in your learning journey!
[1] T. Shin, 14 Data Science Projects to do During Your 14 Day Quarantine (2020), Medium.
[2] N. Piepenbreier, Visualizing COVID-19 Data Beautifully in Python (in 5 Minutes or Less) (2020), Medium.
[3] E. Grimaldi, How to create interactive map plots with Plotly (2020), Medium.
[4] T. Shin, How To Visualize the Coronavirus Pandemic with Choropleth Maps (2020), Medium.