Tutorial: Data Wrangling and Mapping in R | by Pres Nichols | Towards Data Science

I think I’ve said this before ... but I love maps. Almost everyone uses them, making maps an amazing way of instantly communicating huge amounts of data without worrying about overwhelming your audience.
One of my favorite tools for working with spatial data is R. Apart from being great for data wrangling, its broad user-base means that there are loads of packages that make custom map making super quick and easy.
This tutorial is meant to provide a rough, end-to-end example of using R to manipulate and map data. The goal is to create a map that will illustrate the concentration of residential building ownership throughout the borough of Brooklyn, New York.
We’ll be using a number of data sets from New York City’s OpenData portal, in addition to the recently released Cartography package.
To complete this tutorial, you’ll need the following tools:
R
R Studio or any other IDE
1) Clone the GitHub repository
git clone https://github.com/spnichol/mapping_tutorial.gitcd mapping_tutorial
2) Create a new R script and set your working directory
setwd("/home/you/mapping_tutorial")
3) Install packages and load first two data sets
install.packages(c("cartography", "rgdal", "sp"))#ownership registrations owners <- read.csv("data/owners.csv")#building data bldgs <- read.csv("data/bldg.csv")
The first file is the Multiple Dwelling Registrations, a registry of residential buildings with three units or more. The second is Registration Contacts, which contains information relating to the owners of the buildings from the previous file.
Our goal with this project is to get an idea of the concentration of building ownership in Brooklyn. So, a natural place to start out would be with the ownership file.
Let’s look at a few records to get a sense of what we have.
head(owners)
Your output should look like this:
It looks like we have two ID variables: RegistrationContactID and RegistrationID. The first refers to the individual/company that is listed in that row, while the second refers to a specific building registration that person/entity made.
We also have the CorporationName column, which could provide some insight into who owns what in Brooklyn. Let’s use the aggregate function on RegistrationContactID and CorporationName.
owner_count <- aggregate(RegistrationID ~ RegistrationContactID, data=owners, FUN=length)names(owner_count) <- c("RegistrationContactID", "Building_Count")head(owner_count)nrow(owner_count[owner_count$Building_Count > 2 ,])
Your output should look like this:
Now we have one column with each unique RegistrationContactID and the number of buildings corresponding to it. Besides some odd looking IDs (just messy data), it’s a bit strange that we only have 274 owners with more than two buildings. Let’s try with CorporationName.
owner_count <- aggregate(org_agg <- aggregate(RegistrationID ~ CorporationName, data=owners, FUN=length)names(org_agg) <- c("CorporationName", "Building_Count")head(org_agg)nrow(org_agg[org_agg$Building_Count > 2 ,])
The output:
Yikes. The building count is much better, but those are some nasty typos. Let’s fix them by removing all non-alphanumeric characters and then subsetting to keep only non-empty rows.
org_agg$CorporationName<- gsub("[^[:alnum:][:blank:]]", "", org_agg$CorporationName)org_agg <- org_agg[org_agg$CorporationName !="" ,]head(org_agg)nrow(org_agg[org_agg$Building_Count > 2 ,])
The output:
Mmmm. The company names look a bit better, but I’m still not convinced about the number of owners with more than 2 buildings. This data set includes all of New York City. This city has a lot of buildings with a lot of landlords. Could it be that the same landlord or management company is registering different buildings under different names/IDs?
What if we looked at their mailing address? Let’s assume that a landlord registering under different names is still going to want to receive his/her mail at just one office. We can create a new variable by combining the BusinessHouseNumber and BusinessStreetName columns and run our aggregation again.
owners$RealID <- paste(owners$BusinessHouseNumber, owners$BusinessStreetName, sep=" ")real_agg <- aggregate(RegistrationID ~ RealID, data=owners, FUN=length)names(real_agg) <- c("RealID", "Building_Count")nrow(real_agg[real_agg$Building_Count > 2 ,])
Yeah — that makes more sense. We can’t be 100% sure that some of these aren’t actually different landlords that share the same mailing address, but I think we can assume this isn’t the case for the sake of this tutorial.
However, there’s a problem. If you hadn’t already noticed, many of the RegistrationIDs include multiple contacts, each with a different value for Type. Before we start filtering them out, let’s get an idea of the scope of the values.
summary(owners$Type)
This is what we get:
Now we could subset our dataframe to keep only Types that are owner-related roles. But that could lead to some problems, since owners aren’t required to be the “contact” for these types of registrations.
Instead, let’s just remove duplicate RegistrationIDs and re-run our aggregate function.
owners <- owners[! duplicated(owners$RegistrationID) ,]real_agg <- aggregate(RegistrationID ~ RealID, data=owners, FUN=length)names(real_agg) <- c("RealID", "Building_Count")nrow(real_agg[real_agg$Building_Count > 2 ,])
It looks like that, after removing duplicate RegistrationIDs, we have about 5,000 owners with more than two buildings.
Before we move on to the next step of the tutorial, let’s combine the Building_Count column with the main owners dataframe. We can do this with a simple merge.
owners <- merge(x=owners, y=real_agg, by="RealID")
Now that we have a strategy for how to define building ownership, we need to merge the recently updated owners dataframe with the buildings data set. This will give us the GIS information we’ll need for the mapping part of the tutorial.
1) Explore variables for buildings dataframe
head(bldgs)
My output looks like this:
Nice, it looks like both data sets share the RegistrationID variable. This means that each entry in this dataframe corresponds to a RegistrationID in the owners dataframe. We can use this column as our ID for the merge function.
2) Merge on RegistrationID
bldg_counts<- merge(x=bldgs, y=owners, by="RegistrationID")
The merge worked, but we lost a few rows. Annoyingly, it turns out that some types of properties are allowed to use the same RegistrationID, even when they have different addresses. This is explained in more detail in the data documentation.
Now for the fun stuff. We need to figure out what sort of GIS variables we can use to aggregate and plot this information. Let’s take another look at our recently combined dataframe.
bldg_counts<- merge(x=bldgs, y=owners, by="RegistrationID")
Output:
So, it looks like we have two options for geo variables. The first is the building address, which can be derived by combining the HouseNumber and StreetName columns. However, in order to plot this information we’d have to geocode the addresses first. Too much work.
We also have the Block and Lot variables. Block numbers in New York City are unique, meaning we could aggregate the number of buildings per block with landlords owning more than one building. The city also offers shape files with block-level polygons, allowing us to plot our data, as well. However, there are tens of thousands of blocks in Brooklyn, each with only a handful of buildings, so this probably wouldn’t be too meaningful.
In reality, none of these variables offer exactly what we need. Instead, let’s introduce a new data set: PLUTO. The PLUTO data set is the holy grail of New York City building stats, including everything from latitude and longitude to property value.
1) Read in PLUTO CSV file and explore variables
pluto_bk <- read.csv("data/pluto.csv")names(pluto_bk)
Output:
See what I was saying? Let’s keep only the relevant columns.
2) Remove unnecessary columns
pluto_bk <- pluto_bk[, c(4, 5, 6, 58,71, 73, 74, 75)]
We’re left with the following columns:
Now we have tons of options. I chose to go with Census tract, represented in different forms by the CT2010 and Tract2010 variables. This seems to be a good level of aggregation, as there are about 750 Census tracts in Brooklyn. Plus, the Census Bureau offers tract-level shape files, making it a breeze to map.
We need to merge this information with our building list we created earlier. Unfortunately, PLUTO doesn’t have the RegistrationID variable we used earlier. Instead, they use the more common BBL, which stands for “Borough, Block, Lot.”
Our buildings data doesn’t have a BBL variable, but we can make one by combining the borough, building and lot columns. However, first, we’ll use the sprintf function to pad our Block and Lot variables.
Padding adds extra zeros to values that are less than a certain number of digits. This allows our final BBL to be ten digits long, regardless of the value of the components that make it up.
We’ll also take this opportunity to subset our buildings dataframe to only include properties in Brooklyn. We do this using the BoroID of 3.
3) Create BBL variable in buildings dataframe
#subset dataframe for just Brooklyn bldg_counts  <- bldg_counts [bldg_counts $BoroID == 3 ,]#create new columns with padded values bldg_counts ["New_Block"] <- lapply(bldg_counts ["Block"], function(x) sprintf("%05d", x))bldg_counts ["New_Lot"] <- lapply(bldg_counts ["Lot"], function(x) sprintf("%04d", x))#use paste function to combine everything into one variable bldg_counts ["BBL"] <- as.numeric(paste(bldg_counts $BoroID, bldg_counts $New_Block, bldg_counts $New_Lot, sep=""))
Looks good to me. Let’s go ahead and merge this dataframe with the PLUTO data. All we want is each building’s Census tract, so we can subset accordingly.
4) Merge tract info with buildings data set
names(pluto_bk)pluto_bk <- pluto_bk[, c(5, 2)]nrow(bldg_counts)bldg_counts <- merge(x=bldg_counts, y=pluto_bk1, by="BBL")nrow(bldg_counts)
We lost a few hundred rows that, for whatever reason, were not in the PLUTO data set.
One important side-note about BBLs: This value isn’t actually a unique identifier — some Lots have more than one building. This isn’t a huge deal, since all we need to extract from the PLUTO dataframe is Census tract, which would be the same for buildings that reside on the same lot.
If we wanted to be more precise, we could have created a unique ID variable by combining BBL with the street number for each building.
5) Aggregate by Census tract
#aggregate by census tract multiple <- bldg_counts[bldg_counts$Building_Count > 2 ,]tract_counts <- aggregate(Building_Count ~ CT2010, data=multiple, FUN=length )names(tract_counts)[1] <- "CTLabel"nrow(tract_counts)length(unique(pluto_bk$CT2010))
The first thing that jumps out at me is that we lost a bunch of Census tracts. The explanation is simple. While there are a total of 761 tracts in the borough, some of them are either 1) really small or 2) consist almost entirely of one-storey homes, which are not required to register with the city. Since these tracts have no “multiple-building” landlords, they won’t show up on aggregated list.
Alright. We have our dataframe with each Census tract and the number of properties owned by “multiple-building” landlords. It’s surprisingly few, but then again, Census tracts aren’t actually that big. Let’s map!
The first step in any mapping project in R is to read in our shapefile. The shapefile we are going to use for this tutorial is from the New York City Department of Planning. While the Census Bureau is normally the go-to resource for shapefiles, the NYC Department of Planning version already has the water areas clipped, so it is much neater.
1) Load shapefile
bk_shape <- readOGR(dsn = "shapefiles", layer = "nyct2010")head(bk_shape@data)
We can see that this shapefile actually has 2166 different census tracts, as it includes all boroughs. We can also see that it has data bound to the polygons, making it a spatial polygons dataframe (SPDF). This data includes the Census tract label, in addition to information about the polygons themselves. We’re going to want to bind our data to the SPDF, but first we need to fix a few things.
2) Subset SPDFSubsetting an SPDF in R is absurdly easy. You can do it just like a normal dataframe.
#remember "3" is the BoroCode for Brooklyn bk_shape <- bk_shape[bk_shape@data$BoroCode == "3" ,]head(bk_shape@data)
3) Merge our data with SPDF
Adding data to polygons in R is also pretty easy. In this case, we just need to make sure the ID variables have the same name, and we’re good to go.
bk_shape <- merge(bk_shape, tract_counts, by="CTLabel")
There ya go — you can see that we have now bound our data to the SPDF and we are ready to start with the map.
4) Plot base mapHere is where the Cartography package comes in. To start plotting our map, we need to first plot the “extent” of the map (i.e. the spatial region it will occupy), followed by the land mass.
plot(bk_shape, border = NA, col = NA, bg = "#A6CAE0")plot(bk_shape, col  = "#E3DEBF", border=NA, add=TRUE)
It should look like this:
5) Add border linesNow we’re going to add a third layer, which will be the borders of the Census tracts.
plot(bk_shape, col = "grey60",border = "white", lwd=0.4, add=TRUE)
Now you should have this:
6) Choose color paletteThe Cartography package has lots of nice pre-made color palettes. I chose a simple green, but you can have a look at the others here on Page 4. I also chose to use 8 different colors, although this will always depend on the distribution of the data you are using.
cols <- carto.pal(pal1 = "green.pal" ,n1 = 8)
7) Add choropleth layerThis package has loads of different features for displaying data, but we’re going to stick to a simple choropleth map.
choroLayer(spdf = bk_shape,            df = bk_shape@data,            var = "Building_Count",            breaks = c(0, 10, 30, 50, 70, 90, 110, 150),            col = cols,            border = "grey40",            lwd = 0.5,            legend.pos = "left",           legend.title.txt = "Number of Buildings",            legend.values.rnd = 10,           add = TRUE)
Boooom. This actually turned out quite nice, as it corresponds very closely to what I know anecdotally to be true from several years of dealing with Brooklyn landlords. The dark green areas also correspond very closely to the areas of the borough that have experienced the most gentrification.
Many of the maps I made for the Brooklyn Public Library looked very similar, especially when looking at changes in median income and rent pre/post gentrification.
8) Add layout layerWe can add a little bit of metadata to the map by creating a layout layer.
layoutLayer(title = "Census Tracts by Building Ownership Concentration",             author = "Pres Nichols",              sources = "Source: NYC OpenData",             scale = NULL,             col = NA,             coltitle = "black",             frame = FALSE,              bg = NA)
There is one important aspect that I didn’t take into account: what if the northern areas of the map, with supposedly more concentrated ownership, just have more buildings in total?
If you’re following this tutorial, I’ll leave it up to you to figure that out and report back your findings!
Note: Those not familiar with Brooklyn’s geography might be a bit concerned by all the white tracts on the map. It looks way worse than it is. This is because the largest white areas (Prospect Park, Greenwood Cemetery and the Rockaways) aren’t actually inhabited. If they still bother you, you can always give the uninhabited areas a different color.