Moving to Brooklyn
By: Abed Islam
Tags: NYC, Brooklyn, Hive, Big Data, Mapreduce

Sheika Islam, Afnan Haq, and Abed Islam

Abstract

Out of all five boroughs, Brooklyn, New York is the most populated. It has a total amount of 71 square miles of land and approximately 2.51 million people living in it. As a group of young adults, we decided it would be interesting to figure out which neighborhoods in Brooklyn are the safest and most livable. To retrieve this type of information, we looked at poverty rates, 311 calls, energy usage, and automobile accidents. Big data analytics tools such as MapReduce and Hive, allowed us to clean and profile our data, join the new data, and come up with analytics of each zip code. We also used Pandas, Matplotlib, and Excel tools to make visuals so that it would be easier to look at. Using all of these tools, we found that the safest neighborhoods in Brooklyn were East Flatbush and Little Poland.

Introduction

In this project, we were studying the livability and safety of different neighborhoods in Brooklyn. This topic is important to investigate because the neighborhood you live in in New York has a very big impact on the life you live. Therefore, we set out to understand which neighborhoods were highly livable and which were not. Before we started our research, we already had an understanding that many parts of Brooklyn were highly segregated, so we were expecting quite a large variance between different neighborhoods in terms of 311 complaints, energy usage, and automobile accidents. By doing this study, we will further the knowledge that data analysts have regarding urban development in New York and can support urban policy to make the lives of New Yorkers better. We expected that certain zip codes like those in East Flatbush would be far more livable.

The design diagram shown below (Visual 1) highlights our workflow. First, all of our datasets came from the New York City government through NYC Open Data. Then, we used MapReduce on HDFS to remove unnecessary columns. Then we stored our cleaned datasets on Hive which we used to transform each dataset as we needed to. We extracted all the new Hive tables as .csv files and then joined them together as one final Hive table. We extracted that file as a .csv and did data analysis on it.

Visual 1: Data Flow Diagram

Motivation

The motivation behind our work was to understand Brooklyn and understand the issues the borough faces. The goal we had was to find the safest and most livable neighborhoods (zip codes) in Brooklyn, New York. There are two categories of people who will appreciate our research. Firstly, anyone who is planning on moving there soon will find this information useful. They can figure out what would be the most livable neighborhood to rent or buy a house in. Secondly, this is valuable analysis for the city government of New York in designing policy for Brooklyn. They can understand where the problems are in Brooklyn and exactly what those problems are. Based on the overall poverty rates, 311 calls, automobile crashes, energy usage, and further research, we wanted to capture a snapshot of life in different Brooklyn neighborhoods (zip codes).

3.1 Sheika’s Motivation

My love for Brooklyn started during high school when I would go hang out at Brooklyn Bridge Park or go see the Dyker Heights Christmas lights with my family and friends. However, when my sister moved there recently, my curiosity about the neighborhoods of Brooklyn grew even more!

Both the poverty and automobile crash dataset allowed me to explore more about the borough as these datasets contained a lot of valuable information. The poverty dataset contained information such as family type, age category, citizen status, childcare expenses, education attained, income, and many more things. One thing that stood out to me from this data was that some of the information was really specific which I absolutely loved. The automobile crash data was very interesting to me as well since I recently learned how to drive, and I haven’t really driven in Brooklyn as much as I would have liked to. The dataset contained information in regard to the number of accidents, type of vehicles that were involved, and how many people were injured or killed.

3.2 Afnan’s Motivation

I really liked the 311 dataset and felt it was reliable. I liked how big it was, containing a huge amount of government data and was going to appreciate this challenge. This is why I chose this topic. I really loved the idea of getting a bird’s eye view of what life is like in different zip codes of Brooklyn and thought this could be valuable information for policymakers. I am interested in New York City politics so I would find this interesting myself.

3.3 Abed’s Motivation

Lately, I have become more self-conscious about how my actions impact the environment. It comes down to small things like recycling and littering to the big things like where my food comes from. I was very interested in seeing how well neighborhoods use energy, oil, and other necessary resources. For me I find value in how well my neighborhood handles the use of energy. I feel in order to hold myself accountable in my pursuit of trying to be energy efficient, I need to live in a place that also values that.

Related Works

4.1 Poverty Dataset

A study was done regarding the neighborhoods in New York and the chance of getting cancer by Kamath, Geetanjali R. and others. Where one lives can be an indicator of getting multiple myeloma (MM) and impact mortality rates, especially if they live in NYC. The chances of getting MM are higher in NYC areas than other places in the United States. “Trends and racial differences in MM incidence and mortality for the United States [Surveillance, Epidemiology, and End Results Cancer Registry (SEER), National Center for Health Statistics], and NYC [New York State Cancer Registry] were compared using Join point regression.” [3]. It was concluded that specifically places with large minority populations are more vulnerable to poverty related factors associated with MM and mortality.

4.2 Automobile Crash Dataset

A similar study was done regarding automobile crashes by Azad Abdulhafedh. As we know, a lot of mishaps on the road can occur due to how the roads are constructed and any other dangerous road sites. Abdulhafedh collected data related to crashes from various systems and gained a better understanding of the problems that were causing these crashes. The goal was to come up with new measures and road safety programs by locating hazardous road sections, identifying the risk factors, and developing accurate diagnosis and remedial measures [4]. This was necessary as it would allow prevention of future accidents since there’d be less hazardous roads and the road safety programs would also be improved. People would know what to look out for during these programs and roads would be fixed almost immediately.

4.3 311 Calls

A similar study was done in New York by Duygu Pamukcu regarding 311 calls during a global health emergency like COVID. 311 calls can be an indicator of the different dimensions of a community’s reaction to a crisis situation. New York City added new complaint types and descriptors in order to facilitate a better understanding of how citizens were dealing with the crisis. A municipality can aggregate and analyze this data (like we have) to better understand certain information regarding the current situation. This information can be vital in saving lives and solving problems. The paper by Pamukcu analyzes the existing 311 data and explains the actions taken by authorities in response to the data. The conclusion states that “the data from NYC’s existing 311 system can be used to characterize the different impacts of the COVID-19 pandemic on the community. It also showed that the city’s adjustments to that system were subsequently able to provide valuable new information about the ongoing crisis” [2].

4.4. Energy Usage

A similar study on energy usage in New York City buildings was done by John Schofield. Energy consumption is very high in New York City since it is a city that is heavily relied on by the economy. Schofield points out that since 2000, many non-residential buildings have been going for a LEED certification [5]. Since 2011, the city has been making this data public for its non-residential buildings. Schofield says that energy consumption decreased by 20% in buildings that had LEED Gold. This research can give more insight on the dataset used for this project. This report was written in 2013 and a lot has changed since then. It can help me get insight on if anything has changed.

Description of Datasets

All datasets were found on NYC Open Data and had a large amount of data regarding all the boroughs.

5.1 Poverty Dataset

This dataset contained information regarding poverty and certain specifications for each borough from 2018 to August 2021. It had a total of 68.3 rows and 61 columns. The data was provided by the Mayor’s Office for Economic Opportunity. We reduced the number of columns to only 7 columns and primarily focused on the column that had information about residents being in poverty or not, based on the official government threshold, the column that had information about work status, and the column that had information about family structure. This information helped us get an insight of how an average family is in Brooklyn. This dataset, however, could definitely be improved if there were more information, such as dates and zip codes so that we would be able to get a timeline of each neighborhood.

5.2 Automobile Crash Dataset

This dataset contained information about automobile accident reports in all the boroughs from 2014 to present day. It had a total of 1.85 million rows and 29 columns, and the data was provided by the New York Police Department. We reduced the number of columns to only 6 columns and focused on the number of people injured and the number of people killed in each zip code during an automobile crash. This dataset allowed us to get insight into the safety of drivers and pedestrians as cars are continuously developing and more people are getting behind a wheel.

5.3 311 Calls

This dataset contained information on most 311 calls from 2010 to the present day with updates being made each day to the dataset. This dataset was massive containing 27 million rows. It contained information about the type of issue that existed, where the issue occurred, and whether it was resolved. We narrowed the dataset down to only 311 calls from the beginning of 2019 afterwards and only focused on 311 calls in Brooklyn. Otherwise, the data was proving hard to download. We focused on the following columns: Call ID, Date opened, Date accessed, Agency, Complaint Type, Zip Code, Address, Result, Longitude, Latitude. Using this information, we could correlate where 311 calls were coming from and what types of complaints they were. This would help us understand the problems that plague Brooklyn.

5.4. Energy Usage

This dataset has information on the water and energy use in buildings throughout the city. It has data on energy star score, many different metrics on heat and water usage along with property types. For this project we narrowed everything down to some of the simpler metrics to measure energy, water, and gas usage. We also decided to drop all the columns on the property types since we wanted a general viewpoint on all the buildings in one zip code. We further organized the data to only have buildings in the Brooklyn area.

Analytic Stages/Process

We started off by finding common interests of each of our team members and decided we’d like to do something related to New York City as it’s the greatest city in the world. We chose datasets that dealt with common things one would be concerned about before moving to a new neighborhood — poverty rates which would tell us how affordable a neighborhood may be and the costs of living there, 311 calls which would tell us about some of the problems that frequently occur in a certain neighborhood, and energy and water usage which would tell us about the different types of resources are commonly used in each neighborhood and how efficient they are when it comes to waste.

6.1 Poverty dataset

While cleaning and profiling, we narrowed down the number of columns. There were originally 61 unique columns, however we focused on 6 columns which were the borough, official government poverty threshold, number of people in a household, type of work in a household, and estimated nutrition related income. Unfortunately, this dataset wasn’t giving us the type of information we were looking for. It was rather too general, so we had to look for another dataset that would be more specific to each neighborhood. On the bright side, we were able to use information such as common family structures and type of work for Brooklyn as a whole as we used MapReduce to filter out the data for Brooklyn. We also came up with a few calculations to figure out the poverty rate, however due to no dates and different number of entries for each borough, it was a bit hard to tell. We ended up comparing our calculations to the most recent report of poverty rates and concluded that it would be best to just use the poverty rates mentioned in The Good News on NYC Poverty is Old News, Which is Bad News by Jarret Murphy. While our calculations said Brooklyn had a 31% poverty rate, Murphy reported that Brooklyn only had a 17.7% poverty rate, which decreased from previous years. [1]

6.2 Automobile crashes dataset

Due to lack of information from the poverty dataset, we looked at automobile crashes that would allow us to retrieve information based off of zip code. There were originally 29 unique columns, but we only focused on 5 of the columns which were borough, zip code, longitude, latitude, number of persons injured, and number of persons killed. These columns allowed us to analyze how safe it was to drive in each neighborhood. It also gave us a sense of how safe it was for pedestrians as well. We used MapReduce to filter out the content of Brooklyn and used Hive to group by zip code:

​​SELECT zipcode, Sum(persons_injured) AS persons_injured, Sum(persons_killed) AS persons_killed FROM crashes GROUP BY zipcode;

This code above grouped the cleaned data by zip code and output the sum of people injured and killed in automobile crashes.

6.3 311 Calls dataset

We reduced the number of columns from 41 columns down to 10 which were the columns we needed. We wrote a Java MapReduce program for this where we split the columns and combined the specific columns We needed with commas. Then we exported that file as a .csv. We built a Hive table specifically for my new data and then copied the .csv into the Hive table. We wanted to figure out how different zip codes had different complaint types. Therefore, after playing around with HiveQL, we wrote the following code:

SELECT zipcode, Sum(CASE WHEN complaintype = ‘Noise — Residential’ THEN 1 ELSE 0 END) AS Noise_Residential, Sum(CASE WHEN complaintype = ‘Illegal Parking’ THEN 1 ELSE 0 END) AS Illegal_Parking, Sum(CASE WHEN complaintype = ‘HEAT/HOT WATER’ THEN 1 ELSE 0 END) AS Heat_hotwater, Sum(CASE WHEN complaintype = ‘Blocked Driveway’ THEN 1 ELSE 0 END) AS Blocked_driveway, Sum(CASE WHEN complaintype = ‘Noise — Street/Sidewalk’ THEN 1 ELSE 0 END) AS Noise_Street_Sidewalk, Sum(CASE WHEN complaintype = ‘UNSANITARY CONDITION’ THEN 1 ELSE 0 END) AS Unsanitary_condition, Sum(CASE WHEN complaintype = ‘Street Condition’ THEN 1 ELSE 0 END) AS Street_condition, Sum(CASE WHEN complaintype = ‘Water System’ THEN 1 ELSE 0 END) AS Water_system, Sum(CASE WHEN complaintype = ‘General Construction/Plumbing’ THEN 1 ELSE 0 END) AS Construction_or_plumbing, Sum(CASE WHEN complaintype = ‘Noise — Commercial’ THEN 1 ELSE 0 END) AS Noise_commercial, Sum(CASE WHEN complaintype = ‘Noise — Vehicle’ THEN 1 ELSE 0 END) AS Noise_vehicle, Sum(CASE WHEN complaintype = ‘Damaged Tree’ THEN 1 ELSE 0 END) AS damaged_tree, Sum(CASE WHEN complaintype = ‘Sidewalk Condition’ THEN 1 ELSE 0 END) AS sidewalk_condition, Sum(CASE WHEN complaintype = ‘Rodent’ THEN 1 ELSE 0 END) AS Rodent, Sum(CASE WHEN complaintype = ‘Sewer’ THEN 1 ELSE 0 END) AS Sewer, Sum(CASE WHEN complaintype = ‘Graffiti’ THEN 1 ELSE 0 END) AS Graffiti FROM service_calls GROUP BY zipcode;

All that is happening in the code above is we are calculating the number of times different complaint types show up in the dataset and grouping them by zip code. This allowed me to convert the number of times a complaint type shows up into its own unique column. With this information, we can analyze how many times different issues occur in a certain zip code. Then we exported the results into a .csv which we sent to Abed to join together on the other datasets based on zip codes.

6.4 Energy Usage dataset

In order to get the columns, we wanted to first run a simple MapReduce job. However, the difficulty came in organizing the data. We wanted only buildings that were located in Brooklyn. The issue did not come in the logic, rather it was the data itself. The people that made the dataset struggled to spell “Brooklyn” correctly, having a variety of misspelled data. Abed had to analyze the dataset itself to understand this issue. To get around this, when using hive, Abed wrote a simple where clause to account for all the misspellings. He then averaged out all the metrics in each Brooklyn zip code to get the cleaned data. This is the query Abed ran to get his cleaned dataset.

create table energyCleaned as

select postCode, avg(energyStarScore) as eScore, avg(fuelOil1UseKbtu) as fuelOil1,

avg(fuelOil2UseKBtu) as fuelOil2, avg(fuelOil4UseKBtu) as fuelOil4,avg(districtSteamUseKbtu) as districtSteamUse,

avg(naturalGasUseKbtu) as naturalGasUse from energy

where city = “brooklyn” or city = “Bklyn” or city = “Booklyn” or city = “BROOKLYN” or city = “Brooklyn”

group by postCode;

Visuals

Visual 2: Combined Data for all Datasets Used

Visual 3: Number of crashes per zip code

Visual 4: Number of people injured in the crashes per zip code

Visual 5: Number of people killed in the crashes per zip code

Visual 6: Energy scores less than 61 Visual 7: Energy scores more than 61

Visual 8: Energy Score and Noise Complaints

Visual 9: 311 Number of noise complaints per zip code

Visual 10: Street condition complaints and number of people injured

Visual 11: Street condition complaints and number of people killed

Insights and Analysis

The typical family type in Brooklyn is usually husband, wife, and child and majority of the population are working part time, which is less than full time, year round. Brooklyn has a 17.7% poverty rate and had a total number of 400,099 accidents. Of these accidents, zip code 11207 had the highest number of reported accidents while zip code 11241 had the lowest number of reported accidents. In terms of people injured, zip code 11203 had the highest number of people injured with a total number of 123 while zip codes 11241 and 11242 had the lowest number of people injured with a total number of 0. In terms of people killed (may they rest in peace), zip code 11234 had the highest numbers of people killed and zip codes 11241, 11242, 11251, 11385, and 11421 had 0 people killed.

While making further analysis, we found that the number of automobile crashes correlated with street condition complaints. As the number of street conditions complaints increased, the number of persons injured and killed also increased. This could be because of the potholes, road construction, and ongoing construction which usually leads to the usage of less lanes. Neighborhoods with less complaints tend to have better infrastructure avoiding injury and death. On average it takes the city 1–3 days to fix the pothole issue if it gets reported[7]. The response time in Brooklyn is about 3 days and 22 hours. Seeing that it takes this long to fix it, many neighborhoods have bad street conditions and the worse they are, the more people get injured or killed in automobile crashes.

Energy scores a big factor. On average, buildings with lower energy scores were receiving more heat and hot water complaints and buildings with lower energy scores got more complaints for unsanitary conditions. There is not a major difference between water system and sewer complaints. However, there was a negative correlation between energy scores and noise complaints. As the energy score increased, the number of noise complaints decreased. This could be that neighborhoods that are environmentally friendly are also paying attention to the way they contribute to noise pollution. A neighborhood that really stuck out in our analysis is Little Poland.

For Little Poland our analysis has parity to how it is in the real world. Little Poland is a small community that is a subsection of Greenpoint. In Greenpoint, there is a huge community project called the “Greenpoint Community Environmental Fund”. According to their website, they have put more than 11 million dollars into improving the local environment [6]. This claim is backed up by our data well. The buildings in the neighborhood are eco-friendly and there are very few calls for unsanitary conditions. The people and local government are putting in the extra effort to keep the neighborhoods clean and environment friendly.

Conclusion

Based on this project, we were able to find the safest neighborhoods to live in. We prioritized energy score as we felt like that was the leading component of all of the datasets. Energy score ratings that were higher than 61, meaning that they were more eco-friendly, had less complaints and accidents reportings. We did this because typically for a neighborhood to be environmentally friendly, they need to have the proper infrastructure behind it. When seeing if there is parity between the data and work done in the real world, we found that our findings held true and that there is a continuation of improvement.

While we were working on the project, we looked at other boroughs well. There were a few insights that caught our interest; however we didn’t get a chance to analyze them thoroughly. If we perhaps look at additional boroughs in the future, we’d be able to get an insight of the safest neighborhoods in all of New York City. We would also want to look at more features of a neighborhood such as income, food resources, diversity, building infrastructure and see how all that can connect back to the environment and overall living conditions.

Acknowledgement

We would like to thank Professor Malavet for helping us progress through this project by answering emails and the discussion board posts. We’d also like to thank NYC Open Data for allowing us to access this information.

References