Learn Python Data Analytics By Example: NYC Parking Violations | by Nick Cox | Towards Data Science

While working towards my Master’s in Business Analytics, I found that learning by example was the best way for me to learn Python data analytics. Being given a dataset and a set of coding tasks is much more beneficial than reading a textbook or listening to a professor.
I want to share this method of learning with others who will also benefit. All you need is a Python development environment (I recommend Jupyter Notebook) and a willingness to learn and have fun.
Included in this article is a list of data analytics tasks, followed by a detailed walkthrough of how to complete the tasks. Please try to complete the tasks yourself before reading through the walkthrough — you will get more out of it that way. Do keep in mind that there are many many ways to solve coding problems, so your code likely will not match mine word for word.
For this project we will use a dataset of 50,000 parking violations issued in New York City during the 2021 fiscal year. The dataset was created in January 2021 and contains data from April 1 to November 30, 2020 sourced from NYC Open Data.
You will need to install the pandas and matplotlib libraries, if you do not already have them.
Please perform the following tasks in Python using the violations.csv dataset available from the GitHub repo.
Read the CSV file containing the NYC parking violation data. Change the ‘Issue Date’ column to a date format. Then print out the number of rows imported.Perform exploratory data analysis on the imported dataset to identify invalid data. Write code to remove the impacted rows. Then print out the number of rows remaining in the dataset.Display a simple plot that shows the number of parking violations issued for each vehicle year.List the top 5 violation codes for vehicles that are registered in states other than NY.Name the street where Hondas received the most parking violations.For vehicles that are from NY only, create a plot displaying the ratio of plate types that are not passenger, month by month.Determine whether the color of vehicles with passenger plates receiving the most violations is the same as the color of vehicles with commercial plates receiving the most violations.Display the number of registration states represented in the data and the average number of parking violations per registration state.Display the plate type that has the most parking violations for each violation code.Calculate the percentage of parking violations in each county and display in descending order.
Read the CSV file containing the NYC parking violation data. Change the ‘Issue Date’ column to a date format. Then print out the number of rows imported.
Perform exploratory data analysis on the imported dataset to identify invalid data. Write code to remove the impacted rows. Then print out the number of rows remaining in the dataset.
Display a simple plot that shows the number of parking violations issued for each vehicle year.
List the top 5 violation codes for vehicles that are registered in states other than NY.
Name the street where Hondas received the most parking violations.
For vehicles that are from NY only, create a plot displaying the ratio of plate types that are not passenger, month by month.
Determine whether the color of vehicles with passenger plates receiving the most violations is the same as the color of vehicles with commercial plates receiving the most violations.
Display the number of registration states represented in the data and the average number of parking violations per registration state.
Display the plate type that has the most parking violations for each violation code.
Calculate the percentage of parking violations in each county and display in descending order.
Number of Rows: 50000
We start by making the contents of the pandas module available to our program. pandas is an easy to use open source data analysis and manipulation tool, built on top of the Python programming language. We will use it extensively throughout this project.
import pandas as pd
We import the contents of the violations.csv file by calling the read_csv() method and store it in a DataFrame, named df. A DataFrame is a two-dimensional data structure with labeled axes, consisting of data, rows and columns. Think of it like a table built in Microsoft Excel or Microsoft Access.
df = pd.read_csv('violations.csv')
We use the print() function to print the string ‘Number of Rows:’ followed by the number of rows in the DataFrame. The argument that we pass to the print() function is made up of two parts. The first is the string ‘Number of Rows: ‘ surrounded in single quotes, denoting it as a string.
The second part of the argument is calculating the number of rows in df. We use the len() function to tell us the number of rows in df, then wrap it in a call to the str() method to convert the length into a string. Finally, the + concatenates (or joins) the two string parts together. All parts of the argument passed to the print() function must be of type string.
print('Number of Rows: ' + str(len(df)))
The following data is considered to be invalid:
Registration State: values that are not a two letter state or province identifier.
Plate Type: values that are not a three letter identifier (Registration Class Codes)
Issue Date: dates not within the fiscal year (use 2020–11–30 as the end of the fiscal year)
Violation Code: codes other than those between 1 and 99 (Violation Codes)
Vehicle Make: blank values
Violation Time: blank values
Vehicle Year: vehicles with dates in the future (use 2020 as the current year)
Number of Rows: 38937
We need to subset df to filter out records with invalid data. We are able to apply multiple parameters at once by wrapping each in parentheses and using the & character between them. We use != to represent not equal to, >= to represent greater than or equal to and <= to represent less than or equal to. For the Vehicle Make and Violation Time columns, we need to check for null values by calling the notnull() method.
df = df[(df['Registration State'] != "99")     & (df['Plate Type'] != "999")     & (df['Issue Date'] >= '2020-04-01')     & (df['Issue Date'] <= '2020-11-30')     & (df['Violation Code'] != 0)     & (df['Vehicle Make'].notnull())     & (df['Violation Time'].notnull())     & (df['Vehicle Year'] != 0)     & (df['Vehicle Year'] <= 2020)]
We use the print() function in exactly the same way as Step 1 above.
print('Number of Rows: ' + str(len(df)))
We make the contents of the matplotlib library available to our program. matplotlib is a comprehensive library for creating visualizations in Python.
import matplotlib.pyplot as plt
We need to create a dataset for the plot, containing vehicle year and the number of parking violations for each those years. To do this, we group the records in df by vehicle year and count the number of parking violations for each year. Each parking violation has a unique summons number.
df_vehicle_year = df.groupby('Vehicle Year')['Summons Number'].count()
We use the plot() function of matplotlib to create the plot. As an argument, we simply pass the new dataset that we created. Then we use the show() function to display the plot.
plt.plot(df_vehicle_year)plt.show()
There are a lot of elements in this single line of code to unpack, so we will take it one step at a time. Take your time to understand this line, as we will use this structure many more times in the steps to follow.
In the first part of the statement, we filter df to those records with a registration state other than NY.
df[df['Registration State'] != 'NY']
Next we group the resulting records by violation code and count the number of parking violations for each violation code.
.groupby('Violation Code')['Summons Number'].count()
We display the violation codes with the 5 highest counts by using the nlargest() method and passing it the argument 5. The first n rows with the largest values in the count column, in descending order, are displayed.
.nlargest(5)
Finally, we use the reset_index() method to reset the DataFrame index to the original one. We use it here to present the results in a more elegant manner. We pass the name argument of ‘Count’ to name the column containing the count values.
.reset_index(name='Count')
This line of code closely follows the structure of what we used in Step 4 above. In this instance we filter df to only those records with a vehicle make of Honda, group the resulting records by street name, count the number of parking violations for each street and display only the street with the most parking violations.
df[df['Vehicle Make'] == 'HONDA'].groupby('Street Name')['Summons Number'].count().nlargest(1).reset_index(name='Count')
We subset df by selecting only those records with a registration state of NY and store it as df_ny.
df_ny = df[df['Registration State'] == 'NY']
To calculate the requested ratios, we need to create a dataset to represent the numerator (number of parking violations for all non-passenger vehicles by vehicle year) and another to represent the denominator (number of parking violations for all vehicles by vehicle year).
For the numerator we filter df_ny to only those records with a plate type not equal to passenger, group the resulting records by vehicle year and count the number of parking violations for each year. We store this as df_ny_notpas.
df_ny_notpas = df_ny[df_ny['Plate Type'] != 'PAS'].groupby('Vehicle Year')['Summons Number'].count()
For the denominator we don’t need to filter df_ny as we need all records. Instead we just group the records by vehicle year and count the number of parking violations for each year. We store this as df_ny_all.
df_ny_all = df_ny.groupby('Vehicle Year')['Summons Number'].count()
To calculate the ratios for each year, we simply divide the numerator by the denominator. This is part of the magic of pandas, where it applies the division function to each of the count pairs in the DataFrames, year by year. When a vehicle year appears in only one of the DataFrames, the result is NaN.
ratio = df_ny_notpas / df_ny_all
We are unable to plot NaN values, so we replace them with 0 using the fillna() method on the ratio Dataframe. The inplace=True parameter updates the values directly in ratio.
ratio.fillna(0, inplace = True)
We use the plot() function of matplotlib to create the plot. As an argument, we simply pass the new dataset that we created. Then we use the show() function to display the plot.
plt.plot(ratio)plt.show()
We filter df to only those records with a plate type of passenger, group the resulting records by vehicle color, count the number of parking violations for each color and display the color with the most parking violations using nlargest().
df[df['Plate Type'] == 'PAS'].groupby('Vehicle Color')['Summons Number'].count().nlargest(1).reset_index(name='Count')
We filter df to only those records with a plate type of commercial, group the resulting records by vehicle color, count the number of parking violations for each color and display the color with the most parking violations using nlargest().
df[df['Plate Type'] == 'COM'].groupby('Vehicle Color')['Summons Number'].count().nlargest(1).reset_index(name='Count')
Number of Registration States: 45Average Number of Parking Violations per Registration State: 865.2666666666667
We print the string ‘Number of Registration States’ concatenated with the count of registration states in df. To determine the number of registration states we call the nunique() method on the registration state column in df, which returns the count of distinct observations, i.e., the number of unique states listed in the column. As we did before, we use str() to convert the numeric value into a string.
print('Number of Registration States: ' + str(df['Registration State'].nunique()))
We print the string ‘Average Number of Parking Violations per Registration State’ concatenated with the average number of parking violations. To calculate the latter, we group the rows in df by registration state, count the number of parking violations for each state and then average the counts by calling the mean() method.
print('Average Number of Parking Violations per Registration State: ' +  str(df.groupby('Registration State')['Summons Number'].count().mean()))
In this step, we are using more advanced functionality. We will break this code down into snippets to explain.
We start by grouping the records in df by violation code.
df.groupby('Violation Code')['Plate Type']
Next, we want to determine which plate type gets the most parking violations for each of the violation codes. To do this we need to apply a function along an axis of the grouped data. Because we do not specify the axis, it defaults to 0, which is the index.
The lambda function calls the value_counts() method on each group (violation code) and counts the number of instances of each plate type per group, in descending order. The call to the head() method then gives us the top plate type for each violation code group.
.apply(lambda x: x.value_counts().head(1))
We use the reset_index() method to reset the DataFrame index to the original one. We use it here to present the results in a more elegant manner. We pass the name argument of ‘Count’ to name the column containing the count values.
.reset_index(name='Count')
Finally, we call the rename() method to rename the level_1 column as Plate Type, to make the output more readable.
.rename(columns={'level_1': 'Plate Type'})
We group the rows in df by violation county, count the number of parking violations for each violation county and then store it as df_county. We call the reset_index() method to reset the index and to rename the result column as Percentage.
df_county = df.groupby('Violation County')['Summons Number'].count().reset_index(name='Percentage')
We replace the previously calculated values in the Percentage column by dividing them by the total sum of violations to calculate a percentage for each.
df_county['Percentage'] = df_county['Percentage'] / df_county['Percentage'].sum() * 100
Finally, we call the sort_values method on df_county to sort the percentages in descending order.
df_county.sort_values(by='Percentage', ascending=False).reset_index(drop=True)
I hope you have found this project a useful way to learn Python data analytics. Please share your feedback in the comments section.