NYC Taxi Data Analysis Part 1: Clean and Transform Data in BigQuery
By: Linniar Tan
Tags: Sql, Bigquery, Data Cleaning

NYC Taxi & Limousine Commission (TLC) has released public datasets that contain data for taxi trips in NYC, including timestamps, pickup & drop-off locations, number of passengers, type of payment, and fare amount. These data sets can be found in this BigQuery public datasets link:

https://bigquery.cloud.google.com/dataset/bigquery-public-data:new_york

The purpose of this analysis is to understand more about NYC yellow taxi activities, such as which areas are more popular, at what time or day it is the busiest, how much one trip will cost, in general, and which area is more generous in terms of tipping. This analysis can be useful for taxi drivers who want to know where or when to maximise their income or maybe for passengers who want to avoid the traffic.

I have also shared a run-through of my analysis in this link and also the data visualisation in this Public Tableau link.

In this article, I would like to share how I cleaned and prepared the data when analysing 2016 yellow taxi data in BigQuery (new_york.tlc_yellow_trips_2016).

Null and Out-of-Range Pickup & Drop-off Latitude/Longitude

Some pickup & drop-off coordinates are nulls

I checked the minimum and maximum latitude/longitude and figured out that some are out of range. The correct latitudes range from 0 to 90. Longitudes range from 0 to 180.

/*get the range of dropoff latitude & longitude*/

SELECT

MIN(dropoff_longitude) min_long,

MAX(dropoff_longitude) max_long,

MIN(dropoff_latitude) min_lat,

MAX(dropoff_latitude) max_lat

FROM `bigquery-public-data.new_york.tlc_yellow_trips_2016`

max_lat = 405.31668 is way out of range!

Therefore, the latitude & longitude columns need to be filtered to the correct range. By doing this, null values will also be eliminated.

..

/* filter by latitude & longitude that are within the correct range */

WHERE

((pickup_latitude BETWEEN -90 AND 90) AND

(pickup_longitude BETWEEN -180 AND 180))

AND

((dropoff_latitude BETWEEN -90 AND 90) AND

(dropoff_longitude BETWEEN -180 AND 180))

Add Boroughs and Zone Names

BigQuery also provides a public dataset that contains NYC boroughs, zone names, and their geography spatial data. It can be found under this dataset name: new_york_taxi_trips.taxi_zone_geom. Using this dataset, I will be able to identify which coordinates are really under NYC by doing INNER JOIN.

/* find the boroughs and zone names for dropoff locations */

INNER JOIN `bigquery-public-data.new_york_taxi_trips.taxi_zone_geom` tz_do ON

(ST_DWithin(tz_do.zone_geom,ST_GeogPoint(dropoff_longitude, dropoff_latitude), 0)) /* find the boroughs and zone names for pickup locations */

INNER JOIN `bigquery-public-data.new_york_taxi_trips.taxi_zone_geom` tz_pu ON

(ST_DWithin(tz_pu.zone_geom,ST_GeogPoint(pickup_longitude, pickup_latitude), 0))

ST_GEOGPOINT function can only work when the latitudes and longitudes are within the correct range. That is the reason why I did filter the data using the correct latitude & longitude range beforehand.

After I filtered the latitude & longitude to NYC area, I ended up with only January 2016 to June 2016 data (~68.1 million rows). In other words, the data from July 2016 to December 2016 do not have valid latitude & longitude (either nulls or out of range). I will use these 68.1 million rows of data for the analysis, which are good enough to identify the patterns and get some insights on the trend and volume.

The filtered data covers only January 2016 to June 2016 trips

2. Out-of-range timestamp and negative trip duration

There are 5 rows with out-of-range dropoff datetime. Since it is a very small portion of the full dataset, I decided to remove these rows.

Some datetime columns are out of range (not in 2016)

I added a new column “trip_duration_in_secs” and found that there are some rows with trip duration ≤ 0, which is obviously wrong. Therefore, I will remove these rows.

There are some rows with short duration (0 < trip_duration_in_secs ≤ 10 seconds), too. It might be possible that the passenger hopped into a taxi, but then decided to cancel the trip and was charged with only the minimum fare amount. There are around 100K rows with short duration and I will keep these rows for the analysis.

Negative (-) trip duration

...

/*filter the dates to only include 2016 dates and remove those with trip duration <= 0 */

AND

pickup_datetime BETWEEN '2016-01-01' AND '2016-12-31' AND

dropoff_datetime BETWEEN '2016-01-01' AND '2016-12-31'

AND TIMESTAMP_DIFF(dropoff_datetime,pickup_datetime,SECOND) > 0

3. Zero Passenger Count

There is also obvious data error in “passenger_count” column. Since I will use it for the analysis, I will remove these rows.

Zero passenger count

4. Other negative values

There are some negative values for “trip_distance”, “tip_amount”, “tolls_amount”, “mta_tax”, “fare_amount”, and “total_amount”, which I will remove from the dataset because it is incorrect to have negative amount.

After a few steps of cleaning, now I have a dataset with 68,098,319 rows covering yellow taxi trips from January 2016 to June 2016. Before I can use it for analysis and data visualization, I will need to add some additional columns.

5. Add columns: year, month, weekday name, and hour

To help with the analysis, I create new columns that are less granular than dates.

SELECT *,

EXTRACT(YEAR from pickup_datetime) as pickup_year,

EXTRACT(MONTH from pickup_datetime) as pickup_month,

CONCAT(CAST(EXTRACT(YEAR from pickup_datetime) as STRING),

"-",CAST(EXTRACT(MONTH from pickup_datetime) AS STRING)) as pickup_yearmonth,

EXTRACT(DATE from pickup_datetime) as pickup_date,

FORMAT_DATE('%A',DATE(pickup_datetime)) as pickup_weekday_name,

EXTRACT(HOUR from pickup_datetime) as pickup_hour,

EXTRACT(YEAR from dropoff_datetime) as dropoff_year,

EXTRACT(MONTH from dropoff_datetime) as dropoff_month,

CONCAT(CAST(EXTRACT(YEAR from dropoff_datetime) as STRING),

"-",CAST(EXTRACT(MONTH from dropoff_datetime) AS STRING)) as dropoff_yearmonth,

EXTRACT(DATE from dropoff_datetime) as dropoff_date,

FORMAT_DATE('%A',DATE(dropoff_datetime)) as dropoff_weekday_name,

EXTRACT(HOUR from dropoff_datetime) as dropoff_hour FROM `bigquery-public-data.new_york.tlc_yellow_trips_2016`

6. Add columns: trip duration, driving speed, and tip rate

I am also interested to know more about the usual trip duration, driving speed and tip rate for each borough.

time_duration_in_secs = dropoff_datetime - pickup_datetime driving_speed_miles_per_sec = trip_distance*3600/time_duration_in_secs tip_rate = tip_amount/total_amount

SELECT

*,

TIMESTAMP_DIFF(dropoff_datetime,pickup_datetime,SECOND) as time_duration_in_secs,

ROUND(trip_distance/TIMESTAMP_DIFF(dropoff_datetime,pickup_datetime,SECOND),2) as driving_speed_miles_per_sec,

(CASE WHEN total_amount=0 THEN 0

ELSE ROUND(tip_amount*100/total_amount,2) END) as tip_rate ...

7. Add columns: routes

I would also like to see which routes are more popular in NYC. From that, I will gain insight about how busy the taxi activity is at each borough.

CONCAT(tz_pu.borough,"-",tz_do.borough) as route_borough,

CONCAT(tz_pu.zone_name,"-",tz_do.zone_name) as route_zone_name

I am now ready to explore the data and gain some insights about NYC yellow taxi trips. Below is the full SQL query: