MTA Turnstile Data: Exploratory Data Analysis | by Yingzhao Ouyang | Towards Data Science

The first week of the Metis Data Science Bootcamp (Singapore Batch 4) has just concluded, and so far it’s been an interesting experience. Below I detail the first project that we did in that period, which was an exploratory data analysis of the MTA turnstile data set using Python paired with pandas and seaborn.
Prepared MTA turnstile data from February and March 2019 for a campaign signing people up for a summer gala.
Cleaned the data by renaming columns, adding calculated fields, removing erroneous data and aggregating by station and date.
Found that weekdays had the highest traffic, but this was less the case for some stations; certain days also had anomalous traffic flows.
Recommended that the campaign focus on stations with high weekday and low weekend traffic
Anomalies, time of day and other sources of demographic data should be considered for further analysis
Here’s the link to the GitHub repository.
Women Tech Women Yes (WTWY) has an annual gala in New York City (NYC) at the beginning of the summer each year. They try to do double duty with the gala both to fill their event space with individuals passionate about increasing the participation of women in technology, and to concurrently build awareness and reach.
To this end, they place street teams at entrances to NYC subway stations. The street teams collect email addresses and those who sign up are sent free tickets to their gala.
Our task was to use subway data from NYC’s Metropolitian Transit Authority (MTA) to help WTWY optimize the placement of their street teams, such that they can gather the most signatures, ideally from those who will attend the gala and contribute to their cause.
We used historical data from MTA’s turnstiles, deciding to focus on data from February and March 2019.
The months of February and March seem to be enough lead time before the summer months for people to be able to slot the gala into their calendar.
At this point in 2020, the COVID-19 pandemic is raging, with NYC as one of the largest hotspots in the world. As a result, it has locked down, which has heavily disrupted transport patterns and emptied its streets. This is a highly unusual situation. Were this a real-life study, my advice to the organisers would had been to avoid placing street teams entirely, and either postpone/cancel the gala or move it completely online. For the purposes of this exercise, we made the assumption that the gala would happen in the summer of 2019.
To be specific, we took data from a ten-week period from the week ending 2 February 2019 to the week ending 5 March 2019, as per the code snippet below:
We also decided to focus on collecting total traffic for each station each day, leaving out the timing of the entries. The timing would conceivably be added in a later iteration of the data analysis.
Examining mta_test.columns.values, we notice that the EXITS column of the dataset has a whole bunch of trailing spaces:
array(['C/A', 'UNIT', 'SCP', 'STATION', 'LINENAME', 'DIVISION', 'DATE', 'TIME', 'DESC', 'ENTRIES', 'EXITS                                                               '],      dtype=object)
So to make our life easier, we renamed it:
mta_test.rename(columns={'EXITS                                                               ':'EXITS'}, inplace=True)
Based on the field descrpition from MTA, we know that ENTRIES and EXITS are the cumulative entry and exit register values for a given entry and exit point respectively. Since we are aggregating by day rather than by time period, we decided to subtract the first (aka smallest) and last (aka largest) counter values for entries and exits respectively.
We also aggregated entries and exits into a single column, TRAFFIC, as well as appended weekday attributes so as to make it easier to analyse the data by weekday later. Our results were put into mta_entries_exits.
The code snippet for all this is as follows:
When we sort mta_entries_exits by TRAFFIC, we find strangely large values that would appear to throw the data set off:
Further analysis of the data reveals that in certain rows of the MTA data, the counter value jumps by an inexplicable amount (example below):
A look at the distribution for entries and exits reveals that most of these values fall under 20,000:
So we removed all values of entries and exits greater than 20,000:
mta_entries_exits = mta_entries_exits[(mta_entries_exits.ENTRIES < 20000) & (mta_entries_exits.ENTRIES < 20000)]
Note that this removes data for a specific turnstile for an entire day. We deemed this acceptable for the purposes of our initial analysis as stations have many turnstiles per entry point and typically more than one entry point; however, further analysis would need the data removal to be more targeted.
Finally, we aggragated the data by station and date, as we are not looking at individual turnstiles.
mta_bystation = mta_entries_exits.groupby(['STATION','DATE','WEEKDAY', 'WEEKDAY_INDEX']).sum().reset_index()
We found the top 5 stations by aggregate traffic were as follows:
Penn StationGrand CentralHerald Square23rd StreetTimes Square
Penn Station
Grand Central
Herald Square
23rd Street
Times Square
Our recommedation would be to focus on these 5 stations.
We found that weekends see less traffic than weekdays (up to half the usual traffic). This is true in the aggregate:
And is also true for the top 5 stations:
Our recommendation would be to focus on weekdays.
We noted that not all stations followed the same pattern. For instance, the 3rd to 5th rank stations had weekday traffic volumes that were closer to the weekend traffic volumes:
This could be due to these stations being in touristy areas, Times Square being a standout example. Such short-term visitors would be unlikely to attend the summer gala, to say the least. Our recommendation there would be to focus on stations with high weekday and low weekend traffic.
We also found that for at least one of the top 5 stations, there was an unexplained drop in traffic:
Such drops (or peaks) would need to be isolated and investigated, as they could be due to special circumstances such as public holidays or emergencies. It may also be due to loss of data from the earlier data cleaning.
To gain further insights, we could do the following:
Include the TIME column so as to: a) determine peak hours so that WTWY would know when are the best times to place their teams; and b) reduce the loss of data by discarding bad data for specific timings rather than for an entire day for turnstiles.Incorporate demographic and geographical data to determine the makeup of the commuters, any geographical correlations etc.
Include the TIME column so as to: a) determine peak hours so that WTWY would know when are the best times to place their teams; and b) reduce the loss of data by discarding bad data for specific timings rather than for an entire day for turnstiles.
Incorporate demographic and geographical data to determine the makeup of the commuters, any geographical correlations etc.
This exercise in exploratory data analysis taught us the following:
The importance of inspecting column name values; it took a while to realise why we were having trouble pulling the EXITS values from the dataset.The importance of using distribution graphs and filters to tease out outliers in the data.
The importance of inspecting column name values; it took a while to realise why we were having trouble pulling the EXITS values from the dataset.
The importance of using distribution graphs and filters to tease out outliers in the data.