Exploratory Data Analysis — MTA Turnstile Traffic Analysis for Street Engagement | by Tan Pengshi Alvin | Towards Data Science

In the Project 1 of Metis Data Science Bootcamp (Singapore Batch 5), we are tasked on exploratory data analysis (EDA) of MTA turnstile data to advise a fictitious non-profit organization, WomenTechWomenYes (WTWY) on the optimal placement of street teams (at entrances to NYC subway stations) for social engagements. WTWY wants to invite interested individuals to its annual gala to raise awareness and increase participation for women in tech, and the street teams’ agenda is to collect as many emails as possible and give out free tickets to the gala. In my analysis, I have made the following assumptions:
WTWY is constrained by time and manpower resources, hence insights from my analysis should identify top stations by traffic, as well as the peak periods in those stations.
Individuals who are interested in tech have a higher probability to be encountered in city center with a denser cluster of tech corporate offices.
The WTWY gala is imminent, and a week of MTA turnstile data is analyzed as an sample for the weeks leading to the gala.
The MTA turnstile data is scraped for the week 22 August 2020 to 28 August 2020. Taking the first few rows of the data, we observed the following data frame:
The descriptions of the column features are given here. Further exploration reveals that the ‘ENTRIES’ and ‘EXIT’ columns are cumulative serial numbers that increase with time. Also, the rows of the ‘TIME’ column progress at approximately 4 hour blocks and each turnstile has a unique combinations of ‘UNIT’ and ‘SCP’ number as shown. Some rows are also found to be duplicates, and are removed.
df.groupby(['UNIT','SCP'])['STATION'].nunique().sort_values()  
#checking combination of 'UNIT' and 'SCP' is unique to each station,implying each combination refers to a unique turnstile

UNIT  SCP     
R001  00-00-00    1
R273  00-00-01    1
      00-00-00    1
R272  01-06-02    1
      01-06-01    1
                 ..
R112  01-06-01    1
      01-06-00    1
      01-05-01    1
R113  00-00-04    1
R572  01-03-04    1
Name: STATION, Length: 4945, dtype: int64
In order to find number of people that enter and exit each turnstile, I first sort the columns in the data frame according to the following order: [‘UNIT’, ‘SCP’, ‘DATE’, ‘TIME’], then using the diff() method for Series, I take the difference between consecutive rows for ‘ENTRIES’ and ‘EXITS’ respectively to form new columns. The new columns (named as ‘ENTRY_DELTA’ and ‘EXIT_DELTA’) will then represent the actual number of entries and exits through each turnstile during a particular 4 hour period. Nonetheless, some of the values of entries/exit turn out to be either negative or astronomically high numbers.
df['ENTRY_DELTA'][df['ENTRY_DELTA']<0]
# Checking reset of entry counter values

252044   -8291685.0
264128        -91.0
264129         -4.0
264130        -34.0
264131        -25.0
            ...    
341647     -17768.0
339394    -210087.0
342664     -80186.0
337985     -32882.0
337631      -1508.0
Name: ENTRY_DELTA, Length: 1724, dtype: float64
df['ENTRY_DELTA'][df['ENTRY_DELTA']>10000]
# Checking reset of entry counter values

271427    1.057536e+09
271465    1.549702e+09
300430    8.368614e+07
297544    1.443527e+06
297546    1.443555e+06
241440    6.512100e+04
241509    3.245470e+05
241515    1.966110e+05
296920    1.405439e+06
Name: ENTRY_DELTA, dtype: float64
df['EXIT_DELTA'][df['EXIT_DELTA']<0]
# Checking reset of exit counter values

252044   -5618417.0
372575         -1.0
372585         -2.0
371439      -2794.0
378552         -1.0
            ...    
341647      -2261.0
339394     -27583.0
342664    -101904.0
337985    -266037.0
337631      -1199.0
Name: EXIT_DELTA, Length: 1345, dtype: float64
df['EXIT_DELTA'][df['EXIT_DELTA']>10000]
# Checking reset of exit counter values

300430      82744.0
297544    2955446.0
297546    2955411.0
241509      53279.0
296920     228341.0
Name: EXIT_DELTA, dtype: float64
 

Further analysis showed that this is due to the transition of rows between 2 different turnstiles in the data frame, or the reset of the serial number counter in particular turnstiles. To make the corrections, these anomalous values are then replaced with the mean of the preceding and succeeding values of entries/exit. This intervention is reasonable because the entries/exit can be approximated as interpolation between consecutive time periods.
Thereafter, the total traffic for each turnstile in each time period is calculated by summing entries and exit values. I named this column as ‘ENTRY_EXIT’.
Now, we are ready for the data visualization to derive insights for potential placements of street teams for WTWY! Using Panda’s groupby, Matplotlib and Seaborn, I then proceed to plot a histogram, bar chart, and a few heatmaps.
Plotting the histogram shows that the distribution of traffic across all MTA stations in New York City is heavily right-skewed, and that the top 10 stations by traffic are outliers in the distribution. Hence, this gives a clearer indication that WTWY could focus their engagement efforts in the top 10 stations.
Zooming in, plotting the bar chart of the top 10 stations reveal that 34 St-Penn and 34 St-Herald Square stations have notably more traffic than the rest of the stations, and should be taken as priority.
Adding the dimension of day, plotting the heat map of the top 10 stations across the week reveals the trend that people are traveling with the subway more frequently in the weekdays, as compared to the weekends. The trend applies to all the 10 stations.
Further breaking the heat map down into day versus time for each station reveals another interesting fact — stations are generally busier in the late afternoon and in the evenings, even during weekdays. This comes as no surprise, as during the Covid pandemic, many companies in NYC have adopted work-from-home arrangements, thus the morning rush hour crowd was avoided. Furthermore, employing street teams on weekday mornings could be counter-productive, as the rest of the essential workers would be busy reporting to work and are less likely to be successfully approached.
With another heat map, when we analyze the net entry and exits of commuters in each station (red regions represent net entry, and blue regions represent net exit), we can identify the stations that are located in the denser residential/hotel areas — Flushing-Main and 42 St-Port Auth stations. This is evidenced by their net exits during the evenings, which implies that people are returning back home. As these stations are not close to corporate offices, individuals interested in tech are less likely to be found in these pool of commuters.
From our analysis, we can conclude that WTWY should focus their street engagement efforts in the top 10 stations, ideally during weekdays in the late afternoon to evening periods. Moreover, if there are further manpower constraints, Flushing-Main and 42 St-Port Auth stations could be avoided as they are potentially residential and touristy areas where tech corporate offices are not located.
Finally, I hope my exploratory data analysis on MTA turnstile data has generated interesting insights, and I look forward to showcasing other upcoming data projects from the Metis Data Science Bootcamp. Stay tuned!
Find all my codes and presentation slides for this project here on GitHub!
Support me! — If you are not subscribed to Medium, and like my content, do consider supporting me by subscribing via my referral link here.