Analyzing E-Scooter Activity through Visualization and Machine learning in Python | by Himanshu Agarwal | Towards Data Science

According to the recently published 2019 TomTom Traffic Index report, traffic congestion increased in 239 out of 416 cities (57%) worldwide. In the US alone, the traffic congestion level was at +20% among the major cities. Meaning, a trip will take 20% more time than it would during the city’s baseline uncongested conditions. Los Angeles tops the list with +42% and then New York city at +37%. Not to mention the environmental impacts it will have due to the higher emissions.
With the above being said, people are using alternative ways to commute such as carpooling, biking, public transit or a combination of either of those. One such alternative is E-Scooter. E-Scooters are fairly new to the market and mainly target the “last mile” part of a commute.
One such company is Bird. For those of you unfamiliar with the company, Bird operates dockless scooters in more than 100 cities worldwide. There are various models of scooters and I personally rode two of them during my field exploration trip to Washington D.C. Rides are inexpensive, and unless you have a terrible balance, riding this should be very easy.
Scooters are usually installed with lots of sensors that leads to large amounts of data being generated and hence makes for an interesting data exploration.
I set out with the following questions in mind:
1. What is the spread of e-scooters in and around Washington D.C.?
2. What is the average trip distance? Does usage depend on the day and time of the week? The latter seems intuitive but still needs to be proved.
3. What can I learn about the behavior of customers using the scooters?
4. Can I use the findings from above to generate insights?
Certain states require companies to have an open-data policy. But there is none for Washington D.C. Hence Bird does not a publicly accessible API for D.C., but one individual, who goes by ubahnverleih on Github, managed to hack together and develop one. ubahnverleih has developed APIs for multiple other ride sharing companies, so if you’re looking to work on any such data, check out his Github.
Washington, D.C. and its surrounding areas span roughly about 100 square miles. In order to cover the whole region, I carefully chose 49 different coordinate points (roughly 36 sq. mile in area) such that pinging these points in the API, returns me data of all the scooters in the region. Methodology I used for this can be found here.
That was the first part of the task. It was, however, missing one element: time. I wanted to study scooter usage over time, and not merely how scooters were geographically distributed in one instance.
To capture scooter usage over time, I hacked together a script which ran every 5mins and collected scooter data for all the above mentioned coordinate points. I kept this program running for a period of 1.5 months (48 days) resulting in roughly about ~8M data points.
There were a number of technical challenges with the API wrangling phase of the project, and while I won’t list them all, I think it’s informative to mention one in particular, to illustrate the kinds of challenges that arise anytime you try to gather data in the wild:
I had written a script that collected data throughout the day and the night. After initiating the collection process, I noticed the program had crashed for two mornings straight.
My first instinct was to assume it to be due to a bug in the code, but even after examining it, no light was shed into what might be causing the snag.
I then went through the collected data and noticed a pattern, the last API call every night before the crash would always be for sometime before 12am (of the next day). Didn’t provide any insight to the issue at the time but it was a small clue towards learning about the cause of the issue. After a lot of tinkering and unsuccessful test runs with the code, I decided to sit through the night to witness the breakdown myself.
That approach, and the hacky detective work that followed, led me to discover that the API wasn’t returning anything from 12am-4am every night but started working once it was past 4am. Companies often run maintenance runs on their systems, some more frequently than others, and that could be one explanation for that. Anyway, after knowing that, it was an easy fix in that I added a small section to my script that accounted for that nightly event. But it was surely a challenging diagnosis.
With the data now collected regularly, I created a PostgreSQL database to organize and manage the data better. Added another section to the script to do the same from my Jupyter notebook.
After going through the above steps and overcoming challenges, below is what the resulting data looked like.
Each row consists of all the pertinent information for one individual scooter.
For example, going through the first row in the snapshot above, the scooter ‘3db0f8a8–72ed-4d4b-aff8-f0f2de53c875’, is of model type ‘bd’, and was located at (38.927739,-77.097905) on 10/3/19 at 4:00:53 AM and had a battery level of 42, meaning, it would run for another5069 units (about 6 miles, based on the scale from the data).
There are some empty and redundant fields, which weren’t useful and hence weren’t considered for my analysis.
Below is an overview of the approach I took towards answering the above posed questions.
I have gone through the results towards the end of the post.
Data cleaning.Creating dataset of scooter trips from the collected data, and using it to study usage and activity but also aide in further analysis.Defining rules for detecting when scooters are stationary and when moving, in order to analyze and compare usage activity.Feature engineering using the above said rules and clustering on scooter locations to track cluster centers.
Data cleaning.
Creating dataset of scooter trips from the collected data, and using it to study usage and activity but also aide in further analysis.
Defining rules for detecting when scooters are stationary and when moving, in order to analyze and compare usage activity.
Feature engineering using the above said rules and clustering on scooter locations to track cluster centers.
For the sake of this blog post, I will be focusing more on the approach and results and not go too much into the code behind it. I’ll be showing certain sections of code if needed. Full code can be found on my GitHub here.
Two fold:
A. Fixing data types.
B. Creating a datetime feature to allow for slicing and dicing the data.
I created a helper function to do the above for us..
def fix_dtypes(df): int_cols = [‘battery_level’, ‘estimated_range’] float_cols = [‘latitude’, ‘longitude’] df = df.astype({col: ‘int32’ for col in int_cols}) df = df.astype({col: ‘float32’ for col in float_cols}) #Combining date, time column to datetime df[‘date_time’] = df[‘Date’].map(str) + “ “ + df[“Time”].map(str) df[‘date_time’] = pd.to_datetime(df[‘date_time’]) return dfdf_scootsbydate = df_multiple_days.groupby(‘Date’ [‘id’].nunique()df_scootsbydate.mean()[Out]: 5337.555555555556So, on any given day, there are about 5338 scooters spread throughout the D.C. region.
The raw data is essentially a series of snapshots taken each time the API is pinged. Each ping would essentially return data for all available scooters present at that moment of time.
At a very simple level, one can think of a flip book animation. Just like how each page is a snapshot and all pages put together shows us the entire sequence from start to the end, with first page showing the start of a sequence and last page showing the end of a sequence. Similarly, the first API call would show time & location of scooters at the start of the day and the last API call would show it for end of the day.
Below is a snippet of the resulting data and how I accomplished it.
Without going into too much technicality, (‘latitude_first, longitude_first) , are the coordinates of the scooter when it was put into operation at ‘date_time_min’ date & time, and, (‘latitude_last, longitude_last) are the coordinates when it was taken out of operation at ‘date_time_max’ date & time.
‘time_diff_hrs’ : Total operating hours of the scooter for the day. Which was calculated as just ‘date_time_max’ — ‘date_time_min’ .
Calculating distance travelled (‘distance_miles'):
‘distance_miles’ : Distance covered by a scooter in a day (in miles).
From the above, we have (‘latitude_first’, ‘longitude_first’) and (‘latitude_last’, ‘longitude_last’) for each scooter. Now we need to calculate the distance between these two locations. There are a couple of ways to calculate distance between two points. But here, since we are dealing with GPS coordinates, we use Haversine distance for our calculations. Haversine takes the Earth’s spherical shape into account where formulas like Euclidean distance don’t and hence why Haversine is appropriate to use in this case.
There are some good Python packages out there that do that. I used this resource to build a helper function to do that job for me.
At this point, we have a dataset of scooter trips for each day of the 1.5 months the data was collected. We can now analyze the usage and movement of scooters. But it wasn’t as straightforward as just filtering down scooters based on the distance feature, in other words, subsetting data to select only those scooters whose ‘distance_miles’ was 0.
Stationary : Scooter is considered stationary if it met ‘Cond_1’ and 'Cond_2'. Non-stationary : Scooter is considered to be non-stationary (or in a trip) if it doesn’t meet ‘Cond_2’ but meets ‘Cond_3’ . That is, if the ‘distance_miles’ is greater than 0.5 miles and ‘battery_diff’ is less than 50 (explanation given below).
There were a few challenges and I go over them as I explain the three conditions I used to recognize if a scooter was stationary/non-stationary:
1. Cond_1: The ‘time_diff’ should be greater than 4hrs. There were are lot of instances when scooters would just appear for a brief time during the day and disappear for days after that. Could be due to various reasons such as repairs, maintenance etc. This article validates that in some way. The disappearance of scooters doesn’t really count as scooter usage activity and hence weren’t counted towards our analysis. This rule enabled the focus to be on scooters that would “surely” be stationary. 2. Cond_2: ‘distance_miles’ should be less than 0.5miles. One would guess that for a scooter to be deemed stationary, all you need is for the distance feature to be zero, right? Well, the data had a few inconsistencies in that, we noticed instances where scooters were shown to have moved really small distances, less than a few yards or so. A few yards doesn’t really count as a “trip”, so we kept the threshold at 0.5 to avoid taking these scooter movements in our analysis.
3. Cond_3: ‘battery_diff’ should be less than 50. This was a sneaky one, as this was realized after a couple of iterations of the analysis. There were occasions when scooters, after they were taken out for charging, were placed back into a completely different location than to the one they were picked up from. Our initial analysis counted this as scooter movement but after recognizing this irregularity, using this condition helped us avoid making that mistake. 50 was a good threshold because the battery level would change drastically in such situations, for e.g. a scooter with a battery level of 10 was taken out for charging and placed back with a battery level of 100. The difference in battery level in this case will be 90. This condition would rightly recognize this irregularity and not count this scooter towards our analysis.
Feature engineered the original dataset, using rules explained above, into subsets of combinations of stationary / non stationary, and weekdays / weekends.
Below is a snapshot showing some statistics for stationary scooters..
As you can see from the above, there are a pleothera of scooters available around a certain point. In order to generate insights we need to understand the overall movement of scooters rather than focus on individuals. We used K-Means and HDBSCAN clustering methods to achieve that.
We’ve done all the preparation work, now it’s time for the results...
Of course, the number of scooters available will vary over time, but my goal here was just to get a high level understanding of the extent to which Bird had deployed scooters throughout the region.
So, on average, there were about 5338 scooters spread throughout D.C., which is a lot more than I had initially expected.
Tackling the first part of the question.. the average trip distance was approximately 1.88miles.
Curious about verifying my finding, I went about searching any report or article on the web and as it turns out, my finding was pretty close to this report (link below) published online...
“The average length of a Bird trip is 1.6 miles”
This is pretty close, considering this was only based on 1.5 months of data.
Second part... comparing usage activity across different days...
Where 0 — Monday, 1 — Tuesday and so on.
As we can see from above, usage on weekdays seem to be more, slightly, than usage on the weekends.
Which honestly surprised me because I was assuming the scooters to be used more for recreational purposes, and hence more activity on the weekends.
But from my results above, weekday activity seems to be more than weekend. This suggests that people might be using scooters more for commuting purposes. Maybe getting to and from work.
Which, when thought about, can make sense considering the traffic in D.C. region. People could be using scooters to cover their “last-mile” after stepping out of a metro/bus station.
I was able to validate my theory when I came across this report that Bird had published about its scooter usage in D.C...
“....50% of riders reported that they use Bird to get to and from work. But riders are also connecting to public transportation — 29% of riders use Bird to connect to D.C. Metro and the bus...” — Bird
What better way to understand something than to visualize it? I used this cool package in Python called folium whereby you can build custom visualizations on real world maps.
Due to the volume of scooters present and their movements, plotting individual scooters and their movements would just congest a map and make it difficult to interpret. I used the cool heatmapwithtime plugin on folium to draw heatmaps to achieve that. As you all may very well know about a heatmap, the darker the colors, the more the concentration, in this case scooters.
To do this, I clustered scooters based on their geo-coordinates and then visualized these clusters through different periods of time. This will give me a good understanding of the overall movements. I used K-Means and HDBSCAN clustering algorithms to do this and then I compared the results for different situations as shown below.
The images above represent a heartbeat-like pattern of Washington D.C.
The 8am-10am slot on weekdays shows heavy usage, simmers down towards the noon slot but picks up again towards the evening in the 4pm-6pm slot.
The activity is lesser on weekends than on weekdays but follows a similar usage pattern as the weekdays.
On any given day, the activity is concentrated towards the downtown region in the early part of the day and spreads to regions surrounding the downtown towards the latter part of the day. Representing the commuting behavior of the users.
The usage data and insights from above can be used to find the most used routes and using that, public transportation services such as buses, metros etc. can be improved. The routes can be clustered and then ranked based on popularity.
The density of usage can be cross-referenced to population density based on demographic and resources can be focused towards those that rank high on both of these fronts.
Comments or Questions? Please email me at: himanshu.agarwal20792@gmail.com or reach out on LinkedIn