A Practical Approach Using YOUR Uber Rides Dataset | by Felipe Alves Santos | Towards Data Science

Exploring data is certainly one of the most important stages in Data Science processes. Despite its simplicity, it can be a powerful tool to put you ahead on data and business context, as well as to determine crutial treatments before creating machine learning models.
To turn things a little bit more interesting, I’ve decided to have some fun with Python on my personal Uber rides data and see which insights I could extract.
In this post, I will guide you through the following steps:
1. Business Problem Definition
2. Data Discovery
3. Data Preparation
4. Data Analysis & Storytelling
Note: Data Preparation is usually a stage that requires lots of work around data formatting, cleansing and manipulation, but making your data CONSISTENT is surely a success factor for your analysis and future modeling.
Uber’s data download feature provides you in-depth information about your rides. You can request access to your data through the following link: https://myprivacy.uber.com/privacy/exploreyourdata/download
After your request is done, an email with the download link will be sent to you (usually in the same day).
For security purposes, your data is only available during 7 days.
Before starting manipulating and analysing data, the first thing you should do is to think about THE PURPOSE. What mean is that you should think about the reasons why you are up to conduct any analysis. If you are uncertain about this, simply starting formulating questions regarding your subject like What? When? Where? Who? Which? How? How many? How much?.
Depending on how many data and features you have, analysis could go to the infinite and beyond. So that’s why (after thinking process) I decided to focus on the following questions:
a. How many trips I did over the years?b. How many trips were Completed and Canceled?c. Where most of the dropoffs ocurred?d. What product type is usually chosen?e. What is the avg. fare, distance, amount and time spent on rides?f. Which weekdays have the highest average fares?g. Which was the longest/shortest and more expensive/cheaper ride?h. What is the average lead time before begining a trip?
Importing libraries and dataset.
Checking basic dataset information (data types and dimensions)
<class 'pandas.core.frame.DataFrame'>RangeIndex: 554 entries, 0 to 553Data columns (total 13 columns): #   Column                Non-Null Count  Dtype  ---  ------                --------------  -----   0   City                  554 non-null    int64   1   Product Type          551 non-null    object  2   Trip or Order Status  554 non-null    object  3   Request Time          554 non-null    object  4   Begin Trip Time       554 non-null    object  5   Begin Trip Lat        525 non-null    float64 6   Begin Trip Lng        525 non-null    float64 7   Dropoff Time          554 non-null    object  8   Dropoff Lat           525 non-null    float64 9   Dropoff Lng           525 non-null    float64 10  Distance (miles)      554 non-null    float64 11  Fare Amount           554 non-null    float64 12  Fare Currency         551 non-null    object dtypes: float64(6), int64(1), object(6)memory usage: 56.4+ KB
.rename( ) method allows you to rename axis labels (indexes and columns). In this case I decided to normalize column names to clean up coding, as long you can columns by typing <data_frame>.<column>
Use .head( ) method to gain more sensibility around data formatting and understand the overall structure of the dataset values.
Taking a look on the continuous variables, we notice the presence of some outliers. However these outliers do not seem to reflect any abnormal value (e.g. fare_amount = 1000 BRL), which may let us a little bit more comfortable.
P.S. In case abnormal values are found, some treatment should be probably considered (e.g. outliers replacement/removal).
The charts below show a different perspective of the variables distribution. In this case, we see that both variables present an assimetric distribution (positive). For distance we notice that the higher frequency values are shorter distances, and for fare amount we have the same behaviour.
Additionally, we also notice that the standard deviation are high, taking ‘means’ as our reference. This means that values in both variables are very dispersal.
Not surprisingly we have a strong correlation between ‘fare_amount’ and ‘distance_miles’, inferring that as much you stay on the ride, higher will the fare be.
sns.scatterplot(x='distance_miles',y='fare_amount',data=df1);
I decided to remove the column fare_currency, since all my trips happened inside a single country (Brazil).
Now let’s check existance of missing values.
Despite empty Lng and Lat values (29 total), there were found 3 records without product_type. As shown below, these records are insignificant to my dataset, since practically no columns are fulfilled.
So now, let’s get rid of these 3 records before proceding.
While analysing the first categorical column <product_type>, I could clearly see that some work was necessary, since I could find different values referring to the same category. Then, I summarized 15 original categories in 5 ones.
As the scope of this analysis is only around Uber rides, I removed UberEATS records from my dataset.
Our second categorical feature <status> seems well classified in 3 status, which will not require any kind of treatment.
Dates usually increase a lot your power of analysis, since you can break it down to different parts and generate insights from different perspectives. As previously shown, our dates features are in fact object data types, so we need to convert them into datetime format.
Now, let’s break down <request_time> feature into different date parts. I just did that for <request_time>, since I'm assuming that all rides were completed in the same day (believe me, I have already checked that! :D ).
Based on <fare_amount> and <distance_miles> features I've created a new feature called <amount_km>, which would help us understand how much is payed by kilometer ridden.
Delta time between <request_time> and <begin_time> will let us now how much time (in minutes) I usually waited for Uber cars to arrive at my destination. In this case, it was calculated in a minutes base.
Similarly, delta time between <dropoff_time> and <begin_time> will let us now how much time (in minutes) was spent on each trip.
As features in records with Canceled and Driver_Cancelled status will not be useful for my analysis, I set them as null values to clean up a little bit more my dataset.
RECOMMENDATION: Do not start your analysis without completing the Business Problem Definition, since it determines your analysis’ focus and quality. Besides that, this process will help you to think about new possibilities/questions while trying to answer the previous ones set.
NOTE: In order to organize better my analysis, I will create an additional dataframe, removing all trips with status CANCELED and DRIVER_CANCELED, since they should be disconsider in some questions.
A total of 444 trips were completed from Apr’16 to Jan’21. If we disconsider 2016 and 2021 (not full years), we can clearly see that from 2017 to 2019 the average rides per year is 124, and that there is a huge drop from 2019 to 2020 (-51%). This is easily explained by the COVID outbreak.
Now, imagine if we extrapolate this result to all Uber users...
Looking at the stacked bars below, we can see that excluding 2015 and 2021 (due to low trips volume), 2020 has the highest cancelation ratem. This could be an alarming indicator, considering the drastical impacts caused to the businesses after Covid outbreak. In overall, cancelation rate was 17.9% (considering RIDERS and DRIVERS cancelations).
The following heatmap dynamically shows the most frequented areas throughout different hues and intensities. This could be a valuable information for Uber to adjust prices and optimize demand in certain regions, also combining time space data to track users behaviours.
UberX is far the prefered product type with a frequency of 90.3%. So I could say I am the type of user which usually looks for affordable prices.
Considering all trips, the average amount spent by trip is 19.2 BRL, ridding in approx. 8.1 km. So, if we do a quick simulation on how much I would spend in a year to do daily round trips we would have: 365 days * 2 trips * 19.2 BRL/fare = 14,016 BRL/year
Also in average, It was spent approx. 2.4 BRL/km and 21.4 minutes by trip.
According to the chart below, we can see that Mondays, Wednesdays, Fridays and Sundays were (in average) the most expensive weekdays. Therefore, it allows us to better understand the weekly seasonality, and find out days with higher profitability for Uber and its drivers.
The table below show records with the longest (31.77 km) and shortest rides (0.24 km).
Analysing amount paid by km ridden we have: expensive (46.96 BRL/km) and cheaper (0 BRL/km). Cheaper trip certainly refers to a free ride, while the expensive cost 46.96 BRL. This effect is basicaly driven by fixed minimum fare in high demand periods, since the total distance was only 0.24km.
It takes approximately 5 minutes to start trips, after they are requested.
4.9 minutes
Exploratory Data Analysis is not a trivial task! It requires lots of work and patience, however it is surely a powerful tool if correctly applied to your business context.
This post briefly demonstrated some tips and steps to make analysis easier and undoubtely highlighted the crutial importance of a well defined business problem, guiding all coding effort to a specific objective, also highlighting important insights. This business case also tried to reflect a pratical application of python in daily business activities, showing how fun, valuable and interesting it could become.
Thank you so much for getting here! I hope you liked it! 😃