How To Plot Time Series. Produce elegant plots that tell a story | by Nadim Kawwa | Towards Data Science

Dealing with time series can be one of the most insightful parts of exploratory data analysis, if done right. In This post, we are going to use the checkin log from the Yelp Dataset to explore trends across different time periods using Pandas and Matplotlib.
After downloading the data, we need to know what to use. The block below shows a sample entry from the checkin.json file based on the Yelp Documentation:
{    // string, 22 character business id, maps to business in business.json    "business_id": "tnhfDv5Il8EaGSXZGiuQGg"    // string which is a comma-separated list of timestamps for each checkin, each with format YYYY-MM-DD HH:MM:SS    "date": "2016-04-26 19:49:16, 2016-08-30 18:36:57, 2016-10-15 02:45:18, 2016-11-18 01:54:50, 2017-04-20 18:39:06, 2017-05-03 17:58:02"}
We can read the input file with pandas read_json method with arguments orient=columns and Lines=True.
Upon reading the data, our dataframe looks something like this:
The date column entries are strings such that each date is separated by a comma. By looking at them we can tell that the format is indeedYYYY-MM-DD HH:MM:SS.
For readability, we want our dataframe to look more like this:
The unfolded event log makes more sense from an EDA perspective whereas the format in the JSON format makes more sense for memory storage.
We can obtain this result by creating a dictionary where the keys correspond to dates and the values correspond to business_id. Here we assume that the dates are granular enough so that no two dates are the same. Keep in mind that the python documentation does not allow duplicate keys within a single dictionary.
The implementation can be done as follows:
Say we want to know what are the total checkins for all the years available. We will use this as a gateway to introduce the pandas Grouper which can be used inside the groupby method. For more information about frequency aliases refer to the pandas docs.
The code returns this neat plot:
it is a good start, however what if we want to dive deeper? We will zoom in on the year 2014, though any other year will do. Selecting date ranges is an easy one-liner:
A first indicator of seasonality is to look at the weekend, since the data is from the USA & Canada the weekend is Saturday and Sunday. Bear in mind that the weekend varies by country, for example Dubai’s weekend is Friday and Saturday.
We therefore want to detect the days that are a weekend and write a customizable function for it:
The function above returns indices that are weekends. Next for plotting we define another helper function to highlight spans of the plot that would correspond to these indices with the help of pyplot axvspan:
Finally we can use these two functions inside a larger one that takes in a dataframe:
The result is this neat plot:
The plot above matches the intuition that people go out more on weekends than weekdays. To take a better look at that buildup we can take a closer look at April 2014 using the same function.
What if we want to show a more granular view of April? Since each entry in the original dataframe is a single checkin, we can groupby 30 minute frequencies and count as follows:
The result is the plot below:
We saw how we can zoom in on time periods to get a sense of seasonality. However, businesses also want to know what is the expected number of checkins for a given day and how it might vary. For example, we might be solving for a regression.
Therefore we need to build plots that show the average checkin for a given day of the week and some sort of confidence interval.
We will use the checkin_halfhour defined in the previous section to extract the day of the week:
Our dataframe now looks as follows:
Next, we build the day_avg dataframe using pandas aggregate method. For our purposes we will aggregate:
Sum
Mean
Standard Deviation
The code block below does this for us:
We now have a dataframe we can use to draw intervals:
How we define the interval depends on the distribution of the data and the business value of narrow/larger intervals. In our case, we want to generate the expected checkin for each day of the week and grab all the information within two standard deviations.
Doing so is a matter of watching out for the levels in the index of day_avg as shown below:
The resulting plots reveal a lot in terms of checkin behavior, for illustration we show Monday, Wednesday, and Saturday.
The amount and depth of time series analysis depends on the problem we are trying to solve for. For example, we might want to evaluate restaurants, or businesses that only open from 8AM to 3PM such as cafés.
An objective of EDA is to prepare for feature extraction. With the exercise above we can start thinking about features such as the most common checkin time, or checkins by day.