Time Series and Logistic Regression with Plotly and Pandas
By: Justin Chae
Tags: Pandas, Data Science, Plotly, Visualization, Python

Put it All Together

In the preceding sections, we stepped through some of the parts and pieces required to put the whole entire visualization together, but there are some missing items. For instance, with the groupby methods, we lost the type column of categories (a, b) and it is hard to tell if there is any kind of trend at all with just three data points. For this section, let’s switch to a sample dataset with a couple hundred records and two categories (a, b) that span a few years.

Read and Group Data

In the next code block below, a sample CSV table is loaded into a Pandas DataFrame with columns as types and dates. Similarly, as before, we transform the dates column to datetime. This time, notice how we include the types column within the groupby method and then specify types as the column to count.

Group a data frame with aggregated counts by categories in a column.

Group a data frame with aggregated counts by categories in a column.

Sort Data

Before we only sorted by a single column of counts, but we need to sort by dates too. How can we sort order by both dates and counts? For this task, specify the column names in the ‘by=’ parameter of sort_values().

# return a sorted DataFrame by date then count

df = df.sort_values(by=['dates', 'count']) # if you want to reset the index

df = df.reset_index(drop=True)

Plot Data (PX)

As before, let’s see how our graph looks with sample data using Plotly Express.

Plotly Documentation Here

fig = px.area(df, x='dates', y='count', color='types')

Figure 5 — Time series of two data points as area plots. Generated with Plotly Express by the Author Justin Chae.

Now, the same data represented as a regression curve.

fig = px.scatter(df

, x='dates'

, y='count'

, color='types'

, trendline='lowess'

)

Figure 6 — Times series as scatter with Regression line. Generated with Plotly Express by the Author Justin Chae.

This is all great, but how can we overlay the regression curve on top of the time series? There are a few ways to get the job done but after hacking at this, I decided to use Graph Objects to draw the charts and Plotly Express to generate the regression data.

Re-draw time series starting with Plotly Graph Object

This time, in order to fill in the area under each line, add the fill=’tozeroy’ as a parameter to the add_trace() method.

Plotly Community Reference Here

Figure 7 — Similar area chart as before but this time drawn as traces. Generated with Plotly Graph Objects by the Author Justin Chae.

Combine Plotly Express and Graph Objects in a Loop

Here is the trick that I adapted from a Stack Overflow post where someone wanted to add a trendline to a bar chart. The insight — when we use Plotly Express to generate the trendline, it also creates data points — these points can be accessed as plain x, y data just like the counts in our DataFrame. As a result, we can plot them in the loop as a Graph Object.

To reiterate, we are plotting two categories of data to a graph with Graph Objects but using Plotly Express to generate data points for each category’s trend.

Figure 8— A Plotly Graph Object with lines as area charts over time with trend lines. Generated with Plotly by the Author Justin Chae.

Some Housekeeping

At this point we have the foundational graph object with lines and trends but it would be nice to clean up a few things. For example, the labels are not very helpful and the colors are off.

To take care of some housekeeping issues, add more parameters to the go.Scatter() method. Since we are passing a grouped DataFrame in a for loop, we can iteratively access the name of the group and the elements of the DataFrame. In the final version of this code, notice the parameters for line and name in the scatter objects to specify a dashed line among others.

A complete graphing solution.

The final result after grouping an aggregated DataFrame and plotting it with a for-loop.

Figure 9 — Data over time showing counts of different categories with a trend line overlay. Generated with Plotly by the Author Justin Chae.

As a final step to deploy to the web, I am currently developing on Streamlit. You can see a web version of this sample project here and check out another article on how to deploy code with Streamlit in the link below. In addition, the entire source project behind this article and the app is available in my GitHub repo here.