How to make multi-index index charts with Plotly
By: Nikita Khutorni
Tags: Data Visualization, Pandas, Data Science, Plotly, Python

How to make multi-index index charts with Plotly

When working with time series data, raw numbers often provide little insight. If you have multiple variables that vary in magnitude, changes in smaller values are hard to see. There is a simple solution to this problem: Using index charts that index the data.

While regular index charts work fine, by using a simple extension to indexing you can significantly improve your visual presentation and help your charts communicate causalities in the data. This extension is what I refer to as “multi-indexing” — using multiple indices instead of one. That allows you to analyze data from different vantage points. It is especially useful when you suspect that there were events at specific points in time that had a significant influence on the growth of variables in question.

Events with a big impact on stock market prices (Image by Author)

As an example, if you were to compare stock prices for some stocks from 1980 to today, you would find that the Black Monday in 1987, the Dot-com bubble in 2000, and the stock market crash in 2008 were all events that influenced the market heavily and would be interesting to look at. So, you would add them to the indices list, reference data to each of these dates and compare how the prices developed with respect to each reference point. Looking at the prices this way would give you a better intuition of the magnitude of each event compared to just a single index date.

In this article I will teach you how you can implement multi-indexing in time-series plots using the Python Plotly data visualization library. I will present three approaches with complete code and explain them in detail, including the advantages and disadvantages of each approach and when to use which. To keep up with the article, you should be familiar with Python and Pandas and preferably have had some exposure to Plotly.

If you care about only one of the three charts I will talk about, feel free to skip the others — the article is written in a modular way, so you won’t miss anything. However make sure to thoroughly read part 1 (Setup and Preparation), because it contains important concepts and function definitions that will be used in every chart.

[You can find the entire Jupyter Notebook of this post here.]

Article index

Image by Author

3. Dynamic index chart

Image by Author

4. Index chart with variable index in JupyterLab

Image by Author

1. Setup and preparation

1.1 Fetching and cleaning data

As already mentioned, data preparation tasks will be done with Pandas. Most of the plotting will be handled by the Plotly graph_objects module. Additionaly for one plot we will use the ipywidgets library, which provides widgets for interacting with plots in Jupyter Notebook and JupyterLab. For the time series data we are going to use historic stock market data from Yahoo!Finance API. There is a great Python library called yfinance which we will use to interface with the API.

Our goal is to build index charts that will enable the user to compare the development of selected stocks relative to a selected date on the first glance. So, let’s pick some stocks to compare. We will be featuring Facebook (FB), Amazon (AMZN), DowDupont (DD) and also Berkshire Hathaway class A shares (BRK-A) which is the highest-ever priced share for a stock. I purposefully chose stocks from a very broad price range, to demonstrate the effectiveness of indexing.

Image by Author

The call to yf.download() downloads data for the given time period in given intervals, in our case starting from today and dating 3 years back, once per week. The result is stored in a multi-indexed Pandas dataframe.

Setting group_by='column' specifies how the data will be grouped, either by price type (column) or tickers (ticker). This translates to whether price types or tickers will be the top-level index of our dataframe.

The table contains five different prices. To simplify things, we will only look at the adjusted closing price and drop everything else. We’ll remove empty values as well.

df = df.filter(regex="Adj Close")

df = df.dropna()

df.head(5)

Image by Author

1.2 Indexing: Why And How

Now that our data is ready, let’s take a first glance at the values.

df.plot()

Image by Author

Plotting raw values helps little in comparing the stocks. Due to by magnitudes higher prices of BRK-A shares, changes in all other stocks are practically invisible. Maybe scaling the data logarithmically will alleviate the problem?

df.plot(logy=True)

Image by Author

Even at logarithmic scale the discrepancy in values is too big for us to meaningfully compare the stocks. It is evident that we need indexing.

Indexing is simple: Pick a date D, the index, and divide the price at every date by the value at date D. Now, all resulting values are represented as their ratio to the value V at D and can be easily compared to each other. In our case, we need to distinguish between different stocks, so for every stock we divide all values by the value at date D for the respective stock. We can do this easily in Pandas, as we have a multi-indexed dataframe. Calling iloc on a specific index returns a Pandas series containing the values of all stocks for the specific index. When we divide the dataframe by a series, Pandas divides every column by the corresponding element in the series.

We will also add some cosmetics. Multiplying the new values by 100 will give us percentage values, that are more comfortable to work with. The second optional cosmetic is subtracting 100 from the result. This effectively sets 100% to 0% and changes ratio to relative difference. As an example, a drop to 30% of the original value will now be interpreted as original value — 70 % . Personally, I think that this interpretation is easier to comprehend.

Note that I will refer to the index date as reference date and index value as reference value. This is done to avoid confusion with the dataframe index.

# use the first date as index

reference_value = df.iloc[0] # dividing by the series divides each column by the

# corresponding element in the series

tmp_df = df.div(reference_value) * 100 - 100 tmp_df.plot() prepared_df = df.copy()

Image by Author

This looks much better. The stock price trends are immediately visible and comparable.

We are going to use indexing many more times, so let’s write a generalized function create_indexed_columns() for this technique. The function will take the following parameters:

date - the reference date that is the index df - the dataframe to work on top_level_name - (optional) name for the top level index of the normalized dataframe

and return a new dataframe with normalized data and a new top level index.

One important consideration is how to handle an input date that is not in the dataframe. Luckily, dataframe indices have a very useful method get_loc() that will find the integer index of given index. When we pass the method=nearest parameter to the method, Pandas will search for the closest index to the provided input. As our index is of type datetime , the date that is closest to the provided date will be matched.

Before we move on to the first Plotly index chart, let’s recap the concept of “multi-index” charts. As we have seen, indexing data to one point in time is very effective. However, often it is useful to look at multiple reference points. These might be specific events in the time series that had a big influence on our data. Looking at the data from multiple points gives the user the ability to easily see how the variable in question has grown relative to different reference points. The best way to accomplish this is to embed this functionality directly into the chart.

1.3 Very Short Plotly Crashcourse

This quick introduction only briefly touches on the basic concepts of Plotly. Skip it if you have worked with Plotly before.

Having multiple indices in a plot implies that they need to be switchable on the fly, therefore our plot has to be interactive. Plotly is interactive out of the box and extending interactability is relatively simple, which is why Plotly is the library of choice.

In Plotly, every figure is a JSON-like data structure with three main properties (called attributes): data, layout and frames. The first two are self-explanatory while the frames attribute contains optional frames that are used in animations. You can either first write the entire figure structure using JSON or Python dictionaries and then instantiate the figure object, or create the figure object first and update figure attributes incrementally.

As a warm up, let’s recreate the default matplotlib chart that we generated by calling df.plot() in Plotly.

In the above example we instantiate the Figure object and then add individual traces to said object one by one. We can call update_layout() with a dictionary of updated attributes to change the layout of the figure.

All changes to a Plotly figure are done this way, passing a dictionary of attributes to the figure. Plotly supports “magic underscore notation” which means that we don’t need to stepwise deepen into the object to access inner attributes but rather are able to access inner attributes directly. For example, we can simply write fig.update_layout(yaxis_type='log') instead of accessing the type attribute of the yaxis attribute - in many cases a more readable approach.

Now that we are all set, we can get on to our first chart.

2. Index chart with fixed reference dates

For this chart, we are going to use preselected dates around the time of stock market recessions and dates that had an impact on the companies we are analyzing. The dates listed below may not be entirely accurate, but they lay approximately in the timerange of the listed events.

ref_dict = {

"Cambridge Analytica scandal": "2018-03-28",

"DowDuPont separation": "2019-04-15",

"Corona stock market drop": "2020-02-14"

}

We will index our data with respect to each of these dates and include all differently indexed data in our chart. Switching between the different indexed data will be accomplished by using a dropdown menu with an item for every reference date. Plotly refers to this as “buttons”. Clicking a button will display only the data indexed by the respective reference date. To achieve this, we need to switch the visibility of the traces we want to show to on, and switch it off for the remanining traces.

First, let’s calculate values for each reference date and populate our dataframe with the resulting data. We will create a new dataframe for each reference date and then join them into one dataframe. We will also store the closest date to the given reference date, in case that we don’t have values for that reference date.

Now that our table is transformed we can build our Plotly object. We will create a new figure and add a trace for each newly computed column. We are going to provide an additional meta attribute to every trace. The meta attribute can store arbitrary JSON data and has many applications. We will use it to store the reference date that a trace belongs to. This will allow us to identify a group of traces that will be switched to visible/invisible at the same time.

That’s a lot of traces! We want to show only those that belong to one reference date. Just like before, we won’t hardcode the buttons for our exact data. It is better to generalize our procedure so that we’ll be able to handle a different set of reference dates and tickers.

To show or hide traces, we will need to set the visibility attribute of our figure. Because this is an attribute of the figure and not an individual trace, by default changes to this attribute apply to all traces. Internally, visibility is represented as an array of boolean values, where the value at a specific index represents the visibility of the trace with the same index. The trace index is the position of a trace within that array and because traces are added one by one, trace index represents the order in which the traces were added to the figure.

The visibility array allows us to target individual traces, either by passing the index of a specific trace or passing an array of boolean values where the value at each index indicates whether the corresponding trace will be shown. For example, with 4 traces, passing the array [True, False, True, False] will display only the first and third trace.

As you can imagine, relying on the order in which the traces were added to create permanent buttons is a brittle approach. If you add traces in a different order or add more traces, every mention of the visibility array would have to be checked and updated manually. This is why we added the meta attribute - this way, we don't need to rely on the order of the traces to identify which traces to switch on/off.

A simple function create_visibility_array will create the visibility array by comparing the meta value of every trace to the passed in value.

Finally, we will write a function for generating buttons.

Let’s clarify the button description. method specifies the Plotly method that the button calls on clicking. restyle updates the data and the layout of the plot. The args argument takes a list of chart attributes that will be updated by clicking the button, in our case the visibility attribute.

Lastly we will add the buttons to the figure and inspect the result.

Great, the buttons work as expected. The only problem is that it is somewhat hard to see the actual reference date. We can solve this by adding a vertical line at the x-coordinate of every reference date. Naturally, we’ll need to be able to toggle the visibility of these reference lines with our existing buttons.

There are two ways to do this in Plotly: The standard way is by adding a shape object to the figure. In our case we would simply add a line shape by calling figure.add_shape() and specifying the x and y coordinates. While this approach may seem straightforward, there are some difficulties specific to our case. Because shapes aren't traces, we can't change their visibility with our existing method. Rather, the visibility attribute of an individual shape object has to be set, doing which is somewhat obscure (this Stack Overflow question illustrates the process). Furthermore, there seems to be no concise way, similar to the trace visibility array, to change the visibility of multiple shapes at once.

The second way to add reference lines is by adding a trace that will be drawn as a vertical line. To do this, two y-coordinates must be chosen that will set the boundaries of the line. However, we need to ensure that the line covers the entire plot heightwise and in addition to that, the reference line must not affect the scaling of the plot — this means that we can’t simply choose some value below and above the minimum and respectively maximum values in our dataframe as boundaries, as this would skew the scale of the remaining values. The solution is to add a secondary, invisible y-axis to the plot that will be independent of the values in the dataframe and scale the reference lines with respect to itself.

Although the secondary y-axis somewhat increases the complexity of our figure, using traces as reference lines provides a significant benefit: We can toggle the visibility of the reference lines the same way as we do with all other traces. Thus, we only need to add the reference line trace and set its meta parameter to the reference date it refers to.

As you can see, the second approach is a better fit for our situation, which is why we will utilize it. Before we add the reference lines, let’s first add the secondary y-axis to the plot. While we are at it, we might as well add some simple styling to the plot.

The hovermode attribute specifies how the hoverlabels will be displayed in our plot. Setting this value to 'x' will display the values of every trace at the x-value the mouse is hovering over. We set the secondary y-axis to a fixed range. This way, the line will always extend throughout the entire height of the plot. Setting the overlaying attribute to 'y' is very important, as this will force the traces that are defined with respect to the secondary y-axis to behave as if they were defined on the first y-axis - concretely, the combination of this attribute and fixedrange is exactly what allows us to see the reference lines yet makes their scale independent of the y-axis.

Now that the figure is prepared, we can write a create_reference_line() function that will generate the reference lines for us. One important thing to note is that as we specify two values for y-coordinates, symmetrically, we need to provide a list with two x-values as well.

The only additions to our previous scatter trace definitions are:

mode="lines" - specifies that the traces should be drawn as lines only, without marking the points in between

- specifies that the traces should be drawn as lines only, without marking the points in between yaxis="y2" - specifies that the secondary y-axis is to be used as the y-axis

- specifies that the secondary y-axis is to be used as the y-axis line - determines the styling of the line

Finally, the last step consists of adding the reference line traces to the plot and regenerating the visibility buttons. Done!

3. Dynamic index chart

The second chart we will look at is a dynamic index chart. This chart allows the user to quickly move between reference dates, creating a visually appealing chart.

Because we are using only the Plotly graphing library, the best approach here is to create a frame for each reference date and then use a Plotly slider to move between them. By using a slider we can also add an animation effect for transitioning between reference dates, which will make the transitions much smoother.

First off we need to index the data to every date and create traces for it. The resulting data will be stored in a frame which is simply a figure['data'] dictionary that with an identifying name. We will create the traces like in the previous example using the create_indexed_columns and create_reference_line functions. A frame will be identified by the date that the data in the frame is indexed to. The function create_traces_for_date will bundle this functionality.

Next, we will create a slider dictionary that will define our slider. Some important attributes that we use are:

yanchor and xanchor - specify where the slider will be positioned

and - specify where the slider will be positioned pad - padding for the slider

- padding for the slider currentvalue - describes how the current value of the slider will be displayed including position, font size and a prefix.

- describes how the current value of the slider will be displayed including position, font size and a prefix. len - describes the length of the slider relative to the chart. 1 indicates that it matches the length exactly, which is what we want.

- describes the length of the slider relative to the chart. 1 indicates that it matches the length exactly, which is what we want. steps - is a list of steps that the slider will cycle through. We will leave it empty at first and add to it later.

The next thing to do is to create steps that we will add to our slider. A step is described by the attributes label , method and args . method specifies which Plotly function will be called when the slider value changes. By using animate we will get a smooth transition between frames. The args attribute describes the data that will be passed to the Plotly function specifed in method , consisting of a list of frames and a dictionary specifying how to transition between them. You can actually specify multiple frames for each slider step that will all be cycled through when the slider is moved to that step. In our case, we only want one frame and identify it by its name. After that we describe the transitioning properties:

frame - duration specifies the total time the frame will be displayed, redraw whether the chart should be redrawn after each step.

- specifies the total time the frame will be displayed, whether the chart should be redrawn after each step. mode - specifies how a new call to the animate function interacts with current animations. Setting it to "next" means that the current running frame will be finished before switching to the next one.

- specifies how a new call to the function interacts with current animations. Setting it to "next" means that the current running frame will be finished before switching to the next one. transition - specifies the duration of the transitioning part of the total frame duration and the easing function that is used.

There are a few important details to note here. First, setting redraw=False provides significant performance benefits in our case, as we have hundreds of frames, each with hundreds of points. Without that setting, the chart updates visibly lag behind, as you will see soon. However, there are some drawbacks to disabling redraw : For example, a rangeslider which allows you to set the view range will not be updated if redraw is deactivated. Another important drawback is that the range of the y-axis will not be automatically updated, which sometimes may not show all points on the chart. In the third part of this article we will see how to combine the best of both implementations, that is, fast updates and redrawing of the chart, at the cost of some dynamicity and requiring a Jupyter notebook. There is also a way to have it all (for a reasonable amount of data points) without compromising, but this solution requires the Dash server which is beyond the scope of this article.

The other thing to remember is that the transition duration time is a part of the frame duration time, so setting it larger than the frame duration won’t change the total duration.

Furthermore, Plotly will animate transitions between multiple frames during a slider step and the transition between frames of two neighbouring slider steps.

Now, with all functions set, we can actually create frames and steps and populate our figure. For every date, we will create a frame with traces indexed to that date and a slider step for that frame. Note that we set the name of a frame to a iso-formatted string of the date it represents and immediately pass that string to our create_slider_step function. This is done because the slider uses the frame name to determine which frame to display.

Lastly, we build our figure. We simply use the first frame as the data attribute of our figure. We use the style_plot function to configure the secondary y-axis, and we are finished!

I have warned you about enabling redraw , but let's actually see it in action. We will create the exact same index chart as before, but this time with redraw set to true.

As you see, the chart is significantly slower than before. Personally, I find this amount of lag barely acceptable; however this is the limit: if you increase the amount of data points to above 5000, the chart will be to slow for anyone to enjoy using it. To keep the speed acceptable, you need to keep the amount of data points below this number.

4. Index chart with variable index in JupyterLab

The third chart will allow a user to select a custom reference date. If you plan on presenting your work in a JupyterLab, don’t need fancy animations like in the dynamic index chart, yet still want to enbale the user to pick any date as the reference date, this chart is right for you. Note that this chart will work only in JupyterLab and not in Jupyter Notebooks! The chart will be implemented with the ipywidgets library that uses widgets. According to the official docs, widgets are

eventful Python objects that have a representation in the browser, often as a controller like a slider, textbox, etc. [..] You can use widgets to build interactive GUIs for your notebooks. You can also use widgets to synchronize stateful and stateless information between Python and JavaScript.

(Source) The idea is to use widgets to provide input for our chart via GUI and to update the chart. This way, after a different reference date was selected, the chart data will be updated through a Jupyter callback and not a Plotly event.

We will build two version of the chart, using two different widgets. One will be a DatePicker widget and the second will be a SelectionSlider widget. The second version will be a robust implementation of the dynamic index chart from part 2.

First off we copy our initial dataframe and import additional libraries. Ipython.display is used to render the widgets in the browser.

from IPython.display import display

from ipywidgets import widgets as wg dfs = df.copy()

Next, we create the chart object. Because we want our Plotly chart to be compatible with Jupyter widgets, our chart needs to be a FigureWidget object rather than a Figure object.

The style function we defined before works on FigureWidget objects as well, so we can simply apply it to our plot.

Before we can add interactivity, we need to populate our chart with initial data. We need to provide an initial state for our chart. In our case, state represents the selected reference date. We’ll use the starting date as our initial state, index the data to with respect to that date with create_indexed_columns and add all traces including the black reference line to the plot.

We have the base, our initial state. The next step is to write the update_chart function, that will receive a date as an input and property of the widget, normalize our data to that date using create_indexed_columns and update the y-values of the traces with these new values. To update mulitple values at once, we will perform the updates using the figure.batch_update() context manager. Every change to the layout and data inside the with figure.batch_update() block will be sent to the chart in one message, instead of multiple messages like with update and thus prevent stuttering.

Now we are set to implement both the date picker variant and the slider variant. We’ll start off with the first.

There are four steps left:

Create a DatePicker widget. Define a callback function picker_response that listens to changes in the widget and calls update_chart . Connect the callback function to the widget. Put the widgets into containers and display everything.

Step 1 is straightforward.

For step 2 and 3, a special handler function picker_response with the signature handler(change) must be used for the callback. change is a dictionary that holds the information about the change, such as the old and new values and the names of the widget's changed attributes. We'll use the new key to get the new value that was set on change. To bind the handler to the widget we use the observe method of the widget. By setting the names parameter of observe we specify on which attributes of the widget the callback function will be executed. In our case we need to set names='value' because we want to pass the value of the slider, i.e. the selected date to the handler.

In step 4, we simply call the ipython.display function with a vertical box widget, that contains the date picker widget (wrapped in a horizontal box) and the figure widget. The boxes act like containers, similar to HTML divs. And that's it!

Image by Author

The chart works as expected.

Now let’s reimplement the dynamic index chart with the slider using a slider widget. The procedure here is structurally identical to the previous chart. First, we define a selection slider widget, that stores a list of option values. We will use the dates in the index transformed to strings for our options. We use strings, because otherwise the current date will be formatted with zeroes for hours, which isn’t pretty.

Here is the part that will allow us to make this chart a robust alternative for the dynamic index chart: By setting the continuous_update attribute of the widget to false, the callback function will be executed only on release events. This will produce a chart that is not as fancy as the dynamic index chart, but with the same date selection interface and all the benefits of redrawing on every date slider change. If you enable continuous_update , the chart will behave exactly like the dynamic index chart, with the same (if not worse) speed.

Next, we write the callback function and connect it to the slider with observe . Finally, we wrap everything into a VBox widget and call display . Et voilà - the chart is ready!