Don’t Mess With Heisenberg
By: Max Rottersman
Tags: Data Science, Plotly, Dashboard, Python, Data Visualisation

Photo by Rita Morais on Unsplash

Data Visualizations Must Accept Some Uncertainty

Before developing a data visualization app, think about what’s more important, time or space. Do you want to display real-time data? Or do you want to analyze clean data? While developing my app, I wanted both. I became frustrated until I realized I was making a rookie mistake.

In Plotly Dash, I set out to build a dashboard that would get cryptocurrency prices every minute through CryptoCompare’s public API and provide Dash visualizations in which to study the data.

In my first iteration, I didn’t worry about the real-time aspect. I gave the user a dropdown in which to choose their cryptocurrency.

I then checked the Sqlite database and sure enough, every minute, a new value for Bitcoin was added through a cron job. On my Google Cloud Ubuntu Virtual Machine (VM), I tell the crontab service to call a Python script every minute, which fetches data from CrypoCompare and inserts it into a SQLite table.

Then I refreshed my Dash app. But the data didn’t change.

I posted on the Dash forum, looking for a function like RefreshOnPageLoad = True, but none was to be found. At first I was annoyed at the Dash team. Then I began to think about it and realized that if one refreshed a Dash app on every page load you’d eventually question why you’re using Dash in the first place!

First, to be perfectly clear, I am not about to argue that you can’t do real-time data visualization in a framework like Dash — you can. My point is that you should expect the complexity of your programming to rise exponentially as you try to serve two masters, timeliness of data and data quality. Let’s look at some examples from the Dash gallery.

This is the Manufacturing SPC Dashboard app.

The app makes it look like some machine is sending data to the Dash app for visualization

If we look at the code, we’ll notice that an external data source is not driving the visualizations. The app is pulling data based on its timed events.

But what if we want it to reload NEW data into the dataframe every second?

There’s little user control over the data. No fancy sliders. That is, the “streaming” apps in the Dash gallery don’t play to the strengths of Dash. They’d be as easy to do using plain Javascript and Chart.js.

Here’s another Dash “streaming” app.

Is Dash truly wind powered?!

Like the above app, the software uses complex callbacks to re-load components. There is a slider, but it isn’t for data exploration.

Every time I look at the required code to add streaming data into my Dash app my brain freezes. The callbacks in Dash are complicated enough for me. Integrating two sets of callbacks, one for updating data, and the other for handling user input, pushes my skills to the limit.

Let’s look at the strengths of Dash. In the following app, various controls (sliders) can change the graphing of the data. The user can also explore the data by hovering over the graph.

The app does not refresh data, but what if it did? That was my ah-ha moment. What are the consequences?

Where Dash delivers extreme power in a few lines of code

In Dash, HTML controls allow the user to visualize the data according to their chosen parameters.

If an external data source changes the data, can it do so without changing the user’s expectations based on their last chosen parameters?

Let’s say I’ve set my app to look at the the top 5th percentile of cryptocurrency prices. I find that it starts at, say, $100. I filter out all coins below $100? All well and good.

But what if new data enters the system and the percentile changes to $110? My conclusions are now stale and wrong!

Like the Heisenberg Uncertainty Principle, I can study the position of my cryptocurrency, or its momentum, but not both at the same time! In a physics experiment you must always physically touch, in a sense, the thing you are studying. You can only notice one electron by recording, say, another electron that bounced off it and passed through your sensor hitting yet another electron. When your recording electron hits the electron you’re studying it can changes that electron’s position or momentum. You can only study the aspect it didn’t effect. That’s my layman's understanding.

As in my example above, I can very very certain, with any given data visualization data set, that my analysis is correct for that given data set. The uncertainty arises when one asks, but what would new data say?

This is the Achilles heal of Dash and, I expect, all data visualization frameworks.

When I try to add the algorithms to update my app’s data I start making trips to the kitchen for comfort food. Here’s some code from two Dash sample apps to illustrate my point:

Amazing visualizations in 432 lines of code!

The above app plays to Dash’s strength in allowing someone to study a dataset in amazing depth.

Here is a Dash app where it has added streaming data.

Why so many lines of code for such a simple dashboard?

Looking at the big picture, the most difficult part in data visualization is deciding what experiments the user will perform. Do I want them to have the latest data, or the cleanest data? If I try to make the app do both, I will probably lean one way or another, which will anger some users. Either they’ll lose their parameters when they reload the data, or they they won’t be able to view new data.

In conclusion, you can probably do anything you want in Dash. But you should understand fundamental problems in data science. The success of your data visualization app depends partly on managing expectations.

In the future, if I want a data streaming app, I’ll probably build it with different technologies. However, if I want a user to explore a complex dataset to their heart’s content, then Dash delivers the most power in the fewest lines of code.

If I must do both, I better get a team of developers and fully stocked fridge.