How to measure the success of dashboards
Tags: User Experience, Design, UX Design, Product Design, Data Science

How to measure success and what to do about them?

As difficult as it is, there are a couple of things you can monitor to measure success and collect feedback about your dashboards.

Usage Rate

Usage rate is the best indicator of the quality of a dashboard. If a dashboard is used by stakeholders regularly, it’s already reached a level of success. One wouldn’t use something that is not useful.

If you use Tableau or Looker, you can easily see in the backend users and frequency of them accessing the dashboard. If you don’t have those metrics available, try to bitly the links and monitor the number of clicks.

If the usage rate is low, it could mean your dashboard doesn’t help your stakeholders and you need to speak to them about the misalignment. But in my experience, more often than not, they just don’t know the dashboards exist or cannot find them! My trick is to have the right dashboards auto-delivered to their inbox at the right time.

Anecdotes of insights found, hypotheses confirmed, and great decisions made

The second level of success after stakeholders start using the dashboards is when they leverage them for insights and decision making. These feedback usually comes in the form of anecdotes that you can also proactively seek out by asking around. Even better, you know your dashboards are good if you see screenshots of them in presentations, email threads, or slack conversations.

Peer Review

Peer review is an invaluable way to get feedback, especially on your visual designs. Other analysts can spot issues that your stakeholders, who are not experts in the field, cannot easily detect. For example, using a partial y-axis scale to distort perception (per below).

Getting a peer review before sending it to stakeholders is a sure way to remove any misleading visual design landmines that could lead to wrong decisions made based on them.

Response Time

A dashboard that takes too long to load or to interact with is deemed to be unsuccessful, as it will constantly interrupt the user’s flow of thought and fail to keep the user’s attention, making insights discovery, which almost always requires filter/select actions, impossible. According to Miller’s research on “Response time in man-computer conversational transactions”, 1 second is about the limit for the user’s flow of thought to stay uninterrupted, 10 seconds is about the limit for keeping the user’s attention focused on the dialogue.

Munzner advises that “if an action could take significantly longer than a user would naturally expect, some kind of progress indicator should be shown to the user.” (Visualization Analysis & Design).

A simple test would be to interact with the dashboard yourself and see if your flow is interrupted by any response latency. Tableau also offers detailed metrics on load time for each interaction.

Common ways to reduce response time is to reduce input dataset size through change data type (string to number) or add master filters. Alternatively, you can reduce the number of charts or filter options on the dashboard.

All of that being said, you should also remember that visualization is not a simple linear optimization but a series of tradeoffs. A design that does well in one aspect might do poorly in another, such as the tradeoff between interactivity and response time.

I hope by using usage rate, anecdotes, peer review, and response time as proxies of success, visual designers can at least have something to measure against and know what to improve. You cannot improve what you cannot measure.