Creating Notebook-based Dynamic Dashboards
By: Mahdi Karabiben
Tags: Data, Data Mesh, Dashboard, Notebook, Big Data

Photo by Luke Chesser on Unsplash

In an ever-changing tech ecosystem, notebooks are steadily replacing BI dashboards as the standard tool when it comes to extracting value from data.

Just a few years ago, notebooks were merely used for ad-hoc data exploration and analysis, while dashboards were the norm when it came to business-oriented analytics, KPIs, and visualizations. But then slowly the notebook-based use cases started to multiply thanks to the shift towards data-centric approaches for all kinds of businesses and services. This shift necessitates tools that are flexible, extensible, and easy to put in place and evolve — characteristics that BI-era dashboards lack.

Notebooks, whether they’re used to run Spark jobs on top of a Hadoop cluster (Apache Zeppelin) or a Dash web application (JupyterDash), proved to be a viable option to turn data into insights in this data-driven era. And with metadata-based search engines like Lyft’s Amundsen and LinkedIn’s DataHub proving to be a powerful new asset for companies, benefitting from the flexibility of notebooks to add dashboarding capabilities to such tools would be a useful upgrade.

The aim of this article is to present a design in which notebooks are leveraged to generate dynamic dashboards, to support a Google-like metadata search engine.

How would it work?

The design consists of a data-driven search engine that relies on the following workflow:

Users can type natural language search queries via a Google-like UI (that can also be a notebook) The application then extracts keywords, tags, and search terms from the user’s query; and generates a set of similar terms The application runs search queries on the metadata engine (using the aforementioned tags and keywords) and retrieves lists of all the datasets (tables, columns, APIs, etc.) that may interest the user The application retrieves aggregates and data points from the APIs of the data products offering the datasets that are of interest to the user The application determines which chart is the most suitable for every dataset and then relies on the retrieved aggregates to populate a dashboard template (using a notebook) This dynamically-built dashboard gives a big-picture overview of everything related to the query, while also allowing in-depth analysis by the user (leveraging notebooks’ capabilities)

The application’s functional workflow

The application would rely on the existence of aggregates and extensive metadata tags for every dataset available within the company and its external sources. These tags will be stored in a dedicated metadata search tool that the application will use to answer users’ queries by matching keywords in the query with the corresponding metadata tags (via an NLP module). It will then run a dynamic notebook with the corresponding parameters to generate a fully-functioning dashboard, assigning for each chosen dataset a corresponding chart type.

Why would I need it?

With the ongoing paradigm shift towards a data mesh that would replace today’s monolithic datalakes, the proposed design can be useful in a wide range of use-cases for a data mesh architecture, and its main advantages are the following:

Offering access to all of the company’s available datasets (internal and external) via one entry-point

Allowing end-users to focus on value-added tasks and analysis by automating data gathering and aggregations

Allowing end-users to create customized notebook-based dashboards on the fly with no manual operations

Searching through all of the company’s datasets with a single natural language query

Receiving exhaustive insights/predictions/KPIs without costly and time-consuming manual operations

Generating ad-hoc analytics dashboards on-demand without a development phase

Automating data discovery and offering data-based search capabilities

These advantages can greatly ease the path towards having a democratized data platform — one that allows people with different roles to have immediate access to the company’s complete data universe, so that they can contribute to and collaborate on a data-driven vision.

The tech stack

To get this design up and running, three main external components need to be available:

1. Metadata engine

The proposed design relies on the availability of metadata tags for a big number of datasets. These tags need to be provided via the API of a metadata engine that is connected to all of the company’s data products.

Multiple tools offering such capabilities were open-sourced in the past couple of years, like Lyft’s Amundsen and LinkedIn’s DataHub. These tools usually rely on:

A graph-oriented database that stores multiple types of statistics, tags, and metadata generated from the company’s datasets

A search layer that allows us to navigate the data to retrieve all the tags and metadata relevant to a search query

We could also envision storing the following information for the different datasets that are referenced in our metadata engine:

Basic statistics for numerical columns (count, mean, standard deviation, min, and max)

List of most repeated values (with a count for each value) and basic statistics for each value

Relevant tags and metadata related to each column (most suitable chart type, owner, etc)

With that in mind, the vertices and edges of the graph database could be the following (not an exhaustive list):

Vertices: datasource, table, file, column, tag, keyword

Edges: contains, related_to, has_top_value, etc

Metadata layer example

2. Massively parallel aggregation

Since the design works best within a data mesh architecture, the mesh’s data domains need to offer the required aggregates via accessible APIs that our application can interact with.

This functionality allows the application to instantly retrieve different aggregates from multiple data products, that it then uses to populate the dynamically-built dashboard before returning it to the user.

With technologies like Apache Spark, generating such aggregates for all of the company’s datasets is no longer a complicated task, and companies worldwide use such technologies on a daily basis for different tasks and operations. This means that the infrastructure that gives us the ability to generate the aggregates for our architecture is the current industry standard when it comes to parallel data processing.

3. Notebook parameterization

For the application to generate a fully-functioning dashboard based on a set of parameters (dashboard design, chart types, a list of queries that retrieve the aggregations, etc), we need to be able to pass these parameters to a template notebook and run it dynamically.

Luckily for us, the Netflix-backed nteract suite of products provided the open-source community with a powerful toolkit for leveraging notebooks and their interactivity. With technologies like Papermill for parametrizing and executing notebooks, Commuter for their versioning and sharing, the notebook-based infrastructure for a dynamic dashboarding solution is already available for all of us.

POC

A proof of concept that showcases the capabilities of the design was implemented on Google’s GCP using the following components:

Data sources: for the POC, the data was retrieved from News API and Quandl.

for the POC, the data was retrieved from News API and Quandl. nteract Papermill: used to generate dashboards based on a specific template notebook and a set of parameters and metadata tags.

used to generate dashboards based on a specific template notebook and a set of parameters and metadata tags. GCP Stack: AI Platform Notebooks were used to generate the dynamic dashboards (Jupyter notebooks), while Compute Engine VMs were used for the core of the application. External data was stored on Cloud Storage and processed/aggregated using Apache Spark (via Dataproc).

AI Platform Notebooks were used to generate the dynamic dashboards (Jupyter notebooks), while Compute Engine VMs were used for the core of the application. External data was stored on Cloud Storage and processed/aggregated using Apache Spark (via Dataproc). Plotly: the renowned data visualization tool was used to generate the dashboard’s multiple charts.

POC design

The centric piece of the implemented application consists of using Papermill to generate notebooks based on a given template and a set of metadata tags and data sources for each graph on the dashboard.

For each chosen dataset, the type of the chart (whether it’s a pie chart or a bar chart for example) is determined by the application based on the metadata attached to the values (and so if we’re showing counts over a period of time we’d use a bar chart, but if we’re showcasing proportions or percentages the application would instead opt for a pie chart).

Additionally, the core application would determine the dashboard template to be used based on the number of datasets that match the user’s query and the type of the data within them (whether it’s mainly text data for example).

The source code for the POC is also available on GitHub.

Conclusion

The current paradigm shift that is turning our datalakes into data meshes opens the door for new technologies, designs, and tools to integrate our data architectures and make us get more value from our data.

One of these designs, as demonstrated in this post, can be letting go of our BI-era dashboards and leveraging the power of notebooks and their built-in interactivity to create a data search engine that builds dynamic dashboards on demand.

The design, of course, is still a concept at its very early stages. But it shows that our journey towards an optimal data-driven vision still has a lot to offer.