Self Service BI using Tableau
By: Shourya Sengupta
Tags: Excel, Tableau, Business Intelligence, Metabase, Reporting

Contents

Introduction Core factors for Self-Service Business Intelligence Tools Comparison Reporting Requirements Data Management and Governance Evangelizing Tableau in Pasarpolis

Introduction

In our previous blog, we discussed our approach towards data analytics and our goal to be a data-driven organization. In this post, we will elaborate on how we are enabling our users to be more data-driven and therefore efficient.

Fig: Photo by Franki Chamaki on Unsplash

Current Data Analytics Infrastructure

PasarPolis leverages the power of Google BigQuery as a data warehousing platform along with Airflow for data-orchestration. We used a combination of Google Data Studio, Metabase, and Microsoft Excel for data analysis and visualization. We employed database optimization techniques such as partitioning, index clustering, and sharding for quick retrieval of data. Our end-users need to be able to easily query the data and quickly get the output.

Core factors for Self-Service Business Intelligence

The monthly rate of growth of our data is 30%. This has increased the need for quick analysis to get results within every team. The process of getting specific reports includes:

Understand the business requirements Discuss the metrics with the stakeholders Update reports when a new feature is rolled out Deliver reports at scheduled intervals

We realized the core factors needed to drive self-service business intelligence -

Data Exploration and Discovery

Data Automation

Data Visualization

Fig: Core factors for self-service business intelligence

Data Exploration and Discovery

Exploring and deriving insights from our data help us make quick informed decisions. We are striving to make this process of data exploration easy for our decision-makers.

Tableau has enabled our users to use “Investigative Analysis”. It helps because:

It is intuitive to use, thus managers and analysts can drag, drop data fields in dimensions and measures. Convert dimensions to measures with the click of a button. Inbuilt functions enable users to extract data from semi-structured data like JSON fields. It supports a large number of custom user functions/calculations. It is faster for pivoting data and visualization. Data processing is handled in Big Query, thus, visuals render faster in Tableau.

Data Automation

Our organization depends on the latest data that is generated. Insights from it help us tune and improve our products. The data is fetched either using a live connection or, using a store that is batch refreshed.

We explored the following methods for fetching and manipulating the data:

Live Connections : Live connections are set up for tables which have the largest volume of data. Query execution is handled in the BigQuery.

: Live connections are set up for tables which have the largest volume of data. Query execution is handled in the BigQuery. Extract Connections : Extracts are data sources that don’t require live connections, for example, Google Analytics in our use-case. We refresh the data either incrementally or fully.

: Extracts are data sources that don’t require live connections, for example, Google Analytics in our use-case. We refresh the data either incrementally or fully. Subscriptions : Decision-makers subscribe to selected data visualizations. These are sent on a daily, weekly, or monthly basis. Link

: Decision-makers subscribe to selected data visualizations. These are sent on a daily, weekly, or monthly basis. Link Data Blending: We tried data blending and found it to be quite laggy and unresponsive on Tableau Explorer. Possibly because of the volume of data in the underlying BigQuery dataset that made it a very heavy task to aggregate and perform a left join. Comparatively, custom SQL with joins performed better since that computation is done on BigQuery.

We used Tableau Desktop to create data sources. Tableau has a simple UI and this helps us in creating connections and schedules with minimal effort. (Read more here)

Data Visualization

We use data visualization daily, to review things like — the policies sold over months, the amount of Gross Written Premium over time, and detailed partner-product statistics.

We use Tableau to create:

Tableau Dashboards

We set-up partner-specific dashboards. We leveraged Tableau by presenting the information in the form of stories, with multiple views, filters, and in a wide array of layouts and formats.

Advanced Visualizations

Such as —

Histogram Gantt chart Bullet chart Motion chart Treemap Boxplot Maps

Maps

We leverage maps visualization for Travel policies to help us identify hot and cold routes. This further helps us in visualizing risk which translates to policy pricing.

Trend Lines and Predictive Analysis

One of the ways we use this is to forecast how much data will be generated and computed to predict our GCP cost.

Metabase provides the following means of visualization of data. The choices available are few and limited.

Fig: Metabase Chart Choices

The best way to showcase the flexibility of how visualizations can be presented is to visit the Tableau Public Gallery.

Fig: Tableau Public

Tools Comparison

We compared Excel, Metabase, Google Data Studio, and Tableau based on the three aforementioned core factors and came to the conclusion that Tableau made sense for a variety of our use cases.

Fig: Tools Comparison

Conclusion

Fig. An Example Partner Data in Excel

The above figure is an approximate representation of a detailed report which has data about the policies sold every day. It is not possible to make rapid business decisions based on this. One needs a powerful system to process data if downloaded locally (highly discouraged). Excel lags or fails to load data when there are more than 2 million data points. It is difficult to create visualizations and derive meaning when data is presented in the format shown above.

On occasions which require downloading of data we keep in mind the following:

The number of data points is ≤ 10,000 for use in Excel. The data analysis features provided by Metabase are limiting. The latest data is not required for analysis.

These conditions enlisted in the Tools Comparison figure are the reasons we use Metabase and Excel as detailed reporting tools.

Fig. An Example Partner Dashboard in Tableau

For the same data shown earlier in the tabular format, we created a dynamic dashboard hosted in Tableau Server. This dashboard provides more meaningful information and can be shared faster. [Note — we used dummy data for the representational purpose]

We are leveraging Tableau as our Decision-making tool, not as a detailed reporting tool.

Reporting Requirements

This is the current data flow in PasarPolis for analytics.

Fig: Data Flow

Our reporting requirements mainly cater to the following users -

Product Management Partner Management Operations Management / Finance Management (MIS Reporting)

Fig: Internal Hierarchy of Partner Product Managers

Product Management

The data science team works with the product managers to analyze the following -

Study user-application interaction Optimize notifications sent Analyze website traffic and user funnels Perform customer segmentation and targeting A/B testing — Keep a check on machine learning models’ performance Analyze application performance summaries

Partner Management

The partnership team collaborates with our Partners. The reporting requirements for Partner Management are:

Overall Partner Performance metrics across all products for ad-hoc analysis Product growth metric on a weekly, monthly, and quarterly basis Scheduled emails of the same metrics as reports and dashboards

Operations Management / Finance Management

They look into the processes for insurances, claims, and settlement. A few examples of their reporting requirements are -

Calculate Claim Ratio, Total Premium Calculations are within a time period — daily, weekly, monthly, yearly Funnels of the claims settlement process Commissions to our business partners

Our existing method involves these steps:

Requirement Analysis with manager Analyze the structure of report and visualization Discuss the data sources for the report Fetch data using SQL queries Deliver the report and confirm with the manager

When there is any update in a product, steps 1, 3, 4, and 5 are repeated. This results in additional back and forth communication.

To solve this, we created summary tables in Tableau online and they can be explored at daily, weekly, monthly levels. This has enabled managers to use them as data sources. They drag and drop the apt fields to get the information and visuals they require with little or no interaction with the analytics team, thus reducing latency in decision making.

Data Management

Data governance and management tasks include:

Create and impose policies for effective data management Develop management techniques for quality data collection to ensure adequacy, accuracy, and integrity of data Devise and implement efficient and secure procedures for data management

Tableau has a guide on how to get started with data management and governance. We picked up from there and refined our approach as we moved forward.

Tableau Sites

Tableau Sites is a collection of users, data sources, workbooks, and content, which are walled off from all the other groups of users accessing the same Tableau Server instance. Using this feature, every team is assigned a site.

Each employee has a Google account. We enabled Google auth for authenticating and authorizing logins to Tableau. Each member is associated with a site, where we granted necessary Google BigQuery permissions. These permissions include the following IAM permissions on Google console -

BigQuery User

BigQuery Data Viewer

Fig: Data Management

Publishing

We set up Tableau Explorer as a means to self-service business intelligence, thus enabling users to perform analytics from anywhere. We maintain a live shared spreadsheet to manage data sources and workbooks. This spreadsheet contains the data source/workbook name, creator, and requestors’ name.

Tableau Data Management guide inspired the set of rules and guidelines that we use. These include the following:

Data can be published in Tableau Server in the following ways:

Published Separately (Live Connection/Extracts) Embedded in Workbooks

Published Separately (Live Connection/Extracts)

We aim to minimize data source proliferation and help people find the right data for the work they do When a data source is shared; it becomes available for all Tableau users on that site When a data source is added and updated, we make sure that the connected workbooks are updated accordingly. Note, that it is a very cumbersome activity. When we use extracts, a refresh schedule is set up. Our users set up a refresh schedule for an extract, and all workbooks that connect to that extract show the refreshed data per that schedule.

Embedded in Workbooks

We set up embedded data sources for workbooks that require data that answers a very specific question The Embedded Data is available only within that workbook; it is not available for other Tableau users. If the person doesn't have access to the specific workbook then they cannot access the data source. We publish a data source only when multiple workbooks use it. We use embedded extracts that aren’t refreshed for showing snapshots in time

Keeping extracts up-to-date

When we publish a data source as an extract, we set a schedule to refresh the data. The way the schedule refreshes depend on the data source type and whether we are publishing to Tableau Server or Embedding it as a data source in a workbook.

For more information, see the following topics:

Keep Data Fresh on Tableau Online Keep Data Fresh in Workbooks

We set up the following naming convention and metadata requirements per published data source and workbook.

Data Source

Nomenclature <AGG_DETAIL>_<TASKNAME>_<TEAMNAME>_dataSource

Description

Team name Task name Request Date Requestor Author Aggregation detail: Daily/Weekly/Monthly/Quarterly/Yearly Connection Type: Live/Extracts Tables used/Base Query

Workbook

Nomenclature

<TASKNAME>_ <TEAMNAME>_viz

Description

Team name Task name Request Date Requestor Author Connection Type: Live/Extracts Tables used/Base Query

Evangelizing Tableau in Pasarpolis

We took the following steps to make Tableau accessible to our users:

Sent personalized emails to every team manager, explaining the utility of the Tableau platform for their use case Created and shared login ids with those who expressed interest Set-up team-wise sessions presenting a context-sensitive translation of data to information using the Explorer user interface Explained when to use Tableau (create data visualizations) and when not to use Tableau (do not create detailed reports with 16 columns and millions of rows of alphanumeric data). Provided every team with a set of summary tables (at daily grain) and detailed tables as data sources that we found most commonly requested We continue to do 1–1 sessions, creation of custom data sources and workbooks on Tableau, for all members across the organization Given the issues we faced with Metabase for certain types of reports/dashboards and visuals, we transferred all of them to Tableau

One of the major hurdles we faced was to make everyone understand that Tableau is not suitable for creating detailed reports as text-tables with millions of records. We created reports having specific filters for records and fields that our users wanted to view. Further, we demonstrated how to interpret charts instead of looking at the raw data, and then using a 3rd party tool like Excel to derive insights using something like a pivot table. When we receive a request to share near raw data with a user (and it is justified), we choose between publishing a fresh Metabase question and an automated/scheduled reports on Airflow, both of which have the capacity to send Excel attachments. In situations that do not require detailed data as a text-table, we made charts, dashboards, and stories in Tableau with automated emails being sent on a daily, weekly, monthly using Tableau’s subscribe feature.