Harnessing the power of Tableau’s Metadata API with Python
By: Elliott Stam
Tags: Tableau, Automation, Programming, Python, Data Science

This tutorial walks through using the Python tableau-api-lib package and is part of a series on how to tap Tableau Server like a keg, giving you control over Tableau Server’s REST API.

These tutorials assume you have Python 3 installed already. If you do not have Python 3 yet, this will get you started: guide to install Python.

Setting the stage

Behind the scenes of the code we use in this tutorial are two APIs: Tableau’s REST API and Tableau’s Metadata API. The REST API is the gateway through which we will authenticate into our Tableau sites, and the Metadata API is what we will query in order to pull specific information mapping out the relationships between our workbooks and backend data assets.

The Metadata API returns data in JSON format. On one hand that’s great because JSON is a first class citizen in the data world, but on the other hand this can be tricky if you need to convert the JSON into something that more closely resembles a relational database table.

In the steps that follow, we’ll see that the complexity of our metadata queries is directly correlated to the number of nested elements existing within the JSON structure returned to us. Assuming we want to digest that JSON data into a pandas DataFrame for analytical processing, we will want to process it accordingly.

Step 1: install the Python packages used in this tutorial

Even if you’re a pro at these tutorials, go ahead and pull the latest version of these Python libraries. It can’t hurt.

pip install --upgrade tableau-api-lib

pip install --upgrade pandas

New to this Python stuff? Don’t sweat it, you’ll catch on quick. Follow this getting started tutorial. That tutorial walks you through getting connected to Tableau Server using tableau-api-lib.

Step 2: tap into your Tableau Server environment

Use the code below as a template for getting connected to your server. In later steps, we will build upon this boiler plate with additional lines of code. At the end of the article, you’ll find a consolidated block of code you can copy / paste for your convenience.

import pandas as pd

from pandas.io.json import json_normalize

from tableau_api_lib import TableauServerConnection

from tableau_api_lib.utils import flatten_dict_column, flatten_dict_list_column

ts_config = {

'my_env': {

'server': '

'api_version': '<YOUR_API_VERSION>',

'username': '<YOUR_USERNAME>',

'password': '<YOUR_PASSWORD>',

'site_name': '<YOUR_SITE_NAME>',

'site_url': '<YOUR_SITE_CONTENT_URL>'

}

} ts_config = {'my_env': {'server': ' https://YourTableauServer.com' 'api_version': ' ','username': ' ','password': ' ','site_name': ' ','site_url': ' '

conn = TableauServerConnection(ts_config, env='my_env')

conn.sign_in()

Fun fact: you can also use personal access tokens, assuming you are on Tableau Server 2019.4 or newer. If you’re all about the access tokens, check out my article for details on how to use them.

Step 3: practice a basic metadata query

To get comfortable with querying Tableau’s metadata, let’s start simple. We’ll move quickly here in this tutorial, and if you want more in-depth instruction on the basics then check out this tutorial for more details and examples.

Maintaining our focus on the narrow scope of this tutorial rather than explaining the nuances of the Metadata API, let’s define our first query which will be executed via Python. For more details on how this query works, please reference the tutorial linked above or Tableau’s own Metadata API documentation.

basic_query = """

{

workbooks {

name

id

projectName

owner {

name

email

}

}

}

"""

Having defined our GraphQL query, we can now execute it:

query_results = conn.metadata_graphql_query(basic_query)

In my test environment, the above query results look like this:

The blue artistry is me blocking out the associated email address

Step 4: feed the basic query into a pandas DataFrame

When our Metadata API queries are very simple, as is the case for our basic example above, then getting the data into a pandas DataFrame is straightforward:

workbooks_json = query_results.json()['data']['workbooks']

workbooks_df = pd.DataFrame(workbooks_json)

Even in our basic query, we can see that introducing owner details for each workbook has resulted in nested JSON data. In our pandas DataFrame, this results in us having an ‘owner’ column containing a dict / JSON object for each owner.

The details about our workbook owners are nested within another dict / JSON object

It’s often desirable to flatten (or unpack) the nested objects we receive from our queries. In this situation, where our nested objects are in JSON or dict form, it is no trouble to unpack the values using the json_normalize() function built into pandas (from pandas.io.json import json_normalize).

workbooks_df = json_normalize(workbooks_json).head()

The resulting pandas DataFrame looks like this:

The json_normalize() function can unpack nested dict or JSON objects

While this is very helpful for more basic queries, new challenges emerge as our query complexity grows.

Step 5: building a more complex Metadata API workbook query

Let’s up the stakes and query all of the upstream datasources related to our workbooks. That is, let’s identify exactly which of our Tableau datasources feed into each of our workbooks and take a closer look at the nested JSON data returned to us.

wb_ds_query = """

{

workbooks {

name

luid

projectName

owner {

name

email

}

upstreamDatasources {

name

luid

projectName

owner {

name

email

}

}

}

}

"""

The top level of this query is fetching workbook information including the name, ID, project the workbook belongs to. The ‘owner’ and ‘upstreamDatasources’ portions of the query are nested items. As we saw in the basic example, there is only one owner of the workbook so the owner information is returned to us as a single JSON object (or dict) nested into a new layer of the query results.

Similar to our ‘owner’ information, the ‘upstreamDatasources’ data is nested; however, this new data differs in that there can be many datasources associated with any given workbook. For this reason, the ‘upstreamDatasources’ are returned to us as a list of JSON or dict objects.

Let’s see what happens when we attempt to run the resulting data through the json_normalize() function.

wb_ds_query_results = conn.metadata_graphql_query(wb_ds_query)

wb_ds_json = wb_ds_query_results.json()['data']['workbooks']



The ‘owner’ details are still unpacked, but not the ‘upstreamDatasources’ details

As we can see, the ‘upstreamDatasources’ data has been provided to us in the form of a list. It is a list of JSON or dict objects, and the json_normalize() function has not flattened those objects for us.

Before we dive into one viable approach to unpacking this data, let’s first deal with a subtle complexity that snuck in with this more advanced example: repeated column names. While you can pass arguments into the json_normalize() function to specify column prefixes in situations where nested column names overlap with existing column names, I think a better approach is to alias the column keys when writing the GraphQL query.

Step 6: aliasing column names in Metadata API queries

To avoid messy situations where we have multiple column names fighting for the ‘luid’ or ‘name’ alias, let’s get smart about naming these values ahead of time. Here is how I handle this when writing the GraphQL queries.

Using the ‘complex’ query from the previous step, let’s alias the column names so that everything has a meaningful name right from the start:

wb_ds_query_aliased = """

{

workbooks {

workbook_name: name

workbook_id: luid

workbook_project: projectName

owner {

wb_owner_name: name

wb_owner_email: email

}

upstreamDatasources {

datasource_name: name

datasource_id: luid

datasource_project: projectName

owner {

ds_owner_name: name

ds_owner_email: email

}

}

}

}

"""

Do you see what we did there? We have aliased our columns by following a simple pattern:

<desired_column_alias>: <actual_entity_name>

Moving forward, we will use this modified version of our query, setting us up for the next task of unpacking all our nested data. Having aliased our columns appropriately, we won’t have to worry about setting column prefixes to handle overlapping column names.

Step 7: unpacking all our nested Matadata API query results

For this stage, I’m going to move away from the json_normalize() function and instead use two utility functions that come with the tableau-api-lib library: flatten_dict_column() and flatten_dict_list_column().

These two functions are fairly self-explanatory; one flattens a column containing dict / JSON data, while the other flattens any column that contains lists of dict / JSON data.

We can build recursive functions which test our columns and apply these functions accordingly until all nested objects have been iterated over, but let’s not over-complicate this tutorial. While keeping this tutorial as simple as possible, let’s see these functions in action:

query_results = conn.metadata_graphql_query(wb_ds_query_aliased)

workbooks_json = query_results.json()['data']['workbooks']

workbooks_df = pd.DataFrame(workbooks_json)

Having run the code above, we have a familiar-looking pandas DataFrame:

Now we can flatten (unpack) the ‘owner’ column. The ‘keys’ we pass to the flatten_dict() function are the keys for the owner name and owner email address; the ‘col_name’ we pass to the function is the name of the column containing the nested information:

workbooks_df = flatten_dict_column(

df=workbooks_df,

keys=['wb_owner_name', 'wb_owner_email'],

col_name='owner',

add_col_prefix=False)

Now our ‘workbooks_df’ will look like this:

And now to unpack the contents of the ‘upstreamDatasources’ column, we can do the following:

workbooks_df = flatten_dict_list_column(

df=workbooks_df,

col_name='upstreamDatasources')

Let’s quickly verify the changes that have occurred to our DataFrame:

Do be aware that unpacking lists of JSON or dict objects may result in new rows of data in your DataFrame

Finally, we can unpack our new datasource owner column (displayed in the image above as the ‘owner’ column) in the same way we unpacked our workbook owners earlier:

workbooks_df = flatten_dict_column(

df=workbooks_df,

keys=['ds_owner_name', 'ds_owner_email'],

col_name='owner',

add_col_prefix=False)

The result is a flattened, ‘de-normalized’ table in the form of a pandas DataFrame:

Wrapping it up

That does it for this tutorial on extracting Tableau Metadata API queries into pandas DataFrames!

I would like to emphasize that this is one of many viable approaches. This article has showcased how I go about flattening (or unpacking) the JSON data returned by the Metadata API, and it is entirely your decision whether you implement these steps in your own solution. The odds are that if you asked twenty people to solve this problem, you would have twenty unique solutions.

I hope this tutorial helps you get started in tapping into the valuable Metadata API and unlocking its potential!

Consolidated code

Use this GitHub gist as a template to integrate this tutorial into your own workflows.