Data anonymization layer using Snowflake and DBT | by Tommaso Peresson | Towards Data Science

HousingAnywhere is an online marketplace platform for mid to long-term rents. Like any other data-driven business we have to deal with both personal information and GDPR regulations. In this brief article, I’ll walk you through our solution to anonymize PII (Personal Identifiable Information). The technologies that I’ll be using are Snowflake[1], a popular data warehouse cloud solution and DBT[2](Data Build Tool) a data transformation tool.
Starting with some context; Every day our reporting and analytics are served by data that we ingest from various sources, spanning every aspect of our business. This raises a problem, how do we prevent users and services from unnecessarily accessing raw data that exposes PIIs of HousingAnywhere’s clients?
from┌──────────────────┬──────────────────────┬─────────┐│       NAME       │         EMAIL        │ COUNTRY │├──────────────────┼──────────────────────┼─────────┤│ Tommaso Peresson │ placeholder@mail.com │ IT      │└──────────────────┴──────────────────────┴─────────┘to┌──────────────────┬──────────────────────┬─────────┐│       NAME       │         EMAIL        │ COUNTRY │├──────────────────┼──────────────────────┼─────────┤│ 87444be...eb6b21 │ 3c33076...e1ceaf     │ IT      │└──────────────────┴──────────────────────┴─────────┘
In order for this solution to be effective, you will need two areas in your data warehouse: a stage one to store all the raw data coming from external sources, and another to accommodate the results of the anonymization transformation. To do that I will create two databases and call them PRIVATE_SOURCES and SOURCES respectively. If you need some help to get to this stage follow this[3] guide.
Another key element is to manage the access control and the privileges so that the PRIVATE_SOURCES database is only accessible by admins and data engineers, and the SOURCES database by potentially everybody. I will not get into depth on this topic but if you want to learn more about privileges and access control read this article[4].
We will also need a table containing the information on which column names to affect and their respective anonymization methods (anonymization_mapping) a table containing the salt[5] used for secure hashing.
Without anonymization, anybody who has access to the data warehouse could steal and use personal information. A data leak is as simple to perform as typing select * from users . Taking inspiration from this post[6] we decided to build an anonymization layer using DBT.
Let’s start to uncover our solution by looking at an example of the final usage in a model that we want to anonymize.
As you probably can see after select there is a macro call. This is intended to replace the *operator and adding the anonymization transformation. Our goal was to find a maintainable and scalable solution that isn’t text-intensive since we have hundreds of tables and manually selecting each column isn’t, of course, a considerable solution. Let’s now take a deep dive into the macros to understand the implementation details.
Let’s now take a look at the two DBT macros that make up this system.
1.Starting with anonymize_columns() the aim of this macro is to substitute the * operator plus adding the anonymization transformation to specific columns.
We can say that this macro performs two different operations:
(rows 2:13) by fetching all the column names from the private_sources.information_schema.columns and join them with the anonymization_mapping on the column name. As a result, we will have a table containing the columns of the table in object paired with the right anonymization function. For example, the users table has 3 columns: name, email and country. It might be that in our specific case we want to only anonymize the name and email columns, leaving country untouched.
To do this our anonymization_mapping will have to contain these two rows indicating which columns have to be transformed and with which function:
ANONYMIZATION_MAPPING (config)┌───────────────────┬─────────────┬───────────┐│ TABLE_NAME_FILTER │ COLUMN_NAME │ FUNCTION  │├───────────────────┼─────────────┼───────────┤│ production.users  │ name        │ full_hash ││ production.users  │ email       │ full_hash │└───────────────────┴─────────────┴───────────┘
After joining it with the information_schema.columns in the query contained in the statement, the mapping will look like this
mapping (sql statement output)┌───────────────────┬─────────────┬───────────┐│ TABLE_NAME_FILTER │ COLUMN_NAME │ FUNCTION  │├───────────────────┼─────────────┼───────────┤│ production.users  │ name        │ full_hash ││ production.users  │ email       │ full_hash ││ NULL              │ country     │ NULL      │└───────────────────┴─────────────┴───────────┘
N.B. we can also leave the TABLE_NAME_FILTER blank in the anonymization_mapping, this will anonymize every column with a matching name without filtering by table. This can be useful in the case where we have multiple tables with the same column names that have to be anonymized.
(rows 14:27) This part of the macro prints out the right column names and functions to be put in the select statement by mapping column names with the correct anonymization function using the output of the last step (mapping). For example, the output of the macro when following the previous example would be:
SHA256(name) as name,SHA256(email) as email,country
and the final compiled model will look like this:
select SHA256(name) as name, SHA256(email) as email, countryfrom  private_sources.production.users
Effectively anonymizing the sensitive content of our table.
2.The second macro was built to group and easily use the different anonymization transformations. This simply applies the selected transformation to a column. To showcase its use I’ve included two different methods, one generic that hashes the whole content of a cell and one that also applies a REGEX to the content of the cell to anonymized only the user part of an email address.
FROM: tompere@housinganywhere.comTO: SHA256(tompere+salt)@housinganywhere.com
It is also important to stress that whenever using a hashing function it is required to add a salt otherwise our hashes would be vulnerable to dictionary attacks.
The purpose of GDPR regulations is to improve consumer confidence in organizations that hold and process personal data, as well as standardizing and simplifying the free flow of information across the EU. In HousingAnywhere we always believed that the privacy of our users is key in establishing trust and confidence towards our platform and that is why we take so much care in designing our data infrastructure.
A final remark should be also made on the fact that Snowflake offers an out-of-the-box solution for anonymization called Dynamic Data Masking available only for enterprise accounts.
Thanks to Julian Smidek for being such a great manager :)
[1] DBT Cloud. Accessed 20 May 2021. https://cloud.getdbt.com/.
[2] DBT Discourse. ‘PII Anonymization and DBT — Modeling’, 26 April 2018. https://discourse.getdbt.com/t/pii-anonymization-and-dbt/29.
[3] Step 2. Create Snowflake Objects — Snowflake Documentation. Accessed 20 May 2021. https://docs.snowflake.com/en/user-guide/getting-started-tutorial-create-objects.html.
[4] ‘A Comprehensive Tutorial of Snowflake Privileges and Access Control — Trevor’s Code’. Accessed 20 May 2021. https://trevorscode.com/comprehensive-tutorial-of-snowflake-privileges-and-access-control/.
[5] Salt (Cryptography). In Wikipedia, 17 May 2021. https://en.wikipedia.org/w/index.php?title=Salt_(cryptography)&oldid=1023634352.
[6] Snowflake. ‘The Data Cloud | Snowflake | Enable the Most Critical Workloads’. Accessed 20 May 2021. https://www.snowflake.com/.