GDPR compliance in Big Data Infrastructure
By: Salma Bakouk
Tags: Data, Compliance, Regulation, Analytics, Gdpr

What Engineering challenges does it bring?

To understand the challenges GDPR brings to your Data Infrastructure, let’s go over some of the pillars of the framework. Under GDPR, EU citizens are given a certain set of rights:

The right to be forgotten: Complete elimination of users’ data must be conducted upon their request. This represents a real engineering challenge in the era of auto-scheduled backups, non-volatile storage systems, and all-pervasive caching.

Complete elimination of users’ data must be conducted upon their request. This represents a real engineering challenge in the era of auto-scheduled backups, non-volatile storage systems, and all-pervasive caching. The right to data portability: Or the right to request. And this means that users have the right to retrieve all the information a company has collected from them in an exportable, universally readable format, a.k.a. another technical challenge to overcome.

Or the right to request. And this means that users have the right to retrieve all the information a company has collected from them in an exportable, universally readable format, a.k.a. another technical challenge to overcome. The right to object processing: While keeping the data (upon user consent and for “necessary” business operations) is allowed, additional explicit user consent is still required to process the data. Think about how you are going to exclude certain records when writing SQL code…

While keeping the data (upon user consent and for “necessary” business operations) is allowed, additional explicit user consent is still required to process the data. Think about how you are going to exclude certain records when writing SQL code… The right to rectification: This gives users the right to change their PII data in your system as they see fit. From an engineering perspective, this means your company will need to have a way of tagging and enumerating PII-related data.

This gives users the right to change their PII data in your system as they see fit. From an engineering perspective, this means your company will need to have a way of tagging and enumerating PII-related data. The right to be informed: Users need to be aware of when their data is being collected and what purposes. In case of a data breach, users need to be notified as soon as possible, and data protection authorities informed within 72 hours.

Taking all this into account and being mindful of other broader engineering challenges GDPR might bring, let’s dive into the challenges Data Engineers are facing and how to resolve them.

Personal Data

Let’s start here. A key requirement to compliance with GDPR is the ability to locate, enumerate and access all user data classified as PII. You also need to think about PII security measures (encryption and fine-grained security access for PII), but that’s a whole other complex topic.

Gone are the days where you could hoard all sorts of data and wish for the best. Clear scope for data collection, storage, and flow through all the processes within your organization is paramount to ensuring a technical foundation for GDPR compliance.

The right to be forgotten

The idea here is having the ability to remove ALL records related to a user that are distributed across a number of databases, tables, and systems should he/she make the request. Pseudonymization of data (GDPR articles 6, 25, 32, 89) could bring a solution to this, but we’ll dive into that later.

Let’s look at some technical challenges that present themselves here:

Bulk deletion from all storages, especially the places where the entries are used in aggregate metrics or have different identifiers, can be extremely hard to implement properly. Orchestrating a cascade-like deletion without assessing the impact on other related data assets and BI reports is a recipe for disaster. Data lineage here is particularly important; more on this later.

from all storages, especially the places where the entries are used in aggregate metrics or have different identifiers, can be extremely hard to implement properly. Orchestrating a cascade-like deletion without assessing the impact on other related data assets and BI reports is a recipe for disaster. Data lineage here is particularly important; more on this later. Deleting rows that were once allocated to a certain user vs. replacing PII data with a “removed user” placeholder can present some storage challenges, especially in SQL based databases.

that were once allocated to a certain user vs. replacing PII data with a “removed user” placeholder can present some storage challenges, especially in SQL based databases. Backups can quickly go from being a routine procedure to becoming an absolute engineering nightmare. You are faced with either: no backup of PII at all or the pseudonymization of the backup. If you chose pseudonymization, you need to have a proper mechanism that matches Users_IDs to PII identifiers in a way that allows you to get rid of “forgotten” Users_IDs without having to go through the backup again to delete the users’ data. In addition, you should also try to keep a separate table of forgotten user IDs that should be in a separate database with a different backup/restore process so that when you restore a backup, you omit the forgotten users.

The right to object processing

This is arguably the least challenging requirement, although preventing an automated system from processing the data it stores may seem like an arduous and almost illogical task. Few ideas here:

Adding a table of “restricted users” and filtering the outputs against it. Easy and works for most modern databases.

Adding a boolean column/field to all tables and collections containing PII is another way of solving this. Less easy and requires some careful orchestrating.

A more sophisticated solution would be to add in your admin panel as well as within the users’ settings page, a button labeled “restrict processing” when clicked should mark the profile as restricted. That should create a “wall” preventing the back-office staff from accessing the data and hence processing it.

The right to be informed

Under GDPR, when collecting user data for a particular business use case, users will frequently have to understand and agree to whatever it is your organization wants to use their data for. This can be a real nightmare if your organization, like many, doesn’t fully understand the data it has, where it is being stored, or its specific business use case once it’s been stored. Often organizations “hoard” data hoping that the use cases will define themselves later. In order to ensure GDPR compliance, companies will need to understand exactly why they are collecting data and get into the habit of tagging that Data at the time of collection.

Another difficulty here is in the definition of the scope of use. Say, for example, a user only consents to “marketing purposes,” you’re going to need to track and enforce that restriction all the way from collection to use, cf. Data lineage.

The right to rectification

This might seem like an obvious rule but isn’t always diligently followed. A user should be able to access and edit all sorts of personal Data you’ve collected about them, including PII you would’ve fetched from third parties (Salesforce, Apple login, Facebook ID, etc.). As a general rule here, all personal data should be editable through the UI.