Serverless GDPR Workflow — From Design to monitoring with Lumigo
By: Jente Peeraer
Tags: Alerting, Aws Step Functions, AWS, Monitoring, AWS Lambda

Solving the “Right to be forgotten”: A step-by-step guide to AWS Step Function workflows with full monitor and troubleshoot capabilities

This blog was written by Jente Peeraer and Tom Hertogs

Intro

As an enterprise grows, it tends to acquire or develop a few (IT) subsystems containing various forms of personal data about customers. This is perfectly natural and is necessary for the normal operations of a business. But what if you want to delete the data of one or more of these users? How would you coordinate this deletion across the various systems? What happens if one of these systems fails to delete the data of a user, or better yet, how would you know that a system was unable to delete the data of one or more users?

In this blog, we will talk you through a possible way to handle the GDPR regulations of the European Union in your organisation by using a serverless application running on AWS.

We will also talk more in-depth about the difficulties a similar application encounters concerning monitoring, logging and alerting and how to tackle them.

Webinar

Now that you know what this blog post is about; Let us take a short moment of your time to tell you that there is also a webinar for this post.

During the webinar, we have given a demo of the entire workflow and have showed you in detail how Lumgio works and how it can make your monitoring a lot easier. So feel free to have a look at it here.

Outline

What is GDPR?

Before we start architecting a solution, we need to understand GDPR. For this, it’s best to have a look at the GDPR guidelines, or more specifically in our case the “Right to be forgotten”. Making a long (legal) story short, it boils down to: “Enterprises have about a month to delete user data once requested”.

How do we tackle this?

What you need is an overall controlling workflow to manage and monitor the deletion of user data across multiple systems in parallel. You don’t want to wait for one system to have deleted the user data before moving on to the next. You’ve got a deadline to maintain after all. You need to start a workflow once a user has requested his data be deleted, and you need a workflow for each subsystem to request the deletion of that user’s data and monitor the result. This overall workflow could look something like this:

This workflow could delete data in an arbitrary number of subsystems, which would each need a controlling workflow for the deletion in that specific subsystem. These underlying workflows could look like this:

We end up with an overall controlling workflow, managing the underlying workflows, which in turn manage the deletion of a user in a specific subsystem.

Now that we got our high-level overview out of the way let’s deep dive into the implementation.

Technical architecture

Now that we have a general overview of what we need and how it needs to interact, we need to decide which technologies to use, we need:

A way to control the different states/steps in the workflow and manage the transitions To send requests to a subsystem to delete data of a specific user Notify any interested party when something has gone wrong in this process.

We’ve chosen to go for a completely serverless approach by using the following services of the AWS cloud:

AWS Step Functions to manage the workflow AWS Lambda to send the requests AWS SNS to send notifications regarding the failure of a user’s deletion

You might be asking yourself why we’ve chosen these services, and not just created a server specifically for these user deletions, which would then manage the workflow and the deletion in the specific subsystems. Architecturally speaking this would be easier right? Well, yes. Purely architecturally speaking it would be easier, but the combination of the services above has many benefits:

AWS Step Functions will give you a visual representation of where exactly a specific request is at a given time. Along with some information about the various transitions, it went through. We’ll show you what it looks like later! It also gives you a visual representation of which requests are still in progress, which have completed and which failed. A specific workflow can run up to a year (at least at the point of writing this). Since our workflow has about a month to complete, this is more than enough. We don’t need to maintain any state about where a request is at a given time since it’s managed for us, which greatly simplifies what we need to do once the server crashes and the processes need to be resumed from where they left since…. well, there is no server that we manage so… nothing to do here! At least not for resuming the workflow, we still need to handle normal error cases, which can be done by using the retry mechanism built into Step Functions. Since we have a completely serverless approach, we only pay for the moments it’s actually being used. Suppose only one user has requested for his data to be deleted this year, then we only pay for a few transitions, a couple of milliseconds of processing we needed to send the request and a couple of seconds to check whether the user has been deleted. On the other hand, the server would need to run continuously and would continue to increase our billing, why pay for something you don’t use… right? Let’s look at the other side, imagine a million people requesting that their data should be deleted on the same day. While the server would be struggling a lot to comply with this demand and would quite possibly go down a few times, the serverless approach is built for scale and would easily manage this load. You could even add some queuing in between to prevent the subsystems from being overloaded. Because our workflow has been split into various Lambdas, they could be written in different languages or even be maintained by different people. Another one of the benefits of a microservice-like approach.

Now that we’ve chosen our technology stack let’s go a bit deeper into some technical details. For the subsystem, we’ve made a distinction between a synchronous and asynchronous subsystem. The synchronous system would delete the data immediately. In contrast, the asynchronous system simply registers the user deletion and would delete the user over time (possibly with some form of batch processing). Both methods require a slightly different approach regarding the monitoring of a user’s deletion. The synchronous flow is the easiest and is shown below.

The asynchronous flow is slightly more complex and requires a polling process after the delete has been sent, which takes the deadline into account. The flow is shown below.

The last addition is the overall workflow management. In this case, we’ve chosen to start everything on an API request. When someone sends a delete request with his id, we store the id, along with the timestamp of the deletion request. The flow then needs to be completed within 30 days after the original request was made.

Problems

Now we have a fully Serverless architecture managing our GDPR deletion process. Great! Now we’re done, aren’t we? Well, not exactly, while the flow seems to be okay, and most cases seem to be covered, this is only enough in our happy cases. What do we do once we receive an input we didn’t expect? When we misconfigured something? How do we know these things? On top of this, we might want to have an overview of how much activity is going through the system, the number of errors occurring at a given time frame, and if any errors occurred at any given point in time, we’d like to know wouldn’t we? These can be divided into three parts: Logging, Monitoring and Alerting.

Logging

While it’s pretty easy to set up logging using your favourite framework in any of the various programming models of AWS Lambda, consolidating this into one whole is a different story and is not an easy task.

When working with external systems it can also be very useful to see exactly what we’ve received as input at a given time from the system.

Monitoring

The most important things we’d like to know in this architecture would be:

The number of times that our functions are invoked

The number of errors that we’ve encountered

How much of the given memory we consumed

What is our function’s execution time? Did we get any time-outs?

Most of these things are easily visible in the Lambda dashboard in the AWS console, but we’d also like to have a complete overview of all the functions that we manage. To accommodate this we’d need to create a dashboard, which has to be updated every time we create a new function.

Alerting

If something unexpected happens in one of our functions, we’d like to know, or even better, we’d like to be notified. Using the services of AWS itself, we’d need a combination of different alarms and this for every function we’d like to run. While this is all possible, it’s a little cumbersome to do so every time we add a function.

Solution

To provide a solution for the problems above, which arise in most projects when the architecture becomes complex, a framework/tool for monitoring serverless applications becomes more and more a necessity. In this case, the tool we’ve looked at is Lumigo, a monitoring platform specifically designed for serverless applications running on the AWS cloud. We’ll be showing you how to integrate it into your application and how to find what you’re looking for.

Application

We have created an application, which is used to follow the rules imposed on us by the GDPR guidelines. This application will be deployed on AWS and will consist of multiple resources, like API Gateways, Lambdas, DynamoDB, Step Functions and so on.

Organisation

Let’s start by looking at our organisation. We have three departments containing data that should be removed when our deletion process starts. These domains are project, order and customer.

In order to comply with the guidelines, data needs to be removed from all these domains. We’ll have a look at how our application will be set up.

Main Workflow

The main workflow coordinates the process of removing all the user data available throughout the different domains of our organisation.

When we send a rest request to an API Gateway to remove the user data of a specific user, we invoke a Lambda, which stores the provided user id in a DynamoDB table. This DynamoDB table will produce a DynamoDB stream, which invokes another Lambda. This Lambda will kick off the workflow.

As has been mentioned before, our organisation has three domains, which all store user data, the project domain, the order domain and the customer domain. This workflow will send a request to each domain in parallel asking them to kindly remove the data of this user.

In the above image, we see a successful run, the domains successfully removed all user data they had, and all finished executing successfully. However, in practice, things won’t always go so smoothly. What happens if one of the domains is not able to delete the data in time? This flow is illustrated in the flow below.

In this case, the order domain had some problems deleting the user’s data in their systems and failed its flow. This made our main workflow fail as well. Because Step Functions considers the entire parallel step to be failed if the error isn’t properly handled, if you want to know more about Step Functions parallel steps, check the docs here.

Synchronous Workflow

The main workflow invokes three other workflows. The project domain contains very little user data and has a very easy and straightforward way to remove everything they have in a timely manner. Because of this, they can provide a synchronous API to send a removal request.

This allows us to implement a very straightforward synchronous workflow, as shown below.

In the DeleteUserData step we invoke the synchronous API of the project domain, the API returns an error response when it fails to delete the data, which the function in turn translates to a boolean, specifying whether the deletion was successful or not. In this case, the API returned an OK response, and the function translated this to true, so the CheckUserDeleted step goes to the UserDeletedSuccessfully end-stage.

However, this isn’t always the case, as can be seen in the next workflow.

In this example, the project domain wasn’t able to complete the deletion. The CheckUserDelete step will then delegate to the UserDeletionFailed step, sending an email to the admin of the project domain, telling him some manual action is required to get the user data removed.

Asynchronous Workflow

The order and customer domain contain more user data and have it scattered in many places inside their systems. For their domains, queues are being used and you have people doing manual removals of data. Because of this, a simple synchronous call to their API won’t work to request the removal.

An asynchronous workflow has been implemented to alleviate this problem.

The DeleteUserData step will send an asynchronous request to the customer and order domain APIs, which will immediately return an answer marking the request as being submitted.

If the request was submitted successfully the workflow will start polling their API to check if the user data has been removed. If something went wrong during the submission itself, which could be because of network problems for example, the workflow will immediately go to the UserDeletionFailed step.

Then the CheckUserDeleted step says the user hasn't been deleted yet and will provide this information to the IsUserDeleted step. This step will then invoke the WaitForUserDeletion step if our deadline hasn't expired yet. This step waits for a set amount of time before going back to the CheckUserDeleted step.

In the above case, the user was deleted within the allowed amount of time, and the IsUserDeleted step calls the UserDeletedSuccessfully step, ending the workflow successfully.

In the above image the user wasn’t able to be deleted in the maximum defined amount of time. The workflow will fail and an email will be sent to the admin of the required domains, urging him to get a move on with deleting the data. In other words, manual intervention is required.

Integrating Lumigo

To monitor the application, finding out where bottlenecks are, where errors happen and so on we’re going to use Lumigo. Lumigo traces the requests your application receives from source to finish.

Integrating Lumigo into your application is very straightforward, and you have a few possibilities here. We’ve chosen for the lambda layer approach, but you can also auto-instrument your function on the Lumigo site or manually wrap your function using the Lumigo tracer as well. If you want to know more about the different possibilities you can read more here. Since we’ve chosen for the lambda layer approach, we need to include the layer in each of our Lambdas, because it contains all the monitoring code and is fully abstracted away from your application code.

Layers:

- arn:aws:lambda:eu-west-1:114300393969:layer:lumigo-node-tracer:108

If you’d like more information about what the lumigo-node-tracer can do for you, have a look at Lumigo’s Github page, which contains the source code and extra documentation.

Secondly you’re going to need to provide some environment variables, which are used to instrument the layer.

Environment:

Variables:

LUMIGO_TRACER_TOKEN:!Sub '{{resolve:ssm:/${Application}/lumigo/token:${LumigoTokenVersion}}}'

LUMIGO_ORIGINAL_HANDLER: user-api-lambda.handler

LUMIGO_STEP_FUNCTION: true

The LUMIGO_TRACER_TOKEN variable contains the API key we receive from Lumigo, in this case, it's stored on the AWS SSM Parameter Store. When deployed it will be injected from there as an environment variable.

The LUMIGO_ORIGINAL_HANDLER variable contains the handler of our lambda, which is used by the Lumigo layer to trace the function execution.

The LUMIGO_STEP_FUNCTION simply tells the Lambda it's running in a step function.

Now that we’ve added Lumigo to our application, we can set up some alerts. We’ll do so by going to the Lumigo site and going to the Alerts configuration tab. The reason why you can do this now is because alerts work out of the box for most use-cases, as mentioned above.

From here we can add alerts or edit existing alerts, the view should be pretty similar in both cases. Let’s open or create an alert.

Here we can choose which Alert Type we want and for which function the alert should be configured. We can even choose only to get one alert each hour or day in case we’d like to limit the notifications we’ll be getting. When that’s done we still need to set-up where our alarms need to go, we can simply click on the link in the Delivery Channels field.

Here you can choose which integrations you want to enable and configure them. And with that done, your next error will be delivered to you using your preferred channels. For more info on alerts, check the following documentation.

Using Lumigo

We got the set-up out of the way, awesome! Now we can open the Lumigo console and see what we can do with it as a monitoring platform. The best way to see what we can do would be to use our application and see what’s happening. Let’s say we want to delete a user with the id “delete-me-now”, we simply send a delete request with the id in the path.

The request has been sent! Time to open Lumigo, let’s start with the dashboard.

The dashboard gives us an overview about the amount of invocations that we have across our landscape in a specific timeframe. We can also see which functions are most invoked, which functions had a cold start, an estimation of the cost, … . Out of the box, this is already some powerful functionality. Especially if you know that all we did so far was adding a layer and some configuration.

But we want to know more! We need more specific information about how the request we just sent went. We’ll go to the transactions view for that.

Here we can see what transactions just occurred in our system. This already gives us an idea about how long some transaction ran and whether they succeeded. We’ll click on the first one because we’d like to know more about it.

We can see now that our API request saved an item in DynamoDB triggering a Lambda, which started a Step Function, and we can even see the amount of times a given function was invoked. On the right we can also see the logs across all the Lambda functions that ran inside this specific transaction. But we need to know even more, we want to know which response the delete request for the order data received. Let’s click on the API icon.

Here we can see the request duration, response code, request and response headers, request and response body, … . This should be all the information we need to find out why something went south in the system.

Speaking of going south, remember those alerts we talked about earlier. I just got an email and a slack notification.

It seems like there’s an issue in our system. Let’s click on the link and find out what went wrong here.

The link takes us to the transaction overview of erroneous flow. Here we can see which function failed (1) and we can see the logs at the right side to see what went wrong. Clicking on (1) also gives us information about the issue in this case as shown below.

Now we already got deep into troubleshooting: this error indicates that the response from the request was undefined. Weird! Without a strong troubleshooting tool, like Lumigo, finding the root cause might have been a huge time-consumer. Fortunately, we can easily check the call and all its parameters by clicking the AWS API Gateway icon.

Immediately we find the problem: There is no response to this call because this is the wrong api URL! It looks like someone updated the domain name of the external system without checking it properly. Correcting the domain name resolves the issue, and we’re done! Debugging as fast as 3 clicks! Issue solved!

Conclusion

The GDPR guidelines and in our case the “right to be forgotten”, can have a tremendous impact on businesses and the data they have about their users. Failing to meet the deletion policy can amount to enormous fines from the European Union.

By leveraging the power of the cloud and serverless event-driven architectures, you can quickly and cost-effectively deploy a solution which can coordinate the removal of the required data throughout your domains.

To make sure the application is working as intended we integrated Lumigo as a monitoring tool, which worked for us in the background, and abstracted all monitoring, logging and alerting code away, allowing us to focus on what’s important, getting that user data out of our systems before the European Union starts to knock down our doors.

Next steps

If you’re still hungry for more reading material you can check out the next blogs!