How Much Can Bad Data Cost Us?
By: Nam Nguyen
Tags: Criteo, Data Quality, Data, Data Engineering, Data Governance

At Criteo, we know how critical data can be to our business. Our predictive models ingest hundreds of terabytes training data. It learns how to deliver an outstanding outcome to our clients’ advertising campaigns. Our data analysts execute ad-hoc queries or make use of reporting pipelines to answer clients’ needs.

Understanding how vital data infrastructure is, we don’t hesitate to roll up our sleeves and get our hands dirty to keep a decent architecture. We maintain our own data centers, build our in-house data pipelines, and try to keep a viable consistency across the system.

Photo by Kaleidico on Unsplash

We have dedicated teams to regulate data flowing through the whole system. The data observability and transparency enable us to trace back bad data and identify contagious databases to isolate. We create adequate data models suitable for each use case, data integrity monitoring, and inconsistent quality tracking tools.

We don’t neglect the value of data transparency, and we do not internally hide our data behind security walls. We hope to provide a documentation layer to the existed data ecosystem to ease the examination of a distinct data set.

We have some textbook procedures when it comes to handling data-related incidents. A quick response code that requires stakeholders to take actions under a limited period enables us to promptly determine the data origin. The sole bottleneck is to know “the right people to ask for”.

In the near future, we would like to focus on improving the data quality monitor to move it closer to the sources and also defining better the data ownership.

Data governance is to add a security layer

We are the SRE Core Data & Governance group composed of four small teams with a handful of data engineers. Our main products range from reporting data to data documentation. Data quality is on the list of our top concerns to maintain a healthy data system.

We started the initiative by defining a clear vision for data quality across Criteo R&D:

A definition of data quality

Data Quality ensures that data is fit for its intended uses in operations, decision making, and planning.

The current status of data quality efforts on each data producers and data consumers’ entities.

A long-term roadmap on how we SRE Core Data & Governance will attempt to propagate our Data Quality guidelines and best practices to other Criteo R&D squads.

At the time of this writing, we have almost 1000 consistency checks and more than 1500 stability checks actively running on our Core Data Models. The Core Data Models (or CDM in our colloquial language) consists of approximately 200 tables used for reporting purposes.

CDM nowadays provides Criteo data users with a stable, robust, and reliable data source for reporting. Our tables go directly to client reports, performance dashboards, or campaign management centers. One of our core challenges is to maintain high data integrity. It is critical in our job to keep those data clean and accurate.

Besides the data quality checks, we also have monitoring tools such as Tableau dashboards to visualize those checks result for daily surveillance. We also expose our works along with our data documentation tool to help end-users easily navigate through datasets.

Maintaining reliable data pipelines is not all about monitoring. As data has become an indispensable asset, big tech companies now treat data as a part of the development cycle. Think software, but for data. Deployment, debugging, observability. There are more and more promising initiatives coming from tech players across the globe. Each one has its method to combat bad data, but they share a common goal of improving user experience.

As someone who works closely with data, one of my duties is to question the data we consume and ensure the quality of the data we produce. I’ve been intrigued by how we can better supervise our data quality. I’ve come up with a framework named triple As a.k.a the answers to “What, How, and Then?”

Alert: “What is wrong with the data?”. The more we monitor our data quality, the quicker we should be updated with abnormal circumstances. An alerting system consists of multiple means of communication: email, Slack channel, JIRA tickets, etc. Via these communications, we hope to stay by one step ahead of the data-related incidents.

“What is wrong with the data?”. The more we monitor our data quality, the quicker we should be updated with abnormal circumstances. An alerting system consists of multiple means of communication: email, Slack channel, JIRA tickets, etc. Via these communications, we hope to stay by one step ahead of the data-related incidents. Action: “How to do in case of incidents?”. Either to quickly deploy a fix or manually fill a missing period. Not only the actions but the documenting also counts. Our successors depend on how we systematize our responses to bad data detection.

“How to do in case of incidents?”. Either to quickly deploy a fix or manually fill a missing period. Not only the actions but the documenting also counts. Our successors depend on how we systematize our responses to bad data detection. Anticipate: we get alerted, and we correct the issues, and then what? How do we prevent a similar event to occur in the future? That’s not a straightforward question. This requires an understanding of the root cause, and how far we can go to fix the erroneous data.