WhyLogs For Your ML Data Logging
By: Andy Dang
Tags: DevOps, Machine Learning, Data Science, Mlops, Logging

Fire up your MLOps with a scalable, lightweight, open source data logging library

Co-author: Bernease Herman

We are thrilled to announce the open-source package whylogs. It enables data logging for any ML/AI pipeline in a few lines of code. Data logging is a critical component of an ML/AI pipeline, as it provides essential insights into the health and performance of the pipeline. We hope whylogs will make it into the toolset of every data science practitioner and researcher.

We built whylogs because in our years of troubleshooting AI systems, we observed that the vast majority of problems start with the data that the models consume. Teams frequently overestimate the resiliency of machine learning algorithms and underestimate the effects of poor data quality, leading to data dependencies that can cause costly failures. Keeping an AI model healthy requires developers and data scientists to be aware of changes in the quality and consistency of their data. Our team at WhyLabs aims to address these challenges with the open-source data monitoring library whylogs.

Data monitoring, why is it so hard?

Much of the difficulty in maintaining an ML system comes from data drift. This isn’t always due to bad data from upstream; sometimes the data coming in just changes abruptly, and the system is unprepared to deal with it. While monitoring machine learning data is similar in theory to traditional DevOps monitoring, simply applying existing DevOps systems to ML data often leads to trouble. ML data can have unusual data types that DevOps tools aren’t built to handle, or, what’s worse, contain personally identifiable information that needs extra security and privacy measures many DevOps systems are unable to implement.

Some organizations have already started investing in building dedicated monitoring systems for ML pipelines, such as Netflix’s Metaflow and Uber’s Michelangelo. However, most of these systems are so specialized that implementing them requires teams to perform resource-intensive restructuring of the pipelines they monitor.

What does a good data logging solution look like?

Photo by Etienne Girardet on Unsplash

A good data logging system should combine the transparent, flexible methodology of traditional DevOps with the human-centric design of dedicated systems like Metaflow. To do this, it has to accomplish the following:

Log properties of data over time , as that data moves through the ML system

, as that data moves through the ML system Aggregate logs , so you can use distributed systems to process more data at once

, so you can use distributed systems to process more data at once Support a wide range of ML data types , including complex inputs like images and video

, including complex inputs like images and video Tag data segments with labels that the humans involved can use for deeper analysis

In a production setting, there are a few more requirements for a good data logging system:

Use a lightweight structure that can run in parallel with model training and inference

that can run in parallel with model training and inference Integrate easily with a variety of ML pipelines and architectures

with a variety of ML pipelines and architectures Enable data logging at any point in a ML model’s life cycle

Since data scientists lack the specialized tools that software developers have, they often repurpose existing DevOps approaches for ML monitoring. Doing that creates significant problems, some of which we outline below:

Log collection : DevOps solutions like Kafka and Splunk have made log collecting accessible. However, these solutions are designed for application logs, so they struggle with handling the large data volume of MLOps. There is also the risk of losing the structural information associated with ML data if log messages aren’t carefully managed.

: DevOps solutions like Kafka and Splunk have made log collecting accessible. However, these solutions are designed for application logs, so they struggle with handling the large data volume of MLOps. There is also the risk of losing the structural information associated with ML data if log messages aren’t carefully managed. Metrics collection : One popular integration is Prometheus — it’s supported by many systems, including AirFlow and Kubeflow. It can be streamed to different metrics systems for storage (DataDog, New Relic etc.) and visualized using OSS tools like Grafana. However, using Prometheus can limit your ability to monitor various important metrics (average, sum, min, max, count, etc).

: One popular integration is Prometheus — it’s supported by many systems, including AirFlow and Kubeflow. It can be streamed to different metrics systems for storage (DataDog, New Relic etc.) and visualized using OSS tools like Grafana. However, using Prometheus can limit your ability to monitor various important metrics (average, sum, min, max, count, etc). API monitoring: Technologies like Tensorflow Serving and AWS SageMaker Model Hosting make model serving simple. They allow for monitoring REST API requests and responses as a method for extracting insights. While this approach provides deep insights, it doesn’t scale well because it requires every data point to be stored.

As longtime AI practitioners, we grappled with the challenges posed by applying ML to massive amounts of data first-hand. We set out on a mission to find a monitoring approach that delivers on both the DevOps performance requirements, and the data science insight requirements.

Introducing whylogs, the MLdata logging library

We engineered whylogs by working backwards from the requirements. We focused on lightweight data collection, enterprise scalability, and flexibility designed for data science workflow. We added built-in data tagging and aggregation capabilities. Furthermore, we optimized the installation to take minutes and to seamlessly integrate with existing tools . whylogs comes with the following features out of the box: