What exactly is Metaflow?. A high-level look Netflix’s Python... | by Rupert Thomas | Towards Data Science

Metaflow is a Python framework for data science developed by Netflix; it was released as an open source project in December 2019. It addresses some of the challenges data scientists face around scalability and version control. A processing pipeline is constructed as a sequence of steps in a graph. Metaflow makes it easy to move from running the pipeline on a local machine to running on cloud resources (currently AWS only). Each step can be run on a separate node, with unique dependencies, and Metaflow will handle the inter-communication. Below, I break down what I think are the key features.
I’m always on the look out for ways to make life easier, particularly when working on data-driven projects. Metaflow claims to ‘it quick and easy to build and manage real-life data science projects’, so I was all ears. My first question was: “What does it actually do?”, and I had to do a bit of digging to find out. After reading the documentation and working through the tutorials, I summarise my findings here.
Metaflow makes it easy to move from running on a local machine to running on cloud resources.
The main benefit (as I see it) is that Metaflow provides a layer of abstraction over the top of computing resources. This means you can focus on code, and Metaflow will take care of how to run it on one or more machines. In documentation-speak, it provides “a unified API to the infrastructure stack”.
Metaflow provides this abstraction using a Directed Acyclic Graph (DAG) — the flow — that contains the operations to be run. Each operation in the pipeline is a step and is defined as a method inside a Python class with a decorator:
Where steps are defined in parallel (such as the two model fitting steps in the figure above), Metaflow may run them in parallel if resources are available. When running on a single machine this is one route to parallel processing, similar to the multiprocessing package in Python. The main advantage comes, however, when you want to run on cloud resources. Once you have configured your cluster, a single additional command line argument will tell Metaflow to run the code in the cloud: --with batch. Currently only Amazon Web Services is supported though, but I imagine that will change in the future.
A checkpoint is taken at the end of each step, and it is possible to resume execution at each one at a later stage to aid debugging. You can’t step through your code line-by-line though.
Version control of machine learning models is a bit of a challenge, so Metaflow addresses this too. Both code and data are hashed. Each execution of the graph is logged and results are stored, along with the key hyperparameters:
These can be easily manipulated as command line arguments:
python metaflow_parameter.py run --alpha 0.001
Metadata is stored in the file system in JSON format, and you can access variable data stored at each step. Its also easy to read back the last run:
run = Flow(flow_name).latest_successful_run
Metaflow also provides a mechanism for managing dependencies, and they can be specified at the flow level or the step level using decorators. These can specify specific a Python version or specific packages:
The conda environment flag is specified when running from the command line:
python metaflow_conda.py --environment=conda run
If you want to try out metaflow, its simple to install from the command line and there are a set of good tutorials in the documentation:
pip install metaflow
Metaflow aims for “seamless scalability”, and it looks like it does that pretty well. There is a bit of manual setup to get it working with AWS, but its reasonably well documented. If you are likely to need to scale your analytics project to the cloud and are concerned about deployment, you should probably have a look at Metaflow. I haven’t figured out yet how I would get maximum value from the versioning and fine-grained dependency management, but they look like powerful features.
Have you tried Metaflow? Feel free to leave any experiences and feedback in the comments section below.
Rupert Thomas is a technology consultant specialising in machine learning, machine vision, and data-driven products. Rupert Thomas
https://docs.metaflow.org/getting-started/tutorials
Open-Sourcing Metaflow, a Human-Centric Framework for Data Science
Learn Metaflow in 10 mins