What I Learned From Attending #MLOps2020 Production and Engineering World
By: James Le
Tags: Machine Learning, Industry, Conference, Metrics, Software Development

The field of MLOps is rapidly gaining momentum among Data Scientists, Machine Learning Engineers, Machine Learning Researchers, Data Engineers, Machine Learning Product Managers, and any other roles that involve the process of designing, building, and managing ML-powered software. I have been quite interested in learning more about MLOps to get a good understanding of the current state as it is a quickly evolving field.

Two weeks ago, I attended the inaugural MLOps: Production and Engineering World, which is a two-day virtual conference organized by the Toronto Machine Learning Society that explores the best practices, methodologies, and principles of effective MLOps. In this post, I would like to share content from the talks that I attended during this conference.

The post consists of 5 parts:

ML Project Management ML Metrics ML Research ML Case Studies ML Tools

1 — Project Management

In his talk “The Do’s and Dont’s of Delivering AI Projects,” Jan Zawadzki shared his practical tips on ideating, evaluating, and scoping AI projects.

While traditional projects succeed more often than not, AI projects often fail to deliver. This is because it’s quite tricky to define what makes good AI projects. This is also the first step in the AI value stream that includes: (1) Define Project, (2) Prototype Model, (3) Productionize Model, and (4) Measure Results.

Before getting started, we need to understand where our company is on the AI Maturity Journey. More specifically, AI projects can be classified by their potential impact and risk into four groups, as seen below.

Jan Zawadzki — The Beauty of Holy Grail AI Projects (https://www.experfy.com/blog/the-beauty-of-holy-grail-ai-projects-and-how-to-find-them/)

Solid Start: These low-impact, low-risk projects should be the one to start our AI journey.

These low-impact, low-risk projects should be the one to start our AI journey. Holy Grail: These high-impact, low-risk projects fall into the sweet spot of the AI maturity curve. They are likely to succeed and can bring a high impact on your business.

These high-impact, low-risk projects fall into the sweet spot of the AI maturity curve. They are likely to succeed and can bring a high impact on your business. Calculated Bet: These high-impact, high-risk projects are the ones to target as our business becomes more mature.

These high-impact, high-risk projects are the ones to target as our business becomes more mature. No-No-Never: These low-impact, high-risk projects are those to avoid at all costs.

In the beginning, we would want to generate as many AI project ideas as possible, and then we can evaluate and scope them. To discover the ideas, Jan suggested tools like AI The Game or Design Thinking that can help us come up with Solid Starts or Calculated Bets projects.

To scope our discovered AI project ideas, Jan proposed the AI Project Canvas to examine the value proposition, data, customer, and financial aspects of the projects. The ones that are most likely to succeed have clear value definitions, available data, customer needs, and within-reach capital.

And to rank those ideas, Jan proposed the RICE method: