On Being a Data Science Product Manager
By: Wafic El-Assi
Tags: Product Design, Artificial Intelligence, Machine Learning, Product Management, Data Science

On Being a Data Science Product Manager

Seven suggestions from a fellow Data Scientist

A couple of months ago, I left my job as a Data Scientist at Nulogy — A Toronto based SaaS company. During my last 6 months, the Data Science team was transitioning from the POC phase to actually building the company’s first machine learning product. As with any product team, we needed a person to help manage what is left of our data-product life cycle. That is, we needed a Data Science Product Manager (DS PM). And, because of different changes in the organization at the time, I got to temporarily wear that hat. Here are my learnings on how to succeed as a DS PM.

1. Develop an understanding of machine learning

I have a friend who recently started a job in Quebec, Canada. Quebec is the french speaking province in the Country, and as an English-only speaking Canadian, he had a hard time fitting in, up until he learned French! I think you get where I am going with this…

As a DS PM, you must be capable of identifying opportunities where machine learning can be leveraged. Also, you need to be able to have conversations with your data scientists and engineers around their day-to-day. Without developing an understanding of the building blocks of machine learning, it will be very difficult to build empathy with and to advocate for your team. Fortunately, there are countless free resources on the web to help you get up to speed (coursera, medium, youtube, etc.).

2. Understand model evaluation

Selecting the right model evaluation metric and the minimum acceptable error are two of the most challenging tasks in Machine learning. Extremely high accuracy for the task at hand may not be necessary (or even possible). On the other hand, a high error margin can be be very costly for the business. As a result, you can not be divorced from these conversations and should be proactive in learning about evaluation metrics and acceptable error margins. For example, if the ML product is substituting an already existing process, you can use the current process error rate as a base line.

Overall, the evaluation criteria will be dictated by the data available, the model(s) used, and the application at hand. For a more detailed take on evaluation metrics, give this article a read.

3. Be open-minded about experimentation

Deciding on when a machine learning product is ready to be shipped is a challenging task. Should we ship a model when the testing error is low and overfitting is non-existent? Should we test the model in a production environment with a sample of our customer base first before declaring it ready to be used by all customers? I don’t really have the answer (if someone reading this article does, please share it in the comments), and trade-offs between time, cost and accuracy are likely to be made. The reality here is that the machine learning product life cycle is dependent on a lot of experimentation, so my advice is for you include time for experimentation in product roadmapping.

4. Consider a soft launch first

Because it is very difficult to ascertain if a machine learning product is ready to be shipped, consider a soft launch first. A soft lunch can be extremely useful in surfacing blind spots such as silent features, or to collect feedback from customers.

A soft launch in a B2B environment can be very different than that in a B2C environment. In a B2B environment, you can test the ML product with a representative sample of your customer base. Here, you can establish a partnership with your customers requesting product feedback in exchange for discounted SaaS upon release. In a B2C environment on the other hand, you can conduct A/B tests to verify whether your product is generating the intended outcome.

5. You are not done when you ship

You have built your machine learning product. Great job! However, you are still far from done. In addition to your traditional post-deploy PM responsibilities, there are a few things you should be aware of.

First, you need to have a contingency plan. Make sure to have monitoring and alerting systems set-up to warn you and your team when your model’s performance starts to degrade (and yes, it will degrade). It may be ideal if you have a more general (albeit probably less accurate) back-up model or even a rule-based system ready to be deployed in place of your model-of-choice when predictions go south. Second, set clear expectations with your customers around model performance, error margins and latency. Your SLA should also reflect that. Finally, you need to identify how frequently the model needs to be retrained to maintain its SLA and how long it takes to promote the retrained model. This is especially important if you foresee the need for application downtime. Check out this resource for a more comprehensive take on ML engineering best practices.

6. Adopt an agile mindset, not a specific agile framework

Scrum or similar approaches are fairly popular in the software community. Nevertheless, not all stages of the machine learning cycle accept timed or t-shirt sized user stories. At least, in the research or POC stages, a lot of experimentation is necessary and it may be ideal at that stage to adopt a less restrictive agile framework such as Kanban.

7. You are still a PM

Aside from familiarity with machine learning concepts and the machine learning product life cycle, your role as a DS PM is not much different from a regular PM. You are still expected to build a backlog, present release plans, develop business cases, and act as an interface for your team with internal and external stakeholders.

Bringing it all together

If you were to take away one thing from this article, it is that you need to develop a level of understanding and empathy with the roles and responsibilities of the different members of your DS team. Needless to say, that is also true in a traditional software environment. If you understand what your colleagues are going through, you will start thinking about how you can help them succeed. That said, you don’t have to do it alone. If you show that you are trying to learn and that you are there for your team, they will return the favour and be there for you every step of the way. Good luck!