The Data Product Manager and his Experiments
By: Shivalik Sen
Tags: Data Product Manager, Data, Product Manager, Data Science, Product

From Experiments to Products

A Data Science Product Manager should, by dint of being involved in ‘Science’, focus on conceiving, setting-up, running and analysing as many experiments as possible.

However, this is often only an ideal dream, while reality is a lot greyer. Real-world experiments are, very often, expensive and fraught with peril. They have to be extremely carefully thought out and implemented and all possible assumptions have to be detailed before starting and accounted for. If any unforeseen and irreversible interference, whether external like Covid-19 or internal like human error leading to random intervention, happens, experiments may fail — leading to a significant loss of effort, investment and time.

In the best scenario, a question has to be posed first and data has to answer it. If this leads to, or theoretically proves, a Hypothesis, experiments have to be conducted to prove or disprove it. Very often, a single, well-thought question can lead to multiple hypotheses, forming the basis for an experiment, yielding very rich insights.

And not only that, but historical data and experiment data — both have to be analysed. Basis the findings from historical data and experimental A/B tests, algorithms/models have to be built and engineered into systems.

These systems are then integrated with front-facing products (whether internal or external). Additionally, the data collection/pipelines feeding the models, the output of the models, the various system integrations, and the functioning of the end product need to be tested. Only after all of this is in place, the release can be shipped. Congratulations: It all started with a question, and now you have in your hands, an actual product!

Photo by Austin Distel on Unsplash

But that’s not all. In addition to the traditional post-deploy PM responsibilities, there are a few things that need to be factored for. There needs to be a plan to continuously test model performance and accuracy (Pro tip: all models deteriorate with time and new data, in the real world). Future iteration plans need to have this taken into account. You cannot feature pack while your model accuracy starts slipping!

There are a lot of steps to be traversed, and not all of them, necessarily, sequentially.

For example: without asking questions and seeing what the current data says, it is not advised to jump into experimentation. Ideally, experiments should give rise to more questions and more answers, some of which can be used to start building the models. These need to be necessarily sequential.

However, if the final product/feature is a must-have and there is an explicit need/ask for it and a clear idea of what it should look like — it might not be a bad idea to start thinking backward from a first version release.

A good starting point is to sketch out a lean MVP (minimum viable product) first, then work backward, treating each complicated dependency as a black box. Instead of trying to figure out the engineering of the system or the algorithms of the model it can be temporarily marked with a big black box and named MODEL inside it and move on.

This will help getting started on the non-blocking pathways and components almost right away.

Of course, this is subject to enough resources being available, (which according to us Product Managers is an absolute myth anyway!)

Photo by Slidebean on Unsplash

A typical roadmap for building a Data Product

Putting things into perspective, the timeline for building a Data product in User Retention (sample use case) would look like this:

Sprint 1 — Analysis work to answer three fundamental questions about the nature of the product users

Are they discount sensitive? Are they price-sensitive? Are power users more discount sensitive than normal users?

These are answered by individual analysis threads that can spawn more child questions and keep digging till self-evident truths are found.

Simultaneously, a Product thread to draw up a Product Doc (whatever format or process you follow) and work with designers for discount automation dashboard designs.

Sprint 2 — Experiment thread to A/B test on actual users where the insights from previous Analysis thread are being verified (considering that the results indicate that only Power Users are discount sensitive but not price sensitive). The Data Engineering team starts putting in place required datasets like the discount sensitivity of customers. Systems monitoring is also put in place and will be needed soon. Front end team starts working on the discount recommendation dashboard.

Sprint 3 — Experiment thread to continue A/B/N test on actual users to build on previous experiment results (considering that the results indicate that not all power users were discount sensitive, even though all of them were price insensitive). The Data Scientists’ insight is that the power users who are not adequately discount sensitive are the ones who have been acquired through referrals.

To better segment this, the Power Users are split by their channel of acquisition and checked for Discount Sensitivity. The hypothesis here is that Power Users from all other channels except Referrals, should be Discount Sensitive. The Data Engineering team builds endpoint APIs which the Front-end team will integrate with their dashboards. Datasets are filled with dummy data to enable testing.

Sprint 4 — Data Science thread where the experiment results are analyzed, and the insights fed into the User Discount Sensitivity model which buckets Users according to the amount of discount they should be shown. Backtests show that this model has an F1 Score of 75%, which is acceptable, and a rollout is Okay-ed.

Data Engineering and DS teams put the model in production as a service/cron job to update the datasets with the Users, their Acquisition Channels, and the ideal Discount ranges. The Discount Automation dashboard shows for each product line, each type of User (Power or not, Channel of Acquisition), and the recommended discount range. Based on this, marketing teams can see the recommendations and choose the final discount amount to show.

Sprint 5 — Here onwards, the Data Science and Data Engineering teams keep a regular check on model performance, data refresh rates, and other sanity checks.