From Hypothesis to Member Satisfaction: A Scientific Approachto Product ML Innovation
Swanand Joshi

Kriti Kohli

Michelle Kislak

Netflix
swanandj@netflix.com

Netflix
kritik@netflix.com

Netflix
mkislak@netflix.com

Dan Zylberglejd

Ai-Lei Sun

Netflix
dzylberglejd@netflix.com

Netflix
asun@netflix.com

ABSTRACT
Recommendation algorithms are pivotal to Netflix’s services.
They provide our members around the world with personalized
entertainment suggestions that align with their preferences at any
given moment. Delivering truly satisfying member experiences
requires highly cross-functional collaboration between productmanagers, data scientists, and engineers. Our goal is to maximize
subscription value by helping members find entertainment they
truly love so that they keep coming back to the service.
In recommendations - a product Machine Learning (ML) space we personalize the discovery process for members, guiding themto engaging & well-matched titles. Product innovation in thisdomain presents a unique challenge. As our business needs
rapidly grow and change, the research space of possible member
experiences grows exponentially. It becomes critical to identify
which of our hypotheses will lead to higher member satisfaction.
This process relies on collecting quantifiable evidence that
supports investing in a particular experience. The difficulty of thisprocess is compounded further by Netflix’s scale — ourrecommender system generates very large amounts of data, often
influenced by presentation & selection biases among many
confounding factors. Biased evidence can further lead to incorrectconclusions on initial hypotheses and potentially end in suboptimal product experiences that do not satisfy members.

We also talk about an example case study that shows how thisapproach helps in decisions & turning hypotheses into real
product experiences. We conclude with possible enhancements tothis approach.

1

Potential Discussion Points

1.1 Challenges in large scale ML productinnovation••

1.2 Answering prioritization through evidenceanalysis••

In this talk, we share a scientific approach to continuous ML
product innovation. Our approach:1.

Produces trustworthy metric analyses for given
hypotheses that are anchored in member satisfaction
using observational causal inference methods and AB
test meta-analysis,

2.

Delivers a prioritized roadmap of impactful productexperiences,3.

Enables large scale ML experiences through state-ofthe-art personalization algorithms enabled by efficientengineering infrastructure and,4.

Proves high member satisfaction as evidenced by ouronline A/B tests

As the systems scale and business grows, many
hypotheses and ideas become possible, but prioritization
becomes increasingly difficult.
ML technology, member tastes, and product offeringcontinuously evolve, necessitating nimble innovation
strategy.

We describe a set of tools such as correlational analysis,
observational causal inference, A/B test meta-analysisetc. for finding answers to which ideas have more
promise in improving member satisfaction.
We explain what presentation and selection biases are in
ML products and how our approach can help mitigate
these biases affecting our evidence.

1.3 ML engineering for innovation••

We build personalization algorithms that alignrecommendations to member satisfaction. Our previous
work [1] describes the engineering approach.
We provide ideas on creating efficient ML
infrastructure to try a hypothesis once prioritized.

1.4 Generate new learnings through A/B tests•

For brand new product experiences which have little orno observational analysis, this iterative approach
generates reliable learnings for what members value by
setting up online A/B tests, feeding these learnings into
this virtuous cycle.

1.5 Case study & opportunities•

2

We talk about some examples of this approach and how
practitioners can further maximize their cross functional
collaboration with these methods.

Relevance to the Workshop

We believe that our approach is highly transferable to any
industrial application of ML product management andengineering. It offers a solution to best utilize cross functional
collaboration resources, to approach prioritization dilemmas andto improve innovation velocity.
The talk will be a mixture of technical depth and cross functional
learnings - very much in congruence with the workshop’s theme.
Our insights will contribute to the development of best practices
in the field, helping participants in choosing the right things towork on and creating successful ML products within theirindustries.

3Bios

3.1 A short bio of the main presenter
Swanand is an applied machine learning researcher at Netflix,
focusing on developing personalization algorithms and machinelearning models to improve member satisfaction. Before joining
Netflix, he worked at Facebook, where he specialized in feedranking by removing objectionable content such as hate speech
and misinformation, and at Amazon, where he focused on natural
language processing for Amazon Alexa.

3.2 Company Portrait
At Netflix, we want to entertain the world. We give you
personalized access to best-in-class TV series, documentaries,
feature films and games. Our members control what they want towatch, when they want it, in one simple subscription. We’restreaming in more than 30 languages and 190 countries, because
great stories can come from anywhere and be loved everywhere.

REFERENCES[1] Gary Tang, Jiangwei Pan, Henry Wang, and Justin Basilico. 2023. Reward
innovation for long-term member satisfaction. In Proceedings of the 17th ACM
Conference on Recommender Systems (RecSys '23). Association for
Computing
Machinery,
New
York,
NY,
USA,
396–399.
https://doi.org/10.1145/3604915.3608873