The Art of Building ML as a Product: Key Learnings
and Strategies
Shashank Shashikant Rao
IntuitABSTRACT

Despite the promise of Artificial Intelligence (AI)and Machine Learning (ML), many organizations’efforts in the field are falling short with most only
scratching the surface of what’s possible [2]. The
pace of innovation at many organizations has fallen
short of expectations. This is in part due to a lack ofunderstanding of how to build ML as a product,
which requires a different approach than
optimizing for the last mile of performance that ML
practitioners are used to in research settings.
To build ML as a product, technicalleadership must wear multiple hats across theproduct lifecycle, including Product Management,Design, Engineering, Content, and User Experience.
This requires a shift in thinking for research teams,
who must focus on providing value to the businessrather than just optimizing for model performance.
Oftentimes, ML based mission teams aretasked with launching models and experienceswhere a baseline model or data driven approach
does not exist. There is an art to organizing teams
for moving with speed continuing to deliver value
while continuing to optimize for both business andmodel performance.
The talk will share key learnings from theauthor’s work at Intuit in progressively building
towards a larger platform goal of being an AI Driven
Expert Platform [3]. It will discuss data-driven
approaches to machine learning management suchas identifying metrics for measuring impact innascent projects such as customer engagement orrevenue generation. It also involves lessons learnt incrafting A/B tests and designing launch
experiments for new features, which can help tovalidate the effectiveness of ML models andalgorithms before they are deployed in production.
ACM Reference Format:
Shashank Shashikant Rao. 2023. The Art of Building
ML as a Product: Key Learnings and Strategies. In
Proceedings of The Workshop on Applied Machine
Learning Management at The 29th ACM SIGKDD
Conference on Knowledge Discovery and Data
Mining (KDD WAMLM ’23). ACM, New York, NY,
USA, 1 page. https://doi.org/XXXXXXX. XXXXXXX.

DISCUSSION POINTSThis talk discusses some of the principles, key
learnings and example case studies of building anML driven product. As examples, the talk will cover
relevant areas of interest to an ML practitioner
about the processes of managing applied researchfrom the industry.
●●●

Booking.com’s lessons learned inexperiment design sophistication andevaluation of models [1]Learnings from Spotify engineering cultureon the concept of decoupled releases andsmall frequent releases [4]Uber’s approach on organizing Machine
Learning Teams For Model Developer
Velocity [5]Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights forthird-party components of this work must be honored. For all other uses, contact the owner/author(s).
KDD WAMLM ’23, Aug. 6–10, 2023, Long Beach, CA.
© 2023 Copyright held by the owner/author(s). 978-1-4503-0000-0/18/06...$15.00.
https://doi.org/XXXXXXX. XXXXXXX

●Principles of Incrementality and data
driven sufficiency to add value to businessat Doordash [6]The case studies will weave through the following
discussion points:
Stakeholder Management● Influencing business direction withmachine learning.
● Building and designing products to be AIFirst.
● Stewarding business decisions with ML atthe core.
● Architectural oversight from ML teams.
Measuring business impact● Standardized metrics to evaluate theperformance and impact of ML models.
● ML in industries with long productdevelopment cycles.
● Difficulty in quantifying the benefits of ML
in terms of cost savings, revenue
generation, or other business metrics.
● Complexity of ML models and theirinteractions with other systems, making it
difficult to isolate their impact.
Importance of A/B Tests● Reducing the risk of deploying ML models
that may not perform as expected.
● Make data-driven decisions about which
ML models to deploy and how to optimize
them.
● Resolve cross team alignment and pick the“winner” using data.
Responsible AI● Ensuring the product is compliant withrelevant regulations and ethicalconsiderations.
● Moving fast without breaking things.

●

Limited understanding of the impact of ML
on customer experience and satisfaction.

RELEVANCE TO WORKSHOPThis talk is highly relevant to academic and industry
ML practitioners in understanding the nuances ofbuilding ML as a product. The goal is to shareknowledge and ideas between different
organizations on management of ML projects andmeasuring business impact.

MAIN PRESENTER BIO

Shashank is a Senior Data Science Manager atIntuit, part of the AI, Data and Analytics group. As
part of Intuit AI, Shashank is currently focused on
demand forecasting, capacity planning andoptimization for tax expert hiring, training,scheduling and operations. Shashank holds aMaster’s degree from Columbia University, NewYork specializing in Data Science. At Intuit
Shashank has worked on deploying AI solutions in
the Quickbooks suite of products to forecast
business cashflow and to identify cash flow insights
and recommendations. Previously at PayPal andCitrix, Shashank has 7+ years of experience in thefield leading high performing data scientists andmachine learning engineers in mission based teams.
Shashank has received several internal awards with
8 granted patents in the field.

COMPANY PORTRAIT

Intuit helps consumers and small businesses
prosper by delivering financial management andcompliance products and services. We also provide
specialized tax products to accounting
professionals, who are key partners that help us
serve small business customers. Our global financial
technology platform, which includes TurboTax,
Credit Karma, QuickBooks, and Mailchimp, isdesigned to help consumers and small businesses
manage their finances, save money, pay off debt anddo their taxes. For those customers who run small
businesses, we are also focused on helping themfind and keep customers, get paid faster, pay theiremployees, manage and get access to capital, andensure that their books are done right. ProSeries
and Lacerte are our leading tax preparation
offerings for professional accountants. Incorporated
in 1984 and headquartered in Mountain View,California, we sell our products and services
primarily in the United States.

https://www.intuit.com/blog/innovative-thinking/taxpayer-empowerment/paving-a-path-to-virtual-f
inancial-advice-at-scale/

REFERENCES[5] Scaling Michelangelo. Uber Blog. Retrieved June2, 2023 fromhttps://www.uber.com/blog/michelangelo-machine-learning-platform/[1] Lucas Bernardi, Themistoklis Mavridis, andPablo Estevez. 2019. 150 Successful Machine
Learning Models: 6 Lessons Learned atBooking.com. In Proceedings of the 25th ACM
SIGKDD International Conference on Knowledge
Discovery & Data Mining (KDD '19). Association for
Computing Machinery, New York, NY, USA,1743–1751.
https://doi.org/10.1145/3292500.3330744[2] Building the AI-Powered Organization. Harvard
Business Review. Retrieved June 2, 2023 fromhttps://hbr.org/2019/07/building-the-ai-poweredorganization[3] Ashok N. Srivastava. Paving a path to virtual
financial advice at scale. Intuit Blog. Retrieved June2, 2023 from

[4] Spotify engineering culture (part 1) - Spotify
Engineering : Spotify . Retrieved June 2, 2023 fromhttps://engineering.atspotify.com/2014/03/spotifyengineering-culture-part-1/[6] Alok Gupta. Organizing Machine Learning.
Doordash Engineering Blog. Retrieved June 5, 2023
fromhttps://doordash.engineering/2020/02/12/organizi
ng-machine-learning-every-flavor-welcome/