Planning E-commerce search relevance workMICES

June, 2024

© Doug Turnbull (http://softwaredoug.com), all opinions my own, not my employer

Obligatory Bio Slide󰗞 Hi I’m Doug(@softwaredoug everywhere)
Long-time search enthusiast... Not
yet (never?) an expert
I wrote some search books, did some open
sourceI work at Reddit
I worked at Shopify & OpenSource Connections
in search
I blog here: http://softwaredoug.com:itme:

AI Powered Searchhttps://aipoweredsearch.com/

Being published soon!

Join the community!
3Who’s this talk for?
●Talk is good for search teams collaborating across DS,ML, Eng, PM●Especially a DS trying to understand Eng and vice-versa●Anyone trying to see the future, without needing to dothe work!

The currentplanning chaosEvery quarter, so many options!
I read apaper!

󰬂 PM
󰟧 Dev

🤔
LLM synonyms

My gut* says

Q1 Plan

Launch Date

LLM Synonyms

Next Quarter!

* Because my CEO,
Director, etc likes “AI”

󰱢 DataVector searchover productimages!

Sometime later… 📉I read apaper!

󰬂 PM
󰟧 Dev
󰱢 Data

LLMs forsynonyms!

Vector searchover productimages!

We pull ideas out of a hat!

..A/B test months
later...

�Sunk cost trap
I can makethis work!

...even after failure...
We keep digging that hole to save face

(image from Amtec Photos)
https://www.flickr.com/photos/141761303@N08/36547675260/in/photostream/Opportunity cost
BOB! Comeover here,
look what
I found(image from Amtec Photos)
https://www.flickr.com/photos/141761303@N08/36547675260/in/photostream/Opportunity cost
We can’t!
Wepromised
AI…
solution!

(image from Amtec Photos)
https://www.flickr.com/photos/141761303@N08/36547675260/in/photostream/

UsualSolutions

Simple correlation studies

󰬂 PM󰟧 Dev

These results
people likein searchseem to berelated totitle match!

Query

Click-Thru Rate

Add To Cart Rate

shoes

0.09

0.05

running shoes

0.11

0.04

…

…

…

ipad

0.21

0.11

󰱢 DataCorrelation != Causation

Gap analysis

󰬂 PM󰟧 Dev
󰱢 DataWhat’s wrong
with theweakest
queries?

Query

Click-Thru Rate

Add To Cart Rate

shoes

0.09

0.05

running shoes

0.11

0.04

…

…

…

ipad

0.210.11

Beware gap analysis on noisy data
shoes

ipad

Add-to-cart
rate

󰬂 PMMaybe themetric is
too noisy atthe querylevel?

󰟧 Dev
󰱢 DataDay

We can make a judgment list for eval
QueryProductId

30 daynumclicks

30 daytotal
impressio
ns

30 dayadd tocarts

shoes

1234

34

150

2

shoes

5678

32

110

4

shoes

8989

54000

* References: Click models for web search
Rene Kriegler Haystack
AI Powered Search CH 11+12

A miracle
occurs*Judgment ListQueryProduct Id

Grade (0-1)

shoes

12340.7

shoes

5678

0.8

shoes

89890.1

Relevance of
product forquery

Judgment List… to eval search
QueryProduct Id

Grade (0-1)

shoes

1234

0.7

shoes

5678

0.8

shoes

8989

0.1

󰥫 Which search for ‘shoes’ is better?
Rank

Product Id

Grade (0-1)

Rank

Product Id

Grade (0-1)

1

5678

0.8

1

8989

0.1

28989

0.1

25678

0.8

31234

0.7

31234

0.7

Solution 1 ranking of ‘shoes’

Solution 2 ranking of ‘shoes’Now we could check an idea before launch

jupyter

📊NDCG, etc
PM gives👍 to launch��🏽🦳��🏽��🏽🦳��🏽��🏽🦳🦳��🏻��🏻💻🦳 💻��🏻��🏻💻💻��🏼��🏼💻💻��🏼��🏼💻
💻
SHIP IT!

Offline testing during…

Smarter teams offline test while building

📊
NDCG

📊
NDCG

…DS+Dev fine tune…
��🏽🦳��🏽��🏽🦳��🏽��🏽🦳🦳��🏻��🏻💻🦳 💻��🏻
��🏻💻💻
Keep going? 󰛇��🏼��🏼💻💻��🏼��🏼💻
💻

What weWANT!!

… We want to Measure BEFORE committing to project…
We’re all asking ourselves:●

How do we justify a change
before planning?

●How do we predict
outcomes?

●How do we manage
stakeholder expectations?

󰬂 PM󰟧 Dev󰱢 DataWe want to try ALL THE THINGS!

(image from Amtec Photos)
https://www.flickr.com/photos/141761303@N08/36547675260/in/photostream/

LLM Synonyms

I want to try
this paper!

I want to try
what’spopular fromour GoogleSEO(image from Amtec Photos)
https://www.flickr.com/photos/141761303@N08/36547675260/in/photostream/

with ALL THE PEOPLE!
Hello I’m fromthe Content
team, can wesuggest an
idea?

I did a little
search at mylast job, what
about thisidea?

I don’t know
search, but
I’m a DS, canI try
something?

(image from Amtec Photos)
https://www.flickr.com/photos/141761303@N08/36547675260/in/photostream/We want shallow / fast feedback not slow /accurate
I want to try
this paper!

We need to get an idea fast if
an idea is even worth trying.
NOT a super good/accurate
methodology
4 hrs, not 4 weeks(image from Amtec Photos)
https://www.flickr.com/photos/141761303@N08/36547675260/in/photostream/But there’s a problem

Our engineer describing how search actuallyworks

The search system
Index
Enrichment
Indexing
Pipeline

Solr

Pinecone
SearchAPI📊NDCG

📊NDCG

󰱢 DataML
rerankingUmm wear does
my“prioritize
SEO popularproducts”
slot in?

Index
Enrichment
Indexing
Pipeline

Solr

Pinecone
Search
API

📊
NDCG

📊
NDCG

󰱢 DataML
rerankingUmm wear does
my“prioritize
SEO popularproducts”
slot in?

Every hole got deeper
Index
Enrichment
Indexing
Pipeline

Solr

Pinecone
Search
API

📊
NDCG

📊
NDCG

ML
reranking

To try out
this paper is
a 3 monthproject w/ 4
engineers 😭😭

Stuck in a paradox

Can’t do withoutplanning��🏽🦳💻��🏻
��🏼💻Can’t plan
without buildingOvercoming theparadox

Step 1: know this is all just math
Index
Enrichment
Indexing
Pipeline

Solr

Pinecone
SearchAPI📊
NDCG

📊
NDCG

󰱢 DataML
rerankingUmm wear does
my“prioritizeSEO popularproducts”
slot in?

Ranking function
score = f(f1, f2, f3, … fn)
Features:f1 - some BM25 score of the query on somefield
f2 - if query mentions electronics, 1,otherwise 0
f3 - dimension[121] of query embedding...

Unbury the ranking function from infra
Final ranking produced by your API
score = f(f1, f2, f3, … fn)
Features:f1 - some BM25 score of the query on somefield
f2 - if query mentions electronics, 1,
otherwise 0
f3 - dimension[121] of query embedding...

How you query
ElasticsearchSome complicatedfunction in your API
Lookup in the vector

Play with a new feature
Solr/ES/??

GoldedQueries

Golden Corpus

EverylabeledresultA downsample of every
query’s labeled resultfor quick prototyping ina ‘kaggle style’Baseline
Solr/ES/??

Judgments

GoldenQueries

Golden Corpus

Everylabeledresult

A downsample of every
query’s labeled resultfor quick prototypinga ‘kaggle style’Colab notebook or something

LocalIndex

GoldenCorpus +Judgments

Prototyperanking

📈NDCG

󰟧 Dev󰱢 Data

Let’s try an example!

Colab Notebook

What happens when we add fn+1
score = f(f1, f2, f3, … fn, fn+1)
Features:…
fn+1 - that cool thing you read about insome paperAnswer these questions
1. Does fn+1 add information not already present?
2. Does fn+1 add information that helps distinguishrelevant from irrelevant?

Let’s try an example!

FeatureExploration:
Baseline |Add Phrase Search

Make your codeprototypableToo much notebook code is… hell
Not-tested, global state, etc
Index
Enrichment
Indexing
Pipeline

Pinecone

SolrSearch
APIML
rerankingEval code
Prototyperanking“I don’t like notebooks”
Joel Grus

Don’t reimplement search in notebook
Dont put all
this in yourNotebook!

Index
Enrichment
Indexing
Pipeline

Solr

Pinecone
SearchAPIML
reranking(Just create an OK-ish baseline to getquick signal)

Actual goal: just get a signal! Not be 100% accurate(but make in notebook-able)

Solr
SearchArray

Add one field...

Prototyperanking

Did DCG go up?

Wrap ranking smarts in a library
Can work w/ local(colab) or prodbackends
Can run as justa library used
to execute
searches

SearchBackends

SearchBackends

search-ranking
Library/Svc?

Prototyperanking

pip install search-ranking

And use those smarts in prod by search API
Index
Enrichment
Indexing
Pipeline

Solr

Pineconesearch-ranking
Library?
Search APIML
rerankingMake it declarative
Configurable
Strategy over
executable code:

Executable code goeshere
search-ranking
Library/svc?

{“stages”:[{“name”:“parse_entities”,“file”: “foo.txt”},{“name”: “call_search”,“params”: {“qf”: “title desc”“bq”: …
}Prototyperanking

pip install search-ranking

Make it clear to the whole teamConfigurable
Strategy over executablecode:{“stages”:[{“name”: “parse_entities”,“file”: “foo.txt”},{“name”: “call_search”,“params”: {“qf”: “title desc”“bq”: …
},{“name”: “rerank”,“params”: {“depth”: 1000
“model”: “foo”

We all see
how searchworks - it’s
not magic!

󰟧 Dev
󰱢 Data󰬂 PM

Arrange, Act, Assert - but notebooks
1. Declare system state (ie config)
2. Run the queries
3. Measure the relevance state(Reference: ARRANGE, ACT, ASSERT)

Search configs,
expected state of thesystem, etc

Arrange, Act, Assert - but notebooks
1. Declare system state (ie config)
2. Run the queries
3. Measure the relevance state

Run 1K queries

Arrange, Act, Assert - but notebooks
1. Declare system state (ie config)
2. Run the queries
3. Measure the relevance state

Run reports, compute
DCG, etc

And push measurement code to library
IndexEnrichmentIndexing
Pipeline

Solr

Pineconesearch-ranking
Library/svc?

MLreranking

SearchBench

Prototyperanking

pip install search-benchIf you have questions i mightanswer them i might not hahaSlide by ian turnbull… again. (if you havent thrownrotten fruit and stuff at my dad please do it now)

Examples of🍒🪰🍒🪰🍒🪰🍒🪰🍒🪰🍒🪰🍒🪰🍒🪰🍒🪰🍒 rotten fruit