Powering discovery

Semantic Image Retrieval
The Good, The Bad,and The Ugly

Powering discovery

Presenter

Raed Chammam

Paul-Louis NechSr. Software Engineer
Team Recommend at Algolia

Sr. Machine Learning Engineer
Team Recommend at Algolia

1. The Good 😌
Agenda2. The Bad🧐

3. The Ugly 󰷺the World is Visualand so is the Web

The World is Visual: hence, the Web too“97% of top searches are

unbranded [...] people come toPinterest to shop for genericterms and not specific brands”

Places with professional photos can
get upto 20% more bookings

Thumbnails take 82% ofthe browsing focus

https://www.airbnb.com/e/pro-photography
https://www.airbnb.com/help/article/3381
https://about.netflix.com/en/news/the-power-of-a-pictureEtsy: Search by Image

from Building Etsy’s Search by Image Feature
By Eden Dolev, Alaa Awad

Pinterest: Shop from Pins

…and you can have it too

Under the hood: using vectors & metadata together
Vectorization

RetrievalHashing

Rulesboost/bury on demand

Filteringobject-level metadata

See alg.li/ecir23-slides
9the Good󰐬 XTDeco.ro: is it a bird? Is it a plane?

XTDeco.ro 🎩🐇XTDeco.roXTDeco.ro: what does the model see?

XTDeco.roCabinetParts.com: my handle broke…

CabinetParts.com: …and I’m out of words

CabinetParts.com: allow exploration beyond words

CabinetParts.com: great variant-awarenessNooteboom Textiles

Nooteboom Textiles: power users recognize patterns…

Nooteboom Textiles: allow newcomers to deep divethe Badnoftethe Bad lyDONEThe “What you see is what you get” problem

The “What you see is what you get” problem
Nearest neighborInputBest matchnoftethe Bad ly
DONE󰎴 Your next Guitar: Andre købte også?

Your next Guitar~360€

Your next Guitarenop

what do you do when you have no user events ?

Always have something to show

Shoes? Visual scores tell you about specificity

Shoes? Knowing Specificity tells you what to show them󰏃 Shoes - La Redoute

Explore alternatives
in context

Sizes - La Redoute

Recover from‘not available’
with visual
recommendations

Sizes - La Redoute

Sort by similarity - and more

Filtered for inStock in this size
Display theoriginal itemthe UglyMatching random user review pictures

Matching random user review pictures

Matching random user review pictures

Matching random user review pictures

Matching random user review pictures

Matching random user review pictures

Matching hands

GAR

TU
OEGA
B
R
A
G,
N
Try better dataI
BAGE
Focusedimagery
Objectsegmentation

Or use the threshold! “threshold: 0.7”

Matching MUSCLES 💪Pure muscles in…

PURE MUSCLES OUT 💪What to do when an use-case is ugly?

There’s always an alternative to better serve your content!
- Metadata-based carousels 📜 🔢
- User-behavior based suggestions 🫂- Out-of-stock -> suggest alternative items- On the cart page -> offer complementary items
Image-based suggestions is only one way
to surface relevant items to your users!

Faces󰏃 WineConclusion

Conclusion● The Good: great use-cases get immediate uplift● The “Bad”: some should use visual recommendations
alongside other signals to beneﬁt● The Ugly: don’t judge a book by its cover!

Think about your use-caseThink about your use-caseWould image matching:😌 immediately delight ?
🧐 work with other featuresuser’s purchasing criteria?
󰷺 or ultimately be a distraction?

Powering discovery

Get the slides

Thank you!

Try the model