Why we need
User Behavior Insights
MICES, Berlin
June 2024
Stavros Macrakis & Charlie Hull

© 2023, Amazon ©
Web2023,Services,AmazonInc.
Webor its
Services,affiliates.
Inc.
Allorrights
its affiliates.
reserved.
AllAmazonrights reserved.
Confidential
Amazonand Confidential andTrademark.
Trademark.

Who are we?
●
●

●
●
●

Stavros Macrakis, Senior Technical Product Manager for theProject, focusing on document and e-commerce search.
Worked on search at Lycos, FAST, GLG, Google, and AWS for almost
20 years and is passionate about search relevance.

Charlie Hull, UK Director & Marketing Director at
25 years in search as a project lead, manager, trainer, speaker, blogger
Past lives included electronic engineering & circus performer

www.opensourceconnections.com

Who are?

We’ve helped clients around the globe to get good search & AI done:
whether they are stuck with search & AI or whether they want to get theirsearch & AI to the next level.
Our focus areas:●●●

Understand & Delight Your Users
Expert Search & AI Engineering
Build Powerful People and Process

www.opensourceconnections.com

What we offer:●●●

Consulting
Training
Specialist Stafﬁng

What is OpenSearch?

Open source search and analytics suite
used by thousands of companies worldwide
Applications:●●●E-commerce search
Document search
Log analytics© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

Features:●●●

Apache 2.0 license
Fork of Elastic 7.10
AWS hosted service available

All search needs behavioral data

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

5

Search data for e-commerce● Standard today○ Aggregated metrics■ Page ﬂow■ Abandonment■ Click frequency■ Sales● But how can we tune search results for e-commerce?
○ Granular data for analysis■ What searches lead to sales?
■ What is the user journey?
■ What sequence of searches do users use?
■ Which facets are most useful?
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

Every corpus and ever user population is different
Looking for items, not documents (heavy on structured data)• E-commerce search and recommendations: looking for a thing to buy•
••••
••••
•

Real estate
Books
Plumbing parts
Electronics
Media – movies, TV, …
Availability constraints: hotel rooms, flights, fractional jets, …
Geographic constraints: ride share, repair services, maid services, …

Job search; placements
Expert search
Photos by description or by similarity; SmugMug very different from ShutterStock… and various kinds of text-heavy search• Document search and recommendations (heavy on text)• Academic documents• News• Intranet, highly heterogenous, inconsistent formats• Knowledge management• Legal and regulatory, both high-precision (relevant regulation) and high-recall (discovery)• FAQs, call centers, troubleshooting• Embedded• Company search within a financial services app• Person search within email© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andApplications
Trademark. •7

Search=Relevance ranking© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

8

E-commerce search=Relevance ranking +merchandising© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

9

User Behavior Insights (UBI)
Goal: Collect all information needed for evaluating searchat high granularity• What did the user search for?

• Were the results reasonable?
• How did they reﬁne the search – modiﬁed query, facets?
• What search or sequence of searches led to a purchase of thepink Pet Rock with Overalls?
• What did they click on? ... other actions… view details, add tobasket, checkout basket• How are actions related causally, not chronologically?
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

What should all search implementations do?
Berlin Buzzwords 2022Customer• Log search actions• Understand application data
• Evaluate quality of results• Talk to users and sponsors• Regression testing• Performance testing• Multilingual coverage• Privacy• Search specialist engineers• Search is a discipline• Logging• Quality evaluation• Incorporating multiple signals• Tuning© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

11

What should all search implementations do, but don’t?
Berlin Buzzwords 2022 gaps

Customer gaps

• Log search actions• Understand application data
• Evaluate quality of results• Talk to users and sponsors• Regression testing• Performance testing• Multilingual coverage• Privacy• Search specialist engineers• Search is a discipline• Logging• Quality evaluation• Incorporating multiple signals• Tuning© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

12

Calculating search results: search processing
•
•
•
••
•

Lexical (TF/IDF, BM25)
Semantic vector
Sparse neural ≈ Splade
Hybrid semantic, sparse, lexical
RAG summarization
Pipeline processor• Reranking (LLM or other)• Query rewriting (Querqy or other)13© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

The virtuous circle of search relevanceSearch processing
••
•
•
•
•

Lexical (TF/IDF, BM25)
Semantic vector
Sparse neural ≈ Splade
Hybrid semantic, sparse, lexical
RAG summarization
Pipeline processor• Reranking (LLM or other)• Query rewriting (Querqy or other)

Measurement and analysis
•
••••
•

End-to-end search behavior (UBI)
Online evaluation tools
A/B testing
Creation of judgment sets
Offline evaluation tools
Metrics

Search tuning
••
•
•
•

Search configurations
Regression evaluation
Manual tuning, Bayesian optimization
Semantic model fine-tuning
Learning to Rank

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

14

The virtuous circle of search relevanceSearch processing••••••

Lexical (TF/IDF, BM25)
Semantic vector
Sparse neural ≈ Splade
Hybrid semantic, sparse, lexical
RAG summarization
Pipeline processor• Reranking (LLM or other)• Query rewriting (Querqy or other)

Measurement and analysis
•
•
•
•
•
•

End-to-end search behavior (UBI)
Online evaluation tools
A/B testing
Creation of judgment sets
Offline evaluation tools
Metrics

Search tuning•••••

Search configurations
Regression evaluation
Manual tuning, Bayesian optimization
Semantic model fine-tuning
Learning to Rank

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

15

The virtuous circle of search relevanceSearch processing••••••

Lexical (TF/IDF, BM25)
Semantic vector
Sparse neural ≈ Splade
Hybrid semantic, sparse, lexical
RAG summarization
Pipeline processor• Reranking (LLM or other)• Query rewriting (Querqy or other)

Measurement and analysis•
••
•
•
•

End-to-end search behavior (UBI)
Online evaluation tools
A/B testing
Creation of judgment sets
Offline evaluation toolsMetrics

Search tuning•••••Search configurations
Regression evaluation
Manual tuning, Bayesian optimization
Semantic model fine-tuning
Learning to Rank

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

16

How to choose a retrieval technique?
▪ Lexical search▪ Semantic vector search▪ Customization, personalization▪ Neural sparse retrieval (similar to SPLADE)▪ Image▪ Hybrid search: lexical, semantic, multimodal© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

Priorities1. Standardizing the schema
2. Collecting end-to-end search behavior events
3. Analyzing and displaying© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

Principles• Full data available for analysis• All data under customer control• Extensible• Search engine and data store agnostic• Collection of PII under customer control• Secure• Near real-time© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

Search tuning with data
• Manual tuning• Bayesian/grid parameter optimization• Semantic model ﬁne-tuning• Learning to Rank
• Combining lexical and neural/vector rankings© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

Applications• Analysis of customer journeys• Dashboarding of KPIs, anomaly detection• Investigation by relevance engineers, data scientists, andmerchandisers• Collecting judgment data for offline evaluation• Debugging• Tuning of typeahead• Real-time feedback to relevance calculation• Input to machine learning models(Bayesian optimization, Personalize, Learning to Rank, ...)© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

The User Behavior Insights project• Discussion at Haystack/MICES 2022
• OpenSearch RFC 4619 (9/2022); RFC 12084 (1/2024, updated 4/2024)• Community debated names• Frequent presentations and discussion at OpenSearch community meetings• All work done in the open on github• Apache 2.0 license© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.

Why are we so excited about UBI?

www.opensourceconnections.com

Relevancy Framework
CLICKSTREAM

State of play
A/B TESTING

AUTOMATEDTESTINGOPERATIONSONLINEEVALUATIONLABOFFLINEINTEGRATIONRELEVANCE
Search Quality: A Business-Friendly
Perspective
Peter Fries - Haystack 2018www.opensourceconnections.com

Nothing good

CLICKSTREAM

RRE
A/B TESTING

AUTOMATEDTESTING

QuepidOPERATIONSONLINEEVALUATIONLABOFFLINEINTEGRATIONRELEVANCE

www.opensourceconnections.com

State of play

Ugh.. Google Analytics?
(unless you have yourown giant analyticsteam!)Ugh.. Google Analytics?
Nothing good

CLICKSTREAM

RREA/B TESTING

AUTOMATEDTESTING

QuepidOPERATIONSONLINEEVALUATIONLABOFFLINEINTEGRATIONRELEVANCE

www.opensourceconnections.com

State of play

UBI!

What makes up UBI?
1.

A shared schema fora.
b.
c.

deﬁning what the user is asking for ( Query )Deﬁning what documents/objects/records/answers the user is receiving back.
Deﬁning how we track and organize the resulting Events and make the causal connections
between the Query and the follow on actions (click through, add to cart, etc)

www.opensourceconnections.com

What makes up UBI?
1.

A shared schema fora.
b.
c.

2.

deﬁning what the user is asking for ( Query )Deﬁning what documents/objects/records/answers the user is receiving back.
Deﬁning how we track and organize the resulting Events and make the causal connections
between the Query and the follow on actions (click through, add to cart, etc)

Code for collecting speciﬁc search related events by users (often JavaScript)

www.opensourceconnections.com

What makes up UBI?
1.

A shared schema fora.
b.
c.

deﬁning what the user is asking for ( Query )Deﬁning what documents/objects/records/answers the user is receiving back.
Deﬁning how we track and organize the resulting Events and make the causal connections
between the Query and the follow on actions (click through, add to cart, etc)2.

Code for collecting speciﬁc search related events by users (often JavaScript)

3.

An implementation in OpenSearch (and Solr!) to facilitate this process

www.opensourceconnections.com

What makes up UBI?
1.

A shared schema fora.
b.
c.

deﬁning what the user is asking for ( Query )Deﬁning what documents/objects/records/answers the user is receiving back.
Deﬁning how we track and organize the resulting Events and make the causal connections
between the Query and the follow on actions (click through, add to cart, etc)2.

Code for collecting speciﬁc search related events by users (often JavaScript)

3.

An implementation in OpenSearch (and Solr!) to facilitate this process

www.opensourceconnections.com

🔍/searchUBI
ubi_queries

Embeddedclient for events

ubi_events

UBI data ﬂow
www.opensourceconnections.com

🔍

1.

Query OpenSearch/search
UBI

1b. Store query& results

ubi_queriesEmbeddedclient for events

ubi_events

UBI data ﬂow
www.opensourceconnections.com

🔍
1.
2.
3.

Result A
Result B
Result C..
..

Embeddedclient for events

1.

Query OpenSearch

2. Add query_id as part of Response Format/searchUBI

1b. Store query& results

ubi_queries

ubi_events

UBI data ﬂow
www.opensourceconnections.com

🔍
1.
2.
3.

Result A
Result B
Result C..
..

Embeddedclient for events

1.

Query OpenSearch

2. Add query_id as part of Response Format/searchUBI

1b. Store query& results

ubi_queries

3. User clicks on result 2

3b. Store theresult that wasclicked

ubi_events

UBI data ﬂow
www.opensourceconnections.com

1.

🔍
1.
2.
3.

Result A
Result B
Result C..
..

Query OpenSearch

2. Add query_id as part of Response Format/searchUBI

1b. Store query& results

ubi_queries

3. User clicks on result 2
4. Product details shown,passing query_id

Embeddedclient for events

3b. Store theresult that wasclicked &query_id

PDP
Result B

ubi_events

Add to cart

Embeddedwww.opensourceconnections.com

client for events

UBI data ﬂow

1.

🔍
1.
2.
3.

Result A
Result B
Result C..
..

Query OpenSearch

2. Add query_id as part of Response Format/searchUBI

1b. Store query& results

ubi_queries

3. User clicks on result 2
4. Product details shown,passing query_id

Embeddedclient for events

3b. Store theresult that wasclicked &query_id

PDP
Result B

ubi_events

4b. Store the Add to Cart action & query_id

Add to cart

Embeddedwww.opensourceconnections.com

client for events

UBI data ﬂow

1.

🔍
1.
2.
3.

Result A
Result B
Result C..
..

Query OpenSearchclient for events

1b. Store query& results2b. Response Format with query_id

2. Query_id &private data

3. User clicks on result 2
4. Product details shown,passing query_id

Embedded/search

PDP
Result B

Microservice
cacheing
private data
(e.g. ourmargin on aproduct)(DataPrepper?)

3b. Store theresult that wasclicked &query_id &private data

UBI
ubi_queries

ubi_events

4b. Store the Add to Cart action & query_id& private data
Add to cart

Embeddedwww.opensourceconnections.com

client for events

UBI data ﬂow

Blacklight(Solr)

QuerqyDemo UI
SMUI
Search Engine(Elasticsearch/OpenSearch)

DockerRRE(deprecated)
www.opensourceconnections.com

QuepidLet’s watch it working on Chorus Electronics…

www.opensourceconnections.com

Enabling UBI logging per query
GET _search{"ext":"ubi": {"user_query": "the wind rises" ,}},"query": {"query_string": {"query": "the wind AND (rises OR rising)"}}}www.opensourceconnections.com

We track what the user types
AND the search engine queryPassing in our own explicit query_id
GET _search{"ext":"ubi": {"query_id": "USER52-SESSION2-QUERY3","user_query": {"original_query":"the wind rises","aggregates": ....
}}},
"query": {"query_string": {"query": "the wind AND (rises OR rising)"}}}
www.opensourceconnections.com

If you don’t pass in a query_id
then UBI will create one for you.
In both cases we return it in thequery response

Event logging

Events logging is just a standard OS document! You canthus scale UBI just as you would anything else inOpenSearch

POST /ubi_events/doc{"query_id": "USER52-SESSION2-QUERY3" ,"application": "chorus","action_name": "add_to_cart","user_id": "1","event_attributes": {"object": {"object_id": "RESULT_B",....
}"position": {...},"data": {...},}}
www.opensourceconnections.com

Docswww.opensourceconnections.com

Now we know what you’rethinking about a ‘standard’schema....

www.opensourceconnections.com

I know what you are all thinking....

https://xkcd.com/927/

I know what you are all thinking....

0

1 Cooperating

https://xkcd.com/927/

The schema, docs & code
●
●
●

●
●
●
●
●

1.0 release is now available
We want this to be as widely adopted as possible in the community - every
engine, every search! Can you help?
https://github.com/opensearch-project/user-behavior-insights○ Live as of today!
○ Builds for v2.14.0 and upcoming v2.15.0 of OpenSearch
OpenSearch RFC for this work:https://github.com/opensearch-project/OpenSearch/issues/12084
Original pull request for UBI:
https://github.com/opensearch-project/OpenSearch/pull/13546
Chorus w/ UBI: https://github.com/o19s/chorus-opensearch-edition/
Online Demo: http://chorus-opensearch-edition.dev.o19s.com:4000/ andhttp://chorus-opensearch-edition.dev.o19s.com:5601

www.opensourceconnections.com

We need your input!
Just a few questions I promise!
https://forms.gle/7jdnLMJdxqY5j4qb8

Stavros Macrakis
macrakis@amazon.com

Any questions?

Eric Pugh (leading from OSC)
epugh@opensourceconnections.com
Github
OS 2.14, 2.15:
https://github.com/opensearch-project/user-behavior-insights© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential andTrademark.