On Semantic Search
By: Pratik Bhavsar
Tags: Artificial Intelligence, Machine Learning, Deep Learning, Programming, Search

Solution 3: Alternate query generation

If we have a log of queries from user sessions, we can train a generative model to generate (next query | current query). The hypothesis being that all the queries in a session are similar to each other.

The logs can be like this.

Artificial intelligence Tensorflow Neural networks …

Training data

(x, y)

(Artificial intelligence, Tensorflow)

(Tensorflow, Neural networks)

Query generation can help us suggest relevant queries by understanding the buying intent of user.

Solution 4: Using word and document embeddings

Once we have the query entered by the user, instead of representing it in one-hot or TF-IDF normalised form, we can vectorise words, sentences and documents using some approaches.

Simple embedding average

Weighted embedding average by multiplying with IDF values

Sentence embedding using a model like infersent, USE(Universal sentence encoder), sentence-bert

seq2seq autoencoder

This can help us represent all the tokens in a semantic and compressed form of a vector of fixed size irrespective of the vocabulary size. This requires a one-time heavy overhead of vectorising using the model but all the searches later will be vector search in n-dimension.

The current state of the art in vector search is milvus. For approximate nearest neighbour you can use flann and annoy.

Solution 5: Contextualisation

Factors to be considered by the search engine for contextualisation/personalisation

User history — Interests from his past search and if he searched the same in past what did he click

— Interests from his past search and if he searched the same in past what did he click User geography — Searching word ‘president’ gives me ‘Ram Nath Kovind’ → the current president of India

— Searching word ‘president’ gives me ‘Ram Nath Kovind’ → the current president of India Temporal changes in information — The result of query ‘president’ will change with time

User history can be used by encoding each click as a fixed vector and generate score(document | history+query)

Geography and temporal can be taken care either by filtering or adding them as features while training the model.

Personalisation allows us to serve human-level suggestions to the user which can help us improve the conversion.

Solution 6: Learning to rank

Because of the flaws of token matching in TF-IDF scheme, we actually need a layer which can re-rank the results. The scheme has high bias for rare words and also doesn’t account for conversion possibility of an article.

If we have a log of user queries and what they clicked from the search results, we can train the model to rank documents. The data will look like this.

(x, y)

(Artificial intelligence, book 2 title)

(Tensorflow, book 1 title)

(Neural networks, book 4 title)

Next time we want to show results, we first get the top x results from a cheap process of token match through TF-IDF/BM25, mostly through Elasticsearch, and then generate a score for all pairs.

(x, y)

(query, title 1) → score 1 (query, title 2) → score 2 …

Then we sort the titles by the score and show top results.

You can find an implementation of this using BERT in my github.

BERT with NSP head

We can define the learning to rank problem as BERT NextSentencePrediction problem — aka entailment problem.

Solution 7: Ensemble

In most cases, it's advantageous to leverage both the approaches. The book mentions that using a combined score of wordvector and BM25 works best.

From ‘Deep learning for search’

Ensemble allows us to exploit best of both the approaches-hard token match + semantics.

Solution 8: Multilingual search

Approach 1

In certain cases when the application is across the geography, the language of user can be different from the document. Such a search is not possible with classical search approach as tokens of different languages cannot match. This requires the help of machine translation with deep learning.

Detect the language of user query (ex. French) Translate query to all languages for which we have the documents (French to English, German and Spanish) Search documents Translate all top-scoring documents to user’s language (English, German and Spanish to French)

Approach 2

Instead we can use a multilingual sentence encoder to represent text from any language to similar vectors. I have explained this in detail over here.

Conclusion

You might already know that last month Google pushed BERT based implementation in production for enhancing their search results. It seems that semantic search is on a rise and will become common in the industry as we get more understanding of leveraging deep learning for search 😀