Improving the Stack Overflow search algorithm using Semantic Search and NLP
By: Rohit Agrawal
Tags: Search, Machine Learning, Stackoverflow, Deep Learning, Naturallanguageprocessing

If you a person who has ever tried to write a piece of code, you are sure to come across Stack Overflow (it’s that famous). For those who live under a rock, Stack Overflow provides one of the largest QA platforms for programmers. Users post questions/doubts and their fellow peers try to provide solutions in the most helpful manner possible. The better an answer, the higher the votes it gets, which also increase a user’s reputation.

Given its popularity, it’s safe to say that there is a buttload of data on there. However, this huge amount of information also makes it difficult to search for the solution one is looking for. It’s not that big of an issue for programming veterans and other experienced professionals, because they are aware of the correct keywords required to get an appropriate answer. However, for a baby programmer, this poses a great concern. For instance, if he needs to learn ‘how to make a server’ using Python, it is quite unlikely that he would use the terms ‘Django’ or ‘Flask’ in the search box. Thus, this might intimidate the user to use the platform.

Therefore, an optimal search engine is necessary to navigate through this mess. However, as it currently stands, the search engine of Stack Overflow suffers from a few bugs of its own. Let me illustrate it by an example:

Let’s say that I’m a beginner in NodeJS and I want to run my app. So I go over to Stack Overflow and type the following: ‘Node — how to run node app.js?’. This is what I get:

However, there does exist a result for ‘Node — how to run app.js?’:

Ideally, it should have returned the result for both the queries, but it didn’t.

‘Houston, we have a problem!’

But why bother, you ask? A major source of income for Stack Overflow is through Ad Revenue. Therefore, their goal is to maximize user engagement to push more ads and thus earn more money. Due to the suboptimal performance of their search engine, a user would have difficulty getting his doubts cleared through their website and would thus decide to use a more sophisticated search engine like Google for their purposes.

The problem arises when Google suggests a resource other than Stackoverflow. For every user that leaves their website, they lose money they could potentially make.

A possible solution

Our objective is for the platform to actually understand the content of what the user is trying to search for, and then return the most similar results based on that.

Natural Language Processing (NLP) has come a long way since its inception in the 20th century. This subfield of Artificial Intelligence has proven to work very well in the past few years due to fast processors and sophisticated model architectures and thus has immense potential for solving various language comprehension tasks.

Sneak Peek into of the final result

(To view the animated version, click here)

The solution is broken into 2 subtasks:

Tag Prediction: Given the user query, we would like to predict which tags the given query best belongs to Information retrieval: Based on a user query, return an existing question which most closely resembles the user’s query

Giving you a bird-eye view, the flow of the solution is as follows:

Collecting Stackoverflow Question/Answer data from Google BigQuery

Pre-processing and Normalizing the collected data

Filter out only the most common tags (Used 500 tags)

Training Word Embeddings using Word2Vec

Training a Tag Classifier using an LSTM model (hint: you cannot use binary cross-entropy loss to train it. Read more to know why)

Using the trained word embeddings to create sentence embeddings for all questions in our database

Comparing a query to each sentence and ranking semantically similar results using a measure based on cosine distance.

Creating a web application using ReactJS and Flask to deploy the trained model

For people with flowchart fetishes, here you go: