Finding Similar Quora Questions with BOW, TFIDF and Xgboost | by Susan Li | Towards Data Science

Quora is a question-and-answer website where questions are asked, answered, edited, and organized by its community of users in the form of opinions.
In September 2018, Quora reported hitting 300 million monthly users. With over 300 million people visit Quora every month, it’s no surprise that many people ask duplicated questions, that is, questions that have same intent. For example, questions like “How can I be a good geologist?” and “What should I do to be a great geologist?” are duplicate questions because they all have the same intent and should be answered once and once only.
Quora tries very hard to eliminate duplicate questions, but NLP is a very hard problem. With so many ways to describe one and the same meaning, just look at the examples above. Quora users have provided great help that merged similar questions like so:
In this post, we will develop a machine learning and NLP system to classify whether question pairs are duplicates or not and we start from a model that is BOW or TF-IDF with Xgboost.
BOW and TF-IDF are two of the most common methods people use in information retrieval. Generally speaking, SVMs and Naive Bayes are more common for classification problem, however, because their accuracy is dependent of the training data, Xgboost provided the best accuracy in this particular data set. XGBoost is a gradient boosting framework that has become massively popular, especially in the Kaggle community. Therefore, I decided to use this model as a baseline model, because it is simple to set up, easy to understand and has a reasonable chance of providing decent results. Our baseline model will allow us to get a quick performance benchmark. If we find that the performance it provides is not sufficient, then inspecting what the simple model is struggling with can help us choose a next approach.
The Quora duplicate questions public dataset contains over 400K pairs of Quora questions. In our experiments we split the data randomly into 70% of train examples and 30% of test examples.
import numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as plt%matplotlib inlinedf = pd.read_csv('quora_train.csv')df.dropna(axis=0, inplace=True)df.groupby("is_duplicate")['id'].count().plot.bar()
The classes are not perfectly balanced, but it is not bad, we are not going to balance them.
Before cleaning the text, we preview a few question pairs to determine how to clean them.
df.drop(['id', 'qid1', 'qid2'], axis=1, inplace=True)a = 0 for i in range(a,a+10):    print(df.question1[i])    print(df.question2[i])    print()
You may have noticed that we have off a lot work to do in terms of text cleaning. After some inspections, a few tries and ideas from https://www.kaggle.com/currie32/the-importance-of-cleaning-text, I decided to clean the text as follows:
Not to remove stop words, because words like “what”, “which” and “how” may have strong signals.
Not to stem words.
Remove punctuation.
Correct typos.
Change abbreviations to its original terms.
Remove comma between numbers.
Change special chars to words. And so on.
After cleaning text, we preview those question pairs again.
a = 0 for i in range(a,a+10):    print(df.question1[i])    print(df.question2[i])    print()
Much better!
We are well experienced in classifying single texts for some time — but the ability to accurately model the relationships between texts is relative new. Here I use Panda's concatfunction to concatenate two text objects features from question1 and question2 into one. The scipy.sparse.hstackis used to stack sparse matrices horizontally (column wise).
This following strategy (CountVectorizer) is called the Bag of Words. Documents are described by word occurrences while completely ignoring the relative position information of the words in the document. It tokenizes the documents and counts the occurrences of token and return them as a sparse matrix.
The Xgboost was used with the following parameters, selected based on the performance from the validation data:
max_depth=50n_estimators=80objective='binary:logistic'eta=0.3
The tf–idf is the product of two statistics, term frequency and inverse document frequency, it is one of the most popular term-weighting schemes today.
we apply tf-idf normalization to a sparse matrix of occurrence counts, from word level, n-gram level and character level.
A subtle point about feature engineering is that it requires knowing feature statistics that we most likely do not know in practice. In order to compute the tf-idf representation, we have to compute the inverse document frequencies based on the training data and use these statistics to scale both the training and test data. In scikit-learn, fitting the feature transformer on the training data amounts to collecting the relevant statistics. The fitted transformer can then be applied to the test data.
Our highest validation score is 0.80, not bad at all for starters!
So far our best Xgboost model is character level TF-IDF+Xgboost, the recall of duplicate questions, that is, the proportion of duplicate questions that our model is able to detect over the total amount of duplicate questions is 0.67. And this is crucial for the problem at hand, we want to detect and eliminate as many duplicate questions as possible.
With this in mind, we will develop a word2vec and Xgboost model to see whether this result can be improved, which will be the next article.
Jupyter notebook for this post can be found on Github. Happy weekend!