NLP on Data Science Interview Questions & Answers Dataset | by Cheng | Towards Data Science

Recently I had a random question that is there a dataset containing a variety of data science interview questions & answers? For now, I didn’t find any so I decided to create on my own! 🥳
Hence I spent several days gathering over 300 data science interview questions & answers and finally built a dataset large enough to explore.
To be honest, in the beginning, I thought it would be really difficult to cover the majority of types of data science questions. So I decided to gather only the non-coding ones.
Surprisingly, I found that, after reading hundreds of data science questions from websites such as
Simplilearn
Springboard
Towards Data Science
Edureka
Analytics Vidhya
Other Github repos
A conclusion is that there are not as many different interview questions as I expected. After a careful data selection & categorization, I completed a NLP project with the dataset. Hopefully, you will enjoy reading it!
First, let’s see what does the dataset look like.
It has three columns:
Category
Questions
Answers
In terms of category, I manually go through each question and categorize it into either EDA, Methodology, Statistics or a model specified type. However, later in the project, I will further simplify the category into only 4 major types as EDA, Methodology, Statistics and Model.
To clean the Questions and Answers columns I used NLTK. And the cleaning steps include:
Case lowering
Replacing special characters
Word tokenizing
Removing stop words
WordNet Lemmatization
About the step of removing stop words, I didn’t use the one provided by the NLTK library because I think the word list is not long enough. So I created a stop words list on my own with the help of the following Github repo:
github.com
Basically, this repo contains a list of the 10,000 most common English words in order of frequency, as determined by n-gram frequency analysis of the Google’s Trillion Word Corpus. And I used top 500 words from it to build my list of stop words.
Also, the WordNet Lemmatizer is used to link a word back to its semantic common base form.
# WordNet Examples: #> kites ---> kite#> babies ---> baby#> dogs ---> dog#> flying ---> flying#> smiling ---> smiling#> feet ---> foot
gaurav5430.medium.com
Finally, we can integrate all above cleaning steps into a single cleanup function named as nlp_process:
In terms of data visualization, I used word clouds to demo my findings.
The idea of word cloud is to find top most frequent n-gram words inside the interview questions & answers. For each category from EDA, Methodology, Statistics and Model, I made a pair of word cloud graphs to compare. And it’s interesting to see what are the most frequent words being asked & answered.
For a bi-gram word cloud, I used the CountVectorizer first to generate a frequency dictionary for the bi-gram words and used generate_from_frequencies for the plot. You can check the detailed code and plots as following:
Finally, since we have already finished the data cleaning and vectorizing text into count of words, I think we could continue on building a data science interview questions classifier as a conclusion for this NLP project.
For now, the cleaned and re-categorized dataset looks like the one above. And we will only use the Questions column to predict the Category. And I used CountVectorizer to convert text into numerical features. (TF-IDF Vectorizer also works!)
Till this step, we have successfully transforming the text in the Questions column to word count vectors using CountVectorizer. And there is one more optional step that can help us reduce the dimensionality of the word features by performing a Chi_Square test.
Treat each category as binary (for example, the “EDA” category is 1 for the EDA questions and 0 for the others).
Perform a Chi-Square test to determine whether a feature and the binary target are independent.
Keep only the features with a certain p-value from the Chi-Square test.
I learned the above method from this amazing article:
towardsdatascience.com
By following the exact Chi_Square test feature selection steps, below is the result for each category of interview questions:
Hence now we can continue on the data modelling with the selected word features.
I selected ML models including:
Multinomial Naive Bayes Classifier
Random Forest Classifier
Decision Tree Classifier
I specifically choose the multinomial Naive Bayes classifier because it’s suitable for classification with discrete features (e.g. word counts for text classification).
And we can check the result as following:
Thanks for the reading. And I believe the purpose of gathering the data science interview questions into a dataset is to help everyone well-prepared for interviews and have a better study efficiency!
I enjoyed myself reading through every question and labelling it into different category. Also I have learned a lot during preparing the dataset and writing up this article. Hope you enjoyed this reading as well!
Besides, I have found another article about Topic Modeling with LDA that I think may be suitable for the dataset as well. It would be better if the dataset was much larger.
Anyway, thanks for your time again! 🥳
👉 Github repo