NLP: Classification & Recommendation Project | by Alper Çakır | Towards Data Science

NLP is the science of extracting meaning and learning from text data, and It’s one of the most used algorithms in data science projects. Text data is in everywhere, in the conclusion of that, NLP has many application areas, as you can see in the chart below.
In this context, I decided to make an NLP project that covers the arxiv data. By going into this project, I aimed to classify tags of the articles and building a recommendation system via the article’s summary, title, author, and genre features.
To achieve my objectives, I extracted an arxiv dataset which includes 41000 rows and 6 columns: author, link, summary, tag, title, and year. And my dataset contains articles between 1998–2018. As can be seen in the left photo, it contains articles mostly in the computer science area.
Some of my features were in the type of dictionary, so I transformed these columns into list type to have a usable data structure.
from ast import literal_eval# convert 'stringfield' lists to usable structurefeatures = ['author', 'link', 'tag']for feature in features:    arxivData[feature] = arxivData[feature].apply(literal_eval)
After performing literal_eval, I coded three functions for the list transformation:
def get_names(x):    if isinstance(x, list):        names = [i['name'] for i in x]        #Check if more than 3 elements exist. If yes, return only first three. If no, return entire list.        if len(names) > 3:            names = names[:3]        return namesdef get_link(x):    for i in x:        return i['href']    def get_tag(x):    if isinstance(x, list):        terms = [i['term'] for i in x]        #Check if more than 5 elements exist. If yes, return only first five. If no, return entire list.        if len(terms) > 5:            terms = terms[:5]        return terms
Well, I’m ready to write a text cleaning function now. These are one of the most important parts of an NLP project to build a robust model structure. To do this, we need to remove capital letters, punctuation, and numbers:
# Data Cleaning & Preprocessing techniquesdef clean_text(text):    # remove everything except alphabets    text = re.sub("[^a-zA-Z]", " ", text)    # remove whitespaces    text = ' '.join(text.split())    text = text.lower()    return text
Now it’s time to extract features from the text data. Text files are originally just a series of words. With the aim of running machine learning algorithms, we have to convert the text files into numerical feature vectors. There’re two common ways to do so: CountVectorizer and Tfidf.
CountVectorizer: The CountVectorizer provides a simple way to both tokenize a collection of text documents and build a vocabulary of known words, but also to encode new documents using that vocabulary.
Tfidf: TF-IDF is a statistical measure that evaluates how relevant a word is to a document in a collection of documents. This is done by multiplying two metrics: how many times a word appears in a document, and the inverse document frequency of the word across a set of documents.
The difference between them is basically, one counts all the words to vectorize and the other one uses a statistical method instead.
To perform text mining on my dataset, I coded a lemmatize function firstly, and then performed vectorization.
# Lemmatization process'''Words in the third person are changed to first person and verbs in past and future tenses are changed into the present by the lemmatization process. '''lemmatizer = WordNetLemmatizer()def tokenize_and_lemmatize(text):    # tokenization to ensure that punctuation is caught as its own token    tokens = [word.lower() for sent in nltk.sent_tokenize(text) for word in nltk.word_tokenize(sent)]    filtered_tokens = []        for token in tokens:        if re.search('[a-zA-Z]', token):            filtered_tokens.append(token)    lem = [lemmatizer.lemmatize(t) for t in filtered_tokens]    return lem
With this matrix in hand, we can now build a machine learning model.
# Defining a Count Vectorizer objectcount_vec = CountVectorizer(stop_words='english', max_features=10000)# Defining a TF-IDF Vectorizertfidf_vec = TfidfVectorizer(stop_words='english', ngram_range=(1, 2), tokenizer=tokenize_and_lemmatize, max_features=10000, use_idf=True)
There are various algorithms that can be used for text classification. Well, I started by exploring these models: Logistic Regression, Naive Bayes, Linear SVC, and Random Forest. My method was, choosing the best model to optimize, after running all my models in this section. Hence, I ran all the models with their default parameters to see the results.
Here are the results of my classification models. The reason for the conflict between accuracy and f1 scores is there are multiple tags for an article.
Machine Learning models failed to predict all the tags for an article. Still, f1 score is higher than accuracy because I set the average parameter of f1 to ‘micro’.
I skipped to the optimization section following to evaluations of models. For that purpose, I used the GridSearchCV:
param = {'estimator__penalty':['l1', 'l2'], 'estimator__C':[0.001, 0.01, 1, 10]}# GridSearchCVkf=KFold(n_splits=10, shuffle=True, random_state=55)lr_grid = GridSearchCV(oneVsRest, param_grid = param, cv = kf, scoring='f1_micro', n_jobs=-1)lr_grid.fit(xtrain_tfidf, y_train)
I have chosen Logistic Regression for the Grid Search process because of the accuracy score, and as a result, I got an F1-score that has increased to %64,1. Here are some of the results of the optimized regression model:
I made two recommendation systems, first one is based on a summary and the other is based on a title, tags, and authors. To build a recommender, firstly I computed the cosine similarity on vectorized text data.
# TfIdf matrix transformation on clean_summary columntfidf_matrix = tfidf_vec.fit_transform(arxivData['clean_summary'])# Compute the cosine similaritycosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)
Following the calculation, I wrote the recommendation function:
def get_recommendations(title, similarity):    idx = indices[title]    # pairwsie similarity scores    sim_scores = list(enumerate(similarity[idx]))    # sorting    sim_scores = sorted(sim_scores, key=lambda x: x[1],  reverse=True)    sim_scores = sim_scores[1:11]    article_indices = [i[0] for i in sim_scores]    # Return the top 10 most related articles    return arxivData[['link', 'title']].iloc[article_indices]
It gets two parameters and then sorts the articles via their cosine similarity values. Finally, It returns 10 most related articles as a result.
We have learned the general approach to a basic level of NLP projects.
We have learned that there are two common ways to preprocess our text data: Count and Tfidf Vectorizer.
We saw that our classification models failed for a multi-class problem but when we evaluate f1 score as a success metric, we’re not so bad.
The most written articles are from computer science studies and the most frequently used word is ‘neural network’.
Finally, we learned how to use cosine similarity to build a recommender.
As further work, I planned to do a Flask app to make my recommender online.
Well, that’s all for now. See you soon in the next post!
Also, check out my Github profile too if you want to learn more!