NLP-based Data Preprocessing Method to Improve Prediction Model Accuracy | by Sergey Burukin | Towards Data Science

Nowadays, using machine learning for peer-to-peer marketplace is very popular as it can improve the UX and increase customer loyalty. In the part 1, I described the main stages of the ML-based award recommendation system for crowdsourcing platform Arcbazar.com, where a customer initiates a designers’ competition and sets a money prize. The ML system was trained on the dataset of the completed competitions with paid awards to help a customer set the optimal award for a certain architectural project. The optimal regression algorithm was chosen after the data analysis.
As it was mentioned in the previous article, I made some simplifications of the dataset. I replaced three text description fields of the training dataset with one that had a numeric value — a total quantity of chars. So, I got the 5-field dataset instead of 7 fields.
However, the tests showed that the model accuracy could be increased.
In this post, I’m going to tell you how to upgrade the prediction model using Natural Language Processing for the dataset preprocessing.
A classical Greek philosopher Socrates said, “Speak, so that I can see you”. This aphorism means that our speech tells more about our personality than we want to say. Also, it gives a cue for a hypothesis that the project’s text description may be linked with award amount and these relationships are much stronger than a number of chars.
I decided to transform string-type data of the text descriptions to numeric values using Natural Language Processing methods with an additional aim to enrich and homogenize dataset.
The ML system upgrade was divided into three main steps:
Text Cleaning.Text-to-Number Transformation.Choosing the Optimal Regression Method.
Text Cleaning.
Text-to-Number Transformation.
Choosing the Optimal Regression Method.
For each text description field of the database, I applied text cleaning algorithms using Natural Language Toolkit and gensim libraries for Python language. However, you can use any NLP library you prefer. I transformed text to lower case, removed punctuation and English-language stopwords.
#Transform to lower caseimport stringfeatures['description'] = features['description'].str.lower()#Remove punctuationtable = str.maketrans('', '', string.punctuation)features['description'] = [features['description'][row].translate(table) for row in range(len(features['description']))]#Remove stopwordsimport nltknltk.download('stopwords')from nltk.corpus import stopwordsstop = stopwords.words('english')features['description'] = features['description'].apply(lambda x: " ".join(x for x in x.split() if x not in stop))
Then the most and the least frequent words in each field were removed. The meanings of the common (most frequent) words in the semantic analysis are very close to stopwords. They add a noise-like pattern to the texts. Furthermore, the least frequent words have a negligible meaning, and they can be filtered out. The aim of this step was to filter unique words as the most valuable. I used a graphic analysis, plotting words vs their frequency.
#Find words spreading (each word frequency)freq_d = pd.Series(‘ ‘.join(features[‘description’]).split()).value_counts()#Plot the words distributionfreq_d.plot(kind=’line’, ax=None, figsize=None, use_index=True,            title=None, grid=None, legend=False, style=None,            logx=False, logy=False, loglog=False, xticks=None,            yticks=None, xlim=None, ylim=None, rot=None,            fontsize=None, colormap=None, table=False, yerr=None,            xerr=None, label=None, secondary_y=False)
The word frequency visualization helped me filter the words with approximate frequencies between 5 and 200.
#Remove the least frequent wordsrare_d = pd.Series(' '.join(features['description']).split()).value_counts()[-17528:]rare_d = list(rare_d.index)features['description'] = features['description'].apply(lambda x: " ".join(x for x in x.split() if x not in rare_d))#Remove the most frequent wordsfreq_d = pd.Series(' '.join(features['description']).split()).value_counts()[:30]freq_d = list(freq_d.index)features['description'] = features['description'].apply(lambda x: " ".join(x for x in x.split() if x not in freq_d))
Then each text record was tokenized — a text was split into an array of words.
features['description'] = [text.split() for text in features['description']]
The combination of words and their frequency gave a vector of text, where each word is replaced with its index. Similar vectors indicate similar texts. That is how semantic searching engines work. Unlike modern search engines, here I only concentrate on a single aspect of possible similarities — on apparent semantic relatedness of their texts (words). No hyperlinks, no random-walk static ranks, just a semantic extension over the boolean keyword match.
#Create a set of text records for each fieldfrom gensim import corporadict_d = corpora.Dictionary(features['description'])
Converting a text into the bag-of-words (BoW) format, that is a list of (token_id, token_count) tuples, was done by the attribute of a class gensim.corpora.dictionary.Dictionary() — .doc2bow().
#Convert tokenized text (corpora) to vectorscorpus_d = [dict_d.doc2bow(line) for line in features['description']]
To get a real number instead of a vector, I used a norm that is a function that assigns a strictly positive length or size to each vector in a vector space.
#Transform vectors of texts to scalar values (calculating norms of vectors)from numpy import linalg as LAcorpus_d_vec_norm = [LA.norm(vec) for vec in corpus_d]#Replace text descriptions in the database with norms of vectorsfeatures[‘description’] = corpus_d_vec_norm
Finally, I got a homogeneous the 7-field dataset.
I used the best-rated machine learning method from the previous tests — Random Forest Regressor — to calculate how the model fits our new dataset. The coefficient of determination, R-squared, was a twice better (~ 0.75), than with 5-field dataset (0.37).
Initially, I had the mixed dataset: the first three fields (drop-down menu) are obviously interdependent, and the next three (descriptions) are more fluctuant.
Due to my hypothesis, descriptions fields are more philosophically (or even psychologically) close to an award value. So, I changed the ML method to the artificial neural network. By the way, this algorithm was rejected in the previous test with 5-field dataset due to its very low R-squared of 0.05.
However, Multi-layer Perceptron Regressor with just three hidden layers gave a fantastic result with R-squared of 0.999962.
import sklearnfrom sklearn.neural_network import MLPRegressormlpreg = MLPRegressor(hidden_layer_sizes=(3,), activation=’relu’,         solver=’adam’, alpha=0.001, batch_size=’auto’,         learning_rate=’adaptive’, learning_rate_init=0.01,         power_t=0.5, max_iter=1000, shuffle=True, random_state=9,         tol=0.0001, verbose=False, warm_start=False, momentum=0.9,         nesterovs_momentum=True, early_stopping=False,         validation_fraction=0.1, beta_1=0.9, beta_2=0.999,         epsilon=1e-08)
1. The obtained results show the suitability of the norm of the vector for replacing a vector of text without significant data loss.
2. Customers’ verbal description of his/her needs is much closer to a money award amount than drop-down menus, because that text may express human essence or common motive.
3. The ANN-based algorithm is better at ‘understanding’ the human nature than Random Forest, first of all, because of its structural similarity to a human brain.