Case Study 1: Customer satisfaction prediction on Olist Brazillian Dataset | by Ashwin Michael | Towards Data Science

Businesses have always tried to keep their customer base engaged and satisfied with the services provided by them. For remaining relevant in the industry, they need to incorporate the latest technological advances into their services. More than a decade back, it was the internet which was completely new and various industries tried to leverage the capabilities of this technology that effortlessly acted as a medium of communication between various businesses and their customers. In this decade, industries have started to provide services that are catered towards each client’s individual needs. For such services, they are required to leverage the power of artificial intelligence. We shall be exploring an aspect of this technology in this case study. This case study has been divided into the following sections: -
Business problem
ML formulation of business problem
Basic data analysis
Business constraints
Performance metric
Data Description
Exploratory data analysis
Feature engineering
Machine learning
Future work
Links
References
The Olist store is an e-commerce business headquartered in Sao Paulo, Brazil. This firm acts as a single point of contact between various small businesses and the customers who wish to buy their products. Recently, they uploaded a dataset on Kaggle that contains information about 100k orders made at multiple marketplaces between 2016 to 2018. What we purchase on e-commerce websites is affected by the reviews which we read about the product posted on that website. This firm can certainly leverage these reviews to remove those products which consistently receive negative reviews. It could also advertise those items which are popular amongst the customers. The source code for this case study is available here.
We are presented with a 5-star rating system that summarizes the overall satisfaction of the customer with the product which he or she had just purchased. We can convert this aspect into a binary classification problem by treating the 4 and 5-star ratings as the positive class and the rest as the negative class.
We are given multiple tables and a schema depicting how these tables are connected. After merging all the tables, we analyze the entire dataset. It contains multiple categorical and numerical columns. After converting the given 5-star ratings into binary, we see that the positive class occupies 76% of the dataset and the remaining is populated by the negative class. This means that the dataset is heavily imbalanced. Furniture decor, beauty products and sports equipment are the most popular categories as per the sales records. It is known that customers spend around 125 Brazilian real on average for purchasing the products from this website. On analyzing the state-wise distribution of the customers, we can see that the majority of the orders had been placed by customers located in Sao Paulo. The freight cost on most of the items was reasonable. This can be a reason why many were satisfied with the product overall. The majority of the transactions took place using credit cards.
Negative reviews are less, yet important. We should see to it that a lesser number of negative reviews are classified as positive. This means that we need to minimize the number of false positives. No constraint on latency is required for completing this task.
Confusion matrices are used for getting an insight into the type of errors that the model makes. Precision is required for reducing the number of false positives and recall is needed for reducing the number of false negatives. This is why we shall use the macro F1 score as a metric.
When you check the schematic diagram explaining the database connectivity, you’ll find eight tables. The description of these tables is as follows: -
1)olist_orders_dataset: This table is connected to 4 other tables. It is used to connect all the details related to an order.2) olist_order_items_dataset: It contains the details of an item that had been purchased such as shipping date, price and so on. 3) olist_order_reviews_dataset: It contains details related to any reviews posted by the customer on a particular product that he had purchased.4) olist_products_dataset: It contains related to a product such as the ID, category name and measurements.5) olist_order_payments_dataset: The information contained in this table is related to the payment details associated with a particular order.6) olist_customers_dataset: Details the customer base information of this firm.7) olist_geolocation_dataset: It contains geographical information related to both the sellers and customers.8) olist_sellers_dataset: This table contains the information related to all the sellers who have registered with this firm.
All of these tables are connected using primary and foreign keys. We shall join all the individual CSV files to create a large table. All of these keys end with the suffix ‘id’ or ‘prefix’.
Since this dataset is based on a Brazilian e-commerce company, the area of circulation is most probably Brazil and its neighbouring countries. This is why we shall take the plot of the South American continent and use the geographical data mentioned in the geolocation dataset to visually depict the geographical locations of the orders.
As you can see on the map, the majority of the customers are located in Brazil. For further analysis, we won’t be using the geolocation dataset. This is why it won’t be added to the main dataframe.
order_items_products = pd.merge(order_items_dataset,products_dataset,on='product_id')order_items_products_sellers = pd.merge(order_items_products,sellers_dataset,on='seller_id')two_order_items_products_sellers = pd.merge(order_items_products_sellers,orders_dataset,on='order_id')two_order_items_products_sellers_customer = pd.merge(two_order_items_products_sellers,customers_dataset,on='customer_id')two_order_items_products_sellers_customer_reviews = pd.merge(two_order_items_products_sellers_customer,order_reviews_dataset,on='order_id')final_dataframe = pd.merge(two_order_items_products_sellers_customer_reviews,order_payments_dataset,on='order_id')
There is a possibility that some of the observations may be repeated. These are redundant and that is why we need to remove them. We shall be dropping those observations which have the same Order ID, Customer ID, purchase timestamp and review from the dataset. This is because a customer cannot post the same review multiple times for the same product at the same instant of time. Notice that all the ID values, which acted as the primary and foreign keys, are unique. We cannot find a pattern with such rows, and thus, we shall be dropping them.
It is not possible to impute datetime data. That’s why we should remove those rows which contain null values in any of the datetime columns. While parsing through the datetime columns, we and extracting the date information and creating two extra columns using it. The ‘purchase delivery difference’ column gives us the number of days between the time of purchase and delivery. The ‘estimated actual delivery difference’ column gives us the delay or the cut-down in the number of days required for the delivery.
intermediate_time = final_dataframe['order_delivered_customer_date'].apply(lambda x: datetime.strptime(x, "%Y-%m-%d %H:%M:%S").date()) - final_dataframe['order_purchase_timestamp'].apply(lambda x: datetime.strptime(x, "%Y-%m-%d %H:%M:%S").date())final_dataframe['purchase-delivery difference'] = intermediate_time.apply(lambda x:x.days)intermediate_time = final_dataframe['order_estimated_delivery_date'].apply(lambda x: datetime.strptime(x, "%Y-%m-%d %H:%M:%S").date()) - final_dataframe['order_delivered_customer_date'].apply(lambda x: datetime.strptime(x, "%Y-%m-%d %H:%M:%S").date())final_dataframe['estimated-actual delivery difference'] = intermediate_time.apply(lambda x:x.days)
Let us now try to impute the other types of columns. For the categorical columns, we shall use the mode of that column and for numerical columns, we shall use the median of that column for imputation. For reviews comment and title, we use the term ‘indisponível’, which is the Portuguese translation for the English term ‘unavailable’.
final_dataframe['product_category_name'].fillna(value=final_dataframe['product_category_name'].mode()[0], inplace=True)final_dataframe['product_name_lenght'].fillna(value=final_dataframe['product_name_lenght'].mode()[0], inplace=True)final_dataframe['product_description_lenght'].fillna(value=final_dataframe['product_description_lenght'].median(), inplace=True)final_dataframe['product_photos_qty'].fillna(value=final_dataframe['product_photos_qty'].mode()[0], inplace=True)final_dataframe['product_weight_g'].fillna(value=final_dataframe['product_weight_g'].mode()[0], inplace=True)final_dataframe['product_length_cm'].fillna(value=final_dataframe['product_length_cm'].mode()[0], inplace=True)final_dataframe['product_height_cm'].fillna(value=final_dataframe['product_height_cm'].mode()[0], inplace=True)final_dataframe['product_width_cm'].fillna(value=final_dataframe['product_width_cm'].mode()[0], inplace=True)final_dataframe['review_comment_message'].fillna(value='indisponível', inplace=True)
We aim to convert this case study into a binary classification task. For that, we need to create a new column that contains the labels. Values that are rated more than 3 are labelled as positive and those whose values are 3 or below are labelled as negative.
final_dataframe['review_score'] = final_dataframe['review_score'].apply(lambda x: 1 if x > 3 else 0)
On plotting the pie chart showing the labels, we see that the positive class occupies 77.60% of the entire dataset; whereas the negative class only occupies 22.40%. This indicates that the entire dataset is imbalanced.
Some columns contain only numerical data. This means that finding the basic statistics of these columns is one thing that we can try. Notice that there is a significant difference between the mean and median values for the ‘price’ and ‘product length description’ columns.
Let us create a new column called ‘price category’ This column is used to categorize the goods as expensive, cheap and affordable based on their price. We have used the first, second and third quartiles as the conditions for creating these categories.
final_dataframe['price_category'] = final_dataframe['price'].apply(lambda x:'expensive' if x>=139 else ('affordable' if x>=40 and x<139 else 'cheap'))
We are now finding the top 12 most popular product categories in terms of the frequency of purchase. The category ‘bed bath table’ is significantly more popular than most of the other categories. It is the only category whose sales have crossed 10000 units. The category in the 12th position- ‘garden tools’ only had around 3600 units sold.
Sao Paulo, also shortened to ‘SP’ is by leaps and bounds the state with the highest customer base of this firm. It might be because this company’s headquarters is set up in Sao Paulo and more customers were informed about this website due to their heavy marketing strategy around this area. More than 40000 units were sold in Sao Paulo between 2016 and 2018.
When we check the pricing per product category, we can see that the average cost of a product is the highest for the items belonging to the ‘computers’ category. The average cost is around 1150 real. The second-highest category has an average price of nearly half of the highest category.
For this plot, we considered the overall average cost of an item. This means that it is the sum of the freight charges and the original cost of the item. We can see that the ordering of the plot hasn’t changed. It’s only that the average cost of each category that has increased.
We are now finding the city which generates the highest revenue. The city named ‘Pianco’ generates is at the top.
There are only two statuses visible here. Only 6 orders have been cancelled to date. It doesn’t make sense to analyze those products which have been cancelled. Thus, we shall delete them.
final_dataframe = final_dataframe[final_dataframe['order_status'] != 'canceled']
The below-given scatter plot takes the price on the x-axis and the total time difference between delivery and purchase on the y-axis. It can be seen that as the time taken for delivery increases, the chances of being dissatisfied with the product increases significantly. The increase in the price of an item does not cause too much dissatisfaction if it is delivered on time.
We are now creating a new column called ‘purchase delivery diff per price’, which is the time difference between the purchase and delivery of a product given the price.
final_dataframe['purchase_delivery_diff_per_price'] = final_dataframe['purchase-delivery difference']/final_dataframe['price']
The scatter plot between the freight cost and item cost very much descriptive. But it can be said that even if the freight cost is high, customers were satisfied when the item cost is less.
Credit cards are the most popular mode of payment amongst customers. They are much more popular than the other alternatives. The second in line, Boletos, are a type of voucher that can only be found in Brazil. From this plot, we could see that the bad reviews given to a product were most probably not because of a payment related issue.
Now it’s time to drop all the columns which aren’t necessary for the machine learning task. All the datetime columns should be removed.
final_dataframe.drop(['shipping_limit_date','order_purchase_timestamp','order_approved_at','order_delivered_carrier_date','order_delivered_customer_date','order_estimated_delivery_date','customer_id'], axis=1, inplace=True)
We shall now create a new variable called ‘labels’ which contain the edited class polarity. This column should be removed from the original data frame.
labels = final_dataframe['review_score']final_dataframe.drop('review_score', axis=1, inplace=True)
The ‘review availability’ column indicates whether the review of a particular item is available or not. Including this column is better than including a bunch of zeros in the ‘review comment message’ because it avoids sparsity. The problem with sparsity is that it can severely reduce the performance of any model.
final_dataframe['review_availability'] = final_dataframe['review_comment_message'].apply(lambda x: 1 if x != 'indisponível' else 0)
For machine learning, we need to define a train set for training the data and a test set for predicting. Both these sets contain both the classes in equal proportion. Since the dataset is imbalanced, there is a chance that the distribution of these labels could heavily affect the performance of any machine learning model. So, we need to see to it that both the test and train sets contain positive and negative labels in the same proportion. And for reproducibility of the results, we need to add a seed value.
X_train, X_test, y_train, y_test = train_test_split(final_dataframe, labels, stratify=labels, test_size=0.2, random_state=0)
Some columns contain multiple categories. The encoding technique which we normally think about is either one-hot encoding or ordinal encoding. The problem of sparsity arises when we use the one-hot encoding technique because there are too many categories. Ordinal encoding cannot be used unless ordinality is present in the categories. One way to tackle this situation is by using response coding. In this technique, we add the probability of occurrence of all the categories with each label. This means that each category will have two probability values- one for the positive class and another for the negative class. Overall, two distinct columns are being created for each categorical column.
def train_response(frame):  f1 = frame[frame.iloc[:,1] == 0]  f2 = frame[frame.iloc[:,1] == 1]  global dict_frame, dict_f1, dict_f2  dict_frame = dict(frame.iloc[:,0].value_counts())  dict_f1 = dict(f1.iloc[:,0].value_counts())  dict_f2 = dict(f2.iloc[:,0].value_counts())  state_0, state_1 = [],[],  for i in range(len(frame)):    if frame.iloc[:,1][i] == 0:      state_0.append(dict_f1.get(frame.iloc[:,0][i],0) / dict_frame[frame.iloc[:,0][i]])      state_1.append(float(1-state_0[-1]))    else:      state_1.append(dict_f2.get(frame.iloc[:,0][i],0) / dict_frame[frame.iloc[:,0][i]])      state_0.append(float(1-state_1[-1]))   df3 = pd.DataFrame({'State_0':state_0, 'State_1':state_1})  return df3.to_numpy()def test_response(test):  t_state_0, t_state_1 = [],[]  for i in range(len(test)):    if dict_frame.get(test[i]):     t_state_0.append(dict_f1.get(test[i],0) / dict_frame.get(test[i]))     t_state_1.append(dict_f2.get(test[i],0) / dict_frame.get(test[i]))    else:      t_state_0.append(0.5)      t_state_1.append(0.5)  df4 = pd.DataFrame({'State_0':t_state_0, 'State_1':t_state_1})  return df4.to_numpy()def test_response(test):  t_state_0, t_state_1 = [],[]  for i in range(len(test)):    if dict_frame.get(test[i]):    t_state_0.append(dict_f1.get(test[i],0)/dict_frame.get(test[i]))      t_state_1.append(dict_f2.get(test[i],0)/dict_frame.get(test[i]))    else:      t_state_0.append(0.5)      t_state_1.append(0.5)  df4 = pd.DataFrame({'State_0':t_state_0, 'State_1':t_state_1})  return df4.to_numpy()
We shall encode the categorical features which contain fewer categories using either ordinal or one hot encoding and encode them using response coding if there are too many categories. There is no need to encode the numerical features. Note that we aren’t considering all the columns present in the data frame. Only those are encoded which we deem as necessary. The encoding pattern per category is mentioned below: -
Next, we shall look at text data. One of the best ways to handle text data is to convert them into word embeddings. Word embeddings are better than TF-IDF vectors and Bag-of-words because they carry the semantic meaning of words. We shall be using FastText as the word embedding technique in this case study. It would’ve been possible to transform these words into TF-IDF vectors and later reduce the dimensionality using techniques such as Latent semantic analysis. This approach is faster, but the problem is that the dense vectors which we derive using this technique do to contain the semantic representation of these words. This approach would then reduce the accuracy of our models. Before vectorizing the text, we need to clean it. First, we shall remove the stop words and clean the remaining using regex.
sp = spacy.load('pt')all_stopwords = sp.Defaults.stop_wordsdef process_texts(texts):processed_text = []    dates = '^([0]?[1-9]|[1|2][0-9]|[3][0|1])[./-]([0]?[1-9]|[1][0-2])[./-]([0-9]{4}|[0-9]{2})$'        for text in texts:        text = re.sub(r'\r\n|\r|\n', ' ', text)         text = re.sub(r'^https?:\/\/.*[\r\n]*', ' ', text)         text = re.sub(dates, ' ', text)         text = re.sub('[ \t]+$', '', text)        text = re.sub('\W', ' ', text)        text = re.sub('[0-9]+', ' ', text)        text = re.sub('\s+', ' ', text)        text = ' '.join(e for e in text.split() if e.lower() not in all_stopwords)         processed_text.append(text.lower().strip())            return processed_text
We still haven’t brought all the numerical features into the same scale. For this, we shall be using standardization. We are using standardization instead of normalization because as seen in the description of the basic statistics, there are many outliers present in this dataset.
strn = StandardScaler()strn.fit(X_train[['price','freight_value','product_photos_qty','product_weight_g', 'product_length_cm',       'product_height_cm', 'product_width_cm', 'payment_value','purchase-delivery difference','estimated-actual delivery difference','purchase_delivery_diff_per_price']])X_train_strn = strn.transform(X_train[['price','freight_value','product_photos_qty','product_weight_g', 'product_length_cm',       'product_height_cm', 'product_width_cm', 'payment_value','purchase-delivery difference','estimated-actual delivery difference','purchase_delivery_diff_per_price']])X_test_strn = strn.transform(X_test[['price','freight_value','product_photos_qty','product_weight_g', 'product_length_cm',       'product_height_cm', 'product_width_cm', 'payment_value','purchase-delivery difference','estimated-actual delivery difference','purchase_delivery_diff_per_price']])
It is now time to concatenate all the necessary features together and check the shape of the final dataframe. It can be seen that there are 332 features overall.
We shall try to reduce this dimensionality using two techniques. One is by finding the hard threshold using singular value decomposition and the other is by using autoencoders. Singular value decomposition is a widely used matrix factorization technique that can be applied to matrices of any dimension. It converts the given matrix into a set of three matrices, named as U, Σ and V* respectively. Note that the symbol ‘*’ is used to indicate the transpose operator. The singular values of any matrix A of dimensionality MxN are the square roots of the eigenvalues of the square matrix A*A of dimensionality NxN. Here, Σ is a diagonal matrix that contains the singular values as its elements along the diagonal. The rows of U and the columns V* matrices which correspond to the diagonal elements of Σ are known as the left and right singular vectors. Remember that we are taking the transpose of the matrix V by default. Let us explain singular values using a geometric intuition. As we already know, a matrix of dimensionality MxN is a linear transformation from the real space RN into the real space RM, mathematically notated as RN→ RM. SVD is a way to decompose this direct transformation into a set of three transformations, which are- first rotation, scaling and second rotation. Now consider a unit circle with two vectors given in yellow and magenta colours. Here we are considering a transformation from R2 to R2. So, both M and N are equal to 2 in this example. After transformation, this circle changes into an ellipse which has been rotated. This overall transformation can be broken into 3 steps as mentioned in the diagram. After the first rotation(applying V*), the circle rotates through an angle. Then the Σ matrix scales the axis of this figure. The scaling of each axis is proportional to the magnitude of the singular vectors. Note that our two vectors would change in magnitude. After the second rotation(applying U), the ellipse gets rotated by an angle.
Rank reduction is a technique by which high-dimensional noisy data can be transformed into low-dimensional cleaner data. The rank defines a threshold value that we could consider for truncating the U matrix which we had obtained using SVD. Researchers used to take a heuristic approach for finding an optimal rank. The problem with the approach was that they were unsure whether the truncated matrix obtained using this rank contained sufficient information to reconstruct the clean image. Another approach would be to use trial-and-error, which is time-consuming. In a recent paper, a technique named optimal singular valued hard threshold was mentioned which laid the theoretical foundation of finding the optimal threshold value for truncation. We are required to find a threshold value τ*, which is the optimal threshold. The ‘*’ symbol used in this formula does not indicate transpose. The formula depends on whether the value of noise added to the image is known and whether the matrix is square or not. This noise is assumed to have a mean value of 0 and a standard deviation of 1. It is very unlikely that the matrix is square and the noise is known. But if these two conditions are fulfilled, the formula is
Here, ’n’ is the dimensionality of the square matrix and ‘σ’ is the noise. Under most circumstances, we don’t know the amount of noise that has been added. The formula is
Here, β=n/m and ymed is the median of all the singular values. Note that the value of ω(β) which we have used here is just an approximation. Their actual value could also be calculated, but it would require further processing. Since we are using this rank reduction technique in SVD only for dimensionality reduction purposes and not for image denoising, finding the exact value of ω(β) isn’t important.
X_final_sparse = scipy.sparse.csr_matrix(np.vstack((X_train_final,X_test_final)))U, sigma, VT = randomized_svd(X_final_sparse, n_components=min(X_final_sparse.shape[0],X_final_sparse.shape[1])-1,n_iter=5,random_state=45)beta = min(X_final_sparse.shape) / max(X_final_sparse.shape)omega_approx = 0.56*beta**3 - 0.95*beta**2 + 1.82*beta + 1.43tau = np.median(sigma)*omega_approxk = np.max(np.where(sigma>tau))+1
It can be seen that the newly formed matrix has 116 features, which preserves most of the variance. After this, we split the truncated matrix into train and test sets.
It is possible to reduce the dimensionality using neural networks. We shall be using autoencoders, a type of neural networks to encode the data efficiently. Note that autoencoders fall under unsupervised learning. In the autoencoder architecture, the size of the input and output layers are the same. The size of the hidden layers will always be lesser than the size of the external layers. The size of each consecutive hidden layer keeps on reducing till it reaches the bottleneck layer; beyond which the size keeps on increasing. Note that the architecture is symmetrical. The part which compresses the data is known as the encoder and the part which decompresses it is known as the decoder. The encoder extends from the input towards the bottleneck layer and from there on the decoder extends till the end. The bottleneck layer should have the dimensionality which we require for our data to be encoded into. An autoencoder aims to minimize the reconstruction error. The input gets compressed at the bottleneck layer and then gets decompressed at the output layer. The training converges after the error between the input and output gets minimized.
In our architecture, we are using the dimensionality of 116 at the bottleneck layer. Only two hidden layers are being used on each side of the network. After the training converges, we need to drop the decoder part and use the encoder part to transform both our training and test sets into the required dimensionality.
Now that we have created two versions of the data using two different dimensionality reduction techniques, we shall try to model the data using various machine learning and deep learning algorithms and then try to compare their performance using different metrics. We plan to apply five different algorithms to each of them. These algorithms are- K-nearest neighbours(KNN), logistic regression, random forests, xgboost and multi-layered perceptrons(MLP). We shall apply each of them on one form of data and then all of them on the other form.
The hyperparameter used in the case of KNN is the count of the nearest neighbours(K) whose values are odd numbers ranging from 1 to 9. The performance is said to be the best when the value of K is 3. The validation macro F1 score is 0.8587 and it isn’t great. By checking the test confusion matrices, we can see that the count of the false positive points is greater than the count of the false negative points. This isn’t a good classifier.
The hyperparameters used in the case of logistic regression is the inverse regularization constant(C) whose values are the powers of 10 that range from 1e-7 to 1e-3. This model performs the best when the value of C is 1e-5. The validation macro F1 score is 0.9003 and it is good. Unfortunately, we can see that the count of the false positive points is greater than the count of the true negative points when we plot the confusion matrices of the test set. This isn’t a good classifier.
The hyperparameters used in random forest is the count of the base estimators, which in this case are decision trees. Six values are chosen between 50 and 210 as the number of estimators, where the limits are inclusive in the set. There is a difference of 30 between any two consecutive values. The validation macro F1 score is 0.9161 and it is great. The count of true negatives is greater than the count of false positives. This is a good classifier.
The hyperparameters used in xgboost is the count of the base estimators, which also in this case are decision trees. Six values are chosen between 50 and 210 as the number of estimators, where the limits are inclusive in the set as mentioned above. The validation macro F1 score is 0.9191 and it is great. The count of true negatives is greater than the count of false positives. This is a good classifier.
Now we shall use multi-layered perceptrons for classification. The optimizer used in this case is Adam and the learning rate is 1e-3. The validation macro F1 score is 0.9074 and it is good. The count of true negatives is greater than the count of false positives in this case. This is a good classifier.
From what we’ve seen till now, random forest, Xgboost and mlp can properly classify the points. Xgboost performs the best amongst them.
The hyperparameter used in the case of KNN is the count of the nearest neighbours(K) whose values are odd numbers ranging from 1 to 9. The performance is said to be the best when the value of K is 3. The validation macro F1 score is 0.8564 and it isn’t great. By checking the test confusion matrices, we can see that the count of the false positive points is greater than the count of the false negative points. This isn’t a good classifier.
The hyperparameters used in the case of logistic regression is the inverse regularization constant(C) whose values are the powers of 10 that range from 1e-7 to 1e-3. This model performs the best when the value of C is 1e-2. The validation macro F1 score is 0.8874 and it is okay. We can see that the count of the false positive points is greater than the count of the true negative points when we plot the confusion matrices of the test set. This is a bad classifier.
The hyperparameters used in random forest is the count of the base estimators, which in this case are decision trees. Six values are chosen between 50 and 210 as the number of estimators, where the limits are inclusive in the set. There is a difference of 30 between any two consecutive values. The validation macro F1 score is 0.8895 and it is good. Unfortunately, the count of true negatives is lesser than the count of false positives. This is a bad classifier.
The hyperparameters used in xgboost is the count of the base estimators, which also in this case are decision trees. Six values are chosen between 50 and 210 as the number of estimators, where the limits are inclusive in the set as mentioned above. The validation macro F1 score is 0.8867 and it is okay. Unfortunately, the count of true negatives is lesser than the count of false positives. This is a bad classifier.
Now we shall use multi-layered perceptron for classification. The optimizer used in this case is Adam and the learning rate is 1e-3. The validation macro F1 score is 0.8924 and it is okay. The count of true negatives is lesser than the count of false positives in this case. This is a bad classifier.
From what we have here, it is observable that we cannot design a good classifier using the dataset compressed using autoencoders.
We still haven’t tried feeding the tokenized reviews directly as input to the models. For this to happen, we are required to operate on the entire dataset rather than the truncated or compressed version of the same. For this purpose, we shall try two deep learning models.
In the initial model, we shall try using Long short term memory(LSTM) only. We are creating two different inputs for this model. The first input is for numerical data and the other is for text data. An embedding layer converts the tokenized input into a fixed-length vector. We are training the embedding layer using FastText. Thus, the tokenized and padded text is converted into word embeddings using this layer. The next layer on this path is LSTM. This layer is capable of learning the order dependence when given a sequence. The next layer is directly fed to the dense layers. Both these inputs later get combined and there is a sigmoid activation function at the output. The validation macro F1 score is 0.9211 and it is great. The count of true negatives is greater than the count of false positives. This is a good classifier.
In the next model, we are planning to use 1-dimensional convolutional neural networks along with LSTM. Here we are using two pairs of 1-D CNN and LSTM layers consecutively before combining it with the numerical data. Here also, we are using sigmoid as the activation function at the end. The validation macro F1 score is 0.9010 and it is good. The count of true negatives is greater than the count of false positives. This is a good classifier.
When we check the summarized tables, we can see that both dual input LSTM and xgboost on data truncated using hard threshold SVD perform the best on the test data.
We could convert this problem into a multi-class classification problem and check the rating predict. This is a metric called log loss which could also be used as an alternative. There is also the possibility of using other ensembles.
Screenshot of the deployment is mentioned given below. A video recording of the same can be viewed here.
LinkedIn: https://www.linkedin.com/in/ashwin-michael-10b617142/Github repository: https://github.com/Ashcom-git/case-study-1Kaggle dataset: https://www.kaggle.com/olistbr/brazilian-ecommerce
Applied Roots. 2021. Applied AI Course. [online] Available at: <https://www.appliedaicourse.com/.> [Accessed 6 May 2021].Taylor, R., 2021. Optimal Singular Value Hard Threshold. [online] Pyrunner.com. Available at: <http://www.pyrunner.com/weblog/2016/08/01/optimal-svht/> [Accessed 9 May 2021].Brownlee, J., 2021. Autoencoder Feature Extraction for Classification. [online] Machine Learning Mastery. Available at: <https://machinelearningmastery.com/autoencoder-for-classification/> [Accessed 12 May 2021].Calin, T. and Xue, J., 2021. Custom f1_score metric in tensorflow. [online] Stack Overflow. Available at: <https://stackoverflow.com/a/64477588> [Accessed 7 May 2021].Stack Overflow. 2021. How to change legend size with matplotlib.pyplot. [online] Available at: <https://stackoverflow.com/a/7125157> [Accessed 3 May 2021].Aziz, H., Kumar, S., Korshunov, V. and Atha, P., 2021. Regex to match date formats DD-MM-YYYY and DD/MM/YYYY. [online] Stack Overflow. Available at: <https://stackoverflow.com/a/47218282> [Accessed 13 May 2021].Wal, P. and Paving, C., 2021. Match linebreaks — \n or \r\n?. [online] Stack Overflow. Available at: <https://stackoverflow.com/a/52057778> [Accessed 14 May 2021].Python, H. and Martin, L., 2021. How to remove any URL within a string in Python. [online] Stack Overflow. Available at: <https://stackoverflow.com/a/11332580> [Accessed 14 May 2021].Mortensen, P., 2021. How do I remove trailing whitespace using a regular expression?. [online] Stack Overflow. Available at: <https://stackoverflow.com/a/9532388> [Accessed 16 May 2021].