# Developing Data-Conscious Deep Learning Models for Product Classification
## 4 Experiment
### Case Study of Filtering Efficiency  
In this section, we present a case study conducted to see the impact of the filtering process on the performance of the product classification models. Same as in the previous section, accuracy was used as a performance metric. For the top 1% of the frequency of all the words, i.e., 1.3K words, we increased the number of filtered-out stopwords by the top 10% words of the highest frequencies, and examined the accuracy for each case. Figure 6 shows the result of using the CNN model. A similar trend is displayed when the Bi-LSTM model is used. As shown in the figure, the \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{_Selection_} & \multicolumn{3}{c}{_Transformation_} & \multicolumn{3}{c}{_Filtering_} \\ \hline  & \multicolumn{2}{c}{**Worst**} & \multicolumn{1}{c}{**Best**} & \multicolumn{1}{c}{**Avg.**} & \multicolumn{1}{c}{**Worst**} & \multicolumn{1}{c}{**Best**} & \multicolumn{1}{c}{**Avg.**} & \multicolumn{1}{c}{**Worst**} & \multicolumn{1}{c}{**Best**} & \multicolumn{1}{c}{**Avg.**} \\ \hline
**NB** & accuracy 0.2843 & 0.6311 & 0.4634 & 0.3358 & 0.6460 & 0.5218 & 0.3371 & 0.6460 & 0.5228 \\ \hline \multirow{3}{*}{**CNN**} & accuracy 0.4867 & 0.7425 & 0.6574 & 0.5892 & 0.7869 & 0.7314 & 0.5854 & 0.7832 & 0.7345 \\ \cline{2-11}  & time & 0.36:23 & 0.18:17 & - & 0.10:46 & 0.08:57 & - & 0.08:08 & 0.07:09 & - \\ \hline \multirow{3}{*}{**Bi-LSTM**} & accuracy 0.4886 & 0.7266 & 0.6457 & 0.5624 & 0.7784 & 0.7191 & 0.5645 & 0.7839 & 0.7232 \\ \cline{2-11}  & time & 0.36:16 & 0.15:30 & - & 0.10:45 & 0.16:02 & - & 0.07:47 & 0.13:05 & - \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of accuracy and training time between deep learning models (time: hh:mm:ss). accuracy decreases as the number of stopwords increases by the top 10% each time. It can also be observed that after the 70% setting is exceeded, the decreasing rate is relatively flat compared with the 10-70% settings. When the words with high frequencies are filtered-out as stopwords, the impact on the accuracy should be more significant. Furthermore, in our data, we observed that the distribution of the words displaying those frequencies is approximately 70%. Figure 6 shows a case where the experiment involves the observation of only the frequency of the filtered-out stopwords irrespective of the syntactic structure or semantics of those words. This can lead to the following question: would the performance improve if we filter out words that are syntactically wrong or weird, composed of unnecessary characters, or have a highly poor semantic relationship with the category that the product belongs to? Therefore, we conducted the experiments to determine the performance of the models wherein the words considered unnecessary syntactically or semantically were removed using heuristics among the top 1% words with high frequencies. This is similar to the previous experiment. Table 4 shows the type of heuristics, stopword examples, and number and proportion of words removed as stopwords. The data refining process was conducted with the list of stopwords created for each criterion. Then, the training process was performed. Table 5 shows the accuracy measured after performing filtering with each criterion of stopwords, for the NB, CNN, and Bi-LSTM models. Without loss of generality, in this experiment we used only the attributes \"name\" and \"keyword\". In Table 5, the average accuracy of the CNN and Bi-LSTM models for criteria (1)-(6) is 0.7799 and 0.7756, respectively.