# Topic Word Embedding-Based Methods for Automatically Extracting Main Aspects from Product Reviews
## 4 Experimental Validation
### Evaluation of Clean Topics  
Figure 4 shows the Rouge-1 results of the baseline and proposed methods. Each Rouge-1 result is the average of all topic Rouge-1 values. We can indirectly know that the words in each topic are closely relevant each other and the topic words represent the same meaning. The baseline method generates topics by LDA, while the proposed method (Algorithm 2) cleans incomplete topics generated by LDA. The best case is to use fastText in the proposed method. The Rouge-1 values of the baseline and proposed methods are 0.725 and 0.582. Our proposed method improves up to 25% compared to the baseline method. Furthermore, the proposed method outperforms the baseline method in all cases of using different word embedding models.  
Table 5 compares the baseline and proposed methods by counting the number of topics which must be split and merged, and meaningless topics output by each method. For this evaluation, we hired three human evaluators who had nothing to do with our research. After going over the topic results and the review documents, they manually checked to see how many dirty topics are in the output. If the number of the split, merge, and meaningless topics are small, the output by either the baseline method or the proposed method will be more clean. Table 5 clearly shows that the topic results output by the proposed method are more clean than by the baseline method.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Word Embedding Model** & \(\mathbf{\alpha}\) & \(\mathbf{\beta}\) & **Rouge-1** \\ \hline \multirow{4}{*}{Best} & word2 vec & 2.0 & 0.08 & 0.553 \\  & **fastText** & **1.5** & **0.08** & **0.582** \\  & GloVe & 1.0 & 0.04 & 0.569 \\  & ELMo & 2.0 & 0.08 & 0.553 \\  & BERT & 1.5 & 0.16 & 0.536 \\ \hline \multirow{4}{*}{Worst} & word2 vec & 5.0 & 0.2 & 0.508 \\  & fastText & 5.0 & 0.2 & 0.508 \\ \cline{1-1}  & GloVe & 5.0 & 0.2 & 0.508 \\ \cline{1-1}  & ELMo & 5.0 & 0.18 & 0.494 \\ \cline{1-1}  & BERT & 2.0 & 0.1 & 0.520 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The best Dirichlet priors.  
Figure 3: Average topic coherence values of Algorithm 1 according to different \(\alpha\) and \(\beta\) values (In the figure, \(\alpha\) and \(\beta\) are represented as \(a\) and \(b\)).  
Figure 5 shows a clustering result of a topic after DBSCAN is performed. This shows a gigantic cluster (in black) and two small clusters (in blue and red). In this figure, we view the words belonging to the small clusters to the noises in the topic. Thus, we remove the small clusters except for the giant cluster. Because of space limitation, we leave out another cases in which there are 2\(\sim\)3 clusters with different meanings but the numbers of words in those clusters are equal. In this case, the topic is split into 2 or 3 topics.  
Figure 4: The Rouge-1 results of the baseline and proposed methods.  
Figure 5: A clustering result of DBSCAN.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Evaluation** & **Method** & **\# of Split Topics** & **\# of Merge Topics** & **\# of Meaningless Topics** & **Total** \\ \hline \multirow{2}{*}{Human Evaluator 1} & Baseline & 4 & 1 & 0 & 5 \\  & Ours & 1 & 0 & 0 & 1 \\ \hline \multirow{2}{*}{Human Evaluator 2} & Baseline & 1 & 0 & 0 & 1 \\  & Ours & 0 & 0 & 0 & 0 \\ \hline \multirow{2}{*}{Human Evaluator 3} & Baseline & 2 & 1 & 0 & 3 \\  & Ours & 1 & 0 & 0 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of clean topics.  
Figure 6 depicts a real example of the topic split and merge problems. Through word embedding and DBSCAN, a topic including clusters with different meanings is divided to two or more topics. On the other hand, the small clusters will be combined if it turns out that they have the same meaning at the topic merge stage. If each of the sub topics is meaningless, then it is removed. In a nutshell, we can split and merge raw topics by LDA through word embedding and unsupervised clustering like DBSCAN in order to make us clearly understand the semantics of the topics generated by LDA.