# Topic Word Embedding-Based Methods for Automatically Extracting Main Aspects from Product Reviews
## 4 Experimental Validation
### Evaluation of Best Priors and Number of True Topics  
In the experiment, we used about 10,000 review documents about hotels in US, where the total numbers of words, vocabularies, and sentences are 282,934, 12,777, and 40,819, respectively [53]. As a topic model, we executed [PERSON] sampling-based LDA [54] and the numbers of topics and of words per topic are 10 and 50, respectively. Table 3 shows the reason why we selected ten topics as the output of LDA. As the number of topics increases to 3, 5, 7, and 10, we measured the values of Rouge-1 and the numbers of split and merge topics, to see how clean the topic result is. The Rouge-1 is the most well-known metric used to quantify how similar two sets are. In this set-up, for a clean topic \(\theta_{i}\), we first selected the top 100 review documents that are the most relevant with \(\theta_{i}\) retrieved by the LDA, and then several human evaluators made a list of important and meaningful keywords after they went over the documents. We denote such a list by \(X\) as the solution set to \(\theta_{i}\). We also had another set \(Y\) that is a list of topic words in \(\theta_{i}\). Now the Rouge-1 metric is calculated by \(\frac{|X\cap Y|}{|X|}\). \(Y\) set cleaned by the proposed method is very similar to the solution set if the value of the Rouge-1 is close to 1. In addition, if the numbers of split and merge topics are low in the topic result post-processed by the proposed method, the topic result is more clean than the raw topics generated by the existing LDA. While all Rouge-1 results are little different, the total number of the split and merge topics is small when the number of topics is ten. As a result, we selected 10 as the number of topics in all experiments. Unexpectedly, it turns out that only 1\(\sim\)2 topics have three or more groups with different meanings when the number of topics is 3 but each topic has two groups with different meanings when the number of topics is 7.  
For lexical semantics, we focus on word embedding including word2 vec, fastText, GloVe, ELMo, and BERT to surmount the shortcomings of both one-hot encoding and WordNet-based methods. Table 4 supports our claim that using best priors makes topic result to be more clean. In most empirical experiments using LDA, the default \(\alpha=\frac{50}{\text{Number of topics}=10}=5\) and \(\beta=0.1\). According to our manual investigation, the Rouge-1 results are the worst when we used the default values. In contrast, when '\(\alpha=1.5\) and \(\beta=0.08\), the Rouge-1 result is the best, indicating that the topic result is close to the solution set and the fastText is the best among different word embedding models. Please note that Table 4 summarizes the results of the fastText that we selected as the best word embedding model. We can also observe that the word2 vec, GloVe, and fastText are the models re-trained with our own dataset show better Rouge-1 results than ELMo and BERT that are not re-trained with our own dataset.  
To evaluate Algorithm 1, we measured topic coherence values according to different \(\alpha\) and \(\beta\) values. Because of the result in Table 4, we selected fastText as the best word embedding method. Figure 3 clearly shows that the average topic coherence value is about 0.2 when \(\alpha=1.5\) and \(\beta=0.08\). The topic coherence value is the highest in all priors. These results indicate that topic words are closely relevant one another in the particular prior values.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Number of Topics** & **Rouge–1** & **Number of Split Topics** & **Number of Merge Topics** & **Total** \\ \hline
3 & 0.546 & 9 & 0 & 9 \\
5 & 0.511 & 8 & 1 & 9 \\
7 & 0.453 & 10 & 2 & 12 \\
10 & 0.512 & 4 & 1 & 5 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The optimal number of topics.