# BiTTM: A Core Biterms-Based Topic Model for Targeted Analysis
## 4 Experimental Results  
### Baselines and Metrics  
_Baselines_. Three methods are chosen to be compared with BiTTM, including Targeted Topic Model (TTM), Bitterm Topic Model-Partial Data (BTM-PD), and Bitterm Topic Model with a post-filtering strategy (BTM\({}^{\star}\)). * **TTM**. Targeted Topic Model is the first method for focused analysis that extracts related topics according to a target keyword provided by users. We select TTM rather than APSUM as the baseline of specialised topic models for targeted analysis because TTM outperforms APSUM in terms of _topic coherence_ when the number of topics is less than 50 [3]. For targeted analysis of fine-grained topics, we believe the number of topics in a given corpus is usually less than 50. Moreover, TTM serves as the most valuable comparison because APSUM is not exactly designed for targeted analysis. * **BTM\({}^{\star}\)**. As our model is developed based on bitterms, we also compare with two variations of BTM that are adapted for targeted analysis. BTM is a state-of-the-art topic model for short texts, which also applies to long texts [2]. As a typical full-analysis model, BTM aims to find all topics (or all aspects) from the entire corpus. We then use a filtering strategy to eliminate topics that do not contain the target keywords. This approach is named as BTM\({}^{\star}\) for simplicity. * **BTM-PD**. This is another variation of BTM which applies the pre-filtering strategy to perform focused analysis. We use only the subset of documents containing the target keywords to model topics. As discussed before, the pre-filtering strategy is handicapped by the variability of target keyword--relevant documents may be filtered so that topics may be missed out. MetricsWe adopt two techniques to evaluate the quality of topics: _topic coherence_[35] and _precision@n_[1] (_P@n_ for short). The former is a popular evaluation method to evaluate the quality of discovered topics [36; 37; 38; 39; 40]. As an automated evaluation metric, topic coherence mainly measures the interpretability of topics instead of target-relevance. More specifically, topic coherence measures document-level mutual information of keywords in topics, however, it does not reflect the relationship between topics and targets. In order to evaluate whether topics are target-relevant, we employ the metric _P@n_, also used by TTM [1], which is an evaluation based on human judgment to assess the relevance between the target and topics. Considering the \(M\) most probable words in topic \(k\), the topic coherence of \(k\) is defined as Equation (7). \[TC(k)=\sum_{m=2}^{M}\sum_{l=1}^{m-1}log\frac{|\{doc(w_{k,m},w_{k,l})\}|+1}{| \{doc(w_{k,l})\}|} \tag{7}\]  
where \(|\{doc(w_{k,m},w_{k,l})\}|\) is the number of documents containing both \(w_{k,m}\) and \(w_{k,l}\); \(|\{doc(w_{k,l})\}|\) is the number of documents containing \(w_{k,l}\), and \(w_{k,l}\) is the \(l\)th most probable word in topic \(k\). Basically, for the \(m\)th probable word, the measure considers its co-occurrence with the \(m-1\) more probable words. A smoothing count of 1 is added to avoid leading the logarithm to zero. Basically, the more the measure approximates to zero, the more coherent the discovered topics are. Given the set of topics discovered by all models, suppose there are \(Ku\) topics that have been verified by users to be related with the target aspect. Moreover, from all topics discovered by a particular model \(m\), suppose there are \(Km\) topics related with the target. Then, the precision of model \(m\) at rank position \(n\) is defined as follows:  
\[P_{m}@n=\frac{\sum_{i=1}^{Km}|\{CorrectWords(z)\}|}{\sum_{i=1}^{Km}|\{Correct Words(z)\}|} \tag{8}\]  
where \(|\{CorrectWords(z)\}|\) is the number of words, among the top \(n\) words of topic z, which are relevant to the target (Note that, if a discovered topic is potentially related with multiple semantic topics, the best semantic topic based on the top 20 words will be adopted). Therefore, the two evaluation methods have different merits and objectives.