## 2 Segmentation-Enhanced Topic Modeling for Automated Text Annotation
### Text Segmentation with TopicDiff-LDA
Based on the evaluation, it was, therefore, decided to deploy these Linear SVC models for automatic classification and further analysis of tourism online reviews. ### Automatic Classification

New, previously unseen tourism reviews were crawled from the same website as in Section 4.1. Three human annotators different from those in the earlier experiments worked to create a gold-standard labeled data set (322 reviews in total, 1.12 labels per review). The same annotation procedure as in the case of \(B_{g}\) (see Section 4.3) was used. The pairwise-averaged value of the augmented Kappa statistic is 0.633 for the new human-annotated set. The labeled reviews were automatically classified with the three machine learning models pre-trained on the bronze standard data, as described in the previous subsection. To evaluate the classification performance of the models, only those labels were used, on which all three human annotators agreed (226 labels in total). Class-aggregated F1 macro- and micro- scores obtained for the models are as follows: 0.85 and 0.86 for Linear SVC review-level trained on \(B_{tr}\), 0.87 and 0.88 for Linear SVC segment-level trained on \(B_{tr}\), and 0.76 and 0.78 for Linear SVC review-level trained on C\({}_{tr}\), respectively. Figure 7 details results of the automatic classification experiment. Figure 6: F1-score statistics computed through 10-fold cross-validation of the four classifier models trained on (**I**) \(B_{tr}\) (review-level), (**II**) \(B_{tr}\) (segment-level), and (**III**) \(C_{tr}\) (review-level). ## 5 Discussion

### Segmentation Experiments

Results obtained in the segmentation experiments (Table 2, Section 3.2) convincingly demonstrated the superior ability of TopicDiff-LDA, compared to other unsupervised methods. Also, the performance of the proposed algorithm assessed in terms of \(\mathit{PK}\) improved as the segment length was growing. Segmentation error \(\mathit{PK}\) decreased from 0.36, the value obtained for the set with 3-5 sentences in each segment, to 0.22 for the set with 6-8 sentences per segment, and to 0.16 for when there were 9 to 11 sentences. The same tendency was observed for the segmentation error estimated in terms of WindowDiff (\(\mathit{WD}\)), a statistic reportedly less sensitive to internal segment size variance [56]: \(\mathit{WD}\) changed from 0.22 to 0.21, and to 0.19 as the segment size grew. These results should, however, be expected because longer texts allow for more accurate estimation of the multinomial parameters in LDA [47]. As TopicDiff-LDA relies on the perplexity score to determine segment boundaries (see Algorithm 1), the latter statistic is strongly correlated with the two measures of segmentation error. Assessed with Pearson's coefficient, the correlation ranged from 0.995 to 0.999 for \(\mathit{PK}\) and perplexity score, and from 0.974 to 0.998 for \(\mathit{WD}\) and perplexity score on the Choi dataset. Figure 8 illustrates the fact that a segment boundary determined by the minimum of the perplexity score also corresponds to the minimum or near-minimum of the segmentation errors for the data. Furthermore, Figure 8 reveals that locations of the minimums of the perplexity score and segmentation error measures tend to diverge as the segment size grows. This would be explained by the sensitivity of the perplexity statistic to word likelihood that, with a static lexicon, would be higher for longer texts. TopicDiff-LDA may, therefore, not work well when the expected segment size is much greater than 10 sentences. While the latter is an unlikely scenario in the case of online customer reviews (see [57]), this limitation would hamper the application of TopicDiff-LDA to longer documents. A well-known drawback of many unsupervised topic-level segmentation algorithms is their high computational cost (e.g., see [47]).