## 2 Segmentation-Enhanced Topic Modeling for Automated Text Annotation
### Text Segmentation with TopicDiff-LDA
To explore the time-efficiency of TopicDiff-LDA, an experiment was conducted by running the algorithm on datasets of different sizes that were randomly sampled from the tourism review set of Table 3. Figure 9 shows results obtained for a computer with an Intel Xeon E5-1650 3.60 GHz CPU and 128 GB DDR4

Figure 7: Classification performance of models trained on bronze standard sets generated automatically with TopicDiff-LDA (blue and orange bars for training on review- and segment-level, respectively) and MLTM (gray bars). RAM (all data was segmented, using the same LDA model; the model generation time is not included). Figure 8: As segment boundary is determined by the semantic similarity threshold \(t\) corresponding to the minimum of text perplexity (black line), segmentation errors \(Pk\) (blue line) and \(WD\) (red line) are also minimized. Calculations were performed on the Choi data subsets with segment sizes of **(I)** 3–11, **(II)** 3–5, **(III)** 6–8, and **(IV)** 9–11 sentences. (Note that unlike the perplexity score, \(Pk\) and \(WD\), to be computed, both require gold-standard segmentation of documents.) Figure 9: TopicDiff-LDA computing time as a function of the segmented review amount. On the graph, each dot indicates the time averaged over 10 random samples of the given size, and the bars show the standard error. The segmentation time (and, generally, the complexity) of the proposed algorithm is largely determined by the specific implementation of its Search function (see Algorithm 1). In the experiment of Figure 9, this function utilized the standard grid search of a Python library and the segmentation time grew linearly, as the number of reviews increased. For a comparison, some of the best supervised segmentation algorithms also have a linear time-complexity [39]. On the other hand, the runtime of GraphSeg, which is the second-most-accurate among unsupervised algorithms examined (see Table 2), is a polynomial function of the data size [49]. The latter time-complexity holds for many other unsupervised and semi-supervised segmentation algorithms (e.g., [46; 47; 48]). Therefore, one would assert that the algorithm proposed in this study has similar or better scalability than the existing baseline methods, whether supervised or unsupervised. ### Labeling

The reliability analysis of the gold-standard sets conducted in Section 4.3 confirmed that labeling short online texts is a challenging and often confusing task for human annotators (see [31] but also [58]). At the same time, the inter-annotator agreement achieved in this study for the 9 classes is considerably better than in many other reports which dealt with multiple-class annotation of short texts (e.g., [59; 60]). It is understood that the better inter-annotator agreement would be attributed, at least in part, to the fixed (as in [29]) rather than open-ended (as in [59]) labels used in the experiments. The focus of the presented case study was, however, more on automatic classification than on knowledge and label discovery. The results obtained also indicate that for the data, the inter-coder agreement remained nearly the same, regardless of whether single ([PERSON]'s Kappa of 0.658) or multiple (Kappa of 0.609) labels were selected for the reviews. This may be due to the semantic ambiguity associated with interpretation of short and possibly unfocused reviews by annotators lacking contextual domain knowledge. Comparing the final labels with the automated classification results demonstrated that the computer methods tested did not disagree more with the median human assessment (Kappa in the range of 0.612 to 0.625) than the humans disagreed among themselves (Kappa of 0.609) for the data. Furthermore, results depicted in Figure 4 suggest that the human-computer disagreement is not as much dependent on the specific label for TopicDiff-LDA as for MLTM. This may signal that the proposed algorithm is less sensitive to inter-topic variations (in lexicon, scope, etc.) than MLTM.