## 2 Segmentation-Enhanced Topic Modeling for Automated Text Annotation
### Text Segmentation with TopicDiff-LDA
Assessing the overall performances of the two multiple-labeling algorithms (Figure 5), one could conclude that the proposed unsupervised method worked at least as well as, or slightly better than semi-supervised MLTM on the review data. While the considerable difference in the macro-averaged AUROC (0.90 for TopicDiff-LDA vs. 0.78 for MLTM) can be explained by the class imbalance (see Table 7), both algorithms demonstrated \"very good\" [61] performance in terms of micro-averaged AUROC (0.86 and 0.82, respectively). ### Automatic Classification

Results of the automatic classification of the newly collected tourism reviews obtained in the case study suggest that the proposed approach allows for automatically generating bronze standard training sets of an acceptable quality for many practical applications. Performance evaluation of multi-class classification of customer reviews by state-of-the-art application systems trained on gold standard sets typically results in F1-score values ranging from 0.6 to 0.8, depending on the number of classes and the application domain (e.g., see [62; 63] and, more generally, [64]). The F1-scores obtained for the TopicDiff-LDA -trained classifiers are in the upper part of this range. The proposed unsupervised approach achieved better classification results than the MLTM semi-supervised method for all but the \"Beach\" (6) label (Figure 7). By manually inspecting the gold-standard data of this class, it was found that there are several relatively short but multiple-labeled reviews, such as, for example, the following:_\"The Base G Beach (a former American WW2 base) is located about 10 km west of the city of Jayapura, Papua. The beach is beautiful and from here you can look at the Pacific Ocean which is the gateway for ships sailing by from the west. The Base G beach is quiet and still very natural and clean. The water is clear and the beach is made of white sand. The water is so clear you can see clearly through the underwater scenery. Besides enjoying the scenery you can also go swimming, fishing, driving or rent a boat and sail around a bit. Local residents have built some benches and cabins to chill and hang out if you get tired of the sun. There are also several types of trees providing shade. While in Jayapura be sure to visit Base G Beach as it never hurts to spend a day and enjoy this beautiful beach.\"_

The labels manually assigned to this text by all three annotators were \"Beach\" (6) and \"General Information\" (7). The proposed approach failed to assign the \"Beach\" label as TopicDiff-LDA could not break the review into segments and treated it as a single-label (\"General Information\") document. One possible reason for that is the minimum segment size parameter set to 3 sentences (the sliding window size, see Section 3.2) in the presented study that prevented the algorithm from detecting shorter segments. Another reason is that TopicDiff-LDA works only with consecutive (i.e., linear and uninterrupted) segments. In the case of MLTM, the minimum labeled unit is sentence, and it could have correctly assigned both labels to the review in focus. There was practically no impact of the training scheme on the classifier performance. The model trained on (single-labeled) segments did work a little better than the model trained on multiple-labeled reviews, most notably for \"Natural Place\" (3), \"General Information\" (7), and \"Things to Buy\" (8) labels (see Figure 7). This may be due to the classifier being \"noised\" by the label co-occurrence context in the case of whole-review training. However, the class-aggregated difference is not statistically significant. The latter suggests statistical independence of the classes and, in a sense, validates the choice of labels for the review data. It is understood that dealing with dynamic, constantly flowing volumes of customer review data would require a transition from static to dynamic topic modeling in the proposed framework.