## 2 Segmentation-Enhanced Topic Modeling for Automated Text Annotation
### Text Segmentation with TopicDiff-LDA
To assess the level of annotator (dis)agreement in the labeled sets, [PERSON]'s Kappa statistic augmented for multiple labels (see [52]) was calculated. Its pairwise averaged value stays at 0.658 for set \(A_{g}\), and at 0.609 for set \(B_{g}\). The consistency of the machine-labeled sets in respect to human annotation was evaluated with the same statistic. Its values obtained for sets \(A\) and \(A_{g}\) (LDA vs. human), \(B\) and \(B_{g}\) (TopicDiff-LDA vs. human), and \(C\) and \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Segment**} & \multicolumn{8}{c}{**Topic**} \\ \cline{2-10}  & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** \\ \hline
1 & 0.006 & 0.033 & **0.539** & 0.033 & 0.005 & 0.327 & 0.009 & 0.036 & 0.007 \\
2 & 0.006 & 0.004 & 0.146 & 0.107 & 0.004 & 0.241 & 0.008 & **0.344** & 0.136 \\
3 & **0.334** & 0.004 & 0.209 & 0.004 & 0.005 & 0.008 & 0.282 & 0.007 & 0.144 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The updated topic probability distribution of the segmented review (bold numbers indicate the highest probability for each segment). \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Class**} & \multicolumn{2}{c}{**Average Review**} & \multicolumn{2}{c}{**No. of Distinct Labels Assigned (Avg per Review)**} \\ \cline{2-5}  & **Size, Sentences** & \multicolumn{1}{c}{\(A_{g}\)} & \(B_{g}\) \\ \hline
1 & 10 & 24 & 26 (1.35) \\
2 & 11 & 13 & 15 (1.53) \\
3 & 13 & 55 & 65 (1.48) \\
4 & 9 & 18 & 21 (1.29) \\
5 & 11 & 19 & 21 (1.48) \\
6 & 14 & 67 & 73 (1.42) \\
7 & 13 & 43 & 57 (1.46) \\
8 & 17 & 13 & 15 (1.40) \\
9 & 13 & 17 & 25 (1.68) \\ \hline \hline \multicolumn{5}{c}{Total labels:} & 269 & 318 (1.18) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Structure of the gold-standard labeled sets. (MLTM vs. human) are 0.625, 0.615, and 0.612, respectively. Figure 4 shows the per-class agreement in the multiple-labeled sets. To compare the classification performance of the multiple-labeling algorithms, the area under the receiver operating characteristic curve (AUROC) statistic was used [53]. The macro (equal class-label weights) and micro (unequal class-label weights) average AUROC values obtained with the data are 0.90 and 0.86 for TopicDiff-LDA, and 0.78 and 0.82 for MLTM, respectively (also see Figure 5). ### Classifier Model Selection

Two multiple-label bronze standard sets \(B_{tr}\) and \(C_{tr}\) were created by running TopicDiff-LDA and MLTM, respectively, on the 2685 reviews collected in the study. Four machine learning methods frequently used for short-text classification (see [54])-Convolutional Neural Network (CNN), Random Forest, Logistic Regression, and Linear Support Vector Classification (SVC)-were tested. All the methods but CNN were implemented, using the source code from [55], to deal with multiple-label classification. With \(B_{tr}\) set, two different training schemes were used, resulting in two models for each method: one model trained on the whole review texts, and another-trained on the review text segments. (Obviously, with \(C_{tr}\), the training could only be performed on the whole review level.) Figure 6 compares the performance of the methods evaluated through 10-fold cross-validation on sets \(B_{tr}\) and \(C_{tr}\). It should be noted that classifiers trained with a single-labeled set were tested in a previous study but found prone to errors in many cases of automatic review classification [51]. Figure 4: Agreement for sets labeled with TopicDiff-LDA and MLTM relative to human annotation. Figure 5: Multiple-labeling algorithm performance comparison. Linear SVC performed significantly better (\(p<0.001\)) than the other methods in all cases. The averaged macro- and micro-F1 scores achieved the values of 0.78 and 0.78 for the \(B_{tr}\) review-level trained model, 0.78 and 0.79 for the \(B_{tr}\) segment-level trained model, and 0.56 and 0.60 for the \(C_{tr}\) trained model, respectively.