# Paragraph Boundary Recognition in Novels for Story Understanding
## 4 Evaluational Experiment
### Results and Analysis  
We adjusted the \(\gamma\) hyperparameters of the loss function in BERT + FL and BERT + AL, and the decision threshold through the experiment for the validation data. Hyperparameter  
Figure 3: How to split text for each novel into training, validation, and test data. values shown in Table 3 were set in the models and evaluated on the test data. Table 4 shows the experiment results for each model. The model based on BERT outperformed the estimation accuracy of TextTiling and of the model of [PERSON] et al., which is the baseline model. We confirmed that each model using dice loss, focal loss, and anchor loss as the loss functions estimates paragraph boundaries with higher accuracy than that of BERT + CE, which is used in conventional BERT classification tasks. In particular, BERT + DL marked the highest \(F1\) and \(P_{k}\) in all datasets. Introducing threshold moving to the models also improved their performance in each dataset. This may have been because the properties of the validation data and test data were similar, and the determination threshold as a hyperparameter adjusted using validation data could identify paragraph boundaries with higher accuracy, even in the test data. These results suggest that the introduction of the cost-sensitive loss function and threshold moving into BERT improves the accuracy of paragraph-boundary estimation as an imbalanced-classification problem. Figure 4 shows the probability of being a positive example (positive probability) of each sample output by BERT + CE and BERT + DL for the part of the Fitzgerald test dataset. Samples that sunk at the bottom of the figure were predicted to be negative samples, while samples located at the top of the figure were predicted to be positive samples. If the positive probability for a sample was higher than threshold \(\tau\), then the sample was determined to be a positive sample. Therefore, the positive probability output for the sample on the red vertical line, which is the actual position of the paragraph boundary, should exceed the threshold, whereas that for the sample not on the red vertical line should not. Since the number of samples judged as positive samples by threshold moving increases, the \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{**[PERSON]**} & \multicolumn{2}{c}{**[PERSON]**} & \multicolumn{2}{c}{**[PERSON]**} \\ \cline{2-7}
**Model** & **F1(\(\uparrow\))** & **\(P_{k}(\downarrow)\)** & **\(F1(\uparrow)\)** & **\(P_{k}(\downarrow)\)** & **\(F1(\uparrow)\)** & **\(P_{k}(\downarrow)\)** \\ \hline TextTiling & 0.494 & 0.452 & 0.484 & 0.441 & 0.490 & 0.495 \\ [PERSON] et al. model & 0.598 & 0.234 & 0.420 & 0.275 & 0.275 & 0.315 \\ \hline BERT + CE & 0.759 & 0.158 & 0.498 & 0.226 & 0.591 & 0.186 \\ BERT + DL & 0.783 & 0.144 & 0.518 & 0.219 & 0.613 & 0.173 \\ BERT + FL & 0.767 & 0.153 & 0.519 & 0.219 & 0.614 & 0.170 \\ BERT + AL & 0.752 & 0.161 & 0.508 & 0.220 & 0.607 & 0.176 \\ \hline BERT + CE + TM & 0.773 & 0.132 & 0.504 & 0.223 & 0.605 & 0.178 \\ BERT + DL + TM & **0.797** & **0.125** & **0.536** & **0.217** & **0.627** & **0.164** \\ BERT + FL + TM & 0.780 & 0.133 & 0.521 & 0.219 & **0.627** & 0.165 \\ BERT + AL + TM & 0.786 & 0.128 & 0.521 & **0.217** & 0.622 & **0.164** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Experiment results for each model. The higher \(F1\) is, the better the outcome. By contrast, the lower \(P_{k}\) is, the better the results. Bold values represent the best evaluation results for each dataset.