# Paragraph Boundary Recognition in Novels for Story Understanding
## 4 Evaluational Experiment
### Results and Analysis
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{**[PERSON]**} & \multicolumn{2}{c}{**[PERSON]**} & \multicolumn{2}{c}{**Twin**} \\ \cline{2-7}
**Model** & \(\tau\) & \(\gamma\) & \(\tau\) & \(\gamma\) & \(\tau\) & \(\tau\) \\ \hline BERT + FL & 0.5 & 3.0 & 0.5 & 2.0 & 0.5 & 5.0 \\ BERT + AL & 0.5 & 2.0 & 0.5 & 0.5 & 0.5 & 1.0 \\ \hline BERT + CE + TM & 0.13 & - & 0.44 & - & 0.42 & - \\ BERT + DL + TM & 0.20 & - & 0.29 & - & 0.33 & - \\ BERT + FL + TM & 0.14 & 0.5 & 0.42 & 2.0 & 0.43 & 5.0 \\ BERT + AL + TM & 0.14 & 2.0 & 0.39 & 2.0 & 0.24 & 1.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Hyperparameters tuned through grid search on validation data. Value of \(\gamma\) is a hyperparameter for focal loss and anchor loss. Value of \(\tau\) is the decision threshold. Value of \(\tau\) for models with threshold moving could be tuned between 0 and 0.5 in 0.01 increments. However, values for BERT + FL and BERT + AL were fixed at 0.5 because threshold moving was not introduced in these models. number of samples that are correctly identified as positive samples increases, but the number of actually negative samples that are mistakenly identified as positive samples also increases. However, the negative sample, which was mistakenly judged as a positive sample, accounted for a small proportion of the total negative sample. As a result, the value of each evaluation metric indicating the performance of the model improved. Table 5 shows examples of the test samples and outputs of BERT + CE and BERT + DL for the samples. Example 1 is a sample in which both BERT + CE and BERT + DL were judged as positive with high probability. In this sample, Sentence1 and Sentence2 belonged to different sections, and it is clear that the scenes and topics were different; thus, they could easily be identified as positive examples. By contrast, Examples 2 and 3 are samples in which BERT + CE and BERT + DL were both presumed to differ from the correct label with high probability. The dataset also included samples that were difficult for humans to discriminate because the sentences that they contained were short, and little information was given. Example 4 is a sample that was correctly identified as a positive sample by introducing threshold moving. On the other hand, Example 5 is an actually negative sample mistakenly identified as a positive sample by BERT + CE + TM and BERT + DL + TM. Example 6 was correctly determined to be a positive sample by BERT + DL + TM, but was determined to be a negative sample by BERT + CE, BERT + CE + TM, and BERT + DL. Figure 4: Positive probability in paragraph-boundary recognition for BERT + CE and BERT + DL in the Fitzgerald dataset. Samples are from test data based on the novel “The Great Gatsby”. Vertical axis represents the positive probability for each sample \(x_{i}:p_{i}=p(y_{i}=1|x_{i})\). Vertical red lines represent golden paragraph boundaries that were actual positions of paragraph breaks. Each dotted line represents the threshold positive probability. Blue points are BERT + CE predictions, and lime green points are BERT + DL predictions.