SlowGuess/ACL_2020_OCR huggingface dataset:
There are five major takeaways from the experimental results and analysis. First, the jointly trained S-LSTM model shows major improvement over prior work that modeled document segmentation and segment labeling tasks separately. Second, segment alignment and exploration during training reduces error rates. Third, the segment pooling layer leads to improvements for both segmentation and segment labeling. Fourth, S-LSTM outperforms an IOB-tagging CRF-decoded model for single label segment labeling, and also generalizes easily
WikiSection-topics
single-label classification	en_disease	de_disease	en_city	de_city
27 topics	25 topics	30 topics	27 topics
model configuration	↓Pk	↑F1	↑MAP	↓Pk	↑F1	↑MAP	↓Pk	↑F1	↑MAP	↓Pk	↑F1	↑MAP
C99	37.4	n/a	n/a	42.7	n/a	n/a	36.8	n/a	n/a	38.3	n/a	n/a
TopicTiling	43.4	n/a	n/a	45.4	n/a	n/a	30.5	n/a	n/a	41.3	n/a	n/a
TextSeg	24.3	n/a	n/a	35.7	n/a	n/a	19.3	n/a	n/a	27.5	n/a	n/a
SEC>T+emb	26.3	55.8	69.4	27.5	48.9	65.1	15.5	71.6	81.0	16.2	71.0	81.1
LSTM-LSTM-CRF	23.9	57.2	n/a	23.6	51.4	n/a	9.7	77.5	n/a	10.2	74.0	n/a
S-LSTM	20.0	59.3	72.4	18.8	55.6	69.0	9.1	76.1	83.5	9.5	76.5	84.5
Table 1: WikiSection results. Baselines are TopicTiling (Riedl and Biemann, 2012), TextSeg (Koshorek et al., 2018), and C99 (Choi, 2000), and the best neural SECTOR models from Arnold et al..
WikiSection-headings
multi-label classification
model configuration	en_disease
179 topics	de_disease
115 topics	en_city
603 topics	de_city
318 topics
↓Pk	↑Prec@1	↑MAP	↓Pk	↑Prec@1	↑MAP	↓Pk	↑Prec@1	↑MAP	↓Pk	↑Prec@1	↑MAP
C99	37.4	n/a	n/a	42.7	n/a	n/a	36.8	n/a	n/a	38.3	n/a	n/a
TopicTiling	43.4	n/a	n/a	45.4	n/a	n/a	30.5	n/a	n/a	41.3	n/a	n/a
TextSeg	24.3	n/a	n/a	35.7	n/a	n/a	19.3	n/a	n/a	27.5	n/a	n/a
SEC>H+emb	30.7	50.5	57.3	32.9	26.6	36.7	17.9	72.3	71.1	19.3	68.4	70.2
S-LSTM	19.8	53.5	60.3	18.6	36.2	46.1	9.0	73	71.3	8.2	74.1	75.1
S-LSTM, -expl	20.8	52.1	59	19.1	34.7	44.8	9.2	72.7	70.8	8.5	73.8	74.4
S-LSTM, -expl, -pool	21.2	52.3	59.5	19.8	34.4	45	10.4	69.7	67.2	10.2	64.1	66.7
Table 2: WikiSection headings task results, which predicts a multi-label bag-of-words drawn from section headers. To show the effect of the segment pooling and model exploration used in S-LSTM we report two variants where -expl uses only teacher forcing and -pool uses only mean pooling.
and tractably to multi-labeling. Fifth, a deeper analysis of the joint modeling demonstrates that segment labeling and segment bound prediction contain complementary information.
5.1 Structure Predicts Better Structure
Tables 1 and 2 show that by explicitly predicting segment bounds we can improve segmentation by a large margin. On the header prediction task (Table 2), we reduced  $P_{k}$  by an average of over  $30\%$  across the WikiSection datasets.  $P_{k}$  was consistent across both WikiSection tasks, and did not degrade when going from single-label to multi-label prediction, as Arnold et al. had found. This shows that we can achieve a more robust segmentation through jointly modeling segmentation and labeling. This is also clear from Figure 4, where S-LSTM predicts a much more accurate segmentation.
5.2 Exploration Allows Error Recovery
The results of an ablation experiment (Table 2, bottom) show that there is an additional classification gain by allowing the model to explore recovering from segmentation errors. Exploration has the important property of allowing the model to optimize more closely to how it is being evaluated. This follows from a long line of work in NLP that shows
that for tasks such as dependency parsing (Ballesteros et al., 2016), constituency parsing (Goodman, 1996), and machine translation (Och, 2003), all show improvements by optimizing on a loss that aligns with evaluation.
The teacher forcing was important at the beginning of model training. When training variants of S-LSTM that did not use teacher forcing at the beginning, which instead could explore the bad segmentation, the segmentation failed to converge and the model performed universally poorly.
5.3 S-LSTM Can Take Advantage of Both of These, Plus Segment Pooling
S-LSTM is capable of taking advantage of the complementary information by jointly learning to segment and label. It is capable of learning to recover from segmentation errors by exploring towards the end of training. But the ablation study shows that there is one more important component of S-LSTM that allows it to improve over previous baselines: LSTM pooling over segments. The addition of the segment pooling layer improves MAP and Prec@1 across all four datasets in the heading prediction task (Table 2), comparing the model without exploration (S-LSTM,-expl) with the model without exploration (which uses average pooling: S-LSTM,
Segmentation and multi-label classification	Wiki-50	Cities	Elements	Clinical
\( \downarrow P_k \)	\( \uparrow \) MAP	\( \downarrow P_k \)	\( \uparrow \) MAP	\( \downarrow P_k \)	\( \uparrow \) MAP	\( \downarrow P_k \)
GraphSeg	63.6	n/a	40.0	n/a	49.1	n/a	-
BayesSeg	49.2	n/a	36.2	n/a	35.6	n/a	57.8
TextSeg	18.2*	n/a	19.7*	n/a	41.6	n/a	30.8
SEC> H+emb@en_disease	-	-	-	-	43.3	9.5	36.5
SEC> H+emb@en_city	40.5	13.4	33.3	53.6	41.0	7.9	-
S-LSTM@en_city	22.7	16.6	21.2	54.2	34.5	11.0	-
S-LSTM@en_disease	-	-	-	-	30.2	19.1	36.1
Table 3: Transfer results across four datasets. Those marked * are trained on the training portion of the corresponding dataset, whereas those without are either unsupervised or trained on a different dataset. For the Wiki-50, Cities, and Elements datasets, S-LSTM outperforms all models not trained on corresponding training set.
WikiSection-headings	de_disease
multi-label classification	115 topics
model configuration	\( \downarrow P_k \)	\( \uparrow \mathrm{P}@1 \)	\( \uparrow \mathrm{MAP} \)
S-LSTM, w/o Segment Prediction	n/a	42.3	52.1
S-LSTM, w/ Segment Prediction	19.1	43.3	53.3
Table 4: A model trained to jointly predict segment bounds and segment labels improves classification over a baseline which only predicts labels. Both are given oracle segment bounds and do not use exploration.
WikiSection-headings	de_disease
document segmentation	115 topics
model configuration	↓Pk	↑P@1	↑MAP
S-LSTM, w/o Segment Labeling	21.8	n/a	n/a
S-LSTM, w/ Segment Labeling	19.1	34.7	44.8
Table 5: Inverse of the experiment in Table 4. A model that jointly predicts segment bounds and labels outperforms a model that only predicts segment bounds.
expl,-pool). It is the combination of these three improvements that comprise the full S-LSTM.
5.4 S-LSTM Outperforms a CRF Baseline
In Table 1, the results demonstrate that S-LSTM outperforms LSTM-LSTM-CRF baseline in almost every case for single-labeling, and in every case for segmentation. This makes S-LSTM a useful model choice for cases like clinical segmentation and labeling, where segments are drawn from a small fixed vocabulary. S-LSTM also generalizes easily to multi-label problems, in contrast to an IOB-tagging LSTM-LSTM-CRF, since it only requires changing the segment-pooling loss from cross-entropy to binary cross-entropy.
5.5 Predicting Structure Predicts Better Labels (and vice versa)
Though we compare with TextSeg (a neural model that predicts segment bounds) and SECTOR (a neural model that predicts sentence labels and post hoc segments them) and show improvements compared to both models, we also directly test the hypothesis that the segmentation and segment labeling tasks contain complementary information. To do so, we conduct two experiments: (1) we fix the segment bounds at training and evaluation time, only training the model to label known segments (results in Table 5); and (2) we only have the model predict segment bounds (results in Table 4).
In both cases, the addition of the loss from the companion task improves performance on the main task. This shows that the two tasks contain complementary information, and directly validates our core hypothesis that the two tasks are tightly interwoven. Thus, considering them jointly improves performance on both tasks.
In this paper we introduce the Segment Pooling LSTM (S-LSTM) model for joint segmentation and segment labeling tasks. We find that the model dramatically reduces segmentation error (by  $30\%$  on average across four datasets) while improving segment labeling accuracy compared to previous neural and non-neural baselines for both single-label and multi-label tasks. Experiments demonstrate that jointly modeling the segmentation and segment labeling, segmentation alignment and exploration, and segment pooling each contribute to S-LSTM's improved performance.
S-LSTM is agnostic as to the sentence encoder used, so we would like to investigate the potential
usefulness of transformer-based language models as sentence encoders. There are additional engineering challenges associated with using models such as BERT as sentence encoders, since encoding entire documents can be too expensive to fit on a GPU without model parallelism. We would also like to investigate the usefulness of an unconsidered source of document structure: the hierarchical nature of sections and subsections. Like segment bounds and headers, this structure is naturally available in Wikipedia. Having shown that segment bounds contain useful supervisory signal, it would be interesting to examine if segment hierarchies might also contain useful signal.
The authors would like to thank Sebastian Arnold for his feedback and responsiveness. We would also like to thank others for their feedback, including Franck Dernoncourt, Sasha Spala, Nick Miller, Han-Chin Shing, Pedro Rodriguez, Denis Peskov, and Yogarshi Vyas. This work was supported through Adobe Gift Funding, which supports an Adobe Research-University of Maryland collaboration. It was completed while the primary author was interning at Adobe Research.
Parviz Ajideh. 2003. Schema theory-based pre-reading tasks: A neglected essential in the ESL reading class. The Reading Matrix, 3(1).
James Allan, Jaime G Carbonell, George Doddington, Jonathan Yamron, and Yiming Yang. 1998. Topic detection and tracking pilot study final report. In Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop.
Sebastian Arnold, Rudolf Schneider, Philippe Cudre-Mauroux, Felix A Gers, and Alexander Loser. Sector: A neural model for coherent topic segmentation and classification. Transactions of the Association for Computational Linguistics, 7.
Miguel Ballesteros, Yoav Goldberg, Chris Dyer, and Noah A Smith. 2016. Training with exploration improves a greedy stack-LSTM parser. In Proceedings of Empirical Methods in Natural Language Processing.
Doug Beeferman, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Machine Learning, 34(1-3).
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with
subword information. Transactions of the Association for Computational Linguistics, 5.
Harr Chen, SRK Branavan, Regina Barzilay, and David R Karger. 2009. Content modeling using latent permutations. Journal of Artificial Intelligence Research, 36.
Freddy YY Choi. 2000. Advances in domain independent linear text segmentation. Conference of the North American Chapter of the Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics.
Tracy Edinger, Dina Demner-Fushman, Aaron M Cohen, Steven Bedrick, and William Hersh. 2017. Evaluation of clinical text segmentation to facilitate cohort retrieval. In AMIA Annual Symposium Proceedings.
Jacob Eisenstein and Regina Barzilay. 2008. Bayesian unsupervised topic segmentation. In Proceedings of Empirical Methods in Natural Language Processing.
Kavita Ganesan and Michael Subotin. 2014. A general supervised approach to segmentation of clinical texts. In IEEE International Conference on Big Data.
Goran Glavas, Federico Nanni, and Simone Paolo Ponzetto. 2016. Unsupervised text segmentation using semantic relatedness graphs. In Proceedings of the Joint Conference on Lexical and Computational Semantics.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of Artificial Intelligence and Statistics.
Joshua Goodman. 1996. Parsing algorithms and metrics. In Proceedings of the Association for Computational Linguistics.
Marti Hearst and Christian Plaunt. 2002. Subtopic structuring for full-length document access. Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval.
Marti A. Hearst. 1997. Text tiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1).
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations.
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018. Text segmentation as a supervised learning task. In Conference of the North American Chapter of the Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Conference of the North American Chapter of the Association for Computational Linguistics.
Chin-Yew Lin. 2004. Looking for a few good metrics: Automatic summarization evaluation-how many samples are enough? In NTCIR.
Andrew McCallum and Wei Li. 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In Conference of the North American Chapter of the Association for Computational Linguistics.
Viet-An Nguyen, Jordan Boyd-Graber, and Philip Resnik. 2012. SITS: A hierarchical nonparametric model using speaker identity for topic segmentation in multiparty conversations. In Proceedings of the Association for Computational Linguistics.
Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, Sapporo, Japan.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Conference of the North American Chapter of the Association for Computational Linguistics.
Alexandra Pomares-Quimbaya, Markus Kreuzthaler, and Stefan Schulz. 2019. Current approaches to identify sections within clinical narratives from electronic health records: a systematic review. BMC medical research methodology, 19(1).
Lance A Ramshaw and Mitchell P Marcus. 1999. Text chunking using transformation-based learning. In Natural language processing using very large corpora. Springer.
Martin Riedl and Chris Biemann. 2012. *TopicTiling: a text segmentation algorithm based on LDA*. In *Proceedings of ACL 2012 Student Research Workshop*.
Najmeh Sadoughi, Greg P Finley, Erik Edwards, Amanda Robinson, Maxim Korenevsky, Michael Brenndoerfer, Nico Axtmann, Mark Miller, and David Suendermann-Oeft. 2018. Detecting section boundaries in medical dictations: Toward real-time conversion of medical dictations to clinical reports.
In International Conference on Speech and Computer. Springer.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1).
Janet K Swaffar, Katherine Arens, and Heidi Byrnes. 1991. Reading for meaning: An integrated approach to language learning. Pearson College Division.
Michael Tepper, Daniel Capurro, Fei Xia, Lucy Vanderwende, and Meliha Yetisgen-Yildiz. 2012. Statistical section segmentation in free-text clinical records. In Proceedings of the Language Resources and Evaluation Conference.
Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. *Neural Computation*, 1(2).
Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, and Xueqi Cheng. 2019. Outline generation: Understanding the inherent content structure of documents. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval.