Segment Any Text: A Universal Approach for Robust, Efficient and
  Adaptable Sentence Segmentation
Authors: name: Markus Frohmann; name: Igor Sterner; name: Ivan Vulić; name: Benjamin Minixhofer; name: Markus Schedl
Published: 2024-06-24 14:36:11

Abstract: Segmenting text into sentences plays an early and crucial role in many NLP
systems. This is commonly achieved by using rule-based or statistical methods
relying on lexical features such as punctuation. Although some recent works no
longer exclusively rely on punctuation, we find that no prior method achieves
all of (i) robustness to missing punctuation, (ii) effective adaptability to
new domains, and (iii) high efficiency. We introduce a new model - Segment any
Text (SaT) - to solve this problem. To enhance robustness, we propose a new
pretraining scheme that ensures less reliance on punctuation. To address
adaptability, we introduce an extra stage of parameter-efficient fine-tuning,
establishing state-of-the-art performance in distinct domains such as verses
from lyrics and legal documents. Along the way, we introduce architectural
modifications that result in a threefold gain in speed over the previous state
of the art and solve spurious reliance on context far in the future. Finally,
we introduce a variant of our model with fine-tuning on a diverse, multilingual
mixture of sentence-segmented data, acting as a drop-in replacement and
enhancement for existing segmentation tools. Overall, our contributions provide
a universal approach for segmenting any text. Our method outperforms all
baselines - including strong LLMs - across 8 corpora spanning diverse domains
and languages, especially in practically relevant situations where text is
poorly formatted. Our models and code, including documentation, are available
at https://huggingface.co/segment-any-text under the MIT license.