Point Break: Surfing Heterogeneous Data for Subtitle Segmentation
Authors: Alina Karakanta, Matteo Negri, Marco Turchi (Fondazione Bruno Kessler, Università di Trento, 2020)
Topic: Language Models

Abstract
Subtitles, in order to achieve their purpose of transmitting information, need to be easily readable. The segmentation of subtitles into phrases or linguistic units is key to their readability and comprehension. However, automatically segmenting a sentence into subtitles is a challenging task and data containing reliable human segmentation decisions are often scarce. In this paper, we leverage data with noisy segmentation from large subtitle corpora and combine them with smaller amounts of high-quality data in order to train models which perform automatic segmentation of a sentence into subtitles. We show that even a minimum amount of reliable data can lead to readable subtitles and that quality is more important than quantity for the task of subtitle segmentation.

Introduction
In a world dominated by screens, subtitles are a vital means for facilitating access to information for diverse audiences. Subtitles are classified as interlingual (subtitles in a different language as the original video) and intralingual (of the same language as the original video) AUTHOR. Viewers normally resort to interlingual subtitles because they do not speak the language of the original video, while intralingual subtitles (also called captions) are used by people who cannot rely solely on the original audio for comprehension. Such viewers are, for example, the deaf and hard of hearing and language learners. Apart from creating a bridge towards information, entertainment and education, subtitles are a means to improving the reading skills of children and immigrants AUTHOR. Having such a large pool of users and covering a wide variety of functions, subtitling is probably the most dominant form of Audiovisual Translation. Subtitles, however, in order to fulfil their purposes as described above, need to be presented on the screen in a way that facilitates readability and comprehension. AUTHOR claim that subtitles which cannot be read or can be read only with difficulty `are almost as bad as no subtitles at all'. Creating readable subtitles comes with several challenges. The difficulty imposed by the transition to a different semiotic means, which takes place when transcribing or translating the original audio into text, is further exacerbated by the limitations of the medium (time and space on screen). Subtitles should not exceed a maximum length, usually ranging between 35-46 characters, depending on screen size and audience age or preferences. They should also be presented at a comfortable reading speed for the viewer. Moreover, chucking or segmentation, i.e. the way a subtitle is split across the screen, has a great impact on comprehension. Studies have shown that a proper segmentation can balance gazing behaviour and subtitle reading AUTHOR. Each subtitle should -- if possible -- have a logical completion. This is equivalent to a segmentation by phrase, sentence or unit of information. Where and if to insert a subtitle break depends on several factors such as speech rhythm, pauses but also semantic and syntactic properties. This all makes segmenting a full sentence into subtitles a complex and challenging problem. Developing automatic solutions for subtitle segmentation has long been impeded by the lack of representative data. Line breaks are the new lines inside a subtitle block, which are used to split a long subtitle into two shorter lines. This type of breaks is not present in the subtitle files used to create large subtitling corpora such as OpenSubtitles AUTHOR and corpora based on TED Talks AUTHOR, possibly because of encoding issues and the pre-processing of the subtitles into parallel sentences AUTHOR. Recently, MuST-Cinema AUTHOR, a corpus based on TED Talks, was released, which added the missing line breaks from the subtitle files (.srt}) using an automatic annotation procedure. This makes MuST-Cinema a high-quality resource for the task of subtitle segmentation. However, the size of MuST-Cinema (about 270k sentences) might not be sufficient for developing automatic solutions based on data-hungry neural-network approaches, and its language coverage is so far limited to 7 languages. On the other hand, the OpenSubtitles corpus, despite being rather noisy, constitutes a large resource of subtitling data. In this work, we leverage available subtitling resources in different resource conditions %ask the question of what amount and quality of data is necessary to train models which automatically segment sentences into readable subtitles. The goal is to exploit the advantages of the available resources, i.e. size for OpenSubtitles and quality for MuST-Cinema, for maximising segmentation performance, but also taking into account training efficiency and cost. %address the problem of subtitle segmentation, creating a model that, given a full sentence, generates readable subtitles. We experiment with a sequence-to-sequence model, which we train and fine-tune on different amounts of data. More specifically, we hypothesise the condition where data containing high-quality segmentation decisions is scarce or non-existent and we resort to existing resources (OpenSubtitles). We show that high-quality data, representative of the task, even in small amounts, are a key to finding the break points for readable subtitles.

Conclusion
We have presented methods to combine heterogeneous subtitling data in order to improve automatic segmentation of subtitles. We leverage large data containing noisy segmentation decisions from OpenSubtitles and combine them with smaller amounts of high-quality data from MuSTCinema to generate readable subtitles from full sentences. We found that even limited data with reliable segmentation can improve performance. We conclude that quality matters more than size for determining the break points between subtitles.