Automatic Subtitling and Subtitle Compression: FBK at the IWSLT 2024 Subtitling track (2024)

Abstract: The paper describes the FBK submissions to the Subtitling track of the 2024 IWSLT Evaluation Campaign, which covers both the Automatic Subtitling and the Subtitle Compression task for two language pairs: English to German and English to Spanish. For the Automatic Subtitling task, we submitted two systems each covering one of the two proposed training conditions, namely constrained and unconstrained: i) a direct model, trained in constrained conditions, that produces the SRT files from the audio without intermediate outputs (e.g., transcripts), and ii) a cascade solution that integrates only free-to-use and freely trained components, either taken off-the-shelf or developed in-house. Results show that, on both language pairs, our direct model outperforms both cascade and direct systems trained in constrained conditions in last year’s edition of the campaign, while our solution assemblying pre-trained models is competitive with the best 2023 systems, although they were finetuned on task specific training data. For the Subtitle Compression task, our primary submission involved prompting a Large Language Model in zero-shot mode to shorten subtitles that exceed the reading speed limit of 21 characters per second. Our results highlight the challenges inherent in shrinking out-of-context sentence fragments that are automatically generated and potentially error-prone, underscoring the need for future studies to develop targeted solutions.

heading: 1 Introduction
text: In response to the growing amount of audiovisual content produced every day, the task of automatically generating subtitles has seen increasing attention (Álvarez et al., 2015; Vitikainen and Koponen, 2021), with the goal of fostering the accessibility of the material by overcoming language barriers. In light of this, starting from the 2023 edition, the IWSLT Evaluation Campaign includes the Automatic Subtitling task, in which participants had
to generate well-formed subtitles in German and Spanish starting from the corresponding English audio (Agarwal et al., 2023). In addition to requiring high-quality translations of the audio content, correct subtitles also need the translated text to be split into blocks (each of them possibly split into 2 lines) in a way that minimizes the users’ cognitive effort (Bogucki, 2004; Khalaf, 2016; Cintas and Remael, 2021), and these blocks have to be presented on-screen with the correct timing, i.e. in sync with the original audio.
Although there is no absolute rule to determine the cognitive effort required to read a subtitle, typical constraints to keep it low include: i) not having more than 2 lines per block (LPB); ii) keeping the number of characters per line (CPL) below a given threshold, which was set to 42 in the IWSLT 2023 campaign; and iii) avoiding excessive reading speed expressed in the number of characters per second (CPS) to be read by the user, which was set to 21. Good subtitles should hence be displayed in text blocks that conform to these rules, and their adherence to the constraints can be measured as the percentage of blocks compliant with them. Since automatic subtitling systems can fail in fully matching all the above constraints, the IWSLT 2024 campaign introduced an additional Subtitle Compression sub-task,1 which requires to reduce the number of characters in each block of pre-generated subtitles to an extent that satisfies the reading speed constraint, without compromising its semantic content.
This paper describes FBK’s submissions to both tasks (Automatic Subtitling and Subtitle Compression) of the IWSLT 2024 Subtitling track. Our submitted systems cover both language directions under evaluation, namely English-German (en-de) and English-Spanish (en-es).
Regarding Automatic Subtitling, we explored
1https://iwslt.org/2024/subtitling
134
two approaches that led to two submissions, one for each training condition, constrained and unconstrained. On the one hand, following the promising results obtained by the first direct models for automatic subtitling (Papi et al., 2023a), we trained a direct subtitling model (§2.1) in constrained conditions, i.e. using only the data allowed by the organizers for this setting. We call this model direct as it generates the subtitles in the target languages (including block and line delimiters) as well as timestamps without any intermediate discrete content representation, such as textual transcripts of the audio. In this respect, it is different from the two direct models submitted in the 2023 edition as both required the generation of intermediate transcripts for the timestamps estimation, either by using an auxiliary automatic speech recognition (ASR) system (Bahar et al., 2023) or by using auxiliary modules of the direct speech translation (ST) system (Papi et al., 2023b). On the other hand, we created a pipeline system (§2.2) within the AI4Culture2 EU project, which binds us to use only code and models released under licenses as permissive as possible. Lastly, our primary submission to the newly proposed Subtitle Compression task (§2.3) tackled the problem with an LLM-based approach. To this aim, we explored a first basic solution by prompting the model in zero-shot mode to shorten candidate hypotheses exceeding the 21 CPS limit, and compared it with simpler, word/character deletion strategies.

heading: 2 Systems Description
text: In this section, we first describe the direct (§2.1) and cascade (§2.2) Automatic Subtitling systems, and then our Subtitle Compression submissions (§2.3).

heading: 2.1 Direct Subtitling with SBAAM
text: Our direct subtitling system is based on an encoderdecoder architecture, made of a 12-layer Conformer3 encoder (Gulati et al., 2020) and a 6-layer Transformer decoder (Vaswani et al., 2017). It is trained to predict the translation in the target language with end of line (<eol>) and end of block (<eob>) delimiters to learn both to translate and segment into subtitle units. Moreover, we add a Connectionist Temporal Classification (CTC) on
2https://pro.europeana.eu/project/ai4culture-an-aiplatform-for-the-cultural-heritage-data-space
3We use the padding-safe implementation tested with pangolinn by Papi et al. (2024).
target module (Yan et al., 2023) on top of the encoder that is trained with the same target as the autoregressive Transformer decoder. In addition, to reduce the computational cost of our model, we include a CTC compression module in the 8th encoder layer (Gaido et al., 2021). This module is trained to predict the transcription of the audio, but no transcript is generated at inference time and the module only averages similar vectors without producing any textual representation of the source.
The end-to-end training is realized with a composite loss (L) that sums the label smoothing crossentropy (CE) loss (Szegedy et al., 2016) on the decoder outputs with the CTC loss of the CTC on target module, and the CTC loss of the CTC compression module. By defining t as the transcript of an audio sample, and x and y as the target translation augmented with <eob> and <eol> delimiters, we can formalize the loss as:
L = λ1 CTC(h8, t) + λ2 CTC(h, y) + λ3 CE(D(h, y), y)
where λ1,2,3 control the relative weight of the losses, h8 is the output of the 8th encoder layer, h is the encoder output, and D is the Transformer decoder. In our experiments, we follow the indication of (Yan et al., 2023) and set (λ1, λ2, λ3) to (1.0, 2.0, 5.0).
The inference phase, instead, combines only the probabilities predicted by the CTC on target module and by the decoder, following the joint CTC/attention framework with CTC rescoring (Watanabe et al., 2017; Yan et al., 2023). This method involves rescoring the next-token probabilities produced by the decoder using the probabilities of the candidate prefixes obtained from the CTC on target module (TgtCTC):
p = pD(yi|h, y0,...,i−1) + αpTgtCTC(y0,...,i|h)
where α is a hyperparameter that controls the weight of the CTC rescoring.
The output of this inference is the translated text with subtitle boundaries. As such, we still miss a key element for subtitles: the start and end timestamps of each block, which control how long and when they have to be displayed on the screen. To estimate them, we rely on the Speech Block Attention Area Maximization (SBAAM) method (Gaido et al., 2024). SBAAM leverages the encoder-decoder attention to create alignments
between the generated subtitles and the source audio, as done in many works both in text-to-text scenarios (Tang et al., 2018; Zenkel et al., 2019; Garg et al., 2019; Chen et al., 2020) and, more recently, speech-to-text ones (Papi et al., 2023c; Alastruey et al., 2023). In fact, SBAAM first applies a meanstandard deviation normalization to the attention matrix on the text axis (clipping all negative values to a small −ϵ quantity to avoid penalizing in different ways unnecessary areas). Then, for each block boundary (<eob>) in the generated text, it iteratively determines the timing of the <eob> by selecting the splitting point that maximizes the area of the current block with the audio up to that point and the remaining blocks with the rest of the audio.
Once all the <eob>s in the output have been processed, all blocks will have start and end timings.
Experimental Details. The input of our models is represented by 80 Mel-filterbank features extracted every 10 ms with a window of 25 ms. The input features are then processed with two 1D convolutional layers with stride 2 that reduce the input length by a factor of 4. We use 512 for the encoder and the decoder embedding dimensions and 2048 hidden features in the feed-forward layers. The vocabularies are based on unigram SentencePiece (Kudo, 2018), with size 8,000 for the English source and 16,000 for the target (either German or Spanish). The total number of parameters of our models is 133M. The final models are obtained by averaging the last 7 checkpoints obtained from the trainings, which are performed on 4 NVIDIA Ampere GPU A100 (64GB VRAM). At inference time, when long unsegmented audios have to be subtitled, the audio is first segmented into smaller audio chunks with SHAS4 (Tsiamas et al., 2022). The code used to create the models is available at: https://github.com/hlt-mt/FBK-fairseq.
Training Data. The models are trained on most of the datasets admitted for the “constrained” submission type. These include all the available ST corpora, namely MuST-Cinema (Karakanta et al., 2020), EuroParl-ST (Iranzo-Sánchez et al., 2020), and CoVoST v2 (Wang et al., 2020). Also, we leverage most of the available ASR datasets (CommonVoice (Ardila et al., 2020), LibriSpeech (Panayotov et al., 2015), TEDLIUM v3 (Hernandez et al., 2018), and VoxPopuli (Wang et al., 2021)), by automatically translating the transcripts into the
4https://github.com/mt-upc/SHAS
two target languages using the NeMo MT models.5 <eol> and <eob> tags are added to both transcripts and translations of all datasets, except for MuST-Cinema that already include them, using the multimodal segmenter by Papi et al. (2022).

heading: 2.2 Cascade Subtitling
text: As stated in the introduction, within the EU AI4Culture project, we developed a cascade subtitling system combining free-to-use components only. Most of them are taken off-the-shelf, while others were developed in-house. The entire system is publicly available at https://github.com/ hlt-mt/FBK-subtitler.
The pipeline is shown in Figure 1 and concatenates the following modules:
Audio segmenter: Speech recognition and speech translation models are unable to process long audios, which then have to be split into shorter segments. As in the direct architecture, here too SHAS is used to carry out this task. It is worth noting that, in general, each audio segment contains multiple subtitles. SHAS code and models are released under the very permissive MIT license.
Speech recognition system: To transcribe the input speech, we opted for Whisper6 (large-v3) to date one of the best ASR systems covering English, licensed under the MIT license. Whisper generates transcripts already split in subtitles, each supplied with start and end timestamps. However, two main issues can affect Whisper’s outputs: hallucinations and lack of segmentation in lines, both handled by specific modules of the pipeline.
Hallucination removal filter: It removes hallucinations, a well-known concern of LLMs, which refers to the generation of text that is erroneous, nonsensical, or detached from reality. Here, only shallow hallucinations are considered, i.e. those involving the syntax of subtitles but not their semantics. We observed two types of shallow hallucinations, within and across subtitles. The first type refers to the repetition of single words or short n-grams many consecutive times within a subtitle. The second type refers to instances where the same transcript is repeated an anomalous number of times across consecutive subtitles. We implemented a script which heuristically detects and
5Publicly available at: https://docs.nvidia.com/ deeplearning/nemo/user-guide/docs/en/main/nlp/ machine_translation/machine_translation.html
6https://github.com/openai/whisper
removes such phenomena from subtitles; in the pipeline, it is used downstream of both the ASR and the MT models.
Machine translation system: It performs the translation of a text (here: the text in each subtitle generated by Whisper, amended by hallucinations) from a source language into the target language. Various freely usable pre-trained LLMs have been tested in a preliminary investigation, namely NLLB,7 mBART-50,8 Helsinki Opus-MT.9 The outcomes indicated the Helsinki Opus-MT as the best performer. Code and models are released under the MIT license.
Text segmenter: In general, its goal would be splitting the input text into fragments suitable, in terms of both quality and compliance to spatio-temporal constraints, to be displayed on the screen. However, since here the goal is solely to split too long, single line subtitles generated from the previous stages of the pipeline into two lines, we implemented a script that splits subtitles longer than 42 characters into two lines rewarding: the compliance of both lines with the 42-character limit, a similar length of the two lines, and the presence of a punctuation mark at the end of the first line.

heading: 2.3 Subtitle Compression
text: The newly introduced Subtitle Compression task required participants to rephrase subtitles provided by the task organizers that did not comply with the reading speed constraint of 21 CPS.
The material to be automatically processed was presented to participants as standard SRT (SubRip File Format) files that include: i) the text of sequentially numbered subtitles, which can be ei-
7https://github.com/facebookresearch/fairseq/ tree/nllb/
8https://huggingface.co/facebook/ mbart-large-50
9https://huggingface.co/models?sort=trending& search=Helsinki-NLP
ther one or two lines, and ii) timing information for each subtitle (i.e. timestamps in the format hours:minutes:seconds,milliseconds), indicating how long the subtitle should stay on the screen. As per the task guidelines, the goal was to exclusively work at the text level, compressing subtitles’ text when necessary and without modifying the time boundaries. To achieve this, given the lack of indications on which automatic subtitles needed correction, we relied on the subtitle compliance script also provided by the task organizers. This allowed us to reliably identify the subtitle candidates requiring text compression and focus exclusively on rephrasing them.
The identified subtitles (39.8% and 30.0% of the total for en-de and en-es, respectively) underwent the compression phase, for which we devised two strategies. The first one, selected for our primary submission, is user-oriented: its goal is to target the CPS constraint with an LLM-based, fluencydriven approach aimed at preserving the readability of the compressed subtitles and, in turn, user experience. The second strategy, selected for our contrastive submissions, is more metric-oriented. Its goal is to shorten non-CPS-compliant subtitles by removing function words with varying levels of aggressiveness.
User-oriented approach (GPT – primary). Our LLM-based compression approach exploits GPT-4 (OpenAI, 2024) (model gpt-4-0613, with default parameters except for the temperature, which we set to 0), which was prompted in zero-shot mode with the instruction: “Shorten this [LANGUAGE] text to a maximum of [TARGET_NUMCHARS] characters while preserving the original words as much as possible: [TEXT]”, where:
• LANGUAGE indicates the language of the subtitle, either “German” or “Spanish”;
• TARGET_NUMCHARS specifies the maximum al-
lowed length for the compressed subtitle, measured in characters including spaces. The target value is calculated based on the total on-screen time of the subtitle, which is determined by subtracting its start time from its end time and then multiplying this duration by 21 (e.g., with 3.2 seconds of on-screen time, TARGET_NUMCHARS is 67.2, truncated to 67);
• TEXT is the original subtitle that needs to be compressed.
The choice of the overall approach was driven by the aim to preserve the user experience by leveraging the generation capabilities of large language models. In fact, simpler and more aggressive methods, such as the metric-oriented ones presented in the next paragraph, can easily improve the rate of subtitles compliant with the CPS limit but at the cost of losing important information and detracting their readability. In an opposite direction, our LLM-based approach aims to strike a balance between improving CPS values and retaining the original information through targeted and meaningpreserving rephrasing.
Our zero-shot prompting strategy was primarily driven by fast-development reasons. In fact, we expect significant improvements by feeding the model with exemplars, i.e., via in-context learning (Brown et al., 2020). We opted for a simpler, cheaper, and more conservative approach to establish a starting point and a reference baseline for future in-depth comparative experiments. For similar reasons, we opted for a solution that concentrates on individual subtitles instead of operating on full sentences. Though likely more effective, letting the LLM reformulate full sentences in a shorter way would have introduced the additional burden of rearranging the resulting content into timed subtitles afterward. This is certainly a promising direction for future improvements.
Metric-oriented approach (Del_* – contrastive). For our contrastive submissions, we designed “metric-oriented” solutions that aim to improve CPS by aggressively reducing the length of subtitles through simple character or word deletions. The goal was to measure the extent to which this baseline approach affects the readability of subtitles. Along this direction, we explored a range of options which share the common trait of removing from the non-CPS-compliant subtitles specific categories of function words iden-
tified from pre-compiled lists downloaded from the web.10 Word removal is carried out with varying levels of aggressiveness, ranging from i) the deletion of articles (Del_articles) to ii) the deletion of articles, prepositions, and adverbs (Del_art/prep/adv), and iii) the deletion of all function words (Del_all-func-wrds). On the one side, these strategies avoid the loss of important content in the original subtitles and the presence of incomplete words in the output, as it happens in the Baseline approach proposed by the task organizers. On the other side, they intervene in the syntactic structure of the subtitles, altering them in a way that improves CPS but penalizes both readability and automatic evaluation with reference-based metrics.

heading: 3 Results
text: As a recap, FBK submitted the following runs:
Automatic Subtitling task
• Primary run in Constrained condition: FBKdrct24 (§2.1) • Primary run in Unconstrained condition: FBKcscd24 (§2.2)
Subtitle Compression task
• Primary run: GPT (§2.3, paragraph “User-oriented approach”)
• Contrastive1 run: del all func wrds (§2.3, “Metric-oriented approach”)
• Contrastive2 run: del art/prep/adv (§2.3, “Metric-oriented approach”)

heading: 3.1 Automatic Subtitling
text: Results on subtitling task are provided in Tables 1, 2, and 3. Table 1 compares the SubER (Wilken et al., 2022) scores,11 the primary metric of the task, computed on the subtitles of the development set generated by our systems and by the best systems at IWSLT 2023 in constrained and unconstrained conditions. Table 2 shows global results, i.e., on subtitles of all domains, on test23 of our runs as provided to us by organizers, and of the best primary runs at IWSLT 2023, as published in (Agarwal et al., 2023). Table 3 gathers results, global and on each domain, on test24 of our runs
10https://github.com/Yoast/javascript/tree/ develop/packages/yoastseo/src/researches
11When we do state otherwise, we compute SubER without casing and punctuation, as done in the previous evaluation campaign for the sake of fair comparison with previous scores.
as provided to us by organizers. Besides SubER that measures overall subtitle quality, Table 2 and Table 3 include BLEU (Papineni et al., 2002), ChrF (Popović, 2015) and TER (Snover et al., 2006) for translation quality and CPS, CPL and LPB conformity12 for subtitling guideline compliance.
By looking at SubER scores of Table 1 and Table 2, we notice that our direct system outperforms not only the best direct system submitted last year but also the best cascade in constrained conditions. This superiority is consistent over all domains and language pairs. Also, focusing on Table 2, this is confirmed by all the translation quality metrics on test2023. In the unconstrained setting, instead, the results are less clear. Our cascade system achieves a lower (hence, better) SubER than the unconstrained submissions from last year on the en-es section of test2023 while, on the en-de section, it has a higher SubER than AppTekcscd23 , in contrast with the definitely higher translation quality scores.
Back to the comparison between our direct constrained system and our cascade unconstrained solution, we notice consistent trends over all the evaluation sets (validation, test2023, test2024). The direct system achieves better scores on the TED domain, which is the only one covered by the training data allowed for the constrained setting, but falls behind by a large margin on the other two (ITV and PELOTON), especially on en-es. This result is not surprising as the unconstrained system has been trained on a wide range of domains and is therefore more robust to domain shifts. Regarding subtitle compliance, interesting trends emerge: the cascade system has higher CPL compliance (∼+3% across all settings), while the direct system outperforms it in terms of CPS compliance (+6-12%). The latter aspect may be motivated by the direct access to the source audio of the direct system (which is also guided by the CTC module that directly maps the audio sequence to the textual output).

heading: 3.2 Subtitle Compression
text: The results for the subtitle compression task are reported in Table 4 in terms of BLEURT and CPS (as a measure of reading speed compliance). BLEURT results are computed in two ways, either considering the provided subtitles as references or by using the actual subtitle references. The former results
12Computed with the script provided by Papi et al. (2023a): https://github.com/hlt-mt/FBK-fairseq/ blob/master/examples/speech_to_text/scripts/ subtitle_compliance.py
serve as a proxy of translation quality, as well as a way to measure the distance between the original subtitles to be modified and the resulting modified ones (i.e. as an indicator of how radical the applied changes are). The latter ones, instead, provide real translation quality measurements. For the sake of discussion, the table includes the results of the Baseline as provided by the task organizers and those of an unsubmitted metric-oriented solution (Del_articles), besides those of our official primary (GPT) and contrastive submissions (Del_all-func-wrds and Del_art/prep/adv).
Overall, the scores for the two languages indicate different levels of difficulty but exhibit similar trends. Specifically, en-es appears to be an easier direction, as indicated by higher translation quality (BLEURT) and reading speed compliance scores (CPS) compared to en-de. Unsurprisingly, the BLEURT scores computed against the provided original subtitles (i.e., vs. [1]) are significantly higher than those computed against the actual references (vs. [0]). This indicates the tendency of the proposed methods to apply rather conservative changes. This holds particularly for the metricoriented approaches (Del_*), which are actually designed to do so. Still, the relatively high BLEURT results of the user-oriented approach (GPT) are a symptom of local and rather moderate changes, which likely do not suffer from major issues related to hallucinations and/or under-generation into too short subtitles. Regarding the BLEURT scores computed against the actual subtitle references (i.e., vs. [0]), the results drop significantly, attesting that a large quality gap between all methods and human subtitles still exists. Interestingly, however, the gap between metric and user-oriented approaches shrinks on en-es and even disappears on en-de, where GPT achieves results that are substantially equivalent to those of Del_art/prep/adv.
For both languages and evaluation conditions the higher conservativeness of metric-oriented approaches is not sufficient to yield acceptable CPS results. First, the least aggressive one (the unsubmitted Del_articles), which consistently achieves the highest BLEURT computed on the provided references, is definitely the worst one in terms of CPS. Second, also the other ones (our contrastive submissions Del_art/prep/adv and Del_all-func-wrds) attain lower reading speed conformity compared to the LLM-based useroriented approach. Aimed to strike a balance between translation quality and CPS conformity, our
primary submission (GPT) consistently achieves the best CPS scores (84.81 for en-de, 90.66 for en-es). Paired with the above observations about translation quality, these results suggest that LLM-based approaches to subtitle compression are a promising direction for future explorations.
The trade-off between BLEURT and CPS is further highlighted by the plot in Figure 2 where, between the two extremes represented by Provided ([1]) and Baseline ([2]) subtitles, the subtitles generated through metric-oriented strategies ([4] and [5]) are placed according to a nearly linear relationship. The exception are GPT’results which slightly deviate from this linear trend, as a confirmation of our intuition: generative, user-oriented strategies are capable to perform pinpointed text reductions to pursue CPS compliance without a catastrophic loss of the original subtitles’ meaning.
Overall, our results indicate that, even though it is a sub-task of a very complex problem such as automatic subtitling, subtitle compression has its own difficulties. On the one hand, the generative approach based on LLMs is intuitively promising because, unlike rough trimming strategies that are incompatible with the user experience, it targets a compression that is respectful of the subtitles’ semantic content. On the other hand, however, this approach faces the challenge of reformulating text material that is potentially error-prone and often does not come in the form of well-formed sentences but rather as text spans representing sentence portions or words spanning contiguous phrases. At least in the zero-shot prompting modality, the combination of these two aspects makes the task extremely challenging for LLMs. As a matter of fact, upon preliminary analysis of the generated compressions, LLMs often reveal a tendency to generate sentence-like outputs, attempting to “complete” their generations with hallucinated content,
a behavior that can only be exacerbated in the presence of errors in the subtitle to be compressed. The opposite potential issue, represented by “overcompressing” the subtitle beyond the allowed number of characters, is rarely observed.

heading: 4 Conclusions
text: We presented the FBK’s submissions to the Automatic Subtitling and Subtitle Compression tasks of the IWSLT 2024 Evaluation Campaign. For Automatic Subtitling, we proposed two systems: a direct model trained under constrained conditions and a cascade architecture integrating free-to-use components. Our direct model showcased superior performance compared to constrained direct and cascade submissions of the last year. The cascade solution proved competitive with top-performing unconstrained and fine-tuned 2023 runs. For Subtitle Compression, our primary submission exploits GPT in zero-shot prompting mode to shorten subtitles exceeding the reading speed limit of 21 CPS. While promising, this approach revealed the complexities of compressing out-of-context automatically generated sentence fragments, underscoring the necessity for further research in this area.

heading: Acknowledgments
text: The work presented in this paper is co-funded by the European Union under the project AI4Culture: An AI platform for the cultural heritage data space (Action number 101100683). Marco Gaido is supported by the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. We acknowledge the CINECA award (MAGIS) under the ISCRA initiative, for the availability of highperformance computing resources and support.

heading: Future Work
text: Marco Gaido is supported by the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. We acknowledge the CINECA award (MAGIS) under the ISCRA initiative, for the availability of highperformance computing resources and support