SBAAM! Eliminating Transcript Dependency in Automatic Subtitling (2024)

Abstract: Subtitling plays a crucial role in enhancing the accessibility of audiovisual content and encompasses three primary subtasks: translating spoken dialogue, segmenting translations into concise textual units, and estimating timestamps that govern their on-screen duration. Past attempts to automate this process rely, to varying degrees, on automatic transcripts, employed diversely for the three subtasks. In response to the acknowledged limitations associated with this reliance on transcripts, recent research has shifted towards transcription-free solutions for translation and segmentation, leaving the direct generation of timestamps as uncharted territory. To fill this gap, we introduce the first direct model capable of producing automatic subtitles, entirely eliminating any dependence on intermediate transcripts also for timestamp prediction. Experimental results, backed by manual evaluation, showcase our solution’s new state-of-the-art performance across multiple language pairs and diverse conditions.

heading: 1 Introduction
text: Subtitling aims to facilitate the accessibility of audiovisual media, such as movies, TV shows, and video lectures, by providing users with a textual translation of spoken content. Subtitles consist of two components: a textual block, typically encompassing one or two lines, and its corresponding time duration, indicated by start and end timestamps. To ensure effective on-screen presentation and minimize users’ cognitive load, subtitles should conform to spatio-temporal constraints (Bogucki, 2004; Khalaf, 2016). These include restrictions on the maximum number of characters per line and a display duration that guarantees synchronization with the video while granting viewers sufficient time to read the entire text.
Automating the task involves addressing three main subtasks: translation of the spoken content, segmentation of the translated text into
blocks and lines, and estimation of the timestamps for each block. Early approaches to automatic subtitling (AS) adopted a cascade architecture (Piperidis et al., 2004; Oliver Gonzalez, 2006; Alvarez et al., 2017; Bojar et al., 2021), i.e. a pipeline of components, including automatic speech recognition (ASR) and machine translation (MT) models. In these systems, transcripts served as the foundational element for all three subtasks, despite the well-documented limitations of this reliance on them, such as error propagation (Sperber and Paulik, 2020), the loss of useful prosody information (Lakew et al., 2022; Tam et al., 2022), inapplicability to source languages lacking written forms (Lee et al., 2022), and higher computational and environmental cost (Strubell et al., 2019) due to the need to run multiple models.
To cope with these limitations, subsequent studies aimed to streamline the subtitling pipeline by reducing its dependency on transcripts. In this endeavor, direct speech-to-text translation (ST) systems (Bérard et al., 2016; Weiss et al., 2017), capable of translating speech without recourse to intermediate symbolic representations, were successfully used by Karakanta et al. (2020a) for the translation step (subtask ). For subtitle segmentation (subtask ), efforts focused on adapting MT (Etchegoyhen et al., 2014; Bywood et al., 2013; Volk et al., 2010; Matusov et al., 2019; Koponen et al., 2020b; Cherry et al., 2021) and language models (Ponce et al., 2023) to directly produce translations that incorporate block and line boundaries as specific tags, also by exploiting audio information (Papi et al., 2022).
To date, compared to subtasks and , timestamp estimation (subtask ) has received much less attention. This regards not only the elimination of intermediate transcription steps, the primary focus of this work, but also evaluation, as there is still no reliable and informative metric for directly assessing the quality of the predicted timestamps.
3673
As for the task itself, current approaches, including those exclusively based on direct ST models (Papi et al., 2023a,b; Bahar et al., 2023), still require transcripts for timestamp estimation, which involves i) generating captions (i.e., segmented transcripts), ii) estimating timestamps for caption blocks, typically through a Connectionist Temporal Classification (CTC) loss (Graves et al., 2006), and iii) projecting them onto target subtitles. Regarding automatic evaluation, current metrics (Wilken et al., 2022) are by design holistic and therefore inadequate to precisely measure timestamp estimation quality.
To bridge these gaps, we introduce and evaluate the first fully end-to-end AS solution1 that seamlessly produces both subtitles (i.e., segmented translations) and their timestamps without any reliance on intermediate transcripts. This innovation is realized by incorporating into our model the capability to directly determine the temporal alignment between the spoken utterances and their corresponding translation in subtitle form. Along this direction, our contributions are the following:
• We propose two methods for timestamp estimation (§3), respectively based on applying the CTC loss directly on translations (Zhang et al., 2022a), and on estimating the audio-text temporal alignment from the attention mechanism (Papi et al., 2023c; Alastruey et al., 2024). Both approaches are complemented by the joint CTC decoding (Watanabe et al., 2017), which yields higher translation quality and a more precise alignment between the generated text and the corresponding audio (Yan et al., 2023);
• We introduce SubSONAR,2 a novel metric for 1All our code and pre-trained models are available at https://github.com/hlt-mt/FBK-fairseq/ under Apache Licence 2.0.
2Available at https://github.com/hlt-mt/subsonar/ under Apache License 2.0 and on PyPi (https://pypi.org/ project/SubSONAR/).
evaluating timestamp quality (§4), which is based on SONAR (Duquenne et al., 2023) and designed to be sensitive to time shifts, enabling a focused evaluation of timestamps; • We validate our approach through comparative experiments (§5) on 7 language pairs, 2 data conditions, and 4 domains, achieving new state-ofthe-art results on different benchmarks and demonstrating better performance compared to cascade architectures for automatic subtitling; • We verify the efficacy of our model through manual evaluation (§6), attesting a significant reduction in timestamp adjustments of ∼24% compared to the previous state of the art.

heading: 2 Direct Model for Automatic Subtitling
text: Following previous work on direct ST (Liu et al., 2020; Xu et al., 2021a, 2023; Wu et al., 2023), we build a direct autoregressive encoder-decoder model, where the encoder is composed of three blocks: i) an acoustic encoder – made of two 1D convolutions with stride 2 and Na Conformer (Gulati et al., 2020) layers, ii) a length adaptor – a CTC Compression (Gaido et al., 2021) module that averages the vectors corresponding to the same CTC prediction, and iii) a semantic encoder – made of Ns Conformer layers. The encoder output is then fed to an autoregressive decoder D and, in parallel, to a CTC on Target (TgtCTC) module (Zhang et al., 2022a; Yan et al., 2023). The full architecture is shown in Figure 1.
We train our model with a composite loss (L), which is a linear combination of a label smoothing cross-entropy (CE) loss (Szegedy et al., 2016) on the decoder D, a CTC loss on the TgtCTC module, and a CTC loss on the CTC Compression module:
L = λ1 CTC(ha, t) + λ2 CTC(hs, y) + λ3 CE(D(hs, y), y)
where λ1,2,3 are the loss weights, ha is the acoustic encoder output, hs is the encoder output, y is the target subtitle, and t is the caption. It is noteworthy that the caption (t), albeit optionally used for training, is not strictly required by our model. Additionally, captions are neither generated nor utilized by our novel timestamp estimation methods (§3), which exclusively rely on the generated subtitles. Indeed, the auxiliary CTC loss on captions and the CTC compression module can be entirely omitted, at the cost of a slight reduction in quality (Zhang et al., 2022a), and captions can be replaced with speech units (Hsu et al., 2021; Zhang et al., 2022b) as the module target. However, we opt to retain the CTC compression module (as well as the related auxiliary CTC loss on captions), not only for its benefits in terms of computational efficiency and translation quality but, most importantly, to enable the use of the same model with earlier timestamp estimation solutions, which require a CTC module with captions as the target (Bahar et al., 2023; Papi et al., 2023a). This allows a direct comparison between different timestamp estimation methods, employing the same model and generated subtitles.
At inference time, subtitles are generated by estimating their likelihood with the joint CTC/attention framework with CTC rescoring (Yan et al., 2023) through a linear combination of the probabilities of the TgtCTC module and the decoder:
p = pD(yi|hs, y0,...,i−1) + αpTgtCTC(y0,...,i|hs)
where α is a hyperparameter.
Multilinguality. We experiment both in bilingual (English to a target language) and in multilingual (English to many languages) settings. In the latter case, we prepend a learned language embedding to the previous output tokens to be fed to the decoder (Inaguma et al., 2019; Wang et al., 2020a). In addition, we sum the same learned embedding to all the vectors of the encoder output (hs) before processing them with the TgtCTC module, to inform the module about the language to generate.3
3We explored alternative solutions: adding the language embedding at the beginning of the encoder before all Conformer layers, as per (Di Gangi et al., 2019); adding it to the vectors obtained after the CTC compression module; multiplying the language embedding vector instead of summing it; using separate learned language embeddings for the decoder and the encoder. However, none of them led to better results.

heading: 3 Estimation of Block Timestamps
text: In this section, we present novel methods for estimating the block timestamps without requiring the transcripts. These solutions not only avoid error propagation and are applicable to unwritten source languages but also exhibit increased speed compared to current methods, skipping the transcript generation step, thereby minimizing significant overhead. Our methods involve either directly applying the CTC-segmentation algorithm (Kürzinger et al., 2020) to the CTC on Target module (§3.1) or leveraging the encoder-decoder attention scores to find the audio-text alignment (§3.2).

heading: 3.1 Subtitle CTC-based Estimation
text: The error propagation and approximations caused by the cross-lingual Levenshtein alignment proposed by Papi et al. (2023b) can be avoided by directly estimating the timestamps on the target, i.e., on the generated subtitles. This is realized by exploiting the predictions of the TgtCTC module placed on top of the encoder, which are used by the CTC segmentation algorithm (Kürzinger et al., 2020), together with the input audio, to retrieve the timestamp information. This subtitle CTC-based method (SubCTC) enables the direct alignment between the input audio features (representing the time over the speech sequence) and the boundaries of the generated subtitle blocks, eliminating the need for intermediate transcript alignments with the audio and their projection into the target side.
The SubCTC method builds upon the assumption that the CTC module can learn meaningful alignments between source audio and target texts, as it does when predicting transcripts (Sak et al., 2015; Sainath et al., 2020; Chen et al., 2023) without direct supervision. However, the validity of this assumption has not been verified yet. Rather, related works in non-autoregressive translation showed the inability of such models to implicitly learn complex word reordering (Chuang et al., 2021; Ran et al., 2021; Xu et al., 2021b). Motivated by this potential issue, we devise an alternative method that leverages the cross-attention matrix to infer source-target alignments.

heading: 3.2 Attention-based Estimation
text: Building on the extensive literature in text-to-text translation and language modeling discussing the quality of source-target alignments learned by the attention mechanism (Tang et al., 2018; Zenkel
Algorithm 1 Attention-based DTW 1: N,L← nTokens, audioLen 2: B ← blockIdxs ▷ List of block boundary indices 3: A← attnMatrix ▷ A ∈ RN×L 4: A←StdNorm(A, axis = 0) ▷ Column-wise normalize 5: A←MedianFilter(A,width = 7, axis = 1) 6: D ←DTWDistance(−A) ▷ Negated attentions as costs 7: n, l, P, blockT imings← N,L, [(N,L)], [ ] 8: while P [−1] ̸= (0, 0) do ▷ Backtrack best DTW path 9: if n ∈ B then 10: P ← Append(P, (n− 1, l − 1)) 11: blockT imings← Append(blockT imings, l) 12: else 13: P ← Append(P, argmin
i∈{n,n−1},j∈{l,l−1} D[i, j])
14: end if 15: n, l← P [−1] 16: end while 17: return reverse(blockT imings)
et al., 2019; Garg et al., 2019; Chen et al., 2020) and recent works on speech-to-text (Papi et al., 2023c; Alastruey et al., 2024), we propose to exploit the encoder-decoder attention scores to determine subtitle block timings and devise two different modalities of leveraging this information. For the first one, we adapt the Dynamic Time Warping (DTW) algorithm (Sakoe and Chiba, 1978), which is commonly used to determine token-level timestamps in ASR (§3.2.1).4 For the second one, recognizing that DTW relies on the assumption of monotonic source-target alignments, which does not always hold in our case, we introduce a new algorithm named SBAAM (SPEECH BLOCK ATTENTION AREA MAXIMIZATION), specifically designed for subtitles (§3.2.2).

heading: 3.2.1 Attention-based DTW for Subtitling
text: The DTW algorithm is a dynamic programming algorithm similar to Viterbi (Juang, 1984), which finds the best path (i.e., the one that minimizes a distance function) between two temporal sequences with varying speeds. This algorithm operates under the assumption that all the elements of the two sequences are aligned and that the mapping between the two sequences is monotonic. In our case, while the assumption may not be applicable at the token (or word) level, it remains valid at the block level, where the order has to be maintained; we therefore investigate its applicability to our task.
We use the additive inverse of the attention matrix as a distance function and follow the improved
4For example, the DTW is used in the Transformers implementation of Whisper (Radford et al., 2023): https://github.com/huggingface/transformers/ blob/v4.34.0/src/transformers/models/whisper/ modeling_whisper.py#L1817.
Algorithm 2 SBAAM 1: N,L← nTokens, audioLen 2: B ← blockIdxs ▷ List of block boundary indices 3: A← attnMatrix ▷ A ∈ RN×L 4: A←StdNorm(A, axis = 0) ▷ Row-wise normalize 5: A[A < 0]← −ϵ ▷ ϵ set to 0.01 6: n, l, blockT imings← 0, 0, [ ] 7: for all ib ∈ [0, |B|) do
8: l← argmax j∈(l,L−|B|+ib) ( ∑ A[n : B[ib], l : j]+ ∑
A[B[ib] + 1 : N, j + 1 : L])
9: blockT imings← Append(blockT imings, l) 10: n← B[ib] 11: end for 12: return blockT imings
DTW algorithm by Yuxin and Miyanaga (2011) that applies a 1D median filter (Huang et al., 1979)5 over the speech sequence for each text token, after a standard normalization over the text tokens (see Alg. 1). After computing the accumulated DTW distance matrix, we define the best path in the backtracking phase, with three possible moves: i) moving to the preceding element of the first sequence, ii) moving to the preceding element of the second sequence, or iii) moving to the preceding elements of both sequences. In this phase, we force moving across both (speech and text) sequences when the block boundary token (<eob>) is encountered. This decision is based on two observations: i) a manual inspection of attention matrices revealed that the attention of <eob> tokens is often unreliable (e.g., focused on the last token of the speech sequence or silence), and ii) we want to ensure that at least one speech segment (equivalent to 40ms) is assigned to each block by skipping the <eob> token in the DTW search (see lines 9-12 of Alg. 1). As a result, we use the timings assigned to the <eob> tokens as the temporal boundaries (timestamp) for the corresponding subtitle blocks.

heading: 3.2.2 SBAAM
text: To relax the constraint of alignment monotonicity, we devise a new algorithm to estimate the timing of block boundaries. Intuitively, our method (see Alg. 2) maximizes the attention scores within a rectangular area encompassing the tokens belonging to a block (i.e. all the tokens between two <eob>) along one axis, and the assigned span across the temporal dimension of the speech sequence along the other axis. For this reason, we named it SPEECH BLOCK ATTENTION AREA MAXIMIZATION or SBAAM.
5A filter that takes the median value within a pre-defined window defined as the width of the filter itself.
The SBAAM algorithm encompasses two steps. First, the attention matrix is normalized (following the previous procedure) and all negative values are set to a small negative value (−ϵ). This is required as the attention values are typically peaky in the range [0, 1] due to the softmax, with a few large values and random fluctuations close to 0 for all the others. After normalization, some small values may become very negative, while others may be closer to 0, even though both indicate low attention. Despite this, we maintain them as negative values to penalize the integration of areas with very low attention. Second, for each generated subtitle block (i.e., for each <eob>), we iteratively determine the timing by selecting the splitting point that maximizes the area of the first block with the audio up to that point and the rest of the text with the remaining audio. At the end of this process, we obtain the start and end timing (timestamp) for all blocks.

heading: 4 SubSONAR
text: Since it was proposed, the SubER6 metric (Wilken et al., 2022) has been used to provide a holistic evaluation of subtitles, encompassing translation quality, block/line segmentation accuracy, and timing. Specifically, SubER computes the number of word edits and block/line edits required to match the reference, where hypothesis and reference words are allowed to match only within subtitle blocks that overlap in time. This definition highlights how SubER is sensitive only to major errors in terms of timing, as it solely checks whether two blocks overlap, even if only for a few milliseconds. This limitation motivates our proposal of SubSONAR, a new metric designed to be more sensitive to time shifts and, consequently, more suitable for evaluating the quality of subtitle timestamps.
To this aim, we leverage the multimodal and multilingual SONAR model (Duquenne et al., 2023), designed to generate sentence embeddings within a
6In this work, we compute SubER-cased unless specified otherwise as per (Papi et al., 2023b).
shared multimodal (text and audio) semantic space for all languages. Specifically, we calculate the cosine similarity between the SONAR embeddings of the text within a subtitle block and its corresponding audio, determined by the timestamp of the block. Subsequently, we average the similarity scores across all subtitle blocks, which results in a single score in the [−1, 1] range. Being SONAR trained to capture both text and audio semantics, higher SubSONAR scores indicate better alignment between text and audio content, which is influenced by both translation quality and timestamp accuracy. Nevertheless, as revealed by the empirical validation presented in the following sections (§5,6), SubSONAR exhibits higher sensitivity to timing accuracy than to translation quality.

heading: 5 Automatic Evaluation
text: In this section, we evaluate our solutions automatically through comparative experiments in different resource conditions and language settings. First, we validate the adoption of the joint CTC rescoring and compare the timestamp estimation methods introduced in §3 using a multilingual system trained and tested on all the 7 language pairs (en→{de,es,fr,it,nl,pt,ro}) of MuST-Cinema (Karakanta et al., 2020b) (§5.1). Then, we confirm their strength by comparing two bilingual systems (en-de and en-es) trained in the high resource conditions of the IWSLT 2023 subtitling track (Agarwal et al., 2023) with the results reported for the current state of the art in direct AS and production tools on several test sets (§5.2). Lastly, we demonstrate that our solutions close the gap with the cascade approach, outperforming the best IWSLT cascade models on the 4 publicly available validation sets (§5.3). Full experimental settings and details about training data are given in Appendix A to ensure the reproducibility of our work.

heading: 5.1 Timestamp Estimation Methods
text: Table 1 shows the results of our multilingual system with the proposed timestamp estimation methods, comparing them to the Levenshtein-based approach (LEV) of Papi et al. (2023a). Notice that we employ the same model for all rows, meaning that the outputs vary solely in terms of block timings, except for the ablation row (- joint CTC) where we analyze the impact of the joint CTC rescoring on the subtitle quality.
First of all, we notice a notable SubER decrease with joint CTC rescoring (-0.9 on average), while the different timing strategies have a lower impact on it (improvements remain below 0.4 on average), with the two attention-based methods (ATTN DTW and SBAAM) achieving the same scores. On the contrary, SubSONAR is not significantly affected by the decoding strategy but exhibits increasingly higher scores as block timings become more accurate. The substantial gains (>6% relative improvement) of SubCTC over LEV underscore the importance of directly estimating timestamps on the subtitle blocks, thereby avoiding inherent error propagation of mapping them from the caption blocks (as done in LEV). Consistently with the SubER scores, the attention-based methods lead to superior scores, with SBAAM emerging as the best timestamp estimation strategy, outperforming ATTN DTW by a limited, yet consistent, margin across all languages. Overall, SBAAM improves SubSONAR by 11.8% over the current state of the art (LEV) for direct subtitling.
In addition to certifying the effectiveness of our solutions, these results demonstrate that SubER is more sensitive to translation quality than timestamp accuracy. Comparing the results in Table 1 with those in Table 2, it is evident that the substantial gains achieved through joint CTC rescoring are proportional to the improvements in terms of AS-BLEU,7 which measures the pure translation quality. Notably, the languages exhibiting the
7AS-BLEU, computed with SubER tool (AppTek, 2022), is calculated with the popular sacreBLEU metric (Post, 2018) after aligning blocks of the hypothesis and reference with the minimum Levenshtein distance method (Matusov et al., 2005).
smallest decrease in SubER coincide with those where BLEU scores are closer (it and nl), while the language with the highest BLEU increase (es) also demonstrates the widest margin in terms of SubER. Conversely, timestamp accuracy shows limited effects on SubER scores, despite the important difference between methods, as attested by SubSONAR scores and validated by our manual analysis (§6).
Lastly, we verify the effect of the proposed solutions in terms of subtitle conformity, namely the percentage of blocks that respect character-per-line (CPL) and character-per-second (CPS) limits. Such limits, respectively set to 42 and 21 in TED guidelines,8 ensure that subtitles can be understood by users without excessive cognitive effort. As it has been demonstrated that the joint CTC rescoring solves the end detection problem of purely attentional models and promotes hypotheses with correct length (Yan et al., 2023), we hypothesized that its use could improve both CPL and CPS metrics. However, inconsistent outcomes across various settings and languages (shown in Appendix B) dispute this hypothesis, whereas the attention-based timing methods consistently improve the CPS, with SBAAM being superior also in this respect.

heading: 5.2 High-Resource Conditions
text: To further validate the robustness of our findings, we experiment in high-resource conditions with bilingual systems (en-de and en-es) and compare our solutions with the state of the art. To cover different domains and data conditions besides MuSTCinema, we leverage two additional test sets: European Commission (EC) Short Clips (Papi et al., 2023a), made of multi-speaker short informative videos about various topics with background music, and European Parliament (EP) Interviews (Papi et al., 2023a), interviews with non-verbatim subtitles. The results of our solution are reported in Table 3 and compared with the current best scores published on these test sets.
Upon comparing rows 1 and 3, we observe that our baseline model, even without joint CTC rescoring and replicating the LEV method, outperforms previous results across nearly all test sets, with the sole exception of MuST-Cinema en-es, affirming its competitiveness and reinforcing our findings. Furthermore, we observe consistent trends and similar improvements as in the previous section when applying joint CTC rescoring and employing the
8https://www.ted.com/participate/translate/ subtitling-tips
attention-based DTW and SBAAM methods for timestamp estimation. On average across all languages and test sets, joint CTC rescoring decreases the SubER by 1.3 with negligible effects on SubSONAR, while SBAAM and DTW yield limited improvement (0.3) in SubER but a substantial relative increase in SubSONAR, which amounts to 8.8% in the case of SBAAM that confirms to be the best method overall.
In fact, our proposed solution (last row) emerges as the best one on all test sets, except for EP Interviews en-es, where the best production system reported by Papi et al. (2023a)9 has a better SubER (-1.0). On average, our model is 2.2 SubER better than cherry-picking the best production system for each test set and language, and 1.6 SubER better than our baseline. To further confirm its strength, in the next section, we compare it with the best systems of the last IWSLT campaign.

heading: 5.3 Is the Gap with Cascade Systems Closed?
text: In this section, we compare our high-resource models with the best direct (FBK – Papi et al. 2023b)
9Note that, as we did not test such production systems, they may have been improved at the time of writing this paper.
and the best cascade (AppTek – Bahar et al. 2023) systems of the last IWSLT campaign (Agarwal et al., 2023), in the constrained data condition, meaning the models are trained on the same data as ours. As the references for the official test sets are not public, we present the results on the 4 validation sets released for the campaign in Table 4. For the sake of a fair comparison, we report the SubER with and without casing and punctuation, as Bahar et al. (2023) report the latter.
Even in this testing condition, our systems consistently outperform the others, surpassing even the top-performing cascade model on both language pairs. The superiority of our solution is particularly pronounced in en-de, where it achieves the lowest SubER in all conditions by a large margin (-3.4 SubER on average over the best cascade and -4.2 over the direct). On en-es, our solution and the cascade are, instead, close, with our models emerging on TED and ITV, whereas the cascade models are better on on EPTV and PELOTON. On average, however, our solution results slightly superior in this language pair with a 0.3 SubER reduction. All in all, our experiments evidence that our proposed
solution can effectively close the gap between cascade and direct subtitling systems for the first time.

heading: 6 Manual Evaluation
text: To corroborate the findings from our automatic evaluation, we conducted the first-ever manual evaluation of timestamp quality in subtitling. This evaluation was specifically conducted on the two language pairs addressed in our high-resource experiments (en-de, en-es).
To focus only on timestamp quality, we compared different timestamp estimation techniques on the same automatic translations. Specifically, we selected the outputs of our high-resource systems with joint CTC rescoring and confronted the baseline LEV method and our proposed SBAAM.
For each language pair, the manual evaluation set – on which the two systems were run – is composed of subsets of the EC Short Clips and MuST-Cinema test sets, for a total of 22 videos corresponding to approximately 1 hour of audio (see Appendix C.1 for details on the selected videos).
The evaluation was carried out by two annotators per language pair, who are proficient in English (C1) as well as native speakers or very proficient (C2) in the target language (German or Spanish).10 We instructed the annotators with ad-hoc guidelines (described in Appendix C.2) to adjust the start, end, or both timestamps when the generated subtitles are not synchronized with the speech. To collect this information, we used ELAN (Wittenburg et al., 2006), an audio/video annotation tool commonly employed in the literature (Sloetjes and Wittenburg, 2008). To cope with the inherent assessors’ subjectivity (i.e. being more aggressive/tolerant in deciding whether a timing is acceptable or has to be edited) the outputs of the two timestamp estimation methods were randomly assigned to the two annotators, ensuring that each one worked on all 22 audios and annotated half of the outputs from both methods. To prove the quality of the manual evaluation and better understand the difficulty of the task, for each language pair, 25% of the test set was double-annotated. We calculated Cohen’s Kappa (Cohen, 1960) to measure the inter-annotator agreement on whether a start/end timestamp has to be edited or not. The resulting values – 0.65 for en-de and 0.61 for en-es – indicate a “substantial” agreement (Landis and Koch, 1977), confirming both
10The annotators were paid 22C/h gross, in accordance with the average salary of data annotators (https://www.talent. com/salary?job=data+annotator).
the feasibility of this new evaluation task and the reliability of its outcomes.
Table 5 shows the results of the manual analysis in terms of start/end time shifts (in milliseconds) and percentage of modified timestamps. The superiority of SBAAM over LEV is evident across all metrics, domains, and language pairs. The percentage of modified timestamps is less than half in en-de and 3 times lower in en-es. Given that correcting automatic timestamp errors is a major concern for professionals in their subtitling experience with AI-based systems (Koponen et al., 2020a), the reduced error rates of SBAAM are likely to significantly alleviate post-editing effort. Furthermore, SBAAM exhibits a notably lower average time shift (33-40%), indicating not only less frequent but also less severe errors. The lower average time shift is complemented by a significantly lower standard deviation, which further evidences that SBAAM errors are less likely to be large.
Lastly, in Appendix C.3 we present the results of this manual analysis after excluding edits under 120 ms. This evaluation caters to user experience, as users typically perceive audio-visual stimuli under 120 ms as instantaneous (Efron, 1970), while the annotators – aided by ELAN that shows the text and waveform – made numerous fine-grained timestamp edits (even <20 ms). This evaluation accentuates the differences between the two methods and shows that SBAAM requires editing for only 12% of timestamps in en-de and 9% in en-es.
We can conclude that our manual analysis not only confirms the results obtained through the automatic SubSONAR metric, but also further highlights the efficacy of our proposed timestamp estimation method and its substantial superiority over previous solutions.

heading: 7 Conclusions
text: In recent years, research on automatic subtitling has shifted toward direct models that do not rely on the transcription of the input audio. The potential advantages of a transcription-free approach have motivated the use of direct models for the translation and segmentation steps, leaving unexplored the direct generation of timestamps on the target text. In this paper, we filled this gap by introducing the first direct system that does not require the generation of transcripts/captions in any phase of the process, including subtitle timestamp estimation. With the introduction of a new metric, SubSONAR, dedicated to evaluating timestamp quality, and experiments on different domains and 7 language pairs, we demonstrated the effectiveness of our solution, which was further validated by a dedicated manual evaluation. Lastly, we showed that our model closed the gap with the cascade paradigm, approaching and even outperforming the best cascade architectures from the last IWSLT campaign, thereby setting a new state of the art for most of the publicly available benchmarks.
Limitations
While our direct AS model achieves significant advancements, there are still some limitations or problems that have not been addressed in this work and should be the focus of further research on the topic.
First of all, although our models are easily applicable to unwritten languages (the only required change is either to use speech units as targets for the CTC compression module or to remove the module itself), our experiments have not included unwritten languages due to the lack of available benchmarks. Moreover, for the same reason, we have not experimented with source languages different from English. Despite this, we do not foresee any specific problem that could arise in this condition, except for a slight drop in translation quality if the CTC compression and its auxiliary loss are removed (Zhang et al., 2022a).
Another limitation of this work regards the model compliance with constraints posed on CPL and CPS. While our model is trained on subtitles that mostly adhere to spatio-temporal requirements, no specific strategies are adopted for their actual fulfillment. Modifying the model or the training strategy to consistently meet these constraints, especially in complex audiovisual content, should be
the topic of future works. Regarding SubSONAR, its language coverage is currently limited to the languages supported by the SONAR speech encoder.
Lastly, due to the large computational costs required and for the sake of a fair comparison with previous works, we did not experiment with using our proposed method on top of large foundational ST models such as SeamlessM4T (Seamless Communication et al., 2023) and Whisper (Radford et al., 2023), although the latter can not be tested in our settings as it does not support translating from English into other languages. To adopt these models for the subtitling task, line and block boundary tokens should be added to their vocabulary and these models should be then fine-tuned on subtitling data (i.e. translations with line and block boundaries). No other modification is needed as SBAAM requires only looking at the crossattention. Optionally, a CTC on Target module has to be trained on top of their encoder for the joint CTC rescoring. This line of research represents a natural next step of this work.

heading: Acknowledgements
text: We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. The work is co-funded by the European Union under the project AI4Culture: An AI platform for the cultural heritage data space (Action number 101100683). We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support. We also thank Giorgia Pucker, Lea Glaubig, and two anonymous annotators for the manual evaluation.

heading: A Experimental Settings
text: A.1 Data
For the multilingual model, we leverage all the 7 language pairs of MuST-Cinema v1.1 (Karakanta et al., 2020b), namely: English to German, Spanish, French, Italian, Dutch, Portuguese, and Romanian. These texts already contain line and block segmentation, i.e., <eol> and <eob> tags are present in both transcripts and translations.
For the bilingual models, we use the same datasets of (Papi et al., 2023b), encompassing most of the training data admitted by the IWSLT 2023 Evaluation Campaign on automatic subtitling. We collect all the available ST corpora, namely MuST-Cinema, EuroParl-ST (Iranzo-Sánchez et al., 2020), and CoVoST v2 (Wang et al., 2020b). Also, we leverage most of the available ASR datasets (CommonVoice (Ardila et al., 2020), LibriSpeech (Panayotov et al., 2015), TEDLIUM v3 (Hernandez et al., 2018), and VoxPopuli (Wang et al., 2021)), by automatically translating the transcripts into the two target languages (German and Spanish) using the NeMo MT models.11
<eol> and <eob> tags are added to both transcripts and translations of all datasets, except for MuST-Cinema that already include them, using the multimodal segmenter by Papi et al. (2022).
A.2 Model and Training
Our systems are implemented with fairseq-ST (Wang et al., 2020a) using the default settings unless specified otherwise. The input comprises 80 Mel-filterbank audio features extracted every 10 milliseconds, employing a sample window of 25. The input features are then preprocessed with two 1D convolutional layers with stride 2, reducing input length by a factor of 4.
The model architecture follows an encoderdecoder design, consisting of a Conformer encoder (Gulati et al., 2020) and a Transformer decoder (Vaswani et al., 2017), with a total number of 133M for both multilingual and bilingual models. The vocabularies are based on unigram SentencePiece (Kudo, 2018), with size 8,000 for the English source and 16,000 for the target (either German, Spanish, or multilingual). Specific hyperparameters are presented in Table 6.
11Publicly available at: https://docs.nvidia.com/ deeplearning/nemo/user-guide/docs/en/main/nlp/ machine_translation/machine_translation.html
Label-smoothed CE is computed after the decoder, target CTC loss is computed on the output of the semantic encoder (full encoder), and the source CTC loss is computed on the output of the acoustic encoder. The values of the weights for all three losses (λ1, λ2, λ3) are set to (1.0, 2.0, 5.0) according to (Yan et al., 2023). Utterance-level Cepstral Mean and Variance Normalization (CMVN) and SpecAugment (Park et al., 2019) are applied during training and segments longer than 30 seconds are filtered out (fairseq-ST default) to avoid excessive VRAM requirements. All model checkpoints are obtained by averaging the last 7 checkpoints obtained from the training. All trainings are executed on 4 NVIDIA Ampere GPU A100 (64GB VRAM).
For the multilingual case, we train the model in one step on all the MuST-Cinema languages by pre-pending the language tag to the subtitle texts, as already explained in §2. For the bilingual case, we first train the model on the ST and the machinetranslated ASR datasets without <eol> and <eob> tags, and then we continue the training starting from the averaged checkpoint by including <eol>
and <eob> tags in the texts. For both cases (and steps, in the case of bilingual models), we use the training settings provided in Table 7.
For inference, we set the beam size to 5 and, according to (Yan et al., 2023), the joint CTC decoding weight α to 0.2. For the SBAAM timestamp estimation method, we extract the cross-attention from the 4th layer and average the scores across the attention heads, following (Papi et al., 2023c).

heading: B CPL and CPS Conformity
text: Table 8 and 9 report the CPL conformity of, respectively, our multilingual and bilingual models with and without the joint CTC decoding strategy. As we can see, the results are very close and no clear and coherent trend across the different languages and test sets emerges. We can conclude that the decoding strategy does not significantly impact CPL conformity.
Switching to analyze the CPS conformity, Table 10 and 11 show the related percentages for the multilingual and bilingual models. In this case, we do not only study the decoding strategy but also the timing estimation method used, as CPS is influenced both by the generated subtitles and the assigned timestamps. While also in this case the difference between the decoding strategies is limited, on average the joint CTC rescoring leads to a lower conformity in both cases. Looking at the differences between the timing estimation methods (Table 10), we notice that SubCTC is by far the worst method (with a ∼7% degradation). ATTN DTW and SBAAM, instead, lead to subtitles with higher conformity than the LEV baseline, although SBAAM is consistently slightly superior in all languages. The benefits of SBAAM over LEV
are confirmed also by the bilingual systems, where SBAAM has a 0.5% higher compliance on average, although the gains are not coherent across the two language pairs.
We can conclude that we can reject the hypothesis about the potential benefits of the joint CTC rescoring in producing outputs of the correct length. The better timings assigned by the timestamp estimation methods, instead, provide little benefits in terms of CPS conformity.

heading: C Human Evaluation
text: C.1 Audio Selection Table 12 lists the audios included in the manual evaluation carried out by the annotators.
C.2 Guidelines for Subtitle Timestamps Evaluation
The following italic text has been given to the annotators as guidelines for their work.
You are asked to check if the source language speech and its corresponding translated subtitles are synchronized, that is if the subtitle remains on screen for the right amount of time with respect to the corresponding speech.
If the timing is wrong, move the timestamp of each subtitle such that the timing of the subtitle matches the timing of the audio. The timestamp can be adjusted at the beginning by changing the start timestamp, at the end by changing the end timestamp, or both. Change the timestamp only
when the content is not aligned or only partially aligned. Please notice that the content of the subtitles, even if not correct, must not be changed. Only modifications to the timestamps are allowed.
To summarize, please follow these guidelines:
• Adjust the timestamp such that the subtitle is synchronized with the corresponding audio: for this purpose, you can adjust the timestamp either at the beginning, at the end, or both;
• Make only necessary changes to align the partially or completely misaligned subtitle with the audio;
• Make only necessary changes to align the partially or completely misaligned subtitle with the audio;
• Do not change the subtitle content.
You will be given 22 videos to annotate and the work consists of around 6 hours of annotation. Please annotate the audio following the given order (from 1 to 22). To carry out your evaluation work at best, you should work on around 5 minutes of audio without interruptions. Then, you should take a break of around 5 minutes. We suggest to take a break after file numbers: 03, 06, 08, 11, 14, 16, 17, 18, 19, 20, 21, 22.
It can happen that you have to annotate an audio that you have already annotated, please annotate it as if it were the first time.
In addition, specific guidelines on the usage of the annotation tool, ELAN, were provided (Figure 2 shows the interface). An initial training day, conducted with all the annotators and consisting of 3 hours of work, was held to explain the guidelines to the annotators, annotate a pilot audio, and discuss the results and common questions. This, with the 6 hours of annotation, resulted in 9 hours of work.
C.3 Results with perception filtering
As works in the field of human perception (Efron, 1970; Venezia et al., 2016) have demonstrated that humans perceive acoustic and visual stimuli with duration inferior to 120ms as instantaneous, we re-compute the statistics of the manual analysis by filtering out time shifts of less than 120 ms. With this change, the resulting inter-annotator agreement increases to a Cohen’s Kappa of 0.71 for en-de, and 0.68 for en-es, which is again “substatial” in both languages but higher than the one obtained in §6.
The results of the two timestamp estimation methods (LEV and SBAAM) with the filtered shifts are shown in Table 13. First of all, we notice that the time shift mean of SBAAM is 23 of LEV, with important differences also in terms of the standard deviation, resulting in a reduction of 240ms for en-de and 383ms for en-es. Therefore, with filtered shifts, the gap between the two methods is more exacerbated than that shown in Table 5, although with similar conclusions. Regarding the percentage of edited timestamps, we observe a relevant decrease in the number of edits for both methods, with SBAAM achieving less than 9% of edited timestamps in en-es. Also in this case, the superior-
ity of the SBAAM method over LEV is widened, as the number of edits for LEV is slightly less than 3× those for SBAAM in en-de, and nearly 3.5× in en-es.

heading: Future Work
text: We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. The work is co-funded by the European Union under the project AI4Culture: An AI platform for the cultural heritage data space (Action number 101100683). We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support. We also thank Giorgia Pucker, Lea Glaubig, and two anonymous annotators for the manual evaluation