StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection (2024)

Abstract: Streaming speech-to-text translation (StreamST) is the task of automatically translating speech while incrementally receiving an audio stream. Unlike simultaneous ST (SimulST), which deals with pre-segmented speech, StreamST faces the challenges of handling continuous and unbounded audio streams. This requires additional decisions about what to retain of the previous history, which is impractical to keep entirely due to latency and computational constraints. Despite the real-world demand for real-time ST, research on streaming translation remains limited, with existing works solely focusing on SimulST. To fill this gap, we introduce StreamAtt, the first StreamST policy, and propose StreamLAAL, the first StreamST latency metric designed to be comparable with existing metrics for SimulST. Extensive experiments across all 8 languages of MuST-C v1.0 show the effectiveness of StreamAtt compared to a naive streaming baseline and the related state-of-the-art SimulST policy, providing a first step in StreamST research.

heading: 1 Introduction
text: Streaming speech-to-text translation (StreamST) is the task of automatically translating spoken content from the source language into the target language in real-time, while continuously receiving an input audio stream. By processing longer, unsegmented audio, StreamST adds another layer of complexity to the difficulties of simultaneous ST (SimulST) which, instead, operates on – often manually – presegmented speech segments (Ren et al., 2020; Ma et al., 2020b; Liu et al., 2021; Weller et al., 2021; Indurthi et al., 2022; Tang et al., 2023, among others).
In SimulST, the primary objective revolves around finding a balance between producing highquality translations and minimizing latency. This balance is managed by a simultaneous policy,
which is the strategy for determining, at each time step, whether to emit a partial translation hypothesis or to wait for additional audio input. This hypothesis, together with the processed audio, is temporarily stored in memory to provide context for subsequent generations and is automatically removed from memory at the end of each audio segment (Ma et al., 2020a). However, when the input is a continuous, unbounded stream, the memory retained as useful context can indefinitely grow, rendering the direct application of conventional SimulST approaches to StreamST impractical due to latency and computational constraints.1
Despite representing the real-world scenario for providing real-time ST in many applications, such as interpreting (Fantinuoli and Prandi, 2021) and lectures (Fügen et al., 2007), and garnering increasing market interests,2 research on streaming translation remains limited, with existing works solely focusing on text-to-text machine translation (MT) (Iranzo-Sánchez et al., 2022, 2023). Moreover, as these works focus on (unbounded) text streams as input, there is currently no metric in the literature suitable to evaluate the StreamST task, where the input is an audio stream.
To fill these gaps, in this paper we delve into the unexplored domain of StreamST and its associated challenges. First, we define the concept of streaming policy for ST by dividing the decision-making process into two steps: hypothesis selection, to determine which part of the translation hypothesis should be emitted (akin to the simultaneous policy), and history selection, to identify which part of
1For example, in the SeamlessM4T model for simultaneous translation (Barrault et al., 2023), the whole encoder is updated every time a new speech chunk is received (Section 5.2.2 of the paper), which makes its use impracticable for processing continuous, unsegmented audio streams.
2“The Real-Time Language Translation Device market is anticipated to rise astronomically each year.” (https://www.marketreportsworld.com/enquiry/ request-sample/24823921)
3692
past audio and generated partial translations should be retained in memory. Then, motivated by the success of direct ST models (Bérard et al., 2016; Weiss et al., 2017) in overcoming the high latency of cascade architectures in the related field of SimulST (Fügen et al., 2007; Fujita et al., 2013; Oda et al., 2014; Müller et al., 2016; Ren et al., 2020), we propose StreamAtt3 (Section 3), the first StreamST policy designed for direct ST systems. To enable the evaluation of our StreamST solution, we also introduce StreamLAAL3 (Section 4), the first latency metric for StreamST. StreamLAAL is designed to facilitate a direct comparison with SimulST solutions, which provide upper-bound results as they operate on pre-segmented audio. Lastly, we demonstrate the effectiveness of StreamAtt through extensive experiments across all 8 languages of MuST-C v1.0. We show that our policy significantly outperforms a naive streaming baseline (Section 6.1) that relies on a fixed number of past words and audio frames as memory, and is even competitive with the related state-of-the-art SimulST policy at low latency (Section 6.2), providing a first promising step in StreamST research.

heading: 2 Related Works
text: While the terms “streaming” and “simultaneous” translation have often been used interchangeably in the literature, we adhere to the definition of streaming by Iranzo-Sánchez et al. (2022, 2023), which refers to the handling of unbounded and continuous streams that, in our case, are audio streams. Consequently, all works that assume to operate on pre-segmented audio input, and only focus on effective methods to determine when and what to emit (Ma et al., 2020b; Weller et al., 2021; Liu et al., 2021; Indurthi et al., 2022; Zhang and Feng, 2022; Tang et al., 2023), are hereinafter categorized as related to SimulST. Some of these works explore how to automatically detect word boundaries in the audio with dedicated modules integrated into the ST system (Dong et al., 2022; Zhang et al., 2022; Fu et al., 2023; Zhang and Feng, 2023). However, they still use this information only to determine when and what to emit, lacking a mechanism to determine which portion of the memory has to be retained and which can be discarded, hence relying on pre-segmented audio for the simultaneous inference (Ma et al., 2021). It follows that all these
3Code available at https://github.com/hlt-mt/ FBK-fairseq/ under Apache 2.0 license.
works are limited to the hypothesis selection step (i.e., the simultaneous policy) and, differently from this paper, they all ignore the history selection step, which is necessary to deal with continuous audio inputs.
In the context of text-to-text machine translation, the streaming scenario has instead been addressed in (Iranzo-Sánchez et al., 2022, 2023). These works rely on an MT system trained with the wait-k simultaneous policy (Ma et al., 2019), which consists in waiting for a predefined number of source tokens before starting the translation. The same policy is applied at inference time as the hypothesis selection strategy, while the history selection step consists of keeping a fixed number of textual segments (given by a segmenter model in (Iranzo-Sánchez et al., 2022) or a memory mechanism integrated into the MT system in (Iranzo-Sánchez et al., 2023)) as a context for the current translation generation. As these works are tailored for textual inputs, the evaluation metrics they rely on do not directly apply to StreamST, disregarding the crucial aspect of latency measurement from audio streams, which we instead also address in this work.

heading: 3 Streaming Policy
text: The generic decision steps of a StreamST policy (Figure 1) can be schematized as follows:
Hypothesis Selection: Given both audio and textual history and the newly received chunk of audio, the Hypothesis Selection step determines whether and how many of the newly predicted words to emit. This can be easily traced back to the role of a SimulST policy.
History Selection: Given the audio and textual history retained in the previous step, the newly received speech chunk, and the new partial hypothesis selected in the Hypothesis Selection step, the History Selection step decides what part of the new history should be retained for processing the next audio chunk. This process can be further split into:
Textual History Selection: This sub-step selects the new textual history starting from the textual history retained in the previous iteration and the new partial hypothesis obtained by the Hypothesis Selection; Audio History Selection: This sub-step selects the new audio history starting
and step ) is indicated from 1 (first) to 3 (last).
from the audio history retained in the previous iteration and the newly received speech chunk.
Inspired by recent findings on the effectiveness of building SimulST systems by directly applying simultaneous policies to offline-trained ST models without ad-hoc training/fine-tuning (Liu et al., 2020; Nguyen et al., 2021; Papi et al., 2022a), which led to leadership in the IWSLT 2022 Shared Task (Anastasopoulos et al., 2022), we develop a StreamST policy that exploits offline-trained ST models. In particular, building on recent research proposing cross-attention as a reliable guide for SimulST policies (Papi et al., 2023b,a), we introduce StreamAtt, a StreamST policy that leverages cross-attention scores for both hypothesis and audio history selection (steps and ), while using a heuristic for textual history selection (step
).

heading: 3.1 Hypothesis Selection
text: For Hypothesis Selection (step ), we exploit AlignAtt (Papi et al., 2023b), the state-of-theart SimulST policy for offline-trained direct models. AlignAtt outperforms all the alternative solutions, such as the standard wait-k policy (Ma et al., 2019) adapted for speech either with fixed (Ma et al., 2020b; Fukuda et al., 2022; Huang et al., 2023) or adaptive word boundary detection4 (Ren et al., 2020; Zeng et al., 2021, 2022), and the Local Agreement policy (Liu et al., 2020; Polák et al., 2022).
4The fixed word detection assumes that a word lasts 280ms, while adaptive ones leverage the predictions of a CTC (Graves et al., 2006) module to determine when a new word starts.
AlignAtt builds upon the observation that crossattention scores can be used to align the input and the generated translation (Tang et al., 2018; Zenkel et al., 2019; Garg et al., 2019; Chen et al., 2020), also with audio as input (Papi et al., 2023a; Alastruey et al., 2023). Specifically, the alignments between the textual translation Y = [y1, ..., ym] and the encoded input audio X = [x1, ..., xn] are obtained with the following formula:
Align(yi) = arg max j=1,...,|X| Across(xj , yi) (1)
where Across stands for the cross-attention scores (Across)5 computed in the Transformer decoder layers (Vaswani et al., 2017). Align(yi) is, therefore, the index of the frame aligned with the predicted token yi, exploited by AlignAtt to decide which tokens of the partial hypothesis have to be emitted. To this aim, it iterates over the predicted tokens and emits them until the following stopping condition is verified:
Align(yi) > |X| − f
where f is a hyper-parameter that directly controls the latency of the model. The underlying assumption is that if a token is aligned with the most recently received f audio frames, the information provided by these frames can be unstable or insufficiently informative to generate that token (i.e., the system has to wait for additional audio input before generating it). Therefore, smaller f values represent fewer frames that may potentially block the generation if attended and, consequently, a lower
5The cross-attention is the dot-product attention (Chan et al., 2016) between the generated tokens Y and the encoder output X.
chance that the stopping condition is verified, resulting in lower latency.

heading: 3.2 History Selection
text: We design the History Selection of StreamAtt based on two assumptions: i) the audio and textual history should be aligned to provide the model with coherent inputs, and ii) cross-attention scores provide a reliable alignment between the generated text and the input audio, as seen in the previous section. Building on these assumptions, we first perform the Textual History Selection (Section 3.2.1) to determine which part of the generated text has to be retained. Then, we forward the resulting textual history to the Audio History Selection (Section 3.2.2), which discards the audio frames that do not align with the provided text. In the following, we describe both selection steps.

heading: 3.2.1 Textual History Selection
text: For Textual History Selection (step ), we analyze the two different heuristics described below.
Fixed Number of Words (FW). This heuristic retains a fixed number of words (nwords) in the textual history, inspired by the approach of IranzoSánchez et al. (2022) in streaming MT, where a fixed number of segments is retained. In practice, the textual history Yhist at time t is computed as:
Yhist(t) = [Yhist(t− 1), Y (t)][: −nwords]
where Y (t) is the new hypothesis to be emitted at time t and that was selected during step . Specifically, first, the textual history retained from the previous iteration and the new partial hypothesis are concatenated. Then, only the last nwords are preserved as textual history for the next decoding phase (i.e., the next step ). Since nwords is a hyper-parameter, we empirically determined the best value on the validation set, which resulted to be 20. Detailed results are reported in Appendix A.
Punctuation (P). This heuristic simulates what happens in SimulST, where the history is reset at the end of each sentence. As sentence boundaries are not available in StreamST, it considers mediumstrong punctuation marks (“.”, “!”, “?”, “;”, “:”) as sentence boundary proxies. In practice, it retains all the words after the last-predicted medium-strong punctuation mark as the textual history Yhist at time t. As such, Yhist is computed as:
Yhist(t) = [Yhist(t− 1), Y (t)][p+ 1 :]
p = last_index({., !, ?, ; , :}, [Yhist(t− 1), Y (t)]) where Yhist(t− 1) is the textual history of the previous iteration, Y (t) is the new hypothesis selected in step , and the function last_index returns the last occurrence of any of the punctuation marks. Specifically, only the words after the last punctuation mark are preserved as the textual history for the next step, approximating the reset of the textual history at the end of each sentence as done by SimulST systems.

heading: 3.2.2 Audio History Selection
text: For the Audio History Selection (step ), we exploit the cross-attention scores computed by the model and already used for step . However, differently from step , these scores are used to decide which of the currently retained audio frames should be discarded from the audio history.
First of all, we obtain the current audio X by concatenating the audio history retained from the previous iteration Xhist(t− 1) and the new audio input X(t). Then, recalling that we compute the alignment between a textual token and its corresponding audio frame with Eq. 1, we select the audio history Xhist for the iteration t as follows:
Xhist(t) = X[ min hk∈Yhist(t) Align(hk) :]
Here, minhk∈Yhist(t)Align(hk) represents the index of the first frame of the audio sequence X that is attended by at least one of the tokens hk in the textual history Yhist(t) determined in the previous step. By doing so, we discard the audio frames that are no longer attended by the current textual history. Therefore, the textual history Yhist(t) and audio history Xhist(t) preserved at this step, together with the new audio input received in the next step X(t+ 1), constitute the input of the model for the next iteration.

heading: 4 Streaming Latency Metric
text: For evaluating the performance of StreamST, the standard metrics adopted in SimulST cannot be applied as is. In fact, they are not designed to evaluate outputs obtained from entire audio streams but, instead, refer to (manually) segmented audio and their corresponding translations for the computation. We hence adapt them to the streaming scenario to define our StreamST latency metric.
Specifically, we opt to use the family of latency metrics based on Average Lagging (Ma et al.,
2019) for speech (Ma et al., 2020b), given their widespread adoption in SimulST (Anastasopoulos et al., 2022; Agarwal et al., 2023). Among these metrics, we select the Length-Adaptive Average Lagging or LAAL (Papi et al., 2022b; Polák et al., 2022), which corrects the standard AL formulation to avoid the underestimation of the latency when predictions are longer than the reference translation. Given X = [x1, ..., x|X|] as the speech segment, where each element xj has duration Tj , Y∗ = [y∗1, ..., y ∗ |Y∗|] as the reference words, and Y = [y1, ..., y|Y|] as the hypothesis words, LAAL for SimulST is formulated as:
LAAL = 1
τ ′(|X|)
τ ′(|X|)∑
i=1
di − d∗i
d∗i = (i− 1) · ∑|X| j=1 Tj
max{|Y|, |Y∗|} where di is the delay of the predicted words, τ ′(|X|) = min{i|di = ∑|X| j=1 Tj} is the index of the target token when the end of the source sentence is reached, and d∗i represents the delay of an oracle policy that starts to emit words as soon as the speech starts and is perfectly in sync with the speaker. We adapt LAAL by considering the entire (unsegmented) stream of audio S = [X1, ...,X|S|] instead of the single speech segment X, for which we have a continuous stream of predicted translation words YS. Since we have reference translations Y∗X1 , ...,Y ∗ X|S|
only for the segmented audio X1, ...,X|S|, we first obtain the segmented prediction YS = [YX1 , ...,YX|S| ] with their corresponding delays by applying the mWERSegmenter tool (Matusov et al., 2005) between each reference sentence Y∗i and the entire stream of predicted translation YS, in a similar fashion to what has been done for streaming MT (Iranzo-Sánchez et al., 2021). Consequently, we obtain the LAAL for the entire audio stream (StreamLAAL), by computing:
Stream
LAAL =
1 |S| ∑
X1,...,X|S|
1
τ ′(|Xi|)
τ ′(|Xi|)∑
i=1
di − d∗i
d∗i = (i− 1) · ∑|Xi| j=1 Tj
max{|YXi |, |Y∗Xi |} In practice, the LAAL metric is calculated for every speech segment Xi of the stream S and its corresponding reference Y∗Xi with the automatically aligned prediction YXi and then averaged over all the speech segments of the stream
X1, ...,X|S| to obtain StreamLAAL. As this formulation builds upon the original LAAL metric, it enables direct comparisons between the results obtained in StreamST and those reported in related works on SimulST. In this way, we can measure the gap between StreamST systems and their SimulST counterparts, which provide upper-bound results as they operate on pre-segmented audio.

heading: 5 Experimental Settings

heading: 5.1 Data
text: To be comparable with previous works (Ren et al., 2020; Ma et al., 2020b; Zeng et al., 2021; Chen et al., 2021; Liu et al., 2021; Zhang and Feng, 2022; Indurthi et al., 2022; Papi et al., 2022a; Tang et al., 2023), we train our models on all languages of MuST-C v1.0 (Cattoni et al., 2021), namely English (en) to Dutch (nl), French (fr), German (de), Italian (it), Portuguese (pt), Romanian (ro), Russian (ru), and Spanish (es).
To optimize GPU RAM consumption and speed up training, we filter out segments longer than 30s from the training set. The resulting data statistics are presented in Table 1.
We also perform data augmentation by applying sequence-level knowledge distillation (Kim and Rush, 2016; Gaido et al., 2021b) as in previous work on SimulST (Liu et al., 2021; Tang et al., 2023), which consists of translating the transcripts of the training set (MuST-C) with an MT model and using them together with the gold reference during training. As a result, the final number of target sentences used during training is twice the original one, while the speech input remains unaltered. We use NLLB 3.3B (Costa-jussà et al., 2022) as the MT model, whose performance on the MuST-C dataset is presented in Appendix B.

heading: 5.2 Architecture and Training Setup
text: The offline model is a Conformer-based (Gulati et al., 2020) encoder-decoder, which is the state-ofthe-art architecture in ST (Guo et al., 2021). All the model details are provided in Appendix C.
The input is represented by 80 audio features extracted every 10ms with a sample window of 25
and processed by two 1D Convolutional layers with stride 2 to reduce its length by a factor of 4 (Wang et al., 2020). All our models are implemented in fairseq-s2t (Wang et al., 2020). Detailed training settings are described in Appendix C.

heading: 5.3 Inference, Evaluation, and Comparisons
text: As StreamAtt is the first StreamST solution, we compare it with a naive baseline that retains a fixed history both in terms of text and audio. This baseline assumes that each word has a duration of 280ms – following (Ma et al., 2020b) – and keeps the same (fixed) number of words in both the audio history and textual history. For the sake of a fair comparison, we set this number of words to 20, as for StreamAtt (see Section 3.2.1). We also compare StreamAtt with the corresponding state-ofthe-art SimulST policy for offline-trained systems AlignAtt. For both AlignAtt and StreamAtt, we vary the hyperparameter f in the range [2, 4, 6, 8] to obtain results for different latency regimes, while we set the size of the speech segment to 1s (the dimension of the incremental speech chunk) and extract the cross-attention scores from the 4th decoder layer, as per (Papi et al., 2023a).
We use our extension of the SimulEval tool (Ma et al., 2020a) for both SimulST and StreamST evaluation. For the streaming approaches (StreamAtt and Baseline), we simulate streaming conditions by providing as input the entire TED talks of the MuST-C tst-COMMON set. Instead, for the SimulST AlignAtt policy, we provide the manually segmented audio provided for the same test set, following the standard SimulST evaluation settings. We use sacreBLEU (Post, 2018)6 for translation quality, and LAAL – for AlignAtt – and StreamLAAL (Section 4) for latency. Moreover, as recommended by Ma et al. (2020b), we report computationally-aware (CA) StreamLAAL for our streaming comparison, which measures the real elapsed time instead of the ideal latency, as it also accounts for the time required for the model and policy computation. During inference, the features are computed on the fly and CMVN normalization is based on the global mean and variance estimated on the MuST-C training set. Inferences are executed on a NVIDIA K80 GPU with 12GB VRAM.
6BLEU+case.mixed+smooth.exp+tok.13a+version.2.3.1

heading: 6 Results
text: In this section, we first compare our proposed StreamAtt policy for StreamST with the streaming baseline (Section 6.1) and then with the state-of-the-art SimulST policy (Section 6.2). This is followed by an analysis of our approach (Section 6.3).

heading: 6.1 Streaming Results
text: To inspect the streaming ability of the StreamAtt policy equipped either with Fixed Words (FW) or Punctuation (P) Textual History Selection methods (Section 3.2.1), we compare its quality-latency performance with a streaming baseline (Section 5.3).
The translation quality and latency scores, averaged over the 8 languages of MuST-C v1.0, are reported in Table 2. Detailed results for each language pair can be found in Appendix D. As can be observed, both StreamAttFW and StreamAttP outperform the baseline by a large margin, with an increase of 5 BLEU points in quality and a reduction of more than 1s in latency, at every latency regime. The latency gap further increases when considering the computationally aware latency, with improvements of up to 1.7s. This means that the Audio History Selection strategy of StreamAtt based on cross-attention (Section 3.2.2) is crucial not only for obtaining high-quality translations but also for reducing latency, as discarding audio based solely on fixed duration substantially impacts performance and uselessly increases computational costs. Furthermore, the significant translation quality drop observed in the baseline underscores the importance of enforcing alignment between audio and textual history, as StreamAtt does, and the inadequacy of naive heuristics in maintaining this alignment.
Moving to the comparison between StreamAttFW and StreamAttP, the two Textual History Selection methods yield similar BLEU scores, indicating a similar translation quality. However, StreamAttFW consistently achieves lower latency both considering computationally unaware and aware latency measures, with an average reduction of 170ms in NCA-StreamLAAL and 750ms in CAStreamLAAL. This result may be surprising, as StreamAttP is designed to mimic the behavior of SimulST systems, but we explain it in Section 6.3.

heading: 6.2 Comparison with SimulST
text: To further investigate the StreamAtt performance, we compare it with the state-of-the-art AlignAtt
policy for SimulST. Since the SimulST policy is applied to manually segmented audio segments, we consider it as an upper-bound for StreamAtt that, instead, faces the more challenging scenario of unsegmented audio streams. Notice that both StreamST and SimulST policies use the same underlying models since they are directly applied to the offline-trained ST systems.
Figure 2 shows the quality-latency plots for each one of the 8 individual languages of MuST-C v1.0. First, it can be observed that, with the only exception of en-ru and en-fr, StreamAttFW achieves a better quality-latency trade-off compared to StreamAttP since the curve of the first is shifted towards the left-upper part compared to the curve of the second. Second, we notice that at low latency StreamAtt is close to AlignAtt and is even able to out-
perform it in some language pairs. In fact, StreamAttFW yields an improvement of more than 2 BLEU points at 1.2s in Spanish, similar to the gain exhibited in Italian which is of about 1.5 BLEU at the same latency of 1.2s.
Overall, despite being applied to unsegmented speech, the StreamAttFW policy achieves competitive performance at low latency, with less than 1 BLEU of degradation on average across languages, compared to its upper-bound AlignAtt. Instead, both StreamAttFW and StreamAttP performance is not growing as much as that of AlignAtt when the latency increases, exhibiting a drop of about 2 BLEU points on average. We speculate that the root cause of this behavior (comparable or even better performance at low latency but some quality degradation at higher latency) stems from the intrinsic
differences between the simultaneous and streaming tasks: the simultaneous policy benefits from manually segmented audio, while the streaming policy can use both audio and textual history from previous segments, enhancing performance, especially when this context is more useful, as at lower latency. However, when the context becomes too broad, as at higher latency, it can be challenging for the model to effectively select relevant information for the current translation, resulting in performance degradation, as also noted by Iranzo-Sánchez et al. (2022) in text-to-text streaming MT. Further analysis of these aspects presents an interesting avenue for future research.
In summary, despite the added complexities of the streaming task, StreamAtt demonstrates competitive low-latency performance compared to its SimulST upper bound, while closing the quality gap at higher latency is an interesting topic for future StreamST research.

heading: 6.3 Why Punctuation-based Textual History Selection is Worse than Fixed Words?
text: The findings in Sections 6.1 and 6.2 revealed that the streaming solution based on punctuation (StreamAttP) not only exhibits a quality gap at higher latency regimes compared to the SimulST approach but also with the fixed words solution (StreamAttFW). To understand this behavior, we carried out a manual inspection of outputs, revealing a noticeable trend across all streaming solutions: they all tend to generate fewer strong punctuation marks, often substituting them with commas. To corroborate this observation, we computed the average occurrences of punctuation marks in the outputs of both SimulST and StreamST approaches.
As shown in Table 3, the streaming solutions produce approximately half the number of full stops compared to simultaneous systems (and references), while the occurrence of commas is nearly
twice as frequent in streaming outputs compared to simultaneous outputs and references. This particular behavior not only sheds light on the underperformance of the punctuation-based solution compared to fixed words (attributed to the scarcity of strong punctuation marks) but also on its quality gap compared to the simultaneous solution.
The cause of this issue may be attributed to the fact that systems are trained on manually segmented sentences, which typically feature a single full stop at the end. In the streaming setting, such systems face a mismatch during inference that is, instead, absent in the simultaneous approach executed on audio segmented similarly to the training sets. Given this discovery, an interesting future research direction involves experimenting with data augmentation techniques that introduce samples deviating from the conventional single full-stop placement at the end of speech segments in the training data. For instance, exploring the effects of concatenating multiple sentences in training data (Lam et al., 2023) or re-segmenting training data into speech segments that do not correspond to sentences (Gaido et al., 2020; Lam et al., 2022; Tsiamas et al., 2023) represent promising solutions.

heading: 7 Conclusions
text: Our work addressed the underexplored domain of StreamST, which tackles the challenge of translating spoken content from the source language to the target language while incrementally receiving an audio input stream. Unlike SimulST, which deals with pre-segmented speech chunks, StreamST grapples with the inability to retain the entire growing history in memory due to latency and computational constraints. Despite growing interest in its applications, research on streaming translation remains limited, with existing studies solely focusing on text-to-text translation, leaving the domain of StreamST and its challenges, including the absence of a suitable evaluation metric, still unaddressed.
To fill these gaps, in this paper, we delved into the domain of StreamST by first defining the concept of streaming policy for ST. Then, building on insights from SimulST research underscoring the efficacy of direct ST systems in overcoming the latency issues of cascade architectures, we proposed StreamAtt, the first StreamST policy tailored for direct ST models. To enable the evaluation of StreamST solutions, we also introduced StreamLAAL, the first StreamST latency metric designed
to facilitate direct comparisons with SimulST models. Through empirical evaluation on all 8 languages of MuST-C v1.0, we showed that StreamAtt significantly outperforms a naive streaming baseline, and is competitive with the SimulST state-ofthe-art AlignAtt policy at lower latency, providing a first promising step in StreamST research.

heading: Acknowledgements
text: This paper has received funding from the European Union’s Horizon research and innovation programme under grant agreement No 101135798, project Meetween (My Personal AI Mediator for Virtual MEETtings BetWEEN People). We also acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU.
Limitations
Although applicable to any offline-trained ST models, StreamAtt and its behavior have been analyzed only on one architectural configuration (12 Conformer encoder layers and 6 Transformer decoder layers). As a consequence, some hyper-parameters, such as the number of words to preserve in the Fixed Words Textual History Selection (nwords), might vary and depend on the specific ST model, thus requiring a dev set on which to search for the best value before directly testing. Moreover, we applied the Hypothesis Selection-related hyperparameters (e.g., the number of forbidden frames – f , and the decoder layer from which to extract the cross-attention scores) following previous works but we did not validate these choices on our settings nor changed them to be comparable with these works. Concerning the analyzed languages, the StreamAtt policy has been tested and compared with the naive baseline and related SimulST policy on a restricted set of European languages and, even if there is no reason suggesting that cannot be applied to other languages (possibly after a proper hyper-parameter search), its usage on a wider set of target languages and a source language different from English has not been verified in this work and is left for future research.
As already mentioned in Section 6.3, we have noticed a train-test mismatch between the punctuation of the output emitted by our StreamST policy and the SimulST one, despite both being applied to the same underlying ST model. This underscores
that some training or fine-tuning techniques can be applied to further improve StreamAtt performance. However, besides representing an interesting direction for future research, such investigations were beyond the scope of this study, which aimed to move the first step in the exploration of the StreamST domain.

heading: B NLLB 3.3B performance on MuST-C
text: See Table 5.

heading: C Model and Training Settings
text: The Conformer-based model is made of 12 Conformer encoder layers (Gulati et al., 2020) and 6 Transformer (Vaswani et al., 2017) decoder layers with a total of ∼115M parameters. Each encoder/decoder layer has 8 attention heads, 512 as embedding size and 2,048 hidden neurons in the feed-forward layers. We set dropout at 0.1 for feed-forward, attention, and convolution layers. Also, in the convolution layer, we set 31 as the kernel size for the point- and depth-wise convolutions. The vocabularies are based on unigram SentencePiece models (Kudo and Richardson, 2018) with dimensions of 8,000 for the target side and 5,000 for the source side (en). We optimize with Adam (Kingma and Ba, 2015) by using the label-smoothed cross-entropy loss with 0.1 as the smoothing factor (Szegedy et al., 2016). We employ Connectionist Temporal Classification – or CTC – (Graves et al., 2006) as an auxiliary loss
7 BLEU StreamLAALNCA+StreamLAALCA
to avoid pre-training (Gaido et al., 2022) and also to compress the input audio, reducing RAM consumption and speeding up inference (Gaido et al., 2021a). Utterance-level Cepstral Mean and Variance Normalization (CMVN) and SpecAugment (Park et al., 2019) are applied during training. The learning rate is set to 5 · 10−3 with Noam scheduler (Vaswani et al., 2017) and warm-up steps of 25k. We stop the training after 15 epochs without loss decrease on the dev set and average 7 checkpoints around the best (best, three preceding, and three succeeding). Trainings are performed on 4 NVIDIA A40 GPUs with 40GB RAM. We set 40k as the maximum number of tokens per mini-batch, 2 as update frequency, and 100,000 as maximum updates (∼23 hours).

heading: D Streaming Results per Language
text: See Table 6.

heading: Future Work
text: We also acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. Limitations
Although applicable to any offline-trained ST models, StreamAtt and its behavior have been analyzed only on one architectural configuration (12 Conformer encoder layers and 6 Transformer decoder layers). As a consequence, some hyper-parameters, such as the number of words to preserve in the Fixed Words Textual History Selection (nwords), might vary and depend on the specific ST model, thus requiring a dev set on which to search for the best value before directly testing. Moreover, we applied the Hypothesis Selection-related hyperparameters (e. g. , the number of forbidden frames – f , and the decoder layer from which to extract the cross-attention scores) following previous works but we did not validate these choices on our settings nor changed them to be comparable with these works. Concerning the analyzed languages, the StreamAtt policy has been tested and compared with the naive baseline and related SimulST policy on a restricted set of European languages and, even if there is no reason suggesting that cannot be applied to other languages (possibly after a proper hyper-parameter search), its usage on a wider set of target languages and a source language different from English has not been verified in this work and is left for future research. As already mentioned in Section 6. 3, we have noticed a train-test mismatch between the punctuation of the output emitted by our StreamST policy and the SimulST one, despite both being applied to the same underlying ST model. This underscores
that some training or fine-tuning techniques can be applied to further improve StreamAtt performance. However, besides representing an interesting direction for future research, such investigations were beyond the scope of this study, which aimed to move the first step in the exploration of the StreamST domain