End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation
Authors: Juan Pablo Zuluaga Gomez
Zhaocheng Huang
Xing Niu
Rohit Paturi
Sundararajan Srinivasan
Prashant Mathur
Brian Thompson
Marcello Federico
Venue: EMNLP 2023 Main (2023)
Keywords: speech translation
conversation
speaker-turn detection

Abstract: Conventional speech-to-text translation (ST) systems are trained on single-speaker utterances, and they may not generalize to real-life scenarios where the audio contains conversations by multiple speakers. In this paper, we tackle single-channel multi-speaker conversational ST with an end-to-end and multi-task training model, named Speaker-Turn Aware Conversational Speech Translation, that combines automatic speech recognition, speech translation and speaker turn detection using special tokens in a serialized labeling format. We run experiments on the Fisher-CALLHOME corpus, which we adapted by merging the two single-speaker channels into one multi-speaker channel, thus representing the more realistic and challenging scenario with multi-speaker turns and cross-talk. Experimental results across single- and multi-speaker conditions and against conventional ST systems, show that our model outperforms the reference systems on the multi-speaker condition, while attaining comparable performance on the single-speaker condition. We release scripts for data processing and model training.

End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation
Juan Zuluaga-Gomer", Zhaocheng Huang" Xing Niu Rohit Paturi Sundararajan Srinavasan', Prashant Mathurt, Brian Thompson', Marcello Federico iddiap Research Institute & EPFL
#AWS AI Labs
juan. whan.uulugg@eeumm.co
(davidhzc, xingniu, paturi, sundarsr, pramathu, brianjt, marcfede com
Abstract
Spanob-rp Tout
Conventional speech-to-tees translation (ST) systems are trained on single-speaker utter- ances, and they may not generalize to real-life scenarios where the audio contains conversa- tions by multiple speakers. In this paper, tackle single-channel multi-speaker conversa- tional ST with an end-to-end and multi-task training model, named Speaker-Turn Aware Conversational Speech Translation, that com- bines automatic speech recognition, speech translation and speaker turn detection using spe- cial tokens in a serialized labeling format. We experiments the
CHAINEL
Propossed Spench to Tout Taanlittonn mahivv cookeer
CHANNEL: &
speanor
spoaker
oros
Figure 1: A two-speaker multi-ture conversational seg- ment. Previous work focuses on separated channels without considering cross-talks and speaker-turn (top). STAC-ST targets more challenging scenario where mul- tiple speakers converse with occasional cross-talks due merged channels (bottom).
corpus, which we adapted by merging the two single-speaker channels into one multi-speaker channel, thus representing the more realistic and challenging scenario with multi-speaker turns and cross-talk. Experimental results across single- and multi-speaker conditions and against conventional ST systems, show that our model outperforms the reference systems on the multi-speaker condition, while attaining comparable performance on single-speaker condition. We release scripts for data process- ing and model training.'
2017, inter alia) have recently gained increasing interest and popularity thanks their simple archi- tecture, less error propagation (Etchegoyhen et al.,
2022), efficient training process, and competitive performance (Inaguma et al., 2019).
Despite significant recent advances in E2E- (Gheini et al., 2023; Wang al., 2023), most ST systems to date have focused on translat- ing isolated speech utterances from monologue speech (Di Gangi et al., 2019), read speech (Ko- cabiyikoglu et al., 2018) or prompted speech (Wang et al., 2021). Being trained single-turn utter- ances, these systems may lack the ability to handle real-life scenarios in which multiple speakers con- verse, and sometime overlap, in the same audio channel (Post et al., 2013).
1 Introduction
Speech translation (ST) has seen wide adoption in commercial products and the research commu- nity (Anastasopoulos al., 2021, 2022) due to its effectiveness in bridging language barriers. ST aims to translate audio of source languages into text of the target languages. This problem was tackled by cascaded approach that pipelines Au- tomatic Speech Recognition (ASR) and Machine Translation (M') over the last few decades (Waibel et al., 1991; Vidal, 1997; Casacuberta et al., 2008, inter alia). However, end-to-end speech translation (E2E-ST) systems (Berard et al., 2016; Weiss et al.,
In this work, we tackle the more challenging task of multi-speaker conversational ST. We refer to it multi-turn & multi-speaker (MT-MS), opposed to single-turn, which most ST systems implicitly assume. This illustrated in Figure 1, where a "conversation" between two speakers recorded with separate channels (top) becomes more difficult to
"Work conducted during an internship at Amazon. " "Corresponding authors with equal contributions. 'tttps: thub. com/ amazon- amazon-science/ stac- secc-sppech-tranne translation

---

translate if the channels are merged (bottom), due the introduction of speaker-turns and cross-talks, particular, ST with cross-talks and speaker-turns is difficult because speech content of different sen- tences is mixed up or switched. While MT-MS speech has been studied in ASR (Raj et al., 2022), to the best of our knowledge, this is the first pa- per that investigates it in end-to-end ST. We tackle MT-MS ST with an approach named Speaker- Turn Aware Conversational Speech Translation (STAC-ST). STAC-ST is a multi-task training frame- work that combines ASR, ST and speaker-turn de- tection using special tokens in a serialized labeling format. It is inspired by a recent speech founda- tion model, Whisper (Radford et al., 2023), which jointly trains ASR, X-to-English ST, voice activity detection, and language identification with 680k hours of speech data using labeling- -baed multi- task learning. Our contributions are as follows:
requires 3-way parallel data for each training exam- ple, i.c., audio, transcript, and translation, as can be found, in limited amount, the CoVoST (Wang et al., 2020, 2021) and MuST-C (Di Gangi et al., 2019) corpora. Prior work proposed to over- come the 3-way parallel data bottleneck by pseudo- labeling ST data (Gheini et al., 2023), or by pre- training an ASR model (van den Oord et al., 2018) on large multilingual data (Bapna et al., 2022: Zhang et al., 2023b) before training the joint ASR & ST model (Babu et al., 2022). Recently, the Whisper model (Radford et al., 2023) introduced an effective annotation format for jointly training ASR & ST with independent targets.
Conversational Speech Translation Work on conversational ST (Kumar et al., 2014b,a; Zanon Boito et al., 2022) has mainly focused on single-speaker speech, either segmented manually or automatically, via voice activity detection. Man- ual segmentation based on the Fisher and CALLHOME corpora, on cascaded ST (Kumar al., 2014b), E2E-ST (Weiss et al., 2017; Peng et al., 2023), simultaneous ASR & ST (Soky et al., 2022), streamed ST (Deng et al., 2022), and multilingual ST (Inaguma et al., 2019). Automatic segmentation was instead deployed with the MSLT corpus (Federmann and Lewis, 2016) to target streamed ST (Xue et al., 2022) well as language-agnostic streamed ST (Wang al., 2023). In this work, we report results on the Fisher- CALLHOME corpus (Post et al., 2013) which, sim- ilarly to the MSLT corpus, offers the opportunity to run contrasting experiments of single-speaker ST versus MT-MS ST, both without reference segmen- tation.
1. We introduce the task of multi-turn & multi- speaker ST. including cross-talks and speaker- turns, that expands the realm of ST which has been limited to single-speaker utterances.
assumed in recent studies,
2. We propose an end-to-end model (STAC-ST) which achieves state-of-the-art BLEU scores corpus that al-
on Fisher-CALHOMEE lows to target MT-MS without degradation single-turn ST.
3. We explore a zero-shot scenario where MT- MS ST data is not available for training. We show that STAC-ST improves ST up to BLEU by leveraging MT-MS ASR targets, mitigating the necessity of parallel data, which is lacking within the community.
4. Besides serializing transcripts and transla- tions at cross-talks, the STAC-ST model is also shown to learn the task of time-aligned speaker change detection.
Speaker- Turn and Cross-Talk in ASR Speaker- turns and cross-talks have been explored in the ASR field and commonly termed, multi-talker ASR. Kanda al. (2020) proposed serialized output training (SOT) strategy for multi-speaker over- lapped speech recognition with special tokens. At inference time, word and speaker tags are output in serialized manner for an unlimited number of speakers. SOT was later ported to the streaming scenario (Kanda et al., 2022). However, SOT may produce frequent speaker changes, which can de- grade the overall performance. Thus, Liang et al. (2023) proposed to explicitly incorporate bound- ary knowledge with a separate block for speaker change detection task and boundary constraint loss.
5. We conduct extensive ablation studies on im- portant aspects of STAC-ST, including joint modeling of ASR & ST, impact of model size (up to 300M parameters), data size, and inte- gration of task tokens. Thus, we shed light on the best practices for building conversational MT-MS ST systems.
2
Related Work
Joint ST & ASR Modeling Recent works in ST have leveraged ASR training data to improve translation quality. In principle, joint ASR and ST modeling (Gheini et al., 2023; Soky et al., 2022)

---

frame-level 80-dimensional filterbank features are extracted from the audio² every 40ms. Second, we apply SpecAugment (Park et al., 2019) on the in- put audio features, an effective data augmentation technique that masks out certain regions of the in- put filterbank features. Then, the audio augmented features are passed to a 2-layer CNN that outputs 5120-dim vector (flattened 2D-1D output tensor from the CNN layer). Finally, this vector feeds linear layer that generates the input the encoder model. The decoder takes the encoder outputs and generates a sequence of text. Formally, for each speech segment, the filterbank features can be rep- resented as: X {x+ E IF )T= and the reference transcription or translation as: Y {wa, E V) Where, F is the feature dimension, T is the num- ber of speech frames, N is the number of text to- kens, and V is the vocabulary. During training of STAC-ST, we concatenate independent datasets D DASR = (X, YaSR) and DST = (X, YST), for ASR & ST, respectively. Samples of training mini- batches are jointly drawn from D DASH and Dgr-
TURN
XT
moi
Lineer
Liner
25 Hz
xN
Encoder
xN
Decoder
Pesteral
Procl
Mulkiaak triinng
25
Linear Layer
FrontEnd
BCS
lok1
TURN
SpesAugmantt)
FBank Features
Legend
Tassk Tokens
Tuat Tokoons
Figure 2: Proposed model architecture STAC-ST for multi-turn & multi-speakker ST.
Multi-talker ASR has also been explored in the non-
streaming (Huang et al., 2023) and streaming (Raj et al., 2022) setups. Multi-turn ASR has been ex- plored in automatic dubbing (Virkar et al., 2023) of scripted content, challenging case due to the high number of speakers and short segments (Brannon et al., 2023), but improvements have come from aligning (Thompson and Koehn, 2019, 2020) auto- matic transcripts with available production scripts. Another branch of research targets cross-talk & multi-talker ASR (Yang et al., 2023) using speech separation of long-form conversational speech (Pa- turi et al., 2022) but these techniques have diffi- culty handling variable number of speakers and are not optimized end-to-end for ASR improve- ments. However, how to effectively deal with multi- speaker conversational ST has been neglected.
3.2 Serialized Labeling Based on Task Tokens
key component of the model is the serialized multi-task labeling framework based on special to- kens. As shown in Figure 2, besides the text tokens, special tokens are used to specify the task. There are four types of task tokens, i.c., [SL] (source lan- guage), [TL] (target language), [TURN] (speaker- turn), and [XT] (cross-talk).
The first two tokens are language tokens that define the task for either ST (when [SL] * [TLJ) or ASR (when [SL] = [TLJ). At training time, we instantiate language tokens and prepend them to each sample of DST and DASR, such as
3 Speaker-Turn Aware Conversational Speech Translation (STAC-ST)
ST: [ES] [EN] utterance translation. ASR: [ES] [ES] transcripción de enunciados. At inference time, both language tokens are preset to specify the desired task.
This section describes our end-to-end multi-task learning model for multi-turn multi-speaker con- versational ST.
3.1 System Diagram
[TURN] and [XT] specify the auxiliary tasks of detecting speaker-turn changes and cross-talks, which are critical for MT-MS speech processing and more aligned to acoustic tasks. Note that cross- talks always occur during speaker-turn changes, so [XT] always follows [TURN].
Figure 2 illustrates the proposed STAC-ST multi- task learning framework for MT-MS ST. The model is an encoder-decoder Transformer architecture in- spired by Vaswani et al. (2017). The multitask training format using special tokens ($3.2) was in- spired by Whisper (Radford al., 2023), while the integration of Connectionist Temporal Classifica- tion (CTC) loss ($3.3) was inspired by Watanabe et al. (2017).
We concatenate transcripts or translations quentially, inserting [TURN] and [XT] tokens when needed. If utterances u, and ut+1 overlap in time, we append the targets of utterance Ue: after utter-
STAC-ST has a standard front-end module. First,
"The audio is always down- or up-sampled to 16 kHz.

---

ance uy The order of utterances is determined by their start time. demonstration of such serializa- tion is shown below:
Fisher
CALLHOME
dev
dex2
test
Single-Tur Turn Duration [be]
172
4,6
4.7
4.5
14.7
3.8
1.8
Single-Tur MT-MS Duration [hr]
[k]
155
4.1
4.1
4.1
13.8
3.5
1.7
CHANNEL 1: |WORD1|
|WORD2 WORD3
...|
1.9k
CHANNEL 2:
word1 word2|
Speech activity [%]
97
97
98
98
78
80
58
Overlap ratio [16]
Serialization: WORDI [TURN] word1 word2 [TURN]
[XT] NDRD2 WORD3
Table I:
corpus statisties.
3.3 Joint CTC and NLL Loss
0.25
STAC-ST jointly models ASR and ST by balanc- ing CTC (Graves al., 2006) and Negative Log- Likelihood (NLL) losses (Chan et al., 2016), ac- cording to:
0.20
Density
0.15
0.10
0.05-
C= -E-
(I)
0.00
Duration [seconds]
35
Single-Tum 8-
Multi-Turn Multt-Spaakker
SHAS
Lere and LNLL computed by appending lin- ear layers with dimension V on top of the encoder and decoder, respectively. Figure shows the pro- posed joint CTC/NLL loss training scheme (Watan- abe et al., 2017). practice, the CTC loss models a probabilistic distribution by marginalizing over all possible mappings between the input (audio fea- tures, sampled at 40 ms) and output sequence (tran- scription translation). We refer readers to the original implementation by Graves al. (2006), for more details. Moreover, CTC loss has been proven to aid ST by helping to stabilize encoder representations at early stages of training, i.c., al- lowing the decoder to learn soft alignment patterns faster (Yan et al., 2023). Note that we do not in- clude language tokens, [SL] and [TL], for Lere computation because they do not correspond to acoustic features. Following previous work (Zhang al., 2022, 2023a), we set the weight of the CTC loss to 0.3.
Figure 3: Fisher-CALLLOME test set distribution of segment length with three different segmentation ap- proaches: single-turn, MT-MS, and SHAS.
statistics in Table 1. This corpus is well suited for MT-MS ST, as it contains a significant amount of labeled data and non-segmented (audio) long conversation between speakers. We merged Fisher and CALLHOME for training and up-sampled the audio to 16 kHz.
Segmentation. Each conversation on Fisher- CALLHOME occurred between two speakers with multiple turns over channels (one speaker per channel). For MT-MS ST experiments, we merge the two channels into one, which creates natural speaker changes and cross-talks as illus- trated Figure 1. Human annotations Fisher- CALLHOME provide time-aligned audio utter- ances, transcripts and translations, and have been used to segment each channel into single-turn ut- terances in prior work (e.g., Inaguma et al., 2019) Figure plots the distributions of segment dura- tion in the corpus. We observe that the majority of single-turn segments are less than seconds long. To build models with manageable size and compu- tation, following Radford et al. (2023), we segment the cearged-channel conversations into chunks of up to 30 seconds. For this step, we first used an off fff-the-shelf VAD-based segmentation tool, SHAS (Tsiamas et al., 2022), but realized that the sulting duration histogram is almost uniform and far from the natural segmentation. Hence, we de- cided to rely on the manual time annotations as follows. Starting from the first utterance start, we
4 Experimental Setup
This section introduces the datasets and metrics we used for evaluation, as well as architecture and training details of STAC-ST.
4.1 Conversational Multi-Ture & Multi-Speakerr ST
We use the Fisher and CALLHOME corpora which respectively comprises 186 hr and 20 hr of audio and transcripts of telephone conversations in Span- ish.3 The Spanish-to- toeiish-tt-Ennn longnllssh translations are avail- able from Post et al. (2013). We refer to them Fisher-CALLHOME and summarize the data
"LDC2010001, LDC2010T04, LDC96S35, LDC96T17

---

find the farthest utterance end such that end-start is up to 30 seconds. We extract audio within this span as one segment and repeat this procedure un- the last utterance end is reached. Note that one segment may stretch over multiple utterance start and end, so it may include silences, noise, speaker changes and cross-talks. We use this as the primary MT-MS segmentation strategy for both training and test data throughout the paper unless otherwise stated. More discussions can be found in Section 5.
larger sizes when out-of-domain training data are added. We apply BPE sub-wordh (Sennrich et al., 2016) on both translations and transcripts with 5K operations. We create a joint BPE model for the language pair or when we add CV+CoVoST2 cor- pora (only $5.3.2 and $5.3.3).
We train for 100k steps the S-size models and 200k steps the M- and L-size models. We use AdamW (Kingma and Ba, 2015) optimizer with a peak learning rate of 5e for the S model and le for M and L models. The learning rate sched- uler has warmup and cooldown phases, both taking 10% of the total training steps (Zhai et al., 2022). We set dropout (Srivastava et al.. 2014) to 0.1 for the attention and hidden layers, and use GELU (Gaussian Error Linear Units) the activation function (Hendrycks and Gimpel, 2016). We use gradient norm clipping (Pascanu al., 2013) and SpecAugment (Park et al., 2019) for data augmen- tation. The training configuration and architecture are based on LibriSpeech recipe for Transformer- based ASR from the Speech et al., 2021).?
4.2 Additional ASR & ST Corpora
Fisher-CALLHOME has limited training data size, so we explore additional corpora to improve our model and to evaluate its generalization ability. We also use the official CoVoST 2 (Wang et al.. 2021) splits for Spanish-English ST (156 hr) and Common Voice* (CV, Ardila et al., 2020) splits for Spanish ASR (458 hr) as additional training data. Even though these corpora are not in the conversa- tion domain, they may still help speech modeling in general.
SpeechBrain Brain toolkit (Ravanelli
CoVoST 2 and CV corpora are composed of single-turn pre-segmented utterances. To generate data consistent with our MT-MS segmentation, randomly concatenate audio utterances and yield segments of up to 30 seconds. Note that these syn- thetic MT-MS segments contain no silences and cross-talks, but still have speaker-turn changes (la- beled by [TURN]).
5 Results
Our experimental results document three properties of the STAC-ST model: (1) robustness to the MT- MS ST condition with no degradation in the single- turn ST condition; (2) ability to leverage speaker- turn and cross-talk information, which translates into improved WER and BLEU scores; (3) ability to perform time-aligned speaker change detection.
4.3 Evaluation Metrics
We report case-insensitive BLEU using Sacre- BLEU (Post, 2018) for translation and Word Error Rate (WER) for ASR. Note that we (1) remove all special task tokens before computing each metric and (2) evaluate on MT-MS segmentation unless otherwise stated.
5.1 Multi-Task Learning
We explored various training data configurations for multi-task learning (see Table 2). Row-0 in Ta- ble 2 represents how conventional ST system (i.c., trained on only single-turn ST data) performs under the challenging multi-turn multi-speaker scenario. Other systems in Table 2 yield insights into how to boost the performances by augmenting the training data with auxiliary tasks.
4.4 Hyper-Parameters
We experiment with three model sizes, S(mall), M(edium), and L(arge), with increasing dimension (256, 512, 1024), number of encoder layers (12, 14, 16), number of heads (4, 8, 16), with same number of decoder layers (6) and FFN dimension set to 4x the model dimension. Their numbers of parame- ters are 21M, 86M, and 298M, respectively. We use the S-size model by default and scale up to
Joint training of single-turn and multi-turn tasks is beneficial. Adding multi-turn ST data for training gives marginal improvements (Row- I vs. Row-0); this suggests that simply adding limited multi-turn data will not suffice for the MT- MS cases. When either single-turn or multi-turn
'Version: cv-corpus-13, 1..0011..222222 0-202--0----999
"mer_grad norm = 5.0.
'Signature: nrefs:N| case: 1c/ [ tok: 13a| smooth: exp serpvvisiin: 3.1. 1. (Fisher N and CALLHOME N=1).
'https://giit thub. com httm//ppecchraii/ppechhaaaaaa htterdeesspprcccpess////bbeeeeee

---

Training data configuration
Fisher
CALLHOME
Fisher
CALLHOME
Single-Tun Turn
Multi-Turn
WER
BLEU
WER
BLEU
Task tokens
WER L
BLEU
WER
BLEU 1
ASR
ST
ASR
ST
(1)
(t)
(L)
(1)
[SL]. [TL]
26.4
45.0
43.7
16.6
0)
28.3
8.5
1)
30.9
8.7
[[URN]
25.8
45.2
43.1
17.6
+ +XTT] 25.8
46.8
42,1
17.9
2)
40.2
29.3
57.9
8.9
3)
29.4
41.5
49.9
14.7
4) /
25.8
46.8
42.1
17.9
Table 3: ASR and ST performance of STAC-ST with the incremental addition of task tokens. Modeling speaker- turn and cross-talk detection with [TURN] and [XT] tokens enhances ASR and MT accuracy.
V
25.8
35.6
42.3
11.7
6 V
44.9
43.7
68.2
15.5
Table 2: ASR and ST performance of STAC-ST with different training data configurations. Joint training with single-tun turn and multi turtii-uum turn data of both ASR and ST tasks achieves the hest scores.
2023). However, this can be fixed by adding back multi-turn ASR data (Row-4). Note that we use the Row-4 data configuration for the rest of the paper.
data has reasonable size (i.e., augmenting ASR data), combining them yields more pronounced im- provements (Row-4 vs. Row-2/Row-3). Although single-turn and multi-turn data share the same utter- ances, split/concatenation- based data augmentation is known be effective in the low-resource training regime (Nguyen al., 2021; Lupo et al., 2022).
5.2 Speaker- and Cross-Talk Detection
The STAC-ST multi-task learning framework also encodes speaker-turn and cross-talk information with task tokens [TURN] and [XT]. We run exper- iments to study how these task labels impact on ASR and ST performance in MT-MS setting and how they even enable speaker change detection.
Joint training of ST and ASR is beneficial. In- terestingly, training a model with only multi-turn ST data failed to converge, but adding multi-turn ASR data stabilizes the training (Row-3). 8 More- over, by adding both single-turn and multi-turn ASR data for joint training on top of Row-1, 1, both BLEU and WER are improved by significant mar- gin (Row-4).
Modeling speaker-turn and cross-talk detection helps multi-speaker ST and ASR. We run ex- periments by ablating the two task tokens. Eval- uation results in Table 3 show that incrementally adding speaker-turn and cross-talk detection tasks improves translation and transcription quality mea- sured by BLEU and WER. These results support the hypothesis that explicitly learning the two tasks helps the model to better handle MT-MS scenarios.
Multi-turn ASR data helps multi-turn ST. In our training data, there are more labeled single-turn ST data and multi-turn ASR data than multi-turn ST data. We tested zero-shot setting where, for the multi-turn condition is only covered by ASR training data (Row-5). Comparing to training with single-turn ST+ASR data only (Row-2), the result- ing model brings 3-8 BLEU gains. We hypothesize that, as the encoder is aaggr-language-agnosti the acoustic representations and the turn detection capacity learned from multi-turn ASR data does partially transfer to the ST task.
Modeling speaker-turn and cross-talk detection enables the model to perform speaker change detection. The CTC loss helps the encoder to align input audio to text tokens per acoustic frame, including the two task tokens. We trace speaker- turns and cross-talks in the timeline by (1) first rumning forward pass the encoder to extract audio-text temporal alignments and then we (2) locate the spikes of the linear layer on top of the encoder (aka. CTC spikes) only for [TURN] and [XT] tokens. As illustrated in Figure 4, the CTC spikes align remarkably well with actual edges of speaker activities.
Multi-turn ST does not seem to help multi- turn ASR. This can be seen by comparing WER scores in Row-2 and Row-6. We hypothesize that the con-ooootonycity of the multi-turn ST task disrupts multi-turn ASR performance (Yan al.,
By leveraging available annotations in Fisher- CALLHOME test sets, we measure speaker change detection performance with three standard met- rics: False Alarm Rate (FAR), Miss Detection Rate (MDR) and Fl-score. The FAR computes the rate at which STAC-ST outputs a [TURN] CTC spike
*Combining single-turn utterances to create longer (max 30s) multi-tum segments greatly reduces the number of train- ing samples.

---

Activity CH2
Activity CH1
[TURN] spikes [XT] spikes
5
10
15
20
25
Time [seconds]
Figure 4: Speaker activity on Fisher corpus sample. the top, ground truth human annotation on two audio channels. On the bottom, CTC spikes of turn and cross-talk tokens detected by STAC-ST in the merged channel.
Fisher
CALLHOME
5.3.1
MT-MS vs. VAD Segmentation
System
F1
1
FAR
Fl
L
FAR
A common practice for translating long-form dio files is to first segment them into smaller chunks based on voice activity detection (VAD). 13.2
PyAnmote
75.8
26.8
21.4
81.2
20.9
15.0
STAC-ST
74.9
31.3
17.7
80.6
25.6
STAC-ST (L.
77.6
28.6
15.0
81.3
23.5
We compare our MT-MS segmentation approach with two popular VAD-based audio segmenters, i.e., WebRTC (Blum et al., 2021) and SHAS (Tsia- mas et al., 2022), on the channel-merger Fisher- CALLHOME test sets." 10
Table 4: Speaker change detection performance mea- sured by FI, MDR and FAR. We compare STAC-ST with PyAnnote. The strongest L-size STAC-ST model (from Table 5) shows on-par Fi-score with PyAnnote. Tolerance is set to 0.25s.
When the audio and reference translation seg- ments are not aligned, like the case of VAD- based segmentation, the standard process to first concatenate translation hypotheses and then align and re-segment the conversation-level translation based on the segmented reference translation." provides overall assessment of the performance.
when there are actually no speaker changes. The MDR computed the rate that STAC-ST misses gen- erating [TURN] tokens at speaker changes. While the former two are widely used in speaker segmen- tation research (Bredin et al., 2020), the Fl-score
However, our preliminary results show that this process yields poor BLEU scores, partially because VAD treats noise as speech, which leads noisy translation and misalignment. Therefore, we cal- culate BLEU scores on concatenated hypotheses and references for the whole conversation. BLEU scores in this section are not comparable with the ones reported elsewhere.
To compute these metrics, we first prepare Rich Transcription Time Marked (RTTM) files for each test set from the time-aligned CTC [TURN] spikes. We compared performance of two STAC-ST models and L) against a reference system, the speaker segmentation pipeline of the popular PyAnnote toolkit (Bredin and Laurent, 2021).9 From results listed in Table 4, STAC-ST gets on-par F1 F1-score vs. the reference system in the Fisher-CALLHOME: test sets. Using a stronger STAC-ST (L) model im- proves by 2.5 absolute the Fl score. These results corroborate the importance of the [TURN] task to- kens for improving ASR and ST quality.
As shown Figure 5, for both Fisher and CALL- HOME test sets, BLEU scores of using VAD-based tools (either WebRTC or SHAS) for test data seg- mentation are below the ones using our MT-MS segmentation. Despite being popular in conven- tional speech translation, segmenting forg-form audio with VAD-based tools is not the best choice for handling multi-talks conversations with speaker- turns. Thus, we resort to using MT-MS segmenta- tion based on human annotations for preparing the test data. This highlights a potential future work rection of producing robust segmentation on noisy long-form conversational audio.
5.3 Benchmarking STAC-ST
We run extensive benchmarks compare STAC-ST with related work in various settings, including (1) different audio segmentation strategies, (2) model size, and (3) evaluation on single-turn ST.
MoMe details in Appendix F.
'nttps: wttpaaant.cp//aaceee//eea goeaker-segmentation
(Matusov et al., 2005) has been used in IWSLT (Anastasopoulos et al., 2022, 2021) for this purpose.

---

31
Hisher
CALLIOME
30
26
Model
WERJ
BLEU
WER
BLEUT
0.0.0.0
Casc. ST (Post al., 2013)
36.5
65.3
11.6
BLEU
18
Multi-task (Weiss et al,, 2017)
23.2
48.7
45.3
17.4
E2E-ST (Inaguma et al, 2019)
22.9
46.3
44.5
17.2
ESPnet example (2022)
18.7
50.5
37.6
21.7
Whisper-tiny (99M)
58.5
Whisper-base (7-M)
48.7
Fisher
CALLHOME
Whisper-small (2HM)
Segmentation Technique
STAC-ST (21M)
20.9
49.1
36.3
20.1
MTEMS
WebRCT Moaae
SHAS
STAC-ST ST M (86M)
18.9
52.3
31.4
22.1
STAC-ST (298M)
18.8
52.6
31.0
22.4
Figure 5: ST performance on Fisher-CALLHOME test data using different segmentation techniques for form-form audio: MT-M- (ours), WebRTC, and SHAS. BLEU scores of using VAD-based tools (either We- bRTC SHAS) for test data segmentation are lower than BLEU computed using our MT-MS segmentation.
Table 6: ASR and ST performance with the official single-speaker manual segmentation. Previous work re- sults and Whisper baselines are provided. Our strongest model, STAC-ST L yields the best scores.
5.3.3 STAC-ST for Single-Turn ST
Fisher
CALLHOME
To position STAC-ST against previous work ST, we also run experiments under the conventional single-turn ST condition. These experiments en- able us to (1) see how our end-to-end multi-tase learning approach performs on a specific input con- dition, and (2) compare STAC-ST against four pre- vious models trained and evaluated on the same task. To allow for comparing results across single- turn and MS-MT conditions, formance with three Whisper systems. Results of these experiments are reported in Table 6. We observe that all our STAC-ST models are competi- tive with the previous models, also optimized on the Fisher-CALLHOME task. Comparison against the Whisper models confirms the trends observed in Table under the MS-MT condition. Overall, STAC-ST L yields the best BLEU scores both Fisher and CALLHOME.
Model
WER
BLBUT 1
WER
BLEU
Whisper-tiny (39M)
11.5
59.8
Whisper-base (74M)
36.7
29.0
49.2
8.4
Whisper-small (244M)
29.1
46.7
37,9
19.2
STAC-ST S (21M)
25.8
46.8
42.1
17.9
STAC-ST M (86M)
23,8
49.4
38.3
20.4
STAC-ST T. (298M)
23.5
50.0
38.5
21.0
Table 5: ASR
performance with increasing model size of STAC-ST and Whisper. STAC-ST achieves better BLEU and WER scores than Whisper with com- parable model sizes.
also report per-
5.3.2 Scaled STAC-ST vs. Whisper
Given the lack of prior work on MT-MS ST, we compare STAC-ST against strong multi-task model, i.c., Whisper (Radford et al., 2023). Whis- per is trained with over 2,000 times more speech data than our model (although CatLe--MLLLOMEE is not included among them) and its smallest ver- sion is larger than STAC-ST S. To enable more added more speech training data (cf. $4.2) to STAC-ST with size M and L.
Conclusions
In this work, we present STAC-ST, end-to-end system designed for single-channel multi-turn & multi-speaker speech translation that uses multi- task training framework to leverage both ASR and ST datasets. We demonstrate that STAC-ST gener- alizes to both standard pre-segmented ST bench- marks and multi turn conversational ST, the latter being a more challenging scenario. STAC-ST is also shown learn the task of speaker change de- tection, which helps multi-speaker ST and ASR. We investigate different aspects of STAC-ST, in- cluding the impact of model and data size, auto- matic segmentation for long-form conversational ST, zero-shot multi-turn multi spetkrs ST with-
fair comparison,
Results in Table demonstrate that when we add out-of-domain training data and scale the model accordingly (Kaplan et al., 2020; Bapna et al., 2022; Zhai et al., 2022), STAC-ST achieves better BLEU and WER scores than Whisper with comparable model sizes, although our training data is still three orders of magnitude smaller.
12 /tttesss//ttttto.coossseeere/ses

---

out specific training data. Overall, this work sheds light on future work towards more robust conver- sational ST systems that can handle speaker-turns and cross-talks
References
Antonios Anastasopoulos, Loic Barrault, Luisa Ben- tivogli, Marcely Zanon Boito, Ondřej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewiez, Barry Haddow, Benjamin Hsu, Dávid Javorský, Véra Kloudová. Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nädejde, Satoshi Nakamura, Mattco Negri, Jan Niehues, Xing Niu, John Ortega, Juan Pino, Eliz- abeth Salesky, Jiatong Shi, Matthias Sperber, Se- bastian Stüker, Katsuhito Sudoh, Marco Turchi, Yo- gesh Virkar. Alexander Waibel, Changhan Wang, and Shinji Watanabe. 2022. Findings of the IWSLT 2022 evaluation campaign. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 98 157, Dublin, Ireland (in-person and online). Association for Com- putational Linguistics.
Limitations
1. Our primary test sets, Fisher and CALL- HOME, have marrowly one translation direc- tion (Spanish-Enngiish). The only other pub- lic conversational ST dataset we are aware of is MSLT (Federmann and Lewis, 2016), but it only contains independent utterances, which is far from representing a realistic MT-MS use case. We call for more publicly available long- form conversational ST data under a friendly license.
2. Due to the same limitation of publicly avail- able datasets, we do only explore conversa- tions between two speakers.
Antonios Anastasopoulos, Ondiej Bojar, Jacob Bremer- man, Roldano Cattoni, Maha Elbayad, Marcello Fed- erico, Xutai Ma, Satoshi Nakamura, Matteo Negri, Jan Nichues, Juan Pino. Elizabeth Salesky, Sebas- tian Stüker, Katsuhito Sudoh, Marco Turchi, Alexan- der Waibel, Changhan Wang, and Matthew Wiesner. 2021. FINDINGS OF THE IWSLT 2021 EVAL- UATION CAMPAIGN. In Proceedings of the 18th International Conference on Spoken Language Trans- lation (IWSLT 2021), pages 1-29, Bangkok, Thailand (online). Association for Computational Linguistics.
3. We segment the test sets based on human an- notations. Despite being the best choice for the MT-MS data in our study ($5.3.1), it is not a realistic scenario for testing. We leave improving segmentation on noisy long-form conversational audio as future work.
4. We segment long-form audio files into up to 30s pieces following Radford et al. (2023). but we do not use the preceding segments as context. We focus on improving translation quality of conversations by speaker-turn and cross-talk detection, yet using the context in- formation could also help. In addition, within each MT-MS segment, the inter-utterance con- text could have already been leveraged (Zhang et al., 2021). We leave analysis of the inter- and intra-segment context as future work.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gre- gor Weber. 2020. Common voice: A massively- multilingual speech corpus. In Proceedings of the Twelfth Language Rexources and Evaluation Confer- ence, pages 4218 42122 Marseille, France. European Language Resources Association.
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli 2022. XLS-R: Self-supervised Cross-lingual Speech Rep- resentation Learning at Scale. In Proc. Interspeech 2022, pages 2278-2282.
5. We only test the Transformer architecture as we focus on solving a challenging MT-MS ST task with tulti-task learning, which is or- thogonal to the architecture choice. We leave exploring other architecture options, such as Conformer (Radfar et al., 2023), HyperCon- former (Mai et al., 2023) or Conmer (Radfar et al., 2023) as future work.
Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng. Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022. mSLAM: Massively mul- tilingual joint pre-training for speech and text. CoRR, abs/2202.01374.
Ethical Considerations
Alexandre Berard. Olivier Pietquin. Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-tex trans- lation. CoRR, abs/1612.01744.
All speech datasets we use have anonymous speak- ers. We do not have any access nor try cre- ate any PII (Personal Identifiable Information) of speakers, and our model neither identifies speakers nor uses speaker embeddings.
Niklas Blum, Serge Lachapelle, and Harald Alvestrand. 2021. WebRTC: Real-Time Communication for the Open Web Platform. Commun. ACM, 64(8):50-54.

---

William Brannon, Yogesh Virkar, and Brian Thompson.
7637-7650, Toronto, Canada. Association for Com- putational Linguistics.
2023. Dubbing in practice: A large scale study of hu- man localization with insights for automatic dubbing.
Transactions of the Association for Computational Linguistics, 1 11:19-435.
Alex Graves. Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learn- ing. ICML '06, page 369-376, New York, NY, USA. Association for Computing Machinery.
Hervé Bredin and Antoine Laurent. 2021, End-To-End Speaker Segmentation for Perrae--waar Resegmen- tation. In Proc. Interspeech 2021, pages 3111-3115
Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gre- gory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. 2020. Pyannote. audio: Neural building blocks for speaker diarization. ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 7124-7128.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian Er- ror Linear Units (GELUs). CoRR,
Zili Huang, Desh Raj. Paola Garefa, and Sanjeev Khu- danpur. 2023. Adapting self-suuervised models to multi-talker speech recognition using speaker embed- dings. In ICASSP 2023 2023 IEEE International Conference Acoustics, Speech and Signal Process- ing (ICASSP), pages 1-5.
Francisco Casacuberta, Marcello Federico, Hermann Ney, and Enrique Vidal. 2008. Recent efforts in spoken language translation. JEEE Signal Processing Magazine, 25(3):80-88.
Hirofumi Inaguma, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. 2019. Multilingual end-to-end speech translation. In 2019 IEFF Automatic Speech Recognition and Understanding Workshop (ASRU). pages 570-577.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: neural network for large vocabulary conversational speech recognition. In 2016 IEEÉ International Confer- ence on Acousticx, Speech and Signal Procexsing (ICASSP), pages 4960-4964.
Naoyuki Kanda, Yashesh Gaur, Xiaofei Wang. Zhong Meng, and Takuya Yoshioka. 2020. Serialized Out- put Training for End-to-End Overlapped Speech Recognition. In Proc. Interspeech 2020, pages 2797- 2801.
Keqi Deng. Shinji Watanabe. Jiatong Shi, and Siddhant Arora. 2022. Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation. In Proc. Interspeech 2022, pages 1746-1750.
Naoyuki Kanda, Jian Wa, Yu Wu, Xiong Xiao, Zhong Meng. Xiaofei Wang, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Takuya Yoshioka. 2022. Streaming Mulki-Talker ASR with Token-Level Serialized Out- put Training. In Proc. Interspeech 2022, pages 3774- 3778.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: Multilingual Speech Translation Corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2012-2017, Min- neapolis, Minnesota. Association for Computational Linguisties.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, abs/2001.08366
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Thierry Etchegoyhen. Haritz Arzelus, Harritxu Gete, Aitor Alvarez, Iván G. Torre, Juan Manuel Martín- Doñas, Ander González-Docasal, and Edson Benites Fernandez. 2022. Cascade or direct speech transla- tion? a case study. Applied Sciences, 12(3).
Ali Can Kocabiyikoglu, Laurent Besacier, and Olivier Kraif. 2018. Augmenting librispeech with French translations: A multimodal corpus for direct speech translation evaluation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. Eu- ropean Language Resources Association (ELRA).
Christian Federmann and William D. Lewis. 2016. Mi- crosoft speech language translation (MSLT) corpus: The IWSLT 2016 release for English, French and German. In Proceedings of the 13th International Conference on Spoken Language Translation, Seattle, Washington D.C. International Workshop on Spoken Language Translation.
Gaurav Kumar, Yuan Cao, Ryan Cotterell, Chris Callison-Burch, Daniel Povey, and Sanjeev Khudan- pur. 2014a. Translations of the callhome Egyptian Arabic corpus for conversational speech translation. In Proceedings of the 1 11th International Workshop on Spoken Language Trunslation: Pupers, pages 244- 248, Lake Tahoe, California.
Mozhdeh Gheini, Tatiana Likhomanenko, Matthias Sperber, and Hendra Setiawan. 2023. Joint speech transcription and translation: Pseudo-labeling with out-of-distributiin data. In Findings of the Associa- tion.or Computational Linguistics: ACL 2023, pages

---

Gaurav Kumar, Matt Post, Daniel Povey, and Sanjeev Khudanpur. 2014b. Some insights from translating conversational telephone speech. In 2014 IEFE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3231-3235.
Matt Post. 2018. call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages I 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Yuhao Liang, Fan Yu, Yangze Li, Pengcheng Guo, Shil- iang Zhang, Qian Chen, and Lei Xie. 2023. BA- SOT: Sooundayy-Warr Serialized Output Training for Multi-Talker ASR. In Proc. INTERSPEECH 2023, pages 3487-3491.
Matt Post, Gaurav Kumar, Adam Lopez. Damianos Karakos, Chris Callison-Burch, and Sanjeev Khu- danpur. 2013. Improved speech-to-text translation with the fisher and callhome Spanish-Englh speech translation corpus. In Proceedings of the 10th Inter- national Workshop Spoken Language Translation: Papers, Heidelberg. Germany.
Lorenzo Lupo, Marco Dinarelli, and Laurent Besacier. 2022. Divide and rule: Effective pre-training for context-aware multi-encoder translation models. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 4557 4572, Dublin, Ireland. As- sociation for Computational Linguistics.
Martin Radfar, Paulina Lyskawa, Brandon Trujillo, Yi Xie, Kai Zhen, Jahn Heymann, Denis Filimonov, Grant P. Strimel, Nathan Susanj, and Athanasios Mouchtaris. 2023. Conmer: Streaming Conformer Without Self-attention for Interactive Voice Assis- tants, In Proc. INTERSPEECH 2023, pages 2198- 2202.
Florian Mai. Juan Zuluaga-Gome Titouan Parcollet, and Petr Motlicek. 2023. HyperConformer: Multi- head HyperMixer for Efficient Speech Recognition. In Proc. INTERSPEECH 2023, pages 2213-2217.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine Mcleavey, and llya Sutskever. 2023, Robust speech recognition via large-scale weak su- pervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages PMLR.
Evgeny Matusov, Gregor Leusch, Oliver Bender, and Hermann Ney. 2005. Evaluating machine translation output with automatic sentence segmentation. In Pro-
ceedings of the Second International Workshop on Spoken I Language Translation, Pittsburgh, Pennsylva- nia, USA.
Desh Raj. Liang Lu, Zhuo Chen, Yashesh Gaur, and Jinyu Li. 2022. Continuous streaming multi-talker asr with dual-path transducers. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7317- 7321.
Toan Q. Nguyen, Kenton Murray, and David Chiang. 2021. Data augmentation by concatenation for low- resource translation: A mystery and a solution. In Proceedings of the 1 18th International Conference on Spoken Language Translation (/WSLT 2021). J. pages 287-293, Bangkok, Thailand (online), Association for Computational Linguistics.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga. Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. 2021. Speechbrain: A general- purpose speech toolkit. CoRR, abs/2106.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph. Ekin D. Cubuk, and Quoc V. Le. 2019. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Proc. Interspeech 2019, pages 2613-2617.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Pro- ceedings of Machine Learning Research, pages 1310- 1318, Atlanta, Georgia, USA. PMLR.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1722 1725, Berlin, Germany. Association for Computational Lin- guistics.
Rohit Paturi, Sundararajan Srinivasan, Katrin Kirchhoff, and Daniel Marcia-Romero. 2022. Directed speech separation for automatic speech recognition of long form conversational speech. In Proc. Interspeech 2022, pages 3388-5392.
Kak Soky, Sheng Li, Masato Mimura, Chenhui Chu, and Tatsuya Kawahara. 2022. Leveraging Simultane- ous Translation for Enhancing Transcription of Low- resource Language via Cross Attention Mechanism. In Proc. Interspeech 2022, pages 1362-1366.
Yifan Peng, Kwangyoun Kim, Felix Wu, Brian Yan, Siddhant Arora, William Chen, Jiyang Tang. Suwon Shon, Prashant Sridhar, and Shinji Watanabe. 2023.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhuudinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Much. Learn. Res., 15(1:1929 1958.
A Comparative Study on F--Branchformer vs Con- former in Speech Recognition, Translation, and Un- derstanding Tasks, In Proc. INTERSPEECH 2023, pages 2208-2212.

---

Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNILP) pages 1342- 1348, Hong Kong, China. Association for Computa- tional Linguistics.
Peidong Wang, Eric Sun, Jian Xue, Yu Wu, Long Zhou, Yashesh Gaur, Shujie Liu, and Jinyu Li. 2023. LAMASSU: A Streaming
trrguage-Agnosiie Multi- lingual Speech Recognition and Translation Model Using Neural Transducers. In Proc. INTERSPEECH 2023, pages 57-61.
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. 2017. Hybrid CTC/Attention Architecture for End-to-Enn to-End Speech Recognition. IEEE Journal of Selected Topics in Signal Processing, 1188):1240-1253
Brian Thompson and Philipp Koehn. 2020. Exploiting sentence order in document alignment. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5997-6007, Online. Association for Computational Linguisties.
Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-Seqqeee Models Can Directly Translate Foreign Speech. In Proc. Interspeech 2017 7, pages 2625-2629.
Toannis Tsiamas, Gerard I. Gállego, José A. R. Fonol- losa, and Marta R. Costa-jussà, 2022. SHAS: Ap- proaching optimal Segmentation for End-to-End Speech Translation. In Proc. Interspeech 2022, pages 106-110.
Jian Xue, Peidong Wang, Jinyu Li, Matt Post, and Yashesh Gaur. 2022. Large- Scale Streaming End- to-End Speech Translation with Neural Transducers. In Proc. Interspeech 2022, pages 3263-3267.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, and Shinji Watanabe. 2023, CTC aligmments improve autore- gressive translation. In Proceedingy of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics, pages 1623-1639, Dubrovnik, Croatia Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Pro Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
Muqiao Yang. Naoyuki Kanda, Xiaofei Wang. Jian Wu, Sunit Sivasankaran, Zhuo Chen, Jinyu Li, and Takuya Yoshioka. 2023. Simulating realistic speech overlaps improves multi-talker asr. In ICASSP 2023 2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Enrique Vidal. 1997. Finite-state speech-to-speech translation. 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol- ume pages 111-114 vol.1.
Marcely Zanon Boito, John Ortega, Hugo Riguidel. An- toine Laurent, Loic Barrault, Fethi Bougares, Firas Chaabani, Ha Ha Nguyen, Florentin Barbier, Souhir Gah- biche, and Yannick Estève. 2022. ON-TRAC con- sortium systems for the IWSLT 2022 dialect and Low-resource speech translation tasks. In Proceed- ings of the 19th International Conference on Spoken Language Translation (/WSLT 2022), pages 308 318, Dublin, Ireland (in- ter-person and online). Association for Computational Linguistics.
Yogesh Virkar, Brian Thompson, Rohit Paturi, Sun- dararajan Srinivasan, and Marcello Federico. 2023. Speaker diarization of scripted audiovisual content. CORR, abs/2308.021es
Alex Waibel, Ajay N. Jain, Arthur E. McNair, Hiroaki Saito, Alexander G. Hauptmann, and Joe Tebelskis. 1991. JANUS: a speech-to-speedd translation system using connectionist and symbolic processing strate- gies. In [Proceedings) ICASSP 91: 1991 Interna- tional Conference on Acoustics, Speech, and Signal Processing, pages 993-796 vol.2.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlshy, and Lucas Beyer. 2022. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 12 12104-12113.
Changhan Wang. Juan Pino, Anne Wu, and Jiatao Gu. 2020. CoSo A diverse multilingual speech-to-text translation corpus. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 4197-4203, Marseille, France. European Language Resources Association.
Biao Zhang, Barry Haddow, and Rico Sennrich. 2022. Revisiting end-to-end speech-to-text translation from scratch. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26193-26205. PMLR.
Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021. CoVoST 2 and Massively Multilingual Speech Translation. In Proc, Interspeech 2021, pages 2247- 2251.
Biao Zhang, Barry Haddow, and Rico Sennrich. 2023a. Efficient CTC regularization via coarse labels for

---

end-to-end speech translation. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2264- 2276, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Biao Zhang, Ivan Titov, Barry Haddow, and Rico Sennrich. 2021. Beyond sentence-level end-to-end speech translation: Context helps. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 2566-2578, Online. Association for Computational Linguisties.
Yu Zhang, Wei Han, James Qin, Yongqiang Wang. Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng. Ke Hu, Andrew Rosenberg. Rohit Prabhavalkar, Daniel S. Park. Parisa Haghani, Jason Riesa, Ginger Pemng Hagen Soltau, Trevor Strohman, Bhuvana Ramab- hadran, Tara N. Sainath, Pedro J. Moreno, Chung- Cheng Chiu, Johan Schalkwyk, Françoise Beaufays, and Yonghui Wu. 2023b. Google USM:: scaling au- tomatic speech recognition beyond 100 languages. CoRR, abs/2303.01037.

---

Fisher
CALLHOME
Fisher
CALLHOME
Overlap Ratio
BLEUU
#words
BLEU
#words
S 6%
48.15
10,584
21.31
4,228
6% x: 11%
47.43
7,502
19.77
3,962
11% I 17%
45.79
6,901
16.27
4,709
Rate [%6
17% all
10,119
15.82
4,659
46.83
39,095
17.92
18,458
Table 7: We calculate the overlap ratio for each segment in Fisher and CALLHOME and then group the segment- level overlap ratios into bins. We report BLEU score and the number of words in reference within each bin.
0.0
61
02
0.3
0.6
9.0
9.5
8.2
04
65
weight
weight
Wrotadios butnet Corrigaadioss
Singlo-Turn
MT-MS
TOL
Fisher
CALLHOME
Figure Ablation of the CTC weight in the overall loss computation and its impact in BLEU and WERs
(S)
MDR
FAR
MDR
FAR
for Fisher and CALLHOME development & evaluation sets. Eirror bars show the standard deviation between dev/dev2/test sets for Fisher and devset/evlset for CALL- HOME. Single-turn and MS-MS results are shown with straight and dashed lines, respectively.
0.1
58.3
46.2
36.4
67.6
37.5
26.4
0.25
74.9
31.3
17.7
80.6
25.6
12.1
0.5
83.4
23.0
9.0
85.5
20.8
7.2
1
87.3
18.4
6.2
89.3
16.2
4.5
Table Performance STAC-ST on speaker change detection the multi-turn dataset for all official Fisher-
A Evaluating Different CTC Weights
CALLHOME test sets. Tolerance is ablated from 0.1 to second.
In this section, evaluate different CTC weights for joint ASR & ST training under the STAC-ST framework. We show in Figure 6 the results for different S-size models trained on the Fisher- CALLHOME corpora. We confirm that BLEU and WER scores achieve the best with a À = 0.3, akin to previous work (Zhang et al., 2022).
chosen bins are based on [0%, 25% 50%, 75%, 100%) percentiles found on Fisher and remain the same for CALLHOME. These results correspond to Row-4 in Table 2. We can see that the BLEU score decreases with increasing speech overlaps.
B Complete Main Evaluation Results on Fisher-CALLHOME
D More Examples and Analysis on Speaker-Turn and Cross-Taie Detection
We list complete main results on Fisher- CALLHOME corpora for all the official subsets.
Multi-Turn Segments. Table 9 lists BLEU while Single-Tune Turn Segments. For the sake of complete- we also report the performance of STAC-ST each subset of Fisher-CALLHOME with the de- fault utterance segmentation (single-turn). Table 11 lists the BLEU scores, while Table 12 list WER scores.
In Figure 7, we provide additional examples of ground-truth speaker activities vs. CTC spikes of [TURN] and [XT] task tokens (see $ 5.2). The title contains the sample ID, transcript and translation together with the [TURN] and [XT] task tokens.
scores for all subsets of Table 10 lists WER scores.
In Table we evaluate different tolerance values when computing the speaker change detection met- rics con both Fisher-CALLHOME test sets. The tolerance (in seconds) allows us to reduce the gran- ularity that we expect in speaker change detection. Giving the fact that STAC-ST is not directly opti- mized for this task, note that value of at least 0.25 critical to reach acceptable scores by in- creasing the tolerance from 0.1 to 0.25 seconds, we see 22% relative increase in F1 score. Set- ting to 0.5 seconds further brings a 10% relative improvement.
Impact of Speech Overlap Ratio
In MT-MS data, each segment contains different de- gree of overlaps. We calculate the overlap ratio for each segment Fisher and CALLHOME, group the segment-level overlap ratios into 4 bins, and report BLEU scores for each bin in Table The

---

20051115 2112222223555 516 Sp-.6642655-045551
TRANCHIPION: yo creo que la techologia del teléfono han echo avances tamblén porque ya puedo hacer lamadas de lorgas distancles y no me valen nads porque poga cuota
[oum][x] y estados unidos
llamadas que una quien [tum] paro [turn]
años
squi
TRANSLATION: ithikk phone technology has made progress because ican also make long distance phone calls and do not have to pay ah monthly fee [turn] Itum] you years like
calls you wore on [turn] [at] wasn Sappsst united yos
Sappoctt
Activity CH2
Activity CH1
[TURN] spikes [XT] spikes
10
15
20
25
Time (in seconds)
20051102_ 1804 391 029007992111111 631611
TRANCHITION asi que [tum] (xt) pero pero y qué opinas de que esca de que no va tener came compafieros de escuela aso ese [tum] (xt) bueno [tum] coo experienca
[pum] muchos sdu TRANSI ATON: sa [umn [ell what do think about
programas para gente que enseffan
casa
school met ex [tunn [ [xi] well [hurn] tha kuppoçc also [turn) yes there are many programs here mismi for people who teach their children at home [burn) [sl ahe [Tum] then usullly once week they together [tum]
bone
Activity CH2
Activity CH1
[TURN] spikes [XT] spikes
10
15
20
Time (in seconds)
20051030__191000 193924_911111 371, 1p--000011500 008128
TRANCRIFTTON:
ch me pusta modica con rtme tamblén me gusta ballar [bum] okay [tum] to gusta ballar [tum] of me gusta para hablar tamblén [tum] oh que bien [tum) yo
era joven
ballo
[tum] (xt] yo tamblén ahora
estoy
TRANSI ATION: wh like music with rythen alo lioe dance [tum] ok [tum] do like dance [tum] yos also like talk well well [turn] oh that is good [tum] li danced more wheni was young but now idon kappect dance as much st stooped [urnd] [xt] ah me too now dince when Aaprimm alone cleaning my house are you [urn] you Suposom married
children [tum]
6apost children
Activity CH2
Activity CH1
[TURN] spikes
[XT] spikes
5
10
15
20
25
30
Time (in seconds)
Figure 7: Ground-truth speaker activities and CTC spikes of [TURN] and [XT] task tokens three randomly selected Fisher samples. The Tile list the ID (recording, file number, start and end time), the ground truth transcript and translation.

---

Training Data
BLEU score (t)
Single-turn
Multi-turn
Fisher
CALLHOME
ASR
ASR
ST
dev
dev2
test
AVG
devtest
evitest
AVG
26.2
27.0
28.3
27.2
8.6
8.5
8.5
30.31
30.5
30.9
30.5
9.5
8.7
9.1
25.6
27.0
29.3
27.3
8.8
8.9
8.8
40.2
40.0
41.5
40.5
15.0
14.7
14.8
32.7
32,9
35.6
33.7
10.6
11.7
11.1
42.3
42.5
43.7
42.8
15.2
15.5
15.4
45.1
46.1
46.8
46.0
18.4
17.9
18.2
Table BLEU scores on each multi-turn dataset the official Fisher-ALLLHOE development test subset. AVG lists the average between dev and test sets.
Training Data
Word Error Rate
Single-turn
Multi-turn
Fisher
CALLHOME
ASR
ST
ASR
ST
dev
dev2
AVG
deviest
evitest
AVG
29.7
30.0
26.1
28.6
44.0
43.5
43.8
d
45.9
46.6
40.2
44.2
58.0
57.9
58.0
35.2
35.8
29.4
33.5
51.4
49.9
50.7
r
29.4
30,0
25.8
28.4
42.9
42.3
42.6
54.6
44.9
50.8
64.3
68.2
66.3
V
30.2
29.6
25.8
28.5
42.6
42.1
42.4
Table 10: WERs each multi-turn dataset for all the official Fisher-CALLHOME developpeed and test subset. AVG lists the average between dev and test sets.
Training Data
BLEU score (t)
Single-turn
Multi-turn
Fisher
CALLHOME
ASR
ST
ASR
ST
dev
dev2
test
AVG
devtest
evitest
AVG
46.7
47.3
46.5
46.8
18.7
18.9
18.8
34.1
34.5
34.3
34.3
11.4
11.0
11.2
50.2
51.5
50.0
50.5
21.2
21.2
21.2
41.1
41.6
41.7
41.4
14.8
14.9
14.8
47.5
48.1
47.1
47.5
18.5
19.2
18.8
47.2
47.7
46.6
47.2
19.4
18.6
19.0
49.6
50.4
49.1
49.7
20.5
20.1
20.3
Table 11: BLEU scores on each single-tur turn dataset for all the official Hisher-CALLHOMEE development and test subset. AVG lists the average between dev and sets.
Training Data
Word Error Rate (L)
Single-turn
Multi-turn
Fisher
CALLHOME
ASR
ASR
dev2
test
AVG
deviest
evltest
AVG
23.5
22.8
21.0
22.5
35.5
36.3
35,9
22.8
22.2
20.7
21.9
34,0
34.6
34.3
31.5
31.6
27.9
30.3
48.4
48.4
V
f
/
23.1
22.5
20.8
22.1
35.2
35.6
35.4
26.0
26.1
23.4
25.2
38.7
39.7
39.2
23.0
22.2
20.8
22.0
34.6
36.3
35.4
Table WERs on each single-ture turn dataset for all the official Fisher Fisher-CALL CALLHOME development and test subset. AVG lists the average between dev and test sets.

---

Fisher
CALLHOME
Special Tokens
dev
dev2
test
AVG
devtest
evliest
AVG
BLEU score (1)
N/A
43.4
44.2
45.0
44.2
17.0
16.6
[TURN]
44.2
44.7
45,2
44.7
17.6
17.6
17.6
[TURN] [XT]
45.1
46.1
46.8
46.0
18.4
17.9
18.1
Word Error Rate (L)
N/A
29.9
30.3
26.4
28.9
43.9
43.7
43.6
[TURN]
29.2
31.1
25.8
28.7
43.2
43.1
43.2
[TURN] [XT]
30.2
29.6
25.8
28.5
42.6
42.1
Table 13: Ablation of the impact of encoding speaker turn and cross-talk information with [TURN] and [XT]. BLEU
scores and WERs listed for multi-turn dataset for all the official Fisher-CALLLHOM. development and test sets. AVG lists the average between dev and test sets.
Fisher
25
31
30
27
20
1.0
21
20
18:10
Density
0.15
10
Single
Single
0.10
Segmentation Technique
MT-MS
WebRCT Mile SHAS
0.05
Figure 9: We compare different segmentation tech- niques with two training data configurations: only Sin- gle-turn data and Both single-turn and multi-turn data. The bars denote different segmentation techniques for long-form audio, including MT-MS segmentation (pro- posed in this work), VAD via WebRTC (Blum et al., 2021) or SHAS (Tsiamas et al., 2022).
0.00
[seconds]
Single-Tum
Multi-Tum & Multl-Speaker
WebRCT
Figure 8: Data distribution for Fisher test set with dif-
ferent segmentation approaches.
form additional pre-processing step minimize the domain mismatch between SHAS and Fisher- CALLHOME. (1) We extract the speech activity boundaries for each audio file from the original metadata. (2) We modify each audio file by mask- ing with 0 all the regions the signal where there is no speech activity, i.c., setting all the non-speech activity regions to silence. (3) We then use the masked long-form audio files with SHAS. This step decreases the false alarms rate that can be pro- duced by SHAS noisy segments or between contiguous utterances where there are close-talks. Close-talks are areas where two utterances are too close and the segmentation tools might not gener- alize well. In order to keep comparable the experi- mental and evaluation setup, we perform the same pre- pe-preeeesing step when using WebRTC.
E Complete Ablation Results for [TURN] & [XT] Task Tokens
We provide compete ablation results of adding [TURN] & [XT] task tokens on all the official de- velopment and test sets of Risher-ALLHHOME as listed in Table 13.
F
More Details of VAD-Based Segmentation
With WebRTC, audio is split when 90% of consecu- tive frames do not include speech. We set the frame length to 30 ms and the aggressiveness parameter in (Tsiamas et al., 2022). With SHAS, set 1-30 as the min-max sequence length.
SHAS was trained on monologue corpora with MuST-C (Di Gangi et al., 2019). Thus, we per-
Besides SHAS (Figure 3), we also plot the seg- mentation distribution of WebRTC on the Fisher

---

System
Translation
Reference
hello good evening who is this [TURN] [XT] how's going hey this is guillermo..
Baseline
hello good evening how are you i'm guillermo
STAC-ST
hello good evening who is it [TURN] [XT] how is it going eh i' i'm guillermo...
Table 14:
In this example, the second speaker jumps in while the first speaker is saying "who is this". The baseline model trained with only single-turn data fails handle the cross-talk and cuts off "who is this". Our STAC-ST model not only accurately identifies the speaker-turn change and cross-talk (by producing [TURN] [XT]), but also successfully serializes the cross-talk.
Fisher
CALLHOME
Model
Size (0)
dev
dev2
test
AVG
devtest
evitest
AVG
BLEU (t)
Whisper-tiny
39M
8.1
7.5
11.5
9.0
1.9
2.4
2.2
Whisper-base
74M
27.4
23.7
29.0
26.7
8.4
Whisper-small
244M
44.2
44.1
46.7
45.0
19.2
19.2
19.2
Whisper-medium
769M
48.6
47.7
49.2
48.5
22.5
23.1
22.8
STAC-ST (S)
21M
45.1
46,1
46.8
46.0
18.4
17.9
18.2
STAC-ST (M)
86M
48.1
48
49.4
48.5
20.2
20.4
20.3
STAC-ST (L)
298M
48.6
48.9
50.0
49.2
21.0
21.0
21.0
Word Error Rate (L)
Whisper-tiny
39M
51.5
50.1
45.0
48.9
60.3
59.8
60.1
Whisper-
74M
41.8
42.0
36.7
40.2
50.0
49.2
49.6
Whisper-small
244M
33,9
33.7
29.1
32.2
39.1
37.9
38.5
Whisper-meding
769M
31.3
30.9
28.7
30.3
33.9
32.3
33.1
STAC-ST (S)
21M
30.2
29.6
25.8
28.5
42.6
42.1
42.4
STAC-ST (M)
86M
27.0
28.1
23.8
26.3
40.1
38.3
39.2
STAC-ST (L)
298M
27.9
27.9
23.5
26.4
38.98
38.5
38.7
Table 15: Comparison between Whisper scaled STAC-ST using training data. WER and BLEU scores are reported on the multi-turn dataset for all the official Fisher-CALLHHOME development and test subsets. AVG lists the average between dev and test sets.
test set in Figure 8. WebRTC yields a more rea- sonable distribution than SHAS. Note that some samples longer than 30 seconds.
H Complete Results of Scaled STAC-ST vs. Whisper
We list complete evaluation results of scaled STAC-ST vs. Whisper for the MT-MS Fisher- CALLHOME development and test sets in Ta- ble 15.
We compare different segmentation techniques with two training data configurations in Figure 9: only Single-turn data, i.c., Row-2 in Table Both single-turn and multi-turn data, i.e., Row-4 in Ta- ble Using our proposed configuration, Both, helps all segmentation techniques we tested during inference.
Complete Results of STAC-ST for Single-Turn ST
We list complete evaluation results of STAC-ST vs. prior work for the single-turn HALLOOE development and test sets in Table 16. Note that in the main paper, i.e., Table 6, we only list (1) the work that released the Fisher-CALLLOME corpora
G Example Translations With and Without using STAC-ST
(i.e., Casc. ST) and (2) the top three models that report both WER and BLEU scores (i.c., Multi- task, E2E-ST, ESPnet example).
We provide example translations with and without using STAC-ST in Table 14.

---

Fisher
CALLHOME
Model
Size (0)
dev
dev2
test
AVG
devtest
evltest
AVG
BLEU score (1)
Cas. ASR-MT (Post et al., 2013)
Multi-task ASR/ST (Weiss 2017)
48.3
49.1
48.7
48.7
16.8
17.4
17.1
E2E-ST M2Mc (Inaguma al,, 2019)
44.1
45.4
45.2
44.9
16.4
16.2
16.3
EMc2+ASR-PT' (Inaguma al., 2019)
46.3
47.1
46.3
46.6
17.3
17.2
17.3
E2E-ST streaming (Deng et ul., 2022)
47.9
48.2
47.7
47.9
15.5
15.3
15.4
ESPnet example (2022)
51.8
52.3
50.5
51.5
22.3
21.7
22.0
Whisper-tiny
39M
7.4
5.6
9.0
7.3
2.0
2.2
2.1
Whisper-buse
74M
19.1
20.4
25.4
21.6
6.0
6.5
6.2
Whisper-small
244M
45.4
40.7
45.3
43.8
17.5
16.8
17.1
Whisper-medium
769M
51.7
49.2
48.8
49.9
23.5
23.5
23.5
STAC-ST (S)
21M
49.6
50.4
49.1
49.7
20.5
20.1
20.3
STAC-ST (M)
86M
52.0
51.9
52.3
52.1
23.0
22.1
22.6
STAC-ST (L.
298M
52.4
52.8
52.6
52.6
22.7
22.4
22.5
Word Error Rate (L)
SAT-MMLL (Post al., 2013)
41.3
40.0
36.5
39.3
64.7
65.3
65.0
SAT-SGMM (Kumar et al., 2014b)
35.9
34.5
Multi-task ASR/ST (Weiss al., 2017)
25.7
25.1
23.2
24.7
44.5
45.3
44,9
E2E-ST M2Ma (Inaguma al., 2019)
25.6
25.0
22.9
24.5
44.5
44.0
Joint MSR+MT (Soky al., 2022)
22.8
22.3
20.5
21.9
39.5
39.4
39.5
ESPnet example (2022)
20.5
18.7
37.8
37.6
37.7
Whisper-tiny
39M
50.9
44.1
58.5
59.5
Whisper-buse
74M
41.4
39.5
34.8
38.6
49.0
48.7
48.8
Whisper-small
244M
32.2
28.1
30.2
36.5
36.7
Whisper-medium
769M
28.3
26.8
25.8
27.0
29.8
29.3
29.6
STAC-ST (S)
21M
23.0
22.2
20.9
22.0
34.6
36.3
35.4
STAC-ST (M)
86M
21.1
20.4
18.9
20.1
30.2
31.4
30.8
STAC-ST (L)
298M
21.0
20.6
18.8
20.1
30.4
31.0
30.7
Comparison between previous work vs. scaled STAC-ST. WER and BLEU scores are reported on single-turn segments of all the official Fisher-CALLHOME development and test subsets. AVG lists the average between dev and test sets. We list the best BLEU/WER scores for each model from previous work. In some cases, it includes ASR or MT pre-training. "Multilinuual model, name convention in (Inaguma al., 2019).
Table 16:
J Additional Results on CoVoST
tations as introduced in Section 4.2. The Fisher- CALLHOME training data was also used for the Spanish->Englin model. Whisper, XLS-R and CoVoST2 A2E-M are multilingual models. For fair comparison, trained multilingual STAC- ST L-size model by combining data of all related languages. Our languages tokens specify the trans- lation direction.
Traditional speech translation datasets com- posed of single-turn pre-segmented utterances. Fol- lowing Section 5.3.3, we also run experiments on the CoVoST test set." 13 In the following Table 17, we report BLEU scores on 3 translation directions haanaaa/FreenchSppass---ese
and compare with 3 recent papers that report BLEU scores on CoVoST 2: Whisper (Radford et al., 2023), XLS- R (Babu et al., 2022), and CoVoST2 (Wang et al., 2021).14 14 We list our STAC-ST models ranging from S-size L-size. They were trained on CoV- oST 2 ST and Common Voice ASR data with both single-turn and synthetic multi-turn segmen-
The results show that (1) our multilingual large model outperforms Whisper and XLS-R multilin- gual models with comparable sizes, even though Whisper and XLS-R where trained on data ders of magnitude larger: 680k hours for Whisper, 436k hours for XISSR, and 3k hours for STAC- ST L multilingual; (2) our models with smaller sizes sometimes outperform larger Whisper mod-
19 We Common Voice version 13.0 to create the data. 14 CoST2 reports casc-sensitive BLEU.

---

Multilingual?
Model Size
DE - EN
FR - EN
ES - EN
Baselines
(DWhisper-buuee (Radford al., 2023)
Y
74M
11.7
15.4
21.3
(2) W)Whhsper-small (Radford al., 2023)
Y
244M
25.3
27.3
33.0
(3)XLS- (Babu et al., 2022)
Y
300M
32.9
34.1
(4)CoVoST2 Bi-ST (Wang al., 2021)
17.1
26.3
23.0
(5)Co A2E-M (Wang et al., 2021)
Y
18.9
27.0
28.0
Ours
(6) STAC-ST
21M
19.7
29.2
29.1
(7) STAC-ST M
86M
20.5
31.8
32.6
STAC-ST
298M
21.4
25.2
33.0
(9) STAC-ST ST L Multilingual
Y
298M
27.5
34.0
35.8
Table 17: BLEU scores three language directions of the CoVoST 2 corpus test (Wang al., 2021). The results show that (I) our multilingual large model outperforms Whisper and XLS-R multilingual models with comparable sizes, even though Whisper and XLSSR where trained on data two orders magnitude larger; (2) our models with smaller sizes sometimes outperform larger Whisper models, such STAC-ST 21M Whisper 244M on French-
els, such STAC-ST 21M Whisper 244M on French-tEnglish. These results along with our main paper demonstrate that our proposed ap- proach is well-suited for both the novel single- channel multi-speaker speech translation task and the conventional pre-segmented speech transla- tion.


Reviews:
[{"review_id": "8bnRvMhmfD", "reviewer": "Reviewer_BkF5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful reviews.\n\nReason To Reject 1 (I feel the relationship of the proposed method and the task is not tight.):\n- Our ablation study has already provided comprehensive **quantitive** analysis on how each component of our approach contributes to the improvement of multi-turn multi-speaker (MT-MS) ST. Section 5.1 shows that standard training with single-turn data only achieves 50-60% BLEU of our approach. Section 5.2 shows that text-labeling speaker-turn and cross-talk enables speaker change detection at acoustic level, which yields ~1.5 BLEU improvement. We will make these more evident in Section 5.1 and 5.2.\n- We are not sure the meaning of “If the task is not speech translation, it seems the serialized labeling based on task tokens may still work.” But we would like to comment that: (1) We use the generic idea of “serialized labeling based on task tokens” for our specific task of MT-MS ST by injecting speaker-turn and cross-talk information. (2) “Serialized labeling” is just one component of our approach and evaluation. We experimented and analyzed many other factors to the MT-MS problem.\n\nReason To Reject 2 (The findings seem not novel. Joint training of ... sounds intuitive):\n- Our key contribution is not simply using multi-task learning, but exploring an under-studied yet realistic problem: single-channel multi-turn & multi-speaker ST. When tackling the problem, we also explored speaker-turn and cross-talk detection without providing time-aligned signals.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation is clear, and the proposed approach sounds technical.\n- The experimental results show encouraging improvement.", "reasons_to_reject": "- The explanation about the challenge is not detailed. I'd like to see more illustration or even analysis, maybe quantative analysis, about the multi-turn and multi-speaker problem, and a comprehensive study about relationship of the proposed approach and the improvement. After reading the whole paper, I feel the relationship of the proposed method and the task is not tight. If the task is not speech translation, it seems the serialized labeling based on task tokens may still work.\n- The findings seem not novel. Joint training of single-turn and multi-turn, ST and ASR, and multi-turn ASR can improve the performance of multi-turn ST sound intuitive. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work is to address the multi-turn and multi-speaker challenges in conversational speech translation. The paper demonstrates experimental results appealing the proposed end-to-end speaker-turn aware conversational speech translation (STAC-ST), which involves serialized multi-task labeling framework based on special tokens. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IgURYZfevG", "reviewer": "Reviewer_RZ8H", "summary": "", "questions": "Questions:\n1. can you show some exmples of speech translation that are benefit from stac-st? Prefer to learn the cases of phonemes whose translation results are improved by the rich context-based stac-st architecture.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful reviews.\n\nReason To Reject 1 (results of traditional speech translation datasets):\n- Traditional speech translation datasets are composed of single-turn pre-segmented utterances. Following Section 5.3.3 “STAC-ST for Single-Turn ST”, we also run experiments on the CoVoST2 test set. In the following table, we report BLEU scores on 3 translation directions (German/French/Spanish→English) and compare with 3 recent papers that report BLEU scores on CoVoST2: Whisper, XLS-R, and CoVoST2. We list our STAC-ST models ranging from S-size to L-size. They were trained on CoVoST2 ST and Common Voice ASR data with both single-turn and synthetic multi-turn segmentations as introduced in Section 4.2. The Fisher-CALLHOME training data was also used for the Spanish→English model. Whisper, XLS-R and CoVoST2 A2E-M are multilingual models. For fair comparison, we trained a multilingual STAC-ST L-size model by combining data of all related languages. Our languages tokens specify the translation direction. \n- |Model                               |Multilingual?|Size|DE→EN|FR→EN|ES→EN|\n|------------------------------------|:-----------:|---:|----:|----:|----:|\n|Whisper-base (Radford et al., 2022) |      Y      | 74M| 11.7| 15.4| 21.3|\n|Whisper-small (Radford et al., 2022)|      Y      |244M| 25.3| 27.3| 33.0|\n|XLS-R 0.3B (Babu et al., 2021)      |      Y      |300M| 26.7| 32.9| 34.1|\n|CoVoST2 Bi-ST (Wang et al., 2021)   |      -      |  --| 17.1| 26.3| 23.0|\n|CoVoST2 A2E-M (Wang et al., 2021)   |      Y      |  --| 18.9| 27.0| 28.0|\n|(Ours) STAC-ST S                    |      -      | 21M| 19.7| 29.2| 29.1|\n|(Ours) STAC-ST M                    |      -      | 86M| 20.5| 31.8| 32.6|\n|(Ours) STAC-ST L                    |      -      |298M| 21.4| 25.2| 33.0|\n|(Ours) STAC-ST L - Multilingual     |      Y      |298M| **27.5**| **34.0**| **35.8**|\n- The results show that (1) our multilingual large model outperforms Whisper and XLS-R multilingual models with comparable sizes, even though Whisper and XLS-R where trained on data two orders of magnitude larger: 680k hours for Whisper, 436k hours for XLS-R, and ~3k hours for STAC-ST L multilingual; (2) our models with smaller sizes sometimes outperform larger Whisper models, such as STAC-ST 21M vs. Whisper 244M on French→English. These results along with our original paper demonstrate that our proposed approach is well-suited for both the novel single-channel multi-speaker speech translation task and the conventional pre-segmented speech translation.\n\nReason To Reject 2 (more recent st models after 2019):\n- Table 6 lists most recent papers that report scores on Fisher-CALLHOME, and E2E-ST (Inaguma et al., 2019) is the latest we could find. Whisper, which was trained on massive data and released in 2022, is generally considered a very strong baseline. We include more recent models on CoVoST2 above.\n\nQuestion (examples of speech translation that are benefit from stac-st):\n- We provide translations with and without using STAC-ST here.\n- |System   |Translation|\n|---------|-----------|\n|Reference|... hello good evening **who is this [turn] [xt] how's it going** hey this is guillermo ...|\n|Baseline |... hello good evening **how are you** i'm guillermo ...|\n|STAC-ST  |... hello good evening **who is it [turn] [xt] how is it going** eh i'm guillermo ...|\n- In this example, the second speaker jumps in while the first speaker is saying “who is this”. The baseline model trained with only single-turn data fails to handle the cross-talk and cuts off “who is this”. Our STAC-ST model not only accurately identifies the speaker-turn change and cross-talk (by producing [turn] [xt]), but also successfully serializes the cross-talk.\n- Note that we remove all special task tokens before quantitative evaluation as explained in Section 4.3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strong:\n1. multiple speaker conversational speech translation is an impressive direction and this paper is meaningful in practice.\n2. stac-st achieved sotat results with detailed ablation study.", "reasons_to_reject": "weak:\n1. besides fisher/callhome datasets, do we also have results in terms of traditional speech translation datasets?\n2. the paper will be evaluated higher if the baseline models can be enriched by including more recent st models after 2019, besides Whisper model.", "questions_to_authors": "Questions:\n1. can you show some exmples of speech translation that are benefit from stac-st? Prefer to learn the cases of phonemes whose translation results are improved by the rich context-based stac-st architecture.", "paper_topic_and_main_contributions": "This paper tackles single-channel multi-speaker conversational speech translation with an end-to-end and multi-task training model, named Speaker-Turn Aware Conversational Speech Translation (stac-st). It combines automatic speech recognition, speech translation and speaker turn detection using special tokens in a serialized labeling format.\nExperiments on Fisher-callhome corpus show that this paper's proposed method can achieve good results.\nAuthors said that they will release the data processing and model training code.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d2mIbM4BKE", "reviewer": "Reviewer_WTTK", "summary": "", "questions": "See previous section.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful reviews.\n\nReason To Reject 1 (the merit of the SL and TL tokens is not reported on):\n- As introduced in Section 3.2, [SL] and [TL] are language tokens used to “define the task for either ST (when [SL]!=[TL]) or ASR (when [SL]==[TL]).” For a joint ST+ASR model, we have to provide these two tokens to specify which task to perform. We do have a ST-only baseline in Row-1 of Table 2, which is equivalent to not using language tokens.\n\nReason To Reject 2 (comparing table 6 and 5, it seems that Whisper is completely robust):\n1. BLEU scores in Table 5 and Table 6 are not comparable, because Table 5 uses MT-MS segments (see Section 4.3) while Table 6 uses single-turn segments. Even with the same hypothesis, different segmentations lead to different BLEU scores.\n2. We can hardly draw any conclusion on the robustness to MT-MS vs. single-turn scenario given existing data points. Whisper has higher BLEU scores on MT-MS segments than on single-turn segments; STAC-ST as higher BLEU scores on single-turn segments than on MT-MS segments.\n\nReason To Reject 3 (The single speaker results from other papers are 4+ years old; the size of these models is not provided):\n- Table 6 lists most recent papers that report scores on Fisher-CALLHOME, and E2E-ST (Inaguma et al., 2019) is the latest we could find. Whisper, which was trained on massive data and released in 2022, is generally considered a very strong baseline. Only Multi-task (Weiss et al., 2017) reports the model size, which is 9.8M.\n- We compare with more recent models on one additional single-turn test set: CoVoST2. In the following table, we report BLEU scores on 3 translation directions (German/French/Spanish→English) and compare with 3 recent papers that report BLEU scores on CoVoST2: Whisper, XLS-R, and CoVoST2. We list our STAC-ST models ranging from S-size to L-size. They were trained on CoVoST2 ST and Common Voice ASR data with both single-turn and synthetic multi-turn segmentations as introduced in Section 4.2. The Fisher-CALLHOME training data was also used for the Spanish→English model. Whisper, XLS-R and CoVoST2 A2E-M are multilingual models. For fair comparison, we trained a multilingual STAC-ST L-size model by combining data of all related languages. Our languages tokens specify the translation direction. \n- |Model                               |Multilingual?|Size|DE→EN|FR→EN|ES→EN|\n|------------------------------------|:-----------:|---:|----:|----:|----:|\n|Whisper-base (Radford et al., 2022) |      Y      | 74M| 11.7| 15.4| 21.3|\n|Whisper-small (Radford et al., 2022)|      Y      |244M| 25.3| 27.3| 33.0|\n|XLS-R 0.3B (Babu et al., 2021)      |      Y      |300M| 26.7| 32.9| 34.1|\n|CoVoST2 Bi-ST (Wang et al., 2021)   |      -      |  --| 17.1| 26.3| 23.0|\n|CoVoST2 A2E-M (Wang et al., 2021)   |      Y      |  --| 18.9| 27.0| 28.0|\n|(Ours) STAC-ST S                    |      -      | 21M| 19.7| 29.2| 29.1|\n|(Ours) STAC-ST M                    |      -      | 86M| 20.5| 31.8| 32.6|\n|(Ours) STAC-ST L                    |      -      |298M| 21.4| 25.2| 33.0|\n|(Ours) STAC-ST L - Multilingual     |      Y      |298M| **27.5**| **34.0**| **35.8**|\n- The results show that (1) our multilingual large model outperforms Whisper and XLS-R multilingual models with comparable sizes, even though Whisper and XLS-R where trained on data two orders of magnitude larger: 680k hours for Whisper, 436k hours for XLS-R, and ~3k hours for STAC-ST L multilingual; (2) our models with smaller sizes sometimes outperform larger Whisper models, such as STAC-ST 21M vs. Whisper 244M on French→English. These results along with our original paper demonstrate that our proposed approach is well-suited for both the novel single-channel multi-speaker speech translation task and the conventional pre-segmented speech translation.\n\nReason To Reject 4 (Shouldn't finetuning Whisper models be the SOTA baseline?):\n- Fine-tuning Whisper models on Fisher-CALLHOME data is an interesting idea, but it raises many technical questions. For example, should we preserve their original multi-task objectives, including voice activity detection, language identification, and ASR? Should we fine-tune the model with our proposed objectives? The latter would be an interesting extension to leverage pre-trained models for improvements, which is considered as future work.\n\nReason To Reject 5 (Please confirm only using ground-truth for the \"chunking\" of the test data for processing):\n- You are right, we only use ground-truth segmentation information to chunk the test data for pre-processing, and we do not provide ground-truth turn annotations to the model during inference. We will make it more clear in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well written paper. Study seems quite thorough (with some caveats, see limitations section).\n- Experiments demonstrate the utility of their model, as described in the summary.", "reasons_to_reject": "- While the advantage of the TURN and XT tokens is shown in table 3, the merit of the SL and TL tokens is not reported on. If all of the new tokens are omitted, what is the performance of their system on the same training data? This seems like an important baseline.\n- Comparing the single speaker and conversational results of table 6 and 5, it seems that Whisper is completely robust to the conversational vs. single speaker scenario, whereas the presented system degrades signficantly. This warrants discussion.\n- The single speaker results from other papers are all 4 or more years old. Are these results truly representative of the SOTA? Also, the size of these models is not provided.\n- The single speaker results also suggest that the Whisper system may not be well matched to the Fisher and CALLHOME data. Shouldn't finetuning these Whisper models on the training segment of this data (single only and/or conversational) be the SOTA baseline?\n- In the paper, the statements around utilizing human annotations for segmenting the test set are a bit confusing. Please confirm that this refers only to using gt for the \"chunking\" of the test data for processing, as opposed to providing turn segmentations to the model, and please make this more clear in the paper.", "questions_to_authors": "See previous section.", "paper_topic_and_main_contributions": "The authors present a turn-aware transformer-based model for conversational speech translation and asr, using a single audio channel.\n\n\"TURN\" and \"XT\" tokens are introduced into the model (figure 2) to indicate speaker turns (which are provided during training, and I believe inferred at test time), while \"SL\" and \"TL\" tokens are introduced to indicate translation or asr transcription modes (which are fixed and known for each conversation segment that is decoded, during both train and test).\n\nExperimental results on Fisher and CALLHOME show that:\n1) joint training with both single and multi-turn data as well as training on both translation and asr data improves performance (table 2),\n2) Addition of the \"TURN\" and \"XT\" tokens slightly improves ST and WER performance (table 3),\n3) Speaker change detection results showing slightly better performance than PyAnnotate (table 4),\n4) Better ST performance using their conversation segmentation/chunking technique than WebRCT and SHAS (table 5),\n5) Better ASR and ST performance than Whisper models of similar sizes on Fisher and CALLHOME, despite using 3 orders of magnitude less data (table 5), and\n6) Strong single speaker performance relative to Whisper and existing techniques (table 6).\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "spikes of the liner layer -> spikes of the linear layer\nthe STAC-ST model also shows to learn the task of time-aligned speaker change detection.-> is shown?"}, "extra_text": {}}, {"review_id": "coQgCXnrUf", "reviewer": "Reviewer_3SN2", "summary": "", "questions": "Question A: Is there a particular reason why in Figure 4 [XT] does not spike around 0s?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful reviews.\n\nReasons To Reject (the authors only use a cross-talk tag, that learns the beginning of cross-talk, but not when cross talk ends):\n- By placing the cross-talk token **after** the speaker-turn token, we enforce the model to differentiate the **beginning** of cross-talk and non-cross-talk utterances, so the model learns to delay outputting the utterance causing cross-talk and serialize the translation. Our quantitative evaluation verified the effectiveness of this serialization strategy.\n- Using a token to specify the **end** of cross-talk is an interesting idea that probably works as well. In many cases, it equals to placing the cross-talk token **before** the speaker-turn token leading a cross-talk (e.g. the example bellow where word2 and WORD2 overlap in speech.) We leave empirical comparison of these two almost equivalent serialization strategies as future work.\n```\nCHANNEL 1: |word1 word2|\nCHANNEL 2:       |WORD2 WORD3 ...|\n\nSerialization 1: word1 word2 [TURN] [XT-START] WORD2 WORD3 ...\nSerialization 2: word1 word2 [XT-END]   [TURN] WORD2 WORD3 ...\n```\n- Both system output and reference text are serialized, so there are no overlaps and parallel references in text. Serialized text is evaluated in a conventional way.\n\nQuestion (Is there a particular reason why in Figure 4 [XT] does not spike around 0s?):\n- We use [TURN] for speaker-turn **changes**, and [XT] always follows [TURN], so [TURN] and [XT] spikes correspond to speaker activity **changes**. We do not prepend [XT] to a segment, so there is no [XT] spike around 0s.\n\nMissing References (You tend to cite arxiv versions of papers.):\n- We will refresh the bibliography to reflect peer-reviewed versions if available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting set-up for a relevant problem with comprehensive experiments supporting the claims of the paper.", "reasons_to_reject": "Some decisions could be better motivated. E.g., the authors only us a cross-talk tag, that learns the beginning of cross-talk, but not when cross talk ends. It is not clearly motivated how this benefits speech translation. In the case of longer portions of cross-talk it is not clear how WER and BLEU are evaluated during speaker overlap, as in principle one could have two parallel references during the time of the overlap.", "questions_to_authors": "Question A: Is there a particular reason why in Figure 4 [XT] does not spike around 0s?", "paper_topic_and_main_contributions": "The paper describes a multitask framework for speech translation that is specifically target towards speech with multiple speakers per channel. To address this condition it pays special attention to speaker turns and cross-talk detection, by using specific tokens in the training data. Training data is simulated by collapsing the separate channels of training data such as Switchboard, into one channel.\nA comprehensive set of experiments is conducted to measure the influence of single turn and multi turn data for the ASR and Speech Translation task, showing the benefits of multi-task training and training on single-turn and multi-turn data.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "You tend to cite arxiv versions of papers, for which peer-reviewed versions are available. E.g., for Chan et al. 2015 it would be better to cite the ICASSP 2016 version.", "Typos_Grammar_Style_and_Presentation_Improvements": "\"cross-talks\" -> cross-talk"}, "extra_text": {}}, {"review_id": "bFmgYbtEp5", "reviewer": "Reviewer_JKE7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful reviews.\n\nReasons To Reject (It is unclear if there is overlapped speech in the conversation, how to predict the tokens in a serialized format.):\n- As introduced in Section 3.2, “if utterances $u_{t}$ and $u_{t+1}$ overlap in time, we append the targets of utterance $u_{t+1}$ after utterance $u_{t}$. The order of utterances is determined by their start time.” So the model generates the serialized output in a \"First In First Out (FIFO)\" manner — it first predicts the tokens in the former utterance, then adds [TURN] [XT] and predicts the tokens in the latter utterance. We will add a figure to illustrate the procedure mentioned in Section 3.2 more clearly.\n- Serializing the overlapped transcriptions/translations is a word re-ordering problem, and the model uses the cross-attention mechanism to predict the latter utterance tokens after [TURN] [XT].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental results demonstrate the effectiveness of the proposed framework. The visualization of CTC spikes show that the system can model the speaker activities well. ", "reasons_to_reject": "It is unclear if there is overlapped speech in the conversation, how to predict the tokens in a serialized format.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates conversational speech translation using an end-to-end framework. The multiple tasks of automatic speech recognition, speech translation and speaker turn detection, are combined in a multi-task training method. The end-to-end model is trained to generate specific tokens in a serialized labeling format. Such a combination is novel, the experimental results also demonstrate the effectiveness of the proposed framework. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZTmccMFB40", "reviewer": "Reviewer_4M4i", "summary": "", "questions": "Below are some comments\n1.\tIn section 3.2, the description of the auxiliary tasks should be included. Currently, the paper only mentions that task tokens are inserted to specify the task. Still, there is no description of how the turn prediction and cross-talk detection tasks are implemented. I assume that the output simply provides a serialized output containing [Turn] and [cross-talk] output tokens, but this should be explicitly explained, and also how the labels are generated should be mentioned. Probably showing a figure would make this part clearer.\n2.\tIn Table 1, what is speech act. [%]. Is it the speech activity? If so, please explain it in the text. Besides, an essential statistic would be the speech overlap ratio.\n3.\tIn Section 4.1, the authors segment the data in chunks of up to 30 seconds using the true segmentation information. According to the limitation section, it seems that this procedure is performed not only during training but also for testing. This should be clearly mentioned in Section 4.1.\n4.\tFigure 4, I could not perceive from the figure that the turn and cross-talk tokens were correctly predicted. This is related to comment 1., but without a clear explanation of how the turn and cross-talk task is implemented and what should be the correct output. Maybe you could add the reference, including the [turn] and [cross talk] tokens above the figure.\n5.\tTable 4: could you estimate your system in terms of diarization performance and provide DER values?\n6.\tCould you evaluate the performance of different speech overlaps?\n7.\tIn Tables 5 and 6, bold fonts do not always seem to reflect the best performance. Could you double-check?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful reviews.\n\nPlease refer to our answers to your specific questions which are assumed to instantiate “the handling of multi-turn is explained very vaguely” (reason to reject).\n\nQuestion 1 (How the turn prediction and cross-talk detection tasks are implemented? How the labels are generated?):\n- Yes, the auxiliary tasks of detecting speaker-turn changes and cross-talks are implemented by **training** with serialized text containing [TURN] and [XT], and the model learns to predict these two tokens and serialize the output by re-ordering target words in overlaps during **inference**. We will clarify this in Section 3.2.\n- Section 3.2 describes the procedure of generating labels including task tokens. We serialize transcripts or translations and put [TURN] and [XT] between utterances when needed. We will add a figure to illustrate this procedure more clearly.\n\nQuestion 2 (What is speech act.? An essential statistic would be the speech overlap ratio.):\n- Yes, it is the speech activity. We will explain it in the caption.\n- We will add the following statistics of speech overlap ratio to Table 1.\n- |Fisher Statistics|train| dev|dev2|test|\n|-----------------|-----|----|----|----|\n|Overlap ratio (%)| 12.7|14.5|16.8|11.2|\n- |CALLHOME Statistics|train| dev|test|\n|-------------------|-----|----|----|\n|Overlap ratio (%)  | 11.7|14.6|11.8|\n\nQuestion 3 (segment the data using the true segmentation for testing should be clearly mentioned in Section 4.1):\n- Thank you for the suggestion. Besides in the limitation section, we mentioned in Section 5.3.1 that “we resort to using MT-MS segmentation based on human annotations for preparing the test data”. We would also clarify it in Section 4.1.\n\nQuestion 4 (Figure 4, I could not perceive from the figure that the turn and cross-talk tokens were correctly predicted.):\n- As discussed in comment 1, for the speech translation task, the model just needs to correctly serialize the text and put [TURN] and [XT] tokens in the right position.\n- [TURN] and [XT] tokens do not carry exact time information, and the ability of time-aligned speaker change detection shown in Figure 4 was learned implicitly by the model. In fact, we do not provide ground-truth time stamps for training this intermediate task. We’ll add the reference to Figure 4 for comparison.\n\nQuestion 5 (Could you estimate your system in terms of diarization performance and provide DER values?):\n- The system does not perform diarization, which involves clustering segments according to speaker identity. Our model learns speaker-turn and cross-talk information implicitly, which would not provide speaker-related information for evaluating the diarization performance.\n\nQuestion 6 (Could you evaluate the performance of different speech overlaps?):\n- This is a great suggestion! We group Fisher and CALLHOME test segments into 4 bins by their average ratio of speech overlaps and report BLEU for each bin.\n- Fisher:\n\n|Overlap ratio|BLEU|\n|------------:|---:|\n|        x<=6%|48.2|\n|    6%<x<=11%|47.4|\n|   11%<x<=17%|45.8|\n|        17%<x|44.8|\n|          all|46.8|\n\n- CALLHOME:\n\n|Overlap ratio|BLEU|\n|------------:|---:|\n|        x<=6%|21.3|\n|    6%<x<=11%|19.8|\n|   11%<x<=17%|16.3|\n|        17%<x|15.8|\n|          all|17.9|\n\n- These results correspond to Row-3 in Table 2. The BLEU score decreases with increasing speech overlaps.\n\nQuestion 7 (In Tables 5 and 6, bold fonts do not always seem to reflect the best performance.):\n- Thanks for the catch! 37.9 of WER on CALLHOME in Table 5 should be bold.\n- We highlight numbers that are better than the second place by at least 0.2, that is why we bold 3 numbers of WER on Fisher in Table 6.\n\nTypos Grammar Style\n- Thank you for the detailed check. We will check typos and grammar mistakes carefully.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work may be one of the first works tackling E2E multi-speaker translation. The techniques' novelty may be limited, but the problem, task definition, and experimental investigations are novel enough for publication. Besides, the authors have carried out extensive experiments showing the potential of their approach.\n", "reasons_to_reject": "The paper is relatively clearly written, except that the handling of multi-turn is explained very vaguely and could be refined.\n", "questions_to_authors": "Below are some comments\n1.\tIn section 3.2, the description of the auxiliary tasks should be included. Currently, the paper only mentions that task tokens are inserted to specify the task. Still, there is no description of how the turn prediction and cross-talk detection tasks are implemented. I assume that the output simply provides a serialized output containing [Turn] and [cross-talk] output tokens, but this should be explicitly explained, and also how the labels are generated should be mentioned. Probably showing a figure would make this part clearer.\n2.\tIn Table 1, what is speech act. [%]. Is it the speech activity? If so, please explain it in the text. Besides, an essential statistic would be the speech overlap ratio.\n3.\tIn Section 4.1, the authors segment the data in chunks of up to 30 seconds using the true segmentation information. According to the limitation section, it seems that this procedure is performed not only during training but also for testing. This should be clearly mentioned in Section 4.1.\n4.\tFigure 4, I could not perceive from the figure that the turn and cross-talk tokens were correctly predicted. This is related to comment 1., but without a clear explanation of how the turn and cross-talk task is implemented and what should be the correct output. Maybe you could add the reference, including the [turn] and [cross talk] tokens above the figure.\n5.\tTable 4: could you estimate your system in terms of diarization performance and provide DER values?\n6.\tCould you evaluate the performance of different speech overlaps?\n7.\tIn Tables 5 and 6, bold fonts do not always seem to reflect the best performance. Could you double-check?\n", "paper_topic_and_main_contributions": "The paper introduce an end-to-end speech translation system that can translate multi-talker conversations. The model is trained to perform ASR or speech translation of speech segments of up to 30 seconds containing speech from two speakers, with overlap. The authors have explored using task tokens such as source and target languages and turn and overlap prediction to perform multi-tasks with a single model.\n\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are a few typos, e.g., “is know.” Could you double-check the manuscript for typos and grammar mistakes?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I concur with the other reviewers and believe this paper's idea is valuable; I recommend acceptance.
Decision: Accept-Main