SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations
Authors: Ioannis Tsiamas
José A.R. Fonollosa
Marta R. Costa-jussà
Venue: EMNLP 2023 Findings (2023)
Keywords: Speech Translation
Data Augmentation
TL;DR: We improve Speech Translation models by utilizing synthetic data that we created by re-segmenting the training data of ST corpora.

Abstract: End-to-end Speech Translation is hindered by a lack of available data resources. While most of them are based on documents, a sentence-level version is available, which is however single and static, potentially impeding the usefulness of the data. We propose a new data augmentation strategy, SegAugment, to address this issue by generating multiple alternative sentence-level versions of a dataset. Our method utilizes an Audio Segmentation system, which re-segments the speech of each document with different length constraints, after which we obtain the target text via alignment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in mTEDx. Furthermore, when combined with a strong system, SegAugment obtains state-of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it enables Speech Translation models to close the gap between the manual and automatic segmentation at inference time.

SEGAUGMENT: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations
Ioannis Tsiamas and José A. R. Fonollosa
Marta R. Costa-jussà FAIR Meta, Paris
Universitat Politècnica de Catalunya, Barcelona
(ioannis. tsiamas, wos.........o....... wwwollosa yuppc. edu
ta jostaauuss@mett .a. com
Abstract
Original version
Alternative versions
24d
End-to-end Speech Translation is hindered by lack of available data resources. While most of them are based on documents, sentence- level version available, which however single and static, potentially impeding the use- fulness of the data. We propose a new data augmentation strategy, SEGAUGMENT, to ad- dress this issue by generating multiple alterna- tive sentence-level versions dataset. Our method utilizes Audio Segmentation sys- tem, which segmennntss the speech of each doc- ument with different length constraints, after which obtain the target text via aligmment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in m TEDx. Furthermore, when combined with a strong system, SEGAUGMENT obtains state- of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it en- ables Speech Translation models to close the gap between the manual and automatic segmen- tation at inference time.
Sentence-level examples
230
X 68
x160
X220
long
Manual Segmentation
SegAugment
Hil Willt (40) H 44)
Document-level example
Figure 1: Data Augmentation with SEGAUGMENT.
resourceful tasks of ASR and MT, which restricts them from consistently matching the performance of the cascaded models (Bentivogli et al., 2021; Anastasopoulos al., 2022, 2021).
The majority of ST corpora are based document-level speech data, such as MuST-C (Di Gangi al., 2019a) and mTEDx (Salesky et al., 2021), which are derived from TED talks, with du- ration times 10 to 20 minutes. These document- based data are processed into shorter, sentence- level examples through process called manual segmentation, which relies on grammatical features in the text. Still, this sentence- level version is sin- gle and static and is potentially limiting the utility of the already scarce ST datasets.
1 Introduction
The conventional approach for Speech Translation (ST) involves cascading separate systems: an Automatic Speech Recognition (ASR) model fol- lowed by a Machine Translation (MT) model. How- ever, recent advances in deep learning (Vaswani et al., 2017), coupled with an increased availabil- ity of ST corpora (Di Gangi et al., 2019a; Wang et al., 2020a) have enabled the use of end-to-end end models (Weiss et al., 2017). Although end-to-end models can address several shortcomings of cas- caded models, such slow inference times, error propagation, and information loss, they are limited by a data bottleneck (Sperber and Paulik, 2020). This bottleneck arises from the inability of end-to- end models to directly leverage data from the more
To address this limitation, we propose SEGAUG- MENT, a segmentation-ban based data augmentation method that generates multiple alternative sentence- level versions of document-level speech data (Fig 1). SEGAUGMENT employs SHAS (Tsiamas et al., 2022b), an Audio Segmentation method that we tune to yield different re-segmentations of speech document based on duration constraints. For each new segmentation of a document, the correspond- ing transcript is retrieved via CTC-based forced alignment (Kürzinger et al., 2020), and the target text obtained with an MT model.

---

Our contributions are as follows:
(Lam et al., 2022a). Our approach, however, seg- ments documents at arbitrary points, thus providing access to a greater number of synthetic examples. An alternative approach by Lam et al. (2022b) in- volves recombining training data in a linguistically- motivated way, by sampling pivot tokens, retriev- ing possible continuations from suffix memory. combining them to obtain new speech-tranee transcrittinnnn pairs, and finally using an MT model generate the translations. Our method is similar since it also leverages audio alignments and MT, but instead of mixing speech, segments at alternative points.
We present GEGAUGMENT, a novel data aug- mentation method for Speech Translation. We demonstrate its effectiveness across eight language pairs in MuST-C, with average gains of BLEU points and on three low-resource pairs in mTEDx, with gains up to BLEU. When utilized with a strong baseline that com- bines WAV2VEC 2.0 (Baevski et al., 2020) and MBART50 (Tang et al., 2020), it obtains state-of-the-arr results in MuST-C.
We also show its applicability to data not based on documents, providing an increase of 1.9 BLEU in CoVoST2 (Wang et al., 2021). AUGMENT also enables ST models to close the gap between the manual and auto- matic test set segmentations at inference time. Finally, along with our code, we open source all the synthetic data that were created with the proposed method'.
Context-aware ST models have been shown to be robust towards error-prone automatic segmen- tations of the test set at inference time (Zhang et al., 2021a). Our method bears similarities with Gaido et al. (2020b); Papi al. (2021) in that it re-segments the train set to create synthetic data. However, unlike their approach, where they split at random words in the transcript, we use a special- ized Audio Segmentation method (Tsiamas et al., 2022b) to directly split the audio into segments re- sembling proper sentences. Furthermore, instead of using word alignment algorithms to get the target text (Dyer al., 2013), we learn the alignment with an MT model. We thus create high-quality data that can be generally useful, and not only for error- prone test set segmentations. Finally, recent work has demonstrated that context-aware ST models evaluated on fixed-length automatic segmentations can be competitive compared to the manual seg- mentation (Amrhein and Haddow, 2022). Here, we find that utilizing data from SEGAUGMENT yields high translation quality for ST models evaluated on automatic segmentations, even surpassing the translation quality of the manual segmentation, and without explicitly making them context-aware.
2 Relevant Research
SpecAugment (Park et al., 2019), which directly modifies the speech features by wrapping or mask- ing them, is standard approach for data aug- mentation in speech tasks including ST (Bahar et al., 2019; Di Gangi al., 2019b). WavAugment (Kharitonov et al., 2021) is similar technique that modifies the speech wave by introducing effects such as pitch, tempo and echo (Gállego et al., 2021). Instead of altering the speech input, our proposed method generates more synthetic data by altering their points of segmentation, and thus is compli- mentary to techniques such as SpecAugment. An effective way to address data scarcity in ST is by generating synthetic data from external sources. This can be achieved by using an MT model to translate the transcript of an ASR dataset or Text- to- S--Specch (TTS) model to generate speech for the source text of an MT dataset (Jia et al., 2019; Pino et al., 2019; McCarthy et al., 2020). In contrast, SEGAUTMENT generates synthetic data internally, without relying on external datasets.
Background
3.1 ST Corpora and Manual Segmentation
A document-level speech translation corpus DD (Di Gangi et al., 2019a; Salesky et al., 2021) is comprised of triplets that represent the speech wave X, the transeription Z, and the translation Y of each document.
Previous research has established the benefits of generating synthetic examples by cropping or merging the original ones, with sub-sequence sam- pling for ASR (Nguyen et al., 2020), and concatena- tion for MT (Nguyen et al., ; Wu et al., 2021; Kondo et al., 2021), as well as for ASR and ST
= (
Zi Y
(1)
In order for the data to be useful for traditional sentence-level ST, the document-level corpus
ggithhh.com/mm-uup///ggggauu

---

Speech
transcription
translation
test set segmentation is not available, and an au- tomatic one has be inferred, simulated by recent IWSLT evaluations (Anastasopoulos et al., 2021, 2022). They usually rely on acoustic fea- tures (pause-based)) length criteria (length-based), or a combination of both (hybrid). One such hy- brid approach is SHAS (Tsiamas et al., 2022b.a), which uses supervised classification model C and hybrid segmentation algorithm A. The classi- fier C is a Transformer encoder (Vaswani et al., 2017) with frozen WAv2VEC 2.0 (Baevski et al., 2020; Babu et al., 2021) as a backbone. It is trained on the speech documents and segment boundaries of a manually-segmented speech cor- pus, S = {(X', b')] by predicting whether an audio frame belongs to any of the manually- segmented examples. inference time, sequence of binary probabilities p' is obtained by applying the classifier on X' (eq. 4). Following, parameter- ized with Uhr to control the classification threshold, and & = (min, mox) to control the length of the resulting segments, AA produces the automatic seg- mentation b' according to p' (eq. 5). There are two possible choices for A. The Divide-and Toode-e (PDAC) approach progressively splits the audio at the point of lowest probability 1%x thr, until all resulting segments are within é (Tsia- mas et al., 2022b; Potapezyk and Przybysz, 2020). Alternatively, the Streaming (FSTRM) approach takes streams of length max and splits them at the point x with p'c > thr between or uses the whole stream if no such point exists (Tsiamas et al., 2022b; Gaido et al., 2021).
Z
Y
D
(3) spalt on strong punctuation
x
(3) audio
(2) sentence aligrmmnt
h
eliggment
S
segment boondaries
Figure 2: Manual Segmentation of an example i from document-level corpus D into my examples of sentence-level corpus S.
is processed a sentence-level corpus S, with Σ. my examples.
S = {(x'x ('x,z',')) y')
(2)
(E0)) 2/, ) و
Where x² = (xt!1 ..... x ) are the sentence- level speech waves for the i-th document, z' (z61 ..... ,...,,.my. x))))) are the sentence-level transcrip- tions, and y = (y ..... ,...,. m)) are the sentence- level translations. Usually, S obtained by a process of manual segmentation (Fig. 2), where the document-level transcription and translation are split on strong punctuation, and then aligned with cross-lingual sentence alignment (Braune and Fraser, 2010). Finally, the corresponding sentence- level speech waves obtain by audio-text align- ment (McAuliffe et al., 2017). Since speech is continuous, instead of defining the sentence-level speech wave اn common define the start and end points كhي edd R that correspond to document speech X². Thus, S can be re-defined as:
p' = C(X')
(4)
b' = A(p'' min, max, thr)
(5)
S S- s-{(xi (1bid, 243, y (1) }
4 Proposed Methodology
(3)
The proposed data augmentation method SEGAUG- MENT (Fig. 3) aims increase the utility of the training data S, by generating synthetic sentence- level corpora Se, which are based on alternative segmentations of the speech documents D (eq. I). Whereas the manual segmentation (Fig. 2) relies on grammatical features in the text, here propose to split on acoustic features present in the audio, by utilizing SHAS ($3.2). For the i-th speech document X SEGAUGMENT creates alternative segmentation boundaries b' with SHAS ($4.1), obtains the corresponding transcriptions via CTC-based forced alignment ($4.2), and finally,
Where = (bil1 ...., m) ) the segmentation for the i-th speech wave, and nāj = (stJ, eld) is a tuple of the segment boundaries for the j-th seg- ment, for which ' X siss Note that often- times there is gap between consecutive segments (eh5, gi++1) due to silent periods.
3.2 Audio Segmentation and SHAS
In end-to-end ST, Audio Segmentation methods aim find a segmentation b' for speech docu- ment x', without making use its transcription. They are crucial in real world scenarios, when a

---

Glssierr
Model
b
2
y
Se
M
x/4
X
b
yy STRL
S
M
Segm Ale A(
Speech
segroer bounddnee
harscrplmm
Passiatton
wit l
CTC Encoder
ApUs
E
nage!
Classifer C
CTC Brooder
Figure 4: Optional Model Training
Alignmmnt
4.1 Segmentation
b
S
We follow the process described for SHAS ($3.2) and obtain the alternative segmentations b for each X' in the training corpus, by doing inference with C. and applying the segmentation algorithm .A. In contrast to its original use, we use arbitrary val- for min to have more control over the length ranges of the segments and prioritize the classifica- tion threshold requirements thr over the segment length requirements / in the constraint optimization procedure of A, to ensure good data quality.
speech
segisent boundaniss
Varnscipaiinss
transtations
Figure 3: The SEGAUGMENT methodology. Given the i- th beeumennt-eeeel example of D, with m, sentence- -evell examples (S). it creates k synthetic sentence- ex- amples by alternative segmentations with SHAS. Chang- ing the segmentation parameters results in several dif- ferent synthetic corpora Sy.
generates the translations ÿ' with an MT model ($4.3). By repeating this process with different pa- rameterizations é of the segmentation algorithm A, multiple synthetic sentence-leve speech corpora can be generated ($4.4). A synthetic speech corpus S, with
4.2 Audio Alignment
To create the transcriptions 2' the i-th document for the segments b' ($4.1), we are using CTC-based forced aligmment (Kürzinger et al., 2020). We first do inference on the sentence-level speech of the manual segmentation x'= (x*1 ..... x)>";) with CTC encoder E, thus obtaining character-level probabilities u² for the whole audio. We apply a text cleaning process on the transcriptions z which includes spelling-ou numbers, removing unvoiced text such as events and speaker names, removing all remaining characters that are not included in vocabulary, and finally upper casing. The forced alignment algorithm uses the probabilities u' and the cleaned text z' to find the character segmenta- tion along with the starting and ending timestamps of each entry. Following, we merge characters words using the special token for the word bound- aries, and reverse the cleaning step? to recover the original text that corresponds to each segment. For each example j. we obtain the source text 2'd by joining the corresponding words that are within the segment boundary 8',,, and apply post-editing step to fix the casing and punctuation (Alg. 1).
Σ k; examples can be defined as:
St 6', (x, (,)
(6)
x {(bt), 2n
Where X is the original speech for the i-th doc- ument (eq. 1), 6d - (6).1 (i are its alternative segmentations, ž = (2 gilk) are its sentence- level transcriptions, and 9 ( "..,, (. are its synthetic sentence-level translations.
In total, three different models are utilized for creating a synthetic corpus, a classifier C ($3.2) for segmentation, a CTC encoder (Graves et al., 2006) for forced alignment, and an MT model A for text alignment. We can use pre-trained models, optionally learn them from the manually seg- mented examples of S (Fig. 4). The classifier C can be learned from SSGM {(X b') the en- coder& from A (x Fz.zz') 1/ and the model M from MM - yz)
Next, we describe in detail the proposed method.
We re-introduce any potential noise, e.g. "(Laughter)".

---

4.3 Text Alignment
Dataset
Lang. Pair
# Does
Sents
Hours
Unlike the case of manual segmentation (Fig. 2). cross-lingual sentence alignment (Braune and Fraser, 2010) is not applicable, and additionally, word alignment tools (Dyer et al., 2013) yielded sub-optimal results. Thus, we learn the align- ment with an MT model M, which is trained on the manually segmented sentence-level data SMT - {(z', y') y²) The training data modi- fied by concatenating examples to reflect the length of the examples that will be translated, thus learn- ing model Me from Sp where l are the length parameters used in SHAS. To accurately learn the training alignment, we use very little regular- ization, practically overfitting the training data St" ($A.7). Since there are no sentence-leved refer- ences available for the synthetic data, we monitor the document-level BLEU (Papineni et al., 2002) in a small number of training documents, and only end the training when it stops increasing. Finally, we obtain the synthetic sentence-level translations y' with the trained Mg.
En-De
2,093
234K
408
En-Es
2,514
266K
En-Fr
2,460
275K
485
En-It
2.324
254K
457
1.0
MuST-C
En-NI
2,219
248K
434
En-Pt
2,001
206K
377
En-Ro
2,166
236K
424
En-Ru
2,448
265K
482
2.0
En-De
2,537
251K
450
Es-Es
988
102K
178
Pt-Pt
812
90K
153
mTEDx
Es-En
378
36K
64
Pt-En
279
31K
53
Es-Fr
43
4K
CoVoST2
En-De
231K
362
Table 1: Training Data Statisties
($6.8). From mTEEx we use the Es-En, Pt-En, and Es-Fr ST data, well as the Es-Es, and Pt-Pt for the ASR pre-training, and finally the En-De data from Co (Wang al., 2021).
For segmentation ($4.1) we are using the open- sourced pre-trained English and multilingual SHAS classifiers For the SHAS algorithm we set the classification threshold thr = 0.5 and for length constraints & = (min, mar), we use four differ- ent, overllaalg tuples of (0.4, 3), (3, 10), (10, 20) and (20, 30) seconds, resulting in short (s). medium (m), long (f), and extra-long (xl) segmenta- tions. We PDAC and only apply PSTRM for xl, since we observed that PDAC was not able satisfy the length conditions. For the CTC encoder used in audio alignment ($4.2), we use pre-trained WAV2VEC 2.0 (Baevski et al., 2020) models* avail- able on HuggingFace (Wolf et al., 2020). For the MT models used in text aligmment ($4.3) we trained medium-sized Transformers (Vaswani et al., 2017; Ott et al., 2019) with 6 layers (SA.1.3). When train- ing with SEGAUGMENT, we simply concatenate the four synthetic datasets S,, S,.- St, S.I the original one (S), and remove any duplicates.
4.4 Multiple Sentence-level Versions
The parameters é = (min, max) of the segmen- tation algorithm AA allow us to have fine-grained control over the length of the produced segments. Different, non-overlapping tuples of É result in dif- ferent segmentations, providing access multiple synthetic sentence-level versions of each document. Moreover, the additional cost of creating more than one synthetic corpus is relatively low, as the results of the classification with C and the forced align- ment can be cached and reused (Fig. 3).
5 Experimental Setup
For our experiments we use data from three ST datasets, MuST-C (Di Gangi et al., 2019a), mTEDx (Salesky al., 2021) and CoVoST2 (Wang al., 2021) (Table 1). MuST-C and mTEDx are based on TED talks, and have sentence-level examples, which are derived from document-level ones, via manual segmentation (Fig. 2). CoVoST2 is based on the Common Voice (Ardila et al., 2020) corpus and is inherently a sentence- level corpus. For our eight language pairs from MuST-C v1.0, which are English (En) German (De), Spanish (Es), French (FF), Italian (1t), Dutch (NI), Portuguese (Pt), Romanian (Ro), and Russian (Ru). We are also using En-De from v2.0 for cer- tain ablation studies ($A.5, A.6, A.7) and analysis
For Speech Translation we train Speech-to-Ted Transformer basclines (Wang et al., 2020b). Un- less stated otherwise, we use the small architecture (s2t tot_ttranformer___ with 12 encoder layers and 6 decoder layers, and dimensionality of 256, with ASR pre-training using only the original data. The full details of the models and the training proce-
main experiments
ggitub..oom/mttuuu/SSlAA
www-2eee-lange--60---ddd--- waw2vec2-lange-xler-55 spanish, ww-2we22large-xts------orgg

---

Model
En-De
En-Es
En-Fr
En-It
En-NI
En-Pt
En-Ro
En-Ru
Average
Fairseq ST
22.7/
27.2/
32.9/
22.7/
27.3/
28.1
21.9/
15.3/
24.8
Baseline
23,2/
25.0
58.1
22.9 50.4
53.3/
24,7
15.3/ 22.2/49.0
51.6
SEGGAUGMETT
52.6
/
35.7 60.7
25.6/52.9
31.8/ 50.3
57.8
25.0/
16.8
42.5
2 54.0
ASR SEGAUGMENT
52.
29.5/ 56.9
35.8/60/
2 52.9
30.0/ 56.8
57.6
/5.0
14.7
27.5/ 54.0
Table 2: BLE / chrF2(t) scores on MuST-C v1.0 tSt-COMMON. In bold is the best score. All results with SEGAUGMENT and ASR SEGAUGMENT are statistically different from the Baseline with p p= 0.001. All models use the same architecture and results of Fairseq ST are from Wang al. (2020b).
dures are available in $A.1.1, For inference, we average the 10 best checkpoints on the validation set and generate with beam of 5. We evaluate with BLEU (Papineni et al., 2002) and chrF2 (Popović, 2017) using SACREBLEU (Post, 2018), and per- form statistical significance testing using paired bootstrap resampling (Koehn, 2004) to ensure valid comparisons. To evaluate on an automatic seg- mentation ($6.5), the hypotheses are aligned to the references of the manual segmentation with MW- ERSEGMENTER (Matusov et al., 2005).
Model
En-Es
En-Fr
Chimera (Han et al., 2021)
30.6/
35.6/
STEMM (Fang al., 2022)
31.0/
37.4/
ConST (Ye et al., 2022)
32.0/
38.3/
STPT (Tang al., 2022)
33.1/
39.7
SpeechUT (Zhang et al., 20226)
/
41.4/
w2v-mBART
33.3 60.2
40.8 / 64.5
+ SEGAUGMENT
33.7 60.7
41.5 65.1
Table 3: BLEU(T) chrF2(t) scores on MuST-C v1.0 tst--COMON, bold is the best score. Results with SEGAUGMENT are statistically different from the w2v- mBART baseline model with p< 0.001, apart from the BLEU score for En-Es (33.7) which is with p<00.005
6 Results
6.1 Main Results in MuST-C
on ST using MuST-C v1.0, with and without the synthetic data of SEGAUGMENT, and also provide results from other state-of-the-art (SOTA) meth- ods, such as Chimera (Han al., 2021), STEMM (Fang et al., 2022), ConST (Ye et al., 2022), STPT (Tang et al., 2022), and SpeechUT (Zhang et al., 2022b). As shown in Table 3, despite already hav- ing competitive baseline (w2v-mBART), utiliz- ing SEGAUGMENT achieves further significant im- provements", reaching SOTA performance in En- Es and En-Fr.
We compare ST models trained with and with- out SEGAUGMENT on the eight language pairs of MuST-C v1.0, and include results from Wang et al. (2020b), which use the same model architec- ture. In Table 2, we observe that models leveraging SEGAUGMENT achieve significant and consistent improvements in all language pairs, thus confirm- better
ing that the proposed method allows utilize the available ST data. More specifically, the improvements range from 1.5 3.1 BLEU, with an average gain of 2.4 points. We also investigate the application of SEGAUGMENT during the ASR pre-training, which brings further gains in four lan- guage pairs, but the average improvement is only marginal over just using the original ASR data.
6.3 Low-Resource Scenarios
We explore the application of SEGAUGMENT in low-resource and non-Esglish speech settings of mTEDx. In Table 4, we present results of the base- line with and without SEGAUGMENT for Es- En, Pt- En and the extremely low-resource pair of Es-Irr (6 hours). We furthermore provide the BLEU scores from Salesky et al. (2021), which use the extra- small model configuration (10M parameters). We use the extra-small configuration for Es-Fr, while the others use the small one (31M parameters). AEGAUGMENT provides significant improvements in all pairs, with even larger ones when it is also
6.2 Results with SOTA methods
Here we study the impact of the proposed method, when combined with a strong ST model. We use a model with 24 encoder layers, 12 decoder lay- ers, and dimensionality of 1024, where its encoder is initialized from WAV2VEC 2.0 (Baevski al., 2020) and its decoder from MBART50 (Tang et al., 2020) 1S..1.2). We fine-tune this model end-to-end
'nrefs: bs:1000 seed:12344 12345 casermixed eff:no 1ok:13a smootheepp version:2.3.0
CComprred with $ smaller gains are expected since the effect of data augmentation diminishes with increased data availability facilitated here by the large pre-trained models.
"nref's: hs: I sced: 12345 case:mixed I effyyes I ne:6 mw:0 space:no| version:2.3.0

---

Model
Es-En
Pt-En
Es-Fr
6.5
Automatic Segmentations of the test set
Bilingual E2E-ST
7.0/
8.1/
1.7/
Unlike most research settings, in real-works sce- narios, the manual segmentation not typically available, and ST models must rely on automatic segmentation methods. However, evaluating on automatic segmentation is considered sub-optimal, decreasing BLEU scores by 5-10% (Tsiamas et al., 2022b; Gaido et al., 2021) compared to evaluat- ing the manual (gold) segmentation.
Baseline
15.6/40.5
2.7/ 38.3/
SEGAUGMENT
18.5/43.5
17.8/1..1
3.8/23.9
ASR SEGAUGMENT
19.1/43.1
20.3/ 43.7
5.3/
Table 4: BLEU(1) / chrF2(t) scores mTEDx test set. In bold is the best score. All results with SEGAUGMENT and ASR SEGAUGMENT are statistically different from the Baseline with p< 0.001, with the exception of chrF2 in Es-Fr (23.9) which with p < 0.005. Results of Bilingual E2E-ST from Salesky et al. (2021).
Since the synthetic data of SEGAUGMENT orig- inate from an automatic segmentation, we expect they would be useful in bridging the training- inference segmentation mismatch (Papi et al., 2021). We evaluate our baselines with and with- out SEGAUGMENT on MusT-Ctst-CCMMM both the manual segmentation provided with the dataset, and automatic one which is obtained by SHAS. In Table 6 we present results with SHAS- long, which we found be the best, Extended results can be found in $A.4. For the purpose of this experiment, we also train another ST model with SEGAUGMENT where we prepend a special token in each translation, indicating the dataset ori- gin of the example.. When generating with such model, we prompt it with the special token that corresponds to the segmentation of the test set. The results of Table 6 show that the baseline experi- ences a drop of 1.6 BLEU points (or 6%) on av- erage, when evaluated on the automatic segmenta- tion, confirming previous research (Tsiamas et al., 2022b). Applying SEGAUGMENT, validates our hypothesis, since the average increase of 3.5 BLEU (23.4 - 26.9) observed in the automatic segmen- tation is larger than the increase of 2.4 BLEU in the manual one (25.0 + 27.4). Finally, using AEGAUGMENT with special tokens, enables ST models to reach an average score of 27.3 BLEU points, closing the gap with the manual segmenta- tion (27.4), while being better in three language pairs. To the best of our knowledge, this is the first time that ST models can match (or surpass) the performance of the manual segmentation, demon- strating the usefulness of the proposed method in real-world scenarios. Our results also raise in- teresting question, on whether we should continue to consider the manual segmentation as an upper bound of performance for our ST models.
Model
En-De
Bi-AST (Wang et al., 2021)
16.3/
STR (Lam al., 2022b)
18.8/
on
Baseline
17.4 43.2
SEGAUGMENT
19.0 44.9
+ ASR SEGAUGMENT
19.3 / 45.4
Table 5: BLEU()) /chFF2() scores on VoS2 test set. In bold is the best score. All results with SEGAUGMENT and ASR AuGmenT are statistically different from the Baseline with p 0.001.
utilized during the ASR pre-training, improving the BLEU scores by 2.6-5. Our results here con- cerning ASR pre-training are more conclusive than in MuST-C ($6.1), possibly due the better ASR models obtained with SEGAUGMENT (SA.3).
6.4 Application Sentence-level Data
We consider the application of the method to CoV- OST, of which the data do not originate from docu- ments. We treat the sentence-level data as "docu- ments" and apply SEGAUGMENT as before. Due to the relatively short duration of the examples, only apply SEGAUGMENT with short and medium configurations. In Table we provide our results for En-De, with and without SEGAUGMENT, a bilingual baseline from Wang et al. (2020a), and the recently proposed Sample-Translate- Raraaannnnnt--eemmnnneee Recombine (STR) augmentation method (Lam et al., 2022b), which uses the same model architecture as our ex- periments. Although designed for document-level data, SEGAUGMENT brings significant improve- ments to the baseline even outperforming the STR augmentation method by 0.5 BLEU points.
"There is a potential denoising effect, since CoVaST clips include unvoiced audio either at the start or the end, which our method inherently excludes from the synthetic data.
'i. <original>, 
, <m>, <>>, <xl>
10 "Signiffiancc not possible due to different segmentation. Witthout context-aware ST (Amrhein and Haddow, 2022).

---

Lang. Pair
set Segmentation
Model
#p
ASR
En-Es
En-Fr
Model
Manual
SHAS-long
Baseline
31M
/
27.5 55.0
33.1 58.33
+ SEGAUGMENT
31M
29.3 56.8
35.7 60.7
Baseline
23.2
21.2
En-De
+ SEGAUGMENT
25.0
25.1
24.5
25.2
Revisit-ST
48M
28.1
3
special tok.
25.1
25.2
Baseline
31M
K
25.3 52.3
30.1 555.4
Baseline
27.5
26.1
+ SEGAUGMENT
31M
28.5 55.7
34.6 59.6
En-Es
SEGAUGMENT
29.3
29.3
29.0
29.4
special tok.
29.2
29.4
Table 7: BLEU(T)) / chrF2(t) scores on MuST-C sst COMMON. In bold is the best score among the models without ASR pre-training (X). Results of Revisit-ST are from Zhang al. (2022a). #p stands for number parameters.
Buseline
33.1
30.6
En-Fr
SEGAUGMENT
35.7
35.7
34.9
35.3
special tok.
35.6
35.3
Baseline
22.9
21.5
En-It
+ SEGAUGMENT
25.6
25.6
24.4
25.1
special tok.
25.0
25.1
26
Baseline
27.3
25.7
En-NI
SEGAUGMENT
30.3
30.3
29.8
24
special tok.
29.4
29.8
BLEU Score
22
Baseline
28.7
26.9
En-Pt
+ SEGAUGMENT
31.8
31.8
31.4
31.8
20-
special
31.5
31.8
Baseline
22.2
20.5
18
Baseline
SegAugment
En-Ro
SEGAUGMENT
25.0
25.0
24.1
24.6
SegAugment
special tok.
24.8
24.6
20000
40000
60000
80000
Buseline
15.3
14.6
Steps
En-Ru
SEGAUGMENT
16.8
16.8
16.8
17.0
special tok.
16.4
17.0
Figure 5: BLEU(()) scores on MuST-C En-De dev dur- ing training.
Buseline
25.0
23.4
Average
+ SEGAUGMENT
27.4
27.3
+ special tok.
27.1
27.3
6.7 Training Costs
Table 6: BLEU(1) scores MuST-C v1.0 tst COMMON with manual and SHAS-long segmentation. The second column Manual and SHAS-long the best score among the three models, and with bold is the best score overall for each language pair.
In this section we discuss the computational costs involved with SEGAUGMENT during ST model training. We analyze the performance of models with and without SEGAUGMENT from Table 2, at different training steps in MuST-C En-De dev. In Figure 5, observe that models with our proposed method, not only converge better BLEU score, but also consistently surpass the baseline during training. Thus, although utilizing the synthetic data from SEGAUGMENT would naturally result longer training times, it is still better even when we constraint the available resources.
6.6 ST without ASR pre-training
Following, we investigate the importance of the ASR pre-training phase, standard practice (Wang al., 2020b), which usually is also costly. In Table we present the results ST models on MuST-C En-Es and En-Fr trained with and with- out AUGMENT, when skipping the ASR pre- training. We also include the results of the Revisit- ST system proposed by Zhang et al. (2022a). We find that models with SEGAUGMENT itive even without ASR pre-training, surpassing both the baseline with pre-training and the Revisit- ST system. In general, ASR pre-training could be skipped in favor of using SEGAUGMENT, but including both is the best choice.
6.8 Analysis
Here we discuss four potential reasons behind the effectiveness of SEGAUGMENT.
Contextual diversity. The synthetic examples are based on alternative segmentations and are thus presented within different contextual windows, compared the original ones ($A.8). We spec- ulate that this aids the model generalize more, since phrases and sub-words are seen with less or more context, that might or not be essential for
compet-

---

their translation. Adding additional context that is irrelevant was previously found to be beneficial in low-resource MT. by providing negative context to the attention layers (Nguyen et al., 2021).
Model
En-De
Baseline
/4.3/
Baseline in-data-KD
25.2/52.7
Positional diversity. With SEG speech and text units are presented at many more differ- ent absolute positions in the speech target text sequences ($A.11). This is important due to the absolute positional embeddings in the Transformer, which are prone to overfitting (Sinha et al., 2022). We hypothesize that the synthetic data create a di- versification effect on the position of each unit,
Baseline SEGAUGMENT
26.2 53.7
Table 8: BLEU / chrF2(t) scores on MuST-C v2.0 En-De -.MM-CMM.N. bold is best score. Results of Baseline + SEGAUGMENT are statistically different from Baseline + in-data-KD with p 0.001.
based on alternative audio segmentations. Through extensive experimentation across multiple datasets, language pairs, and data conditions, demon- strated the effectiveness of method, in con- sistently improving translation quality by 1.5 BLEU points, and reaching state-of-the re- sults when utilized by a strong ST model. Our method was also able to completely close the gap between the automatic and manual segmentations of the test set. Finally, we analyzed the reasons that contribute to our method's improved perfor- mance. Future work will investigate the extension of our method to ST for spoken-only languages and Speech-to-Se to-Speec translation, by passing the transcription stage.
which can be seen as a form of regularization, es- pecially relevant for rare units. This is also sup- ported for the simpler case of example concatena- tion (Nguyen al., 2021), while in our case the diversificationn effect is richer due the arbitrary document segmentation.
Length Specialization. Synthetic datasets created by SEGAUGMENT supply abundance of exam- ples of extremely long and short lengths, which are relatively infrequent in the original data. This creates specialization effect enabling ST mod- els trained on the synthetic data better translate sequences of extreme lengths in the test set ($A.9). Knowledge Distillation. As translations of the synthetic data generated by MT models, there is effect similar that of Knowledge Distilla- tion (KD) (Liu al., 2019; Gaido al., 2020a). To quantify this effect, we re-translate the train set of MuST-C En-De four times, with the same MT models employed in SEGAUGMENT. Subse- quently, an ST model is trained with the original and re-translated data, referred to as in-data-KD, the MT models did not leverage any external knowledge. In Table 8 we compare in-data-KI KD with SEGAUGMENT, and find that although in- data-KD provides an increase over the baseline, it exhibits significant difference of BLEU point with SEGAUGMENT. Our findings confirm the ex- istence of the KD effect, but suggest that SEGAUG- MENT is more general as it not only formulates different targets, but also diverse inputs (through re-segmentation), thereby amplifying the positive effects of the source-side contextual and positional diversity, In contrast, KD only provides diversity on the tadeet-side.
Limitations
General Applicability. The proposed method requires three steps: Audio Segmentation, CTC- based forced-alignment, and Machine Translation. For Audio Segmentation we used SHAS (Tsiamas et al., 2022b), which requires a classifier that is trained on manually segmented data. Although we demonstrated the method's applicability in CoV- oST En-De, which does include manual segmen- tation, used English classifier that was trained MuST-C En-De. Therefore, cannot be cer- tain of the method's effectiveness without manu- ally segmented data for the source language. A possible alternative would be to use a classifier trained on a different source language, since Tsia- mas et al. (2022b) showed that SHAS has very high zero-shot capabilities, provided the zero-shot language was also included in the pre-training set of XLS-R (Babu et al., 2021). which serves as backbone the classifier. Additionally, tested our method on several languages pairs, and also on an extremely low-resource one, such as Spanish- French (Es-Fr) in mTEDx, with only 4,000 training examples. Although we showed improvement
7 Conclusions
We introduced SEGAUGMENT, a novel data aug- mentation method that generates synthetic data

---

50% in that particular language pair, the two lan- guages involved, Spanish and French, are not by any means considered low-resource. Thus, we can- not be sure about the applicability of the method in truly extremely low-resource languages, such as many African and Native American languages. Fur- thermore, the current version of the method would not support non- written languages, since the tar- get text is obtained by training a MT model which translated the transcription of each audio segment. Biases. The synthetic data is heavily based on the original data, which may result in inheriting any biases present in the original data. We did not observe any signs of this effect during our research, but we did not conduct a proper investigation to assess the degree at which the synthetic data are biased in any way.
Conference on Machine Translation, pages 203-219. Abu Dhabi. Association for Computational Linguis- tics. [Cited on pages 2 and 7.]
Antonios Anastasopoulos, Loic Barrault, Luisa Ben- tivogli, Marcely Zanon Boito, Ondiej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad. Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, Dávid Javorský, Vera Kloudová, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nädejde. Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, John Ortega, Juan Pino. Eliz- abeth Salesky, Jiatong Shi, Matthias Sperber, Se- bastian Stüker, Katsuhito Sudoh, Marco Turchi, Yo- gesh Virkar, Alexander Waibel, Changhan Wang. and Shinji Watanahe. 2022. Findings of the IWSLT 2022 Evaluation Campaign. In Proceedings of the 19th International Conference on Spoken Language Translation (/WSLT 2022), pages 98-157, Dublin, Ireland (in-person and online), Association for Com- putational Linguistics. [Cited on pages 1 and 3.]
Computational and Memory costs. The syn- thetic data have to be created offline, with a pipeline that involves three different models, resulting in increased computational costs. To reduce these costs, we used pre-trained models for the Audio Segmentation and the CTC encoders, and cached the inference results to be re-used. Thus, the com- putational cost of creating the synthetic datasets for given a language pair involves single infer- ence with the classifier and the CTC encoder, and multiple training/inference phases with MT models. This process can take around 24-36 hours to create four new synthetic datasets for pair in MuST-C, using a single GPU. We acknowledge the computa- tional costs but believe the results justify them. The process could be made much lighter by using non- parametric algorithms in the three steps instead of supervised models, which can be investigated in fu- ture work. Finally, despite the computational costs, there is a very small memory cost involved since each synthetic dataset basically a ixt file contain- ing the new target text and a yaml file containing the new segmentation, only requiring 100-200MB of storage.
Antonios Anastasopoulos, Ondiej Bojar, Jacob Bremer- man, Roldano Cattoni, Maha Elbayad. Marcello Fed- erico, Xutai Ma, Satoshi Nakamura, Matto Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Sebas- tian Stüker, Katsuhito Sudoh, Marco Turchi, Alexan- der Waibel, Changhan Wang, and Matthew Wiesner. 2021. FINDINGS OF THE IWSLT 2021 EVALUA- TION CAMPAIGN. In Proceedings of the 18th In- ternational Conference Spoken Language Transla- tion (IWSLT 2021), pages 1-29, Bangkok, Thailand (online), Association for Computational Linguistics. [Cited on pages and 3.]
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuhen Morais, Lindsay Saunders, Francis Tyers, and Gre- gor Weber. 2020. Common Voice: A Massively- Multilingual Speech Corpus. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 4218 22182222, Marseille, France. European Language Resources Association. [Cited on page 5.]
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, al. 2021. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. arXiv preprint arXiv:2111.092%6 [Cited on pages and 9.]
A Acknowledgments
Work UPC was supported by the Spanish State Research Agency (AEI) project PID2019- 107579RB-100/ / AET/ / 10. 10001 1033.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representa- tions. In Advances in Neural Information Processing Systems, volume 33, pages 12449-12460 Curran Associates, Inc. [Cited on pages 2, 3, 5, 6, and 15.]
References
Chantal Amrhein and Barry Haddow. 2022. Don't Dis- card WrnddwW Audio Segmentation in Speech- to-Text Translation. In Proceedings of the Seventh
Parnia Bahar, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. On Using SpecAugment for End-to-End

---

Speech Translation. In Proceedings of the 16th In- ternational Conference on Spoken Language Trans- lation, Hong Kong. Association for Computational Linguistics. [Cited on page 2.]
Marco Gaido, Mattia A. Di Gangi, Matteo Ne- gri, and Marco Turchi. 2020a. End-to-Ee Ban-too-End Sreech-Translation with Knowledge Distillation: FBK @ IWSLT2020. In Proceedings of the 17th In- ternational Conference on Spoken Language Trans- lation, pages 80-88, Online. Association for Compu- tational Linguisties. [Cited on page 9.]
Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alherto Martinelli, Matteo Negri, and Marco Turchi. 2021. Cascade versus Direct Speech Translation: Do the Differences Still Make a Dif- ference? In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Nat- ural Language Processing (Volume I: Long Papers), pages 2873-2887, Online. Association for Computa- tional Linguistics. [Cited on page 1.]
Marco Gaido, Mattia A. Di Gangi. Matteo Negri, Mauro Cettolo, and Mareo Turchi. 2020b. Contextualized Translation of Automatically Segmented Speech. In Proc. Interspeech 2020, pages 1471-1475. [Cited on page 2.]
Marco Gaido, Matteo Negri, Mauro Cettolo, and Marco Turchi. 2021. Beyond Voice Activity Detection: Hy- brid Audio Segmentation for Direct Speech Trans- lation. In Proceedings of the Fourth International Conference on Natural Language and Speech Pro- cessing (ICNLSP 202 pages 55-62, Trento, Italy, Association for Computational Linguistics. [Cited on pages and 7.]
Fabienne Braune and Alexander Fraser. 2010. Improved Unsupervised Sentence Aligmment for Symmetrical and Asymmetrical Parallel Corpora. In Coling 2010: Posters, pages 81 81-89, Beijing, China. Coling 2010 Organizing Committee. [Cited on pages 3 and 5.]
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language Modeling with Gated Convolutional Networks. In Proceedings of the 34th Interaational Conference on Machine Learning- - Vol- ume 70, ICML'17, 17, page 933-941. 1S33-41.JMMRR.0 JMLR. [Cited on page 14.]
Gerard I. Gállego, Toannis Tsiamas, Carlos Escolano,
José A. R. Fonollosa, and Marta R. Costa-jussä, 2021. End-to-End Speech Translation with Pre- trained Models and Adapters: UPC at IWSLT 2021. In Proceedings of the 18th International Confer- ence on Spoken Language Trunslation (IWSLT pages 110-119, Bangkok, Thailand (online). Asso- ciation for Computational Linguistics. [Cited on page 2.]
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019a. MuST-C: a Multilingual Speech Translation Corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012-2017, Min- neapolis, Minnesota. Association for Computational Linguistics. [Cited on pages 1, 2, and 5.]
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist Tempo- ral Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In Proceed- ings of the 23rd International Conference on Ma- chine Learning, ICML '06. page 369-376, New York, NY, USA. Association for Computing Machinery. [Cited on page 4.1
Mattia A. Di Gangi, Matteo Negri, Viet Nhat Nguyen, Amirhossein Tebbifakhr, and Marco Turchi. 2019b. Data Augmentation for End-to-Enn tndEnd Speech Transla- tion: FBK @1 IWSLT '19. In Proceedings of the 16th International Conference on Spoken Language Trans- lation, Hong Kong. Association for Computational Linguistics. [Cited on page 2.]
Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. Learning Shared Semantic Space for Speech-to-Tees Translation. In Findings of the Association for Com- putational Linguistics: ACL-IICNIPP 2021, pages 2214-2225, Online. Association for Computational Linguistics. [Cited on page 6.]
Chris Dyer, Victor Chahuncau, and Noah A. Smith. 2013. A Simple, Fast, and Effective Reparameteri- zation of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644 648, Atlanta, Georgia. Association for Computational Linguistics. [Cited on pages 2 and 5.]
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian Error Linear Units (GELUs). ArXiv, abs/1606.08415. [Cited on pages 14 and 15.]
Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J. Weiss, Yuan Cao, Chung-Chent Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu. 2019. Leverag- ing Weakly Supervised Data to Improve End-to-end Speech-to-text Translation. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7180-7184 [Cited on page 2.1
Qingkai Fang. Rong Ye, Lei Li. Yang Feng, and Mingxuan Wang 2022. STEMM: Self-learnng with Speech-text Manifold Mixup for Speech Translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 7050-7062, Dublin, Ireland. Association for Computational Linguistics. [Cited on page 6.]
J. Kahn, M. Rivière, W. Zheng, F. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Col- lobert, C. Fuegen. T. Likhomanenko, G. Syn- naeve, A. Joulin, A. Mohamed, and E. Dupoux.

---

2020. Light-Light: A Benchmark for ASR with Limited or No Supervision. In ICASSP 2020 2020 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669- 7673. https:/ / /gi thub. com/ Facebookresearch/ 1ibri-light. [Cited on page 15.]
End-to-End Speech Translation with Knowledge Dis- tillation. In Proc. Interspeech 2019, pages 1128- 1132. [Cited on page 9.]
Nyy Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Con- ference on Learning Representations. [Cited page 14.]
Eugene Kharitonov, Morgane Rivière, Gabriel Syn- naeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. 2021. Data Aug- menting Contrastive Learning of Speech Represen- tations in the Time Domain. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 215- 222. [Cited on page 2.]
Exgeny Matusov, Gregor Leusch, Oliver Bender, and Hermann Ney. 2005, Evaluating Machine Transla- tion Output with Automatic Sentence Segmentation. In Proceedings of the Second International Work- shop on Spoken Language Translation, Pittshurgh, Pennsylvania, USA. [Cited on page 6.]
Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natu-
Michael McAuliffe, Michaela Socolof. Sarah Mihuc. Michael Wagner, and Morgan Sonderegger. 2017. Montreal Forced Aligner: Trainable Text-Speech Aligament Using Kaldi. In Interspeech. [Cited page 3.1
ral Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
[Cited on page 6.]
Arya D. McCarthy, Liezl Puzon, and Juan Pino. 2020. SkinAugment: unto-EEcodiing Speaker Conversions for Automatic Speech Translation. In ICASSP 2020 2020 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7924- 7928. [Cited on page 2.]
Seiichiro Kondo, Kengo Hotate, Tosho Hirasawa, Masahiro Kaneko, and Mamoru Komachi. 2021. Sentence Concatenation Approach to Data Augmen- tation for Neural Machine Translation. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Student Research Workshop, pages 143 149, Online. Association for Computational Linguis- tics. [Cited on page 2.]
Thai-Son Nguyen, Sebastian Stüker, Jan Niehues, and Alex Waibel. 2020. Improving Sequence- To- Sequence Speech Recognition Training with On-The- Fly Data Augmentation. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7689-7693. [Cited on page 2.]
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguisties. [Cited on page 14.]
Toan Q. Nguyen, Kenton Murray, and David Chiang. 2021. Data Augmentation by Concatenation for Low- Resource Translation: A Mystery and a Solution. In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 287-293, Bangkok, Thailand (online). Association for Computational Linguistics. [Cited on pages 2 and 9.]
Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. 2020. CTC- Segmentation of Large Corpora for German Find-o- End Speech Recognition. In Speech and Computer, pages 267-278, Cham. Springer International Pub- lishing. [Cited pages and 4.]
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling.
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demon- strations). pages 48-53. Minneapolis. Minnesota. Association for Computational Linguistics. [Cited pages 5 and 14.]
Tsz Kin Lam, Shigehiko Schamoni, and Stefan Rie- zler. 2022a. Make More of Your Data: Min- imal Effort Data Augmentation for Automatic Speech Recognition and Translation. arXiv preprint arXiv:2210.15 15398. [Cited on page 2.]
Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2022b. Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to- end Speech Translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 245- 254, Dublin, Ireland. Association for Computational Linguistics. [Cited on pages and 7.]
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEÉE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206 5210. [Cited on page 15.]
Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2021. Dealing with training and test seg- mentation mismatch: FBK IIL In Pro- ceedings of the 18th International Conference on
Yuchen Liu, Hao Xiong, Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019.

---

Spoken Language Translation (IWSLT 2021), pages 84-91, Bangkok, Thailand (online). Association for Computational Linguistics. [Cited on pages 2 and 7.]
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang. Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, AbdeIrahman Mohamed, Michael Auli, and Juan Pino. 2022. Unified Speech-Tex: Pre-training for Speech Translation and Recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1488-1499, Dublin, Ireland. Association for Computational Linguistics. [Cited on page 6.]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU; A Method for Automatic Evaluation of Machine Translation. In Proceed- ings of the 40th Annual Meeting on Association for Computational Linguisties, ACL '02, page 311-318, USA. Association for Computational Linguistics. [Cited on pages 5 and 6.]
Yuqing Tang. Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual Translation with Exten- sible Multilingual Pretraining and Finetuning. arXiv preprint WrXiv:2008.00401 [Cited on pages 2. 6, and 15.]
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Lc. 2019. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Proc. Interspeech 2019, pages 2613-2617. [Cited on pages 2 and 14.]
Toannis Tsiamas, Gerard I. Gállego, Carlos Escolano, José Fonollosa, and Marta R. Costa-jussà. 2022a. Pretrained Speech Encoders and Efficient Fine- tuning Methods for Speech Translation: UPC at IWSLT 2022. In Proceedings of the 19th Interna- tional Conference on Spoken Language Translation (IWSLT 2022), pages 265-276, Dublin, Ireland (in- person and online). Association for Computational Linguistics. [Cited on pages 3 and 15.]
Juan Pino, Liezl Puzon, Jiatao Gu, Xutai Ma, Arya McCarthy, and Deepak Gopinath. 2019. Harness- ing Indirect Training Data for End Automatic Speech Translation: Tricks of the Trade. In Proceed- ings of the 16th International Conference on Spoken Language Translation, Hong Kong, Association for Computational Linguistics. [Cited on page 2.]
Maja Popovié. 2017. chrF++: Words Helping Character n-grams. In Proceedings of the Second Conference on Machine Trunslation, pages 612-618, Copen- hagen, Denmark. Association for Computational Lin- guistics. [Cited on page 6.]
Toannis Tsiamas, Gerard I. Gállego, José A. R. Fonol- losa, and Marta R. Costa-jussà. 2022b. SHAS: Ap- proaching Optimal Segmentation for End-to-End Speech Translation. In Proc. Interspeech 2022, pages 106-110. [Cited on pages 1, 3, 7, and 9.]
Matt Post. 2018. A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguisties. [Cited on page 6.]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Proceedings of the 31 International Conference on Neural Information Processing Sys- lems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc. [Cited on pages 1, 3, and 5.]
Tomasz Potapezyk and Pawel Przybysz. 2020. SR- POL's System for the IWSLT 2020 End-to-Ennd Speech Translation Task. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 89-94, Online. Association for Computational Linguistics. [Cited on page 3.]
Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018 Context-Aware Neural Machine Trans- lation Learns Anaphora Resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264-1274, Melbourne, Australia. Association for Computational Linguistics. [Cited on page 17.]
Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Dou- glas W. Oard, and Matt Post. 2021. The Multilingual TEDx Corpus for Speech Recognition and Transla- tion. In Proc. Interspeech 2021, pages 3655-3659. [Cited on pages 1, 2. 5, 6, and 7.]
Changhan Wang, Juan Pino, Anne Wu, and Jiatao Gu. 2020a. CoVoST: A Diverse Multilingual Speech- To-Text Translation Corpus. In Proceedings of the Twelfih Language Resources and Evaluation Confer- ence, pages 4197-4203, Marseille, France. European Language Resources Association. [Cited on pages and 7.]
Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, and Adina Williams. 2022. The Curious Case of Absolute Position Em- beddings. [Cited on page 9.]
Matthias Sperber and Matthias Paulik. 2020. Speech Translation and the End-to-End Promise: Taking Stock of Where We Are. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 440917421, Online. Association for Computational Linguistics. [Cited on page 1.]
Changhan Wang, Yun Tang, Xutai Ma. Anne Wu, Dmytro Okhonko, and Juan Pino. 2020b. Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq. In Proceedings of the 1st Conference of the Paci-fo Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference

---

on Natural Language Processing: System Demon- strations, pages 33-39, Suzhou, China. Association for Computational Linguisties. [Cited on pages 5, 6, 8, and 14.]
Biao Zhang, Ivan Titov, Barry Haddow, and Rico Sen- nrich. 2021a. Beyond Seatence-Levee End-to- Bnd-to-Enn Speech Translation: Context Helps. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Join Conference on Natural Language Processing (Volume I: Long Papers), pages 2566-2578, Online. Association for Computational Linguistics. [Cited on page 2.]
Changhan Wang. Anne Wu, Jiatao Gu, and Juan Pino. 2021. CoVoST 2 and Massively Multilingual Speech Translation. In Proc. Interspeech 2021, pages 2247- 2251. [Cited pages 2, 5, and 7.]
Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Loquence-to-Sqqeene Models Can Directly Translate Foreign Speech. In Proc. Interspeech 2017, pages 2625-2629. [Cited on page 1.]
Shucong Zhang, Erfan Loweimi, Peter Bell, and Steve Renals. 2021b. On The Usefulness of Self- Ceff-Attention for Automatic Speech Recognition with Transform- ers. In 2021 IEFE Spoken Language Technology Workshop (SLT), pages 89-96. [Cited on page 17.]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: Sta Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics. [Cited on page 5.]
Ziqiang Zhang, Long Zhou, Junyi Ao, Shujic Liu, Lirong Dai, Jinyu Li, and Furu Wei. 2022b. SpeechUT: Bridging Speech and Text with Hidden- Unit for Encoder- Decoder Based Speech-Tea Pre- training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1663-1676, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [Cited on page 6.]
A Appendix
Xueqing Wu, Yingce Xia, Jinhua Zhu, Lijun Wu, Sh- ufang Xie, Yang Fan, and Tao Qin. 2021. mixSeq: A Simple Data Augmentation Method for Neural Machine Translation. In Proceedings of the 18th In- ternational Conference on Spoken Language Transla- tion (IWSLT 2021), pages 192-197, Bangkok, Thai- land (online). Association for Computational Lin- guistics. [Cited on page 2.]
A.1 Hyperparameters and Training details
A.1.1 Speech-to-Text Transformer models
For all experiments, apart from $6.2 and from those in mTEDx Es-Fr, we use 22__ransformer__ _s models implemented in FAIRSEQ (Ott et al., 2019; Wang et al., 2020b). They have 12 encoder layers, 6 decoder layers, dimensionality of 256, feed- forward dimension of 2048, and 4 heads in the multi-head attention, with 31M parameters in total. We use GELU activations (Hendrycks and Gim- pel, 2016) and pre-layemorm (Xiong et al., 2020a). The input is 80-dimensional log-Mel spectrograms, which are processed by 2-layer convolutional net- work with 1024 inner channels, output dimension of 256, stride of 2, kernel size of 5, and GLU ac- tivations (Dauphin et al., 2017). We do not scale up the output of the convolutional network. The target vocabularies are learned with SentencePieve (Kudo and Richardson, 2018) and have a size of 8,000. We train with Adam W (Loshchilov and Hut- ter, 2019) with learning rate of 0.002, a warm-up of 5, 5,000 steps, and an inverse square root scheduler. We use a gradient accumulation of 6, making the effeetive batch size cqual to 320 thousand tokens, apply SpecAugment (Park al., 2019), and set the dropout to 0.1, applied to attention, activation. and output. The loss function is a standard cross entropy with label smoothing of O. We stop the training when the validation loss does not decrease
Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. 2020a. On Layer Normalization the Transformer Architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. [Cited on page 14.]
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie Liu. 2020b. On Layer Normalization the Transformer Architecture. In Proceedings of the 37th International Conference on Machine Learning, 1CML'20. JMLR.org. [Cited on page 15.]
Rong Ye, Mingxuan Wang, and Lei Li. 2022. Cross- modal Contrastive Learning for Speech Translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5099-5113, Seattle, United States. Association for Computational Linguistics. [Cited on page 6.]
Biao Zhang, Barry Haddow, and Rico Sennrich. 2022a. Revisiting End-to-End Speech-to- Text Translation From Scratch. In International Conference on Ma- chine Learning. [Cited on page 8.]

---

for 10 epochs, and average the 10 best checkpoints according to validation BLEU.
is fine-tuned En-Xx multilingual machine transla- tion. We fine-tune all the parameters of the model, apart from the feature extractor of the encoder and the embedding layer in the decoder. The inputs to the model are raw waveforms sampled at 16kHz, which are normalized to zero mean and unit vari- ance. We train with Adam using a base learning rate of 0.0005, with a warm-up for 2,000 steps and an inverse square root scheduler. In the encoder we use 0.1 activation dropout, time masking with prob- ability of 0.2 and channel masking with probability of 0.1 (Baevski et al., 2020). In the decoder we use a dropout of 0.3, and attention dropout of 0.1 (Tang et al., 2020). All other dropouts are not active. The loss function is a standard cross entropy with label smoothing of 0.2. We use gradient accumulation to have effective batch size of 32M tokens, eval- uate every 250 steps, and stop the training when the performance on the validation set does not im- prove for 20 evaluations. We average the 10 best checkpoints according to the validation BLEU, and generate with a beam search of 5.
The encoders of the ST models are pre-trained on the task of ASR using the same architecture. The only difference is the vocabulary size which is 5,000 and the learning rate which is 0.001. For the models trained without ASR pre-training ($7) we also used learning rate of 0.001. For MuST-C, we pre-train using the ASR data of En-De, for mTEDx we use the Es-Es and Pt-Pt accordingly, and finally for VoST the ASR data of En-De.
For mTEDx Es-Fr we use an extra-small ar- chitecture (s2r [22__transormee__xs It has 6 encoder layers and 3 decoder layers, dimensionality of 256, feed-forward dimension of 1024, 4 heads in the multi-head attention, and vocabulary size of 3,000, having total of 10M parameters. The learn- ing rate is set 0.001 (warm-up of 500), the batch size to 180 thousand tokens, and use a dropout of 0.2. We also share the weights of the embed- ding layer and output projection in the decoder. The same model is pre-trained on ASR, but with a dropout of 0.1. All other hyperparameters are the same as for the small models described before.
A.1.3 Machine Translation models
All our experiments were run on a cluster with 8 NVIDIA GeForce rtx 2080 ti. The running times of each experiment on single GPU ranged from 12 to 36 hours.
For the MT models used for text alignment in SEGAUGMENT ($4.3), we used mediun-sized Transformers, with 6 encoder and decoder layers, dimensionality of 512, feed-forward dimension of 2048, and 8 heads in the multi-head attention. We train with Adam W using learning rate of 0.002, with warm-up for the first 2,500 updates, and the batch size is set to 14 thousand tokens. The vocab- ulary size 8,000, and for regularization. we use a small dropout of 0. only at the outputs of each layer and label smoothing of 0.1. We stop training when the document-level BLEU does not increase for 20 epochs, average the 10 best checkpoints ac- cording to the same metric, and then generate with beam of size 8.
A.1.2 w2y-mBART models
For the experiments of $6.2, we use a strong base- line utilizing pre-trained models and a length adap- tor (Tsiamas et al.. 2022a). The encoder is com- posed of a 7-layer convolutional feature extrac- tor and 24-layer Transformer encoder, while the decoder has 12 layers, and vocabulary of size 250k, with 770M parameters in total. All the layers have an embedding dimensionality of 1024, a feed- forward dimensionality of 4098, GELU activations (Hendrycks and Gimpel, 2016), 16 attention heads, and pre-layer normalization configuration (Xiong et al., 2020b). A strided Id convolutional layer sub-samples the output of the encoder by 2 times. The encoder is initialized from WAV2VEC 1. (Baevski al., 2020), which is pretrained with 60k hours of non-transcribed speech from Libri- Light (Kahn et al., 2020), and fine-tuned for ASR with 960 hours of labeled data from Librispeech (Panayotov et al., 2015). The decoder is initial- ized from MBART50 (Tang et al., 2020), which
For the MT models used in the experiments of $A.6, we use a small architecture with encoder and decoder layers, with dimensionality of 256, feed-forward dimension of 1024, and 4 heads. We share the weights of the embedding and output pro- jection in the decoder and use dropout of 0.1 (ap- plied attention, activation, and output). We stop training when the validation loss does not decrease for 10 epochs, average the 10 best checkpoints ac- cording to validation BLEU, and generate with a beam of 5.
12 "Ffarrse//waw2vee//arr vox 960h new.pt 13 "raaisso//mmdel/mmmaarttt bmbrrt/ In.arr.z

---

A.2 CTC-based Forced Alignment Algorithm
test set Segmentation
Long. Pair
Model
Manual
SHAS
Here we provide the algorithm used for audio-text alignment with SEGAUGMENT ($4.2). The com- plete process for the i-th document is summarized in Algorithm 1, where we omit the index for ease of representation.
Baseline
23.2
20.6
21.7
21.2
20.2
SEGAUGMENT
23.1
24.7
.) special tok.
25.1
23.1
24.5
25.2
25.1
Buseline
27.5
25.0
26.4
26.1
21.8
29.3
20.3
27.3
28.7
29.0
78.9
special tol.
29.2
26,6
25.6
29.4
29.1
Biseline
30,0
31.0
29.0
En-Fr
SEGAUGMENT
35.7
35.7
32.3
34.1
34.9
34.8
35.3
Algorithm 1: CaC- Forced Aligament
sperial
input ix % List[Id Tensor]
Buseline
22.9
20.0
21.6
21.5
20.3
input :z % List(Stringl
En-It
SEGAUGMENT
25.6
25.6
22.7
24.2
24.4
24.8
25.1
special Aol.
25.0
22.3
24.3
25.1
24.5
input :b % LisITu Float]]
Biseline
27.3
24.9
26.5
25.7
24.3
o % List(Strin]
27.5
29.7
u + E(x)
special tol:
29.4
26,9
26.9
29.8
29.3
z CZEAN(Z)
Baseline
25.8
25.44
chars FORCED, ALIGNMENT(Z.U u)
En-Pt
SEGAUGMENT
31.8
31.8
29.2
30.7
31.4
31.1
31.8
words <- JOIN CHARS(Chars)
special tol.
31,5
28.8
30.8
31,8
31.3
words + REVERSE_ CENVERSE_CLEAN(OONN (LEAN(words)
Braeline
19.9
20.8
201.5
18.7
for j - 1 to len(b) do
Ro
SEGAAGMENT
25.0
25.0
22.5
23.4
24.1
24.3
24.6
special tol:
24.0
;² WORDS_W words)
Riseline
15.3
13.4
14.7
14.6
13.8
2/ POST )
En-Ru
SEGAUGMENT
16.8
16.8
16.4
16.7
17.0
special tol.
16.4
14.9
16.4
17.0
16.8
return 2
Biseline
25.0
22,4
23.7
23.4
22.1
Avg
SEGAA GMMNT
27.4
24.9
26.4
26.9
26.9
special tol.
27.1
24.6
26.2
27.3
26.9
A.3 Results of ASR pre-training
Table 10: BLEU(1) scores on manual- and SHAS- segmented MuST-C v1.0 tst- tstMCOMMNN
In Table 9 we present the results of the ASR pre- training, where we observe that by including data from SEGAUGMENT, all ASR models perform bet- ter in terms of WER. This indicates that the pro- posed methods is also useful for ASR augmenta- tions, probably due to the source-side contextual and positional diversity ($6.8).
increases it in the SHAS-long segmentation by 0.4 BLEU (26.9 > 27.3), thus bridging the gap with the manual segmentation.
A.5 Results with different SEGAUGMENT combinations
MuST-C
mTEDx
CoVoST
We perform ablations by training ST models on different combinations of the original and the syn- thetic data of SEGAUGMENT, using MuST-C v2.0 En-De. In Table 11, row is essentially the base- line and row 17 is the baseline + SEGAUGMENT. Firstly, we notice that although training single syn- thetic datasets, but without the original one (rows 2-5) is inferior, the drop is relatively small, con- sidering the absence of the original data. When training with all of them (row 6) actually surpasses the baseline by 1.1 BLEU, without even using the original data. This showcases that SEGAUGMENT is generating high quality data, and even by them- selves can produce very good ST models. Secondly, the results show that the more synthetic datasets use, the better the performance, and that no combination is better than simply using all of them (row 17). We also find that when using I or 2 syn- thetic datasets, the combinations involving longer segmentations (rows 8, 9, 10 and 13, 14) tend to perform better than those involving shorter segmen-
Model
En-De
Es-Es (xs)
Pt-Pt
En-De
Baseline
11.1
18 (19.7)
28.3
26.6
SEGAUGMENT
10.4
16.6 (18.1)
23.8
23.8
Table 9: WER(.) scores for ASR models. indicates the extra-small models pre-trained for mTEDx Es-Fr.
A.4 Extended Results on Automatic Segmentation of MuST-C
Table 10 we provide an extended version of Ta- ble 6, where we evaluate the models on all four SHAS segmentations of the test set. Regarding the choice of automatic segmentation, we find that the long segmentation is the best when using SEGAUG- MENT with special tokens (27.3 BLEU), long and extra-long are equally good without special tokens (26.9 BLEU) and the medium segmentation is best without SEGAUGMENT (23.7 BLEU). Using the special tokens reduces the performance on the man- ual segmentation by 0.3 BLEU (27.4 + 27.1) but

---

Original
SEGAUGMENT
Original
SEGAUGMENT
BLEU
WER()
BLEU(H)
Data
m
xl
Data
8
m
xI
24.3
(1)
11.1
31.0
10.2
31.3
22.6
(3)
10.3
31.6
23.4
(4)
/
11.0
31.7
22.4
10.9
31.7
(5)
22.7
25.4
(6)
/
/ /
10.4
32.1
(7)
24.8
Table 12: WER scores for ASR models and BLEU scores for MT models in MuST-C v2.0 En-De tst- COMMON.
25.1
25.1
(10)
25.1
(11)
25.6
A.7 Results of MT models used in SEGAUGMENT
(12)
25.6
(13)
25.8
(14)
25.8
In table 13 we present results for the MT models trained to generate the new translation for the al- ternative data of SEGAUGMENT for MuST-C v2.0 En-De. For each parameterization E = (min, mor) of the segmentation algorithm A, we train special- ized model MM, as described in $4.3. We evaluate on the original training and development sets from S, well which basically indicates the quality of the target text in the synthetic data. We present both sentence- and document-level BLEU scores for the original data, and only document-level scores for the syn- thetic ones (since no sentence-level references are available). We notice that MT models obtain very high scores in the original train set, as compared to the development set, indicating that the model has indeed overfitted. In any other case that would be very bad news, but here we willingly overfit the model as our goal is learn the training set text alignment, and not having a good and generaliz- able MT model. By looking at the document-level BLEU on the synthetic training set, we can con- firm that the MT models have indeed accurately learned the alignments and thus have generated high-quality translation, that can be utilized during ST training.
(15)
26.0
(16)
26.0
(17)
26.2
Table BLEU(T) scores on tst-COMMON MuST-C v2.0 En-De for models trained with different combina- tions of the original and the synthetic data.
the synthetic training set (S²),
tations (rows and 11, 12), while for combinations of synthetic (rows 15, 16), do not observe any differences.
A.6 Results on ASR and MT
We investigate the impact of the synthetic data when training ASR and MT models. For this ex- periments are using the ASR data and bitext from MuST-C v2.0 En-De. The results of Table 12 show that the synthetic data are in general useful, with 0.7 reduction in WER and 1.1 increase in BLEU (row 6). This indicates that the improve- ments ST are stemming from both the source side (speech, ASR) and the target side (text, MT). We furthermore notice that longer synthetic exam- ples are better for MT, similar to ST in Table 2, while shorter ones better for ASR. We hypothe- size that this is consequence of the significance of long context for each task; ASR models tend use local context (Zhang et al., 2021b), while MT models oftentimes have to use long context, as in pronoun resolution (Voita et al., 2018).
A.8 Contextual Windows of SEGAUGMENT Data
We can categorize the new context of the syn- thetic data of SEGAUGMENT in four types, de- pending the type of overlap between the seg- mentation boundaries b for each document: (1) isolated, when being subset of an original one, (2) expanded, when being a superset of an original one, (3): mixed, when overlapping with an original one, (4): equal, when being exactly the same

---

MT model
Original train
Original development
Synthetic train (St)
Mj
BLEU doc-BLEU
BLEU
doc-BLEU
BLEU
doc-BLEU
95.1
79.2
24.5
26.2
68.3
90.0
74.4
26.0
27.5
70.4
72.9
61.3
25.5
27.0
64.6
x/
59.6
51.8
25.9
27.4
56.8
Table 13: MT models trained for generating the target text of SEGAUGMENT data. BLEU scores for MuST-C v2.0 En-De training and development set, and synthetic training set. BLEU scores are calculated both sentence- and document-level.
context as an original one. In Table 14 we apply this categorization for MuST-C v2.0 En-De, and observe that indeed most of the alternative exam- ples form SEGANGMENT are presented in new context, with only small percentage being equal, which we discard when concatenating the datasets for training.
S.S,
S..
S,S
8,&,
26
BLEU
24
SEGAUGMENT
Expanded
Isolated
Mixed
Equal
12.3%
56.9%
22.3%
8.5%
m
31.1%
33.3%
24.3%
11.3%
20-30
63.2%
7.9%
24.6%
4.2%
Duration buckets (seconds)
55.7%
1.2%
43.0%
0.1%
Figure 6: BLEU in two duration buckets of MuST-C tst-COMMON of models trained with the four different synthetic datasets and with only original data. The legend indicates which data each models is trained The left bucket contains all the examples with duration less than second, and the right contains all those with duration longer than 20 seconds. BLEU is the average of all eight language pairs in MuST-C.
28.3%
38.7%
25.0%
8.0%
Table Percentage of examples (rows) for each type of contextual overlap (columns) in the alternative datasets S, compared to the manual one S for MuST-C v2.0 En-De.
A.10 SEGAUGMENT Data Statistics
A.9 Results on duration buckets
In Table 15 we present the number of examples created with SEGAUGMENT for all the language pairs used in this research. We only consider ex- amples longer than 0.4 seconds and shorter than 30 seconds. Differences across the MuST-C v1.0 datasets created with the short condition are due empty segments which were removed. Following, in Table 16 we are presenting the average duration of each example in each created dataset. The av- erage is around 3.8 seconds for short-segmented datasets, and around 24 seconds for the exta-long ones. In both tables, differences in CoVoST are due to the fact that we prioritize more the length conditions
In Figure 6 we present evidence for the length spe- cialization effect ($6.8). We construct two extreme- length buckets in MuST-C t.s-COMMO;; the first bucket contains all the examples with duration less than 1 second and the second one all those with duration larger than 20 seconds. Then we mea- sure the performance of different types of ST model in the two buckets, which are using the data configurations of rows 1-5 of Table 11. Perfor- mance is measured by average BLEU across all cight language pairs of MuST-C. We notice that indeed, models trained with shorter synthetic data (S,, Sm) are better translating the test bucket with the very short examples, while those trained with longer synthetic data (S,, S.n) are better at translating the bucket with the very long ones.
(min, max) in the segmentation algorithm A ($4.1), while the other datasets are pri- oritizing the classification condition thr. We did this change order to make sure that the CoVoST data are segmented different way, than the

---

Sentence-level version
Dataset
Lang. Pair
SEGAUGMENT
Original
x
Concat
En-De
225K
433K
253K
120K
76K
882K
En-Es
260K
430K
253K
120K
76K
En-Fr
269K
421K
253K
120K
76K
871K
En-It
248K
399K
253K
120K
76K
848K
v1.0
MuST-C
En-NI
244K
433K
253K
120K
76K
882K
En-Pt
201K
433K
253K
120K
76K
882K
En-Ro
231K
433K
253K
120K
76K
882K
En-Ru
260K
419K
253K
76K
868K
v2.0
En-De
249K
401K
231K
109K
70K
812K
Es-Es
101k
150K
88K
43K
27K
308K
Pt-Pt
90K
136K
78K
37K
24K
274K
mTEDx
Es-En
36K
53K
32K
15K
10K
110K
Pt-En
30K
47K
27K
13K
94K
Es-Fr
3.5K
5.5K
3K
1.5K
1K
11K
CoVoST2
En-De
231K
504K
255K
0
759K
Table 15: Number of examples in Sentence-level versions for the manual and SEGAUGMENT processes.
manual segmentation. Due to this they might not always resemble proper sentences, but at least they are diverse enough to be useful during training.
pothesize that this effect could be regularizing the model, aiding in each generalization of this sub- words different positions ($6.8).
Sentence-level el version
Lang
SEGAUGMENT
Original
En-De
6.27
3,83
6.87
15.21
23.96
En-Es
6.63
3.84
6.87
15.21
23.96
15.21
23.96
En-It
6.41
3,83
6.87
15.21
23.96
v1.0
MoST-C
En- NI
6.23
3,84
6,87
15.21
23.96
En-Pt
6.50
3,84
6,87
15.21
23.96
Ea-Ro
En- Ru
6.47
3.83
6.87
15.31
23.96
v2.0
En-De
6:27
3.75
6.83
15.16
23.94
Es-Es
5.90
3.79
6.91
15.24
23.86
iiTEDs
Es-En
6.06
3.87
6.91
15.27
23.78
Pt-En
5,86
3,68
6,83
15.16
24.02
CoVoST2
En-De
Table 16: Average duration (in seconds) of examples in Sentence-level versions for the manual and SEGAUG- MENT processes.
A.11 Sub-word Positional Frequency
In Figure 7 we present examples of semi-rare sub- words in the target languages in MuST-C v2.0 En- De training set. We are counting their frequency in terms of absolute position in the examples they ap- pear in. We can observe that when using AEGAUG- MENT, the positional frequency of this sub-words is much more diverse, covering more space in the possible positions the target sequence. We hy-

---

bäume
kopie
Baseline
Baseline
+SegAugment
+SegAugment
frequency
frequency
8-
8-
6
6
4
2
20
40
60
100
20
40
60
80
100
position
position
.aarkieren
Transistor
16.
Baseline +SegAugment
Baseline
+SegAugment
frequency
frequency
8
8
6
6-
4-
4-
2
2
40
100
100
position
position
Figure 7: Positional frequency of semi-rare sub-words in the original data with and without SEGAUGMENT in MuST-C v2.0 En-De training set.


Reviews:
[{"review_id": "B8TR9vBqHA", "reviewer": "Reviewer_uvqE", "summary": "", "questions": "- Have you tried replacing \"SHAS\" in algorithm 1 with other augmentation tools? If yes, what are the results? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comprehensive review. Your recognition of our method’s strong performance on both manually-segmented and automatically-segmented speech, its thorough experimental design, and the rigorous evaluations across diverse language pairs and datasets is particularly appreciated. Such insights are invaluable for both the paper's refinement and for the broader community's understanding.\n\n**Response to Reasons to Reject**\n\nWe understand the concerns regarding the resemblance of SegAugment’s procedures to Sample-Translate-Recombine (STR) [1]. However, some crucial differences and additional strengths are worth highlighting:\n\n1. **Segmentation Point Difference & Performance on Automatic Segmentations**: Our method focuses on versatile segmentation points as opposed to the linguistic properties centered approach of STR. This fundamental difference not only yields varied training data pairs but also culminates in distinct model behaviors, like contextual & positional diversification and length specialization (Section 6.7 “Analysis”). Most notably, as outlined in section 6.5 \"Automatic Segmentations of the test set\", models with SegAugment demonstrate high performance gains on automatically segmented test sets for the 8 language pairs of MuST-C. This is a direct consequence of our method’s ability to bridge the training-inference segmentation mismatch, a capability that the STR method does not possess.\n2. **Complementarity with STR**: While SegAugment and STR share some foundational procedures, their distinct methodologies, and the nature of the synthetic data they create make them complementary in the ST domain, each addressing different challenges. This inherent diversity in approach positions the two methods as complimentary, suggesting a potential combined usage, further improving performance.\n3. **Thoroughness in Experimental Design**: SegAugment was exhaustively tested across diverse datasets, language pairs, and architectures. This not only validates its versatility but also underscores its novel approach in handling a spectrum of complexities in the field.\n\n**Questions for the Authors & Further Clarifications**\n\n*\"Have you tried replacing \"SHAS\" in algorithm 1 with other augmentation tools? If yes, what are the results?\"*\n\nWe opted to use SHAS due to its documented efficacy as highlighted in the literature [2, 3]. Furthermore, SHAS provides us the flexibility to control the resulting segment lengths through the min-max parameterarization, which in turn allows us to create multiple synthetic datasets for each training corpus. The combination of these multiple synthetic corpora of different segment lengths is key to the strong performance SegAugment, as highlighted by the ablation study in Appendix A.5 “Results with different SegAugment combinations”.\n\n**Typos Grammar Style And Presentation Improvements**\n\nYour feedback regarding Figure 3 is noted. To ensure clarity, we will revise the figure with detailed annotations for each symbol.\n\n**References**\n\n[1] Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2022. [Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation](https://aclanthology.org/2022.acl-short.27). In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 245–254, Dublin, Ireland. Association for Computational Linguistics.\n\n[2] Tsiamas, Ioannis, Gerard I. Gállego, José A. R. Fonollosa and Marta Ruiz Costa-jussà. “SHAS: Approaching optimal Segmentation for End-to-End Speech Translation.” *Interspeech* (2022).\n\n[3] Marco Gaido, Sara Papi, Dennis Fucci, Giuseppe Fiameni, Matteo Negri, and Marco Turchi. 2022. [Efficient yet Competitive Speech Translation: FBK@IWSLT2022](https://aclanthology.org/2022.iwslt-1.13). In *Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)*, pages 177–189, Dublin, Ireland (in-person and online). Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Its evaluation and \"SEGAUGMENT\"'s strong performance on both manually-segmented and automatically-segmented speech, where automatically-segmented speech is a realistic and challenging problem.\n- Its thoroughness in the experimental design, see above. ", "reasons_to_reject": "- \"SEGAUGMENT\" has procedures that are very similar to \"Lam et. al 2022: Sample, Translate, [...]\". This may weaken its novelty as a paper about data augmentation. ", "questions_to_authors": "- Have you tried replacing \"SHAS\" in algorithm 1 with other augmentation tools? If yes, what are the results? \n", "paper_topic_and_main_contributions": "This paper presents \"SEGAUGMENT\", which is a data augmentation method, for speech-to-text translation (S2TT). \n\n\"SEGAUGMENT\" has 3 steps: 1) segmentation on the document-level speech via \"SHAS\" algorithm, 2) CTC-based force alignment between the segmented speech and the corresponding transcripts, and 3) translation of the aligned transcripts via a translation model. The output of step 1) and 3) create new speech-translation data pairs for training. This process is repeated by varying the segmentation threshold (e.g., input length) in step1), resulting in data pairs which cover the input-length distribution more thoroughly. \n\n\"SEGAUGMENT\" is evaluated on three data: 1) MuST-C v1 (a relatively-large corpus with document-level speech), 2) mTEDx (a low-resource corpus with document-level speech) and 3) CoVoST-2 (a relatively-large corpus with sentence-level speech only) in 8, 5, and 1 language direction(s) respectively. In addition to a transformer-based architecture taking log Mel-Fbank as input, \"SEGAUGMENT\" is also applied on a strong model that contains a \"wav2vec2\" acoustic encoder and a \"mBART50\" translation decoder. In all settings, \"SEGAUGMENT\" obtains strong and statistically significant improvements over competitive baselines. \n\n### Comment\nA major weakness of the work is that the procedures of \"SEGAUGMENT\" are very similar to \"Lam et. al 2022: Sample, Translate, [...]\" which is also a segment/alignment-based augmentation method for S2TT. A notable difference, algorithmically, is the choice of segmentation point(s) which Lam et. al 2022 focus on the linguistic property of the transcripts, whereas \"SEGAUGMENT\" is on relatively arbitrary points. \n\nIn spite of the similarity, one major contribution of the work is its evaluation and \"SEGAUGMENT\"'s strong performance on both manually-segmented and automatically-segmented speech, where automatically-segmented speech is a realistic and challenging problem. \n\nApart from solid improvement over baselines, another strength of the work is its thoroughness in the experimental design. In terms of data, the method is evaluated on diverse language pairs and data sets. In terms of architecture, the method is evaluated on both small architecture (with fbank as input) and large architecture (self-supervised pre-training). Its improvements are also supported by significant tests. There are also interesting side experiments which provide details on the segmentation choices. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It might be better to indicate the meaning behind each symbol in the flow-diagrams, e.g. Figure 3. "}, "extra_text": {}}, {"review_id": "VcP6R2FCCi", "reviewer": "Reviewer_Li15", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your review and valuable feedback on our work. We are delighted that you found our methodology clear, appreciated the robustness of our results on MuST-C and low-resource languages, and took note of our efforts to maintain transparency with open-source code and dataset availability. We also value the insights you provided regarding the scope for improvement.\n\nIn response to your feedback:\n\n1. **Relevant Research Section**: We acknowledge that the relevant research section is rather compressed right now, since we opted to assign more space to the methodology and experiments. In the camera-ready version, given the additional page availability, we intend to enhance this section with more relevant works. In particular, our extended relevant research section will include more approaches focused on generating synthetic speech [1, 2] and synthetic text [3]. We will also discuss techniques that aim to modify the fbank or waveform features like SpecAugment [4, 5] and WavAugment [6]. Finally, we will touch the usage of MT data with sequence-level knowledge distillation [7], a technique that can be applied in the standard offline way [8, 9] or even online [10].\n2. **Future Work**: We appreciate your suggestion regarding the elaboration of future directions. In line with this, we aim to expand the discussion on the future research pathways in our revised manuscript.\n    - **Extension to non-written Languages & Speech-to-Speech Translation**: Our ambition is to explore bypassing the transcription stage in the data creation process. Post the segmentation phase with SHAS, we aim to employ an ST model that is overfitted on the training data to generate synthetic translations. This can lift the reliance on the intermediate ASR data, potentially enabling the application of SegAugment on non-written languages.\n    - **Speech-to-Speech Translation**: Given the increased interest in this area [11, 12] and the stark data scarcity, we believe SegAugment's synthetic data can be an asset. We plan to explore its potential for pre-training and even study a potential extension to directly augment speech-to-speech corpora [13, 14]\n    - **Curriculum Learning**: Our preliminary investigations revealed that models trained on “short” synthetic data tend to learn faster. However, their performance tends to plateau or decrease over time compared to models trained on “long” synthetic data. Hence, we plan to explore a curriculum learning [15] strategy that introduces shorter examples initially and gradually transitions to longer examples.\n\n**References**\n\n[1] J. Zhao, G. Haffari, and E. Shareghi. 2023. [Generating Synthetic Speech from SpokenVocab for Speech Translation](https://aclanthology.org/2023.findings-eacl.147). *EACL 2023*, pages 1975–1981, Dubrovnik, Croatia. Association for Computational Linguistics.\n\n[2] A. D. McCarthy, L. Puzon and J. Pino, \"SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translation,\" *ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, Barcelona, Spain, 2020, pp. 7924-7928\n\n[3] C. Mi, L. Xie, and Y. Zhang, \"Improving data augmentation for low resource speech-to-text translation with diverse paraphrasing”. Neural Networks, Volume 148, 2022, Pages 194-205, ISSN 0893-6080, https://doi.org/10.1016/j.neunet.2022.01.016.\n\n[4] Park, D.S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E.D., Le, Q.V. (2019) SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. Proc. Interspeech 2019, 2613-2617, doi: 10.21437/Interspeech.2019-2680\n\n[5] P. Bahar, A. Zeyer, R. Schlüter, and H. Ney, 2019. [On Using SpecAugment for End-to-End Speech Translation](https://aclanthology.org/2019.iwslt-1.22). In *Proceedings of the 16th International Conference on Spoken Language Translation*, Hong Kong. Association for Computational Linguistics.\n\n[6] E. Kharitonov *et al*., \"Data Augmenting Contrastive Learning of Speech Representations in the Time Domain,\" *2021 IEEE Spoken Language Technology Workshop (SLT)*, Shenzhen, China, 2021, pp. 215-222, doi: 10.1109/SLT48900.2021.9383605.\n\n[7] Y. Kim and A. M. Rush. 2016. [Sequence-Level Knowledge Distillation](https://aclanthology.org/D16-1139). In *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.\n\n[8] Liu, Y., Xiong, H., Zhang, J., He, Z., Wu, H., Wang, H., Zong, C. (2019) End-to-End Speech Translation with Knowledge Distillation. Proc. Interspeech 2019, 1128-1132, doi: 10.21437/Interspeech.2019-2582\n\n[9] M. Gaido, M. A. Di Gangi, M. Negri, and M. Turchi. 2020. [End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020](https://aclanthology.org/2020.iwslt-1.8). In *Proceedings of the 17th International Conference on Spoken Language Translation*, pages 80–88, Online. Association for Computational Linguistics.\n\n[10] Y. Tang, J. Pino, X. Li, C. Wang, and D. Genzel. 2021. [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://aclanthology.org/2021.acl-long.328). In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 4252–4261, Online. Association for Computational Linguistics.\n\n[11] Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. 2022. [Direct Speech-to-Speech Translation With Discrete Units](https://aclanthology.org/2022.acl-long.235). In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 3327–3339, Dublin, Ireland. Association for Computational Linguistics.\n\n[12] Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. 2023. [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://aclanthology.org/2023.acl-long.872). In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 15655–15680, Toronto, Canada. Association for Computational Linguistics.\n\n[13] Duquenne, Paul-Ambroise, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswani, Changhan Wang, Juan Miguel Pino, Benoît Sagot and Holger Schwenk. “SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations.” *Annual Meeting of the Association for Computational Linguistics* (2022).\n\n[14] Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. 2022. [CVSS Corpus and Massively Multilingual Speech-to-Speech Translation](https://aclanthology.org/2022.lrec-1.720). In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 6691–6703, Marseille, France. European Language Resources Association.\n\n[15] Bengio Y, Louradour J, Collobert R, Weston J (2009) Curriculum learning. In: Proceedings of ICML, pp 41–48", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The Methodology is clear and detailed. \n2. Strong Results on MuST-C and also on low resource language. \n3. Clear Limitations of work.", "reasons_to_reject": "NA", "questions_to_authors": "", "paper_topic_and_main_contributions": "Proposing data augmentation technique to generate multiple sentence-level versions of a document-level dataset that vary in length. They show strong results in the MuST-C dataset and robust results in low-resource languages. Then they also talk about sentence-level datasets at the inference time. \n\nMain Contributions: \n1. They proposed a data augmentation technique to create varied-length sentence data from document data. \n2. Compared the performance in MuST-C and mTEDx datasets.  \n3. State-of-the-art results in MuST-C\n4. Authors also demonstrated effectiveness on sentence-level datasets. \n4. Reproducibility with open-source code and dataset. \n\nI will recommend authors to add more information on the relevant research sections. The introduction and Methodology are clear and easy to understand. The authors also clearly explained the limitations of the work but lack any mention of future work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Relevant Research Section can be better with more details."}, "extra_text": {}}, {"review_id": "vQsAYqgCeH", "reviewer": "Reviewer_bbzS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our paper's clarity, improvements over the baselines, and comprehensive analysis. We also understand your concerns regarding the extended training time and the associated costs. Let's address these concerns with specific data:\n\n### **Impact of SegAugment on Training Time in Speech Translation**\n\n**Note on the experimental setup**: We train all models until convergence, and thus stop the training when the loss on the development set does not improve for 10 consecutive evaluations, where we evaluate every 500 steps.\n\n- **Baseline Model**: Completes training in about 35,000 steps, with a BLEU score of 22.4.\n- **Model with SegAugment**: Completes training around 70,000 steps, with a BLEU score of 24.8.\n- **Model with also ASR-SegAugment**: Completes training at about 80,000 steps, with a BLEU score of 25.5.\n\nDespite almost doubling the resources for the SegAugment models, if we constrain all models to 35k steps, the models with SegAugment still outperform the baseline, achieving BLEU scores of 24.5/24.8. This implies that SegAugment is consistently advantageous, even with resource constraints. The prolonged training duration for models with SegAugment is probably not only due to the larger training set but also due to the challenges of modeling longer examples, with durations ranging from 20-30 seconds.\n\n**BLEU Scores on MuST-C v1.0 En-De dev set (No checkpoint averaging)**:\n\n| model/steps | 10k | 20k | 35k | Final |\n| --- | --- | --- | --- | --- |\n| Baseline | 20.6 | 22.0 | 22.4 | 22.4 |\n| (+) SegAugment | 21.3 | 23.5 | 24.5 | 24.8 |\n| (+) ASR SegAugment | 21.9 | 23.9 | 24.8 | 25.5 |\n\n### **Impact of SegAugment on Training Time during the ASR Pre-training**\n\nIn the ASR pre-training scenario:\n\n- **Baseline**: Takes 45k steps to train.\n- **SegAugment**: Extends to 70k steps.\n\nAlthough the baseline starts on par with the model that uses SegAugment, by the convergence point (45k steps), the latter one overtakes it. This suggests that while extra resource allocation to train with SegAugment is optional, it ensures superior performance.\n\n**WER Scores on MuST-C v1.0 En-En dev set (No checkpoint averaging)**:\n\n| model/steps | 15k | 30k | 45k | Final |\n| --- | --- | --- | --- | --- |\n| Baseline | 10.6 | 9.0 | 8.6 | 8.6 |\n| (+) SegAugment | 10.6 | 8.7 | 8.1 | 7.6 |\n\nWe hope this detailed explanation alleviates your concerns about the impact of SegAugment on the training costs of ST and ASR models. We acknowledge that we did not touch upon this issue in our initial draft and plan to add a section discussing this matter in the final version of the paper or its Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow\n- The paper shows strong improvements over baselines and contains much detailed analysis", "reasons_to_reject": "What is the training time for running the synthetic data with fine-tuning only and with pertaining?\nThe training time seems to be quite long and with additional pseudo-training data the cost of training seems to be quite high.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a new data augmentation technique for speech translation that generates multiple possible segmentation of the sentences with different length constraints. Methods that are trained on additional data show improvement on two datasets, and show especially improvement on low-resource settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5250l0kbcN", "reviewer": "Reviewer_n8SA", "summary": "", "questions": "How about comparing or combining other data augmentation methods (e.g., SpecAugment, speed perturb)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work.\n\nWe are pleased that you found our paper well-organized and that the motivation behind our work is clear to you. The recognition that our proposed method can potentially benefit the speech translation task is also gratifying.\n\nWe genuinely appreciate your feedback and would like to address your concerns and make some clarifications regarding your summary of the paper.\n\n**Clarifications to Reviewer's Summary**\n\n1. **Segmentation and use of Voice Activity Detection (VAD)**:\n    \n    We wish to clarify a misconception regarding segmentation. While traditional Voice Activity Detection (VAD) tools have been used in similar tasks, our study opted for a more sophisticated method. We are using SHAS [1], which leverages self-supervised speech representations from wav2vec 2.0 to craft a specialized classifier combined with a tunable segmentation algorithm. This not only gives us finer control over segment lengths but also ensures accurate segmentations at semantically meaningful points, unlike traditional VADs which have been shown to be error-prone in recent studies [1, 2]\n    \n2. **Generation of Synthetic Translations and use of document-BLEU**:\nWe would like to clarify that we do not “get the synthetic translation by selecting the document-level BLEU best”. The best doc-BLEU it's not a tool for generating synthetic translations. Instead, it's employed to monitor the training of our MT alignment model. The absence of references for our new synthetic datapoints led us to rely on their document-level BLEU post-concatenation to determine the best in-training alignment model. More details can be found in section 4.3 “Text Alignment” and Appendix A.7 “Results of MT models used in SEGAUGMENT”.\n\n**Addressing the Reasons to Reject:**\n\n1. *“The method has some limitations under several scenarios.”*\n    \n    \n    Although we have discussed extensively the identified limitations of out method in the designated “Limitations” section, the listed limitations here, that are (a) the reliance on MT model for alignment, and (b) effectiveness on sentence-level data, are not valid reasons for concern. We have already provided evidence against them in the paper, but we will elaborate more here.\n    \n    1. *“this method relies on the MT model, and the translation quality of MT trained with limited data is not good. Thus, the alignment effect will not be ideal enough.”*\n        \n        \n        We acknowledge the inherent risks of depending on MT models, especially in low-resource settings. However, our findings from the mTEDx Es-Fr experiments of section 4.3, conducted with a mere 3.5k examples, argue otherwise. The significant increase in translation quality (as evident from Table 4) underscores the robustness of our approach in data-scarce scenarios and consequently indicates the high-quality of the synthetic data and the MT alignment model used to generate them. More concretely, the MT alignment models trained in mTEDx Es-Fr obtain high doc-BLEU scores in the synthetic training set, ranging from 58 to 66 points. Although not directly comparable due to differences in source language and domain, we provide for reference the doc-BLEU scores of the same models for MuST-C En-Fr, which used 270k examples. Despite an 80-fold difference in data size, the scores are not drastically different. The reason that our MT alignment models can still achieve remarkable performance even with limited data is that we deliberately train them past overfitting as we want good alignment within the train set, and not outside of it. Our detailed analysis in section 4.3 “Text Alignment” elaborates on this point. Please also refer to Appendix A.7 “Results of MT models used in SegAugment” for further discussion and analysis on MuST-C En-De.\n        \n        *document-BLEU scores on synthetic train set*\n        \n        | MT alignment model | mTEDx Es-Fr (3.5k) | MuST-C En-Fr (270k) |\n        | --- | --- | --- |\n        | short | 61.7 | 63.2 |\n        | medium | 66.2 | 71.1 |\n        | long | 58.5 | 68.5 |\n        | extra-long | 58.4 | 61.7 |\n    2. *“if the given speeches are not based on a document or have been shuffled, this method will not work well”*\n        \n        \n        Our findings in Section 6.4 “Application on Sentence-level Data” offer empirical evidence against this claim and showcase the method's applicability and effectiveness in these scenarios. By using SegAugment in CoVoST En-De, which is a purely sentence-level corpus, we can obtain a significant increase in translation quality (Table 5). Please refer to that section for more details.\n        \n2. *“The comparison is not sufficient and not fair compared to other methods”*\n    \n    We understand the importance of establishing the rigor and fairness of our comparisons. We have taken diligent efforts to ensure that our method's comparisons span both \"standard\" and \"state-of-the-art\" benchmarks. Our aim has been to transparently demonstrate the efficacy of our method across varying levels of complexities and benchmarks.\n    \n    1. *“The methods compared in Table 2 are weak, and your method is not based on a strong baseline.”*\n        - **Objective of Table 2**: The main purpose of showcasing results in Table 2 (section 6.1 “Main Results in MuST-C”) is to demonstrate the benefits of incorporating our proposed method alongside a widely-recognized baseline, namely the fairseq S2T-Transformer [3]. We believe that it's crucial to benchmark our method not just against the state-of-the-art (SOTA) but also against commonly-used, standard baselines. This ensures broader applicability and understanding.\n        - **Regarding the \"weak\" baseline**: We acknowledge your point on the perceived \"weakness\" of our baseline. However, our results, particularly with our \"weak\" baseline augmented with our method, surpass a much more sophisticated model like Chimera [4]. Notably, Chimera utilizes wav2vec 2.0, relies on external MT data, and has about 5 times the training parameters than our implemented baseline here.\n        - **Transitioning to Stronger Baselines**: We recognize the necessity of comparing against cutting-edge techniques. Hence, we present further experiments in section 6.2 “Results with SOTA methods”.\n    2. *“The model you compared in Table 3 contains about 150M parameters, while your model contains about 800M parameters. Additionally, MBART and w2v-large use much more unlabeled data.”*\n        - **Contextualizing Table 3**: The primary message from section 6.2 and the associated Table 3 is the increase in performance that our already strong baseline of w2v-mBART achieves when enhanced with SegAugment. By presenting other top-performing methods in the MuST-C benchmark within the same table, we provide a frame of reference. This allows readers to understand and appreciate the significance of the gains our method offers relative to existing SOTA models.\n        - **Clarifying our Position**: At no point did we imply a direct equivalence between our w2v-mBART model and other SOTA methods listed in Table 3 concerning training parameters or the use of unlabeled data. Our emphasis remains on the tangible improvements our method brings, regardless of the underlying model's complexities.\n\n**Questions for the authors:**\n\n1. *“How about comparing or combining other data augmentation methods (e.g., SpecAugment, speed perturb)?”*\n    \n    \n    While traditional data augmentation techniques like SpecAugment [5] focus on directly modifying the speech features of the input, our method focuses on modifying the context and length of an example, both in the input speech and target text. They're complementary, not competing. In fact, SpecAugment has been incorporated into all our models*, highlighting its compatibility.\n    \n    *with the exception of w2v-mBART that uses waveforms as input\n    \n\n**References**\n\n[1] Tsiamas, Ioannis, Gerard I. Gállego, José A. R. Fonollosa and Marta Ruiz Costa-jussà. “SHAS: Approaching optimal Segmentation for End-to-End Speech Translation.” *Interspeech* (2022).\n\n[2] Marco Gaido, Matteo Negri, Mauro Cettolo, and Marco Turchi. 2021. [Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation](https://aclanthology.org/2021.icnlsp-1.7). In *Proceedings of the 4th International Conference on Natural Language and Speech Processing (ICNLSP 2021)*, pages 55–62, Trento, Italy. Association for Computational Linguistics.\n\n[3] Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020. [Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq](https://aclanthology.org/2020.aacl-demo.6). In *Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations*, pages 33–39, Suzhou, China. Association for Computational Linguistics.\n\n[4] Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. [Learning Shared Semantic Space for Speech-to-Text Translation](https://aclanthology.org/2021.findings-acl.195). In *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, pages 2214–2225, Online. Association for Computational Linguistics.\n\n[5] Park, Daniel S., William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Dogus Cubuk and Quoc V. Le. “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition.” *Interspeech* (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is clear, and the paper is well-organized. \n\n2. The idea is interesting, and the proposed method can benefit the speech translation task.\n", "reasons_to_reject": "1. The method has some limitations under several scenarios. For example, this method relies on the MT model, and the translation quality of MT trained with limited data is not good. Thus, the alignment effect will not be ideal enough. Furthermore, if the given speeches are not based on a document or have been shuffled, this method will not work well.\n\n2. The comparison is not sufficient and not fair compared to other methods. The methods compared in Table 2 are weak, and your method is not based on a strong baseline. The model you compared in Table 3 contains about 150M parameters, while your model contains about 800M parameters. Additionally, MBART and w2v-large use much more unlabeled data.\n", "questions_to_authors": "How about comparing or combining other data augmentation methods (e.g., SpecAugment, speed perturb)?", "paper_topic_and_main_contributions": "This work proposes a data augmentation method to alleviate the problem of data scarcity in the speech translation task. When given an arbitrary document-level speech and its manual transcription and translation, the authors use a voice activity detection (VAD) tool to re-segment the whole speech and then obtain sentence-level speeches. For each speech segmentation, they use a speech recognition model to search for the corresponding transcription from the document-level transcription. Then, a machine translation (MT) is used to obtain alignment information between the searched transcription and the tagged document-level translation. They can get the synthetic translation by selecting the document-level BLEU best. Finally, they can change the setting of VAD and repeat the above steps to get multiple speech translation triple data. They carry out the experiments on multiple datasets and languages. The results show that their method can improve upon a strong baseline.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel method for data argmentation for speech translation.  By providing different segmentations of the document level triples of the training data, the data size are multiplied.  Intensive experiments are provided showing the effectiveness of the proposed method.  The paper is clearly written and the experiments are well organized and the results show the improvement is significant.   Three reviewers gave positive reviews, while the reviewer n8SA had some concerns on the comparison.  The authors gave detailed response which I think answered the concerns well.
Decision: Accept-Findings