1

Improving Speech Translation Accuracy and Time
Efficiency with Fine-tuned wav2vec 2.0-based
Speech Segmentation
Ryo Fukuda, Nonmember, Katsuhito Sudoh, Nonmember, Satoshi Nakamura, Fellow, IEEE

Abstract—Speech translation (ST) automatically converts utterances in a source language into text in another language. Splitting continuous speech into shorter segments, known as speechsegmentation, plays an important role in ST. Recent segmentation
methods trained to mimic the segmentation of ST corpora havesurpassed traditional approaches. Tsiamas et al. [1] proposeda segmentation frame classifier (SFC) based on a pre-trained
speech encoder called wav2vec 2.0. Their method, named SHAS,
retains 95-98% of the BLEU score for ST corpus segmentation.
However, the segments generated by SHAS are very different
from ST corpus segmentation and tend to be longer with multiple
combined utterances. This is due to SHAS’s reliance on length
heuristics, i.e., it splits speech into segments of easily translatable
length without fully considering the potential for ST improvementby splitting them into even shorter segments. Longer segments
often degrade translation quality and ST’s time efficiency. In thisstudy, we extended SHAS to improve ST translation accuracy
and efficiency by splitting speech into shorter segments that
correspond to sentences. We introduced a simple segmentation
avlgorithm using the moving average of SFC predictions without
relying on length heuristics and explored wav2vec 2.0 fine-tuning
for improved speech segmentation prediction. Our experimental
results reveal that our speech segmentation method significantly
improved the quality and the time efficiency of speech translationcompared to SHAS.
Index Terms—End-to-end speech-to-text translation, speechsegmentation, pretrained speech encoder.

I. IntroductionT

HE segmentation of continuous speech is a fundamental
process required for speech translation (ST) and otherspoken language applications. In text-to-text machine translation (MT), the input text is usually segmented into sentencesusing punctuation marks as boundaries. However, such explicit
boundaries are unavailable in ST. ST corpora usually containspeech segments that are aligned to sentences. For example,
the procedure for creating a multilingual ST corpus, MuSTC [2], first performs sentence alignment between English
transcriptions and its translations and aligns the English speech
and transcriptions with a forced aligner. Much ST research uses
such sentence-aligned speech segments to train and evaluate
Ryo Fukuda is with the Graduate School of Science and Technology,
Nara Institute of Science and Technology, Ikoma 630-0192, Japan (e-mail:fukuda.ryo.fo3@is.naist.jp).
Katsuhito Sudoh is with the Graduate School of Science and Technology,
Nara Institute of Science and Technology, Ikoma 630-0192, Japan (e-mail:sudoh@is.naist.jp).
Satoshi Nakamura is with the Data Science Center and Graduate Schoolof Science and Technology, Nara Institute of Science and Technology, Ikoma
630-0192, Japan (e-mail: s-nakamura@is.naist.jp).

systems, although they cannot be used in practical situations.
In addition, existing ST models cannot directly translate longcontinuous speech without segmentation. One reason is thatthe required computational resources increase with the lengthof the input speech. Even without any constraints on computational resources, an ST model trained on segmented shortspeech struggles to translate extremely long speech that is notincluded in its training data. For these reasons, several efforts
have focused on speech segmentation for ST.
Pause-based segmentation with voice activity detection(VAD) is commonly used as preprocessing for automatic
speech recognition (ASR) and ST. However, pauses in speech
do not necessarily coincide with boundaries of semantic units
such as sentences in text, e.g., there may be long pauses
in an utterance corresponding to a sentence or almost nopauses between utterances. Over-segmentation, in which asilence interval fragments a sentence, and under-segmentation,in which multiple sentences are included in one segment while
ignoring a short pause, reduce the ASR and ST performances[3]. Fixed-length segmentation is the simplest approach thatsegments audio at a predefined segment length [4]. There isalso a combination method that concatenates speech segments
generated by VAD up to a certain length. Such lengthbased segmentation methods are heuristic approaches thatcan split speech into segments of easily translatable length[5]. Punctuation-based segmentation methods are often usedin cascade ST, re-segmenting the ASR results of segments
produced by VAD with a punctuation restoration model ora language model [6], [7]. These methods can improve thetranslation accuracy of MT, but they cannot be used for endto-end ST, where the source language is translated directly
without ASR. In addition, these methods cannot prevent ASR
errors due to improper segmentation.
As mentioned above, ST corpora usually have speechsegments that correspond to sentences, which are suitable
for translation. Recent corpus-based segmentation methods
have been successful using a classification model trained to
predict segmentation of ST corpora. A corpus-based method,
SHAS [1], led to state-of-the-art results with a segmentation
frame classifier (SFC) based on a pre-trained speech encoder
called wav2vec 2.0 [8]. However, the segments generated by
SHAS tend to be significantly longer than segments of ST
corpus. Such long segments can decrease translation quality.
In addition, the longer the segment is, the more computation
time required for translation. These long segments are causedby using segmentation algorithms that place more importance2

on the lengths of segments than SFC prediction. This strategy
allows SHAS to split speech into segments whose lengths
are preferred by ST. However, the following potential remains
unconsidered: improving ST translation accuracy and time
efficiency by splitting these segments even shorter.
In this work, we extend SHAS to improve ST translation
accuracy and efficiency by splitting speech into shorter segments that correspond to sentences, such as those included
in ST corpus. We introduce a simple segmentation algorithm
using the moving average of SFC predictions without relying
on length heuristics to produce shorter segments. We also
introduce an efficient fine-tuning of wav2vec 2.0 to improve
SFC accuracy. We conducted experiments with an end-to-end
ST on MuST-C v2 for English-to-German. Our experimental
results showed that the proposed method retained 97.4%
of BLEU score for MuST-C segmentation included in the
corpus, surpassing 95.1% by SHAS. We also showed that the
proposed method reduced the translation time by about 20%
while improving translation accuracy by generating shorter
segments. Our case analysis revealed that our proposed method
sometimes outperformed MuST-C segmentation and produced
competitive translation results. Furthermore, an evaluation using 8 language pairs from MuST-C v1 and Europarl-ST showedthat the proposed method is effective for target languages anddomains that differ from the dataset used to train the SFC.
II. Related work
Early studies on segmentation for ST considered modelingwith the Markov decision process [4], [9], conditional random
fields [10], [11], and support vector machines [12]–[15].
They focused on cascade ST systems that consist of an ASR
model and a statistical machine translation model, which weresuperseded by newer ST systems based on neural machinetranslation.
In recent studies, many speech segmentation methods based
on VAD have been proposed for ST. Gaido et al. [16] andInaguma et al. [17] used the heuristic concatenation of VAD
segments up to a fixed length to address the over-segmentation
problem. Gállego et al. [18] used a pre-trained ASR model
called wav2vec 2.0 [8] for silence detection. Yoshimura et al.
[19] used an RNN-based ASR model to consider consecutive
blank symbols (“ ”) as a segment boundary in decoding using
connectionist temporal classification (CTC). Such CTC-based
speech segmentation has the following advantage; segment
lengths can be intuitively controlled by adjusting the numberof consecutive blank symbols that are regarded as segment
boundaries. However, these methods often split speech atinappropriate boundaries for ST because they mainly segmentspeech based on long pauses.
Re-segmentation using ASR transcripts is widely used incascade STs. Improvements in MT performance have beenreported by re-segmenting transcriptions to sentence units using punctuation restoration [11], [15], [20]–[22] and language
models [23], [24]. Unfortunately, they are difficult to use in
end-to-end ST and cannot prevent the ASR errors caused byspeech segmentation.
Corpus-based segmentation using manually or semimanually segmented speech corpora is a leading recent ap-

Transformer Encoder

wav2vec 2.0

Fig. 1: SFC of SHAS: Value of 𝑦 = 1 indicates that corresponding
frame is part of a segment of ST corpus, and 𝑦 = 0 indicatesthat it is part of a segment boundary.

proach. Wan et al. [3] introduced a re-segmentation modeltrained with movie and TV subtitle corpora to modify the segment boundaries in ASR output. Wang et al. [25] and IranzoSánchez et al. [26] proposed an RNN-based text segmentation
model trained with a bilingual speech corpus. Methods for
directly segmenting speech with a segmentation model havealso recently been proposed [1], [27]. Fukuda et al. [27] useda Transformer encoder to build a segmentation model andalso proposed a hybrid method that combines VAD and theprediction of the segmentation model. Tsiamas et al. [1] built
an SFC based on a pre-trained speech encoder called wav2vec
2.0. Their method, Supervised Hybrid Audio Segmentation(SHAS), is the current state-of-the-art method of speechsegmentation for ST.
In our approach, we improve the accuracy of the SFC byunfreezing parts or all of the wav2vec 2.0 parameters during
training. Performing full fine-tuning can significantly increase
the training cost compared to the original SHAS, so we use amethod of parameter-efficient transfer learning (PETL). PETL
is a research direction aimed at reducing the computationalcosts of applying large pre-trained models to new tasks [28]–[30].
III. Review of SHAS
In this section, we describe an SFC (III-A) and a probabilistic divide-and-conquer (pDAC) algorithm (III-B) of the stateof-the-art speech segmentation method called SHAS. Then we
describe its drawback: producing lengthy segments (III-C).
A. Segmentation frame classifier
The SFC determines whether each input speech frame
belongs to a segment or a segment boundary. It is implemented
as a neural network model with a single Transformer encoder
layer that is connected to the encoder of the pre-trained, selfsupervised speech model, wav2vec 2.0 (Fig. 1). Given an ST
corpus, each frame of speech is labeled as 1 or 0, dependingon whether it is included in a segment. During training, speechsegments of N seconds split at random positions are used with

3

400
350
300

# samples

250
200
150
100
50
0
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
>20

sequences of labels 𝑦 ∈ {0, 1} that correspond to model output
sequences. 𝑦 = 1 indicates that the corresponding frame is insidea segment, and 𝑦 = 0 indicates that it is outside of the segments,
i.e., belonging to a segment boundary.
At the inference time, given an unlabeled audio waveform, it issplit into contiguous segments of length N, which are then input
to the SFC. These segments are arranged in such a way that there
is no temporal overlap between consecutive segments. For each
input with length N, SFC predicts probabilities corresponding
to audio frames of length 𝑛 = 𝑁/320 due to the convolutionalfeature extractor of wav2vec 2.0. The wav2vec 2.0 parameters
are kept fixed during the training, and only the parameters of the
final Transformer encoder layer and the output layer are updated.

duration (seconds)(a) Sentence-aligned speech segmentation
140

B. Probabilistic divde-and-conquer# samples

120

Algorithm 1 pDAC

18:
19:
20:
21:
22:
23:

60
40

⊲ Init single segment

6: return 𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠
7:
8: Procedure RECURSIVE SPLIT(𝑠𝑔𝑚)9:if 𝑠𝑔𝑚.length < 𝑚𝑎𝑥 then10:
append 𝑠𝑔𝑚 to segments
11:
else
12:
13:
14:
15:
16:
17:

80

𝑗 ←0
𝑖𝑛𝑑𝑖𝑐𝑒𝑠 ← argsort 𝑝𝑟𝑜𝑏𝑠[𝑠𝑔𝑚]while True do
𝑠𝑔𝑚 𝑎 , 𝑠𝑔𝑚 𝑏 ← split 𝑠𝑔𝑚 at 𝑖𝑛𝑑𝑖𝑐𝑒𝑠[ 𝑗]𝑠𝑔𝑚 𝑎 ← trim( 𝑝𝑟𝑜𝑏𝑠[𝑠𝑔𝑚 𝑎 ], 𝑡ℎ𝑟)
𝑠𝑔𝑚 𝑏 ← trim( 𝑝𝑟𝑜𝑏𝑠[𝑠𝑔𝑚 𝑏 ], 𝑡ℎ𝑟)
if 𝑠𝑔𝑚 𝑎 .length > 𝑚𝑖𝑛 and𝑠𝑔𝑚 𝑏 .length > 𝑚𝑖𝑛 then
RECURSIVE SPLIT(𝑠𝑔𝑚 𝑎 )RECURSIVE SPLIT(𝑠𝑔𝑚 𝑏 )break𝑗 ← 𝑗 +1

During inference, pDAC divides speech based on the probability of each frame being included in a segment (𝑝𝑟𝑜𝑏𝑠)
predicted by SFC. pDAC is a recursive algorithm that splits
speech at the point least likely to be in a segment and applies
the same split to the two resulting segments (Algorithm 1). The
algorithm utilizes three hyperparameters: a maximum segment
length (𝑚𝑎𝑥) to regulate the length of the resulting segments,
a minimum segment length (𝑚𝑖𝑛) to prevent excessively small
noisy segments, and a threshold (𝑡ℎ𝑟) to trim a segment’s ends,
which are classified as being excluded from segments. After asplit, the resulting segments are trimmed to the first and last
frames 𝑖, 𝑗 with 𝑝 𝑖 , 𝑝 𝑗 > 𝑡ℎ𝑟. A split is performed until thesegment’s length is less than 𝑚𝑎𝑥. This allows pDAC to keep
the segments’ length within a certain range.

20
0
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
>20

1: Inputs: 𝑝𝑟𝑜𝑏𝑠, 𝑚𝑎𝑥, 𝑚𝑖𝑛, 𝑡ℎ𝑟
2: Initialize:
3:
𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠 ← empty List4:
𝑠𝑔𝑚 ← Tuple(0, 𝑝𝑟𝑜𝑏𝑠.length)
5: RECURSIVE SPLIT(𝑠𝑔𝑚)100

duration (seconds)(b) pDAC segmentation

Fig. 2: Histograms of segment length in each segmentation: Horizontal axis indicates length (seconds) of audio segment, andvertical axis indicates number of samples.

C. Lengthy segments by SHAS
SHAS outperformed the existing pause-based and lengthbased segmentation methods and consistently achieved bettertranslation quality across multiple language pairs. However,the segments generated by SHAS tended to be significantly
longer than the MuST-C segments. For example, the average
length of the speech segments in MuST-C English-to-German
was 5.79 seconds; the average length of segments generated
by pDAC was 9.17 seconds. Fig. 2 shows the segment length
distribution for each segmentation. The mode value for thesentence-aligned speech segmentation is about three seconds,
whereas the mode for the pDAC segmentation is noticeably
longer, about 11 seconds.
The cause of such long segments is that pDAC stops thesegmentation once the segment length falls below a predefinedvalue 𝑚𝑎𝑥, as mentioned. Thus, although SHAS is a corpusbased segmentation method using SFC, it also has aspects oflength-based segmentation. Its advantage is that speech can be
split into segments whose lengths are preferred by ST. Tsiamas etal. [1] found that a 𝑚𝑎𝑥 of values between 14-18 seconds works
well, which are considerably longer than the average MuST-C
segments’ length of 5.79 seconds.
On the other hand, longer segments by SHAS can degrade
the translation quality and the time efficiency of ST. The longer

4

Algorithm 2 pTHR+MA

1

𝑠𝑡𝑎𝑟𝑡 ← 0𝑡ℎ𝑟 𝑠 ← List with size 𝑚𝑎𝑥
𝑡ℎ𝑟 𝑠[: 𝑚𝑖𝑛] ← 0, 𝑡ℎ𝑟 𝑠[𝑚𝑖𝑛 :] ← 𝑡ℎ𝑟⊲ Set threshold filter 𝑡ℎ𝑟 𝑠
𝑡ℎ𝑟 𝑠 ← 𝐿𝑒𝑟 𝑝(𝑡ℎ𝑟 𝑠, 𝑚𝑖𝑛, 𝑙𝑒𝑟 𝑝 𝑚𝑖𝑛 , 0, 𝑡ℎ𝑟)
𝑡ℎ𝑟 𝑠 ← 𝐿𝑒𝑟 𝑝(𝑡ℎ𝑟 𝑠, 𝑙𝑒𝑟 𝑝 𝑚𝑎𝑥 , 𝑚𝑎𝑥, 𝑡ℎ𝑟, 1)
10:⊲ Apply Linear Interpolation
11:
𝑝𝑟𝑜𝑏𝑠 ← 𝑀𝑜𝑣𝑖𝑛𝑔 𝐴𝑣𝑒𝑟𝑎𝑔𝑒( 𝑝𝑟𝑜𝑏𝑠, 𝑛 𝑚𝑎)
12:⊲ Apply Moving Average of 𝑛 𝑚𝑎 frames
13: while 𝑠𝑡𝑎𝑟𝑡 < 𝑝𝑟𝑜𝑏𝑠.length do
14:
if 𝑝𝑟𝑜𝑏𝑠[𝑠𝑡𝑎𝑟𝑡] ≤ 𝑡ℎ𝑟 then15:
𝑠𝑡𝑎𝑟𝑡 ← 𝑠𝑡𝑎𝑟𝑡 + 1
16:
else
17:
𝑒𝑛𝑑 ← min(𝑠𝑡𝑎𝑟𝑡 + 𝑚𝑎𝑥, 𝑝𝑟𝑜𝑏𝑠.length)18:
for 𝑖 = 𝑠𝑡𝑎𝑟𝑡 . . . 𝑒𝑛𝑑 do
19:
if 𝑝𝑟𝑜𝑏𝑠[𝑖] ≤ 𝑡ℎ𝑟 𝑠[𝑖] then20:
𝑒𝑛𝑑 ← 𝑖
21:
break
22:
append Tuple(𝑠𝑡𝑎𝑟𝑡, 𝑒𝑛𝑑) to 𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠
23: return 𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠
4:
5:
6:
7:
8:
9:

a segment is, the more likely that translation omissions will
occur by neural machine translation [31]. In addition, the longer
a segment is, the more computational time that is required fortranslation due to the increased time complexity, longer decoder
outputs, etc.
To bridge the gap between SHAS and ST corpus segmentation, we need a segmentation algorithm that does not heavily
rely on length heuristics. In this case, SFC predictions become
more critical to translation accuracy.
IV. Proposed Method
Next we propose an online decoding algorithm that focuseson SFC prediction rather than segment length to produce shorter
segments (IV-A). We also introduce efficient fine-tuning to
update the parameters of the upper layers of wav2vec 2.0 toimprove SFC accuracy (IV-B) 1.
A. Probability-first decoding algorithm with moving average
We introduce a segmentation algorithm based on probability
thresholds (pTHR) that uses SFC predictions to find sentenceboundaries. pTHR progressively determines where segments
start and end using the probabilities of each speech frame thatis included in a segment corresponding to a sentence (𝑝𝑟𝑜𝑏𝑠),
predicted by SFC.
The algorithm simply takes the point at which the probabilityof being included in a segment exceeds the threshold as starting
point 𝑠𝑖 of segment 𝑖 and the point at which it again falls below it
as end point 𝑒 𝑖 . 𝑡ℎ𝑟 𝑠 is a sequence of probability thresholds that
1The source code
Wav2VecSegmenter

isavailableat

SMA ofprobability

𝑚𝑎, 𝑙𝑒𝑟 𝑝 𝑚𝑖𝑛 , 𝑙𝑒𝑟 𝑝 𝑚𝑎𝑥

https://github.com/ahclab/

Probability

1: Inputs: 𝑝𝑟𝑜𝑏𝑠, 𝑚𝑎𝑥, 𝑚𝑖𝑛, 𝑡ℎ𝑟, 𝑛2: Initialize:
3:
𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠 ← empty ListThreshold for𝑖-th segment

Threshold for𝑖 + 1-th segment0𝑠!

𝑠! + 𝑚𝑖𝑛 + 𝑙𝑒𝑟𝑝"!#𝑠! + 𝑚𝑖𝑛𝑒!

𝑠!&'𝑠! + 𝑚𝑎𝑥Time (s)𝑠! + 𝑚𝑎𝑥 − 𝑙𝑒𝑟𝑝"$%Fig. 3: Schematic diagram of proposed decoding algorithm.

determine the end of a segment, and its length is the numberof speech frames corresponding to 𝑚𝑎𝑥 seconds. The values
contained in 𝑡ℎ𝑟 𝑠 are almost 𝑡ℎ𝑟 (e.g., 0.5), although they are set
to 0 at positions below 𝑚𝑖𝑛 to ensure that the segment length isgreater than or equal to 𝑚𝑖𝑛. We also applied linear interpolation
between 𝑡ℎ𝑟 𝑠[𝑚𝑖𝑛 : 𝑙𝑒𝑟 𝑝 𝑚𝑖𝑛 ] and 𝑡ℎ𝑟 𝑠[𝑚𝑎𝑥−𝑙𝑒𝑟 𝑝 𝑚𝑖𝑛 :] to bias
the segment lengths to fall within the normal range. In Algorithm
2, 𝐿𝑒𝑟 𝑝(𝑙𝑖𝑠𝑡, 𝑠𝑡𝑎𝑟𝑡, 𝑒𝑛𝑑, 𝑎, 𝑏) linearly interpolates the values in𝑙𝑖𝑠𝑡 between 𝑠𝑡𝑎𝑟𝑡 and 𝑒𝑛𝑑, transitioning smoothly from value
𝑎 at 𝑠𝑡𝑎𝑟𝑡 to value 𝑏 at 𝑒𝑛𝑑. The segmentation procedure is asfollows:
1) The algorithm sequentially looks at the 𝑝𝑟𝑜𝑏𝑠 values,
starting with 0. A point at which the value first exceeds
the threshold 𝑡ℎ𝑟 is taken as the starting position of thefirst segment, 𝑠1 .
2) 𝑝𝑟𝑜𝑏𝑠[𝑠1 : 𝑠1 + 𝑚𝑎𝑥] and 𝑡ℎ𝑟 𝑠 are compared, and firstpoint 𝑗, where 𝑝𝑟𝑜𝑏𝑠[ 𝑗] < 𝑡ℎ𝑟 𝑠[ 𝑗], is taken as the end
position of the first segment 𝑒 1 . If no position 𝑗 is found,𝑠1 + 𝑚𝑎𝑥 is set to 𝑒 1 .
3) A point where the probability exceeds 𝑡ℎ𝑟 again is taken
as the starting position of the second segment 𝑠2 . The
positions of the second, third, . . . segments are identicallydetermined as the first segment.
Many existing VAD methods detect speech segments bysuch thresholding values such as acoustic power or CTC
probabilities, etc. Our algorithm, pTHR, differs from them bytaking SFC predictions as input. We automate the sentence-levelsegmentation as given in the ST corpus segmentation instead of
performing VAD.
Since pTHR performs thresholding without past information,
it can be computed in parallel and at high speed. On the other
hand, pTHR is a less stable method than pDAC because its
results are highly dependent on SFC accuracy. To stabilize the
SFC’s prediction, we first tried to use an autoregressive model as
SFC, but the model could not be trained due to data imbalance.
Then, inspired by classical time-series analysis methods, weincorporated a simple moving average (SMA) [32] into pTHR
to smooth the SFC predictions. We applied an SMA with awindow size of 𝑛𝑚𝑎 frames to 𝑝𝑟𝑜𝑏𝑠, which are the pTHR
inputs. Specifically, in line 11 of Algorithm 2, 𝑝𝑟𝑜𝑏𝑠[𝑖] is5FrozenTrainable

TABLE I: Number of segments of MuST-C v2 used in experiments

Transformer Encoderx(

Feed Forward

Feed Forward

LayerNorm

LayerNorm

Attention

Attention

LayerNorm

LayerNorm

Feature Extractor

wav2vec 2.0

Fig. 4: Segmentation frame classifier with efficient fine-tuning

updated as follows:
𝑝𝑟𝑜𝑏𝑠′ [𝑖]

=

𝑖
∑︁

1

𝑝𝑟𝑜𝑏𝑠[𝑘]𝑛𝑚𝑎𝑝𝑟𝑜𝑏𝑠[𝑖]=train
250,942dev
1,415

tst-COMMON
2,580

A. Datax)

Adapter

Language pair
English-to-German

𝑘=max(𝑖−𝑛𝑚𝑎 ,0)′We conducted experiments with the English-to-German (EnDe) ST as our primary forcus. We used MuST-C v2 for theexperiments, which consisted of triplets of segmented English
speech, transcripts, and target language translations. Table I
shows the statistics of the datasets used in the experiments.
MuST-C train and dev split were used to build the SFC models.
tst-COMMON was used as a test set for evaluation. The average
SNRs of the tst-COMMOM are 37dB for the period including
only ambient noise only and 1.52dB for the period including
applause and laughter.
To further validate the effectiveness of the SFC models acrossdifferent languages, we performed supplementary tests using
the 8 language pairs available in MuST-C v1. Specifically, thesetests were conducted from English to German, Spanish (En-Es),
French (En-Fr), Italian (En-It), Dutch (En-Nl), Portuguese (EnPt), Romanian (En-Ro), and Russian (En-Ru). Additionally, toassess the applicability of the method in different domains, we
performed tests using the Europarl-ST En-De dataset [34].

𝑝𝑟𝑜𝑏𝑠 [𝑖]This allows pTHR to stably perform segmentation even when the
SFC prediction accuracy is low while maintaining high speed.
We discuss the relationship between SFC prediction accuracy
and the stability of each algorithm in Section VI-B. Hereafter,
we refer to our proposed algorithm with 𝑛𝑚𝑎 = 0 as pTHR
and with 𝑛𝑚𝑎 > 0 as pTHR+MA. Pseudocode and a schematic
diagram of the pTHR+MA algorithm are shown in Algorithm 2
and Fig. 3.

B. Evaluation
The evaluation process followed [35]. First, the test set
audio files were split using one of the segmentation methods(described in V-C). Then the newly created segments weretranslated using an ST model (V-D), and the translations werealigned to the references in the test set using mwerSegmenter[35]. Finally, the BLEU scores [36] were calculated with
SacreBLEU [37]2 3. We also measured BERTScore [38]4 and
BLEURT [39]5.

B. SFC with memory efficient fine-tuning
In our proposed algorithm, the segmentation heavily relies on
the SFC prediction, and the translation quality is expected to beaffected by SFC accuracy. In SHAS, the wav2vec 2.0 parameters
were frozen during the SFC training. In contrast, we introduce
𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇, which updates the wav2vec 2.0 parameters of
SFC (Fig. 4). The parameters of the upper 𝑁 𝐹𝑇 𝐿𝑎𝑦𝑒𝑟 𝑠 encoder
layers are updated out of 𝑁 𝐴𝑙𝑙𝐿𝑎𝑦𝑒𝑟 𝑠 layers inherited from thewav2vec 2.0. While the self-attention mechanism has quadraticcomplexity with respect to input length, our model operates
with fixed-length inputs during both training and inference,
ensuring consistent memory allocation for self-attention. To
further optimize memory usage, we froze the parameters of
the feed-forward layers, which can be substantial in terms ofparameter count. In their place, we introduced parallel adapters[29]. These parallel adapters, as demonstrated by He et al.
[29], outperform sequentially inserted adapters and have beenvalidated for their efficacy in ST [33].

1) SFC: We trained the SFC models with random segmentsof 20 seconds of audio samples extracted from the training
data, following Tsiamas et al. [1]. As a pre-trained speechencoder of wav2vec 2.0, we used an XLS-R model [40] of
300 million parameters6, with 24 layers and a dimensionality of1024. The Transformer encoder has a single layer, 1024 model
dimensions, 2048 feed-forward dimensions, eight heads, prelayer normalization, GELU activation, and 0.1 dropout. Prior tobeing mapped to probabilities through a linear sigmoid layer, an
additional layer of normalization and 0.1 dropout were applied.
Models were trained for 16 epochs using Adam with an initial
learning rate of 2.5 · 10−4 (decayed with cosine annealing).
After training, the best checkpoint was selected based on theprediction performance of the dev set.
As SFCs of the baseline method SHAS (these models arealso called 𝑆𝐻 𝐴𝑆), we built a 𝑚𝑖𝑑𝑑𝑙𝑒 model that inherited the

V. Experimental Settings
We investigated the effectiveness of our proposed methodby conducting speech translation experiments and compared
several speech segmentation methods.

2https://github.com/mjpost/sacrebleu
3signature: BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.5.0
4bert-base-multilingual-cased
5https://storage.googleapis.com/bleurt-oss-21/BLEURT-20.zip
6https://huggingface.co/facebook/wav2vec2-xls-r-300m

C. Segmentation method

6lower 16 layers of the XLS-R encoder and a 𝑙𝑎𝑟𝑔𝑒 model that
inherited 24 layers. In their preliminary experiments, Tsiamas etal. [1] found that it is beneficial to inherit the lower 14 layers from
XLS-R. We also quoted the scores they reported for comparison.
We created the following six variations of 𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇
shown in Section IV-B. The model settings are shown in brackets
in the format 𝑁 𝐹𝑇 𝐿𝑎𝑦𝑒𝑟 𝑠 /𝑁 𝐴𝑙𝑙𝐿𝑎𝑦𝑒𝑟 𝑠 .
• middle+quarter (4/16)• middle+half (8/16)• middle+all (16/16)• large+quarter (6/24)• large+half (12/24)• large+all (24/24)
2) Segmentation algorithm: We used pDAC (Section III-B)
and pSTRM [5] as baseline segmentation algorithms. pSTRM
splits on the longest pause in the interval (𝑚𝑖𝑛 and 𝑚𝑎𝑥), if any,and otherwise it splits at 𝑚𝑎𝑥. pSTRM also emphasizes the target
segments’ length like pDAC, although it is an online decoding
algorithm like our proposed algorithms. The proposed algorithms are pTHR+MA with moving average and pTHR without
it. We tuned hyperparameters of each segmentation algorithm
using dev set. For pDAC and pSTRM, both of which prioritize
segment length, we fixed the threshold at 0.5, 𝑚𝑖𝑛 = 0.2, andtried 𝑚𝑎𝑥 = {28, 26, 24, 22, 20, 18, 16, 14, 12, 10}. For pTHR
and pTHR+MA, we fixed 𝑚𝑎𝑥 = 28, 𝑚𝑖𝑛 = 0.2, and tried𝑡ℎ𝑟 = {0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1} and a moving
average of {0, 0.1, 0.2, 0.4, 0.8, 1} seconds. As per the above
settings, our proposed algorithm also imposes length constraints
with 𝑚𝑎𝑥 and 𝑚𝑖𝑛. However, these are merely safeguards to
avoid extreme lengths, and usually, the segmentation positions
are determined by probability and threshold 𝑡ℎ𝑟.

leftmost columns in the table present the BLEU results. Fromthe perspective of the SFC model, 𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇 generallyachieved higher translation accuracy than 𝑆𝐻 𝐴𝑆. In terms ofalgorithms, when using 𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇 for SFC, pTHR andpTHR+MA tended to be either comparable to or even better
than the baseline. The best BLEU score (26.30) was obtained
when using the 𝑙𝑎𝑟𝑔𝑒 + 𝑎𝑙𝑙 of the 𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇 for SFC
and pTHR+MA for the segmentation algorithm. We discuss theeffects of the proposed segmentation algorithm in Section VI-B
and fine-tuning the wav2vec 2.0 in Section VI-C. The middle
columns in the table display the results for BERTScore F1.
While the trend of the results was similar to the previously
discussed BLEU scores, differences between our proposed
algorithms and the baselines were more pronounced. This could
potentially be attributed to the longer segments produced by
the baseline algorithms. Specifically, as the translation segments
become longer, there is an increased risk of misalignment during
BERTScore computation, which can lead to a decrease in thescore. The rightmost columns of Table II present the BLEURT
results. These showed a similar trend to BERTScore, suggestingthat shorter segments might receive higher evaluations inembedding-based automatic evaluations.
Table III shows the translation qualities of the segmentationsof MuST-C, SHAS, and the proposed method. The SHAS’s scoreis that reported by Tsiamas et al., and the proposed method’sscore is the best result from Table II. The proposed method
retained 97.4% of the BLEU score for sentence-aligned speechsegmentation, surpassing the 95.1% by SHAS.

D. Speech translation model
Following Tsiamas et al. [1], we used the joint speech-totext model [41] from fairseq [42] for MuST-C v2 En-De andEuroparl-ST En-De. This joint model is a Transformer encoderdecoder that can take both speech and text as input and share the
top layer of the encoder between the two modalities. It performs
knowledge distillation from the text-to-text translation task asa guide for the ST task [43], [44] and applies cross-attention
regularization to the encoder representations to bridge the gap
between the two modalities. We used a model trained on MuSTC En-De7. This model has 12 encoder and 6 decoder layers, witha dimensionality of 512 and 2048 feedforward dimensions. For
the tests on 8 language pairs using MuST-C v1, we employed amultilingual ST model trained on the MuST-C v18. This model
also has 12 encoder and 6 decoder layers, with a dimensionalityof 512 and 2048 feedforward dimensions. During inference,
decoding was performed with a beam search of beam size 5.

In table II, for the baseline SFC models 𝑆𝐻 𝐴𝑆 trainedwith fixed wav2vec 2.0 parameters (𝑚𝑖𝑑𝑑𝑙𝑒 and 𝑙𝑎𝑟𝑔𝑒), thepTHR results had significantly lower BLEU than those of theconventional segmentation algorithms, pDAC and pSTRM. This
result implies that the SFC performance that predicted the
ST corpus segmentation was insufficient, and in such cases,
the conventional segmentation algorithms, which heavily relyon length heuristics, had an advantage. On the other hand,as the number of layers to be trained 𝑁 𝐹𝑇 𝐿𝑎𝑦𝑒𝑟 𝑠 increased,
the pTHR results improved, achieving BLEU scores that werecomparable to those of pDAC and pSTRM. While otheralgorithms demonstrated a statistically significant difference
between 𝑆𝐻 𝐴𝑆 and 𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇 at a 𝑝 < 0.05 level, only
pTHR exhibited significance at 𝑝 < 0.001, underscoring its
substantial improvement. It suggests that the need to consider
segment length decreases with higher SFC accuracy.
Moreover, pTHR with a moving average (pTHR+MA) obtained the best BLEU score in most models. In particular, fora large model, it outperformed pTHR by more than 3 BLEU
points, demonstrating the effect of smoothing the probability
using the moving average to compensate for the model’s
low prediction accuracy. However, we found no significant
difference between pTHR and pTHR+MA for 𝑙𝑎𝑟𝑔𝑒 + ℎ𝑎𝑙 𝑓 ,𝑙𝑎𝑟𝑔𝑒 + 𝑎𝑙𝑙, etc., where there are many trainable parameters.
The best parameters for each segmentation algorithm are shownin Appendix A.

VI. Experimental Results
A. Translation quality
Table II shows the overall results of MuST-C v2 En-De
across BLEU, BERTScore F1, and BLEURT metrics. The
7https://github.com/facebookresearch/fairseq/blob/main/examples/speechtext joint to text/docs/ende-mustc.md
8https://github.com/facebookresearch/fairseq/blob/main/examples/speech
to text/docs/mustc example.md

B. Effectiveness of segmentation algorithm

7

TABLE II: Results by baseline model 𝑆𝐻 𝐴𝑆 and proposed model 𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇, and four decoding algorithms on MuST-C v2 En-De.
Numbers in brackets are the number of encoder layers (fine-tuned/inherited from wav2vec 2.0). For BLEU, †and ‡indicate statisticalsignificance (𝑝 < 0.05 and 𝑝 < 0.001, respectively) in comparison with the top row.
Model \ Decoding
𝑆𝐻 𝐴𝑆
middle (0/16)
large (0/24)
𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇
middle+quarter (4/16)
middle+half (8/16)
middle+all (16/16)
large+quarter (6/24)
large+half (12/24)
large+all (24/24)

pDAC

BLEU
pSTRM
pTHR

+MA

pDAC

BERTScore F1
pSTRM
pTHR

+MA

pDAC

BLEURT
pSTRM
pTHR

+MA

25.42
24.41

25.11
25.18

23.54
21.15

25.73
24.78

0.52010.5072

0.52330.53780.53240.50870.54180.53810.49580.48500.49410.49750.49920.47300.50500.4988

25.84
25.75
25.73
25.73
25.89†
25.95†

25.57†
25.52
25.71†
25.74†
25.58
25.70†

25.67‡
25.92‡
26.13‡
25.73‡
26.26‡
26.28‡

25.96
26.17
26.27†
26.18
26.15
26.30†

0.53810.53440.5394
0.53690.53630.55180.53420.53430.5401
0.54200.53580.53450.56410.57240.56970.56510.57510.56570.55920.57030.5634
0.55600.56230.56980.50820.50460.50540.50580.50770.51610.50350.5046
0.50290.50540.50490.50070.5232
0.5287
0.5264
0.52380.53170.52390.52130.5267
0.52110.51830.52050.5257TABLE III: BLEU scores of sentence-aligned segmentation, SHAS,
and our method in English-to-German Translation. Numbers in parentheses are the percentages of retained BLEU
scores of sentence-aligned speech segmentation
MuST-C v2 En-De
26.99 (100%)
25.67 (95.1%)
26.30 (97.4%)

Sentence-aligned
SHAS (Tsiamas+22)
Proposed method.

TABLE IV: Number of trainable parameters and maximum GPU
memory usage for each SFC
Model
middle (0/16)
middle+quarter (4/16)
middle+half (8/16)
middle+all (16/16)
large (0/24)
large+quarter (6/24)
large+half (12/24)
large+all (24/24)

Trainable / Non-trainable
parameters
8M / 215M
38M / 189M
59M / 173M
101M / 139M
8M / 315M
48M / 282M
80M / 257M
143M / 206M

GPU memory
(MB)
4,469
15,511
15,877
17,053
5,716
21,57023,05825,272

120

Loss

100
80
60

TABLE V: Prediction performance for each SFC model
Model
middle (0/16)
middle+quarter (4/16)
middle+half (8/16)
middle+all (16/16)
large (0/24)
large+quarter (6/24)
large+half (12/24)
large+all (24/24)

Precision
0.98940.98790.9861
0.98340.98020.99080.9896
0.9812

Recall
0.90460.91940.92820.93440.85320.90740.9166
0.9381F1
0.94490.95240.95630.95830.91230.94720.95170.9591the prediction performance for the dev set by each SFC model.
After determining the output to be 0 or 1 with a threshold valueof 0.5 from the probability for each frame, we calculated thePrecision, Recall, and F1 for the correct label. 𝑆𝐻 𝐴𝑆 models(𝑚𝑖𝑑𝑑𝑙𝑒 and 𝑙𝑎𝑟𝑔𝑒) have a high Precision of about 98%, but a
low Recall of about 85% to 90%. Label 𝑦 = 1 indicates that thecorresponding frame is inside the segment, while 𝑦 = 0 indicatesthat it is outside of it. Therefore, low Recall shows that insegment frames are often incorrectly judged as out-of-segment.
In SHAS, the length heuristics with pDAC mitigated the oversegmentation due to this low Recall. On the other hand, 𝑆𝐻 𝐴𝑆 +𝐹𝑇 𝑃𝑇 models (𝑚𝑖𝑑𝑑𝑙𝑒 + 𝑎𝑙𝑙, 𝑙𝑎𝑟𝑔𝑒 + 𝑎𝑙𝑙), with fine-tuning of
all the layers, improved the Recall by 3% to 7% with almost no
drop in Precision. The reduction in the need for length heuristics
proportional to model size, mentioned in Section VI-B, can be
explained by this improvement in Recall.

40
20
0

50,000100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

Step
middle (0/16)

middle+quarter (4/16)

middle+half (8/16)

middle+all (16/16)

Fig. 5: Loss curves of middle SFC models

C. Effectiveness of wav2vec 2.0 fine-tuning
Table IV shows the number of trainable and non-trainable
parameters and the maximum GPU memory usage for each SFC.
The translation quality by pTHR is somewhat proportional to thenumber of trainable parameters in SFC. The explanation is thatthe higher the percentage of parameters that can be updated, theeasier it is to fit a pre-trained speech model to the segmentationtask, as shown by the loss curves in Fig. 5. Table V shows

D. Improving time efficiency of ST
Figure 6 shows the trade-off between time efficiency andtranslation quality for each segmentation algorithm with SFC
model 𝑙𝑎𝑟𝑔𝑒 + 𝑎𝑙𝑙. The number of tokens per mini-batch
was set to 100,000, and ST inference was performed using
NVIDIA GeForce RTX 3090 on the same computer. Segments
were sorted by length before batching. The horizontal axis
shows the average ST inference times of five inferences, andthe vertical axis shows the BLEU. For pDAC and pSTRM,
the conditions were set in the range of 𝑚𝑎𝑥 = [2, 28],
and for pTHR and pTHR+MA, they were set in the rangeof 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 = [0.1, 0.9]. The proposed algorithms (pTHR
and pTHR+MA) achieved higher translation accuracy with
better time efficiency than the baseline algorithms (pDAC andpSTRM). In particular, the segments generated by pTHR were
processed about 25% faster than the pDAC segments while

8

27.5
27.0

BLEU

26.99

26.57

26.5

the proposed method outperformed MuST-C segmentation,
leading to competitive translation results.

26.31

G. Testing on other datasets

25.97

26.0

25.77

25.5
25.0
pDAC
pSTRM
pTHR+MA Oracle

24.5

pTHR

24.0
60

65

70

75

80

85

90

95 100 105 110 115 120

ST inference time (s)Fig. 6: ST inference time and BLEU for each segmentation method.
(model=large+all)

retaining about 97% of the translation accuracy of the sentencealigned speech segmentation. The shorter average segment
length of 5.67 seconds for pTHR compared to 9.17 seconds for
pDAC contributed to the higher speed because shorter segment
lengths are easier to parallelize and reduce the number ofautoregressions.
E. Segment length distribution
Figure 7 shows the length distribution of the segments generated by pSTRM, pTHR, and pTHR+MA using an SFC model
𝑙𝑎𝑟𝑔𝑒 + 𝑎𝑙𝑙. pDAC (Fig. 2b) tends to produce longer segments
compared to the sentence-aligned speech segmentation (Fig.
2a), as shown in Section III-C; the same is true for pSTRM (Fig.
7a). On the other hand, the pTHR (Fig. 7b) and pTHR+MA (Fig.
7c) distributions resemble that of the sentence-aligned speechsegmentation, producing relatively short segments, suggestingthat the segmentation of the proposed method was more faithful
to sentence segmentation than SHAS.
In another aspect, longer pDAC and pSTRM segments
reduced the contextual dependencies between segments, perhaps
improving the translation accuracy. Therefore, combining pTHR
and pTHR+MA with a context-aware ST [45], [46] might further
improve the translation performance.
F. Automatic segmentation repairs improper segmentation in
ST corpus segments
Although the prediction accuracy of the best SFC model has
room for improvement (Table V), it maintains a BLEU scoreas high as 97% of the sentence-aligned speech segmentation.
We conducted a case study, hypothesizing that automatic
segmentation can repair the incorrect segmentation in the ST
corpus. Fig. 8 shows examples of MuST-C and automatic
segmentation and their ST results. Compared to the sentence
segments, it can be seen that MuST-C segmentation often
overlooked sentence boundaries. This was caused by audiencelaughter and short pauses between utterances. Such undersegmentation increases segment lengths, which often degrade
the translation accuracy. In contrast, the proposed method
predicted the sentence segment boundaries more accurately than
MuST-C segmentation. Of course, there are over- and undersegmentations by the proposed method. However, in some cases,

Table VI shows the results of applying a combination of two
SFC models (baseline SHAS and the proposed SHAS+FTPT)
and four algorithms (pDAC, pSTRM, pTHR, and pTHR+MA) tothe 8 languages of MuST-C v1. No hyperparameters were finetuned in these tests, and all segmentation methods were appliedwith exactly the same configuration as used in his MuST-C v2
en-de. From the model perspective, SHAS+FTPT consistently
outperformed SHAS. Observations from the algorithm side werealso in line with our main experiments: the translation accuracyof the SHAS model was particularly low when using pTHR,
while pTHR+MA showed translation accuracy comparable topDAC. These results lead us to conclude that our proposed
method is effective for different target languages. Table VIIshows the test results for a different domain, Europarl-ST.
The improvement in translation accuracy when using pTHR
(from 24.06 to 25.68) suggests that our approach of unfreezing
wav2vec brings about an enhancement in generalization ability,
rather than mere overfitting to the task.
VII. Conclusion
In this study, we addressed a problem caused by the stateof-the-art speech segmentation method, SHAS, which tends
to generate overly long segments, degrading the quality andtime efficiency of speech translation (ST). We extended SHAS
to improve ST translation accuracy and efficiency by splitting
speech into shorter segments that correspond to sentences. We
introduced a simple segmentation algorithm using the moving
average of SFC predictions without relying on length heuristics.
We also introduced efficient fine-tuning of wav2vec 2.0 toimprove the SFC of SHAS and investigated the effects of model
size and trainable parameters on prediction performance. Experiments using ST corpora showed that the proposed segmentation
algorithm improves ST’s time efficiency by generating shorter
segments while maintaining translation quality comparable toexisting algorithms. Experimental results also showed that finetuning wav2vec 2.0 improves the accuracy of SFC, with aconcomitant significant improvement in ST quality.
Future research will focus on the proposed method for simultaneous speech translation. It will also investigate adapting to thenoisy and multi-speaker environments, optimizing segmentation
to maximize translation accuracy, and combining with contextaware ST.
Acknowledgments
Part of this work was supported by JST SPRING Grant
Number JPMJSP2140 and JSPS KAKENHI Grant Numbers
JP21H05054 and JP21H03500.
References[1] I. Tsiamas, G. I. Gállego, J. A. R. Fonollosa, and M. R. Costa-jussà,“SHAS: Approaching optimal Segmentation for End-to-End Speech
Translation,” in Proc. Interspeech 2022, 2022, pp. 106–110.

9

160

350

300

140

300

250

80
60

200
150

0

0

50
0
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
>20

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
>20

20

50

150
100

100

40

200

duration (seconds)

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
>20

100

250

# samples# samples# samples120

duration (seconds)(a) pSTRM segmentation

duration (seconds)(b) pTHR segmentation(c) pTHR+MA segmentation

Fig. 7: Histograms of segment length in each segmentation

Waveform

Intensity

Manually-split
sentence segments

But that being said,
it's a very good media
opportunity.

(Laughter)You know how
many people
watch these TED
Talks?

It’s a lot.

That's just a
So even with that caveat, I knew thatworking title, (Laughter) someone would buy the namingby the way.
rights.

Ground-truthtranslation

Aber davon mal
abgesehen, ist es einesehr gute mediale
Gelegenheit.

(Lachen)Wissen Sie, wieviele Menschen
diese TEDTalks
ansehen?

Eine ganze
Menge.

Das istübrigens nur(Lachen)derArbeitstitel.

ST of MuST-C
segments

Aber obwohl das gesagt wird, ist es eine sehr gute Medienchance. ( Gelächter ) Wissen Sie, wie viele Leute diese TEDTalks sehen ? Es ist eine Menge. Das istübrigens nur ein Arbeitstitel. ( Gelächter ) Selbst mit diesem Hammer wusste ich, dass jemand das Namensrecht kaufen würde.

ST of proposed
method’s segments… wenn ich Sie dazu brüche, dass Ihre Firma dumm aussieht, weil Sie estun, aber wenn ich sage, dass es eine sehr gute Medienchance ist.

Trotz di eser Warnung also wusste
ich, dass irgendjemand die
Namensrechte kauf en würde.

Wissen Sie, wie viele Leute dieseSelbst mit diesem Hammer wusste ich, dassDas ist übrigens nur der
TEDTalks sehen? Es ist einejemand den Namen Reis kaufen würde. Wenn
Arbeitstitel.
Menge.
Sie ihn vor einem Jahr gefragt hätten, …

Fig. 8: Examples of segmentation and its ST result: Vertical lines indicate the split position of a waveform.
TABLE VI: Results in BLEU by 𝑆𝐻 𝐴𝑆 and 𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇 for the 8 language pairs of MuST-C v1 tst-COMMON. Four numbers separated byslashes are the results of different algorithms (pDAC/pSTRM/pTHR/pTHR+MA).
Sentence-aligned
SHAS (middle)
SHAS+FTPT (large+all)
Sentence-aligned
SHAS (middle)
SHAS+FTPT (large+all)

MuST-C v1 En-De
23.31
21.75/21.42/19.69/21.93
22.36/21.85/22.61/22.65
MuST-C v1 En-Nl
27.92
26.28/25.97/22.97/25.20
26.74/26.44/26.48/26.20

MuST-C v1 En-Es
28.12
26.69/26.43/22.84/25.73
26.78/26.68/26.75/26.86
MuST-C v1 En-Pt
30.38
28.55/28.30/25.85/28.39
29.15/28.74/29.02/29.28

TABLE VII: Results in BLEU by 𝑆𝐻 𝐴𝑆 and 𝑆𝐻 𝐴𝑆 + 𝐹𝑇 𝑃𝑇
for En-De of Europarl-ST. Four numbers separated
by slashes are the results of different algorithms(pDAC/pSTRM/pTHR/pTHR+MA).
Sentence-aligned
SHAS (middle)
SHAS+FTPT (large+all)

Europarl-ST En-De
26.17
24.35/22.77/24.06/24.65
25.13/24.15/25.68/25.48[2] M. A. Di Gangi, R. Cattoni, L. Bentivogli, M. Negri, and M. Turchi,“MuST-C: a Multilingual Speech Translation Corpus,” in Proceedings ofthe 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume
1 (Long and Short Papers). Minneapolis, Minnesota: Association
for Computational Linguistics, Jun. 2019, pp. 2012–2017. [Online].
Available: https://aclanthology.org/N19-1202[3] D. Wan, C. Kedzie, F. Ladhak, E. Turcan, P. Galuščáková, E. Zotkina,
Z. P. Jiang, P. Bell, and K. McKeown, “Segmenting subtitles for correcting
asr segmentation errors,” in Proceedings of the 16th Conference of the

MuST-C v1 En-Fr
33.93
31.36/30.84/28.69/31.70
31.96/31.50/32.03/32.07
MuST-C v1 En-Ro
21.93
20.31/19.96/18.62/20.38
21.05/20.50/20.91/20.86

MuST-C v1 En-It
24.20
22.67/22.22/19.68/21.93
23.04/22.61/22.79/22.95
MuST-C v1 En-Ru
15.57
14.59/14.35/12.57/14.19
14.61/14.31/14.43/14.53

European Chapter of the Association for Computational Linguistics:
Main Volume, 2021, pp. 2842–2854.
[4] M. Sinclair, P. Bell, A. Birch, and F. McInnes, “A semi-markov model for
speech segmentation with an utterance-break prior,” in Fifteenth Annual
Conference of the International Speech Communication Association,
2014.
[5] M. Gaido, M. Negri, M. Cettolo, and M. Turchi, “Beyond voice activity
detection: Hybrid audio segmentation for direct speech translation,” inProceedings of the Fourth International Conference on Natural Language
and Speech Processing (ICNLSP 2021). Trento, Italy: Association
for Computational Linguistics, 12–13 Nov. 2021, pp. 55–62. [Online].
Available: https://aclanthology.org/2021.icnlsp-1.7[6] M. Paulik, S. Rao, I. Lane, S. Vogel, and T. Schultz, “Sentence
segmentation and punctuation recovery for spoken language translation,”in 2008 IEEE International Conference on Acoustics, Speech and Signal
Processing, 2008, pp. 5105–5108.
[7] E. Cho, J. Niehues, and A. Waibel, “Segmentation and punctuation
prediction in speech language translation using a monolingualtranslation system,” in Proceedings of the 9th International Workshop
on Spoken Language Translation: Papers, Hong Kong, Tableof contents, Dec. 6-7 2012, pp. 252–259. [Online]. Available:
https://aclanthology.org/2012.iwslt-papers.15

10[8] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0:
A framework for self-supervised learning of speech representations,”
Advances in Neural Information Processing Systems, vol. 33, 2020.
[9] S. Mansour, “Morphtagger: Hmm-based arabic segmentation for statisticalmachine translation,” in Proceedings of the 7th International Workshop
on Spoken Language Translation: Papers, 2010.
[10] T. Nguyen and S. Vogel, “Context-based Arabic morphological analysis
for machine translation,” in CoNLL 2008: Proceedings of the Twelfth
Conference on Computational Natural Language Learning. Manchester,
England: Coling 2008 Organizing Committee, Aug. 2008, pp. 135–142.
[Online]. Available: https://aclanthology.org/W08-2118[11] W. Lu and H. T. Ng, “Better punctuation prediction with dynamic
conditional random fields,” in Proceedings of the 2010 conference on
empirical methods in natural language processing, 2010, pp. 177–186.
[12] M. Diab, K. Hacioglu, and D. Jurafsky, “Automatic tagging of Arabic text:
From raw text to base phrase chunks,” in Proceedings of HLT-NAACL
2004: Short Papers. Boston, Massachusetts, USA: Association for
Computational Linguistics, May 2 - May 7 2004, pp. 149–152. [Online].
Available: https://aclanthology.org/N04-4038[13] F. Sadat and N. Habash, “Combination of Arabic preprocessing
schemes for statistical machine translation,” in Proceedings of the 21stInternational Conference on Computational Linguistics and 44th Annual
Meeting of the Association for Computational Linguistics. Sydney,
Australia: Association for Computational Linguistics, Jul. 2006, pp. 1–8.
[Online]. Available: https://aclanthology.org/P06-1001[14] E. Matusov, D. Hillard, M. Magimai-Doss, D. Hakkani-Tur, M. Ostendorf,
and H. Ney, “Improving speech translation with automatic boundary
prediction,” in Proceedings of Interspeech 2007, 2007, pp. 2449–2452.
[15] V. K. Rangarajan Sridhar, J. Chen, S. Bangalore, A. Ljolje, and
R. Chengalvarayan, “Segmentation strategies for streaming speechtranslation,” in Proceedings of the 2013 Conference of the North
American Chapter of the Association for Computational Linguistics:
Human Language Technologies. Atlanta, Georgia: Association for
Computational Linguistics, Jun. 2013, pp. 230–238. [Online]. Available:
https://aclanthology.org/N13-1023[16] M. Gaido, M. Negri, M. Cettolo, and M. Turchi, “Beyond voiceactivity detection: Hybrid audio segmentation for direct speechtranslation,” CoRR, vol. abs/2104.11710, 2021. [Online]. Available:
https://arxiv.org/abs/2104.11710[17] H. Inaguma, B. Yan, S. Dalmia, P. Guo, J. Shi, K. Duh, andS. Watanabe, “ESPnet-ST IWSLT 2021 offline speech translation
system,” in Proceedings of the 18th International Conference on Spoken
Language Translation. Bangkok, Thailand (online): Association for
Computational Linguistics, Aug. 2021, pp. 100–109. [Online]. Available:
https://aclanthology.org/2021.iwslt-1.10[18] G. I. Gállego, I. Tsiamas, C. Escolano, J. A. Fonollosa, and M. R.
Costa-jussà, “End-to-end speech translation with pre-trained models andadapters: Upc at iwslt 2021,” in Proceedings of the 18th International
Conference on Spoken Language Translation (IWSLT 2021), 2021, pp.
110–119.
[19] T. Yoshimura, T. Hayashi, K. Takeda, and S. Watanabe, “End-to-end
automatic speech recognition integrated with ctc-based voice activity
detection,” in ICASSP 2020-2020 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp.
6999–7003.
[20] E. Cho, J. Niehues, K. Kilgour, and A. Waibel, “Punctuation insertion forreal-time spoken language translation,” in Proceedings of the EleventhInternational Workshop on Spoken Language Translation, 2015.
[21] T.-L. Ha, J. Niehues, E. Cho, M. Mediani, and A. Waibel, “The kit
translation systems for iwslt 2015,” in Proceedings of the EleventhInternational Workshop on Spoken Language Translation, 2015.
[22] E. Cho, J. Niehues, and A. Waibel, “NMT-Based Segmentation andPunctuation Insertion for Real-Time Spoken Language Translation,” inProceedings of Interspeech 2017, 2017, pp. 2645–2649.
[23] A. Stolcke and E. Shriberg, “Automatic linguistic segmentation of
conversational speech,” in Proceeding of Fourth International Conferenceon Spoken Language Processing. ICSLP’96, vol. 2. IEEE, 1996, pp.
1005–1008.
[24] X. Wang, A. Finch, M. Utiyama, and E. Sumita, “An efficient andeffective online sentence segmenter for simultaneous interpretation,” inProceedings of the 3rd Workshop on Asian Translation (WAT2016),
2016, pp. 139–148.
[25] X. Wang, M. Utiyama, and E. Sumita, “Online sentence segmentation forsimultaneous interpretation using multi-shifted recurrent neural network,”in Proceedings of Machine Translation Summit XVII Volume 1: Research
Track, 2019, pp. 1–11.

[26] J. Iranzo-Sánchez, A. Giménez Pastor, J. A. Silvestre-Cerdà, P. BaqueroArnal, J. Civera Saiz, and A. Juan, “Direct segmentation models for
streaming speech translation,” in Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing (EMNLP). Online:
Association for Computational Linguistics, Nov. 2020, pp. 2599–2611.
[Online]. Available: https://aclanthology.org/2020.emnlp-main.206[27] R. Fukuda, K. Sudoh, and S. Nakamura, “Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech
Translation,” in Proc. Interspeech 2022, 2022, pp. 121–125.
[28] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe,
A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer
learning for nlp,” in International Conference on Machine Learning.
PMLR, 2019, pp. 2790–2799.
[29] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards
a unified view of parameter-efficient transfer learning,” in International
Conference on Learning Representations, 2021.
[30] Y.-L. Sung, J. Cho, and M. Bansal, “Lst: Ladder side-tuning for parameterand memory efficient transfer learning,” Advances in Neural Information
Processing Systems, vol. 35, pp. 12 991–13 005, 2022.
[31] K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, “On theproperties of neural machine translation: Encoder–decoder approaches,”in Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics
and Structure in Statistical Translation. Doha, Qatar: Association for
Computational Linguistics, Oct. 2014, pp. 103–111. [Online]. Available:
https://aclanthology.org/W14-4012[32] G. U. Yule, “The applications of the method of correlation to social andeconomic statistics,” Journal of the Royal Statistical Society, vol. 72,
no. 4, pp. 721–730, 1909.
[33] I. Tsiamas, G. I. Gállego, C. Escolano, J. Fonollosa, and M. R.
Costa-jussà, “Pretrained speech encoders and efficient fine-tuning
methods for speech translation: UPC at IWSLT 2022,” in Proceedingsof the 19th International Conference on Spoken Language Translation(IWSLT 2022). Dublin, Ireland (in-person and online): Association for
Computational Linguistics, May 2022, pp. 265–276. [Online]. Available:
https://aclanthology.org/2022.iwslt-1.23[34] J. Iranzo-Sánchez, J. A. Silvestre-Cerda, J. Jorge, N. Roselló, A. Giménez,
A. Sanchis, J. Civera, and A. Juan, “Europarl-st: A multilingual corpus forspeech translation of parliamentary debates,” in ICASSP 2020-2020 IEEE
International Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2020, pp. 8229–8233.
[35] E. Matusov, G. Leusch, O. Bender, and H. Ney, “Evaluating machinetranslation output with automatic sentence segmentation,” in Proceedingsof the Second International Workshop on Spoken Language Translation,
2005.
[36] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu:a method for automatic evaluation of machine translation,” inProceedings of the 40th Annual Meeting of the Association
for Computational Linguistics. Philadelphia, Pennsylvania, USA:
Association for Computational Linguistics, Jul. 2002, pp. 311–318.
[Online]. Available: https://aclanthology.org/P02-1040[37] M. Post, “A call for clarity in reporting BLEU scores,” in Proceedingsof the Third Conference on Machine Translation: Research Papers.
Brussels, Belgium: Association for Computational Linguistics, Oct. 2018,
pp. 186–191. [Online]. Available: https://aclanthology.org/W18-6319[38] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore:
Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675,
2019.
[39] T. Sellam, D. Das, and A. Parikh, “BLEURT: Learning robust metrics
for text generation,” in Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics. Online: Association
for Computational Linguistics, Jul. 2020, pp. 7881–7892. [Online].
Available: https://aclanthology.org/2020.acl-main.704[40] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh,
P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli,“XLS-R: Self-supervised Cross-lingual Speech Representation Learningat Scale,” in Proc. Interspeech 2022, 2022, pp. 2278–2282.
[41] Y. Tang, J. Pino, X. Li, C. Wang, and D. Genzel, “Improving speechtranslation by understanding and learning from the auxiliary texttranslation task,” in Proceedings of the 59th Annual Meeting of the
Association for Computational Linguistics and the 11th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers).
Online: Association for Computational Linguistics, Aug. 2021, pp. 4252–
4261. [Online]. Available: https://aclanthology.org/2021.acl-long.328[42] C. Wang, Y. Tang, X. Ma, A. Wu, D. Okhonko, and J. Pino, “Fairseq
S2T: Fast speech-to-text modeling with fairseq,” in Proceedings ofthe 1st Conference of the Asia-Pacific Chapter of the Association for
Computational Linguistics and the 10th International Joint Conference

11

on Natural Language Processing: System Demonstrations. Suzhou,
China: Association for Computational Linguistics, Dec. 2020, pp. 33–39.
[Online]. Available: https://aclanthology.org/2020.aacl-demo.6[43] Y. Liu, H. Xiong, J. Zhang, Z. He, H. Wu, H. Wang, and C. Zong, “Endto-end speech translation with knowledge distillation,” Proc. Interspeech
2019, pp. 1128–1132, 2019.
[44] M. Gaido, M. A. Di Gangi, M. Negri, and M. Turchi, “On knowledge
distillation for direct speech translation,” Computational Linguistics
CLiC-it 2020, p. 211, 2020.
[45] M. Gaido, M. A. D. Gangi, M. Negri, M. Cettolo, and M. Turchi,“Contextualized Translation of Automatically Segmented Speech,” inProc. Interspeech 2020, 2020, pp. 1471–1475. [Online]. Available:
http://dx.doi.org/10.21437/Interspeech.2020-2860[46] B. Zhang, I. Titov, B. Haddow, and R. Sennrich, “Beyond sentence-levelend-to-end speech translation: Context helps,” in Proceedings of
the 6522th Annual Meeting of the Association for Computational
Linguistics and the 11th International Joint Conference on Natural
Language Processing (Volume 1: Long Papers). Online: Association
for Computational Linguistics, Aug. 2021, pp. 2566–2578. [Online].
Available: https://aclanthology.org/2021.acl-long.200

Appendix
A. Hyperparameters
TABLE VIII: Best parameters of each algorithm chosen by dev set
and results in BLEU and number of segments (#seg).
(a) pDACModel
lna l16
lna l16
lna l16
lna l16
lna l24
lna l24
lna l24
lna l24

ft0
ft4
ft8
ft16
ft0
ft6
ft12
ft24

BLEU
25.42
25.84
25.75
25.73
24.41
25.73
25.89
25.95

#𝑠𝑒𝑔
1073
1656
1268
1855
1031
1636
1389
2279

𝑚𝑎𝑥
26
16
22
14
28
16
20
10

#𝑠𝑒𝑔
953
1186
1294
1300
1648
1584
1304
1292

𝑚𝑎𝑥
28
22
20
20
16
16
20
20

(b) pSTRM
Model
lna l16
lna l16
lna l16
lna l16
lna l24
lna l24
lna l24
lna l24

ft0
ft4
ft8
ft16
ft0
ft6
ft12
ft24

BLEU
25.11
25.57
25.52
25.71
25.18
25.7
25.58
25.70

(c) pTHR
Model
lna l16
lna l16
lna l16
lna l16
lna l24
lna l24
lna l24
lna l24

ft0
ft4
ft8
ft16
ft0
ft6
ft12
ft24

BLEU
23.54
25.67
25.92
26.13
21.15
25.73
26.26
26.28

#𝑠𝑒𝑔
3639
2353
2703
2525
4860
2588
2638
2149

𝑡 ℎ𝑟
0.1
0.1
0.2
0.2
0.1
0.2
0.2
0.1

𝑚𝑎
0
0
0
0
0
0
0
0

𝑡 ℎ𝑟
0.1
0.1
0.2
0.1
0.1
0.1
0.1
0.1

𝑚𝑎
0.2
0.1
0.1
0.1
0.4
0.1
0.1
0.1

(d) pTHR+MA
Model
lna l16
lna l16
lna l16
lna l16
lna l24
lna l24
lna l24
lna l24

ft0
ft4
ft8
ft16
ft0
ft6
ft12
ft24

BLEU
25.73
25.96
26.17
26.27
24.78
26.13
26.15
26.30

#𝑠𝑒𝑔
1944
2055
2464
1981
1816
1914
2005
2044

12

Ryo Fukuda received a B.S. degree in engineeringfrom Toyohashi University of Technology in 2019and master’s degree in engineering from the Nara
Institute of Science and Technology in 2021. He
is currently working toward the Ph.D. degree with
the Nara Institute of Science and Technology. His
research interests include speech and natural language
processing with speech translation.

Katsuhito Sudoh received a B.S. degree in engineering and master’s and Ph.D. degrees in informatics
from Kyoto University in 2000, 2002, and 2015.
He was with NTT Communication Science Laboratories from 2002 to 2017 and is currently an Associate
Professor with the Nara Institute of Science andTechnology. His research interests include machinetranslation and natural language processing.
Dr. Sudoh is a member of the Association for Computational Linguistics (ACL) and the International
Speech Communication Association (ISCA).

Satoshi Nakamura (Fellow, IEEE) received a B.S.
degree from the Kyoto Institute of Technology in 1981
and a Ph.D. degree from Kyoto University in 1992.
From 1994 to 2000, he was an Associate Professor
with the Graduate School of Information Science,
Nara Institute of Science and Technology, Japan.
He was the Director of the ATR Spoken Language
Communication Research Laboratories from 2000
and 2008 and the Vice President of ATR from 2007
to 2008. He was the Director General of the Keihanna
Research Laboratories and the Executive Director of
the Knowledge Creating Communication Research Center, National Institute ofInformation and Communications Technology, Japan from 2009 to 2010. He iscurrently a Professor with the Graduate School of Science and Technology, Nara
Institute of Science and Technology, a Project Leader of the Tourism Information
Analytics Team, RIKEN, Center for Advanced Intelligence Project AIP, an
Honorary Professor with the Karlsruhe Institute of Technology, Germany, andan ATR Fellow. He is also a Director of the Augmented Human Communication
Laboratory and a Full Professor with the Graduate School of Information
Science, Nara Institute of Science and Technology. His research interests include
modeling and systems of speech-to-speech translation and speech recognition.
Prof. Nakamura is one of the leaders of speech-to-speech translation researchand has been serving on various worldwide speech-to-speech translation
research projects, including C-STAR, IWSLT, and A-STAR. He has been an
Elected Board Member of the International Speech Communication Association(ISCA) since June 2011, an IEEE Signal Processing Magazine Editorial Board
Member since April 2012, and an IEEE SPS Speech and Language TechnicalCommittee Member since 2013. He has received the following awards: theYamashita Research Award, the Kiyasu Award from the Information Processing
Society of Japan, the Telecom System Award, the AAMT Nagao Award, theDocomo Mobile Science Award in 2007, and the ASJ Award for DistinguishedAchievements in Acoustics. He received a Commendation for Science andTechnology from the Minister of Education, Science and Technology and theCommendation for Science and Technology by the Minister of Internal Affair
and Communications. He also received the LREC Antonio Zampolli Award in2012.