High-Fidelity Simultaneous Speech-To-Speech Translation
Authors: Tom Labiausse
Laurent Mazaré
Edouard Grave
Alexandre Défossez
Neil Zeghidour
Venue: ICML 2025 poster (2025)
Keywords: audio language models
speech translation
multimodal language models
speech-to-speech
Primary area: applications->language_speech_and_dialog
TL;DR: natural, high-quality simultaneous speech translation with voice preservation

Abstract: We introduce Hibiki, a decoder-only model for simultaneous speech translation. Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation. We furthermore address the fundamental challenge of simultaneous interpretation, which unlike its consecutive counterpart --where one waits for the end of the source utterance to start translating-- adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk. To do so, we introduce a weakly-supervised method that leverages the perplexity of an off-the-shelf text translation system to identify optimal delays on a per-word basis and create aligned synthetic data. After supervised training, Hibiki performs adaptive, simultaneous speech translation with vanilla temperature sampling. On a French-English simultaneous speech translation task, Hibiki demonstrates state-of-the-art performance in translation quality, speaker fidelity and naturalness. Moreover, the simplicity of its inference process makes it compatible with batched translation and even real-time on-device deployment. We provide examples on *huggingface.co/spaces/kyutai/hibiki-samples* as well as models and inference code at *github.com/kyutai-labs/hibiki*.

High-Fidelity Simultaneous Speech-To-Speedh Translation
Tom Labiausse'
Laurent Mazaré!
Edouard Grave!
Alexandre Défossez'
Neil Zeghidour'
Abstract
while fitting many use cases such as offline video dubbing. A more challenging setting is that of sinultaneous trans- lation, where translated speech is produced on-the-fly. This task requires a real-time prcciion-mmaingg process to evalu- ate whether enough context has been accumulated to reliably translate another chunk of speech. When cast as a machine learning problem, this endeavor presents additional chal- lenges such as the lack of speech aligned a chunk-level.
We introduce Hibiki, decoder-only model for simultaneous speech translation. Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech- to-text and speech-to-speess furthermore address the fundamental challenge of simultaneous interpretation, which unlike its one waits for the end of the source utterance to start translating adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk. To do so, we introduce weakly- supervised method that leverages the perplexity of an off-the-shelff text translation system identify optimal delays a per-word basis and create aligned synthetic data. After supervised training, Hibiki performs adaptive, simultaneous speech translation with vanilla temperature sampling. On a French-English simultaneous speech translation task, Hibiki demonstrates state-of-the-art performance in translation quality, speaker fidelity and naturalness. Moreover, the simplicity of its inference process makes it compatible with batched translation and even real-time on-device deployment. We provide examples as well as models and inference code.
translation. We
Hibiki is a decoder-only model which synchronously receives source speech and generates translated speech by leveraging multistream architecture. In this context, nested global and local Transformers (Vaswani et al., 2017) jointly model two audio streams by predicting a hierarchy of text and audio tokens for each of them. At inference time, temperature sampling combined with a causal audio codec allows for streaming inputs and outputs. While this architecture was originally introduced by Défossez et al. (2024) for full-duplex spoken dialogue, we show how it provides a simple and convenient architecture for simultaneous speech translation. To train Hibiki, we generate synthetic parallel data by translating and resynthesizing the transcript of single-language audio. While this provides pairs of inputs and outputs aligned the sequence level, this does not allow learning fine- one-grained alignments. We thus introduce "contextual alignment", simple method based on the perplexity of an off-the-shelf machine translation system (Kudugunta et al., 2023) to derive word-level alignments. By then introducing proper silences into target speech, can train Hibiki adapt its flow in real-time, without the need for complex inference policies. Moreover, observing that training data varies widely in speaker similarity, we propose to label training examples with categories of speaker similarity, which avoids filtering the training data while allowing to favor high speaker similarity at inference with classifier-free guidance.
consecutive counterpart-
1. Introduction
We introduce Hibiki ("echo" in Japanese), a system for streaming and expressive speech-to-speech (S2ST) and specch-to-text (S2TT) translation. Most work in speech translation has focused on the offline (or consecutive) setting where the model has access to the full source utterance before translates, it provides useful context
In a French-te Bennchtto-Eane so-English translation task. Hibiki outperforms previous work in translation quality, speaker similarity and naturalness. We demonstrate how the simplicity of our infer- ence process allows for translating hundreds of sequences in real-time on GPU, and how a distilled model can run in real- time on a smartphone. Human evaluations also show that Hi- biki the first model to provide an experience of interpreta- tion close to human professionals. We will release our code, models, and a high quality 900 hours synthetic dataset.
Kyutai, Paris, France. Correspondence to: Hibiki chibiki@kyutai..o
Proceedings of the ndnd International Conference Machine I Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

High-Fidelity Simultancous Speech-To-Speech Translation
2. Related Work
batched. This makes Hibiki able translate hundreds of sequences on single GPU, provides convenient support for classifier-free guidance, see e.g. Section 3.3, and allows to run in real time on device, as shown in Section 4.6.1. Human evaluations in Section 4.6 show that Hibiki significantly outperforms Seamless in terms of naturalness and audio quality, getting close to human interpretation.
2.1. End-to-end speech translation
Speech translation can be traced back to the early 1990s (Jain et al., 1991) with a first generation of systems that combined automatic speech recognition (ASR), ma- chine translation (MT) and text-to-speech synthesis (TTS). While such cascaded approaches allowed for the growth of speech translation (Wahlster, 2000; Nakamura et al., 2006), they suffer from two main limitations. First, they are subject to compounding errors due to combining separately trained models. This motivated the merging of ASR and MT into single speech-to-tex translation (S2TT) model (Berard et al., 2016; Weiss et al., 2017; Wang et al., 2020; b021b) that can provide inputs to TTS model. However, a second limitation of cascaded systems remains: as the input speech goes through a text bottleneck, the non-linguistic informa- tion it carries- such as speaker identity or prosody- is lost and cannot be transferred to the output speech. End-to-end speech-to-speech translation (S2ST) (Jia et al., 2019; Lee et al., 2022a; Jia et al., 2022a; Rubenstein et al., 2023) addresses this issue by directly predicting target speech from source speech, allowing for retaining paralinguistic information, including voice identity. A notable aspect of most end-to-end S2ST models is that they leverage auxiliary text or phoneme translation tasks in training, that are then discarded (Jia et al., 2022a) or run in parallel (Zhang et al., 2024a) to the main speech translation task at inference. Hibiki performs end-to-end S2ST with voice transfer along with S2TT, but instead of running these tasks in parallel, Hibiki uses the predicted text as a scaffolding for speech generation at inference time. Moreover, since Hibiki predicts aligned speech and text tokens, it provides wevd-level timestamps in the target language.
2.3. Synthetic data for simultaneous translation
As pointed out by the IWSLT community (Ahmad et al., 2024), applying current state-of-the-art simultancous translation technologies to complex scenarios (spontaneous speech, accents, background noise, dialogues, etc.) is still challenging. Unlike the offline text translation task, large interpretation datasets with voice preservation from source to target don't exist thus justifying the efforts to synthesize such data. Recent approaches (Wang et al., 2024) focused on speech-to-text and designed heuristics based on multilin- gual word aligners (Dyer et al., 2013) to create simultaneous translation examples from offiine translation data. Similarly, the data processing behind the training dataset of Hibiki leverages the perplexity of an off-the-shelf translation system to build more robust word aligmments designed for the streaming translation task. It also synthesizes target audios by maintaining high speech naturalness thus being the first implementation of full synthetic data pipeline for simultaneous speech translation with voice preservation.
3. Method
We consider an utterance in a source language represented as a monophonic waveform X E R sampled at a frame rate fa 24 kHz, of duration d. Similarly, its translation is given in a target language, denoted Y E f.dd We assume X is padded to ensure both have the same duration. Our objective is to model PP [X]. We further add the constraint that the modeling of Y knowing X should be causal and of minimal delay with respect to the source utterance, e.g. the same constraints that are imposed on a human interpreter in the context of live translation. To learn this constraint via supervised learning, Y must itself be built to respect this causality constraint. We first assume that Y respects this constraint, and present how to model its distribution. Then, we introduce information theory criterion to verify whether Y is causal with respect to X, and to adapt non- causal interpretation into a causal one.
2.2. Simultancous speech translation
While the first attempts at simultancous speech translation focused on speech-to-text (Ren et al., 2020; Ma al., 2021; Zeng et al., 2021), Seamless (Barrault et al., 2023) and StreamSpeech (Zhang et al., 2024a) have introduced end- to-end simultancous S2ST (Zeng et al., 2021; Barrault et al., 2023; Zhang et al., 2024a). Both systems predict discrete speech units with autoregressive models before decoding them to audio using a neural vocoder, and rely on a specific policy for inference. While StreamSpeech translates into a canonical voice, Seamless performs voice transfer from source to target. Hibiki also performs simultaneous S2ST and S2TT, while transferring voice characteristics. However, Hibiki relies on a decoder-only model which operates at a constant frame rate and performs inference with simple temperature sampling or greedy decoding. In particular, this allows for batching, unlike StreamSpeech's 's and Seamless's 's policies that involve a complex control flow that cannot be
3.1. Modeling
We build on the framework introduced by Défossez et al. (2024) for the joint modeling of multiple sequences of dis- crete tokens, obtained from a neural audio codec.
2

---

High- Simultancous Speech-To-Speech Translation
Vi.a
Vi.a
Temporal Transformer
contect
Idens
Depth Transformer
Embedding
61
V2.4
Figure 2. Joint sequence modeling with contextual alignment.
Figure 1. Architecture of the RQ-Transforme With notations:
From the source stream, Hibiki predicts its Inner Monologue text stream, and audio tokens. Its output is aligned for causality, depicted in Figure 4. Figure adapted from Défossez et al. (2024).
Vi,g T(A)).g Figure adapted from Défossez al. (2024).
3.1.1. NEURAL AUDIO CODEC
We use the pre-trained causal and streaming Mimi codec (Défossez et al., 2024) encode X and Y into low framerate sequences of discrete tokens. Mimi consists of an encoder and decoder from and to the waveform domain, and of an information bottleneck using Residual Vector Quantization (RVQ) (Zeghidour et al., 2022). The encoder transforms an input waveform of duration d into a latent vector U R cd with C' the dimension of the latent space, and fr 12.5 Hz the frame rate. U is then pro- jected to its nearest neighbor in codebook table with NA entries. The residual of the projection is further projected into a second table with the same cardinality, and so forth until Q projections have been performed. The last residual is discarded, and the decoder is trained to reconstruct the input waveform from the sum of the projected tensors. The codebooks are trained through exponential moving average, along with commitment loss (Razavi et al., 2019). The rest of the model is trained only through an adversarial loss with feature matching (Défossez et al., 2024).
cardinality and framerate (Copet et al.. 2023). Following Yang et al. (2023); Défossez et al. (2024), we leverage a RQ- -Q-Tasssformer (Lee et al., 2022b) as shown in Figure I to model (At.g) both over the time /. and quantizer q axes. It consists in large Temporal Transformer (Vaswani et al., 2017), operating the same framerate fr the codec, and being fed all the tokens generated so far, e.g. for all S f.dd.
Z. Temp(Ao. .... A,-1) ERD
(I)
Ao is defined a deterministic token indicating the start of the generation. Then, smaller scale Depth Transformer models auto-regressively the tokens At,,, Ar, over the quantizer axis, e.g. for all / S Sr d and S Q
4.g = Dep( Aro..... At.-1) E RNN with Ar, also special token, and with the goal of having,
(2)
For language modeling, we are not interested in the quan- tized latent vector and its residuals, but in the discrete in- dices of the entry in the codebooks it is projected to. We denote those (Ar,)) (1...., NA W fx-xx For Mimi we have f. 12.5 Hz and Q varies up to 32, but most 16. Following Zhang et al. (2024b); Défossez et al. (2024), the output of the first quantization level is trained to replicate semantic information obtained from a WavLM self-supervised audio model (Chen et al., 2022). Following conventions of Borsos et al. (2022), we refer to Ar as the semantic tokens, and Ar,22 the acoustic tokens. The acoustic tokens are arranged in coarse fine manner, the first ones have the most importance, and the latest model fine details of the audio and ensuring smooth perception.
softmax(le.)) IP [A Au, ................................ At-1,
Aeu, Ai- At.
Following (Copet al., 2023; Défossez et al., 2024), further introduce acoustic delay of 2 time steps, meaning that we model (r(A)t.g) instead of A,
at
TA))a1 = A
T(A)Lg A
Vt 3,Yq
T(A).. =0
A4 < 3, 2 2,
with O being special token. The delay is removed before decoding audio with the codec.
1.3.3. TRANSLATION AS MULTISTREAM MODELING
3.1.2. JOINT MODELING OF DISCRETE AUDIO TOKENS
We have presented how the oQaanssffrrmer given by eq. (1) and (2) allows for jointly modeling multiple discrete streams of tokens. We adapt this framework for the task
The discrete tokens for audio streams cannot easily be sum- marized into single discrete sequence with reasonable
3

---

High-Fidelity Simultancous Speech-To-Speech Translation
of joint speech-to-specch and speech-to-text simultaneous translation illustrated in Figure 2. We concatenate the audio tokens A obtained from the target interpretation Y, with the tokens A from the source utterance X along the q-axis, e.g.
Input context
Output context
Log-Ilkellhodd next output word
transtate
Jo vala
I !wil transiate
Je traduire
transtate
Je vais traduine en
wil transiate
(4) We observe a benefit from modeling the tokens A at train time, although at inference time, predictions for those tokens are skipped and actual tokens of the input are used instead.
A = concaly
vais traduiee en anglais.
transiate
vosi contaol length
Input sentence: Jo vals traduire en anglets. Output sentenca: will translate English.
Figure 3. Contextual alignment. We compute the log- likelihoodd of the word "into" with a pre-trained text translation model, for various input truncations. Once the matching source word "en" appears, we observe large increase in log-likelihood, see cq. (6).
Défossez al. (2024) showed generating an Inner Mono- logue, i.e. padded text tokens aligned with the content of the generated audio, is beneficial to the quality and stability of the generated audio. This is similar to multi-task learn- ing where the translation is predicted both in the audio and text domain. Hibiki thus also predicts a text stream corre- sponding to the transcription of the output Y, with sufficient padding between words to keep them aligned with the au- dio. Note that unlike previous multi-task translation work, Hibiki makes active use of this capability at inference time. We denote We the text stream, with cardinality Nw and the same frame rate fy as the audio streams.
minimize the uncertainty on Tj. Any alignment strictly less conservative than ndتاal would risk the model hallucinating at inference if trained on. Any alignment strictly more con- servative would still be causal, but introduce more latency.
Contextual alignment. We introduce a criterion to esti- mate a ideal Let's denote the conditional log- likellidd
log(p;.)) = log (TP [T)]S), ... S., Tı, -T-1)))
(5)
3.1.4. ARCHITECTURAL DETAILS
we expect logPn, to increase with i, as more context is beneficial. We conjecture that log(p).) log(p;,.-1) is maximal for i ag.- We compute an estimate log(E).)) of log(P;.)) with an off-the-shelff text translation language model MADLAD-3B (Kudugunta et al., 2023), by feeding it truncated input up to the i-th word, which we use to define a contextual alignment, illustrated in Figure 3,
We provide architectural seper-parameters in Section 4.1. At time-step f, the tokens from the step t- L, ((A))--) T(A") ) and Wi-1, are fed into dedicated embedding tables, whose contributions are summed. For the first time step = 1, BOS token is used instead. We then use standard Transformer layers (Vaswani et al., 2017), with gated SiLU activation (Shazeer, 2020; Hendrycks & Gimpel, 2016). A linear layer maps its output Z, to logits for the text token W,, The Depth Transformer then operates for 2 Q steps: the first half to estimate the logits for the output stream, and the next half for the input stream. Each depth step g takes as input Z, summed with a learnt embedding of the previous audio token Ar.q-1, or W, for = 1.
ctx = arg max (log(fp.s) - log((),5-1)]
(6)
is"
Examples of alignments are given in the Appendix, Figure 9.
3.2.2. AUDIO DOMAIN ALIGNMENTS
Given (X, Y), we transcribe both with timestamps with a Whisper model (Radford et al., 2023; Louradour, 2023) and apply eq. (6). The pair (X,Y) respects the alignment (ag*) if the timestamp of the j-th word in Y comes after the timestamp of the ay-th word in X. To reduce the impact of errors, we require Y to lag by at least 80t = 2 sec. compared to the contextual alignment, and cut spikes higher than 25% of the average delay over a window of 5 words.
3.2. Alignment and synthetic interpretation data
We have assumed pairs Y) that respect the constraint of simultaneous interpretation. We now introduce an unsupervised criterion to estimate and enforce causality dependencies between the source and target utterances.
3.2.1. TEXT DOMAIN ALIGNMENTS
Silence insertion. If Y doesn't it respect the alignment, one can simply transform by inserting sufficient silences be- fore word, illustrated in Figure 4, with two limitations: (i) silence insertion can lead to hard cuts when the times- tamps are inaccurate or no pause exists between words; (ii) the corrected Y might be arbitrarily late on the ideal alignment, e.g. if the speech rate is slower in Y than in X. We apply this method during the speech translation training.
Let us first express formally those constrained in the text domain. Let us take S (S,....,5), the sequence of words in the utterance X, and T = (Ti...., Tm, that in Y.
Ideal alignment. We seek to define an ideal alignment (ajiaa)) E {l...., ,n)",, where ideal indicates the index of the word in S that the j-th word in T should wait for to
4

---

High-FFieeltyy Simultancous Speech-To-Speech Translation
CVSST
Input x
4.0
CVSSTT (Retyntheiird)
3.5
Bongaour
3.0
Density
2.5
Output Y
2.0
Helloo
Im
1.5
Siience Insertion
1.0
Hallia
Hilii
0.5
Algromeboonare TTS
Da Doocooo
|M
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Speaker Similarity
Hollo
Hibiki
Figure S. Speaker similarity between source and target speech in CVSS-T training data, before and after resynthesis.
Figure 4. Generating aligned interpretations. We extract unsu- pervised word level contextual alignment, which we lift to audio by either inserting silences, or re-synthesising with an alignment aware TTS. See Section 3.2 for details.
also regenerate CVSS-T with our alignment-aware TTS, as it allows for voice transfer. As shown in Figure 5, the resynthesized CVSS-T displays a higher similarity, with an average of 0.47. Yet, our training mixture which combines synthetic data and resynthesized CVSS-T still covers a wide range, with a significant mass below 0.40.
We obtain more natural aligned data by re)-syythesiiingg Y with a TTS model able to fol- low hard and soft constraints on word locations, along with accurate speaker conditioning. For existing datasets, this can have the added benefit of improving the word error rate and the speaker similarity, as illustrated in Section 3.3. Fol- lowing (Défossez et al., 2024), Appendix C, we train a TTS with both audio and a synced text stream as output, along with voice conditioning. The text stream is constrained to match exactly the text generate, with the model having only the freedom to insert padding tokens. The audio output is late on the text, so that its content is conditioned by it, both for content and timestamps. If the TTS is early on the alignment axx padding tokens are forced to delay the next word. When the TTS is lagging on its target, a penalty is added on the logits the padding token. The penalty scales from -2 as the lag increases from increases smoothly the rate of speech catch up with the source audio. We perform 6 to 8 generations per input, and select the best one based on word error rate first, and speaker similarity second. We apply this only for the fine-tuning speech translation dataset.
Aliggmeet--waree TTS.
Conditional training. Filtering training data to only keep pairs of examples with high similarity would improve voice transfer, but the resulting reduction in training data would likely affect translation quality (e.g. keeping only samples with speaker similarity above 0.40 would re- move 45% of training data). We rather rely on condi- tional training (Keskar et al., 2019; Korbak et al., 2023) to inform the generative model of how reliable each train- ing example is in terms of voice transfer. We label each training sample with a discrete "voice transfer score" in {very bad, bad, neutral, good, very -roddoood based on quantiles of speaker similarity, each label being associated to learnable embedding added to the model's inputs at every timestep. Importantly, the quantiles are com- puted before combining the synthetic data and CVSS-T to guarantee that the model does not associate a specific label to a specific dataset rather than the actual speaker similarity. At inference time, we always pass the very-good label.
seconds. This
3.3. Voice Transfer
Improving voice transfer data. Training speech trans- lation models with voice transfer typically amounts to supervised training on synthetic paired sequences of the same speaker. In particular, CVSS-T (Jia et al.. 2022b) is the standard training set for S2ST with voice transfer and provides such artificial targets. However, Figure 5 shows that the average speaker similarity as measured by the cosine similarity between speaker embeddings of source and target- on this dataset is very low with an average of 0.23. As calibration, state-of-the-ar cross- -ingual voice transfer lies around 0.40 (Rubenstein et al., 2023). We thus
Classifier-free guidance. Following Kreuk et al. (2023) we can increase the impact of the conditioning by using classifier-free guidance. We compute logits both with con- ditionings very good and very bad and sample from:
74, very + (1 - (1
very-good
7) very
which is compatible with real-time inference by producing both sets of logits with a batch size of 2. Section 4.6 shows that it significantly improves voice transfer.
5

---

High-Fidelity Simultancous Speech-To-Speech Translation
4. Experiments
Table I. Comparison with offfine baselines. We also report per- formance from closed-souree streaming model (*) as it uses the same evaluation protocol.
4.1. Architectural hyper- parreeeprsmeeers
Hibiki consists of a Temporal Transformer with a latent dimension of 2560 (7040 for the SiLU gating), 24 layers, 20 heads and local attention over 1500 tokens, i.e. 2.2B parameters and 120s context. The Depth Transformer initially follows Défossez et al. (2024), i.e. 6 layers per codebook, latent dimension of 1024 (2816 for the gating), and 16 heads. It models Q = 16 audio codebooks for the output stream, and the same for the input stream (only at training). To reduce the footprint of the Depth Transformer we distill it post-training into smaller one, with 4 layers per codebook, weight sharing for codebooks 9 to 16. low-rank embedding tables of dimension 128, and dimension of 2048 for the gating. This reduces its size from 1.IB parameters to 449M parameters, for total of 2.7B parameters. We also train Hibiki-M as a distilled version of Hibiki, a 1.8B variant capable of running in real-time on de- vice, using a Temporal Transformer with latent dimension of 2048, 16 layers, and only 8 codebooks levels per stream. Architectural hyper-parametert parameters summarized in Table 6.
MODEL
ASR-BLEU
TRANSLATOTRON (JIA ET AL., 2019)
17.0
TRANSLATOTRON (JIA ET AL.. 2022A)
26.0
S2UT (LEE ET AL., 2022A)
22.2
UNITY (INAGUMA ET AL.. 2023)
27.8
DASPEECH (FANG ET AL.., 2023)
25.0
RNN-TRANSDICERR (ZHAO ET AL.,, 2024)
25.4
STREAMSPEECH INF) (ZHANG ET AI.,, 2024A)
28.5
HIBIKI
30.5
2023) to translate them individually, before joining them back into a translated English transcript. We synthesize each with the TTS described in Section 3.2.2, conditioned on the original French speaker identity with a 10s utterance. We apply the silence insertion technique described in Sec- tion 3.2.2 obtain simultaneous interpretation pairs. We train for 150K steps with batch size of 96, learning rate of 3 10 and compute the loss on both the source and the target streams. We use conditional training on speaker similarity as explained in 3.3 and apply noise augmentation techniques on the source audio. For each training pair, we introduce a special input EOS token on all audio tokens of the source at the first frame after the end of its speech utter- ance and use another special EOS token on the text tokens stream to indicate the end of the model speech utterance.
4.2. Training protocol
We train French-English speech translation system through the following steps, each with cosine learning rate sched- ule and AdamW (Loshchilov & Hutter, 2019), with a weight decay of 0.1, and momentum parameters of (0.9, 0.95).
Text pretraining. We first pretrain the Temporal Trans- former from scratch on multilingual text-only data using next token prediction, for 600K steps, with batch of 1,024 sequences of length 4,096. We use cosine learning rate schedule, with 2K warmup steps and a maximum value of 4.8 10-1 Our training dataset is made of filtered web pages from Common Crawl, as well as curated sources such as Wikipedia, StackExchange or scientific articles and it contains 12.5% of multilingual documents.
Speech translation fine-tuning, We use alignment-awee awwae TTS generations introduced in Section 3.2.2 to build a syn- thetic dataset composed of long-form utterances and an improved version of CVSS-T/train with natural pauses and high speaker similarity, totaling close to 900 hours. We fine-tune for 8K steps with batch size of 8, a learning rate of 2 . 10-5, conditional training on the speaker similarity, special EOS tokens, and apply the loss to both streams.
Audio prefraining. Starting from the pretrained text model, perform audio pretraining on non-parallel French and English data with a single stream as done by Défossez et al. (2024). We train for 1,450K steps with batch size of 1 144 and a learning rate 2 2 10-4, Then we duplicate the weights of the Depth Transformer for multistream modeling.
Training of Hibiki-M. It goes through the same text and audio pre-training stages. During the speech translation training is soft distilled from Hibiki, before going through the same fine-tuning stage without distillation.
4.3. Evaluation metrics and baselines
Speech translation training. We build a French-Englist speech translation dataset of approximately 40K hours in each language. Starting from collection of expressive audio content in French, we extract roughly 2.3M single speaker utterances each with a duration around 60 sec- onds. We transcribe these segments with Whisper (Radford et al., 2023), using the large-v3 model. We rely on PySBD (Sadvilkar& & Neumann, 2020) to segment each transcript into sentences and use MADLAD-3B (Kudugunta et al.,
We first compare Hibiki to several offline baselines as well as a closed source streaming baseline (Zhao al., 2024). We then perform comparison between Hibiki and the two existing methods for simultaneous translation: Seam- less (Barrault et al., 2023) and StreamSpeech (Zhang et al., 2024a) with a chunk size of 2560ms. We evaluate transla- tion and audio quality, naturalness, speaker similarity and translation latency using automatic and human evaluations.
6

---

High-Fidelit Simultancous Speech-To-Speech Translation
Table 2. Objective comparison of Hibiki with StreamSpeech (Zhang et al., 2024a) and Seamless (Barrault al., 2023).
SHORT-PORM (CVSS-C FR-EN FR-EN TEST)
LONG-FORME -TREXX)
ASR
ASR (OMET
SPRAKER
END
ASR BLEU
ASR
SPEAKIR
END
MODEL
BLEU
BLEU
SIM.
OPSET
LAAL
(LEU
COMET
Soa.
OPEEET
LAAL()
MADLAD-38
94.2
72.6
STREAMSPLECH
26.4
25.4
67.0
1.6
2.8
0.1
0.1
18.4
NIA
N/A
SEAMLRSS
77.6
0.30
1.4
25.4
23.9
27.3
0.43
4.2
HIBIKI-M
37.5
33.7
75.7
0.34
2.8
3.5
25.9
25.0
30.5
0.39
2.3
5.5
HIBIKI
39.2
35.5
78.3
0.41
2,9
3.4
27.5
26.6
34,9
0.48
2.7
5.0
Translation quality. We evaluate translation quality by transcribing generated speech and computing BLEU (Post, 2018) and COMET (Rei et al., 2020) scores with the refer- ence, referred to as ASR-BLEU and ASR-COMET. When comparing offline baselines in Table we replicate the setting of Zhang et al. (2024a) for fair comparison. As observed frequent ASR mistakes, we rather used Whisper medium (Radford et al., 2023) in subsequent experiments and compute BLEU/ASR-BLER scores after hypothesis and ground-truth text normalization.' 3 Since Seamless, Stream- Speech and Hibiki also produce a text translation, we also evaluate their BI.EU score using the same text normalization. ASR-COMET scores are calculated without text normaliza- tion and using the XCOMET-XL model.4
speech. We define c maa (Bret) where Asource is the duration of the source speech and Nre the number of words in the reference translation. The LAAL score is then computed as Tmaz minfild; 2 A source
Amerce
Ommax >i di - (i - 1)c where
4.4. Evaluation datasets
Short-form data. We evaluate models on the Fr-En task of CVSS (Jia et al., 2022b). While it is the standard benchmark for S2ST and allows comparisons with previous models, we observe that 99% of its sequences are shorter than 10 seconds. We thus extend our evaluation to long-forms,
Long-form data. We collect long-form speech translations by recording bilingual speakers as they read Fr-En trans- lations from the NTREX (Aepli et al., 2023) text corpus. This speech corpus, that we name Audio-NTREX contains 10 hours of real human speech in each language with 10 different speakers and an average of 50 sec. per utterance.
Audio quality and naturalness. Human raters evaluate the audio quality of generated speech and its naturalness. We evaluate the latter as a proxy for "realism": are the flow and prosody natural, are pauses smooth and properly placed or are there abrupt cuts? We compute each score for each model by averaging Mean Opinion Scores between and 5 across 30 samples, each sample being evaluated by 15 raters.
Real interpretation. To compare with human interpreters, use 90 real interpretations of the European Parliament from VoxPopuli (Wang et al., 202 la) where translations con- tain the source speech at a lower volume. For a fair compari- son, we also add the lowered source speech to generations of Hibiki and Seamless (see our external webpage for samples).
Cross-lingual speaker similarity. For objective evalu- ation, we use a standard model for speaker verification based on WavLM (Chen et al., 2022) and report the cosine similarity between the embeddings of the source and the generated speech. To mitigate potential biases due to using the same speaker verification model for conditional training (see Section 3.3), we also collect human judgments where raters are asked to rate the similarity to the source audio.
4.5. Inference configuration
We encode audio with the streaming codec and feed the tokens to Hibiki while decoding the output tokens to obtain streaming translation. At the end of the input, we send the EOS token to our model, and keep sampling until it produces its own EOS. Inference parameters are cross vali- dated independently for each dataset using held-out 8% of Audio-NTREX and the valid split of CVSS-C. The optimal parameters are = 3.0, a temperature of 0.8, top-k of 250 for audio tokens and 50 for text tokens for Audio-NTREE. On CVSS, the same configuration is used except for text tokens that are sampled with a temperature of 0. We con- jecture that the lower text temperature typically improves translation but can lead to producing an EOS token too early.
Latency. A metric for S2ST latency is the End Offset, which is the time (in seconds) between the end of the last word of the source and that of the last word in the Average
output. We also measure the Sength-AAaptaa Lagging (LAAL) following the method described by Papi et al. (2022): it approximates the average time (in seconds) between the pronunciation of a source word and its translation, without requiring word-level alignments. We rely on word-level emission timestamps (d+))....gee produced by Whisper for Rggen words in the generated
hethhllocwageeeniiiiieehhhnnnnnn github. gehhab.commmcrrood//USSppchh M" Large")

---

High-Fidelit Simultancous Speech-To-Speech Translation
Hibll Scorniess
Table Human evaluation. Raters report Mean Opinion Scores (MOS) between I and 5. Ground Ground-truth is real human interpretation.
MODEL
QUALITY
SPEAKER SIM.
NATURALNESS
GROUND-TRUTH
4.18 +: 0.07
4.12 = 0.08
SEAMLESS
2.22 1: 0.08
2,86 1 0.12
2.18 0.09
3.78 1:
3.43
3.73
LAAL
Figure 6. Quality/Latencc trade-off of multiple Hibiki models vs. Seamless with different decision thresholds Audio-NTREX.
4.6. Results
Translation quality. Table compares Hibiki with of- fline baselines that have access to the complete source audio when translating. Despite performing simultaneous trans- lation, Hibiki outperforms all models, including the offline variant of StreamSpeech. Table 2 benchmarks Hibiki against available baselines for simultaneous translation. In the short- form setting, our model outperforms StreamSpeech and Seamless at the cost of an average 0.7s of additional lag- ging. The long-form dataset represents a more significant challenge, as StreamSpeech does not manage to produce intelligible translations. Hibiki outperforms Seamless, again with latency higher by an average of 0.8s.
Table 4. Ablations.
MODEL
ASR-BLEU
LAAL
No LAG
4.2
2.46
CONSTANT LAG (25)
10.0
2.49
CONSTANT LAG (10S)
22.5
9.02
SENTENCE ALIGNMENT
25.6
21.49
No OGIUE
17.1
14.34
No AUDIO PRETRAINING
14.6
5.12
HIBIKI
26.6
5.0
Audio fidelity. Objective evaluations for speaker similar- ity, as reported in Table 2, show that Hibiki demonstrates significantly better voice transfer than Seamless (we do not evaluate StreamSpeech as it does not perform voice trans- fer). Human evaluations reported in Table 3 confirm this result and furthermore show much higher quality and natu- ralness than Seamless, that get close to that of ground-truth audio from professional human interpreters. This means that Hibiki not only produces high-quality audio, but it inserts smooth and natural pauses into its flow.
in very low translation quality, which can be expected since the model lacks context to produce the translation. Adding lag in training examples improves the ASR-BLEU, with 10 seconds representing reasonable value, however the resulting average latency (as represented by LAAL) is much worse than using a contextual alignment, as the model does not adapt its flow to the context. A middle ground between constant lag and contextual alignment that of "sentence alignment" that simply moves the start of each output sentence the end of the corresponding source sentence. This improves translation quality, however degrading the latency even more. Overall, contextual alignment provides the best trade-off between translation quality and latency.
Quality/Latency trade-off. The quality/latency trade-off of Hibiki cannot be controlled at inference as its transla- tion policy is conditioned by the data preparation process described in Section 3.2.2. However, by varying the lag getx added between contextually aligned words, one can set the quality/latency trade-off at training time. We compared different versions of Hibiki trained with 5exx of 0.5, 1, 1.5 and 3 sec. against Seamless parameterized with different decision threshold values from 0.3 to 0.7. Figure shows that Hibiki achieves better trade-off than Seamless and allows to design more steerable systems.
Ablation: Classifier-free guidance. Table shows that using the very-good label provides a speaker similarity of 0.42, similar to that of Seamless (0.43). Using classifier- free guidance with 7 = 3.0 significantly improves it without significantly hurting translation quality, while increasing its weight too much results in degraded performance due to unintelligible speech. Supplementary material interestingly illustrates how increasing ? to extreme values results in an exaggerated French accent (the source language in our experiments), which we can attribute to biases the speaker model used label our data.
Preliminary English-to-Frenn results. Using the si- lence insertion technique, we also built English-to-Frende speech translation dataset and trained another Hibiki model whose objective evaluation results are given in Table 7.
Ablation: General ablations. Section 3.1.3 describes how jointly predicting text tokens serves as a scaffolding for audio generation. Table 4 illustrates this claim as training Hibiki in a unimodal fashion, without predicting text outputs, results in much worse performance, as does starting from a pretrained text LM and training directly for S2ST.
Ablation: alignment strategies. We compare our contex- tual alignment method with alternatives. Table 4 shows that applying no lag to the target speech during training results
8

---

High- Simultancous Speech-To-Speech Translation
Table 5. Ablations on classifier-free guidance.
boz
bwr
Real-time factor
CFG PARAM.
ASR-BLEU
ASR BLEU
SPEAKER SIM.
No CFG
26.0
0.42
== 3.0 (DEFAULT)
26.6
0.48
2 10.0
18.9
0.44
Time (5)
Real Real-Time Factor (X)
Figure 8. Inference speed of Hibiki-M on an iPhone 16 Pro.
5. Conclusion
We introduce Hibiki, a model for simultaneous and expressive speech and text translation. Hibiki leverages a multistream architecture that casts live translation as simple temperature sampling, thanks to a weakly-supervised method for aligning paired translation data. Hibiki is competitive with the state-of-the-art in terms of translation quality, while demonstrating a much more natural flow close to human interpretation well as a better voice transfer. Moreover, Hibiki is compatible with streaming batched inference, which facilitates large-scale deployment, while the smaller Hibiki-M runs in real-time on smartphone.
Batch Size
Figure 7. Batched inference speed of Hibiki on a H100 SXM.
4.6.1. INFERENCE CAPABILITIES
Batched inference. Hibiki's inference uses temperature sampling with a constant framerate. This makes it easy to perform streaming classifier-free guidance as well as batching the processing of several sources of speech at the same time. This is unlike Seamless and StreamSpeech whose complex inference policies are difficult to batch they require dynamic, irregular decisions for each sequence. Figure shows that Hibiki remains faster than real-time a H100 even when processing 320 sequences in parallel (or 160 with classifier-free guidance).
Impact Statement
Our work aims at bringing simultancous speech translation closer to real-world applications, both by improving its qual- ity and by making it deployable on-device. This carries po- tential for improving communication between humans. We acknowledge that speech translation may affect employment opportunities for interpreters in contexts such interna- tional meetings or live interviews, however we believe that it will will have net positive impact in all contexts where an interpreter would not be considered, e.g. when traveling or streaming videos. We also acknowledge risks associated to voice transfer, while we remark that S2ST with voice trans- fer represents a lower risk of spoofing than TTS as the output textual content is not directly controllable and the model is only given a limited freedom to reformulate its inputs.
On-device inference. Our distilled Hibiki-M competitive with Seamless on short-form and long-form translation as shown in Table 2. We attribute the lower speaker similarity on long-form audio to the lower number of quantizers modeled by Hibiki-M (8 instead of 16) which results in a twice lower audio bitrate. Figure 8 shows inference traces of Hibiki-M on an iPhone 16 Pro. Hibiki-M remains faster than real-time along a minute of inference, even with batch size of which necessary for classifier-free guidance. Training Hibiki-M with sliding window attention would furthermore improve real-timo factor along time.
4.6.2. LIMITATIONS
References
This study focuses on a single translation task (French to English) and scaling to more languages could benefit from MADLAD which massively multilingual, however it would require training TTS systems on more languages. Moreover, while Hibiki reaches 35.5 ASR-BLEU against CVSS-C ground- truth targets, it reaches 47.9 ASR-BLEU if compared to MADLAD text translations instead. This shows that Hibiki is excellent at predicting translations that could be produced by MADLAD, and training it to predict ascudo-targess from better or more diverse translations can improve translation quality w.r.t ground-trutt -ntuh targets.
Aepli, N., Amrhein, C., Schottmann, F., and Sennrich, R. A benchmark for evaluating machine translation metries on dialects without standard orthography. In Koehn, P., Haddon, B., Kocmi, T., and Monz, C. (eds.), Proceedings of the Eighth Conference on Machine Translation, WMT 2023, pp. 1045-1065. Association for Computational Linguistics, 2023. doi: 10.18653/V 1/2023.222222222 WMT-1.99
Ahmad, I. S., Anastasopoulos, A., Bojar, O., Borg, C., Carpuat, M., Cattoni, R., Cettolo, M., Chen, W., Dong, Q., Federico, M., Haddow, B., Javorsky, D., Krubin-

---

High-Fidelit Simultancous Speech-To-Speech Translation
ski, M., Lam, T. K., Ma, X., Mathur, P., Matusov, E., Maurya, C., McCrae, J. P., Murray, K., Nakamura, S., Negri, M., Nichues, J., Niu, X., Ojha, A. K., Ortega, J. E., Papi, S., Polák, P., Pospisil, A., Pecina, P., Salesky, E., Sethiya, N., Sarkar, B., Shi, J., Sikasote, C., Sper- ber, M., Stüker, S., Sudoh, K., Thompson, B., Turchi, M., Waibel, A., Watanabe, S., Wilken, P., Zemánek, P., and Zevallos, R. Findings of the IWSLT 2024 evalu- ation campaign. CoRR, abs/241 1. L05088, 2024. doi: 10. (0-4850/ARXXV:2411000088 URL https: //doi. org/10. 205.4/4550//xxx... 2411 05088.
Défossez, A., Mazaré, L.., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E.,, and Zeghidour, N. Moshi: a speech-text foundation model for real-time dialogue. CoRR, abs/2410.00037,, 2024. doi: 10.48550/ARXIV. 2410.00037.
Dyer, C., Chahuneau, V., and Smith, N. A. A simple, fast, and effective reparameterization of IBM model 2. In Human Language Technologies: Conference of the North American Chapter of the Association of Computa- tional Linguistics, Proceedings, June 9-14, 2013, pp. 644- 648, 2013. URL https://aclant http///ccaaattttoooo/oooo.o orq/ N13 1113-1073/
Barrault, L., Chung. Y., Meglioli, M. C.,, Dale, D., Dong. N., Duppenthaler, M., Duquenne, P., Ellis, B., Elsahar, H., Haaheim, J., Hoffmann J., Hwang, M., Inaguma, H., Klaiber, C., Kulikov, I., Li, P., Licht, D., Maillard, J., Mavlyutov, R., Rakotoarison, A., Sadagopan, K. R., Ra- makrishnan, A., Tran, T., Wenzek, G., Yang, Y., Ye, E., Evtimov, I., Fernandez, P., Gao, C., Hansanti, P., Kalbassi, E., Kallet, A., Kozhevnikov, A., Gonzalez, G. M., Roman, R. S., Touret, C., Wong. C., Wood, C., Yu, B., Andrews, P., Balioglu, C., Chen, P., Costa-jusså, M. R., Elbayad, M., Gong, H., Guzmán, F., Heffernan, K., Jain, S., Kao, J., Lee, A., Ma, X., Mourachko, A., Peloquin, B. N., Pino, J., Popuri, S., Ropers, C., Saleem, S., Schwenk, H., Sun, A. Y., Tomasello, P., Wang, C., Wang. J., Wang. S., and Williamson, M. Seamless: Multilingual expressive and streaming speech translation. CoRR, abs/2312.05187, 2023. doi: 1044850//ARXIV2222205555
Fang. Q.. Zhou, Y., and Feng. Y. Daspeech: Directed acyelic transformer for fast and high-quality speech-to-speed translation. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, 2023.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint /rXiv: (6060044655 2016.
Inaguma, H., Popuri, S., Kulikov, I., Chen, P.-J., Wang, C., Chung, Y.-A., Tang, Y., Lee, A., Watanabe, S.,, and transla- Pino, J. UnitY: Two-pass direct speech-to- tion with discrete units. In Rogers, A., Boyd--ooabaa J., and Okazaki, N. (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 35655-15680 Toronto, Canada, July 2023. Association for Computational Lin- guistics. doi: 10.18653/v http:/33//112222alllllo
Berard, A.,
Pietquin, O., Servan, C., and Besacier, L. Listen and translate: A proof of concept for end-to-end speech- to-text translation. CoRR, abs/1 612.0144,,, 2016.
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grang- ier. D., Tagliasacchi, M., and Zeghidour, N. Audi- olm: A language modeling approach to audio genera- tion. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.
Jain, A. N., McNair, A. E.,, Waibel, A., Saito, H., Haupt- mann, A., and Tebelskis, J. Connectionist and symbolic processing in speech-to- to-ehe translation: The JANUS system. In Proceedings of Machine Translation Summit III: Papers, pp. 113-117, Washington DC, USA, July 1-4 1991.
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Signal Process., 2022.
Jia, Y., Weiss, R. J., Biadsy, F., Macherey, W., Johnson, M., Chen, 7., and Wu, Y. Direct Speech-to-Speede Translation with a Sequence-to-Sequuee Proc. Interspeech 2019, pp. 1123-1127, 2019. doi: 10114477nniexpecc.001111111
Model. In
Jia, Y., Ramanovich, M. T., Remez, T., and Pomerantz, R. Translatotron 2: High-quality direct speech-to-speed translation with voice preservation. In Chaudhuri, K., Jegelka, S., Song, I.., Szepesvari, C., Niu, G., and Sabato, S. (eds.). Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10120-10134 PMLR, 17-23 Jul 2022a.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrlPS 2023, 2023.
10

---

High- Simultancous Speech-To-Specch Translation
Jia, Y., Ramanovich, M. T., Wang, Q. and Zen, H. CVSS corpus and massively multilingual speech-to-speech trans- lation. In Calzolari, N., Béchet, F,, Blache, P., Choukri, K., Cieri, C., Declerek, T., Goggi, S., Isahara, H., Mae- gaard, B., Mariani, J., Mazo, H., Odijk, J., and Piperidis, S. (eds.), Proceedings of the Thirteenth Language Re- sources and Evaluation Conference, LREC 2022, pp. 6691-6703. European Language Resources Association, 2022b.
Loshchilov, and Hutter, F. Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Louradour, J. whisper-tinesstamped https://github. com/Linto-ai/wwiieenon witwwiswwwaieee tttppeerrttmmmttmpeppeee 2023.
Ma, X., Wang, Y., Dousti, M. J., Koehn, P., and Pino, J. M. Streaming simultaneous speech translation with augmented memory transformer. In IEEE International Conference on Acoustics, Speech and Signal Process- ing. ICASSP 2021, pp. 7523-7527. IEEE, 2021. doi: 10. D0.10090CASSPP997822222202 9414997.
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. CTRL: conditional transformer language model for controllable generation. CORR, abs/1909.05858, 2019.
Nakamura, S., Markov, K., Nakaiwa, H., Kikui, G., Kawai, H., Jitsuhiro, T., Zhang, J.-S., Yamamoto, H., Sumita, E., and Yamamoto, S. The ATR multilingual speech-to- speech translation system. IEEE Transactions on Audio, Speech, and Language Processing, 2006.
Korbak, T. Shi, K., Chen, A., Bhalerao, R. V., Buckley, C. Phang. J., Bowman, S. R., and Perez, E. Pretraining language models with human preferences. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, volume 202 of Proceedings of Machine Learning Research, pp. 17506- 17533. PMLR, 2023.
Papi, S., Gaido, M., Negri, M., and Turchi, M. Over- generation cannot be rewarded: Length-adaphnn aver- age lagging for simultaneous speech translation. CORR, bbss/200.008807, 2022. doi: 10.48550/ARXIV.22066 05807.
Kreuk, F,, Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audio- gen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, ICLR 2023. Oee 2023.
Post, M. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Transla- tion: Research Papers, pp. 186-191, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18.
Kudugunta, S., Caswell, I., Zhang, B., Garcia, X., Xin, D., Kusupati, A., Stella, R., Bapna, A., and Firat, O. MADLAD-400: A multilingual and document-level large audited dataset. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 28492-285188 PMIR, 2023.
Razavi, A., van den Oord, A., and Vinyals, O. Generating di- verse high-fidelity images with vq-vae-2. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Lee, A., Chen, P.-J., Wang. C., Gu, J., Popuri, S., Ma, X., Polyak, A., Adi, Y., He, Q. Tang, Y., Pino, J., and Hsu, W.-N. Direct speech-to- -p-eech translation with dis- crete units. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Papers), pp. 3327-3339, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10.18653/v 12tp:://1../22.......og.g......
Rei, R., Stewart, C., Farinha, A. C., and Lavie, A. COMET: A neural framework for MT evaluation. In Proceed- ings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, pp. 2685- 2702, 2020. doi: 10.1663/1122200EMMPPPMM
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W. Autore- gressive image generation using residual quantization. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 11513-11522. IEEE, 2022b. doi: 10. 241100/CVPR52268 2022.01122
213. URL https: UtL.httpsa//aad..oo////... /ttp://......111..1133//////////
2020 . emnlemmppm.mmm semnlp-mainn.. main. 213.
Ren, Y., Liu, J., Tan, X., Zhang, C., Qin, T., Zhao, Z., and Liu, T.-Y. SimulSpeech: End-to-end simultane- ous speech to text translation. In Jurafsky, D., Chai,

---

High-Fidelity Simultancous Speech-To-Speech Translation
J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 3787-3796, Online, July 2020. Association for Computational Linguistics. doi: 10.
Wang, M., Vu, T., Shareghi, E., and Haffari, G. Con- versational simulmt: Efficient simultaneous translation with large language models. CoRR, 10s/2402.105522 2024. doi: 01t.:/5//AAAXX22220000055555 URL https: / oag.i.org/000 4 48550 / aariiriiivvv.. 2402. 10552.
Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry, F., Chen, P., Badawy, D. E.,, Han, W., Kharitonov, E., Muckenhim, H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T. N., Schalkwyk, J., Sharifi, M., Ramanovich, M. T., Tagliasac- chi, M., Tudor, A., Velimirovic, M., Vincent, D., Yu, J., Wang, Y., Zayats, V., Zeghidour, N., Zhang, Y., Zhang. Z., Zilka, I.., and Frank, C. H. Audiopalm: A large language model that can speak and listen. CoRR, abs/2306. 12925, 2023. doi:
Weiss, R. J., Chorowski, J., Jaitly, N., Wu, Y., and Chen, Z. Sequence-to-sequenee
models can directly translate foreign speech. In Lacerda, F. (ed.), 18th Annual Confer- ence of the International Speech Communicatio Associa- tion, Interspeech 2017, pp. 2625-2629, ISCA, 2017. doi: 1021487/NTERSSPICHH22010
Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., Shi, J., Zhao, S., Bian, J., Wu, X., et al. Uniaudio: An au- dio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023.
Sadvilkar, N. and Neumann, M. Pysbd: Pragmatic sentence boundary disambiguation. CoRR, abs/2010.09657, 2020.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process.. 00:4995-507, 2022. doi: 10.1.0. 3129994.
Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., and and, L. K. Attention is all you need. In Advances in Neural Information P Processing Systems (NeurIPS), pp. 5998-599999 00088600822220 2017.
Zeng, X., Li, L., and Liu, Q. Realtrans: End-to-end simulta- neous speech translation with convolutional weighted- shrinking transformer. In Zong, C., Xia. F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, volume ACL/JCNLP 2021 of Findings of ACL, pp. 2461-2474. Association for Computational Linguistics, 2021. doi: 10. 10.18653/V11 1/2021.FINDIINGS.
Wahlster, W. Verbmobil: Foundations of speech-to-speech translation. Springer, 2000.
Wang, C., Tang, Y., Ma, X., Wu, A., Okhonko, D., and Pino, J. Fairseq S2T: Fast speech-to-text modeling with fairseq. In Wong, D. and Kiela, D. (eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Sys- tem Demonstrations, pp. 33-39, Suzhou, China, Decem- ber 2020. Association for Computational Linguistics. doi: 10. N0.18653/v1/2020 (665:///2000..ac/eeemoo...
Zhang, S., Fang, Q., Guo, S., Ma. Z., Zhang, M., and Feng, Y. Streamspeech: Simultaneous speech-to-speech translation with multi-task learning. In Ku, L., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume I: Long Papers), ACL 2024. Bangkok, Thailand, August 11-16, 2024, pp. 8986. Association for Computational Linguistics, 2024a. doi:
Wang, C., Rivière, M., Lee, A., Wu, A., Talnikar, C., Haz- iza, D., Williamson, M., Pino, J. M., and Dupoux, E. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi- supeviuper learning and in- terpretation. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Pro- cessing, ACL/IJCNPP 2021, (Volume 1: Long Papers), pp. 993-1003. Association for Computational Linguistics, 2021a. doi: 10. N0l.18653//111 1008//1/2000 1/0.1//21.1.LLLNG...
10.
Zhang, X., Zhang, D., Li, S., Zhou, Y., and Qiu, X. Speech- tokenizer: Unified speech tokenizer for speech language models. In The Twelfth International Conference on Learning Representations, 2024b.
Zhao, J., Moritz, N., Lakomkin, E., Xie, R., Xiu, Z., Zmolikova, K., Ahmed, Z., Gaur, Y., Le, D., and Fue- gen, C. Textless streaming speech-to-spest tion using semantic speech tokens, 2024. URL https: / aaaiiv org abs 22100.4100. 03298.
transla-
Wang, C., Wu, A., Gu, J., and Pino, J. Covost 2 and mas- sively multilingual speech translation. In Interspeech 2021, pp. 2247-2251, 2021b. doi: 10221437/mterppecch. 2021-2027.
12

---

High- Simultancous Speech-To-Speech Translation
A. Appendix
Sample 65dr0c4430624445- 21-542 549
240805-111810 21-5444440805-1112
Sample b6d2383a4006breel 2202--2222-222222---11111
tinesttamp of imput index af
tivestamp of input word Indox ajt
20
20
10-
10
20
40
50
80
100
120
140
160
20
60
80
100
120
140
Sample 3013000msaartb-114443-----222220
50
timestamp of inout ward Indox a/
timestamp inpuu word
30-
20-
10-
10-
20
40
80
100
60
80
100
120
output word incex
word Index
Figure 9. Example of contextual alignments. We compute the contextual alignment (a²") for four different samples and plot the associated input timestapps. Some results as the two the bottom present extreme spikes interpreted as output words referring input words very far in the future. These spikes are considered as anomalies and are smoothed out as explained in Section 3.2.2.
13

---

High- Simultancous Speeeh-To-Speech Translation
Table 6. Architectural hyper-parameters of Hibiki and Hibiki-M.
HHBIKI
IIBIKI-M
Temporal Transformer
MODEL DIMENSION
2560
2048
GATING DIMENSION
7040
5632
NUMBER OF HEADS
NUMBER OF LAYERS
CONTEXT SIZE
1500
Depth Transformer
MODEL DIMENSION
1024
DIMENSION
2048
2816
LOW-RANK EMBEDDINGS DIMENSION
NUMBER OF HEADS
16
16
NUMBER OF LAYERS
6
Imput Output space
TEXT CARDINALITY
48000
AUDIO CARDINALITY PER CODEBOOK
2048
AUDIO CODEBOOKS PER STREAM
16
8
CODEBOOKS WEIGHT SHARING
9 TO 16
FRAME RATE
12.5 HZ
TOTAL NUMBER OF PARAMETERS
2.7
1.8
Table Objective comparison of Hibiki with Seamless (Barrault al., 2023) for Einllishtoo-French translation task.
SHORT-FORM (FLEURS Ex-FR VAliD)
LONG-PORM (AUDIO-NTREX)
MODEL
BLEU (1)
ASR BLEU (t)
END OFPSET (L)
LAAL (1)
BLEU (1)
ASR BLEU (f)
END OFFSET (L)
LAAL (L)
SEAMLESS
39.5
36.6
2.0
2.9
28.4
23.2
2.5
5.5
HIBIKI
40.0
37.1
3.9
3.8
33.9
29.6
4.6
6.7
14


Reviews:
[{"review_id": "uoIiA4OZ7j", "reviewer": "Reviewer_AWWz", "summary": "This paper proposes a state-of-the-art speech-to-speech translation system called Hibiki. This is a chunk-based decoder-only model based on Mimi codec, and a number of techniques (alignment-related, synthetic data creation, classifier-free guidance, etc.) are introduced to achieve state-of-the-art performance in several public benchmarks. The method seems to be only applied to En-Fr.\n\n## update after rebuttal\nI acknowledge the authors' efforts in presenting the trade-off curve and providing additional English-to-French translation results, and I have accordingly raised my score. I also encourage the authors to follow through on their stated commitments, including open-sourcing their code.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback.\n\n# Updates in the revised version of the paper\nWe first inform the reviewer that we will update the reported results of Hibiki-M in [Table 2](https://hibiki-s2st.github.io/e.png) after fixing an issue that further improves performance.\nFollowing the reviewers' suggestions and thanks to the extra page provided for the camera ready, we will revise the paper on the following aspects:\n* Clarify the **model's architecture, configurations and the nature/size of the datasets used.**\n* Add **quality/latency trade-off curves**  by varying hyperparameters of our contextual alignment method.\n* Add **COMET** evaluation scores.\n* Extend experiments to the **English->French** direction.\n* Improve the **references section** and discuss similar works pointed out by the reviewers.\n\n# Comments\n\n## On the generalization of the contextual alignment method to many language pairs\n\nWe acknowledge that our method is only illustrated by a single language direction in the paper. However, there is no specific aspect of our method with respect to the language pair, but the fact that MADLAD -- which we used to derive contextual alignment -- performs well on these languages. We expect our method to work as strongly on pairs of languages where SOTA text translation models perform well and can thus allow us to derive reliable contextual alignments. Given the massively multilingual nature of MADLAD or even more recent systems like GemmaX2-28-9B ([Cui et al., 2025](https://arxiv.org/abs/2502.02481)), we expect this approach to be a good candidate for scaling to many language pairs. We provide the reviewers examples of [contextual alignments with other languages](https://hibiki-s2st.github.io/d.png). As a first step towards more language directions, we have extended our experiments to the English->French direction and provide [experimental results](https://hibiki-s2st.github.io/b.png) that we will add to the revised paper.\n\n## On the quality/latency trade-off\n\nAs mentioned in Section 3.2.2 (l.183), we enforce a 2s delay between words associated through contextual alignment as we found it to provide a good balance between latency and translation quality. We acknowledge that this choice can be reconsidered and that trade-off curves would provide a clearer picture to the reader. **We thus produced a trade-off curve by varying the delay, as asked by the reviewers.** The results reported in [this quality/latency study](https://hibiki-s2st.github.io/c.png) show that Hibiki provides an overall better trade-off than Seamless. We will add this figure to the revised version of the paper.\n\nWe also acknowledge that the proposed version of Hibiki does not allow for inference-time latency control. We could rely on conditional training, as we did for the speaker similarity, to simultaneously train the model on multiple latency levels making it possible to control the latency at inference by changing the conditioning. We will add this mention to the limitations section.\n\n## On the release of code and data\n\nWe acknowledge that some critical parts of our framework are particularly challenging to reproduce, in particular the synthetic data generation using contextual alignment and **we will release our code for these steps along with training and inference code, trained models and around 900h of synthetic paired data with voice preservation** corresponding to our speech translation fine-tuning dataset introduced in Section 4.2. In order to build the speech translation training dataset, we relied on various data sources and will release the portion that we are allowed to release by their license.\n\n## On references\n\nWe thank the reviewers for their suggestion and acknowledge the contributions made by [Papi et al. (2023)](https://arxiv.org/abs/2305.11408), [Wang et al. (2024)](https://arxiv.org/abs/2402.10552) and [Yu et al. (2025)](https://arxiv.org/abs/2502.00634) that are particularly relevant with respect to our work. We also acknowledge the progress made in streaming and speech translation for complex language pairs such as English-Japanese as highlighted in [Ahmad et al. (2024)](https://arxiv.org/abs/2411.05088). We will add these references to the related work in the updated version of our paper.\n\n## Answer to: *Section 3: \"X is padded\" - What happens when X is longer than Y ? Is Y  padded in that case?*\n\nAt this level of explanation (l.106), we also assume that *the modeling of Y knowing X should be causal*. This implies that Y is longer than X.", "claims_and_evidence": "This paper proposes a number of techniques\n- target text scaffolding\n- contextual alignment based on an off-the-shelf MT system and its application to target text/audio alignment\n- Alignment-aware TTS generation\n- Speaker similarity improvement in TTS generation\n- Conditional training with classifier-free guidance\n\nThe effectiveness of these techniques is validated experimentally (e.g., Albations in Tables 4/5)", "methods_and_evaluation_criteria": "The paper presents four evaluation metrics: fidelity, measured through subjective MOS scores (Table 3), speaker similarity, ASR-BLUE, and latency measures such as LAAL. Given that this study focuses on simultaneous speech translation, it is crucial to analyze the performance-latency tradeoff, particularly through ASR-BLUE and LAAL. However, Table 2 provides only a single condition, making it difficult to thoroughly examine this tradeoff.\n\nI suggest that the authors illustrate trade-off curves by varying latency control parameters and compare their method against competitors, discussing the advantages and limitations of each approach.", "theoretical_claims": "This paper does not have theoretical claims.", "experimental_designs_or_analyses": "As mentioned earlier, the paper should focus more on the performance-latency tradeoff rather than drawing conclusions based on a specific latency setup. For example, Section 4.6 states that Hibiki outperforms Seamless, but a 1.4-second latency is quite large for a speech interface. If generating trade-off curves or testing various latency conditions is not feasible, the authors should at least soften their claims to account for this limitation.", "relation_to_broader_scientific_literature": "SImultaneous speech-to-speech translation is one of the most important human language technologies to remove language barriers in the world.", "essential_references_not_discussed": "The paper sufficiently cites related work. However, I would like to highlight that advancements in simultaneous speech-to-speech translation have not been driven solely by major industries but also by contributions from various researchers in the IWSLT community. I recommend that the authors acknowledge these efforts by citing relevant IWSLT summary papers.", "other_strengths_and_weaknesses": "Strengths\n- Achieves state-of-the-art performance in simultaneous speech-to-speech translation. The results in Table 1 are impressive, as the proposed approach outperforms the offline system despite operating in a streaming setting.\n- Proposes several techniques to enhance performance, with their effectiveness validated through an ablation study.\n\nWeaknesses\n- The training procedure is complex, making it difficult to reproduce the results. While Section 1 states that the authors will release the code, models, and dataset, it is unclear whether the release will include the full dataset creation process and detailed training configurations. I recommend the authors clarify this.\n- The performance-latency tradeoff between Seamless and the proposed method is not clearly analyzed (see my comments above).\n- The alignment methods appear to be tailored to a specific language pair, raising concerns about their applicability to other language pairs.", "other_comments_or_suggestions": "- Section 3.1.1 requires some prior knowledge of the Mimi codec but is well written.\n- Section 3.1.4 is difficult to understand. As this section presents the main proposed architecture, it requires more detailed explanations, such as equations or figures, to enhance clarity.\n- Section 4.6 presents strong results, but it would be more informative if the authors included details on the training data for each system and the number of parameters.", "questions_for_authors": "- Section 3: \"$X$ is padded\"—What happens when $X$ is longer than $Y$? Is $Y$ padded in that case?\n- Will the implementation in Section 4.2 be open-sourced?\n  - \"We build a French-English speech translation dataset of approximately 40K hours in each language.\" Which data sources were used? Will this dataset be released?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Gq7B34B31", "reviewer": "Reviewer_VuJU", "summary": "This paper introduces a model named Hibiki for real-time speech-to-speech translation. Hibiki employs a multi-stream architecture to synchronously process source and target speech, and generates both text and audio through multi-task learning. Trained with a weakly supervised method, Hibiki demonstrates SOTA performance in a French-to-English translation task, achieving good translation quality, speaker fidelity, and naturalness. Its simple inference process supports batch processing and real-time deployment on devices.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback.\n\n# Updates in the revised version of the paper\nWe first inform the reviewer that we will update the reported results of Hibiki-M in [Table 2](https://hibiki-s2st.github.io/e.png) after fixing an issue that further improves performance.\nFollowing the reviewers' suggestions and thanks to the extra page provided for the camera ready, we will revise the paper on the following aspects:\n* Clarify the **model's architecture, configurations and the nature/size of the datasets used.**\n* Add **quality/latency trade-off curves**  by varying hyperparameters of our contextual alignment method.\n* Add **COMET** evaluation scores.\n* Extend experiments to the **English->French** direction.\n* Improve the **references section** and discuss similar works pointed out by the reviewers.\n\n# Comments\n\n## On the generalization of the contextual alignment method to many language pairs\n\nWe acknowledge that our method is only illustrated by a single language direction, however there is no specific aspect of our method with respect to the language pair but the fact that MADLAD -- which we used to derive contextual alignment -- performs well on these languages. We expect our method to work as strongly on pairs of languages where SOTA MT performs well and allows for reliable contextual alignments. Given the massively multilingual nature of MADLAD, it is a good candidate for scaling to many language pairs. We provide the reviewers examples of [contextual alignments with other languages](https://hibiki-s2st.github.io/d.png). As a first step towards more language directions, we have extended our experiments to the English->French direction and provide [experimental results](https://hibiki-s2st.github.io/b.png) that we will add to the revised paper.\n\n## On the quality/latency trade-off and controllable latency\n\nAs mentioned in Section 3.2.2 (l.183), we enforce a 2s delay between words associated through contextual alignment. We acknowledge that this choice can be reconsidered **and produced a trade-off curve by varying the delay, as asked by the reviewers.** The results reported in [this quality/latency study](https://hibiki-s2st.github.io/c.png) show that Hibiki provides a better trade-off than Seamless. We will add this figure to the revised version of the paper.\n\nWe also acknowledge that the proposed version of Hibiki does not allow for inference-time latency control. We could rely on conditional training, as we did for the speaker similarity, to simultaneously train the model on multiple latency levels making it possible to control the latency at inference by changing the conditioning. We will add this mention to the limitations section.\n\n## On incorporating real interpreter speech in training or fine-tuning\n\nReal interpreter speech would indeed be an ideal source of data. However, given the scarcity of such data in terms of volume, number of speakers, covered languages, etc. we believe that developing pipelines for synthetic paired data generation is the best path towards scaling speech translation to more languages and conditions.\n\n## On speaker similarity and Classifier-Free Guidance (CFG)\n\nAccent transfer is indeed a limitation that we mention in the comments on the ablation on classifier-free guidance in Section 4.6. We expect that both a high speaker similarity and a reduced accent can be achieved by labeling our data with a speaker identification system that is invariant to accent and more accurate in terms of identity.\n\nWhile CFG offers fine-grained control on the strength of conditioning, it also doubles the computational cost at inference. [Cideron et al. (2024)](https://arxiv.org/abs/2410.06084) have proposed distilling the post-CFG logits into a student model. Since our submission, we have experimented this method to distill the logits with $\\gamma = 3$ into a student model such that the latter can run without CFG. In our long-form evaluations, while **Hibiki-M without CFG reaches a speaker similarity of 0.33, after CFG-distillation it reaches 0.38 (without CFG), close to the 0.39 obtained with CFG.** This suggest we can use distillation to remove the need for CFG at inference.\n\n## On decoding with stochastic sampling\n\nStochastic sampling indeed induces variability in the output, some of which is desirable (e.g. acoustic diversity) while some is undesirable (unreliable translation). We thus use a lower top-k inference parameter on the text stream compared to the audio streams to disentangle acoustic and linguistic diversity, keeping the former high while lowering the latter.\n\n## On the alignment of text and speech tokens\n\nAs described in Section 3.4.4 of [Défossez et al. (2024)](https://arxiv.org/abs/2410.00037), special `PAD` and `EPAD` (End of PADding) tokens are inserted in the text stream to account for the difference between the constant framerate of the audio tokens and the variable rate of text tokens.", "claims_and_evidence": "All the claims seem reasonable, however, the proposed methods are only validated on English-Franch, and their effects on other languages ​​need study.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria generally make sense.", "theoretical_claims": "I have checked all equations and they are all correct.", "experimental_designs_or_analyses": "I have reviewed the experimental designs and analyses presented in the paper, and they generally appear to be sound and valid.", "relation_to_broader_scientific_literature": "The paper study an important question in speech domain, real-time speech-to-speech translation.", "essential_references_not_discussed": "I think all the related works are discussed in this paper.", "other_strengths_and_weaknesses": "Strengths\n- Hibiki integrates simultaneous speech-to-speech and speech-to-text translation into a single decoder-only model, simplifying inference.\n- Achieves strong BLEU scores, outperforming previous models in both offline and real-time speech translation.\n- Produces fluent, well-paced speech with better voice preservation than prior models.\n- Uses a weakly-supervised alignment method to determine optimal delays, improving real-time accuracy.\n- Simple inference process allows for batched GPU translation and real-time on-device deployment.\n\nWeaknesses\n- Relies heavily on synthetic training data, requiring high-quality ASR, MT, and TTS models.\n- The paper written need to be imporved and is a little difficult to follow, many details are derived from  the Moshi paper.\n-  Currently evaluated only on French-English, raising questions about generalizability.\n- Speaker similarity is improved but not perfect, and accent transfer may not always be desirable.", "other_comments_or_suggestions": "- Testing on more language pairs and domains would strengthen claims of generalizability.\n\n- A more intuitive explanation of multistream decoding would help readers understand the model’s structure.\n\n- Incorporating real interpreter speech in training or fine-tuning could improve performance.\n\n- Exploring alternative speaker adaptation methods might enhance voice retention without needing classifier-free guidance.", "questions_for_authors": "- How well does the approach generalize to other languages, especially low-resource ones?\n\n- Can the latency-quality trade-off be adjusted at inference, or is it fixed based on training?\n\n- Does using stochastic sampling for decoding lead to inconsistent translations across different runs?\n\n- How tightly are the text and speech token streams aligned—does each word directly correspond to a speech segment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VLpubf7jGi", "reviewer": "Reviewer_XmPi", "summary": "This paper introduces Hibiki, a decoder only model for simultaneous speech-to-speech/text translation. Hibiki adapts the architecture of a full-duplex dialogue model Moshi to simultaneous translation by modeling source speech as user input and target speech as agent response. To train Hibiki, the authors synthesize trajectories of simultaneous translation by leveraging the log probabilities of a pretrained machine translation model. The experimental results on Fr-En direction of CVSS dataset demonstrate that 1) Hibiki shows higher speech quality and voice transfer than strong baselines like Seamless while has larger latency; 2) Hibiki is able to conduct efficient batched inference and the distilled version even able to run on a smartphone in realtime.\n\n## update after rebuttal\n\nSome of my major concerns have been addressed. However, the latency remains somewhat high, as shown in the quality-latency trade-off presented in the rebuttal. Combined with the limited coverage of language directions, I will maintain my current score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback.\n\n# Updates in the revised version of the paper\nWe first inform the reviewer that we will update the reported results of Hibiki-M in [Table 2](https://hibiki-s2st.github.io/e.png) after fixing an issue that further improves performance.\nFollowing the reviewers' suggestions and thanks to the extra page provided for the camera ready, we will revise the paper on the following aspects:\n* Clarify the **model's architecture, configurations and the nature/size of the datasets used.**\n* Add **quality/latency trade-off curves**  by varying hyperparameters of our contextual alignment method.\n* Add **COMET** evaluation scores.\n* Extend experiments to the **English->French** direction.\n* Improve the **references section** and discuss similar works pointed out by the reviewers.\n\n# Comments\n\n## On the generalization of the contextual alignment method to many language pairs\n\nWe acknowledge that our method is only illustrated by a single language direction, however there is no specific aspect of our method with respect to the language pair but the fact that MADLAD -- which we used to derive contextual alignment -- performs well on these languages. We expect our method to work as strongly on pairs of languages where SOTA MT performs well and allows for reliable contextual alignments. Given the massively multilingual nature of MADLAD, it is a good candidate for scaling to many language pairs. We provide the reviewers examples of [contextual alignments with other languages](https://hibiki-s2st.github.io/d.png). As a first step towards more language directions, we have extended our experiments to the English->French direction and provide [experimental results](https://hibiki-s2st.github.io/b.png) that we will add to the revised paper.\n\n## On the quality/latency trade-off and controllable latency\n\nAs mentioned in Section 3.2.2 (l.183), we enforce a 2s delay between words associated through contextual alignment. We acknowledge that this choice can be reconsidered **and produced a trade-off curve by varying the delay, as asked by the reviewers.** The results reported in [this quality/latency study](https://hibiki-s2st.github.io/c.png) show that Hibiki provides a better trade-off than Seamless. We will add this figure to the revised version of the paper.\n\nWe also acknowledge that the proposed version of Hibiki does not allow for inference-time latency control. We could rely on conditional training, as we did for the speaker similarity, to simultaneously train the model on multiple latency levels making it possible to control the latency at inference by changing the conditioning. We will add this mention to the limitations section.\n\n## On references\n\nWe acknowledge the contributions made by [Papi et al. (2023)](https://arxiv.org/abs/2305.11408), [Wang et al. (2024)](https://arxiv.org/abs/2402.10552) and [Yu et al. (2025)](https://arxiv.org/abs/2502.00634). We also acknowledge the progress made in streaming and speech translation for complex language pairs such as English-Japanese as highlighted in [Ahmad et al. (2024)](https://arxiv.org/abs/2411.05088). We will add these references to the related work in the updated version of our paper.\n\n## On the usage of a single text translation model\n\nWe used a single model for translation and alignment as we expect this model to be the most appropriate to derive a reliable likelihood-based alignment. However we acknowledge in Section 4.6.2 that we may overfit MADLAD and diversifying the models used to generate and align data may improve the robustness of our system.\n\n## On neural evaluation of quality\n\nWe added [COMET evaluations](https://hibiki-s2st.github.io/a.png) and executed the `comet-compare` script which gave the following system ranking: MADLAD-3B > Hibiki > Seamless with a p-value < 0.05.\n\n## On statistical significance of human evaluation\n\nIndeed we may get more robust estimation from more samples, however the gap between approaches (as demonstrated by error intervals) is wide enough such that we consider these results trustworthy. We also encourage the reviewer to listen to the example webpage.\n\n## Answer: *What is the noise augmentation techniques used in Hibiki?*\n\nWe use samples from [freesound.org](https://freesound.org) that are randomly added with various intensities to the input audio during training. We will add details about it in the revised version of the paper and the associated code will be released with the training code.\n\n## Answer: *Is Hibiki able to generalize to unbounded speech?*\n\nThe updated version of Hibiki that we will release is trained with windowed attention to extrapolate beyond a few minutes.\n\n## Answer: *Is Hibiki still better than StreamSpeech using only CVSS training data?*\n\nWe acknowledge the fact that we never trained Hibiki on CVSS data only as we aimed to handle real-world use cases with longer and diverse speech inputs, while CVSS only contains single sentences of a few seconds.", "claims_and_evidence": "> Claim 1: The architecture of a full-duplex dialogue model Moshi can be adapted for simultaneous translation.\n\nThis claim is supported. It is natural to regard source speech as user speech input and target translation speech as agent response output. Also, the experiments show that this modeling is able to conduct simultaneous translation effectively. \n\n> Claim 2: Decoder-only architecture enables efficient inference.\n\nThis claim is also supported. It is true that prior architectures are hard to do efficient batch inference due to their complex policy design, while a decoder-only model with implicit policy makes it much more convenient.", "methods_and_evaluation_criteria": "**Method** \n1. The method adapts a dialogue model to simultaneous speech to speech translation natually. \n2. The translation and source-target alignment are both generated by MADLAD-3B model. The author lacks quality analysis here. \n3. Hibiki does not support adjusting the latency during inference, which means you need to train multiple models for each latency level. \n\n**Evaluation** \n1. Dataset: CVSS is a common dataset for evaluating speech to speech translation.  \n2. Latency metric: LAAL and Offset are commonly used metrics for latency evaluation. \n3. Translation quality metric: BLEU is a widely used metric for translation quality evaluation. However, BLEU is still a n-gram based method, and is outperformed by many later neural metric like COMET, MetricX as shown in recent WMT workshops. \n4. Speech quality metric: The human evaluation is conducted only on 30 speech samples. May not demonstrate enough statistical significance.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "1. Hibiki is trained on more and refined speech data compared to baseline StreamSpeech. The comparison is not fair.\n2. The comparison in Table 2 not that informative since the comparison is not at the same latency. Vast literatures in both simultaneous speech-to-text and text-to-text translation [e.g., 1-2] already show that the quality can be much higher if the allowed latency is higher. \n3. There are other existing ways to build the source-target alignment, like the one introduced in [3], but the author does not compare with them. \n4. The experiment only tested Fr-En direction, but simultaneous translation could behave very differently on different language directions due to difference in linguistic structures. More language directions are needed. \n\n\n[1] Papi, S., Turchi, M., Negri, M. (2023) AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation. Proc. Interspeech 2023, 3974-3978, doi: 10.21437/Interspeech.2023-170\n[2] Donglei Yu, Xiaomian Kang, Yuchen Liu, Yu Zhou, and Chengqing Zong. 2024. Self-Modifying State Modeling for Simultaneous Machine Translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9781–9795, Bangkok, Thailand. Association for Computational Linguistics.\n[3] Wang, M., Vu, T. T., Wang, Y., Shareghi, E., & Haffari, G. (2024). Conversational simulmt: Efficient simultaneous translation with large language models. arXiv preprint arXiv:2402.10552.", "relation_to_broader_scientific_literature": "1. Hibiki is one of the first decoder-only models for simultaneous speech-to-speech translation and exhibit advantages in efficient batched inference. There are similar findings before in simultaneous text translation [1-2], but not in simultaneous speech-to-speech translation. \n2. Synthesizing source-target alignment is not a new idea, [1] previously proposed a word-alignment-based approach. However, the perplexity-based method introduced in this paper is new, as far as I know.\n\n[1] Wang, M., Vu, T. T., Wang, Y., Shareghi, E., & Haffari, G. (2024). Conversational simulmt: Efficient simultaneous translation with large language models. arXiv preprint arXiv:2402.10552.\n[2] Yu, D., Zhao, Y., Zhu, J., Xu, Y., Zhou, Y., & Zong, C. (2025). SimulPL: Aligning Human Preferences in Simultaneous Machine Translation. arXiv preprint arXiv:2502.00634.", "essential_references_not_discussed": "The key contribution of this paper is the decoder-only architecture for simultaneous speech-to-speech translation and a synthetic alignment building method. Both of which are discussed by [1] in the context of simultaneous text-to-text translation, but not cited nor discussed. \n\n[1] Wang, M., Vu, T. T., Wang, Y., Shareghi, E., & Haffari, G. (2024). Conversational simulmt: Efficient simultaneous translation with large language models. arXiv preprint arXiv:2402.10552.", "other_strengths_and_weaknesses": "The writing needs improvement. The author assumes some prior knowledge of Moshi model, RQ-transformer and related techniques. It would be better to have a figure illustrating the model architecture, so that it is easier to understand for broader audience.", "other_comments_or_suggestions": "1. Figure 2 is a bit confusing at first glance. A more complete illustration of the architecture will be helpful. \n2. What is the noise augmentation techniques used in Hibiki?\n3. line 311-316, the description of EOS tokens is confusing.", "questions_for_authors": "1. Is there way to adjust the latency of Hibiki at inference time? If so, does it provides a better quality-latency trade-off curve than Seamless?\n2. Is Hibiki able to generalize to unbounded speech? By unbounded speech I mean streaming speech input with infinite length. \n3. Is Hibiki still better than StreamSpeech using only CVSS training data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nf8FY6QICF", "reviewer": "Reviewer_sRiZ", "summary": "The paper introduces Hibiki, a decoder-only model for simultaneous speech-to-speech (S2ST) and speech-to-text (S2TT) translation. Unlike offline approaches, Hibiki translates speech in real-time using a multistream language model that synchronously generates text and audio tokens. The model leverages contextual alignment to determine optimal delays for translation, improving fluency and speaker similarity. Experimental results show strong performance in French-English translation, with real-time capabilities on both GPUs and mobile devices.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback.\n\n# Updates in the revised version of the paper\nWe first inform the reviewer that we will update the reported results of Hibiki-M in [Table 2](https://hibiki-s2st.github.io/e.png) after fixing an issue that further improves performance.\nFollowing the reviewers' suggestions and thanks to the extra page provided for the camera ready, we will revise the paper on the following aspects:\n* Clarify the **model's architecture, configurations and the nature/size of the datasets used.**\n* Add **quality/latency trade-off curves**  by varying hyperparameters of our contextual alignment method.\n* Add **COMET** evaluation scores.\n* Extend experiments to the **English->French** direction.\n* Improve the **references section** and discuss similar works pointed out by the reviewers.\n\n# Comments\n\n## On the generalization of the contextual alignment method to many language pairs\n\nWe acknowledge that our method is only illustrated by a single language direction in the paper. However, there is no specific aspect of our method with respect to the language pair, but the fact that MADLAD -- which we used to derive contextual alignment -- performs well on these languages. We expect our method to work as strongly on pairs of languages where SOTA text translation models perform well and can thus allow us to derive reliable contextual alignments. Given the massively multilingual nature of MADLAD or even more recent systems like GemmaX2-28-9B ([Cui et al., 2025](https://arxiv.org/abs/2502.02481)), we expect this approach to be a good candidate for scaling to many language pairs. We provide the reviewers examples of [contextual alignments with other languages](https://hibiki-s2st.github.io/d.png). As a first step towards more language directions, we have extended our experiments to the English->French direction and provide [experimental results](https://hibiki-s2st.github.io/b.png) that we will add to the revised paper.\n\n## On the quality/latency trade-off\n\nAs mentioned in Section 3.2.2 (l.183), we enforce a 2s delay between words associated through contextual alignment as we found it to provide a good balance between latency and translation quality. We acknowledge that this choice can be reconsidered and that trade-off curves would provide a clearer picture to the reader. **We thus produced a trade-off curve by varying the delay, as asked by the reviewers.** The results reported in [this quality/latency study](https://hibiki-s2st.github.io/c.png) show that Hibiki provides an overall better trade-off than Seamless. We will add this figure to the revised version of the paper.\n\n## On the alignment-aware TTS\n\nWe acknowledge that a non-negligible part of the technical details inherited from [Défossez et al. (2024)](https://arxiv.org/abs/2410.00037) were not exposed in our paper as we preferred to focus on the data creation pipelines and the experimental protocol and results. We will take the opportunity of the extra page provided for the camera ready to improve the clarity of technical details such as the alignment-aware TTS. Moreover, we would like to highlight that there is no missing Appendix to our paper as we referrerd to Appendix C of [Défossez et al. (2024)](https://arxiv.org/abs/2410.00037) at Section 3.2 l.203. We will improve the formulation in the updated version of the paper.\n\nAs a rapid summary of the explanations given in Appendix C of [Défossez et al. (2024)](https://arxiv.org/abs/2410.00037), one can force text tokens directly in the text stream of a TTS model derived from the Moshi architecture. Thanks to the contextual alignment, we can ensure that a given text token is fed at the right timestamp (that is not too early) using `PAD` tokens instead to delay its insertion.\n\n## On text-only pretraining \n\nThe text-only pretraining phase is that of standard next token prediction, there is no adaptation to a MT model in the text-only pretraining phase. In early development, we tried alternating batches of text translation and speech translation (starting from a pretrained text model), however while this model did perform quite well in text translation, this did not result in any measurable improvement of the speech translation. We hypothesize that this lack of transferrability is due to the fact that MT samples were built by concatenating the source and target texts in the text stream which is radically different from what is seen in the text stream for a speech translation sample (where the source is audio-only and the target is time-aligned). This highlights a challenge in aligning text and speech representations in speech-text LLMs for each modality to benefit the other, which we believe will be critical to extend speech translation to more language pairs as text translation data is much more accessible than speech translation data.", "claims_and_evidence": "The paper claims Hibiki achieves state-of-the-art translation quality, speaker similarity, and naturalness, supported by BLEU scores, speaker similarity metrics, and human evaluations. The contextual alignment method is validated through ablation studies, demonstrating its impact on latency-quality trade-offs.\n\nHowever, the claim that Hibiki provides an optimal balance between latency and accuracy is questionable, as Seamless achieves lower latency (LAAL and End Offset). So Hibiki does achieve better quality but it might sacrifice the latency to some extent. I think this is okay though because the author adds human evaluation and show Hibiki is preferred.", "methods_and_evaluation_criteria": "The paper employs standard evaluation metrics for S2ST, including BLEU for translation quality, speaker similarity (cosine similarity), and MOS for human evaluation. It also uses LAAL (Length-Adaptative Average Lagging) to measure latency, ensuring a fair comparison with existing models. \n\nOne issue is that the experiments are limited to French-English, making it unclear how Hibiki generalizes to other languages.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is generally strong, with comprehensive comparisons to Seamless and StreamSpeech. Ablation studies effectively highlight the impact of alignment strategies and classifier-free guidance. However, latency trade-offs need more discussion, as Hibiki has higher lag than Seamless. Additionally, the alignment-aware TTS system lacks detail, making it difficult to verify how timing constraints are enforced during synthesis. The missing Appendix C (mentioned in section 3.2 line 203) further limits transparency.", "relation_to_broader_scientific_literature": "Hibiki builds on prior work in S2ST, alignment modeling, and multistream processing. It extends Seamless (Barrault et al., 2023) and StreamSpeech (Zhang et al., 2024a) with an adaptive alignment approach and improved speaker transfer. Its multistream modeling is inspired by Moshi (Défossez et al., 2024), originally designed for full-duplex spoken dialogue.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- I want to know more about the alignment-aware TTS, how is it implemented exactly to consider the alignment during synthesis?\n- For the text-only pertaining, I am wondering what is the machine translation performance after the training? Since the model is trained from scratch, I am assuming it is always trained to perform translation task? Or is it trained for general LM task then adapted into a decoder-only MT model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a decoder-only speech-to-speech translation model Hibiki. The paper interates multiple techniques (alignment-related, synthetic data creation, classifier-free guidance, etc.) and achieves high performance in several public benchmarks. 

Strengths of the paper: 
- The proposed model Hibiki achieves state-of-the-art performance in simultaneous speech-to-speech translation. 
- Hibiki is able to conduct efficient batched inference and the distilled model is able to run on a smartphone in realtime.

Weakness of the paper: 
- The model is only evaluated on French-English direction. It is not enough evidence about its generalization.
- The human evaluation only contains 30 samples. 

All reviewers are satisfied with author responses.
Decision: Accept (poster)