Improving Language and Modality Transfer in Translation by Character-level Modeling
Authors: Ioannis Tsiamas, David Dale, Marta R. Costa-jussà
Accepted at: acl25

Improving Language and Modality Transfer in Translation by Character-level Modeling

Ioannis Tsiamas1,2    David Dale1    Marta R. Costa-jussà1
1FAIR at Meta, Paris    2Universitat Politècnica de Catalunya, Barcelona
Correspondence: ytsiamas@meta.com, daviddale@meta.com

Abstract

Current translation systems, despite being highly multilingual, cover only 5% of the world’s languages. Expanding language coverage to the long-tail of low-resource languages requires data-efficient methods that rely on cross-lingual and cross-modal knowledge transfer. To this end, we propose a character-based approach to improve adaptability to new languages and modalities. Our method leverages SONAR, a multilingual fixed-size embedding space with different modules for encoding and decoding. We use a teacher-student approach with parallel translation data to obtain a character-level encoder. Then, using ASR data, we train a lightweight adapter to connect a massively multilingual CTC ASR model (MMS), to the character-level encoder, potentially enabling speech translation from 1,000+ languages. Experimental results in text translation for 75 languages on FLORES+ demonstrate that our character-based approach can achieve better language transfer than traditional subword-based models, especially out-performing them in low-resource settings, and demonstrating better zero-shot generalizability to unseen languages. Our speech adaptation, maximizing knowledge transfer from the text modality, achieves state-of-the-art results in speech-to-text translation on the FLEURS benchmark on 33 languages, surpassing previous supervised and cascade models, albeit being a zero-shot model with minimal supervision from ASR data.

1 Introduction

Translation has experienced a large growth in terms of language coverage in the last years, with models supporting 200-400 languages in text (NLLB, 2024; Kudugunta et al., 2023), and 100 in speech (SEAMLESS, 2025). Although impressive in terms of population coverage (90%), in terms of actual language coverage we stand at only 5%.1 Moving towards expanding to the long-tail of low-resource languages in the world posses some serious challenges due to the increasingly scarce data sources. For text translation we have to rely on a few thousand parallel sentences, while chances are there are no parallel data for speech translation (ST). To ease the issue of data scarcity in low-resource settings, multilinguality for text (Johnson et al., 2017; Chang et al., 2024) and multimodality for speech (Tang et al., 2021a) can usually be beneficial. But how can we increase cross-lingual and cross-modal knowledge transfer from high-resource languages and modalities? Recent research suggests that character-level models exhibit better cross-lingual transfer in text translation, especially in low-resource scenarios (Edman et al., 2024). Furthermore, for speech translation, methods usually take advantage of a text-based encoder for semantic modeling (Tang et al., 2021a;

![Approach for character-level and speech adaptation using the SONAR space.](page_1016_384_410_384.png)

Figure 1: Approach for character-level and speech adaptation using the SONAR space.

1www.ethnologue.com
Zhang et al., 2023), but the subword-based tokenization is incompatible in terms of length and content with the acoustic representations, thus creating a modality gap that hinders knowledge transfer. Previous research mitigates this by either using a phoneme-based text encoder (Tang et al., 2021a; Le et al., 2023) or converting the acoustic representations to subword-like units (Tsiamas et al., 2024). But a phonemized input degrades performance due to ambiguity\footnote{Homophones, loss of orthographic information, etc.} and furthermore phonemizers for 1000+ languages might be infeasible (Zhao et al., 2024), while subword-based compression requires a substantial amount of data. To this end, we propose to shift towards character-based encoders, that could support data-efficient knowledge transfer both between languages and between text and speech. Our method is based on SONAR (Duquenne et al., 2023), which is an encoder-decoder with a fixed-size semantic embedding space that supports 200 languages, and on MMS (Pratap et al., 2023), which is a CTC-based ASR model that supports 1,000+ languages. Using a teacher-student approach, we obtain a character-based text encoder that embeds sentences in the SONAR space. Then, we propose an adapter that seamlessly connects the CTC output space of MMS to the character-level input space of our encoder, requiring minimal supervision from audio-transcription pairs (Fig. 1). Our experimental results in 75 languages on FLORES+ (NLLB, 2024), show that compared to traditional subword-based models, our multilingual character-level SONAR encoder exhibits better cross-lingual knowledge sharing between known languages and superior zero-shot generalizability to unseen languages. Furthermore, our speech adaptation of the character-based encoder, despite relying only on ASR data, can maximize knowledge transfer from text, and thus surpasses the previous best supervised system (SEAMLESS, 2025) and strong cascades with Whisper (Radford et al., 2022), achieving new state-of-the-art in FLEURS (Conneau et al., 2022).

2 Relevant Research

2.1 Character-level MT

Early works in machine translation investigated character-level approaches due their advantages in understanding and generating rare and unseen words, handling noise, having smaller vocabularies, and being simpler due to the removal of subword tokenization (Sennrich et al., 2016). Several methods using attention-based sequence-to-sequence models (Sutskever et al., 2014; Bahdanau et al., 2015) showed that character-level MT can reach or surpass subword-based approaches (Ling et al., 2015; Costa-jussà and Fonollosa, 2016; Lee et al., 2017; Cherry et al., 2018; Chung et al., 2016). Later, Xue et al. (2022) showed that ByT5, encoder-decoder multilingual language model operating on bytes, is more robust to noise and performs better in spelling-sensitive tasks, than its subword-based counterpart mT5 (Xue et al., 2021). Edman et al. (2024) fine-tuned the ByT5 and mT5 models for translation, and found that character-level modeling is particularly effective when parallel data are limited. Libovický et al. (2022) sought to answer why fully character-level MT has not been widely adopted, which was attributed to lower efficiency, and an inability to confirm previous findings that had been suggesting better domain and morphological generalization. In this work, we propose an encoder-only character-level approach (Cao, 2023) based on SONAR (Duquenne et al., 2023), and study the benefits of cross-lingual transfer in a large group of 75 languages, both in low-resource and in zero-shot settings. Several works have proposed methods that alleviate the additional computational costs stemming from the longer sequences that character- or byte-level models need to process (Clark et al., 2022; Tay et al., 2022; Pagnoni et al., 2024). Since our approach adopts character-level modeling only on the encoder side, and due to the fixed-size embedding bottleneck of SONAR, the computational overhead is minimal, and thus we do not study any architecture-based changes in this work.

2.2 Cross-modal Transfer in ST

Speech translation models have traditionally relied on cross-modal knowledge transfer from the more resourceful task of text translation to improve performance. Several works achieved this by using a multitasking framework of MT and ST, where they share the text modules between the two tasks, and the semantic text encoder accepts either acoustic representations or text embeddings as inputs (Liu et al., 2020; Ye et al., 2021; Tang et al., 2021b; Fang et al., 2022). Another line of work aims at bridging the modality gap by additionally minimizing the distance between the speech-text representations of the encoders (Tang et al., 2021a; Ye et al., 2022; Ouyang et al., 2023). ZeroSwot (Tsiamas et al., 2024) eliminated the dependency on parallel ST
data, and relied only on minimizing the Wasserstein distance (Peyré and Cuturi, 2019) between the speech-text of representations of the encoders using ASR data. In our framework we also follow the paradigm of ZeroSwot, but due to the fixed-size encoder bottleneck of SONAR, our optimization is simpler, and minimizes the MSE distance. Another important consideration in maximizing knowledge transfer from text is unifying the tokenization of acoustic encoder’s output and text encoder’s input space, which are usually phoneme/characters for the the CTC (Graves et al., 2006) of the acoustic encoder, and subwords for the embedding layer of the text encoder. Previous works have either used a phoneme-based input for the text encoder (Tang et al., 2021a; Le et al., 2023), or a subword-based output for the acoustic encoder’s CTC output (Liu et al., 2020; Yang et al., 2023), or more recently a character-to-subword compression adapter (Tsiamas et al., 2024). But phoneme-based text input degrades performance due to ambiguity in meaning. Then, for subword-based output in CTC, it is questionable whether it can scale to massively multilingual vocabularies of hundreds of thousand of tokens (NLLB, 2024). Finally the subword compression adapter requires a substantial amount of ASR data to learn, which can be problematic for the long tail of low-resource languages. Contrary, our approach is based on first modifying the text encoder to work with character-level inputs without degrading MT performance, and then learning a data-efficient and lightweight adapter that connects to it the character-based output space of a CTC acoustic encoder.

3 Methodology

We utilize the multilingual fixed-size embedding space of SONAR (Duquenne et al., 2023), in order to add new languages and modalities (speech) to it. We first obtain a character-level text encoder using a teacher-student approach with parallel translation data (\S3.2), and then adapt it to work with CTC acoustic representations as inputs using a teacher-student approach with paired audio-transcriptions (\S3.3).

3.1 SONAR

The SONAR encoder is a Transformer (Vaswani et al., 2017) with \( N_t \) layers of dimensionality \( d_t \), and a subword-based vocabulary \( \mathcal{V}_t \). The final encoder representation is mean-pooled to obtain a sentence embedding \( \mathbf{e} \in \mathbb{R}^{d_t} \). The SONAR decoder which also has \( N_t \) layers of dimensionality \( d_t \), attends with cross-attention to \( \mathbf{e} \), in order to predict the target sequence.

3.2 Character-level Text Encoder

Our character-level encoder (charSONAR) is initialized from the SONAR encoder, and thus has \( N_t \) layers of dimensionality \( d_t \). As part of the character-based input vocabulary we only keep the tokens of \( \mathcal{V}_t \) that are composed of single characters, thus having a vocabulary \( \mathcal{V}_c \subset \mathcal{V}_t \).

Training Objectives. For training, we follow a student-teacher approach with the SONAR encoder as a teacher, where we minimize the MSE loss between the charSONAR embedding \( \mathbf{c} \) and a SONAR embedding \( \mathbf{e} \in \mathbb{R}^{d_t} \), using monolingual or parallel translation data. We consider three different MSE objectives:

• Reconstruction, where we learn from non-parallel data, and given a sentence \( x \), we minimize \( \mathcal{L}^{recon} = \mathrm{MSE}(\mathbf{c}^x, \mathbf{e}^x) \).
• Translation, where we learn from parallel data, and given a sentence \( x \) with translation \( y \), we minimize \( \mathcal{L}^{trans} = \mathrm{MSE}(\mathbf{c}^x, \mathbf{e}^y) \).
• Interpolation, where we also learn from parallel data, and given a sentence \( x \) with translation \( y \), we minimize the distance from the ‘average’ teacher embedding for that pair (Eq. 1).

\[
\mathcal{L}^{interpol} = \mathrm{MSE}\left( \mathbf{c}^x, \frac{\mathbf{e}^x + \mathbf{e}^y}{2} \right)
\] (1)

Augmentations. We apply ASR-like augmentations to make the character-based encoder robust to the normalized and error-prone output of CTC ASR models and increase cross-modal transfer. Specifically with some probability \( p^{norm} \), we normalize the source text input of the char-based encoder, removing casing and punctuation. Furthermore, with some small probability \( p^{noise} \), we inject different noise perturbations to the text, such as character addition, deletion and replacement.

3.3 Speech Encoder

Our speech encoder is composed of an acoustic encoder, an adapter, and the charSONAR encoder.

3.3.1 Acoustic Encoder

The acoustic encoder consists of a series of strided convolutional layers, followed by a Transformer
encoder with \( N_s \) layers of dimensionality \( d_s \). It is initialized from MMS (Pratap et al., 2023), which was pretrained with the self-supervised objective of wav2vec 2.0 (Baevski et al., 2020) and fine-tuned with CTC (Graves et al., 2006) on 1,000+ languages. Each language \( i \) has its own CTC prediction head \( \mathbf{W}^{(i)} \in \mathbb{R}^{d_s \times |\mathcal{B}_i|} \), where \( \mathcal{B}_i \) is a language-specific character-based vocabulary (including the <blank> token), with \( \mathcal{B}^{(i)} \subset \mathcal{V}_c \).

The acoustic encoder is kept frozen during training, and with it we extract the final encoder representation \( \mathbf{H} \in \mathbb{R}^{m \times d_s} \). Next, we apply CTC-based compression (Gaido et al., 2021) to remove redundancy and obtain a representation that is similar in length as the character-based tokenization of our charSONAR encoder. We label each point \( j \) of \( \mathbf{H} \) with its CTC prediction \( \pi_j = \operatorname{argmax}(\mathbf{W}^{(i)} h_j) \), then average consecutive points corresponding to the same prediction, and drop points corresponding to <blank>. We thus obtain an acoustic representation \( \mathbf{A} \in \mathbb{R}^{n \times d_s} \), where \( n < m \).

3.3.2 Cross-modal Adapter
We use a cross-modal adapter to process the acoustic representation \( \mathbf{A} \) into an embedding-like representation \( \mathbf{E} \in \mathbb{R}^{n \times d_t} \), that aims to match as close as possible the character embedding expected at the input of the charSONAR encoder.

To maximize pretrained knowledge and obtain an adapter that could work out-of-the box in extremely low-resource settings, we propose a minimal (pretrained) two-layer architecture that is fully initialized from MMS and charSONAR. Specially, we use the CTC classification layer \( \mathbf{W}^{(i)} \) of MMS to project \( \mathbf{A} \) to logits, and with a softmax we obtain a probability distribution over the MMS vocabulary \( \mathcal{B}^{(i)} \). Then, since \( \mathcal{B}^{(i)} \subset \mathcal{V}_c \), we can connect the two spaces by doing a soft prediction over the charSONAR vocabulary using its embedding layer.

\[
\mathbf{E}^{\text{pt}} = \operatorname{softmax}\left( \mathbf{A} \mathbf{W}^{(i)} \right) \mathbf{Emb}^{(i)},
\]

where \( \mathbf{Emb}^{(i)} \in \mathbb{R}^{|\mathcal{B}^{(i)}| \times d_t} \) is the embedding layer of charSONAR, indexed by the entries of \( \mathcal{B}^{(i)} \).

Due to the nature of its initialization, the hidden dimension of the pretrained cross-modal adapter is fixed and bound to the size \( |\mathcal{B}^{(i)}| \) of the MMS vocabulary, which is relatively small, usually having 64 tokens. In order to be able to control, and increase, the capacity of the adapter, we also propose a dual cross-modal adapter that combines the pretrained one with another variable-sized adapter that is randomly initialized (Fig. 2).

\[
\mathbf{E}^{\text{rnd}} = \operatorname{ReLU}\left( \mathbf{A} \mathbf{U}^{\text{in}} \right) \mathbf{U}^{\text{out}},
\]

where \( \mathbf{E}^{\text{rnd}} \) is the output of the randomly-initialized adapter, \( \mathbf{U}^{\text{in}} \in \mathbb{R}^{d_s \times d_h} \) and \( \mathbf{U}^{\text{out}} \in \mathbb{R}^{d_h \times d_t} \) are learnable parameters, and \( d_h \) is a hyperparameter that we can control. We concatenate the individual outputs of the pretrained and randomly-initialized adapters and pass them through an MLP: \( 2d_h \rightarrow 1 \), followed by a sigmoid function to obtain a vector of weights \( \mathbf{v} \in (0, 1)^n \). The final representation \( \mathbf{E}^{\text{dual}} \in \mathbb{R}^{n \times d_t} \) is a weighted sum of \( \mathbf{E}^{\text{pt}}, \mathbf{E}^{\text{rnd}} \).

\[
\mathbf{v} = \sigma\left( \operatorname{MLP}\left( [\mathbf{E}^{\text{pt}}, \mathbf{E}^{\text{rnd}}] \right) \right)
\]
\[
\mathbf{E}^{\text{dual}} = \mathbf{v} \mathbf{E}^{\text{pt}} + (1 - \mathbf{v}) \mathbf{E}^{\text{rnd}}
\]

![Cross-modal Adapters diagram](page_1097_1042_563_312.png)

Figure 2: Cross-modal Adapters

To the output of the cross-modal adapter \( \mathbf{E} \) we prepend the corresponding language token embedding, and append the embedding for end-of-sentence from the charSONAR embedding table. After adding positional encoding, \( \mathbf{E} \) is passed through the transformer layers of the (frozen) char-SONAR encoder to obtain a speech embedding \( c^z \in \mathbb{R}^{d_t} \). To train the adapter we use audio-transcription pairs, and minimize the MSE loss between \( c^z \) and the SONAR embedding for the transcription \( e^z \). For speech translation inference, we use the SONAR decoder to generate the translation from the speech embedding \( c^z \).

4 Experimental Setup

4.1 Data
Text. We construct a diverse group of 63 languages in terms of family and script, and with varying degrees of resourcefulness, that are already present
in SONAR. We also add a group of 12 new languages, not present in SONAR, for which we have evaluation data in FLORES+.3 For the 63 known languages, to train charSONAR we used a combination of human-labeled data and mined parallel data (NLLB, 2024), which were filtered with BLASER 2.0 (Dale and Costa-jussà, 2024), discarding pairs with score lower than 4. For the group of 12 new languages, we used various publicly available sources of parallel data. For validation and testing we used the dev and devtest splits of FLORES+.

Speech. For our experiments in speech we used the 33 source languages. Our criteria for choosing these languages where: (a) part of the text training; (b) supported by MMS; (c) included in the Common Voice (CV) (Ardila et al., 2020) dataset (ASR training data); and (d) included in the FLEURS (Conneau et al., 2022) dataset (ST evaluation data). For training we used the train split of the version 17.0 of CommonVoice,4 and in some experiments the small train split of FLEURS which contains 2K examples for each language. Evaluation is done on the dev and test splits of FLEURS, which contain approximately 400 and 900 examples each.
Details regarding the languages and the amount of training data for both text and speech are available in Table 10 in the Appendix.

4.2 Model Architecture
The SONAR encoder (Duquenne et al., 2023) has \( N_t = 24 \) layers, with dimensionality of \( d_t = 1024 \), and an embedding table of size 256K (750M parameters in total). Our charSONAR encoder follows the same architecture, apart from the character-based embedding table with a size of 8K tokens (500M parameters in total). MMS (Pratap et al., 2023) has \( N_s = 48 \) layers with dimensionality \( d_s = 1280 \) (1B parameters).5 It uses language-specific layers (Houlsby et al., 2019) and CTC classification layers. The vocabulary is different for each language, usually having around 64 tokens. Since the size of the vocabulary is also the hidden dimension of our pretrained adapter, this adapter has approximately 200K parameters. The randomly-initialized adapter uses a hidden dimension of \( d_h = 1024 \) (2.2M parameters). For the dual adapter we use an MLP with an inner dimension of 64 (100K parameters) to predict the weight vector, thus having a total of 2.5M parameters. To generate translations, either from text or speech, we couple the encoder with the SONAR decoder which has 24 layers. For X→Eng generation we use the normal SONAR decoder,6 while for all other generation tasks we use the finetuned decoder,7 which according to Duquenne et al. (2023) and our observations here, performs better.

4.3 Training Details
Text. We use AdamW (Loshchilov and Hutter, 2019) with a learning rate of 4e-4, inverse square root scheduler with warmup, a batch size of 12K examples, dropout of 0.1, and train for 128K steps. We up-sample languages with a temperature of 0.5 (NLLB, 2024). We apply ASR-like text normalization by un-casing and removing punctuation to a source sentence with \( p^{norm} = 0.25 \), and inject character-based noise with \( p^{noise} = 0.125 \). Specifically, each character in the source sentence can be deleted, replaced, or a new character is further added, each with a probability of 0.0025. These values were tuned in a small validation set of CV to approximate the character-error-rate of MMS (Pratap et al., 2023). For replacement and addition we sample a new character from the character distribution of that language.

Speech. Both MMS and charSONAR remain frozen during speech training, and only the adapter is finetuned. We minimize the MSE distance with the original SONAR as a teacher. The learning rate is set to 2e-4, the batch size 500 examples, and the adapter dropout to 0.1 (0.3 for the randomly-initialized adapter).

4.4 Evaluation
We apply checkpoint averaging according to the dev set performance, and generate with a beam search of 5. We evaluate primary on two tasks: translation and similarity search. Translation quality is measured with xCOMET-XL (Guerreiro et al., 2024)8. When the target language is not supported, we use case-sensitive detokenized BLEU (Post, 2018) and chrF++ (Popović, 2017). For similarity search we measure xSIM++ (Chen et al., 2023) error rates, by augmenting the English parts of FLORES or FLEURS with 40K hard negatives.

3huggingface.co/datasets/openlanguagedata/flores_plus
4datasets/mozilla-foundation/common_voice_17_0
5huggingface.co/facebook/mms-1b-all
6dl.fbaipublicfiles.com/SONAR/sonar_text_decoder.pt
7dl.fbaipublicfiles.com/SONAR/finetuned_decoder.pt
8huggingface.co/Unbabel/XCOMET-XL
5 Text Results

Here we present our results in text translation and similarity search with charSONAR, and investigate its capacity for cross-lingual knowledge transfer.

5.1 Initial Exploration

Before the main experiments we conduct an exploration regarding the training objectives and augmentations. We used a small subgroup of 15 languages, with 3 languages from the Uralic family, and 12 languages that use the Cyrillic script (Table 10). In the upper part of Table 1 we show that the proposed interpolated MSE objective surpasses both the reconstruction and translation MSE objectives, and additionally their combination. This shows that SONAR embeds sentences in sub-optimal regions, while there are regions in between the languages that are better suited for both translation and similarity search. Further motivation is provided by our results of Table 2, where we show that for a pair of non-English languages Lang_1 and Lang_2, decoding from their average SONAR embedding Emb_{AVG} = (Emb_1 + Emb_2)/2 into English, is better than decoding from each individual embedding, with low-resource languages benefiting the most.9 This finding indicates that our charSONAR encoder can benefit from learning to map sentences to the interpolated or ‘average’ space existing between languages.

In the lower part of Table 1, we find that pre-training charSONAR with the reconstruction MSE before the interpolated MSE is beneficial, since it decouples learning character-level modeling and optimizing the embedding space. Finally, we see that the normalization and noise augmentation do not have an impact in performance. This is expected due to the ground truth source text, but as we show later in the initial exploration for the speech experiments (§6.1), these augmentations are beneficial, as the input to charSONAR is error-prone.

<table>
  <tr>
    <th rowspan="2">Objective</th>
    <th colspan="3">Model</th>
    <th colspan="2">COMET</th>
    <th colspan="2">xSIM++</th>
  </tr>
  <tr>
    <th>Pretrain</th>
    <th>Norm</th>
    <th>Noise</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>recon</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.929</td>
    <td>7.4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>trans</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.924</td>
    <td>6.6</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>recon+trans</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.929</td>
    <td>6.8</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>interpol</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.931</td>
    <td>6.6</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>interpol</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.934</td>
    <td>6.4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>interpol</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>0.934</td>
    <td>6.4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>interpol</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>0.934</td>
    <td>6.5</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>SONAR-200</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.925</td>
    <td></td>
    <td>8.5</td>
    <td></td>
  </tr>
</table>

Table 1: Ablations on training objectives and augmentations for the Ural/Cyril language group (15 langs). Text translation COMET scores and cross-lingual xSIM++(↓) error rates on FLORES dev (X→Eng).

<table>
  <tr>
    <th>Pairs</th>
    <th colspan="4">COMET</th>
    <th colspan="2">Advantage</th>
  </tr>
  <tr>
    <th>Lang_1</th>
    <th>Lang_2</th>
    <th>Emb_1</th>
    <th>Emb_2</th>
    <th>Emb_{AVG}</th>
    <th>Lang_1</th>
    <th>Lang_2</th>
  </tr>
  <tr>
    <td>Low</td>
    <td>Low</td>
    <td>0.788</td>
    <td>0.795</td>
    <td>0.864</td>
    <td>+0.076</td>
    <td>+0.069</td>
  </tr>
  <tr>
    <td>Low</td>
    <td>High</td>
    <td>0.793</td>
    <td>0.937</td>
    <td>0.920</td>
    <td>+0.137</td>
    <td>-0.017</td>
  </tr>
  <tr>
    <td>High</td>
    <td>High</td>
    <td>0.939</td>
    <td>0.937</td>
    <td>0.944</td>
    <td>+0.005</td>
    <td>+0.007</td>
  </tr>
</table>

Table 2: COMET scores of translating from average (interpolated) embeddings, compared to translating from individual embeddings, for different pairs based on resourcefulness. Results in X→Eng FLORES dev averaged over 50 randomly-sampled pairs in each row.

translation MSE objective. We compare against SONAR-200 (Duquenne et al., 2023), and an NLLB-200 (NLLB, 2024) topline, which is not restricted by a bottleneck encoder representation. We also train a comparable subword-based model by further fine-tuning SONAR on the 75 languages with the same setup as we did for charSONAR. We report text translation and cross-lingual similarity search (X→Eng) results, and group results by language resourcefulness according to the amount of our training data (Table 10). Our results of Table 3 show the clear advantage of our character-based encoder, where charSONAR-75 outperforms the comparable SONAR-75, and additionally the NLLB topline in translation. The gains are more evident in the group of 21 low-resource languages, where cross-lingual transfer can be more impactful.

5.2 Scaling to 75 Languages

Next, we present our findings from scaling-up the language coverage of charSONAR to 75 languages. We use the interpolated MSE objective, with reconstruction MSE pretraining and ASR-like text augmentations. For new languages, which are not supported by SONAR, we use the

9High-resource are negatively impacted when paired with low-resource, but this reflects only a very small fraction of their data.

5.3 Zero-shot Generalization

In our next experiment, we only train on the 63 known languages, and evaluate zero-shot on the 12 new ones. SONAR and NLLB encoders require a language tag to be prepended in the source sequence, which is problematic if we want to encode a sentence from a language not seen during train-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">COMET (↑)</th>
    <th colspan="4">xSIM++ (↓)</th>
  </tr>
  <tr>
    <th>Low<br>(21)</th>
    <th>Med<br>(21)</th>
    <th>High<br>(21)</th>
    <th>All<br>(63)</th>
    <th>New<br>(12)</th>
    <th>Low<br>(21)</th>
    <th>Med<br>(21)</th>
    <th>High<br>(21)</th>
    <th>All<br>(63)</th>
    <th>New<br>(12)</th>
  </tr>
  <tr>
    <td colspan="11"><i>Previous Works (trained on 200, not including the 12 new)</i></td>
  </tr>
  <tr>
    <td>NLLB-200</td>
    <td>0.877</td>
    <td><b>0.914</b></td>
    <td><b>0.949</b></td>
    <td>0.913</td>
    <td>0.454<sup>†</sup></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SONAR-200</td>
    <td>0.851</td>
    <td>0.894</td>
    <td>0.944</td>
    <td>0.897</td>
    <td>0.450<sup>†</sup></td>
    <td>13.1</td>
    <td>10.1</td>
    <td>7.3</td>
    <td>10.2</td>
    <td>52.7<sup>†</sup></td>
  </tr>
  <tr>
    <td colspan="11"><i>This work (trained on 63 known + 12 new languages)</i></td>
  </tr>
  <tr>
    <td>SONAR-75</td>
    <td>0.882</td>
    <td>0.909</td>
    <td>0.948</td>
    <td>0.913</td>
    <td>0.859</td>
    <td>9.0</td>
    <td>7.7</td>
    <td>5.8</td>
    <td>7.5</td>
    <td>12.6</td>
  </tr>
  <tr>
    <td>charSONAR-75</td>
    <td><b>0.889</b></td>
    <td><b>0.914</b></td>
    <td><b>0.949</b></td>
    <td><b>0.917</b></td>
    <td><b>0.863</b></td>
    <td>8.4</td>
    <td>7.2</td>
    <td><b>5.5</b></td>
    <td><b>7.0</b></td>
    <td><b>12.3</b></td>
  </tr>
  <tr>
    <td>△</td>
    <td><b>0.007</b></td>
    <td>0.005</td>
    <td>0.001</td>
    <td>0.004</td>
    <td>0.004</td>
    <td><b>0.6</b></td>
    <td>0.5</td>
    <td>0.3</td>
    <td>0.5</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td colspan="11"><i>This work (trained only on the 63 known languages)</i></td>
  </tr>
  <tr>
    <td>SONAR-63</td>
    <td>0.882</td>
    <td>0.909</td>
    <td>0.947</td>
    <td>0.913</td>
    <td>0.517<sup>†</sup></td>
    <td>8.8</td>
    <td>7.7</td>
    <td>5.8</td>
    <td>7.5</td>
    <td>42.9<sup>†</sup></td>
  </tr>
  <tr>
    <td>charSONAR-63</td>
    <td><b>0.889</b></td>
    <td><b>0.914</b></td>
    <td><b>0.949</b></td>
    <td><b>0.917</b></td>
    <td><b>0.530</b><sup>†</sup></td>
    <td><b>8.3</b></td>
    <td><b>7.2</b></td>
    <td><b>5.5</b></td>
    <td><b>7.0</b></td>
    <td><b>42.2</b><sup>†</sup></td>
  </tr>
  <tr>
    <td>△</td>
    <td><b>0.007</b></td>
    <td>0.005</td>
    <td>0.002</td>
    <td>0.004</td>
    <td><b>0.013</b></td>
    <td><b>0.5</b></td>
    <td>0.5</td>
    <td>0.3</td>
    <td>0.5</td>
    <td>0.7</td>
  </tr>
</table>

Table 3: Translation COMET scores and cross-lingual similarity search xSIM++ error rates on FLORES devtest (X→Eng), grouped by All/Low/Med/High) and New languages. † indicates zero-shot evaluation. bold: best overall; underlined: best zero-shot. All models have the same number of parameters (1.3B). Δ refers to the difference between charSONAR-N and SONAR-N models.

<table>
  <tr>
    <th>Model</th>
    <th>BLEU</th>
    <th>chrF++</th>
  </tr>
  <tr>
    <td>NLLB-200</td>
    <td>17.4</td>
    <td>45.3</td>
  </tr>
  <tr>
    <td>SONAR-200</td>
    <td>15.6</td>
    <td>44.0</td>
  </tr>
  <tr>
    <td>SONAR-Eng</td>
    <td><b>15.9</b></td>
    <td><b>44.8</b></td>
  </tr>
  <tr>
    <td>charSONAR-Eng</td>
    <td>15.8</td>
    <td>44.7</td>
  </tr>
</table>

Table 4: Text translation (Eng→200) BLEU and chrF++ scores on FLORES devtest.

<table>
  <tr>
    <th>Model</th>
    <th># Tokens</th>
    <th>Inference Time (s)</th>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>49</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>charSONAR</td>
    <td>158 (×3.2)</td>
    <td>0.94 (×1.1)</td>
  </tr>
</table>

Table 5: Average number of tokens and average inference time in FLORES dev.

ing. To achieve better encoding for these unseen languages, we propose the use of family tokens according to the linguistic family subgroup of each language. Specifically, during training we replace the language token with the corresponding subgroup token with a 20% probability. On inference, we encode a new language, with the appropriate subgroup token. The subgroup tokens are trainable and are initialized from the average of all the language tokens of each family.10 In the last part of Table 3 we observe that charSONAR-63 can generalize better than a subword-based encoder to languages not seen during training, achieving an improvement of 0.013 points in COMET and 0.7 in xSIM++. We also notice a sharp increase for both our encoders, compared to original SONAR-200, showing the benefits of expanding language tokens to subgroup tokens.11

5.4 Are the gains due to language transfer or more compute?

An implicit side-effect of character-level modeling is that sequences are on average 3× longer, which means that the charSONAR encoder is using more FLOPs than the SONAR encoder. To further investigate the source of the advantage shown in Table 3 we conduct an experiment where we train SONAR and charSONAR on only one language, specifically on English. The results of Table 4 show that in the single-language setting, there is no advantage for the character-based model, being slightly behind the subword-based one. This finding indicates that character-level modeling is beneficial due to better cross-lingual knowledge transfer, rather than due to increased compute.

5.5 Efficiency Analysis

To assess the degree of computational overhead due to the longer sequences, we measure the average inference time for the charSONAR and SONAR models in FLORES dev using a batch size of 1. The results of Table 5 show that although sequences

10Information about the linguistic families are available at Table 10 in the Appendix.
11For encoding a new language with SONAR-200 and NLLB-200, we do not use a language tag.
are 3.2× longer for charSONAR, the inference time is only 1.1× longer. This is due to the encoder bottleneck, which decouples the decoder from the source sequence length. Results with batching are available in Table 16 in the Appendix.

6 Speech Results

To investigate the cross-modal benefits of character-level modeling, we present results in zero-shot speech translation and speech-text similarity search with our charSONAR-based speech encoder.

6.1 Initial Exploration

In Table 6 we present zero-shot ST X→Eng results in FLEURS dev for four languages of different families and of varying degree of resourcefulness, ranging from 3K examples (Estonian) to 330K (Spanish). We observe that the pretrained cross-modal adapter (PRETR), despite being significantly smaller, outperforms the large (\( d_h = 1024 \)), yet randomly initialized, adapter (RND). Although for the high-resource Spanish, we notice that the difference is rather small, which indicates that with more data it can be beneficial to increase the capacity. Indeed, our proposed dual adapter (DUAL), with large dimensionality in the random branch, surpasses them both. Finally, we notice further gains when we switch to a robust charSONAR version that was trained with ASR-like Norm/Noise augmentations (\S 3.2).

<table>
  <tr>
    <th rowspan="2">Adapter</th>
    <th rowspan="2">Encoder</th>
    <th colspan="5">COMET</th>
  </tr>
  <tr>
    <th>Type</th>
    <th>Dim</th>
    <th>Train</th>
    <th>Norm/Noise</th>
    <th>Est</th>
    <th>Rus</th>
    <th>Tur</th>
    <th>Spa</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>PRETR</td>
    <td>~64</td>
    <td>x</td>
    <td>x/x</td>
    <td></td>
    <td>0.845 0.849</td>
    <td>0.828 0.837</td>
    <td>0.840</td>
  </tr>
  <tr>
    <td>PRETR</td>
    <td>~64</td>
    <td>✓</td>
    <td>x/x</td>
    <td></td>
    <td>0.901 0.910</td>
    <td>0.877 0.890</td>
    <td>0.894</td>
  </tr>
  <tr>
    <td>RND</td>
    <td>256</td>
    <td>✓</td>
    <td>x/x</td>
    <td></td>
    <td>0.837 0.872</td>
    <td>0.831 0.878</td>
    <td>0.854</td>
  </tr>
  <tr>
    <td>RND</td>
    <td>1024</td>
    <td>✓</td>
    <td>x/x</td>
    <td></td>
    <td>0.882 0.889</td>
    <td>0.869 0.889</td>
    <td>0.882</td>
  </tr>
  <tr>
    <td>DUAL</td>
    <td>256</td>
    <td>✓</td>
    <td>x/x</td>
    <td></td>
    <td>0.914 0.912</td>
    <td>0.889 0.888</td>
    <td>0.901</td>
  </tr>
  <tr>
    <td>DUAL</td>
    <td>1024</td>
    <td>✓</td>
    <td>x/x</td>
    <td></td>
    <td>0.911 0.909</td>
    <td>0.894 0.905</td>
    <td>0.905</td>
  </tr>
  <tr>
    <td>DUAL</td>
    <td>1024</td>
    <td>✓</td>
    <td>x/x</td>
    <td></td>
    <td>0.914 0.910</td>
    <td>0.891 0.897</td>
    <td>0.903</td>
  </tr>
  <tr>
    <td>DUAL</td>
    <td>1024</td>
    <td>✓</td>
    <td>x/x</td>
    <td></td>
    <td>0.915 0.923</td>
    <td>0.906 0.905</td>
    <td>0.912</td>
  </tr>
</table>

Table 6: Ablations in speech adaptation. Speech Translation (X→Eng) results on FLEURS dev.

6.2 Zero-shot Speech Translation

Next, we train adapters for 33 languages using the charSONAR-75 encoder, and compare against strong supervised E2E models, Whisper (Radford et al., 2022) and SeamlessM4T (SEAMLESS, 2025), cascades with MMS/Whisper and NLLB/SONAR, and our own cascades with SONAR-75/charSONAR-75. We report results by grouping languages according to number of examples in CommonVoice (CV) (Table 7). The first version of our system (11) using the pretrained adapter, can work out-of-the-box and without any training, even outperforming Whisper by a large margin, and particularly for low-resource languages. This indicates that the input space of our character-based encoder is fully compatible with the output space of MMS given the initialization of our adapter. Following, by training this adapter with ASR data (12), we surpass the previous state-of-the-art SeamlessM4T-Large-v2. The benefits of cross-modal transfer from charSONAR are evident for some extremely low-resource languages such as Asturian, where with only 400 examples it surpasses SeamlessM4T by 0.1 COMET (Table 12 in Appendix). Furthermore, we observe additional gains when using the proposed dual adapter (13) for medium/high-resource languages, where there are enough data to learn the large, but randomly initialized, branch. Finally, we show that by adding only 2K additional examples from FLEURS train, we can achieve further important gains across all categories (14-15).

Apart from the strong cross-modal transfer showcased by our speech adaptation of charSONAR, significant gains are also observed for the cascade systems that employ it. Specifically a cascade of MMS and charSONAR (9) outperforms all other cascades (3-8) and is on par with SeamlessM4T-Large-v2 in low/medium-resource settings.

6.3 Similarity Search

In Table 8 we present results on cross-lingual and cross-modal similarity search on FLEURS test. We compare our character-based speech encoder against several cascades and the original SONAR speech encoders (Duquenne et al., 2023) that are based on w2v-BERT (Chung et al., 2021), and thus do not transfer knowledge from an acoustic model (MMS) nor the text modality (charSONAR). We observe that our minimal adapters trained on CV outperform the previous SONAR speech encoders and all cascades apart from the charSONAR-based one. Still, by using more data (FLEURS train) and/or more parameters (dual adapter), the proposed encoder surpasses the charSONAR cascades.

6.4 Adapters for Subword-based Encoders

To understand how essential is the character-based encoder in our proposed speech adaptation, we experiment with replacing it with a subword-based
<table>
  <tr>
    <th rowspan="2">id</th>
    <th rowspan="2">Model</th>
    <th colspan="1">Text</th>
    <th colspan="1">Encoder Tokenization</th>
    <th colspan="1">Total Params</th>
    <th colspan="1">Adapter Train Params</th>
    <th colspan="1">Adapter Train Data</th>
    <th colspan="4">Low (11)</th>
    <th colspan="4">Med (11)</th>
    <th colspan="4">High (11)</th>
    <th colspan="4">All (33)</th>
  </tr>
  <tr>
    <th>Low</th>
    <th>Med</th>
    <th>High</th>
    <th>All</th>
    <th>Low</th>
    <th>Med</th>
    <th>High</th>
    <th>All</th>
    <th>Low</th>
    <th>Med</th>
    <th>High</th>
    <th>All</th>
    <th>Low</th>
    <th>Med</th>
    <th>High</th>
    <th>All</th>
  </tr>
  <tr>
    <td colspan="19"><b>Supervised E2E ST (previous)</b></td>
  </tr>
  <tr>
    <td>1</td>
    <td><b>WHISPER-LARGE-v3</b></td>
    <td>/</td>
    <td>1.5B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.598</td>
    <td>0.754</td>
    <td>0.790</td>
    <td>0.714</td>
  </tr>
  <tr>
    <td>2</td>
    <td><b>SEAMLESSM4T-LARGE-v2</b></td>
    <td>/</td>
    <td>2.3B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.829</td>
    <td>0.889</td>
    <td>0.901</td>
    <td>0.873</td>
  </tr>
  <tr>
    <td colspan="19"><b>Cascade ST (previous)</b></td>
  </tr>
  <tr>
    <td>3</td>
    <td>MMS + NLLB-200</td>
    <td>subwords</td>
    <td>2.3B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.786</td>
    <td>0.834</td>
    <td>0.822</td>
    <td>0.814</td>
  </tr>
  <tr>
    <td>4</td>
    <td><b>WHISPER + NLLB-200</b></td>
    <td>subwords</td>
    <td>2.8B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.717</td>
    <td>0.870</td>
    <td>0.863</td>
    <td>0.817</td>
  </tr>
  <tr>
    <td>5</td>
    <td>MMS + SONAR-200</td>
    <td>subwords</td>
    <td>2.3B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.757</td>
    <td>0.839</td>
    <td>0.824</td>
    <td>0.807</td>
  </tr>
  <tr>
    <td>6</td>
    <td><b>WHISPER + SONAR-200</b></td>
    <td>subwords</td>
    <td>2.8B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.684</td>
    <td>0.869</td>
    <td>0.861</td>
    <td>0.804</td>
  </tr>
  <tr>
    <td colspan="19"><b>Cascade ST (ours)</b></td>
  </tr>
  <tr>
    <td>7</td>
    <td>MMS + SONAR-75</td>
    <td>subwords</td>
    <td>2.3B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.811</td>
    <td>0.870</td>
    <td>0.854</td>
    <td>0.845</td>
  </tr>
  <tr>
    <td>8</td>
    <td><b>WHISPER + SONAR-75</b></td>
    <td>subwords</td>
    <td>2.8B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.721</td>
    <td>0.871</td>
    <td>0.865</td>
    <td>0.819</td>
  </tr>
  <tr>
    <td>9</td>
    <td>MMS + charSONAR-75</td>
    <td>chars</td>
    <td>2.3B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.833</td>
    <td>0.889</td>
    <td>0.875</td>
    <td>0.866</td>
  </tr>
  <tr>
    <td>10</td>
    <td><b>WHISPER + charSONAR-75</b></td>
    <td>chars</td>
    <td>2.8B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>0.755</td>
    <td>0.893</td>
    <td>0.882</td>
    <td>0.843</td>
  </tr>
  <tr>
    <td colspan="19"><b>Zero-shot E2E ST (ours)</b></td>
  </tr>
  <tr>
    <td>11</td>
    <td>Speech-charSONAR-75 - PreTr</td>
    <td>chars</td>
    <td>2.3B</td>
    <td>0</td>
    <td>/</td>
    <td>/</td>
    <td>0.772</td>
    <td>0.833</td>
    <td>0.831</td>
    <td>0.812</td>
  </tr>
  <tr>
    <td>12</td>
    <td>Speech-charSONAR-75 - PreTr</td>
    <td>chars</td>
    <td>2.3B</td>
    <td>0.2M</td>
    <td>CV</td>
    <td>/</td>
    <td>0.837</td>
    <td>0.893</td>
    <td>0.894</td>
    <td>0.875</td>
  </tr>
  <tr>
    <td>13</td>
    <td>Speech-charSONAR-75 - DUAL</td>
    <td>chars</td>
    <td>2.3B</td>
    <td>2.5M</td>
    <td>CV</td>
    <td>/</td>
    <td>0.615</td>
    <td>0.899</td>
    <td>0.902</td>
    <td>0.805</td>
  </tr>
  <tr>
    <td>14</td>
    <td>Speech-charSONAR-75 - PreTr</td>
    <td>chars</td>
    <td>2.3B</td>
    <td>0.2M</td>
    <td>CV+FLEURS</td>
    <td>/</td>
    <td>0.852</td>
    <td>0.900</td>
    <td>0.901</td>
    <td>0.884</td>
  </tr>
  <tr>
    <td>15</td>
    <td><b>Speech-charSONAR-75 - DUAL</b></td>
    <td>chars</td>
    <td>2.3B</td>
    <td>2.5M</td>
    <td>CV+FLEURS</td>
    <td>/</td>
    <td><b>0.853</b></td>
    <td><b>0.906</b></td>
    <td><b>0.910</b></td>
    <td><b>0.889</b></td>
  </tr>
</table>

Table 7: Speech Translation (33 → Eng) COMET scores on FLEURS test. Low/Med/High each contain 11 languages, according to amount of Common Voice data. Underlined are the previous best scores. Highlighted are our scores with char-based models that are at least on par with the previous best. In bold are best overall.

<table>
  <tr>
    <th>Model</th>
    <th>avg26</th>
    <th>avg33</th>
  </tr>
  <tr>
    <td>SONAR Speech (<a href="https://arxiv.org/abs/2303.16628">Duquenne et al., 2023</a>)</td>
    <td>14.3</td>
    <td>/</td>
  </tr>
  <tr>
    <td>MMS + SONAR-200</td>
    <td>15.7</td>
    <td>17.2</td>
  </tr>
  <tr>
    <td>MMS + SONAR-75</td>
    <td>12.7</td>
    <td>13.8</td>
  </tr>
  <tr>
    <td>MMS + charSONAR-75</td>
    <td>10.7</td>
    <td>11.5</td>
  </tr>
  <tr>
    <td>Speech-charSONAR-75 - PreTr</td>
    <td>11.6</td>
    <td>12.9</td>
  </tr>
  <tr>
    <td>→ w/ FLEURS train</td>
    <td>10.0</td>
    <td>10.7</td>
  </tr>
  <tr>
    <td>→ w/ DUAL</td>
    <td><b>9.4</b></td>
    <td><b>10.2</b></td>
  </tr>
</table>

Table 8: Cross-modal and cross-lingual retrieval-xSIM++ error rates (.) on FLEURS test (X→Eng). avg26 is the languages supported by SONAR Speech (<a href="https://arxiv.org/abs/2303.16628">Duquenne et al., 2023</a>) and our models.

<table>
  <tr>
    <th>Model</th>
    <th>Oci # 0.3k</th>
    <th>Est # 3k</th>
    <th>Tur # 30k</th>
    <th>Spa # 330k</th>
  </tr>
  <tr>
    <td>Speech-SONAR-75 - RND</td>
    <td>0.199</td>
    <td>0.223</td>
    <td>0.795</td>
    <td>0.841</td>
  </tr>
  <tr>
    <td>Speech-charSONAR-75 - RND</td>
    <td>0.202</td>
    <td>0.877</td>
    <td>0.868</td>
    <td>0.912</td>
  </tr>
  <tr>
    <td>Speech-charSONAR-75 - PreTr</td>
    <td><b>0.795</b></td>
    <td>0.910</td>
    <td>0.885</td>
    <td>0.917</td>
  </tr>
  <tr>
    <td>Speech-charSONAR-75 - DUAL</td>
    <td>0.707</td>
    <td><b>0.912</b></td>
    <td><b>0.903</b></td>
    <td><b>0.920</b></td>
  </tr>
</table>

Table 9: Speech Translation COMET (X→Eng) on FLEURS test for subword-based vs char-based encoders with adapters. Underlined: best among RND adapters; bold: best overall; #: ASR examples.

one. To achieve this we mean-pool the indices of the compressed acoustic representation of MMS that belong to the same subword, as predicted by the CTC. The pretrained adapter version is not possible in this setting and thus we experiment only with the randomly-initialized adapter. The results of Table 9 indicate that we can learn an adapter to connect MMS and (subword-based) SONAR, although the quality is limited, and only works in high resource settings (Turkish, Spanish), while still being several points behind the character-based model. This highlights both the data-efficiency and cross-modal adaptability of our proposed method.

7 Conclusions

We presented a methodology based on character-level modeling that increases cross-lingual transfer and cross-modal transfer in text and speech tasks. For text, our character-based encoder surpasses comparable subword-based encoders, especially in low-resource settings, while exhibiting better zero-shot generalization to unseen languages. For speech, our proposed minimal adapter seamlessly connects an ASR CTC encoder to our character-based encoder, surpassing previous state-of-the-art models. Furthermore it requires minimal supervision from ASR data, and can even work out-of-the-box without any training, surpassing models like Whisper. Future research will focus on target-side cross-lingual and cross-modal transfer, and expanding to more languages.
Limitations

In this work we focused on source-side cross-lingual and cross-modal transfer, leaving target-side transfer for future research. We hypothesize that character-level modeling can be beneficial for target-side, although decoding on the character-level can be problematic and relatively more inefficient than encoding on the character-level. We still believe this is an interesting direction for future work.

Furthermore, we decided to focus on adapting a specific model, SONAR, to work with character-level input. Although the encoder bottleneck reduces the computational overhead in generation, and allowed us to simplify the teacher-student training by using an MSE objective, it also reduces the capacity of the model. We hypothesize that similar gains can be achieved by adapting a traditional encoder-decoder, like NLLB (NLLB, 2024), to work with characters, either by back-propagating the translation signal through the (frozen) decoder or using similar objectives to ZeroSwot (Tsiamas et al., 2024). Also, as discussed in our Relevant Research (§2), we did not experiment with any specific architectural changes in the encoder that are better suited for character-level modeling (Clark et al., 2022; Tay et al., 2022; Pagnoni et al., 2024), as we aimed to study character-based vs subword-based modeling within the same architecture. We believe that by using such techniques further gains in performance and efficiency can be achieved.

Additionally, our proposed methodology for speech adaptation is limited by the language-specific CTC layers of MMS. This forced us to train language-specific cross-modal adapters, which does not allow the speech encoder to generalize to more languages, other than the ones for which we have ASR data. To go around this issue, we carried some experiments with the zero-shot version of MMS (Zhao et al., 2024) that uses a unified model for all languages, but due to decreased ASR quality compared to MMS-1B (Pratap et al., 2023), translation quality was also lagging behind. Still, in the future, and given a supervised MMS-like acoustic model with a unified architecture, our proposed cross-modal adapter could enable generalized speech understand and translation with it.

Acknowledgments

We thank Pere Lluís Huguet Cabot for his helpful comments and suggestions.

References

Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common Voice: A Massively-Multilingual Speech Corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218–4222, Marseille, France. European Language Resources Association.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems, volume 33, pages 12449–12460. Curran Associates, Inc.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.

Kris Cao. 2023. What is the best recipe for character-level encoder-only modelling? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5924–5938, Toronto, Canada. Association for Computational Linguistics.

Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024. When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074–4096, Miami, Florida, USA. Association for Computational Linguistics.

Mingda Chen, Kevin Heffernan, Onur Çelebi, Alexandre Mouchachko, and Holger Schwenk. 2023. xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 101–109, Toronto, Canada. Association for Computational Linguistics.

Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. Revisiting Character-Based Neural Machine Translation with Capacity and Compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4295–4305, Brussels, Belgium. Association for Computational Linguistics.

Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A Character-level Decoder without Explicit Segmentation for Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1693–1703, Berlin, Germany. Association for Computational Linguistics.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250.

Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. Transactions of the Association for Computational Linguistics, 10:73–91.

Alexis Conneau, Min Ma, Simran Khanduja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. arXiv preprint arXiv:2205.12446.

Marta R. Costa-jussà and José A. R. Fonollosa. 2016. Character-based Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 357–361, Berlin, Germany. Association for Computational Linguistics.

David Dale and Marta R. Costa-jussà. 2024. BLASER 2.0: A Metric for Evaluation and Quality Estimation of Massively Multilingual Speech and Text Translation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16075–16085, Miami, Florida, USA. Association for Computational Linguistics.

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. SONAR: Sentence-Level Multimodal and Language-Agnostic Representations. Preprint, arXiv:2308.11466.

Lukas Edman, Gabriele Sarti, Antonio Toral, Gertjan van Noord, and Arianna Bisazza. 2024. Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation. Transactions of the Association for Computational Linguistics, 12:392–410.

Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMm: Self-learning with Speech-text Manifold Mixup for Speech Translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7050–7062, Dublin, Ireland. Association for Computational Linguistics.

Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021. CTC-based Compression for Direct Speech Translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 690–696, Online. Association for Computational Linguistics.

Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, page 369–376, New York, NY, USA. Association for Computing Machinery.

Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection. Transactions of the Association for Computational Linguistics, 12:979–995.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790–2799. PMLR.

Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. Transactions of the Association for Computational Linguistics, 5:339–351.

Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. MADLAD-400: A Multilingual And Document-Level Large Audited Dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023. Pre-training for Speech Translation: CTC Meets Optimal Transport. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.

Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully Character-Level Neural Machine Translation without Explicit Segmentation. Transactions of the Association for Computational Linguistics, 5:365–378.

Jindřich Libovický, Helmut Schmid, and Alexander Fraser. 2022. Why don’t people use character-level machine translation? In Findings of the Association for Computational Linguistics: ACL 2022, pages 2470–2485, Dublin, Ireland. Association for Computational Linguistics.

Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015. Character-based Neural Machine Translation. Preprint, arXiv:1511.04586.

Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. Bridging the Modality Gap for Speech-to-Text Translation. Preprint, arXiv:2010.14920.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations.

NLLB. 2024. Scaling Neural Machine Translation to 200 Languages. Nature, 630:841–846.

Siqi Ouyang, Rong Ye, and Lei Li. 2023. WACO: Word-Aligned Contrastive Learning for Speech Translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3891–3907, Toronto, Canada. Association for Computational Linguistics.

Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. 2024. Byte Latent Transformer: Patches Scale Better Than Tokens. Preprint, arXiv:2412.09871.

Gabriel Peyr´e and Marco Cuturi. 2019. Computational Optimal Transport: With Applications to Data Science. Foundations and Trends® in Machine Learning, 11(5-6):355–607.

Maja Popovi´c. 2017. chrF++: Words Helping Character N-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.

Matt Post. 2018. A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels. Association for Computational Linguistics.

Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023. Scaling Speech Technology to 1,000+ Languages. Preprint, arXiv:2305.13516.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-Scale Weak Supervision. Preprint, arXiv:2212.04356.

SEAMLESS. 2025. Joint Speech and Text Machine Translation for up to 100 Languages. Nature, 637:587–593.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, page 3104–3112, Cambridge, MA, USA. MIT Press.

Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021a. Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4252–4261, Online. Association for Computational Linguistics.

Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 2021b. A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6209–6213.

Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. In International Conference on Learning Representations.

Ioannis Tsiamas, Gerard I. G´allego, Jos´e A. R. Fonollosa, and Marta R. Costa-juss´a. 2024. Pushing the Limits of Zero-shot End-to-End Speech Translation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14245–14267, Bangkok, Thailand. Association for Computational Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharar Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291–306.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.

Jichen Yang, Kai Fan, Minpeng Liao, Boxing Chen, and Zhongqiang Huang. 2023. Towards Zero-shot Learning for End-to-end Cross-modal Translation Models. In Findings of the Association for Computational
Linguistics: EMNLP 2023, pages 13078–13087, Singapore. Association for Computational Linguistics.

Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to-End Speech Translation via Cross-Modal Progressive Training. In Interspeech 2021, pages 2267–2271.

Rong Ye, Mingxuan Wang, and Lei Li. 2022. Cross-modal Contrastive Learning for Speech Translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5099–5113, Seattle, United States. Association for Computational Linguistics.

Yuhao Zhang, Chen Xu, Bei Li, Hao Chen, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2023. Rethinking and Improving Multi-task Learning for End-to-end Speech Translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10753–10765, Singapore. Association for Computational Linguistics.

Jinning Zhao, Vineel Pratap, and Michael Auli. 2024. Scaling A Simple Approach to Zero-Shot Speech Recognition. Preprint, arXiv:2407.17852.

A Data

In Table 10 we provide details about the languages and the amounts of data used in our experiments. The numbers for the MT data indicate the amount after filtering with BLASER 2.0 (Dale and Costa-jussà, 2024). The resourcefulness label (low, medium, or high) of each language is separate for each modality, and indicates in which of the three percentile of the data distribution it belongs.

B Additional Results

In Table 11 we present the per-language text translation results for the models of Table 3.

In Table 12, we present the per-language speech translation results for some of the models of Table 7. We also add results from the SONAR Speech Encoders (Duquenne et al., 2023), which were excluded from the main table since they do not support all the languages with which we experiment. Furthermore, Table 12 includes more than 33 languages, since for ease of presentation, in the main results we presented the ones that were both supported by Whisper and our models. We indicate the languages not taken into account for the results of Table 7.

In Tables 14 and 13, we present the per linguistic subgroup and script results of our text encoders in translation and cross-lingual similarity search. We observe that for known languages, the charSONAR encoder outperforms the subword-based encoder in all categories, apart from the single group that contains the Greek language, and only in cross-lingual similarity search. For the new languages, we notice that charSONAR performs better in all categories for translation, but the subword-based model is better for the Turkic and Uralic subgrouping, and Cyrillic script in cross-lingual similarity search.

In Table 15 we present the text translation results for the three encoders that were used in the initial exploration with the four languages for the speech adaptation (Table 6). We used the Uralic/Cyrillic encoder for Estonian and Russian, the Turkic for Turkish, and the Romance for Spanish.

In Figure 3 we present our ablation for deciding the BLASER 2.0 filtering threshold. To speed-up experimentation and use less data, we filtered with 4.5 for the initial exploration, but for the main experiments we used a threshold of 4.

Finally, in Table 16 we provide an efficiency analysis for SONAR and charSONAR models, similar to the results of §5.5, but now with batching. We use length-based bucketing and a batch size of 5K tokens, which results in 8 batches for SONAR and 31 batches for charSONAR. The results here confirm the findings of Table 5, showing that the impact of the char-based tokenization is minimal with respect to the additional computational overhead.

![COMET scores vs. BLASER 2.0 filtering threshold for charSONAR and SONAR in FLORES dev. Results with the Ural/Cyril group of 15 languages. COMET scores are average of X→Eng for all the 15 languages in the group.](page_1012_180_410_312.png)

Figure 3: COMET scores vs. BLASER 2.0 filtering threshold for charSONAR and SONAR in FLORES dev. Results with the Ural/Cyril group of 15 languages. COMET scores are average of X→Eng for all the 15 languages in the group.
<table>
  <tr>
    <th>Language</th>
    <th>Code</th>
    <th>FLORES+</th>
    <th>MMS</th>
    <th>Family</th>
    <th>Subgrouping</th>
    <th>Script</th>
    <th>MT # (M) / Resource</th>
    <th>ASR # (K) / Resource</th>
  </tr>
  <tr><td>Aragonese</td><td>arg_Latin</td><td>✓</td><td>✗</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.1 new</td><td>0.0 -</td></tr>
  <tr><td>Asturian</td><td>ast_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.2 low</td><td>0.4 -</td></tr>
  <tr><td>Awadhi</td><td>awa_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.4 low</td><td>0.0 -</td></tr>
  <tr><td>South Azerbaijani</td><td>azb_Arab</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Arabic</td><td>0.3 low</td><td>0.0 -</td></tr>
  <tr><td>North Azerbaijani</td><td>azj_Latin</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Latin</td><td>9.4 med</td><td>0.1 low</td></tr>
  <tr><td>Bashkir</td><td>bak_Cyril</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Cyrillic</td><td>1.7 med</td><td>119.2 -</td></tr>
  <tr><td>Belarusian</td><td>bel_Cyril</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Cyrillic</td><td>11.5 med</td><td>347.6 high</td></tr>
  <tr><td>Bhojpuri</td><td>bho_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>1.8 low</td><td>0.0 -</td></tr>
  <tr><td>Bosnian</td><td>bos_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>21.8 high</td><td>0.0 -</td></tr>
  <tr><td>Boro</td><td>brx_Deva</td><td>✓</td><td>✓</td><td>Sino-Tibetan</td><td>Tibeto-Burman</td><td>Devanagari</td><td>0.1 new</td><td>0.0 -</td></tr>
  <tr><td>Bulgarian</td><td>bul_Cyril</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Cyrillic</td><td>39.3 high</td><td>4.8 med</td></tr>
  <tr><td>Catalan</td><td>cat_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>10.1 med</td><td>1146.2 high</td></tr>
  <tr><td>Valencian</td><td>cat_Latin_valle1252</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.0 new</td><td>0.0 -</td></tr>
  <tr><td>Czech</td><td>ces_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>52.3 high</td><td>20.1 med</td></tr>
  <tr><td>Chuvash</td><td>chv_Cyril</td><td>✓</td><td>✓</td><td>Turkic</td><td>Oghuric</td><td>Cyrillic</td><td>1.2 new</td><td>1.4 -</td></tr>
  <tr><td>Central Kurdish</td><td>ckr_Arab</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Iranian</td><td>Arabic</td><td>0.7 med</td><td>7.7 -</td></tr>
  <tr><td>Crimson Jatari</td><td>crh_Arab</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Latin</td><td>0.2 low</td><td>0.0 -</td></tr>
  <tr><td>Dogri</td><td>dgo_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.1 new</td><td>0.0 -</td></tr>
  <tr><td>Greek</td><td>ell_Grek</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Greco-Phrygian</td><td>Greek</td><td>52.6 high</td><td>1.9 low</td></tr>
  <tr><td>Estonian</td><td>est_Latin</td><td>✓</td><td>✓</td><td>Uralic</td><td>Finnic</td><td>Latin</td><td>16.9 high</td><td>3.2 med</td></tr>
  <tr><td>Finnish</td><td>fin_Latin</td><td>✓</td><td>✓</td><td>Uralic</td><td>Finnic</td><td>Latin</td><td>32.6 high</td><td>2.1 low</td></tr>
  <tr><td>French</td><td>fra_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>144.9 high</td><td>558.1 high</td></tr>
  <tr><td>Friulian</td><td>fur_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.2 low</td><td>0.0 -</td></tr>
  <tr><td>Galician</td><td>glg_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>8.3 med</td><td>25.2 high</td></tr>
  <tr><td>Konkani</td><td>gom_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.1 new</td><td>0.0 -</td></tr>
  <tr><td>Hindi</td><td>hin_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>35.6 high</td><td>4.7 med</td></tr>
  <tr><td>Chhattisgarhi</td><td>hne_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.3 low</td><td>0.0 -</td></tr>
  <tr><td>Croatian</td><td>hrv_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>17.1 high</td><td>0.0 -</td></tr>
  <tr><td>Hungarian</td><td>huu_Latin</td><td>✓</td><td>✓</td><td>Uralic</td><td>Italic</td><td>Latin</td><td>32.6 high</td><td>37.1 high</td></tr>
  <tr><td>Italian</td><td>ita_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>95.7 high</td><td>169.8 high</td></tr>
  <tr><td>Karikalap</td><td>kac_Latin</td><td>✓</td><td>✓</td><td>Uralic</td><td>Kipchak</td><td>Latin</td><td>0.9 new</td><td>0.0 -</td></tr>
  <tr><td>Kashmiri</td><td>kas_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.1 new</td><td>0.0 -</td></tr>
  <tr><td>Kazakh</td><td>kaz_Cyril</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Cyrillic</td><td>5.6 med</td><td>0.5 low</td></tr>
  <tr><td>Halt Mongolian</td><td>khk_Cyril</td><td>✓</td><td>✓</td><td>Mongolic-Khitian</td><td>Mongolic</td><td>Cyrillic</td><td>0.5 low</td><td>2.2 low</td></tr>
  <tr><td>Kyrgyz</td><td>kir_Cyril</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Cyrillic</td><td>2.7 med</td><td>1.8 -</td></tr>
  <tr><td>Northern Kurdish</td><td>kmr_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Iranian</td><td>Latin</td><td>0.7 med</td><td>5.1 -</td></tr>
  <tr><td>Ligurian</td><td>lij_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.2 low</td><td>1.6 -</td></tr>
  <tr><td>Lithuanian</td><td>lit_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>14.0 high</td><td>7.3 high</td></tr>
  <tr><td>Lombard</td><td>lmo_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.3 low</td><td>0.0 -</td></tr>
  <tr><td>Latgalian</td><td>lag_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>0.3 low</td><td>0.7 -</td></tr>
  <tr><td>Standard Latvian</td><td>lvv_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>2.8 med</td><td>11.4 med</td></tr>
  <tr><td>Magahi</td><td>mag_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.3 low</td><td>0.0 -</td></tr>
  <tr><td>Maithili</td><td>mai_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.4 low</td><td>0.0 -</td></tr>
  <tr><td>Marathi</td><td>mar_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>11.8 med</td><td>2.2 low</td></tr>
  <tr><td>Meadow Mari</td><td>mhr_Cyril</td><td>✓</td><td>✓</td><td>Uralic</td><td>Finnic-Ugric</td><td>Cyrillic</td><td>0.4 new</td><td>185.9 -</td></tr>
  <tr><td>Macedonian</td><td>mkl_Cyril</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Cyrillic</td><td>6.8 med</td><td>1.7 low</td></tr>
  <tr><td>Erzya</td><td>myr_Cyril</td><td>✓</td><td>✓</td><td>Uralic</td><td>Mordvinic</td><td>Cyrillic</td><td>0.1 new</td><td>1.2 -</td></tr>
  <tr><td>Nepali</td><td>npi_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>4.5 med</td><td>0.3 low</td></tr>
  <tr><td>Occitan</td><td>oci_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.2 low</td><td>0.3 low</td></tr>
  <tr><td>Araneze</td><td>oci_Latin_aran1260</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.0 new</td><td>0.0 -</td></tr>
  <tr><td>Southern Pashto</td><td>pbt_Arab</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Iranian</td><td>Arabic</td><td>0.9 med</td><td>0.0 -</td></tr>
  <tr><td>Western Persian</td><td>pes_Arab</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Iranian</td><td>Arabic</td><td>15.0 high</td><td>28.9 high</td></tr>
  <tr><td>Polish</td><td>pol_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>60.4 high</td><td>20.7 med</td></tr>
  <tr><td>Portuguese</td><td>por_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>11.6 high</td><td>22.0 med</td></tr>
  <tr><td>Dari</td><td>prc_Arab</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Iranian</td><td>Arabic</td><td>0.9 med</td><td>0.0 -</td></tr>
  <tr><td>Romanian</td><td>ron_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>59.9 high</td><td>5.1 med</td></tr>
  <tr><td>Russian</td><td>rus_Cyril</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Cyrillic</td><td>89.0 high</td><td>26.4 high</td></tr>
  <tr><td>Sanskrit</td><td>san_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.3 low</td><td>0.0 -</td></tr>
  <tr><td>Sicilian</td><td>scn_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.2 low</td><td>0.0 -</td></tr>
  <tr><td>Slovak</td><td>sik_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>29.7 high</td><td>3.3 med</td></tr>
  <tr><td>Slovenian</td><td>slv_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>20.8 high</td><td>1.4 low</td></tr>
  <tr><td>Sindhi</td><td>snd_Deva</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Indo-Aryan</td><td>Devanagari</td><td>0.0 new</td><td>0.0 -</td></tr>
  <tr><td>Spanish</td><td>spa_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>202.0 high</td><td>336.8 high</td></tr>
  <tr><td>Sardinian</td><td>srd_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.2 low</td><td>0.5 -</td></tr>
  <tr><td>Serbian</td><td>srp_Cyril</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Cyrillic</td><td>5.5 med</td><td>1.9 low</td></tr>
  <tr><td>Silesian</td><td>szl_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Latin</td><td>0.4 low</td><td>0.0 -</td></tr>
  <tr><td>Tatar</td><td>tat_Cyril</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Cyrillic</td><td>2.1 med</td><td>9.3 -</td></tr>
  <tr><td>Tajik</td><td>tjk_Cyril</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Iranian</td><td>Cyrillic</td><td>1.1 med</td><td>0.0 -</td></tr>
  <tr><td>Turkmen</td><td>tuk_Latin</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Latin</td><td>0.6 low</td><td>0.8 -</td></tr>
  <tr><td>Turkish</td><td>tur_Latin</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Latin</td><td>47.9 high</td><td>35.1 high</td></tr>
  <tr><td>Tuvan</td><td>tyv_Cyril</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Cyrillic</td><td>0.2 new</td><td>0.0 -</td></tr>
  <tr><td>Uyghur</td><td>uig_Arab</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Arabic</td><td>0.8 med</td><td>9.7 -</td></tr>
  <tr><td>Ukrainian</td><td>ukr_Cyril</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Balto-Slavic</td><td>Cyrillic</td><td>12.2 med</td><td>25.1 med</td></tr>
  <tr><td>Northern Uzbek</td><td>uzn_Latin</td><td>✓</td><td>✓</td><td>Turkic</td><td>Common Turkic</td><td>Latin</td><td>4.1 med</td><td>48.5 high</td></tr>
  <tr><td>Venetian</td><td>vec_Latin</td><td>✓</td><td>✓</td><td>Indo-European</td><td>Italic</td><td>Latin</td><td>0.2 low</td><td>0.0 -</td></tr>
</table>

Table 10: Details about the languages used in our experiments.
<table>
  <tr>
    <th></th>
    <th>NLLB-200</th>
    <th>SONAR-200</th>
    <th>SONAR-63</th>
    <th>charSONAR-63</th>
    <th>SONAR-75</th>
    <th>charSONAR-75</th>
  </tr>
  <tr><td>arg_Latn</td><td>0.867<sup>†</sup></td><td>0.860</td><td>0.910<sup>†</sup></td><td>0.912<sup>†</sup></td><td>0.931</td><td>0.917</td></tr>
  <tr><td>ast_Latn</td><td>0.918</td><td>0.909</td><td>0.920</td><td>0.928</td><td>0.920</td><td>0.927</td></tr>
  <tr><td>awa_Deva</td><td><b>0.918</b></td><td>0.861</td><td>0.894</td><td>0.896</td><td>0.898</td><td>0.898</td></tr>
  <tr><td>azb_Arab</td><td>0.632</td><td>0.513</td><td><b>0.698</b></td><td><b>0.698</b></td><td>0.681</td><td>0.696</td></tr>
  <tr><td>azi_Latin</td><td><b>0.910</b></td><td>0.741</td><td>0.863</td><td>0.868</td><td>0.865</td><td>0.870</td></tr>
  <tr><td>bak_Cyrl</td><td>0.916</td><td>0.900</td><td>0.915</td><td>0.920</td><td>0.916</td><td>0.918</td></tr>
  <tr><td>baq_Cyrl</td><td><b>0.925</b></td><td>0.865</td><td>0.922</td><td>0.926</td><td>0.920</td><td>0.926</td></tr>
  <tr><td>bbk_Deva</td><td>0.907</td><td>0.879</td><td>0.902</td><td><b>0.908</b></td><td>0.904</td><td>0.906</td></tr>
  <tr><td>bos_Latn</td><td><b>0.961</b></td><td>0.953</td><td>0.960</td><td>0.960</td><td>0.959</td><td>0.960</td></tr>
  <tr><td>brx_Deva*</td><td>0.208<sup>†</sup></td><td>0.203<sup>†</sup></td><td>0.217<sup>†</sup></td><td>0.216<sup>†</sup></td><td>0.841</td><td><b>0.851</b></td></tr>
  <tr><td>bul_Cyrl</td><td><b>0.954</b></td><td>0.952</td><td>0.952</td><td>0.953</td><td>0.952</td><td><b>0.954</b></td></tr>
  <tr><td>cat_Latin</td><td>0.954</td><td>0.949</td><td>0.952</td><td><b>0.956</b></td><td>0.952</td><td><b>0.956</b></td></tr>
  <tr><td>cat_Latin_vale1252</td><td>0.903<sup>†</sup></td><td>0.886<sup>†</sup></td><td>0.952<sup>†</sup></td><td>0.955<sup>†</sup></td><td>0.952<sup>†</sup></td><td>0.955<sup>†</sup></td></tr>
  <tr><td>ces_Latin</td><td>0.954</td><td>0.953</td><td>0.955</td><td><b>0.956</b></td><td>0.955</td><td><b>0.956</b></td></tr>
  <tr><td>chy_Cyrl</td><td>0.240<sup>†</sup></td><td>0.251<sup>†</sup></td><td>0.274<sup>†</sup></td><td>0.268<sup>†</sup></td><td>0.851</td><td><b>0.855</b></td></tr>
  <tr><td>ckb_Arab</td><td>0.881</td><td>0.872</td><td>0.882</td><td>0.889</td><td>0.884</td><td>0.890</td></tr>
  <tr><td>crh_Latin</td><td>0.907</td><td>0.890</td><td>0.909</td><td>0.920</td><td>0.914</td><td>0.918</td></tr>
  <tr><td>dgo_Deva*</td><td>0.656<sup>†</sup></td><td>0.575<sup>†</sup></td><td>0.735<sup>†</sup></td><td>0.596<sup>†</sup></td><td>0.899</td><td><b>0.899</b></td></tr>
  <tr><td>ell_Greek</td><td><b>0.940</b></td><td>0.932</td><td>0.938</td><td>0.938</td><td>0.937</td><td>0.939</td></tr>
  <tr><td>esg_Latin</td><td>0.940</td><td>0.935</td><td>0.942</td><td><b>0.945</b></td><td>0.942</td><td><b>0.945</b></td></tr>
  <tr><td>fin_Latin</td><td>0.942</td><td>0.936</td><td>0.945</td><td>0.946</td><td>0.944</td><td><b>0.947</b></td></tr>
  <tr><td>fra_Latin</td><td><b>0.965</b></td><td>0.961</td><td>0.961</td><td>0.961</td><td>0.959</td><td>0.961</td></tr>
  <tr><td>fur_Latin</td><td>0.931</td><td>0.930</td><td>0.937</td><td><b>0.938</b></td><td>0.936</td><td><b>0.938</b></td></tr>
  <tr><td>glg_Latin</td><td><b>0.957</b></td><td>0.951</td><td>0.954</td><td><b>0.957</b></td><td>0.954</td><td><b>0.957</b></td></tr>
  <tr><td>gom_Deva*</td><td>0.511<sup>†</sup></td><td>0.477<sup>†</sup></td><td>0.607</td><td>0.635<sup>†</sup></td><td>0.869</td><td><b>0.880</b></td></tr>
  <tr><td>hin_Deva</td><td><b>0.939</b></td><td>0.935</td><td>0.934</td><td>0.935</td><td>0.934</td><td>0.938</td></tr>
  <tr><td>hne_Deva</td><td>0.911</td><td>0.902</td><td>0.913</td><td>0.915</td><td>0.912</td><td><b>0.916</b></td></tr>
  <tr><td>hrv_Latin</td><td>0.940</td><td>0.949</td><td><b>0.954</b></td><td><b>0.954</b></td><td><b>0.954</b></td><td><b>0.954</b></td></tr>
  <tr><td>hum_Latin</td><td>0.946</td><td>0.940</td><td>0.945</td><td>0.948</td><td>0.945</td><td>0.947</td></tr>
  <tr><td>itl_Latin</td><td><b>0.957</b></td><td>0.954</td><td>0.951</td><td>0.954</td><td>0.952</td><td>0.954</td></tr>
  <tr><td>kau_Latin</td><td>0.349<sup>†</sup></td><td>0.359<sup>†</sup></td><td>0.698<sup>†</sup></td><td>0.767<sup>†</sup></td><td>0.917</td><td><b>0.924</b></td></tr>
  <tr><td>kas_Deva</td><td>0.657</td><td>0.600</td><td>0.674</td><td>0.704</td><td>0.681</td><td><b>0.710</b></td></tr>
  <tr><td>kaz_Cyrl</td><td><b>0.918</b></td><td>0.908</td><td>0.912</td><td>0.917</td><td>0.909</td><td>0.917</td></tr>
  <tr><td>khk_Cyrl</td><td>0.852</td><td>0.838</td><td>0.863</td><td>0.873</td><td>0.865</td><td><b>0.876</b></td></tr>
  <tr><td>kir_Cyrl</td><td>0.908</td><td>0.898</td><td>0.911</td><td><b>0.915</b></td><td>0.909</td><td><b>0.915</b></td></tr>
  <tr><td>kmr_Latin</td><td>0.784</td><td>0.776</td><td>0.789</td><td><b>0.804</b></td><td>0.789</td><td>0.800</td></tr>
  <tr><td>lij_Latin</td><td>0.907</td><td>0.896</td><td>0.912</td><td><b>0.914</b></td><td>0.910</td><td><b>0.914</b></td></tr>
  <tr><td>lit_Latin</td><td>0.930</td><td>0.921</td><td>0.933</td><td><b>0.938</b></td><td>0.934</td><td>0.937</td></tr>
  <tr><td>lmo_Latin</td><td>0.866</td><td>0.833</td><td>0.884</td><td>0.900</td><td>0.885</td><td>0.898</td></tr>
  <tr><td>lvk_Latin</td><td>0.888</td><td>0.866</td><td>0.900</td><td>0.917</td><td>0.900</td><td>0.916</td></tr>
  <tr><td>mag_Deva</td><td>0.931</td><td>0.927</td><td>0.926</td><td><b>0.935</b></td><td>0.932</td><td>0.937</td></tr>
  <tr><td>mai_Deva</td><td><b>0.930</b></td><td>0.881</td><td>0.907</td><td>0.905</td><td>0.906</td><td>0.906</td></tr>
  <tr><td>mar_Deva</td><td><b>0.929</b></td><td>0.920</td><td>0.920</td><td>0.925</td><td>0.921</td><td>0.927</td></tr>
  <tr><td>mhr_Cyrl*</td><td>0.262<sup>†</sup></td><td>0.278<sup>†</sup></td><td>0.268<sup>†</sup></td><td>0.307<sup>†</sup></td><td>0.896</td><td><b>0.901</b></td></tr>
  <tr><td>mkd_Cyrl</td><td>0.946</td><td>0.942</td><td>0.946</td><td><b>0.952</b></td><td>0.947</td><td>0.951</td></tr>
  <tr><td>mvv_Cyrl</td><td>0.245<sup>†</sup></td><td>0.243<sup>†</sup></td><td>0.251<sup>†</sup></td><td>0.259<sup>†</sup></td><td><b>0.851</b></td><td>0.846</td></tr>
  <tr><td>npi_Deva</td><td><b>0.926</b></td><td>0.881</td><td>0.900</td><td>0.901</td><td>0.900</td><td>0.901</td></tr>
  <tr><td>oci_Latin</td><td>0.956</td><td>0.952</td><td>0.958</td><td>0.958</td><td>0.957</td><td><b>0.959</b></td></tr>
  <tr><td>oci_Latin_aran1260</td><td>0.505<sup>†</sup></td><td>0.500<sup>†</sup></td><td>0.569<sup>†</sup></td><td>0.576<sup>†</sup></td><td>0.566<sup>†</sup></td><td>0.571<sup>†</sup></tr>
  <tr><td>plh_Arab</td><td>0.968</td><td>0.955</td><td>0.970</td><td>0.972</td><td>0.872</td><td><b>0.874</b></td></tr>
  <tr><td>pec_Arab</td><td>0.924</td><td>0.918</td><td>0.922</td><td><b>0.926</b></td><td>0.925</td><td><b>0.926</b></td></tr>
  <tr><td>pol_Latin</td><td><b>0.950</b></td><td>0.946</td><td>0.946</td><td>0.949</td><td>0.948</td><td>0.948</td></tr>
  <tr><td>por_Latin</td><td><b>0.963</b></td><td>0.960</td><td>0.961</td><td>0.962</td><td>0.960</td><td>0.962</td></tr>
  <tr><td>prs_Arab</td><td>0.901</td><td>0.900</td><td>0.908</td><td><b>0.910</b></td><td>0.908</td><td><b>0.910</b></td></tr>
  <tr><td>ron_Latin</td><td><b>0.964</b></td><td>0.961</td><td>0.963</td><td><b>0.964</b></td><td><b>0.964</b></td><td><b>0.964</b></td></tr>
  <tr><td>rus_Cyrl</td><td><b>0.950</b></td><td>0.942</td><td>0.944</td><td>0.946</td><td>0.944</td><td>0.946</td></tr>
  <tr><td>san_Deva</td><td>0.749</td><td>0.702</td><td>0.731</td><td>0.737</td><td>0.732</td><td>0.742</td></tr>
  <tr><td>scn_Latin</td><td>0.896</td><td>0.877</td><td>0.904</td><td><b>0.915</b></td><td>0.905</td><td>0.914</td></tr>
  <tr><td>sik_Latin</td><td>0.955</td><td>0.951</td><td>0.953</td><td><b>0.956</b></td><td>0.952</td><td>0.955</td></tr>
  <tr><td>slv_Latin</td><td>0.944</td><td>0.943</td><td>0.948</td><td><b>0.951</b></td><td>0.948</td><td><b>0.951</b></td></tr>
  <tr><td>snd_Deva*</td><td>0.466<sup>†</sup></td><td>0.464<sup>†</sup></td><td>0.512<sup>†</sup></td><td>0.497<sup>†</sup></td><td>0.860</td><td><b>0.869</b></td></tr>
  <tr><td>spa_Latin</td><td><b>0.950</b></td><td>0.949</td><td>0.948</td><td>0.948</td><td>0.946</td><td>0.948</td></tr>
  <tr><td>srd_Latin</td><td>0.909</td><td>0.902</td><td>0.917</td><td><b>0.918</b></td><td>0.915</td><td>0.916</td></tr>
  <tr><td>stp_Cyrl</td><td>0.949</td><td>0.943</td><td>0.950</td><td><b>0.954</b></td><td>0.950</td><td><b>0.954</b></td></tr>
  <tr><td>szl_Latin</td><td>0.930</td><td>0.920</td><td>0.934</td><td><b>0.942</b></td><td>0.933</td><td>0.940</td></tr>
  <tr><td>tat_Cyrl</td><td>0.927</td><td>0.915</td><td>0.925</td><td>0.927</td><td>0.925</td><td><b>0.928</b></td></tr>
  <tr><td>tgk_Cyrl</td><td>0.915</td><td>0.903</td><td>0.915</td><td><b>0.925</b></td><td>0.916</td><td>0.923</td></tr>
  <tr><td>tuk_Latin</td><td>0.893</td><td>0.882</td><td>0.898</td><td>0.910</td><td>0.904</td><td><b>0.912</b></td></tr>
  <tr><td>tur_Latin</td><td><b>0.946</b></td><td>0.940</td><td>0.943</td><td><b>0.946</b></td><td>0.945</td><td>0.945</td></tr>
  <tr><td>tyr_Cyrl</td><td>0.281<sup>†</sup></td><td>0.313<sup>†</sup></td><td>0.366<sup>†</sup></td><td>0.381<sup>†</sup></td><td>0.880</td><td><b>0.884</b></td></tr>
  <tr><td>uig_Arab</td><td>0.863</td><td>0.853</td><td>0.859</td><td><b>0.868</b></td><td>0.859</td><td>0.867</td></tr>
  <tr><td>ukr_Cyrl</td><td>0.948</td><td>0.942</td><td>0.945</td><td>0.948</td><td>0.946</td><td><b>0.949</b></td></tr>
  <tr><td>uzn_Latin</td><td><b>0.951</b></td><td>0.914</td><td>0.918</td><td>0.924</td><td>0.920</td><td>0.926</td></tr>
  <tr><td>vec_Latin</td><td>0.920</td><td>0.907</td><td>0.931</td><td>0.933</td><td>0.930</td><td><b>0.936</b></td></tr>
</table>

Table 11: Text Translation COMET (X→Eng) scores in FLORES devtest. * indicates translation is evaluated on dev split. † indicates that the result is zero-shot. Underlined is the best among the zero-shot for each language, if any. In bold is the best for supervised results for each language, if any.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">E2E ST</th>
    <th colspan="4">charSONAR Cascades</th>
    <th colspan="2">Speech-charSONAR (CV)</th>
    <th colspan="2">+ FLEURS</th>
  </tr>
  <tr>
    <th>SONAR</th>
    <th>Whisper</th>
    <th>SeamlessM4T</th>
    <th>w/MMS</th>
    <th>w/Whisper</th>
    <th>Random</th>
    <th>Pretrained</th>
    <th>Dual</th>
    <th>Pretrained</th>
    <th>Dual</th>
  </tr>
  <tr><td>ast_Latn*</td><td>-</td><td>-</td><td>0.752</td><td>0.821</td><td>-</td><td>0.204</td><td>0.816</td><td>0.223</td><td>0.840</td><td>0.849</td></tr>
  <tr><td>azj_Latn</td><td>-</td><td>0.642</td><td><b>0.837</b></td><td>0.762</td><td>0.777</td><td>0.197</td><td>0.788</td><td>0.194</td><td>0.788</td><td>0.764</td></tr>
  <tr><td>bel_Cyril</td><td>0.823</td><td>0.676</td><td>0.885</td><td>0.864</td><td>0.842</td><td>0.856</td><td>0.864</td><td>0.868</td><td>0.874</td><td><b>0.886</b></td></tr>
  <tr><td>bos_Latn</td><td>0.882</td><td>0.829</td><td><b>0.919</b></td><td>0.899</td><td>0.911</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>bul_Cyril</td><td>0.852</td><td>0.816</td><td>0.902</td><td>0.884</td><td>0.891</td><td>0.879</td><td>0.891</td><td>0.898</td><td>0.892</td><td><b>0.907</b></td></tr>
  <tr><td>cat_Latn</td><td>0.886</td><td>0.874</td><td>0.929</td><td>0.893</td><td>0.929</td><td>0.903</td><td>0.900</td><td>0.922</td><td>0.913</td><td><b>0.930</b></td></tr>
  <tr><td>ces_Latn</td><td>0.873</td><td>0.811</td><td>0.910</td><td>0.889</td><td>0.904</td><td>0.888</td><td>0.890</td><td>0.904</td><td>0.902</td><td><b>0.911</b></td></tr>
  <tr><td>ckb_Arab*</td><td>-</td><td>-</td><td>0.766</td><td>0.786</td><td>-</td><td>0.692</td><td>0.800</td><td>0.798</td><td>0.802</td><td><b>0.818</b></td></tr>
  <tr><td>ell_Grek</td><td>-</td><td>0.747</td><td>0.868</td><td>0.860</td><td>0.874</td><td>0.216</td><td>0.866</td><td>0.866</td><td>0.872</td><td><b>0.889</b></td></tr>
  <tr><td>est_Latn</td><td>0.805</td><td>0.629</td><td>0.898</td><td>0.908</td><td>0.902</td><td>0.877</td><td>0.910</td><td>0.912</td><td><b>0.918</b></td><td>0.914</td></tr>
  <tr><td>fin_Latn</td><td>0.799</td><td>0.768</td><td>0.887</td><td>0.900</td><td>0.926</td><td>0.229</td><td>0.902</td><td>0.904</td><td><b>0.906</b></td><td>0.905</td></tr>
  <tr><td>fra_Latn</td><td>0.870</td><td>0.891</td><td>0.910</td><td>0.884</td><td><b>0.935</b></td><td>0.888</td><td>0.906</td><td>0.914</td><td>0.914</td><td>0.922</td></tr>
  <tr><td>glg_Latn</td><td>-</td><td>0.830</td><td>0.917</td><td>0.897</td><td>0.908</td><td>0.859</td><td>0.908</td><td><b>0.913</b></td><td>0.906</td><td>0.910</td></tr>
  <tr><td>hin_Deva</td><td>0.770</td><td>0.746</td><td>0.856</td><td>0.856</td><td>0.834</td><td>0.854</td><td>0.876</td><td>0.875</td><td>0.880</td><td><b>0.882</b></td></tr>
  <tr><td>hrv_Latn*</td><td>0.866</td><td>0.816</td><td>0.895</td><td>0.903</td><td><b>0.912</b></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>hun_Latn</td><td>-</td><td>0.724</td><td>0.878</td><td>0.867</td><td>0.886</td><td>0.870</td><td>0.866</td><td>0.882</td><td>0.884</td><td><b>0.892</b></td></tr>
  <tr><td>ita_Latn</td><td>0.892</td><td>0.898</td><td>0.927</td><td>0.915</td><td><b>0.943</b></td><td>0.914</td><td>0.924</td><td>0.926</td><td>0.932</td><td>0.939</td></tr>
  <tr><td>kaz_Cyril</td><td>-</td><td>0.349</td><td>0.846</td><td>0.844</td><td>0.774</td><td>0.198</td><td>0.849</td><td>0.198</td><td>0.866</td><td><b>0.876</b></td></tr>
  <tr><td>khk_Cyril</td><td>-</td><td>0.207</td><td>0.748</td><td>0.731</td><td>0.342</td><td>0.229</td><td>0.732</td><td>0.213</td><td>0.763</td><td><b>0.764</b></td></tr>
  <tr><td>kir_Cyril*</td><td>-</td><td>-</td><td>0.854</td><td>0.837</td><td>-</td><td>0.197</td><td>0.855</td><td>0.853</td><td>0.861</td><td><b>0.865</b></td></tr>
  <tr><td>lit_Latn</td><td>0.766</td><td>0.579</td><td>0.832</td><td>0.880</td><td>0.853</td><td>0.858</td><td>0.881</td><td>0.885</td><td>0.884</td><td><b>0.897</b></td></tr>
  <tr><td>lvu_Latn</td><td>0.848</td><td>0.589</td><td>0.885</td><td>0.899</td><td>0.888</td><td>0.879</td><td>0.905</td><td>0.909</td><td>0.908</td><td><b>0.910</b></td></tr>
  <tr><td>mar_Deva</td><td>0.734</td><td>0.503</td><td>0.821</td><td>0.812</td><td>0.684</td><td>0.734</td><td>0.836</td><td>0.843</td><td>0.840</td><td><b>0.846</b></td></tr>
  <tr><td>mkd_Cyril</td><td>0.887</td><td>0.808</td><td>0.917</td><td>0.919</td><td>0.912</td><td>0.246</td><td>0.927</td><td>0.925</td><td><b>0.928</b></td><td><b>0.928</b></td></tr>
  <tr><td>npi_Deva</td><td>0.675</td><td>0.538</td><td><b>0.826</b></td><td>0.790</td><td>0.652</td><td>0.196</td><td>0.810</td><td>0.798</td><td>0.801</td><td>0.799</td></tr>
  <tr><td>oci_Latn</td><td>-</td><td>0.483</td><td>0.568</td><td>0.747</td><td>0.583</td><td>0.202</td><td>0.707</td><td>0.707</td><td>0.795</td><td><b>0.805</b></td></tr>
  <tr><td>pes_Arab</td><td>0.810</td><td>0.666</td><td>0.887</td><td>0.867</td><td>0.851</td><td>0.875</td><td>0.890</td><td>0.884</td><td>0.887</td><td><b>0.899</b></td></tr>
  <tr><td>pol_Latn</td><td>0.860</td><td>0.856</td><td>0.893</td><td>0.888</td><td><b>0.923</b></td><td>0.876</td><td>0.888</td><td>0.899</td><td>0.898</td><td>0.908</td></tr>
  <tr><td>por_Latn</td><td>0.878</td><td>0.906</td><td>0.897</td><td>0.902</td><td><b>0.941</b></td><td>0.885</td><td>0.908</td><td>0.917</td><td>0.918</td><td>0.922</td></tr>
  <tr><td>ron_Latn</td><td>0.856</td><td>0.867</td><td>0.909</td><td>0.895</td><td><b>0.919</b></td><td>0.855</td><td>0.900</td><td>0.904</td><td>0.906</td><td>0.905</td></tr>
  <tr><td>rus_Cyril</td><td>0.878</td><td>0.893</td><td><b>0.912</b></td><td>0.883</td><td>0.934</td><td>0.883</td><td>0.898</td><td>0.908</td><td>0.903</td><td>0.910</td></tr>
  <tr><td>sik_Latn</td><td>0.885</td><td>0.822</td><td>0.914</td><td>0.911</td><td>0.924</td><td>0.876</td><td>0.909</td><td>0.917</td><td>0.919</td><td><b>0.921</b></td></tr>
  <tr><td>slv_Latn</td><td>0.843</td><td>0.672</td><td>0.879</td><td>0.871</td><td>0.852</td><td>0.201</td><td>0.871</td><td>0.200</td><td><b>0.883</b></td><td>0.875</td></tr>
  <tr><td>snd_Deva*</td><td>0.360</td><td>0.360</td><td>0.443</td><td><b>0.698</b></td><td>0.423</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>spa_Latn</td><td>0.888</td><td>0.893</td><td>0.908</td><td>0.899</td><td><b>0.934</b></td><td>0.912</td><td>0.917</td><td>0.920</td><td>0.922</td><td>0.930</td></tr>
  <tr><td>srp_Cyril</td><td>0.891</td><td>0.856</td><td>0.924</td><td>0.924</td><td><b>0.929</b></td><td>0.235</td><td>0.922</td><td>0.913</td><td>0.927</td><td>0.928</td></tr>
  <tr><td>tgk_Cyril*</td><td>-</td><td>0.523</td><td>0.858</td><td><b>0.874</b></td><td>0.658</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>tur_Latn</td><td>0.743</td><td>0.827</td><td>0.888</td><td>0.878</td><td><b>0.924</b></td><td>0.868</td><td>0.885</td><td>0.903</td><td>0.899</td><td>0.909</td></tr>
  <tr><td>ukr_Cyril</td><td>0.858</td><td>0.865</td><td>0.912</td><td>0.890</td><td><b>0.931</b></td><td>0.887</td><td>0.903</td><td>0.904</td><td>0.911</td><td>0.915</td></tr>
  <tr><td>uzn_Latn</td><td>0.736</td><td>0.326</td><td>0.846</td><td>0.753</td><td>0.527</td><td>0.798</td><td>0.839</td><td>0.843</td><td>0.846</td><td><b>0.854</b></td></tr>
</table>

Table 12: Speech Translation COMET scores (X→Eng) on FLEURS test. The 6 languages with * where not part of the main results of Table 7, since they were not supported either by our models or by Whisper.
<table>
  <tr>
    <th rowspan="2"># Languages</th>
    <th colspan="4">Subgrouping</th>
    <th colspan="3">Script</th>
  </tr>
  <tr>
    <th>Indic</th>
    <th>Romance</th>
    <th>Turkic</th>
    <th>Uralic</th>
    <th>Devanagari</th>
    <th>Latin</th>
    <th>Cyrillic</th>
  </tr>
  <tr>
    <td>COMET</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>0.428</td>
    <td>0.749</td>
    <td>0.305</td>
    <td>0.260</td>
    <td>0.428</td>
    <td>0.651</td>
    <td>0.269</td>
  </tr>
  <tr>
    <td>SONAR-75</td>
    <td>0.866</td>
    <td>0.813</td>
    <td>0.882</td>
    <td>0.869</td>
    <td>0.866</td>
    <td>0.839</td>
    <td>0.867</td>
  </tr>
  <tr>
    <td>charSONAR-75</td>
    <td><b>0.875</b></td>
    <td><b>0.814</b></td>
    <td><b>0.888</b></td>
    <td><b>0.874</b></td>
    <td><b>0.875</b></td>
    <td><b>0.842</b></td>
    <td><b>0.871</b></td>
  </tr>
  <tr>
    <td>XSIM++</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SONAR-200</td>
    <td>53.8</td>
    <td>29.3</td>
    <td>63.5</td>
    <td>70.2</td>
    <td>53.8</td>
    <td>36.1</td>
    <td>68.6</td>
  </tr>
  <tr>
    <td>SONAR-75</td>
    <td>8.7</td>
    <td>21.6</td>
    <td><b>9.6</b></td>
    <td><b>10.5</b></td>
    <td>8.7</td>
    <td>17.7</td>
    <td><b>11.0</b></td>
  </tr>
  <tr>
    <td>charSONAR-75</td>
    <td><b>8.2</b></td>
    <td><b>20.9</b></td>
    <td>10.1</td>
    <td>10.8</td>
    <td><b>8.2</b></td>
    <td><b>16.9</b></td>
    <td>11.8</td>
  </tr>
</table>

Table 13: Text translation (COMET) and text retrieval (xSIM++) results per language subgroup and script for the 12 newly added languages. Results in FLORES dev/test (X→Eng).

<table>
  <tr>
    <th rowspan="2"># Languages</th>
    <th colspan="6">Subgrouping</th>
    <th colspan="6">Script</th>
  </tr>
  <tr>
    <th>Balto-Slavic</th>
    <th>Romance</th>
    <th>Turkic</th>
    <th>Indic</th>
    <th>Iranian</th>
    <th>Uralic</th>
    <th>Mongolic</th>
    <th>Greek</th>
    <th>Latin</th>
    <th>Cyrillic</th>
    <th>Devanagari</th>
    <th>Arabic</th>
    <th>Greek</th>
  </tr>
  <tr>
    <td>COMET</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SONAR-200</td>
    <td>0.934</td>
    <td>0.926</td>
    <td>0.850</td>
    <td>0.849</td>
    <td>0.871</td>
    <td>0.937</td>
    <td>0.838</td>
    <td>0.932</td>
    <td>0.917</td>
    <td>0.916</td>
    <td>0.849</td>
    <td>0.818</td>
    <td>0.932</td>
  </tr>
  <tr>
    <td>SONAR-75</td>
    <td>0.942</td>
    <td>0.935</td>
    <td>0.884</td>
    <td>0.870</td>
    <td>0.881</td>
    <td>0.944</td>
    <td>0.864</td>
    <td>0.936</td>
    <td>0.929</td>
    <td>0.925</td>
    <td>0.870</td>
    <td>0.852</td>
    <td>0.936</td>
  </tr>
  <tr>
    <td>charSONAR-75</td>
    <td><b>0.946</b></td>
    <td><b>0.940</b></td>
    <td><b>0.892</b></td>
    <td><b>0.877</b></td>
    <td><b>0.887</b></td>
    <td><b>0.946</b></td>
    <td><b>0.876</b></td>
    <td><b>0.939</b></td>
    <td><b>0.934</b></td>
    <td><b>0.930</b></td>
    <td><b>0.877</b></td>
    <td><b>0.860</b></td>
    <td><b>0.939</b></td>
  </tr>
  <tr>
    <td>XSIM++</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>8.1</td>
    <td>8.0</td>
    <td>13.4</td>
    <td>13.5</td>
    <td>10.7</td>
    <td>7.1</td>
    <td>13.3</td>
    <td>9.2</td>
    <td>8.3</td>
    <td>9.7</td>
    <td>13.5</td>
    <td>16.1</td>
    <td>9.2</td>
  </tr>
  <tr>
    <td>SONAR-75</td>
    <td>6.2</td>
    <td>5.5</td>
    <td>10.0</td>
    <td>9.7</td>
    <td>8.2</td>
    <td>5.7</td>
    <td>11.0</td>
    <td><b>6.7</b></td>
    <td>6.1</td>
    <td>7.7</td>
    <td>9.7</td>
    <td>11.6</td>
    <td><b>6.7</b></td>
  </tr>
  <tr>
    <td>charSONAR</td>
    <td>5.8</td>
    <td>5.2</td>
    <td>9.4</td>
    <td>9.1</td>
    <td>7.7</td>
    <td>5.3</td>
    <td>10.1</td>
    <td>6.9</td>
    <td>5.7</td>
    <td>7.1</td>
    <td>9.1</td>
    <td>11.0</td>
    <td>6.9</td>
  </tr>
</table>

Table 14: Text translation (COMET) and text retrieval (xSIM++) results per language subgroup and script for the 63 known training languages. Results in FLORES dev/test (X→Eng).

<table>
  <tr>
    <th>Model</th>
    <th>Ural/Cyril</th>
    <th>Turkic</th>
    <th>Romance</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>SONAR-200</td>
    <td>0.925</td>
    <td>0.857</td>
    <td>0.932</td>
    <td>0.905</td>
  </tr>
  <tr>
    <td>SONAR-group</td>
    <td>0.930</td>
    <td>0.879</td>
    <td>0.942</td>
    <td>0.917</td>
  </tr>
  <tr>
    <td>charSONAR-group</td>
    <td><b>0.934</b></td>
    <td><b>0.881</b></td>
    <td><b>0.946</b></td>
    <td><b>0.920</b></td>
  </tr>
  <tr>
    <td>→ w/ Norm</td>
    <td><b>0.934</b></td>
    <td><b>0.877</b></td>
    <td><b>0.946</b></td>
    <td>0.919</td>
  </tr>
  <tr>
    <td>→ w/ Norm & Noise</td>
    <td><b>0.934</b></td>
    <td><b>0.878</b></td>
    <td><b>0.947</b></td>
    <td><b>0.920</b></td>
  </tr>
</table>

Table 15: Text Translation COMET scores (X→Eng) in FLORES dev. Each encoder was trained on the respective group of languages.

<table>
  <tr>
    <th>Model</th>
    <th># Tokens</th>
    <th>Inference Time (s)</th>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>49</td>
    <td>127</td>
  </tr>
  <tr>
    <td>charSONAR</td>
    <td>158 (\( \times 3.2 \))</td>
    <td>142 (+10%)</td>
  </tr>
</table>

Table 16: Average number of tokens and average inference time in FLORES dev with batching (5K tokens per batch).

Metareview score: 4
Metareview: This paper proposes a character-level encoder (charSONAR) for multilingual text and speech translation, aiming to improve generalization to low-resource and unseen languages. The method involves a two-stage training pipeline:
1. Adapting the SONAR text encoder to operate at the character level;
2. Connecting a multilingual CTC-based ASR model (MMS) to this character-level encoder via a lightweight adapter.

The adapter enables end-to-end differentiable speech-to-text translation without requiring parallel ST data. Experiments across many languages show strong results, especially in low-resource and zero-shot settings, outperforming several baselines and achieving state-of-the-art performance in the 33 -> English direction.

While some reviewers have raised concerns that I believe are relatively minor, the authors have responded thoughtfully. I encourage the authors to consider selectively incorporating parts of their responses into the camera-ready version.
Summary of reasons to publish: 1. This paper is well-written and easy to follow. The method is simple and effective.
2. The method is computationally efficient and accessible to lower-resource communities.
3. The method shows strong results in effective cross-lingual and cross-modal transfer: The character-based model significantly improves transfer to low-resource and zero-shot languages in both text and speech translation. And it demonstrates that speech representations can be successfully projected into a shared text space for translation using only ASR data.