Toward Joint Language Modeling for Speech Units and Text
Authors: Ju-Chieh Chou
Chung-Ming Chien
Wei-Ning Hsu
Karen Livescu
Arun Babu
Alexis Conneau
Alexei Baevski
Michael Auli
Venue: EMNLP 2023 Findings (2023)
Keywords: Language modeling
Speech processing
Spoken Language Understanding
TL;DR: Present a joint LM for speech units and text and propose evaluation metrics to assess the cross-modal ability of the joint LM.

Abstract: Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language modeling for speech units and text. Specifically, we compare different speech tokenizers to transform continuous speech signals into discrete units and use different methods to construct mixed speech-text data. We introduce automatic metrics to evaluate how well the joint LM mixes speech and text. We also fine-tune the LM on downstream spoken language understanding (SLU) tasks with different modalities (speech or text) and test its performance to assess the model's learning of shared representations. Our results show that by mixing speech units and text with our proposed mixing techniques, the joint LM improves over a speech-only baseline on SLU tasks and shows zero-shot cross-modal transferability.

Toward Joint Language Modeling for Speech Units and Text
Ju-Chieh Chou'; Chung-Ming Chien', Wei-Ning Hsu', Karen Livescu', Arun Babu', Alexis Conneau", Alexei Baevski Michael Auli?
Tooyot Technological Institute Chicago Meta Al OOpenAl *Character Al jcchouêttic. edu, wnhsuêmeta. com
Abstract
Speech and text are two major forms of hu- man language. The research community has been focusing mapping speech text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language modeling for speech units and text. Specifically, we com- pare different speech tokenizers to transform continuous speech signals into discrete units and use different methods construct mixed speech-text data. We introduce automatic met- rics to evaluate how well the joint LM mixes speech and text. We also fine-tune the LM on downstream spoken language understanding (SLU) tasks with different modalities (speech text) and test its performance to assess the model's 's learning of shared representations. Our results show that by mixing speech units and text with our proposed mixing techniques, the joint LM improves over a speech-only baseline SLU tasks and shows zero-shot cross-modal transferability.
how are you
HuBERT+ Quantizer
are
SUTLM
$12 566 $17 $18
<U EN> 512 566 <U2T>
Speech S12 $66 $17 S18
(example: AST)
Transcrition: how
Training
Data construction
uLM: <U, <U_EN> $12 566 517 518 <EDU>
566 $18... <EOU>
you
AST: <U_EN> $12 568 <U2T> are you <EOS>
ILM: <LM:<T_ EN_ how are you <EOS>
Figure An illustration of workflow. We tokenize speech signals into discrete units and mix them with text create speech-text data. Our SUTLM then trained on a combination of speech-only, text-only, and speech-text data. More details on the data formats can be found in Table 1.
1 Introduction
Brown et al., 2020). On the other hand, genera- tive spoken language models (GSLM) (Lakhotia et al., 2021), which are LMs trained on discrete speech units derived from self-supervised represen- tations (Hsu al., 2021), also promising for spoken language modeling.
Speech and language processing research has largely focused on spoken and written language separately. However, the integration speech and text in a single model holds potential bene- fits. Speech data contains prosodic information that does not exist in the text, which can help in modeling dialogues. On the other hand, text data from sources like Wikipedia can provide structural knowledge that is not available in most speech datasets. Moreover, the amount of written text on the internet exceeds the size of any available speech dataset.
In this work, we aim fill the gap between text-only and speech-only LMs by developing and studying design choices for a joint Speech Unit and Text Language Model (SUTLM). For speech, we use sule-supe learning (SSL) speech model, i.e. HuBERT (Hsu et al., 2021), to convert con- tinuous speech signals into speech units. We then combine the units with text data to train an LM that models speech units and text jointly. We con- vert speech-only, mixed speech-text, and text-only data into token sequences (as shown in Figure and Table 1), and train the model as an LM.
The impressive performance of text large lan- guage models (LLMs) has caused revolution in natural language processing (Radford et al., 2019;
WWrk done during an internship at Meta AL. Work done while at Meta Al.

---

To evaluate the SUTLM, automatic metrics are developed quantify the cross-modal ability of the LMs. We also fine-tune our models on down- stream tasks for spoken language understanding. We fine-tune the SUTLMs on either the speech or text data and test them on either speech or text to understand how well the models learn to align the two modalities. Our main contributions are:
2.2 Textless NLP
Textless NLP (Lakhotia et al., 2021; Polyak et al., 2021; Kharitonov et al., 2021) is framework model speech in the absence of textual data. It consists of three components: a speech-to-unit to- kenizer, a unit LM (uLM), and a unit-to-speech detokenizer. The tokenizer takes speech signals as inputs to generate discrete speech units. A uLM is trained to predict the next token in an utterance given its prior context. Once the uLM is trained, it can be used to generate unit sequences autoregres- sively. In the end, the detokenizer used convert the generated unit sequences to speech signals.
We present a joint autoregressive LM trained on both speech and text (Sec 3).
We develop automatic metrics that require no fine-tuning for the evaluation of an SUTLM, and show that the proposed metrics are indica- tive of the model's cross-modal transfer ability on downstream tasks (Sec 4).
2.3 Joint speech-text transformers
Transformer models have been extremely suc- cessful in natural language and speech process- ing (Vaswani al., 2017: Gulati al., 2020), with three major configurations: encoder-decoder models (Vaswani al., 2017), encoder-only mod- els (Devlin et al., 2018), and decoder-only mod- els (Radford et al., 2018).
Empirically, we show that units covering a larger span obtained through SentencePiece tokenization (Kudo and Richardson, 2018) outperform local units learned by existing self-supervised models (Hsu et al., 2021) (Sec 5.
Previous works speech-text joint transform- mostly adapt the encoder-decoder (Ao et al., 2021; Tang et al., 2022; Cheng et al., 2022) or encoder-only (Chung et al., 2020; Bapna et al., 2021; Chen et al., 2022b; Zhang et al., 2022b) ar- chitectures. Compared with decoder-only archi- tectures, the training of these models typically re- quires multiple losses and explicit alignments be- tween paired speech and transcriptions. This makes the hyper-parameter selection time-consuming. Also, encoder-only and encoder-decoder models are mostly used in the pre-training + fine-tuning paradigm, which limits the use cases of these mod- els.
We find that mixing speech units and text with our proposed techniques (Sec 5.5.3 & See 5.5.4) improves the cross-modal ability of the model. (Sec 5.4).
2 Related Work
2.1 SSL speech models
Self-supervised pre-training enables speech mod- els to learn the information in speech without paired text transcriptions and show impressive performance on tasks such as automatic speech recognition (ASR) with minimal supervised fine- tuning (Baevski al., 2020; Hsu al., 2021: Chen al., 2021). As SSL speech models learn pho- netically meaningful speech representations (Pasad et al., 2023), they can be used as a feature extrac- tor (Yang et al., 2021) or a quantizer to transform continuous speech into discrete units (Lakhotia et al., 2021; Lee et al., 2021 a,b; Lin et al., 2022; Chen et al., 2022a). In this work, use the HuBERT model (Hsu al., 2021) along with quantizer to tokenize continuous speech into dis- crete representations. The discrete speech units are then combined with text data to train a single LM that is able to model speech and text jointly.
On the other hand, decoder-only models on text (Radford et al., 2019; Brown al., 2020) show the impressive capability of in-context learning. which also reduces the efforts spent on fine-tuning pre-trained models. In light of this, we explore decoder-only models for speech-text joint train- ing. In this under-explored area, the concurrent work VALL-E (Wang et al., 2023) is the only other attempt to build decoder-only model jointly mod- cling speech and text. However, VALL-E's purpose is controllable text-to-speech synthesis (TTS), and the work mainly focuses on the acoustic controlla- bility of the generated speech, while our work aims to build a gencral--urpose purpose joint LM and mainly fo- cuses on modeling the content of spoken language.

---

3 Method
3.2 Text-only: text LM (tLM)
We train another SentencePiece tokenizer (Kudo and Richardson, 2018) using the text-only cor- pus Sec 5.1.3 to convert text into subword tokens. The resulting vocabulary size of the subword kens around 45k.
We start with dataset of sentences D where sentence s¹ is composed of a sequence of T3 tokens (zf, =i 22.......1) 21)) where oj can be either text or speech units. The SUTLM is trained to predict the next token zÿ given its prior context ='-g We maximize the log-probability of
{s², s,s .s".....s" s").
3.3 Concatenated speech-text (CST)
the data
To present paired speech-text data to the SUTLM, we first convert speech units and their transcriptions into the uLM and tLM formats, respectively, and combine them into one sequence by simply concate- nating them shown in Table 1. The CST format explicitly tells the model the correspondence be- tween paired speech and text and thus encourages the model to learn the dependence between speech units and the corresponding text transcriptions.
1
T
log 22))
(1)
In the following sections, we describe how we con- struct token sequences from speech and text. An example of our data formats can be found in Ta- ble 1.
3.1 Speech-only: unit LM (uLM)
3.4 Alternating speech-text (AST)
Prior work has shown that discrete speech units derived from pre-trained HuBERT model can be used as compact representations to encode speech content, enabling the training of unit language model (Lakhotia al., 2021). However, when com- bining speech with text, the time scales of speech units and text differ. HuBERT units are typically on the phone or sub-phone level, as shown in Ta- ble This leads to longer sequences, making difficult for the model to capture long-term depen- dencies. On the other hand, subword tokenizers for text generally break text sequences into chunks of larger size than speech units. This length mismatch between speech and text makes it challenging to model them in single model. Therefore, we use a subword tokenizer (Kudo and Richardson, 2018) to combine HuBERT units into larger chunks as in (Wu et al., 2022) to mitigate the length mis- match.
Aside from simply concatenating the sequences of speech units and text, we also construct mixed speech-text that takes the word-level correspon- dence into consideration.
We use nizer (McAuliffe et al., 2017) to force-align speech and its transcription to obtain the word boundaries in an utterance. We then randomly sample some word boundaries within the utter- ance² the "switching points", which divide the utterance into several chunks. The alternating speech-text (AST) sequence then constructed by alternatively filling in the chunks with uL. speech units and tLM text tokens, resulting in a sequence that switches modalities at every switching point. Special tokens <U2T> and <T2U> are inserted when switching from speech units to text and text to speech units, respectively.
pre-trained speech recog-
The process of generating speech units is as follows. Speech signals are first fed into a Hu- BERT model. The representations in the final layer are then clustered with the k-means algo- rithm. The cluster IDs are used as the discrete speech units after removing consecutive repeating units (Lakhotia et al., 2021).' These units are then further combined by the subword SentencePiece tokenizer (Kudo and Richardson, 2018). The re- sulting average number of tokens per second can be found in Table 2.
Evaluation Metrics
We introduce automatic metrics that require no fine-tuning to evaluate the SUTLM. Fine-tuning is common approach to assess the quality of pre- trained models (Baevski al., 2020; Hsu et al., 2021; Chen et al., 2021). However, it is time- consuming process and the reliability of the ex- periments highly depends on the hyper-parameter selection process. Furthermore, there is no reliable metric to measure the cross-modal ability of LMs.
For example, the unit sequence 13 13 15 88 88 89 hecomes 13 15 80 after removing repetitions.
For sentence with words, we uniformly sample [N] houndarics as the switching points with ~ N 1).

---

Task
Example
uLM
<U_EN> S12 S66 S17 S18
<EOU>
CST
<U_EN> S12 S66 S17 S18
<EOU> <T_EN> how are you <EOS>
CST
<T_EN> how are you <EOS> <U_EN> S12 566 S17 S18
<EOU>
AST
<U EU_EN> S12 S66 <U2T> are you <EOS>
AST
<T_EN> how <T2U> S17 S18 ... <EOU>
ILM
<T_EN> how are you <EOS>
Table I: An example of the formats of unpaired (uLM, tLM) and mixed speech-text (CST, AST) data. For the CST and AST formats, speech units and text can be present in sequence in different orders. <U_EN> and <T <T_EN> are used at the beginning of the unit/text sequence. <EOU> and <EOS> are used at the end the unit/text sequences. <U2T> and <T2U> used when switching from unit text and text to unit at word boundaries.
Average tokens per second
the highest conditional probability among a pool of unmatched prompts.
Phone
20.32
HuBERT
50.00
CRA also has pointwise mutual information (PMT) interpretation:
+ deduplication
33.33
+ SP 10k
17.67
Poly'I
+ SP 32k
14.33
arg max
jE(I..m)
log Po(y(1a')>
max log Po(3 (x²)
Table 2: The average number of tokens per second for different types of speech units. SP 10k and 32k refer to SentencePiece tokenization (Kudo and Richardson, 2018) applied to HuBERT units to create a dictionary with 10k and 32k tokens respectively.
Jc [c
Po(v'(z')
log Po(v'la))
log
Po(y))
jc(l..m)
Po(
PMI(2',y))
max PMI(x
,2²)
jE(l..m)
(3)
In light of this, we propose Context Retrieval Ac- curacy (CRA), new metric that does not require fine-tuning, to evaluate the cross-moda ability of an SUTLM.
That is, correctly identifying the prompt implies the matched pronpt-continuation pair has a higher PMI than all unmatched prompt-continuation pairs.
4.1 Context Retrieval Accuracy (CRA)
Ideally, the model should produce similar rep- resentations given the same content regardless the modality. Hence, in addition to the uni-modal CRA, we also consider cross-modal CRA, where the prompt and the continuation are in different modalities. In practice, for example, when use text as the prompts and speech units as the con- tinuations, we the probability of emitting text tokens to zero and re-normalize the probability to ensure that the continuation y² can be only speech units. Cross-modal CRA can be used as a way to measure whether the SUTLM successfully learns shared representations between text and speech.
The motivation of Context Retrieval Accuracy (CRA) comes from the intuition that good LM should learn to predict the next token based on its prior context. When we divide a sentence into prompt and continuation, a good LM should be able to capture the dependence between them. That is, it should assign a higher conditional probability to the continuation given its corresponding prompt than given random prompt.
To measure CRA, we gather collection of m sentences C
........"! ....... s and break sf into a pair of prompt æ² and continuation y². Given an SUTLM parameterized by 0, we can measure the conditional probabilities Po (a()() with Eq 1. The CRA is then computed as:
{ s
4.2 Perplexity under External LM (PELM)
Following previous work, under external LM (PELM) to measure the quality of the content of generated samples (Lakhotia al., 2021). We sample continuation from the SUTLM given each ground truth prompt. We then use an external text LM, OPT-6.77 6.7B (Zhang et al., 2022a),
the perplexity
m
1 aarg max Po()"( = i), JE(I..m)
(2)
That is, the LM is used as a scorer to classify whether the matched prompt-continuation pair has

---

compute the perplexity of the sequence:
People's 's Speech (Aghajanyan al., 2023). The subsampled datasct consists of 65k hours of speech.
o' ~ Po(y)=²)
x², 30th T(x' 5¹)
5.1.2 Mixed speech-text (CST and AST)
(4)
We use MLS (Pratap et al., 2020) and VoxPop- uli (Wang et al., 2021) to create mixed speech-text data without subsampling. The dataset contains 45k hours of speech and 2.7B of words.
- E, log Port (v") gt(x"))
len("")
(0 =
where x¹ and 8' refer to the prompt and sam- pled continuation, and are the parameters of the SUTLM. Similarly to cross-modal CRA, we con- trol the modality of sampled continuations by ze- roing out the probability of the tokens in the unde- sired modality. Since the prompt and the contin- uation can be either speech units or subword text tokens, we use transcriber T(-) to transcribe the concatenated sequences of || j' into text x" y/" g+()) is function that outputs ground truth tran- scription when the input is speech units and is an identity function when the input text. The exter- nal LM then used to measure the perplexity of the continuation part of the text sequence.
5.1.3 Text-only
We combine OPT web data (Zhang et al., 2022a), Wikipedia, and LibriLM (Panayotov et al., 2015), and then subsample 5% of it, resulting in a total of 8.5B subwords.
5.2 SSL speech tokenizer
We use HuBERT Base model trained on 221K hours of unlabeled speech in 8 languages in (Hsu et al., 2022; Nguyen al., 2023).4 After pre- training, the representations at the last layer (12th) are clustered with k-means using 2000 clusters.
5.3 Model architecture and training
4.3 Evaluation SLUE tasks
We use the 24-layer transformer implementation in fairseq (Ott et al., 2019) with 16 attention heads. The embedding size is 1024, the feed-forwart di- mension is 4096, and the dropout probability is set to 0.1. The weights of the embedding layer are tied to the output layer (Press and Wolf, 2016). The model contains 350M parameters.
We use the SLUE benchmark (Shon et al., 2022) to evaluate our models on downstream tasks. The benchmark includes two tasks, sentiment analysis (SLUE-SA) and named entity recognition (SLUE- NER), with both speech data and transcriptions provided. After pre-training the SUTLM, we fine- tune it on the SLUE dataset with either speech or data as inputs to predict the ground-truth labels, and then evaluate it on either speech or text inputs. We evaluate the model on different input modalities to understand the cross-modal ability of the model as in (Hsu and Shi, 2022; Bapna et al., 2021, 2022). Fine-tuning details can be found in 5.4.2.
The model is trained for 500k updates on 32 V100 GPUs with batch size of 8192 tokens per GPU. We use Adam optimizer (Kingma and Ba, 2014) with (31, (22) (0.9, 0.95). Gradient clipping with threshold 1.0 and weight decay of 0.1 are applied stabilize the training. Since the data size different for different data formats, we resample specch-only specch-test and text-only data equally (1/3 for each in every training batch) to prevent the model from being biased toward any of them.
5 Experiments
5.1 Data
5.1.1 Speech-only
5.4 Evaluation setup
We use 5% of the dataset used in (Aghajanyan et al., 2023) to match the size of the mixed speech- text and text-only data. The dataset includes Multilingual LibriSpeech (MLS) (Pratap et al., 2020), VoxPopuli (Wang et al., 2021), Common- Voice (Ardila et al., 2019) and Spotify Podcast &
5.4.1 Automatic Metries
We use subset of the Multilingual Lib- riSpeech (Pratap al., 2020) dev set evaluate the SUTLM. To provide enough context to the SUTLM, we filter out sentences of less than 20 words. For each sentence, we use the first 10 words as the prompt and the rest as continuation. For the CRA experiments, we evaluate the SUTLM with
'For both speech units and text tokens, we first invert the SentencePiece tokenization process to get raw HuBERT units and raw text. For speech units, we further use a 12-layer Transformer encoder with CTC head to map HuBERT units to text. The transformer is trained on LibriSpecch, with a WER of 5.18% on dev-clean, and 11 11.61% % on dev-other.
'https: /ttts::/ddlffaar fbaipublicfiles http:u/licciless.c//bbbebbeee mhubert_base wwwppppsmm_e____eeeeppppe_pp_ppe www.cc___lnnni....m it3.pt

---

the 100 shortest utterances in the filtered dataset, while for the PELM experiments, we use the 500 shortest utterances. We use fewer utterances in CRA experiments as the computation of CRA is O(N2) for N utterances. We constrain ourselves to sentences with moderate lengths because the continuation part becomes less coherent with the prompt as the sequence length grows, which hurts the sensitivity of the proposed metries.
the original HuBERT units. For CRA for the u2u case (unit prompt, unit continuation), we hypoth- esize that the model uses low-level acoustic infor- mation to make predictions as the CRAs are nearly 1.0 for all types of speech units. Also, HuBERT uses overlapping windows for neighboring tokens, so the first token of the continuation contains infor- mation about the previous token.
For the speech continuation (PELM) experi- ments, the SP 10k-based sequences are shorter than HuBERT unit-based sequences, so the model trained with SP 10k (row (B)) can generate more coherent continuations.
When sampling the speech or text continuations in the PELM experiments, we use temperature 0.6 and nucleus sampling (Holtzman et al., 2019) with = 0.95, and truncate the continuation to 10 words (identical to the length of the prompts).
5.5.2 Do we need paired data to learn shared representations?
5.4.2 Downstream Tasks
For SLUE-SA, we fine-tune SUTLM by adding a self-attention pooling layer on top of the trans- former model after removing the last output layer (Shon et al., 2022). We fine-tune it with a learning rate of 3e-5 for 30k updates and evaluate it with Macro F1 (Shon et al., 2022).
In this section, we compare models trained with and without paired data to investigate the useful- ness of paired data. We can compare the results in row (D) and (F) in Table 3 for automatic met- rics, Table 4 for SLUE-SA and Table for SLUE- NER. For cross-modal cases (u2t and t2u), in terms of automatic metrics, the model trained with un- paired data alone (row (D)) has almost random CRAs and high PELMs, indicating a lack of cross- modal ability.
For SLUE-NER, we follow the SLUE official baseline to formulate the task as an ASR prob- lem and train our model to decode special tokens around each named entity (Shon et al., 2022). We concatenate the output (the text transcription with special tokens before and after each named entity) after the input (speech units when fine-tuned on speech, text tokens when fine-tuned on text) and fine-tune our SUTLM as an LM with the same loss function as Eq 1. The loss is only applied to the out- put part of the sequence. We fine-tune the SUTLM with a learning rate of 3e-5 for 50k updates. During decoding, we use a beam size of 5 to generate the outputs and evaluate them with Micro F1 (Shon et al., 2022). For both SLUE tasks, we report re- sults on the dev set since the test set is not publicly available. We use the fine-tuned HuBERT as the baseline as in (Shon et al.. 2022).
Similarly, for SLUE-SA, the model trained with unpaired data alone (row (D)) shows almost ran- dom macro FI scores for a 3-way classification task when tested on the other modality. For SLUE-NER. the model trained without exposure to paired data (row (D)) performs worse than models trained with paired data (row (F)) when fine-tuned on speech and shows no transferability between modalities, Row (D) also performs worse than its speech unit- only counterpart (row (B), showing that the model trained solely on unpaired data does not demon- strate any cross-modal transfer ability between speech and text.
5.5.3 Does concatenated speech-text (CST) help learn shared representations?
5.5 Results
5.5.1
What kind of speech units works the best?
The next question we want to answer is whether CST is helpful in learning shared representations. Building on the previous findings (rows (A), (B), (C)), we utilize SP 10k as our speech unit vocab- ulary and present the results in row (E) in Table 3 for automatic metrics, Table 4 for SLUE-SA, and Table 5 for SLUE-NER. The results show that, com- pared to using unpaired data alone (row (D)), the model trained with CST (row (E) has higher CRAs for u2t and t2u, which indicates that the model cap-
We utilize HuBERT units described in Sec 5.2 (2000 units) and apply SentencePiece tokenizers on them. Results can be found in rows (A), (B), (C) in Table for automatic metries, Table 4 for SLUE-SA and Table 5 for SLUE-NER.
The model trained with SP 10k has the best per- formance in terms of PELM, SLUE-SA, and SLUE-
NER, but slightly worse CRA than the model using

---

u2u
12u
u2t
121
rOW
unit
uLM
CST
AST
tLM
CRA
PELM
CRA
PELM
CRA
PELM
CRA
PELM
Ground truth continuation
101.4
(A)
HuBERT
1.00
193.3
(B)
SP 10k
0.96
163.6
(C)
SP 32k
V
0.96
177.4
(D)
SP 10k
0.94
175.9
0.03
394.9
0.01
1973.3
0.20*
20.7
(E)
SP 10k
V
0.95
166.0
0.37
39.1"
0.26
43.4"
0.56
34.7
(F)
SP 10k
0.97
162.8
0.70
124.7
0.81
38.7
0.67
28.2
Table 3: Automatic metrics (CRA and PELM). "u2t" denotes that the prompts are speech units and the continuations are text, and so on. (*): for cross-modal cases (u2t and 12u) in row (E), the PELM is low because the continuation simply repeats the prompt. We discuss this issue in Sec 5.6. (*+): The low CRA for 12t is due to the use of MLS as an evaluation set, resulting in distribution mismatch from the text-only training data. Similarly, the use of OPT data to train the SUTLM results in better PELM 12t in row (D).
FT data
SP
TXT
to be that text data provides a less noisy supervi- sory signal compared to using speech units. The model trained with extra speech-text data (row (E)) performs worse than the model trained with only speech units (row (B)). The reason may be similar to the "curse of multilinguality" (Conneau et al., 2019). where sharing the capacity of the model with other languages or modalities hurts perfor- mance.
row
unit
Eval set
SP
TXT
SP
TXT
Baseline
0.46
(A)
HuBERT
uLM
0.51
(B)
10k
uLM
0.56
(C)
SP 32k
uLM
0.54
(D)
SP 10k
uLM+tLM
0.52
0.33
0.35
0.49
(E)
SP 10k
uLMMCST
0.48
0.42
0.51
0.52
uLM+CST
(F)
SP 10k
0.49
0.43
0.52
0.56
+LST+LM
Table 4: Macro F1 score on SLUE-SA. FT data indicates the model is fine-tuned on speech (SP) text (TXT). Eval set denotes the fine-tuned model is tested on speech (SP) or text (TXT).
For SLUE-NER, concatenated speech-text im- proves performance over the model trained with only speech units (row (B)) when fine-tuned on speech. Unlike SLUE-SA, which is a classifica- tion task, here we need to generate the correspond- ing transcription along with the named entity tags for SLUE-NER. Hence, the model (row (E)) fine- tuned on speech benefits directly from the extra speech-text data. We discuss the implications of the fine-tuning results further in Sec 5.7.
FT data
SP
TXT
TOW
unit
Eval set
SP
TXT
SP
TXT
Baseline
54.5
(A)
HuBERT
uLM
62.9
(B)
SP 10k
uLM
64.4
(C)
SP 32k
uLM
62.5
(D)
SP 10k
uLM+1LM
63.2
1.5
0.0
66.8
For speech text continuation, when only using concatenated speech-text data (CST) as our mixed data, there are no special tokens (<U2T>, <T2U>) to trigger modality switching. As shown in Table 6, in the u2t case the model trained with CST simply transcribes the speech prompt into its transcription on u2t and synthesizes the text prompt into speech units, resulting low PELMs for u2t and t2u in row (D) due to the repetition. PELM fails to reflect the quality of the continuation accurately. We discuss this limitation further in See 5.6.
(E) SP 10k
uLM+CST
65.0
3.6
0.5
79.5
uLM+CST
(F) SP 10k
66.6
25.2
0.3
77.2
+AST+LM
Table 5: The F1(%) score on SLUE-NER. FT data in- dicates the model is fine-tuned on speech (SP) or text (TXT). Eval set denotes the fine-tuned model is tested on speech (SP) or text (TXT).
tures the relationship between speech and text bet- ter than models trained with unpaired data alone. For SLUE-SA, the model pre-trained with CST shows comparable performance when fine-tuned on one modality and evaluated on the other. The performance when fine-tuning on text and testing on speech is even better than directly fine-tuning on speech (0.51 vs. 0.48). The reason is likely
5.5.4 Does alternating speech-text (AST) help learn shared representations?
This section discusses the benefits of alternating speech-text (AST). The results are presented (row (F)) in Table 3 for automatic metrics, Table 4

---

for SLUE-SA, and Table for SLUE-NER. By comparing the results of CRA for 12u and u2t in row (F) with those in row (E) in Table 3, we observe an improvement in CRA when the data is directly constructed to switch modalities on word boundaries. We can also that CRA is similar for (2u, u2t, and t2t. It suggests that the model learns match context regardless of modality.
tion of bi-grams in continuations that have ap- peared in the prompt transcription. For row (E), the proportions are 0.02, 0.53, 0.42, and 0.02 for u2u, u2t, 12u, and 12t, respectively. For row (F), the proportions are 0.02, 0.03, 0.01, and 0.03. For row (E), the continuations for u2t and (2u are simply repeating the content of the prompt.
We can see that the u2t and 12t PELMs are lower than the ground truth PELM. This is because of the use of the temperature of 0.6 in the softmax layer, which likely hurts diversity and coherence as in (Caccia et al., 2018; Lakhotia et al., 2021).
In row (F), PELM for 12u is lower than PELM for u2u as the text prompt is less noisy than speech units. PELM for u2t is only marginally worse than 12t. This shows that the LM trained with AST can continue a sentence regardless of the modality. The worse PELM for u2u and 12u than for u2t and t2t could be attributed to the recognition errors within our unit transcriber.
5.7 Implications for SLU Downstream Tasks
We show that mixing speech units and text im- proves the cross-modal ability of the model. In SLUE-SA, the mixed speech-text data enables the model to zero-shot transfer between speech and text. In SLUE-SA, we remove the output layer from the SUTLM and attach classification head so the model will always output valid class.
Regarding SLUE-SA, we can observe that AST and tLM further improve the cross -ross- transfer performance (trained on the text and evaluated on speech, or vice versa) in row (F).
In SLUE-NER, row (F) also shows better per- formance than row (E) when fine-tuned on speech and evaluated on speech. There is also non-trivial speech-to-text transfer (fine-tuned on speech and evaluated on text) in row (F), showing that AST helps in learning transferable features between modalities.
In SLUE-NER, using mixed speech-text data directly improves the performance. Since this is sequence generation task, the mixed speech-text data helps the model generate better text. The trans- fer from speech to text is non-trivial but not vice versa. This finding aligns with the experiments in (Bapna et al., 2022), in which they also find non- non-trivin trivial transfer from speech to text but not the other way around. However, we note that different fine-tuning strategies can produce different results, as demonstrated in (Liu et al., 2021).
In SLUE-NER, when fine-tuned on text and eval- uated speech, there is no transferability between speech and text. The reason can be attributed to the fine-tuning task becoming almost trivial. In text NER, in our formulation, the input and output are nearly identical. The only difference is the named entity tags. Further discussion of downstream task performance can be found in Sec 5.7.
Conclusion
Our study on joint language modeling for speech units and text involved developing evaluation met- rics and fine-tuning the model on speech and text data. We found that using mixed speech-text data improves the model's cross-modal ability and per- formance on both automatic metrics and down- stream tasks.
5.6 Limitations of PELM
We use PELM as a metric to measure the quality of continuations. However, although our SUTLM (row (F)) shows the ability to continue after a cross- modal prompt, the resulting continuation is still only locally consistent as shown in Table 6. This be attributed to the of 250M-parameds model architecture, which is relatively small in the era of LLMs.
Our study sheds light on the benefits of consid- cring both speech and text in building language models. We hope that this research will motivate the research community to further explore the in- tegration of speech and text data for more compre- hensive language modeling.
The PELM metric fails to accurately reflect the result in the case of row (E) when the model sim- ply repeats the prompt. It has been known phc- nomenon that LMs tend to assign high probability repeated tokens (Holtzman et al., 2019).
Future work in this area could involve investigat- ing the optimal balance between speech and text data in model training and exploring ways to handle modal data beyond the speech-text domain.
To quantify repetition, we compute the propor-

---

7 Limitations
A unified encoder for speech and language model- ing via speech-text joint pre- ure-training. arXiv preprint arXiv: 2 10. 10329.
Our approach involves using a speech tokenizer that can encode phonetic information (HuBERT) and an off-the-shelf speech recognizer to generate word-level alignment. For other, lower-resource languages, these components may be harder ob- tain or may not perform as well.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1197-1901.
For our proposed automatic metrics, the com- plexity of CRA grows at a rate of ((2)), which can be expensive when evaluated on a larger num- ber of utterances or when scaling up the model size. PELM, the other hand, also has limitations as stated in Sec 5.6. For the empirical results down- stream tasks, we test our SUTLMs on the SLUE benchmark, which has only two tasks. Extending the experiments to more downstream tasks may provide more insights.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2018. Language gans falling short. arXiv preprint arXiv:
Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul- Ambroise Duquenne, Holger Schwenk, Hongyu Gong, et al. 2022a. Speech-to-speech translation for a real world unwritten language. arXiv preprint M.Ki:22111.04177
Finally, we only study relatively small SUTLMs (350M parameters). It is unclear how scaling it up would affect the results.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li. Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. 2021. WavLM: Large-scale self-supervised pre-training for full stack speech processing. arXiv preprint 0rXiv:2110.139)
References
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei- Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. 2023. Scaling laws for genera- tive
Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen. 2022b. MAESTRO: Matched speech text representations through modality matching, arXiv preprint 21KV::2204003409
language models. arXiv preprint
arXiv:2301.03722
Junyi Ao, Rui Wang. Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko. Qing Li, Yu Zhang. et al. 2021. SpeechT5: Unified-modal encoder- decoder pre-training for spoken language processing. arXiv preprint 10.02255
Yong Cheng. Yu Zhang, Melvin Johnson, Wolfgang Macherey, and Ankur Bapna. 2022. MLA: Multitask, multilingual speech and language mod- els. arXiv preprint arXiv:2212.0955:
Yu-An Chung, Chenguang Zhu, and Michael Zeng. 2020. SPLAT: Speech-language joint pre-training for spoken language understanding. arXiv preprint arXiv:2010.02295.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively- multilingual speech corpus. arXiv preprint arXiv:1912.06670.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised lingual representation learning at scale, arXiv preprint arXiv:
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for supervised (carning of speech representations. Advances in Newral Information Processing Systems, 33: 1 12449-12460.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. 2018, Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint w.0v:1810.04800.
Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022. mSLAM: Massively mul- tilingual joint pre-training for speech and text. arXiv preprint arXivv2202.0137..
Anmol Gulati, James Qin, Chung- -Cenng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu. Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020. Conformer: Convolution-augmentee trans- former for speech recognition. arXiv preprint arXiv:2005.08100
Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. 2021. SLAM:

---

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.00
Xiao Liu, Yanan Zheng. Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too, arXiv preprint arXiv: 103. 10385.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-333 19:3451-3460.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech align- ment using kaldi. In Interpeeech, volume 2017, pages 498-502.
Tu Anh Nguyen, Wei-Ning Hsu, Antony d' Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Re- mez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. 2023, Expresso: benchmark and analysis of discrete expressive speech resynthesis. Interspeech.
Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Don- ley, and Yossi Adi, 2022. ReVISE: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement. arXiv preprint arXiv:2212.1137..
Myle Out, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for se- quence modeling. arXiv preprint arXiv: 7 1904. 01038.
Wei-Ning Hsu and Bowen Shi. 2022. u-HuBERT: Uni- fied mixed-modal speech pretraining and zero-shot transfer to unlabeled modality. Advances Neurul Information Processing Systems.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ast cor- pus based on public domain audio books. In 2015 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEF.
Eugene Kharitonoww, Ann Lee, Adam Polyak, Yossi Adi Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Mor- gane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. 2021. Text-free prosody-aware gen- erative spoken language modeling. urXiv preprint arXiv:2109.03264
Ankita Pasad, Bowen Shi, and Karen Livescu, 2023, Comparative layer-wise analysis of self-supervised speech models. In ICASSP.
Diederik P Kingma and Jimmy Ba. 2014, Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.9900.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deIrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentan- gled self-supervised representations. arXiv preprint a0Xiv:2104.00355.
Taku Kudo and John Richardson. 2018. SentencePiece: simple and language independent subword tok- enizer and detokenizer for neural text processing. arXiv preprini 188.v:1808.06225
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte. Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. 2021. On generative spoken lan- guage modeling from raw audio. Transactions of the Association for Computational Linguisties, 9:1336- 1354.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MI.S: A large- scale multilingual dataset for speech research. arXiv preprint ar 2r1iv:2012.034/1
Ofir Press and Lior Wolf. 2016. Using the output em- bedding to improve language models. arXiv preprint 1rXiv:1688.0558
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri. Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, et al. 2021a. Direct speech-to- speech translation with discrete units. arxiv preprint arXiv:2107.05600
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. 2018 Improving language under- standing by generative pre-training.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. 2021b. Textless speech-to- speech translation on real data. arXiv preprint 12200:2112.08352.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J Han. 2022. SLUE: New benchmark tasks for spoken language un- derstanding evaluation on natural speech. In ICASSP 2022-2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 7927-7931. IEEE.
Guan-Ting Lin, Yung-Sung Chuang, Ho-Lam Chung. Shu-wen Yang. Hsuan-Jui Chen. Shang- Wen Li, Ab- delrahman Mohamed, Hung-yi Lee, and Lin-shan Lee. 2022. DUAL: Textless spoken question answer- ing with speech discrete unit adaptive learning. arXiv preprint 2203.020003449

---

Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, al. 2022. Unified speech-text pre-training for speech translation and recognition. arXiv preprint arXiv:2204.05409.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processing systems, 30.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu. Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPop- uli: large-scale multilingual speech corpus for rep- resentation learning, semi-supervised learning and interpretation. arXiv preprint MxXi:2101.000%%
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023. Neural codec language models are zero-shot text to speech synthe- sizers. arXiv preprint arXiv:2301 22110121.1...
Felix Wu, Kwangyoun Kim, Shinji Watanabe, Kyu Han. Ryan McDonald, Kilian Q Weinberger, and Yoav Artzi. 2022. Wav2Seq: Pre Pre-training speech-to- text encoder-decoder models using pseudo languages. arXiv preprint arXiv:2205.0100 00066.
Shu-wen Yang. Po-Han Chi. Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy Liu, Jiatong Shi, Xuankai Chang, Guan- Ting Lin, et al. 2021. SUPERB: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.01051.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, al. 2022a. OPT: Open pre-trained transformer language models. arXiv preprini arXiv:2205.01068
Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, et al. 2022b. SpeechLM: En hanced speech pre-training with unpaired textual data. arXiv preprint arX9v:2209. 15329.

---

A
Appendix
prompt model
they need the same aid and they
modalities
Continuation
u2u
soon will increase and reach the grave more gradually than ever they seek their own home and many little people suffer from the hospitality and care of the poor
(E)
u2t
they need the same aid and they need the same aid
12u
they need they they need the sameling the need the need the same sense of will that they need for others others to to to they will a and and they to have
12t
need the same help they can make and are able to make the hest of them and the best of them
u2u
certainly do need the same money and they do not give themselves out of either field or settlement but they do need it they say and they need it shall be able to perform all the same the king of egypt is not free from the persecution of the jews he is free from the persecution of the jews
(F)
u2t
12u
can no longer face it but though these men believe women now to be the principal duties the progress made will actually continue
12t
will the same as they have been when they were born and they will be the same they now are if they now are
born they would be the same they
prompt
barrack was just coming through the doorway carrying half
model
modalities
Continuation
u2u
doesn papa and several small scotch girls who had been sitting by the fire with him he said to the girls on whom he kept his second mother mother is good to me and so would not let her go away barrek was just coming through the doorway carrying half a dozen other things round of the door he was just coming through the doorway carrying half around of the door they stopped at the back of the doorway and half a around of the doorway and half a round of the doorway
(E)
u2t
12u
12t
dozen men and a dozen men and women believe and i think could have saved the lives of those who loved me-but don't know how i could have saved them
u2u
dozen boxes when saw the stick black inside in his room stepping out of his way into the hall chuckle of joy fell in the drawing room and he seized the boxes and broke down the door cup of coffee when gertie came up with a basketful of rice and a handful of water and then she came to the house of mrs smiths and she said to gertie dozen packs and wrapper and light sparkling light across the face of jack and burning gold bullet and very sharp thumb
(F)
u2t
12u
12t
a dozen toy guns and hundred toy guns and a hundred toy guns
Table 6: Example for speech and text continuation. Speech continuation has been transcribed by the transcriber.


Reviews:
[{"review_id": "2wi2euqVCq", "reviewer": "Reviewer_rsNT", "summary": "", "questions": "1) Why using multi-task learning (uLM + tLM) harms performance? For example, Table 4 and Table 5 show that (D) model has lower performances than uLM only (B) model. Since speech and text share linguistic information, it is likely that other modal performance improvements can be made.\n\n2) If we apply the joint language model tasks beyond the cross-modal transferring, is it still CST or AST works better than uLM+tLM?\n\n3) Line #493, about the curse of multilinguality; Speech-only dataset also contains multilingual data (Multilingual Librispeech, Voxpopuli, CommonVoice), hence model (B) seems to be trained multilingual data. What is the exact meaning of lines # 489~495?\n\n4) Can the authors provide AST-only and CST-only results?\n\n5) Which benefits we can earn by training a joint speech and text language model? It seems not well described in the manuscript.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the feedback.\n\nRe: reject reason (1): Although CRA is almost the same across the u2u cases, it has differences in the u2t and t2u cases for row (E) and row (F) in Table 3. We don’t suggest using CRA as a formal evaluation metric. Instead, we propose it as a proxy for practitioners to understand how well the model can learn to mix the two modalities without actually fine-tuning the LM on a downstream task (similar to perplexity for LMs). As pointed out in the paper, fine-tuning can have multiple variables that can affect the outcome. \nAlso, including more modalities in the training data can cause competition between modalities as shown in [Conneau et. al.](https://arxiv.org/pdf/1911.02116.pdf) and [Aghajanyan et al.](https://arxiv.org/pdf/2301.03728.pdf) CRA can serve as a way to measure how well the model learns to connect the two modalities. \n\nRe reject reason (2): We agree that applying the model to more downstream tasks can also be a valuable study. However, our goal is to understand how much transferability we can get between the two modalities for a joint LM, so we choose mainly spoken language understanding tasks, i.e, SLUE, so that we can fine-tune/evaluate on speech/text in all four combinations. \n\nRe reject reason (3): We don’t have the results for CST and AST only. Practically, CST and AST are obtained from paired speech and text data. With such data, we can create uLM and tLM. Also, it is much easier to collect unpaired data in speech or text. \n\nRe reject reason (4): We proposed CRA and PELM as a way to probe the model instead of using it for formal evaluation. We agree that exploring more downstream tasks can also be a great study. We leave it as future work. \n\nRe question (1): Our interpretation is that when the model is only exposed to unimodal speech and text without knowing the connection between the two modalities, it shares its capacity between the two. As a result, the two modalities are more competitive than when using mixed speech-text data. As shown in Fig.5 in [Aghajanyan et al.](https://arxiv.org/pdf/2301.03728.pdf), we need enough scaling to break the competition. \n\nRe question (2): \nFrom SLUE-NER experiment, we do see an improvement when adding CST and AST (E, F) but not tLM (D). Also, when fine-tuning on text and testing on text (in this case, it becomes a text-only task) on SLUE-SA and SLUE-NER (Table 4), CST and AST also provide improvements. \n\nRe question (3): We'd like to address what seems to be a misunderstanding. Our model was exclusively trained on English data. The concept of the curse of multi-linguility refers to the phenomenon where two types of data (modalities or languages) can compete with each other, as demonstrated in our SLUE-SA experiment. In this experiment, we observed that introducing text data decreased the performance on speech. We will provide further clarification on this aspect in the final version.\n\nRe question (4): We won’t be able to provide these results due to resource constraints. As pointed out in re q(2), uLM and tLM are easier to collect and always come with AST and CST. As a result, we believe that using uLM and tLM along with CST and AST is a reasonable setup. \n\nRe question (5): Potential benefits: As described in the first paragraph of the introduction, adding the two modalities together can have benefits as the data in speech and text have different characteristics (speech: dialog, text: structural knowledge). Our study also shows that mixing the modalities improves transferability for a classification task (SLUE-SA), and improves a text generation task (NER) on speech.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors explored a timely and interesting topic; developing a unified text and speech language model system.\n2. The authors proposed a Context Retrieval Accuracy (CRA) to measure the performance of a cross-modal language model.", "reasons_to_reject": "1. The proposed CRA appears to lack discernment. Since CRA measures the emission probability of all ground-truth continuation words given prompt words, the probability will be high for any well-trained language models. As the metric shows almost the same results on u2u results, it is wondering if CRA can serve as a reliable evaluation metric.\n\n2. The evaluation is focused on cross-modal transferability only. As the authors proposed a joint language model of speech and text, it seems to be more natural to evaluate the trained model on diverse downstream tasks including speech-only task, text-only task, and speech-text (multimodal) task. The results shown by the authors are only saying that even though we trained a joint language model of audio and speech, the unimodal performance will be degraded and very small cross-modal transferability can be get.\n\n3. The evaluation results of using CST only, AST only, uLM + AST are not shown. Do the CST and AST methods require uLM basically?\n\n4. As the authors pointed out, PELM has limitations for measuring the proposed method. Hence, Table 3 does not give much insight into the benefit of joint language modeling of speech and text. It seems better to explore other downstream tasks instead of using these metrics.", "questions_to_authors": "1) Why using multi-task learning (uLM + tLM) harms performance? For example, Table 4 and Table 5 show that (D) model has lower performances than uLM only (B) model. Since speech and text share linguistic information, it is likely that other modal performance improvements can be made.\n\n2) If we apply the joint language model tasks beyond the cross-modal transferring, is it still CST or AST works better than uLM+tLM?\n\n3) Line #493, about the curse of multilinguality; Speech-only dataset also contains multilingual data (Multilingual Librispeech, Voxpopuli, CommonVoice), hence model (B) seems to be trained multilingual data. What is the exact meaning of lines # 489~495?\n\n4) Can the authors provide AST-only and CST-only results?\n\n5) Which benefits we can earn by training a joint speech and text language model? It seems not well described in the manuscript.", "paper_topic_and_main_contributions": "This paper is about developing a joint language model of speech and text. The authors proposed to learn a language model with both speech and text inputs by concatenating the two representations or alternating them. Specifically, they proposed to use speech units for speech inputs by quantizing features of HuBERT and applying sentencepiece. They also introduce an evaluation metric to measure the cross-modal transfer ability of a trained model. The topic handling in the paper is very interesting but the evaluation seems insufficient.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line #273: highe4 -> higher"}, "extra_text": {"Missing_References": "Zhang, Ziqiang, et al. \"SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022."}}, {"review_id": "9DeBSl0vof", "reviewer": "Reviewer_HZX6", "summary": "", "questions": "A) What is the baseline model mentioned in Tables 4 & 5? I couldn't find a description of the baseline system in the text.\n\nB) Any idea on how the model performs for text-only tasks and generation-tasks? (if y'all have additional experiments that didn't make it to the paper)\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. \n\nRe reject reason (A): \nSimilar to [Conneau et. al., 2022](https://arxiv.org/abs/2205.12446), we also use multi-modal retrieval as a way to probe the model. Instead of using a fixed-size vector in an encoder-only model, we modify the retrieval target to the context to fit the auto-regressive nature of LM. It also has a PMI interpretation as shown in eq(3). While only row (F) is explicitly trained to continue the sentence, row (E) also shows a non-trivial CRA and non-trivial transferability at the same time. \n\nRe reject reason (B): \nNER is not a classification task, since it involves locating and labeling an arbitrary number of NEs.  As in prior work on spoken NER, we formulated the task as a generation task, as described in L405-409. It would be great if we could compare it to SLAM. However, SLAM is not open sourced so we won’t be able to directly compare to it using the same setup. \n\nRe question (A):\nThe baseline is a fine-tuned HuBERT from [Shon et.al.](https://arxiv.org/pdf/2111.10367.pdf). We will add the reference to the final version. \n\nRe question (B):\nIn the SLUE-NER experiment, when fine-tuning on text and testing on text, it can be considered a text-only / generation task. Also, we would like to point out that the NER task is very similar to ASR, but we only evaluate the named entity spans.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A) Analysis: Detailed analysis of the contribution of each pre-training objective: uLM, tLM, CST and AST. The paper goes beyond simply stating results and analyses why they may be so.\n\nB) Writing: Clear writing that is easy to understand. Situates the work well in the related-work section.", "reasons_to_reject": "A) CRA metric: The CRA metric tests what the AST metric is optimizing. While I see value in cross-modal retrieval, i.e., retrieving a relevant speech sample for a given text sample and vice-versa (for example, in audio mining for SSL), I am not sure why intra-sentential cross-modal completion is relevant. Cross-modal retrieval has already been addressed by past works for example in [Conneau et. al., 2022](https://arxiv.org/abs/2205.12446). It is also not clear how CRA correlates with downstream task performance for it to be advocated as a replacement for fine-tuning (as mentioned in L070) because only the final model (Row F) can be used to measure CRA across all 4 directions.\n\nB) Experimental Setup: The baselines used are not mentioned (as far as I could search). Since sentiment analysis and NER are classification tasks, the paper can include encoder or encoder-decoder models as baselines as well (like [SLAM](https://arxiv.org/pdf/2202.01374.pdf)?). Further, since this is a multimodal decoder-only model, I was expecting evaluation on generation tasks like ASR and text-only tasks (like NLI or NER) to fully grasp the capabilities of the model. That said, I completely understand if it was out of scope for the current work given resource constraints. In that case, it may be helpful to lay out the scope as such.", "questions_to_authors": "A) What is the baseline model mentioned in Tables 4 & 5? I couldn't find a description of the baseline system in the text.\n\nB) Any idea on how the model performs for text-only tasks and generation-tasks? (if y'all have additional experiments that didn't make it to the paper)\n", "paper_topic_and_main_contributions": "This paper identifies a gap in the joint modeling of speech and text. Prior works have focused on encoder-only or encoder-decoder architectures while this work proposes to build a multimodal (speech-text) decoder. The decoder is trained with four objectives: a) unitLM (uLM) : maximizes the log-probability of discretized speech-only data (speech units obtained using the pre-trained HuBERT model); b) textLM (tLM): maximizes the log-probability of text-only data; c) concatenated speech-text (CST): maximizes the log-probability of discretized speech and its text transcript concatenated together; and d) alternating speech text (AST): mixed speech-text data is constructed by switching between the two in a single utterance. The paper additionally proposes an evaluation metric for evaluating speech-text models without fine-tuning, called \"context retrieval accuracy\" (CRA), which aims to measure whether a model can retrieve the correct continuation for a given prompt, in a cross-modal way too. Evaluation is done on SLUE sentiment analysis and named entity recognition.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {"Missing_References": "For CRA y'all can cite this work which does cross-modal retrieval:\n\n@inproceedings{conneau2023fleurs,\n  title={Fleurs: Few-shot learning evaluation of universal representations of speech},\n  author={Conneau, Alexis and Ma, Min and Khanuja, Simran and Zhang, Yu and Axelrod, Vera and Dalmia, Siddharth and Riesa, Jason and Rivera, Clara and Bapna, Ankur},\n  booktitle={2022 IEEE Spoken Language Technology Workshop (SLT)},\n  pages={798--805},\n  year={2023},\n  organization={IEEE}\n}"}}, {"review_id": "a2jWnILymj", "reviewer": "Reviewer_oo6z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments.\n\nIn this work, we mainly study tasks related to semantic understanding, which is more likely to benefit from the text. For more acoustic-level speech generation tasks, one may consider extending the joint LM to finer-grained tokens such as in AudioLM, or adding a vocoder to synthesize speech from Hubert units.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The writing in the paper is clear, and the authors introduce a new evaluation metric. Also, experiments comparing the effectiveness of different levels of speech tokenizers and various methods for mixing speech-text data are conducted.", "reasons_to_reject": "In works such as [AudioLM](https://arxiv.org/pdf/2209.03143.pdf) and [SpearTTS](https://arxiv.org/pdf/2302.03540.pdf), HuBERT is understood to be biased towards semantic rather than acoustic tokens. The proposed CRA metric and downstream tasks such as sentiment analysis, NER, and ASR also seem to focus more on language aspects. This paper may be more focused on modeling text rather than jointly modeling speech and text. However, the paper emphasizes the model’s cross-modal capability multiple times. To further support the effectiveness of the model in speech-text joint modeling, it would be beneficial to consider adding tasks related to speech, such as speech reconstruction.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The article focuses on joint modeling of speech units and text, empirically validating that a larger span tokenizer achieves better modeling results. Additionally, the authors propose an automatic evaluation metric that does not require fine-tuning to assess the model’s cross-modal transfer capability.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper looks at combining discrete speech units (automatically extracted with a self-supervised speech model that quantises its input) with text. The proposed approach is applied to spoken language understanding tasks. The reviewers commented positively on the writing of the paper and the thoroughness of the analysis (good soundness scores). But some of the reviewers were concerned about limitations in the experiments, e.g. in the type of downstream tasks used (ambivalent and mediocre excitement).
Decision: Accept-Findings