UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation
Authors: Alexander H. Liu
Sang-gil Lee
Chao-Han Huck Yang
Yuan Gong
Yu-Chiang Frank Wang
James R. Glass
Rafael Valle
Bryan Catanzaro
Venue: ICLR 2025 Poster (2025)
Keywords: speech foundation model
generative pre-training
self-supervised learning
speech generation
speech tokenization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: First unified pre-training method for speech representation learning and generation

Abstract: Pre-training and representation learning have been playing an increasingly important role in modern speech processing.  Nevertheless, different applications have been relying on different foundation models, since predominant pre-training techniques are either designed for discriminative tasks or generative tasks. In this work, we make the first attempt at building a unified pre-training framework for both types of tasks in speech. We show that with the appropriate design choices for pre-training, one can jointly learn a representation encoder and generative audio decoder that can be applied to both types of tasks. We propose UniWav, an encoder-decoder framework designed to unify pre-training representation learning and generative tasks. On speech recognition, text-to-speech, and speech tokenization, UniWav achieves comparable performance to different existing foundation models, each trained on a specific task. Our findings suggest that a single general-purpose foundation model for speech can be built to replace different foundation models, reducing the overhead and cost of pre-training.

Published as a conference paper at ICLR 2025
UNIWAV: TOWARDS UNIFIED PRE-TRAINING FOR SPEECH REPRESENTATION LEARNING AND GENERA- TION
Alexander H. Liu't, Sang-gil Lee", Chao-Han Huck Yang", Yuan Gong't Yu-Chiang Frank James R. Glass', Rafael Valle?, Bryan Catanzaro 'IIT CSAIL, "NVIDIA alexhliu@me edu
ABSTRACT
Pre-training and representation learning have been playing an increasingly im- portant role in modern speech processing. Nevertheless, different applications have been relying on different foundation models, since predominant pre-training techniques either designed for discriminative tasks generative tasks. In this work, we make the first attempt at building a unified pre-training framework for both types of tasks in speech. We show that with the appropriate design choices for pre-training, one can jointly learn a representation encoder and generative audio decoder that can be applied both types of tasks. We propose UniWav, an mncoder-decode framework designed to unify pre- pre-training representation learning and generative tasks. On speech recognition, text-to-speech, and speech tokeniza- tion, UniWav achieves comparable performance to different existing foundation models, each trained on a specific task. Our findings suggest that a single general- purpose foundation model for speech can be built to replace different founda- tion models, reducing the overhead and cost of pre-training. Audio demo page: https://alexander:
github. i /uniiniii wav-demo. githhu..iii io/
INTRODUCTION
Representation learning and generative modeling have seen rapid growth in speech processing in recent years. Researchers have been consistently setting new records on discriminative tasks with representation learning (Baevski et al., 2020b: Hsu et al., 2021: Baevski et al., 2022; Liu et al., 2024). Existing speech representations focus on excelling at specific task. Mainstream pre-training methods (Schneider et al., 2019; Baevski et al., 2020a;b; Hsu et al., 2021), also known self- supervised learning, learn speech representation from massive unlabeled speech. While there are many different prior works in the field, they all shared the same spirit in masked audio modeling, i.e., matching the output extracted from masked audio to that from unmasked audio. These representations tend to capture the phonetic semantic content of speech, leaving out acoustic information such as speaker's 's timbre and the acoustic cavironment. Consequentially, self-supervised representations have dominated classification and recognition tasks (Yang et al., 2021) in speech but not for generation tasks (Tsai et al., 2022). In fact, recent research (Wang et al., 2023) argues that self-supervised representations (Hsu et al., 2021) are sub-optimal for speech synthesis.
On the other hand, generative models have also gained more attention in recent years. State-of-ther models are capable of generating realistic speech with arbitrary voice and text (Wang al., 2023; Le et al., 2023), also supporting a wild range to generation tasks beyond text-to-speech (Wang et al., 2024). Similar representation learning, pre-training methods for speech generative models have also been developed (Liu et al., 2023; Vyas et al., 2023) based on masked audio modeling. These models also are pre-trained on unlabeled data and can be easily transferred to different downstream tasks with minimal supervision.
Work done during an internship Nvidia. Work done MIT, now with xAI Corp.

---

Published as a conference paper at ICLR 2025
Interestingly, while the two families of models are similar in many ways- e.g.. training transformers, applying masked audio modeling, pre-training with unlabeled data, etc. - they are normally designed and trained separately.
Ideally, speech representation learning and generation should be complementary, as representation learning can be used guide generation, and generation can be used to ensure that the learned repre- sentation encodes sufficient information. However, combining them is challenging due to conflicting methods and arguably opposing requirements for each task (Baevski et al., 2022). For example, while speech recognition tasks can benefit from a representation that is invariant to speaker and environment (Schneider et al., 2019; Hsu et al., 2021), speech synthesis tasks require a representation that provides information about speaker and environment (Wang et al., 2023; Zhang et al.. 2023b).
Nevertheless, there are successful examples on text (Devlin al., 2018: Radford al., 2019) and image (Li et al., 2023; Xiang et al., 2023; Chen et al., 2024) that described unified foundation models for representation learning and generation. These prior works showed that a unified pre-training framework for representation learning and generation can reduce the overhead of training, while accommodating both discriminative and generative downstream applications with strong performance.
Our proposed framework, UniWav, is the first to present unified pre-training framework for speech that efficient on both discriminative and generative tasks. It is an encoder-decoder framework, integrating representation encoder and a Flow Matching (Lipman et al., 2023) decoder condi- tioning on the learned representations. The representation encoder is guided simultaneously by self-distillation (Baevski et al., 2022; Liu et al., 2024) and the generative objective through the decoder. By training the encoder and decoder jointly in an end-to-end manner from scratch, we build the first foundation model that excels in both speech recognition and generation. In our experiments, we fine-tune the pre-trained model for speech recognition and in-context text-to-speech synthe- sis. achieving results comparable to satte-of-the-as methods in each task. Additionally, we adapt UniWav for speech tokenization, demonstrating the significant benefits of joint representation and generation learning by achieving high audio quality at low bitrate. Finally, we provide observations and insights on unified pre-training through an ablation study and analysis, aiming to illuminate the path toward a single foundation model for general speech processing.
Our contribution can be highlighted as follows:
We introduce, UniWav, the first unified pre-training framework for speech representation learning and generation.
On speech recognition and in-context text-to-speech generation, we show Uni can compete with different foundation models within each task through fine-tuning.
By introducing discrete bottleneck between the encoder and decoder, UniWav achieves low- bitrate speech tokenization and high-quality resynthesis, outperforming existing methods.
In summary, UniWav marks the first step toward unified pre-training for both representation learning and generation, laying the foundation for more versatile pre-trained models in speech processing.
METHOD
Figure 1 provides an overview of UniWav, an encoder-decoder framework designed to unify pre- training for representation learning and generative tasks. The model takes the surface feature (e.g.. Encodec (Défossez et al., 2022) latent or mel spectrogram) of speech as input. Through masked audio modeling, the model jointly learns encode input into latent representations and generate speech. Below, we detail encoder design in 1, decoder in $2.2. and summarize the pre-training framework in $2.3.
2.1 REPRESENTATION ENCODER
The key to learning representations from unlabeled data is to derive good learning targets from the input itself, commonly referred to as self-supervised learning. While there are different self- supervised learning methods in speech (see related works $4), we adopt oiff-distillation and online clustering from DinoSR (Liu et al., 2024) in this work.
2

---

Published as a conference paper at ICLR 2025
Pscudo-label
Cross Entropy
Flow Matching
Encoder (82.1)
Decoder (52.2)
Teacher (EMA)
Random masking
Figure I: An overview of UniWav. The encoder is trained with masked audio modeling and pseudo- label obtained through a teacher model. The teacher model is the exponential moving average (EMA) of the encoder. The decoder is trained with Flow Matching conditioned z the weighted sum of representations of different encoder layers. All modules are trained jointly from scratch.
x2) where x; R is the input surface feature frame of speech, we randomly mask a subset of the frames, denoted M, and feed them into the Transformer encoder (Vaswani al., 2017) to derive output latent embeddings where 2, Rd. The goal of the encoder is to predict the pseudo label y/ provided by a teacher model by maximizing
Given the sequence of input audio frames x (X1, X2,-..;
(z), 22, Zz,
log (z:)
M
i..EMMM
where ø is the pseudo-label prediction head (linear layer followed by softmax activatiton).
The teacher model for deriving pseudo-label is the exponential moving average (EMA) of the encoder itself, i.e., fleacher <- "eecher Oteacher + (1 "Nencher) fencoder where "eeaher is the decay factor. While both networks are randomly initialized and set to be identical at the beginning of training, only the encoder takes masked audio as input. In contrast, the teacher model takes complete audio (without masking) as the input to derive the pseudo-label
Y/ argmin wEV
eull2 17:1-el12
for input frame x,, where ă; is the output of the teacher model and e, is codeword from codebook of size V. All codewords within the codebook are also updated by taking the EMA of the teacher output embeddings that share the same pseudo-label, i.e..
Sp + Vccade S,, + (1 Tande)
Zis
Thg Vcade nhe + (1 Teude)
1.
Su
C
n2p
where "ouue is the decay factor, the cluster sum s, Rdd is randomly initialized, and the cluster size n. is initialized to 1.
Following DinoSR, we consider pseudo-labels from all the top K layers of the teacher model by repeating (1)(2), and (3) for each layer. The complete loss function of the encoder is therefore
Lencuder
Σ
log (vi
(k) (z:).
K M
KEK
where is the of indices of the teacher model layer to be considered, each with an independent codebook. Note that the prediction is made at the final layer, using z,, of the encoder regardless of the target layer k.
2.2 FLOW MATCHING DECODER
We choose train decoder with Flow Matching (Lipman et al., 2023), which has been shown to be highly effective and generalizable in speech generation tasks (Le et al., 2023; Liu et al., 2023; Kim et al., 2024; Mehta et al., 2024).
3

---

Published as a conference paper at ICLR 2025
Formally, Flow Matching considers the problem of learning to generate samples 21 of unknown data distribution g(æ) by predicting the flow of x along the path between prior distribution po and the target distribution p1 = q. The flow is defined using ordinary differential equation (ODE):
dt 0((x) = (x(()
Φ0(x) = x::
(5)
with the dime-dependent vector field [0, x F - R. Through the change of variable formula, we can generate the distribution p. at any timestep t along the path, allowing us to sample p1 by solving E5.(5) iteratively from i 0 / 1.
Under the framework, the goal of our decoder is to predict the time-dependent vector field (a (deaader) that matches the target vector field 4, that generates the path. While the frame- work is simple, naive Flow Matching regressing decoder output to u, is intractable since the target vector field uN is generally unknown in practice. Fortunately, Lipman al. (2023) showed that we can leverage real data sample Ty to construct the conditional flow of æ with tractable pe and u, Specifically, the Optimal Transport (OT) conditional path is constructed by assuming the mean (z) = tæı and standard deviation oz (x) = 1 - (1 - min)) )t to change linearly in time. with a sufficiently small miin such that p1 (x|(r1) centered around æ1. This results in the OT conditional flow
Pt (x) (1 - (1- min - tæ
(6)
with tractable (a(x)a1)) N/ (11)) (( and u1(x)<1) (2) (01-(1100 (1-((1))))))) deriving the Conditional Flow Matching (CFM) objective
(1-(1-omm))
(a) - -(1 Tmin mmin) ) )|²
E Ee.q(e)). (e( (vo); Odecnder)
(7)
where Po is the standard normal distribution and / is sampled uniformly from [o, 1]. It is worth mentioning that Lipman al. (2023) proved the CFM objective and the naïve Flow Matching share identical gradient w.r.t. the model, therefore (5) can be solved with the model trained with CFM.
We additionally condition the decoder on the weighted sum of the encoder layer features z, yielding the complete loss function for the decoder:
Leccodrr (e,g(z)) 11 (00T 2; decoder) [1
o mia ) )|²
Similar to the representation encoder, the generative decoder is also implemented with Transformer encoder with d-dimensional latent. The input of the decoder is
Worr (zo))
(2OT)))) + 2, where
Σ Wiz(
(9)
Wo E Rd,xd is the noisy data projection layer, and W, E Rddd the linear projection layer for the representation 2(0) from the i-th layer of the encoder. In other words, the decoder takes the noise-injected audio feature x and a learnable combination of representations from all encoder layers z as input.
2.3 UNIWAV FRAMEWORK
Combining the encoder objective (4) and the decoder objective (8), UniWav is trained from scratch an end-to-end manner with the overall objective
Lencuder À Ldecoderss
(10)
where À is the weighting scalar. In practice, we found downstream performance less sensitive to A but a sufficiently small value stabilizes pre-training. We use X 0.25 throughout this work. Note that for both encoder and decoder, we only compute loss at the masked positions.
Unlike prior works that rely on iteratively learning representation from teacher models (Hsu et al., 2021; Chen et al., 2022a), adopting
and online clustering allows training the encoder
In this subsection, denote the sequence of audio frames as Z1, i.e., Ji = (x1: X2. X1), ), to follow the convention in the literature.
4

---

Published as a conference paper at ICLR 2025
from scratch together with the decoder. In practice, also tested self-distillation without cluster- ing (Baevski et al., 2022) and distilling from a pre-trained HuBERT (Hsu et al., 2021), but found the overall encoder-decoder training less stable. For the decoder, tested reconstructing discrete code from Encodec (Défossez et al., 2022) and found Flow Matching better. We leave exploring more combinations of the encoder and decoder
important future work.
After pre-training, the encoder a stand-alone module that can be used to extract representations similar those of prior works. We evaluate the representation encoder with speech recognition in $3.2 and speech tokenization in $3.4. Additional representation analysis is presented in $3.5. For generation, the decoder can be fine-tuned using desired conditions such as text and audio prompt ($3.3) and discretized representations ($3,4).
3 EXPERIMENTS
3.1 PRE-TRAINING SETUP
Model Both the encoder and the decoder follow the Transformer architecture (Vaswani et al., 2017), with 16 attention heads, hidden size d = 1024, and feed-forwar networks with 4096 dimensions. For the encoder, positional information is encoded using convolutional positional embedding (Baevski et al., 2020a). We add skip connections between decoder layers mimic the U-Net (Ronneberger et al. 2013) architecture, following prior works (Le et al., 2023; Liu et al., 2023). Both the encoder and the decoder use the ALiBi self-attention bias (Press al., 2021). We stack 24 layers for the encoder and 12 layers for the decoder. To condition the Flow Matching decoder on ODE timestep I, a sinusoidal embedding of t prepended the input sequence of the decoder. The total number of learnable parameters is around 500M, where the encoder accounts for 60% of the size. Studies on the size of the encoder and the decoder are presented in $3.5.
Data & Input Feature UniWav is trained on LibriLight (Kahn et al., 2019), an audiobook corpus with around 60k hours of untranscribed English speech sampled at 16kHz. We extract the pre- quantized latent feature from 16kHz EnCodec (Défossez et al., 2022) encoder, pre-trained on the same dataset, to serve as the input for our model. The input feature is comprised of 50Hz 128-dimensional vectors, normalized to zero mean and unit variance overall. For generation tasks, the features sampled by the decoder can be projected back to waveform using the EnCodec decoder. Choice of surface feature is studied in $A.1 by replacing EnCodec latent with mel spectrogram.
Syper-parammters & Training details For the encoder teacher model, Tieacerr is set to increase from 0.9997 to 1.0 in the first 400k updates, and "eode is set to 0.9. The top K 10 layers of the teacher model are considered a learning target for the encoder, and the representation from each layer is clustered to a codebook of size V = 256, following Liu et al. (2024). For masking, each input frame has 8% chance of being replaced with learnable mask embedding. We mask 10 consecutive frames a frame is sampled to be masked. We set the decoder loss weight À to 0.25. UniWav is trained using Adam optimizer (Kingma& & Ba, 2014) with cosine learning rate schedule peaking at 2e-4 for a total of 600k updates, including 10k warmup steps. The batch size is 312.5 seconds per GPU, and samples are randomly cropped to cap 20 seconds. The model is trained with bf16 and gradient clipping of 1.0. Pre-training is done on 16 H100 GPUs taking around 9 days.
3.2 SPEECH RECOGNITION
Setup We follow the prior works in self-supervised representation learning (Baevski et al., 2020a; Hsu et al., 2021; Chen et al., 2022a; Baevski et al., 2022) to evaluate the UniWav encoder on speech recognition. A linear layer for letter prediction is attached to the final output layer of encoder, and the whole encoder is fine-tuned with CTC (Graves et al., 2006). In alignment with the prior works, we fine-tune our encoder using either 960 hours of transcribed speech from LibriSpeech (Panayotov et al., 2015) or 100 hours from its decoding script from Baevski et al. (2020a), and perform hyperparameter search over learning rate, mask probability, mask length, and input channel dropout probability. For decoding, the LibriSpeech 4-gram language model is used.
100 subset. We use the open-sourced fine-tuning and
5

---

Published as a conference paper at ICLR 2025
Table Comparison of speech foundation models. For speech recognition, report word error rate (WER) on the test-other subset of the Librispeech (Panayotov et al., 2015) dataset when fine-tuning with 960 hours or 100 hours of labeled data. For speech synthesis, we report ASR-measured WER (ASR-WER) and speaker similarity (Sim.) for in-context text-to-speech.
Pre-training
Speech Recognition
Speech Synthesis
data (hr)
100hr
960hr
data
ASR-WER Sim.
Sclf-supervised Representation Models HuBERT (Hsu et al., 2021)
60k
4.5
3.7
WavLM (Chen et al., 20226)
94k
4.6
data2vec (Baevski et al., 2022)
4.6
3.7
data2vec 2.0 (Baevski et al., 2023)
4.3
3.5
Generative Models
VALL-E (Wang et al., 2023)
60k
3.8
0.508
Voicebox (Le at,,,, 2023)
2023)
60k
2.3
0.616
SpeechFlow" (Liu et al., 2023)
60k
960
2.4
0.629
UniWav w/o pre traitrann
17.5
5.8
960
2.6
0.532
UniWav
60k
4.8
4.0
960
2.5
0.635
Inference with time-aligned phone sequence using force alignment (McAuliffe al., 2017); 32 function evaluations.
We report word error rate (WER) on the test-other subset in Table 1. The clear gap between random initialized and pre-trained weights demonstrated the effectiveness of pre- training. Comparing UniWay with prior works on representation learning, we note that our model falls short of state-to-the-art by around 0.5% WER. Nevertheless, we note that prior works in this direction focused exclusively on discriminative tasks, and none of the prior works supported generation tasks. The speech recognition experiment confirmed that the encoder of WniWav can effectively function as the backbone for discriminative tasks, even when pre-trained with the additional demands of generative objective.
Results
3.3 SPEECH SYNTHESIS
Setup Next, we evaluate the speech generation ability of UniWav against existing foundation models on in-context text-to-speech (Wang al., 2023; Le et al., 2023; Liu et al., 2023). We fine-tune UniWav on LibriSpeech to synthesize speech, conditioned on text and audio prompts that provide speaker information. For text encoding, we use input-franee aligned phone labels with position postfixes (Le et al., 2023). Phone embeddings are randomly initialized and added to the model input. For audio prompts, we directly use the encoder representation (z in (8)) where 70% to 100% of the sequence is randomly masked with single mask. We also drop both audio prompt and text condition with a 20% chance for classifier-fret guidance (CFG; Dhariwal & Nichol, 2021; Le et al., 2023).
We fine- tune our model with le-5 learning rate for 1 150k steps with logit-normal time sampling (Esser et al., 2024) on 8 A100 GPUs Note that since goal is to evaluate acoustic modeling, we Montreal Force Aligner (McAuliffe et al., 2017) to obtain alignment for both training and evaluation (and apply the same for prior works that require alignment). For solving the initial value problems in E5)(5), we use the midpoint method from torchći ffeq (Chen, 2018). With step size of 0.0625 and CFG a 1.9, the number of function evaluations (NFEs) is 32 per sample.
For evaluation, we adapt the protocol introduced by Wang et al. (2023) to perform speaker-conditioned text-to-speech on the test-clean subset with 3-second enrollment. We follow Liu al. (2023) to use fine-tuned HuBERT-L (Hsu et al., 2021) to measure word error rate for audio intelligibiliiyy LASR-WERS and WavLM-TDCNN speaker embedding model (Chen al., 2022a) to measure speaker similarity (Sim) for speaker consistency between the audio prompt and the result.
We compared UniWav against prior works using the same setup in Table 1. Compared to Vall-E (Wang et al., 2023) and Voicebox (Le et al., 2023) that relied on speech data with transcriptions, UniWav achieved better similar results with significantly less data. Compared the generative pre- training method SpeechFlow (Liu et al., 2023),, our method shows equal performance in fine- tuning. This finding is important because, unlike SpeechFlow, UniWav can also be applied for discriminative
Results

---

Published as a conference paper at ICLR 2025
Table 2: Speech tokenization and resynthesis results. All models have a frame rate of 50Hz. Methods sharing the same bitrate are highlighted with the same color. Visualized audio samples are provided in $Appendix A.2 and audible samples are available on demo page.
ASR- SER-
UTMOS?
Spkr. Sim.t
ViSQOLt
quantization
bitrate
dev
test
dev
test
dev
test
dev
test
Ground Truth
256k
3,4
3.87
3.77
VQ1024
500
41.7
41.0
1.24
1.24
0.25
0.25
2.96
2.88
Encodee af
RVO Qax 1024
1k
10.3
1.52
1.50
0.60
0.60
3.59
3.47
RVQ 1024
4k
3.8
4.0
2.94
2.82
0.89
0.89
4.25
4.21
VQ
500
80.9
80.6
1.24
1.24
0.11
0.10
1.88
1.73
DAC
RVQ RVQ
1k
22.8
24.2
1.28
1.28
0.33
0.31
2.95
2.70
RVQss
4k
3.8
4.1
3.09
3.03
0.87
0.87
4.42
4.39
HuBERTRAc Base GE HUBERTTLAGGE"
KM
500
9.8
10.2
2.36
2.37
0.14
0.15
2.14
2.15
KM1024
7.6
2.
2.65
0.15
2.00
2.03
VQ
500
10.5
11.0
1.25
1.26
0.16
0.16
1.97
2.03
SpkechTooeemieer
RVQ 2 1024
1k
9.1
2.20
2.08
0.34
0.34
3.04
3.03
RVQ Qsx 1824
4k
4.5
4.9
3.65
3.56
0.83
0.83
4.22
4.20
KM
500
7.2
7.1
3.64
3.59
0.25
0.26
2.14
2.09
UniWaye
KM2x1024
1k
5.6
5,6
3.79
3.72
0,49
0.50
2.96
2.92
"Defossez et al. (2022); Kumar (2024);
With Hiit-HiiGAN Polyak et (2021); Nguyen (2023);
"Zhang
20233);
Residual quantization, see (lT) Trained on general audio; * Trained on LibriSpecch only;
tasks like speech recognition. To ablate the effectiveness of our pre-training method, we report baseline numbers without pre-training Our result suggest that pre-training has more impact on the model's acoustic modeling ability (measured by SIM) and less on the language modeling ability (measured by ASR-WER).
3.4 SPEECH TOKENIZATION AND RESYNTHESIS
Task introduction In addition to purely discriminative and generative tasks, there are tasks that bridge the gap between the two, such as speech tokenization and resynthesis. With the recent advancements in large language models, there has been growing interest in converting speech into sequences of discrete tokens (Zhang et al., 2023b; Zhu et al., 2023; Huang et al., 2023; Bai et al., 2024), allowing speech to be processed similarly text. These tokens must also be capable of being decoded back into audio signals, enabling token-based language models to generate speech.
Prior works We consider four prior works Encodec (Défossez et al., 2022), DAC (Kumar et al., 2024), unit-HiFiGAN based on HuBERT units (Hsu et al., 2021; Polyak et al., 2021; Nguyen et al., 2023), and Speech Tokenizer (Zhang et al., 2023b). Encodec and DAC are general neural audio codee models that are not designed specifically for speech tokenization. HuBERT units and Unit-HiFiGAN (Polyak et al., 2021) is cascaded system for speech tokenization and resynthesis. Tokenization is done by running kmeans clustering on the representation extracted from a pre- selected layer of HuBERT. Resynthesis is done by training HiFiGAN (Kong et al., 2020) that takes speech tokens as input. For Unit-HiFiGAN, we train the model using the codebase released by the authors (Nguyen al., 2023). In addition to HuBERT BASE used in the original work, also report results using HuBERTLAN that matches the size of UniWav encoder. To ensure a consistent setup, we used 1024 kmeans clusters and did not include speaker and emotion conditions match other models. to audio reconstruction and adversarial training used in Encodec, SpeechTokenizer introduced an additional loss to distill representation from a pre-trained HuBERT into the first vector quantization layer. For SpeechTokenize methods required hand-selecting layer from HuBERT' for quantization or distillation, which is done by identifying the layer that matches the underlying phone label the most. Prior works therefore referred representations from the selected layer to semantic representation. As reference, both unit- Hiti-iFFGAN and Tokemizer use layer of HuBERTBAS. BASE to extract semantic representation, and we found layer 22 works best for HuBERT LARGE- While unit-Hi,GGAN, SpeechTokenizeer and our model are trained on LibriSpeech, Encodec and DAC are trained on a mixture of speech, sound, and music.
is neural audio codec model based on Encodec. In addition
we used the official model released by the authors for evaluation. Both
7

---

Published as a conference paper at ICLR 2025
Table 3: Recognition WER of different model sizes. Results are dev-other with LM after 200k pre-training steps.
Table In-context TTS ASR-WER and speaker simi- larity of different model sizes after 200k pre- pre-training steps. The default encoder/decoder depth is 24/12.
decoder depth
encoder depth
encoder depth
decoder depth
12
24
12
24
12
7.6
8.1
8.2
3 0.45
0.60
2.5 0.66
24
5.8
5.5
5.2
24 2.5/ 2.61
2.5 0.63
2.5 0.66
Fine-tuning UniWav UniWav can be naturally applied to the task by introducing a discrete bottleneck between the encoder and decoder. We follow the prior works to first identify the semantic layer to run kmeans. More information can be passed to the decoder by quantizing the remaining layers a cost of higher bitrate for tokenization. More precisely, we replace the encoder representations in (9) with
2 = W argmin NEQ.
20 all
+ argmin
Σ
اهز W W
(11)
geQuoitall
semaniic quantization
residual quantization (optional)
where = 10 is the index of layer used for extracting semantic representation, Q; and Qresidual are the sets of centroids obtained through k-means clustering. We fine-tune the decoder with the quantized encoder representation on A100 GPUs for 150k steps with a Ie-4 learning rate. For decoding, we use step size of 0.25 and CFG 1.0, resulting in 8 NFEs.
Data & Metrics All models considered are trained on LibriSpeech and tested on the dev and test subsets including both clean and other splits. For audio intelligibiliit, we report ASR-WER (see $3.3) on the resynthesized speech. For the overall audio quality, we use UTMOS (Saeki et al., 2022) following the discrete vocoder training track of the Interspeech 2024 Challenge on Speech Processing Using Discrete Units (Chang al., 2024). UTMOS is a reference-free, machine perception score on audio naturalness on five-point scale that aligns well with human preference. We also measure and speaker similarity (see $3.3) between the original audio and the resynthesized version. For audio reconstruction quality, we measure ViSQOL (Chinen et al., 2020) following Encodec and DAC.
Results Results are presented in Table 2. At 500 bps, UniWav leads all metrics by a margin, outperforming all other methods. The lowest ASR-WER reflected that UniWav carried the most semantic information under the same bitrate. Outstanding UTMOS suggested UniWar is able to generate realistic audio with limited information provided, which emerges from the generative pre- trained decoder. We note that speaker information is still dropped by WniWav, much like all other low-bitrate methods, due to the choice of semantic representation. At 1 bitrate, UniWav maintained a clear advantage over SpeechTookenizee and higher UTMOS (3.72 v.s. 2.08). In fact, 1kbps UniWav is comparable 4kbps SpeechTokenize in terms of ASR-WER and UTMOS. UniWay only falls short on speaker similarity, which requires input speaker information that cannot be filled in by the decoder. Nevertheless, the similarity score still significantly higher (0.499 v.s. 0.340) than SpeechTokenize
with a significantly lower word error rate (5.6% v.s 9.1%)
at the same 1kbps.
Results on speech tokenization and resynthesis provided strong evidence for the usefulness of jointly learning speech representation and generation. UniWav is able to perform low-bitrate speech encoding while decoding audio of substantially better quality than the state-of-the-a Speech Tokenizer.
3.5 ANALYSIS
Scaling Encoder and Decoder Since the encoder and the decoder are trained jointly in UniWav, is interesting to investigate how the scale of the encoder and the decoder changes the overall behavior. In Table and Table 4, we report speech recognition and synthesis results when varying the size of the encoder and the decoder. Zero depth indicates the corresponding training objective was ignored. For speech recognition with a shallow encoder, we found introducing the decoder degrades WER regardless of the size. Interestingly, ie., the encoder benefits from the decoder and the generative objective. For speech generation, we found that encoder depth was critical when working with shallow decoder, and the gain became less
opposite trend is observed when the encoder size is doubled,
8

---

Published as a conference paper at ICLR 2025
DARER
DiWli
1770
1.00
03
Figure 2: Mutual information between quantized representation and phone/speaker (left/right) different layers. Results are computed on the dev set LibriSpeech. For quantization, 1024 clusters are used for k-means.
significant with deeper decoder. More importantly, the size of the decoder became less important as the encoder depth increased.
To summarize, we highlight two important observations regarding unified pre-training:
Discriminative representation learning benefits from generative pre-training only when the encoder has enough capacity.
Representation encoder plays a more important role in unified pre-training, offering a better trade-off between compute and performance.
Mutual Information Between Representation and Labels For speech representation learning, layer-wise analysis has been used to examine what is encoded by the model and its dynamics at different layers (Pasad et al., 2023). In this study, introduce a discrete bottleneck to each layer of UniWav's encoder, using the method described in $3.4 and (11( For each layer, we approximate the mutual information (MI) between quantized representations and the underlying speech label - either phone or speaker - by computing the empirical distribution P(unit), P(label), and Punit, label), and visualize them in Figure 2. The results provide explanations for UniWav's behavior in different applications: (1) In the latter layers, WniWav achieved lower MI in relation to phones compared to HuBERT, potentially making fine-tuning for speech recognition more difficult; (2) For most of the layers, UniWav retained higher MI in relation to speaker label, which explained strong performance on audio quality and speaker similarity metrics in speech generation and resynthesis.
Through the analysis, we conclude some observations and future directions: (1) Due to the existence of the generative decoder, UniWav captured information beyond language from speech. Such property might overturn the belief that self-supervised speech representations are sub-optimal (Wang et al., 2023) for generation tasks. Our speech tokenization experiment is a good evidence. (2) Compared to the pure self-supervised learning algorithm, UniWav learns a more entangled representation. Future works in unified pre-training can potentially benefit from representation disentanglement methods (Qian et al., 2022; Chang et al., 2023) to further improve controllability and downstream performance.
RELATED WORKS
The success of self-supervised representation learning and generative models has emerged promising direction for enhancing model generalization, leading to general-purpose speech and audio- intelligent systems. Our work contributes this growing body of research by rethinking the potential of jointly modeling both speech understanding and generation, aiming for unified framework that can seamlessly integrate these tasks, We would like acknowledge several previous efforts this space, particularly the intersection between speech representation learning and generative modeling.
Speech Representation Learning In recent years, the research community has witnessed signif- icant advancements in self-supervised learning techniques across speech and other domains, with notable developments ranging from vector-quantized methods (Van Den Oord et al., 2017: Dhariwal al., 2020), contrastive pre-training methods (Schneider et al., 2019 Baevski al., 2020b), to more recent random- random-projection projection quantizer methods (Chiu et al., 2022; Yang et al., 2024) or self-labeling
methods (Hsu et al., 2021: Chen et al., 2022a; Baevski et al.. 2022; Liu al.. 2024). These self- supervised representation learning methods have a wide impact in speech processing, but their focus remained primarily on discriminative tasks. Applying these techniques to generative tasks is more

---

Published as a conference paper at ICLR 2025
challenging (Tsai et al., 2022), often relying on a pipelined generative model (Polyak al., 2021; Zhang et al., 2023b). While UniWav's 's encoder follows this line of work, our experiments and analysis suggested that it is significantly different from prior work, due to the generative components involved in pre-training. UniWav can be viewed as a more general representation framework aiming to support more diverse collection of applications.
Besides speech-only models, it is also worth noting that iww--niooeenneeeeeeeeee pre-training could improve speech and text representation on cross-modality matching (Chung al., 2019; Chen et al., 2021; Ao et al., 2021) and feature disentanglement (Chen et al., 2022b). Unified pre-training can potentially be further improved by cross- modsliy alignment in future studies.
Generative Models in Speech Similar to representation learning, generative models have also made significant strides in the field of speech. Methods like adversarial training (Kong et al., 2020; Lee et al., 2022), normalizing flows (Prenger et al., 2019; Valle et al., 2020), auto-regressive models (Wang et al., 2023; 2024), and diffusion-style framework (Koizumi et al., 2022; Le et al., 2023; Kim et al., 2024) have all pushed the edge of speech generation. In this line of work, generative pre- pre-traiming with Flow Matching (SpeechFlow; Liu et al. 2023) is the most relevant work. Our UniWav decoder is built upon SpeechFlow's design and follows similar fine-tuning paradigm. Our decoder can be viewed as representatiooncconitionnn optimized representation encoder and generative decoder substantially different from training generative models on fixed embeddings, and our contribution in this direction is novel.
variant of SpeechFlow. Nevertheless, we note that the jointly
Speech Foundation Models and Large Langauge Models (LLM). In alignment with the recent trend in audio-enabled LLMs (Gong et al., 2023; Tang et al., 2023; Zhang et al., 2023a; Kong al.. 2024; Huang et al., 2024), representation learning and generative models have gained further popularity. For such models, we often desire them to possess the capability of both perceiving (discriminative tasks) and producing (generative tasks) audio content, e.g., GPT-4o (OpenAI, 2024), and Moshi (Défossez et al., 2024). Furthermore, modern LLMs are typically trained in an autore- gressive manner, and thus require the model's input and output to be in a unified representation space. However, previous discriminative representations, such HuBERT (Hsu et al., 2021), are not easily adaptable to generative tasks. Conversely, representations with more detailed acoustic features, such as EnCodec codes (Défossez al., 2022), exhibit limitations in discriminative per- formance and degraded performance in phonetic classification tasks (Yang et al., 2023). As a result, prior models either do not support sound generation (Gong al., 2023; Tang al., 2023; Kong et al., 2024), rely on additional components like a vocoder (Zhang et al., 2023a) or text-to-speech model (Huang et al., 2024), or has higher word error rate than models trained with discriminative representations (Délossez et al., 2024). This highlights the urgent need for a speech representation that excels in both discriminative and generative tasks. UniWav addresses this gap by introducing a unified representation, which is essential for building future foundation models that integrate both understanding and generation capabilities.
5 CONCLUSION
In this work, we introduced UniWav, a unified framework for speech representation learning and generation. By integrating both discriminative and generative capabilities, UniWav demonstrates that single model can effectively bridge the gap between tasks with competing requirements. Through experiments across speech recognition, synthesis, and tokenization tasks, we show that UniWay achieves competitive performance compared to task-specific models, reducing the need for distinet pre-trained models and simplifying the speech processing pipeline. The compactness of the learned representation and the ability to generate high-quality audio underscore the potential of unified foundational models for future Speech and Audio intelligent systems.
Limitations The most significant limitation we noticed is the performance drop on recognition tasks compared to self-supervised learning models at similar scales. To the extent of our experimentation thus far, the cost of enabling generation appears to make this result inevitable. In computer vision, researchers have made similar observations on the small gap between a unified model and single- function state-of-thh -hrtt-r-rtttttt representation model (Li et al., 2023). Whether the gap can be closed or not remains an open research problem. We also note that this work focused exclusively on English Speech, from audiobooks in particular. Applying Wni to more generic speech, more languages, and different audio domains (such sound and music) are left important future works.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unilied-modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205, 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for sulf-supervieed learning of speech representations. Advances in neural information processing systems, 2020a.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 13:12449-12461) 2020b.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pp. 1298-1312 PMLR, 2022.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International Conference on Machine Learning. pp. 1416-1429. PMLR, 2023.
He Bai, Tatiana Likhomanenko, Ruixiang Zhang, Zijin Gu, Zakaria Aldeneh, and Navdeep Jaitly. dmel: Speech tokenization made simple. arXiv preprint 2tti::2007118335 2024.
Heng-Jui Chang, Alexander H Liu, and James Glass. Self-supervised fine-tuning for improved content representations by speaker- invariant clustering. arXiv preprint arXiv:2305.11011 2023.
Xuankai Chang, Jiatong Shi, Jinchuan Tian, Yuning Wu, Yuxun Tang, Yihan Wu, Shinji Watanabe, Yossi Adi, Xie Chen, and Qin Jin. The interspeech 2024 challenge on speech processing using discrete units. arXiv preprint 2rXiv:2406.07725, 2024.
Ricky T. Q. Chen. torchdi f fee.
torchdiffeq, 2018.
URL
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):
1505- 1518, 2022a.
Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401. 2024.
Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Gary Wang, and Pedro Moreno. Injecting text in self-supervised speech pretraining. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 251-258. IEEE, 2021.
Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen. Maestro: Matched speech text representations through modality matching. arXiv preprint 2tt::2200.034009 2022b.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference on quality of multimedia experience (QoMEX), pp. 1-6. IEEE, 2020.
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pp. 3914-3924. PMLR, 2022.
Yu-An Chung. Yuxuan Wang, Wei-Ning Hsu, Yu Zhang, and RJ Skerry- -yann Semi-supervised training for improving data efficiency in end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6940-699 6944. IEEE, 2019.
11

---

Published as a conference paper at ICLR 2025
Yu-An Chung, Yu Zhang, Wei Han, Chung- Channg Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244-250. IEEE, 2021.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. ArXiv, abs/2210. 13338, 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue, 2024.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 2021.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. arXiv preprint wxXi::2402.04822 2024.
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint 27X11:2305.10790 2023.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Yicheng Gu, Xueyao Zhang, Liumeng Xue, and Zhizheng Wu. Multi-scale sub-band constant-q transform discriminator for high-fidelity vocoder. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 10616-10620. IEEE, 2024.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Rongjie Huang, Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 23802 23804, 2024.
Zhichao Huang, Chutong Meng, and Tom Ko. Repcodec: A speech representation codec for speech tokenization. arXiv preprint arXiv:2309.00169, 2023.
Jacob Kahn, Morgane Rivière Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Fierre-Emmanue Mazar'e, e, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdel rahman Mohamed, and Emmanuel Dupoux. Light-Light: A benchmark for asr with limited or supervision. International Confer- ence on Acoustics, Speech and Signal Processing, 2019.
Sungwon Kim, Kevin Shih, Joao Felipe Santos, Evelina Bakhturina, Mikyas Desta, Rafael Valle, Sungroh Yoon, Bryan Catanzaro, et al. P-flow: a fast and data-efficient zero-shot tts through speech prompting. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980,, 2014.
Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. Specgrad: Diffusion probabilisticc model based neural vocoder with adaptive noise spectral shaping. arXiv preprint arXiv:2203.16749 2022.
Jungil Kong, Jaehyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33: 13:1722217703, 2020.
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafacl Valle, and Bryan Catanzaro, Audio flamingo: A novel audio language model with few-shot learning and dialogue abilitics. arXiv preprint arXiv:2402.0183/ 2024.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36, 2024.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306. 250.:22706/6687,, 2023.
Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.04460 2022.
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2142-2152, 2023.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.
Alexander Liu, Matt Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu. Generative pre-training for speech with flow matching, arXiv preprint arXiv:2310.16332 2023.
Alexander H Liu, Heng-Jui Chang, Michael Auli, Wei-Ning Hsu, and Jim Glass. Dinosr: Self- distillation and online clustering for self sulf-super speech representation learning. Advances in Neural Information Processing Systems, 36, 2024.
Michael McAuliffe, Michaela Socolof, Sarah Mihue, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, 2017.
Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Szěkely, and Gustav Eje Henter. Matcha-tts: A fast tts architecture with conditional flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11341-11345 IEEE, 2024.
Tu Anh Nguyen, Wei-Ning Hsu, Antony d' d'Ainoo, Bowen Shi, Itai Gat, Maryam Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arx0v:2308.007725, 2023.
Tal
OpenAl. Gpt-4o system card. https: http://dd.oopaaa.ccoooeeeeeeeeee http://///cch...p.aa.a....//a//p 2024.
pdf,
Vassil Panayotov. Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ast corpus based public domain audio books. International Conference on Acoustics, Speech and Signal Processing, 2015.
Ankita Pasad, Bowen Shi. and Karen Liveseu. Comparative layer-wise analysis of self-supervised speech models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEF, 2023.
13

---

Published as a conference paper at ICLR 2025
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deirahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXhv:2104.00355, 2021.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621. IEEE, 2019.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. ArXiv, abs/2108 108. 12409, 2021.
Kaizhi Qian, Yang Zhang, Heting Gao. Junrui Ni, Cheng-I I Lai, David Cox, Mark Hasegawa-Johnson, and Shiyu Chang. Contentvec: An improved self superssd speech representation by disentangling speakers. In International Conference on Machine Learning, pp. 18003-18017 PMLR, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. CrowdMOS: An approach for crowdsourcing mean opinion score studies. In International Conference Acoustics, Speech and Signal Processing, 2011.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152, 2022.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint 1904 2019.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint MiXiv:2310.13289 2023.
Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy T Liu, Cheng-I Jeff Lai, Jiatong Shi, et al. Superb-sg: Enhanced speech processing universal performance benchmark for semantic and generative capabilities. arXiv preprint 2XKv:2203306844 2022.
Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis. arXiv preprint arXiv:2005.05957. 2020.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. ArXiv, (bs//770600362,, 2017.
Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:23 /582/, 2023.
Chengyi Wang, Sanyuan Chen, Yu Wu, Zi-Hua Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. ArXiv, abs/2301.022 1.120110111,,1 2023.
Xiaofei Wang. Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang. Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as a versatile speech transformer. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
14

---

Published as a conference paper at ICLR 2025
Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders unified self-supervised learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15802-158122 2023.
Dongehao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00744, 2023.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.011 100.0105,, 2021.
Yufeng Yang. Desh Raj, Ju Lin, Niko Moritz, Junteng Jia, Gil Keren, Egor Lakomkin, Yiteng Huang, Jacob Donley, Jay Mahadeokar, et al. M-best-rq: A multi-channel speech foundation model for smart glasses, arXiv preprint arXiv:2409. / 1194, 2024.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305. 1000, 2023a.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. 16692, 2023b.
Xinfa Zhu, Yuanjun Lv, Yi Lei, Tao Li, Wendi He, Hongbin Zhou, Heng Lu, and Lei Xie. Vec- tok speech: Speech vectorization and tokenization for neural speech generation. arXiv preprini arXiv:2310.07246, 2023.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL EXPERIMENTS
Table 5: Comparison of the choice of surface feature. Models are pre-trained for 200k steps. Sim-r refers to the similarity when comparing generated speech with the resynthesized prompt, whereas Sim-o uses original prompt audio the reference.
Recognition
Speech Synthesis
Feature
WER
ASR-W WER
Sim-r
Sim-o
Mel Spectrogram
Decoding ground truth UniWav
2.6
0.638
0,666
5.6
2.5
0.629
0.620
EnCodec latent
Decoding ground truth UniWav
2.5
0.627
0.666
5.5
2.5
0.631
0.614
Choice of Surface Feature Different speech foundation models have been relying on different input representations of audio, ranging from time-domain signal (Baevski al., 2020a;b; Hsu et al., 2021), mel spectrogram (Chung et al., 2021; Le et al., 2023; Liu et al., 2023), to neural auto-encoder features (Wang al., 2023; Vyas al., 2023; Evans et al., 2024). In this work, we choose Encodec (Défossez et al., 2022) latent since it can be easily decoded, making it ideal for audio generation. Besides Encodec latent, we also train UniWav on mel spectrogram ablate the choice of audio feature in Table 5. 80-band mel spectrogram is extracted at a 50Hz frame rate (to match Encodec latent) with the hop size of 320 and window size of 1280. For vocoding, we adopted the BigVGAN-v2 vocoder (Lee et al., 2022) by modifying its configuration to accept 80-band mel spectrogram at 50Hz frame rate and generate a 16kHz waveform. Following the EnCodec architecture,
four residual blocks with upsampling ratios of [8. 5, 4, 2] resulting in a 122M parameter model. We adjust the bi ns per cot.ocaae of the MS-SB-CQT Discriminator (Gu et al., 2024) to [6, 10, 14] to ensure all values remain below the Nyquist frequency for 16kHz audio. We use a learning rate of 5 x 10-5 and batch size of 32, using 2-second chunks per sample.
On both the recognition and synthesis tasks, we also observe similar results regardless of the model input. By decoding the mel spectrogram and Encodec latent extracted from ground truth, we also found that both systems shared similar baseline performance on the quality of generation. We conclude that the proposed pre-training framework is general and robust to the choice of surface feature.
16

---

Published as a conference paper at ICLR 2025
Table Speech representation evaluation results on SUPERB (Yang et al., 2021). works are taken from the official leaderboard.
Results of prior
Pre-training data (hr)
Phoneme Recognition Phone Error Rate
Speaker Identification Accuracy
HuBERT (Hsu et al., 2021)
60k
3,5
90.3
WavLM (Chen et al., 2022a)
94k
3.1
95.5
Uni Way
60k
3.7
93.8
Speech Representation Evaluation To study the representation extracted by UniWav encoder, we conduct experiments on phone recognition and speaker identification using the existing benchmarking toolkit SUPERB (Yang et al., 2021), results are provided in Table 6. We use the default configuration provided by the SUPERB toolkit 2 sweeping only the learning rate for both tasks. In these tasks, representations are kept frozen train downstream models, simulating the use case where a pre- trainined representation model is treated as feature extractor. Results on SUPERB show that representation extracted from UniWav performed slight worse than that of from HuBERT on phoneme recognition, but better speaker identification. WavLM, the leading model SUPERB, outperforms both models with more training data and intensive hyper- eyper-parameter search on downstream tasks (Chen al., 2022a). In short, the conclusion here is consistent with our experiments fine-tuning for speech recognition ($3.2) and layer-wise analysis of speech representation ($3.5).
Table 7: Subjective test on English zero-shot speaker adaptation TTS on filtered LS test-clean. Averaged rating along with 95% confidence interval are reported for Mean Opinion Score (MOS).
subjective MOS
Method
Ground truth
3.82++0.11
SpeechHlow (Liu et al., 2023)
3.59++0.1
UniWav
3.55:10.11
Subjective Evaluation on Text-to-speech Following the evaluation setup of Liu et al. (2023), conducted subjective test on text-to-speech results using mean opinion score (MOS) (Ribeiro et al., 2011). We randomly selected 50 sentences from the test-clean subset of LibriSpeech for the test and collected 10 ratings for each sentence. Each annotator is asked to rate 30 samples, evenly distributed across ground truth, SpecchFlow (Liu al., 2023), and UniWav. The ratings are based on the clarity of speech, sound quality, and naturalness, ranging from I to 5. Results are collected through Amazon Mechanical Turk (AMT) and the listed in Table 7. We found UniWav recieving similar rating that of SpeechFlow, while both methods fall short of the ground truth. The MOS score supported our conclusion from objective tests (see Table 1) that UniWav matches the performance of generative-only state-of-the-arr pre-training method on text-to-speech.
17

---

Published as a conference paper at ICLR 2025
A.2
VISUALIZATED SAMPLES OF SPEECH TOKENIZATION AND RESYNTHESIS
Speech Tokcenizer (500hppe)
Speech Toleenizer (1kbpe)
8000
2000
UniWav (500bps)
UniWav (1kbps)
0000
2000
HuBERT Unit HIII-HI (300bps)
Ground Truth
8000
0000
4000
2000
0.5
1.5
2
2.5
3
3.5
4.5
0.5
1.5
2.5
3
3.5
4
4.5
Time
Time
Figure 3: Utterance 7176-88083-0014 from test-clean
18


Reviews:
[{"review_id": "0VirFpC4bT", "reviewer": "Reviewer_g67C", "summary": "UniWav is an encoder-decoder framework designed to unify pre-training representation learning and generative tasks. With the appropriate design choices for pre-training, UniWav can jointly learn a representation encoder and generative audio decoder that can be applied to both types of tasks.", "questions": "Can the method proposed in this paper be applied to general audio such as sound and music?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces the first unified pre-training framework (UniWav) for speech representation learning and generation.\nUniWav can compete with different foundation models with low bitrate speech tokenization and high-quality resynthesis.", "weaknesses": "1. The results of the method in this paper do not show an advantage over existing methods in speech recognition and speech generation. \n2. In the speech tokenization section, there is a lack of experiments related to the modeling performance of the speech language model (LLM-TTS) as mentioned in SpeechTokenizer[1]. Such experiments could effectively evaluate the potential of the tokenizer proposed in this paper when applied to autoregressive speech synthesis.\n[1]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ryZEhlsIW2", "reviewer": "Reviewer_8mpX", "summary": "The paper proposes UniWav, a unified pre-training framework for speech representation learning and generation. Traditionally, pre-training models for speech have been specialized either for discriminative tasks, like speech recognition, or for generative tasks, such as text-to-speech. UniWav aims to bridge this gap by integrating both functions into a single encoder-decoder model. The encoder is responsible for learning robust speech representations, while the decoder generates speech through Flow Matching.", "questions": "1. How does the experimental section provide evidence for the claim that simultaneous pre-training on speech representation learning and generation enhances performance compared to pre-training on only one task, and what justifies the assertion of reduced overhead and cost?\n\n2. Could you include comparisons with models like Natural Speech 3, Funcodec, Encodec, DAC, and DinoSR to improve the evaluation of the proposed methods in speech generation and tokenization tasks?\n\n3. Could you incorporate additional subjective and objective metrics, such as Mean Opinion Score (MOS) and Mel/STFT distance, provide a more comprehensive assessment of model performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper conducts experiments across multiple tasks like speech recognition, text-to-speech, and tokenization. It includes analyses, such as ablation studies and mutual information metrics.\n2. The paper is well-organized and clearly presents technical details, such as the encoder-decoder structure and the Flow Matching method, making it easy to follow. Visual aids and concise explanations further contribute to the clarity of the complex concepts.", "weaknesses": "1. Firstly, I have some doubts regarding the motivation and novelty validation in this paper. The introduction states that ideally, speech representation learning and generation can mutually enhance each other and that this approach can reduce the overhead and cost of pre-training. However, I did not find evidence for these conclusions in the experimental section. Specifically, it seems that the paper does not address whether pre-training on just one task (either speech representation learning or generation) would yield better performance on downstream tasks compared to simultaneous pre-training on both tasks. Additionally, the model's performance on downstream tasks does not appear to be very strong; for instance, its performance on the speech recognition task is worse than that of other baselines. So the experiments can not show that the overhead and cost can be reduced. Because there is neither proof that pre-training on both tasks together is better, nor evidence that there are significantly better results on a single downstream task. If these advantages cannot be clearly demonstrated, then why combine the two tasks? Merely putting speech representation learning and speech generation together without thoroughly explaining the rationale and benefits of this approach significantly limits the contributions of the paper. Overall, the results seem to contradict the stated motivation and novelty, or these points have not been well validated. \n\n2. Secondly, the selection of baselines in the paper is quite limited, especially for the speech generation and tokenization tasks. Speech generation could be compared with Natural Speech 3 [1], while the speech tokenization task could benefit from comparisons with the latest models like Funcodec [2], Encodec [3], and DAC [4]. Furthermore, the paper should include comparisons with the experimental results of DinoSR for both the speech recognition and tokenization tasks, as the encoder component of the paper is primarily based on the DinoSR model.\n\n3. Lastly, the metrics used for evaluating downstream tasks are insufficient. For example, in the speech generation task, subjective evaluations such as Mean Opinion Score (MOS) should be included, as this is a critical metric. For speech tokenization, additional metrics related to speech reconstruction, such as Mel/STFT distance, could be incorporated.\n\n[1] Ju, Zeqian, et al. \"Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models.\" arXiv preprint arXiv:2403.03100 (2024).\n\n[2] Du, Zhihao, et al. \"Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.\n\n[3] Défossez, Alexandre, et al. \"High fidelity neural audio compression.\" arXiv preprint arXiv:2210.13438 (2022).\n\n[4] Kumar, Rithesh, et al. \"High-fidelity audio compression with improved rvqgan.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PCxnjE9ZrB", "reviewer": "Reviewer_me7k", "summary": "This work proposes a self-supervised speech representation learning objective which combines 1) masked prediction with online clustering from an EMA teacher model (DinoSR) to train a transformer-encoder network and 2) reconstructing noise-inserted input data based on the encoder representations with Flow Matching to train a transformer-decoder network. The aim of this method is to unify the creation of foundation models for discriminative tasks (such as ASR) and generative tasks (such as TTS). \n\nThe method is evaluated by pre-training on 60k hours of LibriLight, and fine-tuning for speech recognition, speech synthesis, and speech tokenization and resynthesis, on Librispeech. \n\nFor speech recognition, they show limited degradation of performance compared to SSL methods like HuBERT, WavLM, and data2vec.   \nFor speech synthesis, they show performance matching contemporary models like VoiceBox and SpeechFlow.  \nFor speech tokenization and reconstruction, performance exceeds SpeechTokenizer and HuBERT+HifiGAN.  \n\nThe method is also ablated on the encoder and decoder depth, which shows that a 12-layer encoder does not benefit from adding the decoder objective, while a 24-layer encoder does benefit. Moreover, it is shown through a mutual-information analysis that their encoder has different characteristics on how speaker and speech information is processed compared to HuBERT.", "questions": "1. Based on Figure 2, have the authors considered only conditioning the decoder on the first, e.g., 12 layers of the encoder? \n2. Can the authors clarify how speaker similarity is computed? Is it the average cosine similarity between the WavLM speaker embedding of pairs of ground-truth and synthesized utterances? If so, I think it would make sense to share the standard deviation as well. \n3. Can the authors comment on their batch size and how they are sampled (e.g., like wav2vec 2.0)? This would help with reproducibility, and figuring out how much data is seen throughout pre-training, and where UniWav lies in Figure 2 of the DinoSR paper (trade-off between performance and data efficiency)\n4. Line 233, \"sinusoidal pos. enc. is appended to the input.\" Is this appended to the feature dimension or the time dimension? Isn't it normally the case that positional embeddings are summed with the input features? Can the authors comments on this design decision?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\n\nThe method proposes a new speech SSL method, combining existing methods DinoSR (encoder-only) and SpeechFlow (encoder-decoder) . This method has strong performance on generative and discriminative tasks compared to foundation models which are generative-only or discriminative-only. \n\n### Quality\n\nThe method is evaluated on multiple speech technology tasks, and a small ablation study is performed for further insights. \n\n### Clarity\n\nThe paper is well-written, easy to follow, and appropriately places itself into existing literature.\n\n### Significance\n\nThis work will definitely spark future work in the speech community.", "weaknesses": "### domain\nThis works only evaluates on Librispeech, i.e., the audio book domain, which has very homogenetic speaker conditions. The observation that ASR performance closely matches HuBERT, WavLM, etc, does not take into account the robustness these models have to other domains. The authors could discuss this limitation or add experiments with the SUPERB benchmark.\n\n### modelling \nAs the proposed method is an extension of DinoSR, it would be nice to see UniWav w/o decoder, as in Table 3 for 200k steps, in Table 1 with 600k steps of pre-training. Moreover, Speechflow also uses an encoder-decoder model. This encoder could in theory be fine-tuned for ASR. The original work did not do this. Can the authors comment on expected results when the SpeechFlow encoder would be fine-tuned for ASR, and how this would compare to UniWav? Could this be discussed in a future work section?\n\n### speaker recognition\n\nOne of the central claims is that UniWav bridges the inherent orthogonality between speech recognition, which normalizes over speaker and environment information, and speech synthesis, which requires speaker and environment information. This is touched upon slightly by the mutual information analysis, where UniWav is seen to lose speaker information through the encoder layer. It would seem to me that UniWav will not be better for the speaker recognition task than, e.g., WavLM, based on the speculation that most environment and speaker information is stored in the decoder. I think evaluating UniWav on the (SUPERB) speaker recognition and speaker verification task would strengthen the claim significantly. I think for now, this limitation should be made more explicit in the limitation section, or if possible, the authors could perform additional experiments on SUPERB.\n\n### Minor comments\n\n1. line 215: Encodec has not been introduced yet, so cite it here. I find this paragraph confusing due to missing context on how the network uses Encodec as input features (line 239) instead of Mel spectrograms as suggested in line 108. \n2. line 236: \\proposed~is\n3. line 242: in~\\ref \n4. line 358: kmeans instead of k-means\n5. line 370: we follow...run k-means, use e.g., by first identifying .. on which to run", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bJ52WcwpLm", "reviewer": "Reviewer_hgj6", "summary": "This paper introduces UniWav, a new pre-training framework focused on both speech representation learning for discriminative task and for generation task. Using an encoder-decoder architecture, UniWav learns speech representations and generates speech together, letting it perform well in both discriminative tasks and generative tasks. UniWav aims to unify the approach, simplifying the speech processing pipeline and reducing need for many specialized pre-trained models. Experiments in speech recognition, text-to-speech synthesis, and speech tokenization show UniWav's effectiveness, achieving results competitive with other top models for specific tasks.", "questions": "On line 108, the authors claim that the model uses mel spectrograms as input, but on lines 239–240, they mention using Encodec features instead.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well motivated to jointly pre-train for discriminative and generative tasks. The paper shows SOTA, if not comparable, results on the generative tasks and lag a little on the discriminative task.\nThey show very good results when tokenizing speech using UniWav, compared to other works in the literature.  \nThe paper is well written.", "weaknesses": "The authors propose UniWav, a pre-training method for jointly learning representations for both discriminative and generative tasks in speech processing. This method uses an encoder-decoder architecture to learn effective representations as part of the training process.\n\nFor the discriminative task of speech recognition, the authors fine-tune the encoder on 100-hour and 960-hour splits, showing competitive results compared to other methods. However, they do not provide results for low-resource scenarios (e.g., 10 hours or 1 hour of data) or results using a frozen encoder (SUPERB benchmark). These two setups are essential to support their claim that the learned encoder representations are effective for speech recognition tasks. Because with large finetuning data, effect of pre-training is less. \n\nFigure 2 shows that the encoder’s full capacity is not optimized for learning features specific to speech recognition. UniWav achieves the highest mutual information (MI) on layer 10 out of 24, which then gradually decreases. This pattern suggests that the model divides its capacity between the discriminative and generative tasks, similar to observations in WavLM and prior works. This could explain UniWav’s strong performance in speech generation tasks, as a significant portion of the model’s capacity is allocated to optimize for generation. This trend is consistent across the paper’s reported results.\n\nRegarding speech tokenization, it’s unclear if the comparison is entirely fair. For speech tokenization, the input to the SpeechTokenizer is raw audio, while for UniWav, the input is Encodec encoder features, which are further transformed by UniWav’s 10-layer encoder. It is as if UniWav is heavily overparameterized and most of the parameters are used for generation task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8TgCanwVci", "reviewer": "Reviewer_v3oX", "summary": "This paper proposes a framework named UniWav. It is an encoder-decoder framework aimed at unifying pre-training representation learning and generative tasks. The authors claimed that this is the first such framework and achieves comparable performance to different existing foundation models, each trained on a specific task, on speech recognition, text-to-speech, and speech tokenization tasks.", "questions": "why \"For speech recognition with a shallow encoder, we found introducing the decoder degrades WER regardless of the size. Interestingly, an opposite trend is observed when the encoder size is doubled\"? This is something requires more analysis and explanation.\n\nIn section 3.1, \"We extract the prequantized latent feature from a 16kHz EnCodec (D´efossez et al., 2022) encoder, pre-trained on the same dataset, to serve as the input for our model.\" This is contradictory to the claim in early sections that the input to your model is mel spectrum?  In addition, this makes the comparison with other approaches unfair since you used another model as the pre-processing module (i.e., total model size is actually increased significantly).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main contribution of this work is the combination of DinoSR and a flow matching model to train a unified representation model. While not considered significant, this special setup is somewhat novel.\n\nThe paper contains sufficient experimental results to support their claims although some setups are questionable (in detail below). \n\nThe paper clearly indicated the limitations of the work.", "weaknesses": "The claim their work is \"the first unified pre-training framework for speech representation learning and generation\" is arguable. For example, the paper \"Speechgptgen: Scaling chain-of-information speech generation.\" introduced an approach to train a discrete representation that works for both understanding and generation. The paper \"Moshi: a speech-text foundation model for real-time dialogue\" further improved the approach. \n\nThe presentation needs to be improved. \n1. the DinoSR part in Figure 1 is confusing since the softmax layer is not clearly shown since the encoder, according to the text description, does not include that layer. In contrast, the figure in the original DinoSR paper is very clear. \n2. The majority of Section 2.1 is describing DinoSR. However, notation is not clearly explained sometimes. for example, what is \"k\" in s^k_v right below eq. 3? it's unclear why Euclidean distance (instead of COS distance) is used given that Cos distance is usually more preferred in higher-dim spaces. \n3. notation \"z\" is overloaded in eq 9.\n4. the footnote under Table 1 is not clear. alignment is only used for generation tasks?\n5. eq 11 is wrong? argmin returns an index instead of a representation.\n\nWhen researchers train a representation model they usually keep the encoder fixed when using them in downstream tasks such as ASR. however, in this work the encoder is finetuned. This causes the claims weaker. Similarlly if flow matching is used in the generation part the quality of the generated speech will of course become better. However, this gain comes from the flow matching model instead of the way the representation is learned. Some clarification here is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)