ProsodyBERT: Self-Supervised Prosody Representation for Style-Controllable TTS
Authors: Yushi Hu
Chunlei Zhang
Jiatong Shi
Jiachen Lian
Mari Ostendorf
Dong Yu
Venue: Submitted to ICLR 2023 (2023)
Keywords: prosody
self-supervised learning
text-to-speech
speech processing
emotion recognition
speech synthesis

Abstract: We propose ProsodyBERT, a self-supervised approach to learning prosody representations from raw audio. Different from most previous works, which use information bottlenecks to disentangle prosody features from speech content and speaker information, we perform an offline clustering of speaker-normalized prosody-related features (energy, pitch, their dynamics, etc.) and use the cluster labels as targets for HuBERT-like masked unit prediction. A span boundary loss is also introduced to capture long-range prosodic information. We demonstrate the effectiveness of ProsodyBERT on a multi-speaker style-controllable text-to-speech (TTS) system. Experiments show that the TTS system trained with ProsodyBERT features can generate natural and expressive speech samples, surpassing the model supervised by energy and pitch on subjective human evaluation. Also, the style and expressiveness of synthesized audio can be controlled by manipulating the prosody features. In addition, We achieve new state-of-the-art results on the IEMOCAP emotion recognition task by combining our prosody features with HuBERT features, showing that ProsodyBERT is complementary to popular pretrained speech self-supervised models.

Under review as a conference paper at ICLR 2023
PROSODYBERT: SELF-SUPERVISER PROSODY REPRE- SENTATION FOR STYLE-CONTROLLABLE TTS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose ProsodyBERT. self-supervised approach to learning prosody represen- tations from raw audio. Different from most previous work, which uses information bottlenecks to disentangle prosody features from lexical content and speaker infor- mation, we perform an offline clustering of speaker-normallee features (energy, pitch, etc.) and use the cluster labels as targets for HuBERT-like masked unit prediction. A span boundary loss is also used to capture long-range range prosodic information. We demonstrate the effectiveness of ProsodyBERT multi-speaker style-controllable text-to-speech (TTS) system, showing that the TTS system trained with ProSodyBERT features generate natural and expressive speech samples, surpassing Fastspeech 2 (which directly models pitch and energy) in subjective human evaluation. In addition, we achieve new tate-of--hhe-ar results on the JEMOCAP emotion recognition task by combining our prosody features with HuBERT features, showing that ProsodyBERT is complementary to popular pretrained speech self-supervised models.
prosody-relane
INTRODUCTION
Human speech contains information beyond the associated word sequence. For example, the in- tonation, stress, rhythm, and tempo of speech carry important cues associated with the speaking style, emotion, and intent. These factors generally referred to as prosody. Prosodic modeling has
been widely investigated in expressive text-to-speech (TTS) synthesis (Valle et al.) 20200] Ren et al.
202T Kenter et al., 2020; Ren al. 2022) and voice conversion (VC) (Kreuk et al.) [02 Zhou
et al. 2022), and has been shown to be important for generating natural and expressive synthesized
speech. Prosody is also applied in spoken language understanding tasks by providing information that disambiguates and complements information in the associated word sequence. Examples include
parsing (Tran et al., 2018), punctuation prediction (Klejch et al. 2017 Cho et al., 2022), emotion recognition (Rao et al. 2013), and other paralinguistic recognition tasks.
Prosody is traditionally defined in terms of its function in communicating linguistic structure and paralinguistic information and/or in terms of the associated acoustic correlates, which includes fundamental frequency (Fo), energy, duration, and other measures associated with vocal effort. In this work, focus acoustic correlates, specifically Fu and energy, with duration implicitly encoded via the temporal dynamics. These features have limitations. Fo tracking algorithms are known to be unreliable in some contexts, with pitch halving and doubling errors. Energy is sensitive to recording conditions. Fo and duration depend on speaker and segmental context. Further, Fo, energy, and duration are highly deeredeeen,,nt, but are often modeled independently, which can lead unnatural prosody in TTS and limit their usefulness in speech understanding. For these reasons, researchers have been exploring automatic methods for learning alternative representations of prosody.
Automatically learned prosody representations have been proposed for speech synthesis using autoen- coders that condition on text and speaker identity, which encourages residual information (assumed to be prosody) to be captured in an information bottleneck (Skerry- -earr et al., 2018]] WWag et al.) 2018, Zhang et al., 2019 Qian al. 2020). These approaches rely on having high-quality speech transcripts, limiting the amount of data that can be used in training and the ability to learn a broadly
generalizable representation. Another paradigm of representation learning is sulf-super learning
Auddio samples available at:

---

Under review as a conference paper at ICLR 2023
(SSL). SSL models are pretrained on large amount of unlabeled examples and then finetuned on task-specific data. This paradigm has been particularly successful for natural language processing (Peters et al. 2018 Devlin et al.) 2019). Recent speech SSL models like wav2vee 2.0 (Baevski et al., 2020), HuBERT (Hsu et al., 2021). and WavLM (Chen et al. 2022) have been proposed to learn acoustic representations from untranscribed speech. Focusing on the phone level, they achieve good performance on speech recognition and understanding tasks, especially when only a small-amoun of lask-specific data available. SSL methods have also been explored for prosody learning in Weston et al. 202I), but the approach requires word time marking, which relies on having human transcripts.
To address the challenge of learning a prosody representation without word transcripts. we propose ProsodyBERI self-supervised learning method that disentangles prosody features from speech content and speaker information. Similar to HuBERT (Hsu et al. 2021], we pretrain an SSL model by masked unit prediction. The pseudo labels are given by K-means clustering on speaker-normalized acoustic-pre -rossodicc- attributes (pitch, energy, and related features), which encourages the model to focus on prosody learning. In addition, inspired by SpanBERT (Joshi et al. 2020), we propose span boundary loss to encourage the model to better represent long-range prosody information. We also substantially compress the model size and reduce the feature dimensions to make the model easy to use. Similar to prior SSL models, ProsodyBERT is first trained on a large amount of raw speech audio and then adapted target tasks. Such design enables ProsodyBERT to learn rich representation of prosody on massive amounts of untranscribed speech.
Our approach follows that of recent speech synthesis work, which aims disentangle prosody from lexical content and speaker identity. With this view, acoustic-prosodic features that can be speaker-dependent, such as Fo range, are accounted for in the speaker representation. Disentangling the speech representation into different factors can improve the model's ability to generalize across different conditions and enable zero-shot speaker models for synthesis. While this factoring could be done in different ways (and include other factors), learning a prosody representation that has minimal speaker information also useful for privacy-sensttivtee speech processing. Using speaker verification experiments, we show that ProsodyBERT is effective at providing de-identified prosody features.
We demonstrate the effectiveness of pretrained ProsodyBERT features text-to-speech (TTS) and emotion recognition. During training, we extract ProsodyBERT features from speech and use them conditional inputs for the TTS decoder. A separate prosody predictor is trained such that it takes text and style as inputs and generates prosody features. During inference, the TTS decoder takes the predicted prosody features as input. Experiments show that the TTS system trained with ProsodyBERT features generates natural and expressive speech, surpassing FastSpeech 2 (Ren) et al. 2021) (trained with energy and Fo) by a large margin in subjective human evaluation. The expressiveness can be controlled by using speaking-style vectors (learned from multi-style data) in prosody prediction. For the emotion recognition task, we simply concatenate ProsodyBERT features to HuBERT features and use them as the inputs for downstream models. We achieve new state of the art on IEMOCAP emotion recognition task, showing that ProsodyBERT features are complementary to HuBERT features.
In summary, the key contribution of this work in the development of new, low-dimensional representation of prosody that can be learned from untranscribed speech. We demonstrate good speaker de-identification and utility of the new features in both TTS and emotion recognition.
2 RELATED WORK
Modeling Prosody in TTS and VC Most prior work on prosody treats prosody feature learning as an auxiliary module for downstream generation tasks. Recent approaches include directly using signal
prosody (Fo. energy, etc.) (Wan et al. 2201 Valle et al.) 2020 Ren et al., [022 Kenter et al. 2020
Liu et al. 2021 Kharitonov al. 2022), learning a latent style embedding (Wang et al. [0188 Zhang
et al. Hsu Hsu et al.) 2019: Sun et al., 2020), learning frame-level or phone-level representation
(Du & Yu! 2021 Kreuk et al. 2021), I). and utilizing reference audios for style (Choi et al. 2020 Yi
et al. 2022). Most of these prosody representations rely on task-specific models.
Learning Prosody Representations Prior work on prosody representation learning approaches are mainly based on either information bottleneck or self-supervised learning. Information bottleneck
2

---

Under review as a conference paper at ICLR 2023
[22 (2 (suo(=1s)
log Pi Roo (is) log Piyau (215
ha Itao, Pt)
Speaker Normalized K-mnans
Feature Extractor
+15
h11
Bag
Bag
kppo
ProsodyBERT Encoder
Anytr
[MASK] -.. [MASK]
[MASK]
Figure 1: Overview of ProsodyBERT pretraining. Frame-level acoustic-prosodee features (NCCF, Fo. energy, deltas) and the log Mel spectrum (bins 500H2) are first extracted from the audio file. Offline K-means clustering is done on the speaker-normalized generate the hidden cluster assignments (211, 212: 220) as the prediction targets. The equation shows the loss terms on the masked frame Z15. The masked unit modeling (MUM) objective uses the corresponding model outputs his to predict 215. The span boundary objective (SBO) uses the model outputs on the boundary to predict the cluster assignments of every frame in the masked span. Here 215 predicted given h11, h200, and the relative position embedding P5-
croustic-prosodit features to
approaches rely on carefully designed bottlenecks that condition lexical and speaker information
to capture the residual prosody information (Qian et al.) 2020; Ren et al. 2022). Other approaches acoustic-prosodic operate more directly on acoustic-pie prosodic correlates. Weston al. (2021) learns word-level prosody features from down-sampled signal by combining a temporal convolutional network and product quantizer with transformer-baser contextualization trained using contrastive task similar to [aevski et al. (2020). However, it relies word boundaries provided by transcripts to learn word-level prosody representation. Zayats et al.] (2019) learn a word-level prosody representation that uses deviations from text-based predictions of Fo, energy and duration compute prosody representation that is disentangled from lexical cues. |Polyak et al. (202T) augments SSL acoustic representation learning with unsupervised, discrete representation of Fo obtained using VQ-VAE (van den Oord et al. 2017). Our approach differs from all of these methods in that it learns frame-level embeddings from untranscribed speech, leveraging multiple acoustic correlates of prosody.
3 PROSODYBERT
An overview of the ProsodyBERT model is given in Figure Our model is based HuBERT (Hsu jet al. 2021) and SpanBERT (Joshi et al.) 2020), but is modified to focus on prosody representation learning that disentangles prosody from lexical content and speaker information.
Notation Given an audio segment A, the feature extractor outputs a sequence of acoustic features X (2,, ) where each I, corresponds to fixed-length frame and n is the number of frames. PRRSodyBERT takes X as input and produces contextualized vector representation for each frame H (h1, h2... h₂) that is trained to predict unsupervised prosody units Z (21, (21,22." ,,,,,,,,,,, The vectors h4 are the ProsodyBERT outputs.
Feature Extractor Given the raw audio segment A, the feature extractor first implements loud- ness normalization on A OdB, and then computes frame-based prosodic and spectral features. For prosodic features, we include log fundamental frequency (log(F))) Normalized Cross Corre- lation Function (NCCF), energy, and log (Fo). Fo and NCCF are extracted via the Kaldi pitch tracker (Povey et al. 2011h; Fo values are interpolated in unvoiced regions. Pitch (log( Fb)) and energy (E) are globally normalized on the corpus-level by z-score, and the deltas are computed on the normalized values. No normalization is done on NCCF features. For spectral features, we take the low-frequency bands (first 20 bins, < 200H2) of log Mel spectrum, focusing on the typical range of Fo and filtering out much of the information associated with lexical content. Mean and variance z-score normalization are performed on these low-frequence bins on the utterance level. log(F,), E, NCCF, A log( Fo), and log Mel features concatenated at each frame the output X.
3

---

Under review as a conference paper at ICLR 2023
Hidden Units for ProsodyBERT Inspired by HuBERT (Hsu et al.
2021), J. get the frame-level
target labels by acoustic unit discovery. Specifically, we train a K K-means clustering model on speaker- normalized prosody features (NCCF, lo)( A log( lo)(Fo), E)) using all the frames in the pretraining corpus. 7-score speaker normalization is done on E and log( Fo) before clustering to disentangle
speaker information. For speech utterance with acoustic features X (11, "2, '.. ,Tn), the discovered acoustic units are Z = (Z1, 22, ,n)) where 2, is a K K-class categorical variable.
Training objectives We adopt the same masking mechanism SpanBERT (Joshi al., 2020). wav2vec 2.0 (Baevski et al., 2020), and HuBERT (Hsu et al.) (021)) Let the input utterance be X = (21, ),,. Denote the set of all masked frames as M C X. Define X as the corrupted input sequence, in which the frames in M are replaced with special mask embedding. ProsodyBERT has two training objectives. The first is the masked unit modeling (MUM) objective. Let the output hidden states at the t-th frame be h2. We compute the masked unit modeling loss of frame as:
MMMM(=)) = - log Pm he).
(I)
The masked prediction model Pm takes hú as input and predicts a distribution over target label 24. It is implemented single-layer network followed by a softmax.
Previous work mostly relies on alignments to learn the span-level representation of speech (Weston
ct al., 2021; Hu et al. 2021). In contrast, our self-supervised method only relies on raw audio. Inspired by SpanBERT (Joshi et al.) 2020), we add span boundary loss to encourage ProsodyBERT to learn a long-range prosodic representation. Given a masked span (I..... Jee, in which s and are the start and ending frames of the span, and let 24
be a frame in this span. The span boundary loss
at time/ is computed as:
(2) in which h2 and he are the output hidden states of the span boundaries æ, and Ze, respectively, and P and q are the relative positional embeddings with respect to the left and right boundaries x, and Te. respectively. The span prediction model Pquan is implemented as 2-layer feedforward network followed by softmax. The span boundary loss forces the model to predict the entire masked span without relying individual tokens within it. The total loss is computed over all masked frames:
Lsno = - log Pspun (2t
Pt- Pe lc-t+1)
C(X,M,2
Σ (2)) + (z))
REM
4 PROSODYBERT FOR LO TTS
4.1 BACKGROUND: UTTS
We demonstrate the effectiveness of ProsodyBERT on UTTS LLan et al. 2022b). UTTS is framework for sulf-supervised multi-speaker TTS acoustic model pretraining that is combined with an alignment mapping model. The acoustic modeling pretraining does not rely on transcribed data. The alignment mapping model is trained on small amounts of paired text-audio data to map a a phone-based forced alignment sequence of unsupervised units learned in pretraining. UTTS is developed from the perspective of disentangling speech representation learning. We choose UTTS because it is self-supervised TTS framework that benefits from pretraining on large amount of data, which aligns with ProsodyBERT's s design. Also, our initial experiments indicate that UTTS synthesizes better quality speech than the trained models provided Lee et al. (2022).
Baseline UTTS Pretraining An overview of UTTS pretraining is shown in Figure 2 Speaker and alignment representation are disentangled via conditional disentangled sequential variational autoencoder (C-DSVAE) (Lian et al.) (022a) during self- supervuee training. Let X be training speech instance of length n frames. Denoting model parameters as 0, let pe and qo be the prior and posterior models, respectively. The training objective of C-DSVAE is a weighted sum of the following losses:
LXLD, Eptx) [KLD go (Zspeaker (X) || (O)
(4)
LKIDD. = Ep(x) [KLD( go (Zalign X) || po (paaiggg)))
(5)
Crecomstruct =Ep,X Exxx) Eng Siggpe x) 2-|-loge (Po &xpeaker: Zalign))
where Zepeaker is the learned utterance-level speaker representation (a vector embedding) and Zalign [Za1; Za2; Ran] is the learned phone alignment (a sequence of discrete unsupervised phonetic
4

---

Under review as a conference paper at ICLR 2023
C-DSVAE
D-DSVE
peecoetrunt
Decoder
ProsodyBERT
Supeoker
Sequsker
Apmoaely
N
Encoder
Acoustic Aligner
Nio, /)
Alncoder
Prosody
(Frocon)
X
(4) baseline UTTS pretraining
(b) UTTS pretraining with PrusodyBERT
An overview of UTTS pretraining, which only requires raw speech. The loss terms during training are marked purple.
Figure 2:
units 2nt). Creconstuutt is the reconstruction loss. KLD the KL-divergence, and CKLD, and CKLD. encourage disentanglement between Espeaker and Zalign-
4.2 UTTS PRETRAINING WITH PROSODY FEATURES
A limitation of the UTTS framework is that only speaker and word content (phone alignment) information are represented during training. There no explicit model of prosody, making the synthesized demos less expressive and potentially unnatural. ProsodyBERT provides a solution. As shown in Figure 2 (b), during training, we condition the UTTS decoder on the prosody features extracted from the speech segment X. Formally, we replace Equation 6 with:
Aprosoyy ProsodyBERT(X)
Lecconstruct ) |x] lo Zspeaker Zalign Aprosody ))]
(8)
ProsodyBERT is designed to focus prosody, disentangled from word content and speaker identity. By adding prosody embeddings (hpeosody) as a third source of reconstruction, the prosodic variations accounted for. Given pretrained (frozen) ProsodyBERT embeddings, the C-DSVAE can focus learning disentangled representations the speaker (Zspeaker) and content (Zalign).
4.3 UTTS INFERENCE AND TRAINING WITH PROSODYBERS
Inference An overview of the UTTS inference pipeline with ProsodyBERT is in Figure (a). Target speaker, text, and style ID are mapped into the learned representations of C-DSVAE to utilize the pretrained decoder for TTS. [Lian [ et al. (2022b) describes the process of mapping target speaker to Aspeaker and mapping phone alignment learned unsupervised alignment žalign Here we focus on component models for predicting the prosody features and phone durations.
In TTS, prosodic structure is typically tied to words and/or syllables. Here, we use word-level prosody vector embeddings, but also explore phone-level vectors. In inference, the prosody features are first predicted given the text and the style, and then the duration of each phone is predicted given the style, the speaker, and the predicted prosody features. (As shown in Section [ the inter-dependence of Fo, energy and duration make the prosody vectors useful for duration prediction.) After duration prediction, the prosody vector sequence is broadcast to the frame-level (Mprrsoddy according to the duration predictions. Similarly, the input phone sequence is broadcast to the frame level for input to the network that predicts the unsupervised acoustic unit sequence (Zalign). The two prosody predictor configurations are shown Figures (b) (phone-level) and (c) (word-level). The architecture of the prosody and duration predictors follows the variance adaptor of [Ren et al. (2021).
Training The UTTS decoder and acoustic unit mapping are trained as in Lian et al. (002bb), using the reference prosody vectors and learned unsupervised alignment. The prosody vector and duration predictors are separately trained. The reference phone- and word-level prosody vectors the mean pooled reference ProsodyBERT vectors using the respective (phone and word) forced alignment times. Unlike the baseline UTTS, our version with prosody conditioning learns the style vector during prosody prediction and not not jointly with the decoder.
5

---

Under review as a conference paper at ICLR 2023
B BSOBD SOSS
S 9
Expended Phone-level
3
Phone-lavel Prosody Predictor
Style
Ward-level Prosody Predictor
Style
Eepeaker
phore Bcadie
Phone Eneoder
RoBERTa (Frezon)
Ensrdde
Duration Pred
Prosody Pred
IH1 T SHI N AHO EY1
ay today
Speaker
Text
Style
It's sunday today
It's sunday today
(e) UTTS training and inference
(b) Phone-level prosody predictor
(c) Word-level prosody predictor
Figure 3: (a) UTTS for speech synthesis. The pretrained UTTS is paired with duration predictor and a prosody predictor for speech synthesis. These predictors are trained on small amount of text-audio pairs; (b) and (c) show phone-level and word-level prosody predictors, respectively.
The configurations of the prosody predictor depends whether the features at the phone-level or word-level. The phone-level prosody predictor takes the phone embedding sequence and a style embedding input, and it is jointly trained with the phone encoder and style embedding. The word- level predictor follows the design of Kenter et al. (2020). The input text is tokenized into WordPieces and passes through a RoBERTa encoder; the embedding corresponding the first WordPiece of each word is used as the word representation. The word-level prosody predictor is trained jointly with the style vector using the frozen RoBERTa embeddings. Given m phones or words (depending on the predictor), let the model output be O = (01, 02, 0m) and the reference ProsodyBERe features be H (h1, h2, hm). The training loss is:
Cpossdy
ΣI
m
1
10 helli log ar (cos(or; h,))]
where is the dimension of hidden states, is the sigmoid function, and cos is the cosine similarity. This loss is proposed Channg et al.) (2022), and we empirically find it yields good performance.
The duration predictor takes the phoneme sequence, the predicted prosody features (word-level features are expanded to phone-level via lexicon), the style embedding, and the speaker embedding as inputs, and predicts the duration (number of frames) of each phoneme. Montreal Forced Alignment (MFA) (McAuliffe et al., 2017) is used extract the target phoneme duration. The duration predictor architecture is the same in Lian et al.) (2022b) but with the added phone-level prosody input vector. The training loss the mean squared error (MSE) between predicted duration and target duration (both transformed to the logarithmic domain).
EXPERIMENTAL SETUP
5.1 PROSODYBERS PRETRAINING
use the full LibriTTS Zen et al. 2019) audio for PrOSodyBERT pretraining. LibriTTS contains 586 hours of audiobook data from 2,456 speakers. To generate the target labels, we run K-means clus- tering with 100 clusters using the MiniBare chKMeans algorithm in scikit-leaze (Pedregosal et al. 2011). The mask span length is set to / 10 for 20ms frames, and / 20 for 11ms frames. Notice that the masks can overlap, so the real span length is variable. We choose this span length because the average word duration in human speech is about 200-300ms. For the sampling of mask starting point, we set the probability to be 65%, resulting in about 50% of frames being masked. We reduce the model size and feature vector (to 32 dimensions) for computational efficiency. The Prosody BERT architecture follows the distilBERT (Sanh et al. 2019) and has 2 parameters. We use Adam (Kingma & Ba 2013) optimizer and linear learning rate schedule, with learning rate le
5.2 EXPERIMENTS ON TEXT-TO- SEXT-TO-SPEECH (TTS)
For text-to-speech experiments, two datasets are used: VCTK (Veaux et al. 2017) and DailyTalk (Lee) al. 2022). VCTK contains 44 hours of read speech from 109 speakers and does not contain much

---

Under review as a conference paper at ICLR 2023
prosody variation. DailyTalk contains 21.6 hours of spontaneous dialogue from 2 speakers and has rich prosody. We pool these two datasets so that the TTS system supports multiple speakers and two styles. VCTK reflects reading speech style. and DailyTail reflects a spontaneous speech style. The architecture of UTTS follows Lian et al. (022b), in which the style and speaker embeddings are learned during training. To support zero-shot scenarios for speakers, we use the speaker embedding from pretrained ECAPA-TDNN (Desplanques et al. 20200 Zhang & Yu 2022).
Baselines. We compare our TTS systems with two baselines. The first is the baseline UTTS, in which there is no explicit prosody component. For fair comparison, we do not extra audio for UTTS pretraining. The second is the official FastSpeech 2 checkpoint (Ren et al., 2021) trained on DailyTalk the system that gets the highest MOS score in Lee et al. (2022), outperforming Tacotron 2 (Shen et al. (018)) and the authors' baseline. FastSpeech 2 contains a pitch predictor and a energy predictor that predicts quantized Fo and energy, respectively. In contrast, our system is predicting PERT features. FastSpeech predicts phone durations from the phone sequence. Our UTTS with ProsodyBERT systems adds the predicted prosody features as inputs for duration prediction. which is shown to improve results.
5.3 EXPERIMENTS ON EMOTION RECOGNITION (ER)
We conduct the ER experiments on JEMOCAP (Busso et al. 2008) dataset, which contains 5 sessions. We follow the saave--one-ssssoonooe others are used for training and validation. The evaluation metric is weighted accuracy, which is the accuracy of all utterances in the test session. We explore the use of ProsodyBERT features together with different pretrained acoustic encoders, including base and large versions of wav2vec 2.0 (Baevski et al., 2020), data2vec (Baevski et al.] (022), and HuBERT (Hsu al. 2021). For all models, the base models have 12 layers with 768 dimensions each, and the large model has 24 layers with 1024 dimensions each. The acoustic encoder vectors are concatenated with 32-dimensional ProsodyBERT feature vectors.
setting. In each round, one session is used for test and the
There are two experiment settings. The first the SUPERB (Yang et al., 2021 probing setting, in which the weights of the pretrained speech models are frozen. The SUPERB S3PRL probing model (an LSTM) takes the weighted average of each model layer (with supervision to learn weights) concatenated with the 32-dim ProsodyBERT features the input. In the second setting, the concate- nated frame-based sequence of acoustic and ProsodyBERT vectors is input to a conformer- based decooer module, and the model is jointly trained with autoregressive setting to predict the emotion label and the corresponding transcription from the given utterance. To maximize the model performance, experiment with HuBERT-large since this model gives the mate-of-the-ee on IEMOCAP. We perform domain-adaptive IEMOCAP, then finetune the whole model, including the pretrained acoustic model and our Prosody- BERT, via ESPne use the raw pitch and energy prosody features instead the ProsodyBERT vectors.
ProsodyBER' pretraining on the training sessions of
Watanabe et al. 2018 Arora et al. 2022). As contrastive experiment, we also
RESULTS AND ANALYSIS
In this section, we first conduct speaker verification experiments assess the extent to which the prosodyBERT learning strategy de-identifies the feature vectors, and then report results of TTS and emotion recognition experiments.
6.1 SPEAKER D
We assess the de- ce-iienntificatinn of pretrained ProsodyBERT features via a speaker verification task, where higher Equal Error Rate (EER) implies less speaker-related information in the features. The task is performed on the VCTK test using the same code (including experiment configuration and random trials) as Lian et al. (2022alb c). We use mean pooling of the frame-level prosody vectors the representation of the speaker for an utterance. Two utterances are randomly drawn from
ht / / thub. www.:hwwww....o.ooaee.ooe http://eeonnee942////e. 1yTa
The paraettass for training and decoding are the same the default ESPnet recipe httpa tig.......o /ttp:hhho.cooeeepeeee///eerr/eee http:t/teree/eee/eee////////a/
7

---

Under review as a conference paper at ICLR 2023
the VCTK test set, and cosine distance is used to determine whether the speakers the same different. The EER computed from 36900 random trials. The ProsodyBERT features are compared to the full feature vector input to ProsodyBERT and the opeaker-normaliin prosody features used with K-means to derive the hidden prosody units. Table shows that Prosody BERT features have less beeaker-identifying information than the full input acoustic-prosodic features (energy, pitch, and low-freq mel), as well as the speaker-normalize representations.
Representation
EER
MOS(1)
WER (4)
Ldur (1)
Energy Pitch + Mel
8.2%
Reference
4.53 +: 0.06
5.6%
Energy + Pitch + Mel (SN)
20.4%
FastSpeech 2
3.37 + 0.09
25.3%
Energy + Pitch (SN)
23.4%
ProsodyBERT
35.3%
Baseline UTTS
3,50 + 0.09
23.2%
0.125
Phone prosody
3.63 1 0.08
21.1%
0.119
Word prosody
3.81 + 0.09
18.9 %
0.117
Table 1: Equal Error Rate (EER) of
speaker verification VCTK test set, us- ing different prosody representations. SN means "speaker normalized." Higher EER implies a higher level of de-identification.
Table 2: MOS scores (with 95% confidence intervals), ASR word error rate (WER), and duration validation loss (Idur) for different TTS settings on DailyTalk,
6.2 STYLE-CONTROLLEC TTS
Quantitative Assessment Results of TTS experiments on the DailyTalk validation set are reported in Table for the two configurations of ProsodyBERT in comparison the two baselines (UTTS and FastSpeech 2) and the reference audio sample. Mean Opinion Score (MOS) tests use 20 subjects who are asked evaluate the naturalness of each sample. Intelligibility is assessed in terms of ASR word error rate (WER) using LibriSpeech ASR model For duration prediction, the validation loss (average MSE of log duration) shown for the UTTS systems. Trends are consistent for criteria, with higher MOS for the TTS systems using ProsodyBERT, as well as lower WER and duration loss. For all criteria, word-level prosody features give the best results, with substantial improvement over FastSpeech 2. We hypothesize that this is due to the fact that word-level context is important for prosody prediction. The reduction in duration loss compared to the UTTS baseline shows that the prosody embeddings are implicitly representing duration effects.
Example comparison of synthesized speech We compare the prosodic contours of the synthesized audio and the ground truth detail on Figure The Fo plots show that both ProsodyBERT models capture the pitch accent on "go," but only the word-level prosody representation gives the final rise in Fo. The figures suggest that there are some issues with prediction of voicing with the ProsodyBERT outputs, but it is imperceptible and may simply be an Fo tracking error. Energy contours also give a better match to the ground truth for the ProsodyBERS Appendix C showing that adding prosodic controls (both in FastSpeech and with ProsodyBERT) impacts the full spectral shape, not just Fo and energy.
models. The spectrograms are given in
Baseline UTTS
FastSpeach
Prosody
Word Prosody
Ground Truth
Comparison of the pitch and energy contours of synthesized speech "where do you want to go?" and the ground truth in Daily (5_l_d1127). The orange line Fg and the puple line is E. The audio samples are available on the demo page.
Figure 4:
Expressiveness Control With the given training data, our system has two speaking styles: read (VCTK) and spontaneous (DailyTalk: However, more variation in expressiveness can be generated
speech_eephccc Barpcch__s__trin____
conformer?_hueee
8

---

Under review as a conference paper at ICLR 2023
using these style vectors, shown in Figure 5 Specifically, change the expressiveness by using prosody vectors that are a weighted sum of the word-level prosody features predicted for the two styles. The number in the figure titles are the weight of VCTK-style prosody features. As the weight becomes bigger, the synthesized speech has less variation in pitch and lower energy. Anecdotally, human evaluation suggests that the naturalness of synthesized speech is kept for all the weights.
Daily Talk Style
0.25
0.75
VCTK Style
Figure 5: The pitch and energy contours conditioned on different level of expressiveness, controlled by weights of ProsodyBERT features of VCTK and DailyTall Talk styles. The text is "How can I solve the problem? I am angry."
6.3 EMOTION RECOGNITION ON IEMOCAP
Model
Acc.
Method
Acc.
Original
+ProsodyBERT
WISE (Shen
2020)
66.5
wave2vec2-PT Pepino et al. 2021
67.2
wav2vec2-base
63.4
64.2
HuBERT-large Gat el al. 2022
71.9
wav2vec2-large
65.6
68.9
data2vec-base
66.3
70.4
HuBERT-large TAP Gat
2022
74.2
data2vec-large
66.3
70.3
Ours
HuBERT-base base
64.9
66.1
HuBERT-lrgee + Signal Prosody
74.3
HuBERT-large 67.6
69.1
HuBERT-large ProsodyBERT
75.8
Table 3: Results on IEMOCAP dataset in SU-
Table 4: Results IEMOCAP dataset in su- pervised setting. The "signal prosody" refers to ProsodyBERT inputs.
PERB (Yang et al., 202T probing setting.
Table shows our results on IEMOCAP in SUPERB probing setting, in which the pretrained models all frozen during training. ProsodyBERT features concatenated the existing self-supervised acoustic model features as the input to the downstream models. In experiments with three different acoustic models, we find consistent improvement after adding prosody features, for both the base and large sizes. These results demonstrate that ProsodyBERT captures additional prosodic information that are complementary to of these pretrained models.
Table compares our results on IEMOCAP with those of recently reported systems in the fully supervised setting, in which all parameters are finetuned. Our method outperforms the current state- of-the-art (Gat al. 2022). For analysis, compare the ER accuracy of "HuBERT ProsodyBERT" with "HuBERT + Signal Prosody" features, which includes Fo, energy, NCCF, deltas, and < 500 Hz log Mel. The relative performance gap shows the benefit of self-supervised learning.
7 CONCLUSION
We propose ProsodyBERT, a self-supervised method to learning prosody representations apart from speech content and speaker information. Our method does not rely on any transcripts external models. This self-supervised framework allows ProsodyBERT to learn the full distribution of prosody variations via pretraining on large amount of data. We demonstrate the effectiveness of ProsodyBERT on a style-controllable TTS system, showing that the TTS model trained with ProsodyBERT features can generate natural and expressive speech, outperforming a model trained with energy and pitch. We also achieve new state-of-the-art on IEMOCAP emotion recognition, showing that ProSodyBERT features are complementary to pretrained models like HuBERT, wav2vec2, and Data2vec. Future work may explore broader usage of BERSodyBERT. For example, ProsodyBERT may be combined with a large language model for spoken language understanding or translation tasks by augmenting text vectors with prosody embeddings.

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
The model details are given in Section 5 The source code for the model architecture, together with all the hyperparameters, are provided in the supplementary material. The trained model checkpoints will be released after the anonymity period, and they can be directly used with the provided source code. For the training procedure, we have accounted for the hardest issues in Appendix. The TTS demos uses in the MOS test are also provided in the supplementary material for reference.
REFERENCES
Siddhant Arora, Siddharth Dalmia, Pavel Denisov, Xuankai Chang, Yushi Ueda, Yifan Peng, Yuekai Zhang. Sujay Kumar, Karthik Ganesan, Brian Yan, et al. Espnet-SI.U: Advancing spoken language understanding through espnet. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7167-7171. IEEE, 2022.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for sulf-supervieed learning of speech representations. Advances in Neural Information Processing Systems, 33:12449-12244 12460, 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-359, 2008.
Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee. DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit bert. ICASSP 2022 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), May 2022. doi: 10. 10.11099/ccassp4422222000 9747490. URL www.htpp//// //xx. doi org.oogg//lo.oo 10. 11w.:11111111111132221112 202. 9747499
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian. Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. WayLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, URL http:/ /ttp://xxx.. doi
pp. 1-14, 2022. ISSN 1941-0484, doi: 10. 1100///ssp.2222.318858. org htc//00.119//stteo 2022. 188113
Jenny Yeonjin Cho. Sara Ng, Trang Tran, and Mari Ostendorf. Leveraging prosody for punctuation prediction of spontaneous speech. Interspeech 2022, 2022.
Seungwoo Choi, Seungju Han, Dongyoung Kim, and Sungjoo Ha. Attentron: Few-shot text-to- speech utilizing attention-based variable-length embedding. Interspeech 2020, Oct 2020. doi: 10.2 10.21437/intesspeechh 2020-202000000222000202222222222 URL httt::/ //xx. orc.lorg111... 2ttr//10..2433//ttereeeeee. 2020-2096
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: Emphasized Channel Attention, propagation and aggregation in TDNN based speaker verification. In Interspeech 2020, pp.
2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Shori Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18655/w1/111-144222 URL https
//aclanthology. org/N19--- -11-NN991110 1423
Chenpeng Du and Kai Yu. Rich prosody diversity modelling with pevne-level mixture density network. Interspeech 2021, Aug 2021. doi: 1022433/mterppecch.0001-1 URL http: 2021
7/dx. doi wtt..oog/o...1337////
10

---

Under review as a conference paper at ICLR 2023
Itai Gat, Hagai Aronowitz, Weizhong Zhu, Edmilson Morais, and Ron Hoory. Speaker normalization for sulf-supe speech emotion recognition. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7342-7346, 2022. doi: 10.1 11000/CASSS44922.222..77777
Wei-Ning Hsu, Yu Zhang. Ron Weiss, Heiga Zen, Yonghui Wu, Yuan Cao, and Yuxuan Wang. Hierar- chical generative modeling for controllable speech synthesis. In International Conference on Learn- ing Representations, 2019. URL httll//tt//pppeeee/rree///
net/ www.e//ooonnieeyygk3300000
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and AbdeIrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451 3460, 2021. ISSN 03329-93044 doi: 10.1 doi.org/10, 10. 1109 las.aaspppp.... 2021.
http://///d
Yushi Hu, Shane Settle, and Karen Livescu. Acoustic span embeddings for multilingual query-by- example search. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 935-942, 2021. doi: 10.11099LL4499002222 231.995555..
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zeutleoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, Dec 2020. ISSN 2307-387X. doi:
10.1 10.11t2/tacl_2 a 00300.
doi. www.olggg..oooo /1c tacclaacclcccccccaaeeaaaeaallcla 00300
Tom Kenter, Manish Sharma, and Robert A. J. Clark. Improving the prosody of rnn-based english text-to-speech synthesis by incorporating a bert model. In Interspeech, 2020.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. Text- free prosody-aware generative spoken language modeling. Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), 2022. doi: 10.18653/v1/
2022.acI-
2el..aclllong.99 long.593. URL/tttp: /xx doi. org/ 10. 10w.::13//11/2222111... acl -ong. 593
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
http
7/arxiv. org/ abs 1412. 6980
Ondfej Klejch, Peter Bell, and Steve Renals. tion combining lexical and acoustic features. In Proc. ICASSP, pp. 5700-5704, 2017.
models for punctuated transcrip-
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using decomposed and discrete representations. CoRR,
2021.
URL httttttsss///axxttt org. aaaaabas/il/ 2111. 07402]
Keon Lee, Kyumin Park, and Dacyoung Kim. Daily Talk: Spoken dialogue dataset for conversational text-to-speech. arXiv preprint arXiv:2207.01663 2022.
Jiachen Lian, Chunlei Zhang, Gopala Krishna Anumanchipalli, and Dong Yu. Towards improved zero-shot voice conversion with conditional dsvae. In Interspeech, 2022a.
Jiachen Lian, Chunlei Zhang, Gopala Krishna Anumanchipalli, and Dong Yu. UTTS: Unsupervised tts with conditional disentangled sequential variational auto-encoder. arXiv preprint arXiv:2206.0251/ 2022b.
Jiachen Lian, Chunlei Zhang, and Dong Yu. Robust disentangled variational speech representation learning for zero-shot voice conversion. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6572-6576. IEFF, 2022c.
Rui Liu, Berrak Sisman, Guanglai Gao, and Haizhou Li. Expressive tts training with frame and style reconstruction loss. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29: 1818, 2021. ISSN 2329-9304. doi: 10. 10.1109/tss.. 3076336.. URL http: /xx doi arg/100.109//1 2022 3076369

---

Under review as a conference paper at ICLR 2023
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, 2017.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825- 2830, 2011.
Leonardo Pepino. Pablo Riera, and Luciana Ferrer. Emotion recognition from speech using wav2vec 2.0 embeddings. Pruc. Interspeech 2021, pp. 3400-3404, 2021.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 1 (Long Papers), pp. 2227-2237, New Orleans. Louisiana, June 2018. Association for Computational I. Linguistics. doi: 10.18653/v1/N166 1202. URL https
//aclanthology Btw.hhhooooy.oog/11--0-100
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal I Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. Interspeech 2021, Aug 2021. doi: 10221457inerspecch222--777 URL httppppp/oooo.....orr/.....2222. http://///tttteeeeeeehee2e
Daniel Povey, Arna Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicck, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society, 2011.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pp. 7836-7846. PMLR, 2020.
K Sreenivasa Rao, Shashidhar G Koolagudi, and Ramu Reddy Vempada. Emotion recognition from speech using global and local prosodic features. International journaa of speech technology. 16(2): 143-160, 2013.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech 2: Fast and high-quality end-to-end text to speech. In International Conference Learning Representations, 2021. URL
net/ forum? ?icerummidippi piiidddppyyxxxxxxx
Yi Ren, Ming Lei, Zhiying Huang, Shiliang Zhang, Qian Chen, Zhijie Yan, and Zhou Zhao. Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech. ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2022. doi: 10. 11///////sspp922.22222.... icassp43922 2022. 974
FRLLhttp:// //dp:::/dd../ doi ort/.o..g/1...1.1
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108, 2019, URL hattp://arxx. orgg/bb/ /1910.91.11 01108
Guang Shen, Riwei Lai, Rui Chen, Yu Zhang, Kejia Zhang, Qilong Han, and Hongtao Song. WISE: Word-level interaction-basee multimodal fusion for speech emotion recognition. In Interspeech, pp. 369-373, 2020.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4779 IEEE, 2018.
RJ Skerry- Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In international conference on machine learning, pp. 4693 4702, PMLR, 2018.
12

---

Under review as a conference paper at ICLR 2023
Guangzhi Sun, Yu Zhang, Ron J. Weiss, Yuan Cao, Heiga Zen, and Yonghui Wu. Fully- helly-hhirarccicall fine-graie fne-grained prosody modeling for interpretable speech synthesis. ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). May 2020. doi: 10. 1109/icasp44076..20000 200.766222200053552 URL http: p: http///xx..oo.oooo////ooooooooot wtt.:/...110///AAS5555.......... 2020.. 9053520
Trang Tran, Shubham Toshniwal, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Mari Osten- dorf. Parsing speech: a neural approach to integrating lexical and acoustic-prosodic informa- tion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 69-81, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi:
10.18653/v1/N188 1-001- URL
Rafael Valle, Jason Li, Ryan Prenger, and Bryan Catanzaro. Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). May 201.///caap447772222000666666
2020. doi:
URL wwt.tttpp// /x www.:ooooooogo..ooooooo 0rt./0.1199/
ICASSP40776 6. 2020. 9054556
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6309-6318, 2017.
Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit. 2017. URL httph:tttt....orgget 7488/ /s//24555 645
Vincent Wan, Tom Kenter, Chun-An Chan, Rob Clark, and Jakub Vit. Chive: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network. In Proc. ICML, 2019. URL lttp::/proceddingg mlr. www.seaww.........ee..e.eeeeeeee prress/v97/kenteee 9a/ kenter19a.pdf
Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference Machine Learning, pp. 5180-5189. PMLR, 2018
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Proceedings of Interspeech, pp. 2207-2211, 2018. doi: 10.21437/ntesspeeech.oo 21437 Interspeech. 2018 12008-111 1456
1456. URL hhtp::t////tt doi org/10
Jack Weston, Raphael Lenain, Udeepa Meepegama, and Emil Fristed. Learning de-identified representations of prosody from raw audio. In International Conference on Machine Learning, pp. 11134-11145. PMLR, 2021.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watan- abe, Abdelrahman Mohamed, and Hung-yi Lee. SUPERB: Speech processing universal per- formance benchmark. Interspeech 2021, Aug 2021. doi: 1021443/meerseccc22211117777 URL 21432114222 www.7/1tterrpeechhh.ooom 2021-e1echhh22117777
http://da
Yuanhao Yi, Lei He, Shifeng Pan, Xi Wang, and Yujia Xiao. Prosodyspeech: Towards advanced prosody model for neural text-to-speech. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7582-7586. 2022. doi: 10.1109/
SCAASP44922..0222.94497
Vicky Zayats, Trang Tran. Richard Wright, Courtney Mansfield, and Mari Ostendorf. Disfluencies and human speech transcription emrors. Interspeech 2019, Sep 2019. doi: 10.214477imtrsseecchh 2019-3134, URL/http: doi httpa/////////o/oooo/o/////////e 201 2-19-331 3134
13

---

Under review as a conference paper at ICLR 2023
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: corpus derived from librispeech for tepech. Interspeech 2019, Sep 2019. doi: 29 /ttp://////////.........///////.
201 2019-2441
Chunlei Zhang and Dong Yu. C3-dino: Joint contrastive and non-contrastive self-supervised learning for speaker verification. IEEE Journal of Selected Topics in Signal Processing, 2022.
Ya-Jie Zhang, Shifeng Pan, Lei He, and Zhen-Hus Ling. Learning latent representations for style control and transfer in end-to-end speech synthesis. ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019. doi: 10. 10.1109icassp. 2019.8683623. URL http::/ doi. org 10 I 2019. 8683623
Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Emotional voice conversion: Theory, databases and esd. Speech Communication,
13: 137:1-18, Feb 2022. ISSN 0167-6393, doi: 10.1016/j.specom.
doi 2021. 122..11.000 11006.. URL Ott.tttppp///x......or//////0010 spwpco.........mmmmmmm.m.mmom
specom. 2021. wwwppppoom2...............
11.006
14

---

Under review as a conference paper at ICLR 2023
Appendices
A
OVERVIEW OF DATA USED
Datasct
Domain
Hours
Total speakers
Usage
LibriTTS (Zen et al.
2019
Audiobook
586
2,456
Prosody BERT pretraining
VCTK (Veaux et al. 2017
Reading
44
109
Text-to-speech
DailyTaie (Lee et al. 2022
Conversation
21.6
2
Text-to-speech
JEMOCAP (Busso et al.
2008
Conversation
12.5
10
Emotion Recognition
Table Summary of the datasets we used.
An overview of the data we use are shown on Table
B
MORE ABOUT DURATION PREDICTION
Inputs
Val loss (predicted prosody)
Val loss (oracle prosody)
Phones
0.125
Phones + Phone Prosody
0.119
0.010
Phones + Word Prosody
0.117
0.077
Table The validation loss of duration prediction on DailyTalk. The loss is computed by MSE on log duration (number of frames of each phone, with frame length about 11ms).
Table shows the duration prediction performance for the baseline UTTS model compared to prediction using prosody vectors, both for automatically predicted and oracle prosody. The results show that the prosody vectors reduce prediction error, which demonstrates that duration effects of prosody are implicitly modeled by mean-pooling the sequence of frame-level vectors.
C SPECTROGRAMS OF SAMPLES
We include the spectrograms of the audio samples presented in Section 6 in Figure 6 and Figure 7 for reference. Figure O shows that besides the energy and pitch contours, prosodic controls also impact the full spectrogram for both UTTS and FastSpeech 2.
Baseline
FastSpeech 2
Phone Prosody
Word Prosody
Ground Truth
Figure 6:
The log Mel spectrograms of the audio samples in Figure
15

---

Under review as a conference paper at ICLR 2023
DailyTalk style
0.25
0.5
0.75
VCTK style
AR
Figure 7: The log Mel spectrograms of the audio samples Figure
D IMPLEMENTATION DETAILS
ProsodyBERT has two configurations. For emotion recognition, following Hsu et al. (2021), I). the input audio sampling rate 16kHz, and the frameshift is 20ms. For TTS, following Ren al. (202T), the input sampling rate is 22.05kHz, and the hop length is 256 (frame shift ~ 1 ms). For both the ER and TTS versions, we train the model for 10 iterations on GPUs, with batch size 8. Training takes about 6 hours for ER and 10 hours for TTS (due to the shorter frame shift).
D.1 ARCHITECTURE
The ProsodyBEe BERT architecture follows the distilBERT architecture (Sanh et al. 2019), with the exception that the positional embeddings are replaced with the convolutional positional embedding as in (Baevski et al. (0200) to support long inputs. The encoder has 6 transformer layers, with hidden dimension 512, 8 heads, feedforward dimension 2048. The final projection layer has dimension 32. The model has 21M parameters.
D.2 K-MEANS ON ACOUSTIC CORRELATES
We train a K-means clustering model on frame-level prosody features (NCCF, (Fo) A (Fo). E). Z-score speaker normalization is performed log(Fo) and E, and A log( Fu) is computed on the normalized log( Fo). NCCF is already normalized, so no further normalization is needed. By such, disentangle speaker information. Each feature is scalar, clustering is done on 4-dim vectors. We have not added any weightings on each dimension because the features have already been normalized before clustering.
D.3 SPAN MASKING
We adopt the same masking mechanism as SpanBERT (Joshi et al.) 2020), wav2vec 2.0 (Baevski ct al. 2020), and HuBERT (Hsu et 2021). Let the input utterance be first randomly select subset of Y C X such that |Y| m |X]. is given hyperparameter, and set to 65% in our experiments. The frames in Y are the starting point of the masked spans, and spans of I frames are masked. Notice that the spans overlap, so the length of the spans is not fixed.
(11, -. Zn). we
D.4 ALIGNMENTS OF WORDPIECE, PHONEMES, AND WORDS
There is no direct mapping between WordPieces and Phones. However, the RoBERTa encoder in word-levee prosody predictor only takes WordPiece as inputs, and the UTTS system takes frame-level inputs. To deal with this mapping issue, we first map words to WordPieces (represent each word by its first WordPiece, and ignore the other WordPieces), and then broadcast this word-level feature to its corresponding phones via a lexicon.
16


Reviews:
[{"review_id": "R2HzU2rKDx-", "reviewer": "Reviewer_925W", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces ProsodyBERT for learning prosody representations to strengthen the prosodic aspects in unsupervised text-to-speech synthesis (TTS). Similar to HuBERT (Hsu et al, 2021), ProsodyBERT is a masked encoder model trained to predict K-means cluster labels, except in this case the K-means is performed on domain-informed features (such as pitch, energy, and dynamics, etc) to capture the long-term temporal structure of prosody. These learned hidden features then become targets for training a prosody predictor, and then downstream a duration predictor. To evaluate the effectiveness of ProsodyBERT, the paper demonstrates expressive TTS, interpolation between speaker’s prosodic styles, and how ProsodyBERT features improve emotion recognition in speech. \n", "summary_of_the_review": "The paper brings to attention the importance of prosody, and provides an effective approach to modeling it and adding it to unsupervised TTS systems. The approach is sound, and the results sound compelling. More discussion on the evaluation and comparison to prior work could help position this work and its contributions. A more complete demo page would also strengthen the paper’s contributions. \n", "strength_and_weaknesses": "Strengths\n\nThis paper provides a targeted and effective solution for modeling prosody, which is crucial to synthesizing expressive and natural speech. The overall design of the modeling, training and inference pipeline is well-thought-out, leveraging strong prior work as building blocks, such as HuBERT (Hsu et al, 2021) and SpanBERT (Joshi et al, 2020) for prosody representation learning and UTTS (Lian et al, 2022a,b) as the unsupervised text-to-speech synthesis, and demonstrates how ProsodyBERT representations can be added to strength prosody generation. The provided synthesis examples are compelling, and the improvement on the emotion recognition task shows that the choice of features and learned representations are useful.\n\nWeaknesses \n\nSince the main contribution of the paper is on learning prosody representations and predicting prosodic features for synthesis, it could be helpful to provide more grounding on what prosody is, and how it motivates subsequent design choices in modeling. Informally, prosody is both a general phenomenon, could also be a person’s signature way of speaking. What is the dependency structure between style, speaker, prosody, duration? How does the paper choose its current ordering of conditioning and predicting them? In the appendix, zero-shot speaker synthesis was mentioned as a motivating factor for prosody features to be not dependent on speaker. Perhaps this is a design choice inherited from prior work that focuses on the zero-shot setup that is also well-suited for the current setting? \n\nFor comparison to prior work, it could be helpful to have a paragraph discussing how the baselines are chosen, what other models are considered but may not be as relevant, or practically not feasible to compare to. In what ways it may or may not be an apples-to-apples comparison. For example, what are the model sizes involved, which may be a result of the different design objectives of the prior models? \n\nSome examples seem to be missing from the demo page, for example “emotion control” examples, and there’s only one example for “expressiveness control”. For the “expressive TTS” examples, would it be informative to include a few examples for each text to show the range of expressiveness that is possible. It seems all the text sentences are from the original dataset. How does the approach perform on unseen text?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NogiQC2wcq", "reviewer": "Reviewer_J32w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work explores utilizing self-supervised pretraining on speech data for improving prosody modeling in TTS and emotion recognization tasks. The proposed model, ProsodyBERT, is similar to HuBERT / SpanBERT, but with more input features added (f0, energy, NCCF). Experiments are conducted on two tasks: TTS (on DailyTalk + VCTK datasets) and emotion recognition (on IEMOCAP dataset).", "summary_of_the_review": "This paper tackle on an important problem (self-supervised prosody modeling). However, the contribution from the works is limited, and there are issues on the technical correctness.", "strength_and_weaknesses": "Strength:\n - The idea of this work is sound\n - The experimental results looks interesting\n\nWeaknesses\n - The novelty is limited.\n - Technical correctness issues, as explained in the next section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JCP1vvhGnV", "reviewer": "Reviewer_PaY1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a BERT-based architecture for extracting prosody embeddings from speech. The pre-training of the model is performed following a self-supervised strategy similar to that of HuBERT but based on prosody features to get the cluster labels. The performance of the proposed approach is evaluated in an unsupervised Text-To-Speech Task proposed in previously published work.", "summary_of_the_review": "The paper proposed a BERT-based architecture and a self-supervised training strategy for prosodic-related embedding representation extraction from speech signals. The prosodic learned features are used in an unsupervised TTS task yielding a real improvement in terms of Mean Opinion Score and an ASR Intelligibility metric (Word Error Rate). The proposed approach is built on top of recent advances in speech processing and unsupervised TTS technologies and proposed new ideas to incorporate prosodic features to enhance the natural and expressive of synthetic speech.\n\nThis is a great work, congratulations to the authors.", "strength_and_weaknesses": "The paper exploits recent ideas in speech processing and text-to-speech technologies, to propose a new and promising idea to incorporate prosodic information into speech synthesis/generation tasks. The proposed approach is theoretical well supported, and the authors provided experiments and results with enough empirical evidence of the improvements achieved by their approach. Nice work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4UNwJU7wGru", "reviewer": "Reviewer_CWFe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The following work proposes a self-supervised representation for learning. Unlike prior methods, the proposed framework relies only on raw audio. Feature vectors are extracted based on a combination of NCCF, log F0, delta log F0, and average energy per frame. The resulting N-D feature vector is k-means clustered to achieve discrete codes. Self-supervised training objectives from HuBERT and SPANBert are then used to learn the final representation. The resulting features are then applied to downstream unsupervised TTS and emotion recognition.", "summary_of_the_review": "Overall, easy to read paper with some promising results. However, I think the novelty is somewhat low and the complexity of the solution (large transformers with self supervised learning objectives) might not be necessary for some of the downstream tasks demonstrated in this work. However, I'm certainly happy to change my review if the authors can demonstrate otherwise.", "strength_and_weaknesses": "Strengths:\n- Fairly simple setup to achieve explicit incorporation of prosodic features\n- Demonstrated applicability in several downstream tasks\n\nWeaknesses:\n- While I have no doubt their setup works in practice, I do have some concerns regarding the K-means clustering.\n  - Not all frames contain valid F0 values. It's not clear how this is being handled, but constants and interpolated values will introduce devations from the true data distribution\n  - It's also not clear how to best combine these individual attributes. Scaling individual features can have drastic effects on which one dominates the clustering task.\n- I don't think there's an ablation on the SpanBert objective\n- Error bars missing in Table 1. Furthermore, MOS is not a good metric for comparing fine grained differences between various models compared to performing pairwise preference testing.\n- While there is no known good way to truly measure the quality of synthesized F0 and energy, I would be very careful about making claims backed up by a single qualitative comparison in Figure 4.\n- I'm not sure I follow the logic behind the claim that wave2vec, hubert, and waveLM filter out long-range prosodic information.\n- It's not fully clear how necessary the large transformer architecture with the self supervised objectives matter at all for some of the downstream tasks. I suppose an even simpler baseline would've been to individually quantize each prosodic feature into K bins using K-means (separately per feature). Then for each downstream task, pair each 1-hot encoding with its own embedding matrix to map to a lower dimensional representation. Alternatively, perhaps for the emotion recognition task, one could even just directly condition on the raw values like in Mellotron (Valle 2020), appending the user-normalized raw values with HuBERT features. \n\nQuestions for clarity:\n  - Can you clarify how EER is computed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject