Adaptive Speech Duration Modification using a Deep-Generative Framework
Authors: Ravi Shankar
Archana Venkataraman
Venue: ICLR 2022 Submitted (2022)
Keywords: Prosody
Encoder-Decoder
Attention
Adaptive Duration Modification
Dynamic Time Warping

Abstract: We propose the first method to adaptively modify the duration of a given speechsignal.  Our approach uses a Bayesian framework to define a latent attention mapthat links frames of the input and target utterances. We train a masked convolu-tional encoder-decoder network to generate this attention map via a stochastic ver-sion of the mean absolute error loss function. Our model also predicts the lengthof the target speech signal using the encoder embeddings, which determines thenumber of time steps for the decoding operation. During testing, we generate theattention map as a proxy for the similarity matrix between the given input speechand an unknown target speech signal. Using this similarity matrix, we compute awarping path of alignment between the two signals. Our experiments demonstratethat this adaptive framework produces similar results to dynamic time warping,which relies on a known target signal, on both voice conversion and emotion con-version tasks. We also show that the modified speech utterances achieve high userquality ratings, thus highlighting the practical utility of our method. 

Under review as a conference paper at ICLR 2022
ADAPTIVE SPEECH DURATION MODIFICATION USING A DEEP-GENERATIVE FRAMEWORK
Anonymous authors Paper under double-blind review
ABSTRACT
We propose the first method to adaptively modify the duration of given speech signal. Our approach uses Bayesian framework to define latent attention map that links frames of the input and target utterances. We train a masked convolu- tional encoder-decoder network to generate this attention map via a stochastic ver- sion of the mean absolute error loss function. Our model also predicts the length of the target speech signal using the encoder embeddings, which determines the number of time steps for the decoding operation. During testing, generate the attention map as a proxy for the similarity matrix between the given input speech and an unknown target speech signal. Using this similarity matrix, we compute warping path of alignment between the two signals. Our experiments demonstrate that this adaptive framework produces similar results to dynamic time warping, which relies on a known target signal, on both voice conversion and emotion con- version tasks. We also show that the modified speech utterances achieve high user quality ratings, thus highlighting the practical utility of our method.
INTRODUCTION
Human speech is a rich and varied mode of communication that encompasses both lan- guage/semantic information and the mood/intent of the speaker. The latter is primarily conveyed by prosodic features, such as pitch, energy, speaking rate, and voice quality. There are many appli- cations where understanding and manipulating these prosodic features is required. Consider voice conversion systems an example. Pitch and energy modifications are used to passively inject emo- tional cues into the neutral speech or to change the overall speaking style (A. Russell et al. 2003
Schacter et al. [001]] Shankar et al. 2019a b. Valle et al. 2019). Prosodic features are also used to evaluate the quality/engagement in human machine dialog systems (Swerts Krahmer 2000), and
they play a significant role in speaker identification and recognition systems (Park et al.
2016
While there are many approaches for automated pitch and energy modification (Toda et al. 2007 Ai-
hara et al., 2012 Kaneko & Kameoka 2017 Shankar et al.) 2020b a), comparatively little progress has been made in changing the duration/speaking rate of an utterance. In fact, the speaking rate plays a crucial role in conveying emotions (Schmidr et al. 2016) and in diagnosing human speech pathologies (Bayerl et al.) (2020). The speaking rate is diffcult to manipulate because, unlike pitch or energy, there is no explicit coding for either the signal duration or speaking rhythm. Rather, these features are implicitly defined by the cardinality of the set of frames over a particular interval of interest. This cardinality is a global parameter that masks subtle variations in the speaker rate over an utterance. As result, duration modification algorithms are not adaptive. Instead, they either require considerable user supervision or they are geared towards aligning known speech signals.
Perhaps the earliest duration modification method is the time-domain pitch synchronous overlap and add (TD-PSOLA) algorithm (Charpentier & Stella, 1986). TD-PSOLA modifies the pitch and duration of speech signal by replicating and interpolating between individual frames centered at the peaks of auto-correlation signal. However, the user must manually specify both the portion of speech to modify and the exact manner in which it should be altered. Hence, the method is nei- ther automated nor adaptive. An alternative approach is dynamic time warping (DTW), which finds the optimal time alignment between two parallel speech utterances (dtw) 2008 DTW constructs pairwise similarity matrix between all frames of the two utterances and estimates a warping path between the starting (0, 0) and ending (T., T:)) points of the utterances based Viterbi-like de-

---

Under review as a conference paper at ICLR 2022
Input Utterance
X
M Mask
Target
A
Target Frame
Figure 1: Graphical model for duration modification. y and 0 are the model parameters inferred during training. Attention A, is conditionally independent of target length T given X and M
coding the similarity matrix. While simple, DTW requires both source and target utterances to be known priori. Hence, it cannot be used for on-the-fly modification of new signals.
Finally, recent advancements deep learning have led to a new generation of neural vocoders that disentangle the semantic content from the speaking style (Oord al. 2016;; [Shnn et al.) 2017 Wang et al. 2017). These vocoders can alter the speaking rate via the learned style embeddings. While these models represent seminal contributions to speech synthesis, the latent representations are learned in an unsupervised manner, which makes it difficult to control the output speaking style in predictable manner. Another drawback of these methods is the large amount of data and compu- tational resources required to train the models and generate new speech (Ysuuda et al. 2020).
In this paper, introduce the first fully-automated adaptive speech duration modification scheme. Our approach combines the representation capabilities of deep neural networks with the structured simplicity of dynamic decoding. Namely, model the alignment between source and target utterance via a latent attention map: these maps are used as replacement of similarity matrix for backtracking. We train masked convolutional encoder-decoder network to estimate these atten- tion maps using a stochastic mean absolute eтror (MAE) formulation. Unlike the conventional DTW (dtw (2008) algorithm, once trained our framework operates entirely on the source utterance without needing reference the target. We demonstrate our framework on a voice conversion task using the CMU-Aretic dataset (Kominek conversion tasks using the VESUS dataset (Sager et al. (219).. Our experiments confirm that the proposed model can perform open-loop duration modification and produces high-quality speech.
Black] 2004) and on three multi-speaker emotion
2
METHOD
Fig. illustrates our underlying generative process. Given an utterance X, we first estimate the length T of the (unknown) target utterance Y and subsequently use it to estimate mask M for the attention map. The mask restricts the domain of the attention vectors A₁ at each frame t during the inference stage mitigate distortion of the output speech. We use paired data (X,r. Y.r) to train a convolutional encoder-decoder network to generate the attention vectors. During testing, we first generate the attention map from the input X and use it produce the target speech Y.
2.1 LOSS FUNCTION
Formally, let E DxT, denote the input speech. In this work, X corresponds to the Mel filter- bank energies extracted from short-time moving window analysis, where D is the number of filter- banks, and T, is the number of temporal frames in the utterance. Similarly, we denote the target speech Y E IR DxT Notice that the target utterance length 7 is usually different from T.,
2

---

Under review as a conference paper at ICLR 2022
Slope 2
Slope 1.6
Slope 1.2
Figure Binary attention masks with different slopes.
Our generative process for single frame of the target speech is represented as follows:
T Laplace(T), br) and Y, Laplace(Y,", by),
(1)
where is the estimated length of the target utterance, and Y is the target Mel filter-bant energy features time t. The parameters (1 br, Y by] of the distributions are unknown and implicitly estimated via a deep neural network. The neural network is parameterized by and (Fig. D.
By treating the unknown parameters as functions of the input X, obtain the following estimating equations for the target sequence length and frame-wiss wise Mel filter-bank energies:
f fx( and Y A + Yo.-1.
(2)
The functions (-) and fa correspond to the length prediction and energy estimation component of the same deep neural network. The variable A, E R is an attention vector that combines frame- wise features of the source utterance X to generate the target frame Ý, Our model differs from standard sequence-to-sequene
model by treating neural net predictions as residuals added input sequence itself. Notice that residuals depend on input and the history of predictions Yo.-- at previous time steps. This autoregressive property allows the neural network to learn segmental and supra-segmental variations that can potentially distinguish between the speakers or emotions.
During training, with respect to the neural network weights {0, This likelihood can be written as:
paired data (X,, Y) and maximize the likelihood of the target speech signal
P(ý, (Ý) P( II P(Y]X f, Po--1),
where, the second term in Eq. (3) be obtained via marginalization A, as follows:
P 10.-1) = Σ
P(G A,X, Y,
Σ P(MXX,T, A., Ar, (o...) Yot-1, A.
(4)
The variable M here denotes the attention mask. We introduce M for convenience, it deter- ministic function of the source length T. and the estimated target length T'. We encode the attention A, one-hot vector across the T, frames of the source speech. Thus, it follows a multinomial distribution. For simplicity, we model A, as conditionally independent of the target length T given the mask M and the input X (see Fig. []. Taking log(-) of Eq. (3) and combining with Eq. (4) yields:
L(0, = log Σ
P(V,, A,X, T, M, - log (P())))
A.
- log Σ 40 (A,, 1o M)
P(Ý. A A, Yo: log (P(T1X))
go ( (Ai|X, Y0.-11,M)
qo( A Pot-1, M\ log (P(8))X, Ar, Pot-1)) log (Pg( + -K((((A))))P(AL))
A.
Σ
go( (Ar|X, Yo.-1, log (P Ar, Fo.-1)) log (P())X) HH const.
3

---

Under review as a conference paper at ICLR 2022
S-E
E qo You M) (P(Y)X, Ar, Yor-)) log (P(TIX) + const.
(5)
The distribution (-)-) above is an approximating distribution for the attention vectors implemented by convolutional network. The first inequality uses the convexity of the log function, and the second inequality comes from the fact that entropy H (qo) 2 O. Notice that we have implicitly as- sumed P P(A,|X, 10 M has a uniform distribution over the masked region. This is reasonable assumption given that the masking process reduces the attention domain to a small region (see Sec- tion 2.3). However, qo is not penalized for deviating from this uniform distribution during training. This flexibility allows the network to learn realistic attention vectors during autoregressive decoding. Eq. (5) can be easily translated into neural network loss function which we minimize for {0,7); 7)
L(O, 7) = X1 x E Au [lg (P(2, X, A,, ao + A x log (P(T
= A1 x EA, 1 11 + X2 x | -
(6)
where X and \2 are the model hyperparameters that adjusts the trade-off between the two objectives and implicitly contain the variances of the Laplace distributions in Eq. (I). Notice that the loss in Eq. (6) computes an expectation over the attention maps. We use the Monte-Carlo estimate by sampling from the attention map at each time-step. The training procedure is therefore stochastic in nature due to this random sampling. We mix this stochastic version with the maximum aposterioni
estimate (MAP) of the attention vector with a probability of 0.2 during the start of training procedure.
Algorithm 1: Strategy for model training
function trainMoodeParrmmeeers (X, Y);
Input :
filterbank energies (X E R RDT, Y RDT.)
Output: model parameters (8,7)
epoch < MaxEpochs then
Set f = 0, predict target length = (X) and create the mask M E REXT; Estimate A E IR 727x7 x using masked convolution and sample u ~ U(0 1);
if < 0.2 then
Sample RT. as 1-hot vectors from A; Reconstruct using y, X fo(X, Yo:-1);;
else
Reconstruct using y, X A + fo(X, Yot-1); end 10
Compute prediction errors and update parameters 0, epoch epoch 1;
13 end
return 0 and 7:
2.2 TRAINING AND TESTING ALGORITHM
Algorithm|| T describes our training strategy, First, we extract the filterbank energies from the paired onput-output utterances and define the ground truth for length prediction as the ratio of the input- to-output sequence length. Next, we construct the attention mask (Sec. 2.3) for each ouputt sample pair in the mini-batch. The neural network then processes the input frames and generates an embedding for the decoder operation and to predict the target sequence length. It further estimates an attention vector region. Finally, to compute the loss, we either sample from this multinomial distribution or directly apply the maximum posteriori attention estimate the input sequence get the target frames.
multinomial distribution at each decoder step inside the specified masked
We use a Bernoulli sampling procedure with probability of 0.2 in line (i.e., low contribution of the stochastic loss) to prevent the model from diverging in sub-optimal directions. The MAP estimate helps in this regard. Empirically, we found this to be extremely helpful in generating monotonic attention that is also group sparse in nature. We fixed the slope of attention mask in
4

---

Under review as a conference paper at ICLR 2022
Algorithm 2: Strategy for model testing (i.e., open-loop duration modification)
function modifyDuration (X);
Input filter-bank energy (X E RDxx, and Yo) Output: alignments ((#1, y1), (122 v2),
Predict length of target sequence 7, (X);
Create attention mask M E R and Set = O; if T, then
Using mask Mr. X, and You-1 estimate As;
Using X, You-1, and A,, predict Y;;
+
end
Run DTW backtracking the attention matrix A; return (alignments (I1, 81), (22, v2), (In, vn)): )):
Input 00
(Y.
Conv Input
Projection
Projection
Projection
Drapout
Dropout
Encoder
Convolve
GLU
Masking
Softmax
Module
(M]
Matmul
Input (X)
Sum
Projection
Sample
Input (X)
Target Len
Fully
Encoder Output
Decoder Output
Output
Encoder Module
Decoder Module
Attention Medule
Figure 3: Neural network architecture used for the sequence-to-ssquenne speech generation. The encoder and decoder modules consist of 10 identical blocks. Projection layers are simple feed- forward layers without any non-linearity to project input features in high dimension.
line 4 to 1.25 based on the relative difference in length observed from the training datasets. The distribution go is the variational distribution which is approximated by fully convolutional neural network shown Fig. 3 The testing procedure is shown Alg. 2 Here, we not sample from the attention distribution at each time step t. Instead, the encoder-decoder model is allowed to autoregressively for the predicted number of time steps (Tt). We this attention map proxy for DTW similarity matrix to run Viterbi decoding strategy and obtain a sequence of co-ordinates representing the frame-wise correspondence between the input and estimated target speech.
The generative/testnng procedure (Alg. 2 is similar to the training strategy except for some minor differences. In the generative mode, ground truth. Second, we do not sample from the attention vector as the estimated target sequence no longer important. We use the attention map as proxy for the DTW similarity/cost matrix for Viterbi alignment. This alignment allows us to rearrange the input frames to give modified speech.
the predicted target sequence's length instead of the
2.3 MASKING
The mask M is used to constrain the scope of the attention mechanism be similar in time-scale to the input. This procedure is important for two reasons. From a speech quality perspective, large local swings in speaking rate may generate unintelligible speech. From an estimation perspective, the speech utterances contains hundreds (sometimes thousands) of frames. difficult to robustly train deep network to generate such long attention vectors using smaller datasets.

---

Under review as a conference paper at ICLR 2022
CMRLAVC
VESLUS-AA
VESUI-NN
VESUS-NS
Longth Prodiction Ertor (mosf (sec)
400
-00
GRU
Trans.
Coms
GRU
Trans:
Cone
6RU
Trons
Com:
CRU
Trans.
Comn
Figure 4: Comparing error in length prediction using encoder embeddings.
We use the masks derived from Itakura parallelogram (Itakura 1975). as illustrated in Fig. 2 The Itakura parallelogram is generally used to speedup the DTW algorithm when the speaking rates in the source and target utterances expected to lie in certain interval. The slope of the Itakura parallelogram specifies the minimum and maximum speaking rates that the reconstructed utterances allowed possess in comparison the input speech. In this paper, fixed the minimum and maximum variation in speaking rate to 0.8 and 1.25, respectively after observing the training data.
2.4 NEURAL NETWORK ARCHITECTURE
We adapt the neural network architecture from (Gehring et al. 2017) by adding residual/skip con- nections to the last layer and reconfiguring the entire aftention module. Fig- Jisows the encoder, de- coder and the modified attention module of the convolutional neural network used for experiments. The encoder is responsible for generating feature embeddings for the decoder and for predicting the relative length of target speech. The sample operation in Fig. 3 is responsible for generating sample from the attention distribution required for reconstruction and backpropagation.
We train model using mini-batch gradient descent and Adam optimizer (Kingma & Ba 2015 with fixed learning rate of 10 We fixed the batch size to 16. The input is 80-dimensional Mcl- filterbank energies spanning O-8kHz. The projection layer expands this input to 256 dimensions. Both the encoder and decoder consist 10 convolutional layers, each followed by gated linear unit. data augmentation to properly estimate the network
Given the small size of training data, parameters. Specifically, we reverse the input-output sequences and randomly extract intervals of variable size (with probability 0.5) from the full speech utterance.
2.5 DTW BACK-TRACKIN
Our final step uses the attention map produced by the encoder-decoder as proxy for the DTW similarity matrix between the source and target speech frames. This strategy allows us to train the convolutional model on relatively small dataset (e.g., 2-3 hours) and still generate intelligible speech during open-loop modification of new utterances. Effectively, apply dynamic program- ming operation to the attention maps produced by the neural network to get a path of alignment from source target, rather than rely the noisy spectral reconstruction for resynthesis (see Al- gorithm 2}. To avoid skipping phonemes, we constrain the dynamic programming path to take at most horizontal vertical at time while backtracking. Once estimated, the path informs reorganization of the source utterance frames via local contraction dilation operations. Following this reorganization, the target speech synthesized via the WORLD vocoder (Morise al. 2016).
EXPERIMENTAL RESULTS
We evaluate our model on two multi-speaker datasets: CMU-ARCTIC (Kominek & W Black 2004)
and VESUS (Sager et al. 2019), We query various properties the model on tasks described below.

---

Under review as a conference paper at ICLR 2022
OMU-VE
VESUS NA
VESUS NH
VESUS-NG NS
Edit Distanze (match ratoo
6RU
Trans.
Coms
GRU
Trans.
Cone
GAU
Tars.
Core
GRU
Trans.
Comy
Figure 5: Comparing alignment similarity between attention map and DTW.
3.1 DATA AND VOICE MORPHING TASKS
CMU-ARCTIC database has American English speakers (two male, two female), who we paired according to their gender, for voice conversion. This gives us total of 2264 sentence pairs. We train our duration modification framework using 2164 utterances from the database and use the remaining 100 utterances (random 50-50 split) for validation and testing the open-loop modification properties.
VESUS is an emotional speech corpus containing 250 phrases read by 10 speakers in 4 emotion classes: neutral, angry, happy, and sad. Each utterance in VESUS corpus contains 10 crowd-sourced emotional annotation representing the saliency intended emotion category. Given the variation in quality, only utterances that are correctly annotated by at least half of the listeners.
Our task on VESUS to inject emotion into a neutral/monotone utterance. Thus, we train three duration models corresponding to the three neutral-emotional pairs, resulting in the following splits:
Neutral to Angry: 2385 utterances for training. 72 for validation and, 61 for testing.
Neutral to Happy: 2431 utterances for training, 43 for validation and, 43 for testing.
Neutral to Sad: 2371 utterances for training. 75 for validation and, 63 for testing.
The utterances in VESUS corpus are short with an average signal duration ranging from 1.5 seconds. Given the small sample size due to shorter length sequences, we train the convolutional neural network on CMU-ARCTIC from scratch and fine-tune it for each emotion conversion task.
3.2 LENGTH PREDICTION
As sanity check, compare the predicted utterance length by framework with that of the ground truth parallel utterance. As seen in Fig. 3 the utterance length is predicted by summing the encoder embeddings across time and feeding the sum into linear layer. summation aggregates information across the entire utterance without the instabilities of recurrent architecture. Our baselines are two most commonly used sequence-to- teeuuuee-to-sequecce se-seqeence frameworks: (i) Gated Recurrent Unit GRU model (Cho et al. 2014), and (ii) Transformer model (Vaswani al.) 2017]. The architecture and training strategy baseline models are described more detail the appendix section. Fig. E shows the error in predicting the length ratio in ms/sec format. Notice that, our framework mispredicts the utterance lengths by only 40ms/sec and 65ms/sec (on average) CMU-ARCTIC and VESUS, respectively. Duration prediction is particularly challenging VESUS due to marked differences between neutral and emotional utterances. The median predic- tion error for GRU model is in the range of 400 600ms per second of the input utterance. The Transformer model fares relatively well in comparison to GRU because of their ability to establish long-range dependency. However, our framework performs best in this challenging scenario, likely due to our multi-task setup and the fusion of deep representation with Bayesian regularization.
high level, the
3.3 ATTENTION ALIGNMENT
Next, we compare the loon alignment between the source and target speech frames estimated via our attention map with the original DWT algorithm (where both utterances are known). To

---

Under review as a conference paper at ICLR 2022
Generated Sequence
(1,25,
(1.25,3)
(1,25,5)
[1.5, 11
(2.5.3)
(1.3, Sj
[2.00
(2.0, 3) Source Sequence
(2.0,
Figure 6: Effect of slope and step constraint the alignment. The tuple under each image is in (slope, constraint) format. Red curves are the optimal path obtained via Viterbi back-tracking.
150
VESUS
VESUS-NH
Vowels
Cormonanss
Pauses
139
Wreeb
Consonants
Rauses
220
339
180
Figure 7: Comparing the duration difference between source/target and source/converted pair of utterances for vowels, consonants, and short pauses. Short pauses have large difference because our method does not generate silence frames by itself if it does not exist in the input utterance.
compare the warping paths, code the horizontal, diagonal, and vertical moves of the backtracking procedure into three classes. We then compute the edit distance between the DTW alignment and the attention map based alignment. Once again, compare method against popular sequence- to-sequence benchmarks. Fig. illustrates the match ratio normalized by the average length of sequences. As seen, the match ratio varies between 0.70 and 0.85, which suggests that our approach is good at learning the general characteristics of duration modification. The GRU model performs poorly in this task due to its inability to learn sequence transformations in the order of 100s of frames. The Transformer model does relatively well on this task as they can handle long sequences. The convolutional model however, performs best because of the Itakura masking constraint and its ability to exploit the continuity of short-time Fourier representation of speech. Furthermore, our method can trained on limited data resources. contrast, sequence-to-sent to-squeecc models for voice conversion require hundreds of hours of training data along with sophisticated noise removal models to generate actual speech. Thus, our method can be used as a tool for manipulation of speaking rate at both, local and global scale. To our knowledge, this is the first demonstration of its kind.
Fig. 6 shows the effect modifying the slope of Itakura parallelogram and the horizontal/vertical movement constraint imposed during the dynamic decoding stage. As expected, relaxing the slope constraint and increasing the number consecutive horizontal/vertical moves provide more flex- ibility in adjusting to the speaking rate of generated speech. However, this flexibility lead to
8

---

Under review as a conference paper at ICLR 2022
MOS Score
3
1
0
CMU-VC
VESUS-NA
VESUS-NS
Figure 8: MOS of speech generated by model evaluated via AMT.
missing or distorted phonemes, suggesting a trade-off between changing the speaking rhythm and preserving naturalness. We allow the users to tune these knobs for their own end application.
3.4 COMPONENT-WISH DURATION ANALYSIS
To further probe into the behavior of our method, Fig- compares the difference in duration between our converted utterances and the ground truth target utterances for vowels, consomants and short pauses (e.g., silence and unvoiced phonemes). We use Penn phonetic forced alignment tool (Yuan) & Libermann 2008) to get the location and duration of each phoneme in given utterance and group them into the above three categories. The results indicate that our proposed method faithfully mod- ifies the duration vowels and consonants. However, is less effective with short pauses, as seen across all four tasks. This trend is intuitive, as we cannot create pauses if these frames do not exist in the source utterances. One direction of future work is to add a branch to our deep network that estimates the duration of short pauses intermittently within the utterance. Nonetheless, our model consistently estimates the difference between vowel and consonant duration across multiple tasks, which corroborates our claim of developing a general purpose speech rate manipulation framework.
3.5 SPEECH RECONSTRUCTION QUALITY
Finally, we use crowd sourcing obtain a mean opinion score (MOS) for the re-synthesized speech quality of the testing utterances. The crowd sourcing was performed using Amazon mechanical turk (AMT). We collect listener ratings for each converted utterance in the test set to leverage the sample averaging effect for reliable estimate of MOS. Further, we also add some noisy utterances to the converted samples set to flag non-invested listeners and bots on AMT. As seen in Fig. our method achieves average MOS between 3.7 - 4.0 across the four tasks. This performance par with the speech quality produced by state-of-the-art neural vocoders such as Oord et al. (2016); Shen et al. (017);; Kalchbrenner et al. 2018). We note that CMU-ARCTIC task has the lowest MOS, perhaps due to the longer and more complex utterances. Interestingly, the MOS is unaffected by errors in length prediction, evidenced by the VESUS neutral-angry emotion conversion task. This suggests that our approach of combining the neural network attention weights with a structured DTW algorithm provides robustness both the speech characteristics and estimation errors.
CONCLUSIONS
We have presented a novel deep- beep-generative framework for adaptive speech duration modification. Our model used a convolutional encoder-decoder architecture to estimate attention maps to associate frames of the input speech with frames of the target speech. The attention maps are modeled as latent variables, which lead to a stochastic formulation of the mean absolute error (MAE) loss for model training. During testing, the attention map is directly used to approximate the similarity matrix for a DtW-style backtracking procedure. We evaluated our framework on a voice conversion and three separate emotion conversion tasks. Overall, our framework produced similar duration modification as the vanilla DTW, but without requiring access to the target utterance. Further, we showed that the re-synthesized speech had similar naturalness to most fate-offthe-art neural vocoders.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Dynamic Time Warping (DTW), pp. 570-570. Springer Netherlands, Dordrecht, 2008. ISBN 978- 1-4020-6754-9. doi: 10. 10.1007/ 1-0000-6754999.444444
James A. Russell, Jo-Anne Bachorowski, and José-Miguel Domanddez-Doss. Facial and vocal expres- 11 2003. doi: 10.1 10/mmuu/e...
sions of emotion. Annual Review of Psychology,
1rcch54.101601.44111
Ryo Aihara, Ryoichi Takashima, Tetsuya Takiguchi, and Yasuo Ariki. Gmm-based emotional voice conversion using spectrum and prosody features. American Journal of Signal Processing, 2:134 138, 12 2012. doi: a1s.23/j.jjp.2212222000
Sebastian P. Bayerl, Florian Hönig, Joelle Reister, and Korbinian Riedhammer. Towards automated assessment of stuttering and stuttering therapy, 2020.
Charpentier and M. Stella. Diphone synthesis using an overlap-add technique for speech wave- forms concatenation. ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing, 11:2015-2018, 1986.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical ma- chine translation. CoRR, 100788,0 2014. URL
1078
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning, 2017.
Itakura. Minimum prediction residual principle applied to speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23(1):67-72, 1975. doi: 10. 10.1109/TASS..1955.
1162641.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan. Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Effi cient neural audio synthesis. In Jennifer G. Dy and Andreas Krause (eds.), Proc. ICMI. 2018, volume 80, pp. 2415-2424. PMLR, 2018.
Takuhiro Kaneko and Hirokazu Kameoka. satalll--dtt--iee voice conversion using cycle-consistent adversarial networks. CoRR, abs/1711.11299, 2017.
Diederik Kingma and Jimmy Ba. abs/1412.6980, 2015.
Adam: A method for stochastic optimization.
CoRR,
John Kominck and Alan W Black. The emu arctic speech databases. SSW5-2004, 01 2004.
Masanori Morise, Fumiya YOKOMORI, and Kenji Ozawa. World: A vocoder-based high-quality speech synthesis system for real-time applications. IEICE Transactions Information and Sys- tems, E999DD:1877-980.. 07 2016. doi: 10. (0.15577rransinn221155111
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalehbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. CoRR, 0bss/1609.0344 2016.
Soo Jin Park, Caroline Sigouin, Jody Kreiman, Patricia Keating, Jinxi Guo, Gary Yeung, Fang- Yu Kuo, and Abeer Alwan. Speaker identity and voice quality: Modeling human responses and automatic speaker recognition. In Interspeech 2016, pp. 1044-1048, 2016. doi: 10.21437/ Interspeech. 2016-553..
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W. W, 2017.
Jacob Sager, Ravi Shankar, Jacob Reinhold, and Archana Venkataraman. VESUS: A Crowd- Annotated Database to Study Emotion Production and Perception in Spoken English. In Proc. Interspeech 2019, pp. 316-320, 2019. doi: 10.21437/lnn (021457/hneesppechh20--1
10

---

Under review as a conference paper at ICLR 2022
D.L. Schacter, D. T. Gilbert, and D. M. Wegner. Psychology (2nd Edition). Worth Publications, 2011.
Juliane Schmidt, Esther Janse, and Odette Scharenborg. Perception of emotion in conversational speech by younger and older listeners. Frontiers in Psychology, 7:781, 2016. ISSN 1664-1078. doi: 10.3389/fssg..201 20100000
Ravi Shankar, Hsi-Wei Hsieh, Nicolas Charon, and Archana Venkataraman. Automated Emotion Morphing in Speech Based on Diffeomorphic Curve Registration and Highway Networks. In Proc. Interspeech 2019, pp. 4499-4503, 2019a. doi: 10.21 10.24437/meerspechh201122222
Ravi Shankar, Jacob Sager, and Archana Venkataraman. A Multi-Speaker Emotion Morphing Model Using Highway Networks and Maximum Likelihood Objective. In Proc. Interspeech 2019, pp- 2848 2852, 2019b. doi: 10.21437/Interspeech.20111
Ravi Shankar, Hsi-Wei Hsich, Nicolas Charon, and Archana Venkataraman. Multi-Speakerr Emotion Conversion via Latent Variable Regularization and Chained work. Proc. Interspeech 2020, pp. 3391-3395, 2020a. doi: 10.21437/mteerspecch..00 2020-1323.
Sewao--Deeooorrreeeeeeeee Net-
Ravi Shankar, Jacob Sager, and Archana Venkataraman. Non-Parallel Emotion Conversion Using Deceraenceeeive Hybrid Network and an Adversarial Pair Discriminator. In Proc. Interspeech 2020, pp. 3396-3400, 2020b. doi: (0221437/merrpecchh2220 1325.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, R. J. Skerry-Ryan, Rif A. Saurous, Yannis Agiomyr- giannakis, and Yonghui Wu. Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions. CoRR, abs/1712.05884, 2017.
Mare Swerts and Emiel Krahmer. On the use of prosody for on-line evaluation of spoken dialogue systems. 04 2000.
T. Toda, A. W. Black, and K. Tokuda. Voice conversion based maximum-likelihood estimation of spectral parameter trajectory. IEEE Transactions on Audio, Speech, and Language Processing, 15(8):2222-2233) Nov 2007. ISSN 1558-7916. doi:
Rafael Valle, Jason Li, Ryan Prenger, and Bryan Catanzaro. Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762. 2017.
URL
org abs/1706. 03762
Yuxuan Wang. R. J. Skerry-Ryan, Ying Xiao, Daisy Stanton, Joel Shor, Eric Battenberg, Rob Clark, and Rif A. Saurous. Uncovering latent style factors for expressive speech synthesis. CoRR,
abs/1711.00522 2017.
Yusuke Yasuda, Xin Wang, and Junichi Yamagishi. Investigation learning abilities on linguistic features in sequence-tor -ousencc----eeuence text-to-speech synthesis, 2020.
Jiahong Yuan and Mark Liberman. Speaker identification on the scotus corpus. Proceedings of Acoustics '08, 2008

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1
DESCRIPTION OF GATED RECURRENT UNIT (GRU) MODEL.
Encoder
Attention
Decoder
h
h
(a, d]
dg
d
GRU
GRU
X
Xm
Vn
Figure GRU model architecture.
We compare the performance of our method with existing state-of-the-arr techniques for sequence- vo-sequence conversion. RNN-LSTM frameworks have been a benchmark for many natural lan- guage processing tasks. They have only been sparingly used for speech conversion tasks due to longer sequence length coupled with the inability RNNs to remember distant past. our base- line, we choose GRU (gated recurrent unit) cells (Cho et al. 2014) based encoder-decoder model due of their low computational footprint. Fig. shows the architecture of the GRU model for length prediction and approximating the DTW similarity matrix. The of encoder embeddings across time-axis is fed into linear layer for target length prediction. The encoder and decoder in the GRU model consist of single layer of 64 cells running in both, forward and backward directions. They are further connected via an attention layer implemented by a single feed-forward layer. We train the model using Adam optimizer with default parameters in PyTorch (Paszke al.. 2017) and fixed learning rate of le - 5. We follow the same data augmentation strategy as the proposed approach but remove the masking constraint for the attention alignment to stay close the vanilla model.
A.2 DESCRIPTION TRANSFORMER MODEL
We also train a vanilla Transformer model from scratch for the voice conversion task and then fine- tune it for emotion conversion. The Transformer model has layers of encoder and decoder and each layer has 4 attention heads. We use the same data augmentation procedure technique and train via Adam optimizer with default PyTorch settings. Similar to the GRU model, there is masking for the encoder-decodee dings across time-axis to predict the target length sequence. Transformer models can handle longer inputs due their non-sequential nature and ability form long-range dependencies between put tokens. Therefore, they naturally perform well on estimating the length of target sequence by learning appropriate encoder representation. The Transformer model however, fails leverage the continuity in the short-time representation of speech signal. An additional constraint we implicitly imposed on the baselines and proposed neural network architecture their ability to run on single NVIDIA K80 GPU. We are able to fit multiple layers of convolutional and Transformer model but only single layer of GRU on this GPU. allowed us to more innovative with our approach.
proposed
attention module. We use the of encoder embed-
A.3 ABLATION ANALYSIS: REMOVING ITAKURA MASKING
There multiple components in the proposed model which work in synchronization to produce naturally sounding speech. In addition to the generative modeling part, the two most important augmentations have made the standard deep neural network pipeline are: (i) using Itakura
12

---

Under review as a conference paper at ICLR 2022
Linear
Add Norm
Feed forward
Add & Norm
Add Norm
Feed forward
Multi-head attention
Add & Norm
Add Norm Masked Multi-head attention
Multi-head attention
Positton encoding
Position encoding
Input
Output
Figure 10: Transformer model architecture.
Length Prediction Error (ms/sec)
0.70
250
Edit Distance (match ratio)
150
0.60
100
0.55
CMU-CC
VESUS-NA (a)
VESUS-NH
VESUS-NS
CMU-VC
VESUS-NA
VESUS-NH VESUS-NS
(b)
Figure 11: (a) Length prediction of target utterances and (b) measuring similarity of attention map DTW cost matrix. The model trained without masking constraint imposed in the attention layer.
masking for attention scope and (ii) using an attention weighted residual connection in the final layer. Therefore, we perform ablation experiments to understand the relative significance of each of these augmentations. Our first experiment removes masking from the attention layers. Fig. [T] shows the model's performance on target length prediction and approximating the DTW similarity matrix. The results Fig. TTaa a) indicate that the length prediction performance roughly similar to the proposed model. This is expected because, we hypothesized that it is the multi-task setup that allows the convolutional network to estimate length with a relatively small error (in ms/sec). The match ratio metric however, (shown in Fig, IT(b) is considerably worse. Itakura masking procedure acts as a good inductive bias/prior on the attention map because the speech rate do not fluctuate drastically in human conversations. Therefore, the attention map is localized in model.
13

---

Under review as a conference paper at ICLR 2022
Length Prediction Error (ms/sec)
250
Edit Distance (match (match ratio)
0.70
200
0.65
150
0.60
50
0.55
0.50
CMU-VC VESUS-NA
VESUS-NH
VESUS-NS
CMU-VC
VESUS-NA
VESUS-NH VESUS-NS
(a)
(b)
Figure 12: (a) Length prediction of target utterances and (b) measuring similarity of attention map to DTW cost matrix. The model is trained without residual connection in the final decoder layer.
A.4 ABLATION ANALYSIS: REMOVING RESIDUAL CONNECTION
Our second ablation experiment involves removing the attention weighted residual connection from the final layer of decoder. Fig- [2]a) a) shows that the model is able estimate the target sequence lengths with a relatively low error rate. This is again due to multi-task training setup. The match ratio (Fig. b)) in this experiment is better than the no-masking results but, worse than the pro- posed model. Therefore, we can confidently say that Itakura masking helps in approximating DTW similarity matrix. Further, the presence of residual connection is extremely important a good starting point for the convolutional network to start predicting the target frames. Since the linguistic content of input and target utterances are same, the residual connection allows the neural network to inherit input speech properties which is helpful in auto-regressive generation mode.
provides
14

---

Under review as a conference paper at ICLR 2022
Figure 13: Examples of alignment path obtained using encoder-decoder attention map (left) and ground truth DTW backtracking procedure (right). The source sequence lies on the x-axis and the target/generated sequence lies y-axis. Note that, unlike DTW, the convolutional neural network does not use the ground truth target utterance. The attention map and the DTW similarity matrix exhibits similar block structure arising due to the short-term stationarity of speech signals.
15

---

Under review as a conference paper at ICLR 2022
Figure 14: Examples of alignment path obtained using encoder-decoder attention map (left) learned without masking constraint and ground truth DTW backtracking procedure (right). The source sequence lies on the x-axis and the target/generated sequence lies on y-axis. The attention map no longer exhibits the block structure and is distributed uniformly across the frames of source sequence.
16

---

Under review as a conference paper at ICLR 2022
Figure 15: Examples of alignment path obtained using encoder-decoder attention map (left) learned without residual connection and ground truth DTW backtracking procedure (right). The source quence lies on the x-ax and the target/generated sequence lies on y-axis. Once again, the attention map no longer exhibits the block structure and is distributed uniformly on y-axis in most cases.
17


Reviews:
[{"review_id": "re0alwsmj9J", "reviewer": "Reviewer_Lbse", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method looks interesting and novel, but there are several concerns over the evaluation, correctness, and clarity.\n\n### Evaluation\n\n- The performance of the proposed method is presented by reporting (1) length prediction error, (2) alignment similarity between the attention map and the original DTW result, and (3) MOS of re-synthesized speech.\n- A major concern over the evaluation is the absence of a proper baseline method, which is crucial for presenting the motivation of the work and the contribution of the paper. This problem applies to all 3 metrics and must be addressed by comparing them with existing works or the simplified version of the proposed method. Replacing the convolution architecture with GRU or Transformers does not look like a proper ablation, since the architecture itself is irrelevant to the proposed generative model.\n- Regarding the length prediction error, the reviewer considers this as an inappropriate metric for generative models on speech. Speech is inherently stochastic - meaning that the duration of the speech can vary even when the text and emotion are kept identical. Recent works on speech synthesis [1, 2] aim to make the duration of generative speech as stochastic as possible. Thus, the reviewer recommends the authors critically think about whether the length prediction error is a suitable metric for the proposed method.\n- Please elaborate on an alignment similarity metric, since the readers won't be able to judge how high/low the values given in Figure 5 are.\n- Providing the audio samples for the generative models on speech is considered necessary to convince the readers. Please supplement them.\n\n### Correctness\n\n- In section 3.3, the authors state that: \"In contrast, the sequence-to-sequence models for voice conversion requires hundreds of hours of training data along with sophisticated noise removal models to generate actual speech.\" The reviewer strongly disagrees with this statement. Recent state-of-the-art works on voice conversion [3] can work with an amount of data that does not differ much from VESUS. It is also unfair to directly compare with previous works since the proposed method has utilized data augmentation, as stated in section 2.4.\n- In section 3.5, the authors state that the speech quality from the proposed method matches that of WaveNet, Tacotron2, and WaveRNN since the average MOS was 3.7 – 4.0. But, this is not correct. Since the MOS results of a single model widely vary across the experiments and the dataset, the MOS of the existing models should be re-measured using the same evaluation protocol to be directly compared with the proposed method.\n- Since the proposed method is working on manipulating the mel-spectrograms, it is basically an acoustic model. Hence, in order to fairly compare the speech quality with existing models, it should not be directly compared with vocoders. For instance, the author might want to use a single vocoder for all models (including baseline).\n\n### Clarity\n\n- There are some unclear parts that might make the reader hard to comprehend the paper. To be specific:\n  - In section 2.2, what does the following sentence mean? \"The neural network then processes the input frames and generates an embedding for the decoder operation and to predict the target sequence length.\" It would be clearer if the referral to the specific line(s) from Algorithm 1 was made.\n  - In Algorithm 1, what is a variable 't' in lines 7, 9?\n  - The location of Figure 2 (start of page 3) is too far away from the section that the reference is made.\n- Some minor mistakes that did not affect the reviewer's judgement include (please fix):\n  - Strange referencing: \"(dtw, 2008)\"\n  - Unclosed parentheses in 4th, 5th line of equation 5\n  - Typo in section 3.3: DWT -> DTW\n\nSome general advice on paper's clarity: Figures do not speak for themselves. The authors might want to add some elaborations on each figure's caption to state what the takeaways are, even if it overlaps with the main text.\n\n### Overall comment\n\nThe reviewer would like to appreciate the amount of work done in this work, however, several issues exist and must be addressed to be presented at ICLR. Thus, the reviewer (initially) recommends rejecting the paper. \n\n### References\n\n- [1] Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro, \"Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis.\", ICLR 2021.\n- [2] Kim, Jaehyeon, Jungil Kong, and Juhee Son. \"Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech.\" arXiv preprint arXiv:2106.06103 (2021).\n- [3] Z. Yi, W.-C. Huang, X. Tian, J. Yamagishi, R.K. Das, T. Kinnunen, Z. Ling, T. Toda, \"Voice Conversion Challenge 2020 -- intra-lingual semi-parallel and cross-lingual voice conversion --\" Proc. Joint workshop for the Blizzard Challenge and Voice Conversion Challenge 2020, pp. 80-98, Oct. 2020.\n", "summary_of_the_paper": "This work proposes a generative model to convert emotion by adaptively modifying the duration of speech components without the need for reference utterance. The proposed method is evaluated via measuring (1) length prediction error, (2) alignment similarity between the attention map and the original DTW result, and (3) MOS of re-synthesized speech.", "summary_of_the_review": "The reviewer vote for rejecting the paper; several issues on evaluation, correctness, and clarity are visible.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "gdJ-pUIT_fr", "reviewer": "Reviewer_rjMF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. A thorough understanding of the problem place (speech parameter modification). \n2. Reexamination of an important and practical problem using recently advances in transformers.\n3. Evaluation on 2 public domain datasets.\n4. Pilot user study.\n\n\nThings to consider\n1. This statement, as it is stated in the abstract is misleading/incorrect. \"We propose the first method to adaptively modify the duration of a given speech signal.\" This relies on a definition of adaptation that is only available to the reader later. \n2. There are lots of strong claims in the Introduction that could be avoided. Drawbacks of DTW and neural vocoders are unwarranted without explicitly stating the practical applications goals being considered by the authors. The statement on data needs is misleading. For example, the present work derive key parameters, such as the slope of the attention mask, based on the training data.\n3. This statement is confusing. \"We mix this stochastic version with the maximum aposteriori estimate (MAP) of the attention vector with a probability of 0.2 during the start of training procedure.\" Do you mean a weighted additive approach?\n4. Lots of hand-wavy details such as \"Empirically, we found this to be extremely helpful in generating monotonic attention that is also group sparse in nature.\"\n5. Small-subject MOS-based comparisons across publications are problematic due to uncalibrated listener expertise. I appreciate the authors' efforts in conducting an user study but conclusions must be more guarded.\n6. The paper references only very early work on pitch and rate modification and a few recent ones. A couple of decades of work after PSOLA (90s - 2005) could be added to provide a more comprehensive review for the reader. ", "summary_of_the_paper": "Speech rate, duration, and pitch modification is of interest in several practical audio applications. The paper proposes the use of an encoder-decoder framework with attention masking to estimate a candidate target utterance length to overcome the need for a priori knowledge of a target utterance on a speaking rate modification task. Evaluation on 2 standard databases show that the proposed approach performs better than 2 sequence-to-sequence models that were originally developed for different domains.", "summary_of_the_review": "The paper brings the state-of-the-art ML research to an important but less active filed of study: speech signal modification. The proposed approach is reasonable and evaluations are solid. However, the authors haven't done a great job of explaining the drawbacks of previous approaches, in practice, and the critical need for unsupervised adaptation. Hence, the overall paper reads as a solution (transformer-based models) looking for a new application. The comparison baselines also reflect this discrepancy. A better comparison would be also related approaches in pitch and style modification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W0pmkqkmTzj", "reviewer": "Reviewer_uzK2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Very clear, easy to read paper\n- Interesting combination of assumptions for loss upper bound derivation\n\nWeakness:\n- Some of the modeling choices are not very clear:\n1. Why Laplace distribution was chosen for length T and target Y_t in Eq1\n2. Conditional independence of output length and A_t given mask M and input X: I can see it is needed for derivation\nof the upper bound (Eq 4), but I am not completely convinced if it is a realistic assumption\n", "summary_of_the_paper": "This paper proposes a model for adaptive duration modification of an input signal. The model is a graphical model with neural components. By making some assumptions, authors derived an upper bound for the likelihood (conditional probability of output sequence and estimated length given input). All the model parameters are updated, by minimizing this upper bound.", "summary_of_the_review": "My main concern with this paper is its scope, novelty and impact.\nFrom modeling and formulation perspective, I do not see much novelty. There is not a novel modeling paradigm which be of interest of general machine learning crowd (as a reader I did not learn a new ml method). Of course, the paper proposes a novel method by combining some existing machine learning tools (like graphical models, sequence-to-sequence modeling). In my opinion, this novelty could have been justified if the method was applied to some high impact problem. Unfortunately, it is not clear to me if the speech duration modification is such a problem.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GhGBJVNyGYA", "reviewer": "Reviewer_EmGa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The introduction glosses over most of the prior work in this area. The authors write: \"comparatively little progress has been made in changing the duration/speaking rate of an utterance\" -- Perhaps the authors have neural/deep models in mind, or some other narrow claim in mind, but time-scale modification is a classical topic in speech and signal processing. The intro, however, jumps from the TD-PSOLA approaches of the mid 80s to a 2017 reference, and skips over intervening work, like sinusoidal-based approaches (to cite one). (I'm not including DTW in this view, since it's not a true time-scale modification algorithm and requires knowledge of target durations.) A better literature review is needed.\n\nRe ealier approaches: \"the user must manually specify both the portion of speech to modify and the exact manner in which it should be altered. Hence, the method is neither automated nor adaptive\" -- One could argue that these approaches are, in fact, more flexible since they allow full (and local) control over what parts of speech to modify, a feature that made them a building block for many years of speech synthesizers that needed precise modification of specific speech segments. They suffer from many drawbacks (e.g., perceptual audible artifacts) that deep approaches can alleviate, and models that allow implicit duration modification (as in this work) are very attractive, so I think it shouldn't be hard to build better arguments to justify the value of this contribution.\n\n\"In this paper, we introduce the first fully-automated adaptive speech duration modification scheme\" -- Existing time-scale algorithms are adaptive and automatic. Perhaps this claim needs to be written in narrower terms to be true (that doesn't require external duration specification?, etc.), but even then it's hard to evaluate it in the absence of a solid prior work comparison.\n\n\n\"Furthermore, our method can be trained on limited data resources. In contrast, sequence-to-sequence models for voice conversion require hundreds of hours of training data along with sophisticated noise removal models to generate actual speech.\" -- I perceive 2 issues with this claim: One is that it seems exaggerated. There will be a natural trade-off with quality, but it doesn't really require hundreds of hours of speech to train a voice-conversion system between, say, a small number of voices (which is the case the authors tackle in the evaluation). And secondly, these bigger voice-conversion systems are tasked with performing a far more complex objective than what the authors have proposed to do, and evaluated, in this submission, namely duration modification. We don't know from this evaluation whether the proposed model has captured any of the target-speaker characteristics, so the comparison between models that are solving different tasks seems unfair.\n\nThe subjective evaluation is somewhat limited by the small number of independent ratings (5) per sample. \"This performance is on par with the speech quality produced by state-of-the-art neural vocoders\" -- We shouldn't compare results across independent listening tests, but generally we should expect the SotA neural vocoders mentioned to outperform a WORLD vocoder. I wonder what the  results would look like if the authors had chosen to include the natural, unmodified, speech in the mix of samples to provide a calibrating topline performance.\n\nClarifications Needed:\n\n-- Is there a point-wise or matrix-vector multiplication in Eqn. 2 between X and A_t? Looks like the former, but X is in R^{DxT_s} and A_t is in R^{T_s}\n\n-- I'm confused by the acoustic features the authors predict with the network (\"In this work, X corresponds to the Mel filterbank energies\") and the fact that the samples are synthesized using the WORLD vocoder (which requires other features like pitch, spectral envelope, aperiodicity).\n\n\nMinor edits:\n** Notice that, our framework --> Notice that our framework\n\n** Y \\in R^{DxT} (in 2.1) --> Elsewhere in the paper this is R^{DxT_t} \n\n", "summary_of_the_paper": "This paper proposes a generative sequence-to-sequence encoder-decoder architecture with attention for modifying the length of an input sequence. The authors derive the training loss for learning the network parameters from a principled Bayesian formulation and a variational inference bound, and use this to train the model from paired {input, output} sequences. During inference, a task-specific network is capable to estimating a target length, and modify the signal accordingly. It would seem that the architecture is capable of handling other transformations, and in fact the authors apply the model to speaker- and emotion-morphing tasks, but the bulk of the evaluation is on how accurate the  target length is estimated, and (briefly) the perceptual quality of the resulting samples. \n", "summary_of_the_review": "The paper provides a novel contribution by bringing neural generative models to the task of time-scale modification of speech signals. The work is evaluated in terms of objective metrics and shown to outperform some baselines. A (somewhat limited) listening test suggests this model attains high-quality samples though there may be some limitations in interpreting these results. The work is well written, but some of the claims need to be revised, and the contributions better situated against prior work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a deep neural network sequence-to-sequence framework for modifying the length of a speech sequence.  It employs a convolutional encoder-decoder architecture optimized under a Bayesian formulation with variational inference.  The proposed framework is evaluated on a voice conversion task and three emotion conversion tasks. The results show that it can successfully change the duration of an utterance without accessing the target utterance.  Almost all reviewers raised concerns with some strong or inaccurate claims made by the authors in the paper.  The literature review on related work also needs to be significantly improved.  Another major concern is on experiments. Other than the DTW compared in the work, the proposed method should also be compared with existing duration modification techniques. The MOS evaluation seems to be limited and needs further improvement to make the results stronger and more convincing.  Since the authors did not provide a rebuttal, all these major concerns remain unanswered.
Decision: Reject