INTERSPEECH 2023
20-24 August 2023, Dublin, Ireland

CFVC: Conditional Filtering for Controllable Voice Conversion
Kou Tanaka, Takuhiro Kaneko, Hirokazu Kameoka, Shogo Seki
NTT Communication Science Laboratories, NTT Corporation, Japan
kou.tanaka.ef@hco.ntt.co.jp

Abstracta) Training
Speaker ID (speaker S)

This paper describes a many-to-many voice conversion modelthat ﬁlters the speaker vector to control high-level attributes
such as speaking rate while preserving voice timbre. In order to control only the speaking rate, it is essential to decompose the speaker vector into a speaking rate vector and others.
The challenge is to train such disentangled representations with
no/few annotation data. Motivated by this difﬁculty, we propose an approach combining the conditional ﬁltering methodwith data augmentation. The experimental results showed that
our method disentangled complex attributes without annotationand separately controlled speaking rate and voice timbre. Audio
samples can be accessed on our web page1 .
Index Terms: Voice conversion, conditional ﬁltering, disentanglement, speaker representation, sequence-to-sequence learning

Augment ↵src
coefficient

Speaker ID (speaker T)↵tgt Augment<latexit sha1_base64="RBzS9yUsEGjuuFJ24uAvdhRQFvo=">AAACb3ichVHLSsNAFD2Nr1pfVRcKghRLRTdlIr5wVXTjsq22CraUJE5raJqEZFqopT/gB+jChQ8QET/DjT/gwk8QV1LBjQtv04CoqHeYmTNn7rlzZka1Dd0VjD0GpK7unt6+YH9oYHBoeCQ8OpZ1raqj8YxmGZazqyouN3STZ4QuDL5rO1ypqAbfUcsb7f2dGndc3TK3Rd3m+YpSMvWirimCqFxOMewDpdAQJdEshKMszryI/ASyD6LwI2mFr5HDPixoqKICDhOCsAEFLrU9yGCwicujQZxDSPf2OZoIkbZKWZwyFGLLNJZoteezJq3bNV1PrdEpBnWHlBHE2AO7YS12z27ZE3v/tVbDq9H2UqdZ7Wi5XRg5mtx6+1dVoVng4FP1p2eBIlY9rzp5tz2mfQuto68dnrS21tKxxiy7ZM/k/4I9sju6gVl71a5SPH2KEH2A/P25f4LsQlxeji+lFqOJdf8rgpjCDObovVeQwCaSyNC5No5xhvPAizQhTUuRTqoU8DXj+BLS/AfQTI8A</latexit><latexit sha1_base64="pH1pY1r24MPYkOziCgebkZs9vnE=">AAACb3ichVHLSsNAFD2N7/po1YWCIMVS0U2Zii9cFd247MM+oC0liaMNpklI0oIGf8AP0IULHyAifoYbf8BFP0FcSQU3LrxJA6JFvcPMnDlzz50zM5KhKpbNWCsg9PT29Q8MDgWHR0bHQuHxibylN0yZ52Rd1c2iJFpcVTSesxVb5UXD5GJdUnlBOthy9wtNblqKru3Yhwav1MV9TdlTZNEmqlwWVaMmVh3LlI+r4SiLMy8i3SDhgyj8SOnhW5SxCx0yGqiDQ4NNWIUIi1oJCTAYxFXgEGcSUrx9jmMESdugLE4ZIrEHNO7TquSzGq3dmpanlukUlbpJyghi7IndsTZ7ZPfsmX38WsvxarheDmmWOlpuVEMn09n3f1V1mm3UvlR/eraxh3XPq0LeDY9xbyF39M2js3Z2IxNz5tk1eyH/V6zFHugGWvNNvknzzDmC9AGJn8/dDfJL8cRqfCW9HE1u+l8xiBnMYYHeew1JbCOFHJ1r4BQXuAy8ClPCrBDppAoBXzOJbyEsfgLCTo75</latexit>

Conditional filtering

coefficient

Decoder
LS2S Data(+ attention)
augment(+ post-net)

SourcespeechDataaugment

Encoder

Targetspeech<latexit sha1_base64="sCxGSOImA4HG9u6vWHIGaNp7I0s=">AAAGSHicnVTPTxNBFH5Ui4g/AL2YeGksGPBApg1R44lETExEBWqBhJJmdzsta/dHs7utLety9MA/4MGTJh6MF6969WK8e+BPMF5MMCEmHvzmdVFbAkvcSTvvfe993868Nzt6wzL9QIidgdSJk+nBU0Onh8+cPXd+ZHTswrLvNj1DFg3Xcr1VXfOlZTqyGJiBJVcbntRs3ZIrev22iq+0pOebrvMo6DTkuq3VHLNqGloAqDw6XmKN0JOVKCzZWrBhaFZmPiqz7dlhIV+IyqNZMS34yRw0crGRpfhZcMdSz6hEFXLJoCbZJMmhALZFGvkYa5QjQQ1g6xQC82CZHJcU0TC4TWRJZGhA6/ivwVuLUQe+0vSZbeAtFn4emBmaEF/EG7ErPom34qv4dahWyBpqLR3MepcrG+WR7UuFvUSWjTmgjb+sREYVK1Q18I/Y336uC9xJqEMAxZu8fxO5DUZUZYzumlqbz3cLt5YmwqvilfiGmrwUO+IjquK0fhivF+XSC6grfQesJ1xDm3floGsh8Ba8iPu0zr5Sb3OXAvZ1rnrlTyUyNE5ZZI+DFR2pXGee0s6z9hyVge7reLBCukfzvI8Sn4gqIpPwlP5TjCyYKuZhNTVwApqCH8U7mgOje9I8WPdj3YeokQSiPHVWrvFaVE6N390GP+zz/1fN5O6Fff7x1ZSWj47q9JjzA8aV4hZGb2SLYwqPEjqqKlthlf5qVxJ71o55Lc5vJ+Z3evI7ifmbPfmbifl3+vahY4SMHs0z+ftTdcwcqEI3lokrebhGDXaTb45/v499LQOzBeQBopN8YqdI3Z+5/tvyoLGcn85dn55ZnMnOzsQ36RBdpivQydENmqW7tEBFvGGb3tF7+pD+nP6e3kv/7KamBmLORep5BlO/Aa+vV4I=</latexit>b) Inference (constant ratio conversion)
Speaker ID (speaker S)
Speaker ID (speaker T)
Augment↵tgt Augment
coefficient
Conditional filtering
coefficient↵src = 1<latexit sha1_base64="RBzS9yUsEGjuuFJ24uAvdhRQFvo=">AAACb3ichVHLSsNAFD2Nr1pfVRcKghRLRTdlIr5wVXTjsq22CraUJE5raJqEZFqopT/gB+jChQ8QET/DjT/gwk8QV1LBjQtv04CoqHeYmTNn7rlzZka1Dd0VjD0GpK7unt6+YH9oYHBoeCQ8OpZ1raqj8YxmGZazqyouN3STZ4QuDL5rO1ypqAbfUcsb7f2dGndc3TK3Rd3m+YpSMvWirimCqFxOMewDpdAQJdEshKMszryI/ASyD6LwI2mFr5HDPixoqKICDhOCsAEFLrU9yGCwicujQZxDSPf2OZoIkbZKWZwyFGLLNJZoteezJq3bNV1PrdEpBnWHlBHE2AO7YS12z27ZE3v/tVbDq9H2UqdZ7Wi5XRg5mtx6+1dVoVng4FP1p2eBIlY9rzp5tz2mfQuto68dnrS21tKxxiy7ZM/k/4I9sju6gVl71a5SPH2KEH2A/P25f4LsQlxeji+lFqOJdf8rgpjCDObovVeQwCaSyNC5No5xhvPAizQhTUuRTqoU8DXj+BLS/AfQTI8A</latexit><latexit sha1_base64="GUkAPMbrkvjxYuyTt4AFbkLoLkE=">AAACcXichVHLSsNAFD2N7/qqulHclBZFEMpEfCEIRTcutdoq+CjJOK2DaRKStFBDf8AfUHClICJ+hht/wIWfIC4ruHHhTRoQFfUOM3PmzD13zszotiFdj7GnmNLW3tHZ1d0T7+3rHxhMDA0XXKvqcJHnlmE5O7rmCkOaIu9JzxA7tiO0im6Ibf14NdjfrgnHlZa55dVtsV/RyqYsSa55RB3saYZ9pBV91+GNZbWYSLMMCyP5E6gRSCOKdStxgz0cwgJHFRUImPAIG9DgUtuFCgabuH34xDmEZLgv0ECctFXKEpShEXtMY5lWuxFr0jqo6YZqTqcY1B1SJjHBHtkta7IHdsee2fuvtfywRuClTrPe0gq7OHg6uvn2r6pCs4ejT9Wfnj2UsBh6leTdDpngFrylr52cNTeXchP+JLtiL+T/kj2xe7qBWXvl1xsid4E4fYD6/bl/gsJMRp3PzG3MprMr0Vd0YxwpTNF7LyCLNawjT+c6OMclrmJNZUxJKqlWqhKLNCP4Esr0B/OKj3s=</latexit>

Decoder(+ attention)(+ post-net)

Pre-trainedneural vocoderc) Inference (vector swapping conversion)
Speaker ID (speaker S)

1. Introduction

Augment
coefficient↵src = 1<latexit sha1_base64="GUkAPMbrkvjxYuyTt4AFbkLoLkE=">AAACcXichVHLSsNAFD2N7/qqulHclBZFEMpEfCEIRTcutdoq+CjJOK2DaRKStFBDf8AfUHClICJ+hht/wIWfIC4ruHHhTRoQFfUOM3PmzD13zszotiFdj7GnmNLW3tHZ1d0T7+3rHxhMDA0XXKvqcJHnlmE5O7rmCkOaIu9JzxA7tiO0im6Ibf14NdjfrgnHlZa55dVtsV/RyqYsSa55RB3saYZ9pBV91+GNZbWYSLMMCyP5E6gRSCOKdStxgz0cwgJHFRUImPAIG9DgUtuFCgabuH34xDmEZLgv0ECctFXKEpShEXtMY5lWuxFr0jqo6YZqTqcY1B1SJjHBHtkta7IHdsee2fuvtfywRuClTrPe0gq7OHg6uvn2r6pCs4ejT9Wfnj2UsBh6leTdDpngFrylr52cNTeXchP+JLtiL+T/kj2xe7qBWXvl1xsid4E4fYD6/bl/gsJMRp3PzG3MprMr0Vd0YxwpTNF7LyCLNawjT+c6OMclrmJNZUxJKqlWqhKLNCP4Esr0B/OKj3s=</latexit>

From
speaker T

Conditional filtering

Encoder

Decoder(+ attention)(+ post-net)

From
speaker S

Pre-trainedneural vocoder

Converted
speechSourcespeechVoice conversion (VC) technology, which converts one speaking style to another without changing linguistic information, hasbeen applied for various tasks; speaker conversion [1,2], singing
conversion [3, 4], assistive systems [5, 6] to overcome speechand hearing impairments, and pronunciation and accent conversions [7] in language learning. Although the recent conversion quality has been improved thanks to sequence-to-sequence(S2S) learning approaches [2,4,6,8,9], the efﬁcient way for optimal control of high-level attributes such as speaking rate andfundamental frequency (F0 ) is still an active research area. In
this work, we focus on speaking rate controllability.
A well-known pure signal processing (SP) method formodifying speaking rate is a waveform similarity-based synchronous overlap addition (WSOLA) method [10]. WSOLA
performs temporal modiﬁcation of speech waveforms by usinga scale factor. For more ﬂexible control, [11] proposed a phonetic posterior-gram (PPG) based VC, which was the best VC
method in VC challenge 2020 [12]. The PPGs are extracted
from the speech in advance using an external automatic speech
recognition (ASR) system trained with a large amount of training data. Then, an inverter from the PPG to the Mel spectrogram is trained. In addition, to change the speaking rate, eachcontext duration per speaker is calculated in advance. The context duration is manually modiﬁed along with the pre-calculated
duration statistics during the inference (called duration adjustment). The converted speech sounds natural but requires an external model and duration statistics for each context.
To reduce tedious labeling tasks, a semi-supervised generative modeling method [13] for text-to-speech synthesis (TTS)
has been proposed. They introduce a variational autoencoder

Speaker ID (speaker T)

Converted
speechSourcespeechEncoder

Figure 1: System overview of the proposed method during thetraining (a) and the inference (b) and (c). The differences
between (b) and (c) are described in Sec. 2.2. In (c), theconverted speech has target speaker’s voice timbre but source
speaker’s speaking rate, for example. (c) is important to conﬁrm
whether the proposed method makes is possible to decompose
the speaker representation.

(VAE) based speaker encoder to the S2S-based TTS system. By
introducing the VAE, speech attributes such as speaking rate,
which are essential but rarely labeled, can be discovered andcontrolled. However, their framework is in the scope of semisupervised learning, requiring small amounts of labeled data as
training data to make reasonable inferences, and their application is TTS.
To avoid the labeling task altogether, an unsupervised
speech disentanglement method [14] has been proposed for the
VC task. They perform aspect-speciﬁc voice conversion by disentangling speech into content, rhythm, pitch, and voice timbre using multiple autoencoders (AEs) in an unsupervised manner. Each AE constrains the ﬂow of information on the speechcomponents to be disentangled using efﬁcient signal processing methods and resampling techniques. Unfortunately, data
modiﬁcation is necessary during the training and testing, andreference speech is also needed to estimate the rhythm corresponding to the speaking rate.
Inspired by [15], we propose a conditional ﬁltered voice

1 http://www.kecl.ntt.co.jp/people/tanaka.ko/
projects/cfvc/

2058

10.21437/Interspeech.2023-1193

(INPUT) 10-dimentional one-hot vector for Speaker ID

conversion (CFVC) model for controlling high-level attributes
such as speaking rate while preserving voice timbre. Unlike [15], we do not require annotation in advance to learn theattribute representation. The ﬁltering coefﬁcients are treatedas continuous values rather than discontinuous values. Furthermore, the conversion framework is within sequence-tosequence learning. Similar to [14], a data augmentation approach is adopted rather than data labeling. However, our approach makes it possible to represent and control the speaker’s
speaking rate as a vector by introducing a conditional ﬁltering
technique, unlike [14]. Our contributions are as follows:• Our model training requires only speech data and no annotation for speaking rate.
• Combining the conditional ﬁltering method with data augmentation makes it possible to learn the represention of eachspeaker’s speaking rate as a multiple-dimensional vector, nota scalar.
• It is possible to convert the speech into a voice with the speaking rate of speaker S and the voice timbre of speaker T by
manipulating the learned speaker vector, as shown in Fig. 1(c).
• We brieﬂy conﬁrmed that our model also allows training andcontrolling other high-level attributes such as F0 (see thebonus track in the supplemental material).

Linear with Tanh
Linear1024
x
3

Variable component

Linear1024
x3

Linear

10
x
1024
1024
x
253

Constant component(INPUT)
Augment
coefficient
3-dimentional vector for speaking rate
Concat.
(OUTPUT) 256-dimentional speaker vector

Figure 2: Conditional ﬁltering module for controlling the speaking rate. “Linear”, “Tanh” and “Concat.” indicate the linear
transformation layer, the hyperbolic tangent function, and thevector concatenation, respectively. Augment coefﬁcients αsrc
and αtgt are deﬁned at Sec. 2.1.

jection, followed by the hyperbolic tangent activation function
and three linear projections to get 3-dimensional variable components cv,src and cv,tgt changed by the data augmentation, 3dimensional constant components cc,src and cc,tgt not changedby the data augmentation, and 253-dimensional remaining components cr,src and cr,tgt . 3-dimensional vectors cs,src andcs,tgt for representing speaking rate are generated as follows:

2. Method
The system overview is shown in Fig. 1. We employ an S2Smodel architecture where the source speech parameters are used
as input, and the target speech parameters are generated as output. All we need for training is a parallel corpus of input andoutput paired speech.
As the speech parameters, we extract 80-dimensional logMel spectrogram features X src = [x1,src , · · · , xi,src ] andX tgt = [x1,tgt , · · · , xj,tgt ] over a range of 80-7600 Hz fromthe given source and target speech signals sampled at 16 kHz.
The requirements for Short-Time Fourier Transform are thesame as reported in [16]; a Hanning window, 64 ms frame
length, eight ms frameshift, and 1024-point Fast Fourier Transform. Since shorter sequences make it easier to train eachmodel, we shortened the length of the sequence by creating subframes containing two time frames.

cs,src = cv,src ∗ αsrc + cc,src ,cs,tgt = cv,tgt ∗ αtgt + cc,tgt .

(1)
(2)

Finally, a 256-dimensional speaker vectors csrc and ctgt conditioning on the encoder and decoder are obtained by concatenating the speaking rate vectors cs,src and cs,tgt and the remainings cr,src and cr,tgt .
At the test time, we can control the speaking rate of the converted speech X̂ crc by changing αtgt . Considering the conditional ﬁltering module makes it possible to represent the speaking rate as cs,tgt , we can try to generate the speech X̂ vsc withtarget speaker’s voice timbre but source speaker’s speaking rateby concatenating cs,src and cr,tgt . We call these tasks ’constantratio conversion’ and ’vector swapping conversion’ (see Fig. 1(b) and (c)).
In the preliminary experiment, we conﬁrmed that changing the speaking rate representation from 3-dimensional vector representation into a scalar representation resulted in performance degradation. Therefore, a multidimensional representation may help to model the high-level attributes. We also conﬁrmed that the constant components were necessary because
the vector swapping conversion failed by removing the constantcomponents.

2.1. Data Augmentation
This paper focuses on speaking rate control and augments
the training data with an efﬁcient signal processing method,WSOLA [10]. Time-domain speech waveforms Y src and Y tgt
are manipulated before the log-Mel spectrogram extraction. Using the scale factors αsrc and αtgt , Y src and Y tgt are stretched
to αsrc and αtgt times their original length. The scale factor of
1 indicates no data augmentation. Therefore, the extracted logMel spectrogram features X src and X tgt are already stretched.
We sampled αsrc and αtgt from the uniform distribution overa range of 0.8-1.25 and performed the data augmentation with
50% probability during the training. In this paper, the uniform distribution range was determined from the fastest andslowest speakers in the training data. To avoid misunderstanding, WSOLA requires a scale factor and does not require precalculation of speaking rate.

2.3. Encoder and Decoder
Inspired by [17], an input log-Mel spectrogram X src and asource speaker vector csrc are passed into an encoder network
fEnc is composed of three 1D non-causal convolutional layers
each containing 512 ﬁlters with the kernel size of 5, followed
by a batch normalization [18] layer and a rectiﬁed linear unit
(ReLU) activation. The output of the ﬁnal convolutional layer
is passed into a single bi-directional long short term memory(LSTM) layer containing 512 units to generate encoded features

2.2. Conditional Filtering
As shown in Fig. 2, input one-hot vectors osrc and otgt corresponding to speaker id are passed into a conditional ﬁltering module fCF consisting of a 1024-dimensional linear pro-2059

Table 1: Averaged speech section length [sec] over the training
dataset. Male speaker (upper) and female speaker (lower). The
lower the value the faster the speaking rate.

Z, as follows:
Z = fEnc (X src , csrc ).

(3)

A decoder network fDec is the same as reported in [17],except for changing the LSTM unit size from 1024 to 512. 2
and taking the target speaker vector ctgt as input. The decoder involving the attention mechanism and post-net predicts an attention matrix A, the output log-Mel spectrogramX̂ = [x̂1 , · · · , x̂j ], and its improved X̂ post from Z, X̃ =[x0 , x1,tgt , · · · , xj−1,tgt ], and ctgt , where x0 indicates a zero
vector, known as start token, as follows:X̂ post , X̂, A = fDec (Z, X̃, ctgt ).

Speaker
Length

(4)

mht
4.18

mmy
3.89

msh
3.93

mtk
4.54

Speaker
Length

fkn
4.42

fks
4.05

ftk
4.48

fym
4.11

myi
3.69

• Baseline: Converted speech by using a sequence-to-sequence
VC model, which is equivalent to the proposed method without the data augmentation and the conditional ﬁltering module.
• Proposed-swap: Converted speech X̂ vsc by using the speaking rate vector cs,src from the source speaker and the remaining cr,tgt from the target speaker in the vector swapping conversion using the proposed method.
• Proposed-0.6∼1.4: Converted speech X̂ crc when αtgt isvaried from 0.6 to 1.4 in the constant ratio conversion using
the proposed method.
As the objective evaluation metrics, we used Mel-cepstral
distortion (MCD) [dB], root mean square error of F0 (F0RMSE) [Hz], and character error rate (CER) [%]. We used
dynamic time warping to get the alignment between the converted sample and the reference sample. To calculate the MCD
and F0 RMSE, we extracted 1-24 order Mel-cepstrums and F0 s
from the raw speech and the converted speech synthesized by
the neural vocoder. Note that the range of F0 RMSE was 2224, and no difference was found, so it was omitted. The CER
was calculated by the Transformer-based ASR model trainedon the corpus of spontaneous Japanese (CSJ) [22], which wasprovided by ESPnet [23].
As the objective evaluation metrics to evaluate the speaking
rate controllability, we deﬁne duration factor (DF) and duration
ratio correlation coefﬁcient (DRCC). DF is proposed to evaluate
the controllability of speaking rate in constant ratio conversion.
In constant ratio conversion, the length of the converted speechshould vary in proportion to the given augment coefﬁcient αtgt .
For example, the length of the converted speech with αtgt =0.8 should be 0.8 times the length of the converted speech with
αtgt = 1.0. Therefore, we deﬁne the DF as follows:

Finally, we minimize the objective function LS2S to train theencoder fEnc and the decoder fDec :LS2S = ||X̂ − X tgt ||2 ,

mho
3.65

(5)

where ||X̂ post − X tgt ||2 and stop token loss [17] are also used
but omitted for brevity.
2.4. Neural Vocoder
In the proposed model, any neural vocoders by conditioning
on Mel spectrograms as input can be used. In the experiment,
HiFi-GAN [19] was used as the neural vocoder. We applied
V2 setting described in [19] except for the eight ms frameshiftcondition.

3. Experiments
The most important thing to prove here is that the proposed
method makes it possible to disentangle the speaker representation and allow the manipulation of each element of the speaker
vector. Therefore, objective experimental evaluations demonstrate that 1) the length of the vector swapping converted speechX̂ vsc using the speaking rate vector cs,src from the source
speaker and the remaining cr,tgt from the target speaker isequivalent to the length of the source speech X src . Then, subjective experimental evaluations demonstrate that 2) the voicetimbre of the converted speech X̂ vsc is similar to that of thetarget speech X tgt .
3.1. Experimental Conditions
We conducted experimental evaluations using a phoneticallybalanced Japanese speech parallel dataset [20] consisting of utterances by six professional male speakers and four professional
female speakers. The speech was recorded in a quiet room withminimal reverberation, and silent sections were removed using
annotation by experts. Therefore, ﬂuctuation of the length of the
speech length caused by ﬂuctuation of the length of the silence
section need not be considered in this experiment. To train VC
models, we used 450 sentences (speech section of around 0.5
hours) per speaker. Table 1 shows speech section length statistics. To evaluate the performance, we used 53 sentences per
speaker. The encoder-decoder models were trained on manyto-many condition, which is 10-speaker input and 10-speaker
output.
The number of training iterations is 100k. The learning
rate and the exponential decay rate for the ﬁrst moment for
Adam [21] were set at 0.0002 and 0.9 after 4k step of warmup.
The mini-batch size was 16. We evaluated the following converted speech.

N

DF =

length(converted speech with αtgt )1 ∑,N n=1 length(converted speech with αtgt = 1.0)
(6)where N is the number of the evaluated audio clips. In thisexperiment, N was 4770 (90 speaker pairs * 53 sentences). The
closer the DF value is to αtgt , the better the performance.
DRCC is also proposed to evaluate the controllability of
speaking rate in vector swapping conversion. First, we deﬁneduration ratio 1 (DR1) and DR2 as follows:
length(original speech X tgt ),length(original speech X src )length(converted speech from X src )DR2 =,length(reconstrcuted speech from X src )

DR1 =

(7)
(8)

If the conversion model is trained well, DR2 should be thesame value as DR1, which is the reference DR. As DRCC for”Baseline” and ”Proposed-1.0”, we calculated the correlation

2 We conﬁrmed no degradation in the preliminary objective experi-ments.

2060

Table 2: Objective evaluation results. The lower the MCD and
CER, the better the performance. The closer the DF and DRCC
are to α and 1, respectively, the better the performance. ⋆ indicates the extrapolation because α is sampled from a uniform
distribution over a range of 0.8-1.25 during the training.
SystemBaseline
Proposed-swap(Faster)↑
Proposed↓(Slower)

α
—
—
0.6⋆
0.7⋆
0.8
0.9
1.0
1.1
1.2
1.3⋆
1.4⋆

MCD

CER

DF

DRCC

5.16
5.20
4.44
4.81
5.08
5.18
5.16
5.33
5.44
5.49
5.51

14.7
14.6
16.1
15.5
15.0
14.8
14.6
14.8
15.0
15.3
15.9

—
—
0.70
0.75
0.81
0.90
—
1.10
1.20
1.31
1.44

0.99
0.95
—
—
—
—
0.98
—
—
—
—

Figure 3: Subjective evaluation results on sound quality (left)
and voice timbre similarity (right). Re-synthesized indicates the
speech synthesized from the ground-truth Mel-spectrogram.
On the other hand, the sound quality and the voice timbre similarity of ”Proposed-swap” were comparable to those of”Baseline,” as shown in Fig. 3. ”Proposed-swap” has the voicetimbre of the target speech even though a part of the speaker
vector was swapped. These results showed that the proposed
method allowed us to decompose the speaker representation
into a speaking rate component and other components.

coefﬁcient between DR1 and DR2 over 4770 audio clips. The
closer the DRCC value is to 1, the better the performance. For”Proposed-swap”, we used the following DR2 to calcurate the
DRCC.
DR2 =

length(converted speech X̂ from X src ), (9)length(converted speech X̂ vsc from X src )3.4. Other Discussions

If the proposed method works as intended, the speech length ofX̂ vsc should be that of the reconstructed speech from X src because of the converted speech X̂ vsc using the source speaker’s
speaking rate vector cs,src . Of course, if the proposed method
does not work well and the length of the converted speech issimilar to that of the target speech, the DRCC of ”Proposedswap” will be close to 0.0.
As the subjective evaluation of sound quality, we conducted
a 5-scaled mean opinion score (MOS): 5 for excellent, 4 forgood, 3 for fair, 2 for poor, and 1 for bad. To conﬁrm voicetimbre similarity, we also conducted a 4-scaled preference test
(PT): 4 for same (sure), 3 for same (not sure), 2 for different(not sure), and 1 for different (sure). 10 native Japanese speakers participated in each subjective evaluation. Each system wasevaluated over 275 times.

For ”Proposed-0.6∼0.8”, as shown in Table 2, the DFs behaved
as if there was an upper limit to how fast we could speak. For”Proposed-0.8∼1.2”, compared with ”Baseline,” the speaking
rate was successfully controlled without degrading CERs. For”Proposed-1.2∼1.4”, MCDs and CERs tended to be degraded,
as in the case of ”Proposed-0.6∼0.8”.
On the other hand, as shown in Fig. 3, the subjective evaluations showed that the sound quality and the voice timbre similarity of the proposed method were comparable to those of thebaseline method except for α = 1.2. A possible reason is thatslower speech may make the difference easier to understand, as
in language learning. In particular, the converted speech further
slowing down an initially slow speech may be out of the natural
speech space.

4. Conclusions

3.2. No Negative Impact of Data Augmentation and Conditional Filtering Modules

We proposed a many-to-many voice conversion using data augmentation and conditional ﬁltering to obtain a speaker representation decomposed into the speaking rate and other components without annotation, precomputation of statistics, or reference speech. Experimental results showed that the proposed
method disentangled complex attributes of the speaker and controlled speaking rate and voice timbre separately. In the future,
we plan to extend the proposed method to an any-to-many/any
voice conversion method.

The comparison of ”Baseline” and ”Proposed-1.0” showed that
there is no difference in objective and subjective evaluation results, as shown in Table 2 and Fig. 3. Therefore, we foundno problems with the addition of data augmentation and conditional ﬁltering modules.
3.3. Disentangled Representation
There was no difference between ”Baseline” and ”Proposedswap” for the objective evaluations, as shown in Table 2. Especially, the DRCC of ”Proposed-swap” was still higher than
0.9 despite a part of the speaker vector being swapped. The results showed that the length of the converted speech ”Proposedswap” is approximately the same as the length of the source
speech.

Acknowledgment
This work was supported by JST CREST Grant Number JPMJCR19A3, Japan.

2061

5. References[19] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efﬁcient and high ﬁdelity speech synthesis,” in
NeurIPS, 2020, pp. 17 022–17 033.

[1] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neuraldiscrete representation learning,” in NeurIPS, 2017, pp. 6309––6 318.

[20] A. Kurematsu, K. Takeda, Y. Sagisaka, S. Katagiri, H. Kuwabara,
and K. Shikano, “ATR Japanese speech database as a tool ofspeech recognition and synthesis,” Speech Communication, vol. 9,
no. 4, pp. 357–363, 1990.

[2] J. Zhang, Z. Ling, L. Liu, Y. Jiang, and L. Dai, “Sequence-tosequence acoustic modeling for voice conversion,” IEEE/ACM
Transactions on Audio, Speech, and Language Processing,vol. 27, no. 3, pp. 631–644, 2019.

[21] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.

[3] K. Kobayashi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura,“Statistical singing voice conversion with direct waveform modiﬁcation based on the spectrum differential,” in INTERSPEECH,
2014, pp. 2514–2518.

[22] K. Maekawa, “Corpus of spontaneous Japanese: Its design andevaluation,” in Workshop on Spontaneous Speech Processing andRecognition, 2003, pp. 7–12.

[4] R. Valle, J. Li, R. Prenger, and B. Catanzaro, “Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm,
pitch and global style tokens,” in ICASSP, 2020, pp. 6189–6193.

[23] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno,
N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen,
A. Renduchintala, and T. Ochiai, “ESPnet: End-to-end speechprocessing toolkit,” in INTERSPEECH, 2018, pp. 2207–2211.

[5] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Ahybrid approach to electrolaryngeal speech enhancement basedon noise reduction and statistical excitation generation,” IEICE
Transactions on Information and Systems, vol. 97, no. 6, pp.
1429–1437, 2014.
[6] F. Biadsy, R. J. Weiss, P. J. Moreno, D. Kanevsky, and Y. Jia, “Parrotron: An end-to-end speech-to-speech conversion model and its
applications to hearing-impaired speech and speech separation,”in INTERSPEECH, 2019, pp. 4115–4119.
[7] D. Felps, H. Bortfeld, and R. Gutierrez-Osuna, “Foreign accent
conversion in computer assisted pronunciation training,” SpeechCommunication, vol. 51, no. 10, pp. 920–932, 2009.
[8] H. Kameoka, W.-C. Huang, K. Tanaka, T. Kaneko, N. Hojo, andT. Toda, “Many-to-many voice transformer network,” IEEE/ACM
Transactions on Audio, Speech, and Language Processing,vol. 29, pp. 656–670, 2020.
[9] K. Tanaka, H. Kameoka, T. Kaneko, and S. Seki, “Distilling sequence-to-sequence voice conversion models for streaming
conversion applications,” in SLT, 2022 (to appear).
[10] W. Verhelst and M. Roelands, “An overlap-add technique based on
waveform similarity (WSOLA) for high quality time-scale modiﬁcation of speech,” in ICASSP, vol. 2, 1993, pp. 554–557.
[11] L.-J. Liu, Y.-N. Chen, J.-X. Zhang, Y. Jiang, Y.-J. Hu, Z.-H. Ling,
and L.-R. Dai, “Non-parallel voice conversion with autoregressive
conversion model and duration adjustment,” in Joint Workshop for
the Blizzard Challenge and Voice Conversion Challenge, 2020,
pp. 126–130.
[12] Z. Yi, W.-C. Huang, X. Tian, J. Yamagishi, R. K. Das, T. Kinnunen, Z.-H. Ling, and T. Toda, “Voice Conversion Challenge
2020 — Intra-lingual semi-parallel and cross-lingual voice conversion —,” in Joint Workshop for the Blizzard Challenge andVoice Conversion Challenge, 2020, pp. 80–98.
[13] R. Habib, S. Mariooryad, M. Shannon, E. Battenberg, R. SkerryRyan, D. Stanton, D. Kao, and T. Bagby, “Semi-supervised generative modeling for controllable speech synthesis,” in ICLR, 2020.
[14] C. H. Chan, K. Qian, Y. Zhang, and M. Hasegawa-Johnson,“SpeechSplit2.0: Unsupervised speech disentanglement for voice
conversion without tuning autoencoder bottlenecks,” in ICASSP,
2022, pp. 6332–6336.
[15] T. Kaneko, K. Hiramatsu, and K. Kashino, “Generative attributecontroller with conditional ﬁltered generative adversarial networks,” in CVPR, 2017, pp. 6089–6098.
[16] H. Kameoka, K. Tanaka, and T. Kaneko, “FastS2S-VC: Streaming non-autoregressive sequence-to-sequence voice conversion,”
arXiv preprint arXiv:2104.06900, 2021.
[17] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang,
Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan et al., “Natural TTS
synthesis by conditioning WaveNet on mel spectrogram predictions,” in ICASSP, 2018, pp. 4779–4783.
[18] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” in ICML,
2015, pp. 448–456.

2062