DISTILLING SEQUENCE-TO-SEQUENCE VOICE CONVERSION MODELSFOR STREAMING CONVERSION APPLICATIONSKou Tanaka, Hirokazu Kameoka, Takuhiro Kaneko, Shogo Seki
NTT Communication Science Laboratories, NTT Corporation, Japan

2022 IEEE Spoken Language Technology Workshop (SLT) | 979-8-3503-9690-4/23/$31.00 ©2023 IEEE | DOI: 10.1109/SLT54892.2023.10023432

ABSTRACT."#/,-0,1*2,&0'-,(This paper describes a method for distilling a recurrentbased sequence-to-sequence (S2S) voice conversion (VC)
model. Although the performance of recent VCs is becoming higher quality, streaming conversion is still a challenge
when considering practical applications. To achieve streaming VC, the conversion model needs a streamable structure, a causal layer rather than a non-causal layer. Motivated by this constraint and recent advances in S2S learning,
we apply the teacher-student framework to recurrent-based
S2S-VC models. A major challenge is how to minimize
degradation due to the use of causal layers which masks
future input information. Experimental evaluations showthat except for male-to-female speaker conversion, our approach is able to maintain the teacher model’s performancein terms of subjective evaluations despite the streamable student model structure. Audio samples can be accessed on
http://www.kecl.ntt.co.jp/people/tanaka.
ko/projects/dists2svc.
Index Terms— Voice conversion, sequence-to-sequencelearning, distillation, streaming conversion.
1. INTRODUCTION
Voice conversion (VC) technology, which converts one speaking style to another without changing linguistic information,
has been applied for various tasks; speaker conversion [1,2], singing conversion [3, 4], assistive systems [5, 6] to overcome speech and hearing impairments, and pronunciation andaccent conversions [7] in language learning. Recently, theconversion quality has been improved thanks to sequenceto-sequence (S2S) learning approaches [2, 4, 6, 8–10]. An
encoder-decoder structure, including an attention mechanism,makes it possible to learn conversion rules that reflect thelong-term dependencies of input and output sequences. However, the conversion process is executable after we obtain theentire sentence because the encoder contains non-causal layers, as shown in Fig. 1. Considering practical use, waiting
for the end of a speech is a major barrier against the smoothspeech communication in daily life. Unlike other speech signal processing, such as text-to-speech synthesis, the effect ofdelayed auditory feedback [11], a well-known problem leadc
978-1-6654-7189-3/22/$31.00 ⃝2023
IEEE!"#$%&'(&)*)&+,-(.&'(&)*)&+,-(

3#4'0*2,&0'-,(Fig. 1. Non-causal (left) and causal (right) layers. The red
and pink boxes show the segments of the input feature andthe frames of the converted feature. The causal layers do nottake future frames as input, contrary to the non-causal layers. Real-time conversion of streaming speech data increases
latency by the amount of look-ahead.

ing to unnatural speech, must also be considered since theconverted speech is fed back to the auditory system. The VC
task requires handling streaming processing with low latency,such as several dozen ms.
In the zero/minimal look-ahead scenario within the S2Slearning, achieving low latency streaming processing witha recurrent-based approach appears possible rather thanconvolution/transformer-based [12, 13]. A recurrent neuralnetwork (RNN) can hold time-series information internallyin a layer, while a convolution/transformer-based model requires a deep architecture to capture such information. Unfortunately, in the preliminary experiments, we confirmed thatjust replacing non-causal layers of the encoder with causal
layers would result in converted speech with significant delay, as shown in Fig. 2. Note that this phenomenon has beenreported in a speech recognition task [14]. Trying to train amodel from scratch without any guides, uni-directional RNN
may have been trained to generate the current output after
considering inputs a little further ahead to capture the timefrequency structure accurately. During the model training, thetime delay is offset by the attention mechanism. However,in the test time on real-time streaming VC, this time delay is1022

Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:37:10 UTC from IEEE Xplore. Restrictions apply.

!"##$%&!"#$%#$&#'()'"*#+,#-'./!$#%&'()*+,-.*/!"#!"##$%&!"#$%#$&#'0)'"*#+,#-'1/'(#)*(+,-#.
/"#+0#(&#-$1.#(!"##$%"#$%#$&#'2"*#+,#-'.

2-$1.#(

3#$1.#(
45&+**#-*,1-6
45&"1/*)-#*6:1-9#(*#.&/"##$%'(#)*(+,-#.
-#7(+8&91$1.#(

0*'()*+,-.*/

Fig. 3. System overview of teacher and student models.
Fig. 2. Attention matrices in reconstruction scenarios; target speech is the same as source speech. a) is generated by
the standard encoder-decoder model, and b) is generated by
the encoder-decoder model replacing non-causal layers withcausal layers. Attention matrix b) is automatically shifted byaround 300 ms. This shifted attention directly increases thelatency of real-time streaming VC.
not canceled and increases the latency because we force theattention matrix to be diagonal.
Motivated by the real-time streaming VC, the recent
advanced VC, and our preliminary experiments, we apply
knowledge distillation approaches, known as teacher-student
frameworks, to a recurrent-based S2S-VC model as the firststep. A major challenge is minimizing degradation due tointroducing the causal structure instead of the non-causal
structure, namely degradation caused by masking future input information. Therefore, we investigate several knowledge
distillation approaches mentioned in Sec. 2.3. One is to copy
some model parameters from the teacher model to the student
model and fix them during training. The second is an attention distillation using Kullback-Leibler divergence (KLD).
The last is to share some model parameters between teacher
and student models and update them during training, known
as joint training. Experimental evaluations show that except
for male-to-female speaker conversion, the first approach isable to maintain the teacher model’s performance in terms ofsubjective evaluations despite the streamable model structure.
2. STRUCTURE AND OBJECTIVE FUNCTION
The system overview is shown in Fig. 3. We employ a S2Smodel architecture where the source speech parameters areused as input and the target speech parameters are generated
as output. All we need for training is a parallel corpus of input
and output paired speech. We introduce pre-trained speaker
encoder [15] to control output speaker styles and pre-trainedneural vocoder [16] to generate speech waveform from speechparameters.
As the speech parameters, we extract 80-dimensionallog-Mel spectrogram features X = [x1 , · · · , xi ] and Y =[y 1 , · · · , y j ] over a range of 80-7600 Hz from the given
source and target speech signals sampled at 16 kHz. The
requirements for Short-Time Fourier Transform are the sameas reported in [12]; a Hanning window, 64 ms frame length,
eight ms frameshift, and 1024-point Fast Fourier Transform.
Since shorter sequences make it easier to train each model, we
shortened the length of the sequence by creating subframes
containing two time-frames.
2.1. Speaker Encoder [15]The network consists three LSTM layers with 768 unis anda fully connected layer with a 256-dimensional linear projection to get a 256-dimensional speaker vector c from an input
log-Mel spectrogram. To train the speaker encoder, we used
Softmax criterion reported in [15] and a speech database containing over 100k files uttered by thousands of speaker. Other
training condition, such as mini-batch size and parameter initialization, is the same as in [15]. The resulting Top-1 accuracies are 99.98 % and 99.65 % for training data and evaluation
data. During S2S model training, to avoid leaking context
information to the S2S modules, the speaker vector c is extracted from different sentences from the source and target
speech of the S2S model.
2.2. Teacher Model
Inspired by [17], an input log-Mel spectrogram X and asource speaker vector cx is passed into an encoder network
fEnc is composed of three 1D non-causal convolutional layers
each containing 512 filters with the kernel size of 5, followed
by a batch normalization [18] layer and a rectified linear unit
(ReLU) activation. The output of the final convolutional layer
is passed into a single bi-directional long short term memory (LSTM) layer containing 512 units to generate encodedfeatures Z x , as follows:
Z x = fEnc (X, cx ).

(1)

A decoder network fDec is the same as reported in [17],except for changing the LSTM unit size from 1024 to 512 1
and taking the target speaker vector cy as input. The decoder
involving the attention mechanism fAtt and post-net fPost
predicts an attention matrix A, the output log-Mel spectrogram Ŷ = [ŷ 1 , · · · , ŷ j ], and its improved Ŷ post from Z x ,1 We confirmed no degradation in the preliminary objective experiments.

1023
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:37:10 UTC from IEEE Xplore. Restrictions apply.

Table 1. A list of investigated approaches. Post-net fPost is omitted for brevity.
Attention
SystemEncoder Decoder Fix
Skip Other explanation
mechanism
Baseline (related with Fig. 2 a) fEnc
fDec
fAtt
LTeacherPreliminary experiment
fDistEnc fDistDec
fDistAtt
LStudent(related with Fig. 2 b)⋆
Approach-1
fDistEnc fDec✓✓
LStudent + copied attention matrix⋆
Approach-2
fDistEnc fDec✓✓
LStudent + self-supervised learning⋆
Approach-3
fDistEnc fDistDec✓
LStudent + copied attention matrix
Approach-4
fDistEnc fDistDec
fDistAtt
LStudent + LAttKLD
fDistEnc fDistDec
fDistAtt
LStudent + LJnt
Approach-5 (excluded)Ỹ = [y 0 , y 1 , · · · , y j−1 ], and cx , where y 0 indicates a zero
vector, known as start token, as follows:Ŷ post , Ŷ , A = fDec (Z x , Ỹ , cy ).

(2)where ⋆ indicates skipping the attention mechanism.
On the other hand, Approach-2 uses the target speech
Y as the source speech, so it is self-supervised learning, likean auto-encoder, as follows:

Finally, we minimize the objective function LTeacher to train
the encoder fEnc and the decoder fDec :LTeacher = ||Ŷ − Y ||2 ,2.3. Student Model
The structure of an encoder network fDistEnc is the same asthat of the teacher encoder fEnc , except for streamable structure replacing the non-causal convolutions and bi-directional
LSTM with the causal convolutions and uni-directional
LSTM. The structure of a decoder network fDistDec is thesame as that of the teacher decoder fDec . A list of investigated approaches is shown in Table 1.
2.3.1. Approach-1 & -2: use a teacher decoder as a student
decoderIn Approach-1 and -2, only the student encoder fDistEnc is
trained, because the teacher decoder is already streamable.
In these approaches, we initialize the model parameters of
the student decoder by those of the teacher decoder and fix
them during training. To do so, we have two training schemes.
Approach-1 uses the source and target parallel speech dataset
for the student encoder training. To handle different lengthsof speech sequence, we skip the attention mechanism fAtt butbring the attention matrix A generated by the teacher model
as follows:⋆
Y ′post , Y ′ = fDec
(Z ′x A, Ỹ , cy ),LStudent = ||Y ′ − Y ||2 ,(7)⋆(Z ′y , Ỹ , cy ).
Y ′post , Y ′ = fDec

(8)

(3)

where ||Ŷ post − Y ||2 and stop token loss [17] are also used
but omitted for brevity of the latter section.

Z ′x = fDistEnc (X, cx ),Z ′y = fDistEnc (Y , cy ),

(4)
(5)
(6)

It is assumed that if the teacher model is successfully trained,
the input of the decoder will be disentangled features; the context information Z ′y extracted by the encoder and the speakerstyle cy extracted by the speaker encoder. Therefore, even if it
is self-supervised learning, we expect the student’s encoder tobe trained as a contextual information extractor that removes
speaking style rather than simple compression.
2.3.2. Approach-3 & -4: train a student decoder
We also investigate schemes in which the student encoder
fDistEnc and decoder fDistDec are trained. As mentioned inSec. 1, we confirmed in the preliminary experiment that an
approach of replacing the non-causal encoder with a causal
encoder and training all parameters resulted in the shifted attention matrices. Therefore, Approach-3 and -4 take the attention matrix generated by the teacher model as a guide to train
the student model. In Approach-3, the given attention matrix
is used for changing the input sequence length as follows:′⋆
Y post, Y ′ = fDistDec(Z ′x A, Ỹ , cy ).

(9)On the other hand, Approach-4 aims to train the attention mechanism fDistAtt , which is contained in the student
decoder fDistDec . Each column of the attention matrix gives
a probability distribution that describes the relationship between the output of the decoder at each step and the source
feature sequence Z x . To guide the attention mechanism training, we introduce KLD between the attention matrix A generated by the teacher and that A′ generated by the student
model. This approach is known as attention distillation. The

1024
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:37:10 UTC from IEEE Xplore. Restrictions apply.

KLD loss LAttKLD is the following;
Y ′post , Y ′ , A′ = fDistDec (Z ′x , Ỹ , cy ),∑LAttKLD =DKL (aj ||a′j ),jTable 2. Speech section length [sec] over the training dataset.
(10)

Speaker
mht
Sourcefym
mshTargetftk(11)

where aj and a′j denote j-th column of the attention matrices
A and A′ .
2.3.3. Approach-5: jointly train non-causal and causal encoders
In Approach-5, we do not use any guides or model parameters from the teacher model but train the non-causal encoder
fJntEnc jointly with the causal encoder. Therefore, in addition
to Eq. (4) and (10), we introduce the following formulation:Z̈ x = fJntEnc (X, cx ),(12)Ÿ post , Ÿ , Ä = fDistDec (Z̈ x , Ỹ , cy ),(13)

LJnt = ||Ÿ − Y ||2 ,(14)

where the structure of fJntEnc is the same as that of theteacher encoder fEnc . From another point of view, we shareand update the decoder between the teacher and student models.
Unfortunately, we confirmed the phenomenon of attention
shifting, similar to Fig. 2b. One possible reason is that the
non-causal encoders were unintentionally trained to match thecharacteristics of the causal encoders, as opposed to knowledge distillation from the non-causal encoders to the causal
encoder, which is our aim. In order to achieve low latency
streaming VC, Approach-5 was excluded from the experimental evaluations.
2.4. Neural Vocoder
In the proposed model, any neural vocoders by conditioning
on Mel spectrograms as input can be used. In the experiment,
HiFi-GAN [16] is used as the neural vocoder. We confirmedthat V2 setting described in [16] is able to generate waveforms faster than real-time for streaming applications underthe eight ms frameshift condition.
3. EXPERIMENTS
3.1. Experimental Conditions
To demonstrate the performance of each method, we conducted experimental evaluations using a phonetically balanced Japanese speech parallel dataset [19] consisting of utterances by two professional male speakers (mht and msh) andtwo professional female speakers (fym and ftk). The speeches
were recorded in a quiet room with minimal reverberation.
To train each VC model, we used 450 sentences (speech
section of around 0.5 hours) of each speaker. Statistics of

Avg.
4.18
4.11
3.93
4.48

Min.
1.48
1.47
1.42
1.45

Max.
8.53
8.31
7.66
8.74

speech section length are shown in Table 2. To evaluate theperformance, we used 53 sentences from each speaker. For
the evaluations, we conducted intra-gender pairs, mht-msh
and fym-ftk, and cross-gender pairs, mht-ftk and fym-msh.
All of the encoder-decoder are trained 100k iterations.
The learning rate and the exponential decay rate for the first
moment for Adam [20] were set at 0.0002 and 0.9 after 4k
step of warmup. The mini-batch size was 16. All of encoderdecoder models were trained on many-to-many condition,
which is four-speaker input and four-speaker output. During test inference time, we forced the attention matrix to bediagonal because real-time streaming VC does not allow tochange the speaking rate.
3.2. Objective Evaluation
As the objective evaluation metrics, we used Mel-cepstral distortion (MCD) [dB], root mean square error of F0 (F0 RMSE)[Hz], and character error rate (CER) [%]. We used dynamic
time warping to get the alignment between the converted sample and the reference sample. To calculate the MCD, we calculated 1-24 order Mel-cepstrum. For VC model evaluation,
we extracted Mel-cepstrum and F0 from the converted speech
waveform synthesized by the neural vocoder because the VC
model generates Mel-spectrogram. The CER was calculatedby the Transformer-based ASR model trained on the corpus of
spontaneous Japanese (CSJ) [21], which was provided by ESPnet [22]. MCD and F0 RMSE reflect speaker, prosody, andphonetic content similarities, and CER represents the intelligibility and correlates to naturalness [23]. Note that MCD and
F0 RMSE are less sensitive to speech discontinuity than CER
because MCD and F0 RMSE are calculated frame-by-frame
while the CER is calculated by ASR model that takes the sequential information into account. The performances of neural vocoder, MCD, F0 RMSE, and CER of the re-synthesized
speech given the ground-truth Mel-spectrogram, were 2.99,
19.50, and 10.6, respectively.
The objective evaluation results are shown in Table 3.
First, we focus on the results of Baseline. The result showsthat the Baseline’s performance depends on who the target
speaker is rather than the gender pairs. The conversion into
the speaker ftk seems to be more difficult than the conversion
into the speaker msh.
Compared to Baseline, the performance of Approach-1,1025
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:37:10 UTC from IEEE Xplore. Restrictions apply.

Table 3. Objective evaluation results on intra-gender (upper) and cross-gender (lower) pairs. The lower the value the better the
performance.

Baseline
Approach-1
Approach-2
Approach-3
Approach-4
SystemBaseline
Approach-1
Approach-2
Approach-3
Approach-4

MCD
5.11 ± 0.08
5.61 ± 0.08
5.51 ± 0.08
5.44 ± 0.08
5.73 ± 0.09

mht-msh
F0 RMSE
18.66 ± 1.18
21.31 ± 1.40
19.09 ± 1.31
21.10 ± 1.43
25.47 ± 1.48

CER
15.8
22.1
17.0
18.9
29.6

MCD
6.01 ± 0.08
6.39 ± 0.09
6.19 ± 0.08
6.43 ± 0.08
6.55 ± 0.09

fym-ftk
F0 RMSE27.49 ± 1.88
31.90 ± 2.53
26.32 ± 1.82
34.57 ± 2.54
39.90 ± 2.89

CER
15.1
19.8
14.2
16.8
26.1

MCD
5.95 ± 0.09
6.38 ± 0.09
6.34 ± 0.06
6.37 ± 0.08
6.55 ± 0.08

mht-ftk
F0 RMSE28.49 ± 1.58
34.87 ± 2.66
28.43 ± 1.56
36.13 ± 2.54
41.01 ± 2.94

CER
15.0
21.2
15.0
16.4
32.9

MCD
5.13 ± 0.08
5.66 ± 0.08
5.82 ± 0.08
5.55 ± 0.09
5.79 ± 0.08

fym-msh
F0 RMSE18.30 ± 1.27
21.23 ± 1.55
22.08 ± 1.74
20.08 ± 1.41
24.14 ± 1.44

CER
16.7
19.6
15.7
17.5
26.0

-3, and -4 was degraded. However, the F0 RMSE and CERof Approach-2 were comparable to the Baseline. MCD of
Approach-2 was also better than Approach-1, -3, and -4,
except for mht-ftk pair. This result indicates that to distill
the S2S-VC model, a self-supervised learning with a pretrained decoder works better than the training approach using
parallel data and attention matrices. One possible reason isthat since the Baseline architecture includes the bi-directional
RNN containing the backward RNN, strange attention shifting, which is acceptable for bi-directional RNN but not foruni-directional RNN, may occur. Moreover, although theparallel data is recorded carefully, it is very challenging to be
exactly parallel in several terms such as pause position andlength, pronunciation error, and different intonation. Even
if the attention mechanism makes it possible to reduce thedegradation caused by the recording mistake, self-supervised
learning, which is under the perfect parallel data condition,
seems to be better. However, it is essential to note that the
self-supervised learning approach has a gap during training
and inference processes because training data never see thespeaker conversion setting, in which the input speaker isdifferent from the target speaker. In the preliminary experiments, we also trained a simple auto-encoder. The result isthe reconstruction of the input speech, and speaker conversion
is a failure 2 . In the next section, we will confirm whether
fixing the decoder achieves the speaker conversion.

2 In AutoVC [24], the authors have reported that the training of simple
auto-encoder to convert the speaker characteristics is very sensitive to thedata setting and model architecture.

5

Re-synthsized
Baseline

Approach-1
Approach-2

Approach-3
Approach-4

Mean opinion scores

System4
3
2
1

mht-mshfym-ftk
Intra-gender

mht-ftkfym-mshCross-genderFig. 4. Subjective evaluation results on sound quality. The
higher the value the better the sound quality. Re-synthesizedindicates the speech synthesized from the ground-truth Melspectrogram.

3.3. Subjective Evaluation
Next, we conducted subjective evaluation tests on sound quality and speaker similarity to check the perceptual quality. For
sound quality, each subject listened to each sample and rated
the sound quality on a 5-point scale: 5 for excellent, 4 forgood, 3 for fair, 2 for poor, and 1 for bad. For speaker similarity, each subject listened to pairs of the target sample andthe converted sample to judge whether the presented samples
were produced by the same speaker with confidence on a 4point scale: 4 for same (sure), 3 for same (not sure), 2 for different (not sure), and 1 for different (sure). 10 native Japanese
speakers participated in each subjective evaluation. Each system was evaluated over 250 times.

1026
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:37:10 UTC from IEEE Xplore. Restrictions apply.

Same (sure)8060

5. ACKNOWLEDGMENTS

Intra-gender

100

mht-mshfym-ftk

Cross-gender

mht-ftkfym-msh40

Baseline
Approach-1
Approach-2
Approach-3
Approach-4

6. REFERENCES

Baseline
Approach-1
Approach-2
Approach-3
Approach-40Baseline
Approach-1
Approach-2
Approach-3
Approach-420This work was supported by JST CREST Grant Number JPMJCR19A3, Japan.
Baseline
Approach-1
Approach-2
Approach-3
Approach-4

Percentage [%]Different (not sure)
Different (sure)self-supervised learning with a fixed decoder can maintainthe teacher model’s performance in terms of subjective evaluations despite the streamable student model structure, except
for male-to-female speaker conversion. In future work, wewill work on reducing the impact of male-to-female speaker
conversion in the self-supervised learning.

Same (not sure)[1] Aaron Van Den Oord, Oriol Vinyals, et al., “Neural discrete representation learning,” Advances in neural information processing systems, vol. 30, 2017.

Fig. 5. Subjective evaluation results on speaker similarity.
The higher the rate of Same the better the performance.
The subjective evaluation results on sound quality andspeaker similarity are shown in Fig. 4 and 5. First, we focus
on the results of Baseline. Although the MCD, F0 RMSE, and
CER of mht-ftk pair were similar to those of fym-ftk pair, thesubjective evaluation results of mht-ftk pair were the worst.
Given that the auto-regressive model also models something
like phoneme duration, a significant change in the speaking
rate may affect conversion performance. However, the speaking rate of the speaker mht was similar to the speaking rate of
the speaker fym, as shown in Table 2. This result showed thatthe Baseline’s performance depended on whether it was maleto-female speaker conversion or not, rather than the speaking
rate changes.
Compared to Baseline, the performance of Approach-1,-3, and -4 was degraded, similar to the objective evaluation
results. However, the sound quality of Approach-2 overcamethat of Baseline, thanks to self-supervised learning, which
was under the perfect parallel data condition. On the other
hand, the speaker similarity of the mht-ftk pair was degraded.
Note that there was no significant difference on the speakersimilarity of the fym-ftk pair. This result indicated that the
self-supervised learning approach might be more likely to beaffected than the S2S learning approaches required the paired
speech data. To tackle this limitation, we will work on it in
future work.
4. CONCLUSIONS
Toward real-time streaming VC applications, this paper describes a method for distilling a recurrent-based S2S-VC
model. The experimental results revealed that using a decoder, which is trained in advance by using parallel speechdata, is a good constraint for the encoder training. Moreover,[2] J. Zhang, Z. Ling, L. Liu, Y. Jiang, and L. Dai,
“Sequence-to-sequence acoustic modeling for voice
conversion,”
IEEE/ACM Transactions on Audio,
Speech, and Language Processing, vol. 27, no. 3, pp.
631–644, 2019.
[3] Kazuhiro Kobayashi, Tomoki Toda, Graham Neubig,
Sakriani Sakti, and Satoshi Nakamura, “Statisticalsinging voice conversion with direct waveform modification based on the spectrum differential,” in INTERSPEECH, 2014, pp. 2514–2518.
[4] Rafael Valle, Jason Li, Ryan Prenger, and Bryan Catanzaro, “Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global styletokens,” in ICASSP, 2020, pp. 6189–6193.
[5] Kou Tanaka, Tomoki Toda, Graham Neubig, Sakriani
Sakti, and Satoshi Nakamura, “A hybrid approach toelectrolaryngeal speech enhancement based on noise reduction and statistical excitation generation,” IEICE
Transactions on Information and Systems, vol. 97, no.
6, pp. 1429–1437, 2014.
[6] Fadi Biadsy, Ron J Weiss, Pedro J Moreno, Dimitri
Kanevsky, and Ye Jia, “Parrotron: An end-to-end
speech-to-speech conversion model and its applications
to hearing-impaired speech and speech separation,” in
INTERSPEECH, 2019, pp. 4115–4119.
[7] Daniel Felps, Heather Bortfeld, and Ricardo GutierrezOsuna, “Foreign accent conversion in computer assisted
pronunciation training,” Speech Communication, vol.
51, no. 10, pp. 920–932, 2009.
[8] Kou Tanaka, Hirokazu Kameoka, Takuhiro Kaneko, andNobukatsu Hojo, “AttS2S-VC: Sequence-to-sequence
voice conversion with attention and context preservation
mechanisms,” in ICASSP, 2019, pp. 6805–6809.

1027
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:37:10 UTC from IEEE Xplore. Restrictions apply.

[9] Hirokazu Kameoka, Kou Tanaka, Damian Kwaśny,
Takuhiro Kaneko, and Nobukatsu Hojo, “ConvS2S-VC:
Fully convolutional sequence-to-sequence voice conversion,” IEEE/ACM Transactions on Audio, Speech, andLanguage Processing, vol. 28, pp. 1849–1863, 2020.
[10] Hirokazu Kameoka, Wen-Chin Huang, Kou Tanaka,
Takuhiro Kaneko, Nobukatsu Hojo, and Tomoki
Toda, “Many-to-many voice transformer network,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 656–670, 2020.
[11] Aubrey J Yates, “Delayed auditory feedback,” Psychological bulletin, vol. 60, no. 3, pp. 213, 1963.
[12] Hirokazu Kameoka, Kou Tanaka, and Takuhiro Kaneko,“FastS2S-VC: Streaming non-autoregressive sequenceto-sequence voice conversion,”
arXiv preprint
arXiv:2104.06900, 2021.
[13] Xie Chen, Yu Wu, Zhenghao Wang, Shujie Liu, andJinyu Li,
“Developing real-time streaming transformer transducer for speech recognition on large-scale
dataset,” in ICASSP, 2021, pp. 5904–5908.

[21] Kikuo Maekawa, “Corpus of spontaneous japanese: Its
design and evaluation,” in Workshop on Spontaneous
Speech Processing and Recognition, 2003.
[22] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki
Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique
Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin
Chen, Adithya Renduchintala, and Tsubasa Ochiai,“ESPnet: End-to-end speech processing toolkit,” in INTERSPEECH, 2018, pp. 2207–2211.
[23] Rohan Kumar Das, Tomi Kinnunen, Wen-Chin Huang,
Zhenhua Ling, Junichi Yamagishi, Yi Zhao, Xiaohai
Tian, and Tomoki Toda, “Predictions of subjective ratings and spoofing assessments of voice conversion challenge 2020 submissions,” in Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge, 2020,
pp. 99–120.
[24] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang,
and Mark Hasegawa-Johnson, “AutoVC: Zero-shot
voice style transfer with only autoencoder loss,” in
ICML, 2019, pp. 5210–5219.

[14] Gakuto Kurata and George Saon, “Knowledge distillation from offline to streaming RNN transducer for endto-end speech recognition,” in INTERSPEECH, 2020,
pp. 2117–2121.
[15] Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez
Moreno, “Generalized end-to-end loss for speaker verification,” in ICASSP, 2018, pp. 4879–4883.
[16] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “Hifigan: Generative adversarial networks for efficient andhigh fidelity speech synthesis,” Advances in Neural
Information Processing Systems, vol. 33, pp. 17022–
17033, 2020.
[17] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike
Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng
Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.,“Natural TTS synthesis by conditioning wavenet on mel
spectrogram predictions,” in ICASSP, 2018, pp. 4779–
4783.
[18] Sergey Ioffe and Christian Szegedy, “Batch normalization: Accelerating deep network training by reducing
internal covariate shift,” in ICML, 2015, pp. 448–456.
[19] Akira Kurematsu, Kazuya Takeda, Yoshinori Sagisaka, Shigeru Katagiri, Hisao Kuwabara, and Kiyohiro
Shikano, “ATR japanese speech database as a tool ofspeech recognition and synthesis,” Speech Communication, vol. 9, no. 4, pp. 357–363, 1990.
[20] Diederik Kingma and Jimmy Ba, “Adam: A method forstochastic optimization,” in ICLR, 2015.

1028
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:37:10 UTC from IEEE Xplore. Restrictions apply.