Non-parallel Accent Transfer based on Fine-grained Controllable Accent Modelling
Authors: linqin wang
Zhengtao Yu
Yuanzhang Yang
Shengxiang Gao
Cunli Mao
Yuxin Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: Accent Transfer，Fine-grained Controllable Accent Modelling，Non-parallel
TL;DR: The accents are modelled at a fine-grained level in terms of tone and rhythm. It also uses mutual information learning to disentangle the accent features of the speaker information and control the accent of the generated speech

Abstract: Existing accent transfer works rely on parallel data or speech recognition models. This paper focuses on the practical application of accent transfer and aims to implement accent transfer using non-parallel datasets. The study has encountered the challenge of speech representation disentanglement and modeling accents. In our accent modeling transfer framework, we manage to solve these problems by two proposed methods. First, we learn the suprasegmental information associated with tone to finely model the accents in terms of tone and rhythm. Second, we propose to use mutual information learning to disentangle the accent features and control the accent of the generated speech during the inference time. Experiments show that the proposed framework attains superior performance to the baseline models in terms of accentedness and audio quality.

Non-parallel Accent Transfer based on Fine-grained Controllable Accent Modeling
Linqin Wangh Zhengtao Yu!.² Yuanzhang Yang!2
Shengxiang Gaot2 Cunli Mao Yuxin Huang!2
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China Yunnan Key Laboratory of Artificial Intelligence, Kunming, China
(1 m 63. com, {ztyu, t com,
(maocunli, Jiaoounli,,y2845535111 yyz845935161, huangyuxin2204)0165555 suaommmii_yyessssssouuuuuuuuaaa
com
Abstract
multiple speakers with multiple accents using in- consistent souree and target speech content (Wang ct al., 2021; Zhao et al., 2018, 2019; Kaneko and Kameoka, 2017; Kaneko et al., 2019, 2020a, 2021; Finkelstein et al., 2022) used multi-stage trained tts model to achieve transfer of North American accents, Australian accents, and British accents, and used CHiVE-BERT pre-training model enhance the audio effect of accent generation. Liu et al. (2022) added an accent variance adaptor model the rhythmicity of accent variance, and also enhanced the accent generation audio by using a consistency constraint module. The use of pho- netic posteriorgrams (PPG) is an essential idea in the application of non-parallel data (Wang et al., 2021; Zhao et al., 2018, 2019). Wang et al. (2021) extracted PPG from Chinese pre-trained speech recognition model and then used them in an end-to- end speech conversion model based on adversarial lcarning disentangling. This approach achieved ac- cent transfer from Chinese Mandarin to Tianjin and obtained decent results.
Existing accent transfer works rely on paral- lel data or speech recognition models. This paper focuses the practical application of accent transfer and aims to implement accent transfer using non-parallel datasets. The study has encountered the challenge of speech rep- resentation disentanglement and modeling ac- cents. In our accent modeling transfer frame- work, we manage to solve these problems by two proposed methods. First, we learn the suprasegmental information associated with tone to finely model the accents in terms of tone and rhythm. Second, we propose to use mutual information learning disentangle the accent features and control the accent of the generated speech during the inference time. Experiments show that the proposed framework attains su- perior performance to the baseline models in terms of accentedness and audio quality.
1 Introduction
The accent transfer task refers to the synthesis of speech with an accent, such as British English for North Americans. Accent pronunciation is a dis- tinctive form of expression influenced by the native language, the speaker's social group speaking in particular region (Loots and Niesler, 201 1). In general, people find it casier to talk with others in their own accent group. The use of speech is now widely adopted, for example in the field of chat- bots and film dubbing requires research on accent transfer speech.
However, existing accent transfer works highly rely on large amount of labelled parallel data or advanced speech recognition models. Working with enormous amount of labeling data is always hectic, labor-intensive, and time-consuming, which is more severe for low-resource languages. This
limitation hinders the wider application of accent transfer in low-resource scenarios. Hence, it is a timely question: Is it feasible to do non-parallel accent transfer task under a unified framework? It is challenging because the speech representation containing various components, including speaker timbre, accent characteristics, and linguistic tent, which difficult to disentangle, especially for non non-parallel accent transfer task.
At present, the accent transfer task for parallel data has achieved sound research results and in- creasing performance. Divided according train- ing data, the methods specifically divided into: (1) Parallel corpus of different accents of the same speaker using souree and target speech content and time alignment (Finkelstein et al., 2022: Liu et al., 2022: Hida et al., 2022; Toda et al., 2007: Oya- mada et al., 2017). (2) Non-parallel corpus of Corresponding author
The dataset for the task in this paper can be represented as (Sa(Aa), (,,())) where the a speaker S, can only speak the accent Ag, and the speakerS), can only speak the accent A,, The objec-

---

tives of this paper are achieve respectively: (1)
2 Method
(Aa) + (.(A),, where the A. accent transfer to the S, speaker without changing the linguistic content of the speech itself. (2) Sb(Ap) - Sa(A), where the As accent transfer to the Sa speaker with- out changing the linguistic content of the speech itself. (3) (Aa) + (s(A)), two-way speaker timbre and accent transfer between the S. speaker and the S₁ speaker.
In this section, we first describe our model archi- tecture. Then we introduce the fine-grained accent modeling and adversarial mutual information learn- ing and show how accent transfer between speaker S. and speaker S on non-paralle datasets with different speakers and accents is achieved.
2.1 Architecture of the proposed model
Following the success of mutual information learning to disentangle speaker information in the One-shot voice conversion (VC) task (Yang et al., 2022), this paper applies the non- non-parallel data- based voice conversion model MaskCycleGAN- VC (Kaneko et al., 2021) to a more challenging task: voice and accent joint conversion. The source speaker's accent can be converted to the target speaker's accent without changing the linguistic content of the speech. The most challenging task is achieve effective disentangling of accent fea- tures, linguistic features, speaker timbre features and fine-grained embodied modeling of accents in a unified model architecture, and to achieve controlled and effective speaker timbre and accent transfer in the prediction phase of the model. The accurate modeling of phonetic pronunciation tones in the task of accent transfer is crucial. The contri- butions of this paper are as follows.
The generator structure of the model is shown in Figure. 1. We improve the generator part of the MaskCycleGAN-VV (Kaneko et al., 2021) for spe- cific data and application scenarios. The genera- tor part is composed five parts: an accent en- coder Eace a speaker encoder Eg, a speech con- tent encoder E., a speech generator G, the feature disentanglements C1 and C2. The model train- ing strategy adopts non-parallel voice conversion approach. Given non-parallel corpus D(z,, v), the training mechanism involves mapping source speech x to converted speech y and then back x', with the primary training objective being the minimization of the mean square error between x and x'. The baseline models training details and our model parameters please refer to Appendix 5.
2.2 Encoder and accent modeling
Encoder:
The accent encoder Eoc takes the mel- spectrogram S and normalized pitch contour P of the speech as inputs. The accent encoder pro- vides global speech accent features to control the speaker's 's accent. We use vector quantized vari- ational autoencoder (VQ-VAE) (Van Den Oord et al., 2017; Polyak et al., 2021) model to learn suprasegmental representations related tone. The speaker encoder the same structure in (Chen et al., 2021), using Conv as the main structure to extract speaker features. The speech content encoder is the downsampling module in MaskCy- cleGAN (Kaneko et al., 2021), the rhythm encoder Ep extract speech rhythm features, have:
(1) For accents being difficult model fine- grained concretely, this paper fine-grained con- cretely models accents in terms of phonetic into- nation, rhythmic pauses and other pronunciation features. Then, an accent feature encoder is pro- posed, which can effectively extract accent features in the inference stage and realise accent controlla- bility modeling.
(2) To address the problem of difficult speaker information disentangling in non-paralled data sets with different speakers with different accents, this paper proposes mutual information learning to max- imize the mutual information upper bound ac- cent features and speech features. It can effectively disentangle the speaker features, accent features and phonetic features of speech.
Zoc - Eac(E, (S), VQ - VAE(P)),
Ze = Ec(S),
(1)
(3) Experimental results show that method con- verts the speech up a MOS score of 4.12. Achiev- ing optimal results compared to baselines the English accent transfer task for the public VCTK dataset and on the Lao accent transfer for the self- constructed Lao dataset. It significantly improves the accentedness and audio quality.
Z, = F),(S,
where Zec represents the accent features, Ze repre- sents the speech content features, 7, represents the speaker features.
Accent modeling: Tones alteration stands out as one of the primary distinguishing features

---

Speech
Spectrogram
Filling in Prames (FIF)
Fac
Cony ID
ReLU
Cony
x4
B
Missing frames Spectrogram
VQVAE
ReLU
E,
Convil
ConvID
E,
Cony
Normalization
ADD
Multiply
Init Random Voctors
Conv Normalization
Normalization sigmand
Conel
Voctor
Matrix Distance Calculation
Quatization
Multiply
Embedding
Residual Convolational
Ze
Z
concal
Cony 1D TEAN
Conv ID TEAN
sigmeid
Multinly
C1
ConiD
C2
Converted Spectrogram S
Figure 1: Framework of proposed model. The Zoe is the output hidden states of accent encoder Eac- Z, is the output hidden states of hidden states rhythm encoder E,,, C'1 speaker identity classify with linear, C2 is accent classify with gradient reverse linear.
speaker encoder E,, Ze is the output hidden states of content encoder Ec, Z, is the output
accents, and these different tones are reflected in the spectrum by different fundamental frequencies, so tone features can be represented by the discrete speech feature FO. To better model the tone at fine- grained degree, we adopt the VQ-VAE framework to train pre-training model for speech tone feature FO. The yet another algorithm for pitch tracking (YAAPT) (Kasi and Zahorian, 2002) algorithm is used to extract the FO from the input signal, x, generating (P,,..., P,v), we have:
Then, the bottleneck maps each latent vector to its nearest vector in the codcbook C. The embed- ded latent vectors are then being fed into the de- coder Decodera, (cfo.) the original FO signal. Similar to (Dhariwal et al., 2020), use Exponential Moving Average up- dates to learn the codebook and employ random restarts for unused embeddings, we use the indices of the mapped latent vectors to generate Zp- The rhythm of the speech extracted using the same structure as E, in SpeechSplit (Qian et al., 2020) to obtain rhythm features output Z,. At last, the Zp @ Z, as representation of accent Z it is incorporated into the speech generation model. We chose F.p and F,, components of the accent coder Egc because pitch and rhythm represent the most significant aspects of accent variation. Addi- tionally, recent works (Qian et al., 2020; Dhariwal et al., 2020) have demonstrated their effectiveness in extracting pitch and rhythm features. We hope that the discrete representations learned from FO capture pitch patterns and/or other suprasegmental information. The proposed extension is straightfor-
) P which reconstructs
F 2t1 E (PuT)))
e VectorQuantizationn, (E1.));
(2)
Puu Decodero, (ef.ur)
Each element in F 21:1/ is an integer 2s E (0, 1,.., K", where K' is the encoder dictio- nary size. The VectorQuuantisationn..
bottle- neck with learned codebook C (e1, ek'). where each item in C 128-dimensional vec- tor. The encoder extracts sequence of latent vectors Encoderg, (P) (hi,.., hzz) from the raw audio, where h; E 128 for all 1 s < L/.

---

ward, but we observe that it results in impressive improvements for accent modeling, especially FO of reconstructed speech waveforms in Lao.
Libear
2.3 Generator and accent transfer
Generator: The speech generator G' is based on the architecture of MaskCycleGAN-VCC (Kaneko et al., 2021) and uses the Filling in Frames (FIF) strategy during training, where random part of the spectrogram is masked. The mel-spectrogram is downsampled and mapped from high dimen- sion to low dimension Ze- Then, the upsampled Ze (speaker features) and Zee (accent features) are combined to map these features into the mel- spectrogram of the target speaker. The discrimina- tor has the same structure as MaskCycleGAN-VV and takes as input the mel-spectrrogam generated by the generator for the target speaker. we have:
Belt
Safirm
Ret
Figure 2: Framework of common classifier C'1 and adversarial speaker classifier C2.
the correlation among different speaker- speaker-irrelewan irrelevant speech representations:
( Ze)min
N Dog 90 (Zoc
[ccc
(5)
N
$ G(Zoss Zoy Zt),
j=1
log go Zucy Zza]
where S represents the converted speech. Controllable accent transfer: During the model inference stage, different target accents can be achieved by controlling the accent feature Zae-
where zo(ZanlZz) is variational distribution with parameter 0 to approximate p(2ncl Zo). Í is the unbiased estimator for vCLUB with samples {Zac Ze The indexes i and j are the samples of Zoc and Ze. The MI loss is:
S5,(1a) = ( $5.(Ab) = (5)) (S.,),
(4)
LMI= Tun=ii(Zoc Zc).
(6)
where (5,(Aa) represents the speaker S₁ with ac- cent Ag, which is not present in training data (only present the speaker S₁ with accent Ag).
Loss: The final training objective of the proposed model to train the generator (As)((AA(((((AAA))))))
achieving bidirectional conversion of speech and accent between speakers Sa. S₁ and accents A., Ag. A full objective Lfull is written as follows:
2.4 Disentanglement and loss
Speaker information disentanglement: As shown in Fig. 1, we use common classifier C'1 and adversarial speaker classifier C2 with gradi- ent reverse linear (GRL) (Ganin et al., 2016) to recognize the identity of speaker, The Fig. illus- trates the model architecture of the two classifiers. Both classifiers, C1 and C22, utilize speaker IDs as supervisory labels during the training process. The objective of C1 is to accurately classify the speaker ID associated with Z, as the training pro- gresses. On the contrary, the training progresses, C2 is designed to gradually struggle in correctly classifying the speaker ID for Z and Z,.
Liull = Cmask- add SS(A..)
mmask-add
+b(Ab) -o(AA)))-SAAAAA)))
+ +Acyo So(Ao) (C >S.(Aa) {8a(Aa)
Laee -
(7)
+Nia (AL))- + idd (s,
Ladu2 + (g.(A)))-55 (
+ + Sp(Au) (c()))))s >S, (Aa) :.5(()))
+aLam-Cy + BL C2 7LMI,
The variational contrastive log-ratio upper bound (yCLUB) (Cheng et al., 2020) used to compute the upper bound of mutual information (MI) for irrelevant information of the speaker, decreasing
where maask-ady Leve, Lid and L are loss function defined in MastCYyleGANN-CC (Kaneko et al., 2021). Lcom-C, and Ludv-C2 are the cross- entropy loss of the classifiers C'l and C2. The

---

MMI loss corresponds minimizing mutual infor- mation of ze and Ze. Aeyer Xid. a, B and y are the hyperparameters.
Table 1: Evaluation results of different models. Aver- age MCD, RMSE, SSIM, MOS with 95% confidence between the converted speech and the ground truth ref- erence. "** denotes the proposed model. "w/o" is short for "without" in ablation study. The "Ap213" " "Aps29", " AAp24a" and "A "Ap244" represent the London accent, American accent. Indian accent and Manch- ester accent of English, respectively. A: the difference value between our model and the best baseline model, t: improved performance compared to the best baseline model. Bold: the best performance under each category. underline: the second best performance, "_" results are not available.
3 Experiments
3.1 Data
The experiments are conducted on the VCTK (Veaux et al., 2016) corpus. For En- glish accent transfer, we conducted speaker selection involving individuals with diverse accents to validate our approach. In addition, we conducted comprchensive experiments accent transfer in the low-resource language of Laotian. We utilize total of 1000 Lao Vientiane accent and 1000 HuaPhan accent utterances, with 100 samples in the validation and testing set, totaling about I hour. All audio data used in the experiment have a sampling rate of 22.05kHz.
Methods
CyclGAN VC2
2.60
CyCGGAN-VC3
6.12
0.84
29.54
4.01
6.82
0.79
31.21
3,89+0 16
Speni-Apaan
SRD
6.94
0.83
30.05
3.83.10111
Our Model
561
0.86
29.40
4,08=0017
00.51
1.002
10.14
10007
CydlGAA VC2
Syoul Aaaus)
CyckGAN
MakkCclkGAN-V
Spanl-Aauul
SRD
6.79
0.78
32.56
3,98 12
Our Model
5,39
0.83
28.15
4.05 018
3.2 Model and training setup
1,20
1
1005
1.41
10.0
VC
In the experiments, accent (a(Aa) is used as the source speech and accent S₂ is used as the target speech for training. We compare our proposed method with the current best non- corpparlllll speech conversion models:
CycleGAN VCI
MunkyykGGA-VC
Speia
SRD
6.56
0.79
35.64
385+0.1
Our Model
537
0.76
32.02
1
1003
1.662
0.007
Speat (paus)
wif Z
6.12
0.69
33.12
381+0. 18
Zogyg-vas
3.23
SpsonlApas))
wlo Zuc
CyC
3,08+0
CycleGAN-VC2 (Kaneko et al., 2019). A GAN- based speech conversion model that uses mel cep- strum input and output.
Spaiel Apris)
CycleGAN
5.65
0.85
33.98
4.01
MaskCcleGAN-VC
5.83
0.69
35.72
3.86.00
Spoul-haau
0.74
32.18
3.92+0
Our Model
533
0.33
34,50
4.0210.15
0002
1.001
CyclGGAN
CycleGAN-VC3 (Kaneko et al., 2020a). This model uses mel spectrogram as input and output instead of mel cepstrum, and incorporates a time- frequency adaptive normalization (TFAN) module on the basis of CycleGAN-VC2.
CyclGAN
Spaialdgais)
MasKCcc&GAN
Syesal Apeus)
SRD
7.12
0.83
33,87
3,60
Our
1.43
1.0.5
1.662
0.26
CyC2GAN-VC2
6.45
3,01+0.10
Spaut--aaui
CycleGAN VC3
5.22
0.87
42.69
3.82.10.1
MukCyccGGAN-C
5.59
0.76
49.41
3.98200
MaskCycleAAN-VC (Kaneko et al., 2021). This model adds a mask mechanism on the basis of CycleGAN-VC2.
SyouttA
35.69
3,92+0
Our Model
6,08
0.79
30.86
4,08 10.1
0.08
1
CyLEGAN-VC
CycleGAN VC3
Spaia(Apaiss)
CycleGAN VC3
SRD (Yang et al., 2022). A method that disentan- gles speaker information based mutual informa- tion learning.
Syosal Ayoual
SRD
7.58
0.60
36.42
3,85+0
Model
1.44
10199
5
1.226
CyckGAN-vCr CycleGAN-VC3
Spawe
In the experimental application, the feature of the speech is an 80-dimensional Mel-spectrogram, and the tone feature is represented FO. The FO is extracted from the raw audio using a window size of 20ms and a 5ms hop. The VQ-VAE quanti- zation described at Sec. 2.2, is applied using FO codebook of KO- 20 tokens and an encoder that downsamples the signal by x16. The con- figurations of MaskCycleGAN-VC followed its original paper (Kaneko et al., 2021). We chose the model checkpoint with the lowest loss on the
MaskCyclGAN VC
Spaial
SRD
7.10
0.59
35.50
3.91
Our Model
6.17
0.68
32.10
4.12-0.18
3.40
0.211
Spait Apaia)
wlo Z
7.69
0.43
29.36
3.0030
Zwag YAE
9.12
0.45
37.52
3.33
Syavel Apus)
who Zack
9.79
0.28
59.52
2.05.0.11
CycleGA -C2
6.76
0.76
54.39
3,34
Ayzul))
CyeleGAN-VC3
5.41
0.83
32.17
3.67+0
MaskCyclGAN VC
5.51
0.79
42.15
3.94
Aptza))))
SRD
Our Model
5.48
0.81
33.01
3.9-0.111
1.007
00002
1.884
10.03
CyRRGANAVCI
CyckGAN-VCC
Syeer Aaau)
CyelGGAN
Spsze
SRD
6.58
0.52
35.32
3.99
Our Model
33.58
4.04=00111
0.42
1 0.01
1.74
1..05

---

validation set. HiFiGAN vocoder (Kong et al., 2020) is employed to generate speech waveforms from mel-spectrograas. In the conversion model training stage, conversion model is trained for 100 epochs using batch size of 1. We use Adam opti- mizer (Kingma and Ba, 2014) with learning rate is 0.0002, 31 = 0.5, 32 0.999 for speech gen- erator. With learning rate is 0.0001, 31 0.5, 82 0.999 for speech discriminator optimizer. All experiments are conducted on a single NVIDIA 3090 for training.
Table 2: Evaluation results of different models. Aver- age MCD, RMSE, SSIM, MOS with 95% confidence between the converted speech and the ground truth ref- erence. "*" denotes the proposed model. "w/o" is short for "without" in ablation study. The "A,." " is Vientiane accent of Lao, "A,, is Huaphan accent of Lao. A: the difference value between our model and the best base- line model, t: improved performance compared to the best baseline model, Bold: the best performance under each category, underline: the second best performance, results are not available.
Accent Transter
Methods
MCD
SSIM
RMSE
MOS
CycletAN--VC
8.40
0.68
40.70
2.82-0.17
3.3 Experimental results and analysis
So(Aa) 6->
CycleGAN-VC3
2,17
0.81
3.95
MeRCCykGGAN-VC
7.65
0.67
35.61
3.79-0.15
S.(As)
SRD
7.70
0.83
33,50
3.8520.12
3.3.1 Accent Similarity and Speaker Similarity
Medel
7,09
0.55
29.87
4.01-0.16
00008
1.002
0.44
0.006
CycRUAN-VC2
Su(Aa)
CycloGANN-VC:
In order to verify the accent similarity and speaker similarity between the converted speech and the original target speech, we perform a accent and speaker visualization using t-SNE method (Van der Maaten and Hinton, 2008) hased on the accent rep- resentation Zac and speaker representation Z, of different speakers utterances. For accent similarity, there are 200 utterances sampled for two speaker (Sp243, Sp248) to calculate the accent representa- tion. Meanwhile, we concatenate Zp and Z, in SRD and compare with our proposed method. As can be seen in Figure. 3(b) and Figure. 3(c), While SRD partially represents different speaker accents in terms of pitch and rhythm, most accent hiiden states are still mixed together. In contrast, the ac- cent embeddings in our approach are separable for different speakers, with only few representations being slightly mixed. Our analysis of the Vientiane and Huaphan accents in Lao shows some similarity certain sentences. These results indicates that our fine-gri une-grained Lao accent modeling encoder Eac is capable of extracting the accent Zac as speaker accent information.
MustCyckGAA-Ve
Sx(A))
7.56
0.1
54.73
Our Modal
7.32
0.5
31.01
3.81011
0.24
1004
10.5
CyclGAA VC2 CycleGAN-VC3
Su(As)
MisCCydkGAN.V
(,(As)
SRD
7,58
0.40
35.64
3.85-0.12
Our Model*
736
0.57
3008
4.02:0016
00:42
10.17
1.662
10.17
Su(Ag)
uho 7,"
7.92
0.49
30.36
3,79-6.12
vioe Zanve
8.62
0.36
3.5320.16
86(A.)
w/o Zy:*
9,88
0.24
65.52
19950.11
3.3.2 Objective Evaluation
For objective evaluation, we use mel-cepstrum distortion (MCD) (Toda et al., 2007), root mean square errors (RMSE) (Luo et al., 2017) between synthesised and reference speech utterances. The lower the MCD is, the smaller the distortion, mean- ing that the two audio segments are more sim- ilar to each other. To evaluate intonation vari- ations of the converted voice, RMSE of source and converted voice is calculated. To account for the temporal difference, dynamic time warp- ing is performed between the converted utter- ance and the target reference to compute MCD and RMSE, where the RMSE of FO is calculated only on the voiced frames in the reference utter- ances. To evaluate the proposed method objectively, 50 conversion utterances pair are randomly se- lected. Table summarizes the MCD, SSIM (Wang et al., 2004) and RMSE evaluation results on VCTK datasets. It is worth noting that in the ac- cent transfer tasks (S/243 (422s)(A22s))S5p2am (Ay243). Sy248 (Ay248) - Sy243 (Ay248)), we do not have data on real labels, so we did a cycle convert, e.g., Sp245( (o243(Ap243) >p224 (Ap243) > Sp243 (S224((A2243)) Is observed that: our model outperforms all bascline models consistently for MCD and achieves the best
For speaker similarity, We randomly selecte 200 utterances and converted them to target speak- ers (Sw. S2)) each with 2 accents (Aw: An) in Lao. The speacker representation is calculated using the speaker encoder. Subsequently, we visualize the speaker representation Ze by t-SNE (Van der Maaten and Hinton, 2008) in Figure. 3(a). The results demonstrate that all samples were grouped into two clusters representing the two target speak- ers. This unveils that the output speech samples from our model, including those converted samples with non non-native new accent, have successfully pre- served the speaker similarity of the target speakers.

---

R. of caneeeeie SalAo)
ef cxnvertted (a(d))
R, ofseiginal SslAs)
R, oforigiaal (a(Ao)
Aum of Viautaone A.
Fu oftahah A
speaker 2e analysis
(b) The accent 2n, analysis of SRD
(6) The accatt Za, analysis of Model
Figure 3: The different encoder output visualization using t-SNE. (a) E, encoder output Z, visualization of our model, (b) Eac encoder output Zec visualization SRD baseline, (c) Eac encoder output Zee visualization of our model.
or second best results of SSIM, which shows the proposed method has better intelligibility while pre- serving the source linguistic content. In addition, in Table 1, we note that our model achieves the lowest RMSE in the task of English accent transfer involv- ing four distinct accents, which shows the ability of our model in transforming and preserving the de- tailed intonation variations from source speech to the converted one. This indicates the effectiveness the proposed fine-graines lao accent modeling encoder. Similar conclusions are obtained in the transfer of the low-resource Lao between Vientiane accents (Aw) and Huaphan accents (A₂) as shown in Table 2.
accents, as well as Indian (p248) and British (p243) accents, and Indian (p248) and American (p329) accents. In addition, the MOS results in Table show that our model also performs well in the low- resource language Lao of both (Am. S (Aw). Sy, (A,) (w,)), (ww) <> S, (AA, accent transfer tasks. Note that the baseline methods (MaskCycleGAN, CycleGAN-VC3, CycleGAN- VC2) cannot generate samples properly with a ac- cent (Aa) and accent (Ay)). Because these non- parallel speech conversion models do not have the ability to disentangle speaker representations and accent representations. Hence these mothods do not have samples for MOS test. SRD has the ability to decouple speech features such rhythm, pitch, and content, which enables it to achieve a certain degree of accent conversion. However, due the lack of modeling of the tone by pre-trained VQ- VAE models, its performance accent conversion tasks is not as good as our proposed method.
3.3.3 Subjective Evaluation
Subjective evaluations are conducted using listen- ing tests with human subjects. AB preference test is performed to evaluate speech quality and speaker similarity, respectively. Additionally, mean opin- ion score (MOS) tests are conducted to determine listeners' preferences across all experimental meth- ods. For each test, 20 samples were randomly se- lected from the converted samples of each experi- mental system and provided 15 participants.
A Accentedness: In the AB test on accentedness, paired speech samples with the same textual con- tent are presented and the listeners are asked to choose samples that more similar the target accent. The results are shown in Figure. 4. Irre- spective of the language, the proposed model is more effective with much more preference.
Audio Quality:
the MOS test, listeners are asked to rate the quality of the converted speech on a 5-point scale. Audios converted from the three systems are randomly shuffled before presenting to listeners. Each group of audio corresponds to the same text content. The MOS results in Table I show that model achieves the best quality MOS compared with other methods on the VCTK dataset. Simultaneously, better performance has been achieved in the task of accent transfer involv- ing English British (p243) and American (p329)
3.3.4 Conversion Visualization
Lao language is a tonal language where the vari- ations in tones become more pronounced with changes in intonation (Brickson, 2001). Fig- ure. shows the spectrogram and FO of the source S.p (Aw), target (Aa) and converted speeches (x(AA) with the same Lao content. Please note
we use parallel speech data to visualize the re-

---

Our Model
No preference
Baseline
(A(A))))8(AA))
5a(A.) + (L(A))
S.(A,) + 9x(As)
vs. CycleGAN
MeskCyeleGAN
vs. SRD
0%
50%
100%
0%
100%
0%
100%
Sysal (pau) Syonn (A,2s))
Syaenl (paur(AAoas) - Syaer(Apaes)
(aan(Auu) Syaus (Apoes)
CyclGAN
vs. MaskCyclGGNN
SRD
50%
100%
0%
50%
100%
0%
50%
100%
Figure 4: Accentedness preference test results
F0
19
Preqerscytzz
8:00
Spectrogram and FU source Sp(Ag)
(h) Speetrrrrrrm and converted 8x(A.)
(c) Spectrugram and PD of 5x(As)
Figure 5: comparison spectrogram FO for source (A,)), target (,,A,) and converted speeches Sn (. Horizontal axis (X-axis) displays time in second, and vertical axis (y-axis) represents spectral frequency and FO frequency respectively
sults. For accent conversion (w (An,). the FO contour of the converted speech matches average pitch of the source speech and retains de- tailed characteristics of the source pitch contour. The spectrogram details of the converted speech match of the target speech, but there are significant differences in the contour of the FO pitch. The re- sults demonstrated that our model has successfully achieved accent conversion non-parallel training data (Sw (Aw), SA (Ax))
the accent transfer task well.
Conclusions
Based on the application scenario of accent trans- fer in non-parallel data sets, this paper proposes a non non-parallel accent transfer method based on fine- grained controllable accent modeling. It applies VQ-VAE network for fine-grained modeling of voice intonation and rhythmic pauses, and then delivers the obtained accent features and speech features to a mutual information based learning feature disentangler. The features extracted by the trained accent encoder can guide the pitch and rhythm variation of the generated speech in the prediction stage of the converted model, achieving controllable modeling of the various accents. The proposed method generates speech with greatly improved fluency and naturalness, and achieves accent transfer in non-parallel dataset through the application of a unified speech conversion frame- work.
3.3.5 Ablation study
Moreover, we conduct ablation study that addresses performance effects from different methods for lao accent modeling with results shown in the last three rows of Table 1. From the results, when the lao accent modeling Zae without the Zp of speech rhythm, the model is still able to perform accent transfer and outperforms most of the baseline mod- els, but the speech naturalness decrease. When the VQ- VAE framework of FO modeling is removed, the audio quality and accentedness significant de- crease. When the lao accent modeling Zac is re- moved, the results are poor and no longer perform

---

Acknowledgements
Rem Hida, Masaki Hamada, Chie Kamada, Emiru Tsunoo, Toshiyuki Sekiya, and Toshiyuki Kumakura. 2022. Polyphone disambiguation and accent predie- tion using pre-trained language models in japanese tts front-end. In ICASSP 2022-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7132-7136, IEEE.
This work was supported in part by the Na- tional Natural Science Foundation of China (Nos. 62376111, U21B2027 and 61972186), Yumnan provincial major science and technology special plan projects (Nos. 202103AA080015 and 202302AD080003), Yunnan Provincial Key Research and Development Plan (Nos. 202303AP140008). The authors would like to thank anonymous reviewers for their comments.
Takuhiro Kaneko and Hirokazu Kameoka. 2017. Parallel-data-free voice conversion using cycle- consistent adversarial networks. arXiv preprint arXiv:1711.11295.
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. 2019. Cyclegan-ve2: Im- proved eyclegan- based non- non-parallel voice conversion. In ICASSP 2019-2019 IEEF International Confer- ence Acoustics, Speech and Signal Processing (ICASSP), pages 6820-6824. IEEE
Limitations
In this paper, we have focused on modeling ac- cents by examining pitch and rhythmic changes in speech. However, in our future work, we plan to an- alyze accents by incorporating additional features of speech. By doing so, we aim to enhance the authenticity of accent performance and improve our understanding of accent variations.
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. 2020a. Cyclegan- Examin- ing and improving cyclegan-ves for mel-spectrogram conversion. arXiv preprint 27Xiv2010..1667
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. 2020b. Cyclegan-ve's Examin- ing and improving cyclegan-ves for mel-spectrogram conversion. In Proceedings of the Annual Conference of the International Speech Communication Associa- tion.
References
Yen-Hao Chen, Da-Yi Wu, Tsung- Han Wu, and Hung- yi Lee. 2021. Again-ve: one-shot voice conver- sion using activation guidance and adaptive instance normalization. In ICASSP 2021-202!! IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5954-5958, IEEE.
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. 2021. Maskeyclegan-ve Learning non non-parallel voice conversion with filling in frames. In ICASSP 2021-2021 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 5919-5923. IEEE.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. 2020. Club: A contrastive log-ratio upper bound of mutual in- formation. In International conference machine learning, pages 1779 1788. PMLR.
Kavita Kasi and Stephen Zahorian. 2002. Yet another algorithm for pitch tracking. In 2002 ieee interna- tional conference on acoustics, speech, and signal processing, volume 1, pages 1-361. IEEE
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Tlya Sutskever. 2020. Jukebox: A generative model for music. arXiv preprint arXiv:2005.003:
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 1:Xi::412.66980
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for effi- cient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022- 17033.
Blaine Erickson. 2001. On the origins of labialized consonants in lao. In Pupers from the sixth annual meeting of the Southeast Asian Linguistic Society, pages 135-148.
Lev Finkelstein, Heiga Zen, Norman Casagrande, Chun an Chan, Ye Jia, Tom Kenter, Alexey Petelin, Jonathan Shen, Vincent Wan, Yu Zhang, et al. 2022. Training text-to-speech systems from synthetic data: practical approach for accent transfer tasks. arXiv preprint 22.8
Rui Liu, Berrak Sisman, Guanglai Gao, and Haizhou Li. 2022. Controllable accented text-to-speech synthesis. arXiv preprint arXiv:2209. 10804.
Linsen Loots and Thomas Niesler. 2011. Automatic conversion between pronunciations of different en- glish accents. Speech Communication, 53(1):75-84
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. 2016. training of neural networks. The journal of machine learning research, 17(1):2096- 2030.
Zhaojie Luo. Jinhui Chen, Tetsuya Takiguchi, and Yasuo Ariki. 2017. Emotional voice conversion using neural networks with arbitrary scales f0 based on wavelet transform. EURASIP Journal on Audio, Speech, and Music Processing. 2017:1-13.

---

Keisuke Oyamada, Hirokazu Kameoka, Takuhiro Kaneko, Hiroyasu Ando, Kaoru Hiramatsu, and Ku- nio Kashino. 2017. Non-native speech conversion with consistency-aware recursive network and gen- erative adversarial network. In 2017 Signal and Information Processing Association An- nual Summit and Conference (APSIPA ASC), pages 182-188. IEEE.
Guanlong Zhao. Sinem Sonsaat, John Levis, Evgeny Chukharev Huuaaaiiien.iiiinnnnnn and Ricardo Gutierrez- Osuna. 2018. Accent conversion using phonetic posteriorgrams. In 2018 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 5314-5318, IEEE.
5 Appendix: CycleGANs
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentan- gled self- supervised representations. arXiv preprint arXiv:2104.00355.
CyclGANwcwc22/C3Mak (Kaneko et al., 2019, 2020a.b, 2021) (CycleGANS) is voice con- version model consisting of two generators, G, and two discriminators, D. CycleGANs has emerged as a novel approach in the domain of voice con- version, demonstrating its effectiveness in learn- ing the transformation between different acoustic feature sequences without the need for parallel data. These advancements contribute to the on- going progress in voice conversion research and its applications in various fields. The primary objec- tive of CycleGANs, as discussed in the research papers by Kaneko and Kameoka (Kancko et al., 2020a,b, 2021), is to acquire the ability to trans- form acoustic feature sequences belonging to the source domain X into those of the target domain Y without relying parallel data. The acoustic fea- ture sequences represented by RºXT and y E πºxT, where Q and T represents the feature dimension and the sequence length respectively.
Kaizhi Qian, Yang Zhang. Shiyu Chang, Mark Hasegawa-Johnson, and David Cox. 2020. Unsu- pervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pages 7836-7846. PMLRR
Tomoki Toda, Alan W Black, and Keichi Tokuda. 2007. Voice conversion based on sokiiumm-ikellhow esti- mation of spectral parameter trajectory. IEEE Trans- actions on Audio, Speech, and Language Processing, 15(8):2222-2235.
Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
The foundation of CycleGANs is rooted the inspiration drawn from CycleGAN, originally pro- posed for image-to-image style transfer in com- puter vision. By applying the principles of Cy- cleGAN, CycleGANs aims to learn the mapping function G(x) + Y, enabling the conversion of input x E X to output y E Y.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDon- ald, al. 2016. Superseded-cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit.
Zhichao Wang, Wenshuo Ge, Xiong Wang, Shan Yang, Wendong Gan, Haitao Chen, Hai Li, Lei Xie, and Xi- ulin Li. 2021. Accent and speaker disentanglement in many-to taayy-tommmm many voice conversion. In 2021 12th In- ternational Symposium on Chinese Spoken Language Processing (ISCSLP), pages 1-5. IEEE.
In pursuit of this goal, CycleGAN semploys several loss functions during the learning process. These include adversarial loss, eyclic consistency loss, and identity mapping loss, collectively con- tributing to the enhancement of the quality and fidelity of the generated outputs. CycleGAN- VC2 (Kaneko et al., 2020a) introduces an addi- tional adversarial loss to further refine and improve the fine-grained details of the reconstructed fea- tures. CycleGAN-VC3 (Kaneko et al., 2020b) incorporats an additional module called time- frequency adaptive normalization (TFAN). Al- though the performance is superior, an increase in the number of converter parameters is necessary (from 16M to 27M). MASKCycleGAN (Kaneko et al., 2021) use a novel auxiliary task called filling in frames (FIF), which apply a temporal mask to
Zhou Wang, Alan C Bovik, Hamid Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transac- tions image processing, (34)6660--61..
SiCheng Yang. Methawee Tantrawenith. Haolin Zhuang. Zhiyong Wu, Aolan Sun, Jianzong Wang, Ning Cheng, Huaizhen Tang, Xintao Zhao, Jie Wang, et al. 2022. Speech representation disentanglement with adversarial mutual information learning for one-shot voice conversion, arXiv preprint 27X1v:2208.08757,
Guanlong Zhao, Shaojin Ding, and Ricardo Gutierrez- Osuna. 2019. Foreign accent conversion by synthe- sizing speech from phonetic posteriorgrams. In Inter- speech, pages 2843-2847.

---

Table 3: Forms and interpretations of notations.
Symbol
Definition
Original data of speech a
y/
Original data of speech b
x²
Generate sample of speech
Generate sample of speech b
Forward conversion from speech speech b with parameters 01
+X
Inverse conversion from speech to speech a with parameters 0₂
82
S
The mel-spectrogram
P
The normalized pitch contour
S.
The speaker identity of speech
S
The speaker identity of speech b
A.
The accent identity of speech
A
The accent identity of speech
Fac
accent encoder
Es
speaker encoder
E
speech content encoder
C
A speaker identity classifier with linear
C
A speaker identity classifier with gradient reverse linear
Zac
The accent feature from Exc
Zu
The speaker feature from Es
Ze
The speaker feature from Ee
Zac
The accent feature from Eac
The unbiased estimator for vCLUB (Cheng et al., 2020)
the input mel-spectrogram and encourage the con- verter to fill in missing frames based on surround- ing frames. These adjustments add some structure to the text and make it even more mader-friendly.
This paper applies the non-parallel data- based voice conversion model MaskCycleGAN- VC (Kaneko al., 2021) more challenging task: voice and accent joint conversion. The source speaker's accent can be converted to the target speaker's accent without changing the linguistic content of the speech. We improve the generator part of the MaskCycleGAN--VC (Kaneko et al., 2021) for specific data and application scenarios. The comprehensive list of the primary symbols used throughout this paper is presented in Table


Reviews:
[{"review_id": "5BruDgGVsa", "reviewer": "Reviewer_hXtk", "summary": "", "questions": "- QUESTION A: Is the E_ac part trained independently for each dataset (Engllish, Lao) or are different depending on experiments?\n- QUESTION B: The same question arises for the E_s and E_r modules? are they retrained for Lao? are you using same English models?\n- QUESTION C: The Lao dataset is a self-constructed dataset so it lacks a reference, is it possible to make it available?\n- QUESTION D: what is z_p in the figure 1? is it the output of the E_ac?\n- QUESTION E: For the vCLUB in the adversarial classifier C_2, what is the q_\\tetha in the equation (6)? and the indexes i and j?\n- QUESTION F: How the results from table 1, where SRD seems to be similar or even better (in SSIM score table 1) relates to the figure 3 where feature confusion looks bigger for SRD? are the same Engilsh speakers involved in both reportings?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(1). The model training strategy adopts a non-parallel voice conversion approach. Given a non-parallel corpus D(x,y), the training mechanism involves mapping x to y and then back to x', with the primary training objective being the minimization of the mean square error between x and x'. The distinctiveness of this research lies in its innovation of incorporating accent feature modeling and disentanglement. During the model inference phase, various latent variables corresponding to different accents are selectively employed in model decoding. This enables the controlled manipulation of accent transfer, serving as a pivotal contribution of this paper.\n(2). Here is a detailed explanation of each L loss function\n $\\mathcal{L}_{cyc}^{X\\longrightarrow Y\\longrightarrow X} =\\mathbb{E}_x~P_X[||G_{Y\\to X}(G_{X\\to Y}(x))-x||_1]$,\n $\\mathcal{L}_{id}^{X\\longrightarrow Y} =\\mathbb{E}_{y\\sim P_Y}[||G_{X\\to Y}(y)-y||_1]$,\n \\mathcal{L}_{adv2}^{X\\longrightarrow Y\\longrightarrow Z} =\\mathbb{E}_{x\\sim P_X}[\\log_{}{D'_X}(x)]\n+\\mathbb{E}_{x\\sim P_X}[\\log_{}{(1-D'_X}(G_{Y\\to X}(G_{X\\to Y}(x))))],\nwhere x represents S_w(A_w), y represents S_h(A_h), G is the generator，D is the discriminator.\nThe alpha, beta, gamma or lambda are set 0.1 in the experiments.\n(3). The MI loss is not obtained from the C_2 discriminator, which is minimizing the unbiased estimator for vCLUB. The C_2 is designed to gradually struggle in correctly classifying the speaker ID for Z_{ac} and Z_c. And \\mathcal{L}_{adv-C_2} is the C_2 loss by using adversarial speaker classifier with GRL.\n(4). We intend to augment the experimentation involving English speakers in Table 1 by incorporating a minimum of 4 English speakers characterized by distinct accent migrations.\n(5).\nQUESTION A: Is the E_ac part trained independently for each dataset (Engllish, Lao) or are different depending on experiments?\nANSWER A: The E_ac part is independently for each dataset，is depending on language. \n\nQUESTION B: The same question arises for the E_s and E_r modules? are they retrained for Lao? are you using same English models?\nANSWER B: The E_s and E_r modules independently for each dataset, and they are retrained for Lao.\n\nQUESTION C: The Lao dataset is a self-constructed dataset so it lacks a reference, is it possible to make it available?\nANSWER C: We'll be open-sourcing the code and the self-constructed dataset.\n\nQUESTION D: what is z_p in the figure 1? is it the output of the E_ac?\nANSWER D: z_p is the output of E_{VQ-VAE}(P)), Z_p \\oplus {Z_r} is the output of the E_ac.\n\nQUESTION E: For the vCLUB in the adversarial classifier C_2, what is the q_\\tetha in the equation (6)? and the indexes i and j?\nANSWER E: q_\\tetha(Z_ac |Z_c) is a variational distribution with parameter \\tetha to approximate p(Z_ac |Z_c). The unbiased estimator for vCLUB with samples {Z_{{ac}_i, Z_c_i}. The indexes i and j are the samples of Z_ac and Z_c.\n\nQUESTION F: How the results from table 1, where SRD seems to be similar or even better (in SSIM score table 1) relates to the figure 3 where feature confusion looks bigger for SRD? are the same Engilsh speakers involved in both reportings?\nANSWER F: The SSIM scores presented in Table 1 offer only a partial representation of the situation. The similarity between the SSIM scores of the SRD model and our proposed model, and in some cases, even the superiority of SRD's SSIM scores, can be attributed to the following reason: In the SSIM calculation of S_{p248}(A_{p248}) \\Longleftrightarrow S_{p329}(A_{p329})}, a step-wise procedure is followed. Initially, the SSIM calculation of S_{p248}(A_{p248}) \\Longleftrightarrow S_{p329}(A_{p329})}\\Longleftrightarrow {\\hatS}_{p248}(A_{p248})} is performed, and subsequently, the SSIM calculation of S_{p248}(A_{p248}) and {\\hatS}_{p248}(A_{p248})} is executed. This involves an x->y->x' transformation process, the SSIM is the distinctions of x and x'. As illustrated in Figure 3. Consequently, even when the features are not entirely disentangled, the SRD model can still achieve commendable performance. However, it's vital to note that this does not align with the primary objective of this paper, which revolves around accent migration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The approach and ideas seem interesting enough and use state of the art methods to address a challenging and novel task as it is speaker accent transfer.\n- The experiments are conducted on two different datasets and the results obtained in the low-resource scenario are above sota.", "reasons_to_reject": "- The training methodology is not well explained and is hard to follow.\n- The section 2.4. is showing the addition of several loss from each of the components. To me is not clear what L function stands for and most of the different losses are not explained in the text or some weighting parameters as alpha, beta, gamma or lambda, the latter seeming to control the cycle content loss and the speaker id loss. \n- It is also hard to understand the MI loss obtained from the C_2 discriminator.\n- Low number of speakers are reported in the tables for English.", "questions_to_authors": "- QUESTION A: Is the E_ac part trained independently for each dataset (Engllish, Lao) or are different depending on experiments?\n- QUESTION B: The same question arises for the E_s and E_r modules? are they retrained for Lao? are you using same English models?\n- QUESTION C: The Lao dataset is a self-constructed dataset so it lacks a reference, is it possible to make it available?\n- QUESTION D: what is z_p in the figure 1? is it the output of the E_ac?\n- QUESTION E: For the vCLUB in the adversarial classifier C_2, what is the q_\\tetha in the equation (6)? and the indexes i and j?\n- QUESTION F: How the results from table 1, where SRD seems to be similar or even better (in SSIM score table 1) relates to the figure 3 where feature confusion looks bigger for SRD? are the same Engilsh speakers involved in both reportings?", "paper_topic_and_main_contributions": "The work proposes a method for the application of accent transfer (F0 and rhythm) between two different speakers and accents. The evaluation is performed on two different datasets, the VCTK corpus and a Lao corpus with no parallel-corpus (same speaker with different accents). The proposed method mixes most of the ideas from the MaskCycleGAN-VC (Kaneko et al. 2021) and combines the mutual information learning from SRD method (Yang et al. 2022), adding the ability of the model to segregate speaker features, accent features and sentence's content. The method also improves the methods from (Kaneko et al. 2021) by adding feature disentanglement components (C_1, C_2) to classify speakers by using speaker features or accent features, respectively.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The reference (Mohsen and Nirmal) is not well chosen and other references will benefit even more the paragraph.\n- In line 46, concatenate all references within the same parenthesis.\n- In lines 79-80: you could remove \"these fine-grained features\" to make shorter and a more readable sentence.\n- In line 119, \"an accent feature...\"\n- In line 152: \"a speach content\"\n- In Eq(1), the first line: S --> E_r(S)\n- In line 191, row --> raw\n- It is hard to follow the text and relate it to the figure 1. Such figure could be improved by, e.g. , a dashed square on the encoder content part, adding E_c. Also adding E_ac in the accent encoder dashed square. Also I would suggest add the references (Chen, 2021) within the E_s box and (Quian, 2020) in the E_r one. To make it consistent with the main text, the output \"Converted Spectogram (\\hat S)\n- In line 332: \"hiiden\"\n- In line 342: \"selecte 200\"\n- In the line 380: the method is not outperforming SRD in SSIM scoring.\n- In the paragraph (lines 383-388): Add a reference to table 1.\n- To improve the contrast in figure 6, I would suggest to switch the color for the F0 contours.\n- In section 3.3.4. line 456-457: the sentence \"but there are significant differences in the contour of the F0 pitch\" is saying the contrary that 6 lines above."}}, {"review_id": "JaHPgXHdpV", "reviewer": "Reviewer_4Wqo", "summary": "", "questions": "- [A] How do these models compare against parallel accent transfer techniques? While I understand that they would do much better, it would be nice to see the comparisons (how close we are to these techniques).\n- [B] How many iterations of experiments have you run? Would like to understand how stable the proposed model is.\n- [C] In Figure 1, what does the arrow joining Z_c and Z_ac indicate?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(1). The E_r is used to extracte the rhythm of the speech, which has the same structure in SpeechSplit\\cite{qian2020unsupervised}. The E_t is the speaker encoder to extract speaker features, which is the same structure in \\cite{chen2021again}. Our selection of these architectures was influenced by their demonstrated capability to effectively extract both speaker timbre features and speech rhythm features in prior research. \n(2). We will provide more comprehensive parameter descriptions pertaining to this paper, along with an elaboration on the workings of MaskcycleGan, within the appendix section.\n(3).\nQUESTION A: How do these models compare against parallel accent transfer techniques? While I understand that they would do much better, it would be nice to see the comparisons (how close we are to these techniques).\nANSWER A: The MOS score of parallel accent transfer techniques is 4.57, closely aligned with the reference waveform score of 4.59\\citet{liu2022controllable},  indicating a remarkably high quality of generated speech that closely approximates actual labeled data. Our model achieves a MOS score of 4.02, implying a slight disparity compared to the reference label.\nQUESTION B: How many iterations of experiments have you run? Would like to understand how stable the proposed model is.\nANSWER B: We conduct each experiment 5 times and report the mean score. \nQUESTION C: In Figure 1, what does the arrow joining Z_c and Z_ac indicate?\nANSWER C: It indicates the Mutual information disentanglement learning between Z_c and Z_ac.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Applications to the research work done (non-parallel accent transfer) are numerous and the topic in general is exciting. \n- The results seem promising, achieving SOTA results in the non-parallel accent transfer domain. \n- It seems to apply for different kinds of languages — English (high-resource, non-tonal) and Laos (low-resource, tonal). \n- Excellent ablation studies to show if all the features used for modelling are truly needed for performance improvement (and also how much each of the features contributes to the performance). \n- Great visualisations to show how well the accent transfers (t-SNE plots) if the proposed model is applied.", "reasons_to_reject": "- I found the paper difficult to follow — possibly because of the way it’s presented.  E_r and E_s seem to be integral part of the network proposed. However, the motivation to choose those specific architectures for E_s and E_r are unclear (at least directly from the paper). Additionally, one has to refer to the papers to understand them (keeping in mind that this is for a wider NLP audience, who are not necessarily up to date with VC task). Instead it would be nice to briefly explain them in the paper (or in the appendix) so that its easy to refer.\n- Again keeping in mind that this is a venue for wider NLP audience, it would be nice to have a section in the appendix explaining what MaskCycleGAN-VC is.\n- The parameters do seem underspecified to me. Would be nice to have them all in the appendix for easy reproducibility.", "questions_to_authors": "- [A] How do these models compare against parallel accent transfer techniques? While I understand that they would do much better, it would be nice to see the comparisons (how close we are to these techniques).\n- [B] How many iterations of experiments have you run? Would like to understand how stable the proposed model is.\n- [C] In Figure 1, what does the arrow joining Z_c and Z_ac indicate?", "paper_topic_and_main_contributions": "The paper proposes a system to model the task of non-parallel accent transfer and achieves promising results. Concretely, the system does two things:\n1. Propose an architecture to create accent embeddings given speech (suprasegmental features) as an input (Z_ac, in the paper)\n2. Disentangle speaker information and accent information, thereby allowing for a non-parallel accent transfer of a particular speaker. (Z_t, in the paper)\n\nUse above two, along with existing method to capture linguistic content (Z_c, in the paper), to generate accented speech using MaskCycleGaN.\n\nThe results are shown in two languages, i.e., English and Laos. The proposed model achieves the best results in terms of Mean Opinion Score (MOS), MCD, SSIM, RMSE compared to other systems that model the same task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper would, in general, benefit from another round of proof-reading. Here is a non-exhaustive list of typos and possible fixes (this did not affect the ratings)\n- Zr in line 203 not defined? Er could be defined better.\n- Equation (3), is Zs supposed to be Zt? Zs and Zt are used interchangeably following this equation (such as Equation (4) as well. Would be nice if the authors fix all such occurrences so that its easier for readers to follow the work.\n- Defining abbreviations — GRL, MI, vCLUB, TFAN, SRD, YAAPT etc.\n- Line 030, “people find it easier”\n- Line 074, “timing question” -> timely question? I am unsure if thats the authors meant.\n- Line 079-081, “which are difficult .. transfer task” doesn’t read well to me. Instead, “. These fine-grained features are difficult to disentangle, especially for non-parallel accent transfer task”\n- Line 116-117, “this paper fine-grained concretely models..” -> “this paper fine-grained concretely models..”\n- Line 119, “a accent” -> “an accent”\n- Line 148, “we” -> “We”\n- Line 160-163, “which using .. related to tone” -> “. We use a VQ-VAE (Van Den et. al.) model to learn suprasegmental representations related to tone” \n- Line 191, “row” -> “raw”\n- Line 332, “most accent hiidden” -> “most accent hidden”\n- Line 358, “root mean square of RMSE”, should be “root mean square of ??”\n- Line 363, “F0RMSE” -> RMSE of F0?\n- Line 382-383, “with preserve the source linguistic content” -> while preserving the source linguistic content\n- Line 383, “We” -> “we”\n- Line 383, “our model achieve” -> “our model achieves”\n- Line 384, “In” -> “in”\n- Line 383-390, “In addition..modelling encoder” is difficult read. Better to paraphrase/split them into multiple sentences\n- Line 426, “distangle” -> “disentangle”\n- Line 440, “mater” -> “matter”\n- I see modelling (Line 432) and modeling (Title, Line 172), would be nice to follow one throughout the paper.\n- Line 440-441, “No mater the language is English or Lao” -> “Irrespective of the language”\n- Line 442, “are” -> “is”"}}, {"review_id": "m5WlZM8fnn", "reviewer": "Reviewer_sM6F", "summary": "", "questions": "In addition to RR:\n1. What is MI? This is not defined at all.\n2. Loads of terms are not defined in Eqn. 5.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(1)Both classifiers, C1 and C2, utilize speaker IDs as supervisory labels during the training process. The objective of C1 is to accurately classify the speaker ID associated with Z_t as the training progresses. On the contrary, as the training progresses, C2 is designed to gradually struggle in correctly classifying the speaker ID for Z_{ac} and Z_c. The input of C1 is Z_t, which is trained to proficiently differentiate between various speaker intonations. The input of C2 is Z_ac, which is trained in an adversarial way to decouple Z_ac and Z_t, transforming accent features into speaker-independent vectors.\n(2)MI is mutual information, .Here is a detailed explanation of each L loss function of Eqn. 5\n$\\mathcal{L}_{cyc}^{X\\longrightarrow Y\\longrightarrow X} =\\mathbb{E}_x~P_X[||G_{Y\\to X}(G_{X\\to Y}(x))-x||_1]$,\n$\\mathcal{L}_{id}^{X\\longrightarrow Y} =\\mathbb{E}_{y\\sim P_Y}[||G_{X\\to Y}(y)-y||_1]$,\n \\mathcal{L}_{adv2}^{X\\longrightarrow Y\\longrightarrow X} =\\mathbb{E}_{x\\sim P_X}[\\log_{}{D'_X}(x)]+\\mathbb{E}_{x\\sim P_X}[\\log_{}{(1-D'_X}(G_{Y\\to X}(G_{X\\to Y}(x))))], \\mathcal{L}_{com-C_1}}  and {\\mathcal{L}_{adv-C_2} are the cross-entropy loss of the classifiers C1 and C2. The \\mathcal{L}_{MI} loss corresponds to minimizing mutual information of Z_ac and Z_c.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method shows good performance with adequate analysis", "reasons_to_reject": "1. There are some technical flaws in this paper: The definition of C1 is speaker classification, and it takes Z_ac as input which is meant to capture accent information (which should be speaker independent?). Also C2 takes the concatenated Z as input but was trained in an adversarial way - isn't it going to confuse Z_ac what it should capture exactly?\n2. The writing of this paper is poor: It makes grammatically mistakes from the very first sentence (works-> work). The nomenclature is weird: e.g. it uses C as the codebook and then C1 and C2 as two tasks.\n", "questions_to_authors": "In addition to RR:\n1. What is MI? This is not defined at all.\n2. Loads of terms are not defined in Eqn. 5.", "paper_topic_and_main_contributions": "This paper proposes a framework that can do both accent and voice conversion without parallel data (I assume that means paired data with the same speech content but different accent etc.). They achieved this by disentangling the latent representations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper looks at the challenging problem of converting speech so that it appears to be produced by a speaker with a different accent from the source. The paper looks specifically at pitch and rhythm modelling. The reviewers agreed that the good performance of the proposed approach to a relevant task was a strength. But they also all highlighted that parts of the paper were difficult to follow, especially with regard to missing technical details. Although the authors provided follow-up explanations in the rebuttal period, this did not alter the reviewers' opinion regarding the clarity. Nevertheless, despite ambivalent excitement, the reviewers seem to agree that the work is generally sound.
Decision: Accept-Findings