Convert and Speak: Zero-shot Accent Conversion with Minimum Supervision

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:09:53

Abstract: Low resource of parallel data is the key challenge of accent conversion(AC) problem in which both the pronunciation units and prosody pattern need to be converted.
We propose a two-stage generative framework "convert-and-speak" in which the conversion is only operated on the semantic token level and the speech is synthesized conditioned on the converted semantic token with a speech generative model in target accent domain. The decoupling design enables the "speaking" module to use massive amount of target accent speech and relieves the parallel data required for the "conversion" module. Conversion with the bridge of semantic token also relieves the requirement for the data with text transcriptions and unlocks the usage of language pre-training technology to further efficiently reduce the need of parallel accent speech data. 
To reduce the complexity and latency of "speaking", a single-stage AR generative model is designed to achieve good quality as well as lower computation cost. Experiments on Indian-English to general American-English conversion show that the proposed framework achieves state-of-the-art performance in accent similarity, speech quality, and speaker maintenance with only 15 minutes of weakly parallel data which is not constrained to the same speaker. 
Extensive experimentation with diverse accent types suggests that this framework possesses a high degree of adaptability, making it readily scalable to accommodate other accents with low-resource data. Audio samples are available at https://convert-and-speak.github.io/demo/

Main paper: 
 background 2.1 accent conversion: 
 For accent conversion task, there has not been a public parallel corpus that contains pairs of audios having the same contents yet coming from the same speakers in different accents. So mainly two kinds of methods are proposed to accomplish this task in the literature. One is to synthesize the dataset containing the pairs of audios in the same voice but in two different accents with another voice conversion model and learn the acoustic mapping between them to accomplish accent conversion. [29] build the golden speaker utterance by converting the general American-English speaker's voice to the source-accent speaker's with a pretrained source speaker's synthesizer. Then use this golden speaker utterance as the target to learn the mapping of the mel-spectrogram based on a seq2seq VC system. [17] use a pretrained VC model to build the parallel data and trained the AC model based on Tacotron [21] conditioned on the semantic representation extracted from wav2vec 2.0 [1] . This end-to-end mapping-based approach needs large amounts of data to achieve a good zero-shot ability and the auxiliary VC model usually needs to be fine-tuned on the AC data set to alleviate the error caused by voice conversion step. These methods also constrain the output to be generated with the same length of the input which limits the conversion quality since the prosody of the speech is largely affected by the accent.Another approach is to regard accent conversion as a decomposition and resynthesis task in which the accent is separated from content, speaker identity, prosody and resynthesize to the target waveform in a TTS manner. [15] disentangles different features in multi-stage with several off-the-shelf models. Specifically, an accent-robust ASR model is trained using source accent speech with text labels to separate the source accent from the content. A multi-speaker TTS model with a global speaker encoder is trained with a large corpus of target-accent speech to map the accentagnostic linguistic features to acoustic features with the voice of source speaker and target accent prosody. Similarly, [32] learns the semantic embeddings directly from the accent speech with the supervision of text embeddings extracted from text labels. Such two-stage mapping-based non-parallel AC approach relieves the burden on the parallel data but needs to leverage large amounts of (text-accent speech) data and dedicated auxiliary tasks. With regard to performance, these methods are not good enough in conversion quality and diversity. It also suffers from poor zero-shot generalization ability with limited high-quality data available. The same accent speaker is used in their training and testing. Another work [9] treats the decomposition and resynthesis in an end-to-end manner. It designs a Pseudo Siamese Disentanglement Network (PSDN) with two streams in which one stream is used to learn the acoustic feature of target accent speech and the other auxiliary stream is used to build the information gap with the target stream to disentangle the content with accent, complemented with another adversarial accent classifier with gradient reversal layer(GRL). This framework can be used in the zero-shot scenario but the performance is not clear since their demo page is out-of-date and the metrics in their paper can not be compared with other public works.Compared to prior AC approaches, the proposed generative framework neither requires large amounts of parallel data spoken by the same speaker nor supervision from text labels or auxiliary tasks to achieve a good conversion quality.Recently, the speech generative models show large potential in generating contextual consistent, natural and diverse audio/speech based on a speech neural codec with the in-context learning of a referenced prompt. AudioLM [2] , designed for zero-shot audio generation, uses Soundstream [26] codes as intermediate representation of acoustic features for speech synthesis. It also shows the strong ability of in-context learning with a short prompt to maintain acoustic information such as speaker identity, prosody style and acoustic environment in the continuations. VALL-E [23] , verified in the TTS task, based on Encodec [4] tokens has also shown better zero-shot ability, speech naturalness and diversity than non-generative based TTS models.For model structure, these models take the decoder-only autoregressive Transformer structure [22] to build the conditional correlation of the acoustic features tokenized by a neural speech codec and the semantic tokens/phoneme sequences. When generating codes, they usually need multiple stages since the neural codecs they use are built on residual vector quantization(RVQ) which consists of a hierarchy of all VQs. In AudioLM, the first several quantizer layers are predicted in the first stage to get the coarse information of the speech and the rest layers are predicted based on the coarse layers to get the fine details of the speech. VALL-E simplifies the generating process by replacing the second stage with nonautoregressive(NAR). In its design, the first quantizer is generated with the AR model and the others are generated with the NAR model(all frames are predicted simultaneously when predicting each codebook) based on the previous quantizers.Different from these existing generative models, we propose a one-stage AR generative model which generates all VQs in one step to achieve lower complexity and latency as well as better quality. 
proposed framework 3.1 overview: 
 In the proposed framework, as shown in Figure 1 , the pronunciation patterns are converted at discrete semantic token level and the prosody is re-synthesized in target accent with a speech generative model. Specifically, we use a pre-trained self-supervised speech representation model, e.g. HuBERT [6] to extract discrete semantic tokens. A neural codec based speech generative model is used to generate the acoustic codes of the codec conditioned on the converted semantic tokens with the style prompt to maintain the source speaker's voice. Both the conversion and generative models are based on autoregressive decoder-only Transformer structure. More details are discussed in Section 3.2 and Section 3.3.The conversion module is designed as a seq2seq task in discrete semantic token space in which the source accent semantic tokens are converted to the target accent semantic tokens iteratively in an autoregressive manner. To handle the shortage of parallel data, inspired by BART and T5 [12, 20] , we use large amounts of target accent data to pre-train the conversion module with a pretext task in our scenario. We then fine-tune the conversion module with a small amount of weakly parallel data.Pre-training. In our scenario, the pretext task is designed to build the probability space of discrete semantic tokens in the target accent domain so that the target accent semantic tokens can be generated according to its context of previous tokens in the target accent domain in a closed-loop manner. In this pretext task, the model is trained in a self-supervised manner which is to produce the original token sequence 𝑌 = {𝑦 0 , ...𝑦 𝑡 }, 𝑡 < 𝑇 conditioned on the corrupted token sequence Ȳ = { ȳ0 , ... ȳ𝑡 }, 𝑡 < 𝑇 , formulated asEQUATIONWe have experimented with corruptions like token masking, token deletion, and token in-filling and we find the token in-filling scheme works the best. Specifically, following the text in-filling scheme in BART [12] , a number of text spans are sampled according to 𝐵𝑒𝑟𝑛𝑜𝑢𝑙𝑖 (𝑝) where 𝑝 = 0.5. The span lengths are drawn from a Poisson distribution(𝜆 = 5). We train the pretext task with large amounts of target accent data which is available in the public corpus.Fine-tuning. Since some phonemes in the source accent need to be converted to the target accent ones, a mapping between these phonemes needs to be learned. Specifically, we fine-tune the pretrained conversion model conditioned on the semantic tokens in source accent with a small amount of weakly parallel accent data. Correspondingly, the training can be formulated asEQUATIONin which 𝑋 = {𝑥 0 , ...𝑥 𝑡 }, 𝑡 < 𝑇 is the source accent semantic token sequence and 𝑌 = {𝑦 0 , ...𝑦 𝑡 }, 𝑡 < 𝑇 is the target accent semantic token sequence.The target accent speech generation is achieved by training a separate generative model on a large target accent speech corpus. We design a new speech generative model based on TF-Codec [10] . This model generates acoustic tokens of TF-Codec iteratively through a single-stage causal speech generation, conditioned on the converted/target accent semantic tokens.3.3.1 Speech tokenizer with TF-Codec. We use the pre-trained causal speech neural codec TF-Codec to extract the acoustic token of each frame. Unlike [10] , we remove the predictive loop and use the non-predictive model at 6 kbps for efficient acoustic modeling with high-quality output. Specifically, the TF-Codec takes the 16kHz magnitude-compressed time-frequency spectrum with a window length of 20 ms and a hop length of 5 ms as input. Then a stack of 2D causal convolutional layers, followed by a temporal convolutional module (TCM) and a gated recurrent unit (GRU) block is used to capture the short-term and long-term temporal dependencies between the input frames in a causal manner. For the quantization, it combines 4 frames together, producing a frame rate of 50 Hz for quantization. Instead of using RVQ, it employs group quantization where the latent embedding is split into 𝐾 groups and each group is quantized by a vector quantizer with a codebook of 1024 codewords. All 𝐾 acoustic codes are concatenated and decoded to get the reconstructed waveform.As the group quantization in TF-Codec encodes each group independently, we leverage a single-stage causal speech generation to generate acoustic codes of all 𝐾 quantizers simultaneously for each frame. As shown in Figure 1 , TF-Codec token, which is the concatenated embeddings corresponding to all 𝐾 quantizers, is generated in one-stage autoregressive manner conditioned on the target accent/converted semantic tokens and style acoustic tokens. For each group embedding in TF-Codec token, the dimension is 𝐷 𝑡𝑜𝑘𝑒𝑛 /𝐾, in which 𝐷 𝑡𝑜𝑘𝑒𝑛 is the dimension of the embedding in transformer. 𝐾 classification heads are employed to predict the 𝐾 acoustic codes for current frame separately. The training target can be formulated asEQUATIONin which 𝑌 = {𝑦 0 , ...𝑦 𝑡 }, 𝑡 < 𝑇 is the semantic token sequence from target accent speech. 𝐶 : is TF-Codec token sequence of target accent speech. 𝐶 : is the TF-Codec token sequence of style acoustic prompt. We do not distinguish 𝐶 : from 𝐶 : in training. The concatenation of 𝐶 : and 𝐶 : is a whole sequence. During inference, the first 3 seconds of the source speech is used as 𝐶 : . 
experiments: 
 To evaluate the performance of the proposed framework, we take Indian-English as source accent and general American-English as target accent which is a common scenario in the research literature.Dataset. To train the speech generative model and pre-train the conversion model, LibriTTS dataset [27] is used as our training data. The dataset contains approximately 585 hours of general American-English speech data, sourced from audiobooks available on the public LibriVox project. To fine-tune the conversion model, L1-L2 ARCTIC dataset [11, 31] is used. L1-L2 ARCTIC dataset is a dataset with accent speakers speaking the same content. To build the parallel data, we select a general American-English speaker named "bdl" as the target accent speaker and "ASI" as the Indian-English speaker. Among all their utterances, 1000 utterances, about 50 minutes of speech are used in the training, 50 utterances are used in validation and the remaining 100 utterances are used for testing.To better verify the zero-shot ability, we also add speaker p248 from VCTK dataset and another 4 Indian-Englsh speakers which are not used in training from L1-L2 ARCTIC dataset into testing.To be noted that the 20 utterances from speaker p248 in VCTK are used to compare with the existing machine-learning based AC method [15] . Besides these 20 cases, we add another 20 utterances of each testing speaker in L1-L2 ARCTIC for objective evaluation, e.g. 120 cases in total, and random 8 utterances per speaker for subjective evaluation, e.g. 60 cases in total. Model and configuration. For semantic tokenizer, we employ the HuBERT-Base model1 and k-means algorithm with 500 clusters to extract semantic tokens. It is trained on LibriSpeech [18] mostly consisting of general American-English. It generates a discrete semantic token sequence at 50Hz framerate for 16kHz audio. Previous studies [7, 19] show that HuBERT is a good representation of speech content and removes most of the speaker identity so that we can use the weakly parallel data in the fine-tuning stage of the conversion module. For the acoustic tokenizer, the number of quantizers in TF-Codec (𝐾) is set to 16. The transformer used in conversion model and generative model is the same structure with 12 layers of 16 attention heads, a feed-forward layer with dimension of 4096, and a dropout layer with rate of 0.1. The embedding dimension in transformer(𝐷 𝑡𝑜𝑘𝑒𝑛 ) is 1024. The generative model and pre-training stage of conversion model are trained on 8 NVIDIA TESLA V100 32GB GPUs with a batch size of 4k tokens per GPU. The ScaledAdam [25] optimizer is used. The learning rate is set to Baseline models. To show the superiority of the proposed framework, we select 3 models as our baselines. The existing machinelearning based AC method [15] , which is the best model available in the public to our knowledge. The generative-only models without the conversion module are also used as our baselines, in which we compare with the commonly-used EnCodec-based multi-stage generative model and the proposed single-stage TF-Codec based generative model.Evaluation methods on accent similarity. To evaluate the performance of accent conversion, both objective and subjective metrics are used. Intuitively, we use the metric Longest Common Subsequence(LCS) to evaluate the similarity of the converted semantic token sequence and referenced target semantic token sequence. To eliminate the disturbance of the duration of each word, the duplicated tokens in the sequence are removed. To remove the effect of the utterance length in the final average statistics, the Longest Common Subsequence Ratio (𝐿𝐶𝑆𝑅 = 𝐿𝐶𝑆/𝑢𝑡𝑡𝑒𝑟𝑎𝑛𝑐𝑒_𝑙𝑒𝑛𝑔𝑡ℎ) is used. The smaller utterance length is used to calculate the LCSR of the testing pair. We also use the latest state-of-the-art English accent classification model CommonAccent2 [33] to identify the accent of the synthesized speech. Besides, we conduct a subjective A/B testing in which participants are asked to choose the one that sounds more close to general American-English accent in an A/B pair. Each A/B pair contains cases chosen from any two of the competitors. The accent source and ground truth are also included to ensure the validity of the testing. To remove the potential factor of speaker identity in the subjective testing, in Table 2 , both the accent source and referenced ground truth are chosen from multiple speaker's utterances, e.g. 5 Indian-English speakers and 4 general American-English speakers from L1-L2 ARCTIC test set. To be noted, the referenced ground truth is used here with the same sentence spoken by different speakers. Particularly, the participants are trained to distinguish the accent difference by listening to several pairs of <Indian-English, general American-English> samples before the formal testing. 20 participants who are proficient in general American-English are invited to conduct these evaluations. MOS-Accent, the percentage of being selected as general American-English accent is used as the metric of this subjective testing.Evaluation methods on speaker similarity and speech quality. To evaluate the speaker identity maintenance, the speaker similarity metric is calculated as the cosine similarity of the two speaker vectors, which are extracted from the source accent speech and the converted speech, correspondingly. WavLM-TDNN3 [3] , a state-ofthe-art speaker verification model, is used to get the speaker vector from a speech. To evaluate the naturalness, we use NISQA-TTS [16] , which is commonly used for synthesized speech. We also conduct MOS testing, in which the raters are asked to give a score ranging from 1 (lowest quality) to 5 (highest quality) according to the overall subjective quality. The MOS-Naturalness with confidence level of 95% is used as the metric.Accent similarity. As shown in Table 1 and Table 2 , the MOS-Accent metric of the proposed framework on both datasets ranks the highest and very close to the ground truth. Compared with the generative-only models, e.g. Generative-only model(EnCodec) and Generative-only model(TF-Codec), the proposed framework highly surpasses them for accent conversion performance, indicating the effectiveness of the conversion module. This is also verified by the LCSR metric on L1-L2 ARCTIC test set in Table 2 , where the proposed framework closely approaches the ground truth LCSR after conversion. The analysis on HuBERT with accent input in Section 4.5.1 also shows the HuBERT tokens are affected by the source Indian-English accent, especially for those accent speech with phonetic changes, indicating the necessity of the semantic conversion module. Figure 2 and Figure 3 show the accent classification results from CommonAccent. Compared with other methods, the proposed framework converts most of the Indian-English accent input to the target general-American English accent. In Figure 3 , for the rest of 11% cases which are identified as Indian-English accent, besides classification error, most of the cases are short and the conversion quality of some cases is not good enough which leaves room for further improvement on the robustness of the generative framework. For Liu. et al [15] 's method, it is interesting to find that there is a big gap between classification metric(as bad as accent In this section, we further reduce the parallel data used in the finetuning stage of the conversion model, from 50 minutes (proposed in Table 1 and Table 2 ) to 30 minutes and 15 minutes, respectively. We use the MOS-Accent as the evaluation metric and test on the VCTK test set. We also add the baseline models into A/B testing for comparison. As shown in Table 4 , by decreasing the data amount, the performance drop is negligible. With the minimum supervision of 15 minutes, the performance is still relatively good, which shows its high potential for extension on other accents with low-resource data, such as Chinese-English accent and Korean-English accent to general American-English accent cases in Section 4.7.4.5.1 HuBERT tokens from accent speech. In this section, we evaluate how accent affects the HuBERT tokens. Specifically, we build a parallel data set from L1-L2 ARCTIC dataset in which the Indian-English speaker and general American-English speaker speak the same content. Both of them are fed into the HuBERT model used in the paper to get the semantic token sequence. The LCSR metric is used to evaluate the content similarity between these two HuBERT token sequences. 1000 pairs are used in this experiment. To further study the phoneme change effect, the accent cases are divided into the one with phoneme changes and without phoneme changes. Since the speakers of each pair are different, the effect of the speaker identity on the LCSR metric is also calculated as a reference. As Table 5 shows, with the source accent introduced, HuBERT tokens have changed a lot, degrading from 0.747 to 0.569 in terms of LCSR.For those cases with specific phoneme changes, more tokens have been changed from the target accent references(LCSR: 0.541).4.5.2 Accent effect on the style prompt in speaking module. We use the accent source as the style prompt in the speech generative model to extract speaker identity of the source speaker. This section is to evaluate if the accent feature will be extended through the in-context learning in the speech generative model. We conduct empirical study to substantiate such usage. Specifically, we design an A/B testing to compare the accent similarity of the synthesized speech generated with two kinds of prompts in different accents conditioned on the same content. For testing, we take Indian-English and general American-English as two prompt types for comparison. We build 100 pairs of samples to test. Each pair contains an utterance in general American-English accent which is used to extract HuBERT semantic tokens, an utterance from a general American-English speaker and from an Indian-English speaker working as the style prompts. The prompts are cut to 3 seconds. Examples can be found in our demo page. For subjective testing, 20 participants who are college students majoring in American-English are asked to distinguish the two synthesized speech and choose the one which sounds closer to general American-English. The percentage of being selected as general American-English accent is used as the evaluation metric. We also use the CommonAccent to identify the two synthesized speeches. As Table 6 shows, no matter general American-English or Indian-English prompt type, the percentage to be selected as general American-English by users is about 50% and almost all the synthesized speeches are identified as general American-English. The similar results are observed from the testing with more accent prompt types as Chinese-English accent and Korean-English accent in Appendix(In-context learning with more accent prompt types). Furthermore, we find accent prompts do have effect on the prosody modeling but are quite limited. According to our experiments on the effect of accent prompt length in Appendix(Effects of accent prompt length), with the prompt in Indian-English accent becomes longer, increased from 3s to 7s, the percentage of predicted general American-English accent drops from 84% to 73%. So we can use 3 seconds of accent source as a prompt to catch the source speaker's identity without bringing the source accent back to the converted speech.Decoupling design We compare with the solution in which the parallel data is used to fine-tune the generative-only model directly. In such a way, the model is guided to learn the phoneme and prosody conversion simultaneously and blindly through an AR Transformer model. As Table 7 shown, the LCSR of generative-only CommonAccent metric shows the percentage of being predicted as general American-English. A/B testing metric shows the percentage of being selected as general American-English in the A/B pair.Prompt type CommonAccent A/B testing General American-English accent 98% 50% Indian-English accent 97% 50% 9 : Results on general American-English accent conversion from Chinese-English accent and Korean-English accent. CommonAccent metric shows the percentage of being predicted as general American-English.Chinese-English accent 95% Korean-English accent 100% model without decoupling on L1-L2 ARCTIC test set is quite low at 0.02, indicating most of the content has been destroyed and the model fails to learn such mapping with so little amount of parallel data.The effect of pre-training for semantic conversion module To verify the validity of the language pre-training technology used for the semantic conversion module, we compare it with the solution where the conversion model is trained from scratch with the weakly parallel data. All parallel data(about 50 mins) are used for training. As Table 8 shows, without pre-training, the results degrade by a large margin. This is reasonable since the pre-training stage lays a good foundation for the fine-tuning stage to just focus on learning few semantic units which are different in the two accents.We extend the proposed framework to Chinese-English accent and Korean-English accent. Specifically, another 15 minutes of weakly parallel data of <Chinese-English accent, general American-English accent> and <Korean-English accent, general American-English accent> from L1-L2 ARCTIC dataset is used in the fine-tuning stage of the conversion module. The conversion accuracy of Chinese-English accent is 95% and Korean-English accent is 100%, identified by the CommonAccent metric as shown in Table 9 . Audio samples can be found in our demo page. 
conclusions: 
 In this work, we propose a two-stage generative framework for accent conversion task in which the conversion is operated on the semantic token level and synthesized to a target accent speech with TF-Codec based generative model. Experimental results show the proposed framework achieves the state-of-the-art performance in terms of accent similarity, speech quality and speaker maintenance with limited parallel data. With the language pre-training technology, only 15 minutes of parallel data, not constrained to the same speaker reaches to a good conversion quality, which shows large potential for an easy extension for other accents with low-resource data. The proposed single-stage AR generative model achieves better speech quality at lower complexity, which can be used for other speech generative tasks. In the future, we will further improve the robustness of the generative framework for AC task. 
: 
 as intonation, rhythm and pronunciation patterns [8] . Take Indian-English for example, they may pronounce ′ 𝑣 ′ as ′ 𝑤 ′ or vice versa, ′ 𝑡ℎ ′ as ′ 𝑡 ′ or ′ 𝑑 ′ and ′ 𝑝 ′ as ′ 𝑏 ′ . Besides such pronunciation units difference, the prosody, e.g. intonation and stress is also changed a lot according to the accent. As the example in Appendix(Examples of accent speech), the pitch contour of the Indian-English accent is presented with more ups and downs. Another key challenge is the lack of parallel data. Strictly parallel data with one speaker speaking the same sentence with two different accents barely existed in the public research area. Some early researches [8] try to explicitly model the pronunciation patterns by building some accent-specific dictionaries to include all possible pronunciations of every word according to the accent type. These methods tend to have poor adaptation because of its assumptions that phonetic knowledge about every accent is available and can be depicted thoroughly in a dictionary and all speakers can be categorized into a few accent clusters are hardly held in the real scenario.One of the conventional AC methods [5, 13, 28, 30] simplify the accent conversion problem by just converting the voice of the target accent speaker to that of the source accent speaker assuming the utterance with the same content in target accent is available. This kind of method just requires to extract the speaker identity from the source speech without disentangling content with prosody which makes it easier to achieve accent conversion. However, these methods hinder their usage in real applications since the target accent reference is hardly available at conversion stage.Therefore, reference-free AC methods are more practical and appealing for usage. Some previous approaches [17, 29] try to learn the acoustic mapping between the source accent speech and target accent speech directly with the parallel data in which the same speaker speaks the same content with two different accents. However, such data is extremely rare. So the main idea of this kind of method is to use the voice conversion(VC) technology [14, 24] to synthesize the data set by converting the speaker identity of the target accent speech to that of the source speaker. Such end-to-end mapping-based methods require large amounts of strictly parallel data to achieve a good conversion quality and generalization ability. However, such massive high-quality data can hardly be achieved and the distortions are introduced from the VC stage, even though these VC models have been fine-tuned on the target AC dataset.To relieve the dependence of parallel data, another kind of approaches [9, 15, 32] which leverage disentanglement technology to remove accent from content, speaker identity, prosody and resynthesize to the target waveform through a synthesizer, e.g. text-tospeech(TTS) model [21] . The synthesizer is trained on the target accent speech to generate speech with prosody in target accent. To remove accent from content and speaker identity, some auxiliary models or tasks are carefully designed, e.g. accent-agnostic automatic speech recognition(ASR) model or phoneme classification task. Text transcriptions are largely used in these solutions to provide the supervision of accent-agnostic semantic representation. Such two-stage mapping-based methods still require large amounts of (text-accent speech) pairs combined with dedicated auxiliary tasks to achieve accent-agnostic semantic feature and generate diverse speech in target accent.In this work, we propose a two-stage generative framework with conversion stage and speaking stage to achieve accent conversion. The conversion stage is operated on semantic level by generating the semantic tokens in target accent from source accent. The speaking stage is using a generative-based synthesis model conditioned on the converted semantic tokens to generate the speech with prosody in target accent. Splitting the AC task into these two subtasks and realizing conversion with the bridge of semantic tokens which are extracted from raw speech enable the "speaking" module to be independent of parallel data and use massive amount of target accent speech without text transcriptions to generate speech with good quality and diversity in the target accent domain. Meanwhile, it makes it easier for the conversion part to just learn the pronunciation pattern/phoneme difference with a small amount of weakly parallel data which is not constrained to the same speaker.Both of the stages are seq2seq tasks based on decoder-only Transformer architectures [22] . Inspired by the ideas from machine translation community to reduce the need for supervision, we leverage the BART/T5-style pre-training [12] to significantly reduce the amount of parallel supervision required to train the conversion part. Such pre-training with a pretext task on target-accent data is designed to learn the pattern of generating semantic units, e.g. the joint probability of phonetic units in target accent space. Furthermore, to reduce the complexity and latency of the speech generation process, we design a single-stage autoregressive generative model which generates all vector quantizers(VQs) in one step based on TF-Codec [10] .Our contributions:(i)We propose a state-of-the-art generativebased framework for accent conversion which is capable of converting prosody pattern as well as pronunciation units, as evaluated with objective and subjective metrics on public Indian-English accent to general American-English accent conversion test set. (ii)We use the pre-training technology on the conversion part to largely reduce the amount of parallel supervision to only 15 minutes of weakly parallel data. (iii)This framework can be easily extended to other low-resource accents as our experiments on Chinese-English accent and Korean-English accent shown. (iv)We propose a singlestage speech generative model based on TF-Codec with better speech quality and speaker similarity at lower computation cost and latency compared with the multi-stage generation process in other generative models based on Encodec (proposed: 50 AR steps/1 sec of audio vs Encodec-based: 75 AR steps+7 NAR steps/1 sec of audio).The paper is organized as follows. Section 2 introduces the background of accent conversion and speech generative models. Section 3 introduces the proposed framework in detail. Section 4 validates the performance of the proposed framework mainly on Indian-English accent to general American-English accent and extensive experiments are undertaken to substantiate the efficacy of our model design. Section 5 concludes the paper. 
 
 Appendix: https://huggingface.co/facebook/hubert-base-ls960https://huggingface.co/Jzuluaga/accent-id-commonaccent_ecapahttps://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_ verification 


Decision: The authors propose a two-stage generative framework "convert-and-speak" in which the conversion is only operated on the semantic token level and the speech is synthesized conditioned on the converted semantic token with a speech generative model in target accent domain. The decoupling design enables the "speaking" module to use massive amount of target accent speech and relieves the parallel data required for the "conversion" module. Conversion with the bridge of semantic token also relieves the requirement for the data with text transcriptions and unlocks the usage of language pre-training technology to further efficiently reduce the need of parallel accent speech data. 

Strengths:
- The paper significantly alleviates the need for parallel data for accent conversion and is highly relevant for real world applications.
- The use of a two-stage generative framework efficiently decouples the tasks of pronunciation and prosody conversion, improving the flexibility and scalability of the model.
- The framework achieves strong results in accent similarity, speech quality, and speaker maintenance compared to existing methods.

Weaknesses:
- Most of the issues pointed by reviewers have been addressed by the rebuttal.

Reviews: review: paper_summary: paper_summary: The paper introduces a two-stage generative framework for zero-shot accent conversion, focusing on efficient use of limited parallel data. The approach decouples the conversion of semantic tokens from speech generation, which allows for leveraging massive amounts of non-parallel data to train a target accent speech synthesis model. This framework achieves state-of-the-art results in accent conversion tests between Indian-English and General American-English.
strength_weakness: strength_weakness: The method introduces innovative techniques for handling semantic tokens in accent conversion, reducing the reliance on extensive parallel datasets, which is a significant advance over previous approaches. The proposed method is particularly valuable for real-world applications where parallel data are scarce but there is a need for high-quality accent conversion.
limitations: limitations: For accent conversion applications, particularly those intended for real-time use, the efficiency of the model during inference is crucial. The paper does not discuss the Real-Time Factor (RTF) or the inference time required by the proposed system.
paper_summary: paper_summary: This paper proposes a two-stage framework for accent conversion, by generating semantic tokens in the conversion stage and generating speech conditioned on the semantic tokens (extracted by HuBERT) in the speaking stage, getting rid of requirements of large amounts of parallel data and text transcriptions. 

The proposed framework is demonstrated effective in converting prosody and pronunciation patterns. Pre-training technologies are adopted to reduce requirement of parallel data by training a Transformer-based decoder for masked token prediction. The decoder is further trained to convert source accent tokens to target accent tokens. The pretrained casual speech neural codec TF-Codec is used to extract acoustic tokens, which are further converted to speech with a single-stage causal speech generation.
strength_weakness: strength_weakness: This paper investigate the novel two-stage framework that leverages pretraining and finetuning to address the data scarcity problem in accent conversion. Experiments on two corpora demonstrate the effectiveness of the proposed framework.
limitations: limitations: The term “stage” is used to refer the “convert” and “speak” stages, and also used to refer to single stage casual speech generation. It would be clearer to use two different terms for them.

In Table 1 and Table 2, MOS-Accent for the reference ground truth is only 70% and 79.3%. Why is it so low for the two datasets? Because the other accent sources are easy to be discerned (hence, MOS-Accent for accent source is as low as 0.5%). Any analysis on this?

The current evaluations can support the effectiveness of the proposed framework in accent conversion. It would be helpful to add more analysis on the prosody patterns and pronunciation units conversion, e.g., by visualization of pitch contours, phonemes, to show the proposed framework’s advantages on prosody and pronunciation conversion.
paper_summary: paper_summary: This paper proposes a novel two-stage framework for accent conversion, aiming to transform speech from a source accent to a target accent. The first stage converts semantic tokens extracted from the source accent speech into target accent tokens. The second stage employs a generative model to synthesize speech in the target accent, conditioned on the converted semantic tokens.
strength_weakness: strength_weakness: 1) The framework leverages pre-training techniques to significantly reduce the amount of parallel data required for training.
2) The proposed single-stage autoregressive generative model for speech synthesis offers improved efficiency compared to existing multi-stage models.
limitations: limitations: The paper presents a promising approach to accent conversion, but it also has some limitations:
1) Although the authors aim to preserve speaker identity, the evaluation of speaker similarity relies on a speaker verification model not trained on accented speech. This could lead to inaccuracies in assessing speaker identity maintenance, as the model might be sensitive to accent-related features. 
2) While subjective A/B testing is conducted, the participant pool is limited to college students majoring in American English. This might introduce bias in the evaluation, as their perception of accent similarity might differ from that of a more diverse group.
paper_summary: paper_summary: The paper introduces "Convert-and-Speak," a two-stage generative framework for accent conversion using minimal supervision. The framework decouples the process into semantic token conversion and speech synthesis in the target accent, significantly reducing the need for parallel data. Experiments show state-of-the-art performance in accent similarity, speech quality, and speaker maintenance using only 15 minutes of weakly parallel data.
strength_weakness: strength_weakness: Reduces dependency on parallel data.
High performance in accent conversion with minimal supervision.
Extensible to other low-resource accents.
limitations: limitations: Some quality loss with reduced training data.
Limited robustness in handling diverse accent challenges.
Needs further improvement for real-world applicability.
paper_summary: paper_summary: The paper proposes a two-stage generative framework for accent conversion named “Convert and Speak.” The primary objective is to tackle the challenge of converting both pronunciation units and prosody patterns in speech from one accent to another with minimal supervision and data requirements. The approach is structured as follows:
1.	Semantic Token Conversion: The first stage involves converting the semantic tokens of the source accent to the target accent using a pre-trained self-supervised speech representation model, such as HuBERT.
2.	Speech Generation: The second stage involves synthesizing speech in the target accent using a single-stage autoregressive generative model based on TF-Codec.
The framework aims to reduce dependency on large amounts of parallel data and text transcriptions by leveraging pre-training techniques. Experiments demonstrate state-of-the-art performance in converting Indian-English to American-English, with the framework showing adaptability to other accents like Chinese-English and Korean-English.
strength_weakness: strength_weakness: 1. Minimal Data Requirement: The proposed method significantly reduces the need for parallel data, achieving high-quality accent conversion with only 15 minutes of weakly parallel data.
2. Generative Framework: The use of a two-stage generative framework efficiently decouples the tasks of pronunciation and prosody conversion, improving the flexibility and scalability of the model.
3. Pre-training Benefits: Incorporating pre-training (e.g., BART/T5-style) helps in reducing the amount of supervised data required for fine-tuning, thus enhancing the model’s performance with limited resources.
4. State-of-the-Art Performance: The framework achieves superior results in accent similarity, speech quality, and speaker maintenance compared to existing methods.
limitations: limitations: 1. This work applies the commonly used strategies from text-to-speech. The Minimum Supervision idea follows Google's SPEAR-TTS [1]. The overall framework is very similar to SPEAR-TTS. 
2. This paper propose a single-stage generation (parallel prediction all of tokens in one frame), which can improve the inference speed. But such strategy has been demonstrated that it may result in suboptimal generation performance in MusicGen [2] and UniAudio [3].
3.  Dependence on Specific Models: The performance heavily relies on the quality of the pre-trained models like HuBERT and TF-Codec, which might not be equally effective for all types of accents or languages.
4. Evaluation Scope: The evaluation is primarily focused on Indian-English to American-English conversion. Although some other accents were tested, a more extensive evaluation across diverse accents and languages would strengthen the findings.

In summary, this paper gives us a empirical study. From the machine learning perspective, this paper does not give enough insight or contributions. From the Accent Conversion task perspective, this paper gives a good results for this direction. If the authors willing to open-source their checkpoints to the community, it will be a good contribution. 

 I am happy to further discuss this paper with authors and other reviewers in the rebuttal stage.

[1] Kharitonov E, Vincent D, Borsos Z, et al. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision[J]. Transactions of the Association for Computational Linguistics, 2023, 11: 1703-1718.

[2] Copet J, Kreuk F, Gat I, et al. Simple and controllable music generation[J]. Advances in Neural Information Processing Systems, 2024, 36.

[3] Yang D, Tian J, Tan X, et al. Uniaudio: An audio foundation model toward universal audio generation[J]. arXiv preprint arXiv:2310.00704, 2023.
paper_summary: paper_summary: This paper addresses the challenge of accent conversion (AC) with limited parallel data. The proposed "convert-and-speak" framework tackles this by decoupling the conversion and speech generation processes. Conversion is performed only at the semantic token level, enabling the use of large amounts of target accent speech for speech synthesis. This design eliminates the need for parallel data for the "conversion" module and reduces the requirement for transcribed data. Furthermore, leveraging language pre-training technology further minimizes the need for parallel accent speech data.

To address complexity and latency, a single-stage AR generative model is employed for speech generation, achieving both high quality and efficient computation.
strength_weakness: strength_weakness: The paper introduces a novel two-stage framework that effectively tackles the data scarcity issue in accent conversion. This approach leverages the power of pre-trained language models and large target-accent speech datasets, making it highly efficient and adaptable.
Semantic Token Level Conversion: It reduces the dependence on parallel data and allows for leveraging large-scale language models for more robust and accurate conversion.
Decoupled Design:  It enables the use of separate and optimized models for each task, leading to improved performance and scalability.
Single-Stage AR Generative Model: The use of a single-stage AR generative model for speech synthesis reduces complexity and latency without compromising quality, making the system more practical and efficient.
limitations: limitations: Semantic Token Representation: The conversion process at the semantic token level might not fully capture nuanced prosodic features that are crucial for natural and accurate accent conversion. This could lead to a loss of fine-grained linguistic information during the conversion process.
So, 
1. Lack of Analysis on Acoustic Fine-grained Loss from HuBERT:
The paper proposes a novel "convert-and-speak" framework for accent conversion, incorporating the HuBERT model for speech representation. However, the analysis lacks a deeper investigation into the specific contributions of HuBERT's acoustic fine-grained loss to the overall performance. It would be beneficial to delve into how HuBERT's fine-grained acoustic features, particularly its ability to capture phonetic details, contribute to the accuracy and naturalness of the accent conversion.

2. Absence of Coarse-level Evaluation Metrics:
While the paper evaluates accent similarity and speech quality, it lacks an assessment of the coarse-level performance of the conversion system. This is crucial for understanding how well the framework translates the overall meaning and intent of the utterance. Metrics like Word Error Rate (WER) could be used to gauge the accuracy of the conversion system in capturing the semantic content of the speech.