StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion (2024)

Abstract: Recent language model (LM) advancements have showcased impressive zero-shot voice conversion (VC) performance. However, existing LM-based VC models usually apply offline conversion from source semantics to acoustic features, demanding the complete source speech and limiting their deployment to realtime applications. In this paper, we introduce StreamVoice, a novel streaming LM-based model for zero-shot VC, facilitating real-time conversion given arbitrary speaker prompts and source speech. Specifically, to enable streaming capability, StreamVoice employs a fully causal context-aware LM with a temporalindependent acoustic predictor, while alternately processing semantic and acoustic features at each time step of autoregression which eliminates the dependence on complete source speech. To address the potential performance degradation from the incomplete context in streaming processing, we enhance the contextawareness of the LM through two strategies: 1) teacher-guided context foresight, using a teacher model to summarize the present and future semantic context during training to guide the model’s forecasting for missing context; 2) semantic masking strategy, promoting acoustic prediction from preceding corrupted semantic and acoustic input, enhancing context-learning ability. Notably, StreamVoice is the first LMbased streaming zero-shot VC model without any future look-ahead. Experiments demonstrate StreamVoice’s streaming conversion capability while achieving zero-shot performance comparable to non-streaming VC systems.

heading: 1 Introduction
text: Voice conversion (VC) aims to transfer a speaker’s voice to that of another speaker without changing the linguistic content. This technique has been deployed in many real-world applications, such as
*Corresponding author
Synthesis: Conversion &
Reconstruction
Streaming Zero-shot VC (Recognition-Synthesis)
User A
User B
Desired Speaker Timbre
...
This syst m can convert my voice to arbitrary speakers in real time! And only one utterance of the speaker is needed.
User A
This system can convert my voice to arbitrary speakers in real time! And only one utterance of the speaker is needed.
Wow! There is almost no delay.
movie dubbing, privacy protection, pronunciation correction, etc. With the help of neural semantic features, such as the bottleneck feature (BNF) from an automatic speech recognition (ASR) system, converting source speech from arbitrary speakers in the wild has been successfully achieved (Sun et al., 2016). Meanwhile, converting to an arbitrary target speaker with only one utterance of this speaker, called zero-shot VC, has also been researched recently (Qian et al., 2019; Wang et al., 2023c). However, most existing zero-shot VC models are designed for offline systems, which are insufficient to meet the recent growing demands of streaming capability in real-time VC applications, such as live broadcasting and real-time communication (RTC). In this study, we focus on the streaming zero-shot VC as illuminated in Fig. 1.
Disentangling speech into different components, e.g., semantic content and speaker timbre, plays an important role in the zero-shot VC task (Chou and Lee, 2019; Wang et al., 2023d, 2021; Qian et al., 2019). Recently, benefiting from the powerful LM framework and the scaling up of training data, LM-based VC models (Wang et al., 2023c;
7328
Yang et al., 2023; Zhu et al., 2023) with built-in in-context learning ability can learn the context relations between source and target speaker’s utterances to capture fine-grained speaker timbre, achieving impressive zero-shot VC performance. However, demanding the complete source speech utterance limits these LM-based VC models to realtime scenarios; thus, they can only be used in offline applications. While several non-LM-based methods (Yang et al., 2022; Wang et al., 2023a) have been proposed for streaming zero-shot VC, the performance fails to generalize well to unseen speakers with high speaker similarity and speech naturalness, mainly due to the limited model capacity to scale up training data, and also the performance degradation caused by the missing future information in streaming scenario.
Inspired by the success of LM-based models in zero-shot VC, we aim to explore the feasibility of LMs for the streaming VC scenario. An intuitive way is to follow the popular recognize-synthesis framework shown in Fig. 1, in which speech is represented in semantic BNF and acoustic features respectively extracted by a streaming ASR and an audio codec. Then, the LM-based VC model undertakes the transformation of semantic information into acoustic features with the target speaker’s timbre. However, the development of the LM-based model in streaming zero-shot VC is hampered by two primary challenges.
• Streamable architecture: streaming models typically produce immediate output upon receiving current input without reliance on future time steps. Current LM-based VC models perform the conversion only when they get a full-utterance of source speech, which fails to meet the demands of streaming applications. The widely adopted multi-stage language modeling for multi-layer codec prediction introduces complexity to system design, posing a potential risk of cumulative errors. Additionally, the dependency models of the streaming pipeline also impact the design and performance of the VC model.
• Performance gap: unlike non-streaming models, streaming models must process frame-wise or chunked input causally on the fly without future information, facing missing context and potential performance degradation. This missing hinders the streaming VC model from achieving high-quality conversion. In addition, as shown in Fig. 1, the VC model relies on the semantic
feature BNF from ASR to achieve conversion, which makes semantic features very important. However, streaming ASR exhibits inferior performance compared to its non-streaming counterpart, leading to the BNF carrying low-quality semantic information but more speaker information. In addition to the inherent unavailable future reception, this low-quality semantic input makes achieving high-quality conversion more difficult. The goal of zero-shot VC amplifies the challenges faced by our streaming VC model.
In this work, we propose StreamVoice, a streaming LM-based model for high-quality zero-shot VC. Specifically, StreamVoice has a streamable architecture that integrates a single-stage language model that casually generates acoustic codecs with the collaboration of an acoustic predictor. Alternating input of semantic and acoustic features at each time step ensures seamless streaming behavior. Two methods are introduced to enhance the context-awareness of the LM to mitigate the performance gap caused by missing contextual information. 1) We incorporate a teacher-guided context foresight, where the VC model is taught by a teacher non-streaming ASR to infer the present and future semantic information summarized by the teacher, which is then used to enhance the acoustic prediction. 2) To enhance the context learning from the input history, semantic masking encourages acoustic prediction from the preceding acoustic and corrupted semantic input, which also implicitly creates an information bottleneck to reduce the source speaker’s information.
Experiments demonstrate StreamVoice’s ability to convert speech in a streaming manner with high speaker similarity for both seen and unseen speakers while maintaining performance comparable to non-streaming VC systems. As the first LM-based zero-shot VC model without any future look-ahead, the total pipeline only has 124 ms latency to perform the conversion, 2.4x faster than real-time on a single A100 GPU without engineering optimizations. Converted samples can be found in https: //kerwinchao.github.io/StreamVoice/.

heading: 2 Related Works
text: Zero-shot Voice Conversion. Zero-shot VC imposes stringent demands on speech decoupling and capturing speaker timbre. Many studies specifically design many disentanglement approaches, incorporating intricate structures (Chou and Lee, 2019),
loss functions (Wang et al., 2021), and training strategies (Ebbers et al., 2021), to achieve speech decoupling. Rather than embedding explicit disentanglement designs in VC training, some approaches (Gu et al., 2021) leverage a speaker verification (SV) model for speaker representation, while linguistic content is extracted using ASR or self-supervised learning (SSL) models (Sun et al., 2016; Choi et al., 2021). To enhance speaker timbre capturing, some fine-grained speaker modeling methods have also been explored (Yin et al., 2021; Wang et al., 2023d). Recent successes of language models in generative tasks have prompted the exploration of LM-based models in zero-shot VC, yielding impressive results. Using the pretrained model to decouple speech, the LM-based VC model (Wang et al., 2023c; Yang et al., 2023; Zhu et al., 2023) captures fine-grained speaker timbre from the speaker prompt and then performs the conversion. However, current LM-based VC models are inapplicable to streaming scenarios, constraining their real-world utility. This paper addresses this gap by investigating the zero-shot capabilities of language models specifically tailored for streaming scenarios. Streaming Voice Conversion. Despite the highquality conversion achieved by non-streaming VC models, their non-streamable structure and reliance on full-utterance input hamper them for real-time streaming applications. For streaming, causal processing and the structure of the streaming pipeline are crucial considerations. Streaming models are compelled to process frame-wise or chunked input on the fly, devoid of access to future information, leading to performance degradation compared to non-streaming counterparts. To address this, a common approach (Hayashi et al., 2022; Kameoka et al., 2021; Ning et al., 2023, 2024) involves the integration of a teacher model to guide the training of the streaming model or the distillation of knowledge from a non-streaming model. Chen et al. (2023b) focus on selecting BNF with minimal semantic information loss through layer-wise analysis, while Chen et al. (2022) incorporate adversarial training to enhance the quality of semantic features. Beyond streaming VC, some efforts have recently been towards streaming zero-shot VC. For instance, VQMIVC (Wang et al., 2021), designed for the nonstreaming application, is modified to be streamable by Yang et al. (2022). ALO-VC (2023a) constructs a streaming system using an SV model, a streaming PPG extractor, and a pitch extractor. However, current streaming zero-shot VC, designed for lowresource devices, has limited model capacity with poor generalization to unseen speakers, leading to inferior similarity and naturalness. Motivated by LM’s successes in zero-shot VC, we design a streamable LM in streaming scenarios. To tackle distinctive challenges in streaming VC, we enhance the context awareness of the LM to improve conversion quality. Language Model-based Speech Generation. Recent advancements in LMs within natural language processing have showcased potent generation capabilities, influencing the development of LMs in speech generation. By employing codec (Zeghidour et al., 2021) or other SSL models (Chung et al., 2021), speech and audio can be efficiently tokenized into discrete units, facilitating low-bitrate audio representation and semantic extraction. This progress allows speech generation to utilize LM frameworks seamlessly. Taking audio generation as a conditional language modeling task, AudioLM (2023) employs hierarchical language modeling for acoustic prediction from coarse to fine units. VALL-E (2023b) and SpearTTS (2023) extend LMs for zero shot-TTS, which can clone a human’s voice with prompt tokens from a short recording. For zero-shot VC, LM-VC (2023c) employs task-oriented optimizations to this task. And some studies (Zhu et al., 2023; Yang et al., 2023) leverage multitask objectives and datasets, achieving high-quality conversion. Despite this progress, existing LM-based VC models usually apply offline processing, demanding complete utterance from the source speech, which hinders their suitability for real-time streaming applications. In contrast to prior studies, we explore the zero-shot capability of the LM-based VC for streaming scenarios. With the enhancement of context awareness, the proposed LM-based VC model achieves results comparable to non-streaming LM-based VC.

heading: 3 StreamVoice

heading: 3.1 Overview
text: As shown in Fig. 2, the development of StreamVoice follows the recognition-synthesis framework. In this framework, speech is first represented as semantic features s = {s1, s2, ...sTs} and acoustic features a = {a1, a2, ..., aTa} by a pre-trained streaming ASR model and a speech codec model respectively. Here, Ts and Ta denote the sequence length. Before inputting to
StreamVoice, s and a are aligned to the same length T . StreamVoice incorporates a context-aware language model and an acoustic predictor to perform a single language modeling process. With the semantic and acoustic features {s̃, ã} of speech from the target speaker as speaker prompt, the LM leverages the semantic information s1:t of source speech to autoregressively predict the hidden output ch. In each autoregression time-step of the LM, the acoustic predictor transforms the hidden output ch to the codec feature â of the converted speech. Finally, the codec model reconstructs the waveform from the predicted codec feature. In the following sections, we will introduce how to build a streamable LM for VC and how to ensure the high-quality conversation of this streaming VC.

heading: 3.2 Streamable Architecture
text: To perform streaming VC, a streamable architecture is necessary. In StreamVoice, the language model is carefully designed to perform full causal
processing in the VC task, and the acoustic predictor is designed to achieve frame-wise prediction without dependency on temporal information.

heading: 3.2.1 Fully Casual Language Model
text: As shown in Fig. 3, inspired by the success of the LM-based VC model, we intend to achieve streaming zero-shot VC by language models. In previous LM-based VC models (Wang et al., 2023c), the demand of the complete semantic feature s from source speech to achieve conversion hinders the deployment for real-time application, which can be formulated as p(at|s1:Ts ,a1:t−1) for each time step. To achieve streaming, any components of the LM cannot rely on future information. As shown in Fig. 3, decoder-only LM with unidirectional attention can easily fit the requirement of casual generation. To eliminate the dependency of the complete semantic input, semantic and acoustic features {s,a} are first aligned with each other to the same sequence length T and then they are alternatively inputted to the LM, forming a crossembedding like {s1, a1, s2, a2, ..., sT , aT }. With these modifications, the LM can achieve streaming processing, modeling p(at|s1:t,a1:t−1).
To be specific, the semantic feature s obtained via an ASR model comprises a sequence of embeddings, denoted as {s1, s2, ..., sT }. On the other hand, the codec tokens obtained from an L-layer codec are discrete units represented by a ∈ RT×L. To obtain the acoustic embedding sequence, the codec tokens from each layer undergo separate embedding into the embedding space, and then they are concatenated along the embedding dimension, resulting in the fused acoustic embedding. Both the fused acoustic embedding and semantic features are transformed into the same dimension using linear layers. Subsequently, they are alternately inputted into the language model, forming the cross-embedding.

heading: 3.2.2 Acoustic Predictor
text: As the preceding LM has essentially encoded content and speaker into its output ch, the acoustic predictor can be designed in temporal irrelevant to transform ch into acoustic codec space, which means the predictor can be easily applied in the streaming scenario. Given that the speech can be represented in acoustic features by neural codec in either continuous or discrete forms, we investigate the incorporation of both features in StreamVoice, which are performed by continuous projection and discrete projection, respectively.
Continuous Projection. Following Shen et al. (2024), the D-dimensional quantized latent vector a ∈ RT×D encoded by the codec model is used as the continuous acoustic representation. The prediction of the continuous representation involves employing a stack of linear layers, as shown in Fig. 4. The continuous projection loss is calculated as the L2 distance between the predicted acoustic feature â and the ground-truth acoustic feature a, which is defined as:
LCont = ||a− â||22. (1)
Discrete Projection. In general, the codec is designed with multi-layer quantizers to compress original speech into L-layer discrete indices a ∈ RT×L at a low bitrate. Most LM-based work (Wang et al., 2023b,c) stacks multiple LMs to predict discrete features, making the pipeline complicated and unsuitable for the streaming scenario. In contrast, StreamVoice adopts a streamlined multi-layer codec prediction method inspired by MQTTS (Chen et al., 2023a). This method, free from temporal dependencies, can seamlessly integrate into the streaming process of the language model. Specifically, a single-layer transformer is used to model the hierarchical conditional distribution of codecs. As depicted in the right of Fig. 4, at time t, the transformer employs the ch as the starting condition and sequentially generates alt from layer 1 to L. Remarkably, this generation process is independent of the preceding or the future ch, rendering it well-suited for the demands of a streaming scenario. Notably, in the proposed StreamVoice, we mainly incorporate the discrete projection to achieve acoustic prediction. The discrete projection loss can be described as:
LDisc = −log T∏
t=1
L∏
l=1
p(alt|a1:t−1,m s1:t, t, a1:l−1t ). (2)

heading: 3.3 Context-aware Enhancement
text: Due to the disadvantage of the causality in the streaming framework, streaming models face missing future reception and potential performance degradation compared to the non-streaming model, while the low-quality semantic input from the streaming ASR, as we mentioned in Section 1, makes achieving high-quality conversion more challenging. To address these issues, a contextaware enhancement method is proposed, which can alleviate incomplete contextual information arising from the semantic input and the absence of future information. Specifically, we introduce contextmasked autoregressive prediction in the LM to enhance the capture of historical context from the given semantic input. Meanwhile, a teacher-guided context foresight is proposed to ensure the model can imagine the future context based on that of its historical context.
Context-masked Autoregressive Prediction. As shown in the left of Fig. 3, the LM is achieved by the multi-layer Transformer with unidirectional attention, following the implementation of LLaMA (Touvron et al., 2023). To enhance contextual awareness from the given semantic input, semantic masking is introduced in the LM to encourage acoustic prediction from the corrupted semantic. Specifically, within a sequence of semantic tokens s = {s1, s2, ...sT }, we randomly select several indices as start indices at a ratio r, and spans of l steps are masked by [M]. After masking, LM takes the corrupted semantic feature ms as input and performs autoregression. With this method, an information bottleneck is also implicitly created in the semantic feature to reduce speaker information. Moreover, during training, we do not explicitly use a speech clip as the speaker prompt. Instead, LM leverages the previous sequence {s1:t−1,a1:t−1, st} as prompts to autoregressively generate hidden representation ht for further acoustic prediction. Notably, when the current input is at, the corresponding output is skipped and does not involve further steps.
Teacher-guided Context Foresight. As previously discussed, the absence of future information resulting in the loss of contextual information leads to a decline in the conversion performance. Inspired by the effective representation learning exhibited by autoregressive predictive coding (2019) (APC), we introduce teacher-guided context foresight guided by a non-streaming ASR to enhance
the autoregression output, as presented in the right of Fig. 3. This allows the model to learn a context vector containing envisioned future information. Specifically, the context representation c is first derived by linear prediction from the hidden features h, which is generated by the LM through historical context. Subsequently, this ct is encouraged to discover more general future context information by minimizing the L2 distance not only with k semantic features from future time steps st+1, ..., st+k but also with the current semantic st. This dual minimization approach contributes to precise content delivery and enhances the ability to forecast future context. The loss can be summarized as
LTF = 1 T − k
T−k∑
1 ∥ct − Concat(st, st+1, ..., st+k)∥22 (3)
where Concat(·) denotes the concatenation of features along the dimensional axe. Unlike the original APC, which operates between the input and output of an autoregressive model, our approach employs a non-streaming ASR model as a teacher to provide semantic information s for guiding this foresight process. This is done to tackle the inherent challenge of obtaining high-quality semantic features from the streaming ASR. After dimensional transformations, the context representation c is then combined with h to form the context-enhanced ch, which is then fed into the acoustic predictor.
Furthermore, since the semantic feature {s, s} still may contain speaker-related information. To further ensure the speech decoupling, the bottleneck regulator (Qian et al., 2019), which squeezes out speaker information by reducing dimension size with a linear layer, is applied in s and c.

heading: 3.4 Training & Inference Procedure
text: Training. During training, the context-enhanced language model and acoustic predictor are trained together. The total loss can be described as Ltotal = LTF + LCont for continuous codec or Ltotal = LTF + LDisc for discrete version. Streaming Inference. We use the semantic and acoustic features from a short speech clip of the target speaker as the speaker prompt. Since this clip is randomly selected, which may contain unfinished pronunciation at the end of the clip, we pad a silence clip after the speaker recording before the conversion process to prevent the unexpected continuation. With this prompt, StreamVoice can stream convert the source speech. In discrete projection, we use greedy decoding to choose the
codec token with the highest probability. Besides, to ensure the real-time streaming inference of StreamVoice, we employ the key-value cache in LM to reduce redundant calculations. In practice, since the beginning and end of the source speech can be determined by ASR or voice activity detection (VAD), we don’t employ techniques, such as window attention or slide attention, to handle the input. StreamVoice’s performance decreases when the long input exceeds the maximum training length. Notably, these techniques can be easily integrated into our framework, providing flexibility for future extensions.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Corpus. A mixed dataset comprising 1,500 hours of Aishell3 (Shi et al., 2021) and an internal Chinese dataset are used to train StreamVoice and Audiodec (Wu et al., 2023). The internal dataset contains recordings from 2679 Chinese speakers, while we use utterances from 200 speakers in Aishell3. To extract semantic features, we incorporate a streaming ASR Fast-U2++ (Liang et al., 2023), which is implemented by WeNet (Yao et al., 2021) and trained on WenetSpeech (Zhang et al., 2022). For zero-shot testing, a set of 400 testing pairs is selected from DIDISpeech (Guo et al., 2021) and EMIME (Wester, 2010), each with a source and target speaker utterance. For evaluation of seen speakers, eight speakers from Aishell3 are selected to form 160 conversion pairs. And 3s speech utterance is used as a speaker prompt in inference. The duration of testing utterances is between 3s and 7s. Implement Details. We use open-sourced code1 of Audiodec, which has 4 quantizer layers with a 1024 codebook size and 64 codebook dimension, representing a 24kHz waveform in 20ms frame length. The Fast-U2++ uses an 80ms chunk size to perform streaming inference and compresses a 16kHz waveform into a semantic feature with a 40ms frame length. StreamVoice contains 101M parameters. For context-enhanced LM, we employ the variant of Transformer, LLaMA (Touvron et al., 2023), with 6 layers and 8 heads. The hidden and intermediate sizes are 1024 and 4096. We use the officially released code2 to implement the acoustic predictor, which uses a layer Transformer decoder with a hidden size 256, feed-forward hidden size
1https://github.com/facebookresearch/AudioDec 2https://github.com/b04901014/MQTTS
1024, and 4 heads. In semantic masking, mask ratio r ranges from 0.01 to 0.02, and span l is set to 10. The foresight step k is set to 4. The bottleneck regulator compresses feature dimensions by 6 times. During training, the max training length is set to 12s. StreamVoice is trained using 8 V100 GPUs with a batch size of 7 utterances per GPU for 700k steps. We use the AdamW optimizer with a learning rate of 5 × 10−4. Exponential decay updates the learning rate after each epoch, using a decay ratio of 0.986. Evaluation Metrics. The mean opinion score (MOS) subjectively measures speech naturalness (NMOS) and speaker similarity (SMOS), which are calculated with 95% confidence intervals. We randomly select 120 testing pairs for subjective evaluations involving a group of 15 listeners. For objective evaluations, a neural network-based system with open-source implementation3 is used to measure speech quality (WV-MOS). Character error rate (CER) measured by an ASR model4 indicates the speech intelligibility. Speaker similarity (SSIM) is calculated by an SV model (Desplanques et al., 2020) to determine if the converted speech matches the target speaker. Real-time factor (RTF) and latency indicate the streaming performance.

heading: 4.2 Experiments Results

heading: 4.2.1 Zero-shot Evaluation
text: To evaluate the zero-shot VC performance, one recent LM-based zero-shot VC system, LMVC (Wang et al., 2023c), is selected as the topline system. Besides, a variant of StreamVoice, referred to as NS-StreamVoice, using a non-streaming ASR for semantic extraction, is also compared. We implement the proposed system StreamVoice integration discrete projection, while C-StreamVoice
3https://github.com/AndreevP/wvmos 4https://github.com/wenet-
e2e/wenet/tree/main/examples/wenetspeech
also involves the evaluation since speech can represented in continuous form by codec model. Table. 1 presents both subjective and objective results. Compared with the non-streaming topline LM-VC, our proposed StreamVoice can achieve close results regarding subjective NMOS and SMOS, while a performance gap still exists. Similar results are also observed in objective results. The non-streaming StreamVoice even surpasses the topline model in certain aspects, indicating the effectiveness of our streamable architecture for zero-shot VC. Additionally, C-StreamVoice exhibits inferior performance compared to the discrete version, which can contribute to the over-smoothness in speech generation (Ren et al., 2022) and the mismatch between the ground truth and predicted features.
As illustrated in Table. 2, the RTF of the entire pipeline is below 1, which meets the real-time requirement. Consisting of chunk-waiting latency (80ms) and model inference latency, the overall pipeline latency is 124.3 ms. If using a V100 GPU, StreamVoice can obtain an RTF of 0.56, and the overall latency reaches 137.2 ms. Importantly, unlike previous streaming VC, our VC model is entirely causal without any future look-ahead, highlighting its powerful modeling capability. These results show that StreamVoice can achieve highquality zero-shot VC in streaming scenarios.

heading: 4.2.2 In-dataset Evaluation
text: To get further insight into StreamVoice, we conducted an in-dataset evaluation on eight seen speak-
ers, as shown in Table. 3. A non-streaming VC system (Tian et al., 2020) achieving any2many VC, is selected, referred to as NS-VC. Also, IBFVC (Chen et al., 2022) and DualVC2 (Ning et al., 2024) are recently proposed streaming models for any2many VC. As observed, a performance gap exists between the strong non-streaming topline and streaming models. Among the streaming models, StreamVoice, designed for the zero-shot scenario, delivers similar results to systems designed for indataset speakers, even though StreamVoice uses a smaller chunk size of 80ms in streaming ASR, achieving lower ASR performance. In contrast, IBF-VC and DualVC2 employ 160ms chunk size of ASR for streaming VC. It indicates StreamVoice’s good conversion ability. With available utterances of target speakers, fine-tuning yields superior performance. This indicates our system can be easily applied to various scenarios with or without the utterances of target speakers.

heading: 4.3 Ablation Study
text: As presented in Table 4, we conducted several ablations studies. In w/o teacher-guided context foresight, we discard the prediction of current and future semantic information, forming two ablations w/o LTF (st) and w/o LTF (st+1:t+k). As can be seen, a noticeable decrease occurs in all evaluation metrics when the LTF (st+1:t+k) is discarded, especially in WVMOS and CER. This indicates that this foresight improves performance in capturing the linguistic content. But when only integrating context from future semantics, the model w/o LTF (st) faces severe performance loss. It shows that only using future information interferes with delivering current linguistic content. In w/o semantic masking, we observe a performance decrease in all evaluation metrics when the semantic masking is discarded. This indicates that StreamVoice, trained with semantic masking, effectively enhances contextual learning from the preceding input while improving speaker timbre capturing. Furthermore, the results
of dropping the bottleneck regulator show that its integration effectively prevents the source speaker information contained in the semantic feature from leaking into the converted speech, with little effect on speech quality.

heading: 4.4 Discussion: Dependency Analysis
text: In this section, we will explore the dependency relations between the selection of ASR and codec and the performance of StreamVoice.
ASR. To investigate the impact of ASR on StreamVoice, three representative ASR systems, including non-streaming ASR 4, widely used CTCbased streaming ASR (Moritz et al., 2019), and the recently proposed streaming Fast-U2++ (Liang et al., 2023), are selected to perform semantic extraction. As can be seen in Table 5, StreamVoice using semantic features of non-streaming ASR outperforms those using streaming ASR. This discrepancy may be attributed to the inherent performance gap between non-streaming and streaming ASR models, resulting in different semantic extraction abilities. Besides, without future look-ahead in StreamVoice, using semantic features from (Moritz et al., 2019) cannot achieve reasonable conversion, while we introduce a 160ms future look-ahead in StreamVoice, i.e., modeling p(at|a1:t−1, s1:t+m, t) with m future look-ahead, yield good conversion results. This issue may arise from delayed CTC spike distributions and token emission latency existing in streaming ASR (Liang et al., 2023), leading to semantic information shifting. Benefiting from the low emission latency of Fast-U2++, StreamVoice can perform conversion without future look-ahead. With a longer chunk size employed in Fast-U2++, StreamVoice can obtain better results while reaching a larger latency of 270ms. A trade-off still exists between performance and speed.
Codec. In StreamVoice, we employ a lowlatency streaming codec Audiodec (Wu et al., 2023). As presented in Table. 6, we validate the
performance of StreamVoice using codecs with different bitrates, including 2kbps and 8kbps, where higher bitrate codecs achieve superior reconstruction quality to lower bitrate ones. The 2kbps Audiodec utilizes 4 layers of quantization and represents audio with a frame length of 20ms, while the 8kbps Audiodec employs 8 layers with a frame length of 10ms. Using the configuration of StreamVoice mentioned in Section 4.1, the results in different bitrates of codec models show no obvious differences. When increasing the number of transformer layers in the codec predictor, forming Large w/ 8kbps Audiodec, conversion performance using 8kbps codec improves noticeably, but resulting in slower inference. This result shows that the design of StreamVoice depends on the codec configuration, affecting both conversion quality and inference speed.

heading: 5 Conclusions
text: This paper introduces StreamVoice, a novel LMbased zero-shot VC system designed for streaming scenarios. Specifically, StreamVoice employs a single-stage framework encompassing a contextaware LM and an acoustic predictor. The casual design of the model’s input and structure ensures compliance with streaming behavior. To address performance degradation caused by missing complete contextual information in streaming scenarios, context-aware LM adopts teacher-guided context foresight to make the model able to forecast the current and future information given by a teacher. Besides, semantic masking is introduced in LM to enhance context learning from historical input and facilitate better disentanglement. Finally, an acoustic predictor collaborates with the LM to generate the target speech. Experiments demonstrate that StreamVoice achieves streaming zero-shot VC while maintaining performance comparable to nonstreaming VC systems.

heading: 6 Limitations
text: We have to point out that StreamVoice still has limitations. In our configuration, StreamVoice
needs a GPU, such as V100 and A100, to achieve real-time streaming inference. The design of streaming VC heavily relies on the ASR and the speech codec as mentioned in Section 4.4. Besides, StreamVoice also faces the out-of-domain problem, which causes performance degradation for utterances with accents, strong emotions, or unseen recording environments. Our future work will first use more training data with diversity coverage to explore StreamVoice’s modeling ability. Also, we will focus on optimizing our streaming pipeline, such as high-fidelity codec with low bitrate and unified streaming model.

heading: 7 Ethics Statement
text: Since StreamVoice can convert source speech to desired speakers, it may carry potential risks of misuse for various purposes, such as spreading fake information or phone fraud. To prevent the abuse of the VC technology, many studies have focused on synthetic speech detection (Yi et al., 2022). Meanwhile, we also encourage the public to report the illegal usage of VC to the appropriate authorities.

heading: Future Work
text: Our future work will first use more training data with diversity coverage to explore StreamVoice’s modeling ability. Also, we will focus on optimizing our streaming pipeline, such as high-fidelity codec with low bitrate and unified streaming model