SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network (2024)

Abstract: Brain-inspired Spiking Neural Network (SNN) has demonstrated its effectiveness and efficiency in vision, natural language, and speech understanding tasks, indicating their capacity to "see", "listen", and "read". In this paper, we design SpikeVoice, which performs high-quality Text-To-Speech (TTS) via SNN, to explore the potential of SNN to “speak”. A major obstacle to using SNN for such generative tasks lies in the demand for models to grasp long-term dependencies. The serial nature of spiking neurons, however, leads to the invisibility of information at future spiking time steps, limiting SNN models to capture sequence dependencies solely within the same time step. We term this phenomenon "partial-time dependency". To address this issue, we introduce Spiking Temporal-Sequential Attention (STSA) in the SpikeVoice. To the best of our knowledge, SpikeVoice is the first TTS work in the SNN field. We perform experiments using four wellestablished datasets that cover both Chinese and English languages, encompassing scenarios with both single-speaker and multi-speaker configurations. The results demonstrate that SpikeVoice can achieve results comparable to Artificial Neural Networks (ANN) with only 10.5% energy consumption of ANN. Both our demo and code are available as supplementary material.

heading: 1 Introduction
text: Since the advent of Artificial Neural Networks (ANN), remarkable achievements have been made in the field of image (Radford et al., 2021; Carion et al., 2020; Liu et al., 2021; Yao et al., 2024b), natural language (Vaswani et al., 2017; Devlin et al., 2019; Brown et al., 2020), and speech (Baevski et al., 2020; Hsu et al., 2021). In recent years, with the success of large language models (OpenAI, 2023; Anil et al., 2023; Touvron et al., 2023; Li et al., 2023a; Sun et al., 2023; Radford et al., 2023), there has been a notable upward trend in energy
consumption. At the same time, Spiking Neural Network (SNN), inspired by the biological nervous system and recognized as the third generation of neural networks (Maass, 1997), employs spiking neurons (Hodgkin and Huxley, 1952; Abbott, 1999; Fang et al., 2023b) with charge-fire-reset temporal dynamic. The temporal dynamic makes SNN to exhibit the event-driven feature of sparse firing and the binary spike communication feature between neurons using 0s and 1s, providing a distinct advantage in energy efficiency (Cao et al., 2015).
Recently, SNN has achieved remarkable progress on several tasks, such as object detection and image classification (Zhao et al., 2021; Rajagopal et al., 2023; Yao et al., 2023a,b, 2024a), speech recognition (Wu et al., 2020; Wang et al., 2023), and text classification tasks (Lv et al., 2023, 2022). It is the success of these tasks that have led us to believe that SNN has preliminarily acquired the abilities of "seeing", "listening", and "reading". However, applying SNN to generative tasks encounters some obstacles, particularly in addressing the challenge of SNN capturing long-term dependencies. As mentioned above, spiking neurons have a temporal dynamic of charge-fire-reset. Such a serial process hinders the capture of information from future time steps in the spiking temporal dimension. Existing SNN models performing attention operations in the spiking sequential dimension can only establish sequence dependencies within the same time step or, in other words, among partial binary embedding (Lv et al., 2023; Li et al., 2023b), hindering the establishment of long-term dependencies. We term this phenomenon as "partial-time dependency".
In this paper, we introduce SpikeVoice, a high-quality Text-To-Speech (TTS) model with a Transformer-based SNN framework (Vaswani et al., 2017) solving the "partial-time dependency" problem, and successfully explore the potential of SNN to "speak". To address the issue of "partial-
7927
time dependency", we propose Spiking TemporalSequential Attention (STSA) in SpikeVoice. STSA performs temporal-mixing in the spiking temporal dimension to capture information from future time steps, enabling access to the global information of binary embedding at each spiking time step. After time-mixing, STSA performs sequential-mixing in the spiking sequential dimension to integrate contextual information. Furthermore, we implement SpikeVoice in a spike-driven manner with the Leaky Integrate-and-Fire (LIF) (Maass, 1997) neurons, fully harnessing the energy efficiency of SNN. Spike-driven denotes the concurrent existence of both the binary spike communication feature and the event-driven feature. To the best of our knowledge, SpikeVoice is the first TTS model within the SNN framework, which not only promotes the development of SNN in generative tasks but also expands the scope of the SNN model in practical applications.
The main contributions are summarized as follows:
• To the best of our knowledge, SpikeVoice is the first TTS model within the SNN framework that endows SNN with the "speaking" capability, enabling high-quality speech synthesis and filling the blank of speech synthesis in the SNN field.
• In SpikeVoice, we introduce STSA, where the temporal-mixing in the spiking temporal dimension enables the access to the global information of binary embedding at each spiking time step, resolving the issue of "partial-time dependency" caused by the serial spiking neurons.
• The results reveal that SpikeVoice achieves synthesis performance close to ANN in both English and Chinese scenarios with both single-speaker and multi-speaker configurations. Remarkably, the energy consumption of SpikeVoice is merely 10.5% of ANN, alleviating the high energy consumption issue associated with ANN.

heading: 2 Related work
text: Transformers in SNN: Training in SNN is primarily categorized into two methods: ANN-to-SNN conversion (ANN2SNN) (Bu et al., 2021; Deng
and Gu, 2020; Han et al., 2020) and surrogate training (Wu et al., 2018a; Shrestha and Orchard, 2018; Wu et al., 2018b; Duan et al., 2022). Leveraging ANN2SNN, (Mueller et al., 2021) integrates the Transformer architecture into SNN. Nevertheless, this approach demands dozens or even hundreds of time steps to attain satisfactory performance. Spikeformer (Zhou et al., 2022) conducts direct training of the Transformer within the SNN framework and achieves state-of-the-art performance on ImageNet with just four time steps. However, it doesn’t fully harness the energy-efficient advantages of SNN due to the presence of Multiplyand-Accumulate (MAC) operations. Spike-driven Transformer (Yao et al., 2023a) incorporates the spike-driven paradigm into Transformer architecture and introduces the Spike-Driven Self-Attenton (SDSA) (Yao et al., 2023a). SDSA utilizes sparse additive operations as a replacement for multiplication operations in attention mechanisms, effectively addressing the issues present in Spikeformer related to MAC operations. SpikeGPT (Zhu et al., 2023) is the first to introduce text generation tasks into the SNN framework. However, it still does not make full of the energy-efficient capabilities of SNN.
Transformers in TTS: Tactron2 (Shen et al., 2018) employs RNN (Hochreiter and Schmidhuber, 1997) for speech synthesis which results in low training efficiency and struggles to establish long-term dependencies. To address these issues, Transformer-TTS (Li et al., 2019) introduces an autoregressive TTS model that combines Tactron2 with the Transformer, enhancing training efficiency while capturing long-term dependencies. However, autoregressive TTS models often suffer from slow synthesis speed and less robust speech synthesis. FastSpeech (Ren et al., 2019), on the other hand, utilizes knowledge distillation during training to build a non-autoregressive TTS model, yet the training process can be complicated. FastSpeech2 (Ren et al., 2020) simplifies the training process by removing knowledge distillation from the FastSpeech training pipeline and adopting the end-to-end training approach, effectively addressing the issue of the extended training duration associated with FastSpeech.

heading: 3 Method
text: In this study, we propose SpikeVoice, the first spikedriven TTS model. The overall model architecture
is illustrated in Fig.1. The Spiking Phoneme Encoder (SPE) performs binary embedding on the input phoneme embedding sequence and generates high-level spiking phoneme representations. The Spiking Variance Adaptor (SVA) enhances the spiking phoneme representations by incorporating variance information related to duration, pitch, and energy. Finally, the Spiking Mel Decoder (SMD) and Spiking PostNet generate Mel-Spectrograms in a non-autoregressive manner. In the following sections, we will first introduce the LIF neurons, and then introduce the components of SpikeVoice.

heading: 3.1 Leaky Integrate-and-Fire Neuron
text: The LIF neuron is a biologically inspired spiking neuron having the charge-fire-reset biological neu-
ronal dynamics as shown in Fig.2. The working process of LIF neuron can be described as:
Ht = Vt−1 + 1
τ (Xt − (Vt−1 − V re)) (1)
St = Θ(Ht − V th) (2) Vt = V reSt +Ht(1− St) (3)
Eq.(1) to (3) respectively represent the charging, firing, and membrane potential resetting of LIF. Xt denotes the input current at time t, Ht signifies the membrane potential after charging, St represents the spike tensor at time t, Θ represents the step function, V th denotes the firing threshold, V re is the reset membrane potential, and Vt signifies the membrane potential after resetting.

heading: 3.2 SpikeVoice
text: Temporal-Sequential Embedding: At spiking temporal wise, we first expand the phoneme embedding sequence z to T time steps. In order to incorporate the position information with STSA, we then apply position embedding in both the spiking temporal dimension and the phoneme sequential dimension.
x0(t,l) = z(t,l) + e tem (t,) + e seq (,l) (4)
where x0 ∈ RT×L×D will be taken as the input to Spiking Phoneme Encoder. L represents the length of the phoneme sequence, D denotes the size of embedding dimension, t ∈ {1, . . . , T} and l ∈ {1, . . . , L}. etem(t,) and e seq (,l) are the position embedding of time step t at temporal wise and position l at sequence wise.
Spiking Phoneme Encoder: Spiking Phoneme Encoders are composed of a stack of N identical layers, each of which consists of an STSA module and a Spiking FeedForward module. As shown on the right side of Fig.1, each Spiking FeedForward module consists of two stacked 1DConvolution layers. To ensure the energy efficiency
of SpikeVoice, we introduce a LIF neuron layer before each 1D-Convolution layer, to convert continuous inputs into sparse spiking tensors. Then the high-level spiking phoneme representations xn of layer n can be obtained as:
un = STSA(xn−1) (5)
xn = LN(un + f(un)) (6)
f(·) = [Conv(SN (·))]2 (7)
where LN is layer nomalization, SN refers to the LIF neuron layer depicted in Eq.(1)-(3). f(·) represents the stacked 1D-Convolution and LIF neuron layers, un is the membrane potential output of STSA.
Spiking Temporal-Sequential Attention: As illustrated in the left block of Fig.1, STSA is composed of a Spiking Temporal Attention and a Spiking Sequential Attention. Due to the serial nature of LIF neurons, it results in the inability to capture information from future time steps along the spiking temporal dimension and leads to the issue of "partial-time dependency". Therefore, we propose the Spiking Temporal Attention to perform temporal-mixing over the spiking temporal dimension obtaining the global information of binary embedding.
Taking STSA in layer n of Spiking Phoneme Encoder as an example, initially, we perform binary embedding on the output of layer n − 1 to obtain the sparse spiking hidden representation sn = SN (xn−1), sn ∈ RT×L×D. Along the spiking temporal dimension T the binary embedding of each token can be obtained. The Spiking Temporal
Attention can be depicted as:
µn = SN (BN(Wn,temµ sn)) (8) sn(t,:) = SN (Σc(qn(t,:) ⊙ kn(t,:)))⊙ vn(t,:) (9) σn = LN(xn−1 + Linear(sn)) (10)
where µ ∈ {q, k, v}, BN represents Batch Normalization, and Wn,temµ is a learnable matrix for Spiking Temporal Attention. For vanilla attention can introduce MAC operations to the SpikeVoice, we utilize the SDSA (Yao et al., 2023a) in Eq.(9) as a substitute for vanilla attention. ⊙ is the Hadamard product and Σc means sum up in column-wise. sn(t,:) denotes the spiking tensor at time step t, which is the output of attention computing on spiking temporal wise. σn represents the membrane potential output of Spiking Temporal Attention.
Then sn = SN (σn) will serve as the sparse input to Spiking Sequential Attention:
µn = SN (BN(Wn,seqµ sn)) (11) sn(:,l) = SN (Σc(qn(:,l) ⊙ kn(:,l)))⊙ vn(:,l) (12) un = LN(un + Linear(sn)) (13)
where sn(:,l) is the spiking tensor at position l in the sequence wise. The computation process above can be easily extended to Spiking Mel Decoder.
Spiking Variance Adaptor: The Spiking Variance Adaptor takes the high-level spiking phoneme representations xN as its input. And then the Duration Predictor Pd, Energy Predictor Pe, and Pitch Predictor Pp impart variance information to xN . The predictors in Spiking Variance Adaptor all take an identical structure, shown in the green block on the right side of Fig.1. Besides, We employ a residual connection around the Energy Predictor and Pitch Predictor. Finally, the Length Regulator LR aligns the hidden sequence to the length of the Mel-Spectrogram:
d = Pd(x N ) (14)
u = Pe(Pp(x N )) (15)
{y0(t,l′)}l′=1,...,L′ = LR ( u(t,l), d(l,) ) l=1,...,L (16)
where d ∈ RL comprises the length of mel frames corresponding to each phoneme. u represents the membrane potential incorporated the pitch and energy variance information. {y0(t,l′)} signifies the mel representations corresponding to u(t,l) after being extended by d(l,) times. L′ represents the total length of the target Mel-Spectrogram.
Spiking Mel Decoder and PostNet: Spiking Phoneme Encoders are composed of a stack of M identical layers, each of which also comprises an STSA and a Spiking FeedForward. The Spiking PostNet is designed to enhance the fine details of Mel-Spectrograms. LIF neuron layers are also added before each linear layer and 1D-convolution layer in the Spiking PostNet to ensure sparse inputs. Then the Mel-Spectrogram can be obtained as:
ym = SFF (STSA(ym−1)) (17)
O = PostNet(yM ) (18)
Oc(l′,) = ȳ M (:,l′), O f (l′,) = Ō(:,l′) (19)
where ym is the output of the mth layer of Spiking Mel Decoder. To calculate the supervised loss with ground truth, we average the output at spiking temporal dimension as the predicted MelSpectrograms, and ·̄ represents the average operation. We denote the Mel-Spectrograms obtained before the Spiking PostNet as Oc and the output obtained from the Spiking PostNet as Of .
The loss function encompasses supervised losses using Mean Squared Error (MSE) for pitch, energy, and duration, as well as Mean Absolute Error (MAE) losses for both the coarse MelSpectrograms Oc and the fine Mel-Spectrograms Of .

heading: 4 Experiments
text: We conducted experiments with SpikeVoice on single-speaker and multi-speaker datasets, encompassing both English and Chinese. The singlespeaker datasets include LJSpeech (Ito and Johnson, 2017) and Baker1, while the multi-speaker datasets comprise LibriTTS (Zen et al., 2019) and AISHELL3 (Yao Shi, 2015). In the following subsections, we present results on subjective and objective metrics for ground truth denoted as ’GT’, ANN baseline denoted as ’FastSpeech2’, SpikeVoice signified as ’SpikeVoice-STSA’, and SNN baselines: SpikeVoice with attention in Spikeformer replacing the STSA, which is denoted as ’SpikeVoice-ATTN’ and SpikeVoice with only Spiking Sequential Attention, which denoted as ’SpikeVoice-SDSA’. Additionally, In Section 4.5, we perform visual analysis, and in Section 4.6, we discuss the balance between SpikeVoice’s energy consumption and the quality of synthesized speech.
1https://www.data-baker.com/data/index/TNtts/

heading: 4.1 Datasets
text: For each of the datasets, we have randomly split the dataset into three sets: the training set, the validation, and the testing sets, both comprising 256 samples.
LJSpeech is a female single-speaker English monolingual dataset. It comprises a collection of 13100 utterances, each lasting between 1 to 10 seconds, amounting to roughly 24 hours of speech material.
Baker is a female single-speaker Chinese dataset. It encompasses a wide range of content domains, including news, novels, technology, and so on. In total, Baker comprises 10000 speech recordings, with approximately a total of 12 hours of speech material.
LibriTTS comprises approximately 191 hours of speech with 1,160 speakers. We utilized the train-clean-360 set from LibriTTS. Within this subset, there are 430 female speakers and 474 male speakers.
AISHELL3 is a multi-speaker Chinese dataset, containing a total of approximately 85 hours of speech, recorded by 218 speakers.

heading: 4.2 Experiments settings
text: Training Settings SpikeVoice is stacked by N = 4 Spiking Phoneme Encoders, a Spiking Variance Adaptor, and M = 6 Spiking Mel Decoders. We transformed the raw speech in all the datasets into mel-spectrograms with a frame length of 1024 and a hop length of 256. The synthesized melspectrograms were uniformly converted into speech using the vocoder HiFiGAN (Kong et al., 2020). We performed the training on four Tesla V100SXM2-32G GPUs with batch size 48. The optimization settings were in line with those defined
in (Ren et al., 2020). The implementation of the SNN framework in SpikeVoice is based on SpikingJelly (Fang et al., 2023a).
Evaluation Settings We employed Word Error Rate (WER) for English and Character Error Rate (CER) for Chinese, along with NISQAV2 (Mittag et al., 2021), as objective metrics to evaluate the quality of single-speaker speech synthesis. For multi-speaker synthesis, we additionally utilized Speaker Embedding Cosine Similarity (SECS) to gauge the similarity between the synthesized speech and the target speech in terms of the speaker’s voice. Specifically, for WER, we utilized Hubert (Hsu et al., 2021) for English ASR transcription and Wav2Vec2 (Baevski et al., 2020) for Chinese ASR transcription. As for SECS, we employed the speaker encoder from the Resemblyzer2 toolkit to extract speaker embeddings and calculate cosine similarity. In assessing both single and multi-speaker synthesis, we relied on 5-scale Mean Opinion Scores (MOS) with 95% confidence intervals as our subjective metric. To obtain these scores, we randomly selected 80 samples from each test set, and a total of 12 participants were asked to provide ratings for the synthesized speech.

heading: 4.3 Performance on Single-Speaker
text: As shown in Tab.2, we conducted experiments on the LJSpeech and Baker datasets, reflecting the synthesis quality of English and Chinese singlespeaker respectively.
For the objective metrics, SpikeVoice surpasses all the SNN and ANN baselines on the WER/CER metric and is the best-performing SNN-based model on NISQA. These results demonstrate that the global information of temporal spike sequence
2https://github.com/resemble-ai/Resemblyzer
in STSA contributes to the synthesis of higherquality and clearer speech.
For the subjective evaluation, SpikeVoice outperforms both SpikeVoice-ATTN and SpikeVoiceSDSA. The difference in MOS scores between SpikeVoice and ANN is merely 0.04 on LJSpeech and SpikeVoice even surpasses the ANN-based model on the Baker dataset, indicating that SpikeVoice’s synthesis quality closely approaches that of ANN in terms of human perception. The results compared to SpikeVoice-SDSA also confirm the effectiveness of temporal-mixing.

heading: 4.4 Model Performance on Multi-Speaker
text: In Tab.3, we respectively present the performance on the AISHELL3 and LibriTTS. In the multispeaker experiments, we have additionally incorporated the SCER metric to assess the speaker similarity between synthesized speech and target speech.
Compared to single-speaker, multi-speaker datasets present more challenges for SNN-based models. SpikeVoice with STSA remains the bestperforming SNN-based model, however, the sparse nature of the spike tensor contributes to energy efficiency at the expense of information loss, leading to a performance gap of MOS scores between the SNN-based models and ANN-based models in multi-speaker datasets, which encompass richer information. Investigating strategies to minimize information loss in the context of spike tensors with low firing rates is worthwhile for future research.

heading: 4.5 Visualized Analysis
text: Visualization of Mel-Spectrograms: Speech synthesized by SpikeVoice exhibits less noise and is clearer compared to the SNN-based baselines, which is evident in Fig.3. As shown in Fig. 3(b)
and 3(c), Mel-Spectrograms synthesized by the SNN baselines become blurry towards the end, losing fine details. In contrast, the Mel-Spectrograms in Fig.3(d) synthesized by SpikeVoice with STSA exhibit minimal sacrifice of details as to ANN in 3(a) and remain notably clearer than those produced by SNN baselines.
Visualization of Spike Patterns: By visualizing spike tensors, more details of SpikeVoice can be observed. As the spike patterns of STSA depicted in Fig.4(a) and Fig.4(b), each dot represents an event, the spike events in the lower layers are sparser, and as the network deepens, more information is incorporated, leading to denser spike events. Spike tensors that convey similar information exhibit similar spike patterns, while others reveal markedly different spike patterns. Spike patterns of the energy and pitch predictors are displayed in Fig.4(c) and Fig.4(d), different from the distribution of spike pattern in 4(a) and 4(b), noticeable channel clustering phenomena can be observed in 4(c) and 4(d).

heading: 4.6 Analysis of Balance between Consumption and Synthesized Speech Quality
text: Apart from its notable biological interpretability, one of the most prominent advantages of SNN lies in its energy efficiency. However, SNN’s binary embedding within a finite time step results in some degree of performance decay. In Tab.4, we present the number of model parameters, time steps of binary embedding, and energy consumption. The term "Spike-Driven" refers to the existence of solely AC operations, and "MOS" here refers to the results on LJSpeech.
While SpikeVoice-STSA comes with a slight in-
crease in the parameter, it takes only 10.5% energy consuming of ANN with 4 time steps and achieves a better performance than SNN baselines. In contrast, SpikeVoice-SDSA exhibits noticeable performance degradation, while the energy consumption is 9.6% of ANN with an equivalent amount of parameters. Similarly, SpikeVoice-ATTN also results in an 88.1% reduction in energy consumption. It is worth to noting that when set time step to 1, the energy consumption of SpikeVoice-STSA can be merely 4.11% of ANN. Hence, when considering both the quality of speech synthesis and energy consumption, SpikeVoice is a superior choice, offering significant energy savings with minimal performance sacrifice.

heading: 5 Conclusion
text: In this paper, we introduce SpikeVoice. To the best of our knowledge, it is the first TTS model that achieves high-quality speech synthesis within the SNN framework and for the first time endows SNN with the ability to "speak". Additionally, SpikeVoice is a spike-driven model with highly energy-efficient. In SpikeVoice, we propose STSA, which performs temporal-mixing in the spiking temporal dimension to address the issue of information invisibility at future time steps on the spiking temporal dimension caused by the serial nature of spiking neurons and thereby address the issue of "partial-time dependency".
We conducted experiments on both singlespeaker and multi-speaker datasets in both Chinese and English. The results demonstrate that SpikeVoice achieves performance comparable to ANN models while consuming only 10.5% of the energy required by ANN. Our successful practice proves the feasibility of TTS tasks within the SNN framework and offers an energy-saving solution for TTS tasks.

heading: 6 Limitation
text: The SpikeVoice within the SNN framework still has several limitations. Primarily, the binary embedding results in inevitably information lost from the input data, leading to a decline in performance. Secondly, due to the inherent sequential mechanism of LIF neurons, the training speed of SpikeVoice is slower than ANN. Finally, as analyzed in section 4.5 with the layers deepen, the firing rate becomes progressively higher, which implies the potential for further reductions in energy consumption. In light of this, we present several prospective exploration directions that reduce information loss during the binary embedding process in SNN, lowering the firing rate in deep neural networks, and parallelization of spike neurons.

heading: 7 Acknowledge
text: This work was partially supported by National Distinguished Young Scholars (62325603), and National Natural Science Foundation of China (62236009,U22A20103,62441606), and Beijing Natural Science Foundation for Distinguished Young Scholars (JQ21015)and CAAI-MindSpore Open Fund, developed on OpenI Community.

heading: A Firing Rate of SpikeVoice
text: In Tab.5, Tab.6, and Tab.7, we respectively present the spike firing rates of Spiking Phoneme Encoder, Spiking Variance Adapter, and Spiking Mel Decoder.

heading: B Examples of Spike Patterns
text: In Fig.5 we present the spike patterns of STSA and also the spike patterns of Pitch Predictor and Energy Predictor.

heading: C Examples of Mel-Spectrograms
text: In Fig.6 we present Mel-Spectrograms of LJSpeech, Baker, LibriTTS, and AISHELL3, and we have magnified the tail of the Mel-Spectrogram for a clearer observation.

heading: Future Work
text: In SpikeVoice, we propose STSA, which performs temporal-mixing in the spiking temporal dimension to address the issue of information invisibility at future time steps on the spiking temporal dimension caused by the serial nature of spiking neurons and thereby address the issue of "partial-time dependency". We conducted experiments on both singlespeaker and multi-speaker datasets in both Chinese and English. The results demonstrate that SpikeVoice achieves performance comparable to ANN models while consuming only 10. 5% of the energy required by ANN. Our successful practice proves the feasibility of TTS tasks within the SNN framework and offers an energy-saving solution for TTS tasks