AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment
Year: 2023

Abstract:
The speech-to-singing (STS) voice conversion task aims to generate singing samples corresponding to speech recordings while facing a major challenge: the alignment between the target (singing) pitch contour and the source (speech) content is difficult to learn in a textfree situation. This paper proposes AlignSTS, an STS model based on explicit cross-modal alignment, we 1) adopt a novel rhythm adaptor to predict the target rhythm representation to bridge the modality gap between content and pitch, where the rhythm representation is disentangled in a simple yet effective way and is quantized into a discrete space; and 2) leverage the cross-modal aligner to re-align the content features explicitly according to the predicted rhythm and conduct a cross-modal fusion for re-synthesis. Experimental results show that AlignSTS achieves superior performance in terms of both objective and subjective metrics. Audio samples are available at https://alignsts.github.io.

heading: 1 Introduction
text: Speech-to-singing (STS) voice conversion (Saitou et al., 2004; Cen et al., 2012; Parekh et al., 2020) aims to transfer speech samples into the corresponding singing samples, with the timbre identity and phoneme information unaltered. An STS system takes a speech sample and a target melody as conditions and then generates a high-quality singing sample following the target musical melody. Speechto-singing research is important for human voice study and useful for practical applications such as computer-aided music production or musical entertainment.
Researchers have developed three major STS approaches: 1) Model-based approaches(Saitou et al., 2004, 2007) use phone-score synchronization information to manually align the phonemes and the target musical notes with artificial control
∗* Corresponding author
models. 2) Template-based STS (Cen et al., 2012; Vijayan et al., 2017, 2018) requires an available high-quality reference vocal input, which will be aligned with the input speech for subsequent musical feature extractions. The alignment is the key part and is mostly based on dynamic time warping (DTW). 3) Style transfer approach (Parekh et al., 2020) views STS as a style-transfer problem. This class of methods considers the specific properties that are transformed during conversion as "style". Parekh et al. (2020) stretch the input speech to the same length as the target F0 contour and concatenate the latent features to fuse the asynchronous representations. Wu and Yang (2020) serve as a continuation and extension of their prior work by leveraging boundary-equilibrium GAN (BEGAN) (Berthelot et al., 2017).
Despite their recent success, however, the style information is complex and is composed of multiple entangled features in the time domain (the duration information and the temporal length) and the frequency domain (the pitch information). Simply stretching the representations temporally or applying implicit self-attention can cause alignment problems. In a larger sense, human voice information (such as speech (Huang et al., 2023b) and singing (Huang et al., 2021, 2022a)) is composed of several variance information, each controlling a specific sensory modality. Plenty of arts focus on the decomposition and resynthesis of speech signal (Qian et al., 2020; Chan et al., 2022; Choi et al., 2021, 2022; Huang et al., 2022b; Huang et al.). They managed to roughly decompose speech signals into components like linguistic content, pitch, rhythm, timbre identity, etc. By manipulating any of these components, one can resynthesize a customized speech waveform. In the context of STS, the components that will be manipulated are 1) the frequency modality, namely the pitch information; and 2) the duration modality, or rhythm. The challenge remains, however, since the rhythm infor-
7074
mation can be indeterminate given only the content and the target pitch information.
In this paper, we present a novel approach AlignSTS based on modality disentanglement and crossmodal alignment. To tackle the alignment problem, we introduce a novel rhythm representation to bridge the modality gap between content and pitch. A rhythm adaptor is designed to predict the target rhythm representation, which is used to guide the content features to perform the alignment and the cross-modal fusion. Further, we explore speech-to-sing conversion in zero-shot scenarios, where we train the model with singing data in a self-supervised manner and test it on unseen speech data. We categorize AlignSTS as one of the style transfer approaches since we only need a source speech and a target pitch contour for conversion. Experimental results demonstrate significant performance improvements over baseline models. The main contributions of this work are summarized as follows:
• We leverage the temporal modality, rhythm, to bridge the modality gap between the speech content and the target pitch. The rhythm representation is carefully designed and quantized into discrete space to model temporal dynamic states.
• We propose AlignSTS, an STS model based on cross-modal alignment, which predicts the target rhythm representation and uses it to conduct an explicit cross-modal alignment to re-align the content information.
• Experimental results demonstrate that AlignSTS achieves state-of-the-art results in both objective and subjective evaluations. Over baseline models, AlignSTS reaches an absolute improvement of 0.39 in MOS for overall quality and 0.36 for prosody naturalness.

heading: 2 Related Works

heading: 2.1 Voice Conversion
text: Voice conversion (VC) focuses on changing the timbre identity of an utterance to a target speaker while keeping the content information (phoneme sequence intact). Inspired by image style transfer, CycleGAN-VC (Kaneko and Kameoka, 2018) combines CycleGAN (Zhu et al., 2017) with gated CNN and identity-mapping loss to capture sequential and hierarchical structures. Similarly, StarGAN-VC (Kameoka et al., 2018) adapts StarGAN (Choi et al., 2018) and manages to train with-
out parallel utterances and pay more attention to real-time processing. Apart from GANs, conditional variational autoencoders (CVAE) are also an important class of approaches. VAE-VC (Hsu et al., 2016) uses the encoder of a VAE to learn the speaker-independent phoneme representations, thus disentangling the timbre identity. ACVAEVC (Kameoka et al., 2019) notices that VAEs easily ignore the attribute class label input, i.e. the speaker identity, which therefore utilizes an auxiliary speaker classifier. AutoVC (Qian et al., 2019) carefully designs a bottleneck mechanism within a simple autoencoder to achieve zero-shot many-tomany voice conversion with non-parallel data.

heading: 2.2 Speech Representation Disentanglement
text: Human speech (Huang et al., 2023a, 2022d) is a severely complicated and comprehensive information stream, where a number of latent units entangled with each other such as content, pitch, timbre, etc. The disentanglement of the speech signal is an attempt to learn factorized and even interpretable generative factors (Bengio et al., 2013) for further application, like style transfer or domain adaptation. NANSY (Choi et al., 2021) manipulates disentangled factors such as content, pitch, and speed of speech signal, where content and pitch are decomposed by wav2vec 2.0 (Baevski et al., 2020) and Yingram, respectively. Information bottleneck (Qian et al., 2019) is also a popular way to disentanglement. Following AutoVC, SpeechSplit (Qian et al., 2020) introduces three carefully designed information bottlenecks to improve decomposition. VoiceMixer (Lee et al., 2021) leverages a similarity-based information bottleneck and adversarial feedback to disentangle content and voice style. SpeechSplit 2.0 (Chan et al., 2022) alleviates the bottleneck tuning in SpeechSplit by applying efficient signal processing techniques on encoder inputs.

heading: 3 AlignSTS
text: In this section, we first define and formulate the problem of speech-to-singing (STS) voice conversion. We then present the information perturbation methods, the rhythm modality modeling, and the cross-modal fusion mechanism. Finally, we introduce the overall architecture of AlignSTS and the training/inference procedure.

heading: 3.1 Problem Formulation
text: Let Ssp and Ssg denote the spectrograms extracted from speech and singing signals. We assume that vocal signals are a comprehensive fusion of several variance information, i.e., content, pitch, rhythm, and timbre. Therefore, this process can be formally defined in Equation 1, where csp, isp, f sp, and rsp denote the representations of content, timbre identity, pitch, and rhythm information of speech data. g(·) denotes the multi-modality fusion.
Ssp = g(csp, isp,f sp, rsp) (1)
Ŝsg = fθ(csp, isp,f sg, rsg(csp,f sg)) (2)
The problem of STS is to learn a neural network fθ, such that if we switch the pitch component f sp to target pitch feature f sg, the corresponding singing spectrogram Ŝsg will be generated while preserving the content and timbre intact, as shown in Equation 2. Note that the rhythm modality implies temporal information, which will also be influenced and should be adapted according to the pitch and content. rsg is the adapted rhythm representation and is generated conditioned on f sg and csp.

heading: 3.2 Method Overview
text: AlignSTS treats speech and singing signals as a comprehensive fusion of several variance information, which can be further regarded as different sensory modalities. Pitch and rhythm features are the main modalities to convert in STS, which can be disentangled in parallel. However, the synthesis logic of variance information needs to be carefully designed during conversion. A phoneme sequence and a pitch contour seem to be uncorrelated with each other at first glance, yet it is highly possible for a human to create a suitable alignment and produce a singing melody. The mechanism behind the human behavior is: (1) find an appropriate sequence of onset and offset timings of phonemes and notes, or as known as rhythm, according to the lyrics and the melody; and (2) place the time-stretched phonemes in order according to the rhythm and combine them with the melody to produce the singing result. Inspired by this mechanism, AlignSTS:
1. Decompose the input speech signal into several disentangled variance information.
2. According to the altered speech component, i.e., the pitch contour, adapt the speech component that controls temporal duration information,
namely the rhythm representation. 3. Perform a cross-modal alignment to re-align
the content according to the adapted rhythm representations and carry out a modality fusion to combine the variance information.

heading: 3.3 Information Perturbation
text: Speech and singing samples are fully complex and can be decomposed into variance information such as content, pitch, rhythm, and timbre. Each feature needs to be extracted and disentangled from the other, which can be achieved by certain information perturbations.
• Linguistic Content. We use a wav2vec 2.0 (Baevski et al., 2020) model pre-trained and finetuned on 960 hours of Librispeech on 16kHz sampled speech audio * to extract the linguistic content. It is shown that extracted features from speech SSP models such as wav2vec 2.0 can be applied to downstream tasks like ASR, indicating that the extracted features should provide rich isolated linguistic information. Prior works like SpeechSplit (Qian et al., 2020) and AutoPST (Qian et al., 2021) utilize random sampling to perturb the rhythm information within the content, given their input and output audios are the same speech. Using the paired speech and singing data, we do not need extra random sampling but leverage the natural discrepancy between speech and singing to perturb the rhythm within the content information.
• Pitch. We extract the fundamental frequency contour F0 of singing data as pitch information. The fundamental frequency contour is then quantized to 256 possible values uniformly. The F0 contour contains minimum rhythm information, given a common singing situation where one phoneme corresponds to several musical notes (or vice versa), making the rhythm theoretically indeterminate.
• Rhythm. Rhythm is a crucial speech component that controls the overall speed, the duration of each phoneme, and the pattern of the onset and offset of syllables. Therefore, the rhythm modality provides the duration information for both content and pitch. A good rhythm representation creates a "fill in the blank" mechanism (Qian et al., 2020) for content information to realign. Meanwhile, the patterns of the rhythms
*https://huggingface.co/facebook/wav2vec2-base-960h
of speech and singing voices differ significantly, the intensity of singing voices is generally more fluctuating and distinct than speech. Therefore, we directly utilize the time-domain energy contour et of singing samples, which is computed by taking the L2-norm of all the frequencies for each time step. To eliminate the relative fluctuation and leave only the rhythm information, we further normalize the energy contour et using the Sigmoid function σ(·):
rt = σ
( β × et − mean(e)
std(e) + ϵ
) (3)
where ϵ is used to avoid division-by-zero error and β is a hyperparameter used to control the normalizing effect. rt is the resulting rhythm representation. We further stabilize the representation by applying Gaussian filtering with a standard deviation σ of 1.0. This scalar representation disentangles the rhythm information the most thoroughly while preserving the duration information. A visualization is shown in Figure 2.
• Timbre. We leverage an open-source speaker identity encoding API Resemblyzer † to extract the timbre representations. Resemblyzer generates a 256-dimensional vector that summarizes the characteristics of the given voice spoken.

heading: 3.4 Rhythm Modality Modeling
text: The rhythm modality of the singing signal can be considered as a series of discrete temporal dynamic states, such as attack, sustain, transition, silence, etc. In automatic music transcription (AMT), the recognition of states refers to onset/offset detection.(Chang and Lee, 2014; Fu and Su, 2019) Similarly, traditional ASR methods like Kaldi (Povey
†https://github.com/resemble-ai/Resemblyzer
et al., 2011) model the intra- and inter-states of phonemes to improve distinguishability. From this perspective, the rhythm modality is a "softened" version of duration with more intermediate states. The onset and offset states (intermediate states) of singing may last longer than that of speech (consider the fade-out effect). Inspired by this, we adopt a Vector Quantization (VQ) module to quantize the continuous rhythm features to model these temporal states and form an information bottleneck (Van Den Oord et al., 2017). The designed discrete latent embedding space can be denoted as e ∈ RK×D where K is the number of clustered categories and D is the dimension of each latent embedding vector ek ∈ RD, k ∈ 1, 2, ...,K. A commitment loss (Van Den Oord et al., 2017) is used to constrain the input representation to commit to a discrete embedding:
LC = ∥ze(x)− sg[e]∥22 (4)
where ze(·) denotes the VQ module and sg denotes the stop-gradient operator.
To model the target rhythm representations conditioned on both input content information and target pitch contour, we design a Cross-Attention module by adopting the Scaled Dot-Product Attention (Vaswani et al., 2017). The encoded pitch representation XP is used as the query, and the encoded content representation XC is used as both the key and the value. A positional encoding embedding is added to both representations before the attention module. The attention mechanism can be formulated as:
Attention(Q,K, V )
= Attention(XP , XC , XC) = Softmax ( XPX T C√
d
) XC (5)
where XP and XC are first projected to the query, the key, and the value representations in practice.
Since the target rhythm features are not available during inference, we only activate the VQ module and use the discrete embeddings generated from it for downstream modules during the training stage. The cross-attention module and a stack of convolution layers are combined to serve as the Rhythm Predictor, which takes the generated discrete rhythm embeddings from the VQ module as the training target. A cross-entropy (CE) loss is applied to train the rhythm predictor so it can generate
the desired rhythm embeddings during inference:
LR = − 1
T T∑ t=1 K∑ c=1 yt,c log(ŷt,c) (6)
where T denotes the number of time frames. yt,c denotes the rhythm embedding indices generated from the VQ module, where yt,c = 1 if c = kt and kt is the rhythm index at time step t. ŷt,c denotes the predicted rhythm indices.

heading: 3.5 Cross-Modal Alignment
text: With the target rhythm sequence generated, we design a cross-modal aligner to place the linguistic features along the time axis according to the target rhythm. This aligner uses rhythm information to bridge the gap between content and pitch modalities. We simply use a stack of two cross-attention layers mentioned before to complete this task. The rhythm embedding XR generated from the rhythm adaptor is used as the query, and the encoded content representation XC is again used as both the key and value.
The alignment using the cross-attention mechanism can be considered a soft-selection operation over the linguistic content representations XC along the time axis at each target time step. Ideally, the resulting attention weight matrix should show a monotonic pattern and the alignment path should be concentrated and nearly diagonal. This requires extra constraints and regulations to make sure the model does not bypass the attention mechanism and simply interpolate the input content to the same length as the required rhythm representation. We mainly apply two techniques: windowing (Chorowski et al., 2015) and guided attention loss (Tachibana et al., 2018), which is described in detail in Appendix B.
With each variance information re-aligned, the content, the rhythm, and the pitch representations should have the same temporal length. We apply the inter-modality fusion across these features by element-wise vector addition. Furthermore, we involve the timbre information by adding the timbre embeddings extracted earlier.

heading: 3.6 Architecture
text: AlignSTS mainly consists of four modules: the encoders, the rhythm adaptor, the cross-modal aligner, and the mel decoder. The overall architecture of AlignSTS is presented in Figure 1. The detailed
description and the overall hyperparameter setting are listed in Appendix A.
To accelerate the re-synthesis process and improve the audio quality, we adopt the teacher model of ProDiff (Huang et al., 2022c), a 4-step generatorbased diffusion model, to be the mel decoder. A generator-based diffusion model parameterizes the denoising model by directly predicting the clean data, instead of estimating the gradient of data density. Therefore, the generator-based method bypasses the difficulty of predicting sample xt using a single network at different diffusion steps t, which allows us to train the decoder with a reconstruction loss. We use two objectives to be the reconstruction loss:
• Mean Absolute Error (MAE). We apply MAE at each random term of the diffusion step t:
LMAE = ∥∥∥∥xθ (αtx0 +√1− α2t ϵ)− x0∥∥∥∥
(7)
where αt denotes a derived constant that αt = t∏
i=1
√ 1− βi, in which βt is the predefined fixed
noise schedule at diffusion step t. ϵ is randomly sampled and ϵ ∈ N (0, I). x0 denotes the clean data and xθ denotes the denoised data sample predicted by the denoising neural networks θ.
• Structural Similarity Index (SSIM) Loss. We adopt SSIM (Wang et al., 2004), one of the stateof-the-art perceptual metrics to measure image quality, to tackle the problem of over-smoothness (Ren et al., 2022):
LSSIM = 1− (8) SSIM ( xθ ( αtx0 + √ 1− α2t ϵ ) ,x0 ) where SSIM(·) is the SSIM function and is between 0 and 1.

heading: 3.7 Training and Inference
text: The final loss of AlignSTS consists of the following loss terms: 1) the commitment loss of VQ module LC; 2) the CE loss of rhythm predictor LR; 3) the guided loss for cross-attention Lattn; 4) the MAE reconstruction loss LMAE; and 5) the SSIM loss LSSIM. During the training stage, the crossmodal aligner takes the rhythm embeddings generated from the VQ module as input directly. At the same time, the rhythm predictor is trained to
predict the correct indices of quantized vectors for rhythm embeddings using CE loss. During the inference stage, the VQ module is deactivated and the predicted rhythm indices from the rhythm predictor are used to look up the embeddings for subsequent modules.

heading: 4 Experiments

heading: 4.1 Experimental Setup

heading: 4.1.1 Dataset
text: We utilize a subset of the PopBuTFy database (Liu et al., 2022) as our dataset. PopBuTFy is originally used for the singing voice beautifying (SVB) task, which consists of paired amateur and professional singing recordings. Additionally, we collected and annotated the speech version of a subset of PopBuTFy to create a paired speech and singing dataset. In all, the dataset consists of 152 English pop songs (∼5.5 hours in total) and the respective speech recordings (∼3.7 hours in total) from 16 singers. More details are listed in Appendix C.

heading: 4.1.2 Implementation Details
text: We use mel-spectrograms extracted from singing samples to be the training target. We transform the raw waveform with the sampling rate of 24000 Hz into mel-spectrograms with window size 512 and hop size 128. We extract the fundamental frequency contour F0 using Parselmouth (Jadoul et al., 2018; Boersma and Weenink, 2021) as pitch information. In addition, we remove all the silent frames to accelerate the training process. The output mel-spectrograms are transformed into audio waveforms using a HiFi-GAN vocoder (Kong et al., 2020) trained with singing data in advance. More details are listed in Appendix A.

heading: 4.1.3 Training and Evaluation
text: We train the proposed model on a single NVIDIA GeForce RTX 3090 GPU with a batch size of 20 sentences for 200k steps. The performance evaluation consists of two parts, objective and subjective evaluations, respectively.
• Objective evaluation. Following (Parekh et al., 2020), we use log-spectral distance (LSD) and F0 raw chroma accuracy (RCA) using mir_eval (Raffel et al., 2014) as the objective metrics. LSD is computed by taking the average of the Euclidean distance between the predicted log-spectrogram and the ground truth recordings. For RCA, we set the maximum tolerance deviation as 50 cents.
In addition, we design a new evaluation metric, rhythm representation distance (RRD), to measure rhythm reconstruction performance by computing the Euclidean distance between the rhythm representations described in Equation 3.
• Subjective evaluation. We conducted crowdsourced mean opinion score (MOS) listening tests for subjective evaluation. Specifically, MOS-Q indicates the overall quality of the audio and MOS-P indicates the naturalness and coherence of prosody. The metrics are rated from 1 to 5 and reported with 95% confidence intervals.

heading: 4.1.4 Baseline Models
text: We compare the quality of the generated audio samples of AlignSTS with other approaches, including 1) GT Mel, in which we first convert the reference audio into mel-spectrograms and then convert them back to audio using HiFi-GAN; 2) (Parekh et al., 2020), an STS model based on encoder-decoder framework; 3) SpeechSplit 2.0 (w/o SE) (Chan et al., 2022), where we train the model only conditioned on the target pitch contour, while the rhythm input is implemented by interpolating the source spectral envelope (from speech) to match the length of pitch contour (in both SpeechSplit 2.0 baselines, we interpolate the content input to match the target length); 4) AlignSTS (zero-shot), where we conduct zero-shot STS and test the model on unseen speech samples; and 5) AlignSTS (GAN), where we change the diffusion mel decoder to the decoder of FastSpeech 2 (Ren et al., 2020), one of the SOTA approaches of non-autoregressive text-to-speech (NAR-TTS), combining a multi-window discriminator (Wu and Luan, 2020). In addition, since SpeechSplit 2.0 is not originally designed for STS tasks, we conduct extra experiments on baseline SpeechSplit 2.0 (w/ SE) in Appendix D, where we involve the target rhythm representations.

heading: 4.2 Main Results
text: The results are shown in Table 1. The quality of GT Mel is the upper limit of STS systems. In both objective and subjective evaluations, AlignSTS outperforms the baseline systems by a large margin. In both objective and subjective evaluations, the results of RCA and MOS-P are better than LSD and MOS-Q, which indicates that the condition of the target pitch contour possesses rich melody and prosody information, making the melody transfer much more effortless than phoneme modeling.
The main challenge remains in the coherence and recognizability of phonemes. The results of RRD show the reconstruction performance of the rhythm representation, demonstrating that AlignSTS with explicit rhythm modeling does the best job. In addition, the results indicate that RRD can be a valid metric for rhythm modeling.
A visualization of output mel-spectrograms is shown in Figure 3. The effectiveness of rhythm modality modeling is clearly demonstrated: 1) (Parekh et al., 2020) barely expresses the dynamic of the melody and the phonemes are scarcely distinguished; 2) the voiced parts are clustered temporally in SpeechSplit 2.0 (w/o SE), but the phonemes are still unrecognizable and the formants are disordered; and 3) AlignSTS successfully re-align the phonemes in order. However, all of them reconstruct the pitch information in various degrees.
We set an additional baseline AlignSTS (GAN) by changing the diffusion decoder to a GAN-based decoder. Results indicate superior performance of diffusion models in singing voice synthesis.

heading: 4.3 Ablation Study
text: As shown in Table 2, we conduct ablation studies to demonstrate the effectiveness of several designs in AlignSTS, including the rhythm adaptor, the crossmodal alignment, and the skip-connection of pitch representation. We conduct CMOS-Q (comparative mean opinion score of quality) and CMOS-P (comparative mean opinion score of prosody) evaluations on each setting: 1) w/o RA: we stretch the speech rhythm representations defined in Equation 3 using linear interpolation and use it for the subsequent cross-modal alignment. 2) w/o CM: we drop the cross-modal alignment operation and stretch the content representation to the same length as the pitch contour and the adapted rhythm and simply fuse them together using element-wise addition. 3) w/o F0: we cut off the skip-connection of pitch representation in the fusion, i.e., we only combine the adapted rhythm and the aligned content to the mel decoder. The results demonstrate a significant loss of performance when dropping any module. Specifically, it indicates that the rhythm adaptor plays an important role in modeling phonemes, replacing the rhythm adaptor will end up with a singing melody with unintelligible syllables. Removing the cross-modal alignment operation witnesses the degradation of both the audio quality and the prosody naturalness, but the pres-
ence of the adapted rhythm representation allows the mel decoder to implicitly re-align the content as thoroughly as possible, resulting in a slightly smaller quantity of loss compared to w/o RA. As expected, The removal of F0 skip-connection drastically drops the prosody naturalness.

heading: 4.4 Zero-Shot Speech-to-Singing Conversion
text: We conduct extensional experiments on zero-shot STS given only the singing samples and test the model on unseen speech samples. Specifically, we use the identical model architecture and the training pipeline, but carry out a singing-to-singing task and train the model to reconstruct singing samples from the proposed dataset. During the inference stage, we input the unseen speech signal and the target pitch contour to generate the corresponding singing samples. The results are also shown in Table 1 and the task is denoted as AlignSTS (zeroshot). AlignSTS demonstrates great potential in zero-shot STS.

heading: 5 Conclusion
text: In this work, we presented AlignSTS, a speechto-singing model based on modality disentanglement and cross-modal alignment. To achieve better voice quality and improve interpretability, we decomposed the input speech waveform into four variance information and proposed a novel cross-modal fusion mechanism. Specifically, we designed a rhythm adaptor to adjust the rhythm representation to deal with the altered pitch modality, and a crossmodal aligner to re-align the content representation. Finally, we conduct a cross-modal fusion to combine the different components together.
Experimental results demonstrated that AlignSTS achieved superior quality and naturalness compared with several baselines. Ablation studies demonstrated that each design in AlignSTS was effective. Extensive experiments showed the great potential of AlignSTS in zero-shot STS. We envisage that our work could serve as a basis for future STS studies.

heading: 6 Limitations
text: Research on the speech-to-singing conversion is important for human voice study and useful for practical applications such as computer-based music productions or entertainment. However, current STS approaches require an input condition of a fine-grained target F0 contour, which is always unavailable. In addition, the F0 contour of a singing utterance often possesses rich speaker-related infor-
mation, which still needs further disentanglement. Finetuning F0 contours in real applications brings significant extra work. One of our future directions is to simplify the input conditions, such as musical scores. Furthermore, the preliminary attempt at the zero-shot STS task may lead to a better perspective.
Except for positive applications, STS systems may face ethical concerns. With the development of speech/singing voice synthesis technology, the cost of faking an utterance of a specific individual is gradually declining. Researchers need further consideration of the regulation and recognition of speech/singing voice synthesis.

heading: 7 Acknowledgement
text: This work was supported in part by the National Key R&D Program of China under Grant No.2022ZD0162000, National Natural Science Foundation of China under Grant No.62222211, Grant No.61836002 and Grant No.62072397.

heading: A Architecture
text: The architecture and hyperparameters are listed in Table 4. We use three encoders EP, ER, and EC to encode the target F0, the target rhythm, and the source content features:
xP = EP(f0), xR = ER(r), xC = EC(x) (9)
where all encoders are stacks of several convolution layers.
These encoded features are fed into the rhythm adaptor, to both generate the discrete rhythm embeddings and train the rhythm predictor. The
rhythm predictor consists of the cross-attention module and several convolutional layers, predicting the target discrete rhythm embeddings conditioned on the content and pitch features. The target rhythm embeddings are then used to re-align the source content representations. Finally, we carry out an inter-modality fusion across four different representations for re-synthesis.
Each encoder (ER, EP, and EC) consists of two 1-D convolutional layers, where the kernel sizes are 7, 5, and 3, respectively. The linguistic hidden features extracted from wav2vec 2.0 are 32- dimensional, which are widely used for downstream tasks. The hidden size of all the model components is 256. The size of the codebook in the VQ module is set to 6. The cross-attention layer is implemented with one multi-head layer and a feed-forward layer, where the latter consists of a 1-D convolution layer and a fully-connected layer.

heading: B Attention and Alignment Regulation
text: B.1 Attention Windowing
Attention windowing is a widely used technique that controls the "field of view" at each time step. Only a subsequence of key x̂ = [xpt−w, ...,xpt+w] are considered at each query time step t, where w is the window width and pt is the middle position of the window along the key time axis. Specifically, we replace all the values outside the window with −108 before Softmax(·) so that the contribution outside the window is reduced significantly.
B.2 Guided Attention Loss
To make sure that the attention weight matrix is nearly diagonal and monotonic, we adopt guided attention loss. Let αt,n denotes the attention weight at query time step t that attends to key time step n, a guided attention loss can be defined as:
Lattn = 1
TN T∑ t=1 N∑ n=1 αt,nwt,n, where (10)
wt,n = 1− exp ( − ( n N − t T )2 2g2 ) (11)
where T and N are the lengths of the query and the key, respectively. wt,n is the weight distribution of the constraint. g is a hyperparameter used to control the concentration degree, which is set g = 0.1 in practice. If αt, n is far from the diagonal, meaning that the key representations (like the content
features) are placed in a random order, it provides a strong penalty.
B.3 Visualization
A visualization of the attention weights is shown in Figure 4. All the attention weights are extracted from the last layer of the cross-modal aligner and averaged across attention heads.
Comparing subfigure (a) the attention weights without alignment regulation and (b) that with regulation, the importance of attention path regulation is clearly demonstrated. Without alignment regulation, skips and non-monotonic situations occur, causing disordered or even indistinguishable phonemes.
As for the zero-shot scenario, we compare the attention weights during the training stage and during the inference stage, which are shown in subfigures (c) and (d), respectively. Since the training is conducted in a self-supervised manner, the attention pattern demonstrates a perfectly linear pattern, as expected. The cross-modal aligner learns how to uniformly interpolate and stretch the input wav2vec 2.0 features to the same length as the target representations. However, in inference, the aligner is still capable of predicting the specific duration information of each linguistic unit of unseen speech data to a certain degree. As shown in subfigure (d), AlignSTSdemonstrates its generalizability to unseen speech data and the ability to explore modality interaction in self-supervised pre-training.

heading: C Dataset
text: We collected and annotated the speech version of a subset of PopBuTFy to create a paired speech and singing dataset. During the collection, the private information of the speakers was protected. The qualified speakers are requested to read the lyrics of songs that have been sung by themselves. The personal vocal timbre is kept still during the recording process. We carefully select a subset of the collected recordings to create a high-quality dataset. In all, the dataset consists of 152 English pop songs (∼5.5 hours in total) and the respective speech recordings (∼3.7 hours in total) from 16 singers. All the audio files are recorded in a professional recording studio by professional singers, male and female. The recordings are sampled at 22050 Hz with 16-bit quantization. We randomly pick 111 pieces for validation and testing.

heading: D Extensional Experiments
text: SpeechSplit 2.0 is originally designed for aspectspecific voice conversion, not STS tasks. Only manipulating the pitch component of SpeechSplit 2.0 input for STS may cause severe alignment problems, since the latent rhythm information is influenced. To lower the training difficulty and explore the importance of rhythm information, we add a new baseline SpeechSplit 2.0 (w/ SE) that involves the target ground truth rhythm information, i.e., the perturbed spectral envelope, in the training and inference procedure. The performances are listed in Table 3. The results show a great improvement
brought by this "information leak", in that the target rhythm information should not be available in a real situation. Also, the perturbed spectral envelope may still carry residual linguistic information for phoneme reconstruction.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 6, after the conclusion

heading: 3 A2. Did you discuss any potential risks of your work?
text: section 6

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The introduction is in section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 3 and 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 3 and 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The license of a GPU can be easily found online and is known to all.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? section 4.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The documentation of a GPU can be easily found online and is known to all.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 3 and 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? section 4
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? It’s not relevant
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? It’s not relevant and this information is redundant.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? There is no space for that, but yes, our instructions explain how the data would be used and the participants agreed with the terms.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We hire a company to collect the data.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: section 4

heading: Future Work
text: One of our future directions is to simplify the input conditions, such as musical scores. Furthermore, the preliminary attempt at the zero-shot STS task may lead to a better perspective. Except for positive applications, STS systems may face ethical concerns. With the development of speech/singing voice synthesis technology, the cost of faking an utterance of a specific individual is gradually declining. Researchers need further consideration of the regulation and recognition of speech/singing voice synthesis