Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling
Authors: Peng Bai
Yue Zhou
Meizhen Zheng
Wujin Sun
Xiaodong Shi
Venue: EMNLP 2023 Main (2023)
Keywords: Singing voice synthesis
local modeling enhancement
local adaptive weights loss
Hokkien Gezi Opera
Chinese pop song
TL;DR: To address local incongruity problem in singing voice synthesis task, we propose two methods to enhance local modeling in the acoustic model.

Abstract: Singing Voice Synthesis (SVS) strives to synthesize pleasing vocals based on music scores and lyrics. The current acoustic models based on Transformer usually process the entire sequence globally and use a simple L1 loss. However, this approach overlooks the significance of local modeling within the sequence and the local optimization of the hard-to-synthesize parts in the predicted mel-spectrogram. Consequently, the synthesized audio exhibits local incongruities (\textsl{e.g.}, local pronunciation jitter or local noise). To address this problem, we propose two methods to enhance local modeling in the acoustic model. First, we devise a nearest neighbor local attention, where each phoneme token focuses only on the adjacent phoneme tokens located before and after it. Second, we propose a phoneme-level local adaptive weights loss function that enables the model to focus more on the hard-to-synthesize parts of the mel-spectrogram. We have verified the universality of our methods on public Chinese pop song and Hokkien Gezi Opera datasets. Extensive experiments have demonstrated the effectiveness of our methods, resulting in significant improvements in both objective and subjective evaluations when compared to the strong baselines. Our code and demonstration samples are available at 
\href{https://github.com/baipeng1/SVSELM}{https://github.com/baipeng1/SVSELM}.

Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling
Peng Bai Yue Zhou Meizhen Zheng!., Wujin Sun and Xiaodong Shi 'Department of Artificial Intelligence, School of Informatics, Xiamen University, China Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China IInstitute of Artificial Intelligence, Xiamen University, China {baipeng, zhouyue1, midon, sunwuj insestu xwu. xmu. edu. cn mandel@xmu edu. cn
Abstract
vocoder. The acoustic model converts music scores and lyries into acoustic features (e.g., mel- spectrogram), and the vocoder synthesizes audio waveform from acoustic features (Liu et al., 2022).
Singing Voice Synthesis (SVS) strives to syn- thesize pleasant vocals based on music scores and lyrics. The current acoustic models based on Transformer usually process the entire se- quence globally and use simple LI loss. How- ever, this approach overlooks the significance local modeling within the sequence and the local optimization of the hard-to-sye tayt-to-synthesize parts in the predicted mel-speetrogram. Con- sequently, the synthesized audio exhibits local incongruities (e.g.. local pronunciation jitter or noise). To address this problem, we pro- pose methods enhance local modeling in the acoustic model. First, we devise a nearest neighbor local attention, where each phoneme token focuses only the adjacent phoneme to- kens located before and after it. Second, we pro- pose phoneme- level local adaptive weights loss function that enables the model to focus more on the hard-to-symtthsizee parts of the mel-speetrogram. We verify the universality of our methods on public Chinese pop song and Hokkien Gezi Opera datasets. Extensive experiments demonstrate the effectiveness of our methods, resulting in significant improve- ments both objective and subjective evalu- ations when compared to the strong baselines. Our code and demonstration samples are avail- able at http://gtthbbbcmmmbapnggg/g I/SSSe /SSL..
Recently, Transformer (Vaswani et al., 2017) has been widely used in sequence modeling tasks. The acoustic models based on Transformer have showed great performance, including FFFT-NPSS (Blaauw and Bonada, 2020), XiaoiceSing (Lu al., 2020), DeepSinger (Ren al., 2020), FFT-Singer (Liu al., 2022). However, these models still ex- hibits local incongruity the synthesized audio, which is characterized by local pronunciation jit- ter or noise. Local incongruity will bring negative experiences to listeners, so this is a problem that urgently needs to be solved. Lee et al. (2021) also focus on local incongruity problem. In order to improve the accuracy of the local pronunciation, they added a postnet to the model and used adver- sarial training methods, where the voicing- aware discriminator was used to capture the harmonic fea- tures of vocal segments and the noise components of silent segments. Unlike their approaches, we ab- stain from employing post processing networks or adversarial training methods. Instead, we address this problem from the perspective of enhancing local attention and refining loss function.
1 Introduction
Some studies (Yang al., 2020; Watzel et al., 2021; Zhu et al., 2021; Cao et al., 2021) discovered that incorporating additional local attention can en- hance model performance text to speech (TTS) and automatic speech recognition (ASR) tasks. addition, some studies (Lin et al., 2017; George and Marcel, 2021; Li et al., 2022) used the local focus loss function in the image reconstruction task to improve the reconstruction effect. Inspired by the above works, in order address the problem of local incongruity in acoustic models based on Transformer, we propose two methods to enhance local modeling. First, the attention mechanism in these acoustic models is the global contextual
Singing Voice Synthesis (SVS) converts the lyrics into natural and humanlike voice audio accord- ing to the music scores (Yi al., 2019)'. Due to its promising application in fields such as vir- tual singer and music education, SVS has attracted the attention of large number of researchers re- cently (Hono et al., 2019; Lu et al., 2020; Gu et al., 2021; Liu al., 2022; He et al., 2023). SVS sys- tems generally consist of an acoustic model and
"Equal contribution. "Corresponding author 'The music score referred to in this paper does not include
lyries.

---

Real mel spectroram
phonema
duration (hame number)
local acaplive weights loss
Output Predicted mel-spedrogram
c0py
Decodar
Layer
Conformer Block
Pusition Embedding
Longth Reguiatar
matrix M
attention matrix
attention matrix
softmax
Encoder
N Transfomerr Block
Local attention
Global self- attention
Values matric
matriix
Position Embedding
20,15,30
0 X
Local reprasentation
Global representation matrix Rg
x (1-o)
Embedding
Embadding
matrix
Input
Phoneme
Duration
Pttch
Fusion representation matrix R4
Lynos
Music scores
Figure 1: The structure of the model.
self-attention of the entire sequence, and such an at- tention mechanism may disperse the local attention in the sequence. So we devise nearest neighbor local attention to only focus the phoneme to- kens that close to cach other in short distance. Second, the loss function in these acoustic models is generally a simple LI loss. It optimizes each part of the mel-spectrogram with equal weights. This will lead to the sondtto-yytthhiie parts of the mel-spectrogram still being in a difficult posi- tion. So we propose a phoneme-level local adap- tive weights loss to mainly optimize the hard-to- synthesize parts.
versality of our local modeling enhancement methods.
Methods
In Section 2.1, first introduce an overview of our model. We then introduce the nearest neighbor local attention method in Section 2.2, and finally introduce the local adaptive weights loss method in Section 2.3.
2.1 Overview of Model
shown Figure 1, our model consists of an encoder, a length regulator, and a decoder. Encoder The encoder in our model is the same the Transformer block in the FastSpeech2 (Ren et al., 2021). The input of the encoder is the lyrics phoneme. After passing through the embed- ding layer, the phoneme sequence is inputted to the encoder with position embedding.
The main contributions of this paper can be sum- marized as follows:
We devise a nearest neighbor local attention to only focus on the adjacent phoneme tokens located before and after the target phoneme token in a short distance.
Length Regulator The length regulator ex- pands the phoneme-level sequence into the frame- level sequence. The duration of phonemes has been obtained during the data processing stage.
We propose a novel phoneme-level local adaptive weights loss to optimize the local hard-to-synthesizze parts in the predicted mel- spectrogram.
Decoder The decoder in model Con- former (Gulati et al., 2020) block with linear layer. Conformer is convolution-augmenees Transformer. The input of the decoder is the output representation of the encoder, pitch embedding, du-
The extensive experiments on public Chinese pop song and Hokkien Gezi Opera datasets have demonstrated the effectiveness and uni-

---

ration information, and position embedding. Pitch and duration are the important content of the music score. After the entire representation is processed by the decoder, the output of the decoder is a pre- dicted mel-spectrogram.
stead of paying too much attention to other words in the entire lyrics at the same time, so we specially devise nearest neighbor local attention that only focuses on the previous and next phoneme token. We first need to construct masked phoneme-level local attention matrix. Figure demonstrates the constructed masked local attention matrix, in which the frame tokens area corresponding to the current phoneme token is shown in pink, and the frame kens area corresponding the previous phoneme token is green, and the frame tokens area corre- sponding to the next phoneme token purple. {PI, P .P.) is a phoneme sequence, where n is the number of phoneme in certain sample. We define the cur- rent phoneme token ID as i, the locally focused phoneme token number before the current phoneme token as L, and the locally focused phoneme token number after the current phoneme token as r. In the masked phoneme-level local attention matrix M, the attended phoneme tokens are set to 0, while the rest phoneme tokens are negative infinity. Thus M can be represented as follows:
2.2 Nearest Neighbor Local A Attention
Specifically,
The visualization result of the global self- attention of the first Transformer block in the FFT- Singer decoder.
Figure 2:
Mp.g - {{ O, -o0,
p.g E ligeli-l,iz otherwise
(1)
where both p and g are the phoneme ID of the entire phoneme-level representation sequence.
Local Attention Layer In the SVS task, each word token is composed of one or more phoneme tokens. Each phoneme token consists of multiple frame tokens. As shown in Figure from the vi- sualization result of the global self-attention of the first Transformer block in the FFT-Singer decoder, it can be seen that the existing global self-attention primarily focuses on the phoneme tokens area ad- jacent to the current phoneme token. The overall attention still exhibits a predominantly diagonal distribution. In addition, we also see that some phoneme tokens tend to focus distant areas, and mark them with red boxes. Based the above observations, we think that the local attention of we add
We add masked local attention matrix M to the global self-attention matrix. In fact, M acts as a mask role, which preserves the content of the global self-attention matrix corresponding to the position with the content 0 in the matrix M. Therefore, the formula for sequence local representation R3 is follows:
R = softmax(M + QK
Vdz. )
(2)
where Q, K and V are the query, key and value for sequence. dg is the dimension keys. softmax(.) a normalization function.
Fusion of local and global representation As shown in Figure 1, on the basis of local rep- resentation Rd, we use gated unit coefficient to fuse it with the original global representation Rg. In this way, will obtain fusion represen- tation R, of local and global representation. The representation of each phoneme token in the Ry is strengthened.
each phoneme token is insufficient. nearest neighbor local attention layer the decoder to enhance the local attention of each phoneme. We first construct a nearest neighbor local attention ma- trix to only focus on the phoneme tokens that are close to each phoneme token in short distance. We then use gated unit mechanism to fuse local attention representation with global self-attention representation.
The formula for Rg is as follows:
Considering that singers usually focus on the word they are currently singing performance in-
QK
Rg softmax Vd
V
(3)

---

where Q, K, V and d& are similar to Eq. 2.
Predicted meksstrrno
uei
[
lon
Ing
The formula for R is as follows:
R aR (1-a)
(4)
where is Iearnable coefficient, and [0, The formula for as follows:
L1 dilbacc
sigmoid (W ([R;; Ryl)),
where sigmoid(-) is an activation function. W is fully connected layer. [:] is concatenation opera- tion and concats the channel dimension of R4 and Rg.
W1
11
14
m1
m2
m2
softmax
Average
2.3 Local adaptive weights loss
Figure 3:
The schematic diagram of the calculation process for local adaptive weights loss.
Ordinary LI loss optimizes each part of the mel- spectrogram with equal weights, it will re- sult in hard-to-synthesize parts in the predicted mel-spectrogram still difficult to synthesize. We are driven by the motivation to optimize each phoneme region, with particular emphasis on the hard-to syythheie parts within the predicted mel- spectrogram. As shown in Figure 3, we propose phoneme-level local adaptive weights loss to replace LI loss. Specifically, We calculate the phoneme-level adaptive confidence based the phoneme region, and the confidence scores repre- sent the synthesis quality of each phoneme region in the current mel-spectrogram. We also normalize the phoneme-level adaptive confidence scores to phoneme-level adaptive weights, which can dynam- ically update the weight of the phoneme region in the predicted mel-spectrogram. We finally multi- ply the adaptive weights by the values of different phoneme regions in the -peetprcco..ram.
Phoneme-level Adaptive Weight We use the softmax function to normalize the confidence scores and obtain the adaptive weights wi:
mm
Wk
k == ...
m
Finally, we multiply the phoneme-level adaptive weights by the original L1 loss matrix according to the phoneme region. The formula for the local adaptive weights loss value F is follows:
M INN
F
ΣΣ 1=0 0=
wk M,((,) - Mp(t,5)
MN
(8)
where M is the frame number of the sample, and N is the number of mel bins.
Phoneme-level Adaptive Confidence We use the confidence phoneme-level mel-spectrogram to determine the emphasis of the model's learn- ing, which quantifies whether the predicted mel- spectrogram phoneme region is close to or far from the real mel-spectrogram. The confidence scores my calculated as follows:
Experiments
3.1 Datasets
PopCS Dataset PopCS (Liu al., 2022) is a Chi- nese Mandarin pop singing dataset recorded by female professional singer a recording studio. This dataset has total of approximately 5.89 hours and contains 117 Chinese songs, with each audio is sampled at 24kHz and quantized using 16 bits. Gezi Opera Dataset Gezi Opera is one of the tradi- tional local operas of the Hokkien dialect in China, and is a national intangible cultural heritage. This dataset is recorded by 5 professional Gezi Opera singers using mobile phones, consisting of ac- tresses and male actors. The total duration approximately 4.5 hours. Each audio is sampled at
my = Ave M - M)(.,))).
(6)
where & is the phoneme ID and its range is 1, 1,2,.... ,n. M,(i,j) is the phoneme-level pre- dicted mel-spectrogram. My(i.)) is the phoneme- level real mel-spectrogram, i the frame ID, and is the mel bins ID. Ave(-) is an operation that aver- ages the mee-speetro of the phoneme region.

---

48kHz and quantized using 16 bits. This dataset is built by our team
coder in PastSpeech2. Both the Transformer block and Conformer block have multi-head attention. The number of head in multi-head attention is 4. The number of block is set to 4 in encoder and decoder. The Transformer block of the encoder contains 1d convolution, and the size of the kernel is 9. The Conformer block of the decoder contains depthwise 1d convolutional layer with kernel size of 31. In addition, there are two pointwise volutional layers with kernel size of 1. The Con- former block includes gated linear unit activation and swish (Ramachandran et al., 2018) activation. Our SVS model is a two-stage model. Our pro- posed methods are added to an acoustic model, so we also choose HiFi-GAN (Kong et al., 2020) singing model pre-trained by the DiffSinger open- source project as the vocoder to synthesize audio. We use the AdamW (Loshchilov and Hutter, 2017) optimizer. Our maximum training steps are 160k. We train the model on a single A40 GPU.
3.2
Comparative models
The Baseline model is the FFT-Singer (Liu et al., 2022) based on FastSpeech2. The en- coder and decoder are Transformer blocks. The training loss is L1 loss.
The Bas is the model that replaces the Baseline decoder with Conformer blocks.
The training loss is LI loss.
model on the basis of Baseline- TCC replaces the training loss with our local adaptive weights loss.
The Baseline-
The Baseline-T+CAr model adds our nearest neighbor local attention to the the Baseline- T+C model decoder. The training loss is L1 loss.
The Baseline-++C+A+E model is our final model. It adds our nearest neighbor local attention to the Baseline-T+C+A model de- coder. The training loss is local adaptive weights loss.
3.4 Evaluation metrics
In order to evaluate the performance of our pro- posed method, we conduct objective and subjective evaluations. In objective evaluation, we use Mean Cepstral Distortion (MCD) and Mean Spectral Dis- tortion (MSD) to evaluate the timbral distortion of synthesized audio. We also select Gross Pitch Error (GPE), Voicing Decision Error (VDE), and Fo Frame Error (FFE) (Chu and Alwan, 2009) evaluate the Fo track in synthesized audio. We use the code implemented in fairseq 4 for objective evaluation.
The N-Singer (Lee et al., 2021) model is Korean SVS model that focuses on addressing the accuracy of pronunciation in local incon- gruity problems. It includes Transformer- based mel- -eneraerr,,,r, convolutional network- based postnct, and voicing-awar discrimina- tors
The Baseline+GAN model is an adversarial training method that we add to the Baseline, and the method used to generative adversar- ial network (GAN) comes from HiFiSinger (Chen et al., 2020).
For subjective evaluation, we conduct a Mean Opinion Score (MOS) for both real and synthesized audio in the test set. MOS mainly evaluates the human subjective perception of audio naturalness. The rating range of MOS is from to 5. repre- sents the lowest naturalness, and represents the highest naturalness. 10 volunteers participated in the evaluation without disclosing the audio source.
3.3 Model configuration
Our model is modified based FastSpeech2. The encoder is Transformer block and the decoder is Conformer block. In terms of the global configura- tion of the model, the audio sampling rate 24kHz. Because we extract the real mel-spectrogram from real audio, the length of the Fast Fourier Trans- form window is 512 and the hop length is 128. The number of mel bins is 80. The representation di- mension of the input token and channels are all 256. Meanwhile, the dimension of attention is also 256. The encoder has the same settings the en-
4 Results
This section shows the results of the experiments. Section 1 is the main comparison and analysis of the overall objective and subjective results. Sec- tion 4.2 is the experiments about the best num- ber selection for the previous and next masked phoneme tokens. Section 4.3 explores how our
"This dataset can authorized by contacting us and used only for scientific research.
'Due to the lack of open official code for N-Singer, we reproduced it.

---

Dataset
Model
MCD(dB))
MSD(dB))
GPE(%))
VDE(%)
FFF(%))
Baseline
3.4065
1.7164
0.74
3.63
4.05
Baseline-T+C
3.2646
1,638
0.75
3.75
4.18
BasCline-++C+A
3.062
1,5475
0.75
3.46
3.89
PopCS
Baseline-T+C+L
2.9991
1,5452
0.64
3.83
4.2
MAseline-T+++H++
2.8735
1,4809
0.65
3.3
3.67
N-Singer
2.9561
1.5523
3,29
3,85
4.95
Basclinc+GAN
3.0324
1.645
5,88
4.05
8,81
Baseline
3.4694
1.7498
1.5
3.7
4.81
Baseline-T+C
3.3911
1,6924
1.75
3.67
4.97
Baseline-++C+A
3,0314
1.5459
1.75
3.78
5.07
Gezi Opera
BasCline-T+C+L
3.017
1,5595
1.47
3.59
4.71
Becline-T+C+A++
2.931
1.5144
1.34
3.57
4.57
N-Singer
3.021
1.582
5.88
3.69
7.82
Baselinc+GAN 3.0728
1.937
6.94
4.12
9.86
Table I: The objective evaluation results on the PopCS dataset and Gezi Opera dataset. The Baseline model is the FFT-Singer. -T means removing Transformer blocks from the decoder of the Baseline model. +C means adding Conformer blocks to the decoder of the Baseline model. +A means that the loss function is replaced by the default LI loss with the local adaptive weights loss. +L. means adding the nearest neighbor local attention in the decoder.
methods flexibly combine with other models, such as DiftSinger. Section 4.4 is study, which visually demonstrates the effectiveness and universality of our proposed methods.
method has worked. The N-Singer in the sixth line and the Baseline+GAN in the seventh line all GAN-based methods. We can see that the objective evaluation metrics of these two models, especially GPE and FFE, are not ideal. However, we find that the increase of GPE and FFE did not cause a significant decrease in subjective perception. It is necessary to conduct subjective evaluation and we cannot rely too much objective evaluation. We should combine these two aspects for comprehen- sive evaluation.
4.1 Main result analysis
Table shows the results of the objective evalua- tion metrics of the models the PopCS dataset and the Gezi Opera dataset. For every dataset, the first line is the result of the Baseline model. In the second line, when replace the Baseline decoder with Conformer block, the results will decrease. This result validates that the performance of Con- former blocks surpasses the Transformer blocks in the SVS task. The convolution module in the Conformer block is more effective. In the third line, we can see that after replacing the ordinary L1 loss with the local adaptive weights loss, the metrics continue to decrease, which also confirms the ef- fectiveness of our proposed loss. The reason that the local adaptive weights loss can dynamically ex- pand the weights mel-spectrogram, making the model to focus on op- timizing the hard-to-synthesse to-synthesize parts. In the fourth and fifth lines, we add the nearest neighbor local see
Dataset
Model
MOS
Ground Truth
4.43+10.08
Bascline
3.55+10.12
PopCS
Baseline-T+C+AHI T+C+A+L
3.71+10.11
N-Singer
3.65+0.1
Baseline+GAN
3.63++0.13
Ground Truth
4.33+10.09
Baseline
3.46:10.15
Gezi Opera
Baseline-T+CH T+C+A+L
3.61
N-Singer
3.58+
tasntti-synt parts in the
Baseline+GAN
3.51
Table 2: The MOS results on the PopCS and the Gezi Opera dataset. MOS is reported with 95% confident intervals.
attention to the previous model, and further decrease in metrics. Especially in the fifth line, after adding both the nearest neighbor local tention and local adaptive weights loss, the metrics achieve the lowest value. These two lines of results indicate that the nearest neighbor local attention
Table shows the results of the subjective evalu- ation metrics of the models on the PopCS dataset and the Gezi Opera dataset. As shown in the PopCS dataset, our final model achieves the highest MOS

---

value of 3.71. As shown in the Gezi Opera dataset, our final model achieves the highest MOS value of 3.61. In the analysis of the synthesized samples, we also find that the two Singer and GAN,, have positive effects in terms of noise, but the GAN-based methods some- times suffer from pitch inaccuracies on the Gezi Opera dataset, which is proved by the GPE and FFE metrics in Table 1.
FFE the highest at 4.02. When "I=1 and r=1". MCD is 2.8735 and FFE is the lowest at 3.67. In subjective evaluation, "I=1 and r=1" the highest at 3.71. Considering both objective and subjec- tive evaluation results, we believe that under the premise of approximate MCD, we should focus on FFE and MOS. So we choose the setting of "I=1 and r=1". In this setting, it is possible to ensure that the attention is focused the initial and final phonemes of each word. Our main experiments adopt this setting.
methods, N-
On the two datasets of SVS tasks, the Baseline model adding our methods achieves the best results in both objective and subjective evaluations, which fully demonstrates the effectiveness and universal- ity of the two local modeling enhancement methods we proposed.
4.3 Method flexibility
The two methods we proposed can be flexibly com- bined with other models. As long as the original acoustic model utilizes global self-attention and L1 loss, our methods be flexibly applied these models improve performance. We validate the performance when using the DiffSinger model (Liu et al., 2022). DiffSinger is SVS model based on the diffusion model. We have set two scenar- ios, one diffusion condition is the mel-spectrogram predicted by FFT-Singer and the other diffusion condition is the mel-spectrogram predicted by FFT- Singer adding our methods.
4.2 Number of phoneme tokens selection
This section is how determine the best number of phoneme tokens before and after the current phoneme token in the masked phoneme-leve local attention matrix. We conduct experiments on the Baseline-T+C+AHr model in the PopCS dataset, employing seven different scenarios for number selection. As we defined in section 2.2, 1 is the number of locally focused phoneme tokens before the current phoneme token, and is the number of locally focused phoneme tokens after the current phoneme token. The seven scenarios we set are "I=0 and r=", "I=0 and r=1", "I=1 and r=0", "I=1 and r=1", 1 "I=1 and r=2", "I=2 and r=1" and "1=2 and r=2". The reason for our setting is that we observe the self-attention matrix from Figure 2. it can be seen that roughly one to two phoneme tokens range before and after each current phoneme token are mainly being focused on. Finally, select the optimal parameters through objective and subjective evaluation metrics.
Dataset
Model
MOS
Ground Truth DiffSinger
4.43::0.08
PopCS
3.86++0.12
DiffSinger+Our 3.91#0.11
Ground Truth
4.33+0.09
Gezi Opera
DiffSinger 3.82+t0.1
DiffSinger+Our 3.86
Table 4:
The MOS results on PopCS and Gezi Opera datasets. MOS is reported with 95% confident inter- vals. Diffsinger represents diffusion based on the mel- spectrogram predicted by FFT-Singer. DiffSinger+ represents the diffusion of DiffSinger based on the mel- spectrogram predicted by FFT-Singer adding our meth- ods.
1 land r
MCD(dB),
FFE(% (FE(%) %)
MOS
1-0 r
2.93
3.77
1 r=
2.9026
3.87
|=1 r=0 2.8581
3.86
3.65+0.13
As shown Table 4, we can see that after adding our methods, the MOS score of DiffSinger+Our is higher than that of the basic DiffSinger. This demonstrates that the audio synthesized by Diff- Singer on the basis of FFT-Singer adding our meth- ods is better. This further validates the flexibility and practicality of our methods. We also find that the DiffSinger+Our model can indeed solve some of the local incongruity problems that exist in the DiffSinger model, especially local pronunciation jitter.
|=] r=1
2.8735
3.67
3.71
|= r=2 2.838
4.02
3.6110.12
r= 2.8615
3.83
1=2
2.883
3.89
Table 3: The MCD, FFE, and MOS results of Baseline- T+C+A+L model on the PopCS dataset. MOS is re- ported with 95% confident intervals.
As shown Table 3, we can see that when "I=1 and r=2", MCD achieves the lowest at 2.838, but

---

4.4 Case study
phoneme sequence of the Iyrics is "sp ua sp k sp m ia e i ien an an ts" sp e e sp specee e e e e sp". At the following segment "e seee e e e e", , the medium and high frequency bands of the corresponding parts in the mel-spectrogram are marked with red boxes. We can see that the predicted content in Figure 5(c) is more detailed than that in Figure 5(b). The Figure 5(c) is closer to the real mel-spectrogram in Figure 5(a). This result also reflects that our methods can indeed improve the local quality of the predicted mel-spectrogram.
In order to more intuitively demonstrate the lo- cal modeling enhancement effects of our proposed methods in the mel-spectrogram prediction pro- cess, Figure 4 and Figure 5 respectively show the mel-speetrogram visualization results of certain sample on the PopCS dataset and the Gezi Opera dataset by different models.
5 Related Works
5.1 Singing voice synthesis
Figure 4: The meesspectr visualization of the sample in the PopCS dataset. (a) is the real mel-spectrogram of the sample. (b) is the mel- spectrogram predicted by the Baseline model. (c) is the mel-spectrogram predicted by the Baseline-T+C+AAr model.
At the end of the 1950s, the earliest computer music project studied by Bell Labs realized the SVS, and representative physical acoustic model Kelly and Lochbaum (Cook, 1996). After the develop- ment of traditional methods such as unit splicing and statistical parameters, the current mainstream method is based on deep learning.
With the rapid development of deep learning (Lin al., 2023), the implementation of SVS search mainly adopts various neural network ar- chitectures. The current SVS research can be di- vided into integrated end-to-end and fully end-to- end model. The integrated end-to-end SVS sys- tem consists of an acoustic model and vocoder. XiaoiceSing (Lu et al., 2020), DeepSinger (Ren et al., 2020), ByteSing (Gu et al., 2021), HiFiSinger (Chen al., 2020), XiaoiceSing2 (Wang al., 2022) are all integrated end-to-end models. Among the above models, the acoustic model ByteSing is based on a recurrent neural network, while the rest acoustic models are all based on Transformer. HiFiSinger and XiaoiceSing2 adopt adversarial training. The vocoders used in these models are usually WORLD(Morise et al., 2016), HiFi-GAN (Kong al., 2020) MelGAN (Kumar al., 2019), etc. Liu al. (2022) designed an acous- tic model named DiffSinger based on the diffusion probabilistic model. They proposed a shallow dif- fusion mechanism to improve audio quality and accelerate inference. Visinger(Zhang et al., 2022a) and Visinger2 (Zhang et al., 2022b) are fully end- to-end models, and the acoustic model is trained together with the vocoder. This type of model can avoid the problem of error accumulation.
Figure 5: The mel-spectrogram visualization of the same sample in the Gezi Opera dataset. (a) is the real mel-spectrogram of the sample. spectrogram predicted by the Baseline model. (c) is the mel-spectrogram predicted by the Buseline-T+C+AHH model.
(b) is the mel-
As shown in Figure 4, the phoneme sequence of the lyrics is "<BOS> ie h ou h uei <SEP>j iou uan Ic uol I g uo <SEP> z ai lill ou <SEP> nil sh ao b u m ian x iang q uo <SEP> h ai Is uan b uul l uo <EOS>". and there are breathing silent segments in "<BOS>", "<SEP>", and "<EOS>" token segments. We highlight these parts with red boxes the mel-spectrogram. In Figure 4(b), these three phoneme segments are relatively noisy com- pared to Figure 4(a). In Figure 4(c), we can see that the mel-spectrogram predicted by our methods is closer to the real mel-spectrogram and has less noise at these three segments.
In Gezi Opera, in order to express a certain emotion, artistic modifications are often made to the pronunciation of the final or after the final. Artistic modifications usually involve long-term multi rhythmic singing. As shown in Figure 5, the
In our work, we propose two local modeling enhancement methods for the acoustic model based on Transformer.

---

5.2 Local modeling enhancement
synthesized audio. We are moving towards a better solution completely solve this problem.
In the research of TTS, Yang et al. (2020) pro- posed two local modeling enhancement methods to improve the performance of models based on the self-attention mechanism. One is the enhancement of local relative position perception representation for sequence representation. Another approach learnable gaussian bias to enhance local represen- tation in self- attentonnni In the research of ASR, some research works (Watzel et al., 2021; Zhu et al., 2021; Cao et al., 2021) also enhance recognition ac- curacy by strengthening local modeling, In natural language processing research, in order to enhance the local attention of sequence, Zhang et al. (2020) added a syntax-guides self-attention layer to im- prove the Transformer's performance in reading comprehension tasks. Li et al. (2021) proposed a syntax-aware local attention method to improve BERT. In the task of facial photo sketch synthesis, Yu et al. (2023) proposed an additional local self- attention for local correlation. Local attention can achieve better synthesis results by integrating with global self-attention.
Limitations
We propose two local modeling enhancement meth- ods the SVS acoustic model. The methods can effectively alleviate the problem of local incon- gruity in synthesized audio. However, our work still has some limitations. (1) The proposed nearest neighbor local attention representation in this paper only verifies the effectiveness of the fusion with the global self-attention representation. (2) The nearest neighbor local attention brings additional compu- tational requirements and increases the demand for GPU resources.
(3) We find that our methods cannot completely solve the problem of local incon- gruity in SVS. Our methods significantly improve the effect of the silent or breathing segments, and can provide some relief for other segments. Our method still has room for improvement, and we think that
further control the high, medium, and low frequency bands in the mel-spectrogram locally through the loss function.
In this work, we enhance the local modeling ability the acoustic model from two perspectives: adding local attention and designing local adaptive weights loss.
Ethics Statement
We use two SVS datasets in the experiments. one is PopCS dataset and the other one Gezi Opera dataset built by us. The PopCS dataset has been authorized by the owner. The Gezi Opera dataset is collected, organized, and produced by ourselves. We have also deleted the singer's personal infor- mation to ensure their privacy. We strictly adopt blind evaluation mechanism during subjective evaluation.
6 Conclusion
In the Chinese pop song and Hokkien Gezi Opera singing voice synthesis tasks, we propose two local modeling enhancement methods acoustic model based on Transformer to improve the quality of the predicted mel-spectrogram. One method is to enhance local attention for each phoneme token in the decoder and fuse local attention representa- tion with the original global self-attention repre- sentation. Another method involves employing novel phoneme-leve local adaptive weights loss optimize the hard-to-symine toytttt---ttiizze parts of the pre- dicted mel-spectrogram. We conduct extensive ex- periments on the Chinese pop song and Hokkien Gezi Opera datasets, and both objective and sub- jective evaluation metrics show the effectiveness and universality of our methods in enhancing lo- cal modeling for mel-spectoogram prediction. Our methods are simple and practical, and can be flexibly incorporated into acoustic models based on Transformer or Conformer. In summary, our meth- ods can greatly alleviate the local inconsistency problem in SVS tasks and improve the quality of
A Acknowledgements
This work supported by the Project of Techno- logical Innovation 2030 "New Generation Artifi- cial Intelligence" (N0.2020AAA0107904) and the Major Scientific Research Project of the State Lan- guage Commission in the 13th Five-Year Plan (No. WT135-38). We thank the reviewers for their valu- able comments. Special thanks to the actors and actresses who participated in the audio recording of Gezi Opera.
References
Merlijn Blaauw and Jordi Bonada. 2020. Sequence-to- sequence singing synthesis using the feed-forward transformer. In IEEE International Conference on

---

Acoustics, Speech and Signal Processing, pages 7229- 7233.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere. Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexan- dre de Brébisson, Yoshua Bengio, and Aaron C. Courville. 2019. Melgan: Generative adversarial net- works for conditional waveform synthesis. In Neu- ral Information Processing Systems, pages 14881- 14892.
Songjun Cao, Yike Zhang, Xiaobing Feng, and Long Ma. 2021. Improving speech recognition accuracy of local POI using geographical models. In IEEE Spoken Language Technology Workshop, pages 180- 185.
Gyeong-Hoon Lee, Tac-Woo Kim, Hanbin Bac, Min- Ji Lee, Young-Ik Kim, and Hoon- Young Cho. 2021. N-singer: A non-autoregressive korean singing voice synthesis system for pronunciation enhancement. In 22nd Annual Conference of the International Speech Communication Association, pages 1589-1593.
Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie- Yan Liu. 2020. Hifisinger: Towards high-fidelity neural singing voice synthesis. CoRR, 0bs/2209.00176.
Wei Chu and Abeer Alwan. 2009. Reducing FO frame
error of FO tracking algorithms under noisy con- unvoiced/voiced classification fron- tend. In IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 3969-3972.
Bo Li, Yonggiang Yao. Jingru Tan, Gang Zhang, Feng- wei Yu, Jianwei Lu, and Ye Luo. 2022. Equalized focal loss for dense long-tailed object detection. In IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 6980-6989.
ditions with
Perry R Cook. 1996. Singing voice synthesis: History, current work, and future directions. Computer Music Journal.
Zhongli Li, Qingyu Zhou, Chao Li, Ke Xu, and Yunbo Cao. 2021. Improving BERT with syntax-aware local attention. In Findings of the Association for Compu- tational Linguistics, pages 645-653.
Anjith George and Sébastien Marcel. 2021. Cross modal focal loss for RGBD face anti-spoofing. In IEEE Conference Computer Vision and Pattern Recognition. pages 1891.
Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023. Automated schol- arly paper review: Concepts, technologies, and chal- lenges. Information Fusion, 98: 18:101330.
Yu Gu, Xiang Yin, Yonghui Rao, Yuan Wan, Benlai Tang, Yang Zhang. Jitong Chen, Yuxuan Wang, and Zejun Ma. 2021. Bytesing: A chinese singing voice synthesis system using duration allocated encoder- decoder acoustic models and wavernn vocoders. In 12th International Symposium on Chinese Spoken Language Processing, pages 1-5.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. IEEE International Conference on Computer Vision, pages 2999-3007.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. 2022. Diffsinger: Singing voice synthe-
Anmol Gulati, James Qin, Chung- Brang-Cheng&aaa Chiu, Niki Parmar, Yu Zhang. Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer:
sis via shallow diffusion mechanism. In Thirty-Sixth AAAI Conference on Artificial Intelligence, pages 11020-11022
trans- former for speech recognition. In 2st Annual Con- ference of the International Speech Communicution Association, pages 5036-5040.
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR,
1.1./001.
Peiling Lu, Jie Wu, Jian Luan, Xu Tan, and Li Zhou. 2020. Xiavicesing: A high-quality and integrated singing voice synthesis system. In 21st Annual Con- ference of the International Speech Communication Association, pages 1306-1310.
Jinzheng He, Jinglin Liu, Zhenhui Ye, Rongjie Huang, Chenye Cui, Huadai Liu, and Zhou Zhao. 2023. based singing voice synthesis. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 236-248.
Rmssinger: Realistic-music-
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. 2016. WORLD: A vocoder- based high-quality speech synthesis system for real-time applications. IEICE Trans. Inf Syst., 19-7(():1187-1188)
Yukiya Hono, Kei Hashimoto, Keiichiro Oura, Yoshi- hiko Nankaku, and Keiichi Tokuda. 2019. Singing voice synthesis based on generative adversarial net- works. In IEEE International Conference on Acous- rics, Speech and Signal Processing, pages 6955- 6959.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. 2018. Searching for activation functions. In Inter- national Conference on Learning Representations, Workshop Track.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for effi- cient and high fidelity speech synthesis. In Annual Conference on Neural Information Processing Sys- lems.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Ye Liu. 2021. Fastspeech 2: Fast and high-quality end-to-end text to speech. In 9th International Conference on Learning Represen- tations.

---

Yi Ren, Xu Tan, Tao Qin, Jian Luan, Zhou Zhao, and Tie Yan Liu. 2020. Deepsinger: Singing voice syn- thesis with data mined from the web. In The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1979-1989.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Annual Conference on Neural Informa- lion Processing Systems, pages 5998-6008.
Chunhui Wang. Chang Zeng, and Xing He. 2022. Xi- aoicesing 2: A high-fidelity singing voice synthe- sizer based generative adversarial network. CoRR. abs/22101 14666.
Tobias Watzel, Ludwig Kürzinger, Lujun Li, and Ger- hard Rigoll. 2021. Induced local attention for trams- former models in speech recognition. In Speech and Computer 23rd International Conference, pages 795-806.
Shan Yang, Heng Lu, Shiyin Kang, Liumeng Xue, Jinba Xiao, Dan Su, Lei Xie, and Dong Yu. 2020, On the localness modeling for the self-attention based end- to-end speech synthesis. Neural Networks, 125:121- 130.
Yuan-Haa Yi. Yang Ai, Zhen- Hua Ling, and Li-R Li-Rong Dai. 2019. Singing voice synthesis using deep autore- gressive neural networks for acoustic modeling. In 20th Annual Conference of the International Speech Communication Association, pages 2593-2597.
Wangbo Yu. Mingrui Zhu, Nannan Wang. Xiaoyu Wang. and Xinbo Gao. 2023. An efficient transformer based global and local self-attention for face photo-sketch synthesis. IEEE Trans. Image Process.. 42:483-495.
Yongmao Zhang, Jian Cong, Heyang Xue, Lei Xie, Pengcheng Zhu, and Mengxiao Bi. 2022a. Visinger: Variational inference with adversarial learning for end-to-end singing voice synthesis. In IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing, pages 7237-7241.
Yongmao Zhang, Heyang Xue, Hanzhao Li, Lei Xie, Tingwei Guo, Ruixiong Zhang, and Caixia Gong. 2022b. Visinger 2: High-fidelity end-to-end singing voice synthesis enhanced by digital signal processing synthesizer. CoRR, abs/221 1. 1.02903.
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2020. Sg-net: Syntax-guided machine reading comprehension. In The Thirty- Fourth AAA/ Conference on Artificial In- lelligence, pages 9636-9643.
Qiu-Shi Zhu, Jie Zhang, Ming-Hui Wu, Xin Fang. and Li-Rong Dai. 2021. An improved wav2vec 2.0 pre- training approach using enhanced local dependency modeling for speech recognition. In 22nd Annual Conference of the International Speech Communica- tion Association, pages 4334-4338.


Reviews:
[{"review_id": "HhuYIYvxeE", "reviewer": "Reviewer_SqYD", "summary": "", "questions": "In Section 4.4, “Figure 5: The mel-spectrogram visualization of same sample in the Gezi Opera dataset”, the mel-spectrogram predictions in Figure 5b and Figure 5c look fairly similar, with no noticeable improvement in Figure 5c generated by the proposed method compared to Figure 5b from the baseline model. So it seems the proposed method may not significantly enhance the predictions beyond the baseline in this case, unless I am missing some finer details.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:In Section 4.4, “Figure 5: The mel-spectrogram visualization of same sample in the Gezi Opera dataset”, the mel-spectrogram predictions in Figure 5b and Figure 5c look fairly similar, with no noticeable improvement in Figure 5c generated by the proposed method compared to Figure 5b from the baseline model. So it seems the proposed method may not significantly enhance the predictions beyond the baseline in this case, unless I am missing some finer details.\n\n**A1**:Thank you for your question. Figure 5c and Figure 5b do not show significant improvement in the red box section. The mel-spectrogram in the example in this paper was not exported as a vectorgraph, and the insertion into the paper resulted in poor performance. We have corrected this issue. Through Figure 5, we want to illustrate that even in non breathing or silent segments, our model has some improvement compared to the baseline model. We can see that Figure 5c shows a significant improvement in the breathing or silent (voiceless) segments compared to Figure 5b. In the voiced segments, there is also an improvement in the medium and high frequency section, where the mel-spectrogram is more detailed and not as blurry as Figure 5b. In summary, we found that our model has significant improvements in the breathing or silent (voiceless) segment, but not particularly significant in the voiced segment. This is because the original model actually has a relatively serious problem of local incongruity in the breathing or silent (voiceless) segments, while the voiced segments are not serious.\n\n**Q2**:The introduction briefly discusses some prior methods for enhancing local modeling in singing voice synthesis, such as N-singer and adversarial training methods. It is necessary to include a comparative analysis of these methods with the proposed approach in the experiments. Merely relying on subjective MOS experiments comparing with DiffSinger is insufficient; additional objective experimental comparisons should be incorporated.\n\n**A2**:Thank you for your question. \nFirstly, we have added a comparative experiment of N-singer and adversarial training methods. The innovation of N-Singer lies in the addition of PostNet to the predicted spectrum and the use of GAN training. N-Singer conducted experiments on Korean datasets. In fact, in our previous research, we found that N-Singer did not have open source code, and we actually reproduced it according to the description in the paper. However, we are not sure if our reproduction is completely consistent with the original paper, so we did not easily release the results. Below is an objective and subjective comparison between our results, N-Singer model results and adversarial training methods (Baseline+GAN) results in PopCS and Gezi Opera dataset.\n\n| Model | MCD(dB)$\\downarrow$ | MSD(dB)$\\downarrow$ | GPE(\\%)$\\downarrow$ | VDE(\\%)$\\downarrow$ | FFE(\\%)$\\downarrow$ |\n| --- | --- | --- | --- | --- | --- |\n| Baseline | 3.4065 | 1.7164 | 0.74 | 3.63 | 4.05 |\n| Baseline-T+C| 3.2646 | 1.638 | 0.75 | 3.75 | 4.18 |\n| Baseline-T+C+A | 3.062 | 1.5475 | 0.75 | 3.46 | 3.89 |\n| Baseline-T+C+L | 2.9991 | 1.5452 | **0.64** | 3.83 | 4.2 |\n| Baseline-T+C+A+L | **2.8735** | **1.4809** | 0.65 | **3.3** | **3.67** |\n| N-Singer | 2.9561 | 1.5523 | 3.29 | 3.85 | 4.95 |\n| Baseline+GAN | 3.0324 | 1.645 | 5.88 | 4.05 | 8.81 |\n#### **Subjective results in PopCS dataset.**\n\n| Model | MOS$\\uparrow$ |\n| --- | --- |\n| Ground Truth | 4.43$\\pm$0.08 |\n| Baseline | 3.55$\\pm$0.12 |\n| Baseline-T+C+A+L | **3.71**$\\pm$0.11 |\n| N-Singer | 3.65$\\pm$0.1 |\n| Baseline+GAN | 3.63$\\pm$0.13 |\n\n#### **Objective results in Gezi Opera dataset.**\n\n| Model | MCD(dB)$\\downarrow$ | MSD(dB)$\\downarrow$ | GPE(\\%)$\\downarrow$ | VDE(\\%)$\\downarrow$ | FFE(\\%)$\\downarrow$ |\n| --- | --- | --- | --- | --- | --- |\n| Baseline | 3.4694 | 1.7498 | 1.5 | 3.7 | 4.81 |\n| Baseline-T+C| 3.3911 | 1.6924 | 1.75 | 3.67 | 4.97 |\n| Baseline-T+C+A | 3.0314 | 1.5459 | 1.75 | 3.78 | 5.07 |\n| Baseline-T+C+L | 3.017 | 1.5595 | 1.47 | 3.59 | 4.71 |\n| Baseline-T+C+A+L | **2.931** | **1.5144** | **1.34** | **3.57** | **4.57** |\n| N-Singer | 3.021 | 1.582 | 5.88 | 3.69 | 7.82 |\n| Baseline+GAN | 3.0728 | 1.937 | 6.94 | 4.12 | 9.86 |\n#### **Subjective results in Gezi Opera dataset.**\n\n| Model | MOS$\\uparrow$ |\n| --- | --- |\n| Ground Truth | 4.33$\\pm$0.09 |\n| Baseline | 3.46$\\pm$0.15 |\n| Baseline-T+C+A+L | **3.61**$\\pm$0.12 |\n| N-Singer | 3.58$\\pm$0.11 |\n| Baseline+GAN | 3.51$\\pm$0.12 |\n\nFrom the above four tables, it can be seen that our model outperforms N-Singer or Baseline+GAN in both objective and subjective evaluation metrics on the PopCS and Gezi Opera dataset. We found that the GAN-based method performed poorly on GPE, VDE and FFE these three metrics. N-Singer performs better than Baseline+GAN, because PostNet in N-Singer is a convolutional network that can pay some attention to local features.\n\nSecondly, we have added objective evaluation results for Section4.3 as shown in the table below .\n#### **Objective evaluation results.**\n\n| Dataset | Model | MCD(dB)$\\downarrow$ | MSD(dB)$\\downarrow$ | GPE(\\%)$\\downarrow$ | VDE(\\%)$\\downarrow$ | FFE(\\%)$\\downarrow$ |\n| --- | --- | --- | --- | --- | --- | --- |\n| PopCS | DiffSinger | 3.1452 | 1.7365 | **0.72** | **3.67** | **4.08** |\n| PopCS | DiffSinger+Our | **3.0564** | **1.6706** | 0.9 | 3.74 | 4.24 |\n| Gezi Opera | DiffSinger | 3.9807 | 2.0012 | 1.7 | 3.84 | 5.1 |\n| Gezi Opera | DiffSinger+Our | **3.9145** | **1.9815** | **1.64** | **3.67** | **4.89** |\n\nFrom the table, we can see that adding our method to DiffSinger resulted in a decrease in both MCD and MSD in the PopCS dataset, while there was almost no significant change in GPE, VDE and FFE. In the Gezi Opera dataset, all metrics have shown a certain decrease. Combining subjective and objective evaluation metrics, the overall improvement effect for DiffSinger is not significant. The actual situation is indeed so, because the DiffSinger model itself has good performance, and our method was not originally designed to improve the performance of the DiffSinger model. Although this method has a small improvement on the DiffSinger model, we did significantly improve the performance of FFT-Singer, which is the starting point of this study. Our purpose in conducting this experiment is to verify that our method is flexible and effective even when combined with the DiffSinger model.\n\n**Q3**:It is customary to provide a demonstration webpage for SVS work, as subjective listening is necessary to demonstrate its superiority. Although the author has included a few samples in the supplementary material, the limited quantity makes it difficult to convince readers.\n\n**A3**:Thank you for your question. A demonstration webpage is indeed necessary. We have created a demonstration webpage, but due to the fact that rebuttal does not allow adding links, we are unable to present it to you at the moment.\n\n**Q4**:This paper is presented with clarity and simplicity, but it lacks innovation. The introduction of a mask matrix to enhance local attention and modifications in the loss function are common tricks that are not novel. These modifications could potentially be implemented with just a few lines of code. Moreover, it is crucial to note that the methods proposed in the paper do not fully address the issue of local incongruity in SVS, which limits the potential impact of this work.\n\n**A4**:Your evaluation is very relevant. The improvement workload for the FFT-Singer model in this paper is a bit small, but the two improvement methods proposed in this paper are both simple and effective, and our methods have good robustness and generalization. As far as we know, we constructed the Gezi Opera dataset for the first time in singing voice synthesis research, and the process of constructing the dataset was very difficult and time-consuming. In addition, we validated the effectiveness of the methods and the effectiveness of the Gezi Opera dataset. We have truly expanded the scope of research on singing voice synthesis. To be honest, the method proposed in this paper has not yet fully solved the problem of local incongruity, but this is only the beginning of our research and we are still exploring.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Demonstrate a comprehensive understanding of earlier studies in SVS tasks through careful summarization.\n2. Present two easy approaches to tackle the challenges of local modeling and local-based enhancement for singing voice synthesis. Fuse local and global representation, introduce a phoneme-level local adaptive weights loss function focusing more on the hard-to-synthesize parts of the mel-spectrogram.\n3. The Method section provides a thorough and clear explanation of each part of the model in detail.\n4. A multitude of ablation experiments were conducted to underscore the effectiveness of the proposed methods.", "reasons_to_reject": "1. The introduction briefly discusses some prior methods for enhancing local modeling in singing voice synthesis, such as N-singer and adversarial training methods. It is necessary to include a comparative analysis of these methods with the proposed approach in the experiments. Merely relying on subjective MOS experiments comparing with DiffSinger is insufficient; additional objective experimental comparisons should be incorporated.\n2. It is customary to provide a demonstration webpage for SVS work, as subjective listening is necessary to demonstrate its superiority. Although the author has included a few samples in the supplementary material, the limited quantity makes it difficult to convince readers.\n3. This paper is presented with clarity and simplicity, but it lacks innovation. The introduction of a mask matrix to enhance local attention and modifications in the loss function are common tricks that are not novel. These modifications could potentially be implemented with just a few lines of code. Moreover, it is crucial to note that the methods proposed in the paper do not fully address the issue of local incongruity in SVS, which limits the potential impact of this work.", "questions_to_authors": "In Section 4.4, “Figure 5: The mel-spectrogram visualization of same sample in the Gezi Opera dataset”, the mel-spectrogram predictions in Figure 5b and Figure 5c look fairly similar, with no noticeable improvement in Figure 5c generated by the proposed method compared to Figure 5b from the baseline model. So it seems the proposed method may not significantly enhance the predictions beyond the baseline in this case, unless I am missing some finer details.", "paper_topic_and_main_contributions": "This paper aims to improve the local modeling ability of the acoustic model in the field of singing voice synthesis. It propose two local modeling enhancement methods, a nearest neighbor local attention and a phoneme-level local adaptive weights loss function which effectively enhance local modeling for mel-spectrogram prediction and greatly alleviate the local incongruity problem in SVS tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mmbpJclDFp", "reviewer": "Reviewer_cHLv", "summary": "", "questions": "in Section 4.3, it is not clear how to make the proposed method effective on the Diffsinger, which is a diffusion model.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:in Section 4.3, it is not clear how to make the proposed method effective on the Diffsinger, which is a diffusion model.\n\n**A1**:DiffSinger is an acoustic model for singing voice synthesis based on the diffusion probabilistic model. When training and inference, the auxiliary encoder-decoder model FFT-Singer is an input for DiffSinger. During training, the output of the FFT-Singer encoder is an input of the DiffSinger denoiser. During inference, the predicted mel-spectrogram by the FFT-Singer decoder is an input to DiffSinger. The DiffSinger model adds noise to predicted mel-spectrogram and then trains the denoiser model to predict the noise. Our local modeling enhancement method effectively improves the performance of FFT-Singer, so we replaced the auxiliary encoder-decoder model of DiffSinger with our model instead of FFT-Singer. These two situations are the two diffusion conditions mentioned in the paper. One diffusion condition is the mel-spectrogram predicted by FFT-Singer and the other diffusion condition is the mel-spectrogram predicted by FFT-Singer added our methods. We have all retrained DiffSinger's denoiser based on these two conditions. The experimental results are shown in Table 4, the audio synthesized by DiffSinger on the basis of FFT-Singer added our methods is better.\n\n**Q2**:The paper claims to address the problem of local modeling in the transformer-based acoustic model. Conformer[1] has been proven to be very effective in modeling both local and global dependencies by combining convolution neural networks and transformers. However, the paper doesn't compare the proposed method with Conformer.\n\n**A2**:In fact, we compared our proposed method with Conformer (Baseline-T+C). As shown in Table 1, Baseline-T+C is the replacement of the baseline model decoder from Transformer to Conformer. Baseline-T+C+A, Baseline-T+C+L and Baseline-T+C+A+L are three scenarios where our proposed methods are added to the Conformer. We can see that Conformer is effective as a decoder, and with the addition of the two methods we proposed, the effect has been further improved.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a nearest neighbor local attention mechanism and a proposed phoneme-level local adaptive weights loss function to enhance local modeling. The experiments have demonstrated the simplicity and effectiveness", "reasons_to_reject": "The paper claims to address the problem of local modeling in the transformer-based acoustic model. Conformer[1] has been proven to be very effective in modeling both local and global dependencies by combining convolution neural networks and transformers. However, the paper doesn't compare the proposed method with Conformer.\n\n[1] Gulati A, Qin J, Chiu C C, et al. Conformer: Convolution-augmented Transformer for Speech Recognition[J]. Interspeech 2020, 2020.", "questions_to_authors": "in Section 4.3, it is not clear how to make the proposed method effective on the Diffsinger, which is a diffusion model.", "paper_topic_and_main_contributions": "This paper proposes two methods to enhance local modeling in the acoustic model of singing voice synthesis, including a nearest neighbor local attention mechanism and a proposed phoneme-level local adaptive weights loss function.  In the nearest neighbor local attention mechanism, each phoneme token focuses only on the adjacent phoneme tokens located before and after it. Additionally, the phoneme-level local adaptive weights loss function enables the model to focus more on the hard-to-synthesize parts of the mel-spectrogram. The evaluations on the publicly available Chinese pop song and Hokkien Gezi Opera datasets showed the effectiveness and universality of the proposed model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hBFP6LwQdS", "reviewer": "Reviewer_DiTD", "summary": "", "questions": "1. The paper relies on a pre-trained HiFi-GAN vocoder to synthesize audio from the predicted mel-spectrogram. However, HiFi-GAN is trained on natural speech data, which may not be suitable for singing voice synthesis. It would be interesting to see how the proposed methods perform with a vocoder that is specifically trained on singing voice data.\n2. How do the proposed methods handle the cases where the phoneme duration is very long or very short, or where the phoneme boundaries are not clear?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:The paper relies on a pre-trained HiFi-GAN vocoder to synthesize audio from the predicted mel-spectrogram. However, HiFi-GAN is trained on natural speech data, which may not be suitable for singing voice synthesis. It would be interesting to see how the proposed methods perform with a vocoder that is specifically trained on singing voice data.\n\n**A1**:Thank you for your question. Sorry for the inaccurate expression of the vocoder used in our paper. The HiFi-GAN vocoder we used is actually a HiFi-GAN singing model pre-trained by the DiffSinger open-source project, which is specially designed for singing voice synthesis. They trained HiFi-GAN singing model using singing data instead of speech data.\n\n**Q2**:How do the proposed methods handle the cases where the phoneme duration is very long or very short, or where the phoneme boundaries are not clear?\n\n**A2**:Thank you for your question. \nFirstly, our method still performs well even when the phoneme duration is very long or very short. In the Gezi Opera dataset, the longest phoneme duration is nearly 10 seconds, and the shortest is a few tenths of a second. After the experiment, we listened to the synthesized samples and found that even in these situations, the synthesized samples performance was good. So our method is not affected by the phoneme duration being too long or too short. Our method has good robustness. \nSecondly, the situation where the phoneme boundary you mentioned is not clear does not exist in our research. Because in the data processing stage, we first need to align the phonemes with the audio in terms of time. Once aligned, we assume that the phoneme boundaries are clear. The phoneme boundary you mentioned is not clear, which means that the alignment model cannot clearly segment the phoneme duration, which was not covered in this research.\n\n**Q3**:The paper only focuses on Chinese pop song and Hokkien Gezi Opera datasets, which are relatively small and specific. It is not clear how well the proposed methods can generalize to other languages, genres, and styles of singing voice synthesis.\n\n**A3**:Thank you for your question. Firstly, our current paper does only demonstrate significant effects on Chinese pop song and Hokkien Gezi Opera datasets. In fact, we have also verified the performance on the Beijing Opera dataset, but due to some copyright and privacy issues of Beijing Opera dataset, we can not display it. After we solve this problem, we will also add the results soon. The above can only indicate that it is effective in Chinese and its dialects, as well as in popular songs and opera styles. Secondly, to our knowledge, there are currently public datasets for singing voice synthesis in English, Japanese, and Korean, and we will also verify them based on your suggestions. In addition, we also conducted experiments on the English speech synthesis dataset LJ Speech, and the subjective and objective experimental results showed that our method is effective.\n\n**Q4**:The paper does not provide a detailed analysis of the local incongruity problem, such as the causes, types, and frequency of local pronunciation jitter or noise.\n\n**A4**:Thank you for your question. As you mentioned, N-Singer (Lee et al., 2021) is indeed a model for solving similar problems. They mentioned in their paper that 'there are word skipping and repetition problems in the synthesized singing voices because of incorrect attention alignment'. From their conclusion, it can be seen that word skipping and repetition problems are due to incorrect attention alignment. Strictly speaking, they are solving word skipping and repetition problems, while we are solving local pronunciation jitter and noise problems. Actually, these are two different problems. We were inspired by them and analyzed our problem from the perspective of attention mechanism. We visualized and analyzed the global self-attention matrix of the first Transformer block in the FFT-Singer decoder, as shown in Figure 2. We found deficiencies in the global self-attention mechanism, so we made improvements. When we wrote the paper, the problem analysis section did not mention why we analyzed the problem from the perspective of attention mechanism. We will supplement the conclusion in the N-Singer.\n\n**Q5**:It also does not compare the proposed methods with other existing methods that aim to address the same problem, such as PostNet (Lee et al., 2021).\n\n**A5**:Thank you for your question. We did not compare our results with the results of N-Singer (Lee et al., 2021) in our paper. The innovation of N-Singer lies in the addition of PostNet to the predicted spectrum and the use of GAN training. N-Singer conducted experiments on Korean datasets. In fact, in our previous research, we found that N-Singer did not have open source code, and we actually reproduced it according to the description in the paper. However, we are not sure if our reproduction is completely consistent with the original paper, so we did not easily release the results. Below is an objective and subjective comparison between our results and N-Singer's results in PopCS and Gezi Opera dataset.\n#### **Objective results in PopCS dataset.**\n\n| Model | MCD(dB)$\\downarrow$ | MSD(dB)$\\downarrow$ | GPE(\\%)$\\downarrow$ | VDE(\\%)$\\downarrow$ | FFE(\\%)$\\downarrow$ |\n| --- | --- | --- | --- | --- | --- |\n| Baseline | 3.4065 | 1.7164 | 0.74 | 3.63 | 4.05 |\n| Baseline-T+C| 3.2646 | 1.638 | 0.75 | 3.75 | 4.18 |\n| Baseline-T+C+A | 3.062 | 1.5475 | 0.75 | 3.46 | 3.89 |\n| Baseline-T+C+L | 2.9991 | 1.5452 | **0.64** | 3.83 | 4.2 |\n| Baseline-T+C+A+L | **2.8735** | **1.4809** | 0.65 | **3.3** | **3.67** |\n| N-Singer | 2.9561 | 1.5523 | 3.29 | 3.85 | 4.95 | \n\n#### **Subjective results in PopCS dataset.**\n\n| Model | MOS$\\uparrow$ |\n| --- | --- |\n| Ground Truth | 4.43$\\pm$0.08 |\n| Baseline | 3.55$\\pm$0.12 |\n| Baseline-T+C+A+L | **3.71**$\\pm$0.11 |\n| N-Singer | 3.65$\\pm$0.1 |\n\n#### **Objective results in Gezi Opera dataset.**\n\n| Model | MCD(dB)$\\downarrow$ | MSD(dB)$\\downarrow$ | GPE(\\%)$\\downarrow$ | VDE(\\%)$\\downarrow$ | FFE(\\%)$\\downarrow$ |\n| --- | --- | --- | --- | --- | --- |\n| Baseline | 3.4694 | 1.7498 | 1.5 | 3.7 | 4.81 |\n| Baseline-T+C| 3.3911 | 1.6924 | 1.75 | 3.67 | 4.97 |\n| Baseline-T+C+A | 3.0314 | 1.5459 | 1.75 | 3.78 | 5.07 |\n| Baseline-T+C+L | 3.017 | 1.5595 | 1.47 | 3.59 | 4.71 |\n| Baseline-T+C+A+L | **2.931** | **1.5144** | **1.34** | **3.57** | **4.57** |\n| N-Singer | 3.021 | 1.582 | 5.88 | 3.69 | 7.82 | \n\n#### **Subjective results in Gezi Opera dataset.**\n\n| Model | MOS$\\uparrow$ |\n| --- | --- |\n| Ground Truth | 4.33$\\pm$0.09 |\n| Baseline | 3.46$\\pm$0.15 |\n| Baseline-T+C+A+L | **3.61**$\\pm$0.12 |\n| N-Singer | 3.58$\\pm$0.11 |\n\nFrom the above four tables, it can be seen that our model outperforms N-Singer model in both objective and subjective evaluation metrics on the PopCS and Gezi Opera dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper conducts extensive experiments on publicly available Chinese pop song and Hokkien Gezi Opera datasets. The results show that the proposed methods achieve significant improvements in both objective and subjective evaluations compared to the strong baseline. The methods can also alleviate the local incongruity problem and improve the quality of synthesized audio.", "reasons_to_reject": "The paper only focuses on Chinese pop song and Hokkien Gezi Opera datasets, which are relatively small and specific. It is not clear how well the proposed methods can generalize to other languages, genres, and styles of singing voice synthesis.\nThe paper does not provide a detailed analysis of the local incongruity problem, such as the causes, types, and frequency of local pronunciation jitter or noise. It also does not compare the proposed methods with other existing methods that aim to address the same problem, such as PostNet (Lee et al., 2021).", "questions_to_authors": "1. The paper relies on a pre-trained HiFi-GAN vocoder to synthesize audio from the predicted mel-spectrogram. However, HiFi-GAN is trained on natural speech data, which may not be suitable for singing voice synthesis. It would be interesting to see how the proposed methods perform with a vocoder that is specifically trained on singing voice data.\n2. How do the proposed methods handle the cases where the phoneme duration is very long or very short, or where the phoneme boundaries are not clear?", "paper_topic_and_main_contributions": "The paper is about improving Chinese pop song and Hokkien Gezi Opera singing voice synthesis by enhancing local modeling. The paper proposes two methods to address the problem of local incongruity in Singing Voice Synthesis (SVS) based on Transformer. Local incongruity is characterized by local pronunciation jitter or local noise in the synthesized audio. The first method is to devise a nearest neighbor local attention that only focuses on the adjacent phoneme tokens located before and after the current phoneme token. The local attention representation is fused with the global self-attention representation using a gated unit mechanism. The second method is to design a phoneme-level local adaptive weights loss that optimizes the hard-to-synthesize parts of the predicted mel-spectrogram. The loss function calculates the phoneme-level adaptive confidence scores based on the distance between the predicted and real mel-spectrogram, and normalizes them to adaptive weights. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9JWVaTu41S", "reviewer": "Reviewer_knLR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:Typos Grammar Style And Presentation Improvements.\n\n**A1**:Thank you for pointing out these obvious casing and singular/plural grammar errors, which were caused by our careless writing. After carefully reviewing the paper, we have corrected the grammar errors that exist in the paper.\n\n**Q2**:The paper has some innovations, but the innovations are a bit small. One is to integrate local attention into global attention, and the other is to replace equal weights on the loss with adaptive weights.\n\n**A2**:Your evaluation is very relevant. The improvement workload for the FFT-Singer model in this paper is a bit small, but the two improvement methods proposed in this paper are both simple and effective, and our methods have good robustness and generalization. As far as we know, we constructed the Gezi Opera dataset for the first time in singing voice  synthesis research, and the process of constructing the dataset was very difficult and time-consuming. In addition, we validated the effectiveness of the methods and the effectiveness of the Gezi Opera dataset. We have truly expanded the scope of research on singing voice  synthesis.\n\n**Q3**:The proposed method is integrated on the DiffSinger model in Section 4.3, but the results show that the improvement is not significant. From the MOS results in Table4, the improvement is weak. It is recommended to add objective evaluation to further prove the improvement effect of the algorithm.\n\n**A3**:We have added objective evaluation results as shown in the table below.\n#### **Objective evaluation results.**\n\n| Dataset | Model | MCD(dB)$\\downarrow$ | MSD(dB)$\\downarrow$ | GPE(\\%)$\\downarrow$ | VDE(\\%)$\\downarrow$ | FFE(\\%)$\\downarrow$ |\n| --- | --- | --- | --- | --- | --- | --- |\n| PopCS | DiffSinger | 3.1452 | 1.7365 | **0.72** | **3.67** | **4.08** |\n| PopCS | DiffSinger+Our | **3.0564** | **1.6706** | 0.9 | 3.74 | 4.24 |\n| Gezi Opera | DiffSinger | 3.9807 | 2.0012 | 1.7 | 3.84 | 5.1 |\n| Gezi Opera | DiffSinger+Our | **3.9145** | **1.9815** | **1.64** | **3.67** | **4.89** |\n\nFrom the table, we can see that adding our method to DiffSinger resulted in a decrease in both MCD and MSD in the PopCS dataset, while there was almost no significant change in GPE, VDE and FFE. In the Gezi Opera dataset, all metrics have shown a certain decrease. Combining subjective and objective evaluation metrics, the overall improvement effect for DiffSinger is not significant. The actual situation is indeed so, because the DiffSinger model itself has good performance, and our method was not originally designed to improve the performance of the DiffSinger model. Although this method has a small improvement on the DiffSinger model, we did significantly improve the performance of FFT-Singer, which is the starting point of this study. Our purpose in conducting this experiment is to verify that our method is flexible and effective even when combined with the DiffSinger model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1)The paper proposes two innovations. One innovation is nearest neighbor local attention. Another innovation is local adaptive weights loss.\n(2)The experimental results of the paper prove the effectiveness of the proposed method. The ablation experiments verify the effectiveness of the two innovations of the method.\n(3)The proposed method can be flexibly integrated on other models. \n", "reasons_to_reject": "(1)The paper has some innovations, but the innovations are a bit small. One is to integrate local attention into global attention, and the other is to replace equal weights on the loss with adaptive weights.\n(2)The proposed method is integrated on the DiffSinger model in Section 4.3, but the results show that the improvement is not significant. From the MOS results in Table4, the improvement is weak. It is recommended to add objective evaluation to further prove the improvement effect of the algorithm.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes two local modeling enhancement methods in acoustic model to improve the quality of the predicted mel-spectrogram. One method is to fuse the local attention representation with the original global self-attention representation. Another method is to replace the L1 loss with a local adaptive weights loss. The experimental results on the Chinese pop song and Hokkien Gezi Opera datasets show that the proposed method can improve the quality of synthesized audio.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "(1)Section 2.3, \"Specifically, We calculate the...\" should be \"Specifically, we calculate the...\".\n(2)Section 6 Limitations, \"Our methods significantly improves the...\" should be \"Our methods significantly improve the...\".\n(3)Section 6 Ethics Statement, \"the experiments. one is PopCS dataset...\" should be \"the experiments. One is PopCS dataset...\". \n"}}]

Author rebuttal:

Meta-review: **Summary:**
The paper presents two methods to improve local modelling to address the issue of local incongruity in the context of singing voice synthesis (SVS):
1)	A nearest neighbour local attention mechanism to enhancing local modelling by incorporating adjacent phoneme tokens' information.
2)	A phoneme-level local adaptive weights loss function, targeting the harder-to-synthesize parts of Mel-spectrograms.
The paper conducts experiments on Chinese pop songs and Hokkien Gezi Opera datasets to validate the effectiveness of the proposed methods.
 
 
**Pros:**
 
- The paper effectively summarizes prior work on SVS task.
 
- The proposed methods are described clearly, and their effects are demonstrated by ablation experiments.
 
- The proposed methods consistently outperform alternatives methods (N-Singer and adversarial methods) in both objective and subjective evaluation metrics.
 
- The authors provide a detailed rebuttal addressing reviewers' questions and concerns, effectively clarifying their approach and results.
 
 
**Cons:**
- Reviewers 2 and 4 found the innovations presented in the paper to be relatively small and simplistic, mostly involving integrating local attention with global attention and adapting loss weights.
 
- While the paper addresses concerns about adding a comparative analysis with N-Singer and adversarial training methods, it lacks a detailed analysis of local incongruity and comparison with existing methods (Reviewer 3, Reviewer 4).
 
- The methods' effectiveness is primarily demonstrated on specific datasets (Chinese pop songs and Hokkien Gezi Opera), raising questions about the generalization of these methods to other languages, genres, and styles of singing (Reviewer 3).

- The suggested dataset lacks both open-source availability and comprehensive explanations, which hinders its ability to facilitate progress in the field (Reviewer 1).



Reviewer 1: SqYD
Reviewer 2: cHLv
Reviewer 3: DiTD
Reviewer 4: knLR
Decision: Accept-Main