UniLG: A Unified Structure-aware Framework for Lyrics Generation
Year: 2023

Abstract:
As a special task of natural language generation, conditional lyrics generation needs to consider the structure of generated lyrics1 and the relationship between lyrics and music. Due to various forms of conditions, a lyrics generation system is expected to generate lyrics conditioned on different signals, such as music scores, music audio, or partially-finished lyrics, etc. However, most of the previous works have ignored the musical attributes hidden behind the lyrics and the structure of the lyrics. Additionally, most works only handle limited lyrics generation conditions, such as lyrics generation based on music score or partial lyrics, they can not be easily extended to other generation conditions with the same framework. In this paper, we propose a unified structure-aware lyrics generation framework named UniLG. Specifically, we design compound templates that incorporate textual and musical information to improve structure modeling and unify the different lyrics generation conditions. Extensive experiments demonstrate the effectiveness of our framework. Both objective and subjective evaluations show significant improvements in generating structural lyrics.

heading: 1 Introduction
text: Great progress has been made in natural language generation (NLG) with pre-trained language models in recent years (Lewis et al., 2020; Radford et al., 2019; Brown et al., 2020). Lyrics generation is also a special task of NLG (Chen and Lerch, 2020; Gill et al., 2020). Different from general natural language, lyrics eventually need to be presented with music after the composition. This requires the lyrics to follow song-writing rules (i.e., the structure of lyrics), such as clear paragraphs with chorus and verse concepts. However, most
∗*Corresponding Author. 1The structure of lyrics in our work means that lyrics have
the chorus and verse parts among the sentences.
previous works ignore the musical concepts behind the lyrics and do not consider the structure of lyrics (Sheng et al., 2021; Qian et al., 2022). To explicitly model the structure of lyrics, some works introduce additional structural labels (e.g., sentence-level chorus and verse label), which inevitably require much effort for additional human annotation (Potash et al., 2015; Lu et al., 2019). To avoid the huge annotation cost, other works either adopt predefined formats (e.g., the number of syllables in each sentence) or linguistic tags (e.g., PoS, Part-of-Speech) to inject structural information (Li et al., 2020; Castro and Attarian, 2018). Nevertheless, given that those methods cannot directly indicate the structure of lyrics, the generated lyrics are still difficult to realize the musical concepts (e.g., chorus and verse). Moreover, most works only focus on certain lyrics generation conditions, such as generating lyrics given music scores or partially-finished lyrics etc., which hinders the application of a lyrics generation model in various scenarios.
To mitigate the issues in previous works, we propose a unified structure-aware lyrics genera-
2https://en.wikipedia.org/wiki/Beat_(music) 3https://en.wikipedia.org/wiki/Bar_(music) 44/4 denotes that each beat is a 1/4 note and each bar has 4 beats. To simplify the description, we state our method with a 4/4 time signature, for it’s widely used in songwriting. The English version is provided in Appendix I due to space issues.
983
tion framework named UniLG. As illustrated in Figure 1, the chorus parts of the songs share the same melody, so that the corresponding lyrics follow a similar pattern. Such a phenomenon inspires us that shared musical signals indicate the structure of lyrics and can be used to infer the relation across lyrics. Therefore, we design a compound template (i.e., a sequence of tuples) that incorporates both textual and musical information to model the structure of lyrics. The template is designed with rhythmic concepts in mind, and it can be extracted from different sources (e.g., audio, music score, etc.). As shown in Figure 2, the general interface in the template enables UniLG to generate lyrics based on various conditional signals without re-training the model. Additionally, we propose a cycle-consistency loss to enforce the reconstruction of the musical information from the generated lyrics to further improves the performance. To verify our proposed framework, we collect a test dataset named Song8k with chorus and verse labels for each sentence. Both objective and subjective evaluations on the test dataset demonstrate the effectiveness of our framework.
In summary, the main contributions of this work are as follows:
• we propose a unified structure-aware lyrics generation framework named UniLG; • we design a compound template that incorporates textual and musical information to achieve structure modeling and enable lyrics generation in various conditions; • we introduce a cycle-consistency loss to validate the impact of musical information and further boost the performance;
• extensive experiments demonstrate the effectiveness of our method that achieves better structural modeling in lyrics generation.

heading: 2 Related Work
text: The existing lyrics generation approaches can be categorized into two types: 1) free lyrics generation, which generates lyrics either from scratch or based on some prefix prompts (Radford et al., 2019; Brown et al., 2020); and 2) conditional lyrics generation, which generates lyrics conditioned on control signals (e.g., music score, audio, etc.) (Saeed et al., 2019; Fan et al., 2019). In this work, we focus on conditional lyrics generation.
Recent works have shown the effectiveness of pre-trained language models in NLG (Lewis et al., 2020; Brown et al., 2020; Radford et al., 2019). As a special task of NLG, lyrics generation also follows the trend of using pre-trained language models. However, the pre-trained language models are trained with general text corpus and fail to consider the structure of lyrics (e.g., chorus and verse parts of the song), which is a salient feature for lyrics. Several works adopt pre-trained Transformer variants, such as GPT-2, as the backbone to improve the performance of lyrics generation but ignore the structure of lyrics as well (Zhang et al., 2020; Lee et al., 2019; Bao et al., 2019; Sheng et al., 2021; Qian et al., 2022). To achieve structural modeling, some works attempt to annotate the structural information of lyrics, however, this requires additional expensive human annotation (Potash et al., 2015; Lu et al., 2019). To avoid human labeling, SongNet chooses corpus with pre-defined formats (e.g., Ci5 and Sonnet), while some works regard the linguistic tags of lyrics as the structure information of lyrics (Li et al., 2020; Castro and Attarian, 2018). However, SongNet can not provide diverse representations for sentences, and the construction of linguistic tags is inconvenient and not humanfriendly. In addition, these methods cannot represent the structure of lyrics explicitly. Moreover, most previous lyrics generation works ignore the musical properties hidden behind the lyrics, that is, the lyrics will eventually be presented together with the music. To overcome all the above limitations, we propose a compound template in our framework that can be conveniently constructed. It provides discriminative representations and incorporates both textual and musical information to
5https://en.wikipedia.org/wiki/CI
achieve structural modeling.

heading: 3 Method
text: Our proposed unified framework for lyrics generation (UniLG) contains two highlights: 1) it considers the structure of lyrics in the generation; 2) it can handle different lyrics generation conditions with different control signals, such as the music score, or music audio, or partial lyrics, etc.
For the structure of lyrics, as illustrated in Figure 1, the melody implies the structure of lyrics, which can be leveraged for lyrics structure modeling. However, large-scale (melody, lyrics) parallel data is generally difficult to obtain. We, therefore, propose using rhythm patterns6 that preserves the inter-correlation of lyrics as musical information to explicitly represent the structure of lyrics. As explored in previous works (Ju et al., 2021; McAuliffe et al., 2017), the defined rhythm patterns can be efficiently extracted from lyrics and different rhythmic sources (e.g., music score, music audio, etc.) without extra human annotation.
For handling various control signals of different
6The rhythm patterns usually indicate when a note happens and how long it lasts. In this paper, the rhythm patterns only indicate when a note happens.
lyrics generation conditions, the model should have the capacity to process different types of inputs, such as music score, music audio, rhythm patterns, partially-finished lyrics, etc. Therefore, we design a compound template (i.e., a sequence of tuples) that can incorporate both textual and musical information. So any type of input can be converted into a compound template, and then lyrics can be generated based on the compound template.
Figure 2 illustrates the overview of our proposed lyrics generation framework UniLG. We propose an intermediate compound template as a bridge between the rhythmic sources (e.g., audio, music score, etc.) and lyrics in UniLG. Specifically, the lyrics generation is decomposed into a two-stage pipeline consisting of an Input-to-Template stage and a Template-to-Lyric stage. In this section, we first describe the compound template in detail. We then present the two stages during training respectively. Finally, we discuss the inference procedure of UniLG to illustrate how to handle various control signals in different lyrics generation conditions with our unified framework.

heading: 3.1 Compound Template
text: To model the structure of the lyrics, the compound templates are designed to incorporate both musical
and textual information. As shown in Figure 3, a compound template consists of five components, Masked Lyric M (or Lyric L), Bar A, Beat B, Segment S, and Intro-position P . These components can be categorized into three aspects: semantic information, musical information, and textual information. The details of these aspects with corresponding components are as follows:
Semantic Information Aspect We introduce Lyric Symbols and Masked Lyric Symbols to leverage the pre-trained language model and achieve semantic control.
Lyric Symbols: We denote the Lyric, a sequence of Chinese character tokens, as L = (l1, l2, ..., ln) = (li) n i=1, where li stands for the ith element of L, li ∈ C ∪ E , n is the length of L. C refers to the set of Chinese characters and E = { ⟨/s⟩, ⟨bos⟩, ⟨eos⟩ } is a set of special tokens, including the separation token between sentences ⟨/s⟩, the start of sequence token ⟨bos⟩, and end of sequence token ⟨eos⟩.
Masked Lyric Symbols: We denote the Masked Lyric as M = (mi)ni=1, where mi stands for the ith element of M , and mi ∈ C ∪ E ∪ {⟨m⟩}, where ⟨m⟩ stands for mask token, which is widely used in masked language modeling (MLM) (Kenton and Toutanova, 2019; Lewis et al., 2020).
Musical Information Aspect As illustrated in Figure 3, the inter-correlation of lyrics can be preserved in the musical information, and two kinds of musical symbols, Beat Symbols and Bar Symbols, are designed to represent the musical information at a different level.
Beat Symbols: The Beat B = (bi)ni=1 denotes the local musical information, where bi (the ith element of B, bi ∈ B∪E), denotes the local musical information of mi and li. B = {Beati}3i=0, and Beat0, Beat1, Beat2, and Beat3 stand for 1st, 2nd, 3rd, and 4th beat in a bar.
Bar Symbols: The Bar A = (ai)ni=1 denotes the global musical information, where ai (the ith element of A, ai ∈ A∪E) denotes the bar information of the bi. A = {Bari}511i=0, and token Barj stands for the jth bar7. And ai also indicates that the word mi and li are supposed to be sung at bar ai.
Textual Information Aspect Similar to SongNet, the Intro-position and segment symbols are adopted to model the textual information at word and sen-
7the number of bars is no more than 512 from our data.
tence level (Li et al., 2020). In the following sections, we name the sub-sequence of any component between special symbols in E as a sentence.
Segment Symbols: The segment symbols provide global textual information to the compound template. We denote Segment as S = (si)ni=1, where si (the ith element of S, si ∈ S ∪ E) denotes the sentence position of the mi and li. S = {Segi}255i=0 and Segj stands for the jth sentence. For example, the lyrics shown in Figure 3 is the 10th and 11th sentences (Seg10 and Seg11) . Intro-Position Symbols: The Intro-Position P = (pi) n i=1 denotes the local textual information, where pi (the ith element of P , pi ∈ P ∪ E) denotes the local position within the sentence of the mi and li. P = {Posi}31i=0 and the token Posi stands for the ith reversed local position within the sentence or the distance to the end of token of the sentence. For example, in Figure 3, the Pos8 means this position is 8 tokens away from the last token of the corresponding segment (Pos0 in Seg10).
The compound template is a tuple sequence consisting of five components, including Masked Lyric M (or Lyric L), Beat B, Bar A, Segment S, and Intro-Position P . As shown in the blue and green dotted box in the bottom of Figure 3, we can construct the template based on these components.

heading: 3.2 Input-to-Template Module
text: In this subsection, we discuss the construction procedure of the compound template given the lyrics L = (li) n i=1 in length n during training. To be specific, we first extract symbols (defined in Section 3.1) from lyrics L. Then, we combine them to construct the compound template:
(1) Masked Lyric M = (mi)ni=1. Similar to MLM, the M is constructed by randomly masking 85% of elements that are not in E of lyrics L.
(2) Bar A = (ai)ni=1 and Beat B = (bi) n i=1. According to the time signatures, the bar information A can be obtained given beat information B. And the Beat B is extracted from lyrics L through a Lyric-to-Beat model (details in Appendix A), which predicts rhythm patterns in B for given lyrics L.
(3) Segment S = (si)ni=1, and Intro-poistion P = (pi) n i=1. As shown in Figure 3, the special tokens of E appear in the position for all components, in other words, they have the same format information, and the S and P can be extracted from either M , B, A, or L. For a sequence Q ∈ {M,B,A,L},
S can be construct by counting the number of ⟨/s⟩ (if the number is c) before each position i and replace Segc with the corresponding ith token not in E . Similarly, for a sequence Q ∈ {M,B,A,L}, P can be constructed by counting the distance away from the nearest ⟨/s⟩ in the right (if the distance is c) for each position i and replace Posc with the corresponding ith token not in E .
As shown in the blue dotted box in the bottom of Figure 3, the compound template T, a tuple sequence including Masked Lyric M , Beat B, Bar A, Segment S, and Intro-Position P , can be formulated as T = (ti)ni=1 = (<mi, bi, ai, si, pi>) n i=1.

heading: 3.3 Template-to-Lyric Module
text: Through the Input-to-Template module, we construct the template T and obtain paired lyrictemplate data. With such data, we adopt a pretrained encoder-decoder Transformer language model MT5 as backbone (Xue et al., 2021b). Figure 3 illustrates the procedure of the Template-toLyric module and the details are as follows:
Encoder Inputs and Decoder Inputs We define H0E and H 0 D as the inputs of the Encoder and the Decoder respectively and their formulations are:
H0E =ET = LN(EM + EB + EA + ES + EP ) H0D =EL = LN(EL + EB + EA + ES + EP ), (1)
where LN(∗) denotes the layer normalization and E∗ stands for token embedding sequences of ∗. Similar to the definition of the T in Section 3.2, the L denote the compound template that is a sequence of tuples: L = (li)ni=1 = (<li, bi, ai, si, pi>) n i=1, where the M is replaced by L in the T to obtain the L as shown in Figure 3.
Encoder and Decoder The Encoder and Decoder each consist of N Transformer layers. HtE and HtD denote the output of the t
th encoder layer and decoder layer respectively. As shown in Figure 3, the output of Encoder and Decoder HNE and H N D are:
HNE =Encoder(H 0 E) HND =Decoder(H N E , H 0 D ∗ MaskD),
(2)
where MaskD denotes a causal decoder mask. And there is a projection layer for HND to get the final distribution of the predicted lyrics.
Training with Cycle-consistency Loss The main training loss is to minimize the negative loglikelihood over the lyrics L = (li)ni=1 given the
template T = (ti)ni=1 as shown in the gray dotted line in Figure 4:
LT2L =− logP (L|L,T) =− Σni=1 logP (li|l<i; t1, ..., tn),
(3)
where L = (li)ni=1 denote the compound template. The l<i stands for sequence (l1,l2,...,li−1).
As illustrated by the orange dotted line in Figure 4, we introduce the cycle-consistency loss (CCL) to enhance the impact of musical information. The Lyric-to-Beat model reconstructs the beat sequence from the predicted lyrics of the language model. The formulation of CCL is as follows:
L′L2B = − logP (B|L′) = −Σni=1 logP (bi|b<i; l′1, ..., l′n),
(4)
where L′ = (l′i) n i=1 denotes the predicted lyrics by the language model, and B = (bi)ni=1 denotes the Beat of input template T as in Figure 4.
Finally, the training objective of Template-toLyric model is to minimize the loss Ltot:
Ltot = LT2L + α ∗ L′L2B, (5)
where α is a hyper-parameter to weigh CCL.

heading: 3.4 Inference Procedure
text: In this subsection, we describe the inference procedure of UniLG for various lyrics generation conditions. The major steps are shown in Algorithm 1,
including "Beat Construction", "Masked Lyric Construction", and "Components Construction". Given the template T by Algorithm 1, the Template-toLyric module generates the Lyric L and the L autoregressively.
"Beat Construction" is a method to construct the Beat B from raw input X (e.g., beat, lyric, music score, audio, etc.)8. "Beat Construction" consists of "Sentence Length Generation" and "Beat Generation". "Sentence Length Generation" generates a sequence of numbers with each number denoting the length of one sentence9. "Beat Generation" generates the Beat based on the outputs from "Sentence Length Generation". For example, if "Sentence Length Generation" generate a sequence S = [3, 2], "Beat Generation" may return B = [⟨bos⟩, Beat1, Beat3, Beat0, ⟨/s⟩, Beat0, Beat1, ⟨/s⟩, ⟨eos⟩].
To achieve content controllable generation, we use keywords K to construct the Masked Lyrics M . Based on the keyword 10, the model generates the lyrics in the MLM manner. The keywords can be either user-specified or sampled from the training corpus, which should appear in the generated lyrics. "Masked Lyric Construction" is a method to construct the masked lyrics M condition on Beat B and keywords K. Similar to the construction of P and S in Section 3.2, given Beat B, M can be constructed by replacing the token that is not in E with the mask token or keywords in K randomly.
"Components Construction" is a method to obtain the other components given M and B as described in Section 3.2, and organize all components for the template.

heading: 4 Experimental Settings

heading: 4.1 Dataset
text: We collect lyrics of 249,007 Chinese pop songs from Internet as the base of our experiments.
Lyric-Template Dataset. We use the pre-trained Lyric-to-Beat model to extract the lyric-template dataset from 249,007 lyrics. We randomly select 8000 songs for the validation and test set respectively, and the remaining songs are used for training. The data statistics are shown in Appendix B.
8The details of the Lyric-to-Beat, MIDI-to-Beat, and Audio-to-Beat modules are discussed in Appendix D.
9The sentence means the sub-sequence between special symbols.
10If keywords are empty, we will randomly select some popular words as keywords.
Algorithm 1 Template Construction In Inference Input: X: the raw input;K: keywords. Output: T, generated compound template. Def Beat Construction(X): Case of X:
a beat sequence : B = X a lyric sequence: B = Lyric-to-Beat(X) a MIDI file: B = MIDI-to-Beat(X) a audio file: B = Audio-to-Beat(X) None: S = Sentence Length Generation()
B = Beat Generation(S) end Case return B B = Beat Construction(X) M = Masked Lyric Construction(B, K) return T = Components Construction(B, M )
Additional Dataset: Song8k. We also annotate 8,000 songs with structure labels (sentence-level chorus and verse label) for evaluation and we name this dataset Song8k. For dataset settings, we use all 8,000 songs for further evaluation in the Templateto-Lyric module.

heading: 4.2 Baselines in Model Comparison
text: We compare with two baselines in the experiments: 1) MT5, a pre-trained Transformer language model (Xue et al., 2021b); 2) SongNet, a format-controlled text generation model (Li et al., 2020). MT5 and SongNet construct their inputs with the same corpus as the lyric-template dataset. MT5, SongNet, and UniLG have similar parameters and all models use the same pre-trained model as initialization for a fair comparison. The details of the model configuration, training, and decoding settings are reported in Appendix E and G.

heading: 4.3 Objective Evaluation Metrics
text: We use three kinds of objective evaluation metrics: general level, low level, and high level (more details can be found in Appendix F).
General Level: Besides perplexity (PPL), we use Integrity metric to evaluate the sentence integrity (Li et al., 2020), which calculates the average probability of the separation token given previous tokens.
Low Level: We use Format F1 and Beat F1 to evaluate the degree of consistency between the generated lyrics and the given textual format (Segment and Intro-Position) and rhythm patterns (Beat) in the template.
High Level: We use Song8k and a pre-trained model (details are in Appendix C) to evaluate the quality of the structure of generated lyrics. Specifically, the model predicts a chorus or verse label for each sentence in generated lyrics and compares it with the human annotations to obtain Structure F1.

heading: 4.4 Subjective Evaluation Metrics
text: As illustrated in Section ??, the Beat is important for the compound template and may have a big impact on our framework. We conduct subjective experiments for "Beat Construction", including "Sentence Length Generation" and "Beat Generation". Besides, we also conduct subjective experiments for model comparison. For each subjective experiment, we invite 43 annotators to evaluate the generated lyrics. Each annotator is required to score lyrics concerning four aspects. Each aspect is rated with an opinion score from 1 to 5 (from bad to excellent). The four aspects are as follows: 1) Coherence: the overall consistency of the topic of the entire song; 2) Fluency: the fluency of the semantic correlation within a sentence and between the sentence; 3) Correlation: the structural or semantic similarity among sentences, such as the distribution of words and corresponding relationships of sentences; 4) Fascination: the degree of fascinating sentences in annotators’ opinion.

heading: 5 Experiments Results
text: In this section, we report and analyze both objective and subjective experimental results. We also show some cases in Appendix H to verify the ability of UniLG to handle different generation conditions.

heading: 5.1 Objective Results
text: Model Comparison We compare MT5, SongNet, and UniLG on the Song8k and the test set of the lyric-template dataset. The results of the model comparison (in Table 1) show that MT5 achieves the best results in PPL and Integrity. Our UniLG outperforms baselines in Format F1, Beat F1, and Structure F1.The Structure F1 shows that our framework does generate the lyrics with better structure, which indicates that the musical information improves the structural modeling.
Ablation Study We further ablate our UniLG to verify the impact of musical and textual information as well as the CCL. From the results shown in Table 2, we see that the textual information (Seg&Pos), musical information (Bar&Beat), and
CCL play crucial roles in the overall performance. These modules of our framework show significant improvement, especially on the metrics of Beat F1 and Structure F1. The CCL may enhance the musical information to boost performance in Format F1, Beat F1, and Structure F1, but at the same time may introduce noise and cause degradation in general metrics (PPL and Integrity). The effectiveness of the CCL further proves that the musical information behind the lyrics does benefit the structure-aware lyric generation. We notice that the musical information (Bar&Beat) degrades the performance of the framework more than the textual information and CCL. This may be due to there being extra position embeddings for input data in MT5 model. When it comes to music information, missing Bar&Beat leads to a complete loss of information, while missing the Seg&Pos only partially loses position information.

heading: 5.2 Subjective Results
text: Template Construction As the template directly affects the Template-to-Lyric module, we perform the subjective evaluation on different settings of "Sentence Length Generation" and "Beat Generation" in Algorithm 1 to investigate the impact of the compound template.
For "Sentence Length Generation", we have 2 candidate settings: 1) Random, the length of the sentence is randomly chosen from 6 to 12; 2) 2- gram, the next sentence length is generated conditioned on the length of the previous sentence. We generate 40 songs in 6 to 16 sentences for each setting. Given two number sequences generated by "Sentence Length Generation" of two settings, the two Beat can be generated by the same method of "Beat Generation", whose setting is chosen randomly. The results in Table 3 show that Random and 2-gram strategies achieve comparable performance and different sentence length generation strategies have little influence on models.
For "Beat Generation", we have 3 candidate settings: 1) Random, the beat information for each character is randomly chosen from B; 2) Rule, the beat is non-decreasing than the previous one; 3) Sample, we compute the statistics of the beat sequence of each length in the lyric-template dataset and sample the beat sequence conditioned on the sequence length. We generate 40 songs in 6 to 16 sentences for each setting ("Sentence Lengths Generation" uses 2-gram). The result in Table 4 shows
that musical information does influence the lyric generation, and the Sample method, which leads to more natural rhythm patterns, achieves the best performance on all metrics.
Model Comparison We also conduct the subjective comparison of UniLG with two baselines: MT5 and SongNet. We adapt 2-gram for "Sentence Length Generation" and Sample for Beat Generation in model comparison. We generate 120 songs in 6 to 16 sentences by each model with the same Masked Lyrics. The results in Table 5 show that our UniLG outperforms the baselines, which further
validates the effectiveness of our framework. Table 1 and 5 prove that our compound template enables a stronger structure modeling ability than SongNet. This may be because that the compound template provides discriminative representations for lyrics under the guidance of musical information. The MT5 achieves better PPL and Integrity in Table 1 but gets lower MOS results in Fluency in Table 5. This indicates that MT5 may pay too much attention to the fluency of the text but lacking the logical correlation between sentences.

heading: 5.3 Case Studies
text: We also show some cases in Appendix H to verify the ability of UniLG to handle different generation conditions. Although lyrics generated conditioned on the templates constructed by automatic method Template Construction are less satisfying (cases in Figure 5), the handcrafted template or template extracted by other resources achieves satisfying results as shown in Figure 6 and 7. These cases demonstrate that the template is human-
understandable and can be manipulated by users directly as in Section 3.1. The results in Figure 8 and 9 indicate that the template acts as a bridge between lyrics and the rhythmic sources (e.g., audio, music score, etc.), which enables our UniLG to generate lyrics conditioned on different signals.

heading: 6 Conclusion
text: In this paper, we propose UniLG, a unified structure-aware lyric generation framework. With our designed compound template to indicate the structure of lyrics with textual and musical information, which acts as a bridge between the rhythmic sources and lyrics, UniLG can handle different lyrics generation conditions. We also introduce a cycle-consistency loss to enhance the impact of musical information to improve performance. Extensive experiments demonstrate the effectiveness of our framework, achieving significant improvement in both objective and subjective evaluations. We will explore topic-driven lyrics generation in our future work.
Limitations
The limitations of our work include: 1) In our work, the structure of lyrics is the chorus and verse parts of songs, and it is learned in a data-driven manner, which highly relies on data quality. 2) The settings of the Lyric-to-Beat model will limit the effect of our model. For this work, we make an assumption that all songs are with 4/4 time signatures for the Lyric-to-Beat model. If the time signature is not 4/4, we need to re-train the Lyricto-Beat model.3) Better ”Beat Construction” can be investigated, such as using a language model to generate the beat sequence. We only explore the simple method and achieve satisfying results. 4) The model trained from scratch may not achieve satisfying results. And a GPU with at least 20G memory may be needed to use the pre-trained language model (MT5) to reproduce our work.
Ethics Statement
Under the review of the company’s legal team, the data collected for research is under legally correct copyright. The artifacts we used (e.g., MT5, other codes, etc) are consistent with their intended use and meet corresponding licenses. The mother tongue of all annotators is Chinese and the annotators are recruited by the human resources departments and the payment is adequate enough (The an-
notators receive an hourly wage of 80 RMB, about 12 USD) according to the laws and regulations of our country. Before the experiments, we report key information about the requirements for human annotators, including the evaluation criteria and the usage of their annotations. We have used the data under the consensus of the industry and research and the final information used for research does not include any protected category.

heading: Acknowledgements
text: This work was partially supported by the National Natural Science Foundation of China (No. 62072462) and the National Key R&D Program of China under Grant No.2020AAA0108600.

heading: A Details of Lyric-to-Beat Model
text: The Lyric-to-Beat model aims to extract the rhythm patterns B from lyrics L. Previous works usually generate a fixed rhythm pattern with a rule-based method and lots of handcraft design is needed which constrains the diversity of rhythm patterns (Fukayama et al., 2010; Monteith et al., 2012). Following the method in previous work and the released project11, we obtain the lyric-beat dataset for training (Ju et al., 2021; Ren et al., 2020; Xue et al., 2021a). With such lyric-beat data, we adopt the sequence-to-sequence (Seq2Seq) framework to train this model:
LL2B = − logP (B|L) = −Σni=1 logP (bi|b<i; l1, ..., ln), (6)
where L = (li)ni=1 and B = (li) n i=1 denote the lyric and beat sequence and n indicate the length of the sequence (Vaswani et al., 2017). The b<i stands for sequence (b1,b2,...,bi−1).
We conduct objective experiments with different settings on the lyric-beat dataset. The results are shown in Table 6 and we chose the MT5-based model for our framework. The Lyric-to-Beat model achieves average perplexity of 1.13 and accuracy of 92.18%. Due to high accuracy, the Lyric-toBeat model provides a more efficient method than previous work to obtain the paired lyrics and beats data (Ju et al., 2021).

heading: B Statics of Lyric-Template Dataset
text: Under the review of our legal team, the data for research is under legally correct copyright. And the statistics of this Lyric-Template dataset are as shown in Table 7.
11https://github.com/microsoft/muzic/tree/main/ telemelody

heading: C Details of Lyric-to-Structure Model
text: Inspired by (Zhang et al., 2019), we train a Lyricto-Structure model on Song8k to verify the performance of our framework. With Song8k, we split 50 songs each for validation and test sets and others for training the Lyric-to-Structure model. Similar to the Lyric-to-Beat model in Appendix A, the Lyricto-Structure model predicts structure information for given lyrics. With Song8k dataset (mentioned in Section 4.1), we construct the lyric-structure dataset to train the Lyric-to-Structure model. We adopt the Seq2Seq framework to train this model:
LL2S = − logP (S|L) = −Σmi=1 logP (si|s<i; l1, ..., ln), (7)
where L = (li)ni=1 and S = (si) m i=1 stands for lyrics sequence, li, si (si ∈ {Chorus,Verse}) stands for ith token in L and S, and n indicate the length of lyrics and m indicate the sentence numbers of lyrics (it’s also the length of S). The Lyric-to-Structure model achieves average perplexity of 1.78 and accuracy of 80.66%.

heading: D Module Details In Inference Procedure Algorithm
text: In this section, we will provide more details about the Lyric-to-Beat, MIDI-to-Beat, and Audio-toBeat modules. Note that the relationship between generated lyrics and inputs is only rhythm patterns and the semantic information should be introduced through masked lyrics (as Algorithm 1 in Section 3.4) in our framework. Notice that the UniLG only produces lyrics and these outputs can be produced by using the alignment of input signals and templates.
D.1 Lyric-to-Beat Module This module contains a Lyric-to-Beat model as described in Appendix A. The Lyric-to-Beat model extracts the beat sequence from the lyrics.
D.2 MIDI-to-Beat Module Similar to TeleMelody (Ju et al., 2021), we extract the melody track from MIDI files and calculate the
beats information of notes in melody track12.
D.3 Audio-to-Beat Module
We use Audio-to-MIDI tools13 to transcript the audio to MIDI file and use MIDI-to-Beat to extract the beats information from audio file.
D.4 Video-to-Beat Module*
Similar to recent video background music generation work, we can extract the visual beats14 from videos and map visual beats to our beats information (Di et al., 2021). But we haven’t done this part yet.

heading: E Model Configuration and Training Settings
text: Lyric-to-Beat Model In recent years, pre-trained auto-regressive language models have improved various downstream tasks’ performance significantly. We adopt MT5-based15 Lyric-to-Beat model in Seq2Seq framework (Xue et al., 2021b). The Lyric-to-Beat model consists of 8 encoder layers and 8 decoder layers and 6 attention heads for each layer. The hidden size of each layer is 512. The model is trained on a GeForce RTX 3090, and the batch size is 32 with 4096 tokens for each sample in the batch. Dropout with the rate of 0.1 is used for training and the activate function is gatedgelu (Shazeer, 2020). The model is finetuning with Adam optimizer with a learning rate of 0.0005 for 40,000 steps on the lyric-beat dataset (Kingma and Ba, 2015).
Lyric-to-Structure Model Inspired by BERTscore, we train a standard Seq2Seq Transformer to evaluate the performance of the structural modeling (Zhang et al., 2019). The Lyric-to-Structure model consists of 4 encoder layers and 4 decoder layers, and 4 attention heads for each layer. The hidden size of each layer is 256. The model is trained on a GeForce RTX 3090, and the batch size is 32 with 4096 tokens for each sample in the batch. Dropout with the rate of 0.2 is used for training. The model is trained with Adam optimizer with a learning rate of 0.0005 for
12https://github.com/microsoft/muzic/blob/main/ telemelody/training/template2melody/gen.py
13https://github.com/SJTMusicTeam/ Audio-to-midi
14https://colab.research.google.com/github/ wzk1015/video-bgm-generation/blob/main/CMT.ipynb
15https://huggingface.co/google/mt5-small
40,000 steps on the Song8k dataset (Kingma and Ba, 2015).
Model Comparison Similar to the Lyric-to-Beat module, we use the MT5-small from hugging face as initialization for following models (Wolf et al., 2020).
MT5 We fine-tune the MT5-small version with masked lyrics and lyrics data with Adam optimizer with a learning rate of 0.0001 and 8,000 warmup steps for 5 epochs on the lyric-template dataset. We use Masked Lyric and Lyric of the compound template as the input for the encoder and decoder of MT5, respectively, under the standard Seq2Seq framework. The MT5 doesn’t incorporate any musical and texture information.
SongNet We rewrite SongNet in the MT5 framework. Based on MT5, we tune the model with Adam optimizer with a learning rate of 0.0001 and 8,000 warmup steps for 5 epochs on the lyrictemplate dataset. SongNet constructs its input with masked lyrics, intro-position, and segments of the compound template. We use Segment, Introposition, Masked Lyric and Lyric of the compound template as the input for the encoder and decoder of SongNet, respectively, under the standard Seq2Seq framework. The SongNet doesn’t incorporate any musical information.
UniLG The parameters are the same as MT5. The Template-to-Lyric model is trained with The hyper-parameter of the CCL α (in Section 3.3) is determined by performance on the validation set and is 0.03. And the Lyric-to-Beat model for CCL is an MT5-based model in Table 6. The UniLG train with Adam optimizer with a learning rate of 0.0001 and 8,000 warmup steps for 5 epochs on the lyric-template dataset.

heading: F Definition of Objective Metrics
text: Integrity The Integrity is the metric to evaluate the sentence integrity (Li et al., 2020). Integrity calculates the average probability of the separation token given previous tokens. The formulation of Integrity is:
Integrity = 1 |Y |Σy∈Y 2 − logP (y|y||y1,...,y|y|−1), (8)
where Y is one piece of song, and y is one sentence of Y . |y| denotes the length of sentence y and |Y | denotes the number of sentences in Y .
F1 scores For given two sequences A = (ai)ni=1 and A′ = (a′i) m i=1, we define the formulation of Format F1 score is that:
F1(A,A′) = 2 ∗ Σmin(n,m)i=1 [ai == a′i]
n+m , (9)
where [*] denote 1 for * is true or 0 otherwise. There are several F1 scores in our experiments, including Format F1, Beat F1, and Structure F1 scores.
Format F1 score We use P = (pi)ni=1 stands for n positions of separation tokens in template of input data. And P ′ = (p′i) m i=1 stands for m positions of separation tokens in corresponding generated results. The formulation of Format F1 score is: Format F1= F1(P, P ′).
Beat F1 score Similarly, we use B = (bi)ni=1 stands for beat sequence with n tokens of input data. With the help of the Lyric-to-Beat model (details in Appendix A), we predicted beat sequence from generated lyrics with m tokens and annotate predicted beat sequence as B′ = {b′1, b′2, ..., b′m}. The formulation of Beat F1 score is: Beat F1= F1(B,B′).
Structure F1 score Similar to Format F1 score, we use S = {s1, s2, ..., sn} stands for structure information sequence with n tokens of annotated 8,000 songs. We use the Lyric-to-Structure model (details in Appendix C) to predict structure information from generated lyrics. We use S′ = {s′1, s′2, ..., s′m} to stand for predicted structure information. The formulation of the Structure F1 score (Struc. F1) is Struc. F1= F1(S, S′).

heading: G Models’ Decoding Settings in Experiments
text: For the Lyric-to-Beat and Lyric-to-Structure modules, we use top-k sampling and the k in top-k is 2, the beams are 1, and the temperature is 0.5.
For the Template-to-Lyric module, we use a sample-based beam search strategy 16 in subjective experiments with temperature 2.0, k of top-k 48, beams is 12, repetition penalty is 1.5, and score time decay is 0.98 and adopts top-k sampling decoding strategy in objective experiments for efficiency with temperature 1.5, k of top-k 32, beams is 1, and repetition penalty is 1.1.
16https://github.com/huggingface/transformers/ blob/c4d4e8bdbd25d9463d41de6398940329c89b7fb6/ src/transformers/generation_utils.py

heading: H Case Studies for Different Conditional Signals
text: We show some cases for different inputs with rhythm resources. In the following figures (Figure 5-9), we use dotted boxes to address the corresponding or chorus parts. It’s obvious that the automatic template construction is less satisfying because the template may not contain chorus and verse parts. The template which is handcrafted or extracted from other resources achieves satisfying results. It’s clear that our framework can capture the correspondence between sentences with musical and textual structural information. And our framework can generate lyrics with structure and handle multiple inputs with rhythm resources.
It is worth noticing that UniLG cannot rely on multiple modalities at the same time. The essence is still to generate lyrics given beats, ignoring the additional information of multimodal inputs. And our new experiments such that the different input modalities of the same song are given as input to the framework show consistency between beats and lyrics as same as Figures 5, 6, 7, 8, and 9. The only differences of results in experiments indicate that the Audio-to-Beat suffers the performance of Audio-to-MIDI tools. To avoid redundant content, we don’t include these results.

heading: I Examples and Instructions For Other Languages
text: In Figure 1 and 10, the rhythm patterns shown are the start beat of the corresponding word and it’s flexible for any time signature and any-to-any correspondence between notes and words. To simplify the description, we only state our method with a 4/4 time signature, for it’s widely used in songwriting. In Figure 1 and 10, the same rhythm pattern may repeat several times when meeting the chorus parts of the song through the melody are not exactly the same. These figures also illustrate the concept of chorus parts of lyrics, similar sentences with correspondence.
It is obvious that the basic elements of Chinese words (Chinese characters) and English words (words) have different attributes in phonemes. The correspondence between notes and Chinese characters is one-to-many. However, the correspondence between notes and English words is many-to-many. Even though, the UniLG needs little adjustments except for generation lyrics on the music score and audio. For many-to-many correspondence lan-
guages, the procedure of extraction beats from MIDI should add a post-procession, that randomly skips 0 - 2 notes for each note and memorizes this information as the alignments for final outputs.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section: Limitation
3 A2. Did you discuss any potential risks of your work? Section: Limitation
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section: Abstract, Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section: Method, Experimental Settings, Experiments Results
3 B1. Did you cite the creators of artifacts you used? Section: Related Work, Method, Experimental Settings, Experiments Results
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section: Experimental Settings, Ethics Statement, Appendix B Statics of Lyric-Template Dataset
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section: Experimental Settings, Experiments Results, Appendix A Details of Lyric-to-Beat Model, E Model Configuration and Training Settings, G Models’ Decoding Settings in Experiments
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section: Appendix B Statics of Lyric-Template Dataset
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section: Experimental Settings, Appendix E Model Configuration and Training Settings
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section: Experimental Settings, Appendix A Details of Lyric-to-Beat Model, B Statics of LyricTemplate Dataset, C Details of Lyric-to-Structure Model
C 3 Did you run computational experiments? Section: Experiments Results, Appendix A Details of Lyric-to-Beat Model, C Details of Lyric-toStructure Model The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section: Appendix A Details of Lyric-to-Beat Model, C Details of Lyric-to-Structure Model, E Model Configuration and Training Settings
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section: Appendix E Model Configuration and Training Settings, G Models’ Decoding Settings in Experiments
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section: Experiments Results, Appendix A Details of Lyric-to-Beat Model, B Statics of Lyric-Template Dataset, C Details of Lyric-to-Structure Model,
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section: Appendix A Details of Lyric-to-Beat Model, C Details of Lyric-to-Structure Model
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section: Experiments Results
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. We report key information about the requirements for human annotators, and we report this in Ethics Statement.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section: Ethics Statement
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section: Ethics Statement
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Section: Ethics Statement
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Section: Ethics Statement

heading: Future Work
text: We will explore topic-driven lyrics generation in our future work. Limitations
The limitations of our work include: 1) In our work, the structure of lyrics is the chorus and verse parts of songs, and it is learned in a data-driven manner, which highly relies on data quality. 2) The settings of the Lyric-to-Beat model will limit the effect of our model. For this work, we make an assumption that all songs are with 4/4 time signatures for the Lyric-to-Beat model. If the time signature is not 4/4, we need to re-train the Lyricto-Beat model. 3) Better ”Beat Construction” can be investigated, such as using a language model to generate the beat sequence. We only explore the simple method and achieve satisfying results. 4) The model trained from scratch may not achieve satisfying results. And a GPU with at least 20G memory may be needed to use the pre-trained language model (MT5) to reproduce our work